title: Pytorch- why is “accumulating” the default mode of .gradient?

body: Why didn't the authors just make it overwrite the gradient? Is there any specific reason for keeping it accumulated?


title: Can GPU supports multiple jobs without delay?

body:
So I am running PyTorch deep learning job using GPU
but the job is pretty light.

My GPU has 8 GB but the job only uses 2 GB. Also GPU-Util is close to 0%.

|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 1080    Off  | 00000000:01:00.0  On |                  N/A |
|  0%   36C    P2    45W / 210W |   1155MiB /  8116MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

based on GPU-Util and memory I might be able to fit in another 3 jobs.

However, I am not sure if that will affect the overall runtime.

If I run multiple jobs on same GPU does that affects the overall runtime?

I think tried once and I think there was delay.


title: Can tensorboard display an interactive plot or 3D plot

body:
I have to visualize the interactive 3D plot on tensorboard. Can the tensorboard visualize this or is there any way to display this on tensorboard.
Thank you.


title: RuntimeError: Error(s) in loading state_dict for Actor - torch.load()

body:
I have created a custom environment in open ai gym  and i am facing error while loading the weights Could some one help me to resolve the issue . I am training a TD3 network in a custom environment and i have trained successfully but while inferencing i am facing this issue

class Actor(nn.Module):

  def __init__(self, state_dim, action_dim, max_action):
    super(Actor, self).__init__()
    self.layer_1 = nn.Linear(state_dim, 400)
    self.layer_2 = nn.Linear(400, 300)
    self.layer_3 = nn.Linear(300, action_dim)
    self.max_action = max_action

  def forward(self, x):
    x = F.relu(self.layer_1(x))
    x = F.relu(self.layer_2(x))
    x = self.max_action * torch.tanh(self.layer_3(x))
    return x

class Critic(nn.Module):

  def __init__(self, state_dim, action_dim):
    super(Critic, self).__init__()
    # Defining the first Critic neural network
    self.layer_1 = nn.Linear(state_dim + action_dim, 400)
    self.layer_2 = nn.Linear(400, 300)
    self.layer_3 = nn.Linear(300, 1)
    # Defining the second Critic neural network
    self.layer_4 = nn.Linear(state_dim + action_dim, 400)
    self.layer_5 = nn.Linear(400, 300)
    self.layer_6 = nn.Linear(300, 1)

  def forward(self, x, u):
    xu = torch.cat([x, u], 1)
    # Forward-Propagation on the first Critic Neural Network
    x1 = F.relu(self.layer_1(xu))
    x1 = F.relu(self.layer_2(x1))
    x1 = self.layer_3(x1)
    # Forward-Propagation on the second Critic Neural Network
    x2 = F.relu(self.layer_4(xu))
    x2 = F.relu(self.layer_5(x2))
    x2 = self.layer_6(x2)
    return x1, x2

  def Q1(self, x, u):
    xu = torch.cat([x, u], 1)
    x1 = F.relu(self.layer_1(xu))
    x1 = F.relu(self.layer_2(x1))
    x1 = self.layer_3(x1)
    return x1

# Selecting the device (CPU or GPU)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# Building the whole Training Process into a class

class TD3(object):

  def __init__(self, state_dim, action_dim, max_action):
    self.actor = Actor(state_dim, action_dim, max_action).to(device)
    self.actor_target = Actor(state_dim, action_dim, max_action).to(device)
    self.actor_target.load_state_dict(self.actor.state_dict())
    self.actor_optimizer = torch.optim.Adam(self.actor.parameters())
    self.critic = Critic(state_dim, action_dim).to(device)
    self.critic_target = Critic(state_dim, action_dim).to(device)
    self.critic_target.load_state_dict(self.critic.state_dict())
    self.critic_optimizer = torch.optim.Adam(self.critic.parameters())
    self.max_action = max_action

  def select_action(self, state):
    state = torch.Tensor(state.reshape(1, -1)).to(device)
    return self.actor(state).cpu().data.numpy().flatten()

  def train(self, replay_buffer, iterations, batch_size=100, discount=0.99, tau=0.005, policy_noise=0.2, noise_clip=0.5, policy_freq=2):

    for it in range(iterations):

      # Step 4: We sample a batch of transitions (s, s’, a, r) from the memory
      batch_states, batch_next_states, batch_actions, batch_rewards, batch_dones = replay_buffer.sample(batch_size)
      state = torch.Tensor(batch_states).to(device)
      next_state = torch.Tensor(batch_next_states).to(device)
      action = torch.Tensor(batch_actions).to(device)
      reward = torch.Tensor(batch_rewards).to(device)
      done = torch.Tensor(batch_dones).to(device)

      # Step 5: From the next state s’, the Actor target plays the next action a’
      next_action = self.actor_target(next_state)

      # Step 6: We add Gaussian noise to this next action a’ and we clamp it in a range of values supported by the environment
      noise = torch.Tensor(batch_actions).data.normal_(0, policy_noise).to(device)
      noise = noise.clamp(-noise_clip, noise_clip)
      next_action = (next_action + noise).clamp(-self.max_action, self.max_action)

      # Step 7: The two Critic targets take each the couple (s’, a’) as input and return two Q-values Qt1(s’,a’) and Qt2(s’,a’) as outputs
      target_Q1, target_Q2 = self.critic_target(next_state, next_action)

      # Step 8: We keep the minimum of these two Q-values: min(Qt1, Qt2)
      target_Q = torch.min(target_Q1, target_Q2)

      # Step 9: We get the final target of the two Critic models, which is: Qt = r + γ * min(Qt1, Qt2), where γ is the discount factor
      target_Q = reward + ((1 - done) * discount * target_Q).detach()

      # Step 10: The two Critic models take each the couple (s, a) as input and return two Q-values Q1(s,a) and Q2(s,a) as outputs
      current_Q1, current_Q2 = self.critic(state, action)

      # Step 11: We compute the loss coming from the two Critic models: Critic Loss = MSE_Loss(Q1(s,a), Qt) + MSE_Loss(Q2(s,a), Qt)
      critic_loss = F.mse_loss(current_Q1, target_Q) + F.mse_loss(current_Q2, target_Q)

      # Step 12: We backpropagate this Critic loss and update the parameters of the two Critic models with a SGD optimizer
      self.critic_optimizer.zero_grad()
      critic_loss.backward()
      self.critic_optimizer.step()

      # Step 13: Once every two iterations, we update our Actor model by performing gradient ascent on the output of the first Critic model
      if it % policy_freq == 0:
        actor_loss = -self.critic.Q1(state, self.actor(state)).mean()
        self.actor_optimizer.zero_grad()
        actor_loss.backward()
        self.actor_optimizer.step()

        # Step 14: Still once every two iterations, we update the weights of the Actor target by polyak averaging
        for param, target_param in zip(self.critic.parameters(), self.critic_target.parameters()):
          target_param.data.copy_(tau * param.data + (1 - tau) * target_param.data)

        # Step 15: Still once every two iterations, we update the weights of the Critic target by polyak averaging
        for param, target_param in zip(self.actor.parameters(), self.actor_target.parameters()):
          target_param.data.copy_(tau * param.data + (1 - tau) * target_param.data)

  # Making a save method to save a trained model
  def save(self, filename, directory):
    torch.save(self.actor.state_dict(), '%s/%s_actor.pth' % (directory, filename))
    torch.save(self.critic.state_dict(), '%s/%s_critic.pth' % (directory, filename))

  # Making a load method to load a pre-trained model
  def load(self, filename, directory):
    self.actor.load_state_dict(torch.load('%s/%s_actor.pth' % (directory, filename)))
    self.critic.load_state_dict(torch.load('%s/%s_critic.pth' % (directory, filename)))

def evaluate_policy(policy, eval_episodes=10):
  avg_reward = 0.
  for _ in range(eval_episodes):
    obs = env.reset()
    done = False
    while not done:
      action = policy.select_action(np.array(obs))
      obs, reward, done, _ = env.step(action)
      avg_reward += reward
  avg_reward /= eval_episodes
  print ("---------------------------------------")
  print ("Average Reward over the Evaluation Step: %f" % (avg_reward))
  print ("---------------------------------------")
  return avg_reward

env_name = "Pygame-v0"
seed = 0

file_name = "%s_%s_%s" % ("TD3", env_name, str(seed))
print ("---------------------------------------")
print ("Settings: %s" % (file_name))
print ("---------------------------------------")

eval_episodes = 10
save_env_vid = True
env = gym.make(env_name)
max_episode_steps = env._max_episode_steps
if save_env_vid:
  env = wrappers.Monitor(env, monitor_dir, force = True)
  env.reset()
env.seed(seed)
torch.manual_seed(seed)
np.random.seed(seed)
state_dim = env.observation_space.shape[0]
action_dim = env.action_space.shape[0]
max_action = float(env.action_space.high[0])
policy = TD3(state_dim, action_dim, max_action)
#policy.load(file_name, './pytorch_models/')
policy.load(file_name,"/content/gdrive/My Drive/reinforce/gym_game/pytorch_models")
_ = evaluate_policy(policy, eval_episodes=eval_episodes)

Traceback:
I am facing a runtime error while loading the state_dict for actor model .I searched google but couldnt find similar issues .

RuntimeError: Error(s) in loading state_dict for Actor:
    Missing key(s) in state_dict: "layer_1.weight", "layer_1.bias", "layer_2.weight", "layer_2.bias", "layer_3.weight", "layer_3.bias".
    Unexpected key(s) in state_dict: "encoder.0.weight", "encoder.0.bias", "encoder.2.weight", "encoder.2.bias", "encoder.2.running_mean", "encoder.2.running_var", "encoder.2.num_batches_tracked", "encoder.3.weight", "encoder.3.bias", "encoder.5.weight", "encoder.5.bias", "encoder.5.running_mean", "encoder.5.running_var", "encoder.5.num_batches_tracked", "encoder.6.weight", "encoder.6.bias", "encoder.8.weight", "encoder.8.bias", "encoder.8.running_mean", "encoder.8.running_var", "encoder.8.num_batches_tracked", "encoder.10.weight", "encoder.10.bias", "encoder.12.weight", "encoder.12.bias", "encoder.12.running_mean", "encoder.12.running_var", "encoder.12.num_batches_tracked", "encoder.13.weight", "encoder.13.bias", "encoder.15.weight", "encoder.15.bias", "encoder.15.running_mean", "encoder.15.running_var", "encoder.15.num_batches_tracked", "encoder.16.weight", "encoder.16.bias", "linear.0.weight", "linear.0.bias", "linear.2.weight", "linear.2.bias".


title: pytorch:Error in using cuda (ValueError: Expected a cuda device with a specified index or an integer, but got: )

body:
Error in using cuda.
Here is the Code:
import robust_loss_pytorch.general

adaptive = robust_loss_pytorch.adaptive.AdaptiveLossFunction(
    num_dims = 4, float_dtype=torch.cuda.FloatTensor, device="cuda")

And here is the document of robust_loss_pytorch.adaptive.AdaptiveLossFunction:
Args:
  num_dims: The number of dimensions of the input to come.
  float_dtype: The floating point precision of the inputs to come.
  device: The device to run on (cpu, cuda, etc).

Got the error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-414-22ef6dd72b45> in <module>()
      2
      3 adaptive = robust_loss_pytorch.adaptive.AdaptiveLossFunction(
----> 4     num_dims = 4, float_dtype=torch.cuda.FloatTensor, device="cuda")

~/anaconda3/envs/torch36/lib/python3.6/site-packages/robust_loss_pytorch/adaptive.py in __init__(self, num_dims, float_dtype, device, alpha_lo, alpha_hi, alpha_init, scale_lo, scale_init)
    130        (isinstance(device, str) and 'cuda' in device) or\
    131        (isinstance(device, torch.device) and device.type == 'cuda'):
--> 132         torch.cuda.set_device(self.device)
    133
    134     self.distribution = distribution.Distribution()

~/anaconda3/envs/torch36/lib/python3.6/site-packages/torch/cuda/__init__.py in set_device(device)
    241             if this argument is negative.
    242     """
--> 243     device = _get_device_index(device)
    244     if device >= 0:
    245         torch._C._cuda_setDevice(device)

~/anaconda3/envs/torch36/lib/python3.6/site-packages/torch/cuda/_utils.py in _get_device_index(device, optional)
     32         else:
     33             raise ValueError('Expected a cuda device with a specified index '
---> 34                              'or an integer, but got: '.format(device))
     35     return device_idx

ValueError: Expected a cuda device with a specified index or an integer, but got:

When I try code adaptive = robust_loss_pytorch.adaptive.AdaptiveLossFunction(num_dims = 4, float_dtype=torch.cuda.FloatTensor, device="cuda:0")
Got error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-415-c7998f8405f9> in <module>()
      2
      3 adaptive = robust_loss_pytorch.adaptive.AdaptiveLossFunction(
----> 4     num_dims = 4, float_dtype=torch.cuda.FloatTensor, device="cuda:0")

~/anaconda3/envs/torch36/lib/python3.6/site-packages/robust_loss_pytorch/adaptive.py in __init__(self, num_dims, float_dtype, device, alpha_lo, alpha_hi, alpha_init, scale_lo, scale_init)
    154               latent_alpha_init.clone().detach().to(
    155                   dtype=self.float_dtype,
--> 156                   device=self.device)[np.newaxis, np.newaxis].repeat(
    157                       1, self.num_dims),
    158               requires_grad=True))

TypeError: to() received an invalid combination of arguments - got (device=str, dtype=torch.tensortype, ), but expected one of:
 * (torch.device device, torch.dtype dtype, bool non_blocking, bool copy, *, torch.memory_format memory_format)
 * (torch.dtype dtype, bool non_blocking, bool copy, *, torch.memory_format memory_format)
 * (Tensor tensor, bool non_blocking, bool copy, *, torch.memory_format memory_format)

Or trying :
adaptive = robust_loss_pytorch.adaptive.AdaptiveLossFunction(
    num_dims = 4, float_dtype=torch.cuda.FloatTensor, device=torch.device("cuda"))

Got the same error as :
ValueError: Expected a cuda device with a specified index or an integer, but got:

Would you please provide me the correct usage of cuda?
If you need detailed information,please let me know!


title: How to get the second moment of the gradient

body:
In the OpenAI Five paper it is mentioned that the "Gradients are additionally clipped per parameter to be within between ±5√v where v is the running estimate of the second moment of the (unclipped) gradient.". This is something I would like to implement in my project, but I am not sure how to do it neihter in theory nor in practice.
From wikipedia I found out that the "The second central moment is the variance. The positive square root of the variance is the standard deviation [...]". My best guess regarding the "running estimate" is that it is the Exponential Moving Average. The gradients of a network can be accessed as this comment suggests.
From these I would assume that √v is the Exponential Running Average of the standard dev. of the gradients and could be calculated via:
estimate = alpha * torch.std(list(param.grad for param in model.parameters())) + (1-alpha) * estimate
Is my theory correct? Is there a better way to do it?
Thanks in advance.
Edit: fixed gradient gathering after Mr. For Example"s answer.


title: How to use different optimizers for each model layer Pytorch?

body: I have a two layer network built in pytorch and two two different optimizers. I would like to use one optimizer on the first layer and the other optimizers on the second layer. Is this possible?


title: pytorch int32 to int64 conversion

body:
I'm trying to convert a simple image mask to int64
image = np.array([[1, 2], [3, 4]], dtype='int32')

transform = Compose([
    torch.from_numpy,
    ConvertImageDtype(torch.int64)
])

However, transform(image) yields
tensor([[ 4294967296,  8589934592],
        [12884901888, 17179869184]])

Is there something wrong, or am I fundamentally misunderstanding something about how the conversion should work?


title: How to slice Torch images as numpy images

body:
I am working on a problem in which I have the coordinates to slice the image like

  X cordinate, Y coordinate, Height, width of the region to crop

So if if I have torch image obtained using

img = Variable(img.cuda())

how can we slice this image to get that specific area of image [y:y+height, x:x+width] .
Thanks


title: Google colab : RuntimeError: Expected object of backend CPU but got backend CUDA for argument #4 'mat1'

body:
I training my model on MNIST dataset, use Google Colab for GPU purpose
device is cuda still getting error, I have tried other solution it does not work as well as code working fine in local pc, Is anything colab required different?
I have done previously training on the aws and there was no issue with the code

epoch = 22
steps = 0
print_every_step = 5

total_train_loss, total_test_loss = [], []

for e in range(epoch):

  train_loss = 0
  test_loss = 0
  accuracy = 0

  for images, labels in train_loader:

    # clear the gradients of all optimized variables

    optimizer.zero_grad()

    steps += 1

    images, labels = images.to(device), labels.to(device)

    # forward pass: compute predicted outputs by passing inputs to the model

    log_ps = model(images)

    loss = criterion(log_ps, labels)

    loss.backward()

    optimizer.step()

    train_loss += loss.item()

  if(steps % print_every_step == 0 ):

    model.eval()

    with torch.no_grad():

      for images, labels in test_loader:

        images, labels = images.to(device), labels.to(device)

        log_ps = model(images)

        loss = criterion(log_ps, labels)

        test_loss += loss.item()

      #calculate accuracy
        ps = torch.exp(log_ps)
        top_p, top_class = ps.topk(1, dim=1)
        equals = top_class == labels.view(*top_class.shape)
        accuracy += torch.mean(equals.type(torch.FloatTensor)).item()

    print(f"Epoch {epoch+1}/{epochs}.. "
          f"Train loss: {train_loss/print_every_step:.3f}.. "
          f"Test loss: {test_loss/len(test_loader):.3f}.. "
          f"Test accuracy: {accuracy/len(testl_oader):.3f}")

    model.train()

    total_train_loss.append(train_loss/print_every_step)
    total_test_loss.append(test_loss/len(testloader))


title: Using annoy with Torchtext for nearest neighbor search

body:
I'm using Torchtext for some NLP tasks, specifically using the built-in embeddings.

I want to be able to do a inverse vector search: Generate a noisy vector, find the vector that is closest to it, then get back the word that is "closest" to the noisy vector.

From the torchtext docs, here's how to attach embeddings to a built-in dataset:

from torchtext.vocab import GloVe
from torchtext import data

embedding = GloVe(name='6B', dim=100)

# Set up fields
TEXT = data.Field(lower=True, include_lengths=True, batch_first=True)
LABEL = data.Field(sequential=False, is_target=True)

# make splits for data
train, test = datasets.IMDB.splits(TEXT, LABEL)

# build the vocabulary
TEXT.build_vocab(train, vectors=embedding, max_size=100000)
LABEL.build_vocab(train)

# Get an example vector
embedding.get_vecs_by_tokens("germany")

Then we can build the annoy index:

from annoy import AnnoyIndex

num_trees = 50

ann_index = AnnoyIndex(embedding_dims, 'angular')

# Iterate through each vector in the embedding and add it to the index
for vector_num, vector in enumerate(TEXT.vocab.vectors):
    ann_index.add_item(vector_num, vector) # Here's the catch: will vector_num correspond to torchtext.vocab.Vocab.itos?

ann_index.build(num_trees)

Then say I want to retrieve a word using a noisy vector:

# Get an existing vector
original_vec = embedding.get_vecs_by_tokens("germany")
# Add some noise to it
noise = generate_noise_vector(ndims=100)
noisy_vector = original_vec + noise
# Get the vector closest to the noisy vector
closest_item_idx = ann_index.get_nns_by_vector(noisy_vector, 1)[0]
# Get word from noisy item
noisy_word = TEXT.vocab.itos[closest_item_idx]

My question comes in for the last two lines above: The ann_index was built using enumerate over the embedding object, which is a Torch tensor.

The [vocab][2] object has its own itos list that given an index returns a word.

My question is this: Can I be certain that the order in which words appear in the itos list, is the same as the order in TEXT.vocab.vectors? How can I map one index to the other?


title: Replacing torch.cdist() function to eliminate GPU out-of-memory runtime error

body:
How shall I modify this new_cdist() function to eliminate GPU out-of-memory runtime error ?

More specifically replacing torch.cdist()with something lighter in memory footprint.

Note: The following code snippet is related to a new type of back-propagation equation and adaptive learning rate.

See here for more information on how the existing code works

def new_cdist(p, eta):
    class cdist(torch.autograd.Function):
        @staticmethod
        def forward(ctx, W, X):
            ctx.save_for_backward(W, X)
            out = -torch.cdist(W, X, p)
            return out

        @staticmethod
        def backward(ctx, grad_output):
            W, X = ctx.saved_tensors
            grad_W = grad_X = None
            if ctx.needs_input_grad[0]:
                _temp1 = torch.unsqueeze(X, 2).expand(X.shape[0], X.shape[1], W.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(W.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                grad_W = torch.matmul(grad_output, _temp)
                # print('before norm: ', torch.norm(grad_W))
                grad_W = eta * np.sqrt(grad_W.numel()) / torch.norm(grad_W) * grad_W
                print('after norm: ', torch.norm(grad_W))
            if ctx.needs_input_grad[1]:
                _temp1 = torch.unsqueeze(W, 2).expand(W.shape[0], W.shape[1], X.shape[0]).permute(1, 0, 2)
                _temp2 = torch.unsqueeze(X.transpose(0, 1), 1)
                _temp = torch.cdist(_temp1, _temp2, p).squeeze().transpose(0, 1)
                _temp = torch.nn.functional.hardtanh(_temp, min_val=-1., max_val=1.)
                grad_X = torch.matmul(grad_output.transpose(0, 1), _temp)
            return grad_W, grad_X
    return cdist().apply


title: Summing over specific indices PyTorch (similar to scatter_add)

body:
I have some matrix where rows belong to some label, unordered. I want to sum all rows for each label.
Here is how it can be done with a loop:
labels = torch.tensor([0, 1, 0])
x = torch.tensor([[1, 2, 3],[4, 5, 6],[7, 8, 9]])
torch.stack([torch.sum(x[labels == i], dim=0) for i in torch.unique(labels)])

desired output:
tensor([[ 8, 10, 12],
        [ 4,  5,  6]])

EDIT: Just to make it clear, I have the labels tensor, I know which labels repeat, I am interested in computing the final line without the use of a loop. I was thinking scatter_add_ or gather might help.


title: Determinant of a complex matrix in PyTorch

body:
Is there a way to calculate the determinant of a complex matrix in PyTroch?
torch.det is not implemented for 'ComplexFloat'


title: How to select indices according to another tensor in pytorch

body:
The task seems to be simple, but I cannot figure out how to do it.
So what I have are two tensors:

an indices tensor indices with shape (2, 5, 2), where the last dimensions corresponds to indices in x and y dimension
a "value tensor" value with shape (2, 5, 2, 16, 16), where I want the last two dimensions to be selected with x and y indices

To be more concrete, the indices are between 0 and 15 and I want to get an output:
out = value[:, :, :, x_indices, y_indices]

The shape of the output should therefore be of (2, 5, 2). Can anybody help me here? Thanks a lot!
Edit:
I tried the suggestion with gather, but unfortunately it does not seem to work (I changed the dimensions, but it doesn't matter):
First I generate a coordinate grid:
y_t = torch.linspace(-1., 1., 16, device='cpu').reshape(16, 1).repeat(1, 16).unsqueeze(-1)
x_t = torch.linspace(-1., 1., 16, device='cpu').reshape(1, 16).repeat(16, 1).unsqueeze(-1)
grid = torch.cat((y_t, x_t), dim=-1).permute(2, 0, 1).unsqueeze(0)
grid = grid.unsqueeze(1).repeat(1, 3, 1, 1, 1)

In the next step, I am creating some indices. In this case, I always take index 1:
indices = torch.ones([1, 3, 2], dtype=torch.int64)

Next, I am using your method:
indices = indices.unsqueeze(-1).unsqueeze(-1)
new_coords = torch.gather(grid, -1, indices).squeeze(-1).squeeze(-1)

Finally, I manually select index 1 for x and y coordinate:
new_coords_manual = grid[:, :, :, 1, 1]

This outputs the following new coordinates:
new_coords
tensor([[[-1.0000, -0.8667],
         [-1.0000, -0.8667],
         [-1.0000, -0.8667]]])

new_coords_manual
tensor([[[-0.8667, -0.8667],
         [-0.8667, -0.8667],
         [-0.8667, -0.8667]]])

As you can see, it only works for one dimension. Do you have an idea how to fix that?


title: Why PyTorch fails to instantiate this neural network from Torch Hub?

body:
I am on Ubuntu, running PyTorch 1.7.1 and torchvision 0.8.2:
print(torch.__version__)
print(torchvision.__version__)

>> 1.7.1
>> 0.8.2

When I attempt to instantiate resnet18 with:
model_resnet18 = torch.hub.load('pytorch/vision', 'resnet18', pretrained=True)

it fails with:
Using cache found in /home/stark/.cache/torch/hub/pytorch_vision_master
Traceback (most recent call last):
  File "/home/stark/Work/test/binary_classifier.py", line 253, in <module>
    build_and_train()
  File "/home/stark/Work/test/binary_classifier.py", line 174, in build_and_train
    model_resnet18 = torch.hub.load('pytorch/vision', 'resnet18', pretrained=True)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/hub.py", line 370, in load
    model = _load_local(repo_or_dir, model, *args, **kwargs)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/hub.py", line 396, in _load_local
    hub_module = import_module(MODULE_HUBCONF, hubconf_path)
  File "/home/stark/anaconda3/envs/torch-env/lib/python3.8/site-packages/torch/hub.py", line 71, in import_module
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/stark/.cache/torch/hub/pytorch_vision_master/hubconf.py", line 14, in <module>
    from torchvision.models.mobilenetv2 import mobilenet_v2
ModuleNotFoundError: No module named 'torchvision.models.mobilenetv2'

I am on Python 3.8 and already on the latest PyTorch version, so what else can I do to fix this?
Any pointers are greatly appreciated!


title: Training step not executing in pytorch lightning

body:
I am working to finetune a t5 model to summarize Amazon reviews. I am following this tutorial here: https://towardsdatascience.com/fine-tuning-a-t5-transformer-for-any-summarization-task-82334c64c81
I noticed that the training_step in my code is never being executed as the training loss remains "NaN" throughout the epoch. However, the validation_step is computed fine.
I already confirmed that there are no empty strings in the data and have tried multiple batch sizes.
This is the error
RuntimeError                              Traceback (most recent call last)
<ipython-input-53-45d4afebefac> in <module>()
----> 1 trainer.fit(model)

8 frames
<ipython-input-46-00fddffa2209> in training_epoch_end(self, outputs)
    134         print("OUTPUTS")
    135         print(outputs)
--> 136         avg_train_loss = torch.stack([x["loss"] for x in outputs]).mean()
    137         tensorboard_logs = {"avg_train_loss": avg_train_loss}
    138         return {"avg_train_loss": avg_train_loss, "log": tensorboard_logs, 'progress_bar': tensorboard_logs}

RuntimeError: stack expects a non-empty TensorList

I found that the training_step function is never being executed by adding print statements inside the training_step function.
Below is my code for the T5FineTuner class (sorry I can't be any more concise):
class T5FineTuner(pl.LightningModule):
    def __init__(self, hparams):
        super(T5FineTuner, self).__init__()
        self.hparams = hparams
        self.model = T5ForConditionalGeneration.from_pretrained(hparams.model_name_or_path)
        self.tokenizer = T5Tokenizer.from_pretrained(hparams.tokenizer_name_or_path)
        self.rouge_metric = load_metric('rouge')

        if self.hparams.freeze_embeds:
            self.freeze_embeds()
        if self.hparams.freeze_encoder:
            self.freeze_params(self.model.get_encoder())
            assert_all_frozen(self.model.get_encoder())

        n_observations_per_split = {
            "train": self.hparams.n_train,
            "validation": self.hparams.n_val,
            "test": self.hparams.n_test,
        }
        self.n_obs = {k: v if v >= 0 else None for k, v in n_observations_per_split.items()}

    def freeze_params(self, model):
        for par in model.parameters():
            par.requires_grad = False

    def freeze_embeds(self):
        """Freeze token embeddings and positional embeddings for bart, just token embeddings for t5."""
        try:
            self.freeze_params(self.model.model.shared)
            for d in [self.model.model.encoder, self.model.model.decoder]:
                freeze_params(d.embed_positions)
                freeze_params(d.embed_tokens)
        except AttributeError:
            self.freeze_params(self.model.shared)
            for d in [self.model.encoder, self.model.decoder]:
                self.freeze_params(d.embed_tokens)

    def lmap(self, f, x):
        """list(map(f, x))"""
        return list(map(f, x))

    def is_logger(self):
        return True

    def parse_score(self, result):
        return {k: round(v.mid.fmeasure * 100, 4) for k, v in result.items()}

    def forward(
      self, input_ids, attention_mask=None, decoder_input_ids=None, decoder_attention_mask=None, labels=None
  ):
        return self.model(
            input_ids,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            decoder_attention_mask=decoder_attention_mask,
            labels=labels,
    )

    def _step(self, batch):
        labels = batch["target_ids"]
        labels[labels[:, :] == self.tokenizer.pad_token_id] = -100
        # print(labels)
        outputs = self(
            input_ids=batch["source_ids"],
            attention_mask=batch["source_mask"],
            labels=labels,
            decoder_attention_mask=batch['target_mask']
        )
        # print(outputs)

        loss = outputs[0]
        return loss

    def ids_to_clean_text(self, generated_ids):
        gen_text = self.tokenizer.batch_decode(
            generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True
        )
        return self.lmap(str.strip, gen_text)

    def _generative_step(self, batch) :

        t0 = time.time()

        generated_ids = self.model.generate(
            batch["source_ids"],
            attention_mask=batch["source_mask"],
            use_cache=True,
            decoder_attention_mask=batch['target_mask'],
            max_length=150,
            num_beams=2,
            repetition_penalty=2.5,
            length_penalty=1.0,
            early_stopping=False,
        )
        preds = self.ids_to_clean_text(generated_ids)
        target = self.ids_to_clean_text(batch["target_ids"])

        gen_time = (time.time() - t0) / batch["source_ids"].shape[0]

        loss = self._step(batch)
        # print("LOSS _generative_step")
        # print(loss)
        base_metrics = {'val_loss': loss}
#         rouge: Dict = self.calc_generative_metrics(preds, target)
        summ_len = np.mean(self.lmap(len, generated_ids))
        base_metrics.update(gen_time=gen_time, gen_len=summ_len, preds=preds, target=target)
        self.rouge_metric.add_batch(preds, target)

        # rouge_results = self.rouge_metric.compute()
        # rouge_dict = self.parse_score(rouge_results)
        # base_metrics.update(rouge1=rouge_dict['rouge1'], rougeL=rouge_dict['rougeL'])

        return base_metrics

    def training_step(self, batch, batch_idx):
        print("training_step")
        print(batch)
        loss = self._step(batch)

        tensorboard_logs = {"train_loss": loss}
        print("LOSS")
        print(loss)
        return {"loss": loss, "log": tensorboard_logs}

    def training_epoch_end(self, outputs):
        print("OUTPUTS")
        print(outputs)
        avg_train_loss = torch.stack([x["loss"] for x in outputs]).mean()
        tensorboard_logs = {"avg_train_loss": avg_train_loss}
        return {"avg_train_loss": avg_train_loss, "log": tensorboard_logs, 'progress_bar': tensorboard_logs}

    def validation_step(self, batch, batch_idx):
        print("validation_step")
        return self._generative_step(batch)

    def validation_epoch_end(self, outputs):

        avg_loss = torch.stack([x["val_loss"] for x in outputs]).mean()
        tensorboard_logs = {"val_loss": avg_loss}

        rouge_results = self.rouge_metric.compute()
        rouge_dict = self.parse_score(rouge_results)

        tensorboard_logs.update(rouge1=rouge_dict['rouge1'], rougeL=rouge_dict['rougeL'])

        ## Clear out the lists for next epoch
        self.target_gen= []
        self.prediction_gen=[]
        return {"avg_val_loss": avg_loss,
                "rouge1" : rouge_results['rouge1'],
                "rougeL" : rouge_results['rougeL'],
                "log": tensorboard_logs, 'progress_bar': tensorboard_logs}

    def configure_optimizers(self):
        "Prepare optimizer and schedule (linear warmup and decay)"

        model = self.model
        no_decay = ["bias", "LayerNorm.weight"]
        optimizer_grouped_parameters = [
            {
                "params": [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)],
                "weight_decay": self.hparams.weight_decay,
            },
            {
                "params": [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)],
                "weight_decay": 0.0,
            },
        ]
        optimizer = AdamW(optimizer_grouped_parameters, lr=self.hparams.learning_rate, eps=self.hparams.adam_epsilon)
        self.opt = optimizer
        return [optimizer]

    def optimizer_step(self, epoch, batch_idx, optimizer, optimizer_idx, second_order_closure=None, using_native_amp=False, optimizer_closure=None, on_tpu=None, using_lbfgs=None):
        # if self.trainer.use_tpu:
        #     xm.optimizer_step(optimizer)
        # else:
        optimizer.step()
        optimizer.zero_grad()
        self.lr_scheduler.step()

    def get_tqdm_dict(self):
        tqdm_dict = {"loss": "{:.3f}".format(self.trainer.avg_loss), "lr": self.lr_scheduler.get_last_lr()[-1]}

        return tqdm_dict

    def train_dataloader(self):
        print("train_dataloader")
        n_samples = self.n_obs['train']
        print(n_samples)
        dataloader = DataLoader(train_dataset, batch_size=self.hparams.train_batch_size, num_workers=4)
        print(len(dataloader.dataset))
        print(self.hparams.train_batch_size * max(1, self.hparams.n_gpu))
        print(self.hparams.gradient_accumulation_steps)
        print(float(self.hparams.num_train_epochs))
        t_total = (
            (len(dataloader.dataset) // (self.hparams.train_batch_size * max(1, self.hparams.n_gpu)))
            # // self.hparams.gradient_accumulation_steps
            * float(self.hparams.num_train_epochs)
        )
        print(t_total)
        scheduler = get_linear_schedule_with_warmup(
            self.opt, num_warmup_steps=self.hparams.warmup_steps, num_training_steps=t_total
        )
        self.lr_scheduler = scheduler
        return dataloader

    def val_dataloader(self):
        n_samples = self.n_obs['validation']
        # validation_dataset = get_dataset(tokenizer=self.tokenizer, type_path="validation", num_samples=n_samples, args=self.hparams)

        return DataLoader(validation_dataset, batch_size=self.hparams.eval_batch_size, num_workers=4)

    def test_dataloader(self):
        n_samples = self.n_obs['test']
        # test_dataset = get_dataset(tokenizer=self.tokenizer, type_path="test", num_samples=n_samples, args=self.hparams)

        return DataLoader(test_dataset, batch_size=self.hparams.test_batch_size, num_workers=4)

Below are my parameters:
args_dict = dict(
    output_dir="", # path to save the checkpoints
    model_name_or_path='t5-small',
    tokenizer_name_or_path='t5-small',
    max_input_length=512,
    max_output_length=150,
    freeze_encoder=False,
    freeze_embeds=False,
    learning_rate=3e-4,
    weight_decay=0.0,
    adam_epsilon=1e-8,
    warmup_steps=0,
    train_batch_size=20,
    eval_batch_size=20,
    num_train_epochs=2,
    gradient_accumulation_steps=8,
    n_gpu=1,
    resume_from_checkpoint=None,
    val_check_interval = 0.05,
    n_val=1000,
    n_train=-1,
    n_test=-1,
    early_stop_callback=False,
    fp_16=False, # if you want to enable 16-bit training then install apex and set this to true
    opt_level='O1', # you can find out more on optimisation levels here https://nvidia.github.io/apex/amp.html#opt-levels-and-properties
    max_grad_norm=1.0, # if you enable 16-bit training then set this to a sensible value, 0.5 is a good default
    seed=42,
)


title: Understanding Actor-Advantage Critic

body:
I am attempting to implement an A2C neural network in C++ from this tutorial. I could only find Python tutorials and this Python tutorial made the most sense to me.

I have already implemented a regular back propagation neural network, however, I want to upgrade this to use reinforcement learning. I have created a C++ version of the Pytorch Tensor and most other things necessary to complete this. I'm having issues going about implementing their Variable class. I've read over the docs and understand the concept of automatic differentiation using the chain rule, but I'm unsure on how to convert Tensors into Variables and what all needs to happen behind the scenes. I might even already have all the necessary gradient math with my original backprop neural network implementation.

It has been very difficult trying to decipher the Python code (I don't have a background in Python) especially since I had to guess the types of complex variables (Variable vs Tensor vs Vector, etc.). For example, I don't understand the line total_loss.backward() (section In [4]: of the tutorial). Is total_loss not a double (or float)? My mean() function of Tensor returns a single value representing the mean of the entire Tensor. total_loss is just simple operations done on floats or doubles. But, then it calls backward() on it. I found three backwards but unsure which one since none make sense in the context. torch.autograd.backward, torch.autograd.Function.backward, and torch.autograd.Variable.backward. None of which make sense because total_loss is a double, right?

Another missing step is the "forward" and "backward" pass implementations. The tutorial says Pytorch does the backward pass for me, but as I am in C++, I need to do that implementation myself and I'm not too sure the purpose of the different passes. Are these the feeding forward and back propagation of neural networks?

I'm also confused about the optim and how it is doing stochastic optimization. I've read the docs on that too but I am still not certain on how that converts to C++. I know it's a first-order gradient-based optimization of stochastic objective functions based on adaptive estimates of lower-order moments.

From what I have researched over the past few weeks, I need to create a forward accumulation computational graph of tensors as nodes with the chain rule performed on the tensor data in order to fix the dependent variable to compute the derivatives with respect to each sub-expression on the nodes recursively. Then, use automatic differentiation of the arbitrary scalar valued functions (which are the neural network activation functions?) to propagate through the tree and result in the respective gradients. The chain rule differentiates the graph and each node would hold the w.r.t value and the function that created it.

I think I am fine for the concepts for the majority of the implementation, just how to go about programming it is a struggle as I don't fully understand how the inter-workings of Python is working.

I've also researched how the implementation of Variable looks in the Pytorch source and read through examples in the examples section, however I am still struggling to understand implementation.

In conclusion (so it doesn't get flagged as too broad), what details do I need to know about Pytorch Variable in order to make an implementation in C++, assuming the concept of automatic differentiation is understood? Does it just need to hold a double for gradient and function pointer for activation? Also, how do the model.variables() in Python work and get passed around? This is so I can understand how to do the Adam algorithm implementation.

If you need to see code I have already, I can post some, but chose not to currently as the code I already have written doesn't necessarily directly pertain the the questions I have.


title: Implementing Group Lasso on PyTorch weight matrices

body:
I am trying to implement Group Lasso on weight matrices of a neural network in PyTorch.

I have written the code to implement Group Lasso but am unsure if this is correct, confirmation or correction of my code will be very helpful.

def gl_norm(model, gl_lambda, num_blk):
    gl_reg = torch.tensor(0., dtype=torch.float32).cuda()
    for key in model:
        for param in model[key].parameters():
            dim = param.size()
            if dim.__len__() > 1 and not model[key].skip_regularization:
                div1 = list(torch.chunk(param,int(num_blk),1))
                all_blks = []
                for div2 in div1:
                    temp = list(torch.chunk(div2,int(num_blk),0))
                    for blk in temp:
                        all_blks.append(blk)
                for l2_param in all_blks:
                    gl_reg += torch.norm(l2_param, 2)
    return gl_reg * float(gl_lambda)

I expect the torch.chunk function to break up the weight matrix into small blocks which then go through L2 norm for the block and L1 norm between all the blocks.


All.

I want to run Libtorch (Pytorch C++ Frontend) in OpenFrameworks.
LibTorch Site: https://pytorch.org/tutorials/advanced/cpp_frontend.html

If LibTorch runs in OpenFrameworks, it will be very powerful Deep Solution IDE (deep learning + GUI Production for OSX, Win10, & Linux).
But I can’t include <torch/torch.h> in XCode.

At CMake method, CMakeLists.txt is like this:

cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(dcgan)

find_package(Torch REQUIRED)

add_executable(dcgan dcgan.cpp)
target_link_libraries(dcgan “${TORCH_LIBRARIES}”)
set_property(TARGET dcgan PROPERTY CXX_STANDARD 11)

and installed libtorch for mac from: https://download.pytorch.org/libtorch/cpu/libtorch-macos-latest.zip

How can I include libtorch header in XCode ??

Thanks in advance.


title: Implementing Batch for Image Segmentation

body:
I wrote a Python 3.5 script for doing street segmentation. Since I'm new in Image Segementation, I did not use predefined dataloaders from pytorch, instead I wrote them by my self (for better understanding). Until now I only use a batch size of 1. Now I want to generalize this for arbitrary batch sizes.

This is a snippet of my Dataloader:

def augment_data(batch_size):

    # [...] defining some paths and data transformation (including ToTensor() function)

    # The images are named by numbers (Frame numbers), this allows me to find the correct label image for a given input image.
    all_input_image_paths = {int(elem.split('\\')[-1].split('.')[0]) : elem for idx, elem in enumerate(glob.glob(input_dir + "*"))}
    all_label_image_paths = {int(elem.split('\\')[-1].split('.')[0]) : elem for idx, elem in enumerate(glob.glob(label_dir + "*"))}

    dataloader = {"train":[], "val":[]}
    all_samples = []
    img_counter = 0

    for key, value in all_input_image_paths.items():
        input_img = Image.open(all_input_image_paths[key])
        label_img = Image.open(all_label_image_paths[key])

        # Here I use my own augmentation function which crops the input and label on the same position and do other things.
        # We get a list of new augmented data
        augmented_images = generate_augmented_images(input_img, label_img)
        for elem in augmented_images:
            input_as_tensor = data_transforms['norm'](elem[0])
            label_as_tensor = data_transforms['val'](elem[1])

            input_as_tensor.unsqueeze_(0)
            label_as_tensor.unsqueeze_(0)

            is_training_data = random.uniform(0.0, 1.0)
            if is_training_data <= 0.7:
                dataloader["train"].append([input_as_tensor, label_as_tensor])
            else:
                dataloader["val"].append([input_as_tensor, label_as_tensor])
            img_counter += 1

    shuffle(dataloader["train"])
    shuffle(dataloader["val"])
    dataloader_batched =  {"train":[], "val":[]}

    # Here I group my data to a given batch size
    for elem in dataloader["train"]:
        batch = []
        for i in range(batch_size):
            batch.append(elem)
        dataloader_batched["train"].append(batch)

    for elem in dataloader["val"]:
        batch = []
        for i in range(batch_size):
            batch.append(elem)
        dataloader_batched["val"].append(batch)

    return dataloader_batched

This is a snippet of my training method with batch size 1:

    while epoch <= num_epochs:
        # Each epoch has a training and validation phase
        for phase in ['train', 'val']:
            if phase == 'train':
                scheduler.step(3)
                model.train()  # Set model to training mode
            else:
                model.eval()  # Set model to evaluate mode

            running_loss = 0.0

            counter = 0
            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                counter += 1
                max_num = len(dataloaders[phase])

                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)

            epoch_loss = running_loss / dataset_sizes[phase]

If I execute this, I get of course the error:

for inputs, labels in dataloaders[phase]:
ValueError: not enough values to unpack (expected 2, got 1)

I understand why, because now I have a list of images and not only an input and label image as before. So guessed I need a second for loop which iterates over these batches. So I tried this:

            # Iterate over data.
            for elem in dataloaders[phase]:
                for inputs, labels in elem:
                    counter += 1
                    max_num = len(dataloaders[phase])

                    inputs = inputs.to(device)
                    labels = labels.to(device)

                    # zero the parameter gradients
                    optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    # _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()

But for me it looks like the optimization step (back-prop) is only applied on the last image of the batch. Is that true? And if so, how can I fix this? I guess if I indent the with-Block, then I get again a batch size 1 optimization.

Thanks in advance


title: Find indices of elements equal to zero in a PyTorch Tensor

body:
My question is nearly identical to this one, with the notable difference of being in PyTorch. I would prefer not to use the Numpy solution as this would involve moving data back to the CPU. I see that, as with Numpy, PyTorch has a nonzero function, however its where function (the solution in the Numpy thread I linked) has behavior different from Numpy's.
The behavior I want is an is_zero() function as follows:

>>> arr.nonzero()
tensor([[0, 1],
        [1, 0]])
>>> arr.is_zero()
tensor([[0, 0],
        [1, 1]])


title: PyTorch Two Losses of Two Variables

body:
This is a question about PyTorch autograd.grad and the backward function specifically.

I have two tensors a, b which are optimized over (i.e. require gradients).

I define loss1, loss2 = f(a,b), g(a,b). Although these are two separate functions f and g, for computational efficiency reasons, I have to compute both of them together as fg(a,b) which returns a tuple (loss1, loss2).

I need to use opt_a and opt_b (optimizers) to step a and b with the following gradients:

a.grad should equal d (loss1)/ d (a)

b.grad should equal d (loss2)/ d (b)

How can I achieve these gradients? I know I can run autograd.grad(loss1, a) and autograd.grad(loss2, b) to get the true gradients and set them to *.grad manually, but I want to use the backward method on the loss1 and loss2.

I want to use the backward method because it is concise code when in my case, a and b are actually the lists of parameters of two neural networks (I don't want to be manually setting param.grad = ... for param in model1.parameters()).

Is there a clean way to do this with .backward()?

My Attempt

I have tried multiple ordered variants of the following (but none of them work because the gradients add for one variable at least):

loss1, loss2 = fg(...)

opt1.zero_grad()
loss1.backward(retain_graph=True)
opt1.step()

opt2.zero_grad()
loss2.backward()
opt2.step()

Different orders of this result in either an accumulation of gradients (d (loss1+loss2)/ d (a)) or they result in one optimizer stepping the value of a, and then I can't run b.backward() because of an in-place operation change.

Some testing code I used:


title: CPU version of "torch._C._nn.nll_loss" function

body:
Is there a function for torch._C._nn.nll_loss that takes in a CPU input? I don't have enough GPU memory to run my function so I'm trying to run everything on CPU.
This is my specific error (look at the anaconda files)

Traceback (most recent call last):
  File "plot_parametric_pytorch.py", line 395, in <module>
    val_result = validate(val_loader, model, criterion, 0)
  File "plot_parametric_pytorch.py", line 228, in validate
    training=False, optimizer=None)
  File "plot_parametric_pytorch.py", line 169, in forward
    loss = criterion(output, target_var)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/modules/loss.py", line 932, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/functional.py", line 2317, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_nll_loss_forward


title: TensorboardX input problem about add_scalar()

body:
When I use the tensorboardX to plot my data loss, it show me that:
AssertionError                            Traceback (most recent call last)
<ipython-input-76-73419a51fcc9> in <module>
----> 1 writer.add_scalar('resnet34_loss', loss)

F:\Program Files\Python\lib\site-packages\tensorboardX\writer.py in add_scalar(self, tag, scalar_value, global_step, walltime)
    403             scalar_value = workspace.FetchBlob(scalar_value)
    404         self._get_file_writer().add_summary(
--> 405             scalar(tag, scalar_value), global_step, walltime)
    406
    407     def add_scalars(self, main_tag, tag_scalar_dict, global_step=None, walltime=None):

F:\Program Files\Python\lib\site-packages\tensorboardX\summary.py in scalar(name, scalar, collections)
    145     name = _clean_tag(name)
    146     scalar = make_np(scalar)
--> 147     assert(scalar.squeeze().ndim == 0), 'scalar should be 0D'
    148     scalar = float(scalar)
    149     return Summary(value=[Summary.Value(tag=name, simple_value=scalar)])

AssertionError: scalar should be 0D

I have turn the loss from float into np.array, and I have read the doc of tensorboardX, it tell me that add_scalar() function must input the scalar data and I do it, but it shows me a bug. Thanks for your help!


title: torch.cat along negative dimension

body:
In the following,
 x_6 = torch.cat((x_1, x_2_1, x_3_1, x_5_1), dim=-3)
 Sizes of tensors x_1, x_2_1, x_3_1, x_5_1 are
 torch.Size([1, 256, 7, 7])
 torch.Size([1, 256, 7, 7])
 torch.Size([1, 256, 7, 7])
 torch.Size([1, 256, 7, 7]) respectively.

 The size of x_6 turns out to be torch.Size([1, 1024, 7, 7])

I couldn't understand & visualise this concatenation along a negative dimension(-3 in this case).
What exactly is happening here?
How does the same go if dim = 3?
Is there any constraint on dim for a given set of tensors?


title: Pip does not recognize torchaudio libary

body:
When i try the command:
pip install torchaudio

i get this error:
 ERROR: Could not find a version that satisfies the requirement torchaudio
 ERROR: No matching distribution found for torchaudio

I use windows 10


title: Pytorch get stuck at .to(‘cuda’) or .cuda()

body:
I use a docker environment to do some pytorch training work, but frequently get stuck at .to(‘cuda’) or .cuda() calls. Not know why.
Some system behaviors I notice: The training process has a cpu usage of 100%, and almost 97% are sys time as shown by top 1. I used strace to debug the training process, and also used the /proc file system, found the process keep getting stuck on a poll syscall to a pipe, but I don’t know where the pipe is used in pytorch.


title: Adding residual connections to simple CNN

body:
I implemented the following CNN with residual connections for classification of 10 classes on CIFAR10:

class ConvolutionalNetwork(nn.Module):

  def __init__(self, filters=[32, 32, 32]):

    super(ConvolutionalNetwork, self).__init__()

    self.conv = []
    self.filters = filters
    in_channels = 3

    for out_channels in self.filters:
      self.conv.append(nn.Conv2d(in_channels, out_channels, 5, padding=2))
      in_channels = out_channels

    self.conv = nn.ModuleList(self.conv)
    self.fc = nn.Linear(self.filters[-1] * 32 * 32, 10)

  def forward(self, X):
    X = X.contiguous()

    for i, conv in enumerate(self.conv):
      res = X
      X = F.relu(conv(X))
      X += res

    X = X.reshape(-1,self.filters[-1]*self.size*self.size)

    X = self.fc(X)
    return F.log_softmax(X, dim=1)

Basically: Conv2d(5x5) --> ReLU --> Conv2d(5x5) --> ReLU --> Conv2d(5x5) --> ReLU --> Fully Connected Layer --> SoftMax
The performence with the residual connections is worse then without. I wonder:

maybe I didn't use it right (should I use nn.Identity() or some other pytorch class?)
what are the best practices for number of filters (I tried [32,32,32] and [64,64,64]) and for skips (how many hops? i did one but maybe this is the problem?)
How can I change the dimention of the identity to fit varying dimentions of the convolutional network (i.e. [16, 32, 64] filters)


title: Do I calculate one loss per mini batch or one loss per entry in mini batch in deep reinforcement learning?

body:
New to neural network and Pytorch.

I have 300 replay memories in each mini batch. I've seen people calculate one loss for the 300 replay memories, but it doesn't really make sense to me. The 300 replay memories are from very different game states and why would it make sense to combine the 300 differences between predictions and targets into one value? Do the gradients get separated into 300 branches, each corresponding to one entry in the mini batch, when the model back propagates?

For example, still using mini batches that have 300 replay memories in each of them. My policy network outputs a probability distribution over 10 actions, or a 300 x 10 tensor, and my target probability distribution is of the same shape. I want to find the cross entropy loss between my predictions and targets. I'm wondering whether I should find 300 cross entropy losses between 300 prediction-target-pairs of size [10] tensors, or find 1 cross entropy loss between 1 prediction-target-pair of size [3000] tensors, if that makes sense. Also how should I implement this in Pytorch? What shape of loss should I expect to get?


title: matplotlib - segmentation fault (core dumped) while trying to plot a grid

body:
I'm trying to plot a grid but exists with the error message -
(feature_vis.py:27730): Gdk-WARNING **: 03:23:47.745: Native Windows wider or taller than 32767 pixels are not supported
Segmentation fault (core dumped)

model_weights = './Mutual-Channel-Loss-master/model_info_best.pth'
model_features = './Mutual-Channel-Loss-master/features_best_new.pth'
weights = torch.load(model_weights, map_location = torch.device('cpu'))
features = torch.load(model_features, map_location = torch.device('cpu'))
activations = features['features']
feature_map = activations[0][0][3]
# feature_maps = np.load(path, allow_pickle = True)
# test = feature_maps[0][0][0]
fig = plt.figure(figsize = (300, 500))
grid = ImageGrid(fig, 111,  # similar to subplot(111)
                 nrows_ncols=(20, 30),  # creates 2x2 grid of axes
                 axes_pad=0.1,  # pad between axes in inch.
                 )
for ax, im in zip(grid, feature_map):
    # Iterating over the grid returns the Axes.
    im = im.to('cpu').numpy()
    ax.imshow(im, cmap = 'gray')

plt.savefig('test.png', ppi = 300)

How should I fix / Debug this?
I am using python 3.6.9 and on ubuntu 18.04


title: Exact coordinates of bounding box for Mask R-CNN?

body:
Given a 5x5 mask (black = 0, red = 1):

I want to get the coordinates for the bounding box.
Intuitively I would think of the image as a grid, and pixels being drawn from top left corner, giving the bounding box coords of (2, 2, 4, 4):

Some image renderers like plt have me believed that pixels are drawn from the middle, resulting either in box coordinates of (2, 2, 3, 3) or possibly (1.5, 1.5, 3.5, 3.5):

Which one of these would be correct? Are any of them?
The regions are to be used in the training of a Mask R-CNN model but I guess that the way of calculating bounding boxes are similar for other models too.


title: Slow multiprocessing when using torch

body:
I have a function func that takes on average 9 sec to run. But when I try to use multiprocessing to parallelize it (even using torch.multiprocessing) each inference takes on average 20 sec why is that ?
func is an inference function which takes in a patient_name and runs a torch model in inference on that patient's data.
device = torch.device(torch.device('cpu'))

def func(patient_name):

    data = np.load(my_dict[system_name]['data_path'])
    model_state = torch.load(my_dict[system_name]['model_state_path'],map_location='cpu')

    model = my_net(my_dict[system_name]['HPs'])
    model = model.to(device)

    model.load_state_dict(model_state)
    model.eval()

    result = model(torch.FloatTensor(data).to(device))
    return result

from torch.multiprocessing import pool

core_cnt = 10

pool = Pool(core_cnt)
out = pool.starmap(func, pool_args)


title: RPC failed with status = "Unavailable: Socket closed" Error when training FairSeq RoBERTa on Cloud TPU using PyTorch

body:
I followed the tutorials "Pre-training FairSeq RoBERTa on Cloud TPU using Pytorch" to setup a Preemptible (v2-8) TPU env and train my RoBERTa model. The PyTorch env is based on torch-xla-1.6 as instructed by the document. However, it does not output any training log as usual in GPU and it throws the RPC failure warning (see below - network endpoint is removed here) twice in 2-3 days (in 12 hours gap).
My training steps per epoch is 161,529. According to the document, v2-8 will take 80 hours for 5 epochs as i configured. However, My job seems hanging there.
Any advice please ?
 W    4566 tensorflow/core/distributed_runtime/rpc/grpc_remote_master.cc:160] RPC failed with status = "Unavailable: Socket closed" and grpc_error_string = "{"created":"@1599580717.037250202","description":"Error received from peer ipv4:<my_network_endpoint>:8470","file":"external/com_github_grpc_grpc/src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket closed","grpc_status":14}", maybe retrying the RPC


title: How can I force PyTorch NOT to use more than 1 threads

body:
I use PyTorch 1.6 on a single CPU with multiple cores and want pytorch not to generate more than 1 threads. By using the following setting, I manage to reduce the number of threads to 1 but as soon as I load and use a model for inference, the number of threads reaches 8.
os.environ[“OMP_NUM_THREADS”] = “1”
os.environ[“MKL_NUM_THREADS”] = “1”
torch.set_num_threads(1)
torch.set_num_interop_threads(1)
os.system(“top -H -b -n1 | grep project_name | wc -l”)# count the number of threads

How can I restrict the number of threads to 1 or 2 (no matter what the number of cores is)?


title: What memory does Transformer Decoder Only use?

body: I've been reading a lot about transformers and self attention and have seen both BERT and GPT-2 are a newer version that only use an encoder transformer (BERT) and decoder transformer (GPT-2). I've been trying to build a decoder only model for myself for next sequence prediction but am confused by one thing. I'm using PyTorch and have looked at thereSeq2Seq tutorial and then looked into the Transformer Decoder Block which is made up of Transformer Decoder Layers. My confusion comes from the memory these need to be passed as well. In the documentation they say memory is the last layer of the encoder block which makes sense for a Seq2Seq model but I'm wanting to make a decoder only model. So my question is what do you pass a decoder only model like GPT-2 for memory if you do not have an encoder?


title: Cannot perform reduction function min on tensor with no elements because the operation does not have an identity at THCTensorMathReduce.cu:64

body:
I am configuring a GitHub repo in which the author stated that you have to install pytorch=0.4 and python=3.7. Now, I have CUDA 11.0 and the Pytorch version is conflicting with CUDA. After installing the Pytorch it gives the below error. Any hint?
My Conda List
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                        main
_pytorch_select           0.1                       cpu_0
blas                      1.0                         mkl
ca-certificates           2021.1.19            h06a4308_1
certifi                   2020.12.5        py37h06a4308_0
cffi                      1.14.0           py37h2e261b9_0
cuda100                   1.0                           0    pytorch
cudatoolkit               9.2                           0
freetype                  2.10.4               h5ab3b9f_0
intel-openmp              2019.4                      243
jpeg                      9b                   h024ee3a_2
lcms2                     2.11                 h396b838_0
libedit                   3.1.20210216         h27cfd23_1
libffi                    3.2.1             hf484d3e_1007
libgcc-ng                 9.1.0                hdf63c60_0
libmklml                  2019.0.5                      0
libpng                    1.6.37               hbc83047_0
libstdcxx-ng              9.1.0                hdf63c60_0
libtiff                   4.2.0                h3942068_0
libwebp-base              1.2.0                h27cfd23_0
lz4-c                     1.9.3                h2531618_0
mkl                       2020.2                      256
mkl-service               2.3.0            py37he8ac12f_0
mkl_fft                   1.3.0            py37h54f3939_0
mkl_random                1.1.1            py37h0573a6f_0
ncurses                   6.2                  he6710b0_1
ninja                     1.10.2           py37hff7bd54_0
numpy                     1.20.1                   pypi_0    pypi
numpy-base                1.19.2           py37hfa32c7d_0
olefile                   0.46                     py37_0
openssl                   1.1.1j               h27cfd23_0
pillow                    8.1.2            py37he98fc37_0
pip                       21.0.1           py37h06a4308_0
pycparser                 2.20                       py_2
python                    3.7.1                h0371630_7
pytorch                   1.2.0           py3.7_cuda9.2.148_cudnn7.6.2_0    pytorch
readline                  7.0                  h7b6447c_5
scipy                     1.6.1                    pypi_0    pypi
setuptools                52.0.0           py37h06a4308_0
six                       1.15.0           py37h06a4308_0
sqlite                    3.33.0               h62c20be_0
tk                        8.6.10               hbc83047_0
torchaudio                0.8.0                    pypi_0    pypi
torchvision               0.4.0                 py37_cu92    pytorch
typing                    3.7.4.3          py37h06a4308_0
typing-extensions         3.7.4.3              hd3eb1b0_0
typing_extensions         3.7.4.3            pyh06a4308_0
wheel                     0.36.2             pyhd3eb1b0_0
xz                        5.2.5                h7b6447c_0
zlib                      1.2.11               h7b6447c_3
zstd                      1.4.5                h9ceee32_0

Error
Model size: 44.17957M
==> Epoch 1/800  lr:0.0003
Traceback (most recent call last):
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 369, in <module>
    main()
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 230, in main
    train(model, criterion_xent, criterion_htri, optimizer, trainloader, use_gpu)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/main_video_person_reid_hypergraphsage_part.py", line 279, in train
    htri_loss = criterion_htri(features, pids)
  File "/home/khawar/anaconda3/envs/hypergraph_reid/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/media/khawar/HDD_Khawar1/hypergraph_reid/losses.py", line 86, in forward
    dist_an.append(dist[i][mask[i] == 0].min())
RuntimeError: invalid argument 1: cannot perform reduction function min on tensor with no elements because the operation does not have an identity at /opt/conda/conda-bld/pytorch_1565287025495/work/aten/src/THC/generic/THCTensorMathReduce.cu:64


title: How is PyTorch's Class BCEWithLogitsLoss exactly implemented?

body:
According to the PyTorch documentation, the advantage of the class BCEWithLogitsLoss() is that one can use the

log-sum-exp trick for numerical stability.

If we use the class BCEWithLogitsLoss() with the parameter reduction set to None, they have a formula for that:

I now simplified the terms, and obtain after some lines of calculation:

I was curious to see whether this is the way how the Source code does it, but I couldn't find it.. The only code they have is this:
Code for BCEWithLogitsLoss


title: "module 'torchtext.data' has no attribute 'Field'"

body:
import torchtext

ENGLISH = torchtext.data.Field(tokenize=tokenizer_english, lower=True, init_token="<sos>", eos_token="<eos>")

Error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-12-2a3d11c77e7d> in <module>
----> 1 ENGLISH = torchtext.data.Field(tokenize=tokenizer_english, lower=True, init_token="<sos>", eos_token="<eos>")

AttributeError: module 'torchtext.data' has no attribute 'Field'

It won't import torchtext.data.Field for some reason even though that's whats in the docs


title: Pytorch custom/modified GAN loss equivalent

body:
I'm trying to implement a Pytorch version of Creative Adversarial Networks, a GAN with a modified/custom loss function.

Here are the formulae for the loss function. I'm using Pytorch's nn.CrossEntropyLoss for the discriminator's modified loss function, and it seems to be working, as its loss decreases over epochs, but I don't think nn.CrossEntropyLoss is suitable for the generator, as nn.CrossEntropyLoss seems to expect Long and not Float tensors, and the paper’s loss function, particularly the generator's loss, seems to me like it would require floats.

This is my current (initial) thinking for the generator's custom loss:

y_dim is the number of classes

disc_class_layer = FC layer that outputs a style/class given an input image

The for loop attempts to be the equivalent to:

(sigma k=1 up to k) ((1/K)log(Dc(ck|G(z)) + (1 − (1/K)log(1 − Dc(ck|G(z)).

class CanGLoss(nn.Module):
def __init__(self,y_dim,labels,disc_class_layer):
    super(CanGLoss,self).__init__()

def forward(self,inp):
    style_loss = 0
    for i in range(1,y_dim+1):
        style_loss += (1/i)*torch.log(disc_class_layer(inp)) + (1 - (1/i))*torch.log(1-disc_class_layer(inp))
    return style_loss*-1

Is this on the right track? I am new to custom loss functions and Pytorch and not sure this is the way to go.

Any help would be great!


title: How does a 2x2 deconv kernel with stride=2 work?

body:
For example, if the feature map is 8x8, than I use such a deconv and the feature map becomes 16x16, I'm confused that what the difference between:

deconv(kernel_size=2, stride=2, padding='valid')

and

deconv(kernel_size=3, stride=2, padding='same')

Since they will both make feature map 2 times larger, how do they work respectively?


title: Using `DataParallel` when network needs a shared (constant) `Tensor`

body:
I would like to use DataParallel to distribute my computations across multiple GPUs along the batch dimension. My network requires a Tensor (let's call it A) internally, which is constant and doens't change through the optimization. It seems that DataParallel does not automatically copy this Tensor to all the GPUs in question, and the network will thus complain that the chunk of the input data x that it sees resides on a different GPU than A.

Is there a way DataParallel can handle this situation automatically? Alternatively, is there a way to copy a Tensor to all GPUs? Or should I just keep one Tensor for each GPU and manually figure out which copy to use depending on where the chunk seen by forward resides?


title: Trouble opening audio files stored on S3 in SageMaker

body: I stored like 300 GB of audio data (mp3/wav mostly) on Amazon S3 and am trying to access it in a SageMaker notebook instance to do some data transformations. I'm trying to use either torchaudio or librosa to load a file as a waveform. torchaudio expects the file path as the input, librosa can either use a file path or file-like object. I tried using s3fs to get the url to the file but torchaudio doesn't recognize it as a file. And apparently SageMaker has problems installing librosa so I can't use that. What should I do?


title: Fast BERT Camembert, arguments located on different GPUs error

body:
I'm trying to implement Fast-Bert with 'camembert-base' model type.
I can easily create my databunch with BertLMDatabunch.From raw_corpus
then I create the learner.
I'm on a cloud env, with 3 GPUs, on ubuntu env, with 32 cores and RAM 130Mo.
When I'm trying to fit the model I always have this error message after this information
databunch_lm = BertLMDataBunch.from_raw_corpus(
                data_dir=DATA_PATH,
                text_list=all_texts,
                tokenizer='camembert-base',
                batch_size_per_gpu=16,
                max_seq_length=512,
                multi_gpu=True,
                model_type='camembert-base',
                logger=logger)
lm_learner = BertLMLearner.from_pretrained_model(
                        dataBunch=databunch_lm,
                        pretrained_path='camembert-base',
                        output_dir=MODEL_PATH,
                        metrics=[],
                        device=device_cuda,
                        logger=logger,
                        multi_gpu=True,
                        logging_steps=50,
                        fp16_opt_level="O2")

lm_learner.fit(epochs=30,
        lr=1e-4,
        validate=True,
        schedule_type="warmup_cosine",
        optimizer_type="adamw")

Selected optimization level O2:  FP16 training with FP32 batchnorm and FP32 master weights.

    Defaults for this optimization level are:
    enabled                : True
    opt_level              : O2
    cast_model_type        : torch.float16
    patch_torch_functions  : False
    keep_batchnorm_fp32    : True
    master_weights         : True
    loss_scale             : dynamic
    Processing user overrides (additional kwargs that are not None)...
    After processing overrides, optimization options are:
    enabled                : True
    opt_level              : O2
    cast_model_type        : torch.float16
    patch_torch_functions  : False
    keep_batchnorm_fp32    : True
    master_weights         : True
    loss_scale             : dynamic
    Warning:  multi_tensor_applier fused unscale kernel is unavailable, possibly because apex was installed without --cuda_ext --cpp_ext. Using Python fallback.  Original ImportError was: ModuleNotFoundError("No module named 'amp_C'")

    0.00% [0/30 00:00<00:00]
    0.00% [0/32 00:00<00:00]
        ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-15-b8d5301e0d4e> in <module>
          3             validate=True,
          4             schedule_type="warmup_cosine",
    ----> 5             optimizer_type="adamw")

    ~/miniconda3/lib/python3.7/site-packages/fast_bert/learner_lm.py in fit(self, epochs, lr, validate, schedule_type, optimizer_type)
        142                 self.model.train()
        143
    --> 144                 outputs = self.model(inputs, masked_lm_labels=labels)
        145                 loss = outputs[0]  # model outputs are always tuple in pytorch-transformers (see doc)
        146

    ~/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
        545             result = self._slow_forward(*input, **kwargs)
        546         else:
    --> 547             result = self.forward(*input, **kwargs)
        548         for hook in self._forward_hooks.values():
        549             hook_result = hook(self, input, result)

    ~/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
        150             return self.module(*inputs[0], **kwargs[0])
        151         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
    --> 152         outputs = self.parallel_apply(replicas, inputs, kwargs)
        153         return self.gather(outputs, self.output_device)
        154

    ~/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs)
        160
        161     def parallel_apply(self, replicas, inputs, kwargs):
    --> 162         return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
        163
        164     def gather(self, outputs, output_device):

    ~/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices)
         83         output = results[i]
         84         if isinstance(output, ExceptionWrapper):
    ---> 85             output.reraise()
         86         outputs.append(output)
         87     return outputs

    ~/miniconda3/lib/python3.7/site-packages/torch/_utils.py in reraise(self)
        367             # (https://bugs.python.org/issue2651), so we work around it.
        368             msg = KeyErrorMessage(msg)
    --> 369         raise self.exc_type(msg)

    RuntimeError: Caught RuntimeError in replica 1 on device 1.
    Original Traceback (most recent call last):
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
        output = module(*input, **kwargs)
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/apex-0.1-py3.7.egg/apex/amp/_initialize.py", line 197, in new_fwd
        **applier(kwargs, input_caster))
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 231, in forward
        inputs_embeds=inputs_embeds,
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 727, in forward
        input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 66, in forward
        input_ids, token_type_ids=token_type_ids, position_ids=position_ids, inputs_embeds=inputs_embeds
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 174, in forward
        inputs_embeds = self.word_embeddings(input_ids)
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
        self.norm_type, self.scale_grad_by_freq, self.sparse)
      File "/home/ubuntu/miniconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 1467, in embedding
        return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
    RuntimeError: arguments are located on different GPUs at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/generic/THCTensorIndex.cu:397


title: Index multidimensional torch tensor with array of variable length

body:
I have a list of indices and a tensor with shape:
shape = [batch_size, d_0, d_1, ..., d_k]
idx = [i_0, i_1, ..., i_k]

Is there a way to efficiently index the tensor on each dim d_0, ..., d_k with the indices i_0, ..., i_k? (k is available only at run time)
The result should be:
tensor[:, i_0, i_1, ..., i_k] #tensor.shape = [batch_size]

At the moment I'm creating a tuple of slices, one for each dimension:
idx = (slice(tensor.shape[0]),) + tuple(slice(i, i+1) for i in idx)
tensor[idx]

but I would prefer something like:
tensor[:, *idx]

Example:
a = torch.randint(0,10,[3,3,3,3])
indexes = torch.LongTensor([1,1,1])

I would like to index only the last len(indexes) dimensions like:
a[:, indexes[0], indexes[1], indexes[2]]

but in the general case where I don't know how long indexes is.

Note: this answer does not help since it indexes all the dimensions, and does not work for a proper subset!


title: Tracing Python Objects in torch.jit.tracing for save and load

body:
I am trying to send a pytorch model from one server to another. I chose to use torch.jit.tracing to save the model for transmission. Now, the problem is that, I want to send encrypted parameters, which is a list of int python objects (to save 100+ digits for each ciphertext). I tried to save it in a dict() object and trace it.
# Model
class Net(nn.Module):
    def __init__(self, input_size, output_size):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_size, 50)
        # nn.init.normal_(self.fc1.weight, mean=0, std=1)
        self.fc2 = nn.Linear(50, 10)
        # nn.init.normal_(self.fc2.weight, mean=0, std=1)
        self.fc3 = nn.Linear(10, output_size)
        nn.init.normal_(self.fc3.weight, mean=0, std=1)
        self.fc1_enc = None
        self.fc2_enc = None
        self.fc3_enc = None
        # a dict of python int objects to save encrypted big ints.
        self.encrypted_params = {"int_wei": [[74812937489217492138741294723198479123749321874,
                                              11111112222222222222888888888888888888888888888]]}
        self.exponent_encrypted_params = dict()

    def forward(self, x):
        x = F.elu(self.fc1(x))
        x = F.elu(self.fc2(x))
        return F.log_softmax(self.fc3(x), dim=-1)

And I trace the model by :
    model = Net(119, 2).to(device)
    traced_model = torch.jit.trace(model, torch.zeros(119, dtype=torch.float))

However, I failed to load the dict wit error: AttributeError: 'ScriptModule' object has no attribute 'encrypted_params'.
How can I serialize the model with big integer objects as parameters? Thanks!


title: File not found or corrupted while loading CUB-200-2011 data set

body:
import os
import pandas as pd
import numpy as np
from torchvision.datasets.folder import default_loader
from torchvision.datasets.utils import download_url
from torch.utils.data import Dataset

class Cub2011(Dataset):
    base_folder = 'CUB_200_2011/images'
    url = 'http://www.vision.caltech.edu/visipedia-data/CUB-200-2011/CUB_200_2011.tgz'
    filename = 'CUB_200_2011.tgz'
    tgz_md5 = '97eceeb196236b17998738112f37df78'

    def __init__(self, root, train=True, transform=None, loader=default_loader, download=True):
        self.root = os.path.expanduser(root)
        self.transform = transform
        self.loader = default_loader
        self.train = train

        if download:
            self._download()

    def _load_metadata(self):
        images = pd.read_csv(os.path.join(self.root, 'CUB_200_2011', 'images.txt'), sep=' ',
                             names=['img_id', 'filepath'])
        image_class_labels = pd.read_csv(os.path.join(self.root, 'CUB_200_2011', 'image_class_labels.txt'),
                                         sep=' ', names=['img_id', 'target'])
        train_test_split = pd.read_csv(os.path.join(self.root, 'CUB_200_2011', 'train_test_split.txt'),
                                       sep=' ', names=['img_id', 'is_training_img'])

        data = images.merge(image_class_labels, on='img_id')
        self.data = data.merge(train_test_split, on='img_id')

        if self.train:
            self.data = self.data[self.data.is_training_img == 1]
        else:
            self.data = self.data[self.data.is_training_img == 0]

    def _download(self):
        import tarfile
        download_url(self.url, self.root, self.filename, self.tgz_md5)
        with tarfile.open(os.path.join(self.root, self.filename), "r:gz") as tar:
            tar.extractall(path=self.root)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        sample = self.data.iloc[idx]
        path = os.path.join(self.root, self.base_folder, sample.filepath)
        target = sample.target - 1
        img = self.loader(path)

        if self.transform is not None:
            img = self.transform(img)

        return img, target

I tried loading CUB-200-2011 dataset by above code,
train_dataset = Cub2011('/content/sample_data')

It shows the following error,
Downloading http://www.vision.caltech.edu/visipedia-data/CUB-200-2011/CUB_200_2011.tgz to /content/sample_data/CUB_200_2011.tgz
49152/? [00:01<00:00, 420940.24it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-19c6ecf1c43a> in <module>()
----> 1 train_dataset = Cub2011('/content/sample_data')

2 frames
/usr/local/lib/python3.6/dist-packages/torchvision/datasets/utils.py in download_url(url, root, filename, md5)
     84         # check integrity of downloaded file
     85         if not check_integrity(fpath, md5):
---> 86             raise RuntimeError("File not found or corrupted.")
     87
     88

RuntimeError: File not found or corrupted.

What is the error in my code and how to correct it?


title: Pytorch: NN function approximator, 2 in 1 out

body:
[Please be aware of the Edit History below, as the major problem statement has changed.]

We are trying to implement a neural network in pytorch, that approximates a function f(x,y)=z. So there are two real numbers as input and one as ouput, we therefore want 2 nodes in the input layer and one in the output layer. We constructed a test set of 5050 samples and had pretty good results for that task in Keras with Tensorflow backend, with 3 hidden layers with a configuration of the nodes like: 2(in) - 4 - 16 - 4 - 1(out); and ReLU activation functions on all hidden layers, linear on in- and output.

Now in Pytorch we tried to implement a similar network but our loss function still literally explodes: It changes in the first few steps and converges then to some value around 10^7. In Keras we had an error around 10 percent. We already tried different network configurations without any improvement. Maybe someone could have a look on our code and suggest any change?

To explain: tr_data is a list, containing 5050 2*1 numpy arrays which are the inputs for the network. tr_labels is a list, containing 5050 numbers which are the outputs we want to learn. loadData() just load those two lists.

import torch.nn as nn
import torch.nn.functional as F

BATCH_SIZE     = 5050
DIM_IN         = 2
DIM_HIDDEN_1   = 4
DIM_HIDDEN_2   = 16
DIM_HIDDEN_3   = 4
DIM_OUT        = 1
LEARN_RATE     = 1e-4
EPOCH_NUM      = 500

class Net(nn.Module):
    def __init__(self):
        #super(Net, self).__init__()
        super().__init__()
        self.hidden1 = nn.Linear(DIM_IN, DIM_HIDDEN_1)
        self.hidden2 = nn.Linear(DIM_HIDDEN_1, DIM_HIDDEN_2)
        self.hidden3 = nn.Linear(DIM_HIDDEN_2, DIM_HIDDEN_3)
        self.out     = nn.Linear(DIM_HIDDEN_3, DIM_OUT)

    def forward(self, x):
        x = F.relu(self.hidden1(x))
        x = F.tanh(self.hidden2(x))
        x = F.tanh(self.hidden3(x))
        x = self.out(x)
        return x

model = Net()

loss_fn = nn.MSELoss(size_average=False)

optimizer = torch.optim.Adam(model.parameters(), lr=LEARN_RATE)

tr_data,tr_labels = loadData()

tr_data_torch    = torch.zeros(BATCH_SIZE, DIM_IN)
tr_labels_torch  = torch.zeros(BATCH_SIZE, DIM_OUT)

for i in range(BATCH_SIZE):
    tr_data_torch[i]   = torch.from_numpy(tr_data[i])
    tr_labels_torch[i] = tr_labels[i]

for t in range(EPOCH_NUM):

    labels_pred = model(tr_data_torch)
    loss = loss_fn(labels_pred, tr_labels_torch)
    #print(t, loss.item())

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

I have to say, those are our first steps in Pytorch, so please forgive me if there are some obvious, dumb mistakes. I appreciate any help or hint,
Thank you!

EDIT 1 ------------------------------------------------------------------

Following the comments and answers, we improved our code. The Loss function has now for the first time reasonable values, around 250. Our new class definition looks like:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        #super().__init__()
        self.hidden1 = nn.Sequential(nn.Linear(DIM_IN, DIM_HIDDEN_1), nn.ReLU())
        self.hidden2 = nn.Sequential(nn.Linear(DIM_HIDDEN_1, DIM_HIDDEN_2), nn.ReLU())
        self.hidden3 = nn.Sequential(nn.Linear(DIM_HIDDEN_2, DIM_HIDDEN_3), nn.ReLU())
        self.out = nn.Linear(DIM_HIDDEN_3, DIM_OUT)

    def forward(self, x):
        x = self.hidden1(x)
        x = self.hidden2(x)
        x = self.hidden3(x)
        x = self.out(x)
        return x

and the loss function:

loss_fn = nn.MSELoss(size_average=True, reduce=True)

As we stated before, we already had far more satisfying results in keras with tensorflow backend. The loss function was around 30, with a similar network configuration. I share the essential parts(!) of our keras code here:

model = Sequential()
model.add(Dense(4, activation="linear", input_shape=(2,)))
model.add(Dense(16, activation="relu"))
model.add(Dense(4, activation="relu"))
model.add(Dense(1, activation="linear" ))
model.summary()

model.compile ( loss="mean_squared_error", optimizer="adam", metrics=["mse"] )

history=model.fit ( np.array(tr_data), np.array(tr_labels), \
                    validation_data = ( np.array(val_data), np.array(val_labels) ),
                    batch_size=50, epochs=200, callbacks = [ cbk ] )

Thank your already for all the help! If anybody still has suggestions to improve the network, we would be happy about it. As somebody already asked for the data, we want to share a pickle file here:

https://mega.nz/#!RDYxSYLY!P4a9mEDtZ7A5Bl7ZRjRk8EzLXQt2gyURa3wN3NCWFPA

together with the code to access it:

import pickle
f=open("data.pcl","rb")
tr_data=pickle.load ( f )
tr_labels=pickle.load ( f )
val_data=pickle.load ( f )
val_labels=pickle.load ( f )
f.close()


title: LSTMCell parameters is not shown Pytorch

body:
I have the following code:

class myLSTM(nn.Module):
def __init__(self, input_size, output_size, hidden_size, num_layers):
    super(myLSTM, self).__init__()
    self.input_size = input_size + 1
    self.output_size = output_size
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.layers = []
    new_input_size = self.input_size
    for i in xrange(num_layers):
        self.layers.append(LSTMCell(new_input_size, hidden_size))
        new_input_size = hidden_size
    self.linear = nn.Linear(hidden_size, output_size)
    self.softmax = nn.Softmax()

def forwardLayers(self, input, hns, cns, layers):
    new_hns = []
    new_cns = []
    (hn, cn) = layers[0](input, (hns[0], cns[0]))
    new_hns.append(hn)
    new_cns.append(cn)
    for i in xrange(1, len(layers)):
        (hn, cn) = layers[i](hn, (hns[i], cns[i]))
        new_hns.append(hn)
        new_cns.append(cn)
    return hn, (new_hns, new_cns)

def forward(self, input, hx):
    actions = []
    hns, cns = hx
    action = torch.Tensor([[0.0]])
    for i in range(len(input)):
        new_input = input[i]
        new_input = new_input.view(1, -1)
        output, (hns, cns) = self.forwardLayers(new_input, hns, cns, self.layers)
        output = self.softmax(self.linear(output))

    return output

Now when I call the following code to see the parameters of my network:

for name, param in myLSTM_object.named_parameters():
        if param.requires_grad:
            print name, param.data

What I get is:

linear.weight tensor([[ 0.5042, -0.6984],
    [ 0.0721, -0.4060]])
linear.bias tensor([ 0.6968, -0.4649])

So, it completely misses the parameters of LSTMCell. Does this mean the parameters of LSTMCell is not trained. What should I do to see LSTMCell parameters?


title: For each layer in neural networks (pytorch), how many biases should be there?

body:
I have a simple model in pytorch.

model = Network()

It's details are:

Network(
  (hidden): Linear(in_features=784, out_features=256, bias=True)
  (output): Linear(in_features=256, out_features=10, bias=True)
  (sigmoid): Sigmoid()
  (softmax): Softmax(dim=1)
)

There are 3 neurons' layers in total. 1 input(786 neurons), 1 hidden(256 neurons) and 1 output (10 neurons). Therefore there'll be two weight layers. So there must be two biases (simply two floating point numbers) for both weight layers right? (correct me if i am wrong).

Now after initializing my network i was curious about the two bias values. So i wanted to check the bias value of my hidden layer so i wrote:

model.hidden.bias

And what i got as the result was not i expected! I actually expected one value! And this is what i actually got:

tensor([-1.6868e-02, -3.5661e-02,  1.2489e-02, -2.7880e-02,  1.4025e-02,
        -2.6085e-02,  1.2625e-02, -3.1748e-02,  5.0335e-03,  3.8031e-03,
        -3.1648e-02, -3.4881e-02, -2.0026e-02,  1.9728e-02,  6.2461e-03,
         9.3936e-04, -5.9270e-03, -2.7183e-02, -1.9850e-02, -3.5693e-02,
        -1.9393e-02,  2.6555e-02,  2.3482e-02,  2.1230e-02, -2.2175e-02,
        -2.4386e-02,  3.4848e-02, -2.6044e-02,  1.3575e-02,  9.4125e-03,
         3.0012e-02, -2.6078e-02,  7.1615e-05, -1.7061e-02,  6.6355e-03,
        -3.4966e-02,  2.9311e-02,  1.4060e-02, -2.5763e-02, -1.4020e-02,
         2.9852e-02, -7.9176e-03, -1.8396e-02,  1.6927e-02, -1.1001e-03,
         1.5595e-02,  1.2169e-02, -1.2275e-02, -2.9270e-03, -6.5685e-04,
        -2.4297e-02,  3.0048e-02,  2.9692e-03, -2.5398e-02,  2.9955e-03,
        -9.3653e-04, -1.2932e-02,  2.4232e-02, -3.5182e-02, -1.6163e-02,
         3.0025e-02,  3.1227e-02, -8.2498e-04,  2.7102e-02, -2.3830e-02,
        -3.4958e-02, -1.1886e-02,  1.6097e-02,  1.4579e-02, -2.6744e-02,
         1.1900e-02, -3.4855e-02, -4.2208e-03, -5.2035e-03,  1.7055e-02,
        -4.8580e-03,  3.4088e-03,  1.6923e-02,  3.5570e-04, -3.0478e-02,
         8.4647e-03,  2.5704e-02, -2.3255e-02,  6.9396e-03, -1.2521e-03,
        -9.4101e-03, -2.5798e-02, -1.4438e-03, -7.2684e-03,  3.5417e-02,
        -3.4388e-02,  1.3706e-02, -5.1430e-03,  1.6174e-02,  1.8135e-03,
        -2.9018e-02, -2.9083e-02,  7.4100e-03, -2.7758e-02,  2.4367e-02,
        -3.8350e-03,  9.4390e-03, -1.0844e-02,  1.6381e-02, -2.5268e-02,
         1.3553e-02, -1.0545e-02, -1.3782e-02,  2.8519e-02,  2.3630e-02,
        -1.9703e-02, -2.0147e-02, -1.0485e-02,  2.4637e-02,  1.9989e-02,
         5.6601e-03,  1.9121e-02, -1.5286e-02,  2.5996e-02, -2.9833e-02,
        -2.9458e-02,  2.3944e-02, -3.0107e-02, -1.2307e-02, -1.8419e-02,
         3.3551e-02,  1.2396e-02,  2.9356e-02,  3.3274e-02,  5.4677e-03,
         3.1715e-02,  1.3361e-02,  3.3042e-02,  2.7843e-03,  2.2837e-02,
        -3.4981e-02,  3.2355e-02, -2.7658e-03,  2.2184e-02, -2.0203e-02,
        -3.3264e-02, -3.4858e-02,  1.0820e-03, -1.4279e-02, -2.8041e-02,
         4.1962e-03,  2.4266e-02, -3.5704e-02, -2.6172e-02,  2.3335e-02,
         2.0657e-02, -3.0387e-03, -5.7096e-03, -1.1062e-02,  1.3450e-02,
        -3.3965e-02,  1.9623e-03, -2.0067e-02, -3.3858e-02, -2.1931e-02,
        -1.5414e-02,  2.4454e-02,  2.5668e-02, -1.1932e-02,  5.7540e-04,
         1.5130e-02,  1.3916e-02, -2.1521e-02, -3.0575e-02,  1.8841e-02,
        -2.3240e-02, -2.7297e-02, -3.2668e-02, -1.5544e-02, -5.9408e-03,
         3.0241e-02,  2.2039e-02, -2.4389e-02,  3.1703e-02,  3.5305e-02,
        -2.7501e-03,  2.0154e-02, -5.3489e-03,  1.4177e-02,  1.6829e-02,
         3.3066e-02, -1.3425e-02, -3.2565e-02,  6.5624e-03, -1.5681e-02,
         2.3047e-02,  6.5880e-03, -3.3803e-02,  2.3790e-02, -5.5061e-03,
         2.9413e-02,  1.2290e-02, -1.0958e-02,  1.2680e-03,  1.3343e-02,
         6.6689e-03, -2.2975e-03, -1.2068e-02,  1.6523e-02, -3.1612e-02,
        -1.7529e-02, -2.2220e-02, -1.4723e-02, -1.3495e-02, -5.1805e-03,
        -2.9620e-02,  3.0571e-02, -3.0999e-02,  3.3681e-03,  1.3579e-02,
         1.4837e-02,  1.5694e-02, -1.1178e-02,  4.6233e-03, -2.2583e-02,
        -3.5281e-03,  3.0918e-02,  2.6407e-02,  1.5822e-04, -3.0181e-03,
         8.6989e-03,  2.8998e-02, -1.5975e-02, -3.1574e-02, -1.5609e-02,
         1.0472e-02,  5.8976e-03,  7.0131e-03, -3.2047e-02,  2.6045e-02,
        -2.8882e-02, -2.2121e-02, -3.2960e-02,  1.8268e-02,  3.0984e-02,
         1.4824e-02,  3.0010e-02, -5.7523e-03, -2.0017e-02,  4.8700e-03,
         1.4997e-02, -1.4898e-02,  6.8572e-03,  9.7713e-03,  1.3410e-02,
         4.9619e-03,  3.1016e-02,  3.1240e-02, -3.0203e-02,  2.1435e-02,
         2.7331e-02], requires_grad=True)

Can someone explain to me this behaviour? Why did i get 256 values instead of one?

Edit1:

Here is my understanding of the layers:
For a whole layer of neurons, a bias is just a single value. Am i right? But what i am seeing as the output about are 256 values ? why ? did pytorch assume that i have a bias with each neuron ? is that okay?


title: azure ml & Pytorch: sample conda-dependencies.yml and docker?

body:
Could you please point me to the documentation sample showcasing how to put together pytorch dependencies for training on AzureML?
Few related questions to the scenario of running pytorch training workloads on AzureML:

How can I set cuda version to 10.1?
Could you please point to sample demonstrating how to use “official” pytorch docker https://hub.docker.com/r/pytorch/pytorch  (which should have all cuda stuff https://github.com/pytorch/pytorch/blob/master/docker/pytorch/Dockerfile)?.
I’ve found distributed-pytorch-with-horovod.yml in the docs but it does not mention any pytorch dependencies  -- am I looking in the right place?


title: Logloss metric in Fastai

body:
i'm doing a competition in zindi plateform which they are using The evaluation metric for this challenge as Log Loss.

so i'm working with fastai library and i want the metric log loss .. i didn't find LogLoss as metric in this library !
i tried some codes like the function provided by sklearn from sklearn.metrics import log_loss but i didn't work
the link of the competition : https://zindi.africa/competitions/basic-needs-basic-rights-kenya-tech4mentalhealth