title: Pytorch: call other Module in for-loop over RNN timesteps

body:
I work with text data from human-machine dialog, where each dialog contains a number of turns (which in turn are a sequence of words), and I want to predict a label for each turn.

My idea is to use a recurrent neural network to loop over fixed-size turn representations that I compute with a simple Multilayer Perceptron based on some features (so no looping over the words in the turn here).

In the code sample below, this is how I do this.

My question: If I compute turn representations timestep by timestep within the for loop and then combine them via torch.stack(), will the loss finally backpropagate through all the MLP timesteps, i.e. all the forward computations? Or will only the last step in the for loop be considered for loss computation? Is there another way to do this?

PS: Because I want to compute different sets of features for different items in my training data, I need the turn MLP to transform turns to a fixed size, otherwise this would of course not be necessary.

class TurnMLP(nn.Module):
  def __init__(self):
    self.linear = nn.Linear(4, 20)
    super(TurnMLP, self).__init__()

  def forward(self, x_turn):
    return torch.sigmoid(self.linear(x_turn))

class DialogRNN(nn.Module):
  def __init__(self):
    self.turn_mlp = TurnMLP()
    self.lstm = nn.LSTM(20, 15)
    self.dense = nn.Linear(15, 3)
    super(DialogRNN, self).__init__()

  def forward(self, x_dialog):  # x_dialog is a list of turns, a turn is a feature vector
    turns_hidden = []

    # Here comes the interesting part: is it okay to call the turn_mlp at each timestep and expect it to store all gradients? I guess not, right?
    for x_turn in x_dialog:
      turn_hidden.append(self.turn_mlp.forward(x_turn)
    turns_hidden = torch.stack(turns_hidden)
    turns_outputs, _ = self.lstm(turns_hidden)
    return torch.sigmoid(self.dense(turns_outputs))

# TRAINING
dialog_rnn = DialogRNN()

# a dialog with 3 turns, converted to feature vectors
sample_dialog = [torch.tensor([0,1,0,1]), torch.tensor([1,0,1,1]), torch.tensor([1,0,0,0])]
sample_labels = torch.tensor([2,0,1])

prediction = dialog_rnn.forward(sample_dialog)
loss = torch.nn.functional.cross_entropy(prediction, sample_labels)
optimizer = torch.optim.Adam(dialog_rnn.parameters())

loss.backward()
optimizer.step()


title: Linear Regresion with PyTorch gives NaN values

body:
I'm learning regression (Profit vs R&D) with PyTorch. I have created the following script:

url =https://raw.githubusercontent.com/LakshmiPanguluri/Linear_Multiple_Regression/master/50_Startups.csv
starup = pd.read_csv(url)

profit = np.array(starup['Profit']).reshape(-1,1)
rd = np.array(starup['R&D Spend']).reshape(-1,1)
marketing = np.array(starup['Marketing Spend']).reshape(-1,1)
administration = np.array(starup['Administration']).reshape(-1,1)

profit_torch = torch.from_numpy(profit).float()
rd_torch = torch.from_numpy(rd).float().requires_grad_(True)

model = nn.Linear(1,1)
loss_function = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.010)

losses = []
iterations = 1000
for i in range(iterations):
  pred = model(rd_torch)
  loss = loss_function(pred, profit_torch)
  losses.append(loss.data)

  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

print(loss)
plt.plot(range(iterations), losses)
tensor(nan, grad_fn=<MeanBackward0>)

My question is why it gives me a tensor with NaN values and why the loss is growing up in every iteration.

The plot of the losses is a line with a positive slope:

Link to the project I've been doing


title: Libtorch:how to create tensor from tensorRT fp16 half type pointer?

body:
how to create tensor from tensorRT fp16 half type pointer in libtorch?
I am working on a detection model. I change the backbone of it to tensorRT to do FP16 inference, and the detection code such as decode boxes and nms is done in libtorch and torchvisoin, so how to create fp16 tensor from tensorRT half type pointers?
The important code is to illustrate the issue:

// tensorRT code to get half type outpus
half_float::half* outputs[18];
doInference(*engine, data, outputs, 1);
// to get the final outputs with libtorch
vector<torch::Tensor> output;
//???? how to feed the date in outpus to output????
// get the result with libtorch method detect_trt->forward
 auto res = detect_trt->forward(output);

Thanks in advance.


title: training Bert for multiple sentences

body:
I have 2 sentences ["first sentence", "another one"] tokenized the input using batch_encode_plus(). While training I'm getting the following error:

RuntimeError: The expanded size of the tensor (64) must match the existing size (2) at non-singleton dimension 2.  Target sizes: [64, 2, 64].  Tensor sizes: [1, 2]


title: How does a pytorch dataset object know whether it has hit the end when used in a for loop?

body:
I am writing a custom pytorch dataset. In __init__ the dataset object loads a file that contains certain data. But in my program I only wish to access part of the data (to achieve train/valid cut, if it helps). Originally I thought this behavior was controlled by overriding __len__, but it turned out that modifying __len__ does not help. A simple example is as follows:
from torch.utils.data import Dataset, DataLoader
import torch

class NewDS(Dataset):
    def __init__(self):
        self.data = torch.randn(10,2) # suppose there are 10 items in the data file

    def __len__(self):
        return len(self.data)-5 # But I only want to access the first 5 items

    def __getitem__(self, index):
        return self.data[index]

ds = NewDS()
for i, x in enumerate(ds):
    print(i)

The output is 0 through 9, while the desired behavior would be 0 through 4.
How does this dataset object know that the enumeration has hit the end when used in a for loop like this? Any other method to achieve a similar effect is also welcome.


title: Pytorch DQN, DDQN using .detach() caused very wield loss (increases exponentially) and do not learn at all

body:
Here is my implementation of DQN and DDQN for CartPole-v0 which I think is correct.
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import gym
import torch.optim as optim
import random
import os
import time

class NETWORK(torch.nn.Module):
    def __init__(self, input_dim: int, output_dim: int, hidden_dim: int) -> None:

        super(NETWORK, self).__init__()

        self.layer1 = torch.nn.Sequential(
            torch.nn.Linear(input_dim, hidden_dim),
            torch.nn.ReLU()
        )

        self.layer2 = torch.nn.Sequential(
            torch.nn.Linear(hidden_dim, hidden_dim),
            torch.nn.ReLU()
        )

        self.final = torch.nn.Linear(hidden_dim, output_dim)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.final(x)

        return x

class ReplayBuffer(object):
    def __init__(self, capacity=50000):
        self.capacity = capacity
        self.memory = []
        self.position = 0

    def push(self, s0, a0, r, s1):
        if len(self.memory) < self.capacity:
            self.memory.append(None)
        self.memory[self.position] = (s0, a0, r, s1)
        self.position = (self.position + 1) % self.capacity

    def sample(self, batch_size=64):
        return random.sample(self.memory, batch_size)

    def __len__(self):
        return len(self.memory)

class DQN(object):
    def __init__(self):
      self.state_dim = 4
      self.action_dim = 2
      self.lr = 0.001
      self.discount_factor = 0.99
      self.epsilon = 1
      self.epsilon_decay = 0.95
      self.num_train = 0
      self.num_train_episodes = 0
      self.batch_size = 64

      self.predict_network = NETWORK(input_dim=4, output_dim=2, hidden_dim=16).double()

      self.memory = ReplayBuffer(capacity=50000)
      self.optimizer = torch.optim.Adam(self.predict_network.parameters(), lr=self.lr)
      self.loss = 0

    def select_action(self, states: np.ndarray) -> int:
      if np.random.uniform(0, 1) < self.epsilon:
        return np.random.choice(self.action_dim)
      else:
        states = torch.from_numpy(states).unsqueeze_(dim=0)
        with torch.no_grad():
          Q_values = self.predict_network(states)
          action = torch.argmax(Q_values).item()
        return action

    def policy(self, states: np.ndarray) -> int:
      states = torch.from_numpy(states).unsqueeze_(dim=0)
      with torch.no_grad():
        Q_values = self.predict_network(states)
        action = torch.argmax(Q_values).item()
      return action

    def train(self, s0, a0, r, s1, sign):
      if sign == 1:
        self.num_train_episodes += 1
        if self.epsilon > 0.01:
          self.epsilon = max(self.epsilon * self.epsilon_decay, 0.01)
        return

      self.num_train += 1
      self.memory.push(s0, a0, r, s1)
      if len(self.memory) < self.batch_size:
        return

      batch = self.memory.sample(self.batch_size)
      state_batch = torch.from_numpy(np.stack([b[0] for b in batch]))
      action_batch = torch.from_numpy(np.stack([b[1] for b in batch]))
      reward_batch = torch.from_numpy(np.stack([b[2] for b in batch]))
      next_state_batch = torch.from_numpy(np.stack([b[3] for b in batch]))

      Q_values = self.predict_network(state_batch)[torch.arange(self.batch_size), action_batch]

      next_state_Q_values = self.predict_network(next_state_batch).max(dim=1)[0]

      Q_targets = self.discount_factor * next_state_Q_values + reward_batch

      loss = F.mse_loss(Q_values, Q_targets.detach())

      self.optimizer.zero_grad()
      loss.backward()
      self.optimizer.step()

      self.loss = loss.data.item()

class DDQN(object):
    def __init__(self):
      self.state_dim = 4
      self.action_dim = 2
      self.lr = 0.001
      self.discount_factor = 0.9
      self.epsilon = 1
      self.epsilon_decay = 0.95
      self.num_train = 0
      self.num_train_episodes = 0
      self.batch_size = 64

      self.predict_network = NETWORK(input_dim=4, output_dim=2, hidden_dim=16).double()
      self.target_network = NETWORK(input_dim=4, output_dim=2, hidden_dim=16).double()
      self.target_network.load_state_dict(self.predict_network.state_dict())
      self.target_network.eval()

      self.memory = ReplayBuffer(capacity=50000)
      self.optimizer = torch.optim.Adam(self.predict_network.parameters(), lr=self.lr)

      self.loss = 0

    def select_action(self, states: np.ndarray) -> int:
      if np.random.uniform(0, 1) < self.epsilon:
        return np.random.choice(self.action_dim)
      else:
        states = torch.from_numpy(states).unsqueeze_(dim=0)
        with torch.no_grad():
          Q_values = self.predict_network(states)
        action = torch.argmax(Q_values).item()
        return action

    def policy(self, states: np.ndarray) -> int:
      states = torch.from_numpy(states).unsqueeze_(dim=0)
      with torch.no_grad():
        Q_values = self.predict_network(states)
        action = torch.argmax(Q_values).item()
      return action

    def train(self, s0, a0, r, s1, sign):
      if sign == 1:
        self.num_train_episodes += 1
        if self.num_train_episodes % 2 == 0:
          self.target_network.load_state_dict(self.predict_network.state_dict())
          self.target_network.eval()

          if self.epsilon > 0.01:
            self.epsilon = max(self.epsilon * self.epsilon_decay, 0.01)
        return

      self.num_train += 1
      self.memory.push(s0, a0, r, s1)
      if len(self.memory) < self.batch_size:
        return
      batch = self.memory.sample(self.batch_size)
      state_batch = torch.from_numpy(np.stack([b[0] for b in batch]))
      action_batch = torch.from_numpy(np.stack([b[1] for b in batch]))
      reward_batch = torch.from_numpy(np.stack([b[2] for b in batch]))
      next_state_batch = torch.from_numpy(np.stack([b[3] for b in batch]))

      Q_values = self.predict_network(state_batch)[torch.arange(self.batch_size), action_batch]

      next_state_action_batch = torch.argmax(self.predict_network(next_state_batch), dim=1)

      next_state_Q_values = self.target_network(next_state_batch)[torch.arange(self.batch_size), next_state_action_batch]

      Q_targets = self.discount_factor * next_state_Q_values + reward_batch

      loss = F.smooth_l1_loss(Q_values, Q_targets.detach())
      self.optimizer.zero_grad()
      loss.backward()
      self.optimizer.step()

      self.loss = loss.data.item()

I used following to evaluate and train my DQN and DDQN.
def eval_policy(agent, env_name, eval_episodes=10):
    eval_env = gym.make(env_name)
    avg_reward = 0.
    for _ in range(eval_episodes):
        state, done = eval_env.reset(), False
        while not done:
            action = agent.policy(state)
            state, reward, done, _ = eval_env.step(action)
            avg_reward += reward
    avg_reward /= eval_episodes
    print("---------------------------------------")
    print(f"Evaluation over {eval_episodes} episodes: {avg_reward:.3f}")
    print("---------------------------------------")
    return avg_reward

env_name = 'CartPole-v0'
env = gym.make(env_name)

agent = DQN() # agent = DDQN()

for i in range(1000):
    state, done = env.reset(), False
    episodic_reward = 0
    while not done:
        action = agent.select_action(np.squeeze(state))
        next_state, reward, done, info = env.step(action)
        episodic_reward += reward
        sign = 1 if done else 0
        agent.train(state, action, reward, next_state, sign)
        state = next_state
    print(f'episode: {i}, reward: {episodic_reward}')
    if i % 20 == 0:
        eval_reward = eval_policy(agent, env_name, eval_episodes=50)
        if eval_reward >= 195:
            print("Problem solved in {} episodes".format(i + 1))
            break

The thing is that my DQN networks do not train and the loss grow exponentially using target.detach() in loss calculation. If I do not use .detach(), the DQN object would train but I believe that is not the correct way. For DDQN, my networks always do not train. Can anyone give some advice on where might be wrong?


title: PyTorch: "ValueError: can't optimize a non-leaf Tensor" after changing pretrained model from 3 RGB Channels to 4 Channels

body:
I have been trying to change the pretrained PyTorch Densenet's first conv layer from 3 channels to 4 channels while maintaining its original RGB channel's pretrained weights. I have done the following codes, but the optimizer part throws me this error:  "ValueError: can't optimize a non-leaf Tensor" .
import torchvision.models as models
import torch.nn as nn
backbone = models.__dict__['densenet169'](pretrained=True)

weight1 = backbone.features.conv0.weight.data.clone()
new_first_layer  = nn.Conv2d(4, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
with torch.no_grad():
    new_first_layer.weight[:,:3] = weight1

backbone.features.conv0 = new_first_layer
optimizer = torch.optim.SGD(backbone.parameters(), 0.001,
                                 weight_decay=0.1)  # Changing this optimizer from SGD to ADAM

I have also tried to remove the argument with torch.no_grad(): but this issue still remains:
  ValueError                                Traceback (most recent call last)
<ipython-input-343-5fc87352da04> in <module>()
     11 backbone.features.conv0 = new_first_layer
     12 optimizer = torch.optim.SGD(res.parameters(), 0.001,
---> 13                                  weight_decay=0.1)  # Changing this optimizer from SGD to ADAM

~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/sgd.py in __init__(self, params, lr, momentum, dampening, weight_decay, nesterov)
     66         if nesterov and (momentum <= 0 or dampening != 0):
     67             raise ValueError("Nesterov momentum requires a momentum and zero dampening")
---> 68         super(SGD, self).__init__(params, defaults)
     69
     70     def __setstate__(self, state):

~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/optimizer.py in __init__(self, params, defaults)
     50
     51         for param_group in param_groups:
---> 52             self.add_param_group(param_group)
     53
     54     def __getstate__(self):

~/anaconda3/envs/detectron2/lib/python3.6/site-packages/torch/optim/optimizer.py in add_param_group(self, param_group)
    231                                 "but one of the params is " + torch.typename(param))
    232             if not param.is_leaf:
--> 233                 raise ValueError("can't optimize a non-leaf Tensor")
    234
    235         for name, default in self.defaults.items():

ValueError: can't optimize a non-leaf Tensor

My PyTorch version is: 1.7.0.
Could you guys please help? Thanks alot!
Regards.


title: How can I find the equivalent 'batch_size' used in Keras from this Pytorch code?

body:
I am using Pytorch code from github

I am trying to port this over to Keras.
In particular, Keras uses model.fit for training the Neural net and has a batch_size parameter. I am trying to set this but cannot determine it in the Pytorch script linked above.

In the script, there is a function called sliding_window in box 4 where there is a parameter called step. I am not sure if this is the equivalent setting of batch_size.

Also, I am looking into how to set the learning rate scheduler from the box 11 in the Pytorch code:

base_lr = 0.01
params_dict = dict(net.named_parameters())
params = []
for key, value in params_dict.items():
    if '_D' in key:
        # Decoder weights are trained at the nominal learning rate
        params += [{'params':[value],'lr': base_lr}]
    else:
        # Encoder weights are trained at lr / 2 (we have VGG-16 weights as initialization)
        params += [{'params':[value],'lr': base_lr / 2}]

optimizer = optim.SGD(net.parameters(), lr=base_lr, momentum=0.9, weight_decay=0.0005)
# We define the scheduler
scheduler = optim.lr_scheduler.MultiStepLR(optimizer, [25, 35, 45], gamma=0.1)

I have always used the default learning rate from Keras. Any insight on how to go about converting this scheduler into Keras code  would also be appreciated.


title: How does pytorch nn.module keep computation graph and when&how release it?

body:
I got a piece of code from:
https://github.com/liorshk/facenet_pytorch/blob/master/train_triplet.py

def train(train_loader, model, optimizer, epoch):
# switch to train mode
model.train()

pbar = tqdm(enumerate(train_loader))
labels, distances = [], []

for batch_idx, (data_a, data_p, data_n,label_p,label_n) in pbar:

    data_a, data_p, data_n = data_a.cuda(), data_p.cuda(), data_n.cuda()
    data_a, data_p, data_n = Variable(data_a), Variable(data_p), \
                             Variable(data_n)

    # compute output
    out_a, out_p, out_n = model(data_a), model(data_p), model(data_n)

    # Choose the hard negatives
    d_p = l2_dist.forward(out_a, out_p)
    d_n = l2_dist.forward(out_a, out_n)
    all = (d_n - d_p < args.margin).cpu().data.numpy().flatten()
    hard_triplets = np.where(all == 1)
    if len(hard_triplets[0]) == 0:
        continue
    out_selected_a = Variable(torch.from_numpy(out_a.cpu().data.numpy()[hard_triplets]).cuda())
    out_selected_p = Variable(torch.from_numpy(out_p.cpu().data.numpy()[hard_triplets]).cuda())
    out_selected_n = Variable(torch.from_numpy(out_n.cpu().data.numpy()[hard_triplets]).cuda())

    selected_data_a = Variable(torch.from_numpy(data_a.cpu().data.numpy()[hard_triplets]).cuda())
    selected_data_p = Variable(torch.from_numpy(data_p.cpu().data.numpy()[hard_triplets]).cuda())
    selected_data_n = Variable(torch.from_numpy(data_n.cpu().data.numpy()[hard_triplets]).cuda())

    selected_label_p = torch.from_numpy(label_p.cpu().numpy()[hard_triplets])
    selected_label_n= torch.from_numpy(label_n.cpu().numpy()[hard_triplets])
    triplet_loss = TripletMarginLoss(args.margin).forward(out_selected_a, out_selected_p, out_selected_n)

    cls_a = model.forward_classifier(selected_data_a)
    cls_p = model.forward_classifier(selected_data_p)
    cls_n = model.forward_classifier(selected_data_n)

    criterion = nn.CrossEntropyLoss()
    predicted_labels = torch.cat([cls_a,cls_p,cls_n])
    true_labels = torch.cat([Variable(selected_label_p.cuda()),Variable(selected_label_p.cuda()),Variable(selected_label_n.cuda())])

    cross_entropy_loss = criterion(predicted_labels.cuda(),true_labels.cuda())

    loss = cross_entropy_loss + triplet_loss
    # compute gradient and update weights
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    # update the optimizer learning rate
    adjust_learning_rate(optimizer)

    # log loss value
    logger.log_value('triplet_loss', triplet_loss.data[0]).step()
    logger.log_value('cross_entropy_loss', cross_entropy_loss.data[0]).step()
    logger.log_value('total_loss', loss.data[0]).step()
    if batch_idx % args.log_interval == 0:
        pbar.set_description(
            'Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f} \t # of Selected Triplets: {}'.format(
                epoch, batch_idx * len(data_a), len(train_loader.dataset),
                100. * batch_idx / len(train_loader),
                loss.data[0],len(hard_triplets[0])))

    dists = l2_dist.forward(out_selected_a,out_selected_n) #torch.sqrt(torch.sum((out_a - out_n) ** 2, 1))  # euclidean distance
    distances.append(dists.data.cpu().numpy())
    labels.append(np.zeros(dists.size(0)))

    dists = l2_dist.forward(out_selected_a,out_selected_p)#torch.sqrt(torch.sum((out_a - out_p) ** 2, 1))  # euclidean distance
    distances.append(dists.data.cpu().numpy())
    labels.append(np.ones(dists.size(0)))

labels = np.array([sublabel for label in labels for sublabel in label])
distances = np.array([subdist[0] for dist in distances for subdist in dist])

tpr, fpr, accuracy, val, val_std, far = evaluate(distances,labels)
print('\33[91mTrain set: Accuracy: {:.8f}\n\33[0m'.format(np.mean(accuracy)))
logger.log_value('Train Accuracy', np.mean(accuracy))

plot_roc(fpr,tpr,figure_name="roc_train_epoch_{}.png".format(epoch))

# do checkpointing
torch.save({'epoch': epoch + 1, 'state_dict': model.state_dict()},
           '{}/checkpoint_{}.pth'.format(LOG_DIR, epoch))

as you can see,there are a bunch of model.forward(..) invoking:

out_a, out_p, out_n = model(data_a), model(data_p), model(data_n)
....
cls_a = model.forward_classifier(selected_data_a)
cls_p = model.forward_classifier(selected_data_p)
cls_n = model.forward_classifier(selected_data_n)
(model.forward_classifier() will invoke model.forward() also)

due to loss.backward() invoked,before second iteration beginning,a little bit gpu memory was release,but more than 30G gpu memory not(I have 4x1080ti),then training crashed with run out of GPU memory,so question are:

1,In a nn.module,how(or where) is computation graph saved,how to release gpu memory?

2,Must to call backward()?if yes,what should be in .backward() method?

3,In my case,which part of GPU memory is released,which part of them is kept?Is there any way to see it?


title: Variable Performance when Training Resnet on Google Compute Engine with data from Google Cloud Storage

body:
I'm attempting to train a resnet18 image classifier on a large number of crops from labeled Google Streeview data. I'm following along with this tutorial. I have two datasets, one of approximately ~20k images and one of approximately ~100k images. Both datasets are stored in the same format, and both have been uploaded to respective Google Cloud Storage buckets. I then mounted both of these buckets in my VM's home directory using gcsfuse with the --implicit-dirs flag.

I then run my train.py file on my Google Compute Engine VM, which was created from the Deep Learning VM image on Google's Cloud Marketplace. The VM has a single vCPU, a single Nvidia Tesla K80 GPU, 3.75gb of memory, and a 100gb persistent disk.

When I run the training script, I make no changes except pointing the dataset_dir variable to the correct gcsfuse-mounted directory on the VM.

When I run train.py on the 100k crops directory, it runs relatively quickly, with a single epoch taking ~30 mins. I jump into top while it's running, and CPU utilization is quite high, staying around 90%.

However, using the same VM, when I run train.py on the 20K crops directory, it runs much more slowly, with a single epoch taking 6-7 hours, despite the smaller size of the dataset. In this case, CPU utilization never jumps above about 5%.

I cannot figure out what is causing the slowdown, as nothing (as far as I can tell) is different between the two runs except for the datasets, which are both formatted identically. I use the same pytorch dataloader with the same number of threads. Both GCS buckets are in the same region, us-west1 which is the same region as my VM instance.

It seems likely that somehow one bucket is IO-limited relative to the other bucket, but I cannot figure out why.

Any thoughts are appreciated!

My train.py file is below.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy
from collections import defaultdict

data_transforms = {
    'Test': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'Val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = 'home/gweld/sliding_window_dataset/'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['Test', 'Val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['Test', 'Val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['Test', 'Val']}
class_names = image_datasets['Test'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['Test', 'Val']:
            if phase == 'Test':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode

            running_loss = 0.0
            running_corrects = 0

            class_corrects = defaultdict(int)
            class_totals   = defaultdict(int)

            # Iterate over data.
            for inputs, labels in dataloaders[phase]:
                inputs = inputs.to(device)
                labels = labels.to(device)

                # zero the parameter gradients
                optimizer.zero_grad()

                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'Test'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)

                    # backward + optimize only if in training phase
                    if phase == 'Test':
                        loss.backward()
                        optimizer.step()

                # statistics
                running_loss += loss.item() * inputs.size(0)
                running_corrects += torch.sum(preds == labels.data)

                for index, pred in enumerate(preds):
                    actual = labels.data[index]
                    class_name = class_names[actual]

                    if actual == pred: class_corrects[class_name] += 1
                    class_totals[class_name] += 1

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]

            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))

            if phase == 'Val':
                print("Validation Class Accuracies")

                for class_name in class_totals:
                    class_acc = float(class_corrects[class_name])
                    class_acc = class_acc/class_totals[class_name]

                    print("{:20}{}%".format(class_name, 100*class_acc))
                print("\n")

            # deep copy the model
            if phase == 'Val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())

        print()

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 5) # last arg here, # classes? -gw

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

# Train and evaluate
# ^^^^^^^^^^^^^^^^^^

print('Beginning Training on {} train and {} val images.'.format(dataset_sizes['Test'], dataset_sizes['Val']))

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=25)

torch.save(model_ft.state_dict(), 'models/test_run_resnet18.pt')


title: Using optim.step() with Pytorch's DataLoader

body:
Usually the learning cycle contains:

optim.zero_grad()
loss(m, op).backward()
optim.step()

But what should be the cycle when the data does not fit in the graphics card?

First option:

for ip, op in DataLoader(TensorDataset(inputs, outputs),
                          batch_size=int(1e4), pin_memory=True):
    m = model(ip.to(dev))
    op = op.to(dev)
    optim.zero_grad()
    loss(m, op).backward()
    optim.step()

Second option:

optim.zero_grad()
for ip, op in DataLoader(TensorDataset(inputs, outputs),
                          batch_size=int(1e4), pin_memory=True):
    m = model(ip.to(dev))
    op = op.to(dev)
    loss(m, op).backward()
optim.step()

The third option:
Accumulate gradients after calling backward().


title: How to create my own loss function in Pytorch?

body:
I'd like to create a model that predicts parameters of a circle (coordinates of center, radius).

Input is an array of points (of arc with noise):

def generate_circle(x0, y0, r, start_angle, phi, N, sigma):
    theta = np.linspace(start_angle*np.pi/180, (start_angle + phi)*np.pi/180, num=N)
    x = np.array([np.random.normal(r*np.cos(t) + x0 , sigma, 1)[0] for t in theta])
    y = np.array([np.random.normal(r*np.sin(t) + y0 , sigma, 1)[0] for t in theta])
    return x, y

n_x = 1000
start_angle = 0
phi = 90
N = 100
sigma = 0.005
x_full = []
for i in range(n_x):
    x0 = np.random.normal(0 , 10, 1)[0]
    y0 = np.random.normal(0 , 10, 1)[0]
    r = np.random.normal(0 , 10, 1)[0]
    x, y = generate_circle(x0, y0, r, start_angle, phi, N, sigma)
    x_full.append(np.array([ [x[i], y[i]] for i in range(len(x))]))
X = torch.from_numpy(np.array(x_full))
print(X.size()) # torch.Size([1000, 100, 2])

Output: [x_c, y_c, r]

As a loss function I need to use this one:

I tried to implement something like the following:

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)
        self.predict = torch.nn.Linear(n_hidden, n_output)

    def forward(self, x):
        x = F.relu(self.hidden(x))
        x = self.predict(x)
        return x

# It doesn't work, it's just an idea
def my_loss(point, params):
    arr = ((point[:, 0] - params[:, 0])**2 + (point[:, 1] - params[:, 1])**2 - params[:, 2]**2)**2
    loss = torch.sum(arr)
    return loss

# For N pairs (x, y) model predicts parameters of circle
net = Net(n_feature=N*2, n_hidden=10, n_output=3)

optimizer = torch.optim.SGD(net.parameters(), lr=1e-4)

for t in range(1000):
    prediction = net(X.view(n_x, N*2).float())
    loss = my_loss(X, prediction)
    print(f"loss: {loss}")

    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

So, the question is how to correctly implement my own loss function in terms of Pytorch in this case?

Or how to change the model's structure to get expected results?


I have this Pytorch Ensemble model but can figure out what is wrong. thanks for the help.

I think the problem is with the first Fc1, I got this from a book in Packt Deep Learning with PyTorch, I have included the definition of fit() so you can see the complete loop. I did no included before because it tells me that it looks tha is to much code and no words...

def fit(epoch,model,data_loader,phase='training',volatile=False):
    if phase == 'training':
        model.train()
    if phase == 'validation':
        model.eval()
        volatile=True
    running_loss = 0.0
    running_correct = 0
    for batch_idx , (data1,data2,data3,target) in enumerate(data_loader):
        if is_cuda:
            data1,data2,data3,target = data1.cuda(),data2.cuda(),data3.cuda(),target.cuda()
        data1,data2,data3,target = Variable(data1,volatile),Variable(data2,volatile),Variable(data3,volatile),Variable(target)
        if phase == 'training':
            optimizer.zero_grad()
        output = model(data1,data2,data3)
        loss = F.cross_entropy(output,target)

        running_loss += F.cross_entropy(output,target,size_average=False).data[0]
        preds = output.data.max(dim=1,keepdim=True)[1]
        running_correct += preds.eq(target.data.view_as(preds)).cpu().sum()
        if phase == 'training':
            loss.backward()
            optimizer.step()

    loss = running_loss/len(data_loader.dataset)
    accuracy = 100. * running_correct/len(data_loader.dataset)

    print(f'{phase} loss is {loss:{5}.{2}} and {phase} accuracy is {running_correct}/{len(data_loader.dataset)}{accuracy:{10}.{4}}')
    return loss,accuracy

class LayerActivations():
    features=[]

    def __init__(self,model):
        self.features = []
        self.hook = model.register_forward_hook(self.hook_fn)

    def hook_fn(self,module,input,output):
        #out = F.avg_pool2d(output, kernel_size=8)
        self.features.extend(output.view(output.size(0),-1).cpu().data)

    def remove(self):

        self.hook.remove()

class EnsembleModel(nn.Module):
    def __init__(self, out_size, training=True):
        super().__init__()
        self.fc1 = nn.Linear(8192, 512)
        self.fc2 = nn.Linear(131072, 512)
        self.fc3 = nn.Linear(82944, 512)
        self.fc4 = nn.Linear(512, out_size)

    def forward(self, inp1, inp2, inp3):
        out1 = self.fc1(F.dropout(inp1, training=self.training))
        out2 = self.fc2(F.dropout(inp2, training=self.training))
        out3 = self.fc3(F.dropout(inp3, training=self.training))
        out = out1 + out2 + out3
        out = self.fc4(F.dropout(out, training=self.training))

        return out

em = EnsembleModel(2)
if is_cuda:
    em = em.cuda()

train_losses, train_accuracy = [], []
val_losses, val_accuracy = [], []

for epoch in range(1, 10):
    epoch_loss, epoch_accuracy = fit(epoch, em, trn_feat_loader, phase="training")

    val_epoch_loss, val_epoch_accuracy = fit(
        epoch, em, val_feat_loader, phase="validation"
    )

    train_losses.append(epoch_loss)
    train_accuracy.append(epoch_accuracy)
    val_losses.append(val_epoch_loss)
    val_accuracy.append(val_epoch_accuracy)

Error:

RuntimeError                              Traceback (most recent call last)
    def extra_repr(self):
RuntimeError: size mismatch, m1: [64 x 512], m2: [8192 x 512] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:290


title: Cannot improve model accuracy

body:
I am building a general-purpose NN that would classify images (Dog/No Dog) and movie reviews(Good/Bad). I have to stick to a very specific architecture and loss function so changing these two seems out of the equation. My architecture is a two-layer network with relu followed by a sigmoid and a cross-entropy loss function. With 1000 epochs and a learning rate of around .001 I am getting 100 percent training accuracy and .72 testing accuracy.I was looking for suggestions to improve my testing accuracy.This is the layout of what I have:

def train_net(epochs,batch_size,train_x,train_y,model_size,lr):
  n_x,n_h,n_y=model_size
  model = Net(n_x, n_h, n_y)
  optim = torch.optim.Adam(model.parameters(),lr=0.005)
  loss_function = nn.BCELoss()
  train_losses = []
  accuracy = []
  for epoch in range(epochs):
    count=0
    model.train()
    train_loss = []
    batch_accuracy = []
    for idx in range(0, train_x.shape[0], batch_size):

      batch_x = torch.from_numpy(train_x[idx : idx + batch_size]).float()
      batch_y = torch.from_numpy(train_y[:,idx : idx + batch_size]).float()
      model_output = model(batch_x)
      batch_accuracy=[]
      loss = loss_function(model_output, batch_y)
      train_loss.append(loss.item())

      preds = model_output > 0.5
      nb_correct = (preds == batch_y).sum()
      count+=nb_correct.item()
      optim.zero_grad()
      loss.backward()
# Scheduler made it worse
# scheduler.step(loss.item())
  optim.step()

    if epoch % 100 == 1:
      train_losses.append(train_loss)
      print("Iteration : {}, Training loss: {} ,Accuracy %: {}".format(epoch,np.mean(train_loss),(count/train_x.shape[0])*100))
  plt.plot(np.squeeze(train_losses))
  plt.ylabel('loss')
  plt.xlabel('iterations (per tens)')
  plt.title("Learning rate =" + str(lr))
  plt.show()
  return model

My model parameters:

batch_size = 32
lr = 0.0001
epochs = 1500

n_x = 12288     # num_px * num_px * 3
n_h = 7
n_y = 1
model_size=n_x,n_h,n_y
model=train_net(epochs,batch_size,train_x,train_y,model_size,or)

and this is the testing phase.

model.eval()  #Setting the model to eval mode, hence making it deterministic.
test_loss = []
count=0;
loss_function = nn.BCELoss()

for idx in range(0, test_x.shape[0], batch_size):
  with torch.no_grad():
    batch_x = torch.from_numpy(test_x[idx : idx + batch_size]).float()
    batch_y = torch.from_numpy(test_y[:,idx : idx + batch_size]).float()
    model_output = model(batch_x)
    preds = model_output > 0.5
    loss = loss_function(model_output, batch_y)
    test_loss.append(loss.item())
    nb_correct = (preds == batch_y).sum()
    count+=nb_correct.item()

print("test loss: {},test accuracy: {}".format(np.mean(test_loss),count/test_x.shape[0]))

Things I have tried:
Messing around with the learning rate, having momentum, using schedulers and changing batch sizes.Of course these were mainly guesses and not based on any valid assumptions.


title: Convert a Keras NN to a Pytorch NN

body:
I'm a beginner with constructing neural networks in pytorch and Keras. I have the following Keras code for a variation of the AlexNet model:

def shallownet(nb_classes):
    global img_size
    model = Sequential()
    model.add(Conv2D(64, (5, 5), input_shape=img_size, data_format='channels_first'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same', data_format='channels_first'))

    model.add(Conv2D(64, (5, 5), padding='same', data_format='channels_first'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(3,3), strides=(2,2), padding='same', data_format='channels_first'))

    model.add(Flatten())
    model.add(Dense(384))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(192))
    model.add(BatchNormalization())
    model.add(Activation('relu'))
    model.add(Dropout(0.5))
    model.add(Dense(nb_classes, activation='softmax'))
    return model

This is based on the C1/C3 model described on page 12 of this paper:
https://arxiv.org/pdf/1609.04836.pdf
And I want to convert this to the Pytorch version of the NN. Specifically I'm trying to get it in the form of:

class AlexNetOWT_BN(nn.Module):

    def __init__(self, num_classes=1000):
        super(AlexNetOWT_BN, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2,
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(64, 192, kernel_size=5, padding=2, bias=False),
            nn.BatchNorm2d(192),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(192, 384, kernel_size=3, padding=1, bias=False),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(384),
            nn.Conv2d(384, 256, kernel_size=3, padding=1, bias=False),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256),
            nn.Conv2d(256, 256, kernel_size=3, padding=1, bias=False),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256)
        )
        self.classifier = nn.Sequential(
            nn.Linear(256 * 6 * 6, 4096, bias=False),
            nn.BatchNorm1d(4096),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(4096, 4096, bias=False),
            nn.BatchNorm1d(4096),
            nn.ReLU(inplace=True),
            nn.Dropout(0.5),
            nn.Linear(4096, num_classes)
        )

        #self.regime = {
        #    0: {'optimizer': 'SGD', 'lr': 1e-2,
        #        'weight_decay': 5e-4, 'momentum': 0.9},
        #    10: {'lr': 5e-3},
        #    15: {'lr': 1e-3, 'weight_decay': 0},
        #    20: {'lr': 5e-4},
        #    25: {'lr': 1e-4}
        #}
        self.regime = {
            0: {'optimizer': 'SGD', 'lr': 1e-2,
                'weight_decay': 5e-4, 'momentum': 0.9},
            20: {'lr': 1e-3},
            40: {'lr': 1e-4}
        }
        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                         std=[0.229, 0.224, 0.225])
        self.input_transform = {
            'train': transforms.Compose([
                transforms.Scale(256),
                transforms.RandomCrop(224),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize
            ]),
            'eval': transforms.Compose([
                transforms.Scale(256),
                transforms.CenterCrop(224),
                transforms.ToTensor(),
                normalize
            ])
        }

    def forward(self, x):
        x = self.features(x)
        x = x.view(-1, 256 * 6 * 6)
        x = self.classifier(x)
        return x

The original AlexNet code for self.features is here (I switched the order of things above; not sure if that's allowed or not):

self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2,
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(64, 192, kernel_size=5, padding=2, bias=False),
            nn.BatchNorm2d(192),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(192, 384, kernel_size=3, padding=1, bias=False),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(384),
            nn.Conv2d(384, 256, kernel_size=3, padding=1, bias=False),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256),
            nn.Conv2d(256, 256, kernel_size=3, padding=1, bias=False),
            nn.MaxPool2d(kernel_size=3, stride=2),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(256)
        )

I am specifically confused about how to change the features and the classifier. The description of the model I want is here:
https://arxiv.org/pdf/1609.04836.pdf on page 12, section B.3.

Thanks so much


title: Numpy memmap throttles with Pytorch Dataloader when available RAM less than file size

body:
I'm working on a dataset that is too big to fit into RAM. The solution I'm trying currently is to use numpy memmap to load one sample/row at a time using Dataloader. The solution looks something like this:

class MMDataset(torch.utils.data.Dataset):
    def __init__(self, path):
        self.file_path = path
        self.dataset_len = 44000000
        self.bytes_per_value = 32/8
        self.num_cols = 512
        self.num_rows = 1

    def __getitem__(self, index):

        x = np.memmap(self.file_path, dtype='float32', mode='r', shape=(
            self.num_rows, self.num_cols), offset=int(index*self.num_cols*self.bytes_per_value))

        return np.array(x)

    def __len__(self):
        return self.dataset_len

dataset = MMDataset('./data/emb.memmap')

data_loader = DataLoader(
    dataset,
    batch_size=4096,
    shuffle=True,
    num_workers=20
)

When the amount of RAM available is greater than the size of the memmap file, the data loading is fast. I get around 60 batches/second. However, when the RAM available is less than the size of the memmap file, I get around 3 batches/second.

I discovered this when trying various sizes for the memmap file.

Why is this the case? If Dataloader + memmap is going to throttle when available RAM < memmap file size, this defeats the point of the solution.

I've observed that disk i/o is at 500MB/s read constantly when available RAM < memmap file size. This is much higher than the theoretical amount of reading required to load a batch of 4096 samples (closer to 8MB/s).


title: Artistic Style Transfer : ValueError: Floating point image RGB values must be in the 0..1 range. using pytorch

body:
Problem
I work on a subject of "artistic transfer", and I want to transfer style of the image Style to content of Image content - but I get this error instead.
erreur :
ValueError                                Traceback (most recent call last)
<ipython-input-1-d26b5592f0b4> in <module>()
    113         plt.imshow(imcnvt(target),label="Epoch "+str(i))
    114         plt.show()
--> 115         plt.imsave(str(i)+'.png',imcnvt(target),format='png')

2 frames
/usr/local/lib/python3.6/dist-packages/matplotlib/cm.py in to_rgba(self, x, alpha, bytes, norm)
    224                 if xx.dtype.kind == 'f':
    225                     if norm and (xx.max() > 1 or xx.min() < 0):
--> 226                         raise ValueError("Floating point image RGB values "
    227                                          "must be in the 0..1 range.")
    228                     if bytes:

ValueError: Floating point image RGB values must be in the 0..1 range.

ValueError: Floating point image RGB values must be in the 0..1 range.
and I cannot resolve this error.
I have tried using content = np.abs(content) but it gives an output very far from the expected result.
I'm using pytorch. thank you!

Codes
import torch
from torchvision import transforms , models
from PIL import Image
import matplotlib.pyplot as plt
import numpy as np

device = ("cuda" if torch.cuda.is_available() else "cpu")

model = models.vgg19(pretrained=True).features
for p in model.parameters():
    p.requires_grad = False
model.to(device)

def model_activations(input,model):
    layers = {
    '0' : 'conv1_1',
    '5' : 'conv2_1',
    '10': 'conv3_1',
    '19': 'conv4_1',
    '21': 'conv4_2',
    '28': 'conv5_1'
    }
    features = {}
    x = input
    x = x.unsqueeze(0)
    for name,layer in model._modules.items():
        x = layer(x)
        if name in layers:
            features[layers[name]] = x

    return features

transform = transforms.Compose([transforms.Resize(300),
                              transforms.ToTensor(),
                              transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])

content = Image.open("/mama.jpg").convert("RGB")
content = transform(content).to(device)
print("Content shape => ", content.shape)
style = Image.open("/picasso.jpg").convert("RGB")
style = transform(style).to(device)

def imcnvt(image):
    x = image.to("cpu").clone().detach().numpy().squeeze()
    x = x.transpose(1,2,0)
    x = x*np.array((0.5,0.5,0.5)) + np.array((0.5,0.5,0.5))
    return x

fig, (ax1,ax2) = plt.subplots(1,2)

ax1.imshow(imcnvt(content),label = "Content")
ax2.imshow(imcnvt(style),label = "Style")
plt.show()

def gram_matrix(imgfeature):
    _,d,h,w = imgfeature.size()
    imgfeature = imgfeature.view(d,h*w)
    gram_mat = torch.mm(imgfeature,imgfeature.t())

    return gram_mat

target = content.clone().requires_grad_(True).to(device)

#set device to cuda if available
print("device = ",device)

style_features = model_activations(style,model)
content_features = model_activations(content,model)

style_wt_meas = {"conv1_1" : 1.0,
                 "conv2_1" : 0.8,
                 "conv3_1" : 0.4,
                 "conv4_1" : 0.2,
                 "conv5_1" : 0.1}

style_grams = {layer:gram_matrix(style_features[layer]) for layer in style_features}

content_wt = 100
style_wt = 1e8

print_after = 500
epochs = 4000
optimizer = torch.optim.Adam([target],lr=0.007)

for i in range(1,epochs+1):
    target_features = model_activations(target,model)
    content_loss = torch.mean((content_features['conv4_2']-target_features['conv4_2'])**2)

    style_loss = 0
    for layer in style_wt_meas:
        style_gram = style_grams[layer]
        target_gram = target_features[layer]
        _,d,w,h = target_gram.shape
        target_gram = gram_matrix(target_gram)

        style_loss += (style_wt_meas[layer]*torch.mean((target_gram-style_gram)**2))/d*w*h

    total_loss = content_wt*content_loss + style_wt*style_loss

    if i%10==0:
        print("epoch ",i," ", total_loss)

    optimizer.zero_grad()
    total_loss.backward()
    optimizer.step()

    if i%print_after == 0:
        plt.imshow(imcnvt(target),label="Epoch "+str(i))
        plt.show()
        plt.imsave(str(i)+'.png',imcnvt(target),format='png')


title: How can I use Librispeech 360h dataset in google-colab?

body:
I'm using Google colab (GPU Enabled) to train my ASR model. The accuracy of the model using the train-clean-100 Librispeech dataset is not great, so i decided to download the train-clean-360 dataset using : torchaudio.datasets.LIBRISPEECH("./data", url='train-clean-360', download=True)
But i can't download it since the disk storage available in google-colab is 36GB
I tried to upload and unzip the train-clean-360 dataset in my google-drive account and mount it to google-colab so i can use the dataset without downloading it with this line of code:
torchaudio.datasets.LIBRISPEECH("/content/gdrive/My Drive/", url="train-clean-360", folder_in_archive="LibriSpeech", download=False)

but it's not working.
Am I doing something wrong?
Is there any other way to use this dataset in google-colab ?


title: Jetson Nano PyTorch Vision v0.7.0-rc2 Installation Error

body:
I am trying to install Pytorch and Pytorch Vision as follows:
pip3 install numpy torch-1.6.0rc2-cp36-cp36m-linux_aarch64.whl

git clone --branch  v0.7.0-rc2 https://github.com/pytorch/vision torchvision

sudo python3 setup.py install

It gives me an error message: "Segmentation fault" and does not install pytorch vision, but Torch 1.6.0 is installed properly.
My environment is:

CUDA 10.2
cuDNN version: 8000

Can anybody help me install pytorch vision?


title: How do I create a scale matrix for rescaling a PyTorch tensor, and then how do I use it?

body:
I need to create a scale matrix that is autograd compatible, works on B,C,H,W tensors, and takes input values (possibly generated randomly) for controlling the scaling. How can I generate and use a scale matrix for this?
import torch
import torch.nn.functional as F
import torchvision.transforms as transforms
from PIL import Image

# Load image
def preprocess_simple(image_name, image_size):
    Loader = transforms.Compose([transforms.Resize(image_size), transforms.ToTensor()])
    image = Image.open(image_name).convert('RGB')
    return Loader(image).unsqueeze(0)

# Save image
def deprocess_simple(output_tensor, output_name):
    output_tensor.clamp_(0, 1)
    Image2PIL = transforms.ToPILImage()
    image = Image2PIL(output_tensor.squeeze(0))
    image.save(output_name)

def get_scale_mat(theta):
    ...
    return scale_mat

def scale_img(x, theta, dtype):
    scale_mat = get_scale_mat(theta)

    # Can F.affine_grid & F.grid_sample be used with a scale matrix?
    grid = F.affine_grid(scale_mat , x.size()).type(dtype)
    x = F.grid_sample(x, grid)
    return x

# Shear tensor
test_input = # Test image
scale = 5 # Example value
scaled_tensor = scale_img(test_input, scale)


title: Trying to accumulate gradients in Pytorch, but getting RuntimeError when calling loss.backward

body:
I'm trying to train a model in Pytorch, and I'd like to have a batch size of 8, but due to memory limitations, I can only have a batch size of at most 4. I've looked all around and read a lot about accumulating gradients, and it seems like the solution to my problem.
However, I seem to have trouble implementing it. Every time I run the code I get RuntimeError: Trying to backward through the graph a second time. I don't understand why since my code looks like all these other examples I've seen (unless I'm just missing something major):

https://stackoverflow.com/a/62076913/1227353
https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255
https://discuss.pytorch.org/t/why-do-we-need-to-set-the-gradients-manually-to-zero-in-pytorch/4903/20

One caveat is that the labels for my images are all different size, so I can't send the output batch and the label batch into the loss function; I have to iterate over them together. This is what an epoch looks like (it's been pared down for the sake of brevity):
  # labels_batch contains labels of different sizes
  for batch_idx, (inputs_batch, labels_batch) in enumerate(dataloader):
    outputs_batch = model(inputs_batch)

    # have to do this because labels can't be stacked into a tensor
    for output, label in zip(outputs_batch, labels_batch):
      output_scaled = interpolate(...)  # make output match label size
      loss = train_criterion(output_scaled, label) / (BATCH_SIZE * 2)
      loss.backward()

    if batch_idx % 2 == 1:
      optimizer.step()
      optimizer.zero_grad()

Is there something I'm missing? If I do the following I also get an error:
  # labels_batch contains labels of different sizes
  for batch_idx, (inputs_batch, labels_batch) in enumerate(dataloader):
    outputs_batch = model(inputs_batch)

    # CHANGE: we're gonna accumulate losses manually
    batch_loss = 0

    # have to do this because labels can't be stacked into a tensor
    for output, label in zip(outputs_batch, labels_batch):
      output_scaled = interpolate(...)  # make output match label size
      loss = train_criterion(output_scaled, label) / (BATCH_SIZE * 2)
      batch_loss += loss # CHANGE: accumulate!

    # CHANGE: do backprop outside for loop
    batch_loss.backward()

    if batch_idx % 2 == 1:
      optimizer.step()
      optimizer.zero_grad()

The error I get in this case is RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn. This happens when the next epoch starts though...   (INCORRECT, SEE EDIT BELOW)
How can I train my model with gradient accumulation? Or am I doomed to train with a batch size of 4 or less?
Oh and as a side question, does the location of where I put loss.backward() affect what I need to normalize the loss by? Or is it always normalized by BATCH_SIZE * 2?
EDIT:
The second code segment was getting an error due to the fact that I was doing torch.set_grad_enabled(phase == 'train') but I had forgotten to wrap the call to batch_loss.backward() with an if phase == 'train'... my bad
So now the second segment of code seems to work and do gradient accumulation, but why doesn't the first bit of code work? It feel equivalent to setting BATCH_SIZE as 1. Furthermore, I'm creating a new loss object each time, so shouldn't the calls to backward() operate on different graphs entirely?


title: Pytorch lightning - amp + cpu is not supported

body:
I'm having trouble with Pytorch Lightning when i fit a model. Raises a misconfiguration exception saying this: "raise MisconfigurationException('amp + cpu is not supported.  Please use a GPU option')"
Some information:

Python version: 3.9
GPU: GTX 1660 Super. I've installed the new CUDA toolkit, but insists on define that the GPU cores are not available.

Could you help me?
Thanks a lot.
Cheers


title: Why does my pytorch NN return a tensor of nan?

body:
I have a quite simple neural network which takes a flattened 6x6 grid as input and should output the values of four actions to take on that grid, so a 1x4 tensor of values.
Sometimes after a few runs though for some reason I am getting a 1x4 tensor of nan
tensor([[nan, nan, nan, nan]], grad_fn=<ReluBackward0>)

My model looks like this with input dim being 36 and output dim being 4:
class Model(nn.Module):
    def __init__(self, input_dim, output_dim):
        # super relates to nn.Module so this initializes nn.Module
        super(Model, self).__init__()
        # Gridsize as input,
        # last layer needs 4 outputs because of 4 possible actions: left, right, up, down
        # output values are Q Values need activation function for those like argmax
        self.lin1 = nn.Linear(input_dim, 24)
        self.lin2 = nn.Linear(24, 24)
        self.lin3 = nn.Linear(24, output_dim)

    # function to feed the input through the net
    def forward(self, x):
        # rectified linear as activation function for the first two layers
        if isinstance(x, np.ndarray):
            x = torch.tensor(x, dtype=torch.float)

        activation1 = F.relu(self.lin1(x))
        activation2 = F.relu(self.lin2(activation1))
        output = F.relu(self.lin3(activation2))

        return output

The input was:
tensor([[0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 1.0000, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3333,
         0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.3333, 0.0000, 0.0000, 0.0000,
         0.0000, 0.0000, 0.3333, 0.0000, 0.0000, 0.0000, 0.0000, 0.0000, 0.6667]])

What are possible causes for a nan output and how can i fix those?


title: Dataloading in PyTorch using custom dataset

body:
I am trying to use the KAIST dataset in PyTorch to train a model and generate weights. I am new to both python and PyTorch and I am struggling using dataloaders with such a dataset. The dataset comes with png image files but the annotations (labels and bounding boxes) are in a .vbb format which i am having trouble converting to a usable form (i.e. csv, text file, etc.). So I guess my first question is, how would I convert the annotations to a format that I can iterate the png file with their associate annotations. The second question is, how would I create the dataloader for this dataset in PyTorch.

Any advice would be greatly appreciated.

Link to dataset - https://github.com/SoonminHwang/rgbt-ped-detection/tree/master/data

There is a method to convert the annotations provided in the link above but I have been unable to get it to produce the correct results. Perhaps there is a simpler method for conversion via python?


title: Using pypi pretrained models vs PyTorch

body:
I have two setups - one takes approx. 10 minutes to run the other is still going after an hour:

10 m:

import pretrainedmodels

def resnext50_32x4d(pretrained=False):
    pretrained = 'imagenet' if pretrained else None
    model = pretrainedmodels.se_resnext50_32x4d(pretrained=pretrained)
    return nn.Sequential(*list(model.children()))

learn = cnn_learner(data, resnext50_32x4d, pretrained=True, cut=-2, split_on=lambda m: (m[0][3], m[1]),metrics=[accuracy, error_rate])

Not finishing:

import torchvision.models as models

def get_model(pretrained=True, model_name = 'resnext50_32x4d', **kwargs ):
    arch = models.resnext50_32x4d(pretrained, **kwargs )
    return arch

learn = Learner(data, get_model(), metrics=[accuracy, error_rate])

This is all copied and hacked from other peoples code so there are parts that I do not understand. But the most perplexing one is why one would be so much faster than the other. I would like to use the second option because its easier for me to understand and I can just swap out the pretrained model to test different ones.


title: How to install torch with conda on window?

body:
I am getting this one issue you can check in picture:

Can you suggest me any solid solution?

I am new here please give me a solution rather than checking my question style. Thank you in advance


title: How to improve code to speed up word embedding with transformer models?

body:
I need to compute words embeddings for a bunch of documents with different language models.
No problem with that, the script is doing fine, except I'm working on a notebook, without GPU and each text needs around 1.5s to be processed which is by far too long (I have thousands of texts to process).

Here is how I'm doing it with pytorch and transformers lib:

import torch
from transformers import CamembertModel, CamembertTokenizer

docs = [text1, text2, ..., text20000]
tok = CamembertTokenizer.from_pretrained('camembert-base')
model = CamembertModel.from_pretrained('camembert-base', output_hidden_states=True)
# let try with a batch size of 64 documents
docids = [tok.encode(
  doc, max_length=512, return_tensors='pt', pad_to_max_length=True) for doc in docs[:64]]
ids=torch.cat(tuple(docids))
device = 'cuda' if torch.cuda.is_available() else 'cpu' # cpu in my case...
model = model.to(device)
ids = ids.to(device)
model.eval()
with torch.no_grad():
    out = model(input_ids=ids)
# 103s later...

Do someone has any idea or suggestions to improve speed?


title: Pytorch word embeddings results in nan values

body:
After training a word embedding model on a large-ish corpus, my embeddings converge to nan values.
The model is very simple (skipgram with negative sampling)
full model:
class NEG_loss(nn.Module):
    def __init__(self, vocab_size, embed_size, neg_sampling_table=None):
        """
        :param vocab_size: An int. The number of possible classes.
        :param embed_size: An int. EmbeddingLockup size
        :param num_sampled: An int. The number of sampled from noise examples
        :param neg_sampling_table: A list of non negative floats. Class neg_sampling_table. None if
            using uniform sampling. The neg_sampling_table are calculated prior to
            estimation and can be of any form, e.g equation (5) in [1]
        """

        super(NEG_loss, self).__init__()

        self.device = t.device("cuda:0" if t.cuda.is_available() else "cpu")

        self.vocab_size = vocab_size
        self.embed_size = embed_size

        self.out_embed.weight = nn.Parameter(
            t.cat(
                [
                    t.zeros(1, self.embed_size),
                    FT(self.vocab_size - 1, self.embed_size).uniform_(
                        -0.5 / self.embed_size, 0.5 / self.embed_size
                    ),
                ]
            )
        )

        self.in_embed.weight = nn.Parameter(
            t.cat(
                [
                    t.zeros(1, self.embed_size),
                    FT(self.vocab_size - 1, self.embed_size).uniform_(
                        -0.5 / self.embed_size, 0.5 / self.embed_size
                    ),
                ]
            )
        )

        self.neg_sampling_table = neg_sampling_table
        if self.neg_sampling_table is not None:
            assert min(self.neg_sampling_table) >= 0, "Each weight should be >= 0"

            self.neg_sampling_table = Variable(t.from_numpy(neg_sampling_table)).float()

    # TODO this is bad - find more elegant solution
    def sample(self, num_sample):
        """
        draws a sample from classes based on neg_sampling_table
        """

        return self.neg_sampling_table[
            t.randint(0, len(self.neg_sampling_table), (num_sample,))
        ]

    def forward(self, input_labels, out_labels, num_sampled):
        """
        :param input_labels: Tensor with shape of [batch_size] of Long type
        :param out_labels: Tensor with shape of [batch_size, window_size] of Long type
        :param num_sampled: An int. The number of sampled from noise examples
        :return: Loss estimation with shape of [1]
            loss defined in Mikolov et al. Distributed Representations of Words and Phrases and their Compositionality
            papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf
        """

        input_labels = input_labels.to("cuda:0")
        out_labels = out_labels.to("cuda:0")

        batch_size = out_labels.size()[0]

        input_ = self.in_embed(input_labels.view(-1))
        output = self.out_embed(out_labels.view(-1))

        if self.neg_sampling_table is not None:

            noise_sample_count = batch_size * num_sampled
            draw = self.sample(noise_sample_count)
            noise = draw.view(batch_size, num_sampled).long()

        else:

            noise = Variable(
                t.Tensor(batch_size, num_sampled)
                .uniform_(0, self.vocab_size - 1)
                .long()
            )
        # if use_cuda:
        noise = noise.to(self.device)
        noise = self.out_embed(noise).neg()

        log_target = (input_ * output).sum(1).squeeze().sigmoid().log()

        """ ∑[batch_size * window_size, num_sampled, embed_size] * [batch_size * window_size, embed_size, 1] ->
            ∑[batch_size, num_sampled, 1] -> [batch_size] """
        sum_log_sampled = (
            t.bmm(noise, input_.unsqueeze(2)).sigmoid().log().sum(1).squeeze()
        )

        loss = log_target + sum_log_sampled

        return -loss.mean()

    def input_embeddings(self):
        return self.in_embed.weight.detach().cpu().numpy()

and training loop and optimizer code
    # NEG loss and optim
    neg = NEG_loss(vocab_size, dim, neg_sampling_table=neg_sampling_dist)
    neg.to("cuda:0")
    optimizer = Adam(neg.parameters(), 0.01)

    sys.stdout.write("BEGINNING TRAINING\n")
    for i in range(epochs):
        sys.stdout.write("-" * 35 + "\n")
        sys.stdout.write(f"EPOCH {i+1}\n")
        for i, batch in enumerate(tqdm(dataloader)):
            input_, output_ = batch
            optimizer.zero_grad()
            loss = neg(input_, output_, neg_samples)
            loss.backward()
            optimizer.step()
        sys.stdout.write("-" * 35 + "\n")

This model is about as simple as it gets so I am a bit surprised I am having this issue. Everything is fine on a smaller corpus (7k batches of size 128) but gets nan-y with a larger corpus (200k batches of size 128).
Anyone see anything immediately wrong or have any tips to figure out what is going?
Any help is appreciated. Also pretty new to pytorch - so if you see anything that is a bit dumb feel free to point it out :)


title: How to change BN to GN and FRN layer in efficientnet pytorch?

body:
i was trying groupnormalization for efficientnet in pytorch : https://github.com/lukemelas/EfficientNet-PyTorch ,
my  model code is :
out_dim = 5
enet_type = 'efficientnet-b0'

pretrained_model = {
    'efficientnet-b0': '../input/efficientnet-pytorch/efficientnet-b0-08094119.pth'
}

class enetv2(nn.Module):
    def __init__(self, backbone, out_dim):
        super(enetv2, self).__init__()
        self.enet = enet.EfficientNet.from_name(backbone)
        self.enet.load_state_dict(torch.load(pretrained_model[backbone]))

        self.myfc = nn.Linear(self.enet._fc.in_features, out_dim)
        self.enet._fc = nn.Identity()

    def extract(self, x):
        return self.enet(x)

    def forward(self, x):
        x = self.extract(x)
        x = self.myfc(x)
        return x
model = enetv2(enet_type, out_dim=out_dim)
model = model.to(device)

if i try model.enet._bn0
then it gives me this output :
BatchNorm2d(32, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
but with this code :
for name, module in model.named_modules():
    if isinstance(module, nn.BatchNorm2d):
        # Get current bn layer
        bn = getattr(model, name)
        # Create new gn layer
        gn = nn.GroupNorm(1, bn.num_features)
        # Assign gn
        print('Swapping {} with {}'.format(bn, gn))
        setattr(model, name, gn)

print(model)

i get this error :
AttributeError Traceback (most recent call last)
in
2 if isinstance(module, nn.BatchNorm2d):
3 # Get current bn layer
----> 4 bn = getattr(model, name)
5 # Create new gn layer
6 gn = nn.GroupNorm(1, bn.num_features)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in getattr(self, name)
592 return modules[name]
593 raise AttributeError("’{}’ object has no attribute ‘{}’".format(
–> 594 type(self).name, name))
595
596 def setattr(self, name, value):

AttributeError: ‘enetv2’ object has no attribute ‘enet._bn0’

now how do i replace BN layers with GN layers in my model?
i would also like to know how to change those BN layers with Filter Response Normalization (FRN) layers in my pytorch model


title: Pytorch: explain torch.argmax

body:
Hello I have the following code:
import torch
x = torch.zeros(1,8,4,576) # create a 4 dimensional tensor
x[0,4,2,333] = 1.0 # put on 1 on a random spot

# I want to find the index of the highest value (0,4,2,333)
print(x.argmax()) # this should return the index

This returns

tensor(10701)

How does this 10701 make sense?
How do I get the actual indices 0,4,2,333 ?


title: Iteration on images with Pytorch: error due to CUDA memory issue with batch size 1

body:
During training, the architecture generates three models and now encoder is used to encode images with iterations=16. After performing 6 iteration, i got an error. "CUDA out of memory". I have 4 gpus and i also implemented dataparallel class but it does not work
Transformation while training
train_transform = transforms.Compose([
#transforms.RandomCrop((32, 32)),
transforms.ToPILImage(),
transforms.Resize((512, 512)),
#transforms.ColorJitter(brightness=0.5),
transforms.ToTensor(),
])

Encoder.py file
import argparse

import numpy as np
from scipy.misc import imread, imresize, imsave

import torch
from torch.autograd import Variable

parser = argparse.ArgumentParser()
parser.add_argument(
    '--model', '-m', required=True, type=str, help='path to model')
parser.add_argument(
    '--input', '-i', required=True, type=str, help='input image')
parser.add_argument(
    '--output', '-o', required=True, type=str, help='output codes')
parser.add_argument('--cuda', '-g', action='store_true', help='enables cuda')
parser.add_argument(
    '--iterations', type=int, default=16, help='unroll iterations')
args = parser.parse_args()

image = imread(args.input, mode='RGB')
image = torch.from_numpy(
    np.expand_dims(
        np.transpose(image.astype(np.float32) / 255.0, (2, 0, 1)), 0))
batch_size, input_channels, height, width = image.size()
assert height % 32 == 0 and width % 32 == 0

image = Variable(image, volatile=True)

torch.cuda.empty_cache()

import network

encoder = network.EncoderCell()
binarizer = network.Binarizer()
decoder = network.DecoderCell()

encoder.eval()
binarizer.eval()
decoder.eval()

print("Khawar", torch.cuda.current_device())
#torch.cuda.set_device(2)

#print("Khawar", torch.cuda.current_device())

encoder.load_state_dict(torch.load(args.model))
binarizer.load_state_dict(
    torch.load(args.model.replace('encoder', 'binarizer')))
decoder.load_state_dict(torch.load(args.model.replace('encoder', 'decoder')))

encoder_h_1 = (Variable(
    torch.zeros(batch_size, 256, height // 4, width // 4), volatile=True),
               Variable(
                   torch.zeros(batch_size, 256, height // 4, width // 4),
                   volatile=True))
encoder_h_2 = (Variable(
    torch.zeros(batch_size, 512, height // 8, width // 8), volatile=True),
               Variable(
                   torch.zeros(batch_size, 512, height // 8, width // 8),
                   volatile=True))
encoder_h_3 = (Variable(
    torch.zeros(batch_size, 512, height // 16, width // 16), volatile=True),
               Variable(
                   torch.zeros(batch_size, 512, height // 16, width // 16),
                   volatile=True))

decoder_h_1 = (Variable(
    torch.zeros(batch_size, 512, height // 16, width // 16), volatile=True),
               Variable(
                   torch.zeros(batch_size, 512, height // 16, width // 16),
                   volatile=True))
decoder_h_2 = (Variable(
    torch.zeros(batch_size, 512, height // 8, width // 8), volatile=True),
               Variable(
                   torch.zeros(batch_size, 512, height // 8, width // 8),
                   volatile=True))
decoder_h_3 = (Variable(
    torch.zeros(batch_size, 256, height // 4, width // 4), volatile=True),
               Variable(
                   torch.zeros(batch_size, 256, height // 4, width // 4),
                   volatile=True))
decoder_h_4 = (Variable(
    torch.zeros(batch_size, 128, height // 2, width // 2), volatile=True),
               Variable(
                   torch.zeros(batch_size, 128, height // 2, width // 2),
                   volatile=True))

if args.cuda:
    encoder = encoder.cuda()
    binarizer = binarizer.cuda()
    decoder = decoder.cuda()

    image = image.cuda()

    encoder_h_1 = (encoder_h_1[0].cuda(), encoder_h_1[1].cuda())
    encoder_h_2 = (encoder_h_2[0].cuda(), encoder_h_2[1].cuda())
    encoder_h_3 = (encoder_h_3[0].cuda(), encoder_h_3[1].cuda())

    decoder_h_1 = (decoder_h_1[0].cuda(), decoder_h_1[1].cuda())
    decoder_h_2 = (decoder_h_2[0].cuda(), decoder_h_2[1].cuda())
    decoder_h_3 = (decoder_h_3[0].cuda(), decoder_h_3[1].cuda())
    decoder_h_4 = (decoder_h_4[0].cuda(), decoder_h_4[1].cuda())

codes = []
res = image - 0.5
for iters in range(args.iterations):
    encoded, encoder_h_1, encoder_h_2, encoder_h_3 = encoder(
        res, encoder_h_1, encoder_h_2, encoder_h_3)

    code = binarizer(encoded)

    output, decoder_h_1, decoder_h_2, decoder_h_3, decoder_h_4 = decoder(
        code, decoder_h_1, decoder_h_2, decoder_h_3, decoder_h_4)

    res = res - output
    codes.append(code.data.cpu().numpy())

    print('Iter: {:02d}; Loss: {:.06f}'.format(iters, res.data.abs().mean()))

codes = (np.stack(codes).astype(np.int8) + 1) // 2

export = np.packbits(codes.reshape(-1))

np.savez_compressed(args.output, shape=codes.shape, codes=export)[!

Error image


title: Artifacts in StyleGAN generated images

body:
I've written my own implementation of StyleGAN (paper here https://arxiv.org/abs/1812.04948), using PyTorch instead of Tensorflow, which is what the official implementation uses. I'm doing this partly as an exercise in implementing a scientific paper from scratch.
I have done my best to reproduce all the features mentioned in the paper and in the ProgressiveGAN paper which it is based on, and the network trains, but I consistently get blurry images and blob-shaped artifacts:

I would very much like to know if anyone with experience of GANs in general or StyleGAN in particular has seen this phenomenon and can give me any insight into possible reasons for it.
(Some detail: I'm training on downsampled CelebA images, 600k images burn-in, 600k images fade-in, but I see very similar phenomena with a tiny toy dataset and a lot fewer iterations.)


title: pytorch - how to troubleshoot device (cpu \ gpu) settings of tensors \ models

body:
I have a torch model that i'm trying to port from CPU do be device independent.
setting the device parameter when creating tensors, or calling model.to(device) to move a full model to the target device solves part of the problem, but there are some "left behind" tensors (like variables created during the forward call)
is there a way to detect these without using an interactive debugger?
something like tracing tensor creation to allow discover of tensors that are created on the wrong device?


title: Model exported from python gives different results when loaded in C++

body:
Problem description
So I have model class written in python using PyTorch:
class ResBlock(nn.Module, ABC):
    def __init__(self, in_channels: int, filters: int, conv_num: int):
        super(ResBlock, self).__init__()
        self.filters = filters
        self.conv_num = conv_num

        self.input_conv = torch.nn.Conv1d(in_channels=in_channels, out_channels=filters, kernel_size=1)
        self.inner_conv = torch.nn.Conv1d(in_channels=in_channels, out_channels=filters, kernel_size=3, padding=1)
        self.outer_conv = torch.nn.Conv1d(in_channels=filters, out_channels=filters, kernel_size=3, padding=1)
        self.max_pool = torch.nn.MaxPool1d(kernel_size=2, stride=2)

    def forward(self, x):
        y = x

        for i in range(self.conv_num - 1):
            if i == 0:
                y = self.inner_conv(y)
            else:
                y = self.outer_conv(y)

            y = torch.relu(y)

        y = self.outer_conv(y)

        s = self.input_conv(x)
        y = s + y
        y = torch.relu(y)

        return self.max_pool(y)

class Net(nn.Module, ABC):
    def __init__(self, num_of_classes: int):
        super(Net, self).__init__()
        self.block_1 = ResBlock(1, 16, 2)
        self.block_2 = ResBlock(16, 32, 2)
        self.block_3 = ResBlock(32, 64, 3)
        self.block_4 = ResBlock(64, 128, 3)
        self.block_5 = ResBlock(128, 128, 3)

        self.avg_pool = torch.nn.AvgPool1d(kernel_size=3, stride=3)
        self.flatten = torch.nn.Flatten()
        self.dense_1 = torch.nn.Linear(
            in_features=self.block_5.filters * (249 // self.avg_pool.kernel_size[0]),
            out_features=256
        )
        self.dense_2 = torch.nn.Linear(in_features=256, out_features=128)
        self.classifier = torch.nn.Linear(in_features=128, out_features=num_of_classes)

    def forward(self, x):
        x = self.block_1(x)
        x = self.block_2(x)
        x = self.block_3(x)
        x = self.block_4(x)
        x = self.block_5(x)

        x = self.avg_pool(x)
        x = self.flatten(x)

        x = self.dense_1(x)
        x = self.dense_2(x)
        x = self.classifier(x)

        return x

It works perfectly fine when using in the python environment. I have trained it, acquired ~65% of accuracy on the test set and wanted to export it using TorchScript, then import in the C++ app.
Here is the code used for exporting:
# Training code skipped for simplification...
# Here the model is actually trained, weights are updated and so on

jit_model = torch.jit.script(model)
jit_model.save('torchscript-model.pt')

Before importing it to the C++ I've checked if the model exported correctly by importing it to the python script using torch.jit.load and checked it against my test dataset again to get the same ~65% of accuracy which was expected.
So the next logical step was to import the model into the C++ program, and evaluate the loaded model using the same data. Here is the code:
#include <torch/torch.h>
#include <torch/script.h>
#include "constants.hh"

int main() {
    torch::manual_seed(1);
    torch::Device device(torch::kCPU);

    auto model = torch::jit::load("./torchscript-model.pt");
    model.to(device);

    auto test_raw_dataset = CsvDataset(constants::kTestCsv);
    auto test_dataset = test_raw_dataset.map(torch::data::transforms::Stack<>());
    auto test_data_loader = torch::data::make_data_loader<torch::data::samplers::SequentialSampler>(std::move(test_dataset), torch::data::DataLoaderOptions(constants::kBatchSize));

    size_t correct_count = 0;
    for (const auto& batch : *test_data_loader) {
        auto inputs = batch.data.to(device);
        auto labels = batch.target.to(device);

        inputs = inputs.unsqueeze(1);
        labels = labels.squeeze(1);

        auto outputs = model.forward(inputs).toTensor();

        auto prediction = outputs.argmax(1);
        correct_count += prediction.eq(labels).sum().item<int64_t>();
    }

    auto accuracy = correct_count / test_dataset.size().value();
    // Rest of code removed for clarity...
}

Actual question
But the thing that happens is the accuracy calculated in the C++ is equal to ~12%. Why?
What have I tried/found so far:

All of the predictions when model is loaded into C++ are the same (equal to 6). When model is loaded in python everything is good.
All of the calculations after acquiring outputs are the same in Python and C++, so problem is not in the way I'm calculating accuracy.
Using torch::manual_seed does nothing. (It should not change anything, but why not to try...)
torch::NoGradGuard also does nothing.
I have also ensured that the test data gets into the model in the same order in Python and in C++.
Model is running in the eval mode in both cases

Update 1
Checked loaded model against static input using torch.ones and python and c++ still yield different results.
Update 2
Python code as @gspr requested:
dataset = CsvDataset(os.path.join(constants.CSV_DIR, 'train.csv'))
sampler = torch.utils.data.sampler.RandomSampler(dataset)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=constants.BATCH_SIZE, sampler=sampler)
train_test_size = len(dataset)
batch_count = math.ceil(train_test_size / constants.BATCH_SIZE)

test_dataset = CsvDataset(os.path.join(constants.CSV_DIR, 'test.csv'))
test_dataloader = torch.utils.data.DataLoader(test_dataset, batch_size=constants.BATCH_SIZE)
test_set_size = len(test_dataset)

model = Net(dataset.num_classes())

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=constants.LEARNING_RATE)

for epoch in range(constants.EPOCHS):
    running_loss = 0.0
    correct_count = 0

    for i, data in enumerate(dataloader, 1):
        inputs, labels = data
        inputs = inputs.unsqueeze(1)
        labels = labels.squeeze(1)

        outputs = model.forward(inputs)

        loss = criterion(outputs, labels)
        running_loss += loss.item()
        prediction = outputs.argmax(1)
        correct_count += sum(prediction.eq(labels)).item()

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

test_loss = 0.0
test_correct = 0.0

with torch.no_grad():
    for i, data in enumerate(test_dataloader, 1):
        inputs, labels = data
        inputs = inputs.unsqueeze(1)
        labels = labels.squeeze(1)

        outputs = model.forward(inputs)

        loss = criterion(outputs, labels)
        test_loss += loss.item()
        prediction = outputs.argmax(1)
        test_correct += sum(prediction.eq(labels)).item()

    test_sample_loss = test_loss / test_set_size
    test_accuracy = test_correct / test_set_size

script_module = torch.jit.script(model)
torch.jit.save(script_module, 'traced_model.torch')


title: pytorch multilabel classification network not training

body:
I'm trying a simple multi label classification example but the network does not seem to be training correctly as the loss is stagnant.
I've used multilabel_soft_margin_loss as the pytorch docs suggest, but there isn't much else to go on..can't find any proper examples in the docs.
Can anyone peer into this and point out whats wrong with it? Fully working example below (also question on prediction below)
Fully working example code
from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.optim.lr_scheduler import StepLR
from sklearn.datasets import make_multilabel_classification
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split
import xgboost as xgb
from sklearn.metrics import accuracy_score
num_classes = 3
X, y = make_multilabel_classification(n_samples=1000,n_classes=num_classes)
X_tensor, y_tensor = torch.tensor(X), torch.tensor(y)
print("X Shape :{}".format(X_tensor.shape))
print("y Shape :{}".format(y_tensor.shape))

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(X.shape[1], 300)
        self.fc2 = nn.Linear(300, 10)
        self.fc3 = nn.Linear(10, num_classes)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

device = torch.device("cpu")
lr = 1
batch_size = 128
gamma = 0.9
epochs = 100
args = {'log_interval': 10, 'dry_run':False}
kwargs = {'batch_size': batch_size}
kwargs.update({'num_workers': 1,
               'pin_memory': True,
               'shuffle': True},
                 )

model = Net().to(device)
optimizer = optim.Adam(model.parameters(), lr=lr, weight_decay=0.1)
scheduler = StepLR(optimizer, step_size=1, gamma=gamma)

# data loader
my_dataset = TensorDataset(X_tensor,y_tensor) # create tensor dataset

train_dataset, test_dataset, = train_test_split(
    my_dataset, test_size=0.2, random_state=42)

train_loader = DataLoader(train_dataset,**kwargs)
test_loader = DataLoader(test_dataset,**kwargs)

## Train step ##
for epoch in range(1, epochs + 1):
    model.train() # set model to train

    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data.float())
        loss = F.multilabel_soft_margin_loss(output,target)
        loss.backward()
        optimizer.step()
        if batch_idx % args['log_interval'] == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))
            if args['dry_run']:
                break

    scheduler.step()

Training Loss Progress
Train Epoch: 1 [0/800 (0%)] Loss: 0.694400
Train Epoch: 2 [0/800 (0%)] Loss: 0.697095
Train Epoch: 3 [0/800 (0%)] Loss: 0.705593
Train Epoch: 4 [0/800 (0%)] Loss: 0.651981
Train Epoch: 5 [0/800 (0%)] Loss: 0.704895
Train Epoch: 6 [0/800 (0%)] Loss: 0.650302
Train Epoch: 7 [0/800 (0%)] Loss: 0.658809
Train Epoch: 8 [0/800 (0%)] Loss: 0.904834
Train Epoch: 9 [0/800 (0%)] Loss: 0.655516
Train Epoch: 10 [0/800 (0%)]    Loss: 0.662808
Train Epoch: 11 [0/800 (0%)]    Loss: 0.664752
Train Epoch: 12 [0/800 (0%)]    Loss: 0.656390
Train Epoch: 13 [0/800 (0%)]    Loss: 0.664982
Train Epoch: 14 [0/800 (0%)]    Loss: 0.664430
Train Epoch: 15 [0/800 (0%)]    Loss: 0.664603 # stagnates

On top of that, how would I obtain predictions for this? It's not the same as taking the argmax anymore as its a multi label problem right? (Example output of the network below)
Output
tensor([[ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354],
    [ 0.2711,  0.1754, -0.3354]]

Thanks!


title: Gradients returning None in huggingface module

body:
I want to get the gradient of an embedding layer from a pytorch/huggingface model. Here's a minimal working example:
from transformers import pipeline

nlp = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

responses = ["I'm having a great day!!"]
hypothesis_template = 'This person feels {}'
candidate_labels = ['happy', 'sad']
nlp(responses, candidate_labels, hypothesis_template=hypothesis_template)

I can extract the logits just fine,
inputs = nlp._parse_and_tokenize(responses, candidate_labels, hypothesis_template)
predictions = nlp.model(**inputs, return_dict=True, output_hidden_states=True)
predictions['logits']

and the model returns a layer I'm interested in. I tried to retain the gradient and backprop with respect to a single logit I'm interested in:
layer = predictions['encoder_hidden_states'][0]
layer.retain_grad()
predictions['logits'][0][2].backward(retain_graph=True)

However, layer.grad == None no matter what I try. The other named parameters of the model have their gradients computed, so I'm not sure what I'm doing wrong. How do I get the grad of the encoder_hidden_states?


title: Extract object features from Detectron2

body:
I am really new to object detection, so sorry if this question seems too obvious.
I have a trained FasterRCNN model on detectron2 that detects objects, and I am trying to extract the features of each detected object for the output of my model's prediction. The tutorials available seems to look for ROI and make new predictions, along with their boxes and features. I have the boxes from inference, I just need to extract the features of each box. I added the code I've been working with below. Thank you
# Preprocessing
images = predictor.model.preprocess_image(inputs)  # don't forget to preprocess
# Run Backbone Res1-Res4
features = predictor.model.backbone(images.tensor)  # set of cnn features

#get proposed boxes + rois + features + predictions
# Run RoI head for each proposal (RoI Pooling + Res5)
proposal_boxes = [x.proposal_boxes for x in proposals]
features = [features[f] for f in predictor.model.roi_heads.in_features]
proposal_rois = predictor.model.roi_heads.box_pooler(features, proposal_boxes)
box_features = predictor.model.roi_heads.box_head(proposal_rois)
predictions = predictor.model.roi_heads.box_predictor(box_features)#found here: https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html
pred_instances, pred_inds = predictor.model.roi_heads.box_predictor.inference(predictions, proposals)
pred_instances = predictor.model.roi_heads.forward_with_given_boxes(features, pred_instances)

# output boxes, masks, scores, etc
pred_instances = predictor.model._postprocess(pred_instances, inputs, images.image_sizes)  # scale box to orig size
# features of the proposed boxes
feats = box_features[pred_inds]
proposal_boxes = proposals[0].proposal_boxes[pred_inds]


title: Pytorch MNIST autoencoder to learn 10-digit classification

body:
I'm trying to build a simple autoencoder for MNIST, where the middle layer is just 10 neurons. My hope is that it will learn to classify the 10 digits, and I assume that would lead to the lowest error in the end (wrt reproducing the original image).
I have the following code, which I've already played around with a fair amount. If I run it for up-to 100 epochs, the loss doesn't really go below 1.0, and if I evaluate it, it's obviously not working. What am I missing?
Training:
import torch
import torchvision as tv
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
from torchvision.utils import save_image

num_epochs = 100
batch_size = 64

transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])
trainset = tv.datasets.MNIST(root='./data',  train=True, download=True, transform=transform)
dataloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=4)

class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder,self).__init__()
        self.encoder = nn.Sequential(
            # 28 x 28
            nn.Conv2d(1, 4, kernel_size=5),
            nn.Dropout2d(p=0.2),
            # 4 x 24 x 24
            nn.ReLU(True),
            nn.Conv2d(4, 8, kernel_size=5),
            nn.Dropout2d(p=0.2),
            # 8 x 20 x 20 = 3200
            nn.ReLU(True),
            nn.Flatten(),
            nn.Linear(3200, 10),
            nn.ReLU(True),
            # 10
            nn.Softmax(),
            # 10
            )
        self.decoder = nn.Sequential(
            # 10
            nn.Linear(10, 400),
            nn.ReLU(True),
            # 400
            nn.Unflatten(1, (1, 20, 20)),
            # 20 x 20
            nn.Dropout2d(p=0.2),
            nn.ConvTranspose2d(1, 10, kernel_size=5),
            # 24 x 24
            nn.ReLU(True),
            nn.Dropout2d(p=0.2),
            nn.ConvTranspose2d(10, 1, kernel_size=5),
            # 28 x 28
            nn.ReLU(True),
            nn.Sigmoid(),
            )
    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = Autoencoder().cpu()
distance = nn.MSELoss()
#optimizer = torch.optim.Adam(model.parameters(), weight_decay=1e-5)
optimizer = torch.optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

for epoch in range(num_epochs):
    for data in dataloader:
        img, _ = data
        img = Variable(img).cpu()
        output = model(img)
        loss = distance(output, img)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    print('epoch [{}/{}], loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

Already the training loss indicates that the thing is not working, but printing out the confusion matrix (which in this case should not necessarily be the identity matrix, since the neurons can be ordered arbitrarily, but should be row-col-reordarable and approximate the identity, if this would work):
import numpy as np

confusion_matrix = np.zeros((10, 10))

batch_size = 20*1000

testset = tv.datasets.MNIST(root='./data',  train=False, download=True, transform=transform)
dataloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=True, num_workers=4)

for data in dataloader:
    imgs, labels = data
    imgs = Variable(imgs).cpu()
    encs = model.encoder(imgs).detach().numpy()
    for i in range(len(encs)):
        predicted = np.argmax(encs[i])
        actual = labels[i]
        confusion_matrix[actual][predicted] += 1
print(confusion_matrix)


title: How can I calculate the grad of this higher dimensional matrix in pytorch?

body:
I got a simple example of using the autograd to do gradient descent using this example from the pytorch docs. Here's my simple working example:

s = np.linspace(-1,1,100)
#The "data"
x = torch.tensor(np.stack((s**2,s**3,s),axis=1), device=device, dtype=dtype)
v_coeffs = np.array([5,-2,4])
#The "target"
v = torch.tensor(np.expand_dims(v_coeffs[0]*s + v_coeffs[1]*s**2 + v_coeffs[2]*s**3,axis=1), device=device, dtype=dtype)
#The weights to determine
w = torch.randn(3, 1, device=device, dtype=dtype, requires_grad=True)
alpha = 10**-4

for t in range(t_range):
    vhat = x.mm(w)
    J = (v - vhat).pow(2).sum()
    J.backward()

    with torch.no_grad():

        w -= alpha*w.grad
        w.grad.zero_()

And it works. If I plot the values of w, I see that they approach the coefficients of the "target" v:

convergence of weights picture

For that example, x.shape = (100,3), v.shape = (100,1), w.shape = (3,1). So x.mm(w).shape = (100,1).

Now I'm trying to do the same thing but my matrix of weights has more dimensions: its shape is W.shape = (40,40,2). My X here will be a batch of (40,40,2) matrices, so it will have X.shape = (N,40,40,2).

Like my simple example above, I want to get a tensor of shape (N,1) by multiplying X*W. I can't figure out how to do it with a built in torch function; I've tried mm(), mul(), matmul(), and bmm(), and none of them seem to do what I want. Most of those throw errors relating to the sizes being different except mul(), but that gives back the same size tensor as X.

I tried to do it in this hack-y way, using list comprehension and doing it element-wise:

Q = torch.Tensor([torch.sum(x*W) for x in X])

but then it throws an error when I calculate J.backward() using Q:

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

I don't know what this means, but I suspect it's because I did the list comp and now Q doesn't "know about" W and its requires_grad() being true. In the simple working example, vhat is directly calculated from w so it seems like J "knows" that w requires grad.

How can I do this?

edit: I figured out that I can do this:

Q = torch.sum(X.mul(W),dim=[1,2,3])

but it still seems like a workaround...


title: Why isn't my RL model behaving the same after being loaded in pytorch?

body:
I'm training some simple neural networks for Reinforcement Learning in Pytorch. At the end of the training, I save the model like so:

torch.save(self.policy_NN.state_dict(), self.model_fname)

It's doing pretty good at this point. Then later, in another script, I load it again, like so:

self.policy_NN.load_state_dict(torch.load(model_fname))

And then just play out the episode, as if the training never stopped (except I'm not doing DQN learning anymore, it's just taking the greedy action at each point). So I'd expect it to behave basically as it did when I saved it.

However, whenever I load it, it behaves completely differently, to the point that it seems like it didn't learn at all before I saved it. For example, if I look at the last 1000 time steps of the training session, it will get many rewards, but after loading it, it gets basically none.

I've verified (by doing print(self.policy_NN.state_dict())) that the weights and biases are in fact the same when I save the model and when I load it again.

What could be going on? Is there something else to the network that might not be getting saved somehow?


title: How to efficiently normalize group of elements in a tensor

body:
Context:

I am trying to replicate Hinton's "Matrix capsules with EM routing" (https://openreview.net/forum?id=HJWLfGWRb).

At some point, there are convolutional operations that are performed (in the sense that an output tensor is connected to an input tensor, and each element in the output tensor is influenced only by the input elements contained in a 2D mask of size K).

Input
Tensor x of shape w_in,w_in
where

w_in=14

Intermediary tensor mapped to the input
Tensor x_mapped of shape w_out,w_out,K,K
where

K=3 is the convolution kernel size
w_out=6, resulting from convolution with stride=2

Summing on the dimensions 2 and 3 (both of size K) means summing on the input elements connected to an output element whose location is given by dimensions 0 and 1.

Question:

How can I efficiently normalize (to 1) groups of elements in x_mapped, based on their location in the input tensor x?

For example:
x_mapped(0,0,2,2)
x_mapped(1,0,0,2)
x_mapped(0,1,2,0)
x_mapped(1,1,0,0)
are all connected to x(2,2) (the formula is i_out*stride + K_index = i_in). For that reason, I would like the sum of those 4 elements to be 1.

And I would like to do that for all the groups of elements in x_mapped that are "connected" to the same element in x.

I can figure out how to do it by:

Building a dictionary with input location as key and list of output elements as value
Looping on the dictionary, summing the elements in the list for a given input location and dividing them by that sum

but that seems really inefficient to me.