title: input for torch.nn.functional.gumbel_softmax

body:
Say I have a tensor named attn_weights of size [1,a], entries of which indicate the attention weights between the given query and |a| keys. I want to select the largest one using torch.nn.functional.gumbel_softmax.
I find docs about this function describe the parameter as logits - […, num_features] unnormalized log probabilities. I wonder whether should I take log of attn_weights before passing it into gumbel_softmax? And I find Wiki defines logit=lg(p/1-p), which is different from barely logrithm. I wonder which one should I pass to the function?
Further, I wonder how to choose tau in gumbel_softmax, any guidelines?


title: Pytorch Where Does resNet add values?

body:
I am working on ResNet and I have found an implementation that does the skip connections with a plus sign. Like the following
Class Net(nn.Module):
    def __init__(self):
        super(Net, self).__int_()
            self.conv = nn.Conv2d(128,128)

    def forward(self, x):
        out = self.conv(x) // line 1
        x = out + x    // skip connection  // line 2

Now I have debugged and printed the values before and after line 1. The output was the following:

after line 1
x = [1,128,32,32]
out = [1,128,32,32]
After line 2
x = [1,128,32,32] // still

Reference link: https://github.com/kuangliu/pytorch-cifar/blob/bf78d3b8b358c4be7a25f9f9438c842d837801fd/models/resnet.py#L62
My question is where did it add the value ?? I mean after

x = out + x

operation, where has the value been added ?
PS: Tensor format is [batch, channel, height, width].


title: No such file or directory: 'docker': 'docker' when running sagemaker studio in local mode

body:
I try to train a pytorch model on amazon sagemaker studio.

It's working when I use an EC2 for training with:

estimator = PyTorch(entry_point='train_script.py',
                role=role,
                sagemaker_session = sess,
                train_instance_count=1,
                train_instance_type='ml.c5.xlarge',
                framework_version='1.4.0',
                source_dir='.',
                git_config=git_config,
               )
estimator.fit({'stockdata': data_path})

and it's work on local mode in classic sagemaker notebook (non studio) with:

 estimator = PyTorch(entry_point='train_script.py',
                role=role,
                train_instance_count=1,
                train_instance_type='local',
                framework_version='1.4.0',
                source_dir='.',
                git_config=git_config,
               )
estimator.fit({'stockdata': data_path})

But when I use it the same code (with train_instance_type='local') on sagemaker studio it doesn't work and I have the following error: No such file or directory: 'docker': 'docker'

I tried to install docker with pip install but the docker command is not found if use it in terminal


title: Save the best model trained on Faster RCNN (COCO dataset) with Pytorch avoiding to "overfitting"

body:
I am training a Faster RCNN neural network on COCO dataset with Pytorch.

I have followed next tutorial:
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

The training results are as follows:

Epoch: [6]  [  0/119]  eta: 0:01:16  lr: 0.000050  loss: 0.3780 (0.3780)  loss_classifier: 0.1290 (0.1290)  loss_box_reg: 0.1848 (0.1848)  loss_objectness: 0.0239 (0.0239)  loss_rpn_box_reg: 0.0403 (0.0403)  time: 0.6451  data: 0.1165  max mem: 3105
Epoch: [6]  [ 10/119]  eta: 0:01:13  lr: 0.000050  loss: 0.4129 (0.4104)  loss_classifier: 0.1277 (0.1263)  loss_box_reg: 0.2164 (0.2059)  loss_objectness: 0.0244 (0.0309)  loss_rpn_box_reg: 0.0487 (0.0473)  time: 0.6770  data: 0.1253  max mem: 3105
Epoch: [6]  [ 20/119]  eta: 0:01:07  lr: 0.000050  loss: 0.4165 (0.4302)  loss_classifier: 0.1277 (0.1290)  loss_box_reg: 0.2180 (0.2136)  loss_objectness: 0.0353 (0.0385)  loss_rpn_box_reg: 0.0499 (0.0491)  time: 0.6843  data: 0.1265  max mem: 3105
Epoch: [6]  [ 30/119]  eta: 0:01:00  lr: 0.000050  loss: 0.4205 (0.4228)  loss_classifier: 0.1271 (0.1277)  loss_box_reg: 0.2125 (0.2093)  loss_objectness: 0.0334 (0.0374)  loss_rpn_box_reg: 0.0499 (0.0484)  time: 0.6819  data: 0.1274  max mem: 3105
Epoch: [6]  [ 40/119]  eta: 0:00:53  lr: 0.000050  loss: 0.4127 (0.4205)  loss_classifier: 0.1209 (0.1265)  loss_box_reg: 0.2102 (0.2085)  loss_objectness: 0.0315 (0.0376)  loss_rpn_box_reg: 0.0475 (0.0479)  time: 0.6748  data: 0.1282  max mem: 3105
Epoch: [6]  [ 50/119]  eta: 0:00:46  lr: 0.000050  loss: 0.3973 (0.4123)  loss_classifier: 0.1202 (0.1248)  loss_box_reg: 0.1947 (0.2039)  loss_objectness: 0.0315 (0.0366)  loss_rpn_box_reg: 0.0459 (0.0470)  time: 0.6730  data: 0.1297  max mem: 3105
Epoch: [6]  [ 60/119]  eta: 0:00:39  lr: 0.000050  loss: 0.3900 (0.4109)  loss_classifier: 0.1206 (0.1248)  loss_box_reg: 0.1876 (0.2030)  loss_objectness: 0.0345 (0.0365)  loss_rpn_box_reg: 0.0431 (0.0467)  time: 0.6692  data: 0.1276  max mem: 3105
Epoch: [6]  [ 70/119]  eta: 0:00:33  lr: 0.000050  loss: 0.3984 (0.4085)  loss_classifier: 0.1172 (0.1242)  loss_box_reg: 0.2069 (0.2024)  loss_objectness: 0.0328 (0.0354)  loss_rpn_box_reg: 0.0458 (0.0464)  time: 0.6707  data: 0.1252  max mem: 3105
Epoch: [6]  [ 80/119]  eta: 0:00:26  lr: 0.000050  loss: 0.4153 (0.4113)  loss_classifier: 0.1178 (0.1246)  loss_box_reg: 0.2123 (0.2036)  loss_objectness: 0.0328 (0.0364)  loss_rpn_box_reg: 0.0480 (0.0468)  time: 0.6744  data: 0.1264  max mem: 3105
Epoch: [6]  [ 90/119]  eta: 0:00:19  lr: 0.000050  loss: 0.4294 (0.4107)  loss_classifier: 0.1178 (0.1238)  loss_box_reg: 0.2098 (0.2021)  loss_objectness: 0.0418 (0.0381)  loss_rpn_box_reg: 0.0495 (0.0466)  time: 0.6856  data: 0.1302  max mem: 3105
Epoch: [6]  [100/119]  eta: 0:00:12  lr: 0.000050  loss: 0.4295 (0.4135)  loss_classifier: 0.1171 (0.1235)  loss_box_reg: 0.2124 (0.2034)  loss_objectness: 0.0460 (0.0397)  loss_rpn_box_reg: 0.0498 (0.0469)  time: 0.6955  data: 0.1345  max mem: 3105
Epoch: [6]  [110/119]  eta: 0:00:06  lr: 0.000050  loss: 0.4126 (0.4117)  loss_classifier: 0.1229 (0.1233)  loss_box_reg: 0.2119 (0.2024)  loss_objectness: 0.0430 (0.0394)  loss_rpn_box_reg: 0.0481 (0.0466)  time: 0.6822  data: 0.1306  max mem: 3105
Epoch: [6]  [118/119]  eta: 0:00:00  lr: 0.000050  loss: 0.4006 (0.4113)  loss_classifier: 0.1171 (0.1227)  loss_box_reg: 0.2028 (0.2028)  loss_objectness: 0.0366 (0.0391)  loss_rpn_box_reg: 0.0481 (0.0466)  time: 0.6583  data: 0.1230  max mem: 3105
Epoch: [6] Total time: 0:01:20 (0.6760 s / it)
creating index...
index created!
Test:  [ 0/59]  eta: 0:00:15  model_time: 0.1188 (0.1188)  evaluator_time: 0.0697 (0.0697)  time: 0.2561  data: 0.0634  max mem: 3105
Test:  [58/59]  eta: 0:00:00  model_time: 0.1086 (0.1092)  evaluator_time: 0.0439 (0.0607)  time: 0.2361  data: 0.0629  max mem: 3105
Test: Total time: 0:00:14 (0.2378 s / it)
Averaged stats: model_time: 0.1086 (0.1092)  evaluator_time: 0.0439 (0.0607)
Accumulating evaluation results...
DONE (t=0.02s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.643
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.079
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.011
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.096
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.333
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.333
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
Epoch: [7]  [  0/119]  eta: 0:01:16  lr: 0.000050  loss: 0.3851 (0.3851)  loss_classifier: 0.1334 (0.1334)  loss_box_reg: 0.1845 (0.1845)  loss_objectness: 0.0287 (0.0287)  loss_rpn_box_reg: 0.0385 (0.0385)  time: 0.6433  data: 0.1150  max mem: 3105
Epoch: [7]  [ 10/119]  eta: 0:01:12  lr: 0.000050  loss: 0.3997 (0.4045)  loss_classifier: 0.1250 (0.1259)  loss_box_reg: 0.1973 (0.2023)  loss_objectness: 0.0292 (0.0303)  loss_rpn_box_reg: 0.0479 (0.0459)  time: 0.6692  data: 0.1252  max mem: 3105
Epoch: [7]  [ 20/119]  eta: 0:01:07  lr: 0.000050  loss: 0.4224 (0.4219)  loss_classifier: 0.1250 (0.1262)  loss_box_reg: 0.2143 (0.2101)  loss_objectness: 0.0333 (0.0373)  loss_rpn_box_reg: 0.0493 (0.0484)  time: 0.6809  data: 0.1286  max mem: 3105
Epoch: [7]  [ 30/119]  eta: 0:01:00  lr: 0.000050  loss: 0.4120 (0.4140)  loss_classifier: 0.1191 (0.1221)  loss_box_reg: 0.2113 (0.2070)  loss_objectness: 0.0357 (0.0374)  loss_rpn_box_reg: 0.0506 (0.0475)  time: 0.6834  data: 0.1316  max mem: 3105
Epoch: [7]  [ 40/119]  eta: 0:00:53  lr: 0.000050  loss: 0.4013 (0.4117)  loss_classifier: 0.1118 (0.1210)  loss_box_reg: 0.2079 (0.2063)  loss_objectness: 0.0357 (0.0371)  loss_rpn_box_reg: 0.0471 (0.0473)  time: 0.6780  data: 0.1304  max mem: 3105
Epoch: [7]  [ 50/119]  eta: 0:00:46  lr: 0.000050  loss: 0.3911 (0.4035)  loss_classifier: 0.1172 (0.1198)  loss_box_reg: 0.1912 (0.2017)  loss_objectness: 0.0341 (0.0356)  loss_rpn_box_reg: 0.0449 (0.0464)  time: 0.6768  data: 0.1314  max mem: 3105
Epoch: [7]  [ 60/119]  eta: 0:00:39  lr: 0.000050  loss: 0.3911 (0.4048)  loss_classifier: 0.1186 (0.1213)  loss_box_reg: 0.1859 (0.2013)  loss_objectness: 0.0334 (0.0360)  loss_rpn_box_reg: 0.0412 (0.0462)  time: 0.6729  data: 0.1306  max mem: 3105
Epoch: [7]  [ 70/119]  eta: 0:00:33  lr: 0.000050  loss: 0.4046 (0.4030)  loss_classifier: 0.1177 (0.1209)  loss_box_reg: 0.2105 (0.2008)  loss_objectness: 0.0359 (0.0354)  loss_rpn_box_reg: 0.0462 (0.0459)  time: 0.6718  data: 0.1282  max mem: 3105
Epoch: [7]  [ 80/119]  eta: 0:00:26  lr: 0.000050  loss: 0.4125 (0.4067)  loss_classifier: 0.1187 (0.1221)  loss_box_reg: 0.2105 (0.2022)  loss_objectness: 0.0362 (0.0362)  loss_rpn_box_reg: 0.0469 (0.0462)  time: 0.6725  data: 0.1285  max mem: 3105
Epoch: [7]  [ 90/119]  eta: 0:00:19  lr: 0.000050  loss: 0.4289 (0.4068)  loss_classifier: 0.1288 (0.1223)  loss_box_reg: 0.2097 (0.2009)  loss_objectness: 0.0434 (0.0375)  loss_rpn_box_reg: 0.0479 (0.0461)  time: 0.6874  data: 0.1327  max mem: 3105
Epoch: [7]  [100/119]  eta: 0:00:12  lr: 0.000050  loss: 0.4222 (0.4086)  loss_classifier: 0.1223 (0.1221)  loss_box_reg: 0.2101 (0.2021)  loss_objectness: 0.0405 (0.0381)  loss_rpn_box_reg: 0.0483 (0.0463)  time: 0.6941  data: 0.1348  max mem: 3105
Epoch: [7]  [110/119]  eta: 0:00:06  lr: 0.000050  loss: 0.4082 (0.4072)  loss_classifier: 0.1196 (0.1220)  loss_box_reg: 0.2081 (0.2013)  loss_objectness: 0.0350 (0.0379)  loss_rpn_box_reg: 0.0475 (0.0461)  time: 0.6792  data: 0.1301  max mem: 3105
Epoch: [7]  [118/119]  eta: 0:00:00  lr: 0.000050  loss: 0.4070 (0.4076)  loss_classifier: 0.1196 (0.1223)  loss_box_reg: 0.2063 (0.2016)  loss_objectness: 0.0313 (0.0375)  loss_rpn_box_reg: 0.0475 (0.0462)  time: 0.6599  data: 0.1255  max mem: 3105
Epoch: [7] Total time: 0:01:20 (0.6763 s / it)
creating index...
index created!
Test:  [ 0/59]  eta: 0:00:14  model_time: 0.1194 (0.1194)  evaluator_time: 0.0633 (0.0633)  time: 0.2511  data: 0.0642  max mem: 3105
Test:  [58/59]  eta: 0:00:00  model_time: 0.1098 (0.1102)  evaluator_time: 0.0481 (0.0590)  time: 0.2353  data: 0.0625  max mem: 3105
Test: Total time: 0:00:13 (0.2371 s / it)
Averaged stats: model_time: 0.1098 (0.1102)  evaluator_time: 0.0481 (0.0590)
Accumulating evaluation results...
DONE (t=0.02s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.649
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.079
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.210
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.011
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.095
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.334
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.334
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000

I have two questions:

Overfitting: I don't know if my model is overfitting or underfitting. How I can find out looking the metrics?
Save the best model of all epochs: How I can save the best model trained during the differents epochs? Which is the best epoch according to the results?

Thank you!


title: Iterating through tensors in pytorch

body:
I have two 1D tensors. One is a vector of predictions, second is vector of labels. I am trying to write a loop that checks the element-wise difference between vectors. If such diff is spotted, I want to do another operation, for simplicity let's say I want to print ("Diff spotted"). So far I came up with this but I got an error: Expected object of scalar type Byte but got scalar type Float for argument #2 'other'. I would appreciate help here. Maybe there is some more efficient way to do it, without loop.

for i in enumerate(t1):
    if t1[i] != t2[i]:
        print("Diff spotted")


title: How to compute the uncertainty of a Monte Carlo Dropout neural network with PyTorch?

body:
I am trying to implement Bayesian CNN using Mc Dropout on Pytorch, the main idea is that by applying dropout at test time and running over many forward passes, you get predictions from a variety of different models. I need to obtain the uncertainty, does anyone have an idea of how I can do it Please
This is how I defined my CNN
'''
  class Net(nn.Module):
   def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 6, 5)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(6, 16, 5)
    self.fc1 = nn.Linear(16 * 5 * 5, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)
    self.dropout = nn.Dropout(p=0.3)

    nn.init.xavier_uniform_(self.conv1.weight)
    nn.init.constant_(self.conv1.bias, 0.0)
    nn.init.xavier_uniform_(self.conv2.weight)
    nn.init.constant_(self.conv2.bias, 0.0)
    nn.init.xavier_uniform_(self.fc1.weight)
    nn.init.constant_(self.fc1.bias, 0.0)
    nn.init.xavier_uniform_(self.fc2.weight)
    nn.init.constant_(self.fc2.bias, 0.0)
    nn.init.xavier_uniform_(self.fc3.weight)
    nn.init.constant_(self.fc3.bias, 0.0)

  def forward(self, x):
    x = self.pool(F.relu(self.dropout(self.conv1(x))))  # recommended to add the relu
    x = self.pool(F.relu(self.dropout(self.conv2(x))))  # recommended to add the relu
    x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(self.dropout(x)))
    x = self.fc3(self.dropout(x))  # no activation function needed for the last layer
    return x

  model = Net().to(device)

  train_accuracies=np.zeros(num_epochs)
  test_accuracies=np.zeros(num_epochs)

  dataiter = iter(trainloader)
  images, labels = dataiter.next()
  #initializing variables

  loss_acc = []
  class_acc_mcdo = []
  start_train = True

  #Defining the Loss Function and Optimizer
  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

  def train():
      loss_vals = []
      acc_vals = []

      for epoch in range(num_epochs):  # loop over the dataset multiple times

          n_correct = 0  # initialize number of correct predictions
          acc = 0  # initialize accuracy of each epoch
          somme = 0  # initialize somme of losses of each epoch
          epoch_loss = []

          for i, (images, labels) in enumerate(trainloader):
              # origin shape: [4, 3, 32, 32] = 4, 3, 1024
              # input_layer: 3 input channels, 6 output channels, 5 kernel size
              images = images.to(device)
              labels = labels.to(device)

              # Forward pass
              outputs = model.train()(images)
              loss = criterion(outputs, labels)

              # Backward and optimize
              optimizer.zero_grad()  # zero the parameter gradients
              loss.backward()
              epoch_loss.append(loss.item())  # add the loss to epoch_loss list
              optimizer.step()
              # max returns (value ,index)
              _, predicted = torch.max(outputs, 1)
              n_correct += (predicted == labels).sum().item()

              # print statistics
              if (i + 1) % 2000 == 0:
                  print(f'Epoch [{epoch + 1}/{num_epochs}], Step [{i + 1}/{n_total_steps}], Loss:
                  {loss.item():.4f}')

          somme = (sum(epoch_loss)) / len(epoch_loss)
          loss_vals.append(somme)  # add the epoch's loss to loss_vals

          print("Loss = {}".format(somme))
          acc = 100 * n_correct / len(trainset)
          acc_vals.append(acc)  # add the epoch's Accuracy to acc_vals
          print("Accuracy = {}".format(acc))

                # SAVE
          PATH = './cnn.pth'
          torch.save(model.state_dict(), PATH)

          loss_acc.append(loss_vals)
          loss_acc.append(acc_vals)
          return loss_acc

And here is the code of the mc dropout
'''
def enable_dropout(model):
    """ Function to enable the dropout layers during test-time """
    for m in model.modules():
        if m.__class__.__name__.startswith('Dropout'):
            m.train()

def test():
    # set non-dropout layers to eval mode
    model.eval()

    # set dropout layers to train mode
    enable_dropout(model)

    test_loss = 0
    correct = 0
    n_samples = 0
    n_class_correct = [0 for i in range(10)]
    n_class_samples = [0 for i in range(10)]
    T = 100

    for images, labels in testloader:
        images = images.to(device)
        labels = labels.to(device)

        with torch.no_grad():
            output_list = []
            # getting outputs for T forward passes
            for i in range(T):
                output_list.append(torch.unsqueeze(model(images), 0))

        # calculating mean
        output_mean = torch.cat(output_list, 0).mean(0)

        test_loss += F.nll_loss(F.log_softmax(output_mean, dim=1), labels,
                            reduction='sum').data  # sum up batch loss
        _, predicted = torch.max(output_mean, 1)  # get the index of the max log-probability
        correct += (predicted == labels).sum().item()  # sum up correct predictions
        n_samples += labels.size(0)

        for i in range(batch_size):
            label = labels[i]
            predi = predicted[i]

            if (label == predi):
                n_class_correct[label] += 1
            n_class_samples[label] += 1

    test_loss /= len(testloader.dataset)

    # PRINT TO HTML PAGE
    print('\n Average loss: {:.4f}, Accuracy:  ({:.3f}%)\n'.format(
        test_loss,
        100. * correct / n_samples))

    # Accuracy for each class
    acc_classes = []
    for i in range(10):
        acc = 100.0 * n_class_correct[i] / n_class_samples[i]
        print(f'Accuracy of {classes[i]}: {acc} %')
        acc_classes.append(acc)

    class_acc_mcdo.extend(acc_classes)
    print('Finished Testing')


title: Pytorch seq2seq learning - using word2vec

body:
I am following a seq2seq tutorial here.

I want to use pretrained vectors. I have edited the code to get the vector of the word rather than index. Following is the code:

#This piece of code loads the vectors from a json file {'word':[vector]..}
class Lang:
    def __init__(self, name, savedVectorsFile):
        def getSavedVectors(filename):
            import json
            word2vec = {}
            with open(filename) as json_data:
                word2vec = json.load(json_data)
            return word2vec

        self.name = name
        self.word2vector = {}
        self.word2count = {}
        self.index2word = {0: "SOS", 1: "EOS"}
        self.n_words = 2  # Count SOS and EOS

        self.get_saved_vector = getSavedVectors(savedVectorsFile)
        self.word2vector['unknown'] = self.get_saved_vector['unknown']

    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2vector:
            self.word2vector[word] = self.get_saved_vector[word]
            self.word2count[word] = 1
            self.index2word[self.n_words] = word
            self.n_words += 1
        else:
            self.word2count[word] += 1

# this piece deals with returning a vector of given word, all vectors are just concatenated into one giant vector
def vectorFromSentence(lang, sentence):
    vectors = []
    for word in sentence.split(' '):
        if word not in lang.word2vector:
            vectors += (lang.word2vector["unknown"])
        else:
            vectors += (lang.word2vector[word])
    return vectors

# in the train method, I am passing  a vector instead of index to the encoder
    for ei in range(0, input_length, VEC_SIZE):
        encoder_output, encoder_hidden = encoder(input_variable[ei*VEC_SIZE:(ei+1)*VEC_SIZE], encoder_hidden)
        encoder_outputs[ei] = encoder_output[0][0]

Now, I cannot figure out how should I change my encoder to incorporate the vectors instead of an index. This is my encoder as of now:

def __init__(self, input_size, hidden_size, n_layers=1):
    super(EncoderRNN, self).__init__()
    self.n_layers = n_layers
    self.hidden_size = hidden_size

    self.embedding = nn.Embedding(input_size, hidden_size)
    self.gru = nn.GRU(hidden_size, hidden_size)

I get this error that

TypeError: torch.index_select received an invalid combination of arguments - got (torch.cuda.FloatTensor, int, torch.cuda.FloatTensor), but expected (torch.cuda.FloatTensor source, int dim, torch.cuda.LongTensor index)

I also tried with using VEC_SIZE instead of input_size, but to no avail.

Following is the trace:

Traceback (most recent call last):
  File "scapula_generation_pretrained_vectors.py", line 710, in <module>
    trainIters(encoder1, attn_decoder1, 50000, print_every=1000)
  File "scapula_generation_pretrained_vectors.py", line 566, in trainIters
    decoder, encoder_optimizer, decoder_optimizer, criterion)
  File "scapula_generation_pretrained_vectors.py", line 472, in train
    encoder_output, encoder_hidden = encoder(input_variable[ei*VEC_SIZE:(ei+1)*VEC_SIZE], encoder_hidden)
  File "/home/sagar/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "scapula_generation_pretrained_vectors.py", line 214, in forward
    embedded = self.embedding(input).view(1, 1, -1)
  File "/home/sagar/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sagar/anaconda3/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 94, in forward
    self.scale_grad_by_freq, self.sparse
  File "/home/sagar/anaconda3/lib/python3.6/site-packages/torch/nn/_functions/thnn/sparse.py", line 53, in forward
    output = torch.index_select(weight, 0, indices.view(-1))

How to write encoder and decoder to occupy the word2vec embeddings? Several things are involved, like does my output of encoder gru has to be a vector of size VEC_SIZE or not, in the decoder - my loss has to be calculated using some similarity metric. I think I will go for cosine similarity, but before that i will have to make sure that decoder takes the output of encoder and generates a vector of size VEC_SIZE.

I would appreciate if someone has already done this homework in the tutorial and has a code readily available?


title: Why randn doesn't always have a mean of 0 and variance of 1?

body:
For the PyTorch.randn() method the documentation says:

  Returns a tensor filled with random numbers from a normal distribution
  with mean 0 and variance 1 (also called the standard normal
  distribution).

So here is an example tensor:

x = torch.randn(4,3)
tensor([[-0.6569, -0.7337, -0.0028],
        [-0.3938,  0.3223,  0.0497],
        [ 0.0129, -2.7546, -2.2488],
        [ 1.6754, -0.1497,  1.8202]])

When I print the mean:

x.mean()
tensor(-0.2550)

When I print the standard deviation:

x.std()
tensor(1.3225)

So why isn't the mean 0 and the standard deviation 1?

Bonus question: How do I generate a random tensor that always has a mean of 0?


title: How embedding_bag exactly works in PyTorch

body: in PyTorch, torch.nn.functional.embedding_bag seems to be the main function responsible for doing the real job of embedding lookup. On PyTorch's documentation, it has been mentioned that embedding_bag does its job > without instantiating the intermediate embeddings. What does that exactly mean? Does this mean for example when the mode is "sum" it does in-place summation? or it just means that no additional Tensors will be produced when calling embedding_bag but still from the system's point of view all the intermediate row-vectors are already fetched into the processor to be used for calculating the final Tensor?


title: Hello World Convolution on Images in PyTorch

body:
I am trying to verify some results with PyTorch's 2D convolution with the following:

Input matrix: X (10, 10, 3) [Dummy numpy image]
Weight matrix: W (3, 3, 3) [My Conv Filter to test]
Output matrix: Y (10, 10, 1)

I have the following code but I am not able to assign the weights properly and run the model without errors.
What am I doing wrong here??

import torch
import torch.nn as nn
import torchvision.transforms
import numpy as np

# Convert image to tensor
image2tensor = torchvision.transforms.ToTensor()

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        # Test layer
        self.layer1 = nn.Conv2d(3, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))

    def forward(self, x):
        out = self.layer1(x)
        return out

# Test image
image = np.ones((10, 10, 3))
tensor = image2tensor(image).unsqueeze(0)

# Create new model
conv = ConvNet()

# Assign test weight - NOT WORKING!!
weight = torch.nn.Parameter(torch.ones(3, 3, 3))
conv.layer1.weight.data = weight

# Run the model
output = conv(tensor)


title: Simple but disturbing problem about function "tensor.backward()" in PyTorch

body:
I'm learning PyTorch these days and the backward() funciton really confused me.
Let's go directly to my problem:

I defined some tensors and ops:

```

import torch
x = torch.ones(2,5,requires_grad=True)
y = x + 2
z = x*x
Y = torch.mean(y)
Z = torch.mean(z)

```

If I run this:

y.backward(torch.ones(2,5))
z.backward(torch.ones(2,5))

no error occured.

But, if I run this:

Y.backward()
Z.backward()

I got:

    RuntimeError                              Traceback (most recent call last)
<ipython-input-7-732c4cd53ca7> in <module>()
      1 Y.backward()
----> 2 Z.backward()

E:\learningsoft\anadonda\lib\site-packages\torch\tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94
     95     def register_hook(self, hook):

E:\learningsoft\anadonda\lib\site-packages\torch\autograd\__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---> 90         allow_unreachable=True)  # allow_unreachable flag
     91
     92

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

Could someone tell me why they have different results?

Let's me give more examples:
example1(picture)


title: How to calculate gradients on a tensor in PyTorch?

body:
I want to calculate the gradient of a tensor and however, it gives error as

RunTimeerror: grad can be implicitly created only for scalar outputs

and here is what I am trying to code:

x = torch.full((2,3), 4,requires_grad=True)
y = (2*x**2+3)
y.backward()

And at this point, it throws an error.


title: Extracting reduced dimension data from autoencoder in pytorch

body:
I have defined my autoencoder in pytorch as following:

self.encoder = nn.Sequential(
    nn.Conv2d(input_shape[0], 32, kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Conv2d(32, 64, kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 64, kernel_size=1, stride=1),
    nn.ReLU()
)

self.decoder = nn.Sequential(
    nn.Conv2d(64, 64, kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 32, kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Conv2d(32, input_shape[0], kernel_size=1, stride=1),
    nn.ReLU(),
    nn.Sigmoid()
)

I need to get a reduced dimension encoding which requires creating a new linear layer of the dimension N much lower than the image dimension so that I can extract the activations.

If anybody can help me with fitting a linear layer in the decoder part I would appreciate (i know how to Flatten() the data, but I guess I need to "unflatten" it again to interface with the Conv2d layer again)

Update:

I have come up with the following based on the first answer (it gives me a 8-dimensional bottleneck at the output of the encoder which works fine torch.Size([1, 8, 1, 1]) ).

self.encoder = nn.Sequential(
    nn.Conv2d(input_shape[0], 32, kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Conv2d(32, 64, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(64, 8, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.MaxPool2d(7, stride=1)
)

self.decoder = nn.Sequential(
    nn.ConvTranspose2d(8, 64, kernel_size=3, stride=1),
    nn.ReLU(),
    nn.Conv2d(64, 32, kernel_size=4, stride=2),
    nn.ReLU(),
    nn.Conv2d(32, input_shape[0], kernel_size=8, stride=4),
    nn.ReLU(),
    nn.Sigmoid()
)

What I cannot do is train the autoencoder with

def forward(self, x):
    x = self.encoder(x)
    x = self.decoder(x)
    return x

The decoder gives me an error:

Calculated padded input size per channel: (3 x 3). Kernel size: (4 x 4). Kernel size can't be greater than actual input size

I would like to thank the person who provided the first answer.


title: PyTorch to Keras model

body:
I'm trying to replicate a model, but I'm having difficulties doing so with Keras. Here is my current implementation:

filters = 256
kernel_size = 3
strides = 1

# Head module
input = Input(shape=(img_height//scale_fact, img_width//scale_fact, img_depth))
conv0 = Conv2D(filters, kernel_size, strides=strides, padding='same',
               kernel_regularizer=regularizers.l2(0.01))(input)

# Body module
res = Conv2D(filters, kernel_size, strides=strides, padding='same')(conv0)
act = ReLU()(res)
res = Conv2D(filters, kernel_size, strides=strides, padding='same')(act)
res_rec = Add()([conv0, res])

for i in range(res_blocks):
    res1 = Conv2D(filters, kernel_size, strides=strides, padding='same')(res_rec)
    act  = ReLU()(res1)
    res2 = Conv2D(filters, kernel_size, strides=strides, padding='same')(act)
    res_rec = Add()([res_rec, res2])

conv = Conv2D(filters, kernel_size, strides=strides, padding='same',
              kernel_regularizer=regularizers.l2(0.01))(res_rec)
add  = Add()([conv0, conv])

# Tail module
conv = Conv2D(filters, kernel_size, strides=strides, padding='same',
              kernel_regularizer=regularizers.l2(0.01))(add)
act = ReLU()(conv)
up  = UpSampling2D(size=scale_fact if scale_fact != 4 else 2)(act)  # TODO: try "Conv2DTranspose"
# mul = Multiply([np.zeros((img_width,img_height,img_depth)).fill(0.1), up])(up)

# When it's a 4X factor, we want the upscale split in two procedures
if(scale_fact == 4):
    conv = Conv2D(filters, kernel_size, strides=strides, padding='same',
                  kernel_regularizer=regularizers.l2(0.01))(up)
    act = ReLU()(conv)
    up  = UpSampling2D(size=2)(act)  # TODO: try "Conv2DTranspose"

output = Conv2D(filters=3,
                kernel_size=1,
                strides=1,
                padding='same',
                kernel_regularizer=regularizers.l2(0.01))(up)

model = Model(inputs=input, outputs=output)

Here is a link to the file I'm trying to replicate. How am I supposed to replicate this custom PyTorch UpSampler that implements a customized PixelShuffling method ?

Here is the relevant part of the UpSampler that I'm having trouble with, for the most part:

import tensorflow as tf
import tensorflow.contrib.slim as slim

"""
Method to upscale an image using
conv2d transpose. Based on upscaling
method defined in the paper
x: input to be upscaled
scale: scale increase of upsample
features: number of features to compute
activation: activation function
"""
def upsample(x,scale=2,features=64,activation=tf.nn.relu):
    assert scale in [2,3,4]
    x = slim.conv2d(x,features,[3,3],activation_fn=activation)
    if scale == 2:
        ps_features = 3*(scale**2)
        x = slim.conv2d(x,ps_features,[3,3],activation_fn=activation)
        #x = slim.conv2d_transpose(x,ps_features,6,stride=1,activation_fn=activation)
        x = PS(x,2,color=True)
    elif scale == 3:
        ps_features =3*(scale**2)
        x = slim.conv2d(x,ps_features,[3,3],activation_fn=activation)
        #x = slim.conv2d_transpose(x,ps_features,9,stride=1,activation_fn=activation)
        x = PS(x,3,color=True)
    elif scale == 4:
        ps_features = 3*(2**2)
        for i in range(2):
            x = slim.conv2d(x,ps_features,[3,3],activation_fn=activation)
            #x = slim.conv2d_transpose(x,ps_features,6,stride=1,activation_fn=activation)
            x = PS(x,2,color=True)
    return x

"""
Borrowed from https://github.com/tetrachrome/subpixel
Used for subpixel phase shifting after deconv operations
"""
def _phase_shift(I, r):
    bsize, a, b, c = I.get_shape().as_list()
    bsize = tf.shape(I)[0] # Handling Dimension(None) type for undefined batch dim
    X = tf.reshape(I, (bsize, a, b, r, r))
    X = tf.transpose(X, (0, 1, 2, 4, 3))  # bsize, a, b, 1, 1
    X = tf.split(X, a, 1)  # a, [bsize, b, r, r]
    X = tf.concat([tf.squeeze(x, axis=1) for x in X],2)  # bsize, b, a*r, r
    X = tf.split(X, b, 1)  # b, [bsize, a*r, r]
    X = tf.concat([tf.squeeze(x, axis=1) for x in X],2)  # bsize, a*r, b*r
    return tf.reshape(X, (bsize, a*r, b*r, 1))

"""
Borrowed from https://github.com/tetrachrome/subpixel
Used for subpixel phase shifting after deconv operations
"""
def PS(X, r, color=False):
    if color:
        Xc = tf.split(X, 3, 3)
        X = tf.concat([_phase_shift(x, r) for x in Xc],3)
    else:
        X = _phase_shift(X, r)
    return X


title: PyTorch RNN-BiLSTM sentiment analysis low accuracy

body:
I'm using PyTorch with a training set of movie reviews each labeled positive or negative. Every review is truncated or padded to be 60 words and I have a batch size of 32.  This 60x32 Tensor is fed to an embedding layer with an embedding dim of 100 resulting in a 60x32x100 Tensor.  Then I use the unpadded lengths of each review to pack the embedding output, and feed that to a BiLSTM layer with hidden dim = 256.
I then pad it back, apply a transformation (to try to get the last hidden state for the forward and backward directions) and feed the transformation to a Linear layer which is 512x1.  Here is my module, I pass the final output through a sigmoid not shown here
class RNN(nn.Module):
    def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers,
                 bidirectional, dropout, pad_idx):

        super().__init__()
        self.el = nn.Embedding(vocab_size, embedding_dim)
        print('vocab size is ', vocab_size)
        print('embedding dim is ', embedding_dim)
        self.hidden_dim = hidden_dim
        self.num_layers = n_layers # 2
        self.lstm = nn.LSTM(input_size=embedding_dim, hidden_size=hidden_dim, num_layers=n_layers, dropout=dropout, bidirectional=bidirectional)
        # Have an output layer for outputting a single output value
        self.linear = nn.Linear(2*hidden_dim, output_dim)

    def init_hidden(self):
        return (torch.zeros(self.n_layers*2, 32, self.hidden_dim).to(device),
                torch.zeros(self.n_layers*2, 32, self.hidden_dim).to(device))

    def forward(self, text, text_lengths):
        print('input text size ', text.size())
        embedded = self.el(text)
        print('embedded size ', embedded.size())
        packed_seq = torch.nn.utils.rnn.pack_padded_sequence(embedded, lengths=text_lengths, enforce_sorted=False)
        packed_out, (ht, ct) = self.lstm(packed_seq, None)
        out_rnn, out_lengths = torch.nn.utils.rnn.pad_packed_sequence(packed_out)
        print('padded lstm out ', out_rnn.size())
        #out_rnn = out_rnn[-1] #this works
        #out_rnn = torch.cat((out_rnn[-1, :, :self.hidden_dim], out_rnn[0, :, self.hidden_dim:]), dim=1) # this works
        out_rnn = torch.cat((ht[-1], ht[0]), dim=1) #this works
        #out_rnn = out_rnn[:, -1, :] #doesn't work maybe should
        print('attempt to get last hidden ', out_rnn.size())
        linear_out = self.linear(out_rnn)
        print('after linear ', linear_out.size())
        return linear_out

I've tried 3 different transformations to get the dimensions correct for the linear layer
out_rnn = out_rnn[-1] #this works
out_rnn = torch.cat((out_rnn[-1, :, :self.hidden_dim], out_rnn[0, :, self.hidden_dim:]), dim=1) # this works
out_rnn = torch.cat((ht[-1], ht[0]), dim=1) #this works

These all produce an output like this

input text size  torch.Size([60, 32])
embedded size  torch.Size([60,32, 100])
padded lstm out  torch.Size([36, 32, 512])
attempt to get last hidden  torch.Size([32, 512])
after linear  torch.Size([32, 1])

I would expect the padded lstm out to be [60, 32, 512] but it is always less than 60 in the first dimension.
I'm training for 10 epochs with optim.SGD and nn.BCEWithLogitsLoss().  My training accuracy is always around 52% and test accuracy is always at like 50%, so the model is doing no better than randomly guessing.  I'm sure that my data is being handled correctly in my tochtext.data.Dataset.  Am I forwarding my tensors along incorrectly?
I have tried using batch_first=True in my lstm, packed_seq function, and pad_packed_seq function and that breaks my transformations before feeding to the linear layer.
Update
I added the init_hidden method and have tried without the pack/pad sequence methods and still get the same results


title: Error in Transforming custom Dataset in Pytorch

body:
I was following this tutorial: https://pytorch.org/tutorials/beginner/data_loading_tutorial.html for making my own custom data loader for MuNuSeg Dataset but I am stuck a point. The dataloader is working fine but when I add transforms to it, I get errors.

I am facing an issue similar to what is mentioned here :
Error Utilizing Pytorch Transforms and Custom Dataset
According to the answer there I have made custom transform for each of the inbuilt transforms so that one whole sample is transformed at the same time. Below are my custom transforms

class AffineTrans(object):
def __init__(self, degrees, translate):
    self.degrees = degrees
    self.translate = translate

def __call__(self, sample):
    image, contour, clrmask = sample['image'], sample['contour'], sample['clrmask']
    TF = transforms.RandomAffine(degrees = self.degrees, translate=self.translate)
    image = TF(image)
    contour = TF(contour)
    clrmask = (clrmask)

class Flip(object):
def __call__(self, sample):
    image, contour, clrmask = sample['image'], sample['contour'], sample['clrmask']
    TF1 = transforms.RandomHorizontalFlip()
    TF2 = transforms.RandomVerticalFlip()
    image = TF1(image)
    contour = TF1(contour)
    clrmask = TF1(clrmask)
    image = TF2(image)
    contour = TF2(contour)
    clrmask = TF2(clrmask)

class ClrJitter(object):
def __init__(self, brightness, contrast, saturation, hue):
    self.brightness = brightness
    self.contrast = contrast
    self.saturation = saturation
    self.hue = hue

def __call__(self, sample):
    image, contour, clrmask = sample['image'], sample['contour'], sample['clrmask']
    TF = transforms.ColorJitter(self.brightness, self.contrast, self.saturation, self.hue)
    image = TF(image)
    contour = TF(contour)
    clrmask = TF(clrmask)

And composed them in the following way

composed = transforms.Compose([RandomCrop(256),
                           AffineTrans(15.0,(0.1,0.1)),
                           Flip(),
                           ClrJitter(10, 10, 10, 0.01),
                           ToTensor()])

And here's the trainLoader Code

class trainLoader(Dataset):
def __init__(self, transform=None):
    """
    Args:
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    [self.train_data , self.test_data1, self.test_data2] = dirload()
    self.transform = transform

def __len__(self):
    return(len(self.train_data[0]))

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()
    img_name = self.train_data[0][idx]
    contour_name = self.train_data[1][idx]
    color_name = self.train_data[2][idx]
    image = cv2.imread(img_name)
    contour = cv2.imread(contour_name)
    clrmask = cv2.imread(color_name)
    sample = {'image': image, 'contour': contour, 'clrmask': clrmask}

    if self.transform:
        sample = self.transform(sample)

    return sample

To check the working of the above code I am doing the following

    train_dat = trainLoader(composed)

for i in range(len(train_dat)):
    sample = train_dat[i]

    print(i, sample['image'].shape, sample['contour'].shape, sample['clrmask'].shape)
    cv2.imshow('sample',sample['image'])
    cv2.waitKey()
    if i == 3:
        break

But I am still, again and again, Encountering the following error

runcell(0, 'F:/Moodle/SEM 8/SRE/code/MoNuSeg/main.py')

runcell(1, 'F:/Moodle/SEM 8/SRE/code/MoNuSeg/main.py')
Traceback (most recent call last):

  File "F:\Moodle\SEM 8\SRE\code\MoNuSeg\main.py", line 212, in <module>
    sample = train_dat[i]

  File "F:\Moodle\SEM 8\SRE\code\MoNuSeg\main.py", line 109, in __getitem__
    sample = self.transform(sample)

  File "F:\Moodle\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 60, in __call__
    img = t(img)

  File "F:\Moodle\SEM 8\SRE\code\MoNuSeg\main.py", line 156, in __call__
    image = TF(image)

  File "F:\Moodle\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 1018, in __call__
    ret = self.get_params(self.degrees, self.translate, self.scale, self.shear, img.size)

  File "F:\Moodle\Anaconda3\lib\site-packages\torchvision\transforms\transforms.py", line 992, in get_params
    max_dx = translate[0] * img_size[0]

TypeError: 'int' object is not subscriptable

This is quite an ambiguous error as I exactly don't get what is the error is

Any help would be really appreciated


title: The result is not fixed after setting random seed in pytorch

body:
def setup_seed(seed):
    np.random.seed(seed)
    random.seed(seed)
    torch.manual_seed(seed)  # cpu
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = True

I set random seed when run the code, but I can not get fixed result with pytorch. Besides, I use batchnorm in my code. When evaluate and test, I have set model.eval(). I cannot figure out the reason for that.


title: implement dropout layer using nn.Sequential()

body:
I am trying to implement a Dropout layer using pytorch as follows:
class DropoutLayer(nn.Module):
    def __init__(self, p):
        super().__init__()
        self.p = p

    def forward(self, input):
        if self.training:
            u1 = (np.random.rand(*input.shape)<self.p) / self.p
            u1 *= u1
            return u1
        else:
            input *= self.p

And then calling a simple NN.sequential:
model = nn.Sequential(nn.Linear(input_size,num_classes), DropoutLayer(.7), nn.Flatten())

opt = torch.optim.Adam(model.parameters(), lr=0.005)
train(model, opt, 5) #train(model, optimizer, epochs #)

But I'm getting the following error:
TypeError: flatten() takes at most 1 argument (2 given)

Not sure what I'm doing wrong.  Still new to pytorch.  Thanks.


title: Query padding mask and key padding mask in Transformer encoder

body:
I'm implementing self-attention part in transformer encoder using pytorch nn.MultiheadAttention and confusing in the padding masking of transformer.
The following picture shows the self-attention weight of the query (row) and key (column).
As you can see, there are some tokens "<PAD>" and I have already mask it in key. Therefore the tokens will not calculate the attention weight.

There are still two questions:

In query part, can I also mask them("<PAD>") except for the red square part? Is this reasonable?

How can I mask "<PAD>" in the query?

The attention weights also use the softmax function along the row by giving mask in src_mask or src_key_padding_mask argument. If I set all the "<PAD>" row into -inf, the softmax will return nan and the loss with be nan


title: Type Conversions

body:
I'm quite new to pytorch so I'm not exactly sure if this is a bug or just an error on my end.

I'm attempting to build a simple LogisticRegression Module on the make_classification dataset in sklearn but for some reason, I keep getting this.

  RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'mat2'

I've tried to convert the both my X and y variables to double but it still seems to think it's float for some reason.

Here's my code:

X, y = make_classification(n_samples=60000,n_features=10, n_redundant=0, n_informative=2)

X_train,y_train, X_test, y_test = train_test_split(X, y, stratify=y, random_state=42)

X_train = Variable(torch.from_numpy(X_train)).double()
y_train = Variable(torch.from_numpy(y_train)).double()
X_test = Variable(torch.from_numpy(X_test)).double()
y_test = Variable(torch.from_numpy(y_test)).double()

Here's the network

input_dim = 40000
output_dim = 2
criterion = nn.CrossEntropyLoss()
learning_rate = 0.001
batch_size = 100
n_iters = 3000
num_epochs = int(n_iters / (len(X_train) / batch_size))

model = SingleLayeredNetwork(input_dim, output_dim)
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

class SingleLayeredNetwork(nn.Module):
    def __init__(self, input_dimension, output_dimension):
        super(SingleLayeredNetwork, self).__init__()
        self.layer = nn.Linear(input_dimension, output_dimension)

    def forward(self, x):
        out = self.layer(x)
        return out

Here's the training loop

iter = 0
for epoch in range(num_epochs):
for train, test in zip(X_train, y_train):
    train = train.double()
    print(train)
    print(train.dtype)
    print(torch.typename(train))
    optimizer.zero_grad()

    outputs = model(train)
    loss = criterion(outputs, test)

    loss.backward()

    # Updating parameters
    optimizer.step()

    iter += 1

    if iter % 500 == 0:
        # Calculate Accuracy
        correct = 0
        total = 0
        # Iterate through test dataset
        for tra, tes in zip(X_test, y_test):
            # Load images to a Torch Variable
            tra = tra

            # Forward pass only to get logits/output
            outputs = model(tra)

            # Get predictions from the maximum value
            # 100 x 1
            _, predicted = torch.max(train, 1)

            # Total number of labels
            total += labels.size(0)

            # Total correct predictions
            correct += (predicted == test).sum()

        accuracy = 100 * correct.item() / total

        # Print Loss
        print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.item(), accuracy))

Here's the error I keep getting

  RuntimeError                              Traceback (most recent call
  last)  in
       10         optimizer.zero_grad()
       11
  ---> 12         outputs = model(train)
       13         loss = criterion(outputs, test)
       14

  ~/miniconda3/envs/mlbook/lib/python3.6/site-packages/torch/nn/modules/module.py
  in call(self, *input, **kwargs)
      487             result = self._slow_forward(*input, **kwargs)
      488         else:
  --> 489             result = self.forward(*input, **kwargs)
      490         for hook in self._forward_hooks.values():
      491             hook_result = hook(self, input, result)

   in forward(self, x)
        5
        6     def forward(self, x):
  ----> 7         out = self.layer(x)
        8         return out

  ~/miniconda3/envs/mlbook/lib/python3.6/site-packages/torch/nn/modules/module.py
  in call(self, *input, **kwargs)
      487             result = self._slow_forward(*input, **kwargs)
      488         else:
  --> 489             result = self.forward(*input, **kwargs)
      490         for hook in self._forward_hooks.values():
      491             hook_result = hook(self, input, result)

  ~/miniconda3/envs/mlbook/lib/python3.6/site-packages/torch/nn/modules/linear.py
  in forward(self, input)
       65     @weak_script_method
       66     def forward(self, input):
  ---> 67         return F.linear(input, self.weight, self.bias)
       68
       69     def extra_repr(self):

  ~/miniconda3/envs/mlbook/lib/python3.6/site-packages/torch/nn/functional.py
  in linear(input, weight, bias)    1352         ret =
  torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())
  1353     else:
  -> 1354         output = input.matmul(weight.t())    1355         if bias is not None:    1356             output +=
  torch.jit._unwrap_optional(bias)

  RuntimeError: Expected object of scalar type Double but got scalar
  type Float for argument #2 'mat2'

What am I missing?


title: Why the pytorch implementation is so inefficient?

body:
I have implemented a paper about a CNN architecture in both Keras and Pytorch but keras implementation is much more efficient it takes 4 gb of gpu for training with 50000 samples and 10000 validation samples but pytorch one takes all the 12 gb of gpu and i cant even use a validation set !
Optimizer for both of them is sgd with momentum and same settings for both.
more info about the paper:[architecture]:https://github.com/Moeinh77/Lightweight-Deep-Convolutional-Network-for-Tiny-Object-Recognition/edit/master/train.py

pytorch code :

class SimpleCNN(torch.nn.Module):

    def __init__(self):
        super(SimpleCNN, self).__init__()

        self.conv2d_11 = torch.nn.Conv2d(3, 64, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_12 = torch.nn.Conv2d(64, 64, kernel_size = 3, stride = 1, padding = 1)

        self.conv2d_21 = torch.nn.Conv2d(64, 128, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_22 = torch.nn.Conv2d(128, 128, kernel_size = 3, stride = 1, padding = 1)

        self.conv2d_31 = torch.nn.Conv2d(128, 256, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_32 = torch.nn.Conv2d(256, 256, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_33 = torch.nn.Conv2d(256, 256, kernel_size = 3, stride = 1, padding = 1)

        self.conv2d_41 = torch.nn.Conv2d(256, 512, kernel_size = 3, stride = 1, padding = 1)
        self.conv2d_42 = torch.nn.Conv2d(512, 512, kernel_size = 3, stride = 1, padding = 1)

        self.conv2d_51 = torch.nn.Conv2d(512, 512, kernel_size = 3, stride = 1, padding = 1)

        self.Batchnorm_1=torch.nn.BatchNorm2d(64)
        self.Batchnorm_2=torch.nn.BatchNorm2d(128)
        self.Batchnorm_3=torch.nn.BatchNorm2d(256)
        self.Batchnorm_4=torch.nn.BatchNorm2d(512)

        self.dropout2d_1=torch.nn.Dropout2d(p=0.3)
        self.dropout2d_2=torch.nn.Dropout2d(p=0.4)
        self.dropout2d_3=torch.nn.Dropout2d(p=0.5)

        self.dropout1d=torch.nn.Dropout(p=0.5)

        self.maxpool2d = torch.nn.MaxPool2d(kernel_size = 2, stride = 2, padding = 0)

        self.avgpool2d = torch.nn.AvgPool2d(kernel_size = 2, stride = 2, padding = 0)

        self.fc = torch.nn.Linear(512, 10)

    def forward(self, x):

        ############################# Phase 1
        #print(x.size())
        x = F.relu(self.conv2d_11(x))
        x = self.dropout2d_1(x) #rate =0.3
        x = self.Batchnorm_1(x) #input 64
        #print(x.size())

        x = F.relu(self.conv2d_12(x))
        x = self.dropout2d_1(x) #rate=0.3
        x = self.Batchnorm_1(x) #input 64
        #print(x.size())

        x = self.maxpool2d(x)
        #print(x.size())
        ############################# Phase 2
        x = F.relu(self.conv2d_21(x))
        x = self.dropout2d_1(x) #rate=0.3
        x = self.Batchnorm_2(x) #input 128
        #print(x.size())

        x = F.relu(self.conv2d_22(x))
        x = self.dropout2d_1(x) #rate=0.3
        x = self.Batchnorm_2(x) #input 128
        #print(x.size())

        x = self.maxpool2d(x)
        #print(x.size())
        ############################# Phase 3
        x = F.relu(self.conv2d_31(x))
        x = self.dropout2d_2(x) #rate=0.4
        x = self.Batchnorm_3(x) #input 256
        #print(x.size())

        x = F.relu(self.conv2d_32(x))
        x = self.dropout2d_2(x) #rate=0.4
        x = self.Batchnorm_3(x) #input 256
        #print(x.size())

        x = F.relu(self.conv2d_33(x))
        x = self.dropout2d_2(x) #rate=0.4
        x = self.Batchnorm_3(x) #input 256
        #print(x.size())

        x = self.maxpool2d(x)
        #print(x.size())
        ############################# Phase 4
        x = F.relu(self.conv2d_41(x))
        x = self.dropout2d_2(x)
        x = self.Batchnorm_4(x)
        #print(x.size())

        x = F.relu(self.conv2d_42(x))
        x = self.dropout2d_2(x)
        x = self.Batchnorm_4(x)
        #print(x.size())

        x = self.maxpool2d(x)
        #print(x.size())
        ############################# Phase 5
        x = F.relu(self.conv2d_51(x))
        x = self.dropout2d_3(x)
        x = self.Batchnorm_4(x)
        #print(x.size())

        x = self.avgpool2d(x)
        #print(x.size())
        x = x.view(x.size(0), -1)
        #print(x.size())
        x = self.dropout1d(x)
        x = F.relu(self.fc(x))
        x = self.dropout1d(x)
        #print(x.size())
        x = F.softmax(x)
        ###############################

        return(x)

import time
from torch.optim.lr_scheduler import ReduceLROnPlateau

def trainNet(model, batch_size, n_epochs, learning_rate):

    lr=learning_rate

    #Print all of the hyperparameters of the training iteration:
    print("======= HYPERPARAMETERS =======")
    print("Batch size=", batch_size)
    print("Epochs=", n_epochs)
    print("Base learning_rate=", learning_rate)
    print("=" * 30)

    #Get training data
    n_batches = len(train_loader)

    #Time for printing
    training_start_time = time.time()

    #Loss function"
    loss = torch.nn.CrossEntropyLoss()
    optimizer = createOptimizer(model, lr)

    scheduler = ReduceLROnPlateau(optimizer, 'min'
                                  ,patience=3,factor=0.9817
                                 ,verbose=True,)

    #Loop for n_epochs
    for epoch in range(n_epochs):

        #save the weightsevery 10 epochs
        if epoch % 10 == 0 :
            torch.save(model.state_dict(), 'model.ckpt')

        #print('learning rate : {:.3f} '.format(lr))
        #Create our loss and optimizer functions

        running_loss = 0.0
        print_every = n_batches // 10
        start_time = time.time()
        total_train_loss = 0
        total_train_acc = 0
        epoch_time = 0

        for i, data in enumerate(train_loader, 0):

            #free up the cuda memory
            inputs=None
            labels=None

            inputs, labels = data

            inputs, labels = Variable(inputs.to(device)), Variable(labels.to(device))

            optimizer.zero_grad()

            outputs = model(inputs)

            score, predictions = torch.max(outputs.data, 1)
            acc = (labels==predictions).sum()
            total_train_acc += acc

            loss_size = loss(outputs, labels)
            loss_size.backward()
            optimizer.step()

            running_loss += loss_size.item()
            total_train_loss += loss_size.item()

            #Print every 10th batch of an epoch
            if (i + 1) % (print_every + 1) == 0:
                print("Epoch {}, {:d} % \t | train_loss: {:.3f} | train_acc:{}% | took: {:.2f}s".format(
                        epoch+1, int(100 * (i+1) / n_batches), running_loss / print_every
                        ,int(acc), time.time() - start_time))

                epoch_time += (time.time() - start_time)

                #Reset running loss and time
                running_loss = 0.0
                start_time = time.time()

        scheduler.step(total_train_loss)
        torch.cuda.empty_cache()
        #At the end of the epoch, do a pass on the validation set
        total_val_loss = 0

        for inputs, labels in val_loader:

            #Wrap tensors in Variables
            inputs, labels = Variable(inputs.to(device)), Variable(labels.to(device))

            #Forward pass
            val_outputs = model(inputs)
            val_loss_size = loss(val_outputs, labels)
            total_val_loss += val_loss_size.item()

        print("-"*30)
        print("Train loss = {:.2f} | Train acc = {:.1f}% | Val loss={:.2f} | took: {:.2f}s".format(
            total_train_loss / len(train_loader),total_train_acc/ len(train_loader)
            ,total_val_loss/len(val_loader),epoch_time))
        print("="*60)

    print("Training finished, took {:.2f}s".format(time.time() - training_start_time))
CNN = SimpleCNN().to(device)
CNN.eval()

trainNet(CNN, batch_size=64, n_epochs=250, learning_rate=0.1)

Keras:

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, Flatten,Activation
from tensorflow.keras.layers import Conv2D, MaxPool2D,BatchNormalization,GlobalAveragePooling2D

model = Sequential()
#####################################################
# Phase 1
model.add(Conv2D(64,(3,3),input_shape=(32,32,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))
model.add(BatchNormalization())

#(32,32,3)

model.add(Conv2D(64,(3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))
model.add(BatchNormalization())
#(32,32,3)

model.add(MaxPool2D((2,2)))
#(16,16,3)

#####################################################
#Phase 2
model.add(Conv2D(128, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))
model.add(BatchNormalization())
#(16,16,3)

model.add(Conv2D(128, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.3))
model.add(BatchNormalization())
#(16,16,3)

model.add(MaxPool2D((2,2),padding='same'))
#(8,8,3)

#####################################################
#Phase 3
model.add(Conv2D(256, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(8,8,3)

model.add(Conv2D(256, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(8,8,3)

model.add(Conv2D(256, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(8,8,3)

model.add(MaxPool2D((2,2)))
#(4,4,3)

#####################################################
#Phase 4
model.add(Conv2D(512, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(4,4,3)

model.add(Conv2D(512, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.4))
model.add(BatchNormalization())
#(4,4,3)

model.add(MaxPool2D((2,2)))
#(2,2,3)

#####################################################
#Phase 5
model.add(Conv2D(512, (3,3),padding='same'))
model.add(Activation('relu'))
model.add(Dropout(rate=0.5))
model.add(BatchNormalization())
#(2,2,3)

model.add(GlobalAveragePooling2D(data_format='channels_last'))
model.add(Flatten())
model.add(Dropout(rate=0.5))

model.add(Dense(10,activation='relu'))
model.add(Dropout(rate=0.5))
model.add(Dense(10, activation='softmax'))
model.compile(optimizer=sgd_optimizer,loss='categorical_crossentropy',metrics=['accuracy'])
history=model.fit(x=x_train,y=y_train,batch_size=64,
                  epochs=250,verbose=1,callbacks=[checkpoint],validation_data=(x_test,y_test))


title: pytorch dataset can't set attribute train_labels

body:
first I set some paraments
import random
import torch
from torchvision import datasets

epochs = 50
lr = 1e-3
batch_size = 128
sample_size = 200
hidden_size = 200
num_task = 3

second I build an dataset,and def get_permute_mnist()
class PermutedMNIST(datasets.MNIST):

    def __init__(self, root="~/.torch/data/mnist", train=True, permute_idx=None):
        super(PermutedMNIST, self).__init__(root, train, download=True)
        assert len(permute_idx) == 28 * 28
        if self.train:
            self.train_data = torch.stack([img.float().view(-1)[permute_idx] / 255 for img in self.train_data])
        else:
            self.test_data = torch.stack([img.float().view(-1)[permute_idx] / 255 for img in self.test_data])

    def __getitem__(self, index):

        if self.train:
            img, target = self.train_data[index], self.train_labels[index]
        else:
            img, target = self.test_data[index], self.test_labels[index]

        return img, target

    def get_sample(self, sample_size):
        sample_idx = random.sample(range(len(self)), sample_size)
        return [img for img in self.train_data[sample_idx]]
def get_permute_mnist():
    train_loader = {}
    test_loader = {}
    idx = list(range(28 * 28))
    for i in range(num_task):
        train_loader[i] = torch.utils.data.DataLoader(PermutedMNIST(train=True, permute_idx=idx),
                                                      batch_size=batch_size,
                                                      num_workers=4)
        test_loader[i] = torch.utils.data.DataLoader(PermutedMNIST(train=False, permute_idx=idx),
                                                     batch_size=batch_size)
        random.shuffle(idx)
    return train_loader, test_loader

last I use the dataset
train_loader, test_loader = get_permute_mnist()

I get the following error AttributeError: can't set attribute, why I can set sel.train_data or give any value to the variable 'train_data'?
the wrong is show in the picture:
enter image description here


title: AssertionError: Torch not compiled with CUDA enabled (problem in torch vision)

body:
so I am trying to run my object detection program and I keep getting the following error message:
AssertionError: Torch not compiled with CUDA enabled.
I don't understand why this happens, as I have a 2017 MacBook Pro with an AMD GPU, so I have no CUDA enabled GPU.
I added this statement in my code to make sure the device is set to 'cpu', however, it looks as if the program keeps trying to run it through a GPU even though it does not exist.
if torch.cuda.is_available():
    device = torch.device('cuda')

else:
    device = torch.device('cpu')

This is the place where the error happens (4th line):
    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        print("Hey")
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        print("Hey")
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

It would be really great, if anyone could help me with this issue!
Thanks everyone in advance!
PS: I already tried updating the Pytorch version, but still same problem.
Error output:

import os
import pandas as pd
import torch
import torch.utils.data
import torchvision
from PIL import Image
import utils
from engine import train_one_epoch, evaluate
import transforms as T
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor

def parse_one_annot(path_to_data_file, filename):
    data = pd.read_csv(path_to_data_file)
    boxes_array = data[data["filename"] == filename][["xmin", "ymin", "xmax", "ymax"]].values

    return boxes_array

class RaccoonDataset(torch.utils.data.Dataset):

    def __init__(self, root, data_file, transforms=None):
        self.root = root
        self.transforms = transforms
        self.imgs = sorted(os.listdir(os.path.join(root, "images")))
        self.path_to_data_file = data_file

    def __getitem__(self, idx):
        # load images and bounding boxes
        img_path = os.path.join(self.root, "images", self.imgs[idx])
        img = Image.open(img_path).convert("RGB")
        box_list = parse_one_annot(self.path_to_data_file,
                                   self.imgs[idx])
        boxes = torch.as_tensor(box_list, dtype=torch.float32)

        num_objs = len(box_list)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)
            return img, target

    def __len__(self):
        return len(self.imgs)

dataset = RaccoonDataset(root="./raccoon_dataset", data_file="./raccoon_dataset/data/raccoon_labels.csv")

dataset.__getitem__(0)

def get_model(num_classes):
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)

    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new on
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    return model

def get_transform(train):
    transforms = []
    # converts the image, a PIL image, into a PyTorch Tensor
    transforms.append(T.ToTensor())

    if train:
        # during training, randomly flip the training images
        # and ground-truth for data augmentation
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def main():

    dataset = RaccoonDataset(root="./raccoon_dataset",
                             data_file="raccoon_dataset/data/raccoon_labels.csv",
                             transforms=get_transform(train=True))

    dataset_test = RaccoonDataset(root="./raccoon_dataset",
                                  data_file="raccoon_dataset/data/raccoon_labels.csv",
                                  transforms=get_transform(train=False))

    torch.manual_seed(1)
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-40])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-40:])

    # define training and validation data loaders
    data_loader = torch.utils.data.DataLoader(dataset, batch_size=2, shuffle=True, num_workers=4,
                                              collate_fn=utils.collate_fn)
    data_loader_test = torch.utils.data.DataLoader(dataset_test, batch_size=1, shuffle=False, num_workers=4,
                                                   collate_fn=utils.collate_fn)

    print("We have: {} examples, {} are training and {} testing".format(len(indices), len(dataset), len(dataset_test)))

    if torch.cuda.is_available():
        device = torch.device('cuda')
    else:
        device = torch.device('cpu')

    num_classes = 2
    model = get_model(num_classes)

    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005, momentum=0.9, weight_decay=0.0005)

    # and a learning rate scheduler which decreases the learning rate by
    # 10x every 3 epochs

    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=3, gamma=0.1)

    # let's train it for 10 epochs

    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

    os.mkdir("pytorch object detection/raccoon/")
    torch.save(model.state_dict(), "pytorch object detection/raccoon/model")

if __name__ == '__main__':
    main()


title: Why does my SRGAN (using PyTorch) result look similar to SRResNet results?

body:
SRGAN was implemented using PyTorch.

Generator pre-train was conducted in 100 times and the SRGAN train was conducted in 200 times.

The code is a combination of the existing github codes.

For the content loss, MSELoss () in PyTorch was used and BCELoss () in PyTorch was used for adversarial loss.

When I ran code, LossD converges to 0, and LossG oscillates around a certain value. So I stopped training because I thought it was not training anymore.

If the training is 1e5 as in the paper, will the result change? Or is it a matter of loss function?

Below is the SRGAN training code.

print('Adversarial training')
for epoch in range(NUM_EPOCHS):
    train_bar = tqdm(train_loader)
    running_results = {'batch_sizes': 0, 'd_loss': 0, 'g_loss': 0, 'd_score': 0, 'g_score': 0}
    # train_bar = tqdm(train_loader)
    for data, target in train_bar:
        batch_size = data.size(0)
        running_results['batch_sizes'] += batch_size

        target_real = Variable(torch.ones(batch_size, 1))
        target_fake = Variable(torch.zeros(batch_size, 1))

        if torch.cuda.is_available():
            target_real = target_real.cuda()
            target_fake = target_fake.cuda()

        real_img = Variable(target)
        z = Variable(data)

        # Generate real and fake inputs
        if torch.cuda.is_available():
            inputsD_real = real_img.cuda()
            inputsD_fake = netG(z.cuda())
        else:
            inputsD_real = real_img
            inputsD_fake = netG(z)

        ######### Train discriminator #########
        netD.zero_grad()

        # With real data
        outputs = netD(inputsD_real)
        D_real = outputs.data.mean()

        lossD_real = adversarial_criterion(outputs, target_real)

        # With fake data
        outputs = netD(inputsD_fake.detach()) # Don't need to compute gradients wrt weights of netG (for efficiency)
        D_fake = outputs.data.mean()

        lossD_fake = adversarial_criterion(outputs, target_fake)

        lossD_total = lossD_real + lossD_fake

        lossD_total.backward()

        # Update discriminator weights
        optimizerD.step()

        ######### Train generator #########
        netG.zero_grad()

        real_features = Variable(feature_extractor(inputsD_real).data)
        fake_features = feature_extractor(inputsD_fake)

        lossG_vgg19 = content_criterion(fake_features, real_features)
        lossG_adversarial = adversarial_criterion(netD(inputsD_fake).detach(), target_real)
        lossG_mse = content_criterion(inputsD_fake, inputsD_real)

        lossG_total = lossG_mse + 2e-6 * lossG_vgg19 + 0.001 * lossG_adversarial
        lossG_total.backward()

        # Update generator weights
        optimizerG.step()


title: Why USE_OPENCV is OFF in PyTorch installation?

body:
In CMakeLists.txt in pytorch folder, use Opencv is ON.

option(USE_OPENCL "Use OpenCL" OFF)
option(USE_OPENCV "Use OpenCV" ON)
option(USE_OPENMP "Use OpenMP for parallel code" OFF)

In setup.py, variable is set with Opencv as

use_env_vars = ['CUDA', 'CUDNN', 'MIOPEN', 'MKLDNN', 'NNPACK', 'DISTRIBUTED',
                'OPENCV', 'SYSTEM_NCCL', 'GLOO_IBVERBS']

My PC has Opencv installed and can check with
pkg-config --modversion opencv gives me
3.3.0.

OpenCV is installed in my PC.
But during Caffe2 installation, it showed me

--   USE_NERVANA_GPU       : OFF
--   USE_NNPACK            : 1
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : OFF
--   USE_PROF              : OFF
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF

OpenCV is OFF. How can I ON OpenCV in Caffe2?


title: How does one determine when the CartPole environment has been solved?

body:
I was going through this tutorial and saw the following piece of code:

        # Calculate score to determine when the environment has been solved
        scores.append(time)
        mean_score = np.mean(scores[-100:])

        if episode % 50 == 0:
            print('Episode {}\tAverage length (last 100 episodes): {:.2f}'.format(
                episode, mean_score))

        if mean_score > env.spec.reward_threshold:
            print("Solved after {} episodes! Running average is now {}. Last episode ran to {} time steps."
                  .format(episode, mean_score, time))
            break

however, it didn't really made sense to me. How does one define when a "RL environment has been solved"? Not sure what that even means. I guess in classification it would make sense to define it to be when loss is zero. In regression maybe when the total l2 loss is less than some value? Perhaps it would have made sense to define it when the expected returns (discounted rewards) is greater than some value.

But here it seems they are counting the # of time steps? This doesn't make any sense to me.

Note the original tutorial had this:

def main(episodes):
    running_reward = 10
    for episode in range(episodes):
        state = env.reset() # Reset environment and record the starting state
        done = False

        for time in range(1000):
            action = select_action(state)
            # Step through environment using chosen action
            state, reward, done, _ = env.step(action.data[0])
# Save reward
            policy.reward_episode.append(reward)
            if done:
                break

        # Used to determine when the environment is solved.
        running_reward = (running_reward * 0.99) + (time * 0.01)
update_policy()
if episode % 50 == 0:
            print('Episode {}\tLast length: {:5d}\tAverage length: {:.2f}'.format(episode, time, running_reward))
if running_reward > env.spec.reward_threshold:
            print("Solved! Running reward is now {} and the last episode runs to {} time steps!".format(running_reward, time))
            break

not sure if this makes much more sense...

is this only a particular quirk of this environment/task? How does the task end in general?


title: PyTorch broadcasting: how this worked?

body:
I'm new to Deep Learning. I'm studying from Udacity.

I came across one of the codes to build up a neural network, where 2 tensors are being added, specifically the 'bias' tensor with the output of the tensor-multiplication product.

It was kind of...

def activation(x):
return (1/(1+torch.exp(-x)))

inputs = images.view(images.shape[0], -1)
w1 = torch.randn(784, 256)
b1 = torch.randn(256)
h = activation(torch.mm(inputs,w1) + b1)

After flattening the MNIST, it came out as [64,784] (inputs).

I'm not getting how the bias tensor (b1) of dimension [256], could be added to the multiplication product of 'inputs' and 'w1' which comes out to be the dimensions of [256, 64].


title: Converting keras code to pytorch code with Conv1D layer

body:
I have some keras code that I need to convert to Pytorch. I've done some research but so far I am not able to reproduce the results I got from keras. I have spent many hours on this any tips or help is very appreciated.

Here is the keras code I am dealing with. The input shape is (None, 105, 768) where None is the batch size and I want to apply Conv1D to the input. The desire output in keras is (None, 105)

x = tf.keras.layers.Dropout(0.2)(input)
x = tf.keras.layers.Conv1D(1,1)(x)
x = tf.keras.layers.Flatten()(x)
x = tf.keras.layers.Activation('softmax')(x)

What I've tried, but worse in term of results:

    self.conv1d = nn.Conv1d(768, 1, 1)
    self.dropout = nn.Dropout(0.2)
    self.softmax = nn.Softmax()

def forward(self, input):

    x = self.dropout(input)
    x = x.view(x.shape[0],x.shape[2],x.shape[1])
    x = self.conv1d(x)
    x = torch.squeeze(x, 1)
    x = self.softmax(x)


title: PyTorch error: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

body:
I have a very simple example
import torch

if __name__ == "__main__":
    DEVICE = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    m = torch.nn.Linear(20, 30).to(DEVICE)
    input = torch.randn(128, 20).to(DEVICE)
    output = m(input)
    print('output', output.size())
    exit()

and I get:
Traceback (most recent call last):
  File "test.py", line 9, in <module>
    output = m(input)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_INTERNAL_ERROR when calling `cublasCreate(handle)`

I'm using PyTorch 1.7.1. Any help would be greatly appreciated.
Thank you.
EDIT. The update of python -m torch.utils.collect_env is:
Collecting environment information...
PyTorch version: 1.8.0
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.2 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: 11.1.0
CMake version: version 3.18.4

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: TITAN RTX
GPU 1: TITAN RTX
GPU 2: TITAN RTX
GPU 3: TITAN RTX
GPU 4: TITAN RTX
GPU 5: TITAN RTX
GPU 6: TITAN RTX
GPU 7: TITAN RTX

Nvidia driver version: 460.39
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.20.1
[pip3] torch==1.8.0
[pip3] torchaudio==0.8.0
[pip3] torchsummary==1.5.1
[conda] Could not collect


title: Gradient flow through torch.nn.Parameter()

body:
I have a toy example

a = torch.ones(10)
b = torch.nn.Parameter(a,requires_grad=True)
c = (b**2).sum()
c.backward()
print(b.grad)
print(a.grad)

b.grad calculated successfully, but a.grad is None. How to make gradient flow through torch.nn.Parameter? This example looks artificial, but I work with class A derived from nn.Module and it's parameters initialized with outputs from some other Module B, and I whant to make gradients flow through A parameters to B parameters.


title: Pytorch DCGAN example doesn't work with different image sizes

body:
I'm trying to use this code as a starting point for building GANs from my own image data set of 224x224 images. Image size is included in the default arguments (e.g. --imageSize 224).

Here is the code snippet with the model:

class _netG(nn.Module):
def __init__(self, ngpu):
    super(_netG, self).__init__()
    self.ngpu = ngpu
    self.main = nn.Sequential(
        # input is Z, going into a convolution
        nn.ConvTranspose2d(     nz, ngf * 8, 4, 1, 0, bias=False),
        nn.BatchNorm2d(ngf * 8),
        nn.ReLU(True),
        # state size. (ngf*8) x 4 x 4
        nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
        nn.BatchNorm2d(ngf * 4),
        nn.ReLU(True),
        # state size. (ngf*4) x 8 x 8
        nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
        nn.BatchNorm2d(ngf * 2),
        nn.ReLU(True),
        # state size. (ngf*2) x 16 x 16
        nn.ConvTranspose2d(ngf * 2,     ngf, 4, 2, 1, bias=False),
        nn.BatchNorm2d(ngf),
        nn.ReLU(True),
        # state size. (ngf) x 32 x 32
        nn.ConvTranspose2d(    ngf,      nc, 4, 2, 1, bias=False),
        nn.Tanh()
        # state size. (nc) x 64 x 64
    )

def forward(self, input):
    if isinstance(input.data, torch.cuda.FloatTensor) and self.ngpu > 1:
        output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))
    else:
        output = self.main(input)
    return output

class _netD(nn.Module):
def __init__(self, ngpu):
    super(_netD, self).__init__()
    self.ngpu = ngpu
    self.main = nn.Sequential(
        # input is (nc) x 64 x 64
        nn.Conv2d(nc, ndf, 4, 2, 1, bias=False),
        nn.LeakyReLU(0.2, inplace=True),
        # state size. (ndf) x 32 x 32
        nn.Conv2d(ndf, ndf * 2, 4, 2, 1, bias=False),
        nn.BatchNorm2d(ndf * 2),
        nn.LeakyReLU(0.2, inplace=True),
        # state size. (ndf*2) x 16 x 16
        nn.Conv2d(ndf * 2, ndf * 4, 4, 2, 1, bias=False),
        nn.BatchNorm2d(ndf * 4),
        nn.LeakyReLU(0.2, inplace=True),
        # state size. (ndf*4) x 8 x 8
        nn.Conv2d(ndf * 4, ndf * 8, 4, 2, 1, bias=False),
        nn.BatchNorm2d(ndf * 8),
        nn.LeakyReLU(0.2, inplace=True),
        # state size. (ndf*8) x 4 x 4
        nn.Conv2d(ndf * 8, 1, 4, 1, 0, bias=False),
        nn.Sigmoid()
    )

def forward(self, input):
    if isinstance(input.data, torch.cuda.FloatTensor) and self.ngpu > 1:
        output = nn.parallel.data_parallel(self.main, input, range(self.ngpu))
    else:
        output = self.main(input)

    return output.view(-1, 1).squeeze(1)

If I change any of the default arguments (e.g. --imageSize 224) I get the following error:

Traceback (most recent call last):
   File "main.py", line 209, in <module>
      errD_real = criterion(output, label)
   File "/opt/python/lib/python3.6/site-
      packages/torch/nn/modules/module.py", line 210, in __call__
        result = self.forward(*input, **kwargs)
     File "/opt/python/lib/python3.6/site-
    packages/torch/nn/modules/loss.py", line 36, in forward
     return backend_fn(self.size_average, weight=self.weight)(input,
   target)
    File "/opt/python/lib/python3.6/site-
      packages/torch/nn/_functions/thnn/loss.py", line 22, in forward
      assert input.nelement() == target.nelement()
     AssertionError

I was told at first to try other models, but I also thought it would be good to try to get around this issue. I've attempted the solutions outline in this github thread, but it still doesn't work. I'm still learning my way around PyTorch so the network architectures that are spit out before the above message don't yet give me much intuition. I appreciate any pointers you can give!


title: Is there any method to generate a piecewise function for tensors in pytorch?

body:
I want to get a piecewise function like this for tensors in pytorch. But I don't know how to define it. I use a very stupid method to do it, but it seems not to work in my code.

    def trapezoid(self, X):
        Y = torch.zeros(X.shape)
        Y[X % (2 * pi) < (0.5 * pi)] = (X[X % (2 * pi) < (0.5 * pi)] % (2 * pi)) * 2 / pi
        Y[(X % (2 * pi) >= (0.5 * pi)) & (X % (2 * pi) < 1.5 * pi)] = 1.0
        Y[X % (2 * pi) >= (1.5 * pi)] = (X[X % (2 * pi) >= (1.5 * pi)] % (2 * pi)) * (-2 / pi) + 4
        return Y

could do you help me find out how to design the function trapezoid, so that for tensor X, I can get the result directly using trapezoid(X)?


title: BertSumExt is not producing Summaries

body:
I was trying to get the extractive BertSUM summarizer working (Paper and Github here)
but i still get the following message

xent 0 at step -1"

and no summary is produced. What i'am doing wrong? Can someone please help me with that,  perhaps provide a working example. the above message apppeared when i did the following in google colab:

1 clone requiered GitHub

!git clone https://github.com/Alcamech/PreSumm.git

2 Change Git-Branch for summarization of raw text data

%cd /content/PreSumm
!git checkout -b  Raw_Input origin/PreSumm_Raw_Input_Text_Setup
!git pull

3 install requirements

!pip install torch==1.1.0 pytorch_transformers tensorboardX multiprocess pyrouge

4 install CNN/DM Extractive bertext_cnndm_transformer.pt

!gdown https://drive.google.com/uc?id=1kKWoV0QCbeIuFt85beQgJ4v0lujaXobJ&export=download
!unzip /content/PreSumm/models/bertext_cnndm_transformer.zip

4.1  Download the Pre-Processed data for CNN/Dailymail

%cd /content/PreSumm/bert_data/
!gdown https://drive.google.com/uc?id=1DN7ClZCCXsk2KegmC6t4ClBwtAf5galI&export=download
!unzip /content/PreSumm/bert_data/bert_data_cnndm_final.zip

5 change to /src folder

cd /content/PreSumm/src/

6 run the extractive summarizer

!python /content/PreSumm/src/train.py -task ext -mode test_text -test_from /content/PreSumm/models/bertext_cnndm_transformer.pt -text_src /content/PreSumm/raw_data/temp_ext.raw_src -text_tgt /content/PreSumm/results/result.txt -log_file /content/PreSumm/logs/ext_bert_cnndm

The Output of Step 6 is:

[2020-05-07 11:20:12,355 INFO] Loading checkpoint from /content/PreSumm/models/bertext_cnndm_transformer.pt
Namespace(accum_count=1, alpha=0.6, batch_size=140, beam_size=5, bert_data_path='../bert_data_new/cnndm', beta1=0.9, beta2=0.999, block_trigram=True, dec_dropout=0.2, dec_ff_size=2048, dec_heads=8, dec_hidden_size=768, dec_layers=6, enc_dropout=0.2, enc_ff_size=512, enc_hidden_size=512, enc_layers=6, encoder='bert', ext_dropout=0.2, ext_ff_size=2048, ext_heads=8, ext_hidden_size=768, ext_layers=2, finetune_bert=True, generator_shard_size=32, gpu_ranks=[0], label_smoothing=0.1, large=False, load_from_extractive='', log_file='/content/PreSumm/logs/ext_bert_cnndm', lr=1, lr_bert=0.002, lr_dec=0.002, max_grad_norm=0, max_length=150, max_ndocs_in_batch=6, max_pos=512, max_tgt_len=140, min_length=15, mode='test_text', model_path='../models/', optim='adam', param_init=0, param_init_glorot=True, recall_eval=False, report_every=1, report_rouge=True, result_path='../results/cnndm', save_checkpoint_steps=5, seed=666, sep_optim=False, share_emb=False, task='ext', temp_dir='../temp', test_all=False, test_batch_size=200, test_from='/content/PreSumm/models/bertext_cnndm_transformer.pt', test_start_from=-1, text_src='/content/PreSumm/raw_data/temp_ext.raw_src', text_tgt='/content/PreSumm/results/result.txt', train_from='', train_steps=1000, use_bert_emb=False, use_interval=True, visible_gpus='-1', warmup_steps=8000, warmup_steps_bert=8000, warmup_steps_dec=8000, world_size=1)
[2020-05-07 11:20:13,361 INFO] https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json not found in cache or force_download set to True, downloading to /tmp/tmpvck0jwoy
100% 433/433 [00:00<00:00, 309339.74B/s]
[2020-05-07 11:20:13,498 INFO] copying /tmp/tmpvck0jwoy to cache at ../temp/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517
[2020-05-07 11:20:13,499 INFO] creating metadata file for ../temp/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517
[2020-05-07 11:20:13,499 INFO] removing temp file /tmp/tmpvck0jwoy
[2020-05-07 11:20:13,499 INFO] loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-config.json from cache at ../temp/4dad0251492946e18ac39290fcfe91b89d370fee250efe9521476438fe8ca185.7156163d5fdc189c3016baca0775ffce230789d7fa2a42ef516483e4ca884517
[2020-05-07 11:20:13,500 INFO] Model config {
  "architectures": [
    "BertForMaskedLM"
  ],
  "attention_probs_dropout_prob": 0.1,
  "finetuning_task": null,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "num_labels": 2,
  "output_attentions": false,
  "output_hidden_states": false,
  "pad_token_id": 0,
  "pruned_heads": {},
  "torchscript": false,
  "type_vocab_size": 2,
  "vocab_size": 30522
}

[2020-05-07 11:20:13,571 INFO] https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin not found in cache or force_download set to True, downloading to /tmp/tmp6b78t4_2
100% 440473133/440473133 [00:06<00:00, 71548841.10B/s]
[2020-05-07 11:20:19,804 INFO] copying /tmp/tmp6b78t4_2 to cache at ../temp/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
[2020-05-07 11:20:21,212 INFO] creating metadata file for ../temp/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
[2020-05-07 11:20:21,212 INFO] removing temp file /tmp/tmp6b78t4_2
[2020-05-07 11:20:21,267 INFO] loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-pytorch_model.bin from cache at ../temp/aa1ef1aede4482d0dbcd4d52baad8ae300e60902e88fcb0bebdec09afd232066.36ca03ab34a1a5d5fa7bc3d03d55c4fa650fed07220e2eeebc06ce58d0e9a157
gpu_rank 0
[2020-05-07 11:20:24,645 INFO] * number of parameters: 120512513
[2020-05-07 11:20:24,736 INFO] https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache or force_download set to True, downloading to /tmp/tmpyv3mwnb6
100% 231508/231508 [00:00<00:00, 4268647.82B/s]
[2020-05-07 11:20:25,044 INFO] copying /tmp/tmpyv3mwnb6 to cache at /root/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
[2020-05-07 11:20:25,045 INFO] creating metadata file for /root/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
[2020-05-07 11:20:25,045 INFO] removing temp file /tmp/tmpyv3mwnb6
[2020-05-07 11:20:25,046 INFO] loading vocabulary file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at /root/.cache/torch/pytorch_transformers/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
  0% 0/2 [00:00<?, ?it/s]
[2020-05-07 11:20:25,115 INFO] Validation xent: 0 at step -1

and the result.txt-file is empty.

Here is a link to  a copy of my google colab, where you can see the full colde.
I also tried these steps  on the origin-github-repo  here and i get the same error.
Thanks for any help.


title: Why doesn't Detectron2 not detect any other instances after transfer learning?

body:
I have followed the tutorial from this website https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5#scrollTo=U5LhISJqWXgM.
The question is: there were images that contained 'person' and many other instances but why weren't they detected and segmented?

from detectron2.utils.visualizer import ColorMode
dataset_dicts = get_balloon_dicts("balloon/val")
for d in random.sample(dataset_dicts, 3):
    im = cv2.imread(<CUSTOM_IMAGE_CONTAINING_PERSON_DETECTED_WHEN_CUSTOM_WASNT_USED>) <--changed
    outputs = predictor(im)
    v = Visualizer(im[:, :, ::-1],
                   metadata=balloon_metadata,
                   scale=0.8,
                   instance_mode=ColorMode.IMAGE_BW   # remove the colors of unsegmented pixels
    )
    v = v.draw_instance_predictions(outputs["instances"].to("cpu"))
    cv2_imshow(v.get_image()[:, :, ::-1])

After downloading a custom photo containing people I ran inference on the image as instructured under Run a pre-trained detectron2 model using the model they used (model_zoo.get_checkpoint_url("COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml")) but when I use the trained model of the balloons it does not detect the people in the image (the threshold was not the problem since I used 0.5 in both cases). Why is it so and how would I be able to make it show all the instances? Help would be greatly appreciated :D