title: Pre-Trained Models in Keras,TorchVision

body:
I Have The Following Code which use pre-trained ResNet50 Model in Keras with imagenet DataSet:

from keras.applications.resnet50 import ResNet50
from keras.preprocessing import image
from keras.applications.resnet50 import preprocess_input, decode_predictions
import numpy as np

model = ResNet50(weights='imagenet')
print(model)

and it works fine..My question is can i find a pre-trained model inside Keras or Torchvision or TensorFlow But to one of the following:

1) LeNet5 for MNIST DataSet
2) 32-Layer ResNet for the CIFAR-10 Dataset

I know the alternative is to train the LeNet5 for example on my own, but a pre-trained model will be preferred and as far as I searched I didn't find them.
thank you.


title: Model() got multiple values for argument 'nr_class' - SpaCy multi-classification model (BERT integration)

body:
Hi I am working on implementing a multi-classification model (5 classes) with the new SpaCy Model en_pytt_bertbaseuncased_lg. The code for the new pipe is here:

nlp = spacy.load('en_pytt_bertbaseuncased_lg')
textcat = nlp.create_pipe(
    'pytt_textcat',
    config={
        "nr_class":5,
        "exclusive_classes": True,
    }
)
nlp.add_pipe(textcat, last = True)

textcat.add_label("class1")
textcat.add_label("class2")
textcat.add_label("class3")
textcat.add_label("class4")
textcat.add_label("class5")

The code for the training is as follows and is based on the example from here(https://pypi.org/project/spacy-pytorch-transformers/):

def extract_cat(x):
    for key in x.keys():
        if x[key]:
            return key

# get names of other pipes to disable them during training
n_iter = 250 # number of epochs

train_data = list(zip(train_texts, [{"cats": cats} for cats in train_cats]))

dev_cats_single   = [extract_cat(x) for x in dev_cats]
train_cats_single = [extract_cat(x) for x in train_cats]
cats = list(set(train_cats_single))
recall = {}
for c in cats:
    if c is not None:
        recall['dev_'+c] = []
        recall['train_'+c] = []

optimizer = nlp.resume_training()
batch_sizes = compounding(1.0, round(len(train_texts)/2), 1.001)

for i in range(n_iter):
    random.shuffle(train_data)
    losses = {}
    batches = minibatch(train_data, size=batch_sizes)
    for batch in batches:
        texts, annotations = zip(*batch)
        nlp.update(texts, annotations, sgd=optimizer, drop=0.2, losses=losses)
    print(i, losses)

So the structure of my data looks like this:

[('TEXT TEXT TEXT',
  {'cats': {'class1': False,
    'class2': False,
    'class3': False,
    'class4': True,
    'class5': False}}), ... ]

I am not sure why I get the following error:

TypeError                                 Traceback (most recent call last)
<ipython-input-32-1588a4eadc8d> in <module>
     21
     22
---> 23 optimizer = nlp.resume_training()
     24 batch_sizes = compounding(1.0, round(len(train_texts)/2), 1.001)
     25

TypeError: Model() got multiple values for argument 'nr_class'

EDIT:

if I take out the nr_class argument, I get this error here:

ValueError: operands could not be broadcast together with shapes (1,2) (1,5)

I actually thought this would happen because I didn't specify the nr_class argument. Is that correct?


title: save embedding layer in pytorch model

body:
i have this model:

class model(nn.Module):
    def __init__(self):
      super().__init__()
      self.conv1 = nn.Conv2d(in_channels=12,out_channels=64,kernel_size=3,stride= 1,padding=1)
      # self.conv2 = nn.Conv2d(in_channels=64,out_channels=64,kernel_size=3,stride= 1,padding=1)
      self.fc1 = nn.Linear(24576, 128)
      self.bn = nn.BatchNorm1d(128)
      self.dropout1 = nn.Dropout2d(0.5)
      self.fc2 = nn.Linear(128, 10)
      self.fc3 = nn.Linear(10, 3)

    def forward(self, x):
      x = F.relu(self.conv1(x))
      # x = F.relu(self.conv2(x))
      x = F.max_pool2d(x, (2,2))
      # print(x.shape)
      x = x.view(-1,24576)
      x = self.bn(F.relu(self.fc1(x)))
      x = self.dropout1(x)
      embeding_stage = F.relu(self.fc2(x))
      x = self.fc3(embeding_stage)

      return x

and i want to save the embeding_stage layer like i save the model here:

model = model()
torch.save(model.state_dict(), 'C:\project\count_speakers\model_pytorch.h5')

thanks,
Ayal


title: Pytorch Dataloader shuffle with multiple dataset

body:
I'm trying to make custom Dataloader with multiple datasets.

My question is that if I use (shuffle = True) in the Dataloader option, is it possible to shuffle the same order in multiple Dataloader?

For example:

dataloader1: label = [5 , 4, 15, 16]

dataloader2: label = [5 , 4, 15, 16]


title: concatenating two tensors in pytorch(with a twist)

body:
I have a word embedding (of type tensor)of size torch.Size([8, 768]) stored in the variable embeddings which looks something like this :-

 tensor([[-0.0687, -0.1327,  0.0112,  ...,  0.0715, -0.0297, -0.0477],
        [ 0.0115, -0.0029,  0.0323,  ...,  0.0277, -0.0297, -0.0599],
        [ 0.0760,  0.0788,  0.1640,  ...,  0.0574, -0.0805,  0.0066],
        ...,
        [-0.0110, -0.1773,  0.1143,  ...,  0.1397,  0.3021,  0.1670],
        [-0.1379, -0.0294, -0.0026,  ..., -0.0966, -0.0726,  0.1160],
        [ 0.0466, -0.0113,  0.0283,  ..., -0.0735,  0.0496,  0.0963]],
       grad_fn=<IndexBackward>)

Now, I wish to take mean of some embeddings and place the mean back in the tensor.
For example,(I'll explain with the help of list and not tensor)

a = [1,2,3,4,5]
output = [1.5, 3, 4, 5]

So, here I have taken mean of 1 and 2 and then placed in the list output by shifting the elements to the  left in the list. I want to do the same thing for tensors as well.

I have the index stored in variable i from where I need to take average and j variable is being used for the stopping index. Now, let's look into the code :-

if i != len(embeddings):
  sum = 0
  count = 0
  #Calculating sum
  for x in range(i-1, j):
    sum += text_index[x]
    count += 1

  avg = sum/count

  #Inserting the average in place of the other embeddings
  embeddings = embeddings[:i-1] + [avg] + embeddings[j:]
else :
  pass

Now, I am getting an error at this line embeddings = embeddings[:i-1] + [avg] + embeddings[j:]
The error is :-

TypeError: unsupported operand type(s) for +: 'Tensor' and 'list'

Now, I understand that the above code would have worked well if embeddings was a list but it is a tensor. How do I do it?

NOTE :

*1. *embeddings.shape : torch.Size([8, 768])
 2.  avg is of type float**


title: CUDA out of memory even though there is free memory when using the simple-transformers pytorch library

body:
Ive been trying to follow this tutorial for the simple transformers library but I keep running into cuda errors that I dont quite understand.
RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 6.00 GiB total capacity; 697.82 MiB already allocated; 3.99 GiB free; 744.00 MiB reserved in total by PyTorch)
I dont understand why its failing to allocate memory when it says there is 4GB free. Im also seeing this error in the stack trace
OSERROR: [WinError 1455] The paging file is too small for this operation to complete
occasionally this one will also pop up.
OSERROR: [WinError 1114] A dynamic link library (DLL) initialization routine failed
Dont really know where to go from here.


title: The bounding box's position and size is incorrect, how to improve it's accuracy?

body:
I'm using detectron2 for solving a segmentation task,
I'm trying to classify an object into 4 classes,
so I have used COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml.
I have applied 4 kind of augmentation transforms and after training I get about 0.1
total loss.
But for some reason the accuracy of the bbox is not great on some images on the test set,
the bbox is drawn either larger or smaller or doesn't cover the whole object.
Moreover sometimes the predictor draws few bboxes, it assumes there are few different objects although there is only a single object.
Are there any suggestions how to improve it's accuracy?
Are there any good practice approaches how to resolve this issue?
Any suggestion or reference material will be helpful.


I want to install torch==1.7.0+cpu from requirements.txt
I have an error

Could not find a version that satisfies the requirement torch==1.7.0+cpu (from -r requirements.txt (line 42)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.3.0, 1.3.1, 1.4.0, 1.5.0, 1.5.1, 1.6.0, 1.7.0, 1.7.1, 1.8.0, 1.8.1)
No matching distribution found for torch==1.7.0+cpu

I am using
Python 3.6.12 (default, Aug 18 2020, 02:08:22)
[GCC 5.4.0 20160609] on linux


title: CUDNN_STATUS_INTERNAL_ERROR when using both Pytorch and TensorFlow

body:
I'm running a program to process some data, and I inference both a TensorFlow model and a Pytorch model.

When inferencing either of the models everything works fine. However, when I add the pytorch input my program crashes with this error:

2018-05-14 12:55:05.525251: E tensorflow/stream_executor/cuda/cuda_dnn.cc:385] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018-05-14 12:55:05.525280: F tensorflow/core/kernels/conv_ops.cc:717] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms)

Note that this already happens before I do anything with Pytorch. No models are loaded, nothing is put on GPU, no devices are checked.

Does anyone know what might be going wrong, how to fix it, and if there are some parameters I can change?

Something I already tried is disabling the PyTorch backend using this code:

import torch.backends.cudnn as cudnn
cudnn.enabled = False

But unfortunately this does not help...


title: PyTorch copy parameter gets stuck in multiprocessing if parameters too big

body:
I'm trying to code an Asynchronous Actor Critic in PyTorch based on this repo: https://github.com/seungeunrho/minimalRL/blob/master/a3c.py
but I'm changing the ActorCritic class to use the one I coded myself.

Basically I have a class A3C, an instance of it, global_model, with shared memory and I use torch.multiprocessing to open some Processes in order to train the model in parallel. In each process at the beginning I have to create a new instance of the model, called local_model, in order to proceed with the training, but the process gets stuck in the initialization of the local model even though the one of the global model works every time.

Trying to debugging it I can see that it enters the A3C.init function and the SharedActorCritic.init too, but there it stops just after I put the checkpoint print. However if I print whatever expression contains list(critic_param_gen) magically everything works. I also noted that printing just critic_param_gen won't do.

Any idea of why is that?

Also a similar thing happens if I use local_model = copy.deepcopy(global_model) as a function create_local_model, i.e. only works if that print is present.

In pseudo-code:

import torch.multiprocessiA3Cng as mp
import torch.nn as nn
import itertools as it

debug = True

A3C(nn.Module):
  def __init__(self, model, n_features):
     ...
     self.AC_architecture = SharedActorCritic(model, n_features)

class SharedActorCritic(nn.Module):
    def __init__(self, model, n_features):
        super(SharedActorCritic, self).__init__()

        self.shared_architecture = model(n_features) # inherits from nn.Module
        self.actor = SharedActor(n_features) # inherits from nn.Module
        self.critic = SharedCritic(n_features) # inherits from nn.Module

        self.critic_target = BaseCritic(model, n_features) # inherits from nn.Module

        critic_param_gen = it.chain(self.shared_architecture.parameters(), self.critic.parameters())
        print("checkpoint")
        if debug: print(list(critic_param_gen)) # this makes the whole thing work
        for trg_params, params in zip(self.critic_target.parameters(), critic_param_gen ):
            trg_params.data.copy_(params.data)

def create_local_model(model, n_features):
    local_model = A3C(model, n_features)
    print("Process ended")

# in the main
global_model = Model() # works
global_model.share_memory() # doesn't really matter

p = mp.Process(target=create_local_model, args=(model, n_features, ))
p.start()
print("Process started")
p.join()

----
# output if debug is True
Process started
checkpoint
[ ...actual list of critic_param_gen ... ]
Process ended

# output if debug is False
Process started
checkpoint
# and then runs forever

Edit: solved the mystery about the print statement thanks to snakecharmerb. I created a minimal reproducible example. It seems that if the network is large enough, the copy operation breaks if executed in a process, but not outside of it (since global model can be instantiated).

import torch.nn as nn
import torch.multiprocessing as mp
import copy

class Net(nn.Module):
    def __init__(self, n_features=256, n_layers=8):
        super(Net, self).__init__()
        self.net1 = nn.Sequential(*nn.ModuleList([nn.Linear(n_features, n_features) for _ in range(n_layers)]))
        self.net2 = nn.Sequential(*nn.ModuleList([nn.Linear(n_features, n_features) for _ in range(n_layers)]))

        for p1, p2 in zip(self.net1.parameters(), self.net2.parameters()):
            p1.data.copy_(p2.data)

    def forward(self, x):
        return self.net(x)

def create_local_model_v1(global_model):
    local_model = copy.deepcopy(global_model)
    print("Process ended")

%%time
global_model = Net(16,2)
print("Global model created")
p = mp.Process(target=create_local_model_v1, args=(global_model,))
p.start()
print("Process started")
p.join()

# Output
Global model created
Process ended
Process started
CPU times: user 3 ms, sys: 11.9 ms, total: 14.9 ms
Wall time: 45.1 ms

%%time
global_model = Net(256,8)
print("Global model created")
p = mp.Process(target=create_local_model_v1, args=(global_model,))
p.start()
print("Process started")
p.join()

# Output - Gets stuck
Global model created
Process started


title: Implementing one to many LSTM/RNN, PyTorch

body:
I have a matrix sized m x n, and want to predict by 1 x n  vector (x at the picture with the network structure) the whole next (m-1) x n matrix (y^{i} at the picture), using RNN or LSTM, I don't understand how to implement feeding each
1 x n vector to the next hidden state and get all the
(m-1) x n vectors simultaneously and how to compute error over all y^{i}
I have this vanilla RNN-model and don't know how to modify it
class RNNModel(nn.Module):
def __init__(self, input_dim, hidden_dim, layer_dim, output_dim):
    super(RNNModel, self).__init__()

    self.hidden_dim = hidden_dim

    self.layer_dim = layer_dim

    # (batch_dim, seq_dim, feature_dim)
    self.RNN = nn.RNN(input_dim, hidden_dim, layer_dim, batch_first=True, nonlinearity='tanh')

    self.fc = nn.Linear(hidden_dim, output_dim)

def forward(self, x):
    # Initialize hidden state with zeros
    h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

    out, h_t = self.RNN(x, h0)

    #out = self.fc(h_t[:, -1, :])
    out = self.fc(out[:, -1, :])

    return out


title: Reset model parameters and weights of a network [pytorch] for cross-validation

body:
I want to implement 5-fold cross validation on my training.  Every fold, I need to reset the parameters of the model. I have the following code added before a certain fold is trained:
for i, (train_index, val_index) in enumerate(kf.split(trainset)):
   for layer in model.children():
       if hasattr(layer, 'reset_parameters'):
           layer.reset_parameters()

Each fold is ran with 5 epochs each.  Observing the results, we have:
Fold 1
Best valid accuracy: 0.806000
Fold 2
Best valid accuracy: 0.852000
Fold 3
Best valid accuracy: 0.930900

The accuracy is increasing which means that the model was not re-initialized after training one fold.  Is there a better way of doing this?


title: How does the groups parameter in torch.nn.conv* influence the convolution process?

body:
I want to convolve a multichannel tensor with the same single channel weight.
I could repeat the weight along the channel dimension, but I thought there might be an other way.

I thought the groups parameter might do the job. However I don't understand the documentation.
That's why I want to ask how the groups parameter influences the convolution process ?


title: How to split MNIST dataset into multiple subsets for distributed nodes using Pytorch?

body:
I am implementing DistributedDataParallel training to a simple CNN for torchvision.datasets.MNIST simultaneously running on 3 distributed nodes. I want to partition the datasets into 3 non-overlapping subsets (A,B,C) and that should contain 20000 images each. Individual subsets should be further split into training and testing partitions, i.e. 0.7% training and 0.3% testing. I plan to provide each subset to each distributed node separately so that they can train and test in a DistributedDataParallel fashion.

The basic code as shown below, downloads MNIST dataset from torchvision.datasets.MNIST and then uses torch.utils.data.distributed.DistributedSampler and torch.utils.data.DataLoader to create data batches for training and testing on a single node.

# TRAINING DATA

train_dataset = datasets.MNIST('data', train=True, download=True, transform=transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.1307,), (0.3081,))]))
train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset, num_replicas=3, rank=dist.get_rank())
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=256, shuffle=False, num_workers=3, pin_memory=True, sampler=True)

# TESTING DATA

test_dataset = datasets.MNIST('data', train=False, download=False, transform=transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))]))
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=256, shuffle=False, num_workers=3, pin_memory=True)

I expect the answer should create train_dataset_a, train_dataset_b, and train_dataset_c, as well as, test_dataset_a, test_dataset_b, and test_dataset_c.


title: how to get jacobian with pytorch for log probability of multivariate normal distribution

body:
I've drawn samples from a multivariate normal distribution and would like to get the gradient of their log probability with respect to the mean. Since there are many samples, this requires a Jacobian:

import torch

mu = torch.ones((2,), requires_grad=True)
sigma = torch.eye(2)
dist = torch.distributions.multivariate_normal.MultivariateNormal(mu, sigma)

num_samples=10
samples = dist.sample((num_samples,))
logprobs = dist.log_prob(samples)

Now I would like to get the derivative of each entry in logprobs with respect to each entry in mu.

A simple solution is a python loop:

grads = []
for logprob in logprobs:
    grad = torch.autograd.grad(logprob, mu, retain_graph=True)
    grads.append(grad)

If you stack the grads, the result is the desired Jacobian. Is there also built-in and vectorized support for this?

Related questions/internet ressources:

This is a huge topic, there are lots of related posts. Nevertheless, I think that this specific question (regarding distributions) hasn't been answered yet:

This question is basically the same as mine (but without example code and solution attempt), sadly it is unanswered: Pytorch custom function jacobian gradient
This question shows the calculation of a jacobian in pytorch, but I don't think the solution is applicable to my problem: Pytorch most efficient Jacobian/Hessian calculation It requires stacking the input in a way that seems incompatible with the distribution. I couldn't make it work.
This gist has some code snippets for Jacobians. In principle they are similar to the approach from the question above.


title: how to convert mnist images to variables images and labels

body:
I have a code as below:

dataset = MNIST(path=data_path, download=True, shuffle=True)

if train:
   images, labels = dataset.get_train()
else:
   images, labels = dataset.get_test()

images, labels = images[:n_examples], labels[:n_examples]
images, labels = iter(images.view(-1, 784) / 255), iter(labels)

but when i run it, it gives me this error:

Traceback (most recent call last):
File "C:\Users\Ati\Downloads\Compressed\bindsnet_experiments-
master\experiments\mnist\two_layer_backprop.py", line 135, in <module>
images, labels = dataset.get_train()
AttributeError: 'TorchvisionDatasetWrapper' object has no attribute 'get_train'

I think because get_train() is out of date , it doesn’t support by torchvision
But i tested different ways for converting mnist data to images and labels variables
Who know how could i change it when get_train() doesn’t work
I will appreciate your help if someone helps me on this


title: Why use relu before maxpooling?

body:
In this pytorch neural network tutorial tutorial link

I'm confuse why we need to use relu before max pooling.
Isn't the pixel values in the image are already positive?
I don't know why relu max(0, x) is needed.
Can anybody give me some advise on this issue?

 class Net(nn.Module):
  ...(init function)

   def forward(self, x):
     x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2)) # Max pooling over a (2, 2) window


title: How to perform Multi output regression using RoBERTa?

body:
I have a problem statement where I want to predict multiple continuous outputs using a text input. I tried using 'robertaforsequenceclassification' from HuggingFace library. But the documentation states that when the number of outputs in the final layer is more than 1, a cross entropy loss is used automatically as mentioned here: https://huggingface.co/transformers/v2.2.0/model_doc/bert.html#transformers.BertForSequenceClassification.
But I want to use an RMSE loss in a regression setting with two classes in the final layer. How would one go about modifying it?


title: Any Tensorflow equivalent of Pytorch's backward()? Trying to send gradients back to TF model to backprop

body:
I'm trying to implement a split learning model, where my TF model on a client takes in the data and produces an intermediate output. This intermediate output will be sent to a server running the Pytorch model that will take it in as input and minimize the loss. Then, my server will send back the client gradients to the TF model for the TF model to update its weights.
How do I get my TF model to update its weights with the gradients sent back from the server?
# pytorch client
client_output.backward(client_grad)
optimizer.step()

With PyTorch, I can just do a client_pred.backward(client_grad) and client_optimizer.step().
How do I achieve the same with a Tensorflow client? I've tried GradientTape with tape.gradient(client_grad, model.trainable_weights) but it just gives me None. I think it's because there's no computation in the tape context and client_grad is just a Tensor holding the gradients and is not connected to the model's layers?
Is there some way I can do this with tf's apply_gradients() or compute_gradients()?
I only have the gradients for the client's last layer (sent by server). I'm trying to compute all the gradients for the client and update the weights.
Thank you.

class TensorflowModel(tf.keras.Model):
        def __init__(self, D_in, H, D_out):
            super(TensorflowModel, self).__init__()
            self.d1 = Dense(H, activation='relu', input_shape=(D_in,))
            self.d2 = Dense(D_out)

        def call(self, x):
            x = self.d1(x)
            return self.d2(x)

tensorflowModel = TensorflowModel(D_in, H, D_out)
tensorflowOptimizer = tf.optimizers.Adam(lr=1e-4)

serverModel = torch.nn.Sequential(
        torch.nn.Linear(10, 50),
        torch.nn.ReLU(),
        torch.nn.Linear(50, 10)
    )
loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(serverModel.parameters(), lr=1e-4)

for t in range(N):
    // let x be minibatch
    // let y be labels of minibatch

    client_pred = tensorflowModel(x)

    client_output = torch.from_numpy(client_pred.numpy())
    client_output.requires_grad = True

    y_pred = serverModel(client_output)
    loss = loss_fn(y_pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step() // update server weights

    // now retrieve client grad for last layer
    client_grad = client_output.grad.detach().clone().numpy()
    client_grad = tf.convert_to_tensor(client_grad) // change to tf tensor

    // now compute all client gradients and update client weights
    // HOW DO I DO THIS?

How should I update the client weights? If the client was a pytorch model I could just do client_pred.backward(client_grad) and client_optimizer.step(). I'm not sure how to use the gradient tape to calculate gradients, since client_grad was computed on the server and was a pytorch tensor that's converted to a tf tensor.


title: output.grad None even after loss.backward()

body:
I'm confused...
My model output:
tensor([[0.0000,0.1537],...],grad_fn=<ReluBackward0>)
If I use print(output.grad) it gives me None but even after gradient computation with
loss.backward() I get the same result, which again is None...
Even with
with torch.set_grad_enabled(True):
added, still the same.
I've tried now with multiple model variants, always the same.
I was achieving good result with my model there seemed no problem, now I see this and I'm not sure any more if there not might be a major flaw I didn't recognized so far.
But my model is learning, it improves so I guess it has to?
Why do I get None instead of an actual value?


title: Pytorch: Accessing batches of images from different classes from the generic dataloader

body:
I'm trying to train a GAN using tensorflow and pytorch. I'm using the torchvision.datasets.ImageFolder(root, transform=None, target_transform=None, loader=) class to load the images. I have two classes: Negatives and Positives. How can I access a batch of negative and positive images from the dataloader when I'm iterating through it. Look at the dataloader iteration in the training loop. Here is the code:

DATA_FOLDER = './tf_data/plasmodium_photos/'
IMAGE_PIXELS = 28*28*3
NOISE_SIZE = 10
BATCH_SIZE = 10

def noise(n_rows, n_cols):
    return np.random.normal(size=(n_rows, n_cols))

def xavier_init(size):
    in_dim = size[0] if len(size) == 1 else size[1]
    stddev = 1. / np.sqrt(float(in_dim))
    return tf.random_uniform(shape=size, minval=-stddev, maxval=stddev)

def images_to_vectors(images):
    return images.reshape(images.shape[0], 2352)

def vectors_to_images(vectors):
    return vectors.reshape(vectors.shape[0], 28, 28, 3)

def plasmodium_data():
    compose = transforms.Compose(
        [transforms.ToTensor(),
         transforms.Normalize((.5, .5, .5), (.5, .5, .5))
        ])
    return datasets.ImageFolder(DATA_FOLDER, transform=compose)

# Load data
data = plasmodium_data()

# Create loader with data, so that we can iterate over it
data_loader = DataLoader(data, batch_size=BATCH_SIZE, shuffle=True)

# Num batches
num_batches = len(data_loader)

## Discriminator

# Input
X = tf.placeholder(tf.float32, shape=(None, IMAGE_PIXELS))

# Layer 1 Variables
D_W1 = tf.Variable(xavier_init([2352, 1024]))
D_B1 = tf.Variable(xavier_init([1024]))

# Layer 2 Variables
D_W2 = tf.Variable(xavier_init([1024, 512]))
D_B2 = tf.Variable(xavier_init([512]))

# Layer 3 Variables
D_W3 = tf.Variable(xavier_init([512, 256]))
D_B3 = tf.Variable(xavier_init([256]))

# Out Layer Variables
D_W4 = tf.Variable(xavier_init([256, 1]))
D_B4 = tf.Variable(xavier_init([1]))

# Store Variables in list
D_var_list = [D_W1, D_B1, D_W2, D_B2, D_W3, D_B3, D_W4, D_B4]

## setting up a Generator network

# Input
Z = tf.placeholder(tf.float32, shape=(None, NOISE_SIZE))

# Layer 1 Variables
G_W1 = tf.Variable(xavier_init([10, 256]))
G_B1 = tf.Variable(xavier_init([256]))

# Layer 2 Variables
G_W2 = tf.Variable(xavier_init([256, 512]))
G_B2 = tf.Variable(xavier_init([512]))

# Layer 3 Variables
G_W3 = tf.Variable(xavier_init([512, 1024]))
G_B3 = tf.Variable(xavier_init([1024]))

# Out Layer Variables
G_W4 = tf.Variable(xavier_init([1024, 2352]))
G_B4 = tf.Variable(xavier_init([2352]))

# Store Variables in list
G_var_list = [G_W1, G_B1, G_W2, G_B2, G_W3, G_B3, G_W4, G_B4]

def discriminator(x):
    l1 = tf.nn.dropout(tf.nn.leaky_relu(tf.matmul(x,   D_W1) + D_B1, .2), .3)
    l2 = tf.nn.dropout(tf.nn.leaky_relu(tf.matmul(l1,  D_W2) + D_B2, .2), .3)
    l3 = tf.nn.dropout(tf.nn.leaky_relu(tf.matmul(l2,  D_W3) + D_B3, .2), .3)
    out = tf.matmul(l3, D_W4) + D_B4
    return out

def generator(z):
    l1 = tf.nn.leaky_relu(tf.matmul(z,  G_W1) + G_B1, .2)
    l2 = tf.nn.leaky_relu(tf.matmul(l1, G_W2) + G_B2, .2)
    l3 = tf.nn.leaky_relu(tf.matmul(l2, G_W3) + G_B3, .2)
    out = tf.nn.tanh(tf.matmul(l3, G_W4) + G_B4)
    return out

G_sample = generator(Z)
D_real = discriminator(X)
D_fake = discriminator(G_sample)

# Losses
D_loss_real = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=D_real, labels=tf.ones_like(D_real)))
D_loss_fake = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=D_fake, labels=tf.zeros_like(D_fake)))
D_loss = D_loss_real + D_loss_fake
G_loss = tf.reduce_mean(tf.nn.sigmoid_cross_entropy_with_logits(logits=D_fake, labels=tf.ones_like(D_fake)))

# Optimizers
D_opt = tf.train.AdamOptimizer(2e-4).minimize(D_loss, var_list=D_var_list)
G_opt = tf.train.AdamOptimizer(2e-4).minimize(G_loss, var_list=G_var_list)

# Testing configuration
num_test_samples = 100
test_noise = noise(num_test_samples, NOISE_SIZE)

# Variable initialization
num_epochs = 200

# Start interactive session
session = tf.InteractiveSession()

# Init Variables
tf.global_variables_initializer().run()

# Init Logger
# Needed to make sure the logging output is visible.
  # See https://github.com/tensorflow/tensorflow/issues/3047
tf.logging.set_verbosity(tf.logging.INFO)

X_batch = None
Z_batch = None
#Training
# Iterate through epochs
for epoch in range(num_epochs):
    for n_batch, (batch, label) in enumerate(data_loader):

        #print(epoch)
        # 1. Train Discriminator
        print(label)
        #print(type(batch))
        #print(batch.permute(0, 2, 3, 1).shape)
        X_batch = images_to_vectors(batch.permute(0, 2, 3, 1).numpy())
        feed_dict = {X: X_batch, Z: noise(BATCH_SIZE, NOISE_SIZE)}
        _, d_error, d_pred_real, d_pred_fake = session.run(
            [D_opt, D_loss, D_real, D_fake], feed_dict=feed_dict
        )

        # 2. Train Generator
        feed_dict = {Z: noise(BATCH_SIZE, NOISE_SIZE)}
        _, g_error = session.run(
            [G_opt, G_loss], feed_dict=feed_dict
        )

        if n_batch % 5 == 0:
            display.clear_output(True)
            # Generate images from test noise
            test_images = session.run(
                G_sample, feed_dict={Z: test_noise}
            )
            test_images = vectors_to_images(test_images)
            # Log Images
            for i in range(len(test_images)):
                imgplot = plt.imshow(test_images[i])

            # Log Status
            #tf.logging.info(
            #    epoch, num_epochs, n_batch, num_batches,
            #    d_error, g_error, d_pred_real, d_pred_fake
            #)


title: Elegant way to generate indexable sliding window timeseries

body:
To implement pytorch's DataSet class __get_item__() method, it requires to support the indexing such that dataset[i] can be used to get ith sample.

Say I have  a time-series ser:

2017-12-29 14:44:00  69.90
2017-12-29 14:45:00  69.91
2017-12-29 14:46:00  69.87
2017-12-29 14:47:00  69.85
2017-12-29 14:48:00  69.86
2017-12-29 14:49:00  69.92
2017-12-29 14:50:00  69.90
2017-12-29 14:51:00  70.00
2017-12-29 14:52:00  69.97
2017-12-29 14:53:00  69.99
2017-12-29 14:54:00  69.99
2017-12-29 14:55:00  69.85

Since I need to index into the rolling window. I  generate a window length 3 time-series by using:

l3_list = list()
def t(x):
  l3_list.append(x.copy())
ser.rolling(3).apply(t)

l3_list becomes:

[array([69.9 , 69.91, 69.87]),
 array([69.91, 69.87, 69.85]),
 array([69.87, 69.85, 69.86]),
 array([69.85, 69.86, 69.92]),
 array([69.86, 69.92, 69.9 ]),
 array([69.92, 69.9 , 70.  ]),
 array([69.9 , 70.  , 69.97]),
 array([70.  , 69.97, 69.99]),
 array([69.97, 69.99, 69.99]),
 array([69.99, 69.99, 69.85])]

So that I can index in l3_list. Namely l3_list[i] is the ith sliding window. Is there a more memory efficient way to do this?


title: How to find the address and port of the rank0 machine in GCP?

body:
I am trying to do distributed training with single node and multi-gpu config on GCP. I have set the IP address of my instance where I have my code and data. I am getting this error. Am I missing something here? Any solutions? More here https://discuss.pytorch.org/t/runtimeerror-with-distributed-package-using-nccl-or-gloo-backend/30992/3. I would like to find the ip address of the rank0 machine.

    File "/opt/conda/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 400, in init_process_group
        store, rank, world_size = next(rendezvous(url))
      File "/opt/conda/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 143, in _env_rendezvous_handler
        store = TCPStore(master_addr, master_port, world_size, start_daemon)
    RuntimeError: Connection timed out


title: Create a new model in pytorch with custom initial value for the weights

body:
I'm new to pytorch and I want to understand how can I set the initial weight for the first hidden layer of my network. I explain a little better: my network is a very simple one layer MLP with 784 inputs values and 10 output values

 class Classifier(nn.Module):
        def __init__(self):
          super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 10)
        # Dropout module with 0.2 drop probability
        self.dropout = nn.Dropout(p=0.2)

    def forward(self, x):
        # make sure input tensor is flattened
        # x = x.view(x.shape[0], -1)

        # Now with dropout
        x = self.dropout(F.relu(self.fc1(x)))

        # output so no dropout here
        x = F.log_softmax(self.fc2(x), dim=1)

        return x

and I have, for now, a numpy matrix of shape(128, 784) witch contain the values that I want for the weight in fc1. How can I initialise the weights of the first layer with the value contained in the matrix?

Searching online in other answers I find out that I have to define the init function for the weight, e.g.

def weights_init(m):
    classname = m.__class__.__name__

    if classname.find('Conv2d') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

but i can not understand the code


title: Facing this error while classifying Images, containing 10 classes in pytorch, in ResNet50. My code is:

body:
This is the code I am implementing: I am using a subset of the CalTech256 dataset to classify images of 10 different kinds of animals. We will go over the dataset preparation, data augmentation and then steps to build the classifier.

def train_and_validate(model, loss_criterion, optimizer, epochs=25):
    '''
    Function to train and validate
    Parameters
        :param model: Model to train and validate
        :param loss_criterion: Loss Criterion to minimize
        :param optimizer: Optimizer for computing gradients
        :param epochs: Number of epochs (default=25)

    Returns
        model: Trained Model with best validation accuracy
        history: (dict object): Having training loss, accuracy and validation loss, accuracy
    '''

    start = time.time()
    history = []
    best_acc = 0.0

    for epoch in range(epochs):
        epoch_start = time.time()
        print("Epoch: {}/{}".format(epoch+1, epochs))

        # Set to training mode
        model.train()

        # Loss and Accuracy within the epoch
        train_loss = 0.0
        train_acc = 0.0

        valid_loss = 0.0
        valid_acc = 0.0

        for i, (inputs, labels) in enumerate(train_data_loader):

            inputs = inputs.to(device)
            labels = labels.to(device)

            # Clean existing gradients
            optimizer.zero_grad()

            # Forward pass - compute outputs on input data using the model
            outputs = model(inputs)

            # Compute loss
            loss = loss_criterion(outputs, labels)

            # Backpropagate the gradients
            loss.backward()

            # Update the parameters
            optimizer.step()

            # Compute the total loss for the batch and add it to train_loss
            train_loss += loss.item() * inputs.size(0)

            # Compute the accuracy
            ret, predictions = torch.max(outputs.data, 1)
            correct_counts = predictions.eq(labels.data.view_as(predictions))

            # Convert correct_counts to float and then compute the mean
            acc = torch.mean(correct_counts.type(torch.FloatTensor))

            # Compute total accuracy in the whole batch and add to train_acc
            train_acc += acc.item() * inputs.size(0)

            #print("Batch number: {:03d}, Training: Loss: {:.4f}, Accuracy: {:.4f}".format(i, loss.item(), acc.item()))

        # Validation - No gradient tracking needed
        with torch.no_grad():

            # Set to evaluation mode
            model.eval()

            # Validation loop
            for j, (inputs, labels) in enumerate(valid_data_loader):
                inputs = inputs.to(device)
                labels = labels.to(device)

                # Forward pass - compute outputs on input data using the model
                outputs = model(inputs)

                # Compute loss
                loss = loss_criterion(outputs, labels)

                # Compute the total loss for the batch and add it to valid_loss
                valid_loss += loss.item() * inputs.size(0)

                # Calculate validation accuracy
                ret, predictions = torch.max(outputs.data, 1)
                correct_counts = predictions.eq(labels.data.view_as(predictions))

                # Convert correct_counts to float and then compute the mean
                acc = torch.mean(correct_counts.type(torch.FloatTensor))

                # Compute total accuracy in the whole batch and add to valid_acc
                valid_acc += acc.item() * inputs.size(0)

                #print("Validation Batch number: {:03d}, Validation: Loss: {:.4f}, Accuracy: {:.4f}".format(j, loss.item(), acc.item()))

        # Find average training loss and training accuracy
        avg_train_loss = train_loss/train_data_size
        avg_train_acc = train_acc/train_data_size

        # Find average training loss and training accuracy
        avg_valid_loss = valid_loss/valid_data_size
        avg_valid_acc = valid_acc/valid_data_size

        history.append([avg_train_loss, avg_valid_loss, avg_train_acc, avg_valid_acc])

        epoch_end = time.time()

        print("Epoch : {:03d}, Training: Loss: {:.4f}, Accuracy: {:.4f}%, \n\t\tValidation : Loss : {:.4f}, Accuracy: {:.4f}%, Time: {:.4f}s".format(epoch, avg_train_loss, avg_train_acc*100, avg_valid_loss, avg_valid_acc*100, epoch_end-epoch_start))

        # Save if the model has best accuracy till now
        torch.save(model, dataset+'_model_'+str(epoch)+'.pt')

    return model, history

# Load pretrained ResNet50 Model
resnet50 = models.resnet50(pretrained=True)
#resnet50 = resnet50.to('cuda:0')

# Freeze model parameters
for param in resnet50.parameters():
    param.requires_grad = False
# Change the final layer of ResNet50 Model for Transfer Learning
fc_inputs = resnet50.fc.in_features

resnet50.fc = nn.Sequential(
    nn.Linear(fc_inputs, 256),
    nn.ReLU(),
    nn.Dropout(0.4),
    nn.Linear(256, num_classes), # Since 10 possible outputs
    nn.LogSoftmax(dim=1) # For using NLLLoss()
)

# Convert model to be used on GPU
# resnet50 = resnet50.to('cuda:0')

# Change the final layer of ResNet50 Model for Transfer Learning
fc_inputs = resnet50.fc.in_features

resnet50.fc = nn.Sequential(
    nn.Linear(fc_inputs, 256),
    nn.ReLU(),
    nn.Dropout(0.4),
    nn.Linear(256, num_classes), # Since 10 possible outputs
    nn.LogSoftmax(dienter code herem=1) # For using NLLLoss()
)

# Convert model to be used on GPU
# resnet50 = resnet50.to('cuda:0')`enter code here`

Error is this:

  RuntimeError                              Traceback (most recent call
  last)  in ()
        6 # Train the model for 25 epochs
        7 num_epochs = 30
  ----> 8 trained_model, history = train_and_validate(resnet50, loss_func, optimizer, num_epochs)
        9
       10 torch.save(history, dataset+'_history.pt')

   in train_and_validate(model,
  loss_criterion, optimizer, epochs)
       43
       44             # Compute loss
  ---> 45             loss = loss_criterion(outputs, labels)
       46
       47             # Backpropagate the gradients

  ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in
  call(self, *input, **kwargs)
      539             result = self._slow_forward(*input, **kwargs)
      540         else:
  --> 541             result = self.forward(*input, **kwargs)
      542         for hook in self._forward_hooks.values():
      543             hook_result = hook(self, input, result)

  ~\Anaconda3\lib\site-packages\torch\nn\modules\loss.py in
  forward(self, input, target)
      202
      203     def forward(self, input, target):
  --> 204         return F.nll_loss(input, target, weight=self.weight, ignore_index=self.ignore_index, reduction=self.reduction)
      205
      206

  ~\Anaconda3\lib\site-packages\torch\nn\functional.py in
  nll_loss(input, target, weight, size_average, ignore_index, reduce,
  reduction)    1836                          .format(input.size(0),
  target.size(0)))    1837     if dim == 2:
  -> 1838         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)    1839     elif dim == 4:    1840         ret = torch._C._nn.nll_loss2d(input, target,
  weight, _Reduction.get_enum(reduction), ignore_index)

  RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes'
  failed.  at
  C:\Users\builder\AppData\Local\Temp\pip-req-build-0i480kur\aten\src\THNN/generic/ClassNLLCriterion.c:97


title: How to solve size mismatch problem in loading a pretrained model in pytorch?

body:
I am trying to load a pretrained model in which the number of filters (channels) of conv layers is 64, into a model with 128 filters in conv layers. I use new_network.load_state_dict(pretrained_model, strict=False) but I face size mismatch error for all of that layers.
How can I initialize some of the filters in each conv layer with available pretrained model weights?


title: Multiple answer spans in context, BERT question answering

body:
I am writing a Question Answering system using pre-trained BERT with a linear layer and a softmax layer on top. When following the templates available on the net the labels of one example usually only consists of one answer_start_index and one answer_end_index. For example, from Huggingface when instantiating a SQUADFeatures object:

```
self.input_ids = input_ids
self.attention_mask = attention_mask
self.token_type_ids = token_type_ids
self.cls_index = cls_index
self.p_mask = p_mask

self.example_index = example_index
self.unique_id = unique_id
self.paragraph_len = paragraph_len
self.token_is_max_context = token_is_max_context
self.tokens = tokens
self.token_to_orig_map = token_to_orig_map

self.start_position = start_position
self.end_position = end_position
self.is_impossible = is_impossible
self.qas_id = qas_id
```

However, in my own dataset I have examples where the answer word is found at several locations in the context, i.e. there may be several correct spans constituting the answer.

My problem is I don't know how to manage such examples? In the templates available on the net labels are usually in a list, say:

[start_example1, start_example2, start_example3]
[end_example1, end_example2, end_example3]

In my case this may look like:

[start_example1, [start_example2_1, start_example2_2], start_example3]
and same for ends of course

In other words, I do not have a list containing one label per example, but a list containing either single-labels or a list of "labels" for an example, i.e. a list consisting of lists.

When following other templates the next step in the process is:

```
input_ids = torch.cat(input_ids, dim=0)
attention_masks = torch.cat(attention_masks, dim=0)
token_type_ids = torch.cat(token_type_ids, dim=0)
span_starts = torch(span_starts) #Something like this
span_ends = torch(span_ends) #Something like this
```

However this of course (?) raises an error as my span_start lists and span_end lists does not contain only single-items but sometimes a list within the list.

Anyone have an idea on how I can tackle this problem? Should I only use examples where there's only one span constituting the answer present in the context?

If I work around the torch-error, will the backpropagation / evaluation/ computation of loss still work?

Thank You! /B


title: PyTorch Average Accuracy after each epoch

body:
I am trying to calculate the accuracy of the model after the end of each epoch. After each epoch I would like to calculate the accuracy over the previous epoch. The model only seems to print the  same value as the mean test error.
model.eval()
for images, paths in tqdm(loader_test):
    images = images.to(device)
    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float().to(device)

    # forward pass:
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

    # logging:
    loss = torch.mean((preds - targets)**2)
    count_error = torch.abs(preds - targets).mean()
    mean_test_error += count_error
    writer.add_scalar('test_loss', loss.item(), global_step=global_step)
    writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)

    global_step += 1

average_accuracy = 0
mean_test_error = mean_test_error / len(loader_test)
writer.add_scalar('mean_test_error', mean_test_error.item(), global_step=global_step)
average_accuracy += mean_test_error
average_accuracy = average_accuracy /len(loader_test)
print("Average accuracy: %f" % average_accuracy)
print("Test count error: %f" % mean_test_error)
if mean_test_error < best_test_error:
    best_test_error = mean_test_error
    torch.save({'state_dict':model.state_dict(),
                'optimizer_state_dict':optimizer.state_dict(),
                'globalStep':global_step,
                'train_paths':dataset_train.files,
                'test_paths':dataset_test.files},checkpoint_path)


title: Print random sample from dataloader in PyTorch

body: I have a certain dataset loaded into a dataloader. For example, if I wanted to save 100 images from this dataloader, how should I iterate over the dataloader to save them?


I am trying to run this code https://github.com/AshwinRJ/Federated-Learning-PyTorch
and i got error RuntimeError: Expected one of cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu device type at start of device string: 0. how to resolve this. any on ecan guide how to resolve thos error .. please help
import os
import copy
import time
import pickle
import numpy as np
from tqdm import tqdm
import torch
from tensorboardX import SummaryWriter

from options import args_parser
from update import LocalUpdate, test_inference
from models import MLP, CNNMnist, CNNFashion_Mnist, CNNCifar
from utils import get_dataset, average_weights, exp_details

if __name__ == '__main__':
    start_time = time.time()

    # define paths
    path_project = os.path.abspath('..')
    logger = SummaryWriter('../logs')

    args = args_parser()
    exp_details(args)

    if args.gpu:
        torch.cuda.set_device(args.gpu)
    device = 'cuda' if args.gpu else 'cpu'

    # load dataset and user groups
    train_dataset, test_dataset, user_groups = get_dataset(args)

    # BUILD MODEL
    if args.model == 'cnn':
        # Convolutional neural netork
        if args.dataset == 'mnist':
            global_model = CNNMnist(args=args)
        elif args.dataset == 'fmnist':
            global_model = CNNFashion_Mnist(args=args)
        elif args.dataset == 'cifar':
            global_model = CNNCifar(args=args)

    elif args.model == 'mlp':
        # Multi-layer preceptron
        img_size = train_dataset[0][0].shape
        len_in = 1
        for x in img_size:
            len_in *= x
            global_model = MLP(dim_in=len_in, dim_hidden=64,
                               dim_out=args.num_classes)
    else:
        exit('Error: unrecognized model')

    # Set the model to train and send it to device.
    global_model.to(device)
    global_model.train()
    print(global_model)

    # copy weights
    global_weights = global_model.state_dict()

    # Training
    train_loss, train_accuracy = [], []
    val_acc_list, net_list = [], []
    cv_loss, cv_acc = [], []
    print_every = 2
    val_loss_pre, counter = 0, 0

    for epoch in tqdm(range(args.epochs)):
        local_weights, local_losses = [], []
        print(f'\n | Global Training Round : {epoch+1} |\n')

        global_model.train()
        m = max(int(args.frac * args.num_users), 1)
        idxs_users = np.random.choice(range(args.num_users), m, replace=False)

        for idx in idxs_users:
            local_model = LocalUpdate(args=args, dataset=train_dataset,
                                      idxs=user_groups[idx], logger=logger)
            w, loss = local_model.update_weights(
                model=copy.deepcopy(global_model), global_round=epoch)
            local_weights.append(copy.deepcopy(w))
            local_losses.append(copy.deepcopy(loss))

        # update global weights
        global_weights = average_weights(local_weights)

        # update global weights
        global_model.load_state_dict(global_weights)

        loss_avg = sum(local_losses) / len(local_losses)
        train_loss.append(loss_avg)

        # Calculate avg training accuracy over all users at every epoch
        list_acc, list_loss = [], []
        global_model.eval()
        for c in range(args.num_users):
            local_model = LocalUpdate(args=args, dataset=train_dataset,
                                      idxs=user_groups[idx], logger=logger)
            acc, loss = local_model.inference(model=global_model)
            list_acc.append(acc)
            list_loss.append(loss)
        train_accuracy.append(sum(list_acc)/len(list_acc))

        # print global training loss after every 'i' rounds
        if (epoch+1) % print_every == 0:
            print(f' \nAvg Training Stats after {epoch+1} global rounds:')
            print(f'Training Loss : {np.mean(np.array(train_loss))}')
            print('Train Accuracy: {:.2f}% \n'.format(100*train_accuracy[-1]))

    # Test inference after completion of training
    test_acc, test_loss = test_inference(args, global_model, test_dataset)

    print(f' \n Results after {args.epochs} global rounds of training:')
    print("|---- Avg Train Accuracy: {:.2f}%".format(100*train_accuracy[-1]))
    print("|---- Test Accuracy: {:.2f}%".format(100*test_acc))

    # Saving the objects train_loss and train_accuracy:
    file_name = '../save/objects/{}_{}_{}_C[{}]_iid[{}]_E[{}]_B[{}].pkl'.\
        format(args.dataset, args.model, args.epochs, args.frac, args.iid,
               args.local_ep, args.local_bs)

    with open(file_name, 'wb') as f:
        pickle.dump([train_loss, train_accuracy], f)

    print('\n Total Run Time: {0:0.4f}'.format(time.time()-start_time))

    # PLOTTING (optional)
    # import matplotlib
    # import matplotlib.pyplot as plt
    # matplotlib.use('Agg')

    # Plot Loss curve
    # plt.figure()
    # plt.title('Training Loss vs Communication rounds')
    # plt.plot(range(len(train_loss)), train_loss, color='r')
    # plt.ylabel('Training loss')
    # plt.xlabel('Communication Rounds')
    # plt.savefig('../save/fed_{}_{}_{}_C[{}]_iid[{}]_E[{}]_B[{}]_loss.png'.
    #             format(args.dataset, args.model, args.epochs, args.frac,
    #                    args.iid, args.local_ep, args.local_bs))
    #
    # # Plot Average Accuracy vs Communication rounds
    # plt.figure()
    # plt.title('Average Accuracy vs Communication rounds')
    # plt.plot(range(len(train_accuracy)), train_accuracy, color='k')
    # plt.ylabel('Average Accuracy')
    # plt.xlabel('Communication Rounds')
    # plt.savefig('../save/fed_{}_{}_{}_C[{}]_iid[{}]_E[{}]_B[{}]_acc.png'.
    #             format(args.dataset, args.model, args.epochs, args.frac,
    #                    args.iid, args.local_ep, args.local_bs))

OUtput log
Experimental details:
    Model     : cnn
    Optimizer : sgd
    Learning  : 0.01
    Global Rounds   : 10

    Federated parameters:
    IID
    Fraction of users  : 0.1
    Local Batch size   : 10
    Local Epochs       : 10

Traceback (most recent call last):
  File "src/federated_main.py", line 32, in <module>
    torch.cuda.set_device(args.gpu)
  File "C:\Users\S\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda\__init__.py", line 298, in set_device
    device = _get_device_index(device)
  File "C:\Users\S\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda\_utils.py", line 20, in _get_device_index
    device = torch.device(device)
RuntimeError: Expected one of cpu, cuda, mkldnn, opengl, opencl, ideep, hip, msnpu device type at start of device string: 0
PS C:\Users\S\Downloads\Federated-Learning-PyTorch-master\Federated-Learning-PyTorch-master>


How can I write the below equivalent code of Keras Neural Net in Pytorch?

actor = Sequential()
        actor.add(Dense(20, input_dim=9, activation='relu', kernel_initializer='he_uniform'))
        actor.add(Dense(20, activation='relu'))
        actor.add(Dense(27, activation='softmax', kernel_initializer='he_uniform'))
        actor.summary()
        # See note regarding crossentropy in cartpole_reinforce.py
        actor.compile(loss='categorical_crossentropy',
                      optimizer=Adam(lr=self.actor_lr))[Please find the image eq here.][1]

  [1]: https://i.stack.imgur.com/gJviP.png


title: PyTorch VAE fails conversion to onnx

body:
I'm trying to convert a PyTorch VAE to onnx, but I'm getting: torch.onnx.symbolic.normal does not exist

The problem appears to originate from a reparametrize() function:

    def reparametrize(self, mu, logvar):
        std = logvar.mul(0.5).exp_()
        if self.have_cuda:
             eps = torch.normal(torch.zeros(std.size()),torch.ones(std.size())).cuda()
        else:
           eps = torch.normal(torch.zeros(std.size()),torch.ones(std.size()))
        return eps.mul(std).add_(mu)

I also tried:

eps = torch.cuda.FloatTensor(std.size()).normal_()

which produced the error:

    Schema not found for node. File a bug report.
    Node: %173 : Float(1, 20) = aten::normal(%169, %170, %171, %172), scope: VAE
    Input types:Float(1, 20), float, float, Generator

and

eps = torch.randn(std.size()).cuda()

which produced the error:

    builtins.TypeError: i_(): incompatible function arguments. The following argument types are supported:
    1. (self: torch._C.Node, arg0: str, arg1: int) -> torch._C.Node
    Invoked with: %137 : Tensor = onnx::RandomNormal(), scope: VAE, 'shape', 133 defined in (%133 : int[] = prim::ListConstruct(%128, %132), scope: VAE) (occurred when translating randn)

I am using cuda.

Any thoughts appreciated. Perhaps I need to approach the z/latent differently for onnx?

NOTE: Stepping through, I can see that it's finding RandomNormal() for torch.randn(), which should be correct. But I don't really have access to the arguments at that point, so how can I fix it?


title: Why Does PyTorch Run Slower Than Keras For Me, and How To Make The Code Concise?

body:
I read PyTorch is better and faster, so I'm trying to see if I want to transition to PyTorch from Keras.

I made the simple testing code below, but Keras consistently runs faster for me, and usually (not always) gets better score than PyTorch.

Also is there a better way to write code for PyTorch more concise like Keras?

#output:
Keras:
Total runtime =  18.451340198516846
LRL: 0.145 LRAP: 0.493
PyTorch:
Total runtime =  19.641956329345703
LRL: 0.092 LRAP: 0.491

def score(true, pred):
    lrl = label_ranking_loss(true, pred)
    lrap = label_ranking_average_precision_score(true, pred)
    print('LRL:', round(lrl), 'LRAP:', round(lrap))

def main():
    x,y = load()
    x_train, x_test, y_train, y_test = train_test_split(x, y)
    scaler = StandardScaler()
    x_train= scaler.fit_transform(x_train)
    x_test= scaler.transform(x_test)
    epochs = 100
    batch_size = 32

    print("Keras:")
    t_start = time.time()
    model= Sequential()
    model.add(Dense(60, activation="relu", input_shape=(120,)))
    model.add(Dense(30, activation="relu"))
    model.add(Dense(10, activation="sigmoid"))
    model.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])
    model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs)
    pred = model.predict(x_test)
    t_finish = time.time()
    total_time = t_finish-t_start
    print('Total runtime = ', total_time)
    score(y_test, pred)

    print("PyTorch:")
    t_start = time.time()
    model = torch.nn.Sequential(
        torch.nn.Linear(120, 60),
        torch.nn.ReLU(),
        torch.nn.Linear(60, 30),
        torch.nn.ReLU(),
        torch.nn.Linear(30, 10),
        torch.nn. Sigmoid())
    loss_fn = torch.nn. BCELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
    n_batch = int(x_train.shape[0]/batch_size)
    for epoch in range(epochs):
        avg_cost = 0
        for i in range(n_batch):
            x_batch = x_train[i*batch_size:(i+1)*batch_size]
            y_batch = y_train[i*batch_size:(i+1)*batch_size]
            x, y = Variable(torch.from_numpy(x_batch).float()), Variable(torch.from_numpy(y_batch).float(), requires_grad=False)
            pred = model(x)
            loss = loss_fn(pred, y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            avg_cost += loss.item()/n_batch
        print(epoch, avg_cost)
        x, y = Variable(torch.from_numpy(x_test).float()), Variable(torch.from_numpy(y_test).float(), requires_grad=False)
    pred = model(x).data.numpy()
    t_finish = time.time()
    total_time = t_finish-t_start
    print('Total runtime = ', total_time)
    score(y_test, pred)

if __name__ == '__main__':
    main()


title: PyTorch - Cuda error: no kernel image is available for execution on the device

body:
I have a problem about CUDA and PyTorch. I am trying to run a simple code from SinGAN implementation but I got a Runtime Cuda error. Here is information about Cuda and PyTorch.

PyTorch: 1.3.1
Cuda: 10.0
Torchvision: 0.4.2

I installed all of them with conda. Here is the simple code that I tried:
import torch
import torch.nn as nn
from skimage import io as img
import numpy as np

x = img.imread('...')
x = x[:,:,:,None]
x = x.transpose((3,2,0,1))/255
x = torch.from_numpy(x)

print(torch.cuda.is_available())

if torch.cuda.is_available():
    x = x.cuda()
    #x = x.to(torch.device('cuda'))

    x = x.type(torch.cuda.FloatTensor)

Output:

True
RuntimeError: CUDA error: no kernel image is available for execution on the device

As a result, although cuda.is_available() returns True, there is a CUDA RuntimeError. And I did not handle this problem.


title: Getting the output's grad with respect to the input

body:
I'm currently trying to implement an ODE Solver with Pytorch, my solution requires computing the gradient of each output wtr to its input.
y = model(x)

for i in range(len(y)): #compute output grad wrt input
       y[i].backward(retain_graph=True)

ydx=x.grad

I was wondering if there is a more elegant way to compute the gradients for each output in the batch, since the code gets messy for higher order ODEs and PDEs.
I tried using:
torch.autograd.backward(x,y,retain_graph=True)

without much success.


title: Validation set augmentations PyTorch example

body:
In this PyTorch vision example for transfer learning, they are performing validation set augmentations, and I can't figure out why.

# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

As far as I know, data augmentations are to be made solely on the training set (and sometimes on the test set, in what is called Test Time Augmentations).

Why is it done here as well?
Also, why not just resizing straight to 224?


title: How to use the past with HuggingFace Transformers GPT-2?

body:
I have:
        context = torch.tensor(context, dtype=torch.long, device=self.device)
        context = context.unsqueeze(0)
        generated = context
        with torch.no_grad():
            past_outputs = None
            for i in trange(num_words):
                print(i, num_words)
                inputs = {"input_ids": generated}

                outputs, past_outputs = self.model(
                    **inputs,
                    past=past_outputs
                )
                next_token_logits = outputs[
                    0, -1, :] / (temperature if temperature > 0 else 1.0)

                # reptition penalty from CTRL
                # (https://arxiv.org/abs/1909.05858)
                for _ in set(generated.view(-1).tolist()):
                    next_token_logits[_] /= repetition_penalty

                filtered_logits = top_k_top_p_filtering(
                    next_token_logits, top_k=top_k, top_p=top_p)
                if temperature == 0:  # greedy sampling:
                    next_token = torch.argmax(filtered_logits).unsqueeze(0)
                else:
                    next_token = torch.multinomial(
                        F.softmax(filtered_logits, dim=-1), num_samples=1)

                generated = torch.cat(
                    (generated, next_token.unsqueeze(0)), dim=1)

This works for the first iteration, but then I get an error for the next iteration:
  File "/Users/shamoon/Sites/wordblot/packages/ml-server/generator.py", line 143, in sample_sequence
    past=past_outputs
  File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 601, in forward
    output_hidden_states=output_hidden_states,
  File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/transformers/modeling_gpt2.py", line 470, in forward
    position_embeds = self.wpe(position_ids)
  File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/modules/sparse.py", line 114, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/Users/shamoon/.local/share/virtualenvs/ml-server-EdimT5-E/lib/python3.7/site-packages/torch/nn/functional.py", line 1724, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

Is there something I'm doing wrong?


title: simple inception block in pytorch taking much much longer time to train on GPU?

body:
I am training very simple inception block followed by a maxpool and fully-connected layer on NVIDIA GeForce RTX 2070 GPU and its taking very long time for an iteration. Just finished 10 iterations in more than 24 hours.
Here is the code for inception model definition
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        input_channels = 3
        conv_block = BasicConv2d(input_channels, 64, kernel_size=1)
        self.branch3x3stack = nn.Sequential(
            BasicConv2d(input_channels, 64, kernel_size=1),
            BasicConv2d(64, 96, kernel_size=3, padding=1),
            BasicConv2d(96, 96, kernel_size=3, padding=1),
        )

        self.branch3x3 = nn.Sequential(
            BasicConv2d(input_channels, 64, kernel_size=1),
            BasicConv2d(64, 96, kernel_size=3, padding=1),
        )

        self.branch1x1 = BasicConv2d(input_channels, 96, kernel_size=1)

        self.branchpool = nn.Sequential(
            nn.AvgPool2d(kernel_size=3, stride=1, padding=1),
            BasicConv2d(input_channels, 96, kernel_size=1),
        )
        self.maxpool = nn.MaxPool2d(kernel_size=8, stride=8)
        self.fc_seqn = nn.Sequential(nn.Linear(301056, 1))

    def forward(self, x):
        x = [
            self.branch3x3stack(x),
            self.branch3x3(x),
            self.branch1x1(x),
            self.branchpool(x),
        ]
        x = torch.cat(x, 1)
        x = self.maxpool(x)
        x = x.view(x.size(0), -1)
        x = self.fc_seqn(x)
        return x  # torch.cat(x, 1)

and the training code I used
def training_code(self, model):
    model = copy.deepcopy(model)
    model = model.to(self.device)
    criterion = nn.MSELoss()
    optimizer = optim.Adam(model.parameters(), lr=self.learning_rate)
    for epoch in range(self.epochs):
        print("\n epoch :", epoch)
        running_loss = 0.0
        start_epoch = time.time()
        for i, (inputs, labels) in enumerate(self.train_data_loader):
            inputs = inputs.to(self.device)
            labels = labels.to(self.device)
            optimizer.zero_grad()
            outputs = (model(inputs)).squeeze()
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()
            if i % log_step == log_step - 1:
                avg_loss = running_loss / float(log_step)
                print("[epoch:%d, batch:%5d] loss: %.3f" % (epoch + 1, i + 1, avg_loss))
                running_loss = 0.0
        if epoch % save_model_step == 0:
            self.path_saved_model = path_saved_model + "_" + str(epoch)
            self.save_model(model)
    print("\nFinished Training\n")
    self.save_model(model)

Here is the summary of the convolution neural network. Can anyone please help me how to speed up the training?

Here is the summary of the convolution neural network after reducing the number of inputs to the fully-connected (FC) layer. But still the training time is very high similar as before. Can suggestion on how to speed up the training will be helpful.


title: Downloading pretrained TimeNet RNN?

body:
I've been looking for the pretrained net behind TimeNet - a resnet equivalent for time series analysis via RNNs.

There's some articles and several publications of the same paper, but the weights and architecture themselves are not easy to find.

Anyone knows how to find it? Is it available to the public?


title: Apply transformation for Tensor without including it in Backward

body:
Let's say I have n layered neural network. After running l layers, I want to apply some transformation to the l^th layer output, without including that transformation in backpropagation.

For e.g. :

output_layer_n = self.LinearLayer(output_layer_prev)
#apply some transformation to output_layer_n, but don't want to take autograd w.r.t. this transformation, basically this transformation function doesn't have any parameter
output_layer_n.data = TransformationFunction(output_layer_n.data)

So how should I go about implementing it? What I want is not to take gradient accounted for TransformationFunction() in my code.


title: nvcc and clang are not working well together when installing pytorch-gpu

body:
I am trying to install pytorch with gpu support on my MacBook Pro following official instructions.

Things go smoothly until an error occurred:

[ 70%] Building NVCC (Device) object caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/caffe2_gpu_generated_THCTensorMath.cu.o
nvcc fatal   : The version ('90000') of the host compiler ('Apple clang') is not supported
nvcc fatal   : The version ('90000') of the host compiler ('Apple clang') is not supported
CMake Error at caffe2_gpu_generated_THCBlas.cu.o.Release.cmake:219 (message):
  Error generating
  /Users/username/Dev/pytorch-gpu/pytorch/build/cCaMfafkee2 /ECrMraokre Faitle sc/acfaffef2e_2g_pgup_ug.ednierr/a_t_e/da_tTeHnC/Sslrece/pT.HcCu/..o/.cRaeflfeea2s_eg.pcum_agkeen:e2r1a9t e(dm_eTsHsCaBglea)s:.

It seems that CUDA and clang are not working well together.

I searched over internet and found these posts, but they did not solve my problem:

Revert Apple Clang Version For NVCC
https://github.com/pytorch/pytorch/issues/3047

Here's my environment:

macOS Sierra 10.12.6 (16G1618)
NVIDIA GeForce GT 750M
CUDA Driver Version: 387.178
GPU Driver Version: 378.05.05.25f11
Cuda compilation tools, release 8.0, V8.0.61
(Previous)Apple LLVM version 9.0.0 (clang-900.0.39.2)
(After downgrade)Apple LLVM version 8.1.0 (clang-802.0.42)
Xcode Version 9.2 (9C40b)


title: How can I solve the pytorch/pylint error: torch.tensor is not callable?

body:
when I make in python
import syft as sy
import torch

hook = sy.TorchHook(torch)
data = torch.tensor([[1.,1],[0,1],[1,0],[0,0]])

an error came up that says: 'torch.tensor is not callable'. And I do not know how to solve it.
I have pytorch version 1.4.0.


title: Scatter homogenous list of values to PyTorch tensor

body:
Consider the following list:

[[3], [1, 2], [4], [0], [2]]

And zeros tensor of size (5, 5)

I want to fill these indices according to their index in the list to the tensor with 1.

So, the expected output should be:

tensor([[0., 0., 0., 1., 0.],
        [0., 1., 1., 0., 0.],
        [0., 0., 0., 0., 1.],
        [1., 0., 0., 0., 0.],
        [0., 0., 1., 0., 0.]])

What happened above is this:

at the index [0, 3] put 1 (the case for the first element in my list).

A very similar case is achievable through using Tensor.scatter_. However, since it takes a tensor as the argument (index); you cannot create a tensor from a list if it contains a sub-list with a different size than the other elements, which is the case with [1, 2] in my list (this is actually the problem).

The scatter method could be used if the list is all of same size as the following:

tensor.scatter_(1, torch.tensor(index), 1)

Numpy solutions are acceptable


title: How to run OpenNMT-py model to translate a given string in jupyter notebook?

body:
The documentation only mentions using the command line to run the translate.py script already given in the OpenNMT-py repo, i.e. use

python translate.py -model model_name.pt -src data/src-test.txt -output pred.txt

and that's fine, but how do I use the onmt library to load the trained model in a jupyter notebook and make translations on the go? I want it to be able to do something like this in jupyter:

model = onmt.module.load(/path/to/model.pt)
text = "hola como estas"
model.translate(text)

Output:
>>>Hello how are you

I can't find any documentation on how to do this. Please help!


title: The train loss and test loss are the same high in CNN Pytorch(FashionMNIST)

body:
The problem is the Training loss and test loss are the same and the loss and accuracy weren't changing, what's wrong with my CNN structure and training process?
Training results:

Epoch: 1/30..  Training Loss: 2.306..  Test Loss: 2.306..  Test Accuracy: 0.100
Epoch: 2/30..  Training Loss: 2.306..  Test Loss: 2.306..  Test Accuracy: 0.100

Class code:
class Model(nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5)
        self.conv2 = nn.Conv2d(in_channels=6, out_channels=12, kernel_size=5)
        self.fc1 = nn.Linear(in_features=12 * 4 * 4, out_features=120)
        self.fc2 = nn.Linear(in_features=120, out_features=60)
        self.out = nn.Linear(in_features=60, out_features=10)
        #the output will be 0~9 (10)

Below is my CNN and training process:
def forward(self, t):
    # implement the forward pass
    # (1)input layer
    t = t
    # (2) hidden conv layer
    t = self.conv1(t)
    t = F.relu(t)
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    # (3) hidden conv layer
    t = self.conv2(t)
    t = F.relu(t)
    t = F.max_pool2d(t, kernel_size=2, stride=2)

    # (4) hidden linear layer
    t = t.reshape(-1, 12 * 4 * 4)
    t = self.fc1(t)
    t = F.relu(t)

    # (5) hidden linear layer
    t = self.fc2(t)
    t = F.relu(t)
    # (6) output layer
    t = self.out(t)
    #t = F.softmax(t, dim=1)
    return t

epoch = 30

train_losses, test_losses = [], []

for e in range(epoch):
    train_loss = 0
    test_loss = 0
    accuracy = 0

    for images, labels in train_loader:

        optimizer.zero_grad()
        op = model(images) #output
        loss = criterion(op, labels)
        train_loss += loss.item()
        loss.backward()
        optimizer.step()

    else:
        with torch.no_grad():
            model.eval()
            for images,labels in testloader:
                log_ps = model(images)
                prob = torch.exp(log_ps)
                top_probs, top_classes = prob.topk(1, dim=1)
                equals = labels == top_classes.view(labels.shape)
                accuracy += equals.type(torch.FloatTensor).mean()
                test_loss += criterion(log_ps, labels)
        model.train()
    print("Epoch: {}/{}.. ".format(e+1, epoch),
              "Training Loss: {:.3f}.. ".format(train_loss/len(train_loader)),
              "Test Loss: {:.3f}.. ".format(test_loss/len(testloader)),
              "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))
    train_losses.append(train_loss/len(train_loader))
    test_losses.append(test_loss/len(testloader))


title: Gradient is None is Pytorch

body:
For the following code, I would wish to get the gradient by get_grads(), but the output is always None, what is the reason?

import torch
import torch.nn as nn
import matplotlib.pyplot as plt
import numpy as np

X = torch.randn(100, 1)*10  #random with 100rows*1col, *10 to make it larger (original is small variance around 0)
y = X + 3*torch.randn(100, 1) #3*torch.randn(100, 1), it's the noise
plt.plot(X.numpy(), y.numpy(), 'o')
plt.ylabel('y')
plt.xlabel('x')

class LR(nn.Module):
  def __init__(self, input_size, output_size):
    super().__init__()
    self.linear = nn.Linear(input_size, output_size)
  def forward(self, x):
    pred = self.linear(x)
    return pred

torch.manual_seed(1)
model = LR(1, 1)

[w, b] = model.parameters()
def get_params():
  return (w[0][0].item(), b[0].item())
def get_grads():
  return (w[0][0].grad, b[0].grad)

criterion = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr = 0.01)

epochs = 30
losses = []
for i in range(epochs):
  y_pred = model.forward(X)
  loss = criterion(y_pred, y)
  print("epoch:", i, "loss:", loss.item())

  losses.append(loss)
  optimizer.zero_grad() #clear gradient for parameter]
  print("model=" + str(get_params())+str(get_grads()))
  loss.backward() #calculate gradient for parameters
  print("model2=" + str(get_params())+str(get_grads()))
  optimizer.step() #update the parameters with gradient
  print("model3=" + str(get_params())+str(get_grads()))
  print()

Output

epoch: 0 loss: 31.3035831451416
model=(0.5152631998062134, -0.44137823581695557)(None, None)
model2=(0.5152631998062134, -0.44137823581695557)(None, None)
model3=(1.447475790977478, -0.4449453055858612)(None, None)

epoch: 1 loss: 31.142377853393555
model=(1.447475790977478, -0.4449453055858612)(None, None)
model2=(1.447475790977478, -0.4449453055858612)(None, None)
model3=(0.5188075304031372, -0.41897052526474)(None, None)

epoch: 2 loss: 30.982675552368164
model=(0.5188075304031372, -0.41897052526474)(None, None)
model2=(0.5188075304031372, -0.41897052526474)(None, None)
model3=(1.4446537494659424, -0.42287370562553406)(None, None)

epoch: 3 loss: 30.824453353881836
model=(1.4446537494659424, -0.42287370562553406)(None, None)
model2=(1.4446537494659424, -0.42287370562553406)(None, None)
model3=(0.5223162770271301, -0.39742958545684814)(None, None)

epoch: 4 loss: 30.667678833007812
model=(0.5223162770271301, -0.39742958545684814)(None, None)
model2=(0.5223162770271301, -0.39742958545684814)(None, None)
model3=(1.4418396949768066, -0.40165263414382935)(None, None)

epoch: 5 loss: 30.51233673095703
model=(1.4418396949768066, -0.40165263414382935)(None, None)
model2=(1.4418396949768066, -0.40165263414382935)(None, None)
model3=(0.5257899761199951, -0.37672188878059387)(None, None)

epoch: 6 loss: 30.358415603637695
model=(0.5257899761199951, -0.37672188878059387)(None, None)
model2=(0.5257899761199951, -0.37672188878059387)(None, None)
model3=(1.4390342235565186, -0.3812492787837982)(None, None)

epoch: 7 loss: 30.205856323242188
model=(1.4390342235565186, -0.3812492787837982)(None, None)
model2=(1.4390342235565186, -0.3812492787837982)(None, None)
model3=(0.5292295813560486, -0.35681530833244324)(None, None)

epoch: 8 loss: 30.054668426513672
model=(0.5292295813560486, -0.35681530833244324)(None, None)
model2=(0.5292295813560486, -0.35681530833244324)(None, None)
model3=(1.4362375736236572, -0.3616321086883545)(None, None)

epoch: 9 loss: 29.904821395874023
model=(1.4362375736236572, -0.3616321086883545)(None, None)
model2=(1.4362375736236572, -0.3616321086883545)(None, None)
model3=(0.532635509967804, -0.3376788794994354)(None, None)

epoch: 10 loss: 29.756284713745117
model=(0.532635509967804, -0.3376788794994354)(None, None)
model2=(0.532635509967804, -0.3376788794994354)(None, None)
model3=(1.4334499835968018, -0.3427707254886627)(None, None)

epoch: 11 loss: 29.609052658081055
model=(1.4334499835968018, -0.3427707254886627)(None, None)
model2=(1.4334499835968018, -0.3427707254886627)(None, None)
model3=(0.5360085964202881, -0.3192828297615051)(None, None)
...


title: Rescale bounding boxes relatively to original image size

body:
I hope everyone’s doing great. Could you please give me a hand with the following problem. I believe there’s no such function in pytorch, opencv etc, so I am trying to implement it manually.

Context: I collect a batch of frames and push them to GPU, keep them there. Then, I have a couple of networks (YOLOs). In order for this to work, I need to resize images on GPU, to the size YOLO expects keeping aspect ratio (there's padding that fills the rest of the image with grey colour). Done as described here: https://discuss.pytorch.org/t/resize-images-torch-tensor-already-loaded-on-gpu-keeping-their-aspect-ratio/83172/9

What I need to do now: rescale bounding box coordinates so that they are relative to the original size of the frames, not the resized ones given to the network for detection. This is how I am doing it now:
code

@staticmethod
def rescale_bounding_box(detections: dict, current_dim: int, original_shape: tuple) -> dict:
    """
    Rescale bounding boxes predicted for image(s) of the size current_dim to the original_shape
    :param detections:
    :param current_dim:
    :param original_shape:
    :return:
    """
    original_h, original_w = original_shape
    # Added padding (check if the image was tall of short)
    pad_x = max(original_h - original_w, 0) * (current_dim / max(original_shape))
    pad_y = max(original_w - original_h, 0) * (current_dim / max(original_shape))

    # Image size after padding's been removed
    unpad_h = current_dim - pad_y
    unpad_w = current_dim - pad_x

    output = dict()
    for img_batch_index, predictions in detections.items():
        output[img_batch_index] = list()
        # On each image in the batch there's a number of detections that need to be rescaled
        for prediction in predictions:
            new_left = int((prediction[0] - pad_x // 2) * (original_w / unpad_w))
            new_top = int((prediction[1] - pad_y // 2) * (original_h / unpad_h))
            new_right = int((prediction[2] - pad_x // 2) * (original_w / unpad_w))
            new_bot = int((prediction[3] - pad_y // 2) * (original_h / unpad_h))
            obj_score = round(prediction[4], 4)
            conf = round(prediction[5], 4)
            index = int(prediction[6])

            print("Rescaled:", new_left, new_top, new_right, new_bot)
            # Save modified results
            assert all((new_left < new_right, new_top < new_bot)), "Coordinates rescaled wrong"
            assert all((new_left > 0, new_top > 0, new_right > 0, new_bot > 0)), "Coordinates rescaled wrong"
            assert all((new_right <= original_w, new_bot <= original_h)), "Coordinates rescaled wrong"
            output[img_batch_index].append(
                [new_left, new_top, new_right, new_bot, obj_score, conf, index]
            )

    return output

Unfortunately, it doesn’t pass the assert statements. My new_top value is usually < 0, new_right can go beyond the right edge.

detections - is a dictionary, where key - index of the image in the batch (1 for images), and values - list of lists. Each nested list is a detection predicted by the network.

If anyone could please give me a hand with it. Spent a lot of time on it, no success.

Thanks in advance.