title: Entropy Coeffient (Temperature) Autotune in SAC not working during continued training on a trained policy

body:
I am training an agent in a custom RL environment using SAC and wasn't sure why the entropy coefficient, when attempting continued training on an already partially-trained policy, wasn't autotuning. The autotuning of temperature when ent_coef is set to auto works perfectly fine, when training starts from scratch.
During continued training, the readout shows that there is a finite ent_coef_loss but the ent_coef itself remains a constant throughout the training. Based on my observation, it is set to the last recorded entropy coefficient of the initial training set. A workaround I found for this was to manually change log_ent_coeff from model.zip to a lower value but even then the autotune doesn't work other than the fact that this time around the training happens at a lower "constant" ent_coeff.
Earlier my ent_coef was set at 124 which I understand is a very large temperature.
What the ent_coeff should be set to in order to strike a balance between exploration and exploitation?
When ent_coeff was at 124, it was noticed that the agent would find great policies that yield higher episode rewards and episode lengths but soon enough revert back to lesser policies. Below are the code snippets I used for continued training and manually changing log_ent_coef in pytorch_variables.pth under model.zip.
Training code:
n_steps = 10000000;
# initially trained model hyperparameters
# model = SAC(MlpPolicy, env,buffer_size=1000000, learning_rate=0.0003, batch_size=64, ent_coef='auto', verbose=1)
print(str(log_dir_path)+'/PrevT_Model/model')
suf="PrevT_Model/model";
model = SAC.load(str(log_dir_path/suf));
model.set_env(env)
model.learn(total_timesteps=n_steps, callback=[checkpoint_callback,eval_callback], log_interval=50)

Manually change log_ent_coeff;
old_model=torch.load(str(log_dir_path_1/suf_1));
old_model['log_ent_coef']=torch.tensor([1.2], device='cuda:0', requires_grad=True)
torch.save(old_model, str(log_dir_path_1/suf_1))


title: How to pass arguments to Sagemaker Predictor endpoint?

body:
I want to pass arguments to a Sagemaker Predictor endpoint. Specifically, I want to pass in the directory from s3 from which to download a model file. This is how I deploy models right now, but I don't believe there are any parameters available for me to pass in arguments to?
import sagemaker
from sagemaker.pytorch import PyTorchModel
from sagemaker.serializers import JSONSerializer
from sagemaker.deserializers import JSONDeserializer

pytorch_predictor = PyTorchModel(
                                model_data=model_data,
                                role=role,
                                entry_point=entry_point,
                                source_dir=source_dir,
                                image_uri=image_uri,
                                framework_version='1.3.1',
                                py_version='py3'
                                )
predictor = pytorch_predictor.deploy(
                                    initial_instance_count=1,
                                    instance_type=instance_type,
                                    accelerator_type=accelerator_type,
                                    serializer=JSONSerializer(),
                                    deserializer=JSONDeserializer()
                                    )

This is my entry_point script:
def model_fn(model_dir):
    #loading model - ideally, I could pass in an argument here to download a model file from s3

def input_fn(request_body, request_content_type):
    #process input

# inference
def predict_fn(input_object, model):
    #inference

# postprocess
def output_fn(predictions, content_type):
    #postprocess


title: Imbalance data of target detection use pytorch

body:
There are two categories of image data, but there are many scenes, different cameras, different distances, daytime, night, rainy days and so on. The resolution of images is different, and there are three specifications.
In addition, there are more than 1000 pictures under the same camera, and only a dozen to hundreds of pictures from other cameras. I tried to take the more than 1000 samples down, but MAP was lower, and only more than 1000 pictures were taken after the sampling, and many scenes were in it.
How to do data enhancement for such images?


title: Load the ,trained model save, in pytorch for chatbots

body:
I ran the code of this tutorial (link) and after a while learning the model was completed and I chatted with the trained model. After exit the program, in the save directory next to the program file, 8 files with a tar extension were created.
I guess they are trained model saved files, what should I do to load those files and reuse them?


title: pytorch is_leaf property within a weights update loop

body:
I'm trying to understand the basic concepts of autograd using a 'deconstructed' piece of code for a simple perceptron
import numpy as np
from Sigmoid import *
import torch
#inputs
X = np.array([[0, 0, 1],
              [0, 1, 1],
              [1, 0, 1],
              [1, 1, 1]])
#targets
D = np.array([[0],
              [0],
              [1],
              [1]])

# Convert numpy arrays to torch tensors
X = torch.from_numpy(X).float()
D = torch.from_numpy(D).float()
W = torch.rand((1,3), requires_grad=True)

x = X[0,:].T
d = D[0]
v = torch.matmul(W, x)
y = Sigmoid(v)
e     = d - y
e2     = torch.square(e)
L = torch.sum(e2)
L.backward(retain_graph=True)

print(W.is_leaf) # gives True
dW = W.grad
W = W-0.9*dW
print(W.is_leaf) # now gives False
...

The variable W.is_leaf=True before W = W-0.9*dW and False afterwards. This is a problem because if you want to loop over the inputs you get the error below:

The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead.

Looking at the online documentation, I understand the reason for this: the operation W = W-0.9*dW recreates W causing W.is_leaf=False.
What is not clear to me is how to resolve this issue. After looking at several posts, I tried things like W.retain_grad() or L.backward(retain_graph=True), but they do not seem to work.
I'm sure I'm missing something very fundamental here and I'm wondering if somebody can point me in the right direction.
Thanks in advance


title: UnboundLocalError: local variable 'labels' referenced before assignment in PyTorch while doing X-Ray classification

body:
I am trying to replicate this Kaggle notebook, using my data. I am not posting the whole code here, because it is huge.
Received an error, which cannot fix myself
----> 2 model, history = run_fold(model, criterion, optimizer, scheduler, device=CFG.device, fold=0, num_epochs=CFG.num_epochs)

1 frames
<ipython-input-63-1ac8b53f265b> in train_model(model, criterion, optimizer, scheduler, num_epochs,
dataloaders, dataset_sizes, device, fold)
 23             for inputs, classes in dataloaders[phase]:
 24                 inputs = inputs.to(CFG.device)
---> 25                 labels = labels.to(CFG.device)
 26
 27                 # forward

UnboundLocalError: local variable 'labels' referenced before assignment

My data structure looks the same as the notebook

I have checked similar questions, there are several of them, but this is more tech question how to fix this thing. Maybe something wrong with index? Appreciate any tips.


title: How to load a pytorch model without access to the class code?

body:
So im looking for a way to load some model without acces to the class code. Is there a way? The only way to make such thing possible is to save the model with torch.jit.save/load but what if im not the person which training/designt the model?
With tensorflow im just used to use the load_model function. Is there something similary within pytorch?


title: Getting Runtime error for cross entropy. What should be changed and why it is coming

body:
I am testing on movie lens data set. I have tried increasing the output in nn.linear(n_act,5) to n_act,10 but nothing is working
The code is below and the error is afterwards
dls=CollabDataLoaders.from_df(ratings,item_name='Title',bs=64)

class deeplearn(Module):
    def __init__(self,user_size,movie_size,n_act=50,y_range=(0,5.5)):
        self.user_factors=create_params(user_size)
        self.movie_factors=create_params(movie_size)
        self.layers=nn.Sequential(
                    nn.Linear(user_size[1]+movie_size[1],n_act),
                    nn.ReLU(),
                    nn.Linear(n_act,5)
        )
        self.y_range=y_range

    def forward(self,x):
        embs=self.user_factors[x[:,0]],self.movie_factors[x[:,1]]
        x1=self.layers(torch.cat(embs,dim=1))
        return sigmoid_range(x1,*self.y_range)

embs=get_emb_sz(dls)
n_users=6041
n_movies=3707

model = deeplearn(*embs)
learn3 = Learner(dls, model, loss_func=CrossEntropyLossFlat)
learn3.fit_one_cycle(5, 5e-3, wd=0.01)

After executing getting
RuntimeError: Boolean value of Tensor with more than one value is ambiguous
RuntimeError                              Traceback (most recent call last)
nvs/tf-gpu/lib/python3.6/site-packages/torch/nn/_reduction.py in legacy_get_string(size_average, reduce, emit_warning)
     35         reduce = True
     36
---> 37     if size_average and reduce:
     38         ret = 'mean'
     39     elif reduce:

RuntimeError: Boolean value of Tensor with more than one value is ambiguous


everybody. I am using jupyter notebook to train a CNN model.I've set up the Pytorch(CUDA version) environment and other packages needed in this program before. Yesterday I just run this code successfully and come up with some expected result. But today I cannot run it. I didn't change anything. It shows that ImportError: cannot import name '_overloadtorch. I don't know how to solve this problem. Could anyone help me? (The picture is also showed via the link)
This is the information I got:
D:\software\Anaconda\anaconda\envs\pytorch\lib\site-packages\torch\nn\functional.py in <module>
      9 from torch._torch_docs import reproducibility_notes, tf32_notes
     10
---> 11 from .._jit_internal import boolean_dispatch, _overloadtorch
     12 from ..overrides import (
     13     has_torch_function, has_torch_function_unary, has_torch_function_variadic,

ImportError: cannot import name '_overloadtorch'

Best regards
enter image description here


title: GPR and kernel with pyro-ai

body:
I’m using pyro.contrib.gp to train my data but I have encountered a weird issue.
Say we define 2 GPR models as follows:
kernel_init = gp.kernels.RBF(input_dim= dimension, variance=torch.tensor(1.),
lengthscale=length_scale_init)
gpr_init = gp.models.GPRegression(train_x, train_y, kernel_init, noise=torch.tensor(0.), jitter = jitter)

kernel = gp.kernels.RBF(input_dim= dimension, variance=torch.tensor(1.),
lengthscale=length_scale_init)
gpr_opt = gp.models.GPRegression(train_x, train_y, kernel, noise=torch.tensor(0.), jitter = jitter)

Then after one optimizes the lengthscale parameters of the kernel by
optimizer = torch.optim.Adam([{‘params’: gpr_opt.kernel.lengthscale_unconstrained}], lr=5e-4)
for i in range(num_steps):
   # Zero gradients from previous iteration
   optimizer.zero_grad()
   # Calc loss and backprop gradients
   loss = loss_function(params)
   loss.backward()
   # Update step
   optimizer.step()
   losses.append(loss.item())

Finally, you want to compute the values of the two models at some test_x points:
with torch.no_grad():
   init_value = gpr_init(test_x, full_cov=False, noiseless=True)
   opt_value = gpr_opt(test_x, full_cov=False, noiseless=True)

And weirdly, one gets the same value equal to opt_value  = init_value but each gpr model has a kernel with different length scales. How come? It seems to work if I remove the init_value=... in the last step after with torch.no_grad():.


title: Visualizing the texture map in PyTorch3D

body:
I was going through the following wonderful Colab Notebook,
PyTorch3D · A library for deep learning with 3D data
This colab Notebook only has the code without the outputs. My personal copy of this Colab Notebook with outputs can be found below,
Copy of render_textured_meshes.ipynb - Colaboratory (google.com)
And when I executed the cell
plt.figure(figsize=(7,7))
texture_image=mesh.textures.maps_padded()
plt.imshow(texture_image.squeeze().cpu().numpy())
plt.grid("off");
plt.axis("off");

It gave the following output,

and when I ran the following cell,
plt.figure(figsize=(7,7))
texturesuv_image_matplotlib(mesh.textures, subsample=None)
plt.grid("off");
plt.axis("off");

it gave the following,

Can someone explain what these output images mean or denote? Are these 3D images open flat for visualization (like earth globe open flat into a world map)
Thankyou very much.


title: Siamese Network doesn't learn in pytroch

body:
I am trying to implement Siamese Network for face recognition, I use contrastive loss, but unfortunately, network doesn't output values more than the margin when two images don't belong to the same person, so when I test it it always outputs that two images belong to same person. here is code:
class Siamese_Network(nn.Module):
    def __init__(self, network):
        super(Siamese_Network, self).__init__()
        self.network = network

    def forward(self, inp1, inp2):
        inp1_pred = self.network(inp1)
        inp2_pred = self.network(inp2)
        return inp1_pred, inp2_pred

class ContrastiveLoss(nn.Module):
    def __init__(self, margin):
        super(ContrastiveLoss, self).__init__()
        self.margin = margin

    def forward(self, output1, output2, target):
        distances = (output2 - output1).pow(2).sum(1)
        losses = 0.2* (target.float() * distances +
                        0.8 * (-1 * target).float() * f.relu(self.margin - distances.sqrt()).pow(2))

        return losses.mean()

how I test: I iterate over images and see outputs of the network after inputting two images, if the distance between these two outputs (embedded vectors) are more than the margin then pictures don't belong to the same person, if the distance is less than a margin then pictures belong to the same person.
As value for margin is 100. (probably this is problem)


title: Initalize using previous .pth and train for further epochs in pytorch

body:
How do I initialize a UNet model from its previous weights and further train the model for more epochs?
This is current model and after training I am saving the state_dict() as .pth file
class UNet(nn.Module):
    def __init__(self):
        super(UNet,self).__init__()
        # encoder

        self.max_pool_2x2 = nn.MaxPool2d(kernel_size=2,stride=2)
        self.down_conv_1 = double_conv(3,64)
        self.down_conv_2 = double_conv(64,128)
        self.down_conv_3 = double_conv(128,256)
        self.down_conv_4 = double_conv(256,512)
        self.down_conv_5 = double_conv(512,1024)

        # decoder

        self.up_trans_1= nn.ConvTranspose2d(in_channels=1024,
                                            out_channels=512,
                                            kernel_size=2,
                                            stride=2)

        self.up_conv_1 = double_conv(1024,512)

        self.up_trans_2= nn.ConvTranspose2d(in_channels=512,
                                            out_channels=256,
                                            kernel_size=2,
                                            stride=2)

        self.up_conv_2 = double_conv(512,256)

        self.up_trans_3= nn.ConvTranspose2d(in_channels=256,
                                            out_channels=128,
                                            kernel_size=2,
                                            stride=2)

        self.up_conv_3 = double_conv(256,128)

        self.up_trans_4 = nn.ConvTranspose2d(in_channels=128,
                                            out_channels=64,
                                            kernel_size=2,
                                            stride=2)

        self.up_conv_4 = double_conv(128,64)

        self.out = nn.Conv2d(64,1,kernel_size=1)

    def forward(self, image):
       ##code for forward prop

model=UNet()
model.cuda()
# y= model(image)

# model.train()
# optimizer = optim.SGD(model.parameters(), lr = 0.1, momentum=0.9)

Now I want to initialize the same model with that .pth file and train further?


title: how to load and reuse saved embeddings which is saved in ".npy" format

body:
I am learning Image similarity with pytorch from someones code. Here I am using model of series of convnet to generate enmbedding name:- encoder , and decoder for convert back to those image in their original shape.
Here I saved generated embeddings in .npy format.
So my question is how to load these embeddings and reuse it.


title: Implementing neural network in pytorch on a non-Image dataset

body:
So I want to implement a neural network on a classification problem. Examples on the internet are of implementing an Image classification problem where they usually load a dataset from the library but I want to implement it on a non image classification problem. (3 classes)
So I have a dataframe of 12000x13 (12000 points with 12 features, the last column is the target variable). I want to train my neural network on this dataset. First I am having issues converting this data frame to the appropriate dimension (and size) tensor and then how to define torch_loader and test_loader. Do we separate the target variable first or Pytorch handles that on its own?
Once I have input in the Pytorch required format, I will mostly don't have issues defining the network structure since that will be similar to the image classification problems on the Internet.
PS: mostly over the Internet they have provided an example of implementation on the MNIST dataset, where I was not able to figure out an answer to my problem


title: Adding a fixed bias only to ouput layer sequential network

body:
Given the following neural network,
model = torch.nn.Sequential(
    torch.nn.Linear(X.shape[-1], 64),
    torch.nn.ReLU(),
    torch.nn.Dropout(0.5),
    torch.nn.Linear(64,1),
    torch.nn.Sigmoid()
    )

I would like to add a fixed bias only to the output layer (Sigmoid). How can that be done?


title: Is it possible to get human readable output when using BERT for multi-class classification?

body:
Just starting with NLP and ML and I've been trying to put together a model using BERT for multi-class sentiment analysis. All the examples e.g. https://pysnacks.com/machine-learning/bert-text-classification-with-fine-tuning/#multi-class-text-classification-using-bertand tutorials seem to consist of the following broad steps:

Input labelled dataset
Encode labelled dataset after converting relevant column to list
Build model, fine-tune model on encoded dataset
Evaluate model on trained dataset

But what I'm looking for is a fifth step where you can output the actual predictions so if I start with a dataset that has Label | Text I'd like to end up with a dataset that has Label | Text | BERT Predicted Lablel.
I can't quite figure out how to go about that step as there doesnt seem to be anyway to map the models predictions back to the observations in the original dataset


title: How augmentation increase number of images

body:
I am confused about how augmentation increases the number of images. Can I calculate it?
Let say I have 10 images in batch
def augmentation():
    aug=transforms.Compose([
        transforms.Resize(size=(248,248)),
        transforms.RandomHorizontalFlip(1),
        transforms.RandomVerticalFlip(1),
        transforms.RandomRotation(degrees=10),
        transforms.ToTensor(),
        transforms.Normalize((0.2, ), (0.2, )),
                          ])
    return aug

After the augmentation in pytorch, how many images my model will be trained on in a single batch. I know at the end the data size remain same, but I want to know does it increase at any point in the pipeline


title: pytorch cross-entropy-loss weights not working

body:
I was playing around with some code and and it behaved differently than what i expected. So i dumbed it down to a minimally working example:
import torch

test_act = torch.tensor([[2.,0.]])
test_target = torch.tensor([0])

loss_function_test = torch.nn.CrossEntropyLoss()
loss_test = loss_function_test(test_act, test_target)
print(loss_test)
> tensor(0.1269)

weights=torch.tensor([0.1,0.5])
loss_function_test = torch.nn.CrossEntropyLoss(weight=weights)
loss_test = loss_function_test(test_act, test_target)
print(loss_test)
> tensor(0.1269)

As you can see the outputs are the same regardless if there are weights present or not. But i would expect the second output to be 0.0127
Is there some normalization going on that I dont know about? Or is it possibly bugged?


title: Evaluating model leads to CUDA out of memory error

body:
my model takes in a sequence RGB images (512 x 512 x 3) and makes a prediction based on that. I was testing a sequence of 224 images when I get this error:

RuntimeError: CUDA out of memory. Tried to allocate 4.00 GiB (GPU 0; 15.78 GiB total capacity; 10.21 GiB already allocated; 1.06 GiB free; 13.43 GiB reserved in total by PyTorch)

This happened on Colab but I also ran into on an AWS ml.p2.xlarge notebook instance and had a similar error.
The solutions I've seen all point to reducing batch size but this doesn't make sense to me because my batch size is already 1:

torch.Size([1, 224, 3, 512, 512])

How do I fix this problem? It seems like I may be loading something wrong for 4.00 gigs to be needed

512 * 512 * 3 * 224 = 176,160,768

EDIT: Also meant to add that I tried changing the number of workers down to 1 and that did not resolve the issue


title: Methods to Load Saved Python List in Pytorch?

body:
I have a question on how to load saved python lists in Pytorch. The reference environment is Google Colab.
Suppose I saved the regular list as follows.
drive.mount('/content/gdrive')
training_sets = ["a","b","c"]
model_save_name = 'training_sets.pt'
path = F"/content/gdrive/My Drive/Documents"
torch.save(training_sets,path)

Does pytorch support loading for regular python lists? Or does it only apply for pytorch tensors?
Thank you.


title: Loss is nan, stopping training when training Mask-RCNN multi-class segmentation

body:
number of train data: 346
number of test data: 69
Epoch: [0] [0/346] eta: 0:35:20 lr: 0.000019 loss: -312.6024 (-312.6024) loss_classifier: 1.5789 (1.5789) loss_box_reg: 0.1299 (0.1299) loss_mask: -314.3485 (-314.3485) loss_objectness: 0.0266 (0.0266) loss_rpn_box_reg: 0.0106 (0.0106) time: 6.1275 data: 0.1599 max mem: 0
Loss is nan, stopping training
{‘loss_classifier’: tensor (nan, grad_fn = ), ‘loss_box_reg’: tensor (nan, grad_fn = ), ‘loss_mask’: tensor (nan, grad_fn = ), ’ tensor (nan, grad_fn = ), ‘loss_rpn_box_reg’: tensor (nan, grad_fn = )}
An exception has occurred, use% tb to see the full traceback.

SystemExit : 1

And this is the dataset code:

class maskrcnn_Dataset(torch.utils.data.Dataset):
   def __init__(self, root, transforms=None):
    self.root = root
    self.transforms = transforms
    # load all image files, sorting them to
    # ensure that they are aligned
    self.imgs = list(sorted(os.listdir(os.path.join(root, "images"))))
    self.masks = list(sorted(os.listdir(os.path.join(root, "masks"))))
    #self.class_masks = list(sorted(os.listdir(os.path.join(root, "SegmentationClass"))))

def __getitem__(self, idx):
    # load images ad masks
    img_path = os.path.join(self.root, "images", self.imgs[idx])
    x=self.imgs[idx].split('.')
    mask_path = os.path.join(self.root, "masks", self.masks[idx])
    #class_mask_path = os.path.join(self.root, "SegmentationClass", self.class_masks[idx])

    #read and convert image to RGB
    img = cv2.imread(img_path)
    mask_for_all=[]
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    # note that we haven't converted the mask to RGB,
    # because each color corresponds to a different instance
    # with 0 being background
    # mask = Image.open(mask_path)
    mask_folder=os.path.join(self.root,"masks")
    source_mask = os.path.join(mask_folder, x[0])
    #print(os.listdir(source_mask))
    boxes = []
    xx=trier(os.listdir(source_mask))
    #print(xx)
    for file_name in xx:
        mask = Image.open(os.path.join(source_mask,file_name))
        mask = np.array(mask)
        mask_for_all.append(mask)
        obj_ids = np.unique(mask)
        obj_ids = obj_ids[1:]
        masks = mask == obj_ids[:, None, None]
        num_objs = len(obj_ids)
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

    num_objs=len(boxes)
    boxes = torch.as_tensor(boxes, dtype=torch.float32)
    # there is only one class
    if(self.root.find("train")!=-1):
        #print("bisgltjf")

        labels =class_ids_train[class_ids_train_names.index(self.imgs[idx])]
        #print(labels)
    else:
        labels =class_ids_val[class_ids_val_names.index(self.imgs[idx])]
        #print('l3assba')

    #labels = np.array([])
    #for i in range(masks.shape[0]):
     #   labels = np.append(labels, (masks[i] * class_mask).max())

    labels = torch.as_tensor(labels, dtype=torch.int64)
    #print(boxes,":",labels)
    masks = torch.as_tensor(mask_for_all, dtype=torch.uint8)
    #print(labels)
    #print(masks)
    #print(masks.shape)

    image_id = torch.tensor([idx])
    area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
    # suppose all instances are not crowd
    iscrowd = torch.zeros((num_objs,), dtype=torch.int64)
    #print(img.shape)
    #print(self.imgs[idx])

    target = {}
    target["boxes"] = boxes
    #print(boxes)
    target["labels"] = labels
    #print(labels.shape)
    target["masks"] = masks
    #print(masks.shape)
    target["image_id"] = image_id
    #print(image_id.shape)
    target["area"] = area
    #print(area)
    target["iscrowd"] = iscrowd
    #print(iscrowd.shape)

    if self.transforms is not None:
        img, target = self.transforms(img, target)

    return img, target

def __len__(self):
    return len(self.imgs)


title: PyTorch MNIST Model RuntimeError: mat1 and mat2 shapes cannot be multiplied (3584x28 and 784x10)

body:
I'm trying to build a simple MNIST Model and this is what I've built -
training_loader = DataLoader(training_dataset, 128, shuffle = True)
validation_loader = DataLoader(validation_dataset, 128)

class mnistmodel(nn.Module):
  def __init__(self):
    super().__init__()
    self.linear1 = nn.Linear(784, 10)
    self.linear2 = nn.Linear(10, 5)
    self.linear3 = nn.Linear(5, 10)

  def forward(self, xb):
    xb.reshape(-1, 784)
    predicted = F.relu(self.linear1(xb))
    predicted.reshape(-1, 10)
    predicted = F.relu(self.linear2(predicted))
    predicted.reshape(-1, 5)
    predicted = self.linear3(predicted)
    return predicted

  def training_step(self, batch):
    images, labels = batch
    predicted = self(images)
    loss = F.cross_entropy(predicted, labels)
    return loss

  def validation_step(self, batch):
    images, labels = batch
    predicted = self(images)
    loss = F.cross_entropy(predicted, labels)
    _, preds = torch.max(predicted, dim=1)
    accuracy = torch.tensor(torch.sum(preds == labels).item() / len(preds))
    return {'validation_loss': loss, 'validation_accuracy': accuracy}

  def validation_epoch_end(self, outputs):
    batch_losses = [x['validation_loss'] for x in outputs]
    epoch_loss = torch.stack(batch_losses).mean()
    batch_accs = [x['validation_acc'] for x in outputs]
    epoch_acc = torch.stack(batch_accs).mean()
    return {'validation_loss': epoch_loss.item(), 'validation_accuracy': epoch_acc.item()}

def epoch_end(self, epoch, result):
    print(f"Epoch [{epoch}], val_loss: {result['validation_loss']}, val_acc: {result['validation_acc']}")

model = mnistmodel()

def fit_mnist(epochs, lr, model, training_loader, validation_loader, optimizer_function=torch.optim.SGD):
    optimizer = optimizer_function(model.parameters(), lr)
    history = []

    for epoch in range(epochs):

        for batch in training_loader:
            loss = model.training_step(batch)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

        result = evaluate(model, validation_loader)
        model.epoch_end(epoch, result)
        history.append(result)

    return history

history1 = fit_mnist(5, 0.001, model, training_loader, validation_loader)

I get the following error -
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-162-48e4fe0cc2d9> in <module>()
----> 1 history1 = fit_mnist(5, 0.001, model, training_loader, validation_loader)

6 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

RuntimeError: mat1 and mat2 shapes cannot be multiplied (3584x28 and 784x10)

I'm new to pytorch but as far as I understand the shapes seem to be fine, what is going wrong here?
I've checked manually and the dimensions seem to be fine. I even tried multiple things removing adding reshape commands but this error persists


title: I am unable to convert pytorch ML model into tflite

body:
I am trying to convert the following PyTorch ML model for car image classification into tflite but it is giving me unknown tensors. Please help me.
import matplotlib.pyplot as plt
import numpy as np

import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.models as models
import torchvision.transforms as transforms

import time
import os
import PIL.Image as Image
from IPython.display import display

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)
print(torch.cuda.get_device_name(device))

dataset_dir = "../input/car_data/car_data/"

train_tfms = transforms.Compose([transforms.Resize((400, 400)),
                             transforms.RandomHorizontalFlip(),
                             transforms.RandomRotation(15),
                             transforms.ToTensor(),
                             transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
test_tfms = transforms.Compose([transforms.Resize((400, 400)),
                            transforms.ToTensor(),
                            transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

dataset = torchvision.datasets.ImageFolder(root=dataset_dir+"train", transform = train_tfms)
trainloader = torch.utils.data.DataLoader(dataset, batch_size = 32, shuffle=True, num_workers = 2)

dataset2 = torchvision.datasets.ImageFolder(root=dataset_dir+"test", transform = test_tfms)
testloader = torch.utils.data.DataLoader(dataset2, batch_size = 32, shuffle=False, num_workers = 2)

def train_model(model, criterion, optimizer, scheduler, n_epochs = 5):

losses = []
accuracies = []
test_accuracies = []
# set the model to train mode initially
model.train()
for epoch in range(n_epochs):
    since = time.time()
    running_loss = 0.0
    running_correct = 0.0
    for i, data in enumerate(trainloader, 0):

        # get the inputs and assign them to cuda
        inputs, labels = data
        #inputs = inputs.to(device).half() # uncomment for half precision model
        inputs = inputs.to(device)
        labels = labels.to(device)
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        _, predicted = torch.max(outputs.data, 1)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # calculate the loss/acc later
        running_loss += loss.item()
        running_correct += (labels==predicted).sum().item()

    epoch_duration = time.time()-since
    epoch_loss = running_loss/len(trainloader)
    epoch_acc = 100/32*running_correct/len(trainloader)
    print("Epoch %s, duration: %d s, loss: %.4f, acc: %.4f" % (epoch+1, epoch_duration, epoch_loss, epoch_acc))

    losses.append(epoch_loss)
    accuracies.append(epoch_acc)

    # switch the model to eval mode to evaluate on test data
    model.eval()
    test_acc = eval_model(model)
    test_accuracies.append(test_acc)

    # re-set the model to train mode after validating
    model.train()
    scheduler.step(test_acc)
    since = time.time()
print('Finished Training')
return model, losses, accuracies, test_accuracies

def eval_model(model):
correct = 0.0
total = 0.0
with torch.no_grad():
    for i, data in enumerate(testloader, 0):
        images, labels = data
        #images = images.to(device).half() # uncomment for half precision model
        images = images.to(device)
        labels = labels.to(device)

        outputs = model_ft(images)
        _, predicted = torch.max(outputs.data, 1)

        total += labels.size(0)
        correct += (predicted == labels).sum().item()

test_acc = 100.0 * correct / total
print('Accuracy of the network on the test images: %d %%' % (
    test_acc))
return test_acc

model_ft = models.resnet34(pretrained=True)
num_ftrs = model_ft.fc.in_features

# replace the last fc layer with an untrained one (requires grad by default)
model_ft.fc = nn.Linear(num_ftrs, 196)
model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model_ft.parameters(), lr=0.01, momentum=0.9)

lrscheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='max', patience=3, threshold = 0.9)

model_ft, training_losses, training_accs, test_accs = train_model(model_ft, criterion, optimizer, lrscheduler, n_epochs=10)

This is the Machine Learning model I am trying to convert but I am not able to identify the declaration of the model to convert it to tflite. Please help me with the steps to convert this model to tflite.


title: How to pass 2 loaders to training of a multi input network

body:
I have a multi input neural network and I am using as dataset MNIST. Half MNIST is for the first data loader and the second half of MNIST is for the second data loader.
This is my training function:
def train(net, loaders, optimizer, criterion, epochs=20, dev=None, save_param=False, model_name="valerio"):
    # try:
      net = net.to(dev)
      #print(net)
      #summary(net,[(net.in_channels,net.in_width,net.in_height)]*2)

      criterion.to(dev)

      # Initialize history
      history_loss = {"train": [], "val": [], "test": []}
      history_accuracy = {"train": [], "val": [], "test": []}
      # Store the best val accuracy
      best_val_accuracy = 0

      # Process each epoch
      for epoch in range(epochs):
        # Initialize epoch variables
        sum_loss = {"train": 0, "val": 0, "test": 0}
        sum_accuracy = {"train": 0, "val": 0, "test": 0}

        progbar = None
        # Process each split
        for split in ["train", "val", "test"]:
          if split == "train":
            net.train()
            widgets = [
              ' [', pb.Timer(), '] ',
              pb.Bar(),
              ' [', pb.ETA(), '] ', pb.Variable('ta','[Train Acc: {formatted_value}]')
            ]

            progbar = pb.ProgressBar(max_value=len(loaders[split]),widgets=widgets,redirect_stdout=True)

          else:
            net.eval()
          # Process each batch
          for j,(input, labels) in enumerate(loaders[split]):
            load_a = enumerate(loaders_a[split])
            load_b = enumerate(loaders_b[split])
            labels = labels.unsqueeze(1)
            labels = labels.float()

            input = input.to(dev)
            labels = labels.to(dev)

            _,(input_a, labels_a) = load_a[j]
            _,(input_b, labels_b) = load_b[j]

            input_a = input.to(dev)
            labels_a = labels.to(dev)
            input_b = input.to(dev)
            labels_b = labels.to(dev)

            # Reset gradients
            optimizer.zero_grad()
            # Compute output
            pred = net(input_a,input_b)

            loss = criterion(pred, [labels_a, labels_b])
            # Update loss
            sum_loss[split] += loss.item()
            # Check parameter update
            if split == "train":
              # Compute gradients
              loss.backward()
              # Optimize
              optimizer.step()

            # Compute accuracy

            #https://discuss.pytorch.org/t/bcewithlogitsloss-and-model-accuracy-calculation/59293/ 2
            pred_labels = (pred[2] >= 0.0).long()  # Binarize predictions to 0 and 1
            pred_labels_a = (pred[0] >= 0.0).long()  # Binarize predictions to 0 and 1
            pred_labels_b = (pred[1] >= 0.0).long()  # Binarize predictions to 0 and 1

            batch_accuracy = (pred_labels == labels).sum().item() / len(labels)
            batch_accuracy_a = (pred_labels_a == labels_a).sum().item() / len(labels_a)
            batch_accuracy_b = (pred_labels_b == labels_b).sum().item() / len(labels_b)
            # Update accuracy
            sum_accuracy[split] += batch_accuracy

            if (split=='train'):
              progbar.update(j, ta=batch_accuracy)

        if (progbar is not None):
          progbar.finish()
        # Compute epoch loss/accuracy
        epoch_loss = {split: sum_loss[split] / len(loaders[split]) for split in ["train", "val", "test"]}
        epoch_accuracy = {split: sum_accuracy[split] / len(loaders[split]) for split in ["train", "val", "test"]}

        # Store params at the best validation accuracy
        if save_param and epoch_accuracy["val"] > best_val_accuracy:
          # torch.save(net.state_dict(), f"{net.__class__.__name__}_best_val.pth")
          torch.save(net.state_dict(), f"{model_name}_best_val.pth")
          best_val_accuracy = epoch_accuracy["val"]

        # Update history
        for split in ["train", "val", "test"]:
          history_loss[split].append(epoch_loss[split])
          history_accuracy[split].append(epoch_accuracy[split])
        # Print info
        print(f"Epoch {epoch + 1}:",
              f"TrL={epoch_loss['train']:.4f},",
              f"TrA={epoch_accuracy['train']:.4f},",
              f"VL={epoch_loss['val']:.4f},",
              f"VA={epoch_accuracy['val']:.4f},",
              f"TeL={epoch_loss['test']:.4f},",
              f"TeA={epoch_accuracy['test']:.4f},")
    # except KeyboardInterrupt:
    #   print("Interrupted")
    # finally:
    #   # Plot loss
    #   plt.title("Loss")
    #   for split in ["train", "val", "test"]:
    #     plt.plot(history_loss[split], label=split)
    #   plt.legend()
    #   plt.show()
    #   # Plot accuracy
    #   plt.title("Accuracy")
    #   for split in ["train", "val", "test"]:
    #     plt.plot(history_accuracy[split], label=split)
    #   plt.legend()
    #   plt.show()

I have 2 data loaders, loaders_a and loaders_b, because my neural network is multi input. How can I modify this training function in order to work with these 2 loaders?
These are the loaders:
# Define dictionary of loaders
loaders = {"train": train_loader,
           "val": val_loader,
           "test": test_loader}

# Define dictionary of loaders
loaders_a = {"train": train_loader_a,
           "val": val_loader_a,
           "test": test_loader_a}

loaders_b = {"train": train_loader_b,
           "val": val_loader_b,
           "test": test_loader_b}


title: Is permute operation differentiable operation in PyTorch

body: For my project I am using permute operation to change the index order of the e tensor in encoder path  but I am not sure if this is  differentiable operation? If it is not, Is there any way to change tensor shape before the convolution.


title: Torch argmax for N highest values

body:
I would like to do something like an argmax but with multiple top values. I know how to use the normal torch.argmax
>>> a = torch.randn(4, 4)
>>> a
tensor([[ 1.3398,  1.2663, -0.2686,  0.2450],
        [-0.7401, -0.8805, -0.3402, -1.1936],
        [ 0.4907, -1.3948, -1.0691, -0.3132],
        [-1.6092,  0.5419, -0.2993,  0.3195]])
>>> torch.argmax(a)
tensor(0)

But now I need to find the indices for the top N values. So something like this
>>> a = torch.randn(4, 4)
>>> a
tensor([[ 1.3398,  1.2663, -0.2686,  0.2450],
        [-0.7401, -0.8805, -0.3402, -1.1936],
        [ 0.4907, -1.3948, -1.0691, -0.3132],
        [-1.6092,  0.5419, -0.2993,  0.3195]])
>>> torch.argmax(a,top_n=2)
tensor([0,1])

I haven't found any function capable of doing this in pytorch, does anyone know?


title: Pytorch problem porting code from mac to windows

body:
I am trying to port my code for training a NN from mac os python 2.7 to python 3.7 windows and I fixed all of my code, my NN trains normally. torch version is 0.4.1

The problem is that when I evaluate the NN python just stops without any error message. It simply stops.

This is the exact line:

correct += pred.eq(target.data.view_as(pred)).sum()

is there a possibility this function is not supported on windows ? But why there is no message for quitting ?


title: FCN dimension in ConvNet+FCN example

body:
So my neural network defined in pytorch is as follows (taken from https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html)

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

Assuming I want to apply NN on a 1 * 32 * 32 tensor, how does self.fc1 come to be 16 * 6 * 6. I understand the 16, not sure how we get 6.


title: How to replace and multiply Pytorch tensor dimensions

body:
I have a tensor with the dimensions of x = torch.Size([128, 99, 8, 31]).
I need to multiply the second position ‘channels’ with itself to get x = torch.Size([128, (99 * 99 ), 8, 31]) = torch.Size([128, 9801, 8, 31]).
Is this logically accepted, and how to do it?
Also
I have a tensor with the dimnsions of y = torch.Size([128, 1, 99, 99]).
I need to replace the third position value (99) with (1), to get y = torch.Size([128, 1, 1, 99]). how I can do that?
I have tried tensor.view and tensor.reshape but didn’t work with me as I expect.


title: Why different shapes in RNNCell created from scratch in numpy to pytorch?

body:
I am trying to recreate pytorch's RNNCell in numpy using the same equation available in the documentation of RNNCell.
import numpy as np

inputs = 10
hiddens = 6
rows = 3

#input data
x = np.random.rand(rows,inputs)

w_ih = np.random.rand(hiddens,inputs)
w_hh = np.random.rand(hiddens,hiddens)
b_ih = np.random.rand(hiddens,1)
b_hh = np.random.rand(hiddens,1)

# initial hidden values
h = np.zeros((hiddens,1))

h = w_ih @ x.T + b_ih + w_hh @ h + b_hh
print("Shape",h.shape)

In this approach the output shape is (6,3). In order to verify this I created a sample code in pythorch RNNCell itself
import torch
torch_rnn = torch.nn.RNNCell(inputs,hiddens)
xx = torch.rand(rows,inputs)
print("shape",torch_rnn(xx).shape)

Here the shape is (3,6) transpose of my numpy implementation.
Why the numpy implementation is giving wrong shape for hidden state?
For initial hidden values I chose this way to h = np.zeros((hiddens,1)) initialize it which is wrong. How should I initialize hidden values for my RNNCell?


title: Pytorch two forwards before a backwards

body:
I'm implementing a timeseries model where I want to train a LSTM to remove the effect of events from the data for the input window, then predict an output window on that de-evented data, before using the same LSTM to predict the effect of events for the output window and adding them back in to produce the final prediction.
for batch, data in enumerate(dataLoader):
            X_time, X_event_in, X_event_out, y, idx = data
            idx = idx.tolist()

            X_time = X_time.to(device)
            X_event_in = X_event_in.to(device)
            X_event_out = X_event_out.to(device)
            y = y.to(device)

            # Find the number of timesteps
            batchsize, n_timesteps, features = X_time.shape

            # Reset the hidden state for each training sample for the time model
            time_model.hidden = time_model.init_hidden()

            for t in range(n_timesteps):

                # Predict the effect of events on the input sequence
                event_model.hidden = event_model.init_hidden()
                events_in = event_model(X_event_in[:,t,:,:])

                # Remove the effect of events
                X_time_events_removed = X_time[:,t,:]/events_in

                # Predict on the timeseries data
                time_output = time_model(X_time_events_removed, idx, t)

                # Predict the effect of events on the output sequence
                event_model.hidden = event_model.init_hidden()
                # For now detach these since I can't figure out how to run backwards on the model twice
                events_out = event_model(X_event_out[:,t,:,:])

                # Add back in the effect of events
                y_pred = time_output*events_out.detach()

                # Optimisation step
                optimizer.zero_grad()
                single_loss = loss_function(y_pred, y[:,t,:])
                single_loss.backward()
                optimizer.step()

where time_model and event_model are PyTorch nn.Module classes.
However, even when I detach the gradient from the second run through events_model, I still get the error python RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
Is it possible to train a model in this way, and if so how do I do it?


title: How to Train End to End model with Part RNN and Part Graph Neural Network?

body:
I am currently working on a project in which I need to use RNNs in part of a neural network. Essentially the RNN would take in a text of variable length and output a feature representation. This feature representation would then be clubbed with some more feature vectors and fed to a different graph neural network. Some loss would be calculated on the output of the graph neural network and this loss would be backpropagated across the entire network including the RNN and be used to train the entire end to end network.
However, I am not able to wrap my head around how I can use the RNN as a part of another different non-sequential model. I use PyTorch for most of my work.
Can anyone suggest any way in which I may address this problem. Or refer to any material which might be useful.
Thanks


title: Why does an indexed tensor retain the original tensor's stride?

body:
My understanding of stride is the number of steps you'd need to take to jump to the next part in an axis. So if you have a stride of (3,1) you'd need to take 3 steps to get to the next row and 1 step to get to the next column (assuming the first axis is row and the second axis is column).
Yet when I index into a pytorch tensor, b[1:, 1:] with shape (3,3) lopping off the first row
and the first column, then query its stride, I get (3,1) instead of (2,1).
Why is this the case?
import unittest
import torch
class TestChapter3(unittest.TestCase):
    def setUp(self):
        self.a = torch.tensor(list(range(9)))
        self.b = self.a.view(3,3)
        self.c = self.b[1:, 1:]

    def test_index_of_view(self):
        print(self.c)
        self.assertEqual(self.c.size(), torch.Size([2, 2]))
        self.assertEqual(self.c.storage_offset(), 4)
        self.assertEqual(self.c.stride(), (2, 1)) # self.c.stride() is actually (3,1)

if __name__ == "__main__":
    unittest.main()


title: Spawn context does not produce any output

body:
I am trying to use PyTorch with multiprocessing, and uses spawn context as others have suggested. However, when used, it does not produce any output.
import torch.multiprocessing as mp
import sys

def worker(num):
    """thread worker function"""

    print('Worker:', num)

if __name__ == '__main__':
    jobs = []
    ctx = mp.get_context("spawn")

    for i in range(4):
        p = ctx.Process(target=worker, args=(i,))
        jobs.append(p)
        p.start()

    map(lambda p: p.join(), jobs)

Any help would be appreciated.


title: deploy 3 NLP Pytorch models to a flask app not working

body:
I followed the notebook 2, 4 and 6 of this repo:https://github.com/bentrevett/pytorch-sentiment-analysis to built a sentiment analyzer. I now want to deploy them using flask but I can't manage to get things working. Can experienced people with this task comment or dm me for me to deploy these models to flask?
I tried saving and loading the vocab using pickle or dill but there were errors with padx and also that the field attribute does not have an attribute stoi.
I am honestly stuck and trying for weeks now. I need to hand in the project next week :/


title: Sudden reinitialization of train and validation loss and slow covergence while training CNN in Pytorch

body:
I am a beginner in deep learning, especially in designing neural networks from scratch and I am trying to build a B/W image colorizer using the Autoencoder concept in Pytorch while referring this Keras architecture from here using the dataset here. The Keras architecture for reference:
#Design the neural network
model = Sequential()
model.add(InputLayer(input_shape=(256, 256, 1)))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same', strides=2))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same', strides=2))
model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(256, (3, 3), activation='relu', padding='same', strides=2))
model.add(Conv2D(512, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(256, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(128, (3, 3), activation='relu', padding='same'))
model.add(UpSampling2D((2, 2)))
model.add(Conv2D(64, (3, 3), activation='relu', padding='same'))
model.add(UpSampling2D((2, 2)))
model.add(Conv2D(32, (3, 3), activation='relu', padding='same'))
model.add(Conv2D(2, (3, 3), activation='tanh', padding='same'))
model.add(UpSampling2D((2, 2)))

# Finish model
model.compile(optimizer='rmsprop', loss='mse')

# Image transformer
datagen = ImageDataGenerator(
    shear_range=0.2,
    zoom_range=0.2,
    rotation_range=20,
    horizontal_flip=True)

# Generate training data
batch_size = 50

def image_a_b_gen(batch_size):
    for batch in datagen.flow(Xtrain, batch_size=batch_size):
        lab_batch = rgb2lab(batch)
        X_batch = lab_batch[:,:,:,0]
        Y_batch = lab_batch[:,:,:,1:] / 128
        yield (X_batch.reshape(X_batch.shape+(1,)), Y_batch)

# Train model
TensorBoard(log_dir='/output')
model.fit_generator(image_a_b_gen(batch_size), steps_per_epoch=10000, epochs=1)

And this is the architecture that I have built in PyTorch:
class AutoEncoder(nn.Module):
    def __init__(self):
        super().__init__()

        self.model = nn.Sequential(
        nn.Conv2d(1, 64, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Conv2d(64, 128, (3,3), padding = (1,1), stride = (2,2)),
        nn.ReLU(),
        nn.Conv2d(128, 128, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Conv2d(128, 256, (3,3), padding = (1,1), stride = (2,2)),
        nn.ReLU(),
        nn.Conv2d(256, 256, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Conv2d(256, 512, (3,3), padding = (1,1), stride = (2,2)),
        nn.ReLU(),
        nn.Conv2d(512, 512, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Conv2d(512, 256, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Conv2d(256, 128, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Upsample(scale_factor= (2,2)),
        nn.Conv2d(128, 64, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Upsample(scale_factor= (2,2)),
        nn.Conv2d(64, 32, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Conv2d(32, 2, (3,3), padding = (1,1)),
        nn.ReLU(),
        nn.Upsample(scale_factor= (2,2)),
)

  def forward(self, input):
       output = self.model(input)
       return output

And initialize like this:
model = AutoEncoder()
loss_func = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

This is the training function (trained on Google Colab) with validation incorporated:
  def train(model, optimizer, loss_func, train_loader, val_loader, epochs, device = 'cuda'):
  model = model.to(device)
  loss_func = loss_func.to(device)
  training_loss = 0
  valid_loss = 0
  save_path = {'grayscale': '/content/outputs/gray/', 'colorized': '/content/outputs/color/',
               'target': '/content/outputs/target/'}

  for epoch in range(e+1, e+1+epochs):
    print('Starting epoch {}'.format(epoch))
    epoch_start = time.time()
    training_loss = 0.0
    valid_loss = 0.0

    model.train()
    b = 1
    for batch in train_loader:
      batch_start = time.time()
      optimizer.zero_grad()
      img_gray, img_ab, og_image = batch
      input, target = img_gray, img_ab
      input, target = input.to(device), target.to(device)
      output = model(input)
      loss = loss_func(output, target)
      loss.backward()
      optimizer.step()

      training_loss += loss.data.item()
      if b % 20 == 0:
        print('Epoch: {} , Batch: {} Training Loss: {:.4f}, Time: {:.2f}'.format(
        epoch, b, loss.data.item(), time.time() - batch_start))
      b += 1

    training_loss /= len(train_loader)

    model.eval()
    b = 1
    for batch in val_loader:
      batch_start = time.time()
      img_gray, img_ab, og_image = batch
      input, target = img_gray, img_ab
      input, target = input.to(device), target.to(device)
      output = model(input)
      loss = loss_func(output, target)

      valid_loss += loss.data.item()

      j = random.randint(1,9)
      save_name = 'epoch-{}batch-{}img-{}.jpg'.format(epoch, b, j)
      save_image(input[j].detach().cpu(), output[j].detach().cpu(), target[j].detach().cpu(),
                 og_image[j].detach().cpu(), save_path=save_path, save_name=save_name)

      print('Epoch: {} , Batch: {} Valid Loss: {:.4f}, Time: {:.2f}'.format(
      epoch, b, loss.data.item(), time.time() - batch_start))

      b += 1

    valid_loss /= len(val_loader)

    # model saving code here

    print('Epoch: {} , Training Loss: {:.4f}, Validation Loss: {:.4f}, Time: {:.2f}'
    .format(epoch, training_loss, valid_loss, time.time() - epoch_start))

However, on training this model, I found that the convergence speed was slower than I expected and also the training and validation loss shoots up suddenly (reinitializes to loss that was there at epoch 1) at a certain epoch (epoch 40) as seen in the logs below and then increases. I cannot ascertain the reason for this happening even though I did search the net for it. I did overfit the model on a single image to check if the model is able to learn images, and it gives great results after about 200 epochs on that single image.
Any other suggestions to improve training speed are also welcome.
The logs:
Starting epoch 35
Epoch: 35 , Batch: 20 Training Loss: 106.5381, Time: 0.68
Epoch: 35 , Batch: 40 Training Loss: 164.4004, Time: 0.67
Epoch: 35 , Batch: 60 Training Loss: 187.2683, Time: 0.67
Epoch: 35 , Batch: 80 Training Loss: 125.9881, Time: 0.68
Epoch: 35 , Batch: 100 Training Loss: 178.2198, Time: 0.67
Epoch: 35 , Batch: 120 Training Loss: 145.3067, Time: 0.68
Epoch: 35 , Batch: 140 Training Loss: 172.2734, Time: 0.67
Epoch: 35 , Batch: 160 Training Loss: 161.0511, Time: 0.68
Epoch: 35 , Batch: 180 Training Loss: 154.7270, Time: 0.67
Epoch: 35 , Batch: 200 Training Loss: 105.7089, Time: 0.67
Epoch: 35 , Batch: 220 Training Loss: 136.7364, Time: 0.67
Epoch: 35 , Batch: 240 Training Loss: 144.3123, Time: 0.67
Epoch: 35 , Batch: 1 Valid Loss: 150.3127, Time: 0.36
Epoch: 35 , Batch: 2 Valid Loss: 136.5788, Time: 0.36
Epoch: 35 , Batch: 3 Valid Loss: 145.5104, Time: 0.37
Epoch: 35 , Batch: 4 Valid Loss: 174.9318, Time: 0.35
Epoch: 35 , Batch: 5 Valid Loss: 169.0320, Time: 0.36
Epoch: 35 , Batch: 6 Valid Loss: 104.9420, Time: 0.36
Epoch: 35 , Batch: 7 Valid Loss: 139.7437, Time: 0.37
Epoch: 35 , Batch: 8 Valid Loss: 142.1164, Time: 0.37
Epoch: 35 , Batch: 9 Valid Loss: 162.5985, Time: 0.35
Epoch: 35 , Batch: 10 Valid Loss: 195.1854, Time: 0.35
Epoch: 35 , Batch: 11 Valid Loss: 134.6287, Time: 0.35
Epoch: 35 , Batch: 12 Valid Loss: 136.0625, Time: 0.35
Epoch: 35 , Batch: 13 Valid Loss: 210.8838, Time: 0.35
Epoch: 35 , Batch: 14 Valid Loss: 171.1289, Time: 0.35
Epoch: 35 , Batch: 15 Valid Loss: 158.6969, Time: 0.34
Epoch: 35 , Batch: 16 Valid Loss: 245.2010, Time: 0.36
Epoch: 35 , Batch: 17 Valid Loss: 163.9438, Time: 0.36
Epoch: 35 , Batch: 18 Valid Loss: 148.4983, Time: 0.36
Epoch: 35 , Batch: 19 Valid Loss: 153.7577, Time: 0.35
Epoch: 35 , Batch: 20 Valid Loss: 180.0079, Time: 0.34
Epoch: 35 , Batch: 21 Valid Loss: 123.2093, Time: 0.36
Epoch: 35 , Batch: 22 Valid Loss: 125.0577, Time: 0.36
Epoch: 35 , Batch: 23 Valid Loss: 167.9037, Time: 0.35
Epoch: 35 , Batch: 24 Valid Loss: 142.5038, Time: 0.35
Epoch: 35 , Batch: 25 Valid Loss: 204.0228, Time: 0.36
Epoch: 35 , Batch: 26 Valid Loss: 138.4986, Time: 0.31
Epoch: 35 , Batch: 27 Valid Loss: 245.4016, Time: 0.35
Epoch: 35 , Batch: 28 Valid Loss: 147.9031, Time: 0.35
Epoch: 35 , Batch: 29 Valid Loss: 86.9302, Time: 0.35
Epoch: 35 , Batch: 30 Valid Loss: 144.6022, Time: 0.35
Epoch: 35 , Batch: 31 Valid Loss: 171.0132, Time: 0.36
Epoch: 35 , Batch: 32 Valid Loss: 171.6215, Time: 0.26

Epoch: 35 , Training Loss: 158.2567, Validation Loss: 159.1384, Time: 419.94

Starting epoch 36
Epoch: 36 , Batch: 20 Training Loss: 105.9032, Time: 0.68
Epoch: 36 , Batch: 40 Training Loss: 166.2677, Time: 0.67
Epoch: 36 , Batch: 60 Training Loss: 187.5138, Time: 0.68
Epoch: 36 , Batch: 80 Training Loss: 126.4907, Time: 0.67
Epoch: 36 , Batch: 100 Training Loss: 177.6801, Time: 0.67
Epoch: 36 , Batch: 120 Training Loss: 145.7679, Time: 0.67
Epoch: 36 , Batch: 140 Training Loss: 172.1092, Time: 0.67
Epoch: 36 , Batch: 160 Training Loss: 161.7235, Time: 0.67
Epoch: 36 , Batch: 180 Training Loss: 154.2764, Time: 0.68
Epoch: 36 , Batch: 200 Training Loss: 105.0589, Time: 0.67
Epoch: 36 , Batch: 220 Training Loss: 137.1901, Time: 0.67
Epoch: 36 , Batch: 240 Training Loss: 143.7092, Time: 0.68
Epoch: 36 , Batch: 1 Valid Loss: 151.7165, Time: 0.36
Epoch: 36 , Batch: 2 Valid Loss: 137.6912, Time: 0.35
Epoch: 36 , Batch: 3 Valid Loss: 148.1911, Time: 0.36
Epoch: 36 , Batch: 4 Valid Loss: 178.2061, Time: 0.36
Epoch: 36 , Batch: 5 Valid Loss: 171.2887, Time: 0.35
Epoch: 36 , Batch: 6 Valid Loss: 105.7988, Time: 0.36
Epoch: 36 , Batch: 7 Valid Loss: 139.2858, Time: 0.35
Epoch: 36 , Batch: 8 Valid Loss: 143.3132, Time: 0.36
Epoch: 36 , Batch: 9 Valid Loss: 164.0428, Time: 0.35
Epoch: 36 , Batch: 10 Valid Loss: 197.8514, Time: 0.35
Epoch: 36 , Batch: 11 Valid Loss: 135.2920, Time: 0.36
Epoch: 36 , Batch: 12 Valid Loss: 136.8798, Time: 0.37
Epoch: 36 , Batch: 13 Valid Loss: 213.7531, Time: 0.36
Epoch: 36 , Batch: 14 Valid Loss: 173.2047, Time: 0.34
Epoch: 36 , Batch: 15 Valid Loss: 159.9836, Time: 0.36
Epoch: 36 , Batch: 16 Valid Loss: 250.1139, Time: 0.36
Epoch: 36 , Batch: 17 Valid Loss: 166.5989, Time: 0.36
Epoch: 36 , Batch: 18 Valid Loss: 149.8430, Time: 0.35
Epoch: 36 , Batch: 19 Valid Loss: 156.4111, Time: 0.32
Epoch: 36 , Batch: 20 Valid Loss: 182.4751, Time: 0.36
Epoch: 36 , Batch: 21 Valid Loss: 125.2163, Time: 0.36
Epoch: 36 , Batch: 22 Valid Loss: 127.7892, Time: 0.36
Epoch: 36 , Batch: 23 Valid Loss: 171.6647, Time: 0.35
Epoch: 36 , Batch: 24 Valid Loss: 145.8683, Time: 0.36
Epoch: 36 , Batch: 25 Valid Loss: 205.0257, Time: 0.35
Epoch: 36 , Batch: 26 Valid Loss: 141.0681, Time: 0.32
Epoch: 36 , Batch: 27 Valid Loss: 248.7428, Time: 0.36
Epoch: 36 , Batch: 28 Valid Loss: 148.8966, Time: 0.36
Epoch: 36 , Batch: 29 Valid Loss: 88.4646, Time: 0.36
Epoch: 36 , Batch: 30 Valid Loss: 143.6590, Time: 0.35
Epoch: 36 , Batch: 31 Valid Loss: 173.3025, Time: 0.35
Epoch: 36 , Batch: 32 Valid Loss: 174.2860, Time: 0.24

Epoch: 36 , Training Loss: 158.3800, Validation Loss: 161.1226, Time: 419.45

Starting epoch 37
Epoch: 37 , Batch: 20 Training Loss: 105.6317, Time: 0.68
Epoch: 37 , Batch: 40 Training Loss: 165.0392, Time: 0.67
Epoch: 37 , Batch: 60 Training Loss: 188.5125, Time: 0.68
Epoch: 37 , Batch: 80 Training Loss: 126.0623, Time: 0.67
Epoch: 37 , Batch: 100 Training Loss: 177.5551, Time: 0.67
Epoch: 37 , Batch: 120 Training Loss: 145.3477, Time: 0.67
Epoch: 37 , Batch: 140 Training Loss: 175.2243, Time: 0.68
Epoch: 37 , Batch: 160 Training Loss: 160.9049, Time: 0.68
Epoch: 37 , Batch: 180 Training Loss: 153.3031, Time: 0.68
Epoch: 37 , Batch: 200 Training Loss: 104.0302, Time: 0.67
Epoch: 37 , Batch: 220 Training Loss: 135.7108, Time: 0.67
Epoch: 37 , Batch: 240 Training Loss: 142.8567, Time: 0.67
Epoch: 37 , Batch: 1 Valid Loss: 150.1673, Time: 0.36
Epoch: 37 , Batch: 2 Valid Loss: 136.7532, Time: 0.36
Epoch: 37 , Batch: 3 Valid Loss: 144.9570, Time: 0.36
Epoch: 37 , Batch: 4 Valid Loss: 173.6571, Time: 0.32
Epoch: 37 , Batch: 5 Valid Loss: 166.9340, Time: 0.35
Epoch: 37 , Batch: 6 Valid Loss: 104.1471, Time: 0.35
Epoch: 37 , Batch: 7 Valid Loss: 138.5468, Time: 0.36
Epoch: 37 , Batch: 8 Valid Loss: 141.9434, Time: 0.35
Epoch: 37 , Batch: 9 Valid Loss: 163.1604, Time: 0.35
Epoch: 37 , Batch: 10 Valid Loss: 195.1403, Time: 0.35
Epoch: 37 , Batch: 11 Valid Loss: 134.0441, Time: 0.36
Epoch: 37 , Batch: 12 Valid Loss: 135.6687, Time: 0.36
Epoch: 37 , Batch: 13 Valid Loss: 210.3593, Time: 0.36
Epoch: 37 , Batch: 14 Valid Loss: 170.5062, Time: 0.35
Epoch: 37 , Batch: 15 Valid Loss: 157.5619, Time: 0.35
Epoch: 37 , Batch: 16 Valid Loss: 243.9199, Time: 0.35
Epoch: 37 , Batch: 17 Valid Loss: 163.0798, Time: 0.36
Epoch: 37 , Batch: 18 Valid Loss: 147.0687, Time: 0.36
Epoch: 37 , Batch: 19 Valid Loss: 152.7307, Time: 0.36
Epoch: 37 , Batch: 20 Valid Loss: 179.4877, Time: 0.34
Epoch: 37 , Batch: 21 Valid Loss: 122.9611, Time: 0.35
Epoch: 37 , Batch: 22 Valid Loss: 124.0443, Time: 0.35
Epoch: 37 , Batch: 23 Valid Loss: 165.3088, Time: 0.36
Epoch: 37 , Batch: 24 Valid Loss: 142.3539, Time: 0.35
Epoch: 37 , Batch: 25 Valid Loss: 202.2605, Time: 0.35
Epoch: 37 , Batch: 26 Valid Loss: 137.7361, Time: 0.35
Epoch: 37 , Batch: 27 Valid Loss: 242.8415, Time: 0.35
Epoch: 37 , Batch: 28 Valid Loss: 148.3895, Time: 0.35
Epoch: 37 , Batch: 29 Valid Loss: 86.0574, Time: 0.36
Epoch: 37 , Batch: 30 Valid Loss: 144.1157, Time: 0.35
Epoch: 37 , Batch: 31 Valid Loss: 170.8322, Time: 0.35
Epoch: 37 , Batch: 32 Valid Loss: 171.6427, Time: 0.25

Epoch: 37 , Training Loss: 158.1220, Validation Loss: 158.3868, Time: 417.97

Starting epoch 38
Epoch: 38 , Batch: 20 Training Loss: 105.7252, Time: 0.68
Epoch: 38 , Batch: 40 Training Loss: 163.4334, Time: 0.67
Epoch: 38 , Batch: 60 Training Loss: 185.7129, Time: 0.68
Epoch: 38 , Batch: 80 Training Loss: 125.1632, Time: 0.68
Epoch: 38 , Batch: 100 Training Loss: 178.0677, Time: 0.67
Epoch: 38 , Batch: 120 Training Loss: 145.0643, Time: 0.67
Epoch: 38 , Batch: 140 Training Loss: 172.1782, Time: 0.67
Epoch: 38 , Batch: 160 Training Loss: 160.3347, Time: 0.68
Epoch: 38 , Batch: 180 Training Loss: 151.9679, Time: 0.68
Epoch: 38 , Batch: 200 Training Loss: 103.2668, Time: 0.67
Epoch: 38 , Batch: 220 Training Loss: 134.6251, Time: 0.67
Epoch: 38 , Batch: 240 Training Loss: 141.8546, Time: 0.67
Epoch: 38 , Batch: 1 Valid Loss: 150.4089, Time: 0.35
Epoch: 38 , Batch: 2 Valid Loss: 136.9565, Time: 0.36
Epoch: 38 , Batch: 3 Valid Loss: 144.5437, Time: 0.35
Epoch: 38 , Batch: 4 Valid Loss: 172.9728, Time: 0.36
Epoch: 38 , Batch: 5 Valid Loss: 166.5804, Time: 0.35
Epoch: 38 , Batch: 6 Valid Loss: 103.4765, Time: 0.35
Epoch: 38 , Batch: 7 Valid Loss: 138.3381, Time: 0.35
Epoch: 38 , Batch: 8 Valid Loss: 141.9496, Time: 0.35
Epoch: 38 , Batch: 9 Valid Loss: 163.2548, Time: 0.35
Epoch: 38 , Batch: 10 Valid Loss: 194.6338, Time: 0.36
Epoch: 38 , Batch: 11 Valid Loss: 134.2889, Time: 0.35
Epoch: 38 , Batch: 12 Valid Loss: 136.0695, Time: 0.35
Epoch: 38 , Batch: 13 Valid Loss: 209.8143, Time: 0.36
Epoch: 38 , Batch: 14 Valid Loss: 170.3180, Time: 0.35
Epoch: 38 , Batch: 15 Valid Loss: 157.3526, Time: 0.35
Epoch: 38 , Batch: 16 Valid Loss: 243.4948, Time: 0.36
Epoch: 38 , Batch: 17 Valid Loss: 162.8065, Time: 0.36
Epoch: 38 , Batch: 18 Valid Loss: 146.0056, Time: 0.36
Epoch: 38 , Batch: 19 Valid Loss: 152.3638, Time: 0.35
Epoch: 38 , Batch: 20 Valid Loss: 179.5833, Time: 0.34
Epoch: 38 , Batch: 21 Valid Loss: 122.8715, Time: 0.36
Epoch: 38 , Batch: 22 Valid Loss: 123.2680, Time: 0.36
Epoch: 38 , Batch: 23 Valid Loss: 164.3962, Time: 0.36
Epoch: 38 , Batch: 24 Valid Loss: 142.2115, Time: 0.33
Epoch: 38 , Batch: 25 Valid Loss: 202.0091, Time: 0.35
Epoch: 38 , Batch: 26 Valid Loss: 137.4782, Time: 0.36
Epoch: 38 , Batch: 27 Valid Loss: 241.4298, Time: 0.35
Epoch: 38 , Batch: 28 Valid Loss: 148.7050, Time: 0.35
Epoch: 38 , Batch: 29 Valid Loss: 86.0582, Time: 0.34
Epoch: 38 , Batch: 30 Valid Loss: 144.0733, Time: 0.33
Epoch: 38 , Batch: 31 Valid Loss: 170.6064, Time: 0.36
Epoch: 38 , Batch: 32 Valid Loss: 171.6329, Time: 0.25

Epoch: 38 , Training Loss: 157.1817, Validation Loss: 158.1235, Time: 418.16

Starting epoch 39
Epoch: 39 , Batch: 20 Training Loss: 105.1230, Time: 0.68
Epoch: 39 , Batch: 40 Training Loss: 165.4860, Time: 0.67
Epoch: 39 , Batch: 60 Training Loss: 185.0349, Time: 0.67
Epoch: 39 , Batch: 80 Training Loss: 124.9884, Time: 0.67
Epoch: 39 , Batch: 100 Training Loss: 176.9023, Time: 0.67
Epoch: 39 , Batch: 120 Training Loss: 144.6198, Time: 0.67
Epoch: 39 , Batch: 140 Training Loss: 170.2020, Time: 0.67
Epoch: 39 , Batch: 160 Training Loss: 158.3938, Time: 0.67
Epoch: 39 , Batch: 180 Training Loss: 151.8574, Time: 0.68
Epoch: 39 , Batch: 200 Training Loss: 103.9747, Time: 0.67
Epoch: 39 , Batch: 220 Training Loss: 134.2870, Time: 0.67
Epoch: 39 , Batch: 240 Training Loss: 141.7696, Time: 0.68
Epoch: 39 , Batch: 1 Valid Loss: 151.1508, Time: 0.34
Epoch: 39 , Batch: 2 Valid Loss: 136.8730, Time: 0.35
Epoch: 39 , Batch: 3 Valid Loss: 145.3472, Time: 0.36
Epoch: 39 , Batch: 4 Valid Loss: 173.3746, Time: 0.36
Epoch: 39 , Batch: 5 Valid Loss: 167.3553, Time: 0.35
Epoch: 39 , Batch: 6 Valid Loss: 103.7990, Time: 0.36
Epoch: 39 , Batch: 7 Valid Loss: 139.3748, Time: 0.36
Epoch: 39 , Batch: 8 Valid Loss: 142.6575, Time: 0.35
Epoch: 39 , Batch: 9 Valid Loss: 163.5934, Time: 0.35
Epoch: 39 , Batch: 10 Valid Loss: 194.6525, Time: 0.35
Epoch: 39 , Batch: 11 Valid Loss: 135.1157, Time: 0.36
Epoch: 39 , Batch: 12 Valid Loss: 136.4958, Time: 0.36
Epoch: 39 , Batch: 13 Valid Loss: 209.0710, Time: 0.36
Epoch: 39 , Batch: 14 Valid Loss: 170.3045, Time: 0.36
Epoch: 39 , Batch: 15 Valid Loss: 157.5657, Time: 0.35
Epoch: 39 , Batch: 16 Valid Loss: 243.3302, Time: 0.36
Epoch: 39 , Batch: 17 Valid Loss: 163.2397, Time: 0.37
Epoch: 39 , Batch: 18 Valid Loss: 146.7547, Time: 0.35
Epoch: 39 , Batch: 19 Valid Loss: 153.3413, Time: 0.36
Epoch: 39 , Batch: 20 Valid Loss: 179.2610, Time: 0.36
Epoch: 39 , Batch: 21 Valid Loss: 123.0379, Time: 0.35
Epoch: 39 , Batch: 22 Valid Loss: 123.5644, Time: 0.36
Epoch: 39 , Batch: 23 Valid Loss: 165.4950, Time: 0.35
Epoch: 39 , Batch: 24 Valid Loss: 142.4183, Time: 0.35
Epoch: 39 , Batch: 25 Valid Loss: 203.7861, Time: 0.35
Epoch: 39 , Batch: 26 Valid Loss: 137.3413, Time: 0.35
Epoch: 39 , Batch: 27 Valid Loss: 241.8771, Time: 0.36
Epoch: 39 , Batch: 28 Valid Loss: 148.8467, Time: 0.36
Epoch: 39 , Batch: 29 Valid Loss: 85.9060, Time: 0.34
Epoch: 39 , Batch: 30 Valid Loss: 145.1030, Time: 0.31
Epoch: 39 , Batch: 31 Valid Loss: 171.1640, Time: 0.36
Epoch: 39 , Batch: 32 Valid Loss: 172.1705, Time: 0.26

Epoch: 39 , Training Loss: 156.5736, Validation Loss: 158.5428, Time: 417.88

Starting epoch 40
Epoch: 40 , Batch: 20 Training Loss: 104.7642, Time: 0.68
Epoch: 40 , Batch: 40 Training Loss: 199.1840, Time: 0.64
Epoch: 40 , Batch: 60 Training Loss: 203.7560, Time: 0.64
Epoch: 40 , Batch: 80 Training Loss: 164.9527, Time: 0.64
Epoch: 40 , Batch: 100 Training Loss: 186.1097, Time: 0.64
Epoch: 40 , Batch: 120 Training Loss: 155.4772, Time: 0.64
Epoch: 40 , Batch: 140 Training Loss: 192.1701, Time: 0.65
Epoch: 40 , Batch: 160 Training Loss: 193.4585, Time: 0.65
Epoch: 40 , Batch: 180 Training Loss: 171.5463, Time: 0.64
Epoch: 40 , Batch: 200 Training Loss: 117.4412, Time: 0.64
Epoch: 40 , Batch: 220 Training Loss: 152.0421, Time: 0.64
Epoch: 40 , Batch: 240 Training Loss: 143.3178, Time: 0.64
Epoch: 40 , Batch: 1 Valid Loss: 156.7238, Time: 0.35
Epoch: 40 , Batch: 2 Valid Loss: 147.8145, Time: 0.36
Epoch: 40 , Batch: 3 Valid Loss: 159.7137, Time: 0.35
Epoch: 40 , Batch: 4 Valid Loss: 191.3246, Time: 0.32
Epoch: 40 , Batch: 5 Valid Loss: 178.6363, Time: 0.36
Epoch: 40 , Batch: 6 Valid Loss: 109.2589, Time: 0.35
Epoch: 40 , Batch: 7 Valid Loss: 144.4912, Time: 0.36
Epoch: 40 , Batch: 8 Valid Loss: 150.4129, Time: 0.36
Epoch: 40 , Batch: 9 Valid Loss: 171.8096, Time: 0.35
Epoch: 40 , Batch: 10 Valid Loss: 210.0710, Time: 0.36
Epoch: 40 , Batch: 11 Valid Loss: 142.5819, Time: 0.35
Epoch: 40 , Batch: 12 Valid Loss: 140.7213, Time: 0.36
Epoch: 40 , Batch: 13 Valid Loss: 221.1351, Time: 0.36
Epoch: 40 , Batch: 14 Valid Loss: 182.1034, Time: 0.36
Epoch: 40 , Batch: 15 Valid Loss: 172.3420, Time: 0.35
Epoch: 40 , Batch: 16 Valid Loss: 264.6909, Time: 0.35
Epoch: 40 , Batch: 17 Valid Loss: 177.7539, Time: 0.36
Epoch: 40 , Batch: 18 Valid Loss: 159.2539, Time: 0.35
Epoch: 40 , Batch: 19 Valid Loss: 167.3184, Time: 0.36
Epoch: 40 , Batch: 20 Valid Loss: 188.2323, Time: 0.35
Epoch: 40 , Batch: 21 Valid Loss: 134.6375, Time: 0.36
Epoch: 40 , Batch: 22 Valid Loss: 137.9939, Time: 0.36
Epoch: 40 , Batch: 23 Valid Loss: 188.5730, Time: 0.36
Epoch: 40 , Batch: 24 Valid Loss: 157.5786, Time: 0.36
Epoch: 40 , Batch: 25 Valid Loss: 215.2611, Time: 0.35
Epoch: 40 , Batch: 26 Valid Loss: 146.4763, Time: 0.36
Epoch: 40 , Batch: 27 Valid Loss: 257.7494, Time: 0.36
Epoch: 40 , Batch: 28 Valid Loss: 156.5165, Time: 0.35
Epoch: 40 , Batch: 29 Valid Loss: 94.2914, Time: 0.34
Epoch: 40 , Batch: 30 Valid Loss: 146.4372, Time: 0.35
Epoch: 40 , Batch: 31 Valid Loss: 180.1848, Time: 0.36
Epoch: 40 , Batch: 32 Valid Loss: 178.5307, Time: 0.24

Epoch: 40 , Training Loss: 176.2027, Validation Loss: 169.7069, Time: 414.79

Starting epoch 41
Epoch: 41 , Batch: 20 Training Loss: 128.6025, Time: 0.64
Epoch: 41 , Batch: 40 Training Loss: 199.1840, Time: 0.64
Epoch: 41 , Batch: 60 Training Loss: 203.7560, Time: 0.64
Epoch: 41 , Batch: 80 Training Loss: 164.9526, Time: 0.64
Epoch: 41 , Batch: 100 Training Loss: 186.1097, Time: 0.64
Epoch: 41 , Batch: 120 Training Loss: 155.4771, Time: 0.65
Epoch: 41 , Batch: 140 Training Loss: 192.1701, Time: 0.65
Epoch: 41 , Batch: 160 Training Loss: 193.4585, Time: 0.64
Epoch: 41 , Batch: 180 Training Loss: 171.5462, Time: 0.64
Epoch: 41 , Batch: 200 Training Loss: 117.4412, Time: 0.64
Epoch: 41 , Batch: 220 Training Loss: 152.0420, Time: 0.65
Epoch: 41 , Batch: 240 Training Loss: 143.3178, Time: 0.65
Epoch: 41 , Batch: 1 Valid Loss: 156.7238, Time: 0.35
Epoch: 41 , Batch: 2 Valid Loss: 147.8145, Time: 0.35
Epoch: 41 , Batch: 3 Valid Loss: 159.7137, Time: 0.35
Epoch: 41 , Batch: 4 Valid Loss: 191.3246, Time: 0.36
Epoch: 41 , Batch: 5 Valid Loss: 178.6363, Time: 0.36
Epoch: 41 , Batch: 6 Valid Loss: 109.2589, Time: 0.35
Epoch: 41 , Batch: 7 Valid Loss: 144.4912, Time: 0.34
Epoch: 41 , Batch: 8 Valid Loss: 150.4129, Time: 0.36
Epoch: 41 , Batch: 9 Valid Loss: 171.8096, Time: 0.36
Epoch: 41 , Batch: 10 Valid Loss: 210.0710, Time: 0.36
Epoch: 41 , Batch: 11 Valid Loss: 142.5818, Time: 0.36
Epoch: 41 , Batch: 12 Valid Loss: 140.7213, Time: 0.36
Epoch: 41 , Batch: 13 Valid Loss: 221.1351, Time: 0.35
Epoch: 41 , Batch: 14 Valid Loss: 182.1034, Time: 0.36
Epoch: 41 , Batch: 15 Valid Loss: 172.3419, Time: 0.35
Epoch: 41 , Batch: 16 Valid Loss: 264.6909, Time: 0.36
Epoch: 41 , Batch: 17 Valid Loss: 177.7538, Time: 0.35
Epoch: 41 , Batch: 18 Valid Loss: 159.2538, Time: 0.36
Epoch: 41 , Batch: 19 Valid Loss: 167.3183, Time: 0.36
Epoch: 41 , Batch: 20 Valid Loss: 188.2322, Time: 0.35
Epoch: 41 , Batch: 21 Valid Loss: 134.6374, Time: 0.36
Epoch: 41 , Batch: 22 Valid Loss: 137.9939, Time: 0.36
Epoch: 41 , Batch: 23 Valid Loss: 188.5730, Time: 0.36
Epoch: 41 , Batch: 24 Valid Loss: 157.5785, Time: 0.36
Epoch: 41 , Batch: 25 Valid Loss: 215.2611, Time: 0.36
Epoch: 41 , Batch: 26 Valid Loss: 146.4763, Time: 0.36
Epoch: 41 , Batch: 27 Valid Loss: 257.7494, Time: 0.36
Epoch: 41 , Batch: 28 Valid Loss: 156.5165, Time: 0.35
Epoch: 41 , Batch: 29 Valid Loss: 94.2914, Time: 0.35
Epoch: 41 , Batch: 30 Valid Loss: 146.4371, Time: 0.35
Epoch: 41 , Batch: 31 Valid Loss: 180.1848, Time: 0.36
Epoch: 41 , Batch: 32 Valid Loss: 178.5307, Time: 0.26
Epoch: 41 , Training Loss: 178.2872, Validation Loss: 169.7068, Time: 414.65


title: Own training data set preparation for Deepmask

body:
For this deepmask training with own dataset, I like to know step by step infos for how to prepart.
(1)What labeling software to use?
(2)Do I need to label every item in the image?
(3)What are the things to take note in labeling to train deepmask and sharpmask?


title: using pandas.io.parser.TextFileReader to draw specific row from dataframe

body:
i want to load only 1 row from a large csv, but i want it to be a specific row.
so far i created an iterable object like so:
image_loader = pd.read_csv('mydata.csv', header=None, chunksize = 1)
image = next(image_loader)

that's great if i want ordered selection or randomized. but what if i want specific selection? i mean something like this (wishful thinking code):
specific_index = 134
image_loader = pd.read_csv('mydata.csv', header=None, chunksize = 1)
image = image_loader.get_idx(specific_index)

i'll describe my general need so maybe you know how to bypass it -
pytorch project, trying to create a torch.dataset object. must define getitem which takes index and returns sample.
shuffling the csv in advance is not an option because it's crucial that after iterating through the entire csv the order is changed. the torch,dataset takes care of that by randomizing the index, so my text.file.reader must be able to draw specific index on demand.


title: Separate data points into two classes with pytorch

body:
I have some data points on the (x,y) field that are supposed to be cotegorised into two classes denoted here with X and O like this
import numpy as np
import matplotlib.pyplot as plt
x1 = np.array([0.1,0.3,0.1,0.6,0.4,0.6,0.5,0.9,0.4,0.7])
x2 = np.array([0.1,0.4,0.5,0.9,0.2,0.3,0.6,0.2,0.4,0.6])
c=np.array([ 1,1,1,1,1,0,0,0,0,0 ])
plt.plot(x1[c==0], x2[c==0], 'bo')
plt.plot(x1[c==1], x2[c==1], 'rx')

Now I want to find a way so I can find the “best fitting curve” separating those like this

First I though maybe I try nearest neighbor method but I’ve been told it cannot apply here and that there’s a much simpler way to do it with an ANN maybe and using PyTorch but I can’t understand how.
Any ideas on what to use and/or how to do it and what should I check to estimate my accuracy?? There are too few points so what's the idea here??
Thank you everyone in advance!


title: Is it possible to input each time sequence separately to LSTM for prediction?

body:
I am working on graph , and I have different graphs each represent a specific time stamp. I want to use each graph time stamp separately as input to LSTM, as what I know, I should combine my inputs before  I go for LSTM, is there any way I can do this without combination of my inputs. I tried to do this but instead I am getting prediction for each graph, my aim is to get the final predictions base on multiple input graph. here is my example code, during training
for epoch in range(epochs):
    for id, data in enumerate(Datasets)... `Here Datasets contains  all graph separately`
     print(data) `i get all graphs`
     result_before_LSTM=Layer(data)
     LSTM_results=LSTM(result_before_LSTM)

Here I am getting output for each graph, but my aim is to get only the last output based on the series of the inputted graphs, Please I need your help


title: Is there any way to save pytorch models with their classes

body:
When i try to load my saved model, i need to import its class. For example:
from module import Net
torch.load('saved_model.pth')

Is there any way to avoid this importing?. For example saving the model with class or something else?


title: Error in Normalizing Network's outputs for loss calculations

body:
I have a working network called self.net, I want to normalize the outputs of that and then calculate the loss. Here what I do;
output_net1, output_net2 = self.net(input_var)
output_net1, output_net2 = normalizeTensors(output_net1, output_net2) # this line I added
loss_net1 = self.criterion(output_net1, target_disp)
loss_net2 = self.criterion(output_net2, target_disp)
loss = loss_net1 + loss_net2
loss.backward()

def normalizeTensors(inputs1,inputs2):
    tuples1 = ()
    for i in range(len(inputs1)):
        data = torch.zeros(inputs1[i].shape).cuda()
        for j in range(inputs1[i].shape[0]):
            data[j,:,:,:] = normalizeOutputs(inputs1[i][j,:,:,:]).clone().unsqueeze(0)
        tuples1 = tuples1 + (data,)
    tuples2 = ()
    for i in range(len(inputs2)):
        data = torch.zeros(inputs2[i].shape).cuda()
        for j in range(inputs2[i].shape[0]):
            data[j,:,:,:] = normalizeOutputs(inputs2[i][j,:,:,:]).clone().unsqueeze(0)
        tuples2 = tuples2 + (data,)
    return tuples1, tuples2

def normalizeOutputs(inpt):
    _min = torch.min(inpt)
    _max = torch.max(inpt)
    inpt -= _min
    inpt /= (_max - _min)
    inpt *= 255
    return inpt

If I call normalizeTensorsfunction, I got an error says;
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 4, 8]], which is output 0 of SliceBackward, is at version 6; expected version 3 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

I didn't understand why it gives such an error. Because, I didn't change anything but datas. I just normalized them. Shapes, lengths etc.  stay all same.


title: Pillow not loadng image from local system - No such file or directory: 'home/hp/DeepLearning/images/picasso.jpg'

body:
imsize = 512 if torch.cuda.is_available() else 128

loader = transforms.Compose([transforms.Resize(imsize),transforms.ToTensor()])

def image_loader(image_name):
    image = Image.open(image_name)
    image = loader(image).unsqueeze(0)
    return image.to(device, torch.float)
style_img = image_loader("home/hp/DeepLearning/images/picasso.jpg")
content_img = image_loader("home/hp/DeepLearning/images/picasso.jpg")
assert style_img.size() == content_img.size(),\"we need to import style and content images of the same size"

When I am trying to load image from my local system,it is showing me error. I have tried os.path.abspath()


title: How to run a single round of validation in PytorchLightning?

body:
I have trained a PytorchLightning model via Trainer.fit and saved the resulting model. Now I would like to load the model and run a single round of validation, without any training. What is the best way to do that? The API seems to only provide Trainer.fit and Trainer.test. I could override MyLightningModule.test_step to MyLightningModule.validation_step and run Trainer.test, but that feels like a hack. Alternatively, I could run Trainer.fit with limit_train_batches=1, max_epochs=1, but that also feels like a hack. What would be the most idiomatic way to do that?
Ideally, I would like to obtain a nice and clean log, without polluting it with unnecessary entries (which is the case with the Trainer.fit workaround).


title: Small mAP difference between pytorch and tf trained model

body: I’m reproducing a model, which was originally implemented on Tensorflow, on PyTorch. The original tf and paper reported mAP tested on PASCAL VOC2012 test is 78.4%. My implementation is 77.9%. Is this 0.5% mAP difference a lot? I followed the same optimization and hyperparameter settings and tested the model right after training. Could this be caused by model random initialization and some TF vs PyTorch platform differences?


title: Feeding different sequences (each of different length and normalization) to LSTM

body:
I am currently feeding a 2 dimensional dataframe to an lstm read in from data.csv. The dataframe contains multiple time series, that are normalized with a minmaxscaler.
However, I would like to train from multiple files. Each of different length, with a different normalization scale, and different files do not sequentially follow each other.
What is the best way to go about this? Do I create a variable length input batch for each file? Or is there some easier way to deal with this?
import torch.nn as nn
from torch.autograd import Variable

f = 'data.csv'
df = pd.read_csv(f, sep=",", index_col=0)
y = df.iloc[:, 10:12]

y = y.astype(int)
X = df.iloc[:, np.r_[0:6]]

X_old = X
y_old = y

from sklearn.preprocessing import StandardScaler, MinMaxScaler
mm = MinMaxScaler()
ss = StandardScaler()

X_ss = ss.fit_transform(X)
y_mm = mm.fit_transform(y)

y_mm = y.to_numpy() #added

#first 200 for training
X_train = X_ss[:1800, :]
X_test = X_ss[1800:, :]

# y_train = y_mm[:1800, :]
# y_test = y_mm[1800:, :]
y_train = y_mm[:1800, :]
y_test = y_mm[1800:, :]

print("Training Shape", X_train.shape, y_train.shape)
print("Testing Shape", X_test.shape, y_test.shape)

X_train_tensors = Variable(torch.Tensor(X_train))
X_test_tensors = Variable(torch.Tensor(X_test))

y_train_tensors = Variable(torch.Tensor(y_train))
y_test_tensors = Variable(torch.Tensor(y_test))

a = X_train_tensors.shape[0]

#reshaping to rows, timestamps, features
X_train_tensors_final = torch.reshape(X_train_tensors,   (X_train_tensors.shape[0], 1, X_train_tensors.shape[1]))
X_test_tensors_final = torch.reshape(X_test_tensors,  (X_test_tensors.shape[0], 1, X_test_tensors.shape[1]))

print("Training Shape", X_train_tensors_final.shape, y_train_tensors.shape)
print("Testing Shape", X_test_tensors_final.shape, y_test_tensors.shape)

class LSTM1(nn.Module):
    def __init__(self, num_classes, input_size, hidden_size, num_layers, seq_length):
        super(LSTM1, self).__init__()
        self.num_classes = num_classes  # number of classes
        self.num_layers = num_layers  # number of layers
        self.input_size = input_size  # input size
        self.hidden_size = hidden_size  # hidden state
        self.seq_length = seq_length  # sequence length

        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
                            num_layers=num_layers, batch_first=True)  # lstm
        self.fc_1 = nn.Linear(hidden_size, 128)  # fully connected 1
        self.fc = nn.Linear(128, num_classes)  # fully connected last layer

        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
        h_0 = Variable(torch.zeros(
            self.num_layers, x.size(0), self.hidden_size))  # hidden state

        c_0 = Variable(torch.zeros(
            self.num_layers, x.size(0), self.hidden_size))  # internal state

        # Propagate input through LSTM

        output, (hn, cn) = self.lstm(x, (h_0, c_0))  # lstm with input, hidden, and internal state

        hn = hn.view(-1, self.hidden_size)  # reshaping the data for Dense layer next

        out = self.relu(hn)

        out = self.fc_1(out)  # first Dense

        out = self.relu(out)  # relu

        out = self.fc(out)  # Final Output
        out = self.sigmoid(out)

        return out

lstm1 = LSTM1(num_classes, input_size, hidden_size, num_layers, X_train_tensors_final.shape[1]) #our lstm class

criterion = torch.nn.BCELoss() #MSELoss()    # mean-squared error for regression
optimizer = torch.optim.Adam(lstm1.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    outputs = lstm1.forward(X_train_tensors_final)  # forward pass
    optimizer.zero_grad()  # caluclate the gradient, manually setting to 0

    # obtain the loss function
    loss = criterion(outputs, y_train_tensors)

    loss.backward()  # calculates the loss of the loss function

    optimizer.step()  # improve from loss, i.e backprop
    if epoch % 100 == 0:
        print("Epoch: %d, loss: %1.5f" % (epoch, loss.item()))


title: How to implement t.utils.data.Dataloader properly

body:
I am working on a project that requires me to fetch data, label from a csv file before going for the training in PyTorch. The structure of the csv file is:
image, animal, extinct
blabla.png, 0, 1
..............
and so on..
I made a class to fetch the dataset like:
from torch.utils.data import Dataset

class CallDataset(Dataset):
    # TODO implement the Dataset class according to the description

    def __init__(self, data, mode):
        #self.img, self.lbl = data
        self.data = data
        self.mode = mode
        self._transform = transforms.Compose([
            transforms.ToPILImage(),
            transforms.ToTensor(),
            transforms.Normalize(mean=train_mean, std=train_std)
        ])
        # self.tab = pd.read_csv(self.data, sep=';')
        self.image = data.iloc[0:, 0]
        self.label = data.iloc[0:, 1:]
        #self.image = self.img
        #self.label = self.lbl

    def __len__(self):
        return len(self.image)

    def __getitem__(self, index):
        if self.mode == 'train':
            x = imread(self.image[index])
            sample = gray2rgb(x)
            transformed_img = self._transform(sample)

            transformed_label = self.label.to_numpy()
            transformed_label = transformed_label[index]
            transformed_label = torch.from_numpy(transformed_label)

            return (transformed_img, transformed_label)

        if self.mode == 'val':
            x = imread(self.image[index])
            sample = gray2rgb(x)
            transformed_img = self._transform(sample)

            transformed_label = self.label.to_numpy()
            transformed_label = transformed_label[index]
            transformed_label = torch.from_numpy(transformed_label)

            return (transformed_img, transformed_label)

So the idea is to return a tuple of 2 torch tensors. The first one is the RGB transformed image tensor of shape (1, 3, height, width) and the second one is a torch tensor consisting the labels (whether it is an animal and whether it is extinct). Its shape if (1, 2).
The csv file and the corresponding image folder contain 1000 images and 1000 respective sets of labels.
For training, I am trying to call them like:
   def train_epoch(self):

        self._model = self._model.train()
        l = 0
        c = 0
        for x, y_true in self._train_dl:
            c +=1
            if self._cuda:

                x = x.to('cuda')
                y_true = y_true.to('cuda')
            l += self.train_step(x, y_true)
        avg_loss = l/c
        return avg_loss

The self.train_dl is defined like this:
def train_dataset():
    train_data = CallDataset(train, 'train')
    return train_data
train_dl = t.utils.data.DataLoader(train_dataset(), batch_size=200, shuffle=True)

So the idea is to fetch at each iteration the image and corresponding label, sending them to cuda, train them (train_step is the function where I called the model, ran forward, got loss, ran backward etc). The initial loss l = 0 is where I defined l. c basically stores the number of interactions that I need to find the average loss (l,c does not bother me at this point).
My question is if I am using the Dataloader correctly. Also, please let me know the best way to use them with explanation.


title: Prepare custom dataset of words in language pytorch

body:
I'm trying to train a binary neural network that will get a word and decide if this word is in language L or not.
The words are bitstrings and the language is defined by a pattern.
my dataset look like this
001101000010010, False
111001001000000, False
111011101001100, False
011111000101101, False
10000110000110001100, True
011100100001010, False
....

but I can create the dataset in other way if needed (I know what is the "real" function that defines the language.
I thought to load the dataset like this but it's not working:
class myDataset(T.utils.data.Dataset):
# WORD  T/f
# 010000101010000  FALSE

def __init__(self, src_file, m_rows=None):
    all_xy = np.loadtxt(src_file, max_rows=m_rows,
                        usecols=[0, 1], delimiter=",", skiprows=0, dtype=np.int)

    tmp_x = all_xy[:, [0]]
    tmp_y = all_xy[:, 1].reshape(-1, 1)  # 2-D required

    self.x_data = T.tensor(tmp_x).to(device)
    self.y_data = T.tensor(tmp_y).to(device)
    self.x_data = tmp_x
    self.y_data = tmp_y

but I get the error

OverflowError: Python int too large to convert to C long

since the number is too big,
when treating the bitstring as text
tmp_x = np.loadtxt(src_file, max_rows=m_rows, usecols=[0], delimiter=",", skiprows=0, dtype=np.str)

I get the error

TypeError: can't convert np.ndarray of type numpy.str_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

I'm new to Pytorch programming and I'm not sure what is the right way to write this dataset and load it.


title: how to get specific elements in tensor (pytorch)

body:
I have a tensor, like:
input:
a = torch.rand(2,3,5)

output:
tensor([[[0.2764, 0.2209, 0.8007, 0.1246, 0.4302],
     [0.9716, 0.8063, 0.3904, 0.7574, 0.2392],
     [0.3366, 0.4209, 0.0527, 0.1328, 0.0441]],

    [[0.8166, 0.6519, 0.5450, 0.3072, 0.2716],
     [0.0583, 0.0613, 0.8984, 0.0110, 0.4744],
     [0.2269, 0.2693, 0.6447, 0.6078, 0.6148]]])

how can I get :
tensor([[[0.2764, 0.2209, 0.8007, 0.1246, 0.4302],
 [0.9716, 0.8063, 0.3904, 0.7574, 0.2392]],

[[0.8166, 0.6519, 0.5450, 0.3072, 0.2716],
 [0.2269, 0.2693, 0.6447, 0.6078, 0.6148]]])

I have tried gather(), but it does not work.


title: How to obtain the values from each row at given position in pytorch?

body:
How to obtain the values from a 2-d torch array based on 1-d array containing the positions at each row:
for example:
a = torch.randn((5,5))
>>> a
tensor([[ 0.0740, -0.3129,  0.7814, -0.0519,  1.3503],
        [ 1.1985,  0.2098, -0.0326,  0.3922,  0.5037],
        [-1.4334,  1.4047, -0.6607, -1.8024, -0.0088],
        [ 1.2116,  0.5928,  1.4041,  1.0494, -0.1146],
        [ 0.4173,  1.0482,  0.5244, -2.1767,  0.5264]])

b = torch.randint(0,5, (5,))
>>> b
tensor([1, 0, 1, 3, 2])

I wanted to get the elements of tensor a at the position given by tensor b
For example:
desired output:
tensor([-0.3129,
        1.1985,
        1.4047,
        1.0494,
        0.5244])

Here, each element in a given position by tensor b is chosen row-wise.
I have tried:
for index in range(b.size(-1)):
    val = torch.cat((val,a[index,b[index]].view(1,-1)), dim=0) if val is not None else a[index,b[index]].view(1,-1)

>>> val
tensor([[-0.3129],
        [ 1.1985],
        [ 1.4047],
        [ 1.0494],
        [ 0.5244]])

However, is there a tensor indexing way to do it?
I tried couple of solutions using tensor indexing, but none of them worked.


title: How to estimate the device computation capacity?

body:
How could I estimate two devices' computing capacity for a specific task? For example, I want to estimate how many times faster to run a task on GPU than CPU.
Right now I could think of , comparing TFlops, Bandwidth. Is there any materials concerning about this?