title: PyTorch: `torch.chunk` source code Github location

body:
I can’t seem to find the source code for torch.chunk in PyTorch’s Github page or in the documentation.
Anyone knows where this is in PyTorch’s Github page?


title: Pytorch: How to train a network with two loss functions?

body:
I want to pretrain a network with reconstruction loss first, then finetune it by crossentropy loss. But it seems that I have to define two network in this two stage. How to achieve it?
class Net():
    def __init__(self,pretrain):
        self.pretrain = pretrain
    def encoder(self,x):
        # do something here
        return x
    def decoder(self,x):
        # do something here
        return x

    def forward(self):
        e_x = self.encoder(x)
        if self.pretrain:
            return decoder(e_x)
        else:
            return e_x

def train(x,y):
    pretrain = True
    if pretrain:
        network = Net(pretrain=True)
        output = network(x)
        loss = MSE(x,output)
     else:
        network = Net(pretrain=False)
        output = network(x)
        loss = crossentropy(output,y)
    loss.backward()


title: How to interpret gradient and parameter histograms in tensorboardX?

body:
I ran AlexNet on an ImageNet database for 3 epochs and got this while plotting grads/params. -
Params in epoch 2

Epoch 2 params distribution

Grads in epoch 2

Epoch 2 grads distribution

Parameters over epochs

Distribution of params over epochs

# Code for Epoch wise plot
# Add this to the training loop in the train_one_epoch function
for name, param in model.named_parameters():
           writer.add_histogram(
               tag=f"epoch_{epoch}_params/{name}", values=param, global_step=batch
           )
           writer.add_histogram(
               tag=f"epoch_{epoch}_grads/{name}", values=param.grad, global_step=batch
           )
# For run parameters -
# Add this to end of epoch loop
for layer, (name, param) in enumerate(model.named_parameters()):
         writer.add_histogram(
             tag=f"run_params/{name}", values=param, global_step=epoch
         )

I have the following questions in mind -

What do these graphs mean and how to read them?
Can we extrapolate if model has learnt anything using these graphs in say, epoch2 or over the whole run?. If so how?
Can we debug vanishing/exploding gradients using these graphs? (It would be awesome if you can share some examples of this.)
Finally, is breaking my head over this even useful?

Thanks in advance!


title: Can't install PyTorch

body:
Encountered the next problem while trying to install pytorch
I've got 64-bit version of python, pip and setuptools were updated
Beginning of the error message
Error message


title: I can not import torch

body:
when I tried the "import torch" ı am getting this error .How can ı fix it?
I left error as a photo
enter image description here


title: CNN in Pytorch: labels for a custom dataset

body:
I have a very basic question that would be really helpful for getting me started on a deep learning project I'm running. I'm planning to apply a 3D convolutional network using pytorch to a large number of medical images to predict a single numerical value. My question is, for the labelled instances (ranging from 0-40) that I'll be using to train my CNN model - how exactly are the labels of the training set fed to the model? Is it usually contained within a csv file that you load in or otherwise is the label embedded in the name of the image or alternatively a folder that contain images of the same class (e.g. folder for "0", folder for "1")?
Any help you could provide would be really appreciated!


title: pytorch,opencv installation problem in python2 having cv library installed in python3

body:
haiii i am running python code in ubuntu 18.04
i needed opencv libarary for the code so i installed opencv using python3
 pip3 install opencv-python

but the code requires and compatible with python 2
so i tried to install the library using just pip instead of pip3
when i tried to install opencv using pip it is saying
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: opencv-python in /home/sri/.local/lib/python3.6/site-packages (4.4.0.46)
Requirement already satisfied: numpy>=1.13.3 in /home/sri/.local/lib/python3.6/site-packages (from opencv-python) (1.19.4)

but running my file python train.py shows no module named cv2
some body please help me to solve this problem
same going with my pytorch library also
 Traceback (most recent call last):
 File "train.py", line 6, in <module>
   from model import CANNet
 File "/home/sri/Downloads/Context-Aware-Crowd-Counting-master/model.py", line 1, in <module>
import torch.nn as nn
ImportError: No module named torch.nn

so i tried installing pytorch
pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

 Defaulting to user installation because normal site-packages is not writeable
 Looking in links: https://download.pytorch.org/whl/torch_stable.html
 Requirement already satisfied: torch==1.7.0+cpu in /home/sri/.local/lib/python3.6/site-packages (1.7.0+cpu)
 Requirement already satisfied: torchvision==0.8.1+cpu in /home/sri/.local/lib/python3.6/site-packages (0.8.1+cpu)
Requirement already satisfied: torchaudio==0.7.0 in /home/sri/.local/lib/python3.6/site-packages (0.7.0)
Requirement already satisfied: numpy in /home/sri/.local/lib/python3.6/site-packages (from torch==1.7.0+cpu) (1.19.4)
Requirement already satisfied: future in /home/sri/.local/lib/python3.6/site-packages (from torch==1.7.0+cpu) (0.18.2)
Requirement already satisfied: dataclasses in /home/sri/.local/lib/python3.6/site-packages (from torch==1.7.0+cpu) (0.8)
Requirement already satisfied: typing-extensions in /home/sri/.local/lib/python3.6/site-packages (from torch==1.7.0+cpu) (3.7.4.3)
Requirement already satisfied: pillow>=4.1.1 in /home/sri/.local/lib/python3.6/site-packages (from torchvision==0.8.1+cpu) (8.0.1)


title: how do i deal with torchvision datasets

body:
I am using a torchvision dataset called MNIST. I run the code in my python project to download it. Do I need to delete the code I wrote, or could I just continue writing my ai? Would the download code disturb it or redownload the whole dataset?
The code I use to download it:
train = datasets.MNIST("", train=True, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

test = datasets.MNIST("", train=False, download=True,
                       transform = transforms.Compose([transforms.ToTensor()]))

The guy from the tutorial is using jupyter notebook, but I am using PyCharm.
What do i need to do?


title: understanding pytorch sample code for gradient calculation

body:
I do not understand the purpose of the following line of code:
external_grad = torch.tensor([1., 1.])
Q.backward(gradient=external_grad)

Here's the complete program from https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html
import torch
import numpy as np
#
a = torch.tensor([2., 3.], requires_grad=True)
b = torch.tensor([6., 4.], requires_grad=True)
Q = 3* a**3 - b**2
#
print('a ',a)
print('---------------------')
print('b ',b)
print('---------------------')
print('Q ',Q)
print('---------------------')
#
# Q.backward()
#print(a.grad) results in 'grad can be implicitly created only for scalar outputs'
#print(b.grad) ditto
# because the tensor a, b contain more than 1 element
# https://discuss.pytorch.org/t/loss-backward-raises-error-grad-can-be-implicitly-created-only-for-scalar-outputs/12152
# pytorch doc : https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html
#
external_grad = torch.tensor([1., 1.])
Q.backward(gradient=external_grad)
print(a.grad)
print(b.grad)

The ouptut is:
a  tensor([2., 3.], requires_grad=True)
---------------------
b  tensor([6., 4.], requires_grad=True)
---------------------
Q  tensor([-12.,  65.], grad_fn=<SubBackward0>)
---------------------
tensor([36., 81.])
tensor([-12.,  -8.])

which matches dq/da and dq/db at the grid points I defined, but how does it work? Also, failing to provide:

gradient=external_grad

results in the exception shown as a comment in the code


title: Multi-Layer Bidirectional LSTM/GRU merge modes in PyTorch

body:
I am trying to replicate my code from Keras into PyTorch to compare the performance of multi-layer bidirectional LSTM/GRU models on CPUs and GPUs. I would like to look into different merge modes such as 'concat' (which is the default mode in PyTorch), sum, mul, average. Merge mode defines how the output from the forward and backward direction will be passed on to the next layer.
In Keras, it's just an argument change for the merge mode for a multi-layer bidirectional LSTM/GRU models, does something similar exist in PyTorch as well? One option is to do the merge mode operation manually after every layer and pass to next layer, but I want to study the performance, so I want to know if there is any other efficient way.
Thanks,


title: How to perform indexing and slicing for sparse tensor

body:
I'm trying to access elements of the sparse coo tensor.
i = [[0,2]]
v = [np.array([1,2,3,4]),[5,6,7,8]]
a = torch.sparse_coo_tensor(i,v,(4,4))
print(a.to_dense())
tensor([[1, 2, 3, 4],
        [0, 0, 0, 0],
        [5, 6, 7, 8],
        [0, 0, 0, 0]])

But I'm unable to acces neither one element, neither the whole row
print(a[0])
RuntimeError: sparse tensors do not have strides
print(a[0,:])
RuntimeError: sparse tensors do not have strides

How to do it correctly?


title: The way to combine two BERT model trained from scratch on different corpus to be one model?

body:
I want to train BERT from scratch on a different language, with almost 150GB data (corpus), and I need to end this in maximum one month.
In some articles, I read that the time it took to train the language models like BERT on like data I have was more than a month over 8 TPUs.
I have two different accounts (we assume it is account-1 and account-2) on Google Cloud TPUs service, which offered me free TPUs for a month. I need to benefit from those different accounts and ending the training process during the free month.
So, I plan to do:
1- Divide the corpus into roughly equal parts, becomes corpus-1, corpus-2.
2- Training two model from scratch (BERT-1, BERT-2), each of them with a section of the corpus(corpus-1, corpus-2), and each model will train on a different account.
3- Saving each model after training.
The question is:
1- Is there any way to combine these two model to be one model?
(i.e, BERT-1 + BERT-2 = BERT-3, where BERT-3 becomes as if it has been trained on all the corpus=150GB), Is this possible?
Lastly, the combined model (BERT-3), I will fine-tune it on different NLP tasks.
I appreciate any contribution. Thank you.


title: Loss is nan, stopping training

body:
I have a program as in the attached file.
The error is
Loss is nan, stopping training
{'loss_classifier': tensor(nan, device='cuda:0', grad_fn=<NllLossBackward>), 'loss_box_reg': tensor(nan, device='cuda:0', grad_fn=<DivBackward0>), 'loss_mask': tensor(-2.1386e+22, device='cuda:0',
       grad_fn=<BinaryCrossEntropyWithLogitsBackward>), 'loss_objectness': tensor(0.7630, device='cuda:0', grad_fn=<BinaryCrossEntropyWithLogitsBackward>), 'loss_rpn_box_reg': tensor(0.0572, device='cuda:0', grad_fn=<DivBackward0>)}
An exception has occurred, use %tb to see the full traceback.

I have suspect to data and printed all
labels, masks, boxes = self.readxml(annotation_path, img.size)
print(labels)
print(masks.shape)
print(boxes)

I don't see any error in data.
What could be wrong?
[3, 3, 3, 3, 3, 3, 3, 1]
(8, 576, 720)
[[101, 351, 124, 394], [96, 430, 126, 482], [110, 509, 141, 571], [472, 332, 515, 375], [541, 272, 585, 309], [487, 269, 529, 316], [379, 149, 418, 176], [441, 141, 508, 189]]
[3, 2, 3, 3, 3, 3, 1, 1]
(8, 576, 720)
[[164, 402, 221, 465], [260, 390, 322, 487], [328, 325, 372, 372], [312, 287, 374, 322], [318, 218, 339, 236], [180, 257, 225, 289], [400, 246, 534, 326], [487, 189, 529, 229]]
[1, 3, 3, 1, 3, 3, 3]
(7, 576, 720)
[[127, 353, 233, 570], [308, 239, 333, 272], [270, 223, 294, 253], [232, 200, 264, 247], [323, 214, 344, 234], [366, 278, 439, 311], [583, 275, 656, 326]]
[3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 4]
(15, 576, 720)
[[240, 173, 289, 220], [83, 190, 127, 233], [158, 243, 213, 305], [79, 300, 109, 359], [58, 403, 101, 478], [297, 348, 354, 403], [301, 222, 347, 270], [402, 297, 448, 361], [455, 324, 506, 390], [482, 445, 546, 500], [486, 480, 517, 508], [529, 419, 600, 483], [561, 46, 604, 82], [559, 138, 603, 189], [187, 308, 210, 345]]
[3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 1, 3, 3, 3, 3, 1, 3]
(18, 576, 720)
[[273, 158, 321, 202], [232, 101, 278, 143], [278, 84, 323, 132], [179, 67, 221, 101], [131, 23, 174, 71], [495, 456, 566, 524], [486, 507, 547, 570], [302, 308, 358, 369], [268, 400, 290, 431], [325, 360, 383, 417], [206, 475, 268, 540], [465, 196, 556, 318], [451, 168, 493, 211], [520, 103, 558, 139], [586, 34, 622, 69], [568, 134, 609, 184], [650, 135, 688, 224], [298, 142, 346, 184]]
/home/centos/anaconda3/lib/python3.8/site-packages/torch/nn/functional.py:3103: UserWarning: The default behavior for interpolate/upsample with float scale_factor changed in 1.6.0 to align with other frameworks/libraries, and now uses scale_factor directly, instead of relying on the computed output size. If you wish to restore the old behavior, please set recompute_scale_factor=True. See the documentation of nn.Upsample for details.
  warnings.warn("The default behavior for interpolate/upsample with float scale_factor changed "
[3, 3, 3, 4, 3, 3, 3, 2, 3, 2]
(10, 576, 720)
[[295, 462, 346, 514], [347, 402, 401, 461], [386, 414, 433, 459], [483, 443, 507, 480], [441, 370, 485, 414], [408, 359, 453, 402], [452, 301, 500, 353], [493, 306, 537, 361], [491, 465, 541, 532], [486, 145, 573, 208]]
[3, 4, 3, 3, 3, 4, 4]
(7, 576, 720)
[[193, 317, 242, 379], [262, 325, 290, 380], [319, 322, 370, 382], [518, 273, 597, 310], [313, 274, 384, 308], [477, 256, 489, 291], [458, 256, 474, 291]]
[3, 3, 3, 3, 2, 3, 3, 3, 4, 3, 3, 3, 2, 3, 3]
(15, 576, 720)
[[136, 109, 182, 151], [184, 129, 228, 178], [85, 65, 129, 115], [541, 122, 575, 172], [230, 157, 305, 233], [300, 214, 349, 270], [344, 265, 385, 328], [89, 200, 137, 240], [109, 244, 131, 274], [84, 309, 114, 371], [187, 322, 248, 380], [58, 401, 103, 474], [557, 421, 636, 507], [554, 149, 598, 206], [633, 60, 676, 107]]
[3, 3, 3, 3, 3, 1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 2, 3, 3, 4, 4, 4, 3, 3, 3, 3]
(27, 576, 720)
[[598, 102, 642, 150], [570, 142, 608, 192], [131, 481, 199, 558], [84, 38, 121, 73], [125, 69, 167, 105], [191, 94, 259, 161], [152, 117, 199, 163], [211, 158, 262, 205], [299, 162, 347, 216], [295, 209, 346, 260], [333, 156, 379, 202], [378, 146, 430, 196], [399, 236, 450, 288], [442, 229, 495, 280], [352, 335, 413, 396], [287, 379, 316, 410], [363, 476, 429, 544], [483, 483, 565, 573], [514, 456, 591, 535], [519, 407, 591, 476], [490, 287, 522, 324], [585, 278, 618, 314], [642, 335, 675, 380], [434, 89, 478, 119], [526, 75, 574, 107], [340, 61, 391, 108], [129, 40, 172, 76]]
[3, 3, 1, 3, 3, 3, 3, 3, 3, 3, 3, 4]
(12, 576, 720)
[[95, 328, 149, 381], [15, 313, 92, 358], [211, 204, 266, 293], [138, 269, 203, 303], [363, 297, 420, 340], [303, 224, 328, 250], [273, 223, 298, 246], [530, 353, 597, 435], [514, 286, 555, 328], [461, 281, 496, 325], [491, 233, 517, 258], [428, 260, 443, 287]]
[3, 3, 3, 1, 2, 3, 3, 3, 3, 3, 1, 3, 3, 3, 3, 3]
(16, 576, 720)
[[124, 100, 168, 142], [293, 208, 345, 262], [176, 124, 224, 173], [143, 247, 210, 333], [79, 311, 115, 385], [301, 305, 366, 370], [335, 256, 381, 315], [317, 430, 374, 483], [467, 347, 518, 408], [503, 455, 578, 519], [437, 152, 495, 269], [582, 450, 646, 524], [23, 482, 86, 571], [556, 150, 600, 203], [592, 109, 633, 157], [632, 60, 676, 106]]
Epoch: [0]  [  0/303]  eta: 0:09:51  lr: 0.000000  loss: -262.4095 (-262.4095)  loss_classifier: 1.5747 (1.5747)  loss_box_reg: 0.6368 (0.6368)  loss_mask: -264.8073 (-264.8073)  loss_objectness: 0.1623 (0.1623)  loss_rpn_box_reg: 0.0240 (0.0240)  time: 1.9534  data: 0.3372  max mem: 4226
[3, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
(13, 576, 720)
[[191, 126, 229, 149], [230, 130, 272, 155], [282, 127, 342, 168], [344, 161, 386, 186], [397, 164, 442, 198], [453, 179, 493, 207], [92, 205, 125, 257], [103, 348, 129, 392], [259, 488, 321, 547], [426, 380, 472, 425], [570, 382, 621, 433], [530, 250, 567, 285], [568, 265, 614, 302]]
[3, 3, 3, 3, 3, 3, 3, 3, 2, 4, 3, 4, 3, 4, 2, 3, 1, 3, 3, 3, 4]
(21, 576, 720)
[[325, 69, 371, 102], [133, 42, 171, 73], [100, 51, 136, 84], [138, 74, 181, 114], [195, 110, 237, 146], [265, 113, 310, 158], [193, 144, 238, 186], [246, 144, 290, 184], [329, 142, 382, 196], [300, 214, 320, 243], [331, 200, 385, 245], [385, 212, 405, 235], [162, 456, 230, 529], [372, 391, 401, 421], [425, 382, 495, 453], [503, 397, 565, 456], [555, 80, 646, 203], [520, 69, 563, 108], [637, 49, 682, 95], [549, 486, 626, 555], [374, 445, 398, 472]]
[3, 3, 1, 3, 3, 4, 3, 3, 3, 3, 3]
(11, 576, 720)
[[101, 387, 158, 451], [459, 303, 499, 351], [503, 242, 569, 333], [450, 245, 476, 275], [479, 220, 503, 246], [429, 257, 442, 291], [326, 234, 349, 260], [295, 227, 321, 250], [234, 277, 271, 315], [338, 211, 358, 231], [181, 256, 236, 291]]
[3, 3, 2, 3, 3, 2, 2, 4, 1, 3, 3, 3, 3, 3, 4]
(15, 576, 720)
[[109, 105, 145, 133], [174, 122, 211, 148], [237, 125, 275, 157], [286, 134, 328, 166], [351, 150, 393, 177], [412, 139, 455, 184], [447, 173, 495, 205], [495, 156, 514, 178], [570, 190, 641, 253], [534, 277, 580, 317], [483, 310, 531, 363], [474, 444, 532, 504], [329, 461, 385, 515], [109, 416, 138, 472], [83, 196, 99, 222]]
[3, 3, 3, 3, 1, 3, 3, 3]
(8, 576, 720)
[[100, 313, 127, 367], [98, 452, 124, 501], [264, 479, 321, 538], [397, 405, 446, 451], [484, 356, 634, 487], [499, 297, 549, 348], [569, 255, 618, 292], [490, 159, 538, 185]]
[3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3]
(11, 576, 720)
[[165, 367, 221, 423], [269, 340, 313, 395], [236, 289, 306, 332], [148, 290, 220, 327], [307, 275, 358, 314], [362, 281, 408, 315], [12, 306, 93, 349], [447, 250, 475, 295], [495, 251, 522, 292], [205, 261, 255, 290], [330, 208, 353, 226]]
[3, 3, 3, 3, 3, 3, 3, 1, 3, 4]
(10, 576, 720)
[[248, 336, 297, 388], [321, 319, 363, 361], [211, 312, 253, 357], [302, 250, 332, 288], [352, 468, 426, 550], [429, 329, 474, 381], [457, 277, 493, 326], [443, 220, 472, 275], [498, 235, 522, 259], [335, 237, 347, 259]]
[3, 3, 3, 3, 3, 3]
(6, 576, 720)
[[165, 427, 240, 505], [373, 449, 457, 528], [336, 371, 395, 438], [250, 363, 312, 431], [219, 271, 296, 319], [250, 237, 279, 267]]
[3, 3, 2, 1, 3, 3, 4, 4, 3, 3, 3, 4, 1, 3]
(14, 576, 720)
[[151, 118, 184, 142], [215, 126, 256, 154], [181, 108, 219, 135], [294, 122, 373, 173], [94, 270, 121, 316], [105, 370, 131, 416], [350, 457, 381, 489], [395, 410, 426, 444], [445, 356, 495, 403], [558, 263, 603, 299], [450, 179, 495, 206], [501, 180, 523, 203], [375, 127, 460, 186], [642, 235, 684, 265]]
[3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 1, 2, 3, 1]
(14, 576, 720)
[[159, 388, 226, 460], [184, 342, 231, 394], [126, 290, 207, 329], [212, 285, 284, 333], [298, 277, 349, 313], [375, 274, 420, 309], [362, 250, 390, 280], [236, 237, 265, 270], [280, 229, 305, 257], [314, 204, 341, 242], [243, 196, 285, 236], [453, 236, 483, 290], [489, 250, 518, 286], [504, 191, 529, 233]]
[3, 3, 3, 3, 4, 4, 2, 3, 3, 3]
(10, 576, 720)
[[103, 363, 157, 422], [338, 341, 384, 391], [210, 259, 243, 296], [260, 248, 288, 282], [395, 288, 423, 336], [458, 269, 473, 305], [495, 342, 615, 499], [292, 230, 318, 262], [331, 214, 351, 232], [448, 239, 475, 270]]
[3, 3, 1, 3, 3, 3, 3, 4, 3, 1, 3]
(11, 576, 720)
[[184, 336, 232, 387], [237, 241, 268, 275], [242, 182, 276, 239], [333, 217, 359, 237], [360, 261, 386, 292], [405, 252, 435, 291], [445, 252, 474, 287], [527, 290, 552, 333], [468, 211, 490, 238], [503, 178, 532, 231], [483, 251, 521, 286]]
[3, 2, 4, 3, 3, 3, 3, 3, 4, 3, 3, 1, 3, 1]
(14, 576, 720)
[[269, 330, 314, 384], [201, 292, 246, 375], [354, 323, 375, 372], [272, 276, 311, 325], [325, 231, 351, 261], [288, 216, 308, 240], [259, 235, 286, 275], [227, 231, 255, 252], [315, 455, 354, 552], [588, 358, 647, 418], [510, 358, 570, 420], [415, 263, 480, 375], [510, 282, 548, 321], [500, 191, 533, 233]]
[3, 3, 4, 4, 1, 3, 3, 2, 3, 3, 3]
(11, 576, 720)
[[335, 311, 375, 372], [221, 254, 276, 288], [294, 236, 308, 261], [321, 241, 337, 273], [288, 191, 333, 230], [346, 208, 369, 226], [382, 287, 432, 333], [453, 326, 502, 394], [646, 269, 719, 312], [436, 258, 469, 291], [315, 277, 366, 316]]
[1, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
(14, 576, 720)
[[280, 169, 401, 300], [607, 91, 647, 135], [565, 139, 606, 193], [173, 129, 219, 174], [522, 120, 555, 161], [452, 312, 500, 376], [330, 382, 392, 440], [178, 220, 230, 266], [94, 230, 148, 287], [66, 266, 99, 324], [70, 375, 108, 442], [469, 431, 542, 502], [601, 472, 687, 544], [639, 53, 679, 96]]
[3, 3, 1, 3, 3, 3, 3, 3, 4, 3]
(10, 576, 720)
[[432, 168, 476, 197], [479, 183, 523, 221], [262, 106, 352, 166], [90, 250, 114, 287], [87, 408, 116, 465], [329, 457, 380, 508], [492, 445, 541, 499], [440, 356, 481, 399], [569, 283, 590, 309], [526, 254, 568, 291]]
[2, 3, 3, 3, 3, 4, 3, 4, 3, 4, 3]
(11, 576, 720)
[[144, 421, 221, 556], [259, 480, 339, 572], [337, 352, 389, 415], [268, 267, 316, 302], [130, 268, 182, 301], [194, 253, 214, 288], [312, 222, 338, 242], [436, 266, 456, 300], [493, 257, 523, 289], [372, 259, 395, 298], [406, 251, 434, 299]]
[3, 3, 3, 3, 4, 3, 4, 4, 3, 3, 3, 3, 3, 3]
(14, 576, 720)
[[259, 340, 302, 400], [106, 355, 158, 412], [88, 289, 172, 330], [175, 284, 246, 332], [36, 284, 76, 335], [279, 238, 311, 266], [352, 288, 393, 326], [396, 275, 431, 315], [360, 261, 391, 294], [494, 274, 572, 313], [620, 262, 694, 306], [397, 218, 420, 243], [297, 206, 319, 226], [580, 398, 661, 482]]
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 3, 3, 3, 4, 3, 3]
(17, 576, 720)
[[273, 158, 321, 202], [337, 134, 382, 171], [278, 84, 323, 132], [179, 67, 221, 101], [131, 23, 174, 71], [495, 456, 566, 524], [486, 507, 547, 570], [239, 446, 295, 509], [316, 462, 380, 546], [325, 361, 383, 417], [381, 154, 489, 290], [446, 231, 491, 279], [598, 17, 632, 53], [653, 139, 676, 193], [487, 203, 503, 239], [298, 142, 346, 184], [677, 240, 720, 300]]
[3, 3, 3, 3, 3, 3, 3, 3, 1, 3, 3, 1, 1]
(13, 576, 720)
[[181, 356, 227, 413], [260, 337, 303, 386], [202, 295, 281, 334], [381, 297, 437, 334], [478, 281, 539, 314], [475, 232, 499, 258], [447, 254, 474, 289], [395, 258, 422, 292], [502, 181, 531, 233], [461, 198, 475, 220], [387, 480, 469, 574], [226, 207, 258, 257], [300, 192, 351, 228]]
[3, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3, 3, 2, 3, 4]
(24, 576, 720)
[[293, 436, 347, 486], [328, 453, 375, 500], [342, 397, 391, 445], [370, 418, 422, 464], [399, 350, 450, 393], [421, 372, 470, 417], [450, 312, 489, 354], [470, 335, 512, 377], [100, 71, 141, 102], [127, 107, 162, 133], [205, 127, 248, 154], [225, 107, 262, 132], [73, 159, 96, 190], [89, 200, 112, 234], [87, 239, 109, 271], [85, 273, 110, 313], [85, 321, 113, 364], [90, 366, 104, 395], [638, 294, 666, 340], [585, 262, 613, 305], [586, 219, 629, 256], [497, 198, 546, 238], [391, 169, 427, 198], [524, 309, 550, 339]]
[1, 3, 3, 3, 3, 3, 3, 3, 3, 1, 3, 1, 3]
(13, 576, 720)
[[274, 166, 395, 297], [83, 318, 112, 375], [539, 417, 612, 485], [469, 348, 530, 419], [609, 91, 647, 135], [565, 139, 606, 193], [636, 487, 718, 560], [139, 106, 185, 151], [55, 420, 107, 495], [458, 472, 539, 559], [281, 391, 346, 462], [235, 285, 352, 398], [639, 53, 679, 96]]
[3, 3, 2, 3, 1, 1, 3, 3, 3, 3, 1]
(11, 576, 720)
[[265, 415, 327, 487], [329, 327, 375, 392], [267, 268, 350, 338], [178, 257, 231, 288], [222, 180, 258, 254], [349, 183, 383, 221], [404, 312, 459, 363], [533, 303, 577, 350], [435, 266, 508, 311], [623, 272, 691, 320], [396, 224, 436, 288]]
[3, 3, 3, 3, 1, 1, 3]
(7, 576, 720)
[[246, 357, 301, 431], [106, 338, 153, 388], [212, 282, 278, 325], [228, 250, 268, 280], [271, 209, 319, 262], [440, 203, 482, 257], [445, 259, 471, 295]]
[1, 3, 2, 3, 3, 3, 3, 2, 3, 3, 3, 2, 3]
(13, 576, 720)
[[152, 65, 236, 114], [270, 120, 311, 145], [336, 127, 380, 174], [435, 184, 475, 210], [504, 266, 544, 302], [554, 270, 592, 309], [484, 320, 529, 361], [428, 357, 477, 414], [378, 366, 427, 410], [366, 421, 416, 467], [264, 466, 314, 513], [86, 330, 116, 391], [601, 225, 643, 254]]
[4, 3, 3, 3, 2, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 2, 3, 3, 3, 3, 3, 2, 3, 3, 2, 3, 3, 3, 3, 3, 3]
(31, 576, 720)
[[45, 62, 59, 82], [150, 87, 188, 118], [212, 105, 252, 129], [157, 141, 202, 166], [210, 179, 256, 214], [268, 164, 315, 192], [284, 143, 327, 172], [317, 125, 361, 155], [361, 164, 397, 190], [375, 127, 416, 153], [421, 149, 464, 182], [414, 177, 455, 204], [299, 208, 346, 240], [382, 211, 427, 240], [431, 245, 489, 274], [505, 232, 559, 271], [518, 202, 559, 236], [549, 195, 594, 224], [504, 158, 550, 197], [587, 174, 631, 200], [604, 267, 657, 299], [470, 319, 518, 376], [432, 328, 473, 373], [432, 372, 477, 414], [385, 349, 434, 408], [384, 409, 432, 456], [326, 410, 382, 459], [309, 470, 363, 522], [207, 487, 274, 554], [84, 204, 108, 237], [467, 188, 509, 217]]
[3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 4, 3, 3]
(13, 576, 720)
[[167, 394, 235, 496], [372, 481, 453, 570], [332, 342, 382, 403], [257, 351, 297, 406], [238, 264, 278, 299], [328, 279, 407, 312], [386, 303, 460, 354], [473, 275, 550, 312], [291, 215, 317, 234], [326, 209, 346, 233], [440, 262, 451, 295], [399, 218, 421, 241], [467, 214, 486, 237]]
[3, 4, 1, 3]
(4, 576, 720)
[[404, 247, 427, 276], [458, 258, 472, 290], [482, 197, 518, 241], [487, 255, 513, 287]]
[3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
(13, 576, 720)
[[129, 353, 180, 409], [41, 402, 115, 507], [271, 305, 305, 348], [232, 282, 263, 323], [162, 254, 219, 293], [297, 260, 325, 289], [332, 228, 354, 251], [364, 467, 444, 551], [496, 365, 560, 443], [600, 354, 671, 422], [454, 268, 485, 303], [486, 251, 512, 284], [477, 204, 493, 225]]
[3, 4, 3, 4, 3, 1, 3, 1, 3, 3, 3, 2, 3, 3]
(14, 576, 720)
[[250, 476, 325, 568], [308, 311, 324, 357], [164, 290, 235, 326], [56, 285, 90, 329], [156, 261, 209, 292], [222, 215, 271, 266], [291, 231, 315, 263], [322, 199, 354, 230], [447, 280, 481, 336], [496, 322, 540, 371], [546, 380, 617, 447], [473, 228, 508, 260], [405, 214, 426, 243], [331, 311, 370, 356]]
[2, 3, 3, 3, 3, 3, 3, 3, 3]
(9, 576, 720)
[[336, 382, 396, 477], [268, 294, 303, 331], [244, 272, 275, 305], [286, 242, 315, 270], [342, 219, 362, 243], [550, 371, 621, 450], [501, 333, 551, 391], [529, 304, 573, 353], [495, 254, 525, 301]]
[3, 3, 1, 1, 3, 4, 3, 3, 2]
(9, 576, 720)
[[130, 104, 164, 127], [175, 121, 212, 146], [294, 123, 374, 171], [377, 126, 458, 185], [450, 176, 492, 206], [621, 216, 644, 241], [570, 255, 615, 289], [456, 323, 493, 360], [107, 447, 137, 517]]
[3, 3, 3, 3, 3, 1, 3, 2, 4, 3, 1, 4, 3, 1]
(14, 576, 720)
[[117, 337, 168, 390], [172, 357, 223, 415], [154, 424, 218, 539], [259, 405, 320, 475], [339, 342, 382, 401], [398, 213, 448, 324], [355, 261, 387, 297], [268, 239, 295, 276], [248, 253, 263, 283], [312, 223, 338, 245], [488, 202, 534, 298], [467, 279, 483, 312], [284, 220, 306, 239], [238, 180, 277, 237]]
[3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3]
(12, 576, 720)
[[299, 219, 349, 272], [540, 124, 584, 174], [189, 136, 238, 185], [578, 87, 615, 128], [623, 27, 658, 71], [79, 71, 119, 107], [132, 104, 174, 145], [95, 299, 127, 360], [59, 426, 102, 494], [484, 149, 527, 196], [573, 133, 617, 185], [654, 78, 686, 140]]
[3, 3, 3, 3, 3, 1, 1, 4, 4, 3, 3, 3, 3, 3]
(14, 576, 720)
[[162, 418, 225, 495], [180, 357, 227, 412], [258, 337, 302, 385], [210, 251, 255, 284], [260, 237, 292, 268], [243, 197, 278, 238], [501, 183, 530, 233], [480, 216, 491, 239], [403, 217, 414, 241], [490, 256, 521, 289], [449, 253, 474, 289], [399, 261, 427, 292], [524, 292, 577, 336], [654, 277, 715, 314]]
[3, 3, 4, 3, 3, 2, 3, 3, 3, 3, 3, 3, 4]
(13, 576, 720)
[[255, 446, 322, 525], [377, 429, 455, 530], [321, 383, 347, 455], [250, 268, 282, 299], [111, 266, 168, 300], [274, 238, 302, 274], [314, 229, 336, 251], [351, 206, 371, 223], [469, 270, 512, 337], [500, 321, 559, 395], [556, 326, 607, 390], [637, 277, 713, 319], [454, 219, 463, 238]]
[4, 3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 1, 3, 3, 1, 3, 3, 3, 3, 2, 3, 3, 3, 3, 1]
(28, 576, 720)
[[237, 69, 256, 96], [259, 76, 305, 121], [314, 95, 377, 165], [268, 120, 311, 160], [253, 182, 304, 230], [281, 166, 336, 214], [317, 151, 361, 196], [508, 454, 588, 532], [204, 150, 249, 197], [106, 19, 145, 56], [119, 66, 158, 101], [280, 54, 322, 87], [110, 210, 155, 246], [273, 414, 329, 478], [319, 253, 382, 318], [462, 205, 512, 260], [411, 88, 462, 121], [477, 57, 542, 112], [614, 6, 647, 41], [585, 44, 619, 78], [541, 75, 579, 125], [579, 82, 621, 131], [528, 130, 574, 182], [557, 150, 599, 203], [662, 193, 700, 256], [617, 40, 652, 78], [632, 62, 672, 106], [384, 253, 543, 389]]
[3, 4, 3, 3, 3, 3, 3, 3]
(8, 576, 720)
[[88, 201, 116, 229], [106, 258, 120, 288], [95, 331, 125, 377], [106, 429, 135, 483], [339, 465, 390, 517], [475, 334, 520, 379], [543, 282, 582, 315], [445, 176, 487, 205]]
[3, 3, 3, 2, 3, 3, 3, 3, 3, 3, 3, 3, 3, 2, 3]
(15, 576, 720)
[[175, 89, 217, 120], [229, 107, 258, 130], [264, 117, 299, 139], [307, 120, 346, 151], [364, 166, 405, 192], [431, 180, 473, 208], [507, 157, 547, 181], [60, 68, 95, 91], [90, 241, 113, 279], [87, 302, 114, 344], [85, 437, 114, 495], [351, 409, 395, 458], [446, 334, 494, 389], [520, 280, 569, 331], [617, 247, 665, 280]]
[3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 3, 4, 3, 3]
(15, 576, 720)
[[393, 165, 430, 198], [342, 162, 380, 189], [270, 124, 308, 157], [233, 108, 270, 134], [294, 151, 334, 176], [196, 113, 233, 139], [127, 98, 163, 123], [99, 77, 133, 104], [411, 377, 463, 425], [507, 397, 576, 470], [538, 276, 583, 313], [80, 178, 105, 209], [97, 225, 110, 251], [85, 268, 108, 308], [323, 454, 378, 504]]
[3, 2, 3, 3, 3, 3, 1, 4, 3, 3, 3, 3]
(12, 576, 720)
[[299, 219, 349, 272], [540, 124, 584, 174], [579, 85, 618, 126], [622, 27, 658, 72], [146, 107, 194, 154], [189, 230, 240, 273], [95, 230, 225, 370], [73, 284, 88, 318], [70, 340, 105, 409], [331, 386, 399, 451], [451, 328, 507, 393], [558, 436, 633, 507]]
Loss is nan, stopping training
{'loss_classifier': tensor(nan, device='cuda:0', grad_fn=<NllLossBackward>), 'loss_box_reg': tensor(nan, device='cuda:0', grad_fn=<DivBackward0>), 'loss_mask': tensor(-2.1386e+22, device='cuda:0',
       grad_fn=<BinaryCrossEntropyWithLogitsBackward>), 'loss_objectness': tensor(0.7630, device='cuda:0', grad_fn=<BinaryCrossEntropyWithLogitsBackward>), 'loss_rpn_box_reg': tensor(0.0572, device='cuda:0', grad_fn=<DivBackward0>)}
An exception has occurred, use %tb to see the full traceback.

SystemExit: 1

/home/centos/anaconda3/lib/python3.8/site-packages/IPython/core/interactiveshell.py:3435: UserWarning: To exit: use 'exit', 'quit', or Ctrl-D.
  warn("To exit: use 'exit', 'quit', or Ctrl-D.", stacklevel=1)


title: Fine Tuning GPT-2

body:
I'm trying to fine tune GPT-2 for the task of if I give five consecutive numbers, what are the next consecutive numbers. For example if input_text = "one | two | three | four | five", output_text = "six | seven... | ten".
The important bits of the model that I have used via the huggingface API is as follows:
class Model(pl.LightningModule):
    def __init__(self,
                 tokenizer,
                 lr: float) -> None:
        super().__init__()
        self.lr = lr
        self.tokenizer = Tokenizer(tokenizer)
        self.model = GPT2LMHeadModel.from_pretrained('gpt2')

    def common_step(self, batch: Tuple[List[str], List[str]]) -> torch.FloatTensor:
        questions, answers = batch
        combined = [input + " <EOS> " + output for input, output in zip(questions, answers)]
        tokens = {k: v.to(self.device) for k, v in self.tokenizer(combined).items()}

        labels = tokens["input_ids"].clone()
        labels[tokens["attention_mask"]==0] = -100

        outputs = self.model(
            input_ids=tokens["input_ids"],
            attention_mask=tokens["attention_mask"],
            labels=labels,
            return_dict=True
        )

        return outputs["loss"]

    def training_step(self, batch: Tuple[List[str], List[str]], *args) -> torch.FloatTensor:
        loss = self.common_step(batch)
        return loss

    def generate_examples(self, batch):
        questions, answers = batch
        combined = [question + " <EOS> " for question in questions]
        tokens = {k: v.to(self.device) for k, v in self.tokenizer(combined).items()}

        generated = self.model.generate(
            input_ids=tokens["input_ids"],
            attention_mask=tokens["attention_mask"],
        )

        print(questions[0])
        print("="*30)
        print(self.tokenizer.decode(generated[0]))

The output I can is trying to spit out numbers but unfortunately looks like this. The output starts from where the  tag is shown, otherwise its just copying stuff. Note that GPT-2 tokenizer does not have :
Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.
<|endoftext|>five thousand, five hundred and ninety-one| five thousand, five hundred and ninety-two| five thousand, five hundred and ninety-three| five thousand, five hundred and ninety-four| five thousand, five hundred and ninety-five <EOS> <|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|> fifteen thousand, four hundred and thirty-six| ten thousand, six hundred and sixty-seven| fifteen thousand and sixty‑eight| 15 thousand and eighty-nine| fifteen hundred and seventy<|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|><|endoftext|>

So the question is why is it generating a possible candidate after a bunch of <|endoftext|> tokens. In the training set combine the input and output via a "" word (its not an actual token) and the output is there straight away without any padding.
Is it to do with the tokenizer that I've used which I've defined below?
# make sure GPT2 appends EOS in begin and end
def build_inputs_with_special_tokens(self, token_ids_0, token_ids_1=None):
    outputs = [self.bos_token_id] + token_ids_0 + [self.eos_token_id]
    return outputs

GPT2Tokenizer.build_inputs_with_special_tokens = build_inputs_with_special_tokens
gpt2_tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
# set pad_token_id to unk_token_id -> be careful here as unk_token_id == eos_token_id == bos_token_id
gpt2_tokenizer.pad_token = gpt2_tokenizer.unk_token

A working example on colab can be found here.


title: how to get gradient of multiple outputs w.r.t. each input in a pytorch network?

body:
Assume that I have a simple neural network with 3 inputs x and 2 outputs y. the training samples would like to be :
x = torch.Tensor([[1,2,3],[4,5,6],[7,8,9]]) # each row is a sample
# [[a1,a2,a3],[b1,b2,b3],[c1,c2,c3]]

I want to get dy/dx for each one of the 2 outputs w.r.t each input in each sample. In other words, I want to get dy1/da1, dy1/da2 ...... dy1/dc3, dy2/da1......
I can hardly achieve this through torch.autograd.grad(). I constructed a code like this:
import torch
from torch import nn
from torch.nn.modules.activation import ReLU
import numpy as np

seed = 0

def seed_torch(seed=0):
    np.random.seed(seed)
    torch.manual_seed(seed) # set cpu random seed
    torch.cuda.manual_seed(seed) # set gpu random seed for single gpu
    torch.cuda.manual_seed_all(seed) # set gpus random seeds for multiple gpus
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

seed_torch()

x = torch.Tensor([[1,2,3],[4,5,6],[7,8,9]])
x.requires_grad = True

class model(nn.Module):
    def __init__(self):
        super(model, self).__init__()
        self.net = nn.Sequential()
        self.net.add_module('input_layer', nn.Linear(3, 10))
        self.net.add_module('activ_layer', nn.ReLU())
        self.net.add_module('output_layer', nn.Linear(10, 2))

    def forward(self, x):
        return self.net(x)

net = model()
row,col = net.net.input_layer.weight.data.shape
net.net.input_layer.weight.data = torch.Tensor(np.ones((row,col)))
num = net.net.input_layer.bias.data.shape
net.net.input_layer.bias.data = torch.Tensor(np.zeros(num))
row,col = net.net.output_layer.weight.data.shape
net.net.output_layer.weight.data = torch.Tensor(np.ones((row,col)))
num = net.net.output_layer.bias.data.shape
net.net.output_layer.bias.data = torch.Tensor(np.zeros(num))

y = net(x)
print(y)
# tensor([[ 60.,  60.],
#         [150., 150.],
#         [ 60.,  60.]], grad_fn=<AddmmBackward>)

dydx = torch.autograd.grad(outputs=y, inputs=x,
    grad_outputs=torch.ones_like(y),
    retain_graph=True)[0]
print(dydx)
# tensor([[20., 20., 20.],
#         [20., 20., 20.],
#         [20., 20., 20.]])

dy1dx = [torch.autograd.grad(outputs=out, inputs=x,
    grad_outputs=torch.ones_like(out),
    retain_graph=True)[0][i]
    for i, out in enumerate(y)]
print(dy1dx)
# [tensor([20., 20., 20.]), tensor([20., 20., 20.]), tensor([20., 20., 20.])]

print('finished')

but I think neither dydx nor dy1dx is what I want. I checked the docs and it says:

Computes and returns the sum of gradients of outputs with respect to the inputs.

It is suitable when the output is a scalar. But I may prefer to compute the gradients of vector or matrix outputs with respect to the inputs (maybe Jacobian matrix?), not the sum of them.
I also tried
dy2dx = torch.autograd.functional.jacobian(y, x, create_graph=True)
# or
dy2dx = torch.autograd.functional.jacobian(net(x), x, create_graph=True)
# meet: TypeError. 'Tensor' object is not callable
# or
dy2dx = torch.autograd.functional.jacobian(net(), x, create_graph=True)
# meet: TypeError. forward() missing 1 required positional argument: 'x'

but meet errors. Could anyone help me with this?


title: Confusion in understand python commands for deep learning

body:
I recently started learning the deep learning with pytorch using this tutorial.
I am having problem with these lines of code.
Parameter train=True means it will take out the training data.
But how much data does it take for the training 50%?
How can we specify the amount of data for training. Similarly, couldn't understand batch_size and num_workers, what that means in loading the data data? Is the batch_size parameter is similar to one used in deep learning for training?
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)


title: How do I use Pytorch's NLLLoss with count data?

body: I have a model where I'm trying to predict with integer data, where an example of an expected output would be [40]. How do I use nn.PoissonNLLLoss to do this? Setting log_input=False slowly decreases my loss below 0.


title: how to use only last few layers of vgg16 using pytorch

body:
To train the whole network OF VGG16 is very lengthy and it takes a lot of time.
i don't know how to select only last few layers of vgg 16 as my network is taking more than 6 hours only for 2 epochs and i want to pre-train my vgg16 model over the last few layers only. is this possible ? if so, then how?


title: RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'

body:
I tried to categorize the image of the skin. The following code is used to train the data, but I will encounter problems like the title when I train halfway.

use
python: 3.7
pytorch: 1.1
system: win10

start_epoch=0
for epoch in range(num_epochs):
    print('Starting train epoch %d / %d' % (start_epoch + epoch + 1, num_epochs))
    model = model.to(device)
    print(device)
    #model.train()
    running_loss = 0
    count = 0
    epoch_loss = 0

    #for i, (input, depth) in enumerate(train_loader):
    for step,(input, depth) in enumerate(train_loader):
        # input, depth = data
        input = input.to(device)
        depth = depth.to(device)

        input_var=torch.tensor(input)
        depth_var=torch.tensor(depth).squeeze(1)
        #input_tensor = input_var.to(device)
        #depth_tensor = depth_var.to(device)

        output = model.forward(input_var)
        #new_output = output.to(device)
        #result = output.type(torch.FloatTensor)
        loss = loss_fn(output, depth_var)
        #loss = loss.type(torch.FloatTensor)
        print('count: ',count,' loss:', loss.item())
        count += 1
        running_loss += loss.data.cpu().numpy()
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        if count%100==0:
            torch.save(model,"./pkl/cifar.pkl")


title: Pytorch's stack() adds dimension?

body:
I am looking to stack one shape=(1,2) tensor array on top of another shape=(1,2) array, across dim=1, using pytorch's stack() method.

>>> import numpy as np
>>> import torch
>>> np_a = np.array([[1,2]])
>>> np_b = np.array([[3,4]])
>>> print(np_a)
[[1 2]]
>>> print(np_b)
[[3 4]]
>>> t_a = torch.from_numpy(np_a)
>>> t_b = torch.from_numpy(np_b)
>>> print(t_a)
tensor([[1, 2]])
>>> print(t_b)
tensor([[3, 4]])
>>> t_stacked = torch.stack((t_a, t_b), dim=1)
>>> print(t_stacked)
tensor([[[1, 2],
         [3, 4]]])

The resulting tensor has an added dimension and now has a shape=(1,2,2).  Why doesn't pytorch's stack() behave like numpy's vstack()?  See below:

>>> import numpy as np
>>> np_a = np.array([[1,2]])
>>> np_b = np.array([[3,4]])
>>> stacked = np.vstack((np_a, np_b))
>>> print(stacked)
[[1 2]
 [3 4]]

How do I make pytorch not add a dimension?


title: What input shapes are required for a classifier that is using a pre-trained network (Pytorch)?

body:
I'm fairly new to deeplearning, python, and pytorch so please bear with me!

I'm trying to understand Transfer Learning in Pytorch using two different Pretrained Networks: Vgg11 and Densenet121.
I've run data of shape (3 x 224 x 224) through the "features" part of the above networks, and the output shapes are as follows:

Vgg11 features output shape: 512 x 7 x 7

Densenet121 features output shape: 1024 x 7 x7

Now, I'm trying to make my own Classifier to use instead of the Pre-trained one. Upon checking both pre-trained classifiers, I see the Vgg11 classifier has in the first layer:

  (0): Linear(in_features=25088, out_features=4096, bias=True)

While the Densenet121 has in the first layer:

  (classifier): Linear(in_features=1024, out_features=1000, bias=True))

The Vgg one makes sense, since if you flatten the output of the "features" part, you get 512 x 7 x 7 = 25,088.

How does the Densenet one have only 1024 dimensions? If you flatten the output of its "features" part, you get 1024 x 7 x 7 = 50,176

Are there steps that I am missing for either of them? Are there ways to check the input and output shapes of each layer and find out exactly what's happening?

Thank you.


title: Shaping Numpy Arrays for pytorch GAN

body:
(pre-processing for qiskit QGAN but the use case is somewhat irrelevant)
I'm a bit lost trying to figure out how to preprocess an image dataset before passing it through a GAN. Below is all the relevant code up to my error. This code is derived from https://github.com/Qiskit/qiskit-tutorials/blob/master/legacy_tutorials/aqua/machine_learning/qgans_for_loading_random_distributions.ipynb and has been (attempted to be) altered to accommodate for a different input dataset. (The original has generated dummy data of much simpler dimensions)
# Root directory for dataset
dataroot = "./data/land"

# Number of workers for dataloader
workers = 2

# Batch size during training
batch_size = 128

#img size
image_size = 64

dataset = dset.ImageFolder(root=dataroot,
                           transform=transforms.Compose([
                               transforms.Resize(image_size),
                               transforms.CenterCrop(image_size),
                               transforms.ToTensor(),
                               transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                           ]))
# Create the dataloader
dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size,
                                         shuffle=True, num_workers=workers)

real_batch = next(iter(dataloader))
real_batch_arr= [t.numpy() for t in real_batch]

# Set number of qubits per data dimension as list of k qubit values[#q_0,...,#q_k-1]
num_qubits = [4]
k = len(num_qubits)

num_epochs = 100

# Initialize qGAN
qgan = QGAN(real_batch_arr,bounds=bounds, num_qubits = num_qubits,batch_size = 128, num_epochs=num_epochs, snapshot_dir=None)

This gives me is the following error.
ValueError                                Traceback (most recent call last)
<ipython-input-42-8cba9a74f024> in <module>
      5
      6  # Initialize qGAN
----> 7 qgan = QGAN(real_batch_arr,bounds=bounds, num_qubits = num_qubits,batch_size = 128, num_epochs=num_epochs, snapshot_dir=None)
      8 qgan.seed = 1
      9 # Set quantum instance to run the quantum generator

~\Anaconda3\lib\site-packages\qiskit\aqua\algorithms\distribution_learners\qgan.py in __init__(self, data, bounds, num_qubits, batch_size, num_epochs, seed, discriminator, generator, tol_rel_ent, snapshot_dir, quantum_instance)
     99         if data is None:
    100             raise AquaError('Training data not given.')
--> 101         self._data = np.array(data)
    102         if bounds is None:
    103             bounds_min = np.percentile(self._data, 5, axis=0)

ValueError: could not broadcast input array from shape (128,3,64,64) into shape (128)

I understand that the qiskit function (QGAN) at some point is attempting to turn real_batch_arr to an array (which is defined as a list when passed to QGAN). This array is expected to be just (128) however, on top of that, an array needs to be passed to QGAN, not a list (based from the original code linked above).
My question is how would I be able to transform my list into the array that I need. There also could be something I am simply fundamentally missing. I truly appreciate any advice or comments.


title: Does creating a data loader inside another data loader in pytorch slow things down (during meta-learning)?

body:
I was trying to create a data loader for meta-learning but got that my code is extremely slow and I can't figure out why. I am doing this because a set of data sets (so I need data loaders for them) is what is used in meta-learning.
I am wondering if it's because I have a collate function generating data loaders.
Here is the collate function that generates data loaders (and receives ALL the data sets):
class GetMetaBatch_NK_WayClassTask:

    def __init__(self, meta_batch_size, n_classes, k_shot, k_eval, shuffle=True, pin_memory=True, original=False, flatten=True):
        self.meta_batch_size = meta_batch_size
        self.n_classes = n_classes
        self.k_shot = k_shot
        self.k_eval = k_eval
        self.shuffle = shuffle
        self.pin_memory = pin_memory
        self.original = original
        self.flatten = flatten

    def __call__(self, all_datasets, verbose=False):
        NUM_WORKERS = 0 # no need to change
        get_data_loader = lambda data_set: iter(data.DataLoader(data_set, batch_size=self.k_shot+self.k_eval, shuffle=self.shuffle, num_workers=NUM_WORKERS, pin_memory=self.pin_memory))
        #assert( len(meta_set) == self.meta_batch_size*self.n_classes )
        # generate M N,K-way classification tasks
        batch_spt_x, batch_spt_y, batch_qry_x, batch_qry_y = [], [], [], []
        for m in range(self.meta_batch_size):
            n_indices = random.sample(range(0,len(all_datasets)), self.n_classes)
            # create N-way, K-shot task instance
            spt_x, spt_y, qry_x, qry_y = [], [], [], []
            for i,n in enumerate(n_indices):
                data_set_n = all_datasets[n]
                dataset_loader_n = get_data_loader(data_set_n) # get data set for class n
                data_x_n, data_y_n = next(dataset_loader_n) # get all data from current class
                spt_x_n, qry_x_n = data_x_n[:self.k_shot], data_x_n[self.k_shot:] # [K, CHW], [K_eval, CHW]
                # get labels
                if self.original:
                    #spt_y_n = torch.tensor([n]).repeat(self.k_shot)
                    #qry_y_n = torch.tensor([n]).repeat(self.k_eval)
                    spt_y_n, qry_y_n = data_y_n[:self.k_shot], data_y_n[self.k_shot:]
                else:
                    spt_y_n = torch.tensor([i]).repeat(self.k_shot)
                    qry_y_n = torch.tensor([i]).repeat(self.k_eval)
                # form K-shot task for current label n
                spt_x.append(spt_x_n); spt_y.append(spt_y_n) # array length N with tensors size [K, CHW]
                qry_x.append(qry_x_n); qry_y.append(qry_y_n) # array length N with tensors size [K, CHW]
            # form N-way, K-shot task with tensor size [N,W, CHW]
            spt_x, spt_y, qry_x, qry_y = torch.stack(spt_x), torch.stack(spt_y), torch.stack(qry_x), torch.stack(qry_y)
            # form N-way, K-shot task with tensor size [N*W, CHW]
            if verbose:
                print(f'spt_x.size() = {spt_x.size()}')
                print(f'spt_y.size() = {spt_y.size()}')
                print(f'qry_x.size() = {qry_x.size()}')
                print(f'spt_y.size() = {qry_y.size()}')
                print()
            if self.flatten:
                CHW = qry_x.shape[-3:]
                spt_x, spt_y, qry_x, qry_y = spt_x.reshape(-1, *CHW), spt_y.reshape(-1), qry_x.reshape(-1, *CHW), qry_y.reshape(-1)
            ## append to N-way, K-shot task to meta-batch of tasks
            batch_spt_x.append(spt_x); batch_spt_y.append(spt_y)
            batch_qry_x.append(qry_x); batch_qry_y.append(qry_y)
        ## get a meta-set of M N-way, K-way classification tasks [M,K*N,C,H,W]
        batch_spt_x, batch_spt_y, batch_qry_x, batch_qry_y = torch.stack(batch_spt_x), torch.stack(batch_spt_y), torch.stack(batch_qry_x), torch.stack(batch_qry_y)
        return batch_spt_x, batch_spt_y, batch_qry_x, batch_qry_y

that is passed to another data loader here:
def get_meta_set_loader(meta_set, meta_batch_size, n_episodes, n_classes, k_shot, k_eval, pin_mem=True, n_workers=4):
    """[summary]

    Args:
        meta_set ([type]): the meta-set
        meta_batch_size ([type]): [description]
        n_classes ([type]): [description]
        pin_mem (bool, optional): [Since returning cuda tensors in dataloaders is not recommended due to cuda subties with multithreading, instead set pin=True for fast transfering of the data to cuda]. Defaults to True.
        n_workers (int, optional): [description]. Defaults to 4.

    Returns:
        [type]: [description]
    """
    if n_classes > len(meta_set):
        raise ValueError(f'You really want a N larger than the # classes in the meta-set? n_classes, len(meta_set = {n_classes, len(meta_set)}')
    collator_nk_way = GetMetaBatch_NK_WayClassTask(meta_batch_size, n_classes, k_shot, k_eval)
    episodic_sampler = EpisodicSampler(total_classes=len(meta_set), n_episodes=n_episodes)
    episodic_metaloader = data.DataLoader(
        meta_set,
        num_workers=n_workers,
        pin_memory=pin_mem, # to make moving to cuda more efficient
        collate_fn=collator_nk_way, # does the collecting to return M N,K-shot task
        batch_sampler=episodic_sampler # for keeping track of the episode
        )
    return episodic_metaloader

(will generate a smaller example)

related:

https://discuss.pytorch.org/t/what-does-runtimeerror-cuda-driver-error-initialization-error-mean/87505/6


title: Training loss is increasing while testing loss is decreasing and Accuracy isn't improving

body:
I am following the Udacity Intro to Pytorch on Exercise 7 which is to make a model which can recognize dogs or cats.
Here is my code.
import matplotlib.pyplot as plt
import torch
from torchvision import datasets, transforms

# %%
base_path = './assets/'
data_dir = 'Cat_Dog_data'

train_folder = base_path + data_dir

# TODO: Define transforms for the training data and testing data
train_transforms = transforms.Compose([
    transforms.RandomRotation(30),
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
])

test_transforms = transforms.Compose([
    transforms.Resize(255),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize([0.5, 0.5, 0.5], [0.5, 0.5, 0.5])
])

# Pass transforms in here, then run the next cell to see how the transforms look
train_data = datasets.ImageFolder(train_folder + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(train_folder + '/test', transform=test_transforms)

trainloader = torch.utils.data.DataLoader(train_data, batch_size=32)
testloader = torch.utils.data.DataLoader(test_data, batch_size=32)

# %%
# Optional TODO: Attempt to build a network to classify cats vs dogs from this dataset
from torch import nn, optim
import torch.nn.functional as F

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(150528, 512)  # use a closer number to 150525 as output features
        self.fc2 = nn.Linear(512, 256)
        self.fc3 = nn.Linear(256, 128)
        self.fc4 = nn.Linear(128, 64)
        self.fc5 = nn.Linear(64, 32)
        self.fc6 = nn.Linear(32, 2)

        self.dropOut = nn.Dropout(p=0.2)

    def forward(self, x):
        # x.shape = [64, 3, 224, 224]
        x = x.view(x.shape[0], -1) # reshape to [63, 3 * 224 * 224]
        output = self.dropOut(F.relu(self.fc1(x)))
        output = self.dropOut(F.relu(self.fc2(output)))
        output = self.dropOut(F.relu(self.fc3(output)))
        output = self.dropOut(F.relu(self.fc4(output)))
        output = self.dropOut(F.relu(self.fc5(output)))

        output = F.log_softmax(self.fc6(output), dim=1)

        return output

# %%
epochs = 5
model = Classifier()
# model.cuda()

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr = 0.0001)

train_losses, test_losses = [], []

printed = False

for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:

        optimizer.zero_grad()
        log_ps = model(images)

        # print(f'labels shape: {labels.shape}')# [32]
        # print(f'log_ps shape: {log_ps.shape}')# [32,1]
        # log_ps = log_ps.view(-1)

        loss = criterion(log_ps, labels)
        loss.backward()
        optimizer.step()

        running_loss += loss.item()

    else:
        test_loss = 0
        accuracy = 0

        with torch.no_grad():
            model.eval()
            for images, labels in testloader:
                log_ps = model(images)

                if printed == False:
                    print(f'images.shape:{images.shape}')
                    printed = True

                test_loss += criterion(log_ps, labels)
                ps = torch.exp(log_ps)

                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor))

        model.train()

        train_losses.append(running_loss/len(trainloader))
        test_losses.append(test_loss/len(testloader))

        print("Epoch: {}/{}.. ".format(e+1, epochs),
              "Training Loss: {:.3f}.. ".format(train_losses[-1]),
              "Test Loss: {:.3f}.. ".format(test_losses[-1]),
              "Test Accuracy: {:.3f}".format(accuracy/len(testloader)))

Here is the output:
images.shape:torch.Size([32, 3, 224, 224])
Epoch: 1/5..  Training Loss: 0.152..  Test Loss: 15.427..  Test Accuracy: 0.506
Epoch: 2/5..  Training Loss: 0.435..  Test Loss: 13.295..  Test Accuracy: 0.506
Epoch: 3/5..  Training Loss: 0.484..  Test Loss: 14.875..  Test Accuracy: 0.506
Epoch: 4/5..  Training Loss: 0.542..  Test Loss: 15.111..  Test Accuracy: 0.506
Epoch: 5/5..  Training Loss: 0.617..  Test Loss: 10.860..  Test Accuracy: 0.506

I am confused that why the test accuracy isn't improving and training loss is increasing.


title: Finding column index of a PyTorch tensor that has most 1's

body:
I have a PyTorch tensor a shaped like below:
import torch
a = torch.tensor([[[1., 0., 0., 0.]],
        [[0., 1., 0., 0.]],
        [[1., 0., 0., 0.]],
        [[0., 0., 0., 1.]],
        [[1., 0., 0., 0.]],
        [[0., 0., 0., 1.]],
        [[1., 0., 0., 0.]]])

Each row of the tensor a has 4 elements, 1's and 0's. Say I index the row and the column of this tensor accordingly. So for instance, the entry in the row 0 (uppermost row) is [[1., 0., 0., 0.]], whereas the entry in the column 3 (rightmost column) is [[0., 0., 0., 1., 0., 1., 0.]].
From a given tensor, I want to identify the index of the column where 1. appear most frequently. For example, for the tensor a, the index of such a column would be 0. If there are ties in the number of 1.'s, I still would like to get all of those tied column indices.
How can I do this task easily on Python?
Thank you,


title: Plot the CIFAR10 Dataset classes frequency on pytorch

body:
I would like to visualize the number of image data for each class of the cifar10 dataset on a graph.I have tried implementing this on pytorch but I couldn't get any satisfying results.
Can anyone please help me to plot the CIFAR10 Dataset classes frequency on pytorch.
Here is an exemple of an implementation i have tried but i keep getting an error when comparing labels with the classes indexes, and the plot function seems to work pretty well
def classes_freq():
    for images, labels in trainloader:
          #print(labels.shape)

          images = images.to(device)
          labels = labels.to(device)
          count = [0]*10
          for i,e in enumerate(classes):
            if (labels.item() == i):
              count[i]+=1
    return count

classes_freq()

import matplotlib.pyplot as plt

def plot(y_units):

  x_units = ['plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

  plt.bar(x_units, y_units,width=0.8, color=['red','blue'])

  plt.xlabel('x - axis')
  plt.ylabel('y - axis')
  plt.title('classes frequency')

  plt.show()

#plot([10,20,23,50,60,70,80,90,10,45])


title: Pytorch Dataloader - TypeError: No loop matching the specified signature and casting was found for ufunc true_divide

body:
I've thrown this error and none of the suggestions I can find fix my problem unfortunately.
The error is coming from my pytorch dataloader, and when I run the contents of it manually there are no issues.
Here's the error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-f7e6029a4c31> in <module>
----> 1 x = runmodel(epochs=20)

<ipython-input-29-3bb7c79f0006> in runmodel(epochs)
     14         print("Epoch = "+str(epoch))
     15         with torch.set_grad_enabled(True):
---> 16             epoch_train_loss = training(model=model)
     17             print("Train Loss: ",epoch_train_loss)
     18             track_epoch_train_loss.append(epoch_train_loss)

<ipython-input-28-626c82187ad7> in training(model)
      3     current_loss = 0
      4     current_correct = 0
----> 5     for t, (train, y_train) in enumerate(training_generator):
      6         model.train()
      7         train = train.to(device=device, dtype=dtype)

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

<ipython-input-8-4711e033b45e> in __getitem__(self, index)
     12
     13         ID = self.ids[index]
---> 14         X = np.array(plt.imread('train/' + ID + '.png' ),dtype = np.float32)
     15         X = np.repeat(X[:, :, np.newaxis], 3, axis=2)
     16         X = np.transpose(X)

~/anaconda3/lib/python3.7/site-packages/matplotlib/pyplot.py in imread(fname, format)
   2228 @_copy_docstring_and_deprecators(matplotlib.image.imread)
   2229 def imread(fname, format=None):
-> 2230     return matplotlib.image.imread(fname, format)
   2231
   2232

~/anaconda3/lib/python3.7/site-packages/matplotlib/image.py in imread(fname, format)
   1486     with img_open(fname) as image:
   1487         return (_pil_png_to_float_array(image)
-> 1488                 if isinstance(image, PIL.PngImagePlugin.PngImageFile) else
   1489                 pil_to_array(image))
   1490

~/anaconda3/lib/python3.7/site-packages/matplotlib/image.py in _pil_png_to_float_array(pil_png)
   1653         return np.divide(pil_png, 2**4 - 1, dtype=np.float32)
   1654     if rawmode == "L":  # Grayscale.
-> 1655         return np.divide(pil_png, 2**8 - 1, dtype=np.float32)
   1656     if rawmode == "I;16B":  # Grayscale.
   1657         return np.divide(pil_png, 2**16 - 1, dtype=np.float32)

TypeError: No loop matching the specified signature and casting
was found for ufunc true_divide

And my dataloader:
class Dataset(Dataset):
    def __init__(self, ids):
        'Initialization'
        self.ids = ids

    def __len__(self):
        'Denotes the total number of samples'
        return len(self.ids)

    def __getitem__(self, index):
        'Generates one sample of data'

        ID = self.ids[index]
        X = np.array(plt.imread('train/' + ID + '.png' ),dtype = np.float32)
        X = np.repeat(X[:, :, np.newaxis], 3, axis=2)
        X = np.transpose(X)

        y = np.array(plt.imread('masks/' + ID + '.png'), dtype = np.float32)

        return X, y

I've tried ensuring the dtype is float, upgrading and downgrading numpy/matplotlib.
Thanks in advance for any help anyone can offer


title: How can I Increase the dimensions of tensor in java?

body:
I have (pytorch) tensor object in java.
(https://pytorch.org/javadoc/org/pytorch/Tensor.html)
The shape of tensor is (3 x H x W) where 3 represent (R,G,B) each.
The tensor is actually gained from bitmap image (shape = H x W)
by performing the following command, outputting (3 x H x W) which is in the TensorImageUtils (https://pytorch.org/docs/1.3.1/org/pytorch/TensorImageUtils.html)
final Tensor inputTensor = TensorImageUtils.bitmapToFloat32Tensor(bitmap,
TensorImageUtils.TORCHVISION_NORM_MEAN_RGB, TensorImageUtils.TORCHVISION_NORM_STD_RGB);

but I need (1 x 3 x H x W) tensor, not (3 x H x W) in java.
In python, that can be performed by
torch.unsqueeze(x, 0) # adding one dimension in x

where x is (3 x H x W) tensor (https://pytorch.org/docs/stable/generated/torch.unsqueeze.html)
in java, I think there is no unsqueeze command, how can i expand the dimension of tensor in java?


title: no pytorch_jni in java.library.path

body:
I am trying to run java-demo on Linux and downloaded pytorch lib from StartLocally page
https://download.pytorch.org/libtorch/cu102/libtorch-shared-with-deps-1.6.0.zip
Unfortunately, this archive file doesn't contain pytorch_jni
What can be done?


title: fastai reloading model and interpret confusion matrix

body:
I have trained a fastai model and saved it.
Can I load it with the same data, path, and files to the cnn_learner and plot the confusion matrix? It seems like the resulting confusion matrix is displaying a much higher accuracy than what was displayed during the training.
If I cannot do that, with a trained model how can I plot the confusion matrix properly?
data = ImageDataLoaders.from_path_func(path, files, label_func, num_workers=0)
model = cnn_learner(data, resnet34, loss_func=LabelSmoothingCrossEntropy(), metrics=error_rate)
model.fine_tune(1)
lr_min, lr_steep = model.lr_find()
model.fine_tune(1, lr_steep)
model.save('model')

# After training

data = ImageDataLoaders.from_path_func(path, files, label_func, num_workers=0)
model = cnn_learner(data, resnet34, loss_func=LabelSmoothingCrossEntropy(), metrics=error_rate)
model.load('model')
interp = ClassificationInterpretation.from_learner(model)
interp.plot_confusion_matrix()


title: Detecting fixed size objects in variable sized images

body:
Neural networks can be trained to recognize an object, then detect occurrences of that object in an image, regardless of their position and apparent size. An example of doing this in PyTorch is at https://towardsdatascience.com/object-detection-and-tracking-in-pytorch-b3cf1a696a98
As the text observes,

Most of the code deals with resizing the image to a 416px square while maintaining its aspect ratio and padding the overflow.

So the idea is that the model always deals with 416px images, both in training and in the actual object detection. Detected objects, being only part of the image, will typically be smaller than 416px, but that's okay because the model has been trained to detect patterns in a scale-invariant way. The only thing fixed is the size in pixels of the input image.
I'm looking at a context in which it is necessary to do the reverse: train to detect patterns of a fixed size, then detect them in a variable sized image. For example, train to detect patterns 10px square, then look for them in an image that could be 500px or 1000px square, without resizing the image, but with the assurance that it is only necessary to look for 10px occurrences of the pattern.
Is there an idiomatic way to do this in PyTorch?


title: How do I add PyTorch w/ CUDA to Dask Helm Chart

body:
Install PyTorch compiled for CUDA into the Dask helm chart, and it failed:
Install PyTorch for CUDA per the instructions on pytorch.org (see image below).
Dask helm chart example fails:
- name: EXTRA_CONDA_PACKAGES
    value: "pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch"


title: Computing intermediate gradients using backward method in Pytorch

body:
I am having trouble in understanding backward method in pytorch
x1 = tensor(2.).requires_grad_()
x2 = tensor(3.).requires_grad_() # or x2 = tensor(3.)
x3 = x1 + x2

l = (x3**2).sum()
l.backward()

print(x1)
print(x3)
print(x1.grad)
print(x3.grad)

Results are
tensor(2., requires_grad=True)
tensor(5., grad_fn=<AddBackward0>)
tensor(10.)
None

Why is x3.grad still None? Shouldn't it be tensor(10.) ?
When I run the following lines of code, x3.grad is evaluated to tensor(10.)
x3 = tensor(5.).requires_grad_()
l = (x3**2).mean()
l.backward()
print(x3.grad)


title: how to use torch dataloader to split data into batches such that they can be aligned with labels well?

body:
I am trying to create batches for torch dataloader on databricks to prepare data for distributed training by torch.dataloader.
My py3 code:
from pyspark.sql.types import StringType, IntegerType, ArrayType

cSchema = StructType([StructField("user_info", ArrayType(StringType()))\
                  ,StructField("contents", ArrayType(StringType()))\
                  ,StructField("target", IntegerType())])

test_list = [[['I', 'am', 'a', 'reader'], ['I', 'like', 'this', 'book'], 1], [['I', 'am', 'a', 'game', 'player'], ['I', 'like', 'this', 'game'], 0]]

df = spark.createDataFrame(test_list,schema=cSchema).cache()
df.show(2, 100)

Output
user_info:array
element:string
contents:array
element:string
target:integer

 +------------------------+---------------------+------+
|               user_info|             contents|target|
+------------------------+---------------------+------+
|      [I, am, a, reader]|[I, like, this, book]|     1|
|[I, am, a, game, player]|[I, like, this, game]|     0|
+------------------------+---------------------+------+

Code
df_pd = df.toPandas()

# I need to use "user_info" and "contents" to predict "target"
# so I need to split the data into batches for distributed training.

from torch.utils.data import DataLoader, Dataset
import numpy as np
import pandas as pd

class MyDataSet(Dataset):
  def __init__(self, df):
    self.data = df[['user_info', 'contents']].values.tolist()
    lt = []
    max_len_user, max_len_cont =0, 0
    for x in self.data:
      t = []
      for y in x:
        t.append(list(y))
      lt.append(t)
      user_len, cont_len = len(t[0]), len(t[1])
      if user_len > max_len_user:
        max_len_user = user_len
      if cont_len > max_len_cont:
        max_len_cont = cont_len

    user_indx, cont_indx = 0, 1
    for l in lt:
      for indx, max_len in zip([user_indx, cont_indx], [max_len_user, max_len_cont]):
        cur_len = len(l[indx])
        if cur_len < max_len:
          l[indx] += [''] * (max_len - cur_len)

    self.data = lt
    self.label = df['target'].tolist()

  def __len__(self):
    return len(self.data)

  def __getitem__(self, indx):
    return self.data[indx], self.label[indx]

ds = MyDataSet(df_pd)

dl = DataLoader(ds, num_workers=1, batch_size=2)

for i, (data, label) in enumerate(dl):
   print(i, data, label)

But, the result is like:
0 [[('I', 'I'), ('am', 'am'), ('a', 'a'), ('reader', 'game'), ('', 'player')], [('I', 'I'), ('like', 'like'), ('this', 'this'), ('book', 'game')]] tensor([1, 0])

I need a batch like :
      [I, am, a, reader] [I, like, this, book]  # row1
[I, am, a, game, player] [I, like, this, game]  # row2

such that each row can be aligned with the target value well.
How does torch.dataloader split the data into a batch as shown above?
How to assure that they are aligned with the target?
Thanks.


title: How do you do data augmentation for segmentation in PyTorch

body:
I am trying to add augmentation to my data loader in torch. All of my example images have masks and none are blank. But when I add an augmentation like RandomRotation I find that some examples in my dataset then yield an image that has not labels anymore and things fall apart.
Is there a way to correctly handle this in PyTorch. Here as example of the problem?
def __getitem__(self, idx):
      # load image and mask
      img_path = os.path.join(self.root, self.imagefolder, self.imgs[idx])
      mask_path = os.path.join(self.root, self.maskfolder , self.masks[idx])
      img = Image.open(img_path).convert("RGB")

      # The problem is that augmentation could remove all the visible labels here
      img, mask = run_augmentations(img, mask)

      mask = Image.open(mask_path)
      mask = np.array(mask)
      obj_ids = np.unique(mask)
      obj_ids = obj_ids[1:]

      masks = mask == obj_ids[:, None, None]
      num_objs = len(obj_ids)
      boxes = []

      for i in range(num_objs):
          pos = np.where(masks[i])

          xmin = np.min(pos[1])
          xmax = np.max(pos[1])
          ymin = np.min(pos[0])
          ymax = np.max(pos[0])

          if xmin == xmax or ymin == ymax:
              print("WARNING ZERO SIZE BOUNDING BOX AFTER AUGMENTATION")
              print(img_path)

          boxes.append([xmin, ymin, xmax, ymax])

      if len(boxes) == 0:
          print("WARNING ZERO BOUNDING BOXES AFTER AUGMENTATION")
          print(img_path)

      ...

EDIT: My best attempt if that if there are no labels after augmentation return the image and mask without any augmentation.


title: Writing PyTorch class in Cython

body:
I'm trying to find examples of a PyTorch nn.Module class written in Cython for speed but haven't found anything. Suppose I have the below class written in Python, what would the best Cython translation be?
class Actor(nn.Module):
    def __init__(self, state_size, action_size, hidden_size):
        super(Actor, self).__init__()
        self.l1 = nn.Linear(state_size, hidden_size)
        self.l2 = nn.Linear(hidden_size, hidden_size)
        self.l3 = nn.Linear(hidden_size, hidden_size)
        self.l4 = nn.Linear(hidden_size, action_size)
        self.log_std = nn.Parameter(-0.5 * torch.ones(action_size, dtype=torch.float32))

    def forward(self, x):
        x = torch.relu(self.l1(x))
        x = torch.relu(self.l2(x))
        x = torch.relu(self.l3(x))
        mu = self.l4(x)
        return mu

    def dist(self, mu):
        pi = Normal(mu, torch.exp(self.log_std))
        return pi

    def log_prob(self, pi, action):
        return pi.log_prob(action).sum(axis=-1)


title: Word-embedding does not provide expected relations between words

body:
I am trying to train a word embedding to a list of repeated sentences where only the subject changes. I expected that the generated vectors corresponding the subjects provide a strong correlation after training as it is expected from a word embedding. However, the angle between the vectors of subjects is not always larger than the angle between subjects and a random word.
Man   is going to write a very long novel that no one can read.
Woman is going to write a very long novel that no one can read.
Boy   is going to write a very long novel that no one can read.

The code is based on pytorch tutorial:
import torch
from torch import nn
import torch.nn.functional as F
import numpy as np

class EmbedTrainer(nn.Module):
    def __init__(self, d_vocab, d_embed, d_context):
        super(EmbedTrainer, self).__init__()
        self.embed = nn.Embedding(d_vocab, d_embed)
        self.fc_1 = nn.Linear(d_embed * d_context, 128)
        self.fc_2 = nn.Linear(128, d_vocab)

    def forward(self, x):
        x = self.embed(x).view((1, -1)) # flatten after embedding
        x = self.fc_2(F.relu(self.fc_1(x)))
        x = F.log_softmax(x, dim=1)
        return x

text = " ".join(["{} is going to write a very long novel that no one can read.".format(x) for x in ["Man", "Woman", "Boy"]])
text_split = text.split()
trigrams = [([text_split[i], text_split[i+1]], text_split[i+2]) for i in range(len(text_split)-2)]
dic = list(set(text.split()))
tok_to_ids = {w:i for i, w in enumerate(dic)}
tokens_text = text.split(" ")
d_vocab, d_embed, d_context = len(dic), 10, 2

""" Train """
loss_func = nn.NLLLoss()
model = EmbedTrainer(d_vocab, d_embed, d_context)
print(model)
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

losses = []
epochs = 10
for epoch in range(epochs):
    total_loss = 0
    for input, target in trigrams:
        tok_ids = torch.tensor([tok_to_ids[tok] for tok in input], dtype=torch.long)
        target_id = torch.tensor([tok_to_ids[target]], dtype=torch.long)
        model.zero_grad()
        log_prob = model(tok_ids)
        #if total_loss == 0: print("train ", log_prob, target_id)
        loss = loss_func(log_prob, target_id)
        total_loss += loss.item()
        loss.backward()
        optimizer.step()
    print(total_loss)
    losses.append(total_loss)

embed_map = {}
for word in ["Man", "Woman", "Boy", "novel"]:
    embed_map[word] = model.embed.weight[tok_to_ids[word]]
    print(word, embed_map[word])

def angle(a, b):
    from numpy.linalg import norm
    a, b = a.detach().numpy(), b.detach().numpy()
    return np.dot(a, b) / norm(a) / norm(b)

print("man.woman", angle(embed_map["Man"], embed_map["Woman"]))
print("man.novel", angle(embed_map["Man"], embed_map["novel"]))


title: Is it possible to resolve TypeError: argument 'input' (position 1) must be Tensor error without retraining the model?

body:
I have made a model in PyTorch for use in an openAI Gym environment. I have made it in the following way:
class Policy(nn.Module):
    def __init__(self, s_size=8, h_size=16, a_size=4):
        super(Policy, self).__init__()
        self.fc1 = nn.Linear(s_size, h_size)
        self.fc2 = nn.Linear(h_size, 32)
        self.fc3 = nn.Linear(32, 64)
        self.fc4 = nn.Linear(64, a_size)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return F.softmax(x, dim=1 )

    def act(self, state):
        state = torch.from_numpy(state).float().unsqueeze(0).to(device)
        probs = self.forward(state).cpu()
        m = Categorical(probs)
        action = m.sample()
        return action.item(), m.log_prob(action)

I then save it's state in a dictionary and use it as following :
env = gym.make('LunarLander-v2')

policy = Policy().to(torch.device('cpu'))
policy.load_state_dict(torch.load('best_params_cloud.ckpt', map_location='cpu'))
policy.eval()
ims = []
rewards = []
state = env.reset()
for step in range(STEPS):
    img = env.render(mode='rgb_array')
    action,log_prob = policy(state)
        # print(action)
    state,reward,done,i_ = env.step(action)
    rewards.append(reward)
    # print(reward,done)
    cv2_im_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    pil_im = Image.fromarray(cv2_im_rgb)

    draw = ImageDraw.Draw(pil_im)

    # Choose a font
    font = ImageFont.truetype("Roboto-Regular.ttf", 20)

    # Draw the text
    draw.text((0, 0), f"Step: {step} Action : {action} Reward: {int(reward)} Total Rewards: {int(np.sum(rewards))} done: {done}", font=font,fill="#FDFEFE")

    # Save the image
    img = cv2.cvtColor(np.array(pil_im), cv2.COLOR_RGB2BGR)
    im = plt.imshow(img, animated=True)
    ims.append([im])
    if done:
        env.close()

        break

Writer = animation.writers['pillow']
writer = Writer(fps=15, metadata=dict(artist='Me'), bitrate=1800)
im_ani = animation.ArtistAnimation(fig, ims, interval=50, repeat_delay=3000,
                                    blit=True)
im_ani.save('ll_train1.gif', writer=writer)

But this returns the error:
TypeError                                 Traceback (most recent call last)
<ipython-input-3-da32222edde2> in <module>
      9 for step in range(STEPS):
     10     img = env.render(mode='rgb_array')
---> 11     action,log_prob = policy(state)
     12         # print(action)
     13     state,reward,done,i_ = env.step(action)

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

<ipython-input-2-66d42ebb791e> in forward(self, x)
     33
     34     def forward(self, x):
---> 35         x = F.relu(self.fc1(x))
     36         x = F.relu(self.fc2(x))
     37         x = F.relu(self.fc3(x))

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\anaconda3\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     92
     93     def forward(self, input: Tensor) -> Tensor:
---> 94         return F.linear(input, self.weight, self.bias)
     95
     96     def extra_repr(self) -> str:

~\anaconda3\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

TypeError: linear(): argument 'input' (position 1) must be Tensor, not numpy.ndarray

I tried to change the forward function by adding the following line of code:
def forward(self, x):
        x = torch.tensor(x,dtype=torch.float32,device=DEVICE).unsqueeze(0) //Added this line
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return F.softmax(x, dim=1 )

But this also returns an error : ValueError: not enough values to unpack (expected 2, got 1)
The policy took a lot of time to train, and I am trying to avoid retraining it, is there a workaround for it to run without retraining?


title: Pytorch TypeError: only integer tensors of a single element can be converted to an index

body:
I am trying to use pytorch in a case for nlp binary classification and i need help to finish the neural network training and validate. I am newbie and this is my first time using pytorch, see the code below and the error...
X_train_tensor = torch.from_numpy(np.asarray(X_train)).type(torch.FloatTensor).to(device)
y_train_tensor = torch.from_numpy(np.asarray(y_train)).type(torch.FloatTensor).unsqueeze(1).to(device)

X_valid_tensor = torch.from_numpy(np.asarray(X_valid)).type(torch.FloatTensor).to(device)
y_valid_tensor = torch.from_numpy(np.asarray(y_valid)).type(torch.FloatTensor).unsqueeze(1).to(device)

X_train_tensor.size()

out: torch.Size([5438, 768])
y_train_tensor.size()

out: torch.Size([5438, 1])
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3, weight_decay=5e-4)

def binary_acc(preds, y_valid):

    y_valid_tag = torch.round(preds)

    correct_results = (y_valid_tag == y_valid).float()
    acc = correct_results.sum() / len(correct_results)

    return acc

def train(model, *var):

    epoch_loss = 0
    epoch_acc = 0

    model.train()

    for x in range(X_train_tensor):

        optimizer.zero_grad()

        predictions = model(X_train_tensor)
        loss = criterion(predictions, y_train_tensor)
        acc = binary_acc(predictions, y_valid_tensor)

        loss.backward()
        optimizer.step()

        epoch_loss += loss.item()
        epoch_acc += acc.item()

    return epoch_loss / len(X_train_tensor), epoch_acc / len(X_train_tensor)

def evaluate(model, *var):

    epoch_acc = 0

    model.eval()

    with torch.no_grad():
        for X in range(X_valid_tensor):
            predictions = model(X_train_tensor)
            acc = binary_acc(predictions, y_valid_tensor)

            epoch_acc += acc.item()

    return epoch_acc / len(X_valid_tensor)

loss=[]
acc=[]
val_acc=[]

for epoch in range(10):

    train_loss, train_acc = train(model, X_train_tensor, y_train_tensor, y_valid_tensor)
    valid_acc = evaluate(model, X_valid_tensor, y_valid_tensor)

    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Acc: {valid_acc*100:.2f}%')

    loss.append(train_loss)
    acc.append(train_acc)
    val_acc.append(valid_acc)

THE OUTPUT ERROR: TypeError: only integer tensors of a single element can be converted to an index
Please help me to fix this
TypeError                                 Traceback (most recent call last)
<ipython-input-46-bfd7a45f13aa> in <module>
      5 for epoch in range(10):
      6
----> 7     train_loss, train_acc = train(model, X_train_tensor, y_train_tensor, y_valid_tensor)
      8     valid_acc = evaluate(model, X_valid_tensor, y_valid_tensor)
      9

<ipython-input-44-689c66e0e9ed> in train(model, *var)
      6     model.train()
      7
----> 8     for x in range(X_train_tensor):
      9
     10         optimizer.zero_grad()

TypeError: only integer tensors of a single element can be converted to an index


title: Neural network - multiple columns per element (Pytorch)

body:
I'm using a neural network for classification, but for each patient/element I want to classify, I have 4 different sets of numbers to learn from (each with their own result to compare to), and obviously I want a single result for each patient.  I can clearly just take the final results and divide into groups of 4 and do something there, but it seems hacky and might cause problems later on when I try to go beyond the most basic net or try shuffling entries first.

Is there a standard method for this?  Everything I'm trying gives me that feeling where you're gluing together a solution that is inelegant and inefficient and not conducive to expanding later on


title: how does convert_pytorch_model_to_tf() work?

body:
I'm playing with the mnist_tutorial_pytorch.py, and I set the eps = 0 in the fgsm_params dictionary.

Based on my understanding this means that fgsm attack should do nothing and the target model's accuracy will not decrease.

But I got a pytorch model which test accuracy is 99.05% and the adv accuracy is 97.94%. I noticed that this pytorch model is converted to a tensorflow model by the convert_pytorch_model_to_tf() function.
So, is the decrease caused by the conversion or something else?

Parameters I have used for replication:

fgsm_params = {"eps": 0.3, "clip_min": 0.0, "clip_max": 1.0}  # I replaced 0.3 to 0.


title: How to understand this on pytorch website?

body:
I notice this on pytorch official website: https://pytorch.org/docs/stable/nn.html

If the following conditions are satisfied:

1) cudnn is enabled,

2) input data is on the GPU,

3) input data has dtype torch.float16,

4) V100 GPU is used, and

5) input data is not in PackedSequence format.

Then, persistent algorithm can be selected to improve performance.

Could anyone explain it? Thanks.


title: Loss decreases but f1 score remains unchanged

body:
Model loss decreases but the performance of model(such as F1-score) does not increase.
I want to fine-tune a pertained language model XLM from Facebook to do NER tasks, so I linked a BiLSTM and CRF.
This is my model architecture. The entire code repo have been uploaded to github    https://github.com/stefensa/XLM_NER
class XLM_BiLSTM_CRF(nn.Module):
    def __init__(self, config, num_labels, params, dico, reloaded):
        super().__init__()
        self.config = config
        self.num_labels = num_labels
        self.batch_size = config.batch_size
        self.hidden_dim = config.hidden_dim

        self.xlm = TransformerModel(params, dico, True, True)
        self.xlm.eval()
        self.xlm.load_state_dict(reloaded['model'])

        self.lstm = nn.LSTM(config.embedding_dim, config.hidden_dim // 2,
                            num_layers=1, bidirectional=True)
        self.dropout = nn.Dropout(config.dropout)
        self.classifier = nn.Linear(config.hidden_dim, config.num_class)
        self.apply(self.init_bert_weights)
        self.crf = CRF(config.num_class)

    def forward(self, word_ids, lengths, langs=None, causal=False):
        sequence_output = self.xlm('fwd', x=word_ids, lengths=lengths, causal=False).contiguous()
        sequence_output, _ = self.lstm(sequence_output)
        sequence_output = self.dropout(sequence_output)
        logits = self.classifier(sequence_output)
        return self.crf.decode(logits)

    def log_likelihood(self, word_ids, lengths, tags):
        sequence_output = self.xlm('fwd', x=word_ids, lengths=lengths, causal=False).contiguous()
        sequence_output, _ = self.lstm(sequence_output)
        sequence_output = self.dropout(sequence_output)
        logits = self.classifier(sequence_output)
        return - self.crf(logits, tags.transpose(0,1))

    def init_bert_weights(self, module):
        """ Initialize the weights.
        """
        if isinstance(module, (nn.Linear, nn.Embedding)):
            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
        if isinstance(module, nn.Linear) and module.bias is not None:
            module.bias.data.zero_()

This is the initial state of my model.

And this is the 9th epoch performance of my model. The metrics do not change.

Can anyone solve my problem?


title: Time series prediction with a recurrent neural neural network: how to prepare the input dataset sequences?

body:
I have 7 continuous input variables and I want to estimate 1 continuous variable  (y = f(x_1, ... x_8)).
The dataset is approximately 26000 measures but it will grow to millions of measures in little time.
I have the time of each measure.
I successfully built with PyTorch a Neural Network with a Linear Layer and a ReLU to do that but I'd like to take into account the past 100 measures for this estimation.
I thought of building a RNN, particularly a GRU or a LSTM since I saw they are less problematic than an Elman RNN.
I built the simplest class ever as a starting point:

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size, num_layers):
        super(RNN, self).__init__()  # compulsory for pytorch
        # Parameters
        self.nHiddenFeatures = hidden_size
        self.nLayers = num_layers
        self.nHiddenNeurons = hidden_size
        # Layers
        self.gru = nn.GRU(input_size=input_size, hidden_size=hidden_size, num_layers=num_layers)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, X, hidden):
          # initial hidden input sequence
        Y, hidden_ = self.gru(X, hidden)
        Y = Y.contiguous().view(-1, self.hidden_dim)
        Y = self.fc(Y)
        return F.relu(Y), hidden_  # output variable can't be negative

# Parameters:
input_size = 7
output_size = 1
hidden_size = 4
num_layers = 2
seq_length = 100

# model
rnn = RNN(input_size=input_size, hidden_size=hidden_size, output_size=output_size, num_layers=num_layers).to(device)
optimizer = torch.optim.Adam(rnn.parameters(), lr=learningRate)
loss_func = nn.MSELoss()

but now I'm struggling to figure out how to train it: in particular in preparing the correct shape for the X_train dataset. My current situation is that X_train.shape = torch.Size([26000, 7])
where I recall 26000 is the number of timestamps and 7 is the number of variables measured at each timestamp.

I've read the documentation on PyTorch RNNs and I know the input tensor should have the shape (seq_len, batch, input_size):
my problem is how to create an input tensor, from the current tensor, in which the sequences runs all the timestamps considering the previous 100 timestamps: I'm I right in trying to build this tensor where sequences overlaps and thus its shape would be approximately X_train.shape = torch.Size([100, 26000, 7]) or should I create separate sequences as I saw in many examples on the internet, ending with X_train.shape = torch.Size([100, 260, 7])? This solution seemed to my like considering previous time just in 260 timestamps and not in the remaining ones...

The final goal would be to have a regular training, like:

hidden = torch.zeros(..., ..., ...)  # initialization
for epoch in range(nEpochs):
    optimizer.zero_grad()  # set gradient to zero in each step
    Y_estTrain= rnn(X_train, hidden)  # prediction of all samples
    loss = loss_func(Y_estTrain, Y_trueTrain)  # difference between predicted and expected
    loss.backward(loss)
    optimizer.step()  # update weights of the NN
    if epoch % printLossOnceEvery == 0:  # check loss decrease during training
        print(f"Epoch = {epoch}, MSE = {loss.item():0.1f}")

Thank you for reading the whole question: is a bit long but I preferred to add as much informations as I could.


title: How to handle entrypoints nested in folders with amazon sagemaker pytorch estimator?

body:
I am attempting to run a training job on amazon sagemaker using the python-sagemaker-sdk, estimator class.

I have the following

estimator = PyTorch(entry_point='training_scripts/train_MSCOCO.py',
                            source_dir='./',
                            role=#dummy_role,
                            train_instance_type='ml.p3.2xlarge',
                            train_instance_count=1,
                            framework_version='1.0.0',
                            output_path=#dummy_output_path,
                            hyperparameters={'lr': 0.001,
                                             'batch_size': 32,
                                             'num_workers': 4,
                                             'description': description})

role and output_path hidden for privacy.

I get the following error, "No module named training_scripts\train_MSCOCO".

When I run python -m training_scripts.train_MSCOCO the script runs fine. However when I pass entry_point='training_script.train_MSCOCO.py it will not run as, "No file named "training_scripts.train_MSCOCO.py" was found in directory "./"".

I am confused as to how to run a nested training script from the top level of my repository within AWS sagemaker, as they seem to have conflicting path needs, one in python module dot notation, the other in standard filepath slash notation.


title: Being confused about tensor dimensions in pytorch

body:
I'm new to pytorch and machine learning in general and I'm trying to create a simple convolutional neural net that classifies the MNIST handwritten digits.
Unfortunately when I'm trying to train it, I get the following error:

ValueError: Expected input batch_size (288) to match target batch_size (64).

Here is the neural network code.

from torch import nn
from torch.nn.functional import relu, log_softmax

class MNIST_SimpleConv(nn.Module):
    def __init__(self):
        super(MNIST_SimpleConv, self).__init__()

        self.conv1 = nn.Conv2d(in_channels=3,  out_channels=32, kernel_size=3, stride=1)
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=1)

        self.pool1 = nn.MaxPool2d(2, 2)

        self.dense1 = nn.Linear(4*4*64, 100)
        self.dense2 = nn.Linear(100, 10)

    def forward(self, x):
        x = relu(self.conv1(x))
        x = relu(self.conv2(x))
        x = self.pool1(x)

        x = x.view(-1, 4*4*64)

        x = relu(self.dense1(x))
        return log_softmax(self.dense2(x), dim=1)

And the training code is as follows:

from nets.conv import MNIST_SimpleConv
from torchvision import datasets, transforms
from torch.utils.data import DataLoader
from torch.nn.functional import nll_loss
import torch.optim as optim
import torch
from torch import nn

MNIST_ROOT = "data/MNIST"

#prepare dataset
mnist_train_ds = datasets.ImageFolder(root=MNIST_ROOT+"/train", transform=transforms.Compose([
                           transforms.ToTensor()]))
mnist_test_ds  = datasets.ImageFolder(root=MNIST_ROOT+"/test", transform=transforms.Compose([
                           transforms.ToTensor()]))
mnist_train = DataLoader(mnist_train_ds, batch_size=64, shuffle=True, num_workers=6)
mnist_test  = DataLoader(mnist_test_ds, batch_size=64, shuffle=True, num_workers=6)

criterion = nn.CrossEntropyLoss()

def train(model, device, train_loader, optimizer, epoch):
    model.train()

    for batch_idx, (data, target) in enumerate(train_loader, 0):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()
        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

model = MNIST_SimpleConv().to(device)
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

for epoch in range(1, 10):
    train(model, device, mnist_train , optimizer, epoch)

So far I have investigated how the dimensions of 'x' change while x is forwarded through the network.

Input:
torch.Size([64, 3, 28, 28])

After x = relu(self.conv1(x)):
torch.Size([64, 32, 26, 26])

After x = relu(self.conv2(x)):
torch.Size([64, 64, 24, 24])

After x = self.pool1(x):
torch.Size([64, 64, 12, 12])

After x = x.view(-1, 4*4*64)
torch.Size([576, 1024])

After x = relu(self.dense1(x))
torch.Size([576, 100])

After x = log_softmax(self.dense2(x), dim=1)
torch.Size([576, 10])

The error is probably caused by x = x.view(-1, 4*4*64) for some reason producing a tensor with the shape [576, 1024] instead of [64, 1024].
(If I understand this correctly the first dimension should be equal to the batch size which in my case is 64.)

What am I doing wrong?


title: Why does Tensorflow have different dimensions for its Conv2D weights then Pytorch does?

body:
Im working on converting and old project written in tensorflow v1.13 to pytorch v1.4.0 when I noticed that tensorflow and pytorch had different size weight tensors for the 2d cnns.

Here is my tensorflow code

cnn = tf.layers.conv2d(img_tensor, 16, (3, 3), (1, 1), padding='SAME', name='cnn_1')
cnn = tf.layers.conv2d(cnn, 32, (3, 3), (1, 1), padding='SAME', name='cnn_2')

init = tf.global_varaibles_initializer()
with tf.Session() as sess:
   sess.run(init)
   vars = {v.name:v for v in tf.trainable_variables()}
   print(sess.run(vars['cnn_2/kernel:0']).shape)

Result

(3, 3, 1, 32)

Here is my pytorch code

class Net(Module):
   def __init__(self):
      super(Net, self).__init__()
      self.create_cnn()

   def create_cnn(self):
      self.cnn_layers = Sequential(
         Conv2d(1,16,3,padding=1)
         Conv2d(16,32,3,padding=1)
      )

   def forward(self, x):
      return self.cnn_layers(x)

def weights_init(m):
   if type(m) == Conv2d:
      if(m.bias.shape[0] == 32):
         print(m.weight.data.shape)

model = Net()
model.apply(weights_init)

Result

torch.Size([32,16,3,3])

The reason this came up was because my pytorch model is not working so I started going a layer at a time and comparing outputs between tensorflow and pytorch. In order for that to work I had to set the weights on both models to the same values. Well I got the 2nd cnn layer and was confused when it failed to set the weights because the size was wrong. A little bit of poking around and I found this difference.

I looks like tensorflow is using the same kernel across all the channels where pytorch has a unique kernel for each channel. If this is the case, how can I replicate this in pytorch?