title: ROC-AUC is high but PR-AUC value is very low

body:
I am working on DNA sequences data and using CNN in Pytorch. My dataset is hugely imbalanced.

positive class samples (~500)

negative class samples (~150,000)

So I am using WeightedRandomSampler to oversample and balance classes before feeding to data loader.

I use a 5-fold cross-validation. When I did few test runs, I could get a decent ROC value but the PR-AUC value seems to be really low.

For fold 1:

roc auc 0.9667848699763594

precision auc 0.055329116326074484

For fold 2:

roc auc 0.8476321207961566

precision auc 0.03307627288669479

For fold 3:

roc auc 0.9528898540612085

precision auc 0.05020178518546394

I suspect that there are lot of false negatives. Since the positive class samples (~500) is very low compared to negative class samples (~150,000) the model learns the negative class better and predicts most of the test samples as negative.

I tried weighing the positive class using

weight = [50.0]

class_weight = torch.FloatTensor(weight).to(device)

criterion = nn.BCEWithLogitsLoss(pos_weight=class_weight)

By doing this, almost all samples are predicted as positive.

I tried Adaptive learning rates as well but the precision-recall values do not seem to improve.
Can someone guide me and let me know the ideas to improve Precision and Recall values?

Thanks!


title: How to make part of the embedding matrix trainable and rest part as not trainable in pytorch?

body:
Codes are in Pytorch.

I want part of my embedding matrix to be trainable and I want rest part to freeze weights as they are pretrained vectors.

My src_weight_matrix is the pretrained embedding matrix of dimension : [vocab_size + 4 special_tokens] X [emb_dim] = [49996 + 4] X [300]

first 49996 rows are for word in vocab and last 4 words in the vocabulary are special token i.e. [UNK], [PAD], [START], [STOP]. I have randomly initialized the embedding vectors for these 4 words.

So I want to train these 4 embedding weights and let other embedding have their own weights.

The code is as follow where all embedding weights are frozen except last 4 which is correct or not I don't know.

class Encoder(nn.Module):
    def __init__(self, src_weights_matrix):
        super(Encoder, self).__init__()
        self.embedding = nn.Embedding(config.vocab_size, config.emb_dim)
        self.embedding.load_state_dict({'weight': src_weights_matrix})
        self.embedding.weight.requires_grad = False
        self.embedding.weight[-4:, :].requires_grad = True
        init_wt_normal(self.embedding.weight)
        self.lstm = nn.LSTM(config.emb_dim, config.hidden_dim, num_layers=1, batch_first=True, bidirectional=True)
        init_lstm_wt(self.lstm)
        self.W_h = nn.Linear(config.hidden_dim * 2, config.hidden_dim * 2, bias=False)

Any lead is greatly appreciated. Thanks in advance


title: Got 512 channels instead of 64 - what should I change in my Autoencoder?

body:
ndf = 128
z_size = 512

# define the model (a simple autoencoder)
class MyNetwork(nn.Module):
    def __init__(self):
        super(MyNetwork, self).__init__()

        self.encoder = nn.Sequential(
            nn.Conv2d(3, 6, kernel_size=3, stride=1, padding=1),
            nn.ReLU(True),
            nn.Conv2d(6,16,kernel_size=3, stride=1, padding=1),
            nn.ReLU(True),
            nn.Conv2d(16,64,kernel_size=3, stride=1, padding=1),
            nn.ReLU(True))

        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(64,16,kernel_size=3, stride=1, padding=1),
            nn.ReLU(True),
            nn.ConvTranspose2d(16,6,kernel_size=3, stride=1, padding=1),
            nn.ReLU(True),
            nn.ConvTranspose2d(6,3,kernel_size=3, stride=1, padding=1),
            nn.ReLU(True))

        self.fc1 = nn.Linear(ndf*4*8*16,z_size)
        self.fc2 = nn.Linear(ndf*4*8*16,z_size)
        self.d1 = nn.Linear(z_size, ndf*4*8*8)
        self.z_size = z_size
        self.d_max = ndf *4

    def encode(self, x):
        x = self.encoder(x)
        x = x.view(x.shape[0], -1)
        mu = self.fc1(x)
        log_var = self.fc2(x)
        return mu, log_var

    def decode(self,x):
        x = x.view(x.shape[0], self.z_size)
        x = self.d1(x)
        x = x.view(x.shape[0], self.d_max, 8,8)
        x = self.decoder(x)
        return x

    def reparameterize(self, mu, log_var):
        std = torch.exp(0.5 * log_var)
        eps = torch.randn_like(std)
        return eps.mul(std).add_(mu)

    def forward(self, x):
        mu, log_var = self.encode(x)
        mu = mu.squeeze()
        log_var = log_var.squeeze()
        z = self.reparameterize(mu, log_var)
        return self.decode(z.view(-1, self.z_size, 1, 1)), mu, log_var

I have adapted code from a tutorial and I'm currently getting the error 'Given transposed=1, weight of size 64 16 3 3, expected input[16, 512, 8, 8] to have 64 channels, but got 512 channels instead' when attempting to run my auto-encoder.

Could someone please explain how I should further adapt this code, using the CIFAR10 dataset, which a batch size of 16.


This PyTorch model evaluation crashes with error c0000374 in Unreal Engine.

The same code is working without any issues on regular desktop App.
Model is loaded from content folder of Plugin:

auto model_path = FPaths::Combine(*BaseDir, TEXT("Content/model.pt"));
module = torch::jit::load(TCHAR_TO_ANSI(*model_path));

Error appears on evaluation:

auto output = module.forward(inputs);

Output window log:

  'UE4Editor.exe' (Win32): Loaded 'D:\projects\MyProject\Plugins\MyPlugin\Binaries\ThirdParty\MyPluginLibrary\Win64\torch_cpu.dll'.
  Module was built without symbols.

  'UE4Editor.exe' (Win32): Loaded 'C:\Windows\System32\vcomp140.dll'.

  Critical error detected c0000374

  UE4Editor.exe has triggered a breakpoint.


title: How can I send a custom dataset through ssh?

body:
I have to train a GAN (coded in Python using pytorch) on a remote GPU that I can only access from my PC via ssh, but I have a custom dataset (that I cannot download from anywhere) which is stored in the PC without the GPU.

I've searched on Google very intensively and tried to use the scp command (which is the only solution that I've found), but it seems that the dataset is too big to be send within an acceptable time (13GB in size).

How can I transfer the dataset to the PC with the GPU within a decent amount of time, given that I cannot access the PC in any other way than an ssh connection, in order to train the network? Moreover, how can I retrieve the state_dict() and store it to my PC, once the training is complete?


title: What does .post2 in pytorch versions means?

body:
I was looking at torch versions
https://pypi.org/project/torch/#history

1.5.0
1.4.0
1.3.1
1.3.0.post2
1.3.0
1.2.0
1.1.0.post2
1.1.0
1.0.1.post2
1.0.1
1.0.0
0.4.1.post2
0.4.1
0.4.0
0.3.1
0.3.0.post4
0.1.2.post2
0.1.2.post1

And I found out that some versions have the suffix .post2 (or .post3, post4).

At first I thought it was a release made after the minor version X release already happened (postX), but then I saw 1.3.0.post2, so that doesn't seem to make sense.

Also, pytorch doesn't seem to follow semver.

What does postX mean?


title: Impact of data shuffling on results reproducibility in Pytorch

body:
Pytorch dataloader class has the following constructor:

DataLoader(dataset, batch_size=1, shuffle=False, sampler=None,
           batch_sampler=None, num_workers=0, collate_fn=None,
           pin_memory=False, drop_last=False, timeout=0,
           worker_init_fn=None)

When shuffle is set to True, data is reshuffled at every epoch. Shuffling the order in which examples are fed to the classifier is helpful so that batches between epochs do not look alike. Doing so will eventually make our model more robust.

However, I'm unable to understand by setting shuffle=True, can we get the same accuracy value for different runs?


title: How to slice multiple bounding boxes from PyTorch Tensor?

body:
So my question is fairly simple one:

I have multiple bounding boxes formed by: [TopLeft_X, TopLeft_Y,
BottomRight_X, BottomRight_Y]
And also I have a Tensor mask sized [900,
1600]
So here is the first 3 bounding boxes I have:

tensor([[1041,  465, 1102,  508],
        [1037,  468, 1108,  519],
        [1041,  465, 1107,  515]], device='cuda:0', dtype=torch.int32)
Essentially, I want to get number of pixels(valued as 1) that each
bounding box intersects with mask.
Currently, I am looping through all the bounding boxes and get the
respective slicing as:

map_mask[y_min:y_max, x_min:x_max] -> 2D Output

How can I do this simultaneously without a for loop in PyTorch?


title: How to keep the accuacy when convert a pytorch model to TensorRT in FP16 mode

body:
I want to use TensorRT FP16 mode to accelerate my pytorch model, but there is a loss of precision.
My question is how can I use the mixed precision training of pytorch, to avoid the loss of accuracy when converting to a TensorRT FP16 model.
I have tried the torch.cuda.amp.autocast to training the model, but it still lost some precision.


title: Training RoBerta using transformers on masked language task giving weird results?

body:
I trained a RoBERTa model following this colab - https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=XaFAsB_fnU3K

Here is how my data looked:

Merkel bemoans lack of rain as Germany fears for its forests .\n
Germany’s forests, covering a third of its territory and as much a part of its cultural landscape as its physical one, are in danger.\n
An aerial view shows a forest near Gummersbach, Germany, April 24, 2020, following an unusually warm, dry winter after a summer of record temperatures leaving forests dried out.\n
Picture taken with a drone.\n
The last two exceptionally hot and dry summers have weakened millions of trees, undermining their defences against the bark beetle, which can be fatal to ancient woodlands.\n
And after an exceptionally dry April, with summer still two months away, a forest fire has already had to be put out near the town of Gummersbach in western Germany this week.\n
“We’re already noticing these days that it’s not raining enough in many areas.

After training the model I used pipeline from the transforms library for the fill_mask task

from transformers import pipeline

fill_mask = pipeline(
    "fill-mask",
    model="./output",
    tokenizer="./output"
fill_mask("Merkel bemoans lack of rain as <mask> fears for its forests")
)

These are the results:

[{'sequence': '<s> Merkel bemoans lack of rain as. fears for its forests</s>',
  'score': 0.040456026792526245,
  'token': 18},
 {'sequence': '<s> Merkel bemoans lack of rain as, fears for its forests</s>',
  'score': 0.03502459451556206,
  'token': 16},
 {'sequence': '<s> Merkel bemoans lack of rain as the fears for its forests</s>',
  'score': 0.03497963398694992,
  'token': 269},
 {'sequence': '<s> Merkel bemoans lack of rain as\n fears for its forests</s>',
  'score': 0.03180328756570816,
  'token': 203},
 {'sequence': '<s> Merkel bemoans lack of rain as to fears for its forests</s>',
  'score': 0.020796578377485275,
  'token': 288}]

As you can see there is no meaningful word(s) returned only punctuations and one other word (to) which doesn't make sense. What am i doing wrong here? Do I have to remove all punctuations?


title: Does torch.utils.cpp_extension.load lead to potential racing condition?

body: If a Pytorch extension (cpp/CUDA) has never been compiled before and I have multiple scripts loading it by torch.utils.cpp_extension.load at the same time. Does that lead to a potential racing condition? I tried to look into the Document but couldn't find a clear answer. I also found a relative pull request at https://github.com/pytorch/pytorch/pull/30956.


title: Saving a torch tensor as multi-dimensional array in C++

body:
Is there any way to save a multi-dimensional tensor created in pytorch as a multi-dimensional array in C++?
for example, in python,
A=numpy(torch.tensor(nn.rand(1,17,512,768)))

This will output an array as the variable A.
I want to know if there is a way to save such a tensor in C++.
Thank you


title: Building Autoencoder with Softmax classifier - Input mismatch error

body:
I am trying to train an auto-encoder with a softmax classifier to replicate the results in this paper Intriguing properties of neural networks.
My implementation is the following:
n_embedded = 400
class AE400_10(nn.Module):
    def __init__(self):
        super(AE400_10, self).__init__()
        self.encoder = nn.Sequential(nn.Linear(28*28, n_embedded), nn.Sigmoid())
        self.decoder = nn.Sequential(nn.Linear(n_embedded, 28*28))
        self.classifier = nn.Sequential(nn.Linear(28*28, 10))

    def forward(self, x):
        x = x.view(-1, 28*28)
        encoded = self.encoder(x)
        decoded = self.decoder(encoded)
        out = self.classifier(decoded) ##NEW UPDATED

        return decoded, F.log_softmax(out)

For the training I have the following:
model = AE400_10().to(device)

criterion1 = nn.MSELoss()
criterion2 = nn.BCELoss()

optimizer = optim.Adam(model.parameters(), lr=0.01)
for epoch in range(epochs):
    total_batch = len(train_set) // batch_size_train

    for batch_idx, (data, target) in enumerate(MNSIT_train):
        X = data.to(device)
        Y = target.to(device)
        optimizer.zero_grad()

        decoded, out = model(X)

        loss1 = criterion1(decoded, inputs)
        loss2 = criterion2(out, labels)
        loss = loss1 + loss2
        loss.backward()
        optimizer.step()
        if (batch_idx+1) % 100 == 0:
            print('Epoch [%d/%d], lter [%d/%d], Loss: %.4f'%(epoch+1, epochs, batch_idx+1, total_batch, cost.item()))

But I am getting the following error:

RuntimeError: size mismatch, m1: [128 x 400], m2: [784 x 10] at
/Users/soumith/mc3build/conda-bld/pytorch_1549593514549/work/aten/src/TH/generic/THTensorMath.cpp:940

I understand this is an error in the dimension but I am not sure why it is happening.
::UPDATE::
I fixed the input to the classifier based on the comments below and now I am getting the following error:

RuntimeError: The size of tensor a (784) must match the size of tensor
b (28) at non-singleton dimension 3


title: PyTorch mutiprocessing: Do I need to use Lock() when accessing a shared model?

body:
I have some questions about using the torch.multiprocessing module. Let’s say I have a torch.nn.Module called model and I call model.share_memory() on it.
What happens if two threads call the forward(), i.e. model(input) at the same time? Is it safe? Or should I use Lock mechanisms to be sure that model is not accessed at the same time by multiple threads?
Similarly, what happens if two or more threads have an optimizer working on model.parameters() and they call optimizer.step() at the same time?
I ask these questions because I often see the optimizer.step() being called on shared models without lock mechanisms (i.e. in RL implementations of A3C or ACER) and I wonder if it is a safe thing to do.


title: PyTorch DataLoader using Mongo DB

body:
I would like to know if using a DataLoader connected to a MongoDB is a sensible thing to do and how this could be implemented.
Background
I have about 20 million documents sitting in a (local) MongoDB. Way more documents than fit in memory. I would like to train a deep neural net on the data. So far, I have been exporting the data to the file system first, with subfolders named as the classes of the documents. But I find this approach nonsensical. Why export first (and later delete) if the data is already well-maintained sitting in a DB.
Question 1:
Am I right? Would it make sense to directly connect to the MongoDB? Or are there reasons not to do it (e.g. DBs generally being too slow etc.)? If DBs are too slow (why?), can one prefetch the data somehow?
Question 2:
How would one implement a PyTorch DataLoader?
I have found only very few code snippets online ([1] and [2]) which makes me doubt my approach.
Code snippet
The general way how I access MongoDB is as follows below. Nothing special about this, I think.
import pymongo
from pymongo import MongoClient

myclient = pymongo.MongoClient("mongodb://localhost:27017/")
mydb = myclient["xyz"]
mycol = mydb["xyz_documents"]

query = {
    # some filters
}

results = mycol.find(query)

# results is now a cursor that can run through all docs
# Assume, for the sake of this example, that each doc contains a class name and some image that I want to train a classifier on


title: PyTorch can't find the name?? (NameError: name 'device' is not defined)

body:
sorry - I'm a complete beginner!
I am trying to build a 'mini-system' using the Torchreid libraries from https://kaiyangzhou.github.io/deep-person-reid/index.html#
In their version they use CUDA but my Mac is not compatible with CUDA and it doesn't have a
CUDA enabled GPU so I installed the CPU-only version of PyTorch instead - therefore I changed model = model.cuda() to model = model.to(device) and added in device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') as you can see below. I thought this would work but I keep getting the NameError: name 'device' is not defined and I don't know what to do.
Please help!
(I also tried putting device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') at the top instead of the bottom to see if it made any difference but I just got another error - NameError: name 'torch' is not defined)
model = torchreid.models.build_model(
    name='resnet50',
    num_classes=datamanager.num_train_pids,
    loss='softmax',
    pretrained=True
)

model = model.to(device)

optimizer = torchreid.optim.build_optimizer(
    model,
    optim='adam',
    lr=0.0003
)

scheduler = torchreid.optim.build_lr_scheduler(
    optimizer,
    lr_scheduler='single_step',
    stepsize=20
)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')


title: PyTorch DataLoader and Parallelism

body:
I have created a class that extends DataSet to load images for a segmentation task, so one input and one output. Every time the method getitem is called, this class performs the necessary operations for data augmentation on both the input and the output, and it works perfectly.

However, when I use this class with PyTorch DataLoader, the input transformation do not match with the output transformations. To perform the same operations, I have to get/set the states of random operations/classes, and my bet is that the DataLoader does the same, so there is a conflict between them.

How can I fix it?


title: Install pytorch fail on win10 external GPU

body:
I can not import pytorch on my gpu conda env:

C:\Users\Jeffy\Desktop
$ python
Python 3.7.2 (default, Feb 11 2019, 14:11:50) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ProgramData\Anaconda3\envs\gpu\lib\site-packages\torch\__init__.py", line 84, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

I have two conda env, one is gpu with external gpu GTX1050, one is base.
On my base env, I have installed pytorch-cpu version and it works well.
However, I cannot install pytorch gpu version on my gpu env.
on my gpu env, I have the following packages installed (including cudnn, intel-openmp, cmake and so on):

$ conda list
 packages in environment at C:\ProgramData\Anaconda3\envs\gpu:

 Name                    Version                   Build  Channel
absl-py                   0.7.0                    pypi_0    pypi
astor                     0.7.1                    pypi_0    pypi
blas                      1.0                         mkl
ca-certificates           2019.1.23                     0
certifi                   2018.11.29               py37_0
cffi                      1.11.5           py37h74b6da3_1
cmake                     3.12.2               he025d50_0
cudatoolkit               10.0.130                      0
cudnn                     7.3.1                cuda10.0_0
freetype                  2.9.1                ha9979f8_1
gast                      0.2.2                    pypi_0    pypi
grpcio                    1.18.0                   pypi_0    pypi
h5py                      2.9.0                    pypi_0    pypi
icc_rt                    2019.0.0             h0cc432a_1
intel-openmp              2019.0                   pypi_0    pypi
jpeg                      9b                   hb83a4c4_2
keras-applications        1.0.7                    pypi_0    pypi
keras-preprocessing       1.0.9                    pypi_0    pypi
libpng                    1.6.36               h2a8f88b_0
libtiff                   4.0.10               hb898794_2
markdown                  3.0.1                    pypi_0    pypi
mkl                       2019.1                      144
mkl-include               2019.1                      144
mkl_fft                   1.0.10           py37h14836fe_0
mkl_random                1.0.2            py37h343c172_0
mock                      2.0.0                    pypi_0    pypi
ninja                     1.8.2.post2              pypi_0    pypi
numpy                     1.15.4           py37h19fb1c0_0
numpy-base                1.15.4           py37hc3f5095_0
olefile                   0.46                     py37_0
openssl                   1.1.1a               he774522_0
pbr                       5.1.2                    pypi_0    pypi
pillow                    5.4.1            py37hdc69c19_0
pip                       19.0.1                   py37_0
protobuf                  3.6.1                    pypi_0    pypi
pycparser                 2.19                     py37_0
python                    3.7.2                h8c8aaf0_2
pytorch                   1.0.1           py3.7_cuda100_cudnn7_1    pytorch
pyyaml                    3.13             py37hfa6e2cd_0
setuptools                40.7.3                   py37_0
six                       1.12.0                   py37_0
sqlite                    3.26.0               he774522_0
tensorboard               1.12.2                   pypi_0    pypi
tensorflow-estimator      1.13.0rc0                pypi_0    pypi
tensorflow-gpu            1.13.0rc1                pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tk                        8.6.8                hfa6e2cd_0
torchvision               0.2.1                      py_2    pytorch
typing                    3.6.4                    py37_0
vc                        14.1                 h21ff451_1    peterjc123
vs2015_runtime            14.15.26706          h3a45250_0
vs2017_runtime            15.4.27004.2010               1    peterjc123
werkzeug                  0.14.1                   pypi_0    pypi
wheel                     0.32.3                   py37_0
wincertstore              0.2                      py37_0
xz                        5.2.4                h2fa13f4_4
yaml                      0.1.7                hc54c509_2
zlib                      1.2.11               h62dcd97_3
zstd                      1.3.7                h508b16e_0


title: PyTorch converting LSTM code to accept batches gives no prediction power

body:
I have an issue where if I feed data row by row, then my binary-classification LSTM model gets an AUC of 0.9+ within a few epochs. When I changed to code so that it accepts batches, the AUC gets stuck at 0.5 despite the loss decreasing.

I created a toy example of the LSTM model to try to test out this issue. I am suspecting that my model architecture is passing in the wrong information, because at some point it predicts everything as "positive", but I don't know where. Code is below:

Import and helper functions:

import pandas as pd
import numpy as np
from sklearn.metrics import roc_curve, auc, roc_auc_score

import multiprocessing

import pickle
import ast
import s3fs
import json

import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.nn.utils.rnn as rnn_utils

import random
from random import randint
import os
import math
import time

# returns the softmax output into a readable prediction
def get_max_prob_result(input, ix_to_word):
    return ix_to_word[get_index_of_max(input)]

# calculates the roc_auc score for the predictions
def get_auroc(truth, pred):
    assert len(truth) == len(pred)
    auc_score = roc_auc_score(np.array(truth),np.array(pred))
    return auc_score

def reorder_list(list, new_index_list):
    new_list = []
    for index in new_index_list:
        new_list.append(list[index])
    return new_list

def grab_batch(batch_size):
    seq=[]
    freq=[]
    target=[]
    time = []
    for k in range(batch_size):
        tseq, tfreq, ttime, ttarget = generate_patient()
        seq.append(tseq)
        freq.append(tfreq)
        time.append(ttime)
        target.append(ttarget)

    return seq, freq, time, target

Function to randomly generate data with structure that matches my real use case (note that I created a rule to define when target is positive that the model will learn):

events_to_ix = {'<PAD>':0,'non':1,'othernon':2,'neutral':3,'trigger':4}

final_seq = []
final_freq = []
final_time = []
final_target = []

dict_keys = list(events_to_ix.keys())[1:]

def generate_patient():
    num_seq = randint(1,100)
    patient_seq = []
    patient_freq = []
    patient_time = []
    patient_target = 0

    final_seq = []
    final_freq = []
    final_time = []

    for i in range(num_seq):
        step_seq = []
        step_freq = []
        step_time = []
        seq_length = randint(1,10)
        for k in range(seq_length):
            event = random.choice(dict_keys)
            if events_to_ix[event] in step_seq:
                continue
            step_seq.append(events_to_ix[event])
            step_freq.append(randint(1, 17))
            step_time.append(randint(0,(5+seq_length-k)))

        patient_seq.append(step_seq)
        patient_freq.append(step_freq)
        patient_time.append(step_time)

    for index, item in enumerate(patient_seq[-1]):
        if item == 4 and patient_freq[-1][index] > 15 and patient_time[-1][index] < 3:
            patient_target = 1
            break

    # Loop through each concet in each timestep
    for step_idx, step in enumerate(patient_seq):
        concepts = [0]*len(events_to_ix)
        frequencies = [0]*len(events_to_ix)
        times = [0]*len(events_to_ix)
        for event_idx, event in enumerate(step):

            # Convert the textual concepts into their index representation
            concepts[event] = event

            # Append the frequencies and time encodings to their appropriate position in the list of 0's
            frequencies[event] = patient_freq[step_idx][event_idx]
            times[event] = patient_time[step_idx][event_idx]

        final_seq.append(concepts)
        final_freq.append(frequencies)
        final_time.append(times)
        #final_static.append(patient_static)

    final_seq = torch.LongTensor(final_seq)
    final_freq = torch.FloatTensor(final_freq).view(-1,len(events_to_ix),1)
    final_time = torch.FloatTensor(final_time).view(-1,len(events_to_ix),1)
    #final_static = torch.FloatTensor(final_static)

    return final_seq, final_freq, final_time, patient_target

The LSTM model:

# Class containing the LSTM model initialization and feed-forward logic
class LSTMClassifier(nn.Module):
    # LSTM initialization
    def __init__(self, embedding_dim, hidden_dim, vocab_size, label_size):
        super(LSTMClassifier, self).__init__()

        # Setting the hidden layer dimension of the LSTM
        self.hidden_dim = hidden_dim
        # Initializing the embedding layer
        self.embeddings = nn.Embedding(vocab_size, embedding_dim-2)
        # Initializing the LSTM layer with one hidden layer
        self.lstm = nn.LSTM(((embedding_dim*vocab_size)), hidden_dim, num_layers = 1, batch_first=False)
        # Initializing linear linear that takes the hidden layer output
        self.hidden2label = nn.Linear(hidden_dim, label_size)

    # Defining the hidden state of the LSTM
    def init_hidden(self,batch_size):
        # the first is the hidden h
        # the second is the cell  c
        return [autograd.Variable(torch.zeros(1,batch_size, self.hidden_dim).cuda()),
                autograd.Variable(torch.zeros(1,batch_size, self.hidden_dim).cuda())]

    # Defining the feed forward logic of the LSTM. It contains:
    # 1. The embedding layer
    # 2. The LSTM layer with one hidden layer
    # 3. The softmax layer
    def forward(self, seq, freq, time_data, seq_lengths):

        # Grab the mini-batch length and max sequence length (pre-ordered)
        # (need to do this in the forward logic because of data parallelism and how the GPU's will split up the batch)
        sequence_length = seq.size()[1]
        batch_length = seq.size()[0]

        # reset the LSTM hidden state.
        # Must be done before you run a new batch. Otherwise the LSTM will treat a new batch as a continuation of a sequence
        self.hidden = self.init_hidden(batch_length)

        # This is the pass to the embedding layer.
        # The sequence is of dimension N and the output is N x Demb
        embeds = self.embeddings(seq)

        # Concatenate the embedding output with the time and frequency vectors
        embeds = torch.cat((embeds,freq), dim=3)
        embeds = torch.cat((embeds,time_data), dim=3)

        # Because the LSTM excepts a dimension of (sequence length, batch size, feature size), and we have (batch size, seq length, feature size),
        # we need to switch the first and second dimension so that we get the correct input format
        embeds = torch.transpose(embeds, 0, 1)

        # Flatten the embedding dimension so that the input to the LSTM remains 3D rather than 4D
        x = embeds.view(sequence_length, batch_length, -1)

        # pack the padded sequence so that paddings are ignored
        packed_x = torch.nn.utils.rnn.pack_padded_sequence(x, seq_lengths, batch_first=False)

        # Feed to the LSTM layer
        lstm_out, self.hidden = self.lstm(packed_x, self.hidden)

        # Feed the last layer of the LSTM into the linear layer
        y = self.hidden2label(self.hidden[0][-1])

        # Produce the softmax probabilities
        log_probs = F.log_softmax(y)

        return log_probs

Function to run a single epoch:

def train_epoch(model, loss_function, optimizer,batch_size, i):
    # Set model to training mode and initialize variables
    model.train()
    avg_loss = 0.0
    count = -1
    truth_res = []
    pred_res = []

    # Group the dataframe into dataframe chunks of length batch size and loop through each batch
    for j in range(1000):
        count += 1

        seq, freq, time_data, target = grab_batch(batch_size)

        # Sort the batches by descending size
        final_seq_ordered = sorted(enumerate(seq), key=lambda x: len(x[-1]), reverse=True)

        # Grab the original indices from final_seq and see how they are now ordered
        final_seq_indices = [item[0] for item in final_seq_ordered]

        # Grab the actual values from the tupled master list of concepts
        seq = [item[1] for item in final_seq_ordered]
        freq = reorder_list(freq, final_seq_indices)
        time_data = reorder_list(time_data, final_seq_indices)
        target = reorder_list(target, final_seq_indices)

        # Grab the list of lengths of sequences, for the purpose of packing the padded sequenes
        seq_lengths = torch.LongTensor(list(map(len, seq)))

        # Grab the targets into a list and append it into the truth_res list in order to measure AUC performance
        truth_res.extend(target)

        # Pad the sequences
        seq = rnn_utils.pad_sequence(seq, batch_first = True)
        freq = rnn_utils.pad_sequence(freq, batch_first = True)
        time_data = rnn_utils.pad_sequence(time_data, batch_first = True)

        # Put the padded sequences into Variable and Cuda cores
        seq = autograd.Variable(seq.cuda())
        freq = autograd.Variable(freq.cuda())
        time_data = autograd.Variable(time_data.cuda())
        target = autograd.Variable(torch.LongTensor(target).cuda())

        # Feed the tensor Variables into the model
        pred = model(seq,freq,time_data,seq_lengths)
        # Append the predictions into a list for future AUC evaluation
        pred_label = pred.detach().max(1)[1].cpu().numpy()
        pred_res.extend(pred_label)

        # Reset the model gradient
        model.zero_grad()
        # Compute the loss
        loss = loss_function(pred, target)
        # Backpropagate
        loss.backward()
        # Update weights
        optimizer.step()

        # Computes the average loss
        avg_loss += loss.detach().item()

    # Computes the AUC score
    auc_score = get_auroc(truth_res, pred_res)
    avg_loss /= (1000/batch_size)
    print('epoch: %d done! \n train avg_loss:%g , auc:%g' % (i, avg_loss, auc_score))

Main training loop:

#############################
### Set hyper parameters ###
############################
EMBEDDING_DIM = 32
HIDDEN_DIM = 50
EPOCH = 10
BATCH_SIZE = 16
best_val_auc = 0.5

model = LSTMClassifier(embedding_dim=EMBEDDING_DIM, hidden_dim=HIDDEN_DIM, vocab_size=(len(events_to_ix)), label_size=2)
model = torch.nn.DataParallel(model.cuda())

weights = [(26/1000), 1]
class_weights = torch.FloatTensor(weights).cuda()
loss_function = nn.NLLLoss(weight=class_weights,reduction="sum",ignore_index=0)
optimizer = optim.Adam(model.parameters(), lr=0.001)

no_up = 0

#####################################################
### Set loop to determine number of EPOCHs to run ###
#####################################################
for i in range(EPOCH):
    #############################################
    ### Run the training on the training data ###
    #############################################
    print('epoch: %d start!' % i)
    start = time.time()

    # Perform the training on the epoch
    train_epoch(model,loss_function, optimizer,BATCH_SIZE, i)

    print("1 epoch length of time")
    print(time.time() - start)


title: How to do batch filling in pytorch

body:
I need to fill images with other image patches during training. Since I’m training with mini-batch, is there any efficient way to do this?

For example, I have a mini-batch of images of size [B, 3, 128, 128]. I also have patches of size [B, 4, 3, 32, 32], where 4 is the number of patches. Besides, I have the bounding box indicating the location of patches of size [B, 4, 4]. How can I fill in the patches in a batch-wise way? You can ignore the overlap between patches.


title: Pytorch mask like Gluon npx.sequence_mask

body:
I am trying to do this tutorial: https://d2l.ai/chapter_attention-mechanisms/attention.html but in Pytorch and I'm stuck in this function:

npx.sequence_mask()

I tried using torch.masked_fill and masked_scatter but to no success. Namely, I want:

a = torch.randn(2, 2, 4)
b = torch.randn(2, 3)

and to get a result like npx.sequence_mask()
 sequence_mask documentation

 ([[[0.488994  , 0.511006  , 0.        , 0.        ],
    [0.43654838, 0.56345165, 0.        , 0.        ]],

   [[0.28817102, 0.3519408 , 0.3598882 , 0.        ],
    [0.29034293, 0.25239873, 0.45725834, 0.        ]]])

Could anyone help me out with any ideas?


title: Mutual information between input training image and difference between two logits

body:
I’m trying to replicate the experiment made by Rafael Muller et al. in “When does label smoothing help?” NIPS, 2019

The method is mainly descibed in section 4 and figure 6 for evaluating the effect of label smoothing in network distillation with classification tasks:
"We measure the mutual information between X and Y , where X is a discrete variable representing the index of the training example and Y is continuous representing the difference between two logits (out of K classes). The exact formula used for mutual information approximation is written in page 7.

I’m trying to replicate this experiment using ResNet18 on Cifar-10 dataset.

After each epoch, I run this piece of code to compute the mutual information as described in the paper. However, the computed mutual information final value is totally far away from its feasible range from 0 to log(N):
batchsize = 300, N = 600 (no. of training instances used in Mutual Inf. calculation), L = 100 (MonteCarlo samples), and classes_mi is a 1-d array containing indices of two classes which we are using during Mutual inf. calculation

with torch.no_grad():
    #Mean calculation
    for i in range(self.L):
        for batch_idx, (inputs, targets) in enumerate(trainloader_sub_transforms[i]):
            inputs, targets = inputs.to(self.device), targets.to(self.device)
            outputs = (self.net(inputs)).cpu().detach().numpy()
            outputs = np.absolute(outputs[:, classes_mi[0]] - outputs[:, classes_mi[1]])
            mu_x[batch_idx*300:batch_idx*300 + len(targets)] += outputs
    mu_x /= self.L
    print('--> Finish Mean Calculation ', mu_x[:10])
    #STD Calculation
    for batch_idx, (inputs, targets) in enumerate(trainloader_sub):
        inputs, targets = inputs.to(self.device), targets.to(self.device)
        outputs = (self.net(inputs)).cpu().detach().numpy()
        outputs = np.absolute(outputs[:, classes_mi[0]] - outputs[:, classes_mi[1]])
        var = np.sum((outputs - mu_x[batch_idx*300:batch_idx*300 + len(targets)]) ** 2)
    var /= self.N
    print('--> Finish VAR Calculation ', var)
    #Mutual Information Calculation
    mutual_info_value = np.zeros(self.N)
    term2 = 0
    for batch_idx, (inputs, targets) in enumerate(trainloader_sub):
        print('batch_idx2 = ', batch_idx)
        inputs, targets = inputs.to(self.device), targets.to(self.device)
        outputs = (self.net(inputs)).cpu().detach().numpy()
        outputs = np.absolute(outputs[:, classes_mi[0]] - outputs[:, classes_mi[1]])
        main_term = -(outputs - mu_x[batch_idx*300:batch_idx*300+len(targets)])**2 / (2 * var)
        term2  +=  np.sum(np.exp( main_term ))
        mutual_info_value[batch_idx*300:batch_idx*300 + len(targets)] = main_term
    term2 = np.log(term2)
    mutual_info_value -= term2
    print(mutual_info_value.shape, mutual_info_value[:10])
    mutual_info_value = np.sum(mutual_info_value)
sum += mutual_info_value
print('trial: ', t, ' --> MI = ', mutual_info_value)


title: Cannot get prediction from google AI platform with Pytorch

body:
I've deployed a custom Pytorch model to the Google AI platform for prediction, but when I try to make a prediction request with image data using gcloud tools I get the following error in response:

{
  "error": "Prediction failed: unknown error."
}

I've tried to encode my image data in b64 format or to place it into a multidimensional python array, by doing the following:

pil_im = Image.open('Pic512.png')
pil_im = pil_im.resize((224,224)).convert('RGB')
im_arr = np.asarray(pil_im)
py_arr = im_arr.tolist()
json_instance_1 = {'instances': py_arr}
with open('json_instance_1.json', 'w') as f:
    json.dump(json_instance_1, f)

I converted it into b64 like so, after adjusting my Predictor code accordingly:

with open('Pic512.png', 'rb') as f:
    byte_im = f.read()
json_instance = {'instances': {'b64': base64.b64encode(byte_im).decode()}}
with open('json_instance.json', 'w') as f:
    json.dump(json_instance, f)

I've tried converting with different file formats and similar methods, but all of them give me the same error.

My predictor module:

from facenet_pytorch import MTCNN, InceptionResnetV1, extract_face
import torch
from torchvision import transforms
from torch.nn import functional as F
from PIL import Image
# from sklearn.externals import joblib
import numpy as np
import os
import io
import base64

class MyPredictor(object):
    """An example Predictor for an AI Platform custom prediction routine."""

    def __init__(self, model, preprocessor, device):
        """Stores artifacts for prediction. Only initialized via `from_path`.
        """
        self._resnet = model
        self._mtcnn_mult = preprocessor
        self._device = device
        self.get_std_tensor = transforms.Compose([
            np.float32,
            np.uint8,
            transforms.ToTensor(),
        ])
        self.tensor2pil = transforms.ToPILImage(mode='RGB')
        self.trans_resnet = transforms.Compose([
            transforms.Resize((100, 100)),
            np.float32,
            transforms.ToTensor()
        ])

    def predict(self, instances, **kwargs):

        pil_transform = transforms.Resize((512, 512))

        imarr = np.uint8(np.array(instances))
        # img_bytes_string = io.BytesIO(base64.b64decode(instances))
        pil_im = Image.fromarray(imarr)
        # pil_im = Image.open(img_bytes_string)
        image = pil_im.convert('RGB')
        pil_im_512 = pil_transform(image)

        boxes, _ = self._mtcnn_mult.detect(pil_im_512)
        box = boxes[0]

        face_tensor = extract_face(pil_im_512, box, margin=40)
        std_tensor = self.get_std_tensor(face_tensor.permute(1, 2, 0))
        cropped_pil_im = self.tensor2pil(std_tensor)

        face_tensor = self.trans_resnet(cropped_pil_im)
        face_tensor4d = face_tensor.unsqueeze(0)
        face_tensor4d = face_tensor4d.to(self._device)

        self._resnet.eval()
        prediction = self._resnet(face_tensor4d)
        preds = F.softmax(prediction, dim=1).detach().numpy().reshape(-1)
        print('probability of (class1, class2) = ({:.4f}, {:.4f})'.format(preds[0], preds[1]))

        return {'probs':preds.tolist()}

    @classmethod
    def from_path(cls, model_dir):

        device_path = os.path.join(model_dir, 'device_cpu.pt')
        device = torch.load(device_path)

        model_path = os.path.join(model_dir, 'FullResNetRefinedExtra_no_norm_100x100_8634.pt')
        classifier = torch.load(model_path, map_location=device)

        mtcnn_path = os.path.join(model_dir, 'mtcnn_mult.pt')
        mtcnn_mult = torch.load(mtcnn_path)

        return cls(classifier, mtcnn_mult, device)

When I test the class locally everything works, so I assume it's a problem related the serialisation and deserialisation on the side of Google Platform. How can I resolve this issue?


title: Is this the right way to compute gradients of two losses from two different NN's in pytorch?

body:
I have a NN defined in pytorch and I have created two instances of that net as self.actor_critic_r1 and self.actor_critic_r2. I calculate the losses of each net i.e. loss1 and loss2 and I sum it up and calculate the grads in the following way,

loss_r1 = value_loss_r1 + action_loss_r1 - dist_entropy_r1 * args.entropy_coef
loss_r2 = value_loss_r2 + action_loss_r2 - dist_entropy_r2 * args.entropy_coef
self.optimizer_r1.zero_grad()
self.optimizer_r2.zero_grad()
loss = loss_r1 + loss_r2
loss.backward()
self.optimizer_r1.step()
self.optimizer_r2.step()
clip_grad_norm_(self.actor_critic_r1.parameters(), args.max_grad_norm)
clip_grad_norm_(self.actor_critic_r2.parameters(), args.max_grad_norm)

Alternatively, should I update the loss individually like this,

self.optimizer_r1.zero_grad()
(value_loss_r1 + action_loss_r1 - dist_entropy_r1 * args.entropy_coef).backward()
self.optimizer_r1.step()
clip_grad_norm_(self.actor_critic_r1.parameters(), args.max_grad_norm)
self.optimizer_r2.zero_grad()
(value_loss_r2 + action_loss_r2 - dist_entropy_r2 * args.entropy_coef).backward()
self.optimizer_r2.step()
clip_grad_norm_(self.actor_critic_r2.parameters(), args.max_grad_norm)

I am not sure if this the right approach to update a network with multiple loss please provide your suggestion.


title: Jensen-Shanon Divergence for two probability distributions in PyTorch

body:
How to calculate js Divergence for two probability distributions in PyTorch?
or how to add two distribution in pytorch?


title: How PyTorch model layer weights get initialized implicitly?

body: I basically defined a model with a Conv2D and linear layer with PyTorch and trained it with a sample dataset. The model seems to run and converge. But I am wondering I did not explicitly initialize the model layer weights (normal or Xavier). Does that mean when I call model.train() before each epoch training, the layer weights will get initialized randomly by default ? If so, how can I explicitly change the type of initializations ?


title: Pytorch Neural Network Trapped in Local Minimum

body:
I apologise in advance for the maths formatting - I do not have the reputation to post equations.
I have been using pytorch to build a neural network to learn the function,
f(x,y,t)=-x.10^y.cos(t)
but so far within a short number (~10) epochs the weights and biases all drop to 0 and never change from there. I believe this is because the network is stuck in a local minimum. The network is structured as:
    def __init__(self,learning_rate,beta_params):
    super(PINN, self).__init__()
    self.E_c=1.0

    self.lin1 = nn.Linear(3,20)
    self.lin2 = nn.Linear(20,20)
    self.lin3 = nn.Linear(20,20)
    self.lin4 = nn.Linear(20,20)
    self.lin5 = nn.Linear(20,1)

    self.optimizer = tr.optim.Adam(self.parameters(),lr=0.01)
    self.LossCrit = tr.nn.MSELoss()
    self.scaler=tr.cuda.amp.GradScaler()

with a forward function defined as:
    def forward(self, x):
    x = tr.sin(self.lin1(x))
    x = tr.tanh(self.lin2(x))
    x = tr.tanh(self.lin3(x))
    x = tr.tanh(self.lin4(x))
    x = self.lin5(x)

    return x

And the training cycle is defined as:
    def InitCycle(self, lb, ub, nTrain, batch_size):
    itNum = []
    itLoss = []
    for it in range(nTrain):
        #Choose a random selection of batch_size points from between lb and ub
        if it % 10 == 0:
            X_f = tr.rand((batch_size, 3)).cuda() * (ub - lb) + lb
            x = X_f[:,0:1]
            B_0 = tr.pow(10.0,X_f[:,2:3])
            t = X_f[:,1:2]

            #define a target
            f_Target = (- x * tr.cos(t))

        #zero the gradients and evaluate the prediction of the neural network
        self.optimizer.zero_grad()
        f_pred = self.forward(X_f)

        #define an 'output' which divides the neural network output by B_0 to keep all values close to unity and calculate loss and scaled gradients.
        output = f_pred / B_0
        loss = self.LossCrit(output,f_Target)
        self.scaler.scale(loss).backward()

        #record loss every 10 iterations
        if it % 10 == 0:
            if it > 0:
                itNum.append(np.log10(float(it)))
                itLoss.append(np.log10(loss.item()))
            print('loss at it. %d:  %.8e, lr at it. %d: %.2e' % (
            it, loss.item(), it, self.optimizer.param_groups[0]['lr']))

        #update the weights and biases
        self.scaler.step(self.optimizer)
        self.scaler.update()

    #save the loss file at the end of the training run
    LFile = pd.DataFrame({'it#': itNum, 'Loss': itLoss})
    LFile.to_csv("PATH.csv")

I have tried every pytorch optimizer with various learning rates and hyperparameters but they all get stuck in the local minimum at {W,b}=0. I have also tried a different loss function which compares the reciprocals (with an added eps for numerical stability) but this also gets stuck at {W,b}=0.
The reason I am using gradient scaling is because with the default pytorch initialization my initial gradients are of the order 1E8 without scaling. I have tried using gradient clipping instead.
I've also tried to train a slightly different network to learn the function:
f(x,y,t)=-x.y.cos(t)
but this didn't work either so I stuck with the original form since I want the network to cover 6 orders of magnitude in 10^y and wanted to keep the values of the input y close to unity (y=-3->3).
I chose the sinusoidal activation function for the first layer in an effort to make it easier for the network to learn the cos(t) term but using a tanh activation function does not work either.
I am now out of ideas for how to get this to train and was hoping someone might be able to shed some light on the situation.


title: Pytorch lightning metrics: ValueError: preds and target must have same number of dimensions, or one additional dimension for preds

body:
Googling this gets you no where, so I decided to help future me and others by posting this as a searchable question.

def __init__():
    ...
    self.val_acc = pl.metrics.Accuracy()

def validation_step(self, batch, batch_index):
    ...
    self.val_acc.update(log_probs, label_batch)

gives
ValueError: preds and target must have same number of dimensions, or one additional dimension for preds

for log_probs.shape == (16, 4) and for label_batch.shape == (16, 4)
What's the issue?


title: Calling the forward method in PyTorch vs. calling the model instance

body:
A lot of the PyTorch tutorials I've been viewing do something like this.
Define model:
class Network(nn.Module):
    def __init__():
        super().__init__()
        self.conv1 = ..
        ...

    def forward(x)
        ...
    ...

Once the Network has been instantiated (net = Network()), the people in the tutorials write net(input_data) instead of net.forward(input_data).
I tried net.forward() and it gives the same results as net().
Why is this a common practice, and also why does this work?


title: PyTorch Autograd Differentiated Tensors appears to not have been used in the graph

body:
I'm trying to improve a CNN I made by implementing a weighted loss method described in this paper. To do this, I looked into this notebook which implements the pseudo-code of the method described in the paper.
When translating their code to my model, I ran into the error RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior when using torch.autograd.grad().
My code and the error is in the second to last line:
for epoch in range(1): #tqdm(range(params['epochs'])):
  model.train()
  text_t, labels_t = next(iter(train_iterator))
  text_t = to_var(text_t, requires_grad=False)
  labels_t = to_var(labels_t, requires_grad=False)
  dummy = L2RWCNN(INPUT_DIM, EMBEDDING_DIM, N_FILTERS, FILTER_SIZES, OUTPUT_DIM,
                DROPOUT, PAD_IDX)
  dummy.state_dict(model.state_dict())
  dummy.cuda()
  y_f_hat = dummy(text_t)
  cost = F.binary_cross_entropy_with_logits(y_f_hat.squeeze(), labels_t, reduce = False)
  eps = to_var(torch.zeros(cost.size()))
  l_f_meta = torch.sum(cost * eps)
  dummy.zero_grad()
  num_params = 0
  grads = torch.autograd.grad(l_f_meta, (dummy.params()), create_graph = True)
  with torch.no_grad():
    for p, grad in zip(dummy.parameters(), grads):
      tmp = p - params['lr'] * grad
      p.copy_(tmp)
  text_v, labels_v = next(iter(valid_iterator))
  y_g_hat = dummy(text_v)
  l_g_meta = F.binary_cross_entropy_with_logits(y_g_hat.squeeze(), labels_v, reduce = False)
  l_g_meta = torch.sum(l_g_meta)
  grad_eps = torch.autograd.grad(l_g_meta, eps, only_inputs=True)[0]
  print(grad_eps)

I think the error is because eps was not in any previous torch.autograd.grad() calls. I tried the suggested solution of setting allow_unused=True but that resulted in a None value. I looked at this post to find a solution, but the method that fixed the problem here (don't slice the tensors) doesn't work for me because I'm not passing in any partial variables. I also tried setting create_graph = False in my first autograd.grad() call, but that didn't fix the issue. Does anyone have a solution?
My full code if that's needed
EDIT:
Created a new post with a different phrasing to the question here


title: How to load a pre-trained PyTorch model?

body:
I'm following this guide on saving and loading checkpoints. However, something is not right. My model would train and the parameters would correctly update during the training phase. However, there seem to be a problem when I load the checkpoints. That is, the parameters are not being updated anymore.
My model:
import torch
import torch.nn as nn
import torch.optim as optim

PATH = 'test.pt'

class model(nn.Module):
        def __init__(self):
            super(model, self).__init__()
            self.a = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.b = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            self.c = torch.nn.Parameter(torch.rand(1, requires_grad=True))
            #print(self.a, self.b, self.c)

        def load(self):
          try:
            checkpoint = torch.load(PATH)
            print('\nloading pre-trained model...')
            self.a = checkpoint['a']
            self.b = checkpoint['b']
            self.c = checkpoint['c']
            optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
            print(self.a, self.b, self.c)
          except: #file doesn't exist yet
            pass

        @property
        def b_opt(self):
            return torch.tanh(self.b)*2

        def train(self):
          print('training...')
          for epoch in range(3):
            print(self.a, self.b, self.c)
            for r in range(5):
              optimizer.zero_grad()
              loss = torch.square(5 * (r > 2) * (3) - model_net.a * torch.sigmoid((r - model_net.b)) * (model_net.c))
              loss.backward(retain_graph=True) #accumulate gradients

            #checkpoint save
            torch.save({
                'model': model_net.state_dict(),
                'a': model_net.a,
                'b': model_net.b,
                'c': model_net.c,
                'optimizer_state_dict': optimizer.state_dict(),
                }, PATH)

            optimizer.step()

model_net = model()
optimizer = optim.Adam(model_net.parameters(), lr = 0.1)

print(model_net.a)
print(model_net.b)
print(model_net.c)

This prints
Parameter containing:
tensor([0.4214], requires_grad=True)
Parameter containing:
tensor([0.3862], requires_grad=True)
Parameter containing:
tensor([0.8812], requires_grad=True)

I then run model_net.train() to see that the parameters are being updated and this outputs:
training...
Parameter containing:
tensor([0.9990], requires_grad=True) Parameter containing:
tensor([0.1580], requires_grad=True) Parameter containing:
tensor([0.1517], requires_grad=True)
Parameter containing:
tensor([1.0990], requires_grad=True) Parameter containing:
tensor([0.0580], requires_grad=True) Parameter containing:
tensor([0.2517], requires_grad=True)
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)

Running model_net.load() outputs:
loading pre-trained model...
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)

And lastly, running model_net.train() again outputs:
training...
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)
Parameter containing:
tensor([1.1974], requires_grad=True) Parameter containing:
tensor([-0.0404], requires_grad=True) Parameter containing:
tensor([0.3518], requires_grad=True)

Update 1.
Following @jhso suggestion I changed my load to:
def load(self):
  try:
    checkpoint = torch.load(PATH)
    print('\nloading pre-trained model...')
    self.load_state_dict(checkpoint['model'])
    self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    print(self.a, self.b, self.c)
  except: #file doesn't exist yet
    pass

This almost seems to work (the network is training now), but I don't think the optimizer is loading correctly. That is because it doesn't go pass the line self.optimizer.load_state_dict(checkpoint['optimizer_state_dict']).
You can see that since it doesn't print(self.a, self.b, self.c) when I run
model_net.load()


title: How could I feed a custom image into this model?

body:
I've been following a course online and one of the exercises was to create a simple image detection model (using MNIST data) to detect written numbers. I've been trying to load a custom image I drew in (128x128 jpg) but I can't seem to figure it out. I'm really close, but I think I'm just confused about what parameters the model takes in. Any help would be appreciated!!

Here is my code


title: Pytorch: multiple datasets with multiple losses

body: I am using multiple datasets. I have multiple losses, each of which must be evaluated on a subset of these datasets. I want to generate a batch from each dataset, and evaluate each loss on all of its appropriate batches. Some of the losses are pairwise (need to load pairs of corresponding datapoints) whereas others are computed on single datapoints. I need to design this in such a way that is open to easily adding new datasets. Is there any pytorch builtin that would help with this? What is the best way to design this in pytorch? Thanks in advance.


title: Tensor Inverse in parallel over multiple GPUs using PyTorch

body:
I want to run over multiple GPUs in parallel torch.inverse(), and am not able to.

I saw this post Matmul on multiple GPUs , which goes over the process for matmul. It shows that if you have multiple tensors allocated to each GPU matmul will be run in parallel. I was able to replicate this behavior for matmul but when I try to do the same thing for torch.inverse() it seems to run sequentially when I check "watch nvidia-smi". Also when I replace the torch.inverse() function with the torch fft function I get parallel GPU usage. Any ideas?

import torch

ngpu = torch.cuda.device_count()
# This is the allocation to each GPU.
lis = []
for i in range(ngpu):
    lis.append(torch.rand(5000,5000,device = 'cuda:'+ str(i)))

# per the matmul on multiple GPUs post this should already be in parallel
# but doesnt seem to be based on watch nvidia-smi
C_ = []
for i in range(ngpu):
    C_.append(torch.inverse(lis[i]))

Edit: This can be compared to the FFT code(below) and the Matmul code in the link above.

import torch

ngpu = torch.cuda.device_count()
# This is the allocation to each GPU.
lis = []
for i in range(ngpu):
    lis.append(torch.rand(5000,5000,2,device = 'cuda:'+ str(i)))

# per the matmul on multiple GPUs post this should already be in parallel
# but doesnt seem to be based on watch nvidia-smi
C_ = []
for i in range(ngpu):
    C_.append(torch.fft(lis[i],2))


title: Pytorch multiplication broadcasting

body:
I have two PyTorch tensors A with shape [30, 11, 32, 64] and B with shape [30, 11, 89].
My goal is to create a tensor C with shape [30, 11, 89, 32, 64] where the tensor A should be extended and replicated in the third dimension and then be multiplied with tensor B at the appropriate dimensions.

How can this be accomplished in PyTorch?


title: How does one implement a meta-trainable step size in Pytorch?

body:
I want to implement a (meta) trainable step size. I tried it with this post:
https://discuss.pytorch.org/t/how-does-one-have-the-parameters-of-a-model-not-be-leafs/70076/17
and with the higher library (https://github.com/facebookresearch/higher) with no luck...
I tried:
eta = torch.tensor([0.5], requires_grad=True).view(1)
inner_opt = torch.optim.Adam(child_model.parameters(), lr=eta)
#meta_params = itertools.chain(child_model.parameters(),eta.parameters())
meta_params = itertools.chain(child_model.parameters())
meta_opt = torch.optim.Adam(meta_params, lr=1e-3)
# do meta-training/outer training minimize outerloop: min_{theta} sum_t L^val( theta^{T} - eta* Grad L^train(theta^{T}) )
nb_outer_steps = 10 # note, in this case it's the same as number of meta-train steps (but it's could not be the same depending how you loop through the val set)
for outer_i, (outer_inputs, outer_targets) in enumerate(testloader, 0):
    meta_opt.zero_grad()
    if outer_i >= nb_outer_steps:
        break
    # do inner-training/MAML; minimize innerloop: theta^{T} - eta* Grad L^train(theta^{T}) ~ argmin L^train(theta)
    nb_inner_steps = 3
    with higher.innerloop_ctx(child_model, inner_opt) as (fmodel, diffopt):

with error:
Exception has occurred: RuntimeError
Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment

which wouldn't work anyway cuz eta might become negative suddenly so I really want to cap it with a sigmoid function but had to try something...

It thinks my step size NN is not in the graph but it is because of this line of code:
            p_new = p + lr*g
            group['params'][p_idx] = p_new

but somehow that is not enough to have gradients...
Full script self contained script:
import torch
import torch.nn as nn
from torch.optim.optimizer import Optimizer

import higher
from higher.optim import DifferentiableOptimizer
from higher.optim import DifferentiableSGD

import torchvision
import torchvision.transforms as transforms

from torchviz import make_dot

import copy

import itertools

from collections import OrderedDict

#mini class to add a flatten layer to the ordered dictionary
class Flatten(nn.Module):
    def forward(self, input):
        '''
        Note that input.size(0) is usually the batch size.
        So what it does is that given any input with input.size(0) # of batches,
        will flatten to be 1 * nb_elements.
        '''
        batch_size = input.size(0)
        out = input.view(batch_size,-1)
        return out # (batch_size, *size)

def get_cifar10():
    transform = transforms.Compose(
        [transforms.ToTensor(),
        transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

    trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                            download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                            shuffle=True, num_workers=2)

    testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                        download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                            shuffle=False, num_workers=2)
    return trainloader, testloader

class MySGD(Optimizer):

    def __init__(self, params, eta, prev_lr):
        defaults = {'eta':eta, 'prev_lr':prev_lr}
        super().__init__(params, defaults)

class TrainableSGD(DifferentiableOptimizer):

    def _update(self, grouped_grads, **kwargs):
        prev_lr = self.param_groups[0]['prev_lr']
        eta = self.param_groups[0]['eta']
        # start differentiable & trainable update
        zipped = zip(self.param_groups, grouped_grads)
        lr = 0.1*eta(prev_lr).view(1)
        for group_idx, (group, grads) in enumerate(zipped):
            for p_idx, (p, g) in enumerate(zip(group['params'], grads)):
                if g is None:
                    continue
                #group['params'][p_idx] = _add(p, -group['lr'], g)
                p_new = p + lr*g
                group['params'][p_idx] = p_new
        # fake returns
        self.param_groups[0]['prev_lr'] = lr

higher.register_optim(MySGD, TrainableSGD)

def main():
    # get dataloaders
    trainloader, testloader = get_cifar10()
    criterion = nn.CrossEntropyLoss()

    child_model = nn.Sequential(OrderedDict([
            ('conv1', nn.Conv2d(in_channels=3,out_channels=2,kernel_size=5)),
            ('relu1', nn.ReLU()),
            ('Flatten', Flatten()),
            ('fc', nn.Linear(in_features=28*28*2,out_features=10) )
        ]))

    hidden = torch.randn(size=(1,1),requires_grad=True)
    print(f'-> hidden = {hidden}')
    eta = nn.Sequential(OrderedDict([
        ('fc', nn.Linear(1,1)),
        ('sigmoid', nn.Sigmoid())
    ]))
    inner_opt = MySGD(child_model.parameters(), eta=eta, prev_lr=hidden)
    meta_params = itertools.chain(child_model.parameters(),eta.parameters())
    #meta_params = itertools.chain(eta.parameters(),[hidden])
    meta_opt = torch.optim.Adam(meta_params, lr=1e-3)
    # do meta-training/outer training minimize outerloop: min_{theta} sum_t L^val( theta^{T} - eta* Grad L^train(theta^{T}) )
    print()
    nb_outer_steps = 1 # note, in this case it's the same as number of meta-train steps (but it's could not be the same depending how you loop through the val set)
    for outer_i, (outer_inputs, outer_targets) in enumerate(testloader, 0):
        meta_opt.zero_grad()
        if outer_i >= nb_outer_steps:
            break
        # do inner-training/MAML; minimize innerloop: theta^{T} - eta * Grad L^train(theta^{T}) ~ argmin L^train(theta)
        nb_inner_steps = 3
        #with higher.innerloop_ctx(child_model, inner_opt, copy_initial_weights=False) as (fmodel, diffopt):
        with higher.innerloop_ctx(child_model, inner_opt) as (fmodel, diffopt):
            for inner_i, (inner_inputs, inner_targets) in enumerate(trainloader, 0):
                if inner_i >= nb_inner_steps:
                    break
                logits = fmodel(inner_inputs)
                inner_loss = criterion(logits, inner_targets)
                print(f'--> inner_i = {inner_i}')
                print(f'inner_loss^<{inner_i}>: {inner_loss}')
                print(f'lr^<{inner_i-1}> = {diffopt.param_groups[0]["prev_lr"]}')
                diffopt.step(inner_loss) # changes params P[t+1] using P[t] and loss[t] in a differentiable manner
                print(f'lr^<{inner_i}> = {diffopt.param_groups[0]["prev_lr"]}')
                print()
            # compute the meta-loss L^val( theta^{T} - eta* Grad L^train(theta^{T}) )
            outer_outputs = fmodel(outer_inputs)
            meta_loss = criterion(outer_outputs, outer_targets) # L^val
            make_dot(meta_loss).render('meta_loss',format='png')
            meta_loss.backward()
            #grad_of_grads = torch.autograd.grad(outputs=meta_loss, inputs=eta.parameters()) # dmeta_loss/dw0
            print(f'----> outer_i = {outer_i}')
            print(f'-> outer_loss/meta_loss^<{outer_i}>: {meta_loss}')
            print(f'child_model.fc.weight.grad = {child_model.fc.weight.grad}')
            print(f'hidden.grad = {hidden.grad}')
            print(f'eta.fc.weight = {eta.fc.weight.grad}')
            meta_opt.step() # meta-optimizer step: more or less theta^<t> := theta^<t> - meta_eta * Grad L^val( theta^{T} - eta* Grad L^train(theta^{T}) )

if __name__ == "__main__":
    main()
    print('---> Done\a')

notice the None's:
Files already downloaded and verifiedFiles already downloaded and verified
-> hidden = tensor([[0.8459]], requires_grad=True)
--> inner_i = 0
inner_loss^<0>: 2.2696359157562256
lr^<-1> = tensor([[0.8459]], requires_grad=True)
lr^<0> = tensor([0.0567], grad_fn=)
--> inner_i = 1
inner_loss^<1>: 2.0114920139312744
lr^<0> = tensor([0.0567], grad_fn=)
lr^<1> = tensor([0.0720], grad_fn=)
--> inner_i = 2
inner_loss^<2>: 2.3866422176361084
lr^<1> = tensor([0.0720], grad_fn=)
lr^<2> = tensor([0.0717], grad_fn=)
----> outer_i = 0
-> outer_loss/meta_loss^<0>: 4.021303176879883
child_model.fc.weight.grad = None
hidden.grad = None
eta.fc.weight = None
---> Done

related:

pytorch forum: https://discuss.pytorch.org/t/implement-a-meta-trainable-step-size/70396
gitissue: https://github.com/facebookresearch/higher/issues/32
related SO Q: How does one have parameters in a pytorch model not be leafs and be in the computation graph?


title: Pytorch model stuck at 0.5 though loss decreases consistently

body:
This is using PyTorch

I have been trying to implement UNet model on my images, however, my model accuracy is always exact 0.5. Loss does decrease.

I have also checked for class imbalance. I have also tried playing with learning rate. Learning rate affects loss but not the accuracy.

My architecture below ( from here )

""" `UNet` class is based on https://arxiv.org/abs/1505.04597

The U-Net is a convolutional encoder-decoder neural network.
Contextual spatial information (from the decoding,
expansive pathway) about an input tensor is merged with
information representing the localization of details
(from the encoding, compressive pathway).

Modifications to the original paper:
(1) padding is used in 3x3 convolutions to prevent loss
    of border pixels
(2) merging outputs does not require cropping due to (1)
(3) residual connections can be used by specifying
    UNet(merge_mode='add')
(4) if non-parametric upsampling is used in the decoder
    pathway (specified by upmode='upsample'), then an
    additional 1x1 2d convolution occurs after upsampling
    to reduce channel dimensionality by a factor of 2.
    This channel halving happens with the convolution in
    the tranpose convolution (specified by upmode='transpose')

    Arguments:
        in_channels: int, number of channels in the input tensor.
                     Default is 3 for RGB images. Our SPARCS dataset is 13 channel.
              depth: int, number of MaxPools in the U-Net. During training, input size needs to be
                     (depth-1) times divisible by 2
        start_filts: int, number of convolutional filters for the first conv.
            up_mode: string, type of upconvolution. Choices: 'transpose' for transpose convolution

"""

class UNet(nn.Module):

    def __init__(self, num_classes, depth, in_channels, start_filts=16, up_mode='transpose', merge_mode='concat'):

        super(UNet, self).__init__()

        if up_mode in ('transpose', 'upsample'):
            self.up_mode = up_mode
        else:
            raise ValueError("\"{}\" is not a valid mode for upsampling. Only \"transpose\" and \"upsample\" are allowed.".format(up_mode))

        if merge_mode in ('concat', 'add'):
            self.merge_mode = merge_mode
        else:
            raise ValueError("\"{}\" is not a valid mode for merging up and down paths.Only \"concat\" and \"add\" are allowed.".format(up_mode))

        # NOTE: up_mode 'upsample' is incompatible with merge_mode 'add'
        if self.up_mode == 'upsample' and self.merge_mode == 'add':
            raise ValueError("up_mode \"upsample\" is incompatible with merge_mode \"add\" at the moment "
                             "because it doesn't make sense to use nearest neighbour to reduce depth channels (by half).")

        self.num_classes = num_classes
        self.in_channels = in_channels
        self.start_filts = start_filts
        self.depth = depth

        self.down_convs = []
        self.up_convs = []

        # create the encoder pathway and add to a list
        for i in range(depth):
            ins = self.in_channels if i == 0 else outs
            outs = self.start_filts*(2**i)
            pooling = True if i < depth-1 else False

            down_conv = DownConv(ins, outs, pooling=pooling)
            self.down_convs.append(down_conv)

        # create the decoder pathway and add to a list
        # - careful! decoding only requires depth-1 blocks
        for i in range(depth-1):
            ins = outs
            outs = ins // 2
            up_conv = UpConv(ins, outs, up_mode=up_mode, merge_mode=merge_mode)
            self.up_convs.append(up_conv)

        self.conv_final = conv1x1(outs, self.num_classes)

        # add the list of modules to current module
        self.down_convs = nn.ModuleList(self.down_convs)
        self.up_convs = nn.ModuleList(self.up_convs)

        self.reset_params()

    @staticmethod
    def weight_init(m):
        if isinstance(m, nn.Conv2d):

            #https://prateekvjoshi.com/2016/03/29/understanding-xavier-initialization-in-deep-neural-networks/
            ##Doc: https://pytorch.org/docs/stable/nn.init.html?highlight=xavier#torch.nn.init.xavier_normal_
            init.xavier_normal_(m.weight)
            init.constant_(m.bias, 0)

    def reset_params(self):
        for i, m in enumerate(self.modules()):
            self.weight_init(m)

    def forward(self, x):
        encoder_outs = []

        # encoder pathway, save outputs for merging
        for i, module in enumerate(self.down_convs):
            x, before_pool = module(x)
            encoder_outs.append(before_pool)

        for i, module in enumerate(self.up_convs):
            before_pool = encoder_outs[-(i+2)]
            x = module(before_pool, x)

        # No softmax is used. This means we need to use
        # nn.CrossEntropyLoss is your training script,
        # as this module includes a softmax already.
        x = self.conv_final(x)
        return x

Parameters are :

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
x,y = train_sequence[0] ; batch_size = x.shape[0]
model = UNet(num_classes = 2, depth=5, in_channels=5, merge_mode='concat').to(device)
optim = torch.optim.Adam(model.parameters(),lr=0.01, weight_decay=1e-3)
criterion = nn.BCEWithLogitsLoss() #has sigmoid internally
epochs = 1000

The function for training is :

import torch.nn.functional as f

def train_model(epoch,train_sequence):
    """Train the model and report validation error with training error
    Args:
        model: the model to be trained
        criterion: loss function
        data_train (DataLoader): training dataset
    """
    model.train()
    for idx in range(len(train_sequence)):
        X, y = train_sequence[idx]
        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device)

        outputs = model(images)
        print(masks.shape, outputs.shape)
        loss = criterion(outputs, masks)
        optim.zero_grad()
        loss.backward()
        # Update weights
        optim.step()
    # total_loss = get_loss_train(model, data_train, criterion)

My function for calculating loss and accuracy is below:

def get_loss_train(model, train_sequence):
    """
        Calculate loss over train set
    """
    model.eval()
    total_acc = 0
    total_loss = 0
    for idx in range(len(train_sequence)):
        with torch.no_grad():
            X, y = train_sequence[idx]
            images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
            masks = Variable(torch.from_numpy(y)).to(device)

            outputs = model(images)
            loss = criterion(outputs, masks)
            preds = torch.argmax(outputs, dim=1).float()
            acc = accuracy_check_for_batch(masks.cpu(), preds.cpu(), images.size()[0])
            total_acc = total_acc + acc
            total_loss = total_loss + loss.cpu().item()
    return total_acc/(len(train_sequence)), total_loss/(len(train_sequence))

Edit : Code which runs (calls) the functions:

for epoch in range(epochs):
    train_model(epoch, train_sequence)
    train_acc, train_loss = get_loss_train(model,train_sequence)
    print("Train Acc:", train_acc)
    print("Train loss:", train_loss)

Can someone help me identify as why is accuracy always exact 0.5?

Edit-2:

As asked accuracy_check_for_batch function is here:

def accuracy_check_for_batch(masks, predictions, batch_size):
    total_acc = 0
    for index in range(batch_size):
        total_acc += accuracy_check(masks[index], predictions[index])
    return total_acc/batch_size

and

def accuracy_check(mask, prediction):
    ims = [mask, prediction]
    np_ims = []
    for item in ims:
        if 'str' in str(type(item)):
            item = np.array(Image.open(item))
        elif 'PIL' in str(type(item)):
            item = np.array(item)
        elif 'torch' in str(type(item)):
            item = item.numpy()
        np_ims.append(item)

    compare = np.equal(np_ims[0], np_ims[1])
    accuracy = np.sum(compare)

    return accuracy/len(np_ims[0].flatten())


title: What is the difference between tensor[:] and tensor in pytorch?

body:
import numpy as np
import time
features, labels = d2l.get_data_ch7()

def init_adam_states():
    v_w, v_b = torch.zeros((features.shape[1], 1),dtype=torch.float32), torch.zeros(1, dtype=torch.float32)
    s_w, s_b = torch.zeros((features.shape[1], 1),dtype=torch.float32), torch.zeros(1, dtype=torch.float32)
    return ((v_w, s_w), (v_b, s_b))

def adam(params, states, hyperparams):
    beta1, beta2, eps = 0.9, 0.999, 1e-6
    for p, (v, s) in zip(params, states):
        v[:] = beta1 * v + (1 - beta1) * p.grad.data
        s = beta2 * s + (1 - beta2) * p.grad.data**2
        v_bias_corr = v / (1 - beta1 ** hyperparams['t'])
        s_bias_corr = s / (1 - beta2 ** hyperparams['t'])
        p.data -= hyperparams['lr'] * v_bias_corr / (torch.sqrt(s_bias_corr) + eps)
    hyperparams['t'] += 1

def train_ch7(optimizer_fn, states, hyperparams, features, labels, batch_size=10, num_epochs=2):
    # 初始化模型
    net, loss = d2l.linreg, d2l.squared_loss

    w = torch.nn.Parameter(torch.tensor(np.random.normal(0, 0.01, size=(features.shape[1], 1)), dtype=torch.float32),
                           requires_grad=True)
    b = torch.nn.Parameter(torch.zeros(1, dtype=torch.float32), requires_grad=True)

    def eval_loss():
        return loss(net(features, w, b), labels).mean().item()

    ls = [eval_loss()]
    data_iter = torch.utils.data.DataLoader(torch.utils.data.TensorDataset(features, labels), batch_size, shuffle=True)

    for _ in range(num_epochs):
        start = time.time()
        print(w)
        print(b)
        for batch_i, (X, y) in enumerate(data_iter):
            l = loss(net(X, w, b), y).mean()  # 使⽤平均损失

            # 梯度清零
            if w.grad is not None:
                w.grad.data.zero_()
                b.grad.data.zero_()

            l.backward()
            optimizer_fn([w, b], states, hyperparams)  # 迭代模型参数
            if (batch_i + 1) * batch_size % 100 == 0:
                ls.append(eval_loss())  # 每100个样本记录下当前训练误差

    # 打印结果和作图
    print('loss: %f, %f sec per epoch' % (ls[-1], time.time() - start))
    d2l.set_figsize()
    d2l.plt.plot(np.linspace(0, num_epochs, len(ls)), ls)
    d2l.plt.xlabel('epoch')
    d2l.plt.ylabel('loss')
train_ch7(adam, init_adam_states(), {'lr': 0.01, 't': 1}, features, labels)

I want to implement the Adam algorithm in the follow code and I feel confused in the function named adam.

v = beta1 * v + (1 - beta1) * p.grad.data
s = beta2 * s + (1 - beta2) * p.grad.data**2

when I use the follow code, the loss function curve is figure 1.
figure 1

v[:] = beta1 * v + (1 - beta1) * p.grad.data
s = beta2 * s + (1 - beta2) * p.grad.data**2
or
v = beta1 * v + (1 - beta1) * p.grad.data
s[:] = beta2 * s + (1 - beta2) * p.grad.data**2

when I use the follow code, the loss function curve is figure 2.
figure 2

v[:] = beta1 * v + (1 - beta1) * p.grad.data
s[:] = beta2 * s + (1 - beta2) * p.grad.data**2

when I use the follow code, the loss function curve is figure 3.
figure 3

The loss function curve in case 3 has always been smoother than that in case 1.
The loss function curve in case 2 sometimes can't converge.

Why is different?


title: In PyTorch, what's the difference between training an RNN to predict the last word given a sequence, vs predicting the entire sequence shifted?

body:
Let's say I'm trying to train an RNN language model in PyTorch. Suppose I iterate over batches of word sequences, and that each training batch tensor has the following shape:

data.shape = [batch_size, sequence_length, vocab_dim]

My question is, what's the difference between using only the last word in each sequence as the target label:

X = data[:,:-1]
y = data[:,-1]

and training to minimize loss using a softmax prediction of the last word,

vs setting the target to be the entire sequence shifted right:

X = data[:,:-1]
y = data[:,1:]

and training to minimize the sum of losses of each predicted word in the shifted sequence?

What's the correct approach here? I feel like i've seen both examples online. Does this also have to do with loop unrolling vs BPTT?


title: Problems using pretrained ResNet50 in PyTorch to solve CIFAR10 Dataset

body:
I got the following error using a pretrained ResNet50 in PyTorch:

RuntimeError
Traceback (most recent call last)
<ipython-input-14-8f0d0641ef12> in <module>()
      28         # Update parameters
      29         optimizer.zero_grad()
 ---> 30         loss.backward()
      31         optimizer.step()
      32

 1 frames
 /usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in
 backward(tensors, grad_tensors, retain_graph, create_graph,
 grad_variables)
      98     Variable._execution_engine.run_backward(
      99         tensors, grad_tensors, retain_graph, create_graph,
 --> 100         allow_unreachable=True)  # allow_unreachable flag
     101
     102

 RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Notebook is in this link: https://colab.research.google.com/drive/1k40NNulSIS6ANagopSPBH4Xty_Cw39qC?usp=sharing


title: Best way to convert a tensor from a condensed representation

body:
I have a Tensor that is in a condensed format representing a sparse 3-D matrix. I need to convert it to a normal matrix (the one that it is actually representing).
So, in my case, each row of any 2-D slice of my matrix can only contain one non-zero element. As data, then, I have for each of these rows, the value, and the index where it appears. For example, the tensor
inp = torch.tensor([[ 1,  2],
 [ 3,  4],
 [-1,  0],
 [45,  1]])

represents a 4x5 matrix (first dimension comes from the first dimension of the tensor, second comes from the metadata) A, where A[0][2] = 1, A[1][4] = 3, A[2][0] = -1, A[3][1] = 45.
This is just one 2-D slice of my Matrix, and I have a variable number of these.
I was able to do this for a 2-D slice as described above in the following way using sparse_coo_tensor:
>>> torch.sparse_coo_tensor(torch.stack([torch.arange(0, 4), inp.t()[1]]), inp.t()[0], [4,5]).to_dense()
tensor([[ 0,  0,  1,  0,  0],
        [ 0,  0,  0,  0,  3],
        [-1,  0,  0,  0,  0],
        [ 0, 45,  0,  0,  0]])

Is this the best way to accomplish this? Is there a simpler, more readable alternative?
How do I extend this to a 3-D matrix without looping?
For a 3-D matrix, you can imagine the input to be something like
inp_list = torch.stack([inp, inp, inp, inp])
and the desired output would be the above output stacked 4 times.
I feel like I should be able to do something if I create an index array correctly, but I cannot think of a way to do this without using some kind of looping.


title: input output mismatch export pytorch to onnx

body:
I am trying to convert this model https://github.com/minivision-ai/photo2cartoon
to onnx
I used this to export
https://gist.github.com/antonpaquin/2614581008c5c84cd4519a4f75735eb7
but the input output does not match
Information
light : True
dataset : photo2cartoon
batch_size : 1
iteration per epoch : 1000000
Discriminator
discriminator layer : 6
Weight
adv_weight : 1
cycle_weight : 50
faceid_weight : 1
identity_weight : 10
cam_weight : 1000
rho_clipper: 1.0
w_clipper: 1.0
[*] Load models/photo2cartoon_weights.pt Success
torch.Size([1, 3, 256, 256])
torch.Size([1, 2])
torch.Size([1, 1, 64, 64]


title: Cannot use pip install pytorch-pretrained-bert

body:
I was trying to use "pip install pytorch-pretrained-bert", but I met a unseen problem. It says that the version of botocore is not satisfied. They need botocore<1.18.0,>=1.17.54. However, according to the rest information, I found that the latest version of botocore is 1.17.53, it never actually reached 1.17.54. So I'm wondering how to solve this problem. Thanks for your help!
The error message is as follows,
ERROR: Could not find a version that satisfies the requirement botocore<1.18.0,>=1.17.54 (from boto3->pytorch-pretrained-bert) (from versions: 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.5.3, 0.5.4, 0.6.0, 0.7.0, 0.8.0, 0.8.1, 0.8.2, 0.8.3, 0.9.0, 0.9.1, 0.9.2, 0.10.0, 0.11.0, 0.12.0, 0.13.0, 0.13.1, 0.14.0, 0.15.0, 0.15.1, 0.16.0, 0.17.0, 0.18.0, 0.19.0, 0.20.0, 0.21.0, 0.22.0, 0.23.0, 0.24.0, 0.25.0, 0.26.0, 0.27.0, 0.28.0, 0.29.0, 0.30.0, 0.31.0, 0.32.0, 0.33.0, 0.34.0, 0.35.0, 0.36.0, 0.37.0, 0.38.0, 0.39.0, 0.40.0, 0.41.0, 0.42.0, 0.43.0, 0.44.0, 0.45.0, 0.46.0, 0.47.0, 0.48.0, 0.49.0, 0.50.0, 0.51.0, 0.52.0, 0.53.0, 0.54.0, 0.55.0, 0.56.0, 0.57.0, 0.58.0, 0.59.0, 0.60.0, 0.61.0, 0.62.0, 0.63.0, 0.64.0, 0.65.0, 0.66.0, 0.67.0, 0.68.0, 0.69.0, 0.70.0, 0.71.0, 0.72.0, 0.73.0, 0.74.0, 0.75.0, 0.76.0, 0.77.0, 0.78.0, 0.79.0, 0.80.0, 0.81.0, 0.82.0, 0.83.0, 0.84.0, 0.85.0, 0.86.0, 0.87.0, 0.88.0, 0.89.0, 0.90.0, 0.91.0, 0.92.0, 0.93.0, 0.94.0, 0.95.0, 0.96.0, 0.97.0, 0.98.0, 0.99.0, 0.100.0, 0.101.0, 0.102.0, 0.103.0, 0.104.0, 0.105.0, 0.106.0, 0.107.0, 0.108.0, 0.109.0, 1.0.0a1, 1.0.0a2, 1.0.0a3, 1.0.0b1, 1.0.0b2, 1.0.0b3, 1.0.0rc1, 1.0.0, 1.0.1, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5, 1.1.6, 1.1.7, 1.1.8, 1.1.9, 1.1.10, 1.1.11, 1.1.12, 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.2.8, 1.2.9, 1.2.10, 1.2.11, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.3.8, 1.3.9, 1.3.10, 1.3.11, 1.3.12, 1.3.13, 1.3.14, 1.3.15, 1.3.16, 1.3.17, 1.3.18, 1.3.19, 1.3.20, 1.3.21, 1.3.22, 1.3.23, 1.3.24, 1.3.25, 1.3.26, 1.3.27, 1.3.28, 1.3.29, 1.3.30, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.4.23, 1.4.24, 1.4.25, 1.4.26, 1.4.27, 1.4.28, 1.4.29, 1.4.30, 1.4.31, 1.4.32, 1.4.33, 1.4.34, 1.4.35, 1.4.36, 1.4.37, 1.4.38, 1.4.39, 1.4.40, 1.4.41, 1.4.42, 1.4.43, 1.4.44, 1.4.46, 1.4.47, 1.4.48, 1.4.49, 1.4.50, 1.4.51, 1.4.52, 1.4.53, 1.4.54, 1.4.55, 1.4.56, 1.4.57, 1.4.58, 1.4.59, 1.4.60, 1.4.61, 1.4.62, 1.4.63, 1.4.64, 1.4.65, 1.4.66, 1.4.67, 1.4.68, 1.4.69, 1.4.70, 1.4.71, 1.4.72, 1.4.73, 1.4.74, 1.4.75, 1.4.76, 1.4.77, 1.4.78, 1.4.79, 1.4.80, 1.4.81, 1.4.82, 1.4.83, 1.4.84, 1.4.85, 1.4.86, 1.4.87, 1.4.88, 1.4.89, 1.4.90, 1.4.91, 1.4.92, 1.4.93, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.13, 1.5.14, 1.5.15, 1.5.16, 1.5.17, 1.5.18, 1.5.19, 1.5.20, 1.5.21, 1.5.22, 1.5.23, 1.5.24, 1.5.25, 1.5.26, 1.5.27, 1.5.28, 1.5.29, 1.5.30, 1.5.31, 1.5.32, 1.5.33, 1.5.34, 1.5.35, 1.5.36, 1.5.37, 1.5.38, 1.5.39, 1.5.40, 1.5.41, 1.5.42, 1.5.43, 1.5.44, 1.5.45, 1.5.46, 1.5.47, 1.5.48, 1.5.49, 1.5.50, 1.5.51, 1.5.52, 1.5.53, 1.5.54, 1.5.55, 1.5.56, 1.5.57, 1.5.58, 1.5.59, 1.5.60, 1.5.61, 1.5.62, 1.5.63, 1.5.64, 1.5.65, 1.5.66, 1.5.67, 1.5.68, 1.5.69, 1.5.70, 1.5.71, 1.5.72, 1.5.73, 1.5.74, 1.5.75, 1.5.76, 1.5.77, 1.5.78, 1.5.79, 1.5.80, 1.5.81, 1.5.82, 1.5.83, 1.5.84, 1.5.85, 1.5.86, 1.5.87, 1.5.88, 1.5.89, 1.5.90, 1.5.91, 1.5.92, 1.5.93, 1.5.94, 1.5.95, 1.6.0, 1.6.1, 1.6.2, 1.6.3, 1.6.4, 1.6.5, 1.6.6, 1.6.7, 1.6.8, 1.7.0, 1.7.1, 1.7.2, 1.7.3, 1.7.4, 1.7.5, 1.7.6, 1.7.7, 1.7.8, 1.7.9, 1.7.10, 1.7.11, 1.7.12, 1.7.13, 1.7.14, 1.7.15, 1.7.16, 1.7.17, 1.7.18, 1.7.19, 1.7.20, 1.7.21, 1.7.22, 1.7.23, 1.7.24, 1.7.25, 1.7.26, 1.7.27, 1.7.28, 1.7.29, 1.7.30, 1.7.31, 1.7.32, 1.7.33, 1.7.34, 1.7.35, 1.7.36, 1.7.37, 1.7.38, 1.7.39, 1.7.40, 1.7.41, 1.7.42, 1.7.43, 1.7.44, 1.7.45, 1.7.46, 1.7.47, 1.7.48, 1.8.0, 1.8.1, 1.8.2, 1.8.3, 1.8.4, 1.8.5, 1.8.6, 1.8.7, 1.8.8, 1.8.9, 1.8.10, 1.8.11, 1.8.12, 1.8.13, 1.8.14, 1.8.15, 1.8.16, 1.8.17, 1.8.18, 1.8.19, 1.8.20, 1.8.21, 1.8.22, 1.8.23, 1.8.24, 1.8.25, 1.8.26, 1.8.27, 1.8.28, 1.8.29, 1.8.30, 1.8.31, 1.8.32, 1.8.33, 1.8.34, 1.8.35, 1.8.36, 1.8.37, 1.8.38, 1.8.39, 1.8.40, 1.8.41, 1.8.42, 1.8.43, 1.8.44, 1.8.45, 1.8.46, 1.8.47, 1.8.48, 1.8.49, 1.8.50, 1.9.0, 1.9.1, 1.9.2, 1.9.3, 1.9.4, 1.9.5, 1.9.6, 1.9.7, 1.9.8, 1.9.9, 1.9.10, 1.9.11, 1.9.12, 1.9.13, 1.9.14, 1.9.15, 1.9.16, 1.9.17, 1.9.18, 1.9.19, 1.9.20, 1.9.21, 1.9.22, 1.9.23, 1.10.0, 1.10.1, 1.10.2, 1.10.3, 1.10.4, 1.10.5, 1.10.6, 1.10.7, 1.10.8, 1.10.9, 1.10.10, 1.10.11, 1.10.12, 1.10.13, 1.10.14, 1.10.15, 1.10.16, 1.10.17, 1.10.18, 1.10.19, 1.10.20, 1.10.21, 1.10.22, 1.10.23, 1.10.24, 1.10.25, 1.10.26, 1.10.27, 1.10.28, 1.10.29, 1.10.30, 1.10.31, 1.10.32, 1.10.33, 1.10.34, 1.10.35, 1.10.36, 1.10.37, 1.10.38, 1.10.39, 1.10.40, 1.10.41, 1.10.42, 1.10.43, 1.10.44, 1.10.45, 1.10.46, 1.10.47, 1.10.48, 1.10.49, 1.10.50, 1.10.51, 1.10.52, 1.10.53, 1.10.54, 1.10.55, 1.10.56, 1.10.57, 1.10.58, 1.10.59, 1.10.60, 1.10.61, 1.10.62, 1.10.63, 1.10.64, 1.10.65, 1.10.66, 1.10.67, 1.10.68, 1.10.69, 1.10.70, 1.10.71, 1.10.72, 1.10.73, 1.10.74, 1.10.75, 1.10.76, 1.10.77, 1.10.78, 1.10.79, 1.10.80, 1.10.81, 1.10.82, 1.10.83, 1.10.84, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.11.4, 1.11.5, 1.11.6, 1.11.7, 1.11.8, 1.11.9, 1.12.0, 1.12.1, 1.12.2, 1.12.3, 1.12.4, 1.12.5, 1.12.6, 1.12.7, 1.12.8, 1.12.9, 1.12.10, 1.12.11, 1.12.12, 1.12.13, 1.12.14, 1.12.15, 1.12.16, 1.12.17, 1.12.18, 1.12.19, 1.12.20, 1.12.21, 1.12.22, 1.12.23, 1.12.24, 1.12.25, 1.12.26, 1.12.27, 1.12.28, 1.12.29, 1.12.30, 1.12.31, 1.12.32, 1.12.33, 1.12.34, 1.12.35, 1.12.36, 1.12.37, 1.12.38, 1.12.39, 1.12.40, 1.12.41, 1.12.42, 1.12.43, 1.12.44, 1.12.45, 1.12.46, 1.12.47, 1.12.48, 1.12.49, 1.12.50, 1.12.51, 1.12.52, 1.12.53, 1.12.54, 1.12.55, 1.12.56, 1.12.57, 1.12.58, 1.12.59, 1.12.60, 1.12.61, 1.12.62, 1.12.63, 1.12.64, 1.12.65, 1.12.66, 1.12.67, 1.12.68, 1.12.69, 1.12.70, 1.12.71, 1.12.72, 1.12.73, 1.12.74, 1.12.75, 1.12.76, 1.12.77, 1.12.78, 1.12.79, 1.12.80, 1.12.81, 1.12.82, 1.12.83, 1.12.84, 1.12.85, 1.12.86, 1.12.87, 1.12.88, 1.12.89, 1.12.90, 1.12.91, 1.12.92, 1.12.93, 1.12.94, 1.12.95, 1.12.96, 1.12.97, 1.12.98, 1.12.99, 1.12.100, 1.12.101, 1.12.102, 1.12.103, 1.12.104, 1.12.105, 1.12.106, 1.12.107, 1.12.108, 1.12.109, 1.12.110, 1.12.111, 1.12.112, 1.12.113, 1.12.114, 1.12.115, 1.12.116, 1.12.117, 1.12.118, 1.12.119, 1.12.120, 1.12.121, 1.12.122, 1.12.123, 1.12.124, 1.12.125, 1.12.126, 1.12.127, 1.12.128, 1.12.129, 1.12.130, 1.12.131, 1.12.132, 1.12.133, 1.12.134, 1.12.135, 1.12.136, 1.12.137, 1.12.138, 1.12.139, 1.12.140, 1.12.141, 1.12.142, 1.12.143, 1.12.144, 1.12.145, 1.12.146, 1.12.147, 1.12.148, 1.12.149, 1.12.150, 1.12.151, 1.12.152, 1.12.153, 1.12.154, 1.12.155, 1.12.156, 1.12.157, 1.12.158, 1.12.159, 1.12.160, 1.12.161, 1.12.162, 1.12.163, 1.12.164, 1.12.165, 1.12.166, 1.12.167, 1.12.168, 1.12.169, 1.12.170, 1.12.171, 1.12.172, 1.12.173, 1.12.174, 1.12.175, 1.12.176, 1.12.177, 1.12.178, 1.12.179, 1.12.180, 1.12.181, 1.12.182, 1.12.183, 1.12.184, 1.12.185, 1.12.186, 1.12.187, 1.12.188, 1.12.189, 1.12.190, 1.12.191, 1.12.192, 1.12.193, 1.12.194, 1.12.195, 1.12.196, 1.12.197, 1.12.198, 1.12.199, 1.12.200, 1.12.201, 1.12.202, 1.12.203, 1.12.204, 1.12.205, 1.12.206, 1.12.207, 1.12.208, 1.12.209, 1.12.210, 1.12.211, 1.12.212, 1.12.213, 1.12.214, 1.12.215, 1.12.216, 1.12.217, 1.12.218, 1.12.219, 1.12.220, 1.12.221, 1.12.222, 1.12.223, 1.12.224, 1.12.225, 1.12.226, 1.12.227, 1.12.228, 1.12.229, 1.12.230, 1.12.231, 1.12.232, 1.12.233, 1.12.234, 1.12.235, 1.12.236, 1.12.237, 1.12.238, 1.12.239, 1.12.240, 1.12.241, 1.12.242, 1.12.243, 1.12.244, 1.12.245, 1.12.246, 1.12.247, 1.12.248, 1.12.249, 1.12.250, 1.12.251, 1.12.252, 1.12.253, 1.13.0, 1.13.1, 1.13.2, 1.13.3, 1.13.4, 1.13.5, 1.13.6, 1.13.7, 1.13.8, 1.13.9, 1.13.10, 1.13.11, 1.13.12, 1.13.13, 1.13.14, 1.13.15, 1.13.16, 1.13.17, 1.13.18, 1.13.19, 1.13.20, 1.13.21, 1.13.22, 1.13.23, 1.13.24, 1.13.25, 1.13.26, 1.13.27, 1.13.28, 1.13.29, 1.13.30, 1.13.31, 1.13.32, 1.13.33, 1.13.34, 1.13.35, 1.13.36, 1.13.37, 1.13.38, 1.13.39, 1.13.40, 1.13.41, 1.13.42, 1.13.43, 1.13.44, 1.13.45, 1.13.46, 1.13.47, 1.13.48, 1.13.49, 1.13.50, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.14.7, 1.14.8, 1.14.9, 1.14.10, 1.14.11, 1.14.12, 1.14.13, 1.14.14, 1.14.15, 1.14.16, 1.14.17, 1.15.0, 1.15.1, 1.15.2, 1.15.3, 1.15.4, 1.15.5, 1.15.6, 1.15.7, 1.15.8, 1.15.9, 1.15.10, 1.15.11, 1.15.12, 1.15.13, 1.15.14, 1.15.15, 1.15.16, 1.15.17, 1.15.18, 1.15.19, 1.15.20, 1.15.21, 1.15.22, 1.15.23, 1.15.24, 1.15.25, 1.15.26, 1.15.27, 1.15.28, 1.15.29, 1.15.30, 1.15.31, 1.15.32, 1.15.33, 1.15.34, 1.15.35, 1.15.36, 1.15.37, 1.15.38, 1.15.39, 1.15.40, 1.15.41, 1.15.42, 1.15.43, 1.15.44, 1.15.45, 1.15.46, 1.15.47, 1.15.48, 1.15.49, 1.16.0, 1.16.1, 1.16.2, 1.16.3, 1.16.4, 1.16.5, 1.16.6, 1.16.7, 1.16.8, 1.16.9, 1.16.10, 1.16.11, 1.16.12, 1.16.13, 1.16.14, 1.16.15, 1.16.16, 1.16.17, 1.16.18, 1.16.19, 1.16.20, 1.16.21, 1.16.22, 1.16.23, 1.16.24, 1.16.25, 1.16.26, 1.17.0, 1.17.1, 1.17.2, 1.17.3, 1.17.4, 1.17.5, 1.17.6, 1.17.7, 1.17.8, 1.17.9, 1.17.10, 1.17.11, 1.17.12, 1.17.13, 1.17.14, 1.17.15, 1.17.16, 1.17.17, 1.17.18, 1.17.19, 1.17.20, 1.17.21, 1.17.22, 1.17.23, 1.17.24, 1.17.25, 1.17.26, 1.17.27, 1.17.28, 1.17.29, 1.17.30, 1.17.31, 1.17.32, 1.17.33, 1.17.34, 1.17.35, 1.17.36, 1.17.37, 1.17.38, 1.17.39, 1.17.40, 1.17.41, 1.17.42, 1.17.43, 1.17.44, 1.17.45, 1.17.46, 1.17.47, 1.17.48, 1.17.49, 1.17.50, 1.17.51, 1.17.52, 1.17.53)
ERROR: No matching distribution found for botocore<1.18.0,>=1.17.54 (from boto3->pytorch-pretrained-bert)


title: Confusion about model.train()

body: I am a beginner in pytorch. I saw on github that some deep learning models have model.train(), and some don’t, but they can run normally. I want to know if model.train() is necessary? what's the effect?


title: Pytorch: Send same batch of data to multiple GPUs, and perform ops on each GPU individually

body:
I have the same dataloader to feed data to 4 models, each with a different hyperparameter loaded on a separate GPU. I want to reduce the bottleneck caused by data-loading, so I intend to load the same batch prepared by the dataloader on all GPUs for them to compute individually and perform a backprop step. I already cache data into RAM to avoid disk-bottlenecks when the dataloader in instantiated.
I am trying to:

Send/Broadcast the same batch of data to N GPUs. I guess this is possible only if we can sync/wait for all GPUs to finish ops for one batch, before we can proceed to the next one.
Bonus: Prefetching next batch as soon as one batch is ready (upto P batches) could help ensure continuous flow of data to the GPUs avoiding the wait.

I am not trying to achieve:

Data Parallelism - Split a large batch into N parts, and compute each part on one GPU
Model Parallelism - Split computation of a large model (that won't fit on one GPU) into N (or less) parts and place each part on one GPU.

Similar questions:

This one is about making a Conv2D operation span across multiple GPUs
This one is about executing different GPU computations in parallel, but I don't know if my problem can be solved with torch.cuda.Stream()
This one is about loading different models, but it does not deal with sharing the same batch.
This one is exactly about what I'm asking, but it's CUDA/PCIe and from 7 years ago.

Update:
I found a very similar question in Pytorch discuss where there is a small example at the end using forward prop using multiprocessing, but I'm wondering how to scale this approach to torch dataloaders.


title: Pytorch, Pre-trained model: How to use feature and classifier at the same time

body:
I'm using vgg16 extracting image feature vector. I want to get 114096 vector from the 2nd-to-last layer.

My code:
def get_model():
    model = models.vgg16(pretrained=True)#.features[:].classifier[:4]
    model = model.eval()
    # model.cuda()  # send the model to GPU, DO NOT include this line if you haven't a GPU
    return model

But I can only get 111000 vector from the last layer.
I know how to use feathers and classifier, but I don't know how to use them at the same time.
use classifier only:

use feathers only:

use them at the same time:

log:
Traceback (most recent call last):
  File "/mnt/c/Users/sunji/PycharmProjects/image_cluster_pytorch/main.py", line 7, in <module>
    model = calc.get_model()
  File "/mnt/c/Users/sunji/PycharmProjects/image_cluster_pytorch/imagecluster/calc.py", line 17, in get_model
    model = models.vgg16(pretrained=True).features[:].classifier[:4]
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 771, in __getattr__
    raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
torch.nn.modules.module.ModuleAttributeError: 'Sequential' object has no attribute 'classifier'


title: PyTorch: What is numpy.linalg.multi_dot() equivalent in PyTorch

body:
I am trying to perform matrix multiplication of multiple matrices in PyTorch and was wondering what is the equivalent of numpy.linalg.multi_dot() in PyTorch?
If there isn't one, what is the next best way (in terms of speed and memory) I can do this in PyTorch?
Code:
import numpy as np
import torch

A = np.random.rand(3, 3)
B = np.random.rand(3, 3)
C = np.random.rand(3, 3)

results = np.linalg.multi_dot(A, B, C)

A_tsr = torch.tensor(A)
B_tsr = torch.tensor(B)
C_tsr = torch.tensor(C)

# What is the PyTorch equivalent of np.linalg.multi_dot()?

Many thanks!