title: How to reuse the backward pass primitives?

body:
I want to implement a PyTorch Function that takes in another function and a tensor as arguments, applies it, and on the backward pass matches the gradient variances between the input and the output. Something like the following pseudo-code.
import torch
from torch.autograd import Function
import torch.nn
import torch.nn.functional

# Is meant to be arbitrary.
def f(x):
    return x*2

class VarianceMatch(Function):
    @staticmethod
    def forward(ctx, x):
        y = f(x)
        ctx.save_for_backward(x,y)
        return y

    @staticmethod
    def backward(ctx, grad_y):
        x,y = ctx.saved_tensors
        y.backward(grad_y)
        x.grad *= torch.sqrt(torch.square(grad_y).sum(1) / torch.square(x.grad).sum(1))
        return torch.zeros_like(x)

i = torch.scalar_tensor(2,requires_grad=True)
x = VarianceMatch.apply(i)
x.backward(torch.scalar_tensor(1))
print(i.grad)

The above is just to express my intent of what the code is supposed to be doing, running the above goes into an infinite loop on the y.backward(grad_y) call. I'd like to somehow extract the backward function and have it give me the gradients so that I can rescale them.


title: Why is the depthwise_conv2d implemented with the group parameter in Pytorch slower than Conv2d?

body:
After referring to the online materials, I used the following code to implement DepthWise_Conv2d under pytorch.
Using the summary test performance of torchinfo, I found that the running speed and the amount of process parameters are very huge.
Depthwise_Con2d
m = nn.Conv2d(16, 16, 3, stride=1, groups=16, dilation=1,bias=False)
n = nn.Conv2d(16, 33, 1, stride=1,  dilation=1, bias=False)

                    Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls

aten::mkldnn_convolution        90.40%      13.248ms        90.88%      13.318ms       6.659ms      17.59 Mb           0 b             2
         model_inference         8.38%       1.228ms        99.81%      14.627ms      14.627ms          -4 b     -17.59 Mb             1
             aten::empty         0.47%      69.400us         0.47%      69.400us      17.350us      17.59 Mb      17.59 Mb             4
      aten::_convolution         0.24%      35.000us        91.12%      13.353ms       6.677ms      17.59 Mb           0 b             2
            aten::conv2d         0.19%      27.900us        91.42%      13.397ms       6.699ms      17.59 Mb           0 b             2
             aten::zeros         0.12%      17.800us         0.19%      27.600us      27.600us           4 b           0 b             1
       aten::convolution         0.11%      16.200us        91.23%      13.369ms       6.685ms      17.59 Mb           0 b             2
       aten::as_strided_         0.07%      10.300us         0.07%      10.300us       5.150us           0 b           0 b             2
             aten::zero_         0.02%       2.600us         0.02%       2.600us       2.600us           0 b           0 b             1

Self CPU time total: 14.655ms

                    Layer (type:depth-idx)                   Output Shape              Param #
                    ├─Conv2d: 1-1                            [20, 16, 48, 98]          144
                    ├─Conv2d: 1-2                            [20, 33, 48, 98]          528

Total params: 672
Trainable params: 672
Non-trainable params: 0
Total mult-adds (M): 3.16

Input size (MB): 6.10
Forward/backward pass size (MB): 35.17
Params size (MB): 0.00
Estimated Total Size (MB): 41.28

Conv2d Summary

                    Name    Self CPU %      Self CPU   CPU total %     CPU total  CPU time avg       CPU Mem  Self CPU Mem    # of Calls

aten::mkldnn_convolution        90.64%       9.066ms        91.02%       9.104ms       9.104ms      11.84 Mb           0 b             1
         model_inference         8.16%     815.700us        99.71%       9.973ms       9.973ms          -4 b     -11.84 Mb             1
             aten::empty         0.44%      43.700us         0.44%      43.700us      14.567us      11.84 Mb      11.84 Mb             3
      aten::_convolution         0.23%      23.500us        91.25%       9.127ms       9.127ms      11.84 Mb           0 b             1
            aten::conv2d         0.19%      19.000us        91.53%       9.155ms       9.155ms      11.84 Mb           0 b             1
             aten::zeros         0.18%      18.100us         0.29%      28.900us      28.900us           4 b           0 b             1
       aten::convolution         0.09%       9.200us        91.34%       9.136ms       9.136ms      11.84 Mb           0 b             1
       aten::as_strided_         0.04%       4.000us         0.04%       4.000us       4.000us           0 b           0 b             1
             aten::zero_         0.03%       3.000us         0.03%       3.000us       3.000us           0 b           0 b             1

                    Layer (type:depth-idx)                   Output Shape              Param #
                    └─Conv2d: 0-1                            [20, 33, 48, 98]          4,785

Total params: 4,785
Trainable params: 4,785
Non-trainable params: 0
Total mult-adds (M): 22.35
Input size (MB): 6.10
Forward/backward pass size (MB): 23.69
Params size (MB): 0.02
Estimated Total Size (MB): 29.81

Could you tell me what went wrong?


title: torch.Tensor() new() received an invalid combination of arguments - got (list, dtype=torch.dtype)

body:
I am facing a very weird error when I am trying to run the following simple line :
a = torch.Tensor([0,0,0],dtype = torch.int64)

TypeError: new() received an invalid combination of arguments - got (list, dtype=torch.dtype), but expected one of:
 * (*, torch.device device)
      didn't match because some of the keywords were incorrect: dtype
 * (torch.Storage storage)
 * (Tensor other)
 * (tuple of ints size, *, torch.device device)
 * (object data, *, torch.device device)

whereas if we look at official documentation
torch.tensor(data, dtype=None, device=None, requires_grad=False, pin_memory=False) → Tensor

Parameters
data (array_like) – Initial data for the tensor. Can be a
list, tuple, NumPy ndarray, scalar, and other types.
dtype (torch.dtype, optional) – the desired data type of returned
tensor. Default: if None, infers data type from data.

Why is the code snippet not working even though the official documentation supports it?


title: How to correctly set input image size for LPRNet model? 'RuntimeError: Sizes of tensors must match except in dimension 1..'

body:
Trying to adapt the LPRNet model for my own dataset. The original model was set for Chinese license plate images with dimensions 24x94 pixels. My dataset consists of plates with only numbers, images come with dimensions 64x128. I got an error when I try to replace 'summary(lprnet, (3,24,94), device="cpu")' to 'summary(lprnet, (3,64,128), device="cpu")'.
The first code is working well, the second gets 'RuntimeError: Sizes of tensors must match except in dimension 1. Got 25 and 24 in dimension 3 (The offending index is 1)', this is the last line of code.
I don't see where else in the code I should change parameters. Will be thankful for any clue!
Original: https://github.com/xuexingyu24/License_Plate_Detection_Pytorch/blob/master/LPRNet/model/LPRNET.py
# LPRNET model
import torch.nn as nn
import torch
from torchsummary import summary

class small_basic_block(nn.Module):
    def __init__(self, ch_in, ch_out):
        super(small_basic_block, self).__init__()
        self.block = nn.Sequential(
            nn.Conv2d(ch_in, ch_out // 4, kernel_size=1),
            nn.ReLU(),
            nn.Conv2d(ch_out // 4, ch_out // 4, kernel_size=(3, 1), padding=(1, 0)),
            nn.ReLU(),
            nn.Conv2d(ch_out // 4, ch_out // 4, kernel_size=(1, 3), padding=(0, 1)),
            nn.ReLU(),
            nn.Conv2d(ch_out // 4, ch_out, kernel_size=1),
        )
    def forward(self, x):
        return self.block(x)

class LPRNet(nn.Module):
    def __init__(self, class_num, dropout_rate):
        super(LPRNet, self).__init__()
        self.class_num = class_num
        self.backbone = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1), # 0
            nn.BatchNorm2d(num_features=64),
            nn.ReLU(),  # 2
            nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(1, 1, 1)),
            small_basic_block(ch_in=64, ch_out=128),    # *** 4 ***
            nn.BatchNorm2d(num_features=128),
            nn.ReLU(),  # 6
            nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(2, 1, 2)),
            small_basic_block(ch_in=64, ch_out=256),   # 8
            nn.BatchNorm2d(num_features=256),
            nn.ReLU(),  # 10
            small_basic_block(ch_in=256, ch_out=256),   # *** 11 ***
            nn.BatchNorm2d(num_features=256),   # 12
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(1, 3, 3), stride=(4, 1, 2)),  # 14
            nn.Dropout(dropout_rate),
            nn.Conv2d(in_channels=64, out_channels=256, kernel_size=(1, 4), stride=1),  # 16
            nn.BatchNorm2d(num_features=256),
            nn.ReLU(),  # 18
            nn.Dropout(dropout_rate),
            nn.Conv2d(in_channels=256, out_channels=class_num, kernel_size=(13, 1), stride=1), # 20
            nn.BatchNorm2d(num_features=class_num),
            nn.ReLU(),  # *** 22 ***
        )
        self.container = nn.Sequential(
            nn.Conv2d(in_channels=256+class_num+128+64, out_channels=self.class_num, kernel_size=(1,1), stride=(1,1)),
            # nn.BatchNorm2d(num_features=self.class_num),
            # nn.ReLU(),
            # nn.Conv2d(in_channels=self.class_num, out_channels=self.lpr_max_len+1, kernel_size=3, stride=2),
            # nn.ReLU(),
        )

    def forward(self, x):
        keep_features = list()
        for i, layer in enumerate(self.backbone.children()):
            x = layer(x)
            if i in [2, 6, 13, 22]: # [2, 4, 8, 11, 22]
                keep_features.append(x)

        global_context = list()
        for i, f in enumerate(keep_features):
            if i in [0, 1]:
                f = nn.AvgPool2d(kernel_size=5, stride=5)(f)
            if i in [2]:
                f = nn.AvgPool2d(kernel_size=(4, 10), stride=(4, 2))(f)
            f_pow = torch.pow(f, 2)
            f_mean = torch.mean(f_pow)
            f = torch.div(f, f_mean)
            global_context.append(f)

        x = torch.cat(global_context, 1)
        x = self.container(x)
        logits = torch.mean(x, dim=2)

        return logits

CHARS = ['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']

if __name__ == "__main__":

    lprnet = LPRNet(class_num=len(CHARS), dropout_rate=0)
    print(lprnet)

    summary(lprnet, (3,24,94), device="cpu")
    #summary(lprnet, (3,64,128), device="cpu")


title: How to make worker machines use RRefs locally, while retaining gradients in PyTorch RPC

body:
Suppose I have a PyTorch module with a component that is an RRef to a model held in a parameter server such as the following example:
class MyRPCModule(nn.Module):
    def __init__(self, ps, x_dim, h_dim, z_dim):
        super(MyRPCModule, self).__init__()

        # Ref for remote embedder on param server
        self.remote_emb_rref = rpc.remote(
            ps, Embedder, args=(x_dim, h_dim)
        )

The Embedder module itself is very small, but I have a huge amount of training data that I'd like to spread out onto multiple different processes/machines.
What would my forward method need to look like such that the computations on that Embedder module happen on the machine calling forward? I've tried
def forward(self, x):
    return self.remote_emb_rref.to_here()(x)

but that doesn't preserve the gradients, so all of the backward and step calls don't change the parameters on the parameter server.
I'm mostly working off of these two tutorials: PyTorch's "getting started with Dist RPC framework, and their Combining DDP with RPC Framework but both use the parameter server to do the computation. Is there some way around this?


title: `register_forward_hook` got error when using multiple GPUs

body:
I'm using register_forward_hook to extract intermediate features from another model and use this feature in another model. However, the code runs well on a single GPU but got a RuntimeError: Expected tensor for 'out' to have the same device as tensor for argument #2 'mat1'; but device 3 does not equal 4 (while checking arguments for addmm)
Here's my code. Could anyone tell me how to fix the problem?
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader

device_id = [3,4]
data_size = 2000
hidden_length = 10
batch_size = 100
device_ = 3
input_size = 1000
output_size = 200
device = torch.device("cuda:{}".format(device_) if torch.cuda.is_available() else "cpu")

class RandomDataset(Dataset):

    def __init__(self, size, length):
        self.len = length
        self.data = torch.randn(length, size)

    def __getitem__(self, index):
        return self.data[index]

    def __len__(self):
        return self.len

rand_loader = DataLoader(dataset=RandomDataset(input_size, data_size),
                         batch_size=batch_size, shuffle=True)

class Model(nn.Module):
    # Model to be extracted

    def __init__(self, input_size, output_size, hidden_length):
        super(Model, self).__init__()
        block = [nn.Linear(input_size, input_size)] * hidden_length
        self.hidden = nn.Sequential(*block)
        self.fc = nn.Linear(input_size, output_size)

    def forward(self, input):
        output = self.hidden(input)
        output = self.fc(output)

        return output

class HookModel(nn.Module):
    # Our model

    def __init__(self, input_size, output_size, hidden_length):
        super(HookModel, self).__init__()
        self.model = Model(input_size, output_size, hidden_length)
        self.layer_extract = self.model._modules.get("hidden")
        self.fc = nn.Linear(input_size, input_size)

    def forward(self, input):
        def model_forward(input):
            model_output = []

            def hook(m, i, o):
                model_output.append(o.clone().detach())

            h = self.layer_extract.register_forward_hook(hook)
            self.model(input)
            h.remove()
            return model_output[0]

        output = model_forward(input)
        output = self.fc(output)

        return output

model = HookModel(input_size, output_size, hidden_length)
if len(device_id) > 1:
    print("Let's use", len(device_id), "GPUs!")
    model = nn.DataParallel(model, device_ids=device_id)
model.to(device)

for data in rand_loader:
    input = data.to(device)
    output = model(input)
    loss = torch.sum((output - 2) ** 2)
    loss.backward()


title: Question about the serial or parallel execution of DGL message passing and aggregation commands

body:
My question is about the order of execution of message passing and aggregation commands from DGL on a GPU.
I would like to use a GatedGraphConv module on a graph with possibly many different edge types and am wondering if, for any given call of apply_edges() below, is the call just scheduling the task on the GPU or is the call blocking the remaining iterations of the loop until after its computations are done?
Here is the main loop of the forward() method of GatedGraphConv (found at https://docs.dgl.ai/_modules/dgl/nn/pytorch/conv/gatedgraphconv.html#GatedGraphConv):
for _ in range(self._n_steps):
    graph.ndata['h'] = feat
    for i in range(self._n_etypes):
        eids = th.nonzero(etypes == i, as_tuple=False).view(-1).type(graph.idtype)
        if len(eids) > 0:
            graph.apply_edges(
                lambda edges: {'W_e*h': self.linears[i](edges.src['h'])},
                eids
            )
    graph.update_all(fn.copy_e('W_e*h', 'm'), fn.sum('m', 'a'))
    a = graph.ndata.pop('a') # (N, D)
    feat = self.gru(a, feat)

I have looked at possibly using a set of cuda streams on a single GPU where each one would be given its own call to apply_edges() (with a different string "W_e*h" for each different edge type so they don't conflict with each other) and then synchronizing them before the call to update_all(), but it occurred to me that this might be overkill and that perhaps the calls to apply_edges() are merely to schedule a task on the GPU and then that the GPU is free to complete all scheduled tasks in whichever way it sees fit (including running multiple tasks simultaneously if it can).
Thanks.


title: Transfer learning on an existing class in object detection with pytorch

body:
I need to perform transfer learning on a class of a pre-trained model using pytorch oriented to object detection.
Basically what I want to do is based on some detections that I have done manually on a series of cars, perform transfer learning to improve the output of the detection model.
To date I have not found information related to this aspect, so I do not know very well how to perform this task.


title: Do we need a ignore_index in F.cross_entropy or a padding_idx in nn.Embedding for padded sequnce in NLP?

body: I know they two can be used to handle the padded sentence. However, which one I should use or both of them for my seq2seq model?


title: How to quantize a module with GRUCell as a submodule

body:
I have a module which needs static quantization as dynamic quantization does not cover many of its components. Within that module I also have a GRUCell. When attempting to run the quantized model I get:
RuntimeError: Could not run 'aten::addmm' with arguments from the 'QuantizedCPU' backend.

I'm aware that dynamic quantization should work for GRUCell and I tried to incorporate it like this:
# set quantization scheme for overall module
model_q.qconfig = torch.quantization.get_default_qconfig("qnnpack")
# drop quantization scheme for embedding layer which is not supported
model_q.model_fp32.decoder.embedding.qconfig = None
# do the same for my GRUCell?
model_q.model_fp32.decoder.decode_step.qconfig = None
# dynamically quantize GRUCell?
model_q.model_fp32.decoder.decode_step = torch.quantization.quantize_dynamic(
    model_q.model_fp32.decoder.decode_step,  # the original model
    {torch.nn.GRUCell},  # a set of layers to dynamically quantize
    dtype=torch.qint8)

but I'm still getting the same error as above


title: cannot import name 'USE_GLOBAL_DEPS' ERROR while using torch

body:
Can someone help me resolve the issue?

ImportError                               Traceback (most recent call last)
 in
----> 1 import torch
~\anaconda3\lib\site-packages\torch\__init__.py in <module>
 21
 22 from ._utils import _import_dotted_name
---> 23 from ._utils_internal import get_file_path, prepare_multiprocessing_environment, \
 24     USE_RTLD_GLOBAL_WITH_LIBTORCH, USE_GLOBAL_DEPS
 25 # TODO(torch_deploy) figure out how to freeze version.py in fbcode build

ImportError: cannot import name 'USE_GLOBAL_DEPS' from 'torch._utils_internal' (C:\Users\ANTHONY\anaconda3\lib\site-packages\torch_utils_internal.py)


title: Detectron2 build error when I try building from source

body:
I am trying to build Detectron2 from source but I keep getting these errors. I have tried many things like downgrading my CUDA version or downgrading my gcc version but none of it seems to work.
ERROR: Command errored out with exit status 1: 'C:\Users\Mathuran\anaconda3\envs\detectron\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\Mathuran\detectron2\setup.py'"'"'; file__='"'"'C:\Users\Mathuran\detectron2\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
ERROR: Command errored out with exit status 1:command: 'C:\Users\Mathuran\anaconda3\envs\detectron\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\Mathuran\detectron2\setup.py'"'"'; file_='"'"'C:\Users\Mathuran\detectron2\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(_file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps


title: How to get around `'Proxy' object does not support item assignment`

body:
Here's the line that triggers this error during torch.quantization.quantize_fx.prepare_fx
patch_embed[:, 1:] = patch_embed[:, 1:] + self.proj(self.norm1_proj(pixel_embed).reshape(B, N - 1, -1))

Is there some way to get around this without having to avoid item assignment? Can I somehow inform that particular proxy instance that this should be okay?


title: Is it sth wrong with the shape of weight files？

body:
lately i've studied the article of Retinaface, and i try to replace the backbone with mobilenetv2.
however it turns out errors when i run train.py:
    Traceback (most recent call last):
  File "E:/study2/retinaface-pytorch/train.py", line 132, in <module>
    model = RetinaFace(cfg=cfg, pretrained=pretrained).train()
  File "E:\study2\retinaface-pytorch\nets\retinaface.py", line 74, in __init__
    self.body = _utils.IntermediateLayerGetter(backbone, cfg['return_layers'])
  File "D:\toolsware\anaconda\envs\pytorch\lib\site-packages\torchvision\models\_utils.py", line 45, in __init__
    if not set(return_layers).issubset([name for name, _ in model.named_children()]):

AttributeError: 'NoneType' object has no attribute 'named_children'

as i rewrite the code of mobilenetv2 to adapt the Retinaface, it could be something wrong with the shape of retrained-weight files, i guess.
but what does it have to do with "named_children" and how can i deal with it?
plz help if someone know anything.
below is the code of train.py, you can look through it if this may help.-v-
from __future__ import print_function
import argparse
import datetime
import math
import os
import time
import numpy as np
import torch
import torch.backends.cudnn as cudnn
import torch.optim as optim
from torch.autograd import Variable
from torch.utils.data import DataLoader
from tqdm import tqdm

from nets.retinaface import RetinaFace
from nets.retinaface_training import (DataGenerator, MultiBoxLoss,
                                      detection_collate)
from utils.anchors import Anchors
from utils.config import cfg_mnet, cfg_re50, cfg_mnetv2

# Used to record the time
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()

def get_lr(optimizer):
    for param_group in optimizer.param_groups:
        return param_group['lr']

def fit_one_epoch(net,criterion,epoch,epoch_size,gen,Epoch,anchors,cfg,cuda):
    total_r_loss = 0
    total_c_loss = 0
    total_landmark_loss = 0

    with tqdm(total=epoch_size,desc=f'Epoch {epoch + 1}/{Epoch}',postfix=dict,mininterval=0.3) as pbar:
        for iteration, batch in enumerate(gen):
            if iteration >= epoch_size:
                break
            images, targets = batch[0], batch[1]
            if len(images) == 0:
                continue

            with torch.no_grad():
                if cuda:
                    images = Variable(torch.from_numpy(images).type(torch.FloatTensor)).cuda()
                    targets = [Variable(torch.from_numpy(ann).type(torch.FloatTensor)).cuda() for ann in targets]
                else:
                    images = Variable(torch.from_numpy(images).type(torch.FloatTensor))
                    targets = [Variable(torch.from_numpy(ann).type(torch.FloatTensor)) for ann in targets]

            optimizer.zero_grad()

            out = net(images)

            r_loss, c_loss, landm_loss = criterion(out, anchors, targets)
            loss = cfg['loc_weight'] * r_loss + c_loss + landm_loss

            loss.backward()
            optimizer.step()

            total_c_loss += c_loss.item()
            total_r_loss += cfg['loc_weight'] * r_loss.item()
            total_landmark_loss += landm_loss.item()

            pbar.set_postfix(**{'Conf Loss'         : total_c_loss / (iteration + 1),
                                'Regression Loss'   : total_r_loss / (iteration + 1),
                                'LandMark Loss'     : total_landmark_loss / (iteration + 1),
                                'lr'                : get_lr(optimizer)})
            pbar.update(1)

    print('Saving state, iter:', str(epoch+1))
    torch.save(model.state_dict(), 'logs/Epoch%d-Total_Loss%.4f.pth'%((epoch+1),(total_c_loss + total_r_loss + total_landmark_loss)/(epoch_size+1)))
    return

import os
os.environ["CUDA_VISIBLE_DEVICES"]="0"

if __name__ == "__main__":
    num_classes = 2

    Cuda = True

    training_dataset_path = './data/widerface/train/label.txt'

    backbone = "mobilenetv2"

    pretrained = True
    # pretrained = False

    if backbone == "mobilenet":
        cfg = cfg_mnet
    elif backbone == "resnet50":
        cfg = cfg_re50
    #####################################
    elif backbone == "mobilenetv2":
        cfg = cfg_mnetv2
    #####################################
    else:
        raise ValueError('Unsupported backbone - `{}`, Use mobilenet, resnet50.'.format(backbone))

    img_dim = cfg['train_image_size']

    anchors = Anchors(cfg, image_size=(img_dim, img_dim)).get_anchors()
    if Cuda:
        anchors = anchors.cuda()

    model = RetinaFace(cfg=cfg, pretrained=pretrained).train()

    # this is downloaded from the links inside "torchvision.models.mobilenetv2"
    model_path = "model_data/mobilenet_v2-b0353104.pth"

    print('Loading weights into state dict...')
    device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
    model_dict = model.state_dict()
    pretrained_dict = torch.load(model_path, map_location=device)
    pretrained_dict = {k: v for k, v in pretrained_dict.items() if np.shape(model_dict[k]) == np.shape(v)}
    model_dict.update(pretrained_dict)
    model.load_state_dict(model_dict)
    print('Finished!')

    net = model
    if Cuda:
        net = torch.nn.DataParallel(model)
        cudnn.benchmark = True
        net = net.cuda()

    criterion = MultiBoxLoss(num_classes, 0.35, 7, Cuda)

    if True:
        lr = 1e-3
        Batch_size = 8
        Init_Epoch = 0
        Freeze_Epoch = 50

        optimizer = optim.Adam(net.parameters(), lr, weight_decay=5e-4)
        lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=1, gamma=0.92)

        train_dataset = DataGenerator(training_dataset_path, img_dim)
        gen = DataLoader(train_dataset, shuffle=True, batch_size=Batch_size, num_workers=4, pin_memory=True,
                                drop_last=True, collate_fn=detection_collate)

        epoch_size = train_dataset.get_len()//Batch_size

        for param in model.body.parameters():
            param.requires_grad = False

        for epoch in range(Init_Epoch,Freeze_Epoch):
            fit_one_epoch(net,criterion,epoch,epoch_size,gen,Freeze_Epoch,anchors,cfg,Cuda)
            lr_scheduler.step()

    if True:

        lr = 1e-4
        Batch_size = 4
        Freeze_Epoch = 50
        Unfreeze_Epoch = 100

        optimizer = optim.Adam(net.parameters(),lr,weight_decay=5e-4)
        lr_scheduler = optim.lr_scheduler.StepLR(optimizer,step_size=1,gamma=0.92)

        train_dataset = DataGenerator(training_dataset_path,img_dim)
        gen = DataLoader(train_dataset, shuffle=True, batch_size=Batch_size, num_workers=4, pin_memory=True,
                                drop_last=True, collate_fn=detection_collate)

        epoch_size = train_dataset.get_len()//Batch_size
        for param in model.body.parameters():
            param.requires_grad = True

        for epoch in range(Freeze_Epoch,Unfreeze_Epoch):
            fit_one_epoch(net,criterion,epoch,epoch_size,gen,Unfreeze_Epoch,anchors,cfg,Cuda)
            lr_scheduler.step()

end.record()
# Waits for everything to finish running
torch.cuda.synchronize()
print("total training time:")
print(start.elapsed_time(end))


title: How can scatter_std influence the value of the weights in a GNN programmed with pytorch?

body:
I'm trying to program a GNN using pytorch. I currently use scatter_add and scatter_mean to summerize the edge input for the node update, and then everything works fine. However if I also want to use scatter_std, things go wrong: when the model updates it's parameters after the first batch, all weights are set to 'nan'. I have no idea how scatter_std() can influence this, but apparently it does. Has anyone seen this before?
This is my node update:
class NodeModel_1(torch.nn.Module):
def __init__(self):
    super(NodeModel_1, self).__init__()
    self.node_mlp = nn.ModuleList()

    for l in range(len(NN_node_layers_1)-1):
        # Add linear layers to the neural networks
        self.node_mlp.append(nn.Linear(NN_node_layers_1[l], NN_node_layers_1[l+1]))

def forward(self, node_attr, edge_index, edge_attr, u = None, batch = None):
    row, col = edge_index

    out = torch.cat([scatter_mean(edge_attr, col, dim=0, dim_size=node_attr.size(0)),scatter_add(edge_attr, row, dim=0, dim_size=node_attr.size(0)),scatter_std(edge_attr, row, dim=0, dim_size=node_attr.size(0), unbiased = False)], dim = 1)
    out = torch.cat([out, node_attr],dim = 1)

    for layer in self.node_mlp[:-1]:
        out =  F.elu(layer(out))
    out = self.node_mlp[-1](out)
    return out

And this is my network function:
    def forward(self, data):
    print(self.encoder_edge[0].weight)
    node_attr, edge_index, edge_attr = data.x, data.edge_index, data.edge_attr
    #First put data through the encoders

    for layer in self.encoder_node[:-1]:
        node_attr =  F.elu(layer(node_attr))
    node_attr = self.encoder_node[-1](node_attr)
    for layer in self.encoder_edge[:-1]:
        edge_attr =  F.elu(layer(edge_attr))
    edge_attr = self.encoder_edge[-1](edge_attr)

    print(self.encoder_node[0].weight)
    # Save initial input to feed to the  neural network
    node_attr_zero, edge_attr_zero = 1*node_attr, 1*edge_attr

    # Update network for the first time
    node_attr , edge_attr, u = self.network_1(node_attr, edge_index, edge_attr)

    edge_attr = torch.cat([edge_attr,edge_attr_zero],dim =1)
    node_attr = torch.cat([node_attr,node_attr_zero],dim =1)

    node_attr , edge_attr, u = self.network_2(node_attr, edge_index, edge_attr)

    edge_attr = torch.cat([edge_attr,edge_attr_zero],dim =1)
    node_attr = torch.cat([node_attr,node_attr_zero],dim =1)
    node_attr , edge_attr, u = self.network_3(node_attr, edge_index, edge_attr)

    # Decode node information
    for layer in self.decoder_node[:-1]:
        node_attr =  F.elu(layer(node_attr))
    node_attr = self.decoder_node[-1](node_attr)
    print(self.encoder_edge[0].weight)
    return node_attr


title: Ax Platform optimize returns ValueError Argument to 'not_none' was None

body:
I am trying to perform Bayesian optimization on my PyTorch model using the Ax Platform however
whenever I try to run the optimize function an error is thrown:
ValueError: Argument to `not_none` was None.

This is my call to my optimize function:
best_parameters, values, experiment, model = optimize(
    parameters=[
        {"name": "max_lr", "type": "range", "bounds": [1e-4, 0.4], "log_scale": True},
        {"name": "epoch", "type": "range", "bounds": [5, 20]},
        {"name": "encoder_depth", "type": "range", "bounds": [3, 5]},
        {"name": "arch_name", "type": "choice", "values": ["UNET", "DEEPLABV3+"]},
        {"name": "encoder", "type": "choice", "values": ["mobilenet_v2", "resnet18", "resnet34"]},
        {"name": "activation", "type": "choice", "values": [None, "softmax"]},
    ],
    evaluation_function=train_evaluate,
    objective_name="objective",
)

This is my train and evaluate function:
def train_evaluate(parameterization):

  model = init_model(parameterization)

  max_lr = parameterization["max_lr"]
  epoch = parameterization["epoch"]
  weight_decay = 1e-4

  criterion = nn.CrossEntropyLoss()
  optimizer = torch.optim.AdamW(model.parameters(), lr=max_lr, weight_decay=weight_decay)
  sched = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr, epochs=epoch,
                                              steps_per_epoch=len(train_loader))

  history = fit(epoch, model, train_loader, val_loader, criterion, optimizer, sched)

  torch.save(model, '/mydrive/backup/' + str(parameterization["arch_name"]) + "_" + str(parameterization["epoch"]) + "_" + str(parameterization["max_lr"]) + ".pt")

  return evaluate_model(model=model, test_set=test_set)

And finally this is my evaluation function:
def evaluate_model(model, test_set):
    score_iou = []
    for i in tqdm(range(len(test_set))):
        img, mask = test_set[i]
        pred_mask, score = predict_image_mask_miou(model, img, mask)
        score_iou.append(score)
    return {"objective": sum(score_iou) / len(score_iou)}

I am running this code in a Google Colab notebook. I tried removing and changing some of the arguments in the optimize function however the same error kept being thrown.


title: How to clear gradients for the manual optimizer in Pytorch

body:
What I want to do is to calculate Projected Gradient Ascent w.r.t input X(in my case images) and at the same time disable parameter gradients.
Here is my code:
def func(x: torch.Tensor, y: torch.Tensor,
                       network: nn.Module, loss_func: nn.Module, eta: float = .1, steps: int = 10):

    network.requires_grad_(False)#don't calculate grads for the parameters of the network
    x_copy = x.clone().requires_grad_(True)#copy from the main input
    for i in range(steps):
        pred = network(x_copy)#calculating predictions for the network
        loss = loss_func(pred,y)#calculation of the loss
        x_copy.retain_grad()#retain gradients of the input

        loss.backward()

        x_copy = x_copy + eta * x_copy.grad.sign()#gradient calculation

    return x_copy

Usually, in order to clear grads we do optimiser.zero_grad() before each loss.backward().
But how to do it with manual gradient optimizer? Now I am getting an error:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time


title: How to make this nn.Embedding example work with FX tracing?

body:
I'm working with an RNN decoder and I have some lines in my code like:
start_tokens = encoder_out.new_ones(batch_size, dtype=torch.int) * sos_token
embeddings = self.embedding(start_tokens)

The problem is that the tracer raises this error:
RuntimeError: _th_histc_out not supported on CPUType for Int
presumably when it's attempting to quantize start tokens.
I've also tried with
start_tokens = torch.full(batch_size, sos_token, dtype=torch.int)

but that raised this error TypeError: full() received an invalid combination of arguments - got (Proxy, int, dtype=torch.dtype). In fact, it's the reason I use new_ones (see this discussion).
How can I initialize my embeddings then?


title: adding deteron2 in heroku

body:
ERROR: Invalid requirement: 'detectron2 =0.4' (from line 20 of /tmp/build_4627fb0c/requirements.txt)

   Hint: = is not a valid operator. Did you mean == ?

!     Push rejected, failed to compile Python app.


title: Bizzare extra time consumption in Pytorch (GPU) 1.1.0 / 1.2.0?

body:
I have been struggling with an extra and confusing time problem which I encountered for twice:
First time:
I have two model running simultaneously with a joint loss but it is quite slow(up to 0.4s~0.5s+). So I was trying to improve my efficiency through checking some time spot using time.time(). And I found out The first small model is just fine(like 0.002s). However when I was checking the second model. At the beginning of my second model’s forwarding, Something weird just happen:
def forward(feature, coords, permutation):
    """
    feature : [B, d, N, 1]
    coords: [B, N ,3]
    """
    t1 = time.time()
    coords: [:, permutation]
    t2 =  time.time()
    feature = feature[:, :, permutation]
    t3 = time.time()
    print(t2 - t1) # running for 0.24s
    print(t3 - t2)  # 0.00...  very short anyway

When I change the sequence of these tow:
def forward(feature, coords, permutation):
    """
    feature : [B, d, N, 1]
    coords: [B, N ,3]
    """
    t1 = time.time()
    feature = feature[:, :, permutation]
    t2 =  time.time()
    coords: [:, permutation]
    t3 = time.time()
    print(t2 - t1) # running for 0.24s
    print(t3 - t2)  # 0.00...  very small anyway

It seems that the what really counts is its sequence instead of what the code truly does.
Second:
I was debugging my code using time.time() like before, and the time consumption was becoming even weirder. In my second model, I found a very time consuming line : a[b] , especially when there is an action that put b.to(device), it is just a simple slice operation but it took 0.4s+, 95%+ of the whole model. So I change a way to get the slice, it decreased to several ms like it is solved. However, unfortunately, I found the total time still remain unchanged , what is more, I found another line of code became suddenly very time consuming with no reason just like a ghost with absolutely no changes done to it. So I did rewrite the sentence again. But it is vain, because the “time consuming” part just moved to next part like a virus.
However I found a common feature between them. Every “ghost time consuming” part will be related to (to(device) / device / … etc.) , and when you fix one of them, another one will just become the “ghost time consuming” part again.
Is there a preparation stage of GPU like it can’t be removed ? If yes, why dose every step needs this ? Literally every step in my iteration has a “ghost” time spot.
Yours sincerely,
ZD


title: Switching model during training in Pytorch Lightening

body:
I am experiencing knowledge distillation in Incremental Learning. Basically , at each phase I init a new model and train it on the current data as a student model , and use the old model ( trained on previous phase) as a teacher. So far , there's no error in the code but the problem is that the loss never decrease after each phase. Also, is there a way in PL to init new optimizer in on_train_epoch_start ?
def on_train_epoch_start(self) :
   if self.new_phase:
       self.old_backbone = copy.deepcopy(self.backbone)
       self.old_head = copy.deepcopy(self.head)
       self.backbone = None
       self.head = None
       for p in self.old_backbone.parameters():
           p.requires_grad = False
       for p in self.old_head.parameters():
           p.requires_grad = False
       self.backbone = create_backbone(model_name=self.params.backbone_name,
                                        **self.params.backbone_params)
        # create LINEAR head
       self.params.head_params['in_features'] = self.backbone.num_features
       self.head = HEADS.get(self.params.head_name)(**self.params.head_params)
       self.backbone = self.backbone.to(self.device)
       self.head = self.head.to(self.device)
       self.old_backbone.eval()
       self.old_head.eval()
       self.backbone.train()
       self.head.train()


title: Strange behaviour with qnnpack accuracy

body:
I'm working with timms mobilenetv2_100
import timm
model = timm.create_model('mobilenetv2_100')

and FX post-training static quantization.
I'm getting a very strange behaviour regarding the quantized model accuracy. I'd be happy to provide more detail if this question gets interest but for now here are the clues:

Using get_default_qconfig("fgemm") I get 100% accuracy (I'm only testing 10 samples so this is fair).
Using get_default_qconfig("qnnpack") I get 0% accuracy BUT read on for the interesting clues.
If I only quantize some of the backbone blocks rather than the whole model I can recover 100% accuracy with get_default_qconfig("qnnpack")

Quantize only blocks [0] -> 100%
[0, 2, 3, 4, 5] -> 100%
[0, 1, 2, 3, 4, 5] -> 0%
[1] -> 100%
[0, 2, 3, 4, 5, 6] -> 0%
[6] -> 0%

All the above results are gathered by running the model with the regular torch backend. When I set `torch.backends.quantized.engine = 'qnnpack' even the cases where I previously recovered 100% go to 0%.

Where could I go from here to understand what's going on? I'm new to quantization so I'm not necessarily aware of the options.
Could it be that there's some under/overflow issue which just randomly happens according to some combination of blocks quantized?
For reference, the blocks of the model look like:
(blocks): Sequential(
    (0): Sequential(
      (0): DepthwiseSeparableConv(
        (conv_dw): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=32, bias=False)
        (bn1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (se): Identity()
        (conv_pw): Conv2d(32, 16, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn2): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): Identity()
      )
    )
    (1): Sequential(
      (0): InvertedResidual(
        (conv_pw): Conv2d(16, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(96, 96, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=96, bias=False)
        (bn2): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(96, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): InvertedResidual(
        (conv_pw): Conv2d(24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(144, 144, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=144, bias=False)
        (bn2): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(144, 24, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(24, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (2): Sequential(
      (0): InvertedResidual(
        (conv_pw): Conv2d(24, 144, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(144, 144, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=144, bias=False)
        (bn2): BatchNorm2d(144, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(144, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): InvertedResidual(
        (conv_pw): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)
        (bn2): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): InvertedResidual(
        (conv_pw): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(192, 192, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=192, bias=False)
        (bn2): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(192, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (3): Sequential(
      (0): InvertedResidual(
        (conv_pw): Conv2d(32, 192, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(192, 192, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=192, bias=False)
        (bn2): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(192, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): InvertedResidual(
        (conv_pw): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)
        (bn2): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): InvertedResidual(
        (conv_pw): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)
        (bn2): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (3): InvertedResidual(
        (conv_pw): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)
        (bn2): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(384, 64, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (4): Sequential(
      (0): InvertedResidual(
        (conv_pw): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(384, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=384, bias=False)
        (bn2): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(384, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): InvertedResidual(
        (conv_pw): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)
        (bn2): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): InvertedResidual(
        (conv_pw): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(576, 576, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=576, bias=False)
        (bn2): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(576, 96, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (5): Sequential(
      (0): InvertedResidual(
        (conv_pw): Conv2d(96, 576, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(576, 576, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), groups=576, bias=False)
        (bn2): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(576, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (1): InvertedResidual(
        (conv_pw): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
        (bn2): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (2): InvertedResidual(
        (conv_pw): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
        (bn2): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(960, 160, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (6): Sequential(
      (0): InvertedResidual(
        (conv_pw): Conv2d(160, 960, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act1): ReLU6(inplace=True)
        (conv_dw): Conv2d(960, 960, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), groups=960, bias=False)
        (bn2): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (act2): ReLU6(inplace=True)
        (se): Identity()
        (conv_pwl): Conv2d(960, 320, kernel_size=(1, 1), stride=(1, 1), bias=False)
        (bn3): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )