title: Pytorch model loading performance

body:
Does the performance of loading a model here seem correct?

Total time: 5.4545 s
Function: load_model at line 81

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
    81                                            @profile
    82                                            def load_model(dirname, device):
    82                                            """ load model from disk """
    83         1         90.0      90.0     0.0       device = torch.device(device)
    84         1         26.0      26.0     0.0       modelfile = os.path.join(dirname, 'model.py')
    85         1          7.0       7.0     0.0       weights = os.path.join(dirname, 'weights_%s.tar' % weights)
    86         1    5379756.0 5379756.0     98.6      model = torch.load(modelfile, map_location=device)
    87         1      72966.0   72966.0     1.3       model.load_state_dict(torch.load(weights, map_location=device))
    88         1       1648.0    1648.0     0.0       model.eval()
    89         1          1.0       1.0     0.0       return model

The initial called to load a 26MB model on line 86 is ~100 times slower than loading the 26MB checkpoint on 87 and 5.45 seconds is a lot longer than I would expect?

I'm using pytorch 1.2 and a device="cuda" and have confirmed the same performance with 1.3.1. Adding in calls to sync confirms that all of the time is actually spend in torch.load.

    86         1    5395545.0 5395545.0     98.6      model = torch.load(modelfile, map_location=device)
    87         1         76.0      76.0      0.0      torch.cuda.synchronize(device=device)
    88         1      72403.0   72403.0      1.3      model.load_state_dict(torch.load(weights, map_location=device))
    89         1         52.0      52.0      0.0      torch.cuda.synchronize(device=device)
    90         1       1640.0    1640.0      0.0      model.eval()
    91         1         21.0      21.0      0.0      torch.cuda.synchronize(device=device)

Am I doing something wrong here or is this typical?


title: What is a good way to construct a PyTorch Dataset that loads (an unknown number of) slices from 3D images?

body:
I am trying to construct a PyTorch Dataset that returns 21x512x512 slices from 3D images of shape ?x512x512. I know how many images there are, but I do not know how many slices there are in each image. Therefore, I would intuitively make the __len__() function of the Dataset return the total number of images I have. I could technically check the shapes of all images beforehand, but the dataset might change over time, so I would greatly prefer a scalable software solution.

With this, I need some functionality that breaks the image into slices (of size mentioned above), and return these instead of the whole image. This is also not a problem, I have a function that can do this.

Here comes the problem. If I add this slicing functionality in the __getitem__() function of the Dataset, then I will only get one slice per image, since the PyTorch DataLoader will think there are len(dataset) datapoints, which is not the case anymore. But I also cannot specify the correct number of samples, as I do not know it beforehand.

I tried some solutions:

Return a generator function in __getitem__() which yields slices per image. This does not work because __getitem__() needs to return something of type list, tuple, tensor etc.
Just return the whole image and break it up in the train loop. This can work, but is both bad programming style (as I want to hide data selection in the dataset) and not very compatible with the batching of the DataLoader, as one image might have 100 slices in it, while another might only have 5 slices. In this case, making batches from these images would result in very only 5 batches with the actual batch_size, and 95 others with less samples per batch. Resolving this would require some ugly check to see if another image needs to be loaded, which I again would like to hide in a Dataset.
Yield results in a for loop in the __getitem__() function of the Dataset. This does not work for the same reason as point 1: a generator cannot be returned in a Dataset.

So in short, what is a clean way to load an unknown number of slices from 3D images in a PyTorch Dataset?


title: Resolution preserving Fully Convolutional Network

body:
I am new to ML and Pytorch and I have the following problem:

I am looking for a Fully Convolutional Network architecture in Pytorch, so that the input would be an RGB image (HxWxC or 480x640x3) and the output would be a single channel image (HxW or 480x640). In other words, I am looking for a network that will preserve the resolution of the input (HxW), and will loose the channel dimension. All of the networks that I've came across (ResNet, Densenet, ...) end with a fully connected layer (without any upsampling or deconvolution). This is problematic for two reasons:

I am restricted with the choice of the input size (HxWxC).
It has nothing to do with the output that I expect to get (a single channel image HxW).

What am I missing? Why is there even a FC layer? Why is there no up-sampling, or  some deconvolution layers after feature extraction? Is there any build-in torchvision.model that might suit my requirements? Where can I find such pytorch architecture? As I said, I am new in this field so I don't really like the idea of building such a network from scratch.

Thanks.


title: How to output the loss gradient backpropagation path through a PyTorch computational graph

body:
I have implemented a new loss function in PyTorch.

#model_1 needs to be trained
outputs = model_1(input)
loss = myloss(outputs,labels)

#output is how much to resize an image
#label give the image file index
#Below I am explaining myloss() function
org_file_name = pic_ + str(labels[0]) + ".png"
new_image = compress(org_image,outputs)
accuracy_loss = run_pretrained_yolov3(org_image, new_image)

#next two lines are modifying the same DAG
prev_loss = torch.mean((outputs-labels)**2)
new_loss = (accuracy_loss/prev_loss.item())*prev_loss
new_loss.backward()

Can anyone plz help me suggesting how can I know regarding how the loss gradient backpropagation through the computational graph?

[i.e., Actually, inside the myloss() function, I used some other pre-trained model applied in testing mode to get the difference or final loss value.] Now I want to know whether my new_loss.grad backpropagated through model1 or first through yolov3 then through model1? pretrained yolov3 is used on testing mode only.

I have tried tensorboard, it's not providing me that option. Any suggestions will be highly helpful.


title: Python code slowing down when wrapped in a function

body:
I'm reading and processing a file (using the same bit of code) which runs at two very different speeds: 1. scripted (50K+ iterations per second) and 2. wrapped in function (~300 iterations per second). I really can't figure out why there are such massive differences in reading time consumption.

The module structure (unused and irrelevant files omitted. The code is at the end.):

| experiments/
|--| experiment_runner.py
|
| module/
|--| shared/
|--|--| dataloaders.py
|--|--| data.py

In data.py we have the method (load, the class wrapping the method inherits from torch.utils.data.Dataset) actually loading the file. In dataloaders.py I prepare the arguments to pass to load, wrapped in a function for each dataset that I'm using. This is then passed to a loader function, which handles splitting the dataset and so on.

In experiment_runner then is where the differences in speed happen. If I use the dataset functions in dataloaders.py the loading happens at around 300 iterations/second. If I copy the code from the function and throw it directly into the experiment_runner, still using the loader function from dataloaders.py (so, not wrapped in a function for each dataset), the loading happens at roughly 50000 iterations / second. I am at a complete loss as to why wrapping code in a function would alter the speed of it that drastically.

Now the actual code:

data.py:

    def load(self, dataset: str = 'train', skip_header = True, **kwargs) -> None:
        fp = open(self.data_files[dataset])

        if skip_header:
            next(fp)

        data = []
        for line in tqdm(self.reader(fp), desc = f'loading {self.name} ({dataset})'):
            data_line, datapoint = {}, base.Datapoint()

            for field in self.train_fields:
                idx = field.index if self.ftype in ['CSV', 'TSV'] else field.cname
                data_line[field.name] = self.process_doc(line[idx].rstrip())
                data_line['original'] = line[idx].rstrip()

            for field in self.label_fields:
                idx = field.index if self.ftype in ['CSV', 'TSV'] else field.cname
                if self.label_preprocessor:
                    data_line[field.name] = self.label_preprocessor(line[idx].rstrip())
                else:
                    data_line[field.name] = line[idx].rstrip()

            for key, val in data_line.items():
                setattr(datapoint, key, val)
            data.append(datapoint)
        fp.close()

        if self.length is None:
            # Get the max length
            lens = []
            for doc in data:
                for f in self.train_fields:
                    lens.append(len([tok for tok in getattr(doc, getattr(f, 'name'))]))
            self.length = max(lens)

        if dataset == 'train':
            self.data = data
        elif dataset == 'dev':
            self.dev = data
        elif dataset == 'test':
            self.test = data

dataloaders.py:

def loader(args: dict, **kwargs):
    """Loads the dataset.
    :args (dict): Dict containing arguments to load dataaset.
    :returns: Loaded and splitted dataset.
    """
    dataset = GeneralDataset(**args)
    dataset.load('train', **kwargs)

    if (args['dev'], args['test']) == (None, None):  # Only train set is given.
        dataset.split(dataset.data, [0.8, 0.1, 0.1], **kwargs)

    elif args['dev'] is not None and args['test'] is None:  # Dev set is given, test it not.
        dataset.load('dev', **kwargs)
        dataset.split(dataset.data, [0.8], **kwargs)

    elif args['dev'] is None and args['test'] is not None:  # Test is given, dev is not.
        dataset.split(dataset.data, [0.8], **kwargs)
        dataset.dev_set = dataset.test
        dataset.load('test', **kwargs)

    else:  # Both dev and test sets are given.
        dataset.load('dev', **kwargs)
        dataset.load('test', **kwargs)

    return dataset

def binarize(label: str) -> str:
    if label in ['0', '1']:
        return 'pos'
    else:
        return 'neg'

def datal(path: str, cleaners: base.Callable, preprocessor: base.Callable = None):
    args = {'data_dir': path,
            'ftype': 'csv',
            'fields': None,
            'train': 'dataset.csv', 'dev': None, 'test': None,
            'train_labels': None, 'dev_labels': None, 'test_labels': None,
            'sep': ',',
            'tokenizer': lambda x: x.split(),
            'preprocessor': preprocessor,
            'transformations': None,
            'length': None,
            'label_preprocessor': binarize,
            'name': 'First dataset.'
            }

    ignore = base.Field('ignore', train = False, label = False, ignore = True)
    d_text = base.Field('text', train = True, label = False, ignore = False, ix = 6, cname = 'text')
    d_label = base.Field('label', train = False, label = True, cname = 'label', ignore = False, ix = 5)

    args['fields'] = [ignore, ignore, ignore, ignore, ignore, d_label, d_text]

    return loader(args)

And for the purposes:
experiment_runner.py

from module.dataloaders import datal, loader

dataset = datal() # Slow: 300-ish iterations/second

# Fast version: 50000 iter/second
def binarize(label: str) -> str:
    if label in ['0', '1']:
        return 'pos'
    else:
        return 'neg'

args = {'data_dir': path,
        'ftype': 'csv',
        'fields': None,
        'train': 'dataset.csv', 'dev': None, 'test': None,
        'train_labels': None, 'dev_labels': None, 'test_labels': None,
        'sep': ',',
        'tokenizer': lambda x: x.split(),
        'preprocessor': preprocessor,
        'transformations': None,
        'length': None,
        'label_preprocessor': binarize,
        'name': 'First dataset.'
        }

ignore = base.Field('ignore', train = False, label = False, ignore = True)
d_text = base.Field('text', train = True, label = False, ignore = False, ix = 6, cname = 'text')
d_label = base.Field('label', train = False, label = True, cname = 'label', ignore = False, ix = 5)

args['fields'] = [ignore, ignore, ignore, ignore, ignore, d_label, d_text]

dataset = loader(args)

I would ideally prefer to keep the dataset functions (e.g. datal) wrapped to keep the logic separate but with this speed decrease, that's not feasible.


title: Custom layer with B-splines approximation and backpropagation

body:
I am currently trying to implement a neural network with a custom layer.
This custom layer takes as input a sequence of 2d points p and a parameterization t for these points
and approximates the sequence of points p using BSplines functions. The loss is then calculated as the
distance between the sequence of points p and the approximation calculated by this approach.

Checking Scipy's documentation I found

https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.BSpline.html#scipy.interpolate.BSpline

which allows you to represent a Bspline and

https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.make_lsq_spline.html

usable to approximate a sequence of points provided with parameterization using Bsplines.

So I was wondering in what way I could implement this layer in PyTorch in order to correctly perform
backpropagation. It is possible to somehow reuse the methods in the links above that use Numpy arrays
or is it necessary to implement these operations using PyTorch Tensors?


title: How to load images from a folder whose names are stored in a csv file in pytorch

body:
I have a csv file which has the names of the specific images which have to be loaded into a dataloader from a folder. This csv file also contains the labels/classes of the images. I tried using a dataset class. I had written the below code:

class data(Dataset):
   def __init__(self,type='train',transform=None):
     self.imagecsv=''
     if type=='train':
       self.imagecsv=traincsv
     else:
       self.imagecsv=testcsv
   def __getitem__(self):
     imagelabels=self.imagecsv[['image_id','Labels']]
     images=[]
     for i in imagelabels['image_id']:
       image=imread(os.path.join(path,i+'.jpg'))
       images.append(image)
     imagelabels['images']=images
     imagelabels.drop(columns='image_id')
     return imagelabels

When I try to run this code, the system crashes everytime. I am running it on google colab with a GPU.  I know I haven't added an index parameter in the getitem function, it is because I need all the images whose names are specified in the training csv file. Is there a cleaner implementation which won't crash my system? I also tried using Image.open but the system crashed again.


title: Unknown behaviour of HOOKS in PyTorch

body:
I have a straightforward and simple CNN below,

# creat a dummy deep net
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1,2, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv2 = nn.Conv2d(2,3, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv3 = nn.Conv2d(3,1, kernel_size=3, stride=1, padding=1, bias=True)
        self.seq = nn.Sequential(
                    nn.Conv2d(1,5, kernel_size=3, stride=1, padding=1, bias=True),
                    nn.LeakyReLU(negative_slope=0.2, inplace=True),
                    nn.Conv2d(5,1, kernel_size=3, stride=1, padding=1, bias=True),
                    )
        self.relu = nn.LeakyReLU(negative_slope=0.2, inplace=True)

    def forward(self, x):

        out = self.relu(self.conv1(x))
        out = self.conv3(self.conv2(out))
        out = out + x
        out = self.seq(x)

        return out

5 hooks have been applied to each layer for the forward pass.

Hooked 0 to Conv2d(1, 2, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Hooked 1 to Conv2d(2, 3, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Hooked 2 to Conv2d(3, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
Hooked 3 to Sequential(
  (0): Conv2d(1, 5, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (1): LeakyReLU(negative_slope=0.2, inplace=True)
  (2): Conv2d(5, 1, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
)
Hooked 4 to LeakyReLU(negative_slope=0.2, inplace=True)

These hooks have been created using following class

# ------------------The Hook class begins to calculate each layer stats
class Hook():
    def __init__(self, module, backward=False):
        if backward==False:
            self.hook = module.register_forward_hook(self.hook_fn)
        else:
            self.hook = module.register_backward_hook(self.hook_fn)

        self.inputMean = []
        self.outputMean = []

    def hook_fn(self, module, input, output):
        self.inputMean.append(input[0][0,...].mean().item())#calculate only for 1st image in the batch
        print('\nIn hook class input {}'.format(input[0].size()))
        self.outputMean.append(output[0][0,...].mean().item())
        print('In hook class outout {}'.format(output[0].size()))

# create hooks on each layer
hookF = []
for i,layer in enumerate(list(net.children())):
    print('Hooked to {}'.format(layer))
    hookF.append(Hook(layer))

Please note between Hook 1 and Hook 2 there is no ReLU
self.conv3(self.conv2(out)). Thus OUTPUT of HOOK1 is INPUT to HOOK2 and should be identical. BUT THIS DOES NOT TURNS OUT TO BE WHY? Below is output for HOOK1 and HOOK2

Hook of layer 1 (HOOK on layer 1 which is self.conv2)
... OutputMean: [0.2381615787744522, 0.2710852324962616, 0.30706286430358887, 0.26064932346343994, 0.24395985901355743]

 Hook of layer 2 (HOOK on layer 2 which is self.conv3)
InputMean: [0.13127394020557404, 0.1611362248659134, 0.1457807868719101, 0.17380955815315247, 0.1537724733352661], OutputMean: ...

These two values should have been the same but do not turn out to be.

------ The Full code is shown below -------

import torch
import torch.nn as nn

# creat a dummy deep net
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1,2, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv2 = nn.Conv2d(2,3, kernel_size=3, stride=1, padding=1, bias=True)
        self.conv3 = nn.Conv2d(3,1, kernel_size=3, stride=1, padding=1, bias=True)
        self.seq = nn.Sequential(
                    nn.Conv2d(1,5, kernel_size=3, stride=1, padding=1, bias=True),
                    nn.LeakyReLU(negative_slope=0.2, inplace=True),
                    nn.Conv2d(5,1, kernel_size=3, stride=1, padding=1, bias=True),
                    )
        self.relu = nn.LeakyReLU(negative_slope=0.2, inplace=True)

    def forward(self, x):

        out = self.relu(self.conv1(x))
        out = self.conv3(self.conv2(out))
        out = out + x
        out = self.seq(x)

        return out

net = Net()
print(net)
criterion = nn.MSELoss()

# ------------------The Hook class begins to calculate each layer stats
class Hook():
    def __init__(self, module, backward=False):
        if backward==False:
            self.hook = module.register_forward_hook(self.hook_fn)
        else:
            self.hook = module.register_backward_hook(self.hook_fn)

        self.inputMean = []
        self.outputMean = []

    def hook_fn(self, module, input, output):
        self.inputMean.append(input[0][0,...].mean().item())#calculate only for 1st image in the batch
        print('\nIn hook class input {}'.format(input[0].size()))
        self.outputMean.append(output[0][0,...].mean().item())
        print('In hook class outout {}'.format(output[0].size()))

# create hooks on each layer
hookF = []
for i,layer in enumerate(list(net.children())):
    print('Hooked to {}'.format(layer))
    hookF.append(Hook(layer))

optimizer = torch.optim.Adam(net.parameters())

# Do 5 forward pass
for _ in range(5):
    print('Iteration --------')
    data = torch.rand(2,1,10,10)*10
    print('Input mean is {}'.format(data[0,...].mean()))
    target = data.clone()

    out = net(data)
    loss = criterion(out, target)
    print('backward')
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

for i,h in enumerate(hookF):
    print('\n Hook of layer {}'.format(i))
    print('InputMean: {}, OutputMean: {}'.format(h.inputMean, h.outputMean))
    h.hook.remove()


title: how convert string to path in opemnmt-py

body:
I use opennmt-py for MT and in the code any time I want to set a path I have to write all directory and it's not good looking when I have long directory. is there any way to set a string as the main directory and just add the file name to the end.
I use google colab to train the model
The code is like:

!onmt_preprocess  \\
-train_src //content//drive//My\ Drive//Colab\ Notebooks//NLP//spring99//CA6//Corpora//En2Fa-Translation//train.en \\
-train_tgt //content//drive//My\ Drive//Colab\ Notebooks//NLP//spring99//CA6//Corpora//En2Fa-Translation//train.fa \\
-valid_src //content//drive//My\ Drive//Colab\ Notebooks//NLP//spring99//CA6//Corpora//En2Fa-Translation//dev.en \\
-valid_tgt  //content//drive//My\ Drive//Colab\ Notebooks//NLP//spring99//CA6//Corpora//En2Fa-Translation//dev.fa \\
-save_data //content//drive//My\ Drive//Colab\ Notebooks//NLP//spring99//CA6//Corpora//En2Fa-Translation//demo//

and the code I want to be like:

path ='//content//dri`ve//My\ Drive//Colab\ Notebooks//NLP//spring99//CA6//Corpora//En2Fa-Translation//'

!onmt_preprocess  \\
-train_src path+'train.en' \\
-train_tgt path+'train.fa' \\
-valid_src path+'dev.en' \\
-valid_tgt  path++'dev.fa' \\
-save_data path+'demo//'

or maybe just can write all path in a variable and use it like:

path_train ='//content//dri`ve//My\ Drive//Colab\ Notebooks//NLP//spring99//CA6//Corpora//En2Fa-Translation//'

!onmt_preprocess  \\
-train_src path_train \\


title: How to get an individual weight in pytorch neural network model without reference

body:
I want to change individual weight's value in the neural network in pytorch(it is fully connected network with 2 layers).

for k in range(len(list(net.parameters()))):
  cur_params = list(net.parameters())[k]
  cur_params_size = list(list(net.parameters())[k].size())
  if(len(cur_params_size) == 2):
    for l in range(len(cur_params)):
      for m in range(len(cur_params[l])):
        weight_val = cur_params[l][m]
        print(weight_val)
        with torch.no_grad():
          next(islice(net.parameters(), k, None))[l][m] = 0
        print(weight_val)

This leads to the second printed weight_val to be changed to 0.
I think saved weight_val is iterator, so changing parameter to 0 leads to changing weight_val to 0.
Is there a standard way to get an individual weight value without reference, or any way to fix this problem?

I tried to solve this with code:

weight_val = [cur_params[l][m]]
weight_val = copy.deepcopy(weight_val)

but the error:

RuntimeError: Only Tensors created explicitly by the user (graph leaves) support the deepcopy protocol at the moment

has occurred.


title: How to define edge_func parameter in dgl.nn.pytorch.conv.NNConv?

body:
I have a graph with nodes, edges, and edge features. I want to make a model with SAGEConv but SAGEConv only look at node features. So I thought of making node feature by aggregating edge features of each node and I found out about NNConv, where they're able to do that. The problem is I'm not sure how to use NNConv, especially what do I do with edge_func that is required as a parameter for NNConv?

Let say I have 3 nodes and 4 edges, each edge has 2 features, and I want the each nodes to have 2 features as well. I defined edge_func to be

  edge_func = nn.Sequential(nn.Linear(2, 2), nn.ReLU(), nn.Linear(2, 4))

and edge_feature to be

  tensor([[2., 3.],
          [3., 1.],
          [0., 4.],
          [1., 1.]])

and build a DGLGraph with that. However after computing the layer

  NNConv(in_feats=2, out_feats=2, edge_nn, aggregator_type='max')

in the forward function

  feature = self.layer(self.graph, feature, edge_feature)

returns feature (it was originally tensor of zeros) as

  tensor([[-inf, -inf],
          [0., 0.],
          [0., 0.]], grad_fn=<AddBackward0>)

What am I doing wrong?


title: How to use PyTorch's torchaudio in Android?

body:
Starting from here, I downloaded the tutorial project and got it to build and run. Then I tried adding this to the project's app build.gradle (after upping the pytorch version to 1.5.0):
implementation 'org.pytorch:pytorch_android_torchaudio:1.5.0'

And I got this error:
Could not find org.pytorch:pytorch_android_torchaudio:1.5.0.

Anyone else had any luck getting PyTorch's torchaudio to work in Android?


title: Best way to apply per neuron activation functions in pytorch

body:
I want to use a custom activation function that has a random component that gets applied to every neuron individually.
If I use the standard method and call the activation function on a layer, it applies the same value to every neuron in that layer.
I am looking for the most efficient way to have the activation function affect every neuron individually and would appreciate any advise on the topic.
The only method I currently know about involves splitting the input tensor and applying the function each part and then concatenating the result, but I am hoping there is a more efficient way


title: How do I limit using one CPU per python processes launched via gnu parallel?

body:
If I run this script
$ seq 1 4 | taskset -c 0-3 parallel -j4 -u <my_bash_script.sh>

Then each python process contained in the <my_bash_script.sh> runs on multiple cpus instead of one. The python function use both numpy and pytorch. So the option taskset -c 0-4 impose the max number of CPUs but it doesn't guarantee that each process will be limited to one CPU.
I've tried
$ export OPENBLAS_NUM_THREADS=1
$ export MKL_NUM_THREADS=1

but it didn't work
I've also added to the python script
import mkl
mkl.set_num_threads(1)

but it didn't help


title: What does this strange C++ syntax do?

body:
From what I understand, 3 arguments are being passed to m.def(). I don't understand the syntax of the second argument passed, i.e. [] (Observable...){} What does this mean?
m.def(
  "DumpHistogramFile",
  [](Observable<NetBase>::Observer* ob) {
    HistogramNetObserver* hist_ob =
        dynamic_cast_if_rtti<HistogramNetObserver*>(ob);
    hist_ob->DumpHistogramFile();
  },
  pybind11::arg("ob"));

I found this syntax in the pytorch source code


title: Is there MBART tensorflow 2.0 model ready for use?

body:
I am looking for MBART pretrained model and found that was published in hugging face. https://huggingface.co/facebook/mbart-large-cc25#. But it seems like there is only a pytorch version. When clicking the "List all files in model" link, there is only a pytorch model file

Also, when looking the document for MBART, there is no class like "TFMBart".
I am wondering whether there is a tf2.0 version for this model. If not, can I convert this pytorch version to tf2.0 version?


title: PyTorch Conv2D returns non-zero output for an input tensor of zeros?

body:
If you input an array consisting only of zeros to a Conv2D layer, the output should also consist only of zeros. In TensorFlow, this is the case. But, in PyTorch, it isn't. Here is some very simple sample Python code to demonstrate this. Why does PyTorch output non-zero numbers in this situation?
import torch
import numpy as np

image = np.zeros((3,3,3), dtype=np.float32)
batch = np.asarray([image])

a = torch.nn.Conv2d(3,3,1)
b = a(torch.tensor(batch).permute(0,3,1,2))

print(b.permute(0,2,3,1))


title: "view_as_windows" from skimage but in Pytorch

body: Is there any Pytorch version of view_as_windows from skimage? I want to create the view while the tensor is on the GPU.


title: Sequence of for loops to access different dimensions of a tensor

body:
My problem is a bit hard to explain.

I want to update probability distributions recursively in [0, 1]^n.
I discretized it with 10 bins per dimension, so there are 10^n cells indexed by an n-tuple.
Initially, I should have a uniform distribution; but then I take a threshold parameter that dictates how I propagate the distribution: every point that is "below" all the n thresholds must receive zero probability.

Here is how I hardcoded it for 1 or 3 dimensions:
import torch
import numpy as np

dim = 3 # or 1
bins = 10

# form the vector [10, 10, ..., 10] (n dimensions)
probabilityStarter = []
for counter in range(dim):
    probabilityStarter.append(bins)

# form the n-cube with 10^n cells
probabilityDistribution = torch.ones(probabilityStarter, dtype = torch.int8)
zeroProbability = torch.ones(probabilityStarter, dtype = torch.int8)

# if lower than the threshold, I want to zero the probability
threshold = 0.5

# for 1 dim, I just need to re-arrange, skewing the probabilities "to the right"
if dim == 1:
    for j in range(bins):
        step = (j + 0.5) / bins
        if step < threshold:
            probabilityDistribution[j] = 0

    probabilityDistribution = probabilityDistribution.float()
    probabilityDistribution = probabilityDistribution / torch.sum(probabilityDistribution)

# for n dim, I need to do n for loops and update the d-th dimension [:, ..., d, :, ..., :]
elif dim == 3:
    for j in range(bins):
        step = (j + 0.5) / bins
        if step > threshold:
            zeroProbability[j, :, :] = 0 # for dim 1

    for j in range(bins):
        step = (j + 0.5) / bins
        if step > threshold:
            zeroProbability[:, j, :] = 0 # for dim 2

    for j in range(bins):
        step = (j + 0.5) / bins
        if step > threshold:
            zeroProbability[:, :, j] = 0 # for dim 3

    probabilityDistribution = probabilityDistribution - zeroProbability
    probabilityDistribution = probabilityDistribution.float()
    probabilityDistribution = probabilityDistribution / torch.sum(probabilityDistribution)

However, if I want the 'dim' to be dynamic, how could I implement it?


title: How to alternatively train 3 models with shared encoders in pytorch?

body:
I want to build 3 U-Net like models which share the same encoder but have independent decoders (first 2 models are the same as U-Net for segmentation, and the third one without short connection is for reconstruction). And I want to train the 3 models alternatively because I have 3 different datasets with their own labels.
model0, model1, model2 = build3models()
models = [model0, model1, model2]

dataloaders = [None, None, None]
dataloaders[0] = Dataloader_for_model0
dataloaders[1] = Dataloader_for_model1
dataloaders[2] = Dataloader_for_model2

max_steps = 10000
for step in range(max_steps):
  for data, model in zip(dataloaders, models]:
    x, y = next(dataloader[i])
    pred = model(x)
    opt = torch.optim.Adam(model.parameters(), lr=1e-4)
    opt.zero_grad()
    criterion = nn.MSE()
    loss = criterion_seg(pred , y)
    loss.backward()
    opt.step()

My question is: How to build 3 models with shared encoder and train them independently and alternatively with different loss/opt/lr etc. for different models?
I have already built the following encoder and decoders, but I donot know how to connect and use them correctly.
class Encoder(nn.Module):
    def __init__(
            self,
            dimensions: int = 3,
            in_channels: int = 1,
            features: Sequence[int] = (32, 32, 64, 128, 256, 32),
            act: Union[str, tuple] = ("LeakyReLU", {"negative_slope": 0.1, "inplace": True}),
            norm: Union[str, tuple] = ("instance", {"affine": True}),
            dropout: Union[float, tuple] = 0.0,
            upsample: str = "deconv",
    ):
        """
        A UNet implementation with 1D/2D/3D supports.

        Based on:

            Falk et al. "U-Net – Deep Learning for Cell Counting, Detection, and
            Morphometry". Nature Methods 16, 67–70 (2019), DOI:
            http://dx.doi.org/10.1038/s41592-018-0261-2

        Args:
            dimensions: number of spatial dimensions. Defaults to 3 for spatial 3D inputs.
            in_channels: number of input channels. Defaults to 1.
            features: six integers as numbers of features.
                Defaults to ``(32, 32, 64, 128, 256, 32)``,

                - the first five values correspond to the five-level encoder feature sizes.
                - the last value corresponds to the feature size after the last upsampling.

            act: activation type and arguments. Defaults to LeakyReLU.
            norm: feature normalization type and arguments. Defaults to instance norm.
            dropout: dropout ratio. Defaults to no dropout.
            upsample: upsampling mode, available options are
                ``"deconv"``, ``"pixelshuffle"``, ``"nontrainable"``.

        Examples::

            # for spatial 2D
            >>> net = BasicUNet(dimensions=2, features=(64, 128, 256, 512, 1024, 128))

            # for spatial 2D, with group norm
            >>> net = BasicUNet(dimensions=2, features=(64, 128, 256, 512, 1024, 128), norm=("group", {"num_groups": 4}))

            # for spatial 3D
            >>> net = BasicUNet(dimensions=3, features=(32, 32, 64, 128, 256, 32))

        See Also

            - :py:class:`monai.networks.nets.DynUNet`
            - :py:class:`monai.networks.nets.UNet`

        """
        super().__init__()

        fea = ensure_tuple_rep(features, 6)
        print(f"BasicUNet features: {fea}.")

        self.conv_0 = TwoConv(dimensions, in_channels, features[0], act, norm, dropout)
        self.down_1 = Down(dimensions, fea[0], fea[1], act, norm, dropout)
        self.down_2 = Down(dimensions, fea[1], fea[2], act, norm, dropout)
        self.down_3 = Down(dimensions, fea[2], fea[3], act, norm, dropout)
        self.down_4 = Down(dimensions, fea[3], fea[4], act, norm, dropout)

    def forward(self, x: torch.Tensor):
        """
        Args:
            x: input should have spatially N dimensions
                ``(Batch, in_channels, dim_0[, dim_1, ..., dim_N])``, N is defined by `dimensions`.
                It is recommended to have ``dim_n % 16 == 0`` to ensure all maxpooling inputs have
                even edge lengths.

        Returns:
            A torch Tensor of "raw" predictions in shape
            ``(Batch, out_channels, dim_0[, dim_1, ..., dim_N])``.
        """
        x0 = self.conv_0(x)

        x1 = self.down_1(x0)
        x2 = self.down_2(x1)
        x3 = self.down_3(x2)
        x4 = self.down_4(x3)

        return x0, x1, x2, x3, x4

class DecoderSeg(nn.Module):
    def __init__(
            self,
            dimensions: int = 3,
            in_channels: int = 1,
            out_channels: int = 2,
            features: Sequence[int] = (32, 32, 64, 128, 256, 32),
            act: Union[str, tuple] = ("LeakyReLU", {"negative_slope": 0.1, "inplace": True}),
            norm: Union[str, tuple] = ("instance", {"affine": True}),
            dropout: Union[float, tuple] = 0.0,
            upsample: str = "deconv",
    ):
        """
        A UNet implementation with 1D/2D/3D supports.

        Based on:

            Falk et al. "U-Net – Deep Learning for Cell Counting, Detection, and
            Morphometry". Nature Methods 16, 67–70 (2019), DOI:
            http://dx.doi.org/10.1038/s41592-018-0261-2

        Args:
            dimensions: number of spatial dimensions. Defaults to 3 for spatial 3D inputs.
            in_channels: number of input channels. Defaults to 1.
            out_channels: number of output channels. Defaults to 2.
            features: six integers as numbers of features.
                Defaults to ``(32, 32, 64, 128, 256, 32)``,

                - the first five values correspond to the five-level encoder feature sizes.
                - the last value corresponds to the feature size after the last upsampling.

            act: activation type and arguments. Defaults to LeakyReLU.
            norm: feature normalization type and arguments. Defaults to instance norm.
            dropout: dropout ratio. Defaults to no dropout.
            upsample: upsampling mode, available options are
                ``"deconv"``, ``"pixelshuffle"``, ``"nontrainable"``.

        Examples::

            # for spatial 2D
            >>> net = BasicUNet(dimensions=2, features=(64, 128, 256, 512, 1024, 128))

            # for spatial 2D, with group norm
            >>> net = BasicUNet(dimensions=2, features=(64, 128, 256, 512, 1024, 128), norm=("group", {"num_groups": 4}))

            # for spatial 3D
            >>> net = BasicUNet(dimensions=3, features=(32, 32, 64, 128, 256, 32))

        See Also

            - :py:class:`monai.networks.nets.DynUNet`
            - :py:class:`monai.networks.nets.UNet`

        """
        super().__init__()

        fea = ensure_tuple_rep(features, 6)
        print(f"BasicUNet features: {fea}.")

        self.upcat_4 = UpCat(dimensions, fea[4], fea[3], fea[3], act, norm, dropout, upsample)
        self.upcat_3 = UpCat(dimensions, fea[3], fea[2], fea[2], act, norm, dropout, upsample)
        self.upcat_2 = UpCat(dimensions, fea[2], fea[1], fea[1], act, norm, dropout, upsample)
        self.upcat_1 = UpCat(dimensions, fea[1], fea[0], fea[5], act, norm, dropout, upsample, halves=False)

        self.final_conv = Conv["conv", dimensions](fea[5], out_channels, kernel_size=1)

    def forward(self,
                x0: torch.Tensor,
                x1: torch.Tensor,
                x2: torch.Tensor,
                x3: torch.Tensor,
                x4: torch.Tensor):
        """
        Args:
            x: input should have spatially N dimensions
                ``(Batch, in_channels, dim_0[, dim_1, ..., dim_N])``, N is defined by `dimensions`.
                It is recommended to have ``dim_n % 16 == 0`` to ensure all maxpooling inputs have
                even edge lengths.

        Returns:
            A torch Tensor of "raw" predictions in shape
            ``(Batch, out_channels, dim_0[, dim_1, ..., dim_N])``.
        """

        u4 = self.upcat_4(x4, x3)
        u3 = self.upcat_3(u4, x2)
        u2 = self.upcat_2(u3, x1)
        u1 = self.upcat_1(u2, x0)

        u_out = self.final_conv(u1)

        return u_out

class DecoderRec(nn.Module):
    def __init__(
            self,
            dimensions: int = 3,
            in_channels: int = 1,
            out_channels: int = 1,
            features: Sequence[int] = (32, 32, 64, 128, 256, 32),
            act: Union[str, tuple] = ("LeakyReLU", {"negative_slope": 0.1, "inplace": True}),
            norm: Union[str, tuple] = ("instance", {"affine": True}),
            dropout: Union[float, tuple] = 0.0,
            upsample: str = "deconv",
    ):
        """
        A UNet implementation with 1D/2D/3D supports.

        Based on:

            Falk et al. "U-Net – Deep Learning for Cell Counting, Detection, and
            Morphometry". Nature Methods 16, 67–70 (2019), DOI:
            http://dx.doi.org/10.1038/s41592-018-0261-2

        Args:
            dimensions: number of spatial dimensions. Defaults to 3 for spatial 3D inputs.
            in_channels: number of input channels. Defaults to 1.
            out_channels: number of output channels. Defaults to 2.
            features: six integers as numbers of features.
                Defaults to ``(32, 32, 64, 128, 256, 32)``,

                - the first five values correspond to the five-level encoder feature sizes.
                - the last value corresponds to the feature size after the last upsampling.

            act: activation type and arguments. Defaults to LeakyReLU.
            norm: feature normalization type and arguments. Defaults to instance norm.
            dropout: dropout ratio. Defaults to no dropout.
            upsample: upsampling mode, available options are
                ``"deconv"``, ``"pixelshuffle"``, ``"nontrainable"``.

        Examples::

            # for spatial 2D
            >>> net = BasicUNet(dimensions=2, features=(64, 128, 256, 512, 1024, 128))

            # for spatial 2D, with group norm
            >>> net = BasicUNet(dimensions=2, features=(64, 128, 256, 512, 1024, 128), norm=("group", {"num_groups": 4}))

            # for spatial 3D
            >>> net = BasicUNet(dimensions=3, features=(32, 32, 64, 128, 256, 32))

        See Also

            - :py:class:`monai.networks.nets.DynUNet`
            - :py:class:`monai.networks.nets.UNet`

        """
        super().__init__()

        fea = ensure_tuple_rep(features, 6)
        print(f"BasicUNet features: {fea}.")

        self.up_4 = Up(dimensions, fea[4], fea[3], fea[3], act, norm, dropout, upsample)
        self.up_3 = Up(dimensions, fea[3], fea[2], fea[2], act, norm, dropout, upsample)
        self.up_2 = Up(dimensions, fea[2], fea[1], fea[1], act, norm, dropout, upsample)
        self.up_1 = Up(dimensions, fea[1], fea[0], fea[5], act, norm, dropout, upsample, halves=False)

        self.final_conv = Conv["conv", dimensions](fea[5], out_channels, kernel_size=1)

    def forward(self, x: torch.Tensor):
        """
        Args:
            x: input should have spatially N dimensions
                ``(Batch, in_channels, dim_0[, dim_1, ..., dim_N])``, N is defined by `dimensions`.
                It is recommended to have ``dim_n % 16 == 0`` to ensure all maxpooling inputs have
                even edge lengths.

        Returns:
            A torch Tensor of "raw" predictions in shape
            ``(Batch, out_channels, dim_0[, dim_1, ..., dim_N])``.
        """

        r4 = self.up_4(x)
        r3 = self.up_3(r4)
        r2 = self.up_2(r3)
        r1 = self.up_1(r2)
        r_out = self.final_conv(r1)
        return r_out


title: How to shuffle an iterator of known size in Python?

body:
I am trying to bijectively map a range of indices to another set of indices.
So I came up with this:
def get_index(i):
    # Does more complicated stuff, but just to illustrate.
    return 2*i + 1

my_iterator = iter(get_index(i) for i in range(100))

# then I use my_iterator
for a in my_iterator:
    ...

Now I would like to provide a randomly permuted version of this iterator (without repetition) and my question is how to do that without having to explicit the range into a list. The number of elements is huge and I would not like to store the whole thing in memory.
EDIT
The number of elements is known beforehand.
I am doing all this to make a pytorch sampler subclass.
I am asking this question so I can make my Sampler support random shuffling. The whole thing looks something like:
from torch.utils.data import Sampler

class CustomSampler(Sampler):
    def __init__(self,n_samples,shuffle=False,data_source=None):
        super().__init__(data_source)
        self.shuffle = shuffle
        self.n_samples = n_samples

    def get_index(self,i):
        # Does more complicated stuff, but just to illustrate.
        return 2*i + 1

    def __len__(self):
        return self.n_samples

    def __iter__(self):
        if not self.shuffle:
            return iter(self.get_index(i) for i in range(self.n_samples))
        else:
            # HELP
            # return (iter(self.get_index(i) for i in permutation_of(range(self.n_samples))))


title: How to train the Shared Layers in PyTorch

body:
I have the follow code
import torch
import torch.nn as nn
from torchviz import make_dot, make_dot_from_trace

class Net(nn.Module):
    def __init__(self, input, output):
        super(Net, self).__init__()
        self.fc = nn.Linear(input, output)

    def forward(self, x):
        x = self.fc(x)
        x = self.fc(x)
        return x

model = Net(12, 12)
print(model)

x = torch.rand(1, 12)
y = model(x)
make_dot(y, params = dict(model.named_parameters()))

Here I reuse the self.fc twice in the forward.
The computational graph is look

I am confused about the computational graph and,
I am curious how to train this model in back propagation? It seem for me the gradient will live in a loop forever. Thanks a lot.


title: Why does torchvision pretrained model perform better than scratch trained?

body:
I followed the torchvision tutorial (https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html) and trained a segmentation model on my own dataset of aerial images. Which only has 2 classes. The result were:
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.640
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.700

Decent but I would like to improve the accuracy of the masks.
So I then trained exactly the same with pretrained=False and pretrained_backbone=False and was very surprised that the performance was about half of the pre-trained model.
Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.328
Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.472

I'm trying to understand why this would be the case and where I should be directing my effort to improve the model:

Training parameters like: lr, momentum, dropout, batch size ..
Architecture of the MaskRCNN network: roi, anchors ..
Something about my dataset: adding augmentation ..

I'm not sure where to look to understand how I can improve the performance. What is a decent way to approach that?
EDIT: My model is:
def get_model_instance_segmentation(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, num_classes)
    return model


title: pytorch runs slow when data are pre-transported to GPU

body:
I have a model written in pytorch. Since my dataset is small, I can directly load all of the data to GPU. However, I found the forward speed becomes slow if I do so. The following is a runnable example. Specifically, I have the model:
import numpy as np
from time import time
import random
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader

def knn(x, k):
    inner = -2*torch.matmul(x.transpose(2, 1), x)
    xx = torch.sum(x**2, dim=1, keepdim=True)
    pairwise_distance = -xx - inner - xx.transpose(2, 1)
    idx = pairwise_distance.topk(k=k, dim=-1)[1]   # (batch_size, num_points, k)
    return idx

def get_graph_feature(x, k=20, idx=None):
    batch_size = x.size(0)
    num_points = x.size(2)
    x = x.view(batch_size, -1, num_points)
    if idx is None:
        idx = knn(x, k=k)   # (batch_size, num_points, k)
    idx_base = torch.arange(0, batch_size, device=x.device).view(-1, 1, 1)*num_points
    idx = idx + idx_base
    idx = idx.view(-1)
    _, num_dims, _ = x.size()
    x = x.transpose(2, 1).contiguous()   # (batch_size, num_points, num_dims)  -> (batch_size*num_points, num_dims) #   batch_size * num_points * k + range(0, batch_size*num_points)
    feature = x.view(batch_size*num_points, -1)[idx, :]
    feature = feature.view(batch_size, num_points, k, num_dims)
    x = x.view(batch_size, num_points, 1, num_dims).repeat(1, 1, k, 1)
    feature = torch.cat((feature-x, x), dim=3).permute(0, 3, 1, 2).contiguous()
    return feature

class DGCNN(nn.Module):
    def __init__(self, k=25, output_channels=10):
        super(DGCNN, self).__init__()
        self.k = k
        self.bn1 = nn.BatchNorm2d(64)
        self.bn2 = nn.BatchNorm2d(64)
        self.bn3 = nn.BatchNorm2d(128)
        self.bn4 = nn.BatchNorm2d(256)
        self.bn5 = nn.BatchNorm1d(1024)
        self.conv1 = nn.Sequential(nn.Conv2d(6, 64, kernel_size=1, bias=False),
                                   self.bn1,
                                   nn.LeakyReLU(negative_slope=0.2))
        self.conv2 = nn.Sequential(nn.Conv2d(64*2, 64, kernel_size=1, bias=False),
                                   self.bn2,
                                   nn.LeakyReLU(negative_slope=0.2))
        self.conv3 = nn.Sequential(nn.Conv2d(64*2, 128, kernel_size=1, bias=False),
                                   self.bn3,
                                   nn.LeakyReLU(negative_slope=0.2))
        self.conv4 = nn.Sequential(nn.Conv2d(128*2, 256, kernel_size=1, bias=False),
                                   self.bn4,
                                   nn.LeakyReLU(negative_slope=0.2))
        self.conv5 = nn.Sequential(nn.Conv1d(512, 1024, kernel_size=1, bias=False),
                                   self.bn5,
                                   nn.LeakyReLU(negative_slope=0.2))
        self.linear1 = nn.Linear(1024*2, 512, bias=False)
        self.bn6 = nn.BatchNorm1d(512)
        self.dp1 = nn.Dropout()
        self.linear2 = nn.Linear(512, 256)
        self.bn7 = nn.BatchNorm1d(256)
        self.dp2 = nn.Dropout()
        self.linear3 = nn.Linear(256, output_channels)

    def forward(self, x):
        x = x.transpose(2, 1)
        batch_size = x.size(0)
        x = get_graph_feature(x, k=self.k)
        x = self.conv1(x)
        x1 = x.max(dim=-1, keepdim=False)[0]
        x = get_graph_feature(x1, k=self.k)
        x = self.conv2(x)
        x2 = x.max(dim=-1, keepdim=False)[0]
        x = get_graph_feature(x2, k=self.k)
        x = self.conv3(x)
        x3 = x.max(dim=-1, keepdim=False)[0]
        x = get_graph_feature(x3, k=self.k)
        x = self.conv4(x)
        x4 = x.max(dim=-1, keepdim=False)[0]
        x = torch.cat((x1, x2, x3, x4), dim=1)
        x = self.conv5(x)
        x1 = F.adaptive_max_pool1d(x, 1).view(batch_size, -1)
        x2 = F.adaptive_avg_pool1d(x, 1).view(batch_size, -1)
        x = torch.cat((x1, x2), 1)
        x = F.leaky_relu(self.bn6(self.linear1(x)), negative_slope=0.2)
        x = self.dp1(x)
        x = F.leaky_relu(self.bn7(self.linear2(x)), negative_slope=0.2)
        x = self.dp2(x)
        x = self.linear3(x)
        return x

Here is what the dataloader and test function looks like:
class my_loader(Dataset):
    def __init__(self, device):
        self.data = torch.rand(256, 2048, 3).to(device).float()
        self.labels = torch.rand(256).to(device).long()

    def __getitem__(self, ind):
        return self.data[ind], self.labels[ind]

    def __len__(self):
        return len(self.data)

def test():
    device = torch.device('cuda:2')
    test_set = my_loader(device)
    test_loader = DataLoader(test_set, batch_size=16, shuffle=True, num_workers=0)
    model = DGCNN().to(device)
    model.eval()

    #---------- this one is 0.12s --------------#
    for inputs, labels in test_loader:
        tic = time()
        pred = model(inputs)
        print('time1 {}'.format(time() - tic))
    print('------------------')

    #---------- this one is 0.004s --------------#
    for inputs, labels in test_loader:
        inputs = inputs.detach().cpu().to(device)
        tic = time()
        pred = model(inputs)
        print('time2 {}'.format(time() - tic))
    print('------------------')

    #---------- this one is 0.12s --------------#
    for inputs, labels in test_loader:
        tic = time()
        inputs = inputs.detach().cpu().to(device)
        pred = model(inputs)
        print('time3 {}'.format(time() - tic))
    print('------------------')

Basically, it seems that if there is no explicit call of gpu to cpu transportation either before or after the forward propagation, the forward propagation would cost more time. It just seems like that the forward propagation is implicitly doing gpu->cpu transportation.


title: PyTorch out of GPU memory in test loop

body:
For the following training program, training and validation are all ok.
Once reach to Test method, I have CUDA out of memory. What should I change so that I have enough memory to test as well.
import torch
from torchvision import datasets, transforms
import torch.nn.functional as f
class CnnLstm(nn.Module):
    def __init__(self):
        super(CnnLstm, self).__init__()
        self.cnn = CNN()
        self.rnn = nn.LSTM(input_size=180000, hidden_size=256, num_layers=2, batch_first=True)#stacked LSTM with 2 layers
        #print(num_classes)
        self.linear = nn.Linear(256, num_classes)
        #print('after num_classes')

    def forward(self, x):
        #print(x.shape)
        batch_size, time_steps, channels, height, width = x.size()
        c_in = x.view(batch_size * time_steps, channels, height, width)
        _, c_out = self.cnn(c_in)
        r_in = c_out.view(batch_size, time_steps, -1)
        r_out, (_, _) = self.rnn(r_in)
        r_out2 = self.linear(r_out[:, -1, :])
        return f.log_softmax(r_out2, dim=1)

class TrainCNNLSTM:
    def __init__(self):
        self.seed = 1
        self.batch_size = 8
        self.validate_batch_size = 8
        self.test_batch_size = 1
        self.epoch = 20
        self.learning_rate = 0.01
        self.step = 100
        self.train_loader = None
        self.validate_loader = None
        self.test_loader = None
        #print('before')
        self.model = CnnLstm().to(device)
        #print('after')
        self.criterion = nn.CrossEntropyLoss()

    def load_data(self):
        data_loader = DataLoader()
        self.train_loader = data_loader.get_train_data(self.batch_size)
        self.validate_loader = data_loader.get_validate_data(self.validate_batch_size)
        self.test_loader = data_loader.get_test_data(self.test_batch_size)

    def train(self):
        optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate, momentum=0.9)
        scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=self.learning_rate/100.0, max_lr=self.learning_rate, step_size_up=13)
        #optimizer = torch.optim.SGD(self.model.parameters(), lr=self.learning_rate)
        for epoch in range(self.epoch):
            t_losses=[]
            for iteration, (data, target) in enumerate(self.train_loader):
                data = np.expand_dims(data, axis=1)
                data = torch.FloatTensor(data)
                data, target = data.cuda(), target.cuda()
                data, target = Variable(data), Variable(target)
                optimizer.zero_grad()
                output = self.model(data)
                loss = self.criterion(output, target)
                #loss = f.nll_loss(output, target)
                t_losses.append(loss)
                loss.backward()
                optimizer.step()
                scheduler.step()
                if iteration % self.step == 0:
                   print('Epoch: {} | train loss: {:.4f}'.format(epoch, loss.item()))
            avgd_trainloss = sum(t_losses)/len(t_losses)
            self.validate(epoch, avgd_trainloss)

    def validate(self, epoch, avg_tloss):
        v_losses=[]
        with torch.no_grad():
            for iteration, (data, target) in enumerate(self.validate_loader):
                data = np.expand_dims(data, axis=1)
                data = torch.FloatTensor(data)
                data, target = data.cuda(), target.cuda()
                data, target = Variable(data), Variable(target)
                output = self.model(data)
                loss = self.criterion(output, target)
                #loss = f.nll_loss(output, target)
                v_losses.append(loss)
        avgd_validloss = sum(v_losses)/len(v_losses)
        print('Epoch: {} | train loss: {:.4f} | validate loss: {:.4f}'.format(epoch, avg_tloss, avgd_validloss))

    def test(self):
        test_loss = []
        correct = 0
        for data, target in self.test_loader:
            data = np.expand_dims(data, axis=1)
            data = torch.FloatTensor(data)
            data, target = data.cuda(), target.cuda()
            data, target = Variable(data, volatile=True), Variable(target)
            output = self.model(data)
            loss = self.criterion(output, target)
            #f.nll_loss(output, target, size_average=False).item()  # sum up batch loss
            test_loss.append(loss)
            pred = torch.max(output, 1)[1].data.squeeze()
            correct += pred.eq(target.data.view_as(pred)).long().cpu().sum()

        test_loss = sum(test_loss)/len(test_loss)
        print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(test_loss, correct, len(self.test_loader.dataset),
                100. * correct / len(self.test_loader.dataset)))

train = TrainCNNLSTM()
train.load_data()
train.train()
train.test()


title: Error while implenting the Faster R-CNN Object detection algorithm

body:
I am trying to implement the Faster R-CNN object detection algorithm and I have an unusual error.
While trying to call the train_one_epoch function in this colab tutorial I had an error in the loss_dict = model(images, targets)which is mentioned here .
The exact error that i had is :
    101         cell_anchors = self.cell_anchors
    102         assert cell_anchors is not None
--> 103         assert len(grid_sizes) == len(strides) == len(cell_anchors)
    104
    105         for size, stride, base_anchors in zip(

AssertionError:

Anyone have an idea ? And thanks in advance !


title: ML classification model for any pic input size

body:
I'm learning for weeks now how to train own models to classify images.
So far I trained some models and got a rough overview about classification.
The training pictures for my model are 3x32x32, training works fine. Predictions with test images works also fine, as long as they come in the same dimensions as the training pics.
I already added adaptive max pooling before fully connected layers, any image size can also be used with the model but the predictions are always wrong.
Can anyone tell me why it behaves like this and how can I improve it?
Thats may model:
class ResNet9(ImageClassificationBase):
def __init__(self, in_channels, num_classes):
    super().__init__()

    self.conv1 = conv_block(in_channels, 64)
    self.conv2 = conv_block(64, 128, pool=True)
    self.res1 = nn.Sequential(conv_block(128, 128), conv_block(128, 128))

    self.conv3 = conv_block(128, 256, pool=True)
    self.conv4 = conv_block(256, 512, pool=True)
    self.res2 = nn.Sequential(conv_block(512, 512), conv_block(512, 512))

    self.classifier = nn.Sequential(nn.AdaptiveMaxPool2d(1),
                                  nn.Flatten(),
                                  nn.Dropout(0.2),
                                  nn.Linear(512, num_classes))

def forward(self, xb):
    out = self.conv1(xb)
    out = self.conv2(out)
    out = self.res1(out) + out
    out = self.conv3(out)
    out = self.conv4(out)
    out = self.res2(out) + out
    out = self.classifier(out)
    return out

Normalization part:
# create dataset
stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
train_trans = tt.Compose(
    [tt.RandomCrop(32, padding=4, padding_mode='reflect'),
     tt.ToTensor(),
     tt.Normalize(*stats, inplace=True)])
test_trans = tt.Compose([tt.ToTensor(),
             tt.Normalize(*stats, inplace=True)])


title: Why does my ONNX model have 0 inputs after converting from PyTorch?

body:
I'm looking to convert a PyTorch model to Tensorflow using ONNX as a intermediate format using the tutorial here. However, after conversion, when I run
output =[node.name for node in model.graph.output]
input_all = [node.name for node in model.graph.input]
input_initializer =  [node.name for node in model.graph.initializer]
net_feed_input = list(set(input_all)  - set(input_initializer))

print('Inputs: ', net_feed_input)
print('Outputs: ', output)

it says there are zero inputs. Here is the model architecture
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden_x1 = nn.Sequential(
            nn.Conv1d(in_channels=2, out_channels=8, kernel_size=7),
            nn.SELU(),
            nn.Conv1d(in_channels=8, out_channels=6, kernel_size=7),
            nn.SELU(),
            nn.Conv1d(in_channels=6, out_channels=4, kernel_size=5),
            nn.SELU(),
        )
        self.hidden_xfc = nn.Sequential(
            nn.Linear(256, 20),
            nn.SELU(),
            nn.Linear(20, 20),
            nn.SELU(),
        )
        self.hidden_x2 = nn.Sequential(
            nn.MaxPool1d(kernel_size=2),
            nn.Conv1d(in_channels=2, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.AvgPool1d(kernel_size=2),
            nn.Conv1d(in_channels=4, out_channels=2, kernel_size=3),
            nn.SELU(),
            nn.AvgPool1d(kernel_size=2),
            nn.Conv1d(in_channels=2, out_channels=2, kernel_size=3),
            nn.SELU(),
            nn.AvgPool1d(kernel_size=2),
        )
        self.hidden_encoded_1 = nn.Flatten()
        self.hidden_embedding = nn.Sequential(
            nn.Linear(26, 16),
            nn.SELU(),
            nn.Linear(16, 8),
            nn.SELU(),
            nn.Linear(8, 4),
        )

    def forward(self, x, n=-1):
      x = torch.transpose(x, 1, 2)
      x = self.hidden_x1(x)
      xfc = torch.reshape(x, (n, 256))
      xfc = self.hidden_xfc(xfc)
      x = torch.reshape(x, (n, 2, 128))
      x = self.hidden_x2(x)
      encoded = self.hidden_encoded_1(x)
      encoded = torch.cat((encoded, xfc), 1)
      embedding = self.hidden_embedding(encoded)

      unscaled_param = embedding*torch.tensor(params_scaler.var_[0:4]**0.5).cuda() + torch.tensor(params_scaler.mean_[0:4]).cuda()
      return unscaled_param

and I pass in this random input when doing torch.onnx.export(trained_model, dummy_input, "output/trained_model.onnx")
dummy_input = Variable(torch.randn(20,80,2))
dummy_input = dummy_input.type(torch.cuda.DoubleTensor)

After converting the ONNX model TensorFlow it also says there are zero inputs as well. Does anyone know what I might be doing wrong?


title: Create class that is both a nn.Parameter and nn.Module

body:
To facilitate managing trainable hyperparameters, I am looking for a way to create a class Hyperparameter that acts both as a nn.Parameter and a nn.Module. In particular, I would like to use Hyperparameter objects both as a nn.Parameter (e.g. for tensor operations) but still have access to the interface provided by nn.Module to for example store the objects in a nn.ModuleDict along with other modules or use methods like zero_grad(), parameters().
I tried to accomplish this through multiple inheritance but it didn't quite work out:
import torch

class Hyperparameter(torch.nn.Parameter, torch.nn.Module):
    def __new__(cls, tensor, name):
        return torch.nn.Parameter.__new__(cls, data=tensor)

    def __init__(self, tensor, name):
        torch.nn.Parameter.__init__(self)
        torch.nn.Module.__init__(self)
        self.register_parameter(name, self)

hp1 = Hyperparameter(torch.ones(5), "test1")
hp2 = Hyperparameter(torch.ones(8), "test2")

# Examples of what I want to do, which already work
tmp = hp1 * hp2
hp_dict = torch.nn.ModuleDict({"hp1": hp1, "hp2": hp2})

# What does not work with this solution
hp_dict.to(torch.device("cpu"))
# KeyError: "attribute 'data' already exists"

This works for the things I described (can be added to ModuleDict, can perform algebraic operations, ...), but calling to() throws an error. I think something is no longer as nn.Module expects it to be, but I do not understand what it is.
EDIT: Here is the full stack trace as requested:
Traceback (most recent call last):
  File "tmp.py", line 16, in <module>
    hp_dict.to(torch.device("cpu"))
  File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 612, in to
    return self._apply(convert)
  File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 359, in _apply
    module._apply(fn)
  File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 384, in _apply
    param.data = param_applied
  File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 796, in __setattr__
    self.register_parameter(name, value)
  File ".myenv/lib/python3.9/site-packages/torch/nn/modules/module.py", line 316, in register_parameter
    raise KeyError("attribute '{}' already exists".format(name))
KeyError: "attribute 'data' already exists"


title: check if a tensor included in bigger tensor in pytorch

body:
how to check for example if:
torch.tensor([1, 3]) belongs to torch.tensor([3, 1], [1, 1], [3, 1])
a in b methods compare element-wise and thus here not correct
what I want to compute is whether the whole tensor [1, 3] is in the bigger one
thanks


title: Manually update momentum terms in pytorch optimizers

body:
The Adam optimizer has several terms that are used to add "momentum" to the gradient descent algorithm, making the step size for each variable adaptive:
Specifically, in the case of Adam here, I refer to the m-hat and v-hat terms.
There are times, however, when you may wish to manually update their states, such as resetting training for a subset of variables.  Is there a way to do this in PyTorch's momentum-based optimizers (especially Adam)?


title: I can't figure out why the size of the tensors doesn't match in Pytorch

body:
Some context:
I have been studying AI and ML for the last couple of month now and finally I am studying neural nets. Great! The problem is that when I follow a tutorial everything seems to be OK, but when I try to implement a NN by my self I always face issues related to the size of the tensors.
I have seem the answer to other questions (like this one) but they face the exact problem of the post. I am not looking for a code to just copy and paste. I want to understand why I am facing this problem, how to handle it and avoid it.
The error message:

/home/devops/aic/venv/lib/python3.8/site-packages/torch/nn/modules/loss.py:528: UserWarning: Using a target size (torch.Size([16, 2])) that is different to the input size (torch.Size([9, 2])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
return F.mse_loss(input, target, reduction=self.reduction)
Traceback (most recent call last):
File "nn_conv.py", line 195, in
loss = loss_function(outputs, targets)

File "/home/devops/aic/venv/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)

File "/home/devops/aic/venv/lib/python3.8/site-packages/torch/nn/modules/loss.py", line 528, in forward
return F.mse_loss(input, target, reduction=self.reduction)

File "/home/devops/aic/venv/lib/python3.8/site-packages/torch/nn/functional.py", line 2928, in mse_loss
expanded_input, expanded_target = torch.broadcast_tensors(input, target)

File "/home/devops/aic/venv/lib/python3.8/site-packages/torch/functional.py", line 74, in broadcast_tensors
return _VF.broadcast_tensors(tensors)  # type: ignore

RuntimeError: The size of tensor a (9) must match the size of tensor b (16) at non-singleton dimension 0

This is my code:
import os

import cv2
import numpy as np
from tqdm import tqdm
import matplotlib.pyplot as plt

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class DogsVSCats():

    IMG_SIZE = 50
    CATS = 'PetImages/Cat'
    DOGS = 'PetImages/Dog'
    LABELS = {CATS: 0, DOGS: 1}

    training_data = []
    cats_count = 0
    dogs_count = 0

    def make_training_data(self):
        for label in self.LABELS.keys():

            for f in tqdm(os.listdir(label)):
                try:
                    path = os.path.join(label, f)
                    # convert image to grayscale
                    img = cv2.imread(path)
                    if img is not None:
                        height, width = img.shape[:2]
                        if width > height:
                            height = round((height * self.IMG_SIZE) / width)
                            width = self.IMG_SIZE
                            right = 0
                            bottom = self.IMG_SIZE - height
                        else:
                            width = round((width * self.IMG_SIZE) / height)
                            height = self.IMG_SIZE
                            right = self.IMG_SIZE - width
                            bottom = 0

                        img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
                        img = cv2.cvtColor(img, cv2.COLOR_GRAY2BGR)
                        img = cv2.resize(img, (width, height))
                        img = cv2.copyMakeBorder(img,
                                                 top=0,
                                                 bottom=bottom,
                                                 left=0,
                                                 right=right,
                                                 borderType=cv2.BORDER_CONSTANT)

                        # Add a One-hot-vector of label of the image to self.training_data
                        self.training_data.append([np.array(img), np.eye(len(self.LABELS))[self.LABELS[label]]])

                        if label == self.CATS:
                            self.cats_count += 1
                        elif label == self.DOGS:
                            self.dogs_count += 1
                except cv2.error as e:
                    pass

        np.random.shuffle(self.training_data)
        np.save("PetImages/training_data.npy", self.training_data)

        print("Cats:", self.cats_count)
        print("Dogs:", self.dogs_count)

        training_data = np.load('PetImages/training_data.npy', allow_pickle=True)
        plt.imsave('PetImages/trained_example.png', training_data[1][0])

class RunningMetrics():

    def __init__(self):
        self._sum = 0
        self._count = 0

    def __call__(self):
        return self._sum/float(self._count)

    def update(self, val, size):
        self._sum += val
        self._count += size

class Net(nn.Module):

    def __init__(self, num_channels, conv_kernel_size=3, stride=1, padding=1, max_pool_kernel_size=2):
        super(Net, self).__init__()

        self._num_channels = num_channels
        self._max_pool_kernel_size = max_pool_kernel_size

        self.conv1 = nn.Conv2d(1, self._num_channels, conv_kernel_size, stride, padding)
        self.conv2 = nn.Conv2d(self._num_channels, self._num_channels*2, conv_kernel_size, stride, padding)
        self.conv3 = nn.Conv2d(self._num_channels*2, self._num_channels*4, conv_kernel_size, stride, padding)

        # Calc input of first
        self.fc1 = nn.Linear(self._num_channels*4*8*8, self._num_channels*8)
        self.fc2 = nn.Linear(self._num_channels*8, 2)

    def forward(self, x):

        # Conv
        x = self.conv1(x)
        x = F.relu(F.max_pool2d(x, self._max_pool_kernel_size))
        x = self.conv2(x)
        x = F.relu(F.max_pool2d(x, self._max_pool_kernel_size))
        x = self.conv3(x)
        x = F.relu(F.max_pool2d(x, self._max_pool_kernel_size))

        # Flatten
        x = x.view(-1, self._num_channels*4*8*8)

        # Fully Connected
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)

        # return F.log_softmax(x, dim=1)
        return F.softmax(x, dim=1)

    def save_model(path):
        torch.save(save, path)

    def load_model(path):
        self = torch.load(PATH)
        self.eval()

if __name__ == '__main__':

    print('Loading dataset')
    if not os.path.exists("PetImages/training_data.npy"):
        dogsvcats = DogsVSCats()
        dogsvcats.make_training_data()

    training_data = np.load('PetImages/training_data.npy', allow_pickle=True)

    print('Loading Net')
    net = Net(num_channels=32)
    # net = net.to(device)

    # optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9  )
    optimizer = optim.Adam(net.parameters(), lr=0.001)

    # loss_function = nn.NLLLoss()
    loss_function = nn.MSELoss()

    print('Converting X tensor')
    X = torch.Tensor([i[0] for i in training_data]).view(-1, 50, 50)
    X = X/255.0

    print('Converting Y tensor')
    y = torch.Tensor([i[1] for i in training_data])

    # Validation data
    VAL_PERCENT = 0.1
    val_size = int(len(X)*VAL_PERCENT)

    X_train = X[:-val_size]
    y_train = y[:-val_size]

    X_test = X[-val_size:]
    y_test = y[-val_size:]

    print('Training Set:', len(X_train))
    print('Testing Set:', len(X_test))

    BATCH_SIZE = 16
    EPOCHS = 2
    IMG_SIZE=50
    for epoch in range(EPOCHS):
        print(f'Epoch {epoch+1}/{EPOCHS}')

        running_loss = RunningMetrics()
        running_acc = RunningMetrics()

        for i in tqdm(range(0, len(X_train), BATCH_SIZE)):
            inputs = X_train[i:i+BATCH_SIZE].view(-1,1, IMG_SIZE, IMG_SIZE)
            targets = y_train[i:i+BATCH_SIZE]
            # inputs, targets = inputs.to(device), targets.to(device)

            optimizer.zero_grad()
            outputs = net(inputs)
            _, preds = torch.max(outputs, 1)

            loss = loss_function(outputs, targets)
            loss.backward()

            optimizer.step()

            running_loss.update(loss.item()*BATCH_SIZE,
                                BATCH_SIZE)

            running_acc.update(toch.sum(preds == targets).float(),
                                BATCH_SIZE)

        print(f'Loss: {running_loss:.4f}, Acc: {running_acc:.4f}')
        print('-'*10)

Dataset:
I am using the Microsoft's dataset of cats and dogs images
EDIT:
The error previous message has been solved following Anonymous' advice but now I am getting another error:

Traceback (most recent call last):
File "nn_conv.py", line 203, in
running_acc.update(torch.sum(preds == targets).float(),

RuntimeError: The size of tensor a (16) must match the size of tensor b (2) at non-singleton dimension 1


title: Difference between torch.Size([64]) and (64,)?

body:
I created a Pytorch dataset class to store 64 lines of text. The file only has text, no label so I artificially generated an index list y (just to follow along with a tutorial https://medium.com/swlh/how-to-use-pytorch-dataloaders-to-work-with-enormously-large-text-files-bbd672e955a0#4fe0). After I created the dataset object and wrapped it around a dataloader, y.shape returned torch.Size([64]) while the tutorial said it would return (64,). (torch version is 1.8.1. torchvision version is 0.9.1. Python is 3.7.10.)
Is there a difference between torch.Size([64]) and (64,)? Thank you.
##### IMPORT PACKAGES #####
import nltk
import string
from nltk import word_tokenize
from torch.utils.data import IterableDataset, DataLoader, Dataset

##### DEFINE CLASS #####
class CustomDataset(Dataset):
    # A Pytorch Dataset class to store text
    def __init__(self, filename):
        '''
        Input: filename (Each line is a string.)
        Output: member variable X (list of unprocessed strings)
                member variable y (index list of X)
        '''
        # Open file and store contents in list
        with open(filename) as f:
            lines = f.read().split('\n')
        X, y = [], []
        i = 0
        for line in lines:
          X.append(line)
          y.append(i)
          i +=1
        # Store in member variables
        self.X = X
        self.y = y

     def preprocess(self, text):
        '''
        Input: a string from X
        Output: a preprocessed string
        '''
        text_pp = text.lower() # lower case
        return text_pp

    def __len__(self):
        return len(self.y)

    def __getitem__(self, index):
       '''
       Input: a number (within range of X's indices)
       Output: string at specified index
       '''
       return self.preprocess(self.X[index]), self.y[index]

##### CREATE OBJECT #####
dataset = CustomDataset('micro.txt')
dataloader = DataLoader(dataset, batch_size = 64, num_workers = 2)
for X, y in dataloader:
  print(y.shape) # torch.Size([64]) [Is it same as (64,)?])