title: How to pass a list of numbers as a single feature to a neural network?

body:
I am trying to cluster sentences by clustering the sentence embedding of them taken from fasttext model. Each sentence embedding has 300 dimensions, and I want to reduce them to 50 (say). I tried t-SNE, PCA, UMAP. I wanted to see how Auto Encoder works for my data.

Now passing those 300 numbers for each sentence as separate features to the NN would make sense or they should be passed as a single entity? If so, is there an way to pass a list as a feature to NN?

I tried passing the 300 numbers as individual features and with the output I tried clustering. Could get very few meaningful clusters rest were either noise or clusters with no similar sentences but being grouped (But with other techniques like UMAP I could get far more meaningful clusters in more number). Any leads would be helpful. Thanks in advance :)


title: Pytorch - Inferring linear layer in_features

body:
I am building a toy model to take in some images and give me a classification. My model looks like:

conv2d -> pool -> conv2d -> linear -> linear.

My issue is that when we create the model, we have to calculate the size of the first linear layer in_features based on the size of the input image. If we get new images of different sizes, we have to recalculate in_features for our linear layer. Why do we have to do this? Can't it just be inferred?


title: Volatile gpu-util is 100% and no progress on PyTorch1.1 after infering lots of images

body:
I deploy a pretrained deep model on NVIDIA V100 GPU. After infering lots of images, I find Volatile GPU-Utils still shows 100%, but no infering is running now. How to fix this problem?

When I type nvidia-smi, it shows:

  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 384.183      Driver Version: 384.183      CUDA Version: 9.0      |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |=============+=============+===============|
  |   0  Tesla V100-PCIE...  On   | 00000000:3B:00.0 Off |                    0 |
  | N/A   43C    P0    51W / 250W |   2530MiB / 32502MiB |    100%      Default |

I have tried sudo nvidia-smi -pm 1 to enable Persistence Mode. But the problem still remains.


title: Is there another way to save the Graph of preservation loss and learning rate

body:
I use fastai's learn class to train on a remote cloud server.
Generally, we will adjust the learning rate based on the relationship between the learning rate and the loss.

learn.lr_find()

learn.recorder.plot()

However, for some reason, the remote server cannot return this relationship graph.

Are there other ways we can Save the relationship between them and draw locally?


title: Overlap and add tensor in pytorch using nn.fold

body:
I have the following problem, I have tensor with shape (1,16198), first i would like to dive it into chunks so I used unfold like:
my_tensor.unfold(-1, 178, 89) -> tensor with shape (1, 182, 178)

which means 182 overlapping chunks of size 178, that's perfect.
But now I would like to undo the operation adding the overlapping chunks and get back a (1,16198) tensor. I believe that the fold method would do that, but I've burned through several hours trying to understand how it works to no progress at all and I cant seem to find a good explanation source for how to use it and I am afraid the official documentation is way to complex for my understanding.


title: Retrieve elements from a 3D tensor with a 2D index tensor

body:
I am playing around with GPT2 and I have 2 tensors:
O: An output tensor of shaped (B, S-1, V) where B is the batch size S is the the number of timestep and V is the vocabulary size. This is the output of a generative model and is softmaxed along the 2nd dimension.
L: A 2D tensor shaped (B, S-1) where each element is the index of the correct token for each timestep for each sample. This is basically the labels.
I want to extract the predicted probability of the corresponding correct token from tensor O based on tensor L such that I will end up with a 2D tensor shaped (B, S). Is there an efficient way of doing this apart from using loops?


title: Reading keys from an .npz file with multiple workers in pytorch dataloader?

body:
I have an .npz file where I have stored a dictionary. The dictionary has some keys and the values are numpy arrays. I want to read the dictionary in my getitem() method of the dataloader. When I set the dataloader num_workers to 1, everything runs fine. But when I increase the num workers, it throws the following error when reading the data from that npz file:
   Traceback (most recent call last):
  File "scripts/train.py", line 235, in <module>
    train(args)
  File "scripts/train.py", line 186, in train
    solver(args.epoch, args.verbose)
  File "/local-scratch/codebase/cap/lib/solver.py", line 174, in __call__
    self._feed(self.dataloader["train"], "train", epoch_id)
  File "/local-scratch/codebase/cap/lib/solver.py", line 366, in _feed
    for data_dict in dataloader:
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 819, in __next__
    return self._process_data(data)
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
    data.reraise()
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/_utils.py", line 369, in reraise
    raise self.exc_type(msg)
zipfile.BadZipFile: Caught BadZipFile in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/local-scratch/codebase/cap/lib/dataset.py", line 947, in __getitem__
    other_bbox_feat = self.box_features['{}-{}_{}.{}'.format(scene_id, target_object_id, ann_id, object_id)]
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/numpy/lib/npyio.py", line 255, in __getitem__
    pickle_kwargs=self.pickle_kwargs)
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/numpy/lib/format.py", line 763, in read_array
    data = _read_bytes(fp, read_size, "array data")
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/site-packages/numpy/lib/format.py", line 892, in _read_bytes
    r = fp.read(size - len(data))
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/zipfile.py", line 872, in read
    data = self._read1(n)
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/zipfile.py", line 962, in _read1
    self._update_crc(data)
  File "/local-scratch/anaconda3/envs/scanenv/lib/python3.6/zipfile.py", line 890, in _update_crc
    raise BadZipFile("Bad CRC-32 for file %r" % self.name)
zipfile.BadZipFile: Bad CRC-32 for file 'scene0519_00-13_1.0.npy'

As far as I know, pytorch dataloader uses multiprocessing to for data loading. Perhaps the issue is with multiprocessing and .npz files. I really appreciate any help.


title: pipenv install pytorch cpu + specific version

body:
I ned to install a specific version of pytorch cpu mode.
With pip I would do it like this:
pip install torch==1.2.0+cpu torchvision==0.4.0+cpu -f https://download.pytorch.org/whl/torch_stable.html --trusted-host download.pytorch.org
How can I achieve the same using Pipenv?
I tried having the following Pipfile:
[[source]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
verify_ssl = false

[packages]
torchvision = {index = "pytorch", version = "==0.4.0"}
torch = {index = "pytorch", version = "==1.2.0"}

but didn't work


title: ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory(Ubuntu)

body:
I was trying to run demo code in python object detection package mmdetection(on ubuntu 16.04), but got error as follow:
Traceback (most recent call last):
File "image_demo.py", line 3, in
from mmdet.apis import inference_detector, init_detector, show_result_pyplot
File "/home/hustwen/mmdetection/mmdet/apis/init.py", line 1, in
from .inference import (async_inference_detector, inference_detector,
File "/home/hustwen/mmdetection/mmdet/apis/inference.py", line 7, in
from mmcv.ops import RoIPool
File "/home/hustwen/anaconda3/envs/mmdet/lib/python3.7/site-packages/mmcv/ops/init.py", line 1, in
from .bbox import bbox_overlaps
File "/home/hustwen/anaconda3/envs/mmdet/lib/python3.7/site-packages/mmcv/ops/bbox.py", line 3, in
ext_module = ext_loader.load_ext('_ext', ['bbox_overlaps'])
File "/home/hustwen/anaconda3/envs/mmdet/lib/python3.7/site-packages/mmcv/utils/ext_loader.py", line 11, in load_ext
ext = importlib.import_module('mmcv.' + name)
File "/home/hustwen/anaconda3/envs/mmdet/lib/python3.7/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory

To run this package, I have upgrade my nvidia driver and cuda(from cuda9.0 to cuda10.1), and install pytorch,mmcv,mmdetection as instruction according to the new cuda version. But still got this error, I think this error has something with the older cuda(cuda9.0), but cannot figure out where the error is.
I checked libcuadrt, but only got libcudart.so.10
(mmdet) hustwen@TitanXP:~$ function lib_installed() { /sbin/ldconfig -N -v $(sed 's/:/ /' <<< $LD_LIBRARY_PATH) 2>/dev/null | grep $1; }
(mmdet) hustwen@TitanXP:~$ function check() { lib_installed $1 && echo "$1 is installed" || echo "ERROR: $1 is NOT installed"; }
(mmdet) hustwen@TitanXP:~$ check libcuda
    libcudart.so.10.1 -> libcudart.so.10.1.243
    libcuda.so.1 -> libcuda.so.430.64
    libcuda is installed
(mmdet) hustwen@TitanXP:~$ check libcudart
    libcudart.so.10.1 -> libcudart.so.10.1.243
    libcudart is installed

some necessary info:
pytorch 1.7.1 py3.7_cuda10.1.243_cudnn7.6.3_0 pytorch
torchaudio 0.7.2 py37 pytorch
torchvision 0.8.2 py37_cu101 pytorch

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243

mmcv-full 1.2.1

mmdet version = '2.7.0'

gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.12)


title: PyTorch plot loss curve

body:
I am new to PyTorch and i am trying to plot the loss curve of my training.
I use the first model (matrixfactorizaton) of this page with the movielens dataset. They say that they achieve a RMSE of 0.66, however my plot looks pretty weird.
https://blog.fastforwardlabs.com/2018/04/10/pytorch-for-recommenders-101.html
I also normalized the ratings between [0,1] like described. My code looks as follows:
index = 0
running_loss = 0
avg_loss = 0
loss_values = []
for user, item in zip(users, items):
    # get user, item and rating data
    # rating = Variable(torch.FloatTensor([ratings[user, item]]))
    rating = normalize(rating_values[index])
    rating = Variable(torch.FloatTensor([rating]))
    user = Variable(torch.LongTensor([int(user)]))
    item = Variable(torch.LongTensor([int(item)]))

    index += 1
    # predict
    prediction = model.predict(user, item)
    loss = loss_fn(prediction, rating)

    print(loss)

    running_loss += loss.item() / len(users)
    avg_loss += loss.item()
    loss_values.append(running_loss)

    optimizer.zero_grad()
    # backpropagate
    loss.backward()

    # update weights
    optimizer.step()

avg_loss = avg_loss / len(users)
print("avg loss: " + str(avg_loss))
plt.plot(loss_values)
plt.show()

normal model of the webpage
small adaption to the model. i applied the hardsigmoid function before returning the prediction, because i thought something in the model is wrong
I have read that i have to do something with the batch size but I don't know how. So i think something is wrong in my computation than with the model? The average loss sometimes outputs values close to the 0.66 RSME.


title: Flutter- Connecting to Python/Pytorch Backend

body:
I'm building a mobile app using Flutter that translates American Sign Language (ASL) to written English. As a prototype, I've limited my scope to translating photos of single alphanumeric characters. I have two parts to the project right now:

A python script that takes the path to an image, processes it with Torch/Torchvision and a custom model, and returns the alphanumeric character that is most likely to have been in the image. This part works great.

A Flutter front end that connects to the user's camera and saves the path to a taken image.

Q: What is the best way to connect these two parts of my project?
So far, I've tried two methods:

Using Starflut, which allows me to call Python code from Flutter and receive return values. I've tested it with some basic Python code (no dependencies) and it works great. However, for my more complex use case, where my Python code imports "torch", the code crashes.

Error Message:
:/data/data/com.example.asl/files/flutter_assets/starfiles/image_prediction.py, run failed

ModuleNotFoundError: No module named 'torch'

This happens when calling the following line of code:
    var result = await widget.srvGroup.loadRawModule("python", "",
    resPath + "/flutter_assets/starfiles/" + "image_prediction.py", false);

I've also tried Pytorch_mobile Flutter package, but beyond not being working exactly in the way I want and implemented in my own Python script, also crashes.

Error Message:
java.lang.NullPointerException: Attempt to invoke virtual method 'org.pytorch.IValue org.pytorch.Module.forward(org.pytorch.IValue[])' on a null object reference

This happens when calling the following line of code:
String prediction =
    await TorchMobile.getPrediction(_image, maxWidth: 256, maxHeight: 256);

Any suggestions as to how to get Starflut functioning would be ideal, as I'd like to use my Python interface, but if that's not possible, I'd love suggestions as to how to approach this problem.


title: Training speed on GPU become slower overtime

body:
My model training speed becomes slower over time. Every epoch take longer time to train.
Here is the full source code with my preprocess sentiment tree bank data (put glove.840B.300d.txt into data/glove).
Install some python packages:
pip install meowlogtool
pip install tqdm

Command to run:
python sentiment.py --emblr 0 --rel_dim 0 --tag_dim 0 --optim adagrad --name basic --lr 0.05 --wd 1e-4 --at_hid_dim 0

Model source code for you to read
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable as Var
import utils
import Constants
from model import SentimentModule
from embedding_model import EmbeddingModel

class SimpleGRU(nn.Module):
    """
    w[i] : (300, 1)
    h[i] : (150, 1)
    p[i] : (20, 1)
    r[i] : (20, 1)
    k[i] : (150, 1)
    x[i] : (20 + 150 + 300 + 20 = 490, 1) (490, 1)
    Uz, Ur, Uh : (150, 150) => 67500 => (450, 450)
    Wz, Wr, Wh : (150, 20 + 150 + 300 + 20) (150, 490)
    """
    def __init__(self, cuda, in_dim, hid_dim):
        super(SimpleGRU, self).__init__()
        self.cudaFlag = cuda

        self.Uz = nn.Linear(hid_dim, hid_dim)
        self.Ur = nn.Linear(hid_dim, hid_dim)
        self.Uh = nn.Linear(hid_dim, hid_dim)

        self.Wz = nn.Linear(in_dim, hid_dim)
        self.Wr = nn.Linear(in_dim, hid_dim)
        self.Wh = nn.Linear(in_dim, hid_dim)

        if self.cudaFlag:
            self.Uz = self.Uz.cuda()
            self.Ur = self.Uz.cuda()
            self.Uh = self.Uz.cuda()

            self.Wz = self.Wz.cuda()
            self.Wr = self.Wr.cuda()
            self.Wh = self.Wh.cuda()

    def forward(self, x, h_prev):
        """
    Simple-GRU(compress_x[v], h[t-1]) :
    z[t]         := s(Wz *compress_x[t]+ Uz * h[t-1] + bz)
    r[t]         := s(Wr * compress_x[t] + Ur * h[t-1] + br)
    h_temp[t]     := g(Wh * compress_x[t] + Uh * h[t-1] + bh)
    h[t]         := r[t] .* h[t-1] + (1 - z[t]) .* h_temp[t]
    return h[t]
        :param x: compress_x[t]
        :param h_prev: h[t-1]
        :return:
        """
        z = F.sigmoid(self.Wz(x) + self.Uz(h_prev))
        r = F.sigmoid(self.Wr(x) + self.Ur(h_prev))
        h_temp = F.tanh(self.Wh(x) + self.Uh(h_prev))
        h = r*h_prev + (1-z)*h_temp
        return h

class TreeSimpleGRU(nn.Module):
    def __init__(self, cuda, word_dim, tag_dim, rel_dim, mem_dim, at_hid_dim, criterion, leaf_h = None):
        super(TreeSimpleGRU, self).__init__()
        self.cudaFlag = cuda
        # self.gru_cell = nn.GRUCell(word_dim + tag_dim, mem_dim)
        self.gru_cell = SimpleGRU(self.cudaFlag, word_dim+tag_dim, mem_dim)
        self.gru_at = GRU_AT(self.cudaFlag, word_dim + tag_dim + rel_dim + mem_dim, at_hid_dim ,mem_dim)
        self.mem_dim = mem_dim
        self.in_dim = word_dim
        self.tag_dim = tag_dim
        self.rel_dim = rel_dim
        self.leaf_h = leaf_h # init h for leaf node
        if self.leaf_h == None:
            self.leaf_h = Var(torch.rand(1, self.mem_dim))
            torch.save(self.leaf_h, 'leaf_h.pth')

        if self.cudaFlag:
            self.leaf_h = self.leaf_h.cuda()

        self.criterion = criterion
        self.output_module = None

    def getParameters(self):
        """
        Get flatParameters
        note that getParameters and parameters is not equal in this case
        getParameters do not get parameters of output module
        :return: 1d tensor
        """
        params = []
        for m in [self.gru_cell, self.gru_at]:
            # we do not get param of output module
            l = list(m.parameters())
            params.extend(l)

        one_dim = [p.view(p.numel()) for p in params]
        params = F.torch.cat(one_dim)
        return params

    def set_output_module(self, output_module):
        self.output_module = output_module

    def forward(self, tree, w_emb, tag_emb, rel_emb, training = False):
        loss = Var(torch.zeros(1))  # init zero loss
        if self.cudaFlag:
            loss = loss.cuda()

        for idx in xrange(tree.num_children):
            _, child_loss = self.forward(tree.children[idx], w_emb, tag_emb, rel_emb, training)
            loss = loss + child_loss

        if tree.num_children > 0:
            child_rels, child_k  = self.get_child_state(tree, rel_emb)
            if self.tag_dim > 0:
                tree.state = self.node_forward(w_emb[tree.idx - 1], tag_emb[tree.idx -1], child_rels, child_k)
            else:
                tree.state = self.node_forward(w_emb[tree.idx - 1], None, child_rels, child_k)
        elif tree.num_children == 0:
            if self.tag_dim > 0:
                tree.state = self.leaf_forward(w_emb[tree.idx - 1], tag_emb[tree.idx -1])
            else:
                tree.state = self.leaf_forward(w_emb[tree.idx - 1], None)

        if self.output_module != None:
            output = self.output_module.forward(tree.state, training)
            tree.output = output
            if training and tree.gold_label != None:
                target = Var(utils.map_label_to_target_sentiment(tree.gold_label))
                if self.cudaFlag:
                    target = target.cuda()
                loss = loss + self.criterion(output, target)
        return tree.state, loss

    def leaf_forward(self, word_emb, tag_emb):
        """
        Forward function for leaf node
        :param word_emb:  word embedding of current node u
        :param tag_emb: tag embedding of current node u
        :return: k of current node u
        """
        h = self.leaf_h
        if self.cudaFlag:
            h = h.cuda()
        if self.tag_dim > 0:
            x = F.torch.cat([word_emb, tag_emb], 1)
        else:
            x = word_emb
        k = self.gru_cell(x, h)
        return k

    def node_forward(self, word_emb, tag_emb, child_rels, child_k):
        """
        Foward function for inner node
        :param word_emb: word embedding of current node u
        :param tag_emb: tag embedding of current node u
        :param child_rels (tensor): rels embedding of child node v
        :param child_k (tensor): k of child node v
        :return:
        """
        n_child = child_k.size(0)
        h = Var(torch.zeros(1, self.mem_dim))
        if self.cudaFlag:
            h = h.cuda()

        for i in range(0, n_child):
            k = child_k[i]
            x_list = [word_emb, k]
            if self.rel_dim >0:
                rel = child_rels[i]
                x_list.append(rel)
            if self.tag_dim > 0:
                x_list.append(tag_emb)
            x = F.torch.cat(x_list, 1)
            h = self.gru_at(x, h)
        k = h
        return k

    def get_child_state(self, tree, rels_emb):
        """
        Get child rels, get child k
        :param tree: tree we need to get child
        :param rels_emb (tensor):
        :return:
        """
        if tree.num_children == 0:
            assert False #  never get here
        else:
            child_k = Var(torch.Tensor(tree.num_children, 1, self.mem_dim))
            if self.rel_dim>0:
                child_rels = Var(torch.Tensor(tree.num_children, 1, self.rel_dim))
            else:
                child_rels = None
            if self.cudaFlag:
                child_k = child_k.cuda()
                if self.rel_dim > 0:
                    child_rels = child_rels.cuda()
            for idx in xrange(tree.num_children):
                child_k[idx] = tree.children[idx].state
                if self.rel_dim > 0:
                    child_rels[idx] = rels_emb[tree.children[idx].idx - 1]
        return child_rels, child_k

class AT(nn.Module):
    """
    AT(compress_x[v]) := sigmoid(Wa * tanh(Wb * compress_x[v] + bb) + ba)
    """
    def __init__(self, cuda, in_dim, hid_dim):
        super(AT, self).__init__()
        self.cudaFlag = cuda
        self.in_dim = in_dim
        self.hid_dim = hid_dim

        self.Wa = nn.Linear(hid_dim, 1)
        self.Wb = nn.Linear(in_dim, hid_dim)

        if self.cudaFlag:
            self.Wa = self.Wa.cuda()
            self.Wb = self.Wb.cuda()

    def forward(self, x):
        out = F.sigmoid(self.Wa(F.tanh(self.Wb(x))))
        return out

class GRU_AT(nn.Module):
    def __init__(self, cuda, in_dim, at_hid_dim ,mem_dim):
        super(GRU_AT, self).__init__()
        self.cudaFlag = cuda
        self.in_dim = in_dim
        self.mem_dim = mem_dim
        self.at_hid_dim = at_hid_dim
        if at_hid_dim > 0:
            self.at = AT(cuda, in_dim, at_hid_dim)
        self.gru_cell = SimpleGRU(self.cudaFlag, in_dim, mem_dim)

        if self.cudaFlag:
            if at_hid_dim > 0:
                self.at = self.at.cuda()
            self.gru_cell = self.gru_cell.cuda()

    def forward(self, x, h_prev):
        """

        :param x:
        :param h_prev:
        :return: a * m + (1 - a) * h[t-1]
        """
        m = self.gru_cell(x, h_prev)
        if self.at_hid_dim > 0:
            a = self.at.forward(x)
            h = torch.mm(a, m) + torch.mm((1-a), h_prev)
        else:
            h = m
        return h

class TreeGRUSentiment(nn.Module):
    def __init__(self, cuda, in_dim, tag_dim, rel_dim, mem_dim, at_hid_dim, num_classes, criterion):
        super(TreeGRUSentiment, self).__init__()
        self.cudaFlag = cuda
        self.tree_module = TreeSimpleGRU(cuda, in_dim, tag_dim, rel_dim, mem_dim, at_hid_dim, criterion)
        self.output_module = SentimentModule(cuda, mem_dim, num_classes, dropout=True)
        self.tree_module.set_output_module(self.output_module)

    def get_tree_parameters(self):
        return self.tree_module.getParameters()

    def forward(self, tree, sent_emb, tag_emb, rel_emb, training = False):
        # sent_emb = F.torch.unsqueeze(self.word_embedding.forward(sent_inputs), 1)
        # tag_emb = F.torch.unsqueeze(self.tag_emb.forward(tag_inputs), 1)
        # rel_emb = F.torch.unsqueeze(self.rel_emb.forward(rel_inputs), 1)
        # sent_emb, tag_emb, rel_emb = self.embedding_model(sent_inputs, tag_inputs, rel_inputs)

        tree_state, loss = self.tree_module(tree, sent_emb, tag_emb, rel_emb, training)
        output = tree.output
        return output, loss


title: Is "input" a keyword that causes errors when used as a parameter name (in PyTorch)?

body:
So I have a line of code:

packed_embeddings = pack_padded_sequence(input=embeddings,
                                                 lengths=lengths,
                                                 batch_first=True)

That throws me this error:

  File "/Users/kwj/anaconda3/lib/python3.6/site-packages/torch/onnx/__init__.py", line 130, in might_trace
    first_arg = args[0]

IndexError: tuple index out of range

But magically fixes itself if I take out the "input":

    packed_embeddings = pack_padded_sequence(embeddings,
                                             lengths=lengths,
                                             batch_first=True)

Here is the function specification in the PyTorch docs:

https://pytorch.org/docs/stable/_modules/torch/nn/utils/rnn.html#pack_padded_sequence

I'm using Python3 and PyTorch 0.4. Am I missing something really basic?
Not sure if this is my issue, or a PyTorch specific issue...pretty confused here.

Thanks


title: Does setting a manual seed in torch and numpy propagate into all imported scripts, nested functions, etc?

body:
I have a script structure like this:

main.py -> launcher_util.py: run_experiment(experiment, ...) -> experiment.py: experiment(...)

In main.py, I call:

import torch
import numpy
torch.manual_seed(1)
np.random.seed(1)

But when looking at multiple runs of my experiments, the results are not identical. The only two sources of randomness come from PyTorch and Numpy (i.e. due to random environment variables generation in OpenAI Gym), yet the experiment result plots are not identical. Why?


title: PyTorch: Slightly different results for different batch sizes

body:
I am using a VGG19 network to classify CIFAR10 images, yet I have slightly different results for data of different batch sizes. Code:

net.eval()

f_image = net.forward(Variable(image, requires_grad=True)).data

f_image2 = net.forward(Variable(image[0].unsqueeze(0), requires_grad=True)).data

f_image3 = net.forward(Variable(image[0].unsqueeze(0), requires_grad=True)).data

print(‘f_image - f_image2 = {}’.format((f_image[0] - f_image2).cpu().data.numpy()))
print(‘f_image - f_image3 = {}’.format((f_image[0] - f_image3).cpu().data.numpy()))
print(‘f_image2 - f_image3 = {}’.format((f_image2 - f_image3).cpu().data.numpy()))

Specifically, image is of size [batch_size, n_channels, height, width]. And I expect f_image[0].unsqueeze(0) to be the same as f_image2 and f_image3. However, they are slightly different:

f_image - f_image2 = [[ 0.0000000e+00 3.7252903e-09 -1.8626451e-09 3.7252903e-09
0.0000000e+00 -3.7252903e-09 3.7252903e-09 -1.8626451e-09
3.7252903e-09 -1.3969839e-09]]
f_image - f_image3 = [[ 0.0000000e+00 3.7252903e-09 -1.8626451e-09 3.7252903e-09
0.0000000e+00 -3.7252903e-09 3.7252903e-09 -1.8626451e-09
3.7252903e-09 -1.3969839e-09]]
f_image2 - f_image3 = [[0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]

The difference is very small, is it a bug or expected behavior of PyTorch?


title: Given groups=1, weight of size 16 16 3 3, expected input[16, 64, 222, 222] to have 16 channels, but got 64 channels instead?

body:
I try to run the following program for image classification problems in Pytorch.
I am new to PyTorch and I am not getting what's wrong with the code. I tried reshaping the images but no help.
I am running this code with Cuda. I have around 750 classes and 10-20 images in one class. My dataset is a benchmark dataset and every image has a size of 60*160.

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.ConvLayer1 = nn.Sequential(
            nn.Conv2d(3, 64, 3), # inp (3, 512, 512) changes doing here original (3, 64, 3)
            nn.Conv2d(8, 16, 3), # original (8,16,3)
            nn.MaxPool2d(2),
            nn.ReLU() # op (16, 256, 256)
        )
        self.ConvLayer2 = nn.Sequential(
            nn.Conv2d(16, 32, 5), # inp (16, 256, 256)
            nn.Conv2d(32, 32, 3),
            nn.MaxPool2d(4),
            nn.ReLU() # op (32, 64, 64)
        )
        self.ConvLayer3 = nn.Sequential(
            nn.Conv2d(32, 64, 3), # inp (32, 64, 64) original (32,64,3)
            nn.Conv2d(64, 64, 5),
            nn.MaxPool2d(2),
            nn.ReLU() # op (64, 32, 32)
        )
        self.ConvLayer4 = nn.Sequential(
            nn.Conv2d(64, 128, 5), # inp (64, 32, 32)
            nn.Conv2d(128, 128, 3),
            nn.MaxPool2d(2),
            nn.ReLU() # op (128, 16, 16)
        )
        self.Lin1 = nn.Linear(15488, 15)
        self.Lin2 = nn.Linear(1500, 150)
        self.Lin3 = nn.Linear(150, 15)

    def forward(self, x):
        x = self.ConvLayer1(x)
        x = self.ConvLayer2(x)
        x = self.ConvLayer3(x)
        x = self.ConvLayer4(x)
        x = x.view(x.size(0), -1)
        x = self.Lin1(x)

        return F.log_softmax(x, dim = 1)
'''

'''
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr=0.0001, momentum=0.5)
for epoch in tqdm(range(2)):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(dataloaders['train']):
        # get the inputs; data is a list of [inputs, labels]
        inputs, class_names = data
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, class_names)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
       # if i % 10 == 0:    # print every 10 mini-batches
        #    print('[%d, %5d] loss: %.3f' %
         #         (epoch + 1, i + 1, running_loss / 2000))
          #  running_loss = 0.0
    break
print('Finished Training')

Getting this error and I don't know where to make changes.
Given groups=1, the weight of size 16 16 3 3, expected input[16, 64, 222, 222] to have 16 channels, but got 64 channels instead.


title: Obtain a set of embedding from pretrained model - vgg16 pytorch

body:
For a certain project purpose I am trying to store the 1 * 4096 embeddings (The output right before the final layer) of around 6000 images into a pkl file. For the same, I am running an iteration over the 6000 images on vgg16 modified model in google colab. But it returns 'CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 15.90 GiB total capacity; 14.86 GiB already allocated; 1.88 MiB free; 342.26 MiB cached)' error.

Whereas I have used the same dataset split into test-train for training and validating my model and that runs fine. I am wondering why obtaining and storing the embedding alone is becoming a heavy task in colab.
Is there any other way I can obtain the embeddings and store in a pkl file other than the below code.

embedding = []
vgg16 = vgg16.to(device)
for x in range (0, len(inputImages)) :
  input = transformations(inputImages[x]) //pre processing
  input = torch.unsqueeze(input, 0)
  input = input.to(device)
  embedding.append(vgg16(input))

The code is interupted at the last line with the CUDA out of memory error.


title: How to calculate a Forward Pass with matrix operations in PyTorch?

body:
I have got an input x, layer 1 weight matrix, and layer 2 weight matrix.

Now I want to calculate the output of this pre-trained neural network via hand:

x * weights1 * weights2

While doing this I receive a RuntimeError: The size of tensor a (6) must match the size of tensor b (4) at non-singleton dimension 1

class Net(nn.Module):

def __init__(self):
    super().__init__()
    self.fc1 = nn.Linear(4,6)
    self.fc2 = nn.Linear(6,2)
    self.fc3 = nn.Linear(2,1)

def forward(self, x):
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = F.relu(self.fc3(x))
    return x

net = Net()
X = torch.randn(1000,4)
net.fc2.weight*(net.fc1.weight * X[0])


title: TypeError: can't multiply sequence by non-int type of 'tuple' in pytorch

body:
The code as below:

class L2Norm(nn.Module):
    def __init__(self):
        super(L2Norm, self).__init__()
        self.eps = 1e-10
    def forward(self, x):
        norm = torch.sqrt(torch.sum(x * x, dim = 1) + self.eps)
        x = x / norm.unsqueeze(-1).expand_as(x)
        return x

I want to normalize the features. The input x is the output of nn.Linear() in FC layer (x = self.fc1(x)).

    def forward(self, x):
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)

        x = x.view(x.size(0), -1)
        x = self.fc1(x)
        if self.feature:
            return x
#        x = self.last_bn(x)
        x = self.fc2(x)

        return x

I print the x as below:

(tensor([[-0.8409,  8.6126, -1.6639,  ..., -3.3563, 10.0872,  2.4730],
        [-1.3959,  0.5608, -0.9233,  ...,  0.4385, -0.7089, -1.3401],
        [ 0.5742, -3.8479,  1.7756,  ..., -4.2798, -5.0684, -0.9032],
        ...,
        [ 0.9205,  3.1602, -3.9247,  ..., -2.1396,  4.0262,  2.8075],
        [-0.2024,  0.5603,  0.0491,  ..., -0.1716, -0.2513,  0.1179],
        [ 4.8053,  0.3062, -1.6867,  ..., -1.5749,  0.5193,  0.8671]],
       device='cuda:0', grad_fn=<GatherBackward>), tensor([[-156.3423, -145.1505, -156.6586,  ..., -157.9570, -141.1895,
         -155.2964],
        [ -31.9854,  -30.2333,  -31.1459,  ...,  -30.3290,  -30.8740,
          -31.8696],
        [-141.5926, -144.1404, -141.1151,  ..., -144.7264, -145.9508,
         -141.7867],
        ...,
        [-193.6224, -192.4931, -195.6285,  ..., -194.2939, -191.7269,
         -192.7527],
        [  -5.8791,   -4.5035,   -5.6987,  ...,   -5.8316,   -5.9696,
           -5.6506],
        [ -77.5002,  -83.8829,  -84.7204,  ...,  -84.6169,  -83.8326,
          -83.6949]], device='cuda:0', grad_fn=<GatherBackward>))

However, the mistake occurs in the torch.sum(x*x, dim=1). I have any solution for the mistake.


title: SpaCy-transformers regression output

body:
I would like to have a regression output instead of the classification. For instance: instead of n classes I want a floating point output value from 0 to 1.

Here is the minimalistic example from the package github page:

    import spacy
    from spacy.util import minibatch
    import random
    import torch

    is_using_gpu = spacy.prefer_gpu()
    if is_using_gpu:
        torch.set_default_tensor_type("torch.cuda.FloatTensor")

    nlp = spacy.load("en_trf_bertbaseuncased_lg")
    print(nlp.pipe_names) # ["sentencizer", "trf_wordpiecer", "trf_tok2vec"]
    textcat = nlp.create_pipe("trf_textcat", config={"exclusive_classes": True})
    for label in ("POSITIVE", "NEGATIVE"):
        textcat.add_label(label)
    nlp.add_pipe(textcat)

    optimizer = nlp.resume_training()
    for i in range(10):
        random.shuffle(TRAIN_DATA)
        losses = {}
        for batch in minibatch(TRAIN_DATA, size=8):
            texts, cats = zip(*batch)
            nlp.update(texts, cats, sgd=optimizer, losses=losses)
        print(i, losses)
    nlp.to_disk("/bert-textcat")

Is there an easy way to make trf_textcat work as a regressor? Or would it mean extending the library?


title: How can i convert pytorch 3d cnn code to keras

body:
I have a 3D CNN network in pytorch that I have tried to convert into keras, but I am not quite sure about the conversion. Also, when I run the keras code, I have this error:

  ValueError: Negative dimension size caused by subtracting 3 from 2 for 'conv3d_13/convolution' (op: 'Conv3D) with input shapes [?,10,2,2,512],[3,3,3,512,512]

Pytorch code:

class semi_C3D(nn.Module):

    def __init__(self, num_classes = 101):
        super(semi_C3D, self).__init__()

        self.features = nn.Sequential(
            nn.Conv3d(3, 64, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.Conv3d(64, 64, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.MaxPool3d(kernel_size=(1,2,2), stride=(1,2,2)),

            nn.Conv3d(64, 128, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.Conv3d(128, 128, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.MaxPool3d(kernel_size=(1,2,2), stride=(1,2,2)),

            nn.Conv3d(128, 256, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.Conv3d(256, 256, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.Conv3d(256, 256, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.MaxPool3d(kernel_size=(1,2,2), stride=(1,2,2)),

            nn.Conv3d(256, 512, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.Conv3d(512, 512, kernel_size=(1,3,3), padding=(0,1,1)),
            nn.ReLU(inplace=True),
            nn.MaxPool3d(kernel_size=(1,2,2), stride=(1,2,2)),

        )

        self.conv3d = nn.Sequential(
            nn.Conv3d(512, 512, kernel_size=(3,1,1), padding=(1,0,0)),
            nn.ReLU(inplace=True),
            nn.MaxPool3d(kernel_size=(3,1,1), stride=(2,1,1), padding=(1,0,0)),

            nn.Conv3d(512, 512, kernel_size=(3,3,3), padding=(1,1,1)),
            nn.ReLU(),
            nn.Conv3d(512, 512, kernel_size=(3,3,3), padding=(1,1,1)),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(2,2,2), stride=(2,2,2)),

            nn.Conv3d(512, 512, kernel_size=(3,3,3), padding=(1,1,1)),
            nn.ReLU(),
            nn.MaxPool3d(kernel_size=(2, 2, 2), stride=(2, 2, 2)),

        )

        self.st_classifier = nn.Sequential(
            nn.Linear(9216, 4096),
            nn.ReLU(),
            nn.Dropout(p=0.5),
            nn.Linear(4096, 4096),
            nn.ReLU(),
            nn.Dropout(p=0.5),
            nn.Linear(4096, num_classes),
        )

    def forward(self, input):

        x = self.features(input)
        x = self.conv3d(x)

        x = x.view(-1, 9216)
        x = self.st_classifier(x)
        return x

Keras code:

def semi_3d(self):

        model = Sequential()
    #First bloc of layers (self.features = nn.Sequential part for the pytorch code)
        model.add(Conv3D(64,kernel_size=(1, 3, 3),activation='relu',
                 input_shape=self.input_shape))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(Conv3D(64,kernel_size=(1, 3, 3),activation='relu',))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(MaxPooling3D(pool_size=(1, 2, 2), strides=(1,2,2)))

        model.add(Conv3D(128,kernel_size=(1, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(Conv3D(128,kernel_size=(1, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(MaxPooling3D(pool_size=(1, 2, 2), strides=(1,2,2)))

        model.add(Conv3D(256,kernel_size=(1, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(Conv3D(256,kernel_size=(1, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(Conv3D(256,kernel_size=(1, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(MaxPooling3D(pool_size=(1, 2, 2), strides=(1,2,2)))

        model.add(Conv3D(512,kernel_size=(1, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(Conv3D(512,kernel_size=(1, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(0, 1, 1)))
        model.add(MaxPooling3D(pool_size=(1, 2, 2), strides=(1,2,2)))

    #Second bloc of layers (self.conv3d = nn.Sequential part for the pytorch code)
        model.add(Conv3D(512,kernel_size=(3, 1, 1),activation='relu'))
        model.add(ZeroPadding3D(padding=(1, 0, 0)))
        model.add(MaxPooling3D(pool_size=(3, 1, 1), strides=(2, 1, 1)))
        model.add(ZeroPadding3D(padding=(1, 0, 0)))

        model.add(Conv3D(512,kernel_size=(3, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(1, 1, 1)))
        model.add(Conv3D(512,kernel_size=(3, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(1, 1, 1)))
        model.add(MaxPooling3D(pool_size=(2, 2, 2), strides=(2, 2, 2)))

        model.add(Conv3D(512,kernel_size=(3, 3, 3),activation='relu'))
        model.add(ZeroPadding3D(padding=(1, 1, 1)))
        model.add(MaxPooling3D(pool_size=(2, 2, 2), strides=(2, 2, 2)))

    #FC Layers
        model.add(Dense(4096, activation='softmax', name='fc1'))
        model.add(Dropout(0.5))
        model.add(Dense(4096, activation='softmax', name='fc2'))
        model.add(Dropout(0.5))
        model.add(Dense(self.nb_classes, activation='softmax', name='fc3'))
        return model


title: How to convert pytorch code that uses SqueezeNet pretrained model to tensorflow

body:
i have the following pytorch code :

model = models.squeezenet1_1(pretrained=True)

num_classes = 2
model.classifier = nn.Sequential (
    nn.Dropout(0.1),
    nn.Conv2d(512, num_classes, kernel_size= (1,1), stride= (1,1)),
    nn.ReLU(),
    nn.AvgPool2d(kernel_size= 13, stride= 1, padding= 0)
    )
model.num_classes = num_classes
criterion = nn.CrossEntropyLoss()

optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

and id like to do convert this code into tensorflow as i need to use tensorflow and not pytorch. I've searched up some things and couldn't quite understand how to do it.

I have been able to get a pretrained SqueezeNet model thanks to the keras_squeezenet package (although it causes some future deprecation warnings with numpy when i load the model) , but then how can i change the Classifier , optimizer and number of classes for the model that i loaded like i did in pytorch ?

Im using tensorflow 1.14.0 and keras 2.3.1   (for some reason i couldn't get to launch tensorflow 2)

Also im only using the CPU because of session rights

EDIT thanks to @Albert :

Right now i have i simply have loaded the model from the package i took from https://github.com/rcmalli/keras-squeezenet , and i did setup my different Image Folders  that i will use :

datagen = ImageDataGenerator()
# load and iterate training dataset
train_it = datagen.flow_from_directory('path to my training', class_mode='binary', batch_size=32)
# load and iterate validation dataset
val_it = datagen.flow_from_directory('path to my validation', class_mode='binary', batch_size=32)
# load and iterate test dataset
test_it = datagen.flow_from_directory('path to my test', class_mode='binary', batch_size=32)

model = SqueezeNet(weights='imagenet')

I have found you must use model.compile to specify optimizer and loss with keras so i think it might look something like this , correct me if i am wrong :

model.compile(loss='BinaryCrossentropy',optimizer='Adam',metrics=['accuracy'])

If it is correct then i simply need to do

model.classifier = nn.Sequential (
    nn.Dropout(0.1),
    nn.Conv2d(512, num_classes, kernel_size= (1,1), stride= (1,1)),
    nn.ReLU(),
    nn.AvgPool2d(kernel_size= 13, stride= 1, padding= 0)
    )
model.num_classes = num_classes

in tenserflow/keras ; how can i do that ?


title: One of the defferent Tensors appear to not have been used in the graph

body:
I want to get outputs from lstm model and update the model's parameters() artificially, but I get the following error:
Runtime ERROR: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.

My RNN model is below:
class RNN(nn.Module):
    def __init__(self):
        super(RNN,self).__init__()
        self.vars = nn.ParameterList() ## parameter need to be updated
        self.lstm = nn.LSTM(input_size=2,hidden_size=128,num_layers=1,batch_first=True)
        self.vars.extend(self.lstm.parameters())
        self.out = nn.Linear(128,5)
        self.vars.extend(self.out.parameters())
    def forward(self,x,params=None):
        if params==None:
            params=self.vars
        self.lstm.weight_ih_l0.data=params[0]
        self.lstm.weight_hh_l0.data=params[1]
        self.lstm.bias_ih_l0.data=params[2]
        self.lstm.bias_hh_l0.data=params[3]

        output, (hidden, cell)=self.lstm(x,None)
        self.out.weight.data=params[4]
        self.out.bias.data=params[5]

        out = self.out(output[:,-1,:])

        return out

    def parameters(self):

        return self.vars

y_hat = RNN(x_spt[i], params = None)
loss = F.cross_entropy(y_hat, y_spt[i])   ## y_spt[i] is labels
grad = torch.autograd.grad(loss, RNN.parameters())

tuples = zip(grad, RNN.parameters())
fast_weights = list(map(lambda p: p[1] - base_lr * p[0], tuples)) ## update RNN.parameters()

for k in range(1, update_step):
    y_hat = RNN(x_spt[i], params = fast_weights)
    loss = F.cross_entropy(y_hat, y_spt[i])

    grad = torch.autograd.grad(loss, fast_weights)  ## error occurs
    tuples = zip(grad, fast_weights)

    fast_weights = list(map(lambda p: p[1] - base_lr * p[0], tuples))


title: In knowledge distillation, how to run the student and the teacher models in parallel?

body:
I am implementing fast DNN model training using knowledge distillation, as illustrated in the figure below, to run the teacher and student models in parallel.
I checked some popular repos like NervanaSystems/distiller and peterliht/knowledge-distillation-pytorch. They execute the forward operations of the student and teacher models step by step, i.e., not in parallel on different devices (GPU or CPU).
I am trying to speed up this training process to run the 2 models at the same time using multiple devices (e.g., loading one model on CPU and not interrupting the GPU training of the other model).
What is the proper way to run 2 models in parallel? Can I use Python multiprocessing library to start 2 processes for the 2 models, i.e., loading 2 model instances and running forward()? I am using MXNet but this is a general question for all ML frameworks.
Edit:
My plan is to put a light-weight pre-trained teacher model on CPU which only runs forward pass with frozen parameters.
The student model is a large model to be trained on GPU (distributedly).
This task is not for model compression.
I suppose moving a light task (teacher's forward pass) to CPU can increase the overlap and make this pipeline faster.
The idea is from a workshop paper: Infer2Train: leveraging inference for better training of deep networks.


title: How to mutate weights of a NN in pytorch

body:
Im playing around with genetic algorithms with pytorch and I'm looking for a more efficient way of mutating the weights of the network (applying a small modification to them)
Right now I have a suboptimal solution where I loop through the parameters and apply a random modification.
child_agent = network()
for param in child_agent.parameters():
        if len(param.shape) == 4:  # weights of Conv2D
            for i0 in range(param.shape[0]):
                for i1 in range(param.shape[1]):
                    for i2 in range(param.shape[2]):
                        for i3 in range(param.shape[3]):
                            param[i0][i1][i2][i3] += mutation_power * np.random.randn()
        elif len(param.shape) == 2:  # weights of linear layer
            for i0 in range(param.shape[0]):
                for i1 in range(param.shape[1]):
                    param[i0][i1] += mutation_power * np.random.randn()
        elif len(param.shape) == 1:  # biases of linear layer or conv layer
            for i0 in range(param.shape[0]):
                param[i0] += mutation_power * np.random.randn()

This solution is bound to my architecture and needs recoding If I decide to add more layers. Is there any way to do this more efficiently and clean? Preferable that it works regardless of how the architecture of my network looks like.
Thanks


title: Why embed dimemsion must be divisible by num of heads in MultiheadAttention?

body:
I am learning the Transformer. Here is the pytorch document for MultiheadAttention. In their implementation, I saw there is a constraint:
 assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"

Why require the constraint: embed_dim must be divisible by num_heads?  If we go back to the equation

Assume:
Q, K,V are n x emded_dim matrices; all the weight matrices W is emded_dim x head_dim,
Then, the concat [head_i, ..., head_h] will be a n x (num_heads*head_dim) matrix;
W^O with size (num_heads*head_dim) x embed_dim
[head_i, ..., head_h] * W^O will become a n x embed_dim output
I don't know why we require embed_dim must be divisible by num_heads.
Let say we have num_heads=10000, the resuts are the same, since the matrix-matrix product will absort this information.


title: How fast is PlaidML in comparison to TF or Pytorch on GPU

body:
I want to know about the training performance of PlaidML, on Nvidia or AMD GPU.
But I couldn't find data that comparing with TF or Pytorch on GPU(CUDA or ROCm). What I only found is comparing with CPU.
Does anyone know how fast is PlaidML for training CNN in comparison to TF or Pytorch on GPU?


title: HTTPError : HTTP Error 403: Forbidden when download MNIST dataset using pytorch

body:
Please help me to solve this problem
I am using colab notebook and loaded it to my drive. When I run the following code to get the dataset using the following code:
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms

train_dataset = torchvision.datasets.MNIST(root="/content/drive/MyDrive/Colab Notebooks/pytorch/Data",
                                           train=True,
                                           transform=transforms.ToTensor(),
                                           download=True
                                           )

This is my error
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to /content/drive/MyDrive/Colab Notebooks/pytorch/Data/MNIST/raw/train-images-idx3-ubyte.gz
0/? [00:00<?, ?it/s]
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-12-32702cf15284> in <module>()
      3                                            train=True,
      4                                            transform=transforms.ToTensor(),
----> 5                                            download=True
      6                                            )

11 frames
/usr/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650
    651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden


title: Wild discrepancies between training DeepLab ResNet V3 on Google Colab versus on local machine

body:
I am attempting to train Deeplab Resnet V3 to perform semantic segmentation on a custom dataset. I had been working on my local machine however my GPU is just a small Quadro T1000 so I decided to move my model onto Google Colab to take advantage of their GPU instances and get better results.
Whilst I get the speed increase I was hoping for, I am getting wildly different training losses on colab compared to my local machine. I have copied and pasted the exact same code so the only difference I can find would be in the dataset. I am using the exact same dataset except for the one on colab is a copy of the local dataset on Google Drive. I have noticed that Drive orders file differently on Windows but I can't see how this is a problem since I randomly shuffle the dataset. I understand that these random splitting can cause small differences in the outputs however a difference of about 10x in the training losses doesn't make sense.
I have also tried running the version on colab with different random seeds, different batch sizes, different train_test_split parameters, and changing the optimizer from SGD to Adam, however, this still causes the model to converge very early at a loss of around 0.5.
Here is my code:
import torch
from torch.utils import data
from torchvision import transforms
from customdatasets import SegmentationDataSet
import pathlib
from sklearn.model_selection import train_test_split
from customtransforms import Compose, AlbuSeg2d, DenseTarget
from customtransforms import MoveAxis, Normalize01, Resize
import albumentations
import matplotlib.pyplot as plt
import time
import GPUtil

def get_filenames_of_path(path: pathlib.Path, ext: str = '*'):
    """Returns a list of files in a directory/path. Uses pathlib."""
    filenames = [file for file in path.glob(ext) if file.is_file()]
    return filenames

if __name__ == '__main__':

    root = pathlib.Path.cwd() / 'train'
    inputs = get_filenames_of_path(root / 'input')
    targets = get_filenames_of_path(root / 'target')

# training transformations and augmentations
    transforms_training = Compose([
        Resize(input_size=(128, 128, 3), target_size=(128, 128)),
        AlbuSeg2d(albu=albumentations.HorizontalFlip(p=0.5)),
        MoveAxis(),
        Normalize01()
    ])
# validation transformations
    transforms_validation = Compose([
        Resize(input_size=(128, 128, 3), target_size=(128, 128)),
        MoveAxis(),
        Normalize01()
    ])
    if torch.cuda.is_available():
        device = torch.device('cuda')
    else:
        device = torch.device('cpu')

    random_seed = 142
    train_size = 0.8

    inputs_train, inputs_valid = train_test_split(
        inputs,
        random_state=random_seed,
        train_size=train_size,
        shuffle=True)
    targets_train, targets_valid = train_test_split(
        targets,
        random_state=random_seed,
        train_size=train_size,
        shuffle=True)

    dataset_train = SegmentationDataSet(inputs=inputs_train,
                                    targets=targets_train,
                                    transform=transforms_training,
                                    device=device)

    dataset_valid = SegmentationDataSet(inputs=inputs_valid,
                                    targets=targets_valid,
                                    transform=transforms_validation,
                                    device=device)

    dataloader_training = data.DataLoader(dataset=dataset_train,
                                      batch_size=15,
                                      shuffle=True,
                                      num_workers=4,
                                      pin_memory=True)

    dataloader_validation = data.DataLoader(dataset=dataset_valid,
                                        batch_size=15,
                                        shuffle=True,
                                        num_workers=4,
                                        pin_memory=True)

    model = torch.hub.load('pytorch/vision:v0.6.0', 'deeplabv3_resnet101', pretrained=False)

    criterion = torch.nn.CrossEntropyLoss()

    model = model.to(device)

    optimizer = torch.optim.SGD(model.parameters(), lr=0.001, momentum=0.99)

    epochs = 10
    steps = 0
    running_loss = 0
    print_every = 10
    train_losses, valid_losses = [], []

    start_time = time.time()
    prev_time = time.time()

    for epoch in range(epochs):
        #Training
        for inputs, labels in dataloader_training:
            steps += 1
            inputs, labels = inputs.to(device, non_blocking=True), labels.to(device,non_blocking=True)
            optimizer.zero_grad()
            logps = model(inputs)
            loss = criterion(logps['out'], labels)
            loss.backward()
            optimizer.step()
            running_loss += loss.item()

            if steps % print_every == 0:
                train_losses.append(running_loss / len(dataloader_training))
                epoch_time = time.time()
                elasped_time = epoch_time - prev_time
                prev_time = epoch_time
                print(f"Epoch {epoch + 1}/{epochs}.. "
                    f"Train loss: {running_loss / print_every:.3f}.. "
                    f"Elapsed time: {elasped_time}")

                running_loss = 0
                model.train()
        # Evaluation
        valid_loss = 0
        accuracy = 0
        model.eval()
        with torch.no_grad():
            for inputs, labels in dataloader_validation:
                inputs, labels = inputs.to(device, non_blocking=True), labels.to(device, non_blocking=True)
                logps = model.forward(inputs)
                batch_loss = criterion(logps['out'], labels)
                valid_loss += batch_loss.item()

                ps = torch.exp(logps['out'])
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class == labels.view(*top_class.shape)
                accuracy += torch.mean(equals.type(torch.FloatTensor)).item()
        valid_losses.append(valid_loss / len(dataloader_validation))
        print(f"Epoch {epoch + 1}/{epochs}.. "
            f"Validation loss: {valid_loss / len(dataloader_training):.3f}.. "
            f"Validation accuracy: {accuracy / len(dataloader_training):.3f} ")
        model.train()
    torch.save(model, 'model.pth')

    end_time = time.time()
    total_time = end_time - start_time
    print("Total Time: ", total_time)
    plt.plot(train_losses, label='Training loss')
    plt.plot(valid_losses, label='Validation loss')
    plt.legend(frameon=False)
    plt.show()

This is the output for one epoch on Colab:
Epoch 1/10.. Train loss: 2.080.. Elapsed time: 12.156640768051147
Epoch 1/10.. Train loss: 1.231.. Elapsed time: 8.76858925819397
Epoch 1/10.. Train loss: 1.051.. Elapsed time: 8.315532445907593
Epoch 1/10.. Train loss: 0.890.. Elapsed time: 8.249168634414673
Epoch 1/10.. Train loss: 0.839.. Elapsed time: 8.248667478561401
Epoch 1/10.. Train loss: 0.807.. Elapsed time: 8.120820999145508
Epoch 1/10.. Train loss: 0.742.. Elapsed time: 8.298616886138916
Epoch 1/10.. Train loss: 0.726.. Elapsed time: 8.170734167098999
Epoch 1/10.. Train loss: 0.677.. Elapsed time: 8.221246004104614
Epoch 1/10.. Train loss: 0.698.. Elapsed time: 8.124614000320435
Epoch 1/10.. Train loss: 0.675.. Elapsed time: 8.197462558746338
Epoch 1/10.. Train loss: 0.682.. Elapsed time: 8.263437509536743
Epoch 1/10.. Train loss: 0.626.. Elapsed time: 8.156179189682007
Epoch 1/10.. Train loss: 0.632.. Elapsed time: 8.268096446990967
Epoch 1/10.. Train loss: 0.616.. Elapsed time: 8.214547872543335
Epoch 1/10.. Train loss: 0.585.. Elapsed time: 8.31475019454956
Epoch 1/10.. Train loss: 0.598.. Elapsed time: 8.388074398040771
Epoch 1/10.. Train loss: 0.626.. Elapsed time: 8.179292440414429
Epoch 1/10.. Train loss: 0.612.. Elapsed time: 8.252359390258789
Epoch 1/10.. Train loss: 0.592.. Elapsed time: 8.284745693206787
Epoch 1/10.. Train loss: 0.597.. Elapsed time: 8.31213927268982
Epoch 1/10.. Train loss: 0.566.. Elapsed time: 8.164374113082886
Epoch 1/10.. Train loss: 0.556.. Elapsed time: 8.300082206726074
Epoch 1/10.. Train loss: 0.568.. Elapsed time: 8.26304841041565
Epoch 1/10.. Train loss: 0.572.. Elapsed time: 8.309881448745728
Epoch 1/10.. Train loss: 0.586.. Elapsed time: 8.211671352386475
Epoch 1/10.. Train loss: 0.586.. Elapsed time: 8.321797609329224
Epoch 1/10.. Train loss: 0.535.. Elapsed time: 8.318871021270752
Epoch 1/10.. Train loss: 0.543.. Elapsed time: 8.152915239334106
Epoch 1/10.. Train loss: 0.569.. Elapsed time: 8.251380205154419
Epoch 1/10.. Train loss: 0.526.. Elapsed time: 8.29153847694397
Epoch 1/10.. Train loss: 0.565.. Elapsed time: 8.15071702003479
Epoch 1/10.. Train loss: 0.542.. Elapsed time: 8.253364562988281
Epoch 1/10.. Validation loss: 0.182.. Validation accuracy: 0.271

And here is the output on my local machine:
Epoch 1/10.. Train loss: 2.932.. Elapsed time: 32.148621797561646
Epoch 1/10.. Train loss: 1.852.. Elapsed time: 14.120505809783936
Epoch 1/10.. Train loss: 0.887.. Elapsed time: 14.210048198699951
Epoch 1/10.. Train loss: 0.618.. Elapsed time: 14.23294186592102
Epoch 1/10.. Train loss: 0.549.. Elapsed time: 14.212541103363037
Epoch 1/10.. Train loss: 0.519.. Elapsed time: 14.047481775283813
Epoch 1/10.. Train loss: 0.506.. Elapsed time: 14.060708045959473
Epoch 1/10.. Train loss: 0.347.. Elapsed time: 14.301624059677124
Epoch 1/10.. Train loss: 0.399.. Elapsed time: 13.9844491481781
Epoch 1/10.. Train loss: 0.361.. Elapsed time: 13.957871913909912
Epoch 1/10.. Train loss: 0.305.. Elapsed time: 14.164010763168335
Epoch 1/10.. Train loss: 0.296.. Elapsed time: 14.001536846160889
Epoch 1/10.. Train loss: 0.298.. Elapsed time: 14.019971132278442
Epoch 1/10.. Train loss: 0.271.. Elapsed time: 13.951345443725586
Epoch 1/10.. Train loss: 0.252.. Elapsed time: 14.037938594818115
Epoch 1/10.. Train loss: 0.283.. Elapsed time: 13.944657564163208
Epoch 1/10.. Train loss: 0.299.. Elapsed time: 13.977224826812744
Epoch 1/10.. Train loss: 0.219.. Elapsed time: 13.941975355148315
Epoch 1/10.. Train loss: 0.242.. Elapsed time: 13.936140060424805
Epoch 1/10.. Train loss: 0.244.. Elapsed time: 13.942122459411621
Epoch 1/10.. Train loss: 0.216.. Elapsed time: 13.960899114608765
Epoch 1/10.. Train loss: 0.186.. Elapsed time: 13.956881523132324
Epoch 1/10.. Train loss: 0.241.. Elapsed time: 13.944581985473633
Epoch 1/10.. Train loss: 0.203.. Elapsed time: 13.934357404708862
Epoch 1/10.. Train loss: 0.189.. Elapsed time: 13.938358306884766
Epoch 1/10.. Train loss: 0.181.. Elapsed time: 13.944468021392822
Epoch 1/10.. Train loss: 0.186.. Elapsed time: 13.946297407150269
Epoch 1/10.. Train loss: 0.164.. Elapsed time: 13.940366744995117
Epoch 1/10.. Train loss: 0.165.. Elapsed time: 13.938241720199585
Epoch 1/10.. Train loss: 0.176.. Elapsed time: 14.015569925308228
Epoch 1/10.. Train loss: 0.165.. Elapsed time: 14.019208669662476
Epoch 1/10.. Train loss: 0.175.. Elapsed time: 14.149503469467163
Epoch 1/10.. Train loss: 0.159.. Elapsed time: 14.128302097320557
Epoch 1/10.. Train loss: 0.155.. Elapsed time: 13.935027837753296
Epoch 1/10.. Train loss: 0.137.. Elapsed time: 13.937382221221924
Epoch 1/10.. Train loss: 0.127.. Elapsed time: 13.929635524749756
Epoch 1/10.. Train loss: 0.133.. Elapsed time: 13.935472011566162
Epoch 1/10.. Train loss: 0.152.. Elapsed time: 13.922808647155762
Epoch 1/10.. Validation loss: 0.032.. Validation accuracy: 0.239

I won't paste more than this since it's long and takes a while to run but by the end of the 3rd epoch, the loss on the Colab model is still bouncing around 0.5 whereas locally it reaches 0.02.
If anyone could help me resolve this issue it would be greatly appreciated.


title: Time series prediction (GRU) with several meta information

body:
I'm currently trying to build a deep learning model for time series prediction. For the trainings data I don't only have the actual time series, but also some meta information for every time step of the series.

I'm implementing the model using pytorch. My problem is, that I don't know how to use those meta information to build an embedding layer containing all those information, because it only can be 2d. I know that in keras, it is possible to fit the model when putting multiple time series in a list. But I'm not sure if pytorch has the same option?

It should be the same idea as using word and character embeddings all at once for text classification/prediction. Does anyone have an idea?

Thank you! :3


title: Same working model in Keras not being improved in Pytorch

body:
I'm converting a basic LSTM many-to-one architecture to predict the next single element in a sequence, written in Keras to Pytorch. NN architecture is the following (whole code can be found here):

model = Sequential()
model.add(LSTM(
    512,
    input_shape=(network_input.shape[1], network_input.shape[2]),
    return_sequences=True
))
model.add(Dropout(0.3))
model.add(LSTM(512, return_sequences=True))
model.add(Dropout(0.3))
model.add(LSTM(512))
model.add(Dense(256))
model.add(Dropout(0.3))
model.add(Dense(n_vocab))
model.add(Activation('softmax'))
model.compile(loss='categorical_crossentropy', optimizer='rmsprop')

Running both models with the same data (yes, I've explicitly checked that), both start with a loss value ~ 4, but after 100 epochs or so, Keras already reached a loss ~ 0.02, which gives the desired results.

However, Pytorch model is stuck around ~ 3.4 after 20 epochs. I've tried many things:

Play with LR: It explodes when LR is too high, so this means that at least parameters are being updated.
Different optimizers, SGD, Adam, RMSprop, but same results with all.
Swap between .view[], .squeeze_ and indexing when accessing last sequence element.
Add, remove and modify non-linear activation functions and dropout.
Remove manual initialization for x_0 and h_0.

Here is the code for my model:

class NNP_RNN(nn.Module):
    def __init__(self):
        super(NNP_RNN, self).__init__()
        self.lstm_1 = nn.LSTM(input_size=1, hidden_size=512, batch_first=True)
        self.lstm_2 = nn.LSTM(input_size=512, hidden_size=512, batch_first=True)
        self.lstm_3 = nn.LSTM(input_size=512, hidden_size=512, batch_first=True)

        self.dense_1 = nn.Linear(in_features=512, out_features=256)
        self.dense_2 = nn.Linear(in_features=256, out_features=58)

    def forward(self, x):
        batch_size = x.size(0)
        h_0 = NNP_RNN.init_hidden((1, batch_size, 512))
        c_0 = NNP_RNN.init_hidden((1, batch_size, 512))

        x, _ = self.lstm_1(x, (h_0, c_0))
        x = F.dropout(x, 0.3)

        x, _ = self.lstm_2(x, (h_0, c_0))
        x = F.dropout(x, 0.2)

        _, (x, _) = self.lstm_3(x, (h_0, c_0))
        x = x.squeeze_(0)

        x = self.dense_1(x)
        x = F.dropout(x, 0.1)

        x = self.dense_2(x)

        return x

    @staticmethod
    def init_hidden(dims):
        return torch.zeros(dims, device=device)

And the training process:

optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, factor=0.2, verbose=True, patience=5)
criterion = nn.CrossEntropyLoss()

for epoch in range(1, epochs + 1):
    epoch_loss = 0
    epoch_corrects = 0
    for features, labels in tqdm(data, ncols=800):
        features = features.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        batch_size = features.size(0)
        output = model(features)

        loss = criterion(output, labels)
        loss.backward()

        optimizer.step()

        corrects = torch.argmax(output, dim=1)
        corrects = torch.eq(corrects, labels).sum().item()
        epoch_corrects += corrects

        epoch_loss += loss.clone() * batch_size

    epoch_loss /= len(data.dataset)
    epoch_corrects /= len(data.dataset)
    print(f'Loss epoch #{epoch} = {epoch_loss:.10f}, Accuracy = {epoch_corrects}')

    scheduler.step(epoch_loss)


title: Pytorch how to multiply tensors of variable size except the first dimention

body:
I have one tensor which is A = 40x1.

i need to multiply this one with 3 other tensors: B = 40x100x384, C = 40x10, D=40x10.

for example in tensor B, we got 40 100x384 matrixes and i need each one of these matrixes to be multiplied with its corresponding element from A

what is the best way to do this in pytorch? Suppose that we could have more matrixes like B,C,D they will always be in the style 40xKxL or 40xJ


title: What does this warning mean in PATE analysis?

body:
Got warning while doing PATE analysis:

Warning: May not have used enough values of l. Increase 'moments' variable and run again.

from syft.frameworks.torch.differential_privacy import pate

data_dep_eps, data_ind_eps = pate.perform_analysis(teacher_preds=preds, indices=indices, noise_eps=0.1)
print("Data Independent Epsilon:", data_ind_eps)
print("Data Dependent Epsilon:", data_dep_eps)

It has gone after increasing the value of the "moment" parameter in the "pate.perform_analysis" analysis function. But I want to know why this was so.
data_dep_eps, data_ind_eps = pate.perform_analysis(teacher_preds=preds, indices=indices, noise_eps=0.1,moments=20)
print("Data Independent Epsilon:", data_ind_eps)
print("Data Dependent Epsilon:", data_dep_eps)


title: Pytorch error on multi-GPU setup: cannot import name 'invoke_remote_python_udf' from 'torch.distributed'

body:
I'm getting this below error when importing Pytorch.

"cannot import name 'invoke_remote_python_udf' from 'torch.distributed'"


title: pytorch .cuda() can't get the tensor to cuda

body:
I try to get my data onto gpu,but it doesn't work;

in my train.py

if __name__ == '__main__':
ten = torch.FloatTensor(2)
ten = ten.cuda()
print(ten)

args = config()
train_net(args, args.train_net, loss_config=net_loss_config[args.train_net])

when it runs ,it prints this

tensor([0., 0.])

the tensor is not on cuda

but in test.py

import torch
ten=torch.FloatTensor(2)
ten=ten.cuda()
print(ten)

it prints this

tensor([1.4013e-45, 0.0000e+00], device='cuda:0')

now the tensor is on cuda


title: Remove RELU activation from Resnet model in Pytorch

body: How to remove all RELU activation layers from Resnet model in pytorch and if possible replace it by a linear activation layer?


title: Using Two Data Iterators Under One Loop in Pytorch

body:
I would like to ask a question regarding Data Iterators in Pytorch.

I have two datasets, and would like to use a different data iterator for each of them to train one model.

To do so, I have used DataLoader for each of them to get data iterators, and shuffle argument was set to True.

Then, I have used them as following under one loop, for each epoch:

for idx, datum in enumerate(zip(data_iterator1, data_iterator2)):

Here, the length of data_iterator1 is ~28 times longer than the length of data_iterator2.

Here I have the following questions in my mind:

1) When the loop ends? Does it end when the loop sees all the batches belonging to data_iterator2, or it ends until seeing all the batches belonging to data_iterator1?

2) If the loop ends after seeing all the batches belonging to data_iterator2, at the next epoch, are the data belonging to both data iterators shuffled again?


title: Why use Caffe2 or Core-ML instead of LibTorch(.pt file) on iOS?

body:
It seems like there are several ways to run Pytorch models on iOS.

PyTorch(.pt) -> onnx -> caffe2
PyTorch(.pt) -> onnx -> Core-ML (.mlmodel)
PyTorch(.pt) -> LibTorch (.pt)
PyTorch Mobile?

What is the difference between the above methods?
Why people use caffe2 or Core-ml (.mlmodel), which requires model format conversion, instead of LibTorch?


title: Can anyone explain about the "non-sliding window" statement in Feature Pyramid Networks for Object Detection paper?

body:
Feature Pyramid Networks for Object Detection adopt RPN technique to create the detector, and it use sliding window technique to classify. How come there is a statement for "non-sliding window" in 5.2 section?

The extended statement in the paper :
5.2. Object Detection with Fast/Faster R-CNN
Next we investigate FPN for region-based (non-sliding window) detectors.

In my understanding, FPN using sliding window in detection task. This is also mentioned in
https://medium.com/@jonathan_hui/understanding-feature-pyramid-networks-for-object-detection-fpn-45b227b9106c the statement is

"FPN extracts feature maps and later feeds into a detector, says RPN, for object detection. RPN applies a sliding window over the feature maps to make predictions on the objectness (has an object or not) and the object boundary box at each location."

Thank you in advanced.


title: Can't parse arguments (deep learning tutorial using pytorch)

body:
I'm following this tutorial.
The first scripts run fine and I have a "data" folder in the folder of my scripts containing the MRI data downloaded from MRnet.
However when it comes to the "train" script I get an error. Here's the full script and the error (using jupyter notebook):
import shutil
import os
import time
from datetime import datetime
import argparse
import numpy as np
from tqdm import tqdm

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable
from torchsample.transforms import RandomRotate, RandomTranslate, RandomFlip, ToTensor, Compose, RandomAffine
from torchvision import transforms
import torch.nn.functional as F
from tensorboardX import SummaryWriter
import nbimporter

from dataloader import MRDataset
import model

from sklearn import metrics

def train_model(model, train_loader, epoch, num_epochs, optimizer, writer, current_lr, log_every=100):
    _ = model.train()

    if torch.cuda.is_available():
        model.cuda()

    y_preds = []
    y_trues = []
    losses = []

    for i, (image, label, weight) in enumerate(train_loader):
        optimizer.zero_grad()

        if torch.cuda.is_available():
            image = image.cuda()
            label = label.cuda()
            weight = weight.cuda()

        label = label[0]
        weight = weight[0]

        prediction = model.forward(image.float())

        loss = torch.nn.BCEWithLogitsLoss(weight=weight)(prediction, label)
        loss.backward()
        optimizer.step()

        loss_value = loss.item()
        losses.append(loss_value)

        probas = torch.sigmoid(prediction)

        y_trues.append(int(label[0][1]))
        y_preds.append(probas[0][1].item())

        try:
            auc = metrics.roc_auc_score(y_trues, y_preds)
        except:
            auc = 0.5

        writer.add_scalar('Train/Loss', loss_value,
                          epoch * len(train_loader) + i)
        writer.add_scalar('Train/AUC', auc, epoch * len(train_loader) + i)

        if (i % log_every == 0) & (i > 0):
            print('''[Epoch: {0} / {1} |Single batch number : {2} / {3} ]| avg train loss {4} | train auc : {5} | lr : {6}'''.
                  format(
                      epoch + 1,
                      num_epochs,
                      i,
                      len(train_loader),
                      np.round(np.mean(losses), 4),
                      np.round(auc, 4),
                      current_lr
                  )
                  )

    writer.add_scalar('Train/AUC_epoch', auc, epoch + i)

    train_loss_epoch = np.round(np.mean(losses), 4)
    train_auc_epoch = np.round(auc, 4)
    return train_loss_epoch, train_auc_epoch

def evaluate_model(model, val_loader, epoch, num_epochs, writer, current_lr, log_every=20):
    _ = model.eval()

    if torch.cuda.is_available():
        model.cuda()

    y_trues = []
    y_preds = []
    losses = []

    for i, (image, label, weight) in enumerate(val_loader):

        if torch.cuda.is_available():
            image = image.cuda()
            label = label.cuda()
            weight = weight.cuda()

        label = label[0]
        weight = weight[0]

        prediction = model.forward(image.float())

        loss = torch.nn.BCEWithLogitsLoss(weight=weight)(prediction, label)

        loss_value = loss.item()
        losses.append(loss_value)

        probas = torch.sigmoid(prediction)

        y_trues.append(int(label[0][1]))
        y_preds.append(probas[0][1].item())

        try:
            auc = metrics.roc_auc_score(y_trues, y_preds)
        except:
            auc = 0.5

        writer.add_scalar('Val/Loss', loss_value, epoch * len(val_loader) + i)
        writer.add_scalar('Val/AUC', auc, epoch * len(val_loader) + i)

        if (i % log_every == 0) & (i > 0):
            print('''[Epoch: {0} / {1} |Single batch number : {2} / {3} ] | avg val loss {4} | val auc : {5} | lr : {6}'''.
                  format(
                      epoch + 1,
                      num_epochs,
                      i,
                      len(val_loader),
                      np.round(np.mean(losses), 4),
                      np.round(auc, 4),
                      current_lr
                  )
                  )

    writer.add_scalar('Val/AUC_epoch', auc, epoch + i)

    val_loss_epoch = np.round(np.mean(losses), 4)
    val_auc_epoch = np.round(auc, 4)
    return val_loss_epoch, val_auc_epoch

def get_lr(optimizer):
    for param_group in optimizer.param_groups:
        return param_group['lr']

def run(args):
    log_root_folder = "./logs/{0}/{1}/".format(args.task, args.plane)
    if args.flush_history == 1:
        objects = os.listdir(log_root_folder)
        for f in objects:
            if os.path.isdir(log_root_folder + f):
                shutil.rmtree(log_root_folder + f)

    now = datetime.now()
    logdir = log_root_folder + now.strftime("%Y%m%d-%H%M%S") + "/"
    os.makedirs(logdir)

    writer = SummaryWriter(logdir)

    augmentor = Compose([
        transforms.Lambda(lambda x: torch.Tensor(x)),
        RandomRotate(25),
        RandomTranslate([0.11, 0.11]),
        RandomFlip(),
        transforms.Lambda(lambda x: x.repeat(3, 1, 1, 1).permute(1, 0, 2, 3)),
    ])

    train_dataset = MRDataset('./data/', args.task,
                              args.plane, transform=augmentor, train=True)
    train_loader = torch.utils.data.DataLoader(
        train_dataset, batch_size=1, shuffle=True, num_workers=11, drop_last=False)

    validation_dataset = MRDataset(
        './data/', args.task, args.plane, train=False)
    validation_loader = torch.utils.data.DataLoader(
        validation_dataset, batch_size=1, shuffle=-True, num_workers=11, drop_last=False)

    mrnet = model.MRNet()

    if torch.cuda.is_available():
        mrnet = mrnet.cuda()

    optimizer = optim.Adam(mrnet.parameters(), lr=args.lr, weight_decay=0.1)

    if args.lr_scheduler == "plateau":
        scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
            optimizer, patience=3, factor=.3, threshold=1e-4, verbose=True)
    elif args.lr_scheduler == "step":
        scheduler = torch.optim.lr_scheduler.StepLR(
            optimizer, step_size=3, gamma=args.gamma)

    best_val_loss = float('inf')
    best_val_auc = float(0)

    num_epochs = args.epochs
    iteration_change_loss = 0
    patience = args.patience
    log_every = args.log_every

    t_start_training = time.time()

    for epoch in range(num_epochs):
        current_lr = get_lr(optimizer)

        t_start = time.time()

        train_loss, train_auc = train_model(
            mrnet, train_loader, epoch, num_epochs, optimizer, writer, current_lr, log_every)
        val_loss, val_auc = evaluate_model(
            mrnet, validation_loader, epoch, num_epochs, writer, current_lr)

        if args.lr_scheduler == 'plateau':
            scheduler.step(val_loss)
        elif args.lr_scheduler == 'step':
            scheduler.step()

        t_end = time.time()
        delta = t_end - t_start

        print("train loss : {0} | train auc {1} | val loss {2} | val auc {3} | elapsed time {4} s".format(
            train_loss, train_auc, val_loss, val_auc, delta))

        iteration_change_loss += 1
        print('-' * 30)

        if val_auc > best_val_auc:
            best_val_auc = val_auc
            if bool(args.save_model):
                file_name = f'model_{args.prefix_name}_{args.task}_{args.plane}_val_auc_{val_auc:0.4f}_train_auc_{train_auc:0.4f}_epoch_{epoch+1}.pth'
                for f in os.listdir('./models/'):
                    if (args.task in f) and (args.plane in f) and (args.prefix_name in f):
                        os.remove(f'./models/{f}')
                torch.save(mrnet, f'./models/{file_name}')

        if val_loss < best_val_loss:
            best_val_loss = val_loss
            iteration_change_loss = 0

        if iteration_change_loss == patience:
            print('Early stopping after {0} iterations without the decrease of the val loss'.
                  format(iteration_change_loss))
            break

    t_end_training = time.time()
    print(f'training took {t_end_training - t_start_training} s')

def parse_arguments():
    parser = argparse.ArgumentParser()
    parser.add_argument('-t', '--task', type=str, required=True,
                        choices=['abnormal', 'acl', 'meniscus'])
    parser.add_argument('-p', '--plane', type=str, required=True,
                        choices=['sagittal', 'coronal', 'axial'])
    parser.add_argument('--prefix_name', type=str, required=True)
    parser.add_argument('--augment', type=int, choices=[0, 1], default=1)
    parser.add_argument('--lr_scheduler', type=str,
                        default='plateau', choices=['plateau', 'step'])
    parser.add_argument('--gamma', type=float, default=0.5)
    parser.add_argument('--epochs', type=int, default=50)
    parser.add_argument('--lr', type=float, default=1e-5)
    parser.add_argument('--flush_history', type=int, choices=[0, 1], default=0)
    parser.add_argument('--save_model', type=int, choices=[0, 1], default=1)
    parser.add_argument('--patience', type=int, default=5)
    parser.add_argument('--log_every', type=int, default=100)
    args = parser.parse_args()
    return args

if __name__ == "__main__":
    args = parse_arguments()
    run(args)

Error:
usage: ipykernel_launcher.py [-h] -t {abnormal,acl,meniscus} -p
                             {sagittal,coronal,axial} --prefix_name
                             PREFIX_NAME [--augment {0,1}]
                             [--lr_scheduler {plateau,step}] [--gamma GAMMA]
                             [--epochs EPOCHS] [--lr LR]
                             [--flush_history {0,1}] [--save_model {0,1}]
                             [--patience PATIENCE] [--log_every LOG_EVERY]
ipykernel_launcher.py: error: the following arguments are required: -t/--task, -p/--plane, --prefix_name

%tb:
    ---------------------------------------------------------------------------
SystemExit                                Traceback (most recent call last)
<ipython-input-3-e6a34ab63dc0> in <module>
    275
    276 if __name__ == "__main__":
--> 277     args = parse_arguments()
    278     run(args)

<ipython-input-3-e6a34ab63dc0> in parse_arguments()
    270     parser.add_argument('--patience', type=int, default=5)
    271     parser.add_argument('--log_every', type=int, default=100)
--> 272     args = parser.parse_args()
    273     return args
    274

~\anaconda3\envs\Pytorch\lib\argparse.py in parse_args(self, args, namespace)
   1753     # =====================================
   1754     def parse_args(self, args=None, namespace=None):
-> 1755         args, argv = self.parse_known_args(args, namespace)
   1756         if argv:
   1757             msg = _('unrecognized arguments: %s')

~\anaconda3\envs\Pytorch\lib\argparse.py in parse_known_args(self, args, namespace)
   1785         # parse the arguments and exit if there are any errors
   1786         try:
-> 1787             namespace, args = self._parse_known_args(args, namespace)
   1788             if hasattr(namespace, _UNRECOGNIZED_ARGS_ATTR):
   1789                 args.extend(getattr(namespace, _UNRECOGNIZED_ARGS_ATTR))

~\anaconda3\envs\Pytorch\lib\argparse.py in _parse_known_args(self, arg_strings, namespace)
   2020         if required_actions:
   2021             self.error(_('the following arguments are required: %s') %
-> 2022                        ', '.join(required_actions))
   2023
   2024         # make sure all required groups had one option present

~\anaconda3\envs\Pytorch\lib\argparse.py in error(self, message)
   2506         self.print_usage(_sys.stderr)
   2507         args = {'prog': self.prog, 'message': message}
-> 2508         self.exit(2, _('%(prog)s: error: %(message)s\n') % args)

~\anaconda3\envs\Pytorch\lib\argparse.py in exit(self, status, message)
   2493         if message:
   2494             self._print_message(message, _sys.stderr)
-> 2495         _sys.exit(status)
   2496
   2497     def error(self, message):

SystemExit: 2

I have no clue on how to move forwards. I'm stranded here. Anyone know where to go from here?


title: PyTorch gets "CUDA error: device-side assert triggered" - how do I fix it?

body:
When trying to run PyTorch with GPU, I get a strange, cryptic error message: "device-side assert triggered".
For example, if I type
import torch
torch.randn(1, device='cuda')

I get:

RuntimeError: CUDA error: device-side assert triggered

How do I fix this error? Typing torch.cuda.is_available() returns True, and restarting my computer doesn't help either.


title: Extracting execution DAG from PyTorch Module

body:
Let us consider a simple feed-forward deep net or even a resnet. In both case, you can think of the execution flow as a DAG (directed acyclic graph) where nodes are layers (it's a chain if there's no skip connexion).
For some reason, I need this DAG for pyTorch models. Up to now I have been using custom object subclassing nn.Module (that stores the information that I need) but I would like to be able ultimately to use an arbitrary model in input of my pipeline. Is there a way to extract this DAG automatically ? (it does not matter if it's not working in all cases, I am mainly looking for general ideas / principles to achieve my goal).


title: Pytorch Conv2d Autoencoder Output Shape

body:
I built the bellow convolution autoencoder and trying to tune it to get encoder output shape (x_encoder) of [NxHxW] = 1024 without increasing loss. Currently my output shape is [4, 64, 64] Any ideas?
# define the NN architecture
class ConvAutoencoder(nn.Module):
    def __init__(self):
        super(ConvAutoencoder, self).__init__()
        ## encoder layers ##
        # conv layer (depth from in --> 16), 3x3 kernels
        self.conv1 = nn.Conv2d(1, 16, 3, padding=1)
        # conv layer (depth from 16 --> 4), 3x3 kernels
        self.conv2 = nn.Conv2d(16, 4, 3, padding=1)
        # pooling layer to reduce x-y dims by two; kernel and stride of 2
        self.pool = nn.MaxPool2d(2, 2)

        ## decoder layers ##
        ## a kernel of 2 and a stride of 2 will increase the spatial dims by 2
        self.t_conv1 = nn.ConvTranspose2d(4, 16, 2, stride=2)
        self.t_conv2 = nn.ConvTranspose2d(16, 1, 2, stride=2)

    def forward(self, x):
        ## encode ##
        # add hidden layers with relu activation function
        # and maxpooling after
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        # add second hidden layer
        x = F.relu(self.conv2(x))
        x = self.pool(x)  # compressed representation
        x_encoder = x
        ## decode ##
        # add transpose conv layers, with relu activation function
        x = F.relu(self.t_conv1(x))
        # output layer (with sigmoid for scaling from 0 to 1)
        x = F.sigmoid(self.t_conv2(x))

        return x, x_encoder


title: (Differentiable Image Sampling) Custom Integer Sampling Kernel, Spatial Transformer Network

body:
I was looking through the Spatial Transformer Network paper, and I am trying to implement a custom grid_sample function (inheriting the autograd.Function class) in PyTorch for the Integer Sampling Kernel.
While defining the backward function, I have come across the following conundrum.
Given that the integer sampling works as the following:

I think that the gradients w.r.t the input map and the transformed grid (x_i^s, y_i^s) should be like the following:
Gradient w.r.t. input map:

Gradient w.r.t transformed grid (x_i^s):

Gradient w.r.t transformed grid (y_i^s):

as the derivative of the Kronecker delta function is zero (I'm unsure about this!! - HELP)
Derivative of the Kronecker delta?

Thus I am reaching a conclusion that the gradient w.r.t to the input should be: a tensor of the same size as the input filled with ones if the pixel was sampled and 0 if it wasn't sampled, and the gradient w.r.t the transformed grid should be a tensor full of zeros.
However, if the gradient of the transformed grid is 0, then due to the chain rule, no information will be passed on to the layers before the integer sampler. Therefore I think the derivate with respect to the grid should be something else. Could anybody point out what I'm doing wrong?
Many thanks in advance!


title: Pytorch adaptive_avg_pool2d algorithm

body:
Anyone knows the algorithm for pytorch adaptive_avg_pool2d, for example,

adaptive_avg_pool2d(image,[14,14])

so question:
    I want to do the same in keras neural network, for any give inputs, want to get 14*14 output. Any suggestions?


title: How to select subtensor from a tensor

body:
I have a tensor A with the dimension of [N1, N2, N3/2, 2, N4, N5]. Another tensor B is an boolean index tensor with the dimension of [N1, N2, N3/2] = 1,0,0,0,1....
Now, I want to get a subtensor C with the dimension of [N1, N2, N3/2, N4, N5] using the index of B .
How could I implement this with pytorch?
Note that I don't incline to use for-loop because it is very slow.

I have looked up some functions in pytorch but found none of them are suitable for my task


title: What does 'Epoch' mean in training Generative Adversarial Networks

body:
I am training a GAN with text data. When I train the discriminator, I randomly sample m positive data from the dataset and generate m negative data with the generator. I found many papers mention about details of implementation such as training epochs. About the training epochs, I have a question about sampling positive data:

Sample from the dataset (maybe shuffled) in order, when the whole dataset is covered, we call 1 epoch
Just as I did, randomly sample positive data, when the total amount of sampled data is the same size as the dataset, we call 1 epoch

Which one is right? or which one is commonly used? or which one is better?


title: Weak optimizers in Pytorch

body:
Consider a simple line fitting a * x + b = x, where a, b are the optimized parameters and x is the observed vector given by

import torch
X = torch.randn(1000,1,1)

One can immediately see that the exact solution is a=1, b=0 for any x and it can be found as easily as:

import numpy as np
np.polyfit(X.numpy().flatten(), X.numpy().flatten(), 1)

I am trying now to find this solution by means of gradient descent in PyTorch, where the mean square error is used as an optimization criterion.

import matplotlib.pyplot as plt
import numpy as np

import torch
import torch.nn as nn
from torch.optim import Adam, SGD, Adagrad, ASGD

X = torch.randn(1000,1,1) # Sample data

class SimpleNet(nn.Module): # Trivial neural network containing two weights
    def __init__(self):
        super(SimpleNet, self).__init__()
        self.f1 = nn.Linear(1,1)

    def forward(self, x):
        x = self.f1(x)
        return x

# Testing default setting of 3 basic optimizers

K = 500
net = SimpleNet()
optimizer = Adam(params=net.parameters())
Adam_losses = []
optimizer.zero_grad()   # zero the gradient buffers
for k in range(K):
    for b in range(1): # single batch
        loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
        loss.backward()
        optimizer.step()
        Adam_losses.append(float(loss.detach()))

net = SimpleNet()
optimizer = SGD(params=net.parameters(), lr=0.0001)
SGD_losses = []
optimizer.zero_grad()   # zero the gradient buffers
for k in range(K):
    for b in range(1): # single batch
        loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
        loss.backward()
        optimizer.step()
        SGD_losses.append(float(loss.detach()))

net = SimpleNet()
optimizer = Adagrad(params=net.parameters())
Adagrad_losses = []
optimizer.zero_grad()   # zero the gradient buffers
for k in range(K):
    for b in range(1): # single batch
        loss = torch.mean((net.forward(X[b,:,:]) - X[b,:, :])**2)
        loss.backward()
        optimizer.step()
        Adagrad_losses.append(float(loss.detach()))

The training progress in terms of loss evolution can be shown as

What is surprising for me is a very slow convergence of the algorithms in default setting. I have thus 2 questions:

1) Is it possible to achieve an arbitrary small error (loss) purely by means of some Pytorch optimizer? Since the loss function is convex, it should be definitely possible, however, I am not able to figure out, how to achieve this using PyTorch. Note that the above 3 optimizers cannot do that - see the loss progress in log scale for 20000 iterations:

2) I am wondering how the optimizers can work well in complex examples, when they does not work well even in this extremely simple example. Or (and that is the second question) is it something wrong in their application above that I missed?