title: PyTorch: Infer dtype from device capability, not input data

body:
I am trying to write a device-agnostic library for PyTorch, and I have stumbled across the problem of PyTorch using dtypes that are not compatible with my compute device:

import scipy.signal
import torch

raw_window = scipy.signal.windows.cosine(128)
print(raw_window.dtype)  # float64

device = torch.device("cpu")
window = torch.as_tensor(raw_window, device=device)
print(window.device)  # cpu
print(window.dtype)   # torch.float64

device = torch.device("cuda")
window = torch.as_tensor(raw_window, device=device)
print(window.device)  # cuda:0
print(window.dtype)   # torch.float64

As you can see in the last line, torch assigns the dtype torch.float64, even though my CUDA device is not able to handle double precision float values.

Is there a way to make PyTorch use the most suitable device dtype instead of the input data dtype? Or am I misunderstanding that whole concept of setting devices and dtypes completely?


title: PyTorch LSTM crashing on colab gpu (works fine on cpu)

body:
Hello I have following LSTM which runs fine on a CPU.

import torch

class LSTMForecast(torch.nn.Module):
    """
    A very simple baseline LSTM model that returns
    an output sequence given a multi-dimensional input seq. Inspired by the StackOverflow link below.
    https://stackoverflow.com/questions/56858924/multivariate-input-lstm-in-pytorch
    """
    def __init__(self, seq_length: int, n_time_series: int, output_seq_len=1, hidden_states:int=20, num_layers=2, bias=True, batch_size=100):
        super().__init__()
        self.forecast_history = seq_length
        self.n_time_series = n_time_series
        self.hidden_dim = hidden_states
        self.num_layers = num_layers
        self.lstm = torch.nn.LSTM(n_time_series, hidden_states, num_layers, bias, batch_first=True)
        self.final_layer = torch.nn.Linear(seq_length*hidden_states, output_seq_len)
        self.init_hidden(batch_size)

    def init_hidden(self, batch_size)->None:
        # This is what we'll initialise our hidden state
        self.hidden = (torch.zeros(self.num_layers, batch_size, self.hidden_dim), torch.zeros(self.num_layers, batch_size, self.hidden_dim))

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        batch_size = x.size()[0]
        self.init_hidden(batch_size)
        out_x, self.hidden = self.lstm(x, self.hidden)
        x = self.final_layer(out_x.contiguous().view(batch_size, -1))
        return x

However, when I try to run on colab GPU it crashes without even an error message.

   model = LSTMForecast(1, 1, batch_size=1).to('cuda')
   a = torch.rand(1, 1, 1).to('cuda')
   model(a)

The logs don't tell me anything either. I'm really at a loss.


title: Train n% last layers of BERT in Pytorch using HuggingFace Library (train Last 5 BERTLAYER out of 12 .)

body:
Bert has an Architecture something like encoder -> 12 BertLayer -> Pooling. I want to train the last 40% layers of Bert Model. I can freeze all the layers as:
# freeze parameters
bert = AutoModel.from_pretrained('bert-base-uncased')
for param in bert.parameters():
    param.requires_grad = False

But I want to Train last 40% layers. When I do len(list(bert.parameters())), it gives me 199. So let us suppose 79 is the 40% of parameters. Can I do something like:
for param in list(bert.parameters())[-79:]: # total  trainable 199 Params: 79 is 40%
    param.requires_grad = False

I think it will freeze first 60% layers.
Also, can someone tell me that which layers it will freeze according to architecture?


title: unable to import pytorch-lightning

body:
I installed pytorch-lightning using pip, and I'm running on Mac.
I tried:
! pip install pytorch-lightning --upgrade
! pip install pytorch-lightning-bolts

(finished successfully)
and then:
import pytorch_lightning as pl

and what I get is:
--
-------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-3-f3b4217dcea1> in <module>
      7 from torchvision.datasets import MNIST
      8 from torchvision import transforms
----> 9 import pytorch_lightning as pl
     10 from pytorch_lightning.metrics.functional import accuracy
     11 tmpdir = os.getcwd()

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/__init__.py in <module>
     60     # We are not importing the rest of the lightning during the build process, as it may not be compiled yet
     61 else:
---> 62     from pytorch_lightning import metrics
     63     from pytorch_lightning.callbacks import Callback
     64     from pytorch_lightning.core import LightningDataModule, LightningModule

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/__init__.py in <module>
     12 # See the License for the specific language governing permissions and
     13 # limitations under the License.
---> 14 from pytorch_lightning.metrics.classification import (  # noqa: F401
     15     Accuracy,
     16     AUC,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/classification/__init__.py in <module>
     12 # See the License for the specific language governing permissions and
     13 # limitations under the License.
---> 14 from pytorch_lightning.metrics.classification.accuracy import Accuracy  # noqa: F401
     15 from pytorch_lightning.metrics.classification.auc import AUC  # noqa: F401
     16 from pytorch_lightning.metrics.classification.auroc import AUROC  # noqa: F401

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/classification/accuracy.py in <module>
     16 import torch
     17
---> 18 from pytorch_lightning.metrics.functional.accuracy import _accuracy_compute, _accuracy_update
     19 from pytorch_lightning.metrics.metric import Metric
     20

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/functional/__init__.py in <module>
     12 # See the License for the specific language governing permissions and
     13 # limitations under the License.
---> 14 from pytorch_lightning.metrics.functional.accuracy import accuracy  # noqa: F401
     15 from pytorch_lightning.metrics.functional.auc import auc  # noqa: F401
     16 from pytorch_lightning.metrics.functional.auroc import auroc  # noqa: F401

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/functional/accuracy.py in <module>
     16 import torch
     17
---> 18 from pytorch_lightning.metrics.classification.helpers import _input_format_classification, DataType
     19
     20

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/classification/helpers.py in <module>
     17 import torch
     18
---> 19 from pytorch_lightning.metrics.utils import select_topk, to_onehot
     20 from pytorch_lightning.utilities import LightningEnum
     21

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/metrics/utils.py in <module>
     16 import torch
     17
---> 18 from pytorch_lightning.utilities import rank_zero_warn
     19
     20 METRIC_EPS = 1e-6

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/__init__.py in <module>
     16 import numpy
     17
---> 18 from pytorch_lightning.utilities.apply_func import move_data_to_device  # noqa: F401
     19 from pytorch_lightning.utilities.distributed import (  # noqa: F401
     20     AllGatherGrad,

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/apply_func.py in <module>
     23
     24 from pytorch_lightning.utilities.exceptions import MisconfigurationException
---> 25 from pytorch_lightning.utilities.imports import _TORCHTEXT_AVAILABLE
     26
     27 if _TORCHTEXT_AVAILABLE:

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/imports.py in <module>
     54 _TORCH_GREATER_EQUAL_1_7 = _compare_version("torch", operator.ge, "1.7.0")
     55 _TORCH_QUANTIZE_AVAILABLE = bool([eg for eg in torch.backends.quantized.supported_engines if eg != 'none'])
---> 56 _APEX_AVAILABLE = _module_available("apex.amp")
     57 _BOLTS_AVAILABLE = _module_available('pl_bolts')
     58 _DEEPSPEED_AVAILABLE = not _IS_WINDOWS and _module_available('deepspeed')

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/pytorch_lightning/utilities/imports.py in _module_available(module_path)
     32     """
     33     try:
---> 34         return find_spec(module_path) is not None
     35     except AttributeError:
     36         # Python 3.6

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/importlib/util.py in find_spec(name, package)
     92         parent_name = fullname.rpartition('.')[0]
     93         if parent_name:
---> 94             parent = __import__(parent_name, fromlist=['__path__'])
     95             try:
     96                 parent_path = parent.__path__

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/apex/__init__.py in <module>
     11                                 ISessionFactory)
     12 from pyramid.security import NO_PERMISSION_REQUIRED
---> 13 from pyramid.session import UnencryptedCookieSessionFactoryConfig
     14 from pyramid.settings import asbool
     15

ImportError: cannot import name 'UnencryptedCookieSessionFactoryConfig' from 'pyramid.session' (unknown location


title: Why unintialized tensor in Pytorch have initial values?

body:
The torch command

x = torch.Tensor(4, 3)

is supposed to create an uninitialized tensor (based on documentations).
But when we try to print the content of x, there are values there.

>>>from __future__ import print_function
>>>print(x)

 0.0000e+00 -8.5899e+09  6.1021e-38
 8.5920e+09  1.7470e-21  4.5806e-41
 0.0000e+00  0.0000e+00  0.0000e+00
 0.0000e+00  0.0000e+00  0.0000e+00
[torch.FloatTensor of size 4x3]

So what is the meaning of uninitialized here?


title: Detectron2 Panoptic FPN Model Partial Execution - TypeError: 'NoneType' object is not iterable

body:
I am trying to extract the pre-output feature-map for the Panoptic Output of Detectron2 ResNet50 - based FPN model.

Hence, In order to get partial model outputs, I am following the official Detectron2 Modeling Documentation to Partially Execute Models.

Please find the code below:

# Setting the model Configurations
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file(
    "COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml")
)
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(
    "COCO-PanopticSegmentation/panoptic_fpn_R_50_1x.yaml"
)

# Build the model from config
model = build_model(cfg)

# Loading an image just for testing
im = cv2.imread("./detectron/input.jpg")
im = torch.from_numpy(im).cuda().permute(2, 0, 1).unsqueeze(0).float()

# Extracting Features - This part works fine
features = model.backbone(im)

# Extracting the proposals - Throws Error
proposals = model.proposal_generator(im, features)

Please find the error thrown by the above step shown below:

TypeError                                 Traceback (most recent call last)
<ipython-input-17-dd53471cf2d2> in <module>
----> 1 proposals = model.proposal_generator(im, features)

~/miniconda3/envs/pytorch/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/media/Data/Documents/Python-Codes/Freelance/detectron2/detectron2/modeling/proposal_generator/rpn.py in forward(self, images, features, gt_instances)
    409
    410         if self.training:
--> 411             gt_labels, gt_boxes = self.label_and_sample_anchors(anchors, gt_instances)
    412             losses = self.losses(
    413                 anchors, pred_objectness_logits, gt_labels, pred_anchor_deltas, gt_boxes

~/miniconda3/envs/pytorch/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17

/media/Data/Documents/Python-Codes/Freelance/detectron2/detectron2/modeling/proposal_generator/rpn.py in label_and_sample_anchors(self, anchors, gt_instances)
    272         anchors = Boxes.cat(anchors)
    273
--> 274         gt_boxes = [x.gt_boxes for x in gt_instances]
    275         image_sizes = [x.image_size for x in gt_instances]
    276         del gt_instances

TypeError: 'NoneType' object is not iterable

Please let me know what am I doing wrong, and how to fix it. An explanation regarding why did this error come up would also be very helpful.

Please let me know if any other information is required.


title: How to run Tutorial: Simple LSTM on fairseq

body:
While trying to learn fairseq, I was following the tutorials on the website and implementing:
https://fairseq.readthedocs.io/en/latest/tutorial_simple_lstm.html#training-the-model
However, after following all the steps, when I try to train the model using the following:
! fairseq-train data-bin/iwslt14.tokenized.de-en \ --arch tutorial_simple_lstm \ --encoder-dropout 0.2 --decoder-dropout 0.2 \ --optimizer adam --lr 0.005 --lr-shrink 0.5 \ --max-tokens 12000
I receive an error:
`fairseq-train: error: argument --arch/-a: invalid choice: 'tutorial_simple_lstm' (choose from 'fconv', 'fconv_iwslt_de_en', 'fconv_wmt_en_ro', 'fconv_wmt_en_de', 'fconv_wmt_en_fr', 'fconv_lm', 'fconv_lm_dauphin_wikitext103', 'fconv_lm_dauphin_gbw', 'transformer', 'transformer_iwslt_de_en', 'transformer_wmt_en_de', 'transformer_vaswani_wmt_en_de_big', 'transformer_vaswani_wmt_en_fr_big', 'transformer_wmt_en_de_big', 'transformer_wmt_en_de_big_t2t', 'bart_large', 'bart_base', 'mbart_large', 'mbart_base', 'mbart_base_wmt20', 'nonautoregressive_transformer', 'nonautoregressive_transformer_wmt_en_de', 'nacrf_transformer', 'iterative_nonautoregressive_transformer', 'iterative_nonautoregressive_transformer_wmt_en_de', 'cmlm_transformer', 'cmlm_transformer_wmt_en_de', 'levenshtein_transformer', 'levenshtein_transformer_wmt_en_de', 'levenshtein_transformer_vaswani_wmt_en_de_big',....

Some additional info: I am using google colab. And I am writing the entire code until train step into .py file and uploading it to fairseq/models/... path as per my interpretation of the instructions. I am following the exact tutorial in the link.
And, before running it on colab, I am installing fairseq using:
!git clone https://github.com/pytorch/fairseq %cd fairseq !pip install --editable ./
I think this error happens because the command line argument created as per the tutorial has not been set properly.
Can anyone please explain if on any step I would need to do something else.
I would be grateful for your inputs as for a beginner learner such help from the community goes a long way.


title: RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[4, 4, 1024, 2048] to have 3 channels, but got 4 channels instead

body:
My model is something like this and designed to work with RGB images. Previously I tested with both RGB and Grayscale images and it was fine. Suddenly it is not working with a new RGB dataset.
class UNet(nn.Module):
    def __init__(self):
        super(UNet, self).__init__()
        self.inc = inconv(3, 64)
        self.down1 = down(64, 128)
        self.down2 = down(128, 256)
        self.down3 = down(256, 512)
        self.down4 = down(512, 512)
        self.up1 = up(1024, 256)
        self.up2 = up(512, 128)
        self.up3 = up(256, 64)
        self.up4 = up(128, 64)
        self.sem_out = outconv(64, 2)
        self.ins_out = outconv(64, 70)

Datasetloader :
train_dataset = DataLoaderInstanceSegmentation()
train_dataloader = DataLoader(train_dataset, batch_size=4)

Training :
for epoch in range(5):
    for batched in train_dataloader:
        images, sem_labels, ins_labels = batched
        images = Variable(images)
        sem_labels = Variable(sem_labels)
        ins_labels = Variable(ins_labels)
        model.zero_grad()
        sem_predict, ins_predict = model(images)


title: NaN error when testing Network, what could be the reasons?

body:
It contains the implementation of Recognizing Challenging Handwritten Annotations with Fully Convolutional Networks.

While training, an error occurs, NaN loss during training. What could be the reasons?

I tried changing loss function to tanh and changing hyper-parameters, but the problem still persists

import torch

from PIL import Image
from matplotlib import pyplot as plt

import os
import numpy as np
from tqdm import trange, tqdm

import torch.optim as optim
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

from torchvision import transforms, datasets
from torch.utils.data import Dataset, DataLoader

import random
import time

import math
import numbers

import torchfcn

import subprocess

def make_cv_folds(img_root, gt_root, num_folds, random_state):
    samples = []
    folds = [list() for i in range(num_folds)]
    lengths = [0]*num_folds
    img_root = os.path.expanduser(img_root)
    gt_root = os.path.expanduser(gt_root)
    for f in sorted(os.listdir(img_root)):
        if not os.path.isfile(os.path.join(img_root, f)) or not os.path.isfile(os.path.join(gt_root, f.rsplit(".", 1)[0] + ".png")):
            raise Exception("GT fehlt")
        samples.append((os.path.join(img_root, f), os.path.join(gt_root, f.rsplit(".", 1)[0] + ".png")))

    np.random.seed(random_state)
    np.random.shuffle(samples)
    np.random.seed()

    for s in samples:
        idx = np.argmin(lengths)
        folds[idx].append(s)
        lengths[idx] += 1
    return folds

def load_sample(img_path, gt_path):
    img = Image.open(img_path)
    gt = Image.open(gt_path)
    gt = np.array(gt)[:,:,2]

    #binary format
    gt[gt == 0] = 2
    gt[gt == 255] = 1

    #hisdb format
    gt[gt == 1] = 1
    gt[(gt%8) == 0] = 1
    gt[(gt%4) == 0] = 1
    gt[(gt%2) == 0] = 0
    gt = Image.fromarray(gt)
    return img, gt

class Annotations(Dataset):
    class_names = np.array(['other', 'annotation'])

    def __init__(self, img_root, gt_root, loader=load_sample, num_folds=5, preprocess=None, random_state=None):
        self.folds = make_cv_folds(img_root, gt_root, num_folds=num_folds, random_state=random_state)
        self.img_root = img_root
        self.num_folds = num_folds
        self.preprocess = preprocess
        self.loader = loader
        self.is_training = True
        self.load_split(num=0)

    def train(self, val):
        if val:
            self.is_training = True
            self.samples = self.train_samples
        else:
            self.is_training = False
            self.samples = self.test_samples

    def load_split(self, num=0):
        if len(self.folds) == 1:
            self.train_samples = self.folds[0]
            self.test_samples = self.folds[0]
        else:
            num = num%len(self.folds)
            train_folds = list(range(0,num)) + list(range(num+1, len(self.folds)))
            test_fold = num
            self.train_samples = []
            for i in train_folds:
                self.train_samples.extend(self.folds[i])
            self.test_samples = self.folds[num]
        if self.is_training:
            self.samples = self.train_samples
        else:
            self.samples = self.test_samples

    def untransform(self, img, gt):
        img = img.numpy()
        img = img.transpose(1, 2, 0)
        img = img.astype(np.uint8)
        img = img[:, :, ::-1]
        gt = gt.numpy()
        return img, gt

    def __getitem__(self, index):
        img_path, gt_path = self.samples[index]
        img, gt = self.loader(img_path, gt_path)
        if self.preprocess is not None:
            state = time.time()
            img = self.preprocess(img, random_state=state)
            gt = self.preprocess(gt, random_state=state)

        img = np.array(img, dtype=np.uint8)
        img = img[:, :, ::-1]  # RGB -> BGR
        img = img.astype(np.float64)
        img = img.transpose(2, 0, 1)
        img = torch.from_numpy(img).float()
        gt = np.array(gt, dtype=np.int32)
        gt = torch.from_numpy(gt).long()
        return img, gt

    def __len__(self):
        return len(self.samples)

    def __repr__(self):
        fmt_str = 'Dataset ' + self.__class__.__name__ + '\n'
        fmt_str += '    Number of datapoints: {}\n'.format(self.__len__())
        fmt_str += '    Number of training samples: {}\n'.format(len(self.train_samples))
        fmt_str += '    Number of testing samples: {}\n'.format(len(self.test_samples))
        return fmt_str

class CenterCrop(object):
    """Crops the given PIL Image at the center.
    Args:
        size (sequence or int): Desired output size of the crop. If size is an
            int instead of sequence like (h, w), a square crop (size, size) is
            made.
    """

    def __init__(self, size):
        if isinstance(size, numbers.Number):
            self.size = (int(size), int(size))
        else:
            self.size = size

    def __call__(self, img, random_state=None):
        """
        Args:
            img (PIL Image): Image to be cropped.
        Returns:
            PIL Image: Cropped image.
        """
        return transforms.functional.center_crop(img, self.size)

    def __repr__(self):
        return self.__class__.__name__ + '(size={0})'.format(self.size)

class RandomResizedCrop(object):
    """Crop the given PIL Image to random size and aspect ratio.
    A crop of random size (default: of 0.08 to 1.0) of the original size and a random
    aspect ratio (default: of 3/4 to 4/3) of the original aspect ratio is made. This crop
    is finally resized to given size.
    This is popularly used to train the Inception networks.
    Args:
        size: expected output size of each edge
        scale: range of size of the origin size cropped
        ratio: range of aspect ratio of the origin aspect ratio cropped
        interpolation: Default: PIL.Image.BILINEAR
    """

    def __init__(self, size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolation=Image.NEAREST):
        self.size = (size, size)
        self.interpolation = interpolation
        self.scale = scale
        self.ratio = ratio

    @staticmethod
    def get_params(img, scale, ratio, random_state=None):
        """Get parameters for ``crop`` for a random sized crop.
        Args:
            img (PIL Image): Image to be cropped.
            scale (tuple): range of size of the origin size cropped
            ratio (tuple): range of aspect ratio of the origin aspect ratio cropped
        Returns:
            tuple: params (i, j, h, w) to be passed to ``crop`` for a random
                sized crop.
        """

        random.seed(random_state)
        for attempt in range(10):
            area = img.size[0] * img.size[1]
            target_area = random.uniform(*scale) * area
            aspect_ratio = random.uniform(*ratio)

            w = int(round(math.sqrt(target_area * aspect_ratio)))
            h = int(round(math.sqrt(target_area / aspect_ratio)))

            if random.random() < 0.5:
                w, h = h, w

            if w <= img.size[0] and h <= img.size[1]:
                i = random.randint(0, img.size[1] - h)
                j = random.randint(0, img.size[0] - w)
                return i, j, h, w

        # Fallback
        w = min(img.size[0], img.size[1])
        i = (img.size[1] - w) // 2
        j = (img.size[0] - w) // 2
        return i, j, w, w

    def __call__(self, img, random_state=None):
        """
        Args:
            img (PIL Image): Image to be cropped and resized.
        Returns:
            PIL Image: Randomly cropped and resized image.
        """
        i, j, h, w = self.get_params(img, self.scale, self.ratio, random_state=random_state)
        return transforms.functional.resized_crop(img, i, j, h, w, self.size, self.interpolation)

    def __repr__(self):
        interpolate_str = _pil_interpolation_to_str[self.interpolation]
        format_string = self.__class__.__name__ + '(size={0}'.format(self.size)
        format_string += ', scale={0}'.format(round(self.scale, 4))
        format_string += ', ratio={0}'.format(round(self.ratio, 4))
        format_string += ', interpolation={0})'.format(interpolate_str)
        return format_string

class RandomCrop(object):
    """Crop the given PIL Image at a random location.
    Args:
        size (sequence or int): Desired output size of the crop. If size is an
            int instead of sequence like (h, w), a square crop (size, size) is
            made.
        padding (int or sequence, optional): Optional padding on each border
            of the image. Default is 0, i.e no padding. If a sequence of length
            4 is provided, it is used to pad left, top, right, bottom borders
            respectively.
    """

    def __init__(self, size, padding=0):
        if isinstance(size, numbers.Number):
            self.size = (int(size), int(size))
        else:
            self.size = size
        self.padding = padding

    @staticmethod
    def get_params(img, output_size, random_state=None):
        """Get parameters for ``crop`` for a random crop.
        Args:
            img (PIL Image): Image to be cropped.
            output_size (tuple): Expected output size of the crop.
        Returns:
            tuple: params (i, j, h, w) to be passed to ``crop`` for random crop.
        """
        random.seed(random_state)

        w, h = img.size
        th, tw = output_size
        if w == tw and h == th:
            return 0, 0, h, w

        i = random.randint(0, h - th)
        j = random.randint(0, w - tw)
        return i, j, th, tw

    def __call__(self, img, random_state=None):
        """
        Args:
            img (PIL Image): Image to be cropped.
        Returns:
            PIL Image: Cropped image.
        """
        if self.padding > 0:
            img = F.pad(img, self.padding)

        i, j, h, w = self.get_params(img, self.size, random_state=random_state)

        return transforms.functional.crop(img, i, j, h, w)

    def __repr__(self):
        return self.__class__.__name__ + '(size={0}, padding={1})'.format(self.size, self.padding)

preprocess_train = RandomResizedCrop(size=1024)
preprocess_test = RandomResizedCrop(size=1024)

trainset = Annotations(img_root='public_shared/input/',
                      gt_root='public_shared/new_bin',
                      preprocess=preprocess_train,
                      num_folds=1)

testset = Annotations(img_root='test/input/',
                      gt_root='test/new_bin/',
                      preprocess=preprocess_test,
                      num_folds=1)

testset.train(False)

train_loader = DataLoader(trainset, batch_size=1, shuffle=True, num_workers=8, drop_last=True)
test_loader = DataLoader(testset, batch_size=1, shuffle=False, num_workers=8, drop_last=True)

dat = trainset[0]
img = dat[0].numpy().transpose(1,2,0)
gt = dat[1].numpy()

print(img.shape)
print(gt.shape)

plt.imshow(img.squeeze())
plt.show()
plt.imshow(gt.squeeze())
plt.show()

log_dir = "log-mg3/"

def get_parameters(model, bias=False):
    import torch.nn as nn
    modules_skipped = (
        nn.ReLU,
        nn.MaxPool2d,
        nn.Dropout2d,
        nn.Sequential,
        torchfcn.models.FCN8s,
    )
    for m in model.modules():
        if isinstance(m, nn.Conv2d):
            if bias:
                yield m.bias
            else:
                yield m.weight
        elif isinstance(m, nn.ConvTranspose2d):
            # weight is frozen because it is just a bilinear upsampling
            if bias:
                assert m.bias is None
        elif isinstance(m, modules_skipped):
            continue
        else:
            raise ValueError('Unexpected module: %s' % str(m))

configurations = {
    # same configuration as original work
    # https://github.com/shelhamer/fcn.berkeleyvision.org
    1: dict(
        max_iteration=200000,
        lr=1.0e-10,
        momentum=0.99,
        weight_decay=0.0005,
        interval_validate=4000,
    )
}

cfg = configurations[1]
out = log_dir

cuda = torch.cuda.is_available()

torch.manual_seed(1337)
if cuda:
    torch.cuda.manual_seed(1337)

resume = ""

model = torchfcn.models.FCN8sAtOnce(n_class=2)
start_epoch = 0
start_iteration = 0
if resume:
    checkpoint = torch.load(resume)
    model.load_state_dict(checkpoint['model_state_dict'])
    start_epoch = checkpoint['epoch']
    start_iteration = checkpoint['iteration']
else:
    vgg16 = torchfcn.models.VGG16(pretrained=True)
    model.copy_params_from_vgg16(vgg16)
if cuda:
    model = model.cuda()

optimizer = torch.optim.SGD(
    [
        {'params': get_parameters(model, bias=False)},
        {'params': get_parameters(model, bias=True),
         'lr': cfg['lr'] * 2, 'weight_decay': 0},
    ],
    lr=cfg['lr'],
    momentum=cfg['momentum'],
    weight_decay=cfg['weight_decay'])
if resume:
    optimizer.load_state_dict(checkpoint['optim_state_dict'])

trainer = torchfcn.Trainer(
    cuda=cuda,
    model=model,
    optimizer=optimizer,
    train_loader=train_loader,
    val_loader=test_loader,
    out=out,
    max_iter=cfg['max_iteration'],
    interval_validate=cfg.get('interval_validate', len(train_loader)),
)
trainer.epoch = start_epoch
trainer.iteration = start_iteration
# trainer.train()

def evaluate_model(model, data_loader):
    model.eval()

    processes = []
    mius = []

    for index in tqdm(range(len(data_loader.dataset))):
        _, gt_path = data_loader.dataset.samples[index]
        image, _ = data_loader.dataset[index]
        image = image.numpy()
        image.shape = (1, image.shape[0], image.shape[1], image.shape[2])
        prediction = np.zeros((image.shape[2], image.shape[3], 3), dtype=np.uint8)
        div_arr = np.zeros((image.shape[2], image.shape[3]), dtype=np.uint8)

        offsets_vertical = list(range(0, image.shape[2], 256))
        offsets_horizontal = list(range(0, image.shape[3], 256))

        for v in offsets_vertical:
            for h in offsets_horizontal:
                data = image[:, :, v:v+1024, h:h+1024]
                data = torch.from_numpy(data)
                data = data.cuda()
                data = Variable(data, volatile=True)
                score = model(data)
                lbl_pred = score.data.max(1)[1].cpu().numpy()[:, :, :]
                lbl_pred[lbl_pred == 0] = 2
                prediction[v:v+1024, h:h+1024, 2] += lbl_pred.astype(np.uint8).squeeze()
                div_arr[v:v+1024, h:h+1024] += 1

        prediction[:,:,2] = np.round(prediction[:,:,2]/div_arr)

        im = Image.fromarray(prediction)
        prediction_path = os.path.join(log_dir, "prediction-private")
        if not os.path.isdir(prediction_path):
            os.makedirs(prediction_path)
        prediction_filename = os.path.join(prediction_path, os.path.basename(gt_path))
        im.save(prediction_filename)

        processes.append(subprocess.Popen(["java", "-jar", "DIVA_Layout_Analysis_Evaluator/out/artifacts/LayoutAnalysisEvaluator.jar", "-p", prediction_filename, "-gt", gt_path], stdout=subprocess.PIPE))

    for p in processes:
        miu = float(p.communicate()[0].splitlines()[0].split()[-1])
        mius.append(miu)

    print(mius)
    print("average:", np.mean(mius))
    return np.mean(mius)

testset = Annotations(img_root='test/input/',
                      gt_root='test/new_gt/',
                      preprocess=None,
                      num_folds=1)

testset.train(False)

test_loader = DataLoader(testset, batch_size=1, shuffle=False, num_workers=8, drop_last=True)

evaluate_model(model, test_loader)

Error:

0%|          | 0/10 [00:00<?, ?it/s]/home/harsh/anaconda3/envs/vdlproject/lib/python3.6/site-packages/ipykernel_launcher.py:23: UserWarning: volatile was removed and now has no effect. Use `with torch.no_grad():` instead.
100%|██████████| 10/10 [00:13<00:00,  1.28s/it]
[]
average: nan

/home/harsh/anaconda3/envs/vdlproject/lib/python3.6/site-packages/numpy/core/fromnumeric.py:2920: RuntimeWarning: Mean of empty slice.
  out=out, **kwargs)
/home/harsh/anaconda3/envs/vdlproject/lib/python3.6/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)

Output:  average: nan

Can anyone explain cause of error?


title: Invalid argument tensors for DataLoader PyTorch

body:
I have made a dataset using PyTorch's DataLoader and Imagefolder, my dataset class has two ImageFolder datasets. These two datasets are paired (original and ground truth image). I want to feed these to a PyTorch neural network.

Dataset class:

class bsds_dataset(Dataset):
    def __init__(self, ds_main, ds_energy):
        self.dataset1 = ds_main
        self.dataset2 = ds_energy

    def __getitem__(self, index):
        x1 = self.dataset1[index]
        x2 = self.dataset2[index]

        return x1, x2

    def __len__(self):
        return len(self.dataset1)

And I'm loading images with Imagefolder:

original_imagefolder = './images/whole'
target_imagefolder = './results/whole'

original_ds = ImageFolder(original_imagefolder,
transform=transforms.ToTensor())
energy_ds = ImageFolder(target_imagefolder, transform=transforms.ToTensor())

dataset = bsds_dataset(original_ds, energy_ds)
loader = DataLoader(dataset, batch_size=16)

Then I tried to iterate in batches:

for i, x, y in enumerate(loader):
    print(x)

This error happened:

  RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 321 and 481 in dimension 2 at ..\aten\src\TH/generic/THTensor.cpp:711

The dataset is BSDS500:

https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html

All images in the dataset are in 481x321 or 321x481 pixels.
I think some transform is needed but I don't want to demolish the images and stretch them.

Full traceback:

C:\Anaconda3\envs\torchgpu\lib\site-packages\ipykernel_launcher.py:77: UserWarning: nn.init.xavier_normal is now deprecated in favor of nn.init.xavier_normal_.
C:\Anaconda3\envs\torchgpu\lib\site-packages\ipykernel_launcher.py:78: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-42-4c4ba0a13c32> in <module>
      5 optimizer = optim.SGD(model.parameters(), lr=0.001)
      6 for epoch in range(epochs):
----> 7     for i, batch in enumerate(loader):
      8         print(batch)

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    558         if self.num_workers == 0:  # same-process loading
    559             indices = next(self.sample_iter)  # may raise StopIteration
--> 560             batch = self.collate_fn([self.dataset[i] for i in indices])
    561             if self.pin_memory:
    562                 batch = _utils.pin_memory.pin_memory_batch(batch)

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     66     elif isinstance(batch[0], container_abcs.Sequence):
     67         transposed = zip(*batch)
---> 68         return [default_collate(samples) for samples in transposed]
     69
     70     raise TypeError((error_msg_fmt.format(type(batch[0]))))

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\utils\data\_utils\collate.py in <listcomp>(.0)
     66     elif isinstance(batch[0], container_abcs.Sequence):
     67         transposed = zip(*batch)
---> 68         return [default_collate(samples) for samples in transposed]
     69
     70     raise TypeError((error_msg_fmt.format(type(batch[0]))))

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     66     elif isinstance(batch[0], container_abcs.Sequence):
     67         transposed = zip(*batch)
---> 68         return [default_collate(samples) for samples in transposed]
     69
     70     raise TypeError((error_msg_fmt.format(type(batch[0]))))

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\utils\data\_utils\collate.py in <listcomp>(.0)
     66     elif isinstance(batch[0], container_abcs.Sequence):
     67         transposed = zip(*batch)
---> 68         return [default_collate(samples) for samples in transposed]
     69
     70     raise TypeError((error_msg_fmt.format(type(batch[0]))))

C:\Anaconda3\envs\torchgpu\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     41             storage = batch[0].storage()._new_shared(numel)
     42             out = batch[0].new(storage)
---> 43         return torch.stack(batch, 0, out=out)
     44     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
     45             and elem_type.__name__ != 'string_':

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 321 and 481 in dimension 2 at ..\aten\src\TH/generic/THTensor.cpp:711


title: Torch.cuda.empty_cache() replacement in case of CPU only enviroment

body:
Currently, I am using PyTorch built with CPU only support. When I run inference, somehow information for that input file is stored in cache and memory keeps on increasing for every new unique file used for inference. On the other hand, memory usage does not increase if i use the same file again and again.

Is there a way to clear cache like cuda.empty_cache() in case of CPUs only.


title: pytorch debugging timeout with PyCharm

body:
I have a frustrating problem, where I cannot debug my pytorch code while in Pycharm.
While trying to inspect (breakpoint, then print e.g.) the code below, I receive a "Loading time out"

import torch

    tensors = []
    num_tensors = 16
    shape = (1, 3, 512, 512)
    for i in range(num_tensors):
        tensors.append(torch.zeros(shape))

I saw this[1,2] posts, set variable loading policy to syncronyous, disabled Qt debugger options, and all the options specified, But I believe there is something basic I'm missing.
pycharm 2019.2.5, happens both in python2 and python3.


title: How to concurrently run multiple branches in pytorch?

body:
I was trying to build a network with multiple branches in pytorch. But how can I run multiple branches in parallel instead of run them one by one?

Not like tensorflow or keras, pytorch use dynamic graph, so I can't define concurrent processing beforehand.

I looked up for some similar official implement of pytorch network like InceptionNet, only to find out pytorch runs consecutively with multiple branches.

from inception.py

def _forward(self, x):
    branch1x1 = self.branch1x1(x)

    branch5x5 = self.branch5x5_1(x)
    branch5x5 = self.branch5x5_2(branch5x5)

    branch3x3dbl = self.branch3x3dbl_1(x)
    branch3x3dbl = self.branch3x3dbl_2(branch3x3dbl)
    branch3x3dbl = self.branch3x3dbl_3(branch3x3dbl)

    branch_pool = F.avg_pool2d(x, kernel_size=3, stride=1, padding=1)
    branch_pool = self.branch_pool(branch_pool)

    outputs = [branch1x1, branch5x5, branch3x3dbl, branch_pool]
    return outputs

Four branches run one by one, first branch1x1, then branch5x5, and branch3x3dbl, branch_pool. Then outputs stores their results and they will be concatenated later.

Would't it be a waste of performance? And how can we deal with that?


title: Loss Function in Multi-GPUs training (PyTorch)

body:
I use Pytorch and BERT to traing a model. Everithing works great on one GPU, but when I try to use multi GPUs I am getting an error:

ValueError                                Traceback (most recent call last)
<ipython-input-168-507223f9879c> in <module>()
     92         # single value; the `.item()` function just returns the Python value
     93         # from the tensor.
---> 94         total_loss += loss.item()
     95
     96         # Perform a backward pass to calculate the gradients.

ValueError: only one element tensors can be converted to Python scalars

Can someone help me what I am missing and how I should fix it?

Here is my code for training:

import random
seed_val = 42
random.seed(seed_val)
np.random.seed(seed_val)
torch.manual_seed(seed_val)
torch.cuda.manual_seed_all(seed_val)
loss_values = []
for epoch_i in range(0, epochs):

    t0 = time.time()
    total_loss = 0
    for step, batch in enumerate(train_dataloader):
        if step % 40 == 0 and not step == 0:
            elapsed = format_time(time.time() - t0)

        b_input_ids = batch[0].to(device).long()
        b_input_mask = batch[1].to(device).long()
        b_labels = batch[2].to(device).long()
        model.zero_grad()
        outputs = model(b_input_ids,
                    token_type_ids=None,
                    attention_mask=b_input_mask,
                    labels=b_labels)

        loss = outputs[0]
        total_loss += loss.item()
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        scheduler.step()
    avg_train_loss = total_loss / len(train_dataloader)
    loss_values.append(avg_train_loss)
    print("")
    print("  Average training loss: {0:.2f}".format(avg_train_loss))
    print("  Training epcoh took: {:}".format(format_time(time.time() - t0)))

And here is my code for the model:

from transformers import BertForSequenceClassification, AdamW, BertConfig
model_to_parallel = BertForSequenceClassification.from_pretrained(
    "./bert_cache.zip",
    num_labels = 2,
    output_attentions = False,
    output_hidden_states = False,
)
model = nn.DataParallel(model_to_parallel,  device_ids=[0,1,2,3])
model.to(device)


title: PyTorch augmentation

body:
i'm new to machine learning and pytorch. I'm using imgaug library for images augmentation (https://github.com/aleju/imgaug)

I have this code:

class ImgAugTransform:
    def __init__(self):
        self.aug = seq = iaa.Sequential(
            [
                # Apply the following augmenters to most images
                iaa.Fliplr(0.5), # horizontally flip 50% of all images
                iaa.Flipud(0.2),  # vertically flip 20% of all images
                random_aug_use(iaa.CropAndPad( # crop images by -5% to 10% of their height/width
                    percent=(-0.1, 0.2),
                    pad_mode=ia.ALL,
                    pad_cval=(0.,255)
                )),
                random_aug_use(iaa.Affine(
                    scale={"x": (0.8, 1.2), "y": (0.8, 1.2)}, # scale images to 80-120% of their size, individually per axis
                    translate_percent={"x": (-0.2, 0.2), "y": (-0.2, 0.2)}, # translate by -20 to +20 percent (per axis)
                    rotate=(-45, 45), # rotate by -45 to +45 degrees
                    shear=(-16, 16), # shear by -16 to +16 degrees
                    order=[0, 1], # use nearest neighbour or bilinear interpolation (fast)
                    cval=(0, 255), # if mode is constant, use a cval between 0 and 255
                    mode=ia.ALL # use any of scikit-image's warping modes (see 2nd image from the top for examples)
                ))
            ],
            random_order=True)

    def __call__(self, img):
        img = np.array(img)
        return self.aug.augment_image(img)

train_transforms = ImgAugTransform()

train_dataset = torchvision.datasets.ImageFolder(train_dir, train_transforms)

train_dataloader = torch.utils.data.DataLoader(
    train_dataset, batch_size=batch_size, shuffle=True, num_workers=batch_size)

So now i cant do this:

X_batch, y_batch = next(iter(train_dataloader))

I get error:

ValueError: some of the strides of a given numpy array are negative. This is currently not supported, but will be added in future releases.

Please, help me to deal with it. I just dont understand what i should do


title: Memory Issue while following LM tutorial

body:
SPECS:
OS: Windows 10
CUDA: 10.1
GPU: RTX 2060 6G VRAM (x2)
RAM: 32GB
tutorial: https://huggingface.co/blog/how-to-train
Hello I am trying to train my own language model and I have had some memory issues. I have tried to run some of this code in Pycharm on my computer and then trying to replicate in my Collab Pro Notebook.
First, my code
from transformers import RobertaConfig, RobertaTokenizerFast, RobertaForMaskedLM, LineByLineTextDataset
from transformers import DataCollatorForLanguageModeling, Trainer, TrainingArguments

config = RobertaConfig(vocab_size=60000, max_position_embeddings=514, num_attention_heads=12, num_hidden_layers=6,
                       type_vocab_size=1)

tokenizer = RobertaTokenizerFast.from_pretrained("./MODEL DIRECTORY", max_len=512)

model = RobertaForMaskedLM(config=config)

print("making dataset")

dataset = LineByLineTextDataset(tokenizer=tokenizer, file_path="./total_text.txt", block_size=128)

print("making c")

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)

training_args = TrainingArguments(output_dir="./MODEL DIRECTORY", overwrite_output_dir=True, num_train_epochs=1,
                                  per_gpu_train_batch_size=64, save_steps=10000, save_total_limit=2)
print("Building trainer")
trainer = Trainer(model=model, args=training_args, data_collator=data_collator, train_dataset=dataset,
                  prediction_loss_only=True)
trainer.train()

trainer.save_model("./MODEL DIRECTORY")

"./total_text.txt" being a 1.7GB text file.
PyCharm Attempt
This code on pycharm builds the dataset and then would throw an error saying that my preferred gpu was running out of memory, and that Torch was already using 3.7GiB of memory.
I tried:

import gc doing a gc clear to try to flush what ever was going on my gpu
Decreasing my batch size for my gpu (training only happened on a batch size of 8 resulting in 200,000+ epochs that all took 1.17 seconds)
Setting my os.environ["CUDA_VISIBLE_OBJECTS"] ="" so that torch would have to use my CPU and not my GPU. Still threw same gpu memory error...

So succumbing to the fact that torch, for the time being, was forcing itself to use my gpu, I decided to go to Collab.
Collab Attempt
Collab has different issues with my code. It does not have the memory to build the dataset, and crashes due to RAM shortages. I purchased a Pro account and then increased the usable RAM to 25GB, still memory shortages.
Cheers!


title: How to load CUB-200-2011 dataset in pytorch?

body: I am trying to do fine-grained image classification for CUB-200-2011 dataset, couldn't figure out the correct way for loading the data. I'm using pre-trained ImageNet ResNet-101 for training by freezing all layers except last two layers. Suggest a way to for data loading in Pytorch.


title: Output of DecoderRNN contains extra dimensions (Pytorch)

body:
I have developed an Encoder(CNN)-Decoder (RNN) network for image captioning in pytorch.  The decoder network takes in two inputs- Context feature vector from the Encoder and the word embeddings of the caption for training. The context feature vector is of size = embed_size , which is also the embedding size of each word in the caption. My question here is more concerned with the output of the Class DecoderRNN. Please refer to the code below.

class DecoderRNN(nn.Module):
def __init__(self, embed_size, hidden_size, vocab_size, num_layers=1):
    super(DecoderRNN, self).__init__()
    self.embed_size = embed_size
    self.hidden_size = hidden_size
    self.vocab_size = vocab_size
    self.num_layers = num_layers
    self.linear = nn.Linear(hidden_size, vocab_size)
    self.embed = nn.Embedding(vocab_size, embed_size)
    self.lstm = nn.LSTM(embed_size, hidden_size, num_layers, batch_first = True)

def forward(self, features, captions):
    embeddings = self.embed(captions)
    embeddings = torch.cat((features.unsqueeze(1), embeddings),1)
    hiddens,_ = self.lstm(embeddings)
    outputs = self.linear(hiddens)
    return outputs

In the forward function , I send in a sequence of (batch_size, caption_length+1, embed_size) (concatenated tensor of context feature vector and the embedded caption) . The output of the sequence should be captions and  of the shape (batch_size, caption_length, vocab_size) , but I am still receiving an output of shape (batch_size, caption_length +1, vocab_size) .  Can anyone please suggest what should I alter in my forward function so that extra 2nd dimension is not received? Thanks in advance


title: Transformers summarization with Python Pytorch - how to get longer output?

body:
I use Ai-powered summarization from https://github.com/huggingface/transformers/tree/master/examples/summarization - state of the art results.

Should i train it myself to get summary output longer than used in original huggingface github training script?
:

python run_summarization.py \
    --documents_dir $DATA_PATH \
    --summaries_output_dir $SUMMARIES_PATH \ # optional
    --no_cuda false \
    --batch_size 4 \
    --min_length 50 \
    --max_length 200 \
    --beam_size 5 \
    --alpha 0.95 \
    --block_trigram true \
    --compute_rouge true

When i do inference with

--min_length 500 \
--max_length 600 \

I got a good output for 200 tokens, but the rest of the text is

. . . [unused7] [unused7] [unused7] [unused8] [unused4] [unused7] [unused7]  [unused4] [unused7] [unused8]. [unused4] [unused7] . [unused4] [unused8] [unused4] [unused8].  [unused4]  [unused4] [unused8]  [unused4] . .  [unused4] [unused6] [unused4] [unused7] [unused6] [unused4] [unused8] [unused5] [unused4] [unused7] [unused4] [unused4] [unused7]. [unused4] [unused6]. [unused4] [unused4] [unused4] [unused8]  [unused4] [unused7]  [unused4] [unused8] [unused6] [unused4]   [unused4] [unused4]. [unused4].  [unused5] [unused4] [unused8] [unused7] [unused4] [unused7] [unused9] [unused4] [unused7]  [unused4] [unused7] [unused5] [unused4]  [unused5] [unused4] [unused6]  [unused4]. .  . [unused5]. [unused4]  [unused4]   [unused4] [unused6] [unused5] [unused4] [unused4]  [unused6] [unused4] [unused6]  [unused4] [unused4] [unused5] [unused4]. [unused5]  [unused4] . [unused4]  [unused4] [unused8] [unused8] [unused4]  [unused7] [unused4] [unused8]  [unused4] [unused7]  [unused4] [unused8]  [unused4]  [unused8] [unused4] [unused6]


title: Calculating negative ELBO

body:
I am going through the tutorial of deep Markov models where they are trying to learn the polyphonic dataset. The Link to tutorial is:

https://pyro.ai/examples/dmm.html

This model parameterises transitions and emissions with using a neural network and for variational inference part they use RNN to map observable 'x' to latent space. And in order to ensure that their model is learning something they try to maximise ELBO or minimise negative ELBO. They refer to negative ELBO as NLL. So far I understand what they are doing. However, the next step confuses me. Once they have their NLL, they divide it by sum of sequence lengths.

times = [time.time()]
for epoch in range(args.num_epochs):
    # accumulator for our estimate of the negative log likelihood
    # (or rather -elbo) for this epoch
    epoch_nll = 0.0
    # prepare mini-batch subsampling indices for this epoch
    shuffled_indices = np.arange(N_train_data)
    np.random.shuffle(shuffled_indices)

    # process each mini-batch; this is where we take gradient steps
    for which_mini_batch in range(N_mini_batches):
        epoch_nll += process_minibatch(epoch, which_mini_batch, shuffled_indices)

    # report training diagnostics
    times.append(time.time())
    epoch_time = times[-1] - times[-2]
    log("[training epoch %04d]  %.4f \t\t\t\t(dt = %.3f sec)" %
        (epoch, epoch_nll / N_train_time_slices, epoch_time))

And I don't quite understand why they are doing that. Can some explain? Are they averaging here? Insights would be appreciated.


title: Pytorch: Calculating running time on GPU and CPU of a for loop

body:
I am really new to pytorch. And I got really confused the whole day while I was trying out to figure out why my nn runs slower on GPU than CPU. I do not understand when I calculated the running time using time.time(), the time of the whole loop is a lot different with the sum of every single running time. Here is part of my code. Could anybody help me? Appreciate it!
    time_out = 0
    time_in = 0

    for epoch in tqdm(range(self.n_epoch)):

        running_loss = 0
        running_error = 0
        running_acc = 0

        if self.cuda:
            torch.cuda.synchronize()                #time_out_start
        epst1 = time.time()

        for step, (batch_x, batch_y) in enumerate(self.normal_loader):

            if self.cuda:
                torch.cuda.synchronize()                        #time_in_start
            t1 = time.time()

            batch_x, batch_y = batch_x.to(self.device), batch_y.to(self.device)

            b_x = Variable(batch_x)
            b_y = Variable(batch_y)

            pred_y = self.model(b_x)
            #print (pred_y)

            loss = self.criterion(pred_y, b_y)

            error = mae(pred_y.detach().cpu().numpy(),b_y.detach().cpu().numpy())
            acc = r2(b_y.detach().cpu().numpy(),pred_y.detach().cpu().numpy())

            #print (loss)
            self.optimizer.zero_grad()
            loss.backward()
            self.optimizer.step()

            running_acc += acc
            running_loss += loss.item()
            running_error += error

            if self.cuda:
                torch.cuda.synchronize()                        #time_in_end
            t6 = time.time()

            time_in += t6-t1

        if self.cuda:
            torch.cuda.synchronize()                    #time_out_end
        eped1 = time.time()

        time_out += eped1-epst1

    print ('loop time(out)',time_out)
    print ('loop time(in)',time_in)

The result is:
CPU:
EPOCH 10: out: 1.283s in: 0.695s
EPOCH 50: out: 6.43s  in: 3.288s
EPOCH 100: out:12.646s  in:6.386s
GPU:
EPOCH 10: out: 3.92s  in: 1.471s
EPOCH 50: out: 9.35s  in:3.04s
EPOCH 100: out: 18.418s  in:5.655
I understand that transferring data from cpu to gpu cost some time. So as the epochs go up, the calculation time of GPU should become less than CPU time. My question is:

why the time I record outside of the loop is so different from the inside one? Is there any step that I missed to record the running time?
And why GPU costs more outside-time even the inside-time has been less than the CPU time?

The Network is really simple, which is:
class Model(nn.Module):
def __init__(self,n_input,n_nodes1,n_nodes2):
    super(Model, self).__init__()

    self.n_input = n_input
    self.n_nodes1 = n_nodes1
    self.n_nodes2 = n_nodes2

    self.l1 = nn.Linear(self.n_input, self.n_nodes1)
    self.l2 = nn.Linear(self.n_nodes1, self.n_nodes2)
    self.l3 = nn.Linear(self.n_nodes2, 1)

def forward(self,x):

    h1 = F.relu(self.l1(x))
    h2 = F.relu(self.l2(h1))
    h = self.l3(h2)

    return h

the training data is formed as:(regression problem, input_x are descriptors and y is the target value)
def load_train_normal(self,x,y,batch_size = 100):
    if batch_size:
        self.batch_size = batch_size

    self.x_train_n, self.y_train_n = Variable(torch.from_numpy(x).float()), Variable(torch.from_numpy(y).float())

    #x, y = Variable(torch.from_numpy(x).float()), Variable(torch.from_numpy(y).float())
    self.dataset = Data.TensorDataset(self.x_train_n,self.y_train_n)
    self.normal_loader = Data.DataLoader(
                        dataset = self.dataset,
                        batch_size = self.batch_size,
                        shuffle = True, num_workers=2,)


title: Pytorch transforms.Compose usage for pair of images in segmentation tasks

body:
I'm trying to use the transforms.Compose() in my segmentation task. But I'm not sure how to use the same (almost) random transforms for both the image and the mask.
So in my segmentation task, I have the raw picture and the corresponding mask, I'd like to generate more random transformed image pairs for training popurse. Meaning if I do some transform on my raw pictures, and this transformation should also happen on my mask pictures, and then this pair can go into my CNN. My transformer is something like:
train_transform = transforms.Compose([
            transforms.Resize(512), # resize, the smaller edge will be matched.
            transforms.RandomHorizontalFlip(p=0.5),
            transforms.RandomVerticalFlip(p=0.5),
            transforms.RandomRotation(90),
            transforms.RandomResizedCrop(320,scale=(0.3, 1.0)),
            AddGaussianNoise(0., 1.),
            transforms.ToTensor(), # convert a PIL image or ndarray to tensor.
            transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)) # normalize to Imagenet mean and std
])

mask_transform = transforms.Compose([
            transforms.Resize(512), # resize, the smaller edge will be matched.
            transforms.RandomHorizontalFlip(p=0.5),
            transforms.RandomVerticalFlip(p=0.5),
            transforms.RandomRotation(90),
            transforms.RandomResizedCrop(320,scale=(0.3, 1.0)),
            ##---------------------!------------------
            transforms.ToTensor(), # convert a PIL image or ndarray to tensor.
            transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)) # normalize to Imagenet mean and std
])

Notice, in the code block, I added a class that can add random noise to the raw images transformation, which is not in the mask_transformation, that I want my mask images follow the raw image transformation, but ignore the random noise. So how can these two transformations happen in pairs (with the same random act)?


title: Pytorch 1D tensors expected but got 2D tensors

body:
I have been working on making neural network from scratch in python. The input tensor is of shape [400,3] and target_tensor has the shape [400]. I am getting error during taking derivative of weights. Below are functions
    def sigmoid(z):
           return 1 / (1 + torch.exp(-z))

    def nueral_net(data,weights,bias):
           return sigmoid( ( data @ weights ) + bias )

    def loss_function(prediction,actual,m):
           return (-1/m) * (torch.sum(actual * torch.log(prediction) + (1-actual)
           * torch.log(1- prediction)))

    w = torch.randn(input_tensor.shape[1],1)
    b = torch.randn(1,1)

    predictions = nueral_net(input_tensor.float() , w, b) #Applying model
    loss = loss_function(predictions,target_tensor.unsqueeze(1),400)
    dw = (1/400) * torch.dot(input_tensor,(predictions - target_tensor).T)

Running this throws an error.
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-26-632338d8fd16> in <module>
          1 predictions = nueral_net(input_tensor.float() , w, b) #Applying model
          2 loss = loss_function(predictions,target_tensor.unsqueeze(1),400)
    ----> 3 dw = (1/400) * torch.dot(input_tensor,(predictions - target_tensor).T)
          4 db = (1/400) * torch.sum(predictions - target_tensor)
          5 #m = input_tensor.shape[0]

    RuntimeError: 1D tensors expected, but got 2D and 2D tensor


title: Python - Multiprocessing getting stuck

body:
I am trying to perform certain operations on the MNIST dataset in parallel. With the multiprocessing, it gets stuck and never ends. I tried putting the close and join functions inside or outside the for loop. I also tried putting the pool instantiation inside the function, doesn't seem to work either. What am I doing wrong here? No idea where should I be looking to solve this. Thank you very much for your help in advance

class MnistPartialDataset(Dataset):
    def __init__(self, data, transform=None):
        self.data = data
    def __getitem__(self, index):
    return self.data[index]
    def __len__(self):
        return len(self.data)

def create_tasks(train_pool, test_pool):
    train_thread_result = []
    start = time.time()
    for k, v in train_tasks.items():
        tc_data = train_pool.map_async(test_function, v)
        train_thread_result.append(tc_data)
    train_pool.close()
    train_pool.join()
    for i in range(len(train_thread_result)):
        train_dataset[i] = MnistPartialDataset(train_thread_result[i].get())
    end = time.time()
    print("tc time: " + str(end-start))

    test_thread_result = []
    start = time.time()
    for k, v in test_tasks.items():
        tc_data = test_pool.map_async(test_function, v)
        test_thread_result.append(tc_data)
    test_pool.close()
    test_pool.join()
    for i in range(len(test_thread_result)):
        test_dataset[i] = MnistPartialDataset(test_thread_result[i].get())
    end = time.time()
def main():
    train_pool =  multiprocessing.Pool(processes = multiprocessing.cpu_count()-1)
    test_pool =  multiprocessing.Pool(processes = multiprocessing.cpu_count()-1)
    create_tasks(train_pool, test_pool)
    pool.close()
    pool.terminate()


title: 'An attempt has been made to start' in lr_find with fast.ai

body:
I am running this small piece of code to identify learning rate:

import cv2
from fastai.vision import *
from fastai.callbacks.hooks import *
path = untar_data(URLs.CAMVID)
path_lbl = path/'labels'
path_img = path/'images'
fnames = get_image_files(path_img)
lbl_names = get_image_files(path_lbl)
img_f = fnames[0]
img = open_image(img_f)
get_y_fn = lambda x: path_lbl/f'{x.stem}_P{x.suffix}'
mask = open_mask(get_y_fn(img_f))
src_size = np.array(mask.shape[1:])
src_size,mask.data
codes = np.loadtxt(path/'codes.txt', dtype=str); codes
size = src_size//2
bs=4
src = (SegmentationItemList.from_folder(path_img)
       .split_by_fname_file('../valid.txt')
       .label_from_func(get_y_fn, classes=codes))
data = (src.transform(get_transforms(), size=size, tfm_y=True)
        .databunch(bs=bs)
        .normalize(imagenet_stats))
name2id = {v:k for k,v in enumerate(codes)}
void_code = name2id['Void']
def acc_camvid(input, target):
    target = target.squeeze(1)
    mask = target != void_code
    return (input.argmax(dim=1)[mask]==target[mask]).float().mean()
wd=1e-2
learn = unet_learner(data, models.resnet34, metrics=acc_camvid, wd=wd)
lr_find(learn)
print("end")

And I get this error:

RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

and also this one:

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.4\helpers\pydev\pydevd.py", line 1664, in <module>
    main()
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.4\helpers\pydev\pydevd.py", line 1658, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.4\helpers\pydev\pydevd.py", line 1068, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2018.1.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/steve/Project/fastai_unet/main.py", line 32, in <module>
    lr_find(learn)
  File "C:\Users\steve\Miniconda3\lib\site-packages\fastai\train.py", line 32, in lr_find
    learn.fit(epochs, start_lr, callbacks=[cb], wd=wd)
  File "C:\Users\steve\Miniconda3\lib\site-packages\fastai\basic_train.py", line 199, in fit
    fit(epochs, self, metrics=self.metrics, callbacks=self.callbacks+callbacks)
  File "C:\Users\steve\Miniconda3\lib\site-packages\fastai\basic_train.py", line 99, in fit
    for xb,yb in progress_bar(learn.data.train_dl, parent=pbar):
  File "C:\Users\steve\Miniconda3\lib\site-packages\fastprogress\fastprogress.py", line 72, in __iter__
    for i,o in enumerate(self._gen):
  File "C:\Users\steve\Miniconda3\lib\site-packages\fastai\basic_data.py", line 75, in __iter__
    for b in self.dl: yield self.proc_batch(b)
  File "C:\Users\steve\Miniconda3\lib\site-packages\torch\utils\data\dataloader.py", line 193, in __iter__
    return _DataLoaderIter(self)
  File "C:\Users\steve\Miniconda3\lib\site-packages\torch\utils\data\dataloader.py", line 469, in __init__
    w.start()
  File "C:\Users\steve\Miniconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\steve\Miniconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\steve\Miniconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\steve\Miniconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\steve\Miniconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe

how can I fix this?


title: What is the gradient of pytorch floor() gradient method?

body:
I am looking to use floor() method in one of my models. I would like to understand what pytorch does with its gradient propagation since as such floor is a discontinuous method.

If there is no gradient defined, I could override the backward method to define my own gradient as necessary but I would like to understand what the default behavior is and the corresponding source code if possible.

import torch

x = torch.rand(20, requires_grad=True)
y = 20*x
z = y.floor().sum()
z.backward()

x.grad returns zeros.

z has a grad_fn=

So FloorBackward is the gradient method. But there is no reference to the source code of FloorBackward in pytorch repository.


title: TensorFlow / PyTorch: Gradient for loss which is measured externally

body:
I am relatively new to Machine Learning and Python.

I have a system, which consists of a NN whose output is fed into an unknown nonlinear function F, e.g. some hardware. The idea is to train the NN to be an inverse F^(-1) of that unknown nonlinear function F. This means that a loss L is calculated at the output of F. However, backpropagation cannot be used in a straightforward manner for calculating the gradients and updating the NN weights because the gradient of F is not known either.

Is there any way how to use a loss function L, which is not directly connected to the NN, for the calculation of the gradients in TensorFlow or PyTorch? Or to take a loss that was obtained with any other software (Matlab, C, etc.) use it for backpropagation?

As far as I know, Keras keras.backend.gradients only allows to calculate gradients with respect to connected weights, otherwise the gradient is either zero or NoneType.
I read about the stop_gradient() function in TensorFlow. But I am not sure whether this is what I am looking for. It allows to not compute the gradient with respect to some variables during backpropagation. But I think the operation F is not interpreted as a variable anyway.
Can I define any arbitrary loss function (including a hardware measurement) and use it for backpropagation in TensorFlow or is it required to be connected to the graph as well?

Please, let me know if my question is not specific enough.


title: updating the code for newer Pytorch and getting RuntimeError: Tensors must have same number of dimensions: got 2 and 1

body:
I am studying AI and now I am trying to understand and update a code which was made for older version of Pytorch. I have tried it for versions 1.0 & 1.5 with similar results. Variable has been deprecated and when creating a tensor requires_grad=True should be used.
Also I was advised that instead of torch.Tensor , which is an alias of torch.FloatTensor , I should use torch.tensor , which should automatically determine the data type.

But there are errors which I am not able to handle:

return map(lambda x: Variable(torch.cat(x, 0)), samples) #putting the samples in to pytorch variable

as well as when I remove Variable

return map(lambda x: torch.cat(x, 0), samples)

gets the same error:

RuntimeError: Tensors must have same number of dimensions: got 2 and 1

Code is:

# AI for Self Driving Car

# Importing the libraries

import numpy as np
import random
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torch.autograd as autograd
from torch.autograd import Variable

# Creating the architecture of the Neural Network

class Network(nn.Module):

    def __init__(self, input_size, nb_action):
        super(Network, self).__init__()  #to use all tools from nn.Module
        self.input_size = input_size
        self.nb_action = nb_action
        self.fc1 = nn.Linear(input_size, 30) #creating the full connection between input & hidden layer
        self.fc2 = nn.Linear(30, nb_action)

    def forward(self, state):   #forward propagation
        x = F.relu(self.fc1(state))
        q_values = self.fc2(x)
        return q_values

# Implementing Experience Replay

class ReplayMemory(object):

    def __init__(self, capacity):
        self.capacity = capacity  #maximum number of events in memory
        self.memory = []

    def push(self, event):  #append new event in to the memory up to a maximum memory size
        self.memory.append(event)
        if len(self.memory) > self.capacity:
            del self.memory[0]

    def sample(self, batch_size):   #take random samples form the memory
        # if list=((1,2,3),(4,5,6)) then zip(*list)=((1,4),(2,5),(3,6))
        # events = (state,action,reward) we need (state1,state2), (action1,action2), (reward1,reward2)
        samples = zip(*random.sample(self.memory, batch_size))
        # torch.cat aligns everything as (state, action, reward)
# OLD-> return map(lambda x: Variable(torch.cat(x, 0)), samples) #putting samples in a pytorch variable
        return map(lambda x: torch.cat(x, 0), samples) #<-NEW    #putting samples in a pytorch variable

# Implementing Deep Q Learning

class Dqn():

    def __init__(self, input_size, nb_action, gamma):
        self.gamma = gamma
        self.reward_window = []
        self.model = Network(input_size, nb_action)
        self.memory = ReplayMemory(100000)   #memory capacity
        self.optimizer = optim.Adam(self.model.parameters(), lr = 0.001)
#OLD-> self.last_state = torch.Tensor(input_size).unsqueeze(0)
        self.last_state = torch.tensor(input_size, requires_grad=True).unsqueeze(0) #<-NEW
        self.last_action = 0
        self.last_reward = 0

    def select_action(self, state):
        with torch.no_grad():
            probs = F.softmax(self.model(state), dim=1)*100 # T=100
            action = probs.multinomial(num_samples=1)
            return action.data[0,0]

# obsolete
#    def select_action(self, state):
#        probs = F.softmax(self.model(Variable(state, volatile = True))*100) # T=100
#        action = probs.multinomial()
#        return action.data[0,0]

    def learn(self, batch_state, batch_next_state, batch_reward, batch_action):
        outputs = self.model(batch_state).gather(1, batch_action.unsqueeze(1)).squeeze(1)
        next_outputs = self.model(batch_next_state).detach().max(1)[0]
        target = self.gamma*next_outputs + batch_reward
        td_loss = F.smooth_l1_loss(outputs, target)
        self.optimizer.zero_grad()
        td_loss.backward(retain_graph = True)
        self.optimizer.step()

    def update(self, reward, new_signal):
#OLD-> new_state = torch.Tensor(new_signal).float().unsqueeze(0)
        new_state = torch.tensor(new_signal, requires_grad=True, dtype=torch.float).float().unsqueeze(0)   #<-NEW

        self.memory.push((self.last_state, new_state, torch.LongTensor([int(self.last_action)]), torch.Tensor([self.last_reward])))
        action = self.select_action(new_state)
        if len(self.memory.memory) > 100:
            batch_state, batch_next_state, batch_action, batch_reward = self.memory.sample(100)
            self.learn(batch_state, batch_next_state, batch_reward, batch_action)
        self.last_action = action
        self.last_state = new_state
        self.last_reward = reward
        self.reward_window.append(reward)
        if len(self.reward_window) > 1000:
            del self.reward_window[0]
        return action

    def score(self):
        return sum(self.reward_window)/(len(self.reward_window)+1.)

    def save(self):
        torch.save({'state_dict': self.model.state_dict(),
                    'optimizer' : self.optimizer.state_dict(),
                   }, 'last_brain.pth')

    def load(self):
        if os.path.isfile('last_brain.pth'):
            print("=> loading checkpoint... ")
            checkpoint = torch.load('last_brain.pth')
            self.model.load_state_dict(checkpoint['state_dict'])
            self.optimizer.load_state_dict(checkpoint['optimizer'])
            print("done !")
        else:
            print("no checkpoint found...")


title: A custom loss function with Keras model

body:
I have a custom loss function and I want to use it in a Keras model but it give me the below errors could you please help me to solve this problem.  It works with simple custom loss function. I have provided data and the model to be easily checked.

the custom loss function which is Maximum mean discrepancy

import keras.backend as K

def guassian_kernel( source, target, kernel_mul = 2.0, kernel_num = 5, fix_sigma = None):

    total = K.concatenate([source, target], axis=0)
    n_samples = K.int_shape(source)[0] + K.int_shape(target)[0]

    total0= tf.broadcast_to(K.expand_dims(total, 0),shape=(K.int_shape(total)[0],K.int_shape(total)[0],K.int_shape(total)[1]))
    total1= tf.broadcast_to(K.expand_dims(total, 1),shape=(K.int_shape(total)[0],K.int_shape(total)[0],K.int_shape(total)[1]))

    print(K.int_shape(total0))

    print(K.int_shape(total1))
    print(total0)
    print(total1)

    # L2_distance = ((total0-total1)**2).sum(2)
    L2_distance = K.sum(((total0-total1)**2),axis=2)

    if fix_sigma:
      bandwidth = fix_sigma
    else:
      bandwidth = K.sum(L2_distance.data) / (n_samples**2-n_samples)
    bandwidth /= kernel_mul ** (kernel_num // 2)

    bandwidth_list = [bandwidth * (kernel_mul**i) for i in range(kernel_num)]

    kernel_val = [K.exp(-L2_distance / bandwidth_temp) for bandwidth_temp in bandwidth_list]
    # print(K.sum((kernel_val),axis=0))
    # return K.sum((kernel_val),axis=0)
    return sum(kernel_val)

def MMD_loss( source, target):
  kernel_mul = 2.0
  kernel_num = 5
  # sample_weight=torch.FloatTensor(sample_weight)
  print(K.int_shape(source))
  print(source)

  print(target)

  print(K.int_shape(target)[1])
  fix_sigma  = K.constant([1e-6])
  # source=torch.FloatTensor(source)
  # target=torch.FloatTensor(target)
  # print(K.shape(source))
  batch_size = K.int_shape(source)[0]
  kernels = guassian_kernel(source, target, kernel_mul,  kernel_num,  fix_sigma)
  XX = kernels[:batch_size, :batch_size]
  YY = kernels[batch_size:, batch_size:]
  XY = kernels[:batch_size, batch_size:]
  YX = kernels[batch_size:, :batch_size]
  loss = K.mean(XX + YY - XY -YX)
  print("weighted loss........")
  # return K.sum(np.dot(loss.numpy(), sample_weight),axis=1)
  return loss #.numpy()

y_pred = [[0, 0.95, 0]]
weight = [[1, 1, 1]]
MMD_loss(K.constant(y_true),K.constant(y_pred))

Data

from sklearn.datasets import make_blobs
from matplotlib import pyplot
from pandas import DataFrame
import numpy as np
# generate 2d classification dataset
X, y = make_blobs(n_samples=100, centers=3, n_features=2)
# scatter plot, dots colored by class value
df = DataFrame(dict(x=X[:,0], y=X[:,1], label=y))
colors = {0:'red', 1:'blue', 2:'green'}
fig, ax = pyplot.subplots()
grouped = df.groupby('label')
for key, group in grouped:
    group.plot(ax=ax, kind='scatter', x='x', y='y', label=key, color=colors[key])
pyplot.show()

Encoding

def one_hot(labels, n_class = 3):
    """ One-hot encoding """
    expansion = np.eye(n_class)
    y = expansion[:, labels-1].T
    assert y.shape[1] == n_class, "Wrong number of labels!"

    return y

Model

import keras
from keras.models import Sequential
from keras.layers import Dense
from keras import optimizers
from sklearn.model_selection import train_test_split

dfclass=df['label'].values
df1 = df.drop([ "label"], axis=1).values.astype("float32")

X_train, X_test, y_train, y_test = train_test_split(df1, dfclass, train_size=0.8)
y_train = one_hot(y_train,3)
model = Sequential()
model.add(Dense(6, input_dim=2, activation='relu'))
# model.add(Dense(32, activation='relu'))
model.add(Dense(3, activation='softmax'))
adam=optimizers.Adam(lr=0.001)

# model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
model.compile(loss = MMD_loss, optimizer=adam, metrics=['accuracy'])
sam_weight = np.array( [1] * 80)
# sam_weight = np.ones(shape=(len(y_train),))

history = model.fit(X_train, y_train, epochs=10, batch_size=10)#,sample_weight=sam_weight)
loss, acc = model.evaluate(X_test, one_hot(y_test,3), batch_size=80,verbose=1)
print("accuracy ",acc)

Error

(None, None)
Tensor("dense_48_target:0", shape=(None, None), dtype=float32)
Tensor("dense_48/Softmax:0", shape=(None, 3), dtype=float32)
3
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-206-3fc90c13c85e> in <module>()
     17
     18 # model.compile(loss='categorical_crossentropy', optimizer=adam, metrics=['accuracy'])
---> 19 model.compile(loss = MMD_loss, optimizer=adam, metrics=['accuracy'])
     20 sam_weight = np.array( [1] * 80)
     21 # sam_weight = np.ones(shape=(len(y_train),))

6 frames
<ipython-input-205-859bcd423c45> in guassian_kernel(source, target, kernel_mul, kernel_num, fix_sigma)
      6
      7     total = K.concatenate([source, target], axis=0)
----> 8     n_samples = K.int_shape(source)[0] + K.int_shape(target)[0]
      9
     10     # total0 = K.expand_dims(total, 0)

TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'


title: Extending PyTorch nn.Sequential class

body:
I'm pretty new to OOP in Python, and rusty in general. I'd like to extend PyTorch's 'nn.Sequential' object in such a way that passing it a tuple of containing the number of node in each layer automatically generates an OrderedDict according to those nodes. For a functional example:

layers = (784, 392, 196, 98, 10)
n_layers = len(layers)
modules = OrderedDict()

# Layer definitions for inner layers:
for i in range(n_layers - 2):
    modules[f'fc{i}']   = nn.Linear(layers[i], layers[i+1])
    modules[f'relu{i}'] = nn.ReLU()

# Definition for output layer:
modules['fc_out'] = nn.Linear(layers[-2], layers[-1])
modules['smax_out'] = nn.LogSoftmax(dim=1)

# Define model and check attributes:
model = nn.Sequential(modules)

So, rather than pass the 'OrderedDict' object when initializing nn.Sequential, I want my class to take the tuple instead.

class Network(nn.Sequential):
   def__init__(self, n_nodes):
      super().__init__()

      **** INSERT LOGIC FROM LAST SNIPPET ***

So it seems like this won't work, because when my Network class calls super().__init__(), it's going to want to see the dictionary of layer activations. How might I go about writing my own network in such a way that it circumvents this problem, but still has all the functionality of PyTorche's sequential object?

I was thinking along the lines of something like:

class Network(nn.Sequential):
    def __init__(self, layers):
        super().__init__(self.init_modules(layers))

    def init_modules(self, layers):
        n_layers = len(layers)
        modules = OrderedDict()

        # Layer definitions for inner layers:
        for i in range(n_layers - 2):
            modules[f'fc{i}']   = nn.Linear(layers[i], layers[i+1])
            modules[f'relu{i}'] = nn.ReLU()

        # Definition for output layer:
        modules['fc_out'] = nn.Linear(layers[-2], layers[-1])
        modules['smax_out'] = nn.LogSoftmax(dim=1)

        return modules

I'm not sure if this sort of thing is allowed and/or good practice in Python.


title: ValueError: optimizer got an empty parameter list in pytorch

body:
I want to implement Kalman Filter in PyTorch. I built the following model:
import torch
from torch import matmul, mm, nn

class DepthV1Acceleration(torch.nn.Module):
    def __init__(self, prediction_steps):
        super(DepthV1Acceleration, self).__init__()
        t = 1 / 30
        self.initial_P = torch.tensor([[0, 0, 0],
                                       [0, 1000, 0],
                                       [0, 0, 1000]])
        self.P = self.initial_P.clone()
        self.F = torch.tensor([[1, t, 0.5 * t ** 2],
                               [0, 1, t],
                               [0, 0, 0.9]])
        self.depth_acceleration = torch.tensor([0.9], requires_grad=True)
        self.F[2, 2] = self.depth_acceleration
        self.H = torch.tensor([[1, 0, 0]])
        self.R = torch.tensor([[0.01]], requires_grad=True)
        self.prediction_steps = prediction_steps

    def forward(self, measurements):
        output = torch.zeros(measurements[0], measurements[1] - self.prediction_steps, measurements[2])
        for i in range(measurements.shape[0]):
            self.P = self.initial_P.clone()
            x = measurements[i, 0, :]
            for j in range(measurements.shape[1], -1 * self.prediction_steps):
                z = measurements[i, j, :]
                y = z - mm(self.H, x)
                S = mm(mm(self.H, self.P), torch.transpose(self.H, 0, 1)) + self.R
                K = mm(mm(self.P, torch.transpose(self.H, 0, 1)) @ torch.inverse(S))
                x = x + mm(K, y)
                self.P = mm((torch.eye(10) - mm(K, self.H)), self.P)
                # prediction
                output_element = x.clone()
                x = mm(self.F, x)
                self.P = mm(mm(self.F, self.P), torch.transpose(self.F, 0, 1))
                for _ in range(self.prediction_steps):
                    output_element = mm(self.F, output_element)
                output[i, j, :] = output_element
        return output

It gives me ValueError: optimizer got an empty parameter list. I use this code:
model = DepthV1Acceleration(3)
optimizer = torch.optim.Adam(model.parameters(), lr=0.0001)

I checked other questions with the same error but didn't find anything like my case.


title: Clarification of a Faster R-CNN torchvision implementation

body:
I'm digging through the source code of the Faster R-CNN implementation of torchvision and I'm facing some things I don't quite understand. Namely, assuming that I want to create a Faster R-CNN model, not pretrained on COCO, with a backbone pre-trained on ImageNet, and then just get the backbone I do the following:
plain_backbone = fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=True).backbone.body

Which is consistent with how the backbone is set-up as indicated here and here. However, when I pass an image through the model, the results don't correspond to what I would obtain if I just set-up a resnet50 directly. Namely:
# Regular resnet50, pretrained on ImageNet, without the classifier and the average pooling layer
resnet50_1 = torch.nn.Sequential(*(list(torchvision.models.resnet50(pretrained=True).children())[:-2]))
resnet50_1.eval()
# Resnet50, extract from the Faster R-CNN, also pre-trained on ImageNet
resnet50_2 = fasterrcnn_resnet50_fpn(pretrained=False, pretrained_backbone=True).backbone.body
resnet50_2.eval()
# Loading a random image, converted to torch.Tensor, rescalled to [0, 1] (not that it matters)
image = transforms.ToTensor()(Image.open("random_images/random.jpg")).unsqueeze(0)
# Obtaining the model outputs
with torch.no_grad():
    # Output from the regular resnet50
    output_1 = resnet50_1(image)
    # Output from the resnet50 extracted from the Faster R-CNN
    output_2 = resnet50_2(image)["3"]
    # Their outputs aren't the same, which I would assume they should be
    np.testing.assert_almost_equal(output_1.numpy(), output_2.numpy())

Looking forward to your thoughts!


title: What parameters do I change to train a pytorch model from scratch?

body:
I followed this tutorial to train a pytorch model for instance segmentation:
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html
I would not like to train a model on entirely different data and classes, totally unrelated to COCO. What changes do I need to make to retrain the model. From my reading I'm guessing besides have the correct number of classes I just need to train this line:
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

to
model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False)

But I notice there is another parameters: pretrained_backbone=True, trainable_backbone_layers=None should they be changed too?


title: RTX 3080, Ubuntu 18.04, cuda 11.1, gcc 9.1.0, fail to install pytorch 1.7.1

body:
The NVIDIA driver is shown following:
1
I used anaconda and create a virtual environment to install pytroch 1.7.1, when I type the test code:
print(torch.rand(3,3).cuda())

There came the error:
2
This may caused by the error Geforce driver. But I have confirmed that the cuda, gcc and pytorch version are matched.
I installed pytorch 1.7.0，1.6.0 and I got the same results.
Next, I created another environment and installed pytorch 1.5.0. After I typed the same test code, 5 minutes later I could see the result:
3
If I run this code again, I can get the result right away. But if I exit the python and come into python again, when run this test code in the first time I have to wait again.
How to solve this problem or which pytorch and cuda version should I install? Thanks a lot.


title: PyTorch loss function evaluation

body:
I am trying to optimise a function (fit) using the acceleration of GPU in PyTorch. This is the straight Python code, where I doing the evaluation of fit:

import numpy as np
...
for j in range(P):
    e[:,j] = z - h[:,j];
    fit[j] = 1/(sqrt(2*pi)*sigma*N)*np.sum(exp(-(e[:,j]**2)/(2*sigma**2)));

  The dimension of the variables are:
  z[Nx1], h[NxP], e[NxP], fit[1xP].
  where P is the number of dimension of fit and N is the length of each dimension.

  I am aware that for loops should be avoided, so where it is my attempt to do it using PyTorch through torch.cuda.FloatTensor.

import torch
dtype = torch.cuda.FloatTensor
e     = z - h;
fit   = 1/(torch.sqrt(2*pi)*sigma*N)*torch.sum(torch.exp(-(torch.pw(e,2))/(2*torch.pow(sigma,2))));

  Unfortunately it is not working. What is wrong?
  Thank you!


title: Language translation using TorchText (PyTorch)

body:
I have recently started with ML/DL using PyTorch. The following pytorch example explains how we can train a simple model for translating from German to English.

https://pytorch.org/tutorials/beginner/torchtext_translation_tutorial.html

However I am confused on how to use the model for running inference on custom input. From my understanding so far :

1) We will need to save the "vocab" for both German (input) and English(output) [using torch.save()] so that they can be used later for running predictions.

2) At the time of running inference on a German paragraph, we will first need to convert the German text to tensor using the german vocab file.

3) The above tensor will be passed to the model's forward method for translation

4) The model will again return a tensor for the destination language i.e., English in current example.

5) We will use the English vocab saved in first step to convert this tensor back to English text.

Questions:

1) If the above understanding is correct, can the above steps be treated as a generic approach for running inference on any language translation model if we know the source and destination language and have the vocab files for the same? Or can we use the vocab provided by third party libraries like spacy?

2) How do we convert the output tensor returned from model back to target language? I couldn't find any example on how to do that. The above blog explains how to convert the input text to tensor using source-language vocab.

I could easily find various examples and detailed explanation for image/vision models but not much for text.


title: Changing attribute values via Pyqgis

body:
I would like to change the attribute value according to the excel table.
I write the following code but the newly added field cannot change value.
The original shapefile just have one field.

import pandas as pd

file='/home/xj/Research/Data/Map/province.csv'

add = pd.read_csv(file)

for i in range(0,len(add)):
    add_name = add.iloc[i,2]
    add_code = add.iloc[i,1]

    print(add_code,add_name,type(add_name))

    uri = '/home/xj/Research/Data/Map/Province/'+add_name+'.shp'
    layer = iface.addVectorLayer(uri, '', 'ogr')

    layer_provider=layer.dataProvider()
    layer_provider.addAttributes([QgsField("PCode",QVariant.Double)])
    layer_provider.addAttributes([QgsField("PName",QVariant.String)])
    layer.updateFields()

    layer.startEditing()
    features = layer.getFeatures()
    for f in features:
        id=f.id()
        attr_value={1:add_name}
        layer_provider.changeAttributeValues({id:attr_value})
        attr_code={2:add_code}
        layer_provider.changeAttributeValues({id:attr_code})

    layer.commitChanges()

    QgsProject.instance().removeAllMapLayers()

print(f"done!")


title: GPT2 on Hugging face(pytorch transformers) RuntimeError: grad can be implicitly created only for scalar outputs

body:
I am trying to fine-tune gpt2 with a custom dataset of mine. I created a basic example with the documentation from hugging-face transformers. I receive the mentioned error. I know what it means: (basically it is calling backward on a non-scalar tensor) but since I almost use only API calls, I have no idea how to fix this issue. Any suggestions?
from pathlib import Path
from absl import flags, app
import IPython
import torch
from transformers import GPT2LMHeadModel, Trainer,  TrainingArguments
from data_reader import GetDataAsPython

# this is my custom data, but i get the same error for the basic case below
# data = GetDataAsPython('data.json')
# data = [data_point.GetText2Text() for data_point in data]
# print("Number of data samples is", len(data))

data = ["this is a trial text", "this is another trial text"]

train_texts = data

from transformers import GPT2Tokenizer
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')

special_tokens_dict = {'pad_token': '<PAD>'}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
train_encodigs = tokenizer(train_texts, truncation=True, padding=True)

class BugFixDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings

    def __getitem__(self, index):
        item = {key: torch.tensor(val[index]) for key, val in self.encodings.items()}
        return item

    def __len__(self):
        return len(self.encodings['input_ids'])

train_dataset = BugFixDataset(train_encodigs)

training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=10,
)

model = GPT2LMHeadModel.from_pretrained('gpt2', return_dict=True)
model.resize_token_embeddings(len(tokenizer))

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset,
)

trainer.train()


title: Python, Fast Bert, predict with pre trained model

body:
I'm using the fast Bert package to train a Bert model.
the fast bert save model outputs the following files:
-Resources
-- config.json
-- pytorch_model.bin
-- specail_tokens_map.json
-- tokenizer_config.json
-- vocab.txt
-- events.out.tfevents.1601151257 (1).ffce4853f2c9

I'm trying to load to model and use it to predict, again with fast bert:
from fast_bert.prediction import BertClassificationPredictor
def predictor(texts, MODEL_PATH=None):
        """
        :param MODEL_PATH: path to trained model
        :type texts: list
        :param: texts: texts to run prediction on
        """
        MODEL_PATH = '/content/Resources/'
        LABEL_DATA = '/content/data/' # same path as when trained so it's valid

        if MODEL_PATH is None:
            raise LookupError("This Path is either wrong or No Trained Model exists")
        predictor = BertClassificationPredictor(
            model_path=MODEL_PATH,
            label_path=LABEL_DATA,  # location for labels.csv file
            multi_label=False,
            model_type='xlnet',
            do_lower_case=False)

        # Batch predictions
        multiple_predictions = predictor.predict_batch(texts)
        pprint(("Predicting texts accuracy",) multiple_predictions)
        return multiple_predictions

and so, I get the following error:
OSError: Unable to load weights from pytorch checkpoint file. If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.


title: Pytorch: How to map a model zoo pre-trained model to a new GPU

body:
I am trying to load one of the pre-trained models

model_urls = { 'resnet18': 'https://download.pytorch.org/models/resnet18-5c106cde.pth', 'resnet34': 'https://download.pytorch.org/models/resnet34-333f7ec4.pth'}

When I used the following code, it always loads the model to cuda:0. What if I want to load it to cuda:3?

model = ResNet(BasicBlock, [3, 4, 6, 3])
device = 3
model.load_state_dict(model_zoo.load_url(model_urls['resnet34'],
                      map_location=lambda storage, loc: storage.cuda(device)))


title: CMake not finding files - Can't build detectron with pytorch

body:
make ops for detectron isn't working

export Caffe2_DIR=<path to my pytorch>/torch
make ops

Error Message

[ 40%] Building CXX object CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o
In file included from /home/ubtuntu/detectron/detectron/ops/zero_even_op.cc:17:0:
/home/ubtuntu/detectron/detectron/ops/zero_even_op.h:20:33: fatal error: caffe2/core/context.h: No such file or directory
compilation terminated.
CMakeFiles/caffe2_detectron_custom_ops.dir/build.make:62: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o' failed
make[3]: *** [CMakeFiles/caffe2_detectron_custom_ops.dir/detectron/ops/zero_even_op.cc.o] Error 1
make[3]: Leaving directory '/home/ubtuntu/detectron/build'
CMakeFiles/Makefile2:72: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops.dir/all' failed
make[2]: *** [CMakeFiles/caffe2_detectron_custom_ops.dir/all] Error 2
make[2]: *** Waiting for unfinished jobs....
/home/ubtuntu/detectron/detectron/ops/zero_even_op.cu:17:37: fatal error: caffe2/core/context_gpu.h: No such file or directory
compilation terminated.
CMake Error at caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o.cmake:219 (message):
  Error generating
  /home/ubtuntu/detectron/build/CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/./caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o

CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/build.make:63: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o' failed
make[3]: *** [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/detectron/ops/caffe2_detectron_custom_ops_gpu_generated_zero_even_op.cu.o] Error 1
make[3]: Leaving directory '/home/ubtuntu/detectron/build'
CMakeFiles/Makefile2:109: recipe for target 'CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all' failed
make[2]: *** [CMakeFiles/caffe2_detectron_custom_ops_gpu.dir/all] Error 2
make[2]: Leaving directory '/home/ubtuntu/detectron/build'
Makefile:129: recipe for target 'all' failed
make[1]: *** [all] Error 2
make[1]: Leaving directory '/home/ubtuntu/detectron/build'
Makefile:13: recipe for target 'ops' failed
make: *** [ops] Error 2

What I've Tried

In the error message, it says it can't find caffe2/core/context.h however the file exists in pytorch/caffe2/core/context.h. So I've tried exporting the caffe2 folder instead of the torch folder, but then the build fails earlier
- I've also tried setting it to pytorch/cmake and pytorch/build folders

I then tried creating a caffe2 link to pytorch/caffe2 inside of pytorch/torch, however that had no effect. I'm out of ideas :(


title: pytorch modify array with list of indices

body:
Suppose I have a list of indices and wish to modify an existing array with this list. Currently the only way I can do this is by using a for loop as follows. Just wondering if there is a faster/ efficient way.

torch.manual_seed(0)
a = torch.randn(5,3)
idx = torch.Tensor([[1,2], [3,2]], dtype=torch.long)
for i,j in idx:
    a[i,j] = 1

I initially assumed that gather or index_select would go some way in answering this question, but looking at documentation this doesn't seem to be the answer.

In my particular case, a is a 5 dimensional vector and idx is a Nx5 vector. So the output (after subscripting with something like a[idx]) I'd expect is a (N,) shaped vector.

Answer

Thanks to @shai below, the answer that I was seeking was:
a[idx.t().chunk(chunks=2,dim=0)]. Taken from this SO answer.


title: How to vectorize the following python code

body:
I'm trying to obtain a matrix, where each element is calculated as follows:

X = torch.ones(batch_size, dim)
X_ = torch.ones(batch_size, dim)
Y = torch.ones(batch_size, dim)
M = torch.zeros(batch_size, batch_size)
for i in range(batch_size):
    for j in range(batch_size):
        M[i, j] = ((X[i] - X_[i] * Y[j])**2).sum()

It's very slow to calculate M element-wise, is there any suggestion about how to use matrix multiplication to replace the for loops?

Thanks.


title: Querying an image with bilinear interpolation e.e. finding RGB value in the fractional coordinates using Pytorch

body:
I have a input T1 of size (1,256,256,3) i.e. images/grid of batch size 1. I have another tensor T2 of size (1, N, 2) i.e. tensor consisting of coordinates i.e. [ [10.5 , 200.787], [150.568, 190.456], …]. How do I compute functional values (using bilinear interpolation) of coordinates in T2 from T1 data?

Thanks for any help

I have tested the same functionality using tensorflow with the function "tf.contrib.resampler.resampler"


title: Transfer learning with OpenNMT

body:
I'm training a transformer model with OpenNMT-py on MIDI music files, but results are poor because I only have access to a small dataset pertaining to the style I want to study. To help the model learn something useful, I would like to use a much larger dataset of other styles of music for a pre-training and then fine-tune the results using the small dataset.

I was thinking of freezing the encoder side of the transformer after the pre-training and letting the decoder part free to do the fine-tuning. How would one do this with OpenNMT-py?