title: Input tensor size doesnt inherit training dataset labels count

body:
Recently I've stumbled upon an issue that I'm unable to resolve myself.
I've been reproducing examples for using text classification scripts:
https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_glue.py
https://github.com/huggingface/transformers/blob/master/examples/text-classification/run_tf_glue.py
Both PyTorch and Tensorflow version of them.
I've conducted few tests and one of its results are unclear to me.
These tests were including mostly switching between both ML framework scripts (PyTorch/Tensorflow) and using different models with respective datasets.
I was trying to fine-tune base language models like bert-base-uncased (uploaded via https://huggingface.co/models) with my custom CSV dataset to perform well at sequence classification task.
Most of these tests were entirely succesful, and I was able to verify models inference and predictions.
The issue began when I've tried to use different model with PyTorch framework script.
Maybe it is worth to add that same model worked flawlessly on Tensorflow fine-tuning script. Errors occur only when I'm fine-tuning with PyTorch script.
This model's config is as below:
{
  "architectures": [
    "BertForPreTraining"
  ],
  "attention_probs_dropout_prob": 0.1,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 768,
  "initializer_range": 0.02,
  "intermediate_size": 3072,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 12,
  "num_hidden_layers": 12,
  "output_past": true,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 60000
}

I kept receiving errors like:
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle)

  File "/home/konrad/miniconda3/envs/test_1_3/lib/python3.8/site-packages/torch/nn/functional.py", line 2387, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: cuda runtime error (710) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1614378083779/work/aten/src/THCUNN/generic/ClassNLLCriterion.cu:115

Which eventually led me to investigate the sizes of input and target tensors.
I've found out that input tensor size was not inheriting the count of labels.
It was:
torch.Size([16, 2])

while it should be: (77 unique labels in dataset)
torch.Size([16, 77])

I'm pretty sure it should be like that, because other successful test's input tensor sizes were correct.
So my question emerged.
What kind of methods/parameters one may use/set to ensure that correct input tensor size will be adjusted accordingly to the number of unique labels in my dataset?
May the BERT model transformers architecture be relevant?
Because when I was using models of architecture like BertModel or BertForMaskedLM there were no issues like that, everything was going smoothly.
That experimental model that caused errors was of different architecture - BertForPretraining.
The same questions I'd like to ask You.
I would appreciate any tips and further insights that could help me find a solution.
EDIT: Added code below.
#!/usr/bin/env python
# coding=utf-8
# Copyright 2020 The HuggingFace Inc. team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
""" Finetuning the library models for sequence classification on GLUE."""
# You can also adapt this script on your own text classification task. Pointers for this are left as comments.

import logging
import os
import random
import sys
from dataclasses import dataclass, field
from typing import Optional

import numpy as np
from datasets import load_dataset, load_metric

import transformers
from transformers import (
    AutoConfig,
    BertForSequenceClassification,
    AutoTokenizer,
    DataCollatorWithPadding,
    EvalPrediction,
    HfArgumentParser,
    PretrainedConfig,
    Trainer,
    TrainingArguments,
    default_data_collator,
    set_seed,
)
from transformers.trainer_utils import get_last_checkpoint, is_main_process

task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

logger = logging.getLogger(__name__)

@dataclass
class DataTrainingArguments:
    """
    Arguments pertaining to what data we are going to input our model for training and eval.
    Using `HfArgumentParser` we can turn this class
    into argparse arguments to be able to specify them on
    the command line.
    """

    task_name: Optional[str] = field(
        default=None,
        metadata={"help": "The name of the task to train on: " + ", ".join(task_to_keys.keys())},
    )
    max_seq_length: int = field(
        default=128,
        metadata={
            "help": "The maximum total input sequence length after tokenization. Sequences longer "
            "than this will be truncated, sequences shorter will be padded."
        },
    )
    overwrite_cache: bool = field(
        default=False, metadata={"help": "Overwrite the cached preprocessed datasets or not."}
    )
    pad_to_max_length: bool = field(
        default=True,
        metadata={
            "help": "Whether to pad all samples to `max_seq_length`. "
            "If False, will pad the samples dynamically when batching to the maximum length in the batch."
        },
    )
    max_train_samples: Optional[int] = field(
        default=None,
        metadata={
            "help": "For debugging purposes or quicker training, truncate the number of training examples to this "
            "value if set."
        },
    )
    max_val_samples: Optional[int] = field(
        default=None,
        metadata={
            "help": "For debugging purposes or quicker training, truncate the number of validation examples to this "
            "value if set."
        },
    )
    max_test_samples: Optional[int] = field(
        default=None,
        metadata={
            "help": "For debugging purposes or quicker training, truncate the number of test examples to this "
            "value if set."
        },
    )
    train_file: Optional[str] = field(
        default=None, metadata={"help": "A csv or a json file containing the training data."}
    )
    validation_file: Optional[str] = field(
        default=None, metadata={"help": "A csv or a json file containing the validation data."}
    )
    test_file: Optional[str] = field(default=None, metadata={"help": "A csv or a json file containing the test data."})

    def __post_init__(self):
        if self.task_name is not None:
            self.task_name = self.task_name.lower()
            if self.task_name not in task_to_keys.keys():
                raise ValueError("Unknown task, you should pick one in " + ",".join(task_to_keys.keys()))
        elif self.train_file is None or self.validation_file is None:
            raise ValueError("Need either a GLUE task or a training/validation file.")
        else:
            train_extension = self.train_file.split(".")[-1]
            assert train_extension in ["csv", "json"], "`train_file` should be a csv or a json file."
            validation_extension = self.validation_file.split(".")[-1]
            assert (
                validation_extension == train_extension
            ), "`validation_file` should have the same extension (csv or json) as `train_file`."

@dataclass
class ModelArguments:
    """
    Arguments pertaining to which model/config/tokenizer we are going to fine-tune from.
    """

    model_name_or_path: str = field(
        metadata={"help": "Path to pretrained model or model identifier from huggingface.co/models"}
    )
    config_name: Optional[str] = field(
        default=None, metadata={"help": "Pretrained config name or path if not the same as model_name"}
    )
    tokenizer_name: Optional[str] = field(
        default=None, metadata={"help": "Pretrained tokenizer name or path if not the same as model_name"}
    )
    cache_dir: Optional[str] = field(
        default=None,
        metadata={"help": "Where do you want to store the pretrained models downloaded from huggingface.co"},
    )
    use_fast_tokenizer: bool = field(
        default=True,
        metadata={"help": "Whether to use one of the fast tokenizer (backed by the tokenizers library) or not."},
    )
    model_revision: str = field(
        default="main",
        metadata={"help": "The specific model version to use (can be a branch name, tag name or commit id)."},
    )
    use_auth_token: bool = field(
        default=False,
        metadata={
            "help": "Will use the token generated when running `transformers-cli login` (necessary to use this script "
            "with private models)."
        },
    )

def main():
    # See all possible arguments in src/transformers/training_args.py
    # or by passing the --help flag to this script.
    # We now keep distinct sets of args, for a cleaner separation of concerns.

    parser = HfArgumentParser((ModelArguments, DataTrainingArguments, TrainingArguments))
    if len(sys.argv) == 2 and sys.argv[1].endswith(".json"):
        # If we pass only one argument to the script and it's the path to a json file,
        # let's parse it to get our arguments.
        model_args, data_args, training_args = parser.parse_json_file(json_file=os.path.abspath(sys.argv[1]))
    else:
        model_args, data_args, training_args = parser.parse_args_into_dataclasses()

    # Detecting last checkpoint.
    last_checkpoint = None
    if os.path.isdir(training_args.output_dir) and training_args.do_train and not training_args.overwrite_output_dir:
        last_checkpoint = get_last_checkpoint(training_args.output_dir)
        if last_checkpoint is None and len(os.listdir(training_args.output_dir)) > 0:
            raise ValueError(
                f"Output directory ({training_args.output_dir}) already exists and is not empty. "
                "Use --overwrite_output_dir to overcome."
            )
        elif last_checkpoint is not None:
            logger.info(
                f"Checkpoint detected, resuming training at {last_checkpoint}. To avoid this behavior, change "
                "the `--output_dir` or add `--overwrite_output_dir` to train from scratch."
            )

    # Setup logging
    logging.basicConfig(
        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
        datefmt="%m/%d/%Y %H:%M:%S",
        handlers=[logging.StreamHandler(sys.stdout)],
    )
    logger.setLevel(logging.INFO if is_main_process(training_args.local_rank) else logging.WARN)

    # Log on each process the small summary:
    logger.warning(
        f"Process rank: {training_args.local_rank}, device: {training_args.device}, n_gpu: {training_args.n_gpu}"
        + f"distributed training: {bool(training_args.local_rank != -1)}, 16-bits training: {training_args.fp16}"
    )
    # Set the verbosity to info of the Transformers logger (on main process only):
    if is_main_process(training_args.local_rank):
        transformers.utils.logging.set_verbosity_info()
        transformers.utils.logging.enable_default_handler()
        transformers.utils.logging.enable_explicit_format()
    logger.info(f"Training/evaluation parameters {training_args}")

    # Set seed before initializing model.
    set_seed(training_args.seed)

    # Get the datasets: you can either provide your own CSV/JSON training and evaluation files (see below)
    # or specify a GLUE benchmark task (the dataset will be downloaded automatically from the datasets Hub).
    #
    # For CSV/JSON files, this script will use as labels the column called 'label' and as pair of sentences the
    # sentences in columns called 'sentence1' and 'sentence2' if such column exists or the first two columns not named
    # label if at least two columns are provided.
    #
    # If the CSVs/JSONs contain only one non-label column, the script does single sentence classification on this
    # single column. You can easily tweak this behavior (see below)
    #
    # In distributed training, the load_dataset function guarantee that only one local process can concurrently
    # download the dataset.
    if data_args.task_name is not None:
        # Downloading and loading a dataset from the hub.
        datasets = load_dataset("glue", data_args.task_name)
    else:
        # Loading a dataset from your local files.
        # CSV/JSON training and evaluation files are needed.
        data_files = {"train": data_args.train_file, "validation": data_args.validation_file}

        # Get the test dataset: you can provide your own CSV/JSON test file (see below)
        # when you use `do_predict` without specifying a GLUE benchmark task.
        if training_args.do_predict:
            if data_args.test_file is not None:
                train_extension = data_args.train_file.split(".")[-1]
                test_extension = data_args.test_file.split(".")[-1]
                assert (
                    test_extension == train_extension
                ), "`test_file` should have the same extension (csv or json) as `train_file`."
                data_files["test"] = data_args.test_file
            else:
                raise ValueError("Need either a GLUE task or a test file for `do_predict`.")

        for key in data_files.keys():
            logger.info(f"load a local file for {key}: {data_files[key]}")

        if data_args.train_file.endswith(".csv"):
            # Loading a dataset from local csv files
            datasets = load_dataset("csv", data_files=data_files)
        else:
            # Loading a dataset from local json files
            datasets = load_dataset("json", data_files=data_files)
    # See more about loading any type of standard or custom dataset at
    # https://huggingface.co/docs/datasets/loading_datasets.html.

    # Labels
    if data_args.task_name is not None:
        is_regression = data_args.task_name == "stsb"
        if not is_regression:
            label_list = datasets["train"].features["label"].names
            num_labels = len(label_list)
        else:
            num_labels = 1
    else:
        # Trying to have good defaults here, don't hesitate to tweak to your needs.
        is_regression = datasets["train"].features["label"].dtype in ["float32", "float64"]
        if is_regression:
            num_labels = 1
        else:
            # A useful fast method:
            # https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.unique
            label_list = datasets["train"].unique("label")
            label_list.sort()  # Let's sort it for determinism
            num_labels = len(label_list)

    print("\n\n")
    print("label_list", label_list[:10])
    print("\n", len(label_list))

    ############################################################
    # Crucial addition
    label2id = {label: i for i, label in enumerate(label_list)}

    # output sorted labels to json file
    import json

    sorted_labels_dict = {k: v for k, v in sorted(label2id.items(), key=lambda item: item[1])}
    with open("sorted_labels.json", "w") as outfile:
        json.dump(sorted_labels_dict, outfile)

    ############################################################

    # Load pretrained model and tokenizer
    #
    # In distributed training, the .from_pretrained methods guarantee that only one local process can concurrently
    # download model & vocab.
    config = AutoConfig.from_pretrained(
        model_args.config_name if model_args.config_name else model_args.model_name_or_path,
        num_labels=num_labels,

        ############################################################
        # Crucial addition
        label2id=label2id,
        id2label={id: label for label, id in label2id.items()},
        ############################################################

        finetuning_task=data_args.task_name,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        use_auth_token=True if model_args.use_auth_token else None,
    )
    config.num_labels = 77
    tokenizer = AutoTokenizer.from_pretrained(
        model_args.tokenizer_name if model_args.tokenizer_name else model_args.model_name_or_path,
        cache_dir=model_args.cache_dir,
        use_fast=model_args.use_fast_tokenizer,
        revision=model_args.model_revision,
        use_auth_token=True if model_args.use_auth_token else None,
    )
    model = BertForSequenceClassification.from_pretrained(
        model_args.model_name_or_path,
        from_tf=bool(".ckpt" in model_args.model_name_or_path),
        config=config,
        cache_dir=model_args.cache_dir,
        revision=model_args.model_revision,
        use_auth_token=True if model_args.use_auth_token else None,
    )
    print("\n\n\n\n", model.parameters, "\n\n\n\n")

    # Preprocessing the datasets
    if data_args.task_name is not None:
        sentence1_key, sentence2_key = task_to_keys[data_args.task_name]
    else:
        # Again, we try to have some nice defaults but don't hesitate to tweak to your use case.
        non_label_column_names = [name for name in datasets["train"].column_names if name != "label"]
        if "sentence1" in non_label_column_names and "sentence2" in non_label_column_names:
            sentence1_key, sentence2_key = "sentence1", "sentence2"
        else:
            if len(non_label_column_names) >= 2:
                sentence1_key, sentence2_key = non_label_column_names[:2]
            else:
                sentence1_key, sentence2_key = non_label_column_names[0], None

    # Padding strategy
    if data_args.pad_to_max_length:
        padding = "max_length"
    else:
        # We will pad later, dynamically at batch creation, to the max sequence length in each batch
        padding = False

    # Some models have set the order of the labels to use, so let's make sure we do use it.
    label_to_id = None
    if (
        model.config.label2id != PretrainedConfig(num_labels=num_labels).label2id
        and data_args.task_name is not None
        and not is_regression
    ):
        # Some have all caps in their config, some don't.
        label_name_to_id = {k.lower(): v for k, v in model.config.label2id.items()}

        if list(sorted(label_name_to_id.keys())) == list(sorted(label_list)):
            label_to_id = {i: int(label_name_to_id[label_list[i]]) for i in range(num_labels)}
        else:
            logger.warn(
                "Your model seems to have been trained with labels, but they don't match the dataset: ",
                f"model labels: {list(sorted(label_name_to_id.keys()))}, dataset labels: {list(sorted(label_list))}."
                "\nIgnoring the model labels as a result.",
            )
    elif data_args.task_name is None and not is_regression:
        label_to_id = {v: i for i, v in enumerate(label_list)}

    if data_args.max_seq_length > tokenizer.model_max_length:
        logger.warn(
            f"The max_seq_length passed ({data_args.max_seq_length}) is larger than the maximum length for the"
            f"model ({tokenizer.model_max_length}). Using max_seq_length={tokenizer.model_max_length}."
        )
    max_seq_length = min(data_args.max_seq_length, tokenizer.model_max_length)

    def preprocess_function(examples):
        # Tokenize the texts
        args = (
            (examples[sentence1_key],) if sentence2_key is None else (examples[sentence1_key], examples[sentence2_key])
        )
        result = tokenizer(*args, padding=padding, max_length=max_seq_length, truncation=True)

        # Map labels to IDs (not necessary for GLUE tasks)
        if label_to_id is not None and "label" in examples:
            result["label"] = [(label_to_id[l] if l != -1 else -1) for l in examples["label"]]
        return result

    if training_args.do_train:
        if "train" not in datasets:
            raise ValueError("--do_train requires a train dataset")
        train_dataset = datasets["train"]
        if data_args.max_train_samples is not None:
            train_dataset = train_dataset.select(range(data_args.max_train_samples))
        train_dataset = train_dataset.map(
            preprocess_function,
            batched=True,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    if training_args.do_eval:
        if "validation" not in datasets and "validation_matched" not in datasets:
            raise ValueError("--do_eval requires a validation dataset")
        eval_dataset = datasets["validation_matched" if data_args.task_name == "mnli" else "validation"]
        if data_args.max_val_samples is not None:
            eval_dataset = eval_dataset.select(range(data_args.max_val_samples))
        eval_dataset = eval_dataset.map(
            preprocess_function,
            batched=True,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    if training_args.do_predict or data_args.task_name is not None or data_args.test_file is not None:
        if "test" not in datasets and "test_matched" not in datasets:
            raise ValueError("--do_predict requires a test dataset")
        test_dataset = datasets["test_matched" if data_args.task_name == "mnli" else "test"]
        if data_args.max_test_samples is not None:
            test_dataset = test_dataset.select(range(data_args.max_test_samples))
        test_dataset = test_dataset.map(
            preprocess_function,
            batched=True,
            load_from_cache_file=not data_args.overwrite_cache,
        )

    # Log a few random samples from the training set:
    for index in random.sample(range(len(train_dataset)), 3):
        logger.info(f"Sample {index} of the training set: {train_dataset[index]}.")

    # Get the metric function
    if data_args.task_name is not None:
        metric = load_metric("glue", data_args.task_name)
    # TODO: When datasets metrics include regular accuracy, make an else here and remove special branch from
    # compute_metrics

    # You can define your custom compute_metrics function. It takes an `EvalPrediction` object (a namedtuple with a
    # predictions and label_ids field) and has to return a dictionary string to float.
    def compute_metrics(p: EvalPrediction):
        preds = p.predictions[0] if isinstance(p.predictions, tuple) else p.predictions
        preds = np.squeeze(preds) if is_regression else np.argmax(preds, axis=1)
        if data_args.task_name is not None:
            result = metric.compute(predictions=preds, references=p.label_ids)
            if len(result) > 1:
                result["combined_score"] = np.mean(list(result.values())).item()
            return result
        elif is_regression:
            return {"mse": ((preds - p.label_ids) ** 2).mean().item()}
        else:
            return {"accuracy": (preds == p.label_ids).astype(np.float32).mean().item()}

    # Data collator will default to DataCollatorWithPadding, so we change it if we already did the padding.
    if data_args.pad_to_max_length:
        data_collator = default_data_collator
    elif training_args.fp16:
        data_collator = DataCollatorWithPadding(tokenizer, pad_to_multiple_of=8)
    else:
        data_collator = None

    # Initialize our Trainer
    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        compute_metrics=compute_metrics,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )

    config.num_labels = 77

    print("\n\n\n AutoConfig.from_pretrained(model_args.model_name_or_path).num_label\n", AutoConfig.from_pretrained(model_args.model_name_or_path).num_labels)

    if training_args.do_train:
        if last_checkpoint is not None:
            model_path = last_checkpoint
        elif os.path.isdir(model_args.model_name_or_path):
            model_path = model_args.model_name_or_path
        else:
            model_path = None

        config.num_labels = 77
        train_result = trainer.train(model_path=model_path)
        metrics = train_result.metrics

        trainer.save_model()  # Saves the tokenizer too for easy upload

        output_train_file = os.path.join(training_args.output_dir, "train_results.txt")
        if trainer.is_world_process_zero():
            with open(output_train_file, "w") as writer:
                logger.info("***** Train results *****")
                for key, value in sorted(metrics.items()):
                    logger.info(f"  {key} = {value}")
                    writer.write(f"{key} = {value}\n")

            # Need to save the state, since Trainer.save_model saves only the tokenizer with the model
            trainer.state.save_to_json(os.path.join(training_args.output_dir, "trainer_state.json"))

    eval_results = {}
    if training_args.do_eval:
        logger.info("*** Evaluate ***")

        # Loop to handle MNLI double evaluation (matched, mis-matched)
        tasks = [data_args.task_name]
        eval_datasets = [eval_dataset]
        if data_args.task_name == "mnli":
            tasks.append("mnli-mm")
            eval_datasets.append(datasets["validation_mismatched"])

        for eval_dataset, task in zip(eval_datasets, tasks):
            eval_result = trainer.evaluate(eval_dataset=eval_dataset)

            output_eval_file = os.path.join(training_args.output_dir, f"eval_results_{task}.txt")
            if trainer.is_world_process_zero():
                with open(output_eval_file, "w") as writer:
                    logger.info(f"***** Eval results {task} *****")
                    for key, value in sorted(eval_result.items()):
                        logger.info(f"  {key} = {value}")
                        writer.write(f"{key} = {value}\n")

            eval_results.update(eval_result)

    if training_args.do_predict:
        logger.info("*** Test ***")

        # Loop to handle MNLI double evaluation (matched, mis-matched)
        tasks = [data_args.task_name]
        test_datasets = [test_dataset]
        if data_args.task_name == "mnli":
            tasks.append("mnli-mm")
            test_datasets.append(datasets["test_mismatched"])

        for test_dataset, task in zip(test_datasets, tasks):
            # Removing the `label` columns because it contains -1 and Trainer won't like that.
            test_dataset.remove_columns_("label")
            predictions = trainer.predict(test_dataset=test_dataset).predictions
            predictions = np.squeeze(predictions) if is_regression else np.argmax(predictions, axis=1)

            output_test_file = os.path.join(training_args.output_dir, f"test_results_{task}.txt")
            if trainer.is_world_process_zero():
                with open(output_test_file, "w") as writer:
                    logger.info(f"***** Test results {task} *****")
                    writer.write("index\tprediction\n")
                    for index, item in enumerate(predictions):
                        if is_regression:
                            writer.write(f"{index}\t{item:3.3f}\n")
                        else:
                            item = label_list[item]
                            writer.write(f"{index}\t{item}\n")

def _mp_fn(index):
    # For xla_spawn (TPUs)
    main()

if __name__ == "__main__":
    main()


title: GPU memory increasing at each batch (PyTorch)

body:
I am trying to build a convolutionnal network using ConvLSTM layer (LSTM cell but with convolutions instead of matrix multiplications), but the problem is that my GPU memory increases at each batch, even if I'm deleting variables, and getting the true value for the loss (and not the graph) for each iteration. I may be doing something wrong but that exact same script ran without issues with another model (with more parameters and also using ConvLSTM layer).
Each batch is composed of num_batch x 3 images (grayscale) and I'm trying to predict the difference |Im(t+1)-Im(t)| with the input Im(t)
def main():
    config = Config()

    train_dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=config.batch_size, num_workers=0, shuffle=True, drop_last=True)

    nb_img = len(train_dataset)
    util.clear_progress_dir()

    step_tensorboard = 0
    ###################################
    #          Model Setup            #
    ###################################

    model = fully_convLSTM()
    if torch.cuda.is_available():
        model = model.float().cuda()

    lr = 0.001
    optimizer = torch.optim.Adam(model.parameters(),lr=lr)

    util.enumerate_params([model])

    ###################################
    #          Training Loop          #
    ###################################

    model.train() #Put model in training mode

    train_loss_recon = []
    train_loss_recon2 = []

    for epoch in tqdm(range(config.num_epochs)):

        running_loss1 = 0.0
        running_loss2 = 0.0

        for i, (inputs, outputs) in enumerate(train_dataloader, 0):
            print(i)
            torch.cuda.empty_cache()
            gc.collect()

           # if torch.cuda.is_available():
            inputs  = autograd.Variable(inputs.float()).cuda()
            outputs = autograd.Variable(outputs.float()).cuda()

            im1 =  inputs[:,0,:,:,:]
            im2 =  inputs[:,1,:,:,:]
            im3 =  inputs[:,2,:,:,:]

            diff1 = torch.abs(im2 - im1).cuda().float()
            diff2 = torch.abs(im3 - im2).cuda().float()

            model.initialize_hidden()

            optimizer.zero_grad()
            pred1 = model.forward(im1)
            loss = reconstruction_loss(diff1, pred1)
            loss.backward()
            # optimizer.step()

            model.update_hidden()

            optimizer.zero_grad()
            pred2 = model.forward(im2)
            loss2 = reconstruction_loss(diff2, pred2)
            loss2.backward()
            optimizer.step()

            model.update_hidden()

            ## print statistics

            running_loss1 += loss.detach().data
            running_loss2 += loss2.detach().data

            if i==0:

                with torch.no_grad():
                    img_grid_diff_true = (diff2).cpu()
                    img_grid_diff_pred = (pred2).cpu()

                    f, axes = plt.subplots(2, 4, figsize=(48,48))
                    for l in range(4):
                        axes[0, l].imshow(img_grid_diff_true[l].squeeze(0).squeeze(0), cmap='gray')
                        axes[1, l].imshow(img_grid_diff_pred[l].squeeze(0).squeeze(0), cmap='gray')

                    plt.show()
                    plt.close()

                    writer_recon_loss.add_scalar('Reconstruction loss', running_loss1, step_tensorboard)
                    writer_recon_loss2.add_scalar('Reconstruction loss2', running_loss2, step_tensorboard)

                    step_tensorboard += 1

            del pred1
            del pred2
            del im1
            del im2
            del im3
            del diff1
            del diff2#, im1_noised, im2_noised
            del inputs
            del outputs
            del loss
            del loss2
            for obj in gc.get_objects():
                if torch.is_tensor(obj) :
                    del obj

            torch.cuda.empty_cache()
            gc.collect()

        epoch_loss = running_loss1 / len(train_dataloader.dataset)
        epoch_loss2 = running_loss2/ len(train_dataloader.dataset)
        print(f"Epoch {epoch} loss reconstruction1: {epoch_loss:.6f}")
        print(f"Epoch {epoch} loss reconstruction2: {epoch_loss2:.6f}")

        train_loss_recon.append(epoch_loss)
        train_loss_recon2.append(epoch_loss2)

        del running_loss1, running_loss2, epoch_loss, epoch_loss2

Here is the model used :
class ConvLSTMCell(nn.Module):
    def __init__(self, input_channels, hidden_channels, kernel_size):
        super(ConvLSTMCell, self).__init__()

        # assert hidden_channels % 2 == 0

        self.input_channels = input_channels
        self.hidden_channels = hidden_channels
        self.kernel_size = kernel_size
        # self.num_features = 4

        self.padding = 1

        self.Wxi = nn.Conv2d(self.input_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=True)
        self.Whi = nn.Conv2d(self.hidden_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=False)
        self.Wxf = nn.Conv2d(self.input_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=True)
        self.Whf = nn.Conv2d(self.hidden_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=False)
        self.Wxc = nn.Conv2d(self.input_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=True)
        self.Whc = nn.Conv2d(self.hidden_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=False)
        self.Wxo = nn.Conv2d(self.input_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=True)
        self.Who = nn.Conv2d(self.hidden_channels, self.hidden_channels, self.kernel_size, 1, self.padding, bias=False)

        self.Wci = None
        self.Wcf = None
        self.Wco = None

    def forward(self, x, h, c): ## Equation (3) dans Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
        ci = torch.sigmoid(self.Wxi(x) + self.Whi(h) + c * self.Wci)
        cf = torch.sigmoid(self.Wxf(x) + self.Whf(h) + c * self.Wcf)
        cc = cf * c + ci * torch.tanh(self.Wxc(x) + self.Whc(h)) ###gt= tanh(cc)
        co = torch.sigmoid(self.Wxo(x) + self.Who(h) + cc * self.Wco) ##channel out = hidden channel
        ch = co * torch.tanh(cc)
        return ch, cc #short memory, long memory

    def init_hidden(self, batch_size, hidden, shape):
        if self.Wci is None:
            self.Wci = nn.Parameter(torch.zeros(1, hidden, shape[0], shape[1])).cuda()
            self.Wcf = nn.Parameter(torch.zeros(1, hidden, shape[0], shape[1])).cuda()
            self.Wco = nn.Parameter(torch.zeros(1, hidden, shape[0], shape[1])).cuda()
        else:
            assert shape[0] == self.Wci.size()[2], 'Input Height Mismatched!'
            assert shape[1] == self.Wci.size()[3], 'Input Width Mismatched!'
        return (autograd.Variable(torch.zeros(batch_size, hidden, shape[0], shape[1])).cuda(),
                autograd.Variable(torch.zeros(batch_size, hidden, shape[0], shape[1])).cuda())

class fully_convLSTM(nn.Module):
    def __init__(self):
        super(fully_convLSTM, self).__init__()
        layers = []
        self.hidden_list = [1,32,32,1]#,32,64,32,
        for k in range(len(self.hidden_list)-1):   # Define blocks of [ConvLSTM,BatchNorm,Relu]
            name_conv = "self.convLSTM" +str(k)
            cell_conv = ConvLSTMCell(self.hidden_list[k],self.hidden_list[k+1],3)
            setattr(self, name_conv, cell_conv)
            name_batchnorm = "self.batchnorm"+str(k)
            batchnorm=nn.BatchNorm2d(self.hidden_list[k+1])
            setattr(self, name_batchnorm, batchnorm)
            name_relu =" self.relu"+str(k)
            relu=nn.ReLU()
            setattr(self, name_relu, relu)
        self.sigmoid = nn.Sigmoid()

        self.internal_state=[]

    def initialize_hidden(self):
        for k in range(len(self.hidden_list)-1):
            name_conv = "self.convLSTM" +str(k)
            (h,c) = getattr(self,name_conv).init_hidden(config.batch_size, self.hidden_list[k+1],(256,256))
            self.internal_state.append((h,c))
        self.internal_state_new=[]
    def update_hidden(self):
        for i, hidden in enumerate(self.internal_state_new):
            self.internal_state[i] = (hidden[0].detach(), hidden[1].detach())
        self.internal_state_new = []
    def forward(self, input):
        x = input
        for k in range(len(self.hidden_list)-1):
            name_conv = "self.convLSTM" +str(k)
            name_batchnorm = "self.batchnorm"+str(k)
            name_relu =" self.relu"+str(k)
            x, c = getattr(self,name_conv)(x, self.internal_state[k][1], self.internal_state[k][0])
            self.internal_state_new.append((x.detach(),c.detach()))
            x = getattr(self,name_batchnorm)(x)
            if k!= len(self.hidden_list)-2:
                x = getattr(self,name_relu)(x)
            else :
                x = self.sigmoid(x)
        return x

So my question is, what in my code is causing memory to accumulate during the training phase?


title: RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)` ```

body:
I'm trying to convert code from java to python. For that I'm using Facebook's Transcoder and while running the following command on my google colab notebook, I'm getting the following error (I guess all the dependencies have been installed):
Command
!python TransCoder/translate.py --src_lang java --tgt_lang python \\
--BPE_path TransCoder/data/BPE_with_comments_codes --model_path model_1.pth < a.java

Error
Loading codes from /content/TransCoder/data/BPE_with_comments_codes ...
Read 50000 codes from the codes file.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: indexSelectLargeIndex: block: [59,0,0], thread: [64,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: indexSelectLargeIndex: block: [59,0,0], thread: [65,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: indexSelectLargeIndex: block: [59,0,0], thread: [66,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
.
.
.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:662: indexSelectLargeIndex: block: [59,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
Traceback (most recent call last):
  File "TransCoder/translate.py", line 179, in <module>
    input, lang1=params.src_lang, lang2=params.tgt_lang, beam_size=params.beam_size)
  File "TransCoder/translate.py", line 129, in translate
    langs=langs1, causal=False)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/content/TransCoder/XLM/src/model/transformer.py", line 328, in forward
    return self.fwd(**kwargs)
  File "/content/TransCoder/XLM/src/model/transformer.py", line 400, in fwd
    attn = self.attentions[i](tensor, attn_mask, use_cache=use_cache)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/content/TransCoder/XLM/src/model/transformer.py", line 182, in forward
    q = shape(self.q_lin(input))
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling `cublasCreate(handle)`


title: Dynamic Programming w/ PyTorch: Longest Common Subsequence

body:
I am currently wondering how could we make a efficient implementation of LCS problem.
I found a way to find consecutive match (i.e. ngrams match) with tensor operation by comparing and shifting.
With two sequences x (len: n), y (len: m), the matrix:

e = x.eq(y.unsqueeze(1)) # [n x m]

we have: e[i, j] == 1 <=> x[i] == y[j], a n-gram match will be visible as a diagonal of ones.
Thus we can do the following:

# match_1 = [n x m] of {0, 1}
match_1 = x.eq(y.unsqueeze(1))

# match_2 = [(n-1) x (m-1)] matrix of {0, 1}
match_2 = match_1[:-1, :-1] * match_1[1:, 1:]

# etcetc

The LCS problem is more complicated as we allow gaps. It can be implemented using dynamic programing, in O(n x m), but it is not really protable to Pytorch right?
I tried, it’s super slow.

# considering two "sentences" (LongTensors of word indices)
# _ts, _tr of respective length n and m
table = torch.zeros(n+1, m+1)
_ts, _tr = ts, tr
for i in range(1, n+1):
    for j in range(1, m+1):
        if _ts[i-1] == _tr[j-1]:
            _table[i, j] = _table[i-1, j-1] + 1
        else:
            _table[i, j] = max(_table[i-1, j], _table[i, j-1])
 lcs = _table[n][m]

Any idea to make it more efficient?


title: Can I implement kaldi-pytorch on windows?

body: kaldi can only be used on windows via VM configuration (fedora 29 for example) , which massively consumes ressources of computations and late working flow . is there any other suitable way to configure and implement kaldi-pytorch on windows10 ? thanks


title: Pytorch neural network for classification

body:
The task is to train a neural network to choose the value v in a list for given input x that minimizes (x - v).
For example I have 100x1 dimension tensor of integers as input. My neural network should output probability distribution for those values.

It works properly for 1x1 tensor, but outputs the same distribution for each example in the input. I want it to output proper distribution for every example separately. How should I fix that?

Here is the code:

N, D_in, D_out, H = 20, 1, 5, 10
X = torch.LongTensor(100, 1).random_(0, 500).float()
values = torch.FloatTensor([0, 1, 3, 5, 100])

model = nn.Sequential(
    nn.Linear(D_in, H),
    nn.Sigmoid(),
    nn.Linear(H, D_out),
    nn.Softmax(dim=1)
)

def loss_function(x, y, probability):
    return torch.mean(torch.sum(probability*((x-y)**2), 1).div(probability.size(1)))

loss_fn = loss_function
learning_rate = 1e-3
optimizer = optim.SGD(model.parameters(), lr=learning_rate)

model.train(True)
for epoch in range(200):
    pred = model(X)
    loss = loss_fn(X, values, pred)
    print('loss: ', loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

And it is the input and the output for 20 examples:
Input:

tensor([[ 16.],
         [ 28.],
         [188.],
         [ 16.],
         [379.],
         [366.],
         [366.],
         [421.],
         [ 13.],
         [ 86.],
         [ 38.],
         [135.],
         [173.],
         [153.],
         [223.],
         [133.],
         [316.],
         [108.],
         [133.],
         [295.]])

Output:

tensor([[1.2926e-04, 1.3165e-04, 1.2761e-04, 1.3385e-04, 9.9948e-01],
         [9.9158e-05, 1.0086e-04, 9.8492e-05, 1.0170e-04, 9.9960e-01],
         [3.3826e-05, 3.4437e-05, 3.4443e-05, 3.2122e-05, 9.9987e-01],
         [1.2926e-04, 1.3165e-04, 1.2761e-04, 1.3385e-04, 9.9948e-01],
         [3.3173e-05, 3.3778e-05, 3.3758e-05, 3.1419e-05, 9.9987e-01],
         [3.3175e-05, 3.3780e-05, 3.3759e-05, 3.1421e-05, 9.9987e-01],
         [3.3175e-05, 3.3780e-05, 3.3759e-05, 3.1421e-05, 9.9987e-01],
         [3.3170e-05, 3.3775e-05, 3.3754e-05, 3.1416e-05, 9.9987e-01],
         [1.4026e-04, 1.4281e-04, 1.3817e-04, 1.4524e-04, 9.9943e-01],
         [4.4578e-05, 4.5341e-05, 4.5195e-05, 4.3355e-05, 9.9982e-01],
         [8.1636e-05, 8.2994e-05, 8.1468e-05, 8.2810e-05, 9.9967e-01],
         [3.5991e-05, 3.6629e-05, 3.6643e-05, 3.4392e-05, 9.9986e-01],
         [3.4158e-05, 3.4773e-05, 3.4784e-05, 3.2472e-05, 9.9986e-01],
         [3.4881e-05, 3.5505e-05, 3.5521e-05, 3.3232e-05, 9.9986e-01],
         [3.3425e-05, 3.4031e-05, 3.4026e-05, 3.1694e-05, 9.9987e-01],
         [3.6153e-05, 3.6794e-05, 3.6807e-05, 3.4561e-05, 9.9986e-01],
         [3.3191e-05, 3.3796e-05, 3.3777e-05, 3.1439e-05, 9.9987e-01],
         [3.9212e-05, 3.9897e-05, 3.9870e-05, 3.7751e-05, 9.9984e-01],
         [3.6153e-05, 3.6794e-05, 3.6807e-05, 3.4561e-05, 9.9986e-01],
         [3.3207e-05, 3.3812e-05, 3.3795e-05, 3.1457e-05, 9.9987e-01]],
        grad_fn=<SoftmaxBackward>)


title: What considerations should be taken into account with directly setting Pytorch model variables and optimizer parameters?

body:
From the Pytorch forums
https://discuss.pytorch.org/t/layer-weight-vs-weight-data/24271/2
it is mentioned that setting variable weights directly could result in

"Using .data on the other side would work, but is generally not recommended, as changing it after the model was used would yield weird results and autograd cannot throw an error."

I am wonder what would cause weird results. Also, I am also thinking of setting optimizer parameters directly, specifically the momentum/sum of gradients for optimizers which have those parameters. Is there any considerations needed for that case as well?


title: In python3: strange behaviour of list(iterables)

body:
I have a specific question regarding the behaviour of iterables in python. My iterable is a custom built Dataset class in pytorch:

import torch
from torch.utils.data import Dataset
class datasetTest(Dataset):
    def __init__(self, X):
        self.X = X

    def __len__(self):
        return len(self.X)

    def __getitem__(self, x):
        print('***********')
        print('getitem x = ', x)
        print('###########')
        y = self.X[x]
        print('getitem y = ', y)
        return y

The weird behaviour now comes about when I initialize a specific instance of that datasetTest class. Depending on what data structure I pass as an argument X, it behaves differently when I call list(datasetTestInstance). In particular, when passing a torch.tensor as argument there is no problem, however when passing a dict as argument it will throw a KeyError. The reason for this is that list(iterable) not just calls i=0, ..., len(iterable)-1, but it calls i=0, ..., len(iterable). That is, it will iterate until (inclusive) the index equal to the length of the iterable. Obviously, this index is not definied in any python datastructure, as the last element has always the index len(datastructure)-1 and not len(datastructure). If X is a torch.tensor or a list, no error will be risen, even though I think the should be an error. It will still call getitem even for the (non-existent) element with index len(datasetTestinstance) but it will not compute y=self.X[len(datasetTestInstance]. Does anyone know if pytorch handels this somehow gracefully internally?

When passing a dict as data it will throw an error in the last iteration, when x=len(datasetTestInstance). This is actually the expected behaviour I guess. But why does this only happen for a dict and not for a list or torch.tensor?

if __name__ == "__main__":
    a = datasetTest(torch.randn(5,2))
    print(len(a))
    print('++++++++++++')
    for i in range(len(a)):
        print(i)
        print(a[i])
    print('++++++++++++')
    print(list(a))

    print('++++++++++++')
    b = datasetTest({0: 12, 1:35, 2:99, 3:27, 4:33})
    print(len(b))
    print('++++++++++++')
    for i in range(len(b)):
        print(i)
        print(b[i])
    print('++++++++++++')
    print(list(b))

You could try out that snippet of code if you want to understand better what I have observed.

My questions are:

1.) Why does list(iterable) iterate until (including) the len(iterable)? A for loop doesnt do that.

2.) In case of a torch.tensor or a list passed as data X: Why does it not throw an error even when calling the getitem method for the index len(datasetTestInstance) which should actually be out of range since it is not defined as an index in the tensor/list? Or, in other words, when having reached the index len(datasetTestInstance) and then going into the getitem method, what happens exactly? It obviously doesnt make the call 'y = self.X[x]' anymore (otherwiese there would be an IndexError) but it DOES enter the getitem method which I can see as it prints the index x from within the getitem method. So what happens in that method? And why does it behave different depending on whether having a torch.tensor/list or a dict?


title: Using CNN with Dataset that has different depths between volumes

body:
I am working with Medical Images, where I have 130 Patient Volumes, each volume consists of N number of  DICOM Images/slices.

The problem is that between the volumes the the number of slices N, varies.

Majority, 50% of volumes have 20 Slices, rest varies by 3 or 4 slices, some even more than 10 slices (so much so that interpolation to make number of slices equal between volumes is not possible)

I am able to use Conv3d for volumes where the depth N (number of slices) is same between volumes, but I have to make use of entire data set for the classification task. So how do I incorporate entire dataset and feed it to my network model ?


title: PyTorch : Different output on different cuda device for FasterRCNN/MaskRCNN

body:
I am trying out pretrained faster-rcnn model for object detection in PyTorch and observed a weird behavior on executing the following code on different cuda devices.

import io
import torch
import torchvision.transforms as transforms
from torchvision.models.detection.faster_rcnn import FasterRCNN
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone
from PIL import Image
from torch.autograd import Variable

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
backbone = resnet_fpn_backbone('resnet50', True)
model = FasterRCNN(backbone, num_classes=91)
state_dict = torch.load('/home/ubuntu/state_dicts/fasterrcnn_resnet50_fpn_coco-258fb6c6.pth')
model.load_state_dict(state_dict)
model.to(device)
model.eval()

def pre_process(image_bytes):
    my_preprocess = transforms.Compose([transforms.ToTensor()])
    image = Image.open(io.BytesIO(image_bytes))
    image = my_preprocess(image)
    return image

def get_prediction(image_bytes, threshold=0.5):
    tensor = pre_process(image_bytes=image_bytes)
    tensor = Variable(tensor).to(device)
    pred = model([tensor])
    print(pred)

with open("/home/ubuntu/persons.jpg", 'rb') as f:
    image_bytes = f.read()
    get_prediction(image_bytes=image_bytes)

The above code returns different output for device cuda:0 and cuda:1. Can someone help understand this difference in output for the same model on different cuda devices on the same machine.

On using cuda:0 it returns same response as it does on CPU.

Observed same behavior while using MaskRCNN model.

Output on cuda:0

[{'boxes': tensor([[167.4223,  57.0383, 301.3054, 436.6868],
        [ 89.6149,  64.8980, 191.4021, 446.6606],
        [362.3454, 161.9877, 515.5366, 385.2343],
        [ 67.3742, 277.6379, 111.6810, 400.2647],
        [228.7159, 145.8775, 303.5066, 231.1051],
        [379.4247, 259.9776, 419.0149, 317.9510],
        [517.9014, 149.5500, 636.5953, 365.5251],
        [268.9992, 217.2433, 423.9517, 390.4785],
        [539.6832, 157.8171, 616.1689, 253.0961],
        [477.1378, 147.9255, 611.0255, 297.9276],
        [286.6689, 216.3575, 550.4538, 383.1956],
        [627.4468, 177.1990, 640.0000, 247.3514],
        [ 88.3993, 226.4796, 560.9189, 421.6618],
        [406.9602, 261.8285, 453.7620, 357.5365],
        [451.3659, 207.4905, 504.6570, 287.6619],
        [454.3897, 207.9612, 487.7692, 270.3133],
        [451.8828, 208.3855, 631.0622, 355.3239],
        [497.1180, 289.9157, 581.5941, 356.1050],
        [600.6650, 183.4176, 621.5589, 250.3380],
        [559.7050, 202.6747, 608.1462, 250.1502],
        [375.3307, 245.6641, 444.8958, 333.0625],
        [453.1024, 210.8463, 553.8406, 296.7747],
        [555.2745, 199.9524, 611.2347, 250.5636],
        [359.7946, 219.5903, 425.5572, 316.5619],
        [476.7842, 249.0592, 583.8101, 354.6469],
        [ 71.4854, 333.2897, 108.0255, 399.1010],
        [207.6522, 121.4260, 301.1808, 251.5350],
        [550.4424, 175.4845, 621.4010, 317.4897],
        [445.1313, 209.7148, 519.7682, 331.3234],
        [523.6974, 193.5186, 548.5457, 234.6627],
        [449.0608, 229.3627, 572.3047, 293.8238],
        [348.8312, 185.0679, 620.9442, 368.1201],
        [578.4594, 232.6871, 586.2761, 246.6013],
        [359.9344, 166.1812, 502.6697, 287.2637],
        [ 43.1700, 244.8350, 407.5768, 394.7983],
        [115.0793, 126.5799, 177.2827, 198.4358],
        [476.8102, 147.0127, 566.3655, 260.0383],
        [410.9664, 258.0466, 514.5250, 357.0403],
        [450.8164, 277.2901, 521.0891, 359.8105],
        [ 63.9356, 221.3673, 126.4192, 409.7991],
        [625.5704, 189.2636, 640.0000, 256.4739],
        [  1.7555, 174.2491,  86.2912, 436.6681],
        [ 65.3964, 274.4007, 106.8389, 349.2521],
        [558.3841, 197.9385, 639.8632, 368.0412],
        [193.0894, 164.9078, 599.5771, 384.6865],
        [269.0641, 126.7004, 324.2201, 146.3630],
        [359.1832, 201.2081, 484.3798, 276.5368],
        [580.0465, 231.4633, 593.2866, 247.9024],
        [454.5699, 142.0131, 634.2507, 258.4456],
        [616.1375, 246.1040, 639.7282, 255.8053],
        [309.7035, 151.7276, 518.3733, 249.3150],
        [615.1505, 246.0356, 639.2537, 255.4936],
        [452.0419, 199.0634, 584.8884, 357.6918],
        [270.1078, 216.1271, 408.6000, 395.1962],
        [564.9176, 199.7667, 606.9827, 245.9028],
        [  1.7000, 279.6961,  92.9089, 393.7010],
        [495.4763, 253.3147, 640.0000, 361.1835],
        [452.0239, 208.3828, 502.1486, 285.4540],
        [554.9769, 214.0762, 601.4109, 248.5285],
        [473.0355, 251.5581, 575.2361, 298.9354],
        [383.1731, 259.1596, 418.4447, 312.5125],
        [265.9569, 143.7254, 640.0000, 311.1364],
        [353.1688, 200.4693, 494.6974, 272.1262],
        [229.8953, 142.8851, 254.5031, 226.0164]], device='cuda:0',
       grad_fn=<StackBackward>), 'labels': tensor([ 1,  1,  1, 31, 31, 31,  1, 15,  1,  1, 15,  1, 15, 31, 62, 62, 15, 15,
         1, 18, 31, 62,  1, 31, 15, 31, 31,  1, 31, 32, 15, 15, 77,  1, 15, 27,
         1, 31, 31, 31, 62, 64, 31,  1, 15, 15, 62, 77, 15, 15, 15, 67, 62, 62,
        27, 64, 15, 15, 31, 15, 44, 15, 15, 31], device='cuda:0'), 'scores': tensor([0.9995, 0.9995, 0.9978, 0.9925, 0.9922, 0.9896, 0.9828, 0.9582, 0.8994,
        0.8727, 0.8438, 0.8364, 0.7470, 0.7322, 0.6674, 0.5940, 0.4650, 0.3875,
        0.3826, 0.3792, 0.3722, 0.3720, 0.3480, 0.3407, 0.2381, 0.2210, 0.2163,
        0.2060, 0.1994, 0.1939, 0.1769, 0.1652, 0.1589, 0.1521, 0.1516, 0.1499,
        0.1495, 0.1419, 0.1248, 0.1184, 0.1124, 0.1098, 0.1077, 0.1059, 0.1035,
        0.0986, 0.0975, 0.0910, 0.0909, 0.0882, 0.0863, 0.0802, 0.0733, 0.0709,
        0.0699, 0.0668, 0.0662, 0.0651, 0.0600, 0.0586, 0.0578, 0.0578, 0.0577,
        0.0540], device='cuda:0', grad_fn=<IndexBackward>)}]

Output on cuda:1 (or any GPU other then cuda:0)

[{'boxes': tensor([[218.7705,   0.0000, 640.0000, 491.0000]], device='cuda:1',
       grad_fn=<StackBackward>), 'labels': tensor([77], device='cuda:1'), 'scores': tensor([0.0646], device='cuda:1', grad_fn=<IndexBackward>)}]


title: Unable to reproduce results using PyTorch in Colab

body:
I’m running a model written in PyTorch in Colab on P100 GPU and I have set all the seeds using the following code

torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(seed)
random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)

Apart from the above, I have set seed value wherever it is required in the code.

I’m able to get reproducible results only when I run the code in the same session. When I close the session and then rerun the code in a new session, results are varying.
Is it not possible to get reproducible results using PyTorch in Colab?


title: How does one Dockerize and deploy a Pytorch deep learning model via SSH?

body:
I'm having trouble understanding how to deploy a PyTorch docker image on a remote machine that has an Nvidia GPU. I'm relatively unfamiliar with Docker in general, and this is my first time using Docker for the purposes of deep learning.
I am able to train my models on my local machine without problems via a trivial command-line call to python main.py, which is what I'm familiar with. But now that I have to SSH into this machine I'm required to use Docker.
So far I have pulled a few images without really knowing what I'm doing. So, here's what got me the furthest so far:
First, I transferred my code to the machine via a GitLab repository -- nothing fancy. I also transferred some large datasets via SSH into the parent directory of my GitLab repository. I.e., it looks something like this:
DeepLearningProject/
├── Data_folder/
│   ├── datasets/
│   ├── results/
│   └── etc...
└── MyGitFolder/
    ├── lots_of_python_files.py
    └── main.py

Next, I pulled the following image via docker pull anibali/pytorch:1.4.0-cuda10.1 as the version numbers satisfy our requirements.
Here's where the guessing game got out of hand. So, I have an image with Pytorch and all the Nvidia stuff, but I'm still missing my own python package dependencies. I'm not sure where the installation hereof takes place, for example in a requirements.txt file. Thus, python reports missing packages when I run the following, as per a tutorial I found online:
docker run --rm -it --init \
  --gpus=all \
  --ipc=host \
  --user="$(id -u):$(id -g)" \
  --volume="$PWD:/app" \
  anibali/pytorch:1.4.0-cuda10.1 python3 main.py

So, I thought I could perhaps enter a bash terminal by excluding the python3 main.py part of the above command. This just opens the python3 interpreter interface. I assume this is due to the image's Dockerfile(?) ending on CMD ["python3"], and I'm unsure as to how I can install my requirements through this.
As requested, please find an example of the error message for the missing python packages below:
Traceback (most recent call last):
  File "main.py", line 16, in <module>
    from model import (generate_model, load_pretrained_model, make_data_parallel,
  File "/app/model.py", line 4, in <module>
    from models import resnet, resnet2p1d, pre_act_resnet, wide_resnet, resnext, densenet
  File "/app/models/resnext.py", line 9, in <module>
    from utils import partialclass
  File "/app/utils.py", line 7, in <module>
    from sklearn.metrics import precision_recall_fscore_support
ModuleNotFoundError: No module named 'sklearn'

I assume this image does not have sklearn in its requirements.txt. Is there a way I could 'pipe' in my own requirements.txt when executing docker run? I believe this to be my main issue here.
Any advice would be appreciated.


I have been working with a PyTorch neural network for a while now. I decided I wanted to add a permutation feature importance scorer, and this started to cause some issues.
I get" TypeError: If no scoring is specified, the estimator passed should have a 'score' method. The estimator <class 'skorch.net.NeuralNet'>[uninitialized](
module=<class 'main.run..MultiLayerPredictor'>,
) does not. " - error message. Here's my code:
class MultiLayerPredictor(torch.nn.Module):
    def __init__(self, input_shape=9152, output_shape=1, hidden_dim=1024, **kwargs):
        super().__init__()
        self.fc1 = torch.nn.Linear(in_features=input_shape, out_features=hidden_dim)
        self.fc2 = torch.nn.Linear(in_features=hidden_dim, out_features=hidden_dim)
        self.fc3 = torch.nn.Linear(in_features=hidden_dim, out_features=output_shape)

    def forward(self, x):
        l1 = torch.relu(self.fc1(x))
        l2 = torch.relu(self.fc2(l1))
        return torch.sigmoid(self.fc3(l2)).reshape(-1)

print("Moving to wrapping the neural net")
net = NeuralNet(
    MultiLayerPredictor,
    criterion=nn.MSELoss,
    max_epochs=10,
    optimizer=optim.Adam,
    lr=0.1,
    iterator_train__shuffle=True
)

print("Moving to finding optimal hyperparameters")

lr = (10**np.random.uniform(-5,-2.5,1000)).tolist()
params = {
    'optimizer__lr': lr,
    'max_epochs':[300,400,500],
    'module__num_units': [14,20,28,36,42],
    'module__drop' : [0,.1,.2,.3,.4]
}

gs = RandomizedSearchCV(net,params,refit=True,cv=3,scoring='neg_mean_squared_error',n_iter=100)
gs.fit(X_train_scaled,y_train);

def report(results, n_top=3):
    for i in range(1, n_top + 1):
        candidates = np.flatnonzero(results['rank_test_score'] == i)
    for candidate in candidates:
        print("Model with rank: {0}".format(i))
        print("Mean validation score: {0:.3f} (std: {1:.3f})".format(
              results['mean_test_score'][candidate],
              results['std_test_score'][candidate]))
        print("Parameters: {0}".format(results['params'][candidate]))
        print("")

print(report(gs.cv_results_,10))

epochs = [i for i in range(len(gs.best_estimator_.history))]
train_loss = gs.best_estimator_.history[:,'train_loss']
valid_loss = gs.best_estimator_.history[:,'valid_loss']

plt.plot(epochs,train_loss,'g-');
plt.plot(epochs,valid_loss,'r-');
plt.title('Training Loss Curves');
plt.xlabel('Epochs');
plt.ylabel('Mean Squared Error');
plt.legend(['Train','Validation']);
plt.show()

r = permutation_importance(net, X_test, y_test, n_repeats=30,random_state=0)

for i in r.importances_mean.argsort()[::-1]:
    if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
        print(f"{metabolites.feature_names[i]:<8}"
              f"{r.importances_mean[i]:.3f}"
              f" +/- {r.importances_std[i]:.3f}")

y_pred_acc = gs.predict(X_test)
print('Accuracy : ' + str(accuracy_score(y_test,y_pred_acc)))

Stacktrace would point that the error stems from the line where I set the permutation importance. How can I fix this?
Full stacktrace:
*Traceback (most recent call last):
  File "//ad..fi/home/h//Desktop/neuralnet/neuralnet_wrapped.py", line 141, in <module>
    run()
  File "//ad..fi/home/h//Desktop/neuralnet/neuralnet_wrapped.py", line 119, in run
    r = permutation_importance(net, X_test, y_test,
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\utils\validation.py", line 73, in inner_f
    return f(**kwargs)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\inspection\_permutation_importance.py", line 132, in permutation_importance
    scorer = check_scoring(estimator, scoring=scoring)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\utils\validation.py", line 73, in inner_f
    return f(**kwargs)
  File "C:\Users\\AppData\Roaming\Python\Python38\site-packages\sklearn\metrics\_scorer.py", line 425, in check_scoring
    raise TypeError(
TypeError: If no scoring is specified, the estimator passed should have a 'score' method. The estimator <class 'skorch.net.NeuralNet'>[uninitialized](
  module=<class '__main__.run.<locals>.MultiLayerPredictor'>,
) does not.*


I am trying to use a customized loss function for my NN. I've implemented all operations in torch and I have complex numbers among my data.
I get the error while training a NN:
RuntimeError: _th_addr_out not supported on CPUType for ComplexFloat

Do you know any possible solution to deal with it?


title: How to use Pytorch to create a custom EfficientNet with the last layer written correctly

body:
I have a classification problem to predict 8 classes for example, I am using EfficientNetB3 in pytorch from here. However, I got confused on whether my custom class is correctly written. I think I want to strip the last layer of the pre-trained model to suit the 8 outputs right? Did I do it correctly? Because when I print y_preds = model(images) in my DataLoader, it seems to give me 1536 predictions. Is this an expected behavior?
!pip install geffnet
import geffnet

class EfficientNet(nn.Module):
    def __init__(self, config):
        super().__init__()
        self.config = config
        self.model = geffnet.create_model(config.effnet, pretrained=True)
        n_features = self.model.classifier.in_features
        # does the name fc matter?
        self.fc = nn.Linear(n_features, config.num_classes)
        self.model.classifier = nn.Identity()

    def extract(self, x):
        x = self.model(x)
        return x

    def forward(self, x):
        x = self.extract(x).squeeze(-1).squeeze(-1)
        return x

model = EfficientNet(config=config)
if torch.cuda.is_available():
    model.cuda()

Sample code for printing y_pred:
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
for step, (images, labels) in enumerate(sample_loader):
    images = images.to(device)
    labels = labels.to(device)
    batch_size = images.shape[0]
    y_preds = model(images)
    print('The predictions of the 4 images is as follows\n', y_preds)
    break


title: what does it mean when kernel depth=1 in conv3d pytorch

body:
I would like to understand the difference between conv2d and conv3d in PyTorch.
What is the difference between:
conv3d(in, out, kernel_size(1,3,3))

and
conv2d(in,out,kernel_size(3,3))

I checked the official documentation but I couldn't quite understand the difference between the two. Should conv3d in this case be the same as conv2d since the depth is 1? Any help would be appreciated.


title: RuntimeError: "upsample_bilinear2d_channels_last" not implemented for 'Long' after converting np array to list

body:
I am trying to send image data across modules.
The following works fine:
def process_image(pdf_path, page_dimensions):
  pdf_path = get_pdf(pdf_path, False)
  pdf_name = os.path.basename(pdf_path)
  with tempfile.TemporaryDirectory() as path:
    conversion_time = time.time()
    chart_images = convert_from_path(
          pdf_path=pdf_path,
          dpi=300,
          fmt="jpg",
          output_file=os.path.basename(str(pdf_path)).split(".")[0],
          output_folder=path,
          use_pdftocairo=False,
          paths_only=True,
          thread_count=8,
      )
    pg_dim, pg_image_path = page_dimensions[0], chart_images[0]
    pg_image = cv2.imread(pg_image_path, cv2.IMREAD_UNCHANGED)
    pg_image = cv2.resize(pg_image, (pg_dim[1], pg_dim[2]))
  return pg_image

The result of this function (pg_image) works fine when taken as input to my Detectron2 model.
However, when I send pg_image.tolist() and convert back to np array on receiving (np.array(pg_image)) and send it to my Detectron2 model, I keep getting the following error:
RuntimeError: "upsample_bilinear2d_channels_last" not implemented for 'Long'

So, if converting the np array to a list and converting back changing the data?


title: How can I make this PyTorch heatmap function faster and more efficient?

body:
I have this function that creates a sort if heatmap for 2d tensors, but it's painfully slow when using larger tensor inputs. How can I speed it up and make it more efficient?
import torch
import numpy as np
import matplotlib.pyplot as plt

def heatmap(
    tensor: torch.Tensor,
) -> torch.Tensor:
    assert tensor.dim() == 2

    def color_tensor(x: torch.Tensor) -> torch.Tensor:
        if x < 0:
            x = -x
            if x < 0.5:
                x = x * 2
                return (1 - x) * torch.tensor(
                    [0.9686, 0.9686, 0.9686]
                ) + x * torch.tensor([0.5725, 0.7725, 0.8706])
            else:
                x = (x - 0.5) * 2
                return (1 - x) * torch.tensor(
                    [0.5725, 0.7725, 0.8706]
                ) + x * torch.tensor([0.0196, 0.4431, 0.6902])
        else:
            if x < 0.5:
                x = x * 2
                return (1 - x) * torch.tensor(
                    [0.9686, 0.9686, 0.9686]
                ) + x * torch.tensor([0.9569, 0.6471, 0.5098])
            else:
                x = (x - 0.5) * 2
                return (1 - x) * torch.tensor(
                    [0.9569, 0.6471, 0.5098]
                ) + x * torch.tensor([0.7922, 0.0000, 0.1255])

    return torch.stack(
        [torch.stack([color_tensor(x) for x in t]) for t in tensor]
    ).permute(2, 0, 1)

x = torch.randn(3,3)
x = x / x.max()
x_out = heatmap(x)

x_out = (x_out.permute(1, 2, 0) * 255).numpy()
plt.imshow(x_out.astype(np.uint8))
plt.axis("off")
plt.show()

An example of the output:


title: Conditionally apply tensor operations in PyTorch

body:
I know PyTorch doesn't have a map-like function to apply a function to each element of a tensor. So, could I do something like the following without a map-like function in PyTorch?
if tensor_a * tensor_b.matmul(tensor_c) < 1:
    return -tensor_a*tensor_b
else:
    return 0

This would work if the tensors were 1D. However, I need this to work when tensor_b is 2D (tensor_a needs to be unsqueezed in the return statement). This means a 2D tensor should be returned where some of the rows will be 0 vectors.
Happy to use the latest features of the most recent Python version.


title: How to find partial derivative in pytorch

body:
I have a model u(x,t) with layers 2X50, then 50X50, and 50X1.
I train the model with input x,t of size [100,2]. In the final layer I get u and now I want to differentiate it w.r.t to x and t and double differentiate w.r.t to x.  How do I do this in PyTorch?


title: Dice and CE loss not training network together

body:
I am training a segmentation network on the Kaggle Salt challenge. My dice and ce decrease, but then suddenly dice increases and CE jumps up a bit, this keeps happening to dice. I have been trying all day to fix this but can’t get my code to run. I am running on only 10 data points to overfit my data but it just is not happening. Any help would be greatly appreciated.
Plots of dice(top) and CE:
Loss curve
Heres my dice and train:
def dice(input, target,weights=torch.tensor([1,1]).float().cuda()):
    smooth=.001

    dummy=np.zeros([batch_size,2,100,100]) # create dummy to one hot encode target for weighted dice
    dummy[:,0,:,:][target==0]=1 # background class is 0
    dummy[:,1,:,:][target==1]=1 # salt class is 1

    target=torch.tensor(dummy).float().cuda()

#     print(input.size(),input[:,0,:,:].size())
    input1=input[:,0,:,:].contiguous().view(-1) #flatten both classes seperately
    target1=target[:,0,:,:].contiguous().view(-1)

    input2=input[:,1,:,:].contiguous().view(-1)
    target2=target[:,1,:,:].contiguous().view(-1)

    score1=2*(input1*target1).sum()/(input1.sum()+target1.sum()+smooth) #back
    score2=2*(input2*target2).sum()/(input2.sum()+target2.sum()+smooth) #salt

    score=1-(weights[0]*score1+weights[1]*score2)/2
    if score<0:
        score=score-score

    return(score)
Heres the train:

def train(epoch):
    for idx, batch_data in enumerate(dataloader) :
        x, target=batch_data['image'].float().cuda(),batch_data['label'].float().cuda()

        optimizer.zero_grad()
        output = net(x)
#         print(output.size())
        output.squeeze_(1)

#         print('out',output.size(),target.size())
        bce_loss = criterion(output, target.long())
        lc.append(bce_loss.item())

        dice_loss = dice((output), target)
        ld.append(dice_loss.item())
        loss =  dice_loss + bce_loss
        l.append(loss.item())

        loss.backward()
        optimizer.step()

        print('Epoch {}, loss {}, bce {}, dice {}'.format(
            epoch, sum(l)/len(l), sum(lc)/len(lc) , sum(ld)/len(ld) ))

Heres the rest of the code( I downed from gaggle kernel): https://github.com/bluesky314/Salt-Segmentation/blob/master/kernel-2.ipynb 1 (the training showed here is when I ran that cell(14) a second time so the ups and downs don’t appear but can be seen in the plot)
dataset=DatasetSalt(limit_paths=10) just limits the dataset to any number by only taking the top paths to get the images from
Would really appreciate any help, have been literally struggling on this for 8+ hrs


title: SageMaker Neo PyTorch 1.0.0

body:
I've updated the torch version in my SageMaker pytorch_36 kernel to torch version 1.0.0. I then tried running the example notebook pytorch_torchvision_neo.ipynb, also changing the framework_version to 1.0.0. Neo compilation then fails.

Any idea why it isn't working with 1.0.0? The console error message actually tells me to make sure I'm using 1.0.0, but the example notebook seems to only work with 0.4.0.


title: PyTorch Image transformation with gridsampler, weird behaviour while optimizing grid

body:
Im trying to get the per-pixel transformation to fit one image (+ background) to a result.
Background Image + Input image should be transformed into desired result

to achieve this i am using the PyTorch gridsampler and autograd to optimize the grid.
The transformed input will be added to the unchanged background.

ToTensor = torchvision.transforms.ToTensor()
FromTensor = torchvision.transforms.ToPILImage()

backround= ToTensor(Image.open("backround.png"))
pic = ToTensor(Image.open("pic.png"))
goal = ToTensor(Image.open("goal.png"))

empty = empty.expand(1, 3, empty.size()[1], empty.size()[2])
pic = pic.expand(1, 3, pic.size()[1], pic.size()[2])
goal = goal.expand(1, 3, goal.size()[1], goal.size()[2])

def createIdentityGrid(w, h):
    grid = torch.zeros(1, w, h, 2);
    for x in range(w):
        for y in range(h):
            grid[0][x][y][1] = 2 / w * (0.5 + x) - 1
            grid[0][x][y][0] = 2 / h * (0.5 + y) - 1
    return grid

w = 256; h=256 #hardcoded imagesize

grid = createIdentityGrid(w, h)
grid.requires_grad = True

for i in range(300):
    goal_pred = torch.nn.functional.grid_sample(pic, grid)[0]
    goal_pred = (empty + 0.75 * goal_pred).clamp(min=0, max=1)
    out = goal_pred

    loss = (goal_pred - goal).pow(2).sum()
    loss.backward()

    with torch.no_grad():
        grid -= grid.grad * (1e-2)
        grid.grad.zero_()

FromTensor(out[0]).show()

these are the result:

After 0 steps
After 300 steps
After 600 steps
After 900 steps
After 1200 steps

It is actualy working with this this easy example, but I observe some weird behaviour.
The grid just starts to change on one side.
Why is this so, and why doesn't the complete grid change instantly?
Is there some obvious part I am missing?


title: Write validation metrics on pytorch-ignite ProgressBar on same line

body:
Is there a way to write validation metrics at the end of the progress bar using pytorch-ignite like in keras?

Right now, I'm using ProgressBar.log_message (example below) to print the relevant metrics, but doing so prints on a new line at the end of each epoch. I'd like the messages to print at the end of the bar, without a newline character.

@trainer.on(Events.EPOCH_COMPLETED)
def run_evaluator(trainer):
    evaluator.run(val_loader)
    val_metrics = evaluator.state.metrics

    # ProgBar
    pbar.log_message(f"Val_loss: {val_metrics['Val_loss']:.2f}")
    pbar.log_message(f"RMSE: {val_metrics['RMSE']:.2f}")

To further clarify, here's the output I'm getting now:

Epoch [1/3]: [3/3] 100%|██████████| [00:00/00:00], Train_loss=1.59e+4
Val_loss: 26333.96
RMSE: 72572.66
Epoch [2/3]: [3/3] 100%|██████████| [00:00/00:00], Train_loss=2.35e+4
Val_loss: 25508.46
RMSE: 71426.14
Epoch [3/3]: [3/3] 100%|██████████| [00:00/00:00], Train_loss=2.26e+4
Val_loss: 24721.82
RMSE: 70316.17

And here's what I want:

Epoch [1/3]: [3/3] 100%|██████████| [00:00/00:00], Train_loss=1.59e+4  Val_loss: 26333.96  RMSE: 72572.66
Epoch [2/3]: [3/3] 100%|██████████| [00:00/00:00], Train_loss=2.35e+4  Val_loss: 25508.46  RMSE: 71426.14
Epoch [3/3]: [3/3] 100%|██████████| [00:00/00:00], Train_loss=2.26e+4  Val_loss: 24721.82  RMSE: 70316.17


title: How to find the arguments for torch.nn.conv_transpose2d and max_unpool2d?

body:
In a given forward function of a convolutional layer I have:

def forward(self, x):
  c = torch.nn.functional.conv2d(x)
  a,i = torch.maxpool2d(c)
  o = torch.relu(a)
  return a,i

I'm looking to undo this function with torch.nn.functional.max_unpool2d and torch.nn.functional.conv_transpose2d.

So far I have (not entirely sure about this either):

a = torch.relu(o)
c = torch.nn.functional.max_unpool2d(a, i, kernel_size=c.size[1])
x = torch.nn.functional.conv_transpose2d(c,..)

My questions:

How come the call to conv2d doesn't require any other arguments than the one given?
How do I get the parameters for conv_transpose2d from only a and i (e. g. from their sizes)? Or is there some other way I'm not seeing?
Is there a default kernel_size conv2d uses?


title: Convert from .npy to MLMultiArray for CoreML prediction in swift

body: I have exported a PyTorch model to CoreML and want to do inference in swift. I have my input data stored on disk as a 2D float32 numpy ndarray .npy and need load into a MLMultiArray in swift. Is there a convenient way to do this?


title: FastAI Segmentation Problem: Updating Model Weights with custom Item- and LabelList

body:
This might be a stupid question as nobody on fastai is trying to answer it. If it is one you can go ahead and tell me but also please tell me the answer, because right now I am completely lost.

I am currently working on a U-Net model for the segmentation of cells in microscopy images. Due to class imbalances and to amplify the importance of cell boundaries, I calculated a pixelwise weightmap for each image that I pass into fastai. Therefore I created a new ItemBase class to save labels and weights together:

class WeightedLabels(ItemBase):
"""
Custom ItemBase to store and process labels and pixelwise weights together.
Also handling the target_size of the labels.
"""

def __init__(self, lbl: Image, wgt: Image, target_size: Tuple = None):
    self.lbl, self.wgt = lbl, wgt
    self.obj, self.data = (lbl, wgt), [lbl.data, wgt.data]

    self.target_size = target_size

...

I use extensive augmentation, like elastic deformation, mirroring and rotations on both weights and labels, as well as the original image. I determine the Loss with a custom Cross-entropy loss function that uses the weights to get the weighted loss for each pixel and averages them.

My problem is, that I do not get a very good performace. I have the feeling that might be because of fastai trying to predict the weights as well. My questions are:

Am I right to assume my model tries to predict both?
If so, how do I tell the learner what to use for updating the layers and to only predict part of my labels, while still applying augmentation to both?

Here's the code for how I implemented my custom LabelList and my custom ItemList:

class CustomSegmentationLabelList(ImageList):
"'Item List' suitable for WeightedLabels containing labels and pixelweights"
_processor = vision.data.SegmentationProcessor

def __init__(self,
             items: Iterator,
             wghts = None,
             classes: Collection = None,
             target_size: Tuple = None,
             loss_func=CrossEntropyFlat(axis=1),
             **kwargs):

    super().__init__(items, **kwargs)
    self.copy_new.append('classes')
    self.copy_new.append('wghts')
    self.classes, self.loss_func, self.wghts = classes, loss_func, wghts
    self.target_size = target_size

def open(self, fn):
    res = io.imread(fn)
    res = pil2tensor(res, np.float32)
    return Image(res)

def get(self, i):
    fn = super().get(i)
    wt = self.wghts[i]
    return WeightedLabels(fn, self.open(wt), self.target_size)

def reconstruct(self, t: Tensor):
    return WeightedLabels(Image(t[0]), Image(t[1]), self.target_size)

class CustomSegmentationItemList(ImageList):
"'ItemList' suitable for segmentation with pixelwise weighted loss"
_label_cls, _square_show_res = CustomSegmentationLabelList, False

def label_from_funcs(self, get_labels: Callable, get_weights: Callable,
                     label_cls: Callable = None, classes=None,
                     target_size: Tuple = None, **kwargs) -> 'LabelList':
    "Get weights and labels from two functions. Saves them in a CustomSegmentationLabelList"
    kwargs = {}
    wghts = [get_weights(o) for o in self.items]
    labels = [get_labels(o) for o in self.items]

    if target_size:
        print(
            f'Masks will be cropped to {target_size}. Choose \'target_size \\= None \' to keep initial size.')
    else:
        print(f'Masks will not be cropped.')

    y = CustomSegmentationLabelList(
        labels, wghts, classes, target_size, path=self.path)
    res = self._label_list(x=self, y=y)
    return res

Also here the part, where I initiate my databunch object:

data = (CustomSegmentationItemList.from_df(img_df,IMG_PATH, convert_mode=IMAGE_TYPE)
  .split_by_rand_pct(valid_pct=(1/N_SPLITS),seed=SEED)
  .label_from_funcs(get_labels, get_weights, target_size=MASK_SHAPE, classes = array(['background','cell']))
  .transform(tfms=tfms, tfm_y=True)
  .databunch(bs=BATCH_SIZE))

I use the regular learner, where I pass in my U-Net model, the data, my loss function and some additional arguments that shouldn't really matter here. When trying to apply my model, after training it, it gives me two identical output tensors. I assume that one is probably for the labels and the other for the weights. Both have the following dimensions: (WxHxC). I do not understand why this is happening, because my model is supposed to only have one output in form of (WxHxC). If this happens during prediction, this probably also happens during training. How can I overcome this?


title: In pytorch, I want to save the the output in every epoch for late caculation. But it leads to OUT OF MEMORY ERROR after several epochs,

body:
In pytorch, I want to save the output in every epoch for late caculation. But it leads to OUT OF MEMORY ERROR after several epochs. The code is like below:
    L=[]
    optimizer.zero_grad()
    for i, (input, target) in enumerate(train_loader):
        output = model(input)
        L.append(output)
    *** updata my model to minimize a loss function. List L will be used here.

I know the reason is because pytorch save all computation graphs from every epoch.
But the loss function can only be calculated after obtaining all of the prediction results
Is there a way I can train my model?


title: How to use a for loop length as an optimization parameter in Pytorch

body:
I am trying to register my loop's length as an optimization parameter so that the optimizer will automatically adjust the length of the loop.
Here is an example code (the real problem is more complex but you will get the idea):
import torch

positive_iter = torch.tensor([10.0], requires_grad=True)
negative_iter = torch.tensor([20.0], requires_grad=True)

optimizer = torch.optim.Adam([positive_iter, negative_iter], lr=0.02, betas=(0.5, 0.999))

for i in range(100):
    loss = torch.tensor([0.0], requires_grad=True)
    for g in range(int(positive_iter)):
        loss = loss + torch.rand(1)
    for d in range(int(negative_iter)):
        loss = loss - torch.rand(1) * 2

    loss = torch.abs(loss)
    loss.backward()
    optimizer.step()
    print(i, loss.item(), positive_iter.item(), negative_iter.item())

The optimization does not seem to work, here is the output:

0 19.467784881591797 10.0 20.0
1 14.334418296813965 10.0 20.0
2 13.515042304992676 10.0 20.0
3 13.477707862854004 10.0 20.0
4 15.240434646606445 10.0 20.0
5 18.45014190673828 10.0 20.0
6 18.557266235351562 10.0 20.0
7 16.325769424438477 10.0 20.0
8 13.95105266571045 10.0 20.0
9 12.435094833374023 10.0 20.0
10 13.70322322845459 10.0 20.0
11 10.128765106201172 10.0 20.0
12 16.986034393310547 10.0 20.0
13 15.652003288269043 10.0 20.0
14 10.300052642822266 10.0 20.0
15 18.038368225097656 10.0 20.0
16 11.830389022827148 10.0 20.0
17 14.917057037353516 10.0 20.0
18 18.603071212768555 10.0 20.0
19 17.595298767089844 10.0 20.0
20 17.17181968688965 10.0 20.0
21 14.548274993896484 10.0 20.0
22 18.839675903320312 10.0 20.0
23 13.375761032104492 10.0 20.0
24 14.045333862304688 10.0 20.0
25 13.088285446166992 10.0 20.0
26 15.019135475158691 10.0 20.0
27 16.992284774780273 10.0 20.0
28 13.883159637451172 10.0 20.0
29 12.695013999938965 10.0 20.0
30 17.23816680908203 10.0 20.0
...continued

Could you advise on how to make the for loop length an optimization parameter.
Thank you


title: Python block/halt on importing torch

body:
I have developed a deep-learning object-detect program based on pytorch and it works very well. Today I deploy this program on a PC, everything goes well, but the program cannot be launched. Debug and find out that, the program blocks, or halts, on importing pytorch.
Simply start a python prompt, type import torch and the prompt blocks. top command shows that CPU/memory usage is very low. Press ctrl-c cannot stop the prompt. While other library importing is fine. I have tried pycrypto and the one I wrote myself, all work but pytorch cannot.
I have deployed more than 100 times, but never meet this situation. I also tried to reinstall pytorch, from 1.6 to 1.4, torchvision from 0.7 to 0.5, still not work. No error printed, no complain shown.
Environment:

OS: centos 7.4
CUDA: 10.0
NVIDIA driver: 440.82
GPU: GTX 1660
python: 3.6

pytorch version: 1.6 and 1.4.
Any information are welcome, thanks in advance.

Edit:
According to Szymon's idea, running python3 foo.py, which with only import torch in it, and press ctrl-c, the prompt prints:
Traceback (most recent call last):
  File "foo.py", line 1, in <module>
    import torch
  File "/usr/local/lib64/python3.6/site-packages/torch/__init__.py", line 48, in <module>
    if platform.system() == 'Windows':
  File "/usr/lib64/python3.6/platform.py", line 1068, in system
    return uname().system
  File "/usr/lib64/python3.6/platform.py", line 1034, in uname
    processor = _syscmd_uname('-p', '')
  File "/usr/lib64/python3.6/platform.py", line 788, in _syscmd_uname
    f = os.popen('uname %s 2> %s' % (option, DEV_NULL))
  File "/usr/lib64/python3.6/os.py", line 980, in popen
    bufsize=buffering)
  File "/usr/lib64/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/usr/lib64/python3.6/subprocess.py", line 1318, in _execute_child
    part = os.read(errpipe_read, 50000)

It seems that python hangs on running uname <option>, so I tried uname -a 2> /dev/null on command line, it returned immediately and nothing strange. Also created a file named bar.py with the content below:
import platform
print(platform.system())

and run it with python3, works well, printed 'Linux'. I don't think it is the reason, maybe just a coincidence?
I also tried more times, other situation looked like running import torch in python prompt, could not kill the process and nothing printed. Once it printed Soft lock up on CPU#4, I thought it was caused by the test process in the last disconnected session.


title: Deep learning chatbot specific Index error list index out of range

body:
I am trying to follow a tutorial on how to make a deeplearning chatbot with pytorch. However, this code is quite complex for me and it has stopped with a "IndexError: list index out of range". I looked the error up and get the gist of what it usually means, but seeing as this code is very complex for me I can't figure out how to solve the error.
this is the source tutorial: [https://colab.research.google.com/github/pytorch/tutorials/blob/gh-pages/_downloads/chatbot_tutorial.ipynb#scrollTo=LTzdbPF-OBL9][1]
Line 198 seems to be causing the error
return len(p[0].split(' ')) < MAX_LENGTH and len(p[1].split(' ')) < MAX_LENGTH

This is the error log
Start preparing training data ...
Reading lines...
Traceback (most recent call last):
  File "D:\Documents\Python\python pycharm files\pythonProject4\3.9 Chatbot.py", line 221, in <module>
    voc, pairs = loadPrepareData(corpus, corpus_name, datafile, save_dir)
  File "D:\Documents\Python\python pycharm files\pythonProject4\3.9 Chatbot.py", line 209, in loadPrepareData
    pairs = filterPairs(pairs)
  File "D:\Documents\Python\python pycharm files\pythonProject4\3.9 Chatbot.py", line 202, in filterPairs
    return [pair for pair in pairs if filterPair(pair)]
  File "D:\Documents\Python\python pycharm files\pythonProject4\3.9 Chatbot.py", line 202, in <listcomp>
    return [pair for pair in pairs if filterPair(pair)]
  File "D:\Documents\Python\python pycharm files\pythonProject4\3.9 Chatbot.py", line 198, in filterPair
    return len(p[0].split(' ')) < MAX_LENGTH and len(p[1].split(' ')) < MAX_LENGTH
IndexError: list index out of range
Read 442563 sentence pairs

Process finished with exit code 1

And this is my code copied from my pycharm up to the block with the error. Seeing as its a huge code I could not copy the entire code. The rest of the code can be found in the github source link above.
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals

import torch
from torch.jit import script, trace
import torch.nn as nn
from torch import optim
import torch.nn.functional as F
import csv
import random
import re
import os
import unicodedata
import codecs
from io import open
import itertools
import math

USE_CUDA = torch.cuda.is_available()
device = torch.device("cuda" if USE_CUDA else "cpu")

corpus_name = "cornell movie-dialogs corpus"
corpus = os.path.join("D:\Documents\Python\intents", corpus_name)

def printLines(file, n=10):
    with open(file, 'rb') as datafile:
        lines = datafile.readlines()
    for line in lines[:n]:
        print(line)

printLines(os.path.join(corpus, "movie_lines.txt"))

# Splits each line of the file into a dictionary of fields
def loadLines(fileName, fields):
    lines = {}
    with open(fileName, 'r', encoding='iso-8859-1') as f:
        for line in f:
            values = line.split(" +++$+++ ")
            # Extract fields
            lineObj = {}
            for i, field in enumerate(fields):
                lineObj[field] = values[i]
            lines[lineObj['lineID']] = lineObj
    return lines

# Groups fields of lines from `loadLines` into conversations based on *movie_conversations.txt*
def loadConversations(fileName, lines, fields):
    conversations = []
    with open(fileName, 'r', encoding='iso-8859-1') as f:
        for line in f:
            values = line.split(" +++$+++ ")
            # Extract fields
            convObj = {}
            for i, field in enumerate(fields):
                convObj[field] = values[i]
            # Convert string to list (convObj["utteranceIDs"] == "['L598485', 'L598486', ...]")
            lineIds = eval(convObj["utteranceIDs"])
            # Reassemble lines
            convObj["lines"] = []
            for lineId in lineIds:
                convObj["lines"].append(lines[lineId])
            conversations.append(convObj)
    return conversations

# Extracts pairs of sentences from conversations
def extractSentencePairs(conversations):
    qa_pairs = []
    for conversation in conversations:
        # Iterate over all the lines of the conversation
        for i in range(len(conversation["lines"]) - 1):  # We ignore the last line (no answer for it)
            inputLine = conversation["lines"][i]["text"].strip()
            targetLine = conversation["lines"][i+1]["text"].strip()
            # Filter wrong samples (if one of the lists is empty)
            if inputLine and targetLine:
                qa_pairs.append([inputLine, targetLine])
    return qa_pairs

# Define path to new file
datafile = os.path.join(corpus, "formatted_movie_lines.txt")

delimiter = '\t'
# Unescape the delimiter
delimiter = str(codecs.decode(delimiter, "unicode_escape"))

# Initialize lines dict, conversations list, and field ids
lines = {}
conversations = []
MOVIE_LINES_FIELDS = ["lineID", "characterID", "movieID", "character", "text"]
MOVIE_CONVERSATIONS_FIELDS = ["character1ID", "character2ID", "movieID", "utteranceIDs"]

# Load lines and process conversations
print("\nProcessing corpus...")
lines = loadLines(os.path.join(corpus, "movie_lines.txt"), MOVIE_LINES_FIELDS)
print("\nLoading conversations...")
conversations = loadConversations(os.path.join(corpus, "movie_conversations.txt"),
                                  lines, MOVIE_CONVERSATIONS_FIELDS)

# Write new csv file
print("\nWriting newly formatted file...")
with open(datafile, 'w', encoding='utf-8') as outputfile:
    writer = csv.writer(outputfile, delimiter=delimiter)
    for pair in extractSentencePairs(conversations):
        writer.writerow(pair)

# Print a sample of lines
print("\nSample lines from file:")
printLines(datafile)

# Default word tokens
PAD_token = 0  # Used for padding short sentences
SOS_token = 1  # Start-of-sentence token
EOS_token = 2  # End-of-sentence token

class Voc:
    def __init__(self, name):
        self.name = name
        self.trimmed = False
        self.word2index = {}
        self.word2count = {}
        self.index2word = {PAD_token: "PAD", SOS_token: "SOS", EOS_token: "EOS"}
        self.num_words = 3  # Count SOS, EOS, PAD

    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2index:
            self.word2index[word] = self.num_words
            self.word2count[word] = 1
            self.index2word[self.num_words] = word
            self.num_words += 1
        else:
            self.word2count[word] += 1

    # Remove words below a certain count threshold
    def trim(self, min_count):
        if self.trimmed:
            return
        self.trimmed = True

        keep_words = []

        for k, v in self.word2count.items():
            if v >= min_count:
                keep_words.append(k)

        print('keep_words {} / {} = {:.4f}'.format(
            len(keep_words), len(self.word2index), len(keep_words) / len(self.word2index)
        ))

        # Reinitialize dictionaries
        self.word2index = {}
        self.word2count = {}
        self.index2word = {PAD_token: "PAD", SOS_token: "SOS", EOS_token: "EOS"}
        self.num_words = 3 # Count default tokens

        for word in keep_words:
            self.addWord(word)

MAX_LENGTH = 10  # Maximum sentence length to consider

# Turn a Unicode string to plain ASCII, thanks to
# http://stackoverflow.com/a/518232/2809427
def unicodeToAscii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
    )

# Lowercase, trim, and remove non-letter characters
def normalizeString(s):
    s = unicodeToAscii(s.lower().strip())
    s = re.sub(r"([.!?])", r" \1", s)
    s = re.sub(r"[^a-zA-Z.!?]+", r" ", s)
    s = re.sub(r"\s+", r" ", s).strip()
    return s

# Read query/response pairs and return a voc object
def readVocs(datafile, corpus_name):
    print("Reading lines...")
    # Read the file and split into lines
    lines = open(datafile, encoding='utf-8').\
        read().strip().split('\n')
    # Split every line into pairs and normalize
    pairs = [[normalizeString(s) for s in l.split('\t')] for l in lines]
    voc = Voc(corpus_name)
    return voc, pairs

# Returns True iff both sentences in a pair 'p' are under the MAX_LENGTH threshold
def filterPair(p):
    # Input sequences need to preserve the last word for EOS token
    return len(p[0].split(' ')) < MAX_LENGTH and len(p[1].split(' ')) < MAX_LENGTH

# Filter pairs using filterPair condition
def filterPairs(pairs):
    return [pair for pair in pairs if filterPair(pair)]

# Using the functions defined above, return a populated voc object and pairs list
def loadPrepareData(corpus, corpus_name, datafile, save_dir):
    print("Start preparing training data ...")
    voc, pairs = readVocs(datafile, corpus_name)
    print("Read {!s} sentence pairs".format(len(pairs)))
    pairs = filterPairs(pairs)
    print("Trimmed to {!s} sentence pairs".format(len(pairs)))
    print("Counting words...")
    for pair in pairs:
        voc.addSentence(pair[0])
        voc.addSentence(pair[1])
    print("Counted words:", voc.num_words)
    return voc, pairs

# Load/Assemble voc and pairs
save_dir = os.path.join("data", "save")
voc, pairs = loadPrepareData(corpus, corpus_name, datafile, save_dir)
# Print some pairs to validate
print("\npairs:")
for pair in pairs[:10]:
    print(pair)

MIN_COUNT = 3    # Minimum word count threshold for trimming

I really hope someone can help me fix this problem and help me understand why it happens.


What is the way dimension extension with pytorch tensors?
-before:
torch.Size([3, 3, 3])
tensor([[[ 0.,  1.,  2.],
         [ 3.,  4.,  5.],
         [ 6.,  7.,  8.]],

        [[ 9., 10., 11.],
         [12., 13., 14.],
         [15., 16., 17.]],

        [[18., 19., 20.],
         [21., 22., 23.],
         [24., 25., 26.]]], device='cuda:0', dtype=torch.float64)

-after:
torch.Size([2, 3, 3, 3])
tensor([[[[ 0.,  1.,  2.],
          [ 3.,  4.,  5.],
          [ 6.,  7.,  8.]],

         [[ 9., 10., 11.],
          [12., 13., 14.],
          [15., 16., 17.]],

         [[18., 19., 20.],
          [21., 22., 23.],
          [24., 25., 26.]]],

        [[[0., 1., 2.],
          [ 3.,  4.,  5.],
          [ 6.,  7.,  8.]],

         [[ 9., 10., 11.],
          [12., 13., 14.],
          [15., 16., 17.]],

         [[18., 19., 20.],
          [21., 22., 23.],
          [24., 25., 26.]]]], device='cuda:0', dtype=torch.float64)

under numpy would work like this:
b =  np.broadcast_to(a1[None, :,:,:], (2,3,3,3))

How does this work under pytorch? I want to take advantage of the gpu. Thanks in advance for the help!


title: Use of PyTorch permute in RCNN

body:
I am looking at an implementation of RCNN for text classification using PyTorch. Full Code. There are two points where the dimensions of tensors are permuted using the permute function. The first is after the LSTM layer and before tanh. The second is after a linear layer and before a max pooling layer.
Could you please explain why the permutation is necessary or useful?
Relevant Code
    def forward(self, x):
        # x.shape = (seq_len, batch_size)
        embedded_sent = self.embeddings(x)
        # embedded_sent.shape = (seq_len, batch_size, embed_size)

        lstm_out, (h_n,c_n) = self.lstm(embedded_sent)
        # lstm_out.shape = (seq_len, batch_size, 2 * hidden_size)

        input_features = torch.cat([lstm_out,embedded_sent], 2).permute(1,0,2)
        # final_features.shape = (batch_size, seq_len, embed_size + 2*hidden_size)

        linear_output = self.tanh(
            self.W(input_features)
        )
        # linear_output.shape = (batch_size, seq_len, hidden_size_linear)

        linear_output = linear_output.permute(0,2,1) # Reshaping fot max_pool

        max_out_features = F.max_pool1d(linear_output, linear_output.shape[2]).squeeze(2)
        # max_out_features.shape = (batch_size, hidden_size_linear)

        max_out_features = self.dropout(max_out_features)
        final_out = self.fc(max_out_features)
        return self.softmax(final_out)

Similar Code in other Repositories
Similar implementations of RCNN use permute or transpose. Here are examples:

https://github.com/prakashpandey9/Text-Classification-Pytorch/blob/master/models/RCNN.py
https://github.com/jungwhank/rcnn-text-classification-pytorch/blob/master/model.py