title: Why my predictions are all zeros during the training phase of my Neural Network?

body:
I am a beginner with PyTorch, and I'm working on a small project which consists in predicting some output. To do this I'm using a Neural Network that has a few layers with simple transforms.
When I try to run my code I'm faced with the following problem : every prediction of my neural network is 0 (sometimes it gives other values in the first batch when I'm lucky enough...)
Here is the code of my project :
import torch
from torch import nn
from torch.utils.data import DataLoader
from torch.utils.data import Dataset
from torchvision.transforms import Lambda, Compose

import matplotlib.pyplot as plt
import numpy as np

n = 20      #Nombre de variables de direction résumant la variance du payoff
d = 20      #Dimension de l'option

N_data_training = 5000      #Taille de la base de donnée utile à l'apprentissage du réseau de neurones
N_data_test = 2500      #Taille de la base de donnée utile au test de la précision du réseau de neurones entrainé

type_product = "basket call"    #Type de produit dérivé
S0 = 100      #Spot price
K = 100     #Strike
T = 1     #Maturité
r = 0.05      #Taux d'intérêt sans risque
sigma = 0.2     #Volatilité
# Nmc = 1000      #Nombre de trajectoires générées pour l'estimation de Monte-Carlo
learning_rate = 1e-3

device = "cuda" if torch.cuda.is_available() else "cpu"
print("Using {} device".format(device))

def payoff_derivative(N_data, type_product, d, S0, r, sigma, T): #### pay-offs test
  payoff = np.zeros(N_data)
  Z = np.zeros((N_data, d))

  if type_product == "basket call":
    for i in range(N_data):
      Z[i, :] = S0*np.exp((r-0.5*sigma**2)*T + sigma*np.sqrt(T)*np.random.randn(d))
      payoff[i] = np.maximum((1/d)*np.sum(Z[i, :], axis=0) - K, 0)

  return (Z, payoff)

pure_data_training = payoff_derivative(N_data_training, type_product, d, S0, r, sigma, T) ##### pay-offs pour l'apprentissage du réseau de neurones
pure_data_test = payoff_derivative(N_data_test, type_product, d, S0, r, sigma, T) ##### pay-offs pour le test de précision

class Creation_Dataset(Dataset):
  def __init__(self, pure_data):
    self.Z = pure_data[0]
    self.payoff = pure_data[1]

  def __len__(self):
    return len(self.payoff)

  def __getitem__(self, idx):
    return torch.tensor(self.Z[idx],dtype=torch.float), torch.tensor(self.payoff[idx],dtype=torch.float)

dataset_training = Creation_Dataset(pure_data_training)
dataset_test = Creation_Dataset(pure_data_test)

# Creation des data loaders
batch_size = 100
train_dataloader = DataLoader(dataset_training, batch_size)
test_dataloader = DataLoader(dataset_test, batch_size)

for X, y in train_dataloader:
    print("Shape of X : ", X.shape, X.dtype)
    print("Shape of y: ", y.shape, y.dtype)
    print(X[0])
    print(y[0])
    break

class NeuralNetwork(nn.Module):
    def __init__(self): ######méthode dans laquelle on définit les couches du réseau
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten() ########## convertit en tenseur 1D
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(d, n, bias=False),
            nn.Linear(n, 6),
            nn.ReLU(),
            nn.Linear(6, 15),
            nn.ReLU(),
            nn.Linear(15, 10),
            nn.ReLU(),
            nn.Linear(10, 5),
            nn.ReLU(),
            nn.Linear(5, 1),
            nn.ReLU()
          )

    def forward(self, x):
        logits = self.linear_relu_stack(x)
        return torch.reshape(logits, (-1,))

model = NeuralNetwork().to(device)
print(model)

loss_fn = nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        print("Shape of X : ", X.shape, X.dtype)
        print("Shape of y: ", y.shape, y.dtype)
        pred = model(X)
        print(pred)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")

def test(dataloader, model):
    model.eval()
    test_loss, correct = 0, 0
    with torch.no_grad():
        for X, y in dataloader:
            X, y = X.to(device), y.to(device)
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += float((pred == y).sum())
    size = len(dataloader.dataset)
    test_loss /= size
    correct /= size
    print(f"Accuracy: {100*correct:.2f}%, Avg loss (sqrt(MSE)) : {np.sqrt(test_loss):.3f} \n")

epochs = 2
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train(train_dataloader, model, loss_fn, optimizer)
    test(test_dataloader, model)

So, using a DataLoader that creates batches of batch_size, I train my Neural Network in the train_loop() function and I print out my predictions' tensor, and it turns out that all my predictions are zeros. I've seen a few topics with people facing this kind of problems with classification problems, but I'm still trying to solve it for predicting correct outputs.
Thank you for your time


title: Pytorch Lightning Inference

body:
I trained a model using pytorch lightning and especially appreciated the ease of using multiple GPU's. Now after training, how can I still make use of lightnings GPU features to run inference on a test set and store/export the predictions?
The documentation on inference does not target that.
Thanks in advance.


title: Pytorch Lightning: Get model's output on full train data, during training

body:
TL;DR: During on_train_epoch_start, I want to get the model's output on ALL the training data, as part of some pre-training calculations. I'm asking what the lightning-friendly way to do that is.
This is an odd question.
In my project, every 10 epochs I select a subset of the full training data, and train only on that subset. During part of the calculation of which subset to use, I compute the model's output on every datapoint in the train dataset.
My question is, what's the best way to do this in pytorch lightning? Currently I have a callback with an on_train_epoch_start hook. During this hook, the callback makes its own dataloader from trainer.datamodule.train_dataloader(), and manually iterates over the dataloader, computing the model outputs. That's not ideal, right?
This makes me run into problems with pytorch lightning. For instance, when training on the GPU, I get an error, since my callback is using its own dataloader, not the trainer's train_dataloader, and so it isn't on the GPU. However, I can't use the trainer's train_dataloader, since after my callback selects its subset, it changes the trainer's train_dataloader to be just that subset, instead of the full dataset.
I guess I have two main questions:

Is there any way to avoid making a separate dataloader? Can I call a train loop somehow? Getting a models output on the full dataset seems like such a simple operation, I would think it'd be a one-liner.
How can I get/use a dataloader that syncs with all of the Pytorch Lightning automatic modifications? (eg. GPU/CPU, dataloading workers, pin_memory)


title: Sequential network with the VGG layers

body:
I want to have a sequential network with the characteristics of VGG network (I want to pass my network to another function, which doesn't support VGG objects and supports nn.sequential).
I added the function getSequentialVersion method to VGG class to have the sequential network with the linear layer. However, apparently, there is a size mismatch in the network.
'''VGG for CIFAR10. FC layers are removed.
(c) YANG, Wei
'''
import torch.nn as nn
import torch.utils.model_zoo as model_zoo
import math

__all__ = [
    'VGG','vgg16_bn',
]

model_urls = {
    'vgg16': 'https://download.pytorch.org/models/vgg16-397923af.pth',
}

class VGG(nn.Module):
    def __init__(self, features, num_classes=1000, cfg_type=None, batch_norm=False, **kwargs):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Linear(512, num_classes)
        self._initialize_weights()
        self.cfg_type = cfg_type
        self.batch_norm = batch_norm

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return x

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                if m.bias is not None:
                    m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()
            elif isinstance(m, nn.Linear):
                n = m.weight.size(1)
                m.weight.data.normal_(0, 0.01)
                m.bias.data.zero_()

    def getSequentialVersion(self):
        return make_layers(cfg[self.cfg_type], batch_norm=self.batch_norm, flag=True)

def make_layers(cfg, batch_norm=False, flag=False):
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1, bias=False)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = v
    if flag:
        #for Cifar10
        layers += [nn.Linear(512, 10)]
    return nn.Sequential(*layers)

cfg = {
    'A': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'B': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'E': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

def vgg16_bn(**kwargs):
    """VGG 16-layer model (configuration "D") with batch normalization"""
    print("VGG16-bn")
    model = VGG(make_layers(cfg['D'], batch_norm=True), cfg_type='D', batch_norm=True,**kwargs)
    return model

When I call summary(net, ( 3, 32, 32)) (for cifar10), I get the mismatch error. In other words, the main problem is when I add this line layers+= [nn.linear(512, 10)].
Can anyone help me? Thanks a lot.
The error message:
  File "./main.py", line 284, in <module>
    summary(net, ( 3, 32, 32))
  File "./anaconda3/envs/my_env/lib/python3.8/site-packages/torchsummary/torchsummary.py", line 72, in summary
    model(*x)
  File ".anaconda3/envs/my_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "./anaconda3/envs/my_env/lib/python3.8/site-packages/torch/nn/modules/container.py", line 119, in forward
    input = module(input)
  File "./anaconda3/envs/my_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "./anaconda3/envs/my_env/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 94, in forward
    return F.linear(input, self.weight, self.bias)
  File "./envs/my_env/lib/python3.8/site-packages/torch/nn/functional.py", line 1753, in linear
    return torch._C._nn.linear(input, weight, bias)
RuntimeError: mat1 dim 1 must match mat2 dim 0

Additional information:
This is how exactly I initialize and use my network:
net = vgg16_bn(depth=args.depth,
                  num_classes=num_classes,
                  growthRate=args.growthRate,
                  compressionRate=args.compressionRate,
                  widen_factor=args.widen_factor,
                  dropRate=args.dropRate,
                  base_width=args.base_width,
                  cardinality=args.cardinality).getSequentialVersion()
net = net.to(args.device)
module_names = ''
if hasattr(net, 'features'):
    module_names = 'features'
elif hasattr(net, 'children'):
    module_names = 'children'
else:
    print('unknown net modules...')

summary(net, ( 3, 32, 32))


title: PyTorch - Import dataset with images as labels

body:
I have a dataset containing images as inputs and labels/targets as images as well. The structure in the folder is as follows:
> DATASET/
> ---TRAIN/
> ------image_xx.png
> ------label_xx.png
> ---TEST/
> ------image_xx.png
> ------label_xx.png

I've currently tried to use "ImageFolder" from torchvisions datasets to load the images as follows:
TRAIN_PATH = '/path/to/dataset/DATASET'
train_data = datasets.ImageFolder(root=TRAIN_PATH, transform=transforms.ToTensor())
train_loader = DataLoader(train_data, batch_size=16, shuffle=True)

However as shown below:
for img, label in train_loader:
    print(img.shape)
    print(label.shape)
    break

torch.Size([16, 3, 128, 128])
torch.Size([16])

The labels aren't images but rather indicies or something similar. Is there a convenient way of importing this dataset with the aforementioned structure?


title: difference between initializing bias by nn.init.constant and (bias=1) in pytorch

body:
I'm making a code for AlexNet and i'm confused with how to initialize the weights
what is the difference between:
        for layer in self.cnnnet:
            if isinstance(layer, nn.Conv2d):
                 nn.init.constant_(layer.bias, 0)

and
nn.Linear(shape, bias=0)


title: PyTorch: how to let only chosen elements in a tensor be differentiable?

body:
Right now, I have a use case like following
A is a torch.tensor like following
A =
[[1,x,y],
 [1,2,3],
 [1,z,3]]

only elements x, y, z in A are differentiable, other elements are just constant.
For example, if the cost = tr(A.A)
cost = 14 + 2 x + 2 y + 6 z
When I do backtracking, I only want to differentiate and update with respect to x, y, z. Certainly this example is just a toy example, not the true complicated one.
How can I realize such use case?


title: ImportError: No module named 'knn'

body:
I am trying to install PU_GAN repository and after installing knn_cuda module and trying to call it on python shell I get this error:
import knn_cuda
        Traceback (most recent call last):
        File "", line 1, in
        File "C:\Users\49152\anaconda3\envs\torch\lib\site-packages\knn_cuda_init_.py", line 38, in
        knn = load_cpp_ext("knn")
        File "C:\Users\49152\anaconda3\envs\torch\lib\site-packages\knn_cuda_init.py", line 26, in load_cpp_ext
        ext = load(
        File "C:\Users\49152\anaconda3\envs\torch\lib\site-packages\torch\utils\cpp_extension.py", line 1079, in load
        return _jit_compile(
        File "C:\Users\49152\anaconda3\envs\torch\lib\site-packages\torch\utils\cpp_extension.py", line 1317, in _jit_compile
        return _import_module_from_library(name, build_directory, is_python_module)
        File "C:\Users\49152\anaconda3\envs\torch\lib\site-packages\torch\utils\cpp_extension.py", line 1699, in _import_module_from_library
        file, path, description = imp.find_module(module_name, [path])
        File "C:\Users\49152\anaconda3\envs\torch\lib\imp.py", line 296, in find_module
        raise ImportError(_ERR_MSG.format(name), name=name)
        ImportError: No module named 'knn'

I am using :
pytorch                   1.8.1           py3.8_cuda10.1_cudnn7_0    pytorch
knn-cuda                  0.2                      pypi_0    pypi
cudatoolkit               10.1.243             h74a9793_0
python                    3.8.8                hdbf39b2_5


title: Python: OSError can't load config for bert

body:
I am trying to train a bert-base-multilingual-uncased model for a task. I have all the required files present in my dataset including the config.json bert file but when I run the model it gives an error.
Config
class config:
    DEVICE = "cuda:2"
    MAX_LEN = 256
    TRAIN_BATCH_SIZE = 8
    VALID_BATCH_SIZE = 4
    EPOCHS = 1
    BERT_PATH = "workspace/data/jigsaw-multilingual/input/bert-base-multilingual-uncased"
    MODEL_PATH = "workspace/data/jigsaw-multilingual/model.bin"
    TOKENIZER = BertTokenizer.from_pretrained('bert-base-multilingual-uncased', do_lower_case=True)

Model
class BERTBaseUncased(nn.Module):
    def __init__(self):
        super(BERTBaseUncased, self).__init__()
        self.bert = transformers.BertModel.from_pretrained(config.BERT_PATH)
        self.bert_drop = nn.Dropout(0.3)
        self.out = nn.Linear(768 * 2, 1) # *2 since we have 2 pooling layers

    def forward(self, ids, mask, token_type_ids):
        o1, _ = self.bert(ids, attention_mask=mask, token_type_ids=token_type_ids)

        mean_pooling = torch.mean(o1, 1)
        max_pooling, _ = torch.max(o1, 1)
        cat = torch.cat((mean_pooling, max_pooling), 1)

        bo = self.bert_drop(cat)
        output = self.out(bo)
        return output

Error
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/opt/conda/lib/python3.6/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    241             if resolved_config_file is None:
--> 242                 raise EnvironmentError
    243             config_dict = cls._dict_from_json_file(resolved_config_file)

OSError:

During handling of the above exception, another exception occurred:

OSError                                   Traceback (most recent call last)
<ipython-input-64-9f2999c88020> in <module>
     79
     80 if __name__ == "__main__":
---> 81     run()

<ipython-input-64-9f2999c88020> in run()
     38
     39     device = torch.device(config.DEVICE)
---> 40     model = BERTBaseUncased()
     41     model.to(device)
     42

<ipython-input-60-8e1508eac60a> in __init__(self)
      2     def __init__(self):
      3         super(BERTBaseUncased, self).__init__()
----> 4         self.bert = transformers.BertModel.from_pretrained(config.BERT_PATH)
      5         self.bert_drop = nn.Dropout(0.3)
      6         self.out = nn.Linear(768 * 2, 1) # *2 since we have 2 pooling layers

/opt/conda/lib/python3.6/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
    601                 proxies=proxies,
    602                 local_files_only=local_files_only,
--> 603                 **kwargs,
    604             )
    605         else:

/opt/conda/lib/python3.6/site-packages/transformers/configuration_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    198
    199         """
--> 200         config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
    201         return cls.from_dict(config_dict, **kwargs)
    202

/opt/conda/lib/python3.6/site-packages/transformers/configuration_utils.py in get_config_dict(cls, pretrained_model_name_or_path, **kwargs)
    249                 f"- or '{pretrained_model_name_or_path}' is the correct path to a directory containing a {CONFIG_NAME} file\n\n"
    250             )
--> 251             raise EnvironmentError(msg)
    252
    253         except json.JSONDecodeError:

OSError: Can't load config for 'workspace/data/jigsaw-multilingual/input/bert-base-multilingual-uncased'. Make sure that:

- 'workspace/data/jigsaw-multilingual/input/bert-base-multilingual-uncased' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'workspace/data/jigsaw-multilingual/input/bert-base-multilingual-uncased' is the correct path to a directory containing a config.json file

These are the files present in my bert dataset:
-> config.json
->pytorch_model.bin
-> vocab.txt
How to fix this issue?


title: RuntimeError: CUDA out of memory. When training with Yolact

body:
I have been trying to train with PyTorch for Yolact following the guide: https://github.com/dbolya/yolact
Current GPU is RTX2070 and cudatoolkit of 11.1.1 is used.
When I run the following:

python train.py --config=yolact_base_config --batch_size=8

I keep running into this error

RuntimeError: CUDA out of memory. Tried to allocate 50.00 MiB (GPU 0;
7.79 GiB total capacity; 5.98 GiB already allocated; 22.88 MiB free; 6.40 GiB reserved in total by PyTorch)

I have tried decreasing the maximum size of the pictures in max_size under config.py but the CUDA error continue to persist. I decrease the batch size as well but it does not seem to improve.


title: Pytorch Data Loader concatenate an image to input images

body:
In PyTorch data loader, how could I concatenate an image (let's say x.jpg) in-band wise to each and every input images. ie, in effect I will have 4 band input ( 3 band input jpg with 1 band x.jpg. How to implement it.
Please find below example of my current dataloader just to load the images. To this, I want to add x.jpg to "image"(ie input image, not to mask)
from PIL import Image

class lakeDataSet(Dataset):
   def __init__(self, root, transform):
   super().__init__()
   self.root = root
   self.img_dir = os.path.join(root,'image-c3/c3-crop')   #9UAV
   self.mask_dir = os.path.join(root,'label-c3/c3-crop')
   # self.mask_dir = os.path.join(root,'test')
   self.files = [fname for fname in os.listdir(self.img_dir) if fname.endswith('.jpg')]
self.transform = transform

   def __len__(self):
     return len(self.files)

   def __getitem__(self,I):
     fname = self.files[i]
     img_path = os.path.join(self.img_dir, fname)
     mask_path = os.path.join(self.mask_dir, fname)

     img = self.transform(Image.open(img_path))
     mask = self.transform(Image.open(mask_path))
     return img, mask


title: Pytorch List of all gradients in a model

body:
I'm trying to clip my gradients in a simple deep network model (for RL). But for that I want to fetch statistics of gradients in each epochs, e.g. mean, max etc. Through this I will be able to determine the threshold value to clip my gradients to.
So the way I can approach this was if there was any way to fetch all the calculated gradients as an array after model.backwards() step.
How can I do this? Or is there any other way to determine this hyper-parameter?


title: Pytorch: RuntimeError: Caught RuntimeError in replica 0 on device 0

body:
I have eight different models in a project in which two are works fine without giving any error on multiple GPUs.  When I integrated a new model, I got a replica error and the point is that I made a config file and this file remains unchanged for both models.
Config.py file
import torch, os
import yaml
from IPython import embed

def get_config(args):
    configuration = dict(
        SEED=1337,  # random seed for reproduce results
        INPUT_SIZE=[112, 112],  # support: [112, 112] and [224, 224]
        EMBEDDING_SIZE=512,  # feature dimension
    )

    if args.workers_id == 'cpu' or not torch.cuda.is_available():
        configuration['GPU_ID'] = []
        print("check", args.workers_id, torch.cuda.is_available())
    else:
        configuration['GPU_ID'] = [int(i) for i in args.workers_id.split(',')]
    if len(configuration['GPU_ID']) == 0:
        configuration['DEVICE'] = torch.device('cpu')
        configuration['MULTI_GPU'] = False
    else:
        configuration['DEVICE'] = torch.device('cuda:%d' % configuration['GPU_ID'][0])
        if len(configuration['GPU_ID']) == 1:
            configuration['MULTI_GPU'] = False
        else:
            configuration['MULTI_GPU'] = True

    configuration['NUM_EPOCH'] = args.epochs
    configuration['BATCH_SIZE'] = args.batch_size

    if args.data_mode == 'casia':
        configuration['DATA_ROOT'] = '/media/khawar/HDD_Khawar/face_datasets/WIDER_train/images/'
    elif args.data_mode == "CelebA":
        configuration['DATA_ROOT'] = '/media/khawar/HDD_Khawar/face_datasets/CelebA'
    elif args.data_mode == "faces":
        configuration['DATA_ROOT'] = '/raid/khawar/dataset/faces/'
    else:
        print('fff')
        # raise Exception(args.data_mode)

    configuration['EVAL_PATH'] = './eval/'
    assert args.net in ['VIT', 'VITs', 'VITs_Eff', 'CAiT', 'DeepViT', 'PiT', 'LeViT','Comb_ViT', 'Dino_VIT', 'CvT','Swin', 'T2TViT']
    configuration['BACKBONE_NAME'] = args.net
    assert args.head in ['Softmax', 'ArcFace', 'CosFace', 'SFaceLoss']
    configuration['HEAD_NAME'] = args.head
    configuration['TARGET'] = [i for i in args.target.split(',')]

    if args.resume:
        configuration['BACKBONE_RESUME_ROOT'] = args.resume
    else:
        configuration['BACKBONE_RESUME_ROOT'] = ''  # the root to resume training from a saved checkpoint
    configuration['WORK_PATH'] = args.outdir  # the root to buffer your checkpoints
    if not os.path.exists(args.outdir):
        os.makedirs(args.outdir)

    return configuration

main.py
GPU_ID = cfg['GPU_ID']  # specify your GPU ids
print('GPU_ID', GPU_ID)

Traceback (most recent call last):
  File "train.py", line 367, in <module>
    outputs, emb = BACKBONE(inputs.float(), labels)
  File "/raid/khawar/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/raid/khawar/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 167, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/raid/khawar/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 177, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/raid/khawar/.local/lib/python3.6/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/raid/khawar/.local/lib/python3.6/site-packages/torch/_utils.py", line 429, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.


title: How to integrate a pytorch model into a dynamic optimization, for example in Pyomo or gekko

body:
Let's say I have a pytorch-model describing the evolution of some multidimensional system based on its own state x and an external actuator u. So x_(t+1) = f(x_t, u_t) with f being the artificial neural network from pytorch.
Now i want to solve a dynamic optimization problem to find an optimal sequence of u-values to minimize an objective that depends on x. Something like this:
min sum over all timesteps phi(x_t)
s.t.: x_(t+1) = f(x_t, u_t)
Additionally I also have some upper and lower bounds on some of the variables in x.
Is there an easy way to do this using a dynamic optimization toolbox like pyomo or gekko?
I already wrote some code that transforms a feedforward neural network to a numpy-function which can then be passed as a constraint to pyomo. The problem with this approach is, that it requires significant reprogramming-effort every time the structure of the neural network changes, so quick testing becomes difficult. Also integration of recurrent neural networks gets difficult because hidden cell states would have to be added as additional variables to the optimization problem.
I think a good solution could be to do the function evaluations and gradient calculations in torch and somehow pass the results to the dynamic optimizer. I'm just not sure how to do this.
Thanks a lot for your help!


title: Negative values in negative log likelihood loss function of mixture density networks

body:
I am trying to implement mixture density networks (MDN), which can learn a mixture Gaussion distribution. I want to use MDN to fit a conditional probability distribution (p(y|x)). The dimensionality of the model input x is (batch_size, 1), y (label) is (batch_size, 1). here is my code.
Is that something wrong with data? When I use generated dataset, result is right.
Thanks.
class MDN(nn.Module):
    """A mixture density network layer
    The input maps to the parameters of a MoG probability distribution, where
    each Gaussian has O dimensions and diagonal covariance.
    Arguments:
        in_features (int): the number of dimensions in the input
        out_features (int): the number of dimensions in the output
        num_gaussians (int): the number of Gaussians per output dimensions
    Input:
        minibatch (BxD): B is the batch size and D is the number of input
            dimensions.
    Output:
        (pi, sigma, mu) (BxG, BxGxO, BxGxO): B is the batch size, G is the
            number of Gaussians, and O is the number of dimensions for each
            Gaussian. Pi is a multinomial distribution of the Gaussians. Sigma
            is the standard deviation of each Gaussian. Mu is the mean of each
            Gaussian.
    """

    def __init__(self, n_hidden, num_gaussians, in_features=1, out_features=1):
        super(MDN, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.num_gaussians = num_gaussians
        self.z_h = nn.Sequential(
            nn.Linear(1, n_hidden)
        )
        self.pi = nn.Sequential(
            nn.Linear(n_hidden, num_gaussians),
            nn.Softmax(dim=1)
        )
        self.sigma = nn.Linear(n_hidden, out_features * num_gaussians)
        self.mu = nn.Linear(n_hidden, out_features * num_gaussians)

    def forward(self, minibatch):
        z_h = self.z_h(minibatch)
        pi = self.pi(z_h)

        sigma = torch.exp(self.sigma(z_h))
        sigma = sigma.view(-1, self.num_gaussians, self.out_features)
        mu = self.mu(z_h)
        mu = mu.view(-1, self.num_gaussians, self.out_features)
        return pi, sigma, mu

ONEOVERSQRT2PI = 1.0 / math.sqrt(2 * math.pi)
def gaussian_probability(sigma, mu, target):
    """Returns the probability of `target` given MoG parameters `sigma` and `mu`.
    Arguments:
        sigma (BxGxO): The standard deviation of the Gaussians. B is the batch
            size, G is the number of Gaussians, and O is the number of
            dimensions per Gaussian.
        mu (BxGxO): The means of the Gaussians. B is the batch size, G is the
            number of Gaussians, and O is the number of dimensions per Gaussian.
        target (BxI): A batch of target. B is the batch size and I is the number of
            input dimensions.
    Returns:
        probabilities (BxG): The probability of each point in the probability
            of the distribution in the corresponding sigma/mu index.
    """
    target = target.unsqueeze(1).expand_as(sigma)
    ret = ONEOVERSQRT2PI * torch.exp(-0.5 * ((target - mu) / sigma)**2) / sigma
    return torch.prod(ret, 2)

def mdn_loss(pi, sigma, mu, target):
    """Calculates the error, given the MoG parameters and the target
    The loss is the negative log likelihood of the data given the MoG
    parameters.
    """
    prob = pi * gaussian_probability(sigma, mu, target)

    nll = -torch.log(torch.sum(prob, dim=1))
    return torch.mean(nll)
network = MDN(n_hidden=20, num_gaussians=5)
optimizer = torch.optim.Adam(network.parameters(), lr=0.0001)

data = pd.read_csv("./experiment/data/sorted_lineitem_1g_quantity_extendedprice.csv")
data.sort_values("extendedprice", inplace=True)
data.reset_index(drop=True, inplace=True)
sample = data.sample(frac=0.001)

quantity = np.array(sample["quantity"])
extendedprice = np.array(sample["extendedprice"])

max_x = np.max(extendedprice)
min_x = np.min(extendedprice)
mean_x = np.mean(extendedprice)
stdev_x = np.std(extendedprice)
max_y = np.max(quantity)
min_y = np.min(quantity)
mean_y = np.mean(quantity)
stdev_y = np.std(quantity)
# mean_y = np.mean(quantity)

def normalize(x, min_x, max_x):
    return (x - min_x) / (max_x - min_x)

def standardization(x, mean, stdev):
    return (x - mean) / stdev

x_data = np.array([standardization(i, mean_x, stdev_x) for i in extendedprice])
# x_data = extendedprice
y_data = np.array([standardization(i, mean_y, stdev_y) for i in quantity])
# y_data = quantity

x_tensor = torch.from_numpy(np.float32(x_data).reshape(-1, 1))
y_tensor = torch.from_numpy(np.float32(y_data).reshape(-1, 1))

mdn_x_data = x_data
mdn_y_data = y_data

mdn_x_tensor = x_tensor
mdn_y_tensor = y_tensor

x_variable = torch.tensor(mdn_x_tensor)
y_variable = torch.tensor(mdn_y_tensor, requires_grad=False)
def train_mdn():
    for epoch in range(10000):
        pi_variable, sigma_variable, mu_variable = network(x_variable)
        loss = mdn_loss(pi_variable, sigma_variable, mu_variable, y_variable)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if epoch % 500 == 0:
            print(epoch, loss.item())

train_mdn()

result of code
0 1.9179250001907349
500 0.6071352362632751
1000 0.12722773849964142
1500 0.05316021665930748
2000 0.026438187807798386
2500 0.011702506802976131
3000 -0.00010265530727338046
3500 -0.011923379264771938

My question is: why the value of the loss function becomes negative with the training process?


title: "RuntimeError: mat1 dim 1 must match mat2 dim 0" PyTorch

body:
I've been trying to build a simple neural network with linear layers:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.lin1 = nn.Linear(64 * 1 * 28 * 28, 6)
        self.lin2 = nn.Linear(6, 4)
        self.out = nn.Linear(4, 10)

    def forward(self, x):
        print(x.shape)

        x = self.lin1(x)
        x = F.relu(x)

        x = self.lin2(x)
        x = F.relu(x)

        x = self.out(x)
        x = F.softmax(x)

        return x

However I run into the error:

mat1 dim 1 must match mat2 dim 0

for the line
x = self.lin1(x)

I tried flattening x before lin1 or changing the lin1 input size, but nothing worked.
The output of the print statement is:
torch.Size([64, 1, 28, 28])


title: Why does Loss Diagram of LSTM Cell in my Algorithm show many upper and down fluctuation?

body:
Hier I attached Image of my Loss Diagram. I am very new to stack overflow, if  possible, please try to answer here.

My Data stands here for one Graph with  approximately 3600 nodes. And i want to predict next nodes with help of known data of last nodes of Graph.

Here I used one LSTM Cell layer for my 6 diemnsional Data.[x,y,u,v,p,type]. And then  at the end I want to get only 3D[u,v,p] output.
Actually it does very well.

But when I see the lossHisto after 30,000 iteration for same graph.
it looks like, as I attached one Photo for that above.

I don't know, why my Loss is not reduced as we expected under 0.025. And why sometimes shows very high fluctuation instead curve.

import numpy as np
import matplotlib.pyplot as plt
import torch.nn as nn
import torch
plt.switch_backend('agg')

class Sequence(nn.Module):
    def __init__(self):
        super(Sequence, self).__init__()
        self.lstmcell = nn.LSTMCell(5, 256)
        self.linear = nn.Linear(256, 3)

    def forward(self, x, path):

        outputs = []
        h_t = torch.zeros(1, 256)
        c_t = torch.zeros(1, 256)

        for i in range(x.shape[0]):
            if  trainData[path[i],5] == 0:
                neighbours = graphDict[str(path[i])]
                y = graphFlags[neighbours]
                idx = np.where(y !=0)[0]
                z = torch.Tensor(x[neighbours[idx],:])
                u = torch.mean(z,0)
            else:
                u = (x[path[i],:]).clone()
            input_t = u.view(1,5)
            h_t, c_t = self.lstmcell(input_t, (h_t, c_t))
            output = torch.tanh(self.linear(h_t))
            outputs += [output]
            graphFlags[path[i]] = 1
            x[path[i], 0:3] = output
        outputs = torch.cat((outputs), 0)
        return outputs

for t in range(30000):
    graphFlags = np.copy(trainData[:,5])
    trueData_ = torch.Tensor(trueData[:, 2:5])

    binary_repr = np.unpackbits(np.copy(trainData)[:,5].astype(np.uint8).reshape(-1,1), axis=1)[:,6:]
    x = np.concatenate((np.copy(trainData)[:,2:5], binary_repr), axis=1)
    x = torch.Tensor(x)

    optimizer.zero_grad()
    out = seq(x, path)

    loss = criterion(out, trueData_)
    counter+=1
    loss.backward()
    optimizer.step()
    lossHisto.append(loss.item())

    if counter == 100:

        saveName = 'out'
        SaveName = 'lossHisto'
        torch.save(out, '/mnt/fs2/home/bhavesh/Ergebnisse/test1/'+saveName+str(t))
        BILD = torch.load('/mnt/fs2/home/bhavesh/Ergebnisse/test1/'+saveName+str(t))
        plt.figure(figsize=(20,10))
        plt.scatter(trueData[:,0],trueData[:,1],c=torch.Tensor(BILD.data[:,2]),marker='.')
        plt.colorbar()
        plt.savefig("/mnt/fs2/home/bhavesh/Ergebnisse/test1/out%d.png"%t)
        plt.clf()
        torch.save(lossHisto, '/mnt/fs2/home/bhavesh/Ergebnisse/test1/'+SaveName+str(t))
        LOSS = torch.load('/mnt/fs2/home/bhavesh/Ergebnisse/test1/'+SaveName+str(t))
        plt.plot(LOSS)
        plt.grid()
        plt.savefig("/mnt/fs2/home/bhavesh/Ergebnisse/test/lossHisto%d.png"%t)
        counter = 0

    print(loss.item())


title: How to do one line flag embed in PyTorch (not nn.Embedding)?

body:
With a generator I create the random batch like:

import torch

n = 10
batch_size = 2

x = torch.zeros((batch_size, n), dtype=torch.float)
in_flags = torch.randint(n, (batch_size,), dtype=torch.long)

for idx, row in enumerate(x):
    row[in_flags[idx]] = 1.0

But the disadvantage of that is that loop runs in Python.
That is the original meaning of embedding (do not confuse that with PyTorch nn.embedding). Is it possible to do with one PyTorch operator to make it be executed native or in GPU?


title: How to generate new tensor by given indexes and tensor in pytorch?

body:
I have a tensor like this:

x = torch.tensor([[3, 4, 2], [0, 1, 5]])

and I have a indexes like this:

ind = torch.tensor([[1, 1, 0], [0, 0, 1]])

then I want to generate a new tensor by x and ind:

z = torch.tensor([0, 1, 2], [3, 4, 5])

I impement it with python like this:

# -*- coding: utf-8 -*-

import torch

x = torch.tensor([[3, 4, 2], [0, 1, 5]])
ind = torch.tensor([[1, 1, 0], [0, 0, 1]])

z = torch.zeros_like(x)
for i in range(x.shape[0]):
    for j in range(x.shape[1]):
        z[i, j] = x[ind[i][j]][j]

print(z)

I want to know how to solve this by pytorch?


title: How can adapt my data to this linear matrix pytorch optimization method?

body:
I try my code is not working I'm using a simple  dataset:

epochs = 100
losses = []
for i in range(epochs):
  y_pred = model.forward(X)
  loss = criterion(y_pred, y)
  print("epoch:", i, "loss:", loss.item())
  losses.append(loss)
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

thanks,

Philippe.


title: In Pytorch, what is the most efficient way to copy the learned params of a model as the initialization for a second model of the same architecture?

body:
I have a CNN Model which has the following architecture:

class Model(nn.Module):

    def __init__(self):
        super().__init__()

        self.conv1 = nn.Conv2d(4, 32, (8, 8), 4)
        self.conv2 = nn.Conv2d(32, 64, (4, 4), 2)
        self.conv3 = nn.Conv2d(64, 64, (3, 3), 1)
        self.dense = nn.Linear(4*4*64, 512)
        self.out = nn.Linear(512, 18)

I am training it using a certain optimizer. I then want to use these learned parameters from the first model as the initialization scheme for a second model of the exact same architecture (as opposed to using, say, Xavier). I am aware that I need to use model_object.apply(initalization_function), but what would be the most efficient way to do this vis-a-vis the initialization scheme I described where I am using the learned parameters from another model as initialization for a new model?


title: AI with un-enumerable action output?

body: I'm getting into AI as a hobby and all of my previous projects had the action output enumerated and specific choices however the new project I want to begin working on will require for example the AI to decide the best place to tap on a screen. Can anyone point me some resources/information that I can research to learn this task? I just need to know what to search to educate myself.


title: While calculating posterior using the parameter(say w_mu) vs using the parameter.data(w_mu.data) makes a difference?

body:
I am trying to implement Bayes By Backprop. While calculating posterior if I use the parameter as an input vs parameter.data as input the result accuracy drastically changes.

    self.w_post = Normal(self.w_mu.data, torch.log(1+torch.exp(self.w_rho)))
    self.b_post = Normal(self.b_mu.data, torch.log(1+torch.exp(self.b_rho)))
    self.log_post = self.w_post.log_prob(self.w).sum() + self.b_post.log_prob(self.b).sum()

This works, while the next block doesn't.

    self.w_post = Normal(self.w_mu, torch.log(1+torch.exp(self.w_rho)))
    self.b_post = Normal(self.b_mu, torch.log(1+torch.exp(self.b_rho)))
    self.log_post = self.w_post.log_prob(self.w).sum() + self.b_post.log_prob(self.b).sum()

Since w_post and b_post aren't the parameters so why does this affect my answer. This snip of code lies in the forward function of a custom-defined linear layer.

While the value of log_posterior does not change through the epochs. Can it have something to do with the seed?


title: Adding a Linear layer to my LSTM made the validation loss skyrocket in PyTorch

body:
My model was:

    def forward(self, x, hidden=None):
        lstm_out, hidden = self.lstm(x, hidden)

        lstm_out = (lstm_out[:, :, :self.hidden_size] +
                    lstm_out[:, :, self.hidden_size:])

        out = torch.nn.SELU()(lstm_out)
        return out, hidden

It now is:

    def forward(self, x, hidden=None):
        lstm_out, hidden = self.lstm(x, hidden)
        batch_size = lstm_out.size(0)

        flattened_out = lstm_out.view(-1, self.hidden_size * 2)

        lstm_out = (lstm_out[:, :, :self.hidden_size] +
                    lstm_out[:, :, self.hidden_size:])
        out = self.linear(flattened_out)

        out = torch.nn.functional.relu(out)

        view_out = out.view(batch_size, self.seq_length, -1)

        return view_out, hidden

I used to get validation loss (with MSELoss) under 1000 after 2-3 epochs. Now with the Linear layer, it is skyrocketing up to 15000 even after 10 epochs. Why would this be?


title: Mapping a list of label with one hot encoding

body:
How can I do the same thing when 'label' is a list?
For example: label = [2,4,6,1,7...,9]

label = 3
NumClass = 10
NumRows  = 100

mask    =  torch.zeros(100, 64)
ones     =  torch.ones(1, 64)
ElementsPerClass = NumRows//NumClass
mask [ ElementsPerClass*label : ElementsPerClass*(label+1) ] = ones


title: How have the number of dimensions after LSTM been decided in Pointer Generator Model in PyTorch?

body:
I don't understand why is the number of input and output dimensions 2 * config.hidden_dim while applying a fully connected layer in the encode class (mentioned in the last line)?

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.embedding = nn.Embedding(config.vocab_size, config.emb_dim)
        init_wt_normal(self.embedding.weight)

        self.lstm = nn.LSTM(
            config.emb_dim, config.hidden_dim, num_layers=1,
            batch_first=True, bidirectional=True)
        init_lstm_wt(self.lstm)

        self.W_h = nn.Linear(
            config.hidden_dim * 2, config.hidden_dim * 2, bias=False)

The code has been taken from https://github.com/atulkum/pointer_summarizer/blob/master/training_ptr_gen/model.py
Please Explain


title: How to show image from compressed representation (RNN)?

body:
I am learning RNN with pytorch from this github.

Pytorch RNN Tutorial

I'm a little bit confused, because the code didn't show result of the training. Only show the accuracy. Now I want to show image that output from model.

I try to save image from output, but the result I get is a compressed representation (below is the image I got from output of the model)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = images.reshape(-1, sequence_length, input_size).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                    .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

I am trying to show the image of the output by adding this code

plt.imshow(outputs.reshape(28, 28))

But I got this error

RuntimeError: shape '[28, 28]' is invalid for input of size 1000

So, I assume that the output is 1000 size. but I want it to show as a mnist image (784 size (28*28)).


title: Artificial Neural Networks | Which loss function is used for the optimization of the model?

body:
I have a code sample from github. Link is here: github code sample

My question is "Which loss function is used for the optimization of the model?"

Adam Optimizer or CrossEntropyLoss

What are the differences between them?


title: PyTorch ConvNet randomly learns a single label and gets stuck

body:
I am trying to classify two types of images (i.e. 2 labels). I have over 30k images for my training dataset, and have built the CNN that's described below.

Here's the weird part - I've trained this network over 20 times, and I get 2 completely different behaviours -

1. The network will in fact "learn", loss decreases and after about 10 epochs I'll correctly classify about 80% of my test dataset.

2. After seeing ~1k pictures (not even a single epoch), the network will "decide" to always classify the images as only one of the labels! Loss get's stuck and nothing happens

The strange part - it's the exact same piece of code! nothing changes.
I tried debugging for so many hours and got to a point where I don't have a clue what's going on.

The only thing that I noticed is when the network fails to learn, then the initial output of the network (w/o performing even one backprop iteration) is something like -

model = MyNetwork()
images, labels  = iter(train_dataset_loader).next()
fw_pass_output = model(images)
print(fw_pass_output[:5])
---
tensor([[9.4696e-09, 1.0000e+00],
        [2.8105e-08, 1.0000e+00],
        [7.4285e-09, 1.0000e+00],
        [4.3701e-09, 1.0000e+00],
        [4.4942e-08, 1.0000e+00]], grad_fn=<SliceBackward>)

And on the other hand when the network does succeed in learning, it'll look like this -

tensor([[0.4982, 0.5018],
        [0.4353, 0.5647],
        [0.3051, 0.6949],
        [0.4823, 0.5177],
        [0.4342, 0.5658]], grad_fn=<SliceBackward>)

So as you can see - when the network manages to learn, seems like the initial weight initialisation is allowing more balanced results, while on another occasion it'll just arbitrarily assign everything to one of the classes, and from there it never manages to learn anything.

I tried swapping between Adam/SGD, lot's of different learning rates, different weight decay values, nothing helped...

What am I missing? What's causing this behaviour? Is it a bug in my code, or a concept that I'm missing?

class MyNetwork(nn.Module):
    def __init__(self):
        super(MyNetwork, self).__init__()

        self.conv1_layer = nn.Conv2d(3, 32, 5)
        self.conv2_layer = nn.Conv2d(32, 16, 3)
        self.conv3_layer = nn.Conv2d(16, 8, 2)

        self.layer_size_after_convs = 8 * 5 * 5
        TOTAL_NUM_OF_CLASSES = 2
        self.fc1 = nn.Linear(self.layer_size_after_convs, TOTAL_NUM_OF_CLASSES)

    def forward(self, x):
        """
        Perform a forward pass on the network
        """
        x = F.relu(self.conv1_layer(x))
        x = F.max_pool2d(x, (3, 3))

        x = F.relu(self.conv2_layer(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv3_layer(x))
        x = F.max_pool2d(x, (2, 2))
            x = x.view(-1, self.layer_size_after_convs)
        x = self.fc1(x)
        x = F.softmax(x, dim=1)
        return x

model = MyNetwork()
loss_func = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001, weight_decay=0.1)

total_steps = len(train_dataset_loader)

epochs = 100
for epoch_num in range(epochs):
    for i, (img_batch, labels) in enumerate(train_dataset_loader):
        optimizer.zero_grad()
        fw_pass_output = model(img_batch)
        loss_values = loss_func(fw_pass_output, labels)
        loss_values.backward()
        optimizer.step()

        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch_num+1, epochs, i+1, total_steps, loss_values.item()))


title: Why does my accuracy not improve after training for 10 epochs?

body:
I've programmed an image classifier with PyTorch.
When I train the net with several parameters, it randomly either makes progress and learns or it doesn't learn anything. I ran the code several times and for the same parameters I have different results.

class Network(nn.Module):
  def __init__(self):
    super(Network, self).__init__()

    self.conv1 = nn.Conv2d(3, 6, kernel_size=5)
    self.conv2 = nn.Conv2d(6, 12, kernel_size=5,)
    self.conv3 = nn.Conv2d(12, 24, kernel_size=3 )
    self.conv4 = nn.Conv2d(24, 24, kernel_size=3 )
    self.fc1 = nn.Linear(864, 300)
    self.fc2 = nn.Linear(300,2)

  def forward(self, x):
    x = F.relu(self.conv1(x))
    x = F.max_pool2d(x,2)

    x = F.relu(self.conv2(x))
    x = F.max_pool2d(x,2)

    x = F.relu(self.conv3(x))
    x = F.max_pool2d(x,2)

    x = F.relu(self.conv4(x))
    x = F.max_pool2d(x,2)
    x = F.max_pool2d(x,2)
    #print(x.shape)
    x = x.reshape(-1, 864)

    x = F.relu(self.fc1(x))
    x = self.fc2(x)
    return x

device= torch.device("cuda" if torch.cuda.is_available() else "cpu")

data = CatDogs('train/',transforms.Compose([
                                                transforms.Resize(256),
                                                transforms.CenterCrop(256),
                                                transforms.ToTensor(),
                                                transforms.Normalize(
                                                    mean = [0.485,0.456,0.406],
                                                    std= [0.229,0.224,0.225])
                                                ]))
train_set, test_set = torch.utils.data.random_split(data, [20000, 5000])

params = OrderedDict(
    batch_size = [10,20,30,40,50,60,70,80,90,100,110,120]
    ,lr = [.005]
    ,shuffle = [True]
    ,pin_memory = [True]
    ,num_workers = [2]
    )

m = RunManager()

for run in RunBuilder.get_runs(params):
    network = Network()
    network = network.to(device)
    loader = DataLoader(train_set,
                        pin_memory=run.pin_memory,
                        batch_size=run.batch_size,
                        shuffle=run.shuffle,
                        num_workers=run.num_workers)

    optimizer = optim.Adam(network.parameters(), lr=run.lr)
    network.train()
    m.begin_run(run, network, loader)
    for epoch in range(10):
      m.begin_epoch()
      for batch in loader: # Get Batch
        images, labels = batch
        images = images.to(device)
        labels = labels.to(device)
        preds = network(images) # Pass Batch
        loss = F.cross_entropy(preds, labels) # Calculate Loss
        optimizer.zero_grad()
        loss.backward() # Calculate Gradients
        optimizer.step() # Update Weights

        m.track_loss(loss, images)
        m.track_num_correct(preds, labels)
      m.end_epoch()
    m.end_run()
m.save('results_batchsize_3')

Example for my Output:

The Class RunManager is tracking several stats like loss, accuracy, and time, but it doesn't intervene with the training.

As a result, the network returns positive results and the loss decreases, while sometimes the loss stays pretty much the same.

If you have any suggestion whats causing this, please let me know.
Thank you very much for your help!


title: Size mismatch when fitting ResNet34 with 3x390x320 pixel images

body:
I tried to apply code for data valuation methods on my dataset.
In the original source code, images of size 3x32x32 pixels are fed into a fit-method for a ResNet34.

When I try to feed my 3x390x320 pixel images into a fit-method for the ResNet34 the forward method of ResNet throws a RuntimeError: size mismatch, m1: [10 x 61440], m2: [512 x 10]

I double checked and the input data for the forward method has the shape [10, 3, 390, 320] (10 pictures). In the source code below I always commented the shape of the tensors. In the original source code, the input data has the shape [10, 3, 32, 32] and works just fine.

How can I modify the model to run on my pictures?

def conv3x3(in_planes, out_planes, stride=1):
    return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False)

class BasicBlock(FitModule):
    expansion = 1

    def __init__(self, in_planes, planes, stride=1):
        super(BasicBlock, self).__init__()
        self.conv1 = conv3x3(in_planes, planes, stride)
        self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = conv3x3(planes, planes)
        self.bn2 = nn.BatchNorm2d(planes)
        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != self.expansion * planes:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_planes, self.expansion * planes,
                          kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(self.expansion * planes))

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)
        out = F.relu(out)
        return out

class ResNet(FitModule):
    def __init__(self, block, num_blocks, num_classes=10):
        super(ResNet, self).__init__()
        self.in_planes = 64

        self.conv1 = conv3x3(3, 64)
        self.bn1 = nn.BatchNorm2d(64)
        self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
        self.linear = nn.Linear(512 * block.expansion, num_classes)

    def _make_layer(self, block, planes, num_blocks, stride):
        strides = [stride] + [1] * (num_blocks - 1)
        layers = []
        for stride in strides:
            layers.append(block(self.in_planes, planes, stride))
            self.in_planes = planes * block.expansion
        return nn.Sequential(*layers)

    def forward(self, x):
        x = x.float()                                         # 10, 3, 390, 320
        out = F.relu(self.bn1(self.conv1(x).float()).float()) # 10, 64, 390, 320
        out = self.layer1(out)                                # 10, 64, 390, 320
        out = self.layer2(out)                                # 10, 128, 195, 160
        out = self.layer3(out)                                # 10, 256, 98, 80
        out = self.layer4(out)                                # 10, 512, 49, 40
        out = F.avg_pool2d(out, 4)                            # 10, 512, 12, 10
        out = out.view(out.size(0), -1)                       # 10, 61440
        out = self.linear(out)                          # runtime exception!
        return out

def ResNet18(num_classes=10):
    return ResNet(PreActBlock, [2, 2, 2, 2], num_classes)

Thanks in advance.

Best regards,
Fabian


title: Resolving dimensions mismatch while designing autoencoder

body:
I have the following Auto Encoder architecture (Pytorch):

        channels = 12
        # encoder
        self.conv0 = nn.Conv2d(1, channels, kernel_size=3, stride=1)
        self.mp0 = nn.MaxPool2d((2, 2), stride=2, return_indices=True)
        self.conv1 = nn.Conv2d(channels, channels * 2, padding=1, kernel_size=3, stride=1)
        self.mp1 = nn.MaxPool2d((3, 3), stride=2, return_indices=True)
        self.conv2 = nn.Conv2d(channels * 2, channels * 2, kernel_size=3, stride=1)
        # decoder
        self.convtransp0 = nn.ConvTranspose2d(channels * 2, channels * 2, kernel_size=3, stride=1)
        self.mup0 = nn.MaxUnpool2d((2, 2), stride=2)
        self.convtransp1 = nn.ConvTranspose2d(channels * 2, channels, kernel_size=3, stride=1)
        self.mup1 = nn.MaxUnpool2d((2, 2), stride=2)
        self.convtransp2 = nn.ConvTranspose2d(channels, 1, kernel_size=3, stride=1)

I intend to make it symmetric. My images are of size 512x512.
When I try to run it I get an exception because of mismatching dimensions between convtransp1  and mup1: torch.Size([5, 12, 256, 256]) and torch.Size([5, 12, 255, 255]) correspondingly. My question is regarding general attitude to solve such mismatch?

currently I'm taking the most simple approach:

def forward(self, x):
...
        y = self.convtransp1(y)
        y = y[:, :, :255, :255]
        y = F.relu(y)
        y = self.mup1(y, ixs0)

...


title: Graph classification given node-level outputs using global pooling in pytorch

body: I have node-level outputs for a graph classification task (using Graph neural nets). Based on this article on GCN, it seems like I have to introduce a pooling layer to transform my outputs into graph-level outputs, which makes sense. In Pytorch geometric, it seems like there are multiple options for this, under the "Global pooling layer" here. How do I choose which one to use?


title: Pytorch-Change Minibatch Size while training

body: How do you change the minibatch size while training? I do not want to re-initialize the dataloader because I want to keep track of the index.


title: Pipeline for Deep Learning Models

body:
I am working on a project, it has 5 different deep models shown in the picture. I want to execute the pipeline in a way so that all the model should work in a streamline without waiting to complete processing of all the input images for other models. Also in the pipeline, I need to off/on some model occasionally.

I tested the concept of pipeline in sklearn, I want to use the same type of concept to deploy models in production. Please need your help if you know any good way to implement that. I have all the model in pytorch.


title: Reshaping a tensor with one dimension having different dimension size

body:
Suppose I have a tensor x of dimension [num_elem, 128] and another tensor batch of dimension [num_elem, 1]. The entries in batch are integers from 0 to 511, indicating which batch each entry in x corresponds to. In addition, each batch may have different number of counts (e.g 0 appears 22 times, but 29 appears only 5 times).

I want to reshape x into a tensor of dimension [batch_size (512), unk, 128], where unk refers to the maximum count of an integer in batch. Currently I am finding out the frequency of the mode of batch, then construct a tensor x_reshaped of dimensions [512, mode_freq, 128]. Thereafter, I use a for loop to iterate through x, assign it to the correct batch (by referring to batch) and second dimension of x_reshaped. However, this is very slow.

Is there any other more efficient ways to do this? Thank you.


title: Cannot obtain same output tensor values in C++ side

body:
I try to make predictions for a 4-class classification project in C++ side using libtorch 1.4. However, I cannot obtain same predictions compared to Python side. Firstly, I obtain same input tensor values just before prediction. When I compare the output tensor values, I noticed that they are different. You can find those values in that picture:

Left side includes Python output tensor values and the prediction results for each input picture.

Right side includes C++ output tensor values and the prediction results for each input picture.

Could you offer a solution to obtain same output tensor values and prediction results?


title: How to train multiple losses in one model with freezing some part of the network optionally in pytorch?

body:
I'm new to pytorch.
I'm implementing a network that has two classifiers(yellow and purple) as shown in the figure.
The problem is that I want to freeze the red part when the network trains the yellow classifier and unfreeze the red part when the network trains the purple classifier.

A brief code that I imagine to implement is bellow

# x is input and y_yellow, y_purple are labels of yellow and purple classifiers respectively.
criterion = CrossEntropyLoss()
opt = SGD()
model = my_model()

opt.zero_grad()
yellow_out, purple_out = model(x)

# freeze red part requires_grad = False

yellow_loss = criterion(yellow_out, y_yellow)
yellow_loss.backward()
opt.step()

opt.zero_grad()
# unfreeze red part requires_grad = True

purple_loss = criterion(purple_out, y_purple)
purple_loss.backward()
opt.step

Please let me know the exact way to implement the idea.

Is the sequence of the code right?
Did I use correctly the zero_grad?
Did I miss something?
Is there any optional parameter that I need to use?


title: .detach() in Neural Style transfer Content and Style loss function

body:
I have grabbed this from Pytorch doc on Neural style transfer,https://pytorch.org/tutorials/advanced/neural_style_tutorial.html
Here in the content loss function (similarly in style loss function) we have self.target = target.detach(). I am not sure what it does, I know that detach gets a variable out of computational graph but I am unable to understand why it is done here?


title: How to adjust the batch data by the amount of labels in PyTorch

body:
I have made n-grams / doc-ids for document classification,
def create_dataset(tok_docs, vocab, n):
  n_grams = []
  document_ids = []
  for i, doc in enumerate(tok_docs):
    for n_gram in [doc[0][i:i+n] for i in range(len(doc[0]) - 1)]:
       n_grams.append(n_gram)
       document_ids.append(i)
  return n_grams, document_ids

def create_pytorch_datasets(n_grams, doc_ids):
  n_grams_tensor = torch.tensor(n_grams)
  doc_ids_tensor = troch.tensor(doc_ids)
  full_dataset = TensorDataset(n_grams_tensor, doc_ids_tensor)
  return full_dataset

create_dataset returns pair of (n-grams, document_ids) like below:
n_grams, doc_ids = create_dataset( ... )
train_data = create_pytorch_datasets(n_grams, doc_ids)
>>> train_data[0:100]
(tensor([[2076, 517, 54, 3647, 1182, 7086],
         [517, 54, 3647, 1182, 7086, 1149],
         ...
         ]),
 tensor(([0, 0, 0, 0, 0, ..., 3, 3, 3]))

train_loader = DataLoader(train_data, batch_size = batch_size, shuffle = True)

The first of tensor content means n-grams and the second one does doc_id.
But as you know, by the length of documents, the amount of training data according to the label would changes.
If one document has very long length, there would be so many pairs that have its label in training data.
I think it can cause overfitting in model, because the classification model tends to classify inputs to long length documents.
So, I want to extract input batches from a uniform distribution for label (doc_ids). How can I fix it in code above?
p.s)
If there is train_data like below, I want to extract batch by the probability like that:
  n-grams        doc_ids
([1, 2, 3, 4],      1)       ====> 0.33
([1, 3, 5, 7],      2)       ====> 0.33
([2, 3, 4, 5],      3)       ====> 0.33 * 0.25
([3, 5, 2, 5],      3)       ====> 0.33 * 0.25
([6, 3, 4, 5],      3)       ====> 0.33 * 0.25
([2, 3, 1, 5],      3)       ====> 0.33 * 0.25


title: Is the definition of the model created in pytorch based on a paper correct?

body:
I am using pytorch 1.4.0 to build a model based on this paper.
https://arxiv.org/abs/1806.11368[1]
However, the loss function doesn't go down at all, and I'm wondering if the code defining the model is wrong.
This image is an overview of the model.
https://i.stack.imgur.com/DTLJk.png
Here is code defining the model．
In the paper, the output is a single channel, but here, the model output a two-channel probability map in order to handle a one-hot representation.
#model.py

import torch
import torchvision.models as models
import torch.nn as nn

class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()

    def forward(self, x):
        return x

class MLP1(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(512,1024)
        self.r1 = nn.ReLU()
        self.dropout = nn.Dropout2d(p=0.5)
        self.fc2 = nn.Linear(1024,2)
        self.r2 = nn.ReLU()
        self.softmax = nn.Softmax2d()

    def forward(self, x):
        x = x.view(-1,512,32,32)
        x = x.permute(0,2,3,1)
        x = self.fc1(x)
        x = self.r1(x)
        x = self.dropout(x)
        x = self.fc2(x)
        x = self.r2(x)
        x = x.permute(0,3,1,2)
        x = x.reshape(-1,2,32,32)
        x = x.softmax(x)
        return x

And this is the call to the model.
Resnet18 is used for feature extraction.
import torch
import torch.nn as nn
from torchvision import models
from model import Identity,MLP1

resnet_model = models.resnet18(pretrained=True)
resnet_model.conv1 = nn.Conv2d(3, 64, kernel_size=(7, 7), stride=(1, 1), padding=(3, 3), bias=False)
resnet_model.avgpool = Identity()
resnet_model.fc = MLP1()

If this model is fine, I'd like to look for other causes.


title: Pytorch Biggraph (PBG) checkpoint save best only

body:
I'm using torchbiggraph_train script to tune a pytorch biggraph model over a graph with 2 bilion edges and 180 milion nodes, but with storage limitation of 300 GB. Each checkpoint of the embeddings consumes nearly 90 GB and I want to keep only the best set.
Is there any way to only keep the model (and embeddings) that has achieved the "best performance" so far? Like save_best_only option in keras model_checkpoint callback function (https://keras.io/api/callbacks/model_checkpoint/) or even some kind of early stopping?


title: PyTorch: Not being able to run my fit function

body:
I am using this table btw
input = np.array([
                  [[313, 1], #HCL
                   [323, 1],
                   [333, 1],
                   [343, 1]],
                  [[313, 10e-3], #Ortho
                   [323, 10e-3],
                   [333, 10e-3],
                   [343, 10e-3]],
                  [[313, 10e-3], #Para
                   [323, 10e-3],
                   [333, 10e-3],
                   [343, 10e-3]]
                  ], dtype='float32')   target = np.array([[[14.76, 16.42, 18.08, 23.41]],
                    [[5.87, 11.14, 13.20, 25.72]],
                    [[2.73, 4.42, 8.04, 13.68]]], dtype='float32') loss_fn = F.mse_loss loss = loss_fn(model(input), target)

Output:
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: UserWarning: Using a target size (torch.Size([3, 1, 4])) that is different to the input size (torch.Size([3, 4, 4])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

def fit(num_epochs, model, loss_fn, opt):
  for epoch in range(num_epochs):
    for xb, yb in train_dl:
      pred = model(xb)
      loss = loss_fn(preds, yb)
      loss.backward()
      opt.step()
      opt.zero_grad()

    if (epoch+1) % 10 == 0:
      print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

fit(100, model, loss_fn, opt)

Output:
Epoch [10/100], Loss: 25.1053
Epoch [20/100], Loss: 25.1050
Epoch [30/100], Loss: 25.1047
Epoch [40/100], Loss: 25.1043
Epoch [50/100], Loss: 25.1040
Epoch [60/100], Loss: 25.1036
Epoch [70/100], Loss: 25.1033
Epoch [80/100], Loss: 25.1030
Epoch [90/100], Loss: 25.1026
Epoch [100/100], Loss: 25.1023

/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: UserWarning: Using a target size (torch.Size([3, 1, 4])) that is different to the input size (torch.Size([3, 4, 4])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  """

I'm wondering if the shape mismatching gives an error in the fit function. Also I would like some general tips on how to make these matrices. So I don't run into this, maybe like a general rule of thumb. Or things I can do when I do run into this problem, because I don't wanna not be able to do anything once this happens.
EDIT: I fixed the error while in the fit function. But the warning persists. I am not sure what to do, because this gives very wrong data.


title: Welch Transformation to keep the dimensions

body:
I am trying to extract the scipy.signal.welch signal from a temporal 1d time series and since I am not adept in signal processing I have no idea why the dimensions shrink when returned.
I need to concatenate the temporal to the spectral as another channel so if the temporal is of shape:
[batches, channels, sample_length]

then I expect to get after the concatenation:
[batches, 2*channels, sample_length]

but when I try to train my model an error is thrown because the size of the spectral doesn't match the temporal (the temporal size is 16):
size mismatch, m1: [2 x 9], m2: [16 x 16]

I tried to look at the documentation but they don't mention why it shrinks it and how it can be avoided.


title: Does the sparse model influence inference time?

body:
I used the same model structure and the same dataset to train the CNN model several times, however, I got different inference time. I only changed the learning rate or repeated the training process with all same parameters, and those models show different inference times when all the settings were the same.
So, I propose the above question.
MacBook Pro 16', PyTorch=1.12.0, CPU is only used.
Thanks!


title: Custom CNN gives wrong output shape

body:
I need some help. I am trying to make a custom CNN, which should accept one channel images and do binary classification. This is the model:
class custom_small_CNN(nn.Module):

    def __init__(self, input_channels=1, output_features=1):
        super(custom_small_CNN, self).__init__()

        self.input_channels = input_channels
        self.output_features = output_features

        self.conv1 = nn.Conv2d(self.input_channels, 8, kernel_size=(7, 7), stride=(2, 2), padding=(6, 6), dilation=(2, 2))
        self.conv2 = nn.Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), dilation=(1, 1))
        self.pool = nn.MaxPool2d(kernel_size=(2, 2))
        self.fc1 = nn.Linear(in_features=1024, out_features=self.output_features, bias=True)
        self.dropout = nn.Dropout(p=0.5)
        self.softmax = nn.Softmax(dim=1)
        self.net_name = 'Custom_Small_CNN'

        self.net = nn.Sequential(self.conv1, self.pool, self.conv2, self.pool, self.fc1)

    def forward(self, x):
        x = self.conv1(x)
        x = self.pool(x)
        #x = self.dropout(x)
        x = self.conv2(x)
        x = self.pool(x)
        x = x.view(-1, 1024)
        x = self.dropout(x)
        x = self.fc1(x)
        if not self.output_features == 1:
            x = self.softmax(x)
        return x

However, when I put an example batch with 4 images (all zeros) in the model like this:
x = torch.from_numpy(np.zeros((4, 1, 256, 256))).float()
net = custom_small_CNN(output_features=2, input_channels=1).float()
output = net(x)

the output has shape torch.Size([16, 2]) instead of torch.Size([4, 2]), which is what I want and what e.g. a ResNet delivers as an output. What am I missing?
Thanks!


title: Why is the output of a convolutional network so exponentially large?

body:
I am trying to reproduce a unet result on Carvana dataset using Ternausnet in PyTorch using Lightning.
I am using DiceLoss for that with sigmoid activation function. I think I am running into an issue of a vanishing gradient, because all gradients of weights are 0, and I see the output of the network with min value of order 10^8.
What could be the issue here? How can I address the vanishing gradient? Also, if I use a different criterion, I see a problem of loss going into negative values without stopping (for BCE with logits, for instance).
Here is the code for my Dice loss:
class DiceLoss(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, logits, targets, eps=0, threshold=None):

        # comment out if your model contains a sigmoid or
        # equivalent activation layer
        proba = torch.sigmoid(logits)
        proba = proba.view(proba.shape[0], 1, -1)
        targets = targets.view(targets.shape[0], 1, -1)
        if threshold:
            proba = (proba > threshold).float()
        # flatten label and prediction tensors

        intersection = torch.sum(proba * targets, dim=1)
        summation = torch.sum(proba, dim=1) + torch.sum(targets, dim=1)
        dice = (2.0 * intersection + eps) / (summation + eps)
        # print(intersection, summation, dice)
        return (1 - dice).mean()


title: How to display the file path of the incorrectly classified instances?

body:
I am attempting to display the file path of the incorrectly classified instances of my model (using Pytorch) and have written the following code:
incorrect_examples = []

model_gpu.eval()
for data in test_loader:
   images, labels = data
   images = images.to(device)
   labels = labels.to(device)
   output = model_gpu(images)
   _, pred = torch.max(output,1)
   idxs_mask = ((pred == labels) == False).nonzero()
   incorrect_examples.append(images[idxs_mask].cpu().numpy())
   print(images[idxs_mask].cpu().numpy())
   print(test_loader.dataset.df.path[idxs_mask])

print(incorrect_examples)
print(len(incorrect_examples))

However, I am receiving the following error:
KeyError                                  Traceback (most recent call last)
<ipython-input-45-02179a61953e> in <module>()
     11    incorrect_examples.append(images[idxs_mask].cpu().numpy())
     12    print(images[idxs_mask].cpu().numpy())
---> 13    print(test_loader.dataset.df.path[idxs_mask])
     14
     15 print(incorrect_examples)

1 frames
/usr/local/lib/python3.6/dist-packages/pandas/core/series.py in __getitem__(self, key)
    869         key = com.apply_if_callable(key, self)
    870         try:
--> 871             result = self.index.get_value(self, key)
    872
    873             if not is_scalar(result):

/usr/local/lib/python3.6/dist-packages/pandas/core/indexes/base.py in get_value(self, series, key)
   4403         k = self._convert_scalar_indexer(k, kind="getitem")
   4404         try:
-> 4405             return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None))
   4406         except KeyError as e1:
   4407             if len(self) > 0 and (self.holds_integer() or self.is_boolean()):

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index_class_helper.pxi in pandas._libs.index.Int64Engine._check_type()

KeyError: tensor([[ 1],
        [ 2],
        [ 5],
        [12],
        [14],
        [15]], device='cuda:0')

Can anyone advise where I am going wrong? Thanks in advance!


title: How to create dictionary with names of files in folder in the same orderof files dataset.ImageFolder creates dataset

body:
I have a folder with test 3 files [img1.jpg, img2.jpg, img3.jpg] for prediction. I  used class ImageFolder and Dataloader to serve my pretrained Pytorch model.
I assume, ImageFolder class creates a dataset in ascending format.
Following code gives me prediction value to index for dictionary in format results ={0:'class0',1:'class1'..}.
for idx, (data, labels) in enumerate(test_loader):
        output = model(data)
        pred = torch.argmax(output, 1)
        pred = pred.detach().cpu()
        if pred.item() == 0:
            results[idx] = 'class0'

        else:
            results[idx] = 'class1'

How to feed on dataset names of files to the dictionary, because what I need is to provide JSON file with {'image1.jpg': 'class1', ...}?
os.listdir(path) is not working due to unsporting manners - it returns random order of file, but I am sensitive for order when the dataset is created.


title: Model predicts only wrong label despite only using the other label and has 0 specificity but high sensitivity?

body:
This might be a bit of a long shot but I'm not sure what else I can do. My issue is that I currently have a model that's supposed to be performing binary classification. After training I noticed that the sensitivity is very high (around 98%) but the specificity is close to 0. This was due to the fact that my model was only predicting 1's.
In order to check if the model's properly training or not, I filtered the data so that I would have data samples with only label 0's. However, the result was the same (only 1 predictions) which leads me to believe that my model is not training properly (if at all).
The code is below:
Model
import torch
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self, num, dim):
        super().__init__()
        self.hidden_dim = 100
        self.embedding_dim = 300
        self.dropout_p = 0.8
        self.feature_dim = dim
        self.num = num

        self.features_linear = nn.Linear(in_features=self.feature_dim, out_features=self.hidden_dim)
        nn.init.xavier_uniform_(self.features_linear.weight)

        self.embedding = nn.Embedding(num_embeddings=self.num, embedding_dim=self.embedding_dim)
        nn.init.xavier_uniform_(self.embedding.weight)

        self.output_linear = nn.Linear(in_features=(self.hidden_dim + self.embedding_dim), out_features=1)
        nn.init.xavier_uniform_(self.output_linear.weight)

    def forward(self, a, b):
        a_out = F.relu(self.features_linear(a))
        b_emb = self.embedding(b)
        b_sum = torch.sum(b_emb, dim=1)

        input_rep = torch.cat((a_out, b_sum), dim=1)
        input_rep = self.dropout(input_rep)
        output = self.output_linear(input_rep)

        return output

Solver code.
from sklearn.metrics import confusion_matrix
import torch
import torch.optim as optim

model = Model(num=16, dim=189)
criterion = nn.BCELoss(reduction='mean')
optimizer = optim.Adam(params=model.parameters())

with open(file='train_data.pkl', mode='rb') as f:
    train_data = pickle.load(f)

for epoch in range(200):
    epoch_loss = 0.0
    predicted_targets = []
    ground_truth_targets = []

    for batch in train_data:
        optimizer.zero_grad()

        input_1, input_2, input_3, input_4, label = batch

        if torch.cuda.is_available():
            input_1 = input_1.to('cuda')
            input_2 = input_2.to('cuda')
            input_3 = input_3.to('cuda')
            input_4 = input_4.to('cuda')
            label = label.to('cuda')

        output = input_2

        for i in range(len(input_3)):
            input_data = torch.cat([input_1, output.view(1, -1), input4[i].view(1, -1)], dim=1)
            output = torch.sigmoid(model(input_data, input_3[i].unsqueeze(dim=0)))

        loss = criterion(output, label.float().view(1, -1))
        epoch_loss += loss.item()

        loss.backward()
        optimizer.step()

        output_start = input_2.cpu().item()
        output_end = output.cpu().item()
        label_threshold = output_end / output_start
        prediction = 1 if label_threshold <= 0.1 else 0

        predicted_targets.append(prediction)
        ground_truth_targets.append(label.cpu().item())

    tn, fp, fn, tp = confusion_matrix(ground_truth_targets, predicted_targets).ravel()

    sensitivity = tp / (tp + fn)
    specificity = tn / (tn + fp)

    if epoch % 5 == 0:
        print('Epoch: %d | loss: %.6f | sens: %.4f | spec: %.4f' % (epoch, epoch_loss, sensitivity, specificity)

Sorry if the code is a bit long, I tried to keep the minimal essential framework in tact without cluttering too much. The data can be found here.
If anyone can provide some feedback or any ideas on what might be the problem, that'd be greatly appreciated. I've been trying to find out what's wrong for the past couple of weeks but haven't had any luck. Thanks!


title: How to leave no storage footprint in pytorch?

body:
I am running a temporal fusion transformer in pytorch in a loop where I need the model to be retrained every x time intervals.  I am using a model checkpoint to save the best model in each train.
chkpt_callback = ModelCheckpoint('C:\\Users\\Administrator\\Documents\\chk.ckpt')

I am having several problems here.  The model checkpoint does not overwrite the last model checkpoint and keeps creating new checkpoint files.  Additionally, there are new pytorch lightning model versions that are added every loop that I have to manually delete.
Additionally, previous models I have trained with the EarlyStopping feature are taking up storage and I cannot locate them.
Has anyone successfully built a pytorch model using the Trainer without leaving a storage footprint?


title: BucketIterator returning labels instead of text

body:
This is what my code looks like:
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Fields

label_field = Field(sequential=False, use_vocab=False, batch_first=True, dtype=torch.float)
text_field = Field(tokenize='spacy', lower=True, include_lengths=True, batch_first=True)
fields = [('labels', label_field), ('Text', text_field)]

# TabularDataset

source_folder = '/dir/'
train, valid, test = TabularDataset.splits(path=source_folder, train='train.csv', validation='val.csv', test = 'test.csv',
                                           format='CSV', fields=fields, skip_header=True)

# Iterators

train_iter = BucketIterator(train, batch_size=32, sort_key=lambda x: len(x.Text),
                            device=device, sort=True, sort_within_batch=True)
valid_iter = BucketIterator(valid, batch_size=32, sort_key=lambda x: len(x.Text),
                            device=device, sort=True, sort_within_batch=True)
test_iter = BucketIterator(valid, batch_size=32, sort_key=lambda x: len(x.Text),
                            device=device, sort=True, sort_within_batch=True)

# Vocabulary

text_field.build_vocab(train, min_freq=3)
label_field.build_vocab(train, min_freq=3)

When I try to unpack the variables by:
for x in train_iter:
    print(x.Text)

I end up with the labels instead of the the Text, and vice versa if I try print(x.labels)


title: How to accelerate staged-model execution with GPU？

body:
The whole model has a hierarchical structure (e.g. 2 levels). That is, the model on the first stage takes the input and gives an output, then based on this output to choose a model from many on the next stage to use. Then I give the same input to the picked second-stage model and get an output as the final prediction of the entire process. All models on each stage is a fairly simple BPNN with 2 hidden layers. Each input is an integer vector.
Say I have a middle-sized test set (sized 0.1M - 1M) and I want to do the process above for every test case.
Here is the example code I use now:
import time, torch

model = [[m0],
         [m1, m2, m3, m4, m5, m6, m7, m8, m9, m10]] # mi is a BPNN on GPU with two hidden layers
inputs = [[1,5,6,2,1,8], [3,5,34,26,1,2],...]    # size from 100,000 to 1,000,000
device = torch.decice('cuda:0' if torch.cuda.is_available() else 'cpu')    # 'cuda:0'

torch.cuda.synchronize()
start = time.time()
for input in inputs:
    vector = torch.tensor(input,dtype=torch.float32,device=device)
    pos = model[0][0](vector).item() # This output should be an integer to index the next model to use
    prediction = model[1][pos](vector)

torch.cuda.synchronize()
end = time.time()
print(end-start)

What kind of adjustments could I make to accelerate the whole process to reduce the execution time? (e.g. reduce the transfer to and from GPU memory? Or recreate a whole model and make the model selection inside the model rather than between the stages?)


title: Tensor size mismatch using fastai Learner

body:
I'm having some trouble building an autoencoder with MVTec AD dataset.
In particular, I'm trying to replicate this guide, which is based on MNIST.
The code is the following:
m = torchvision.models.resnet34(pretrained = True).cuda()
m = nn.Sequential(*list(m.children())[:-3]).cuda()

t,_ = next(iter(load_train_dataset()))

t.size()
  > torch.Size([1, 3, 224, 224])

m(t).size()
  > torch.Size([1, 256, 14, 14])

code_sz = 32

conv = nn.Conv2d(256, code_sz, kernel_size=(2,2)).cuda()

m.add_module('CodeIn',conv)
m(t).size()
  > torch.Size([1, 32, 13, 13])

class UpSample(nn.Module):
    def __init__(self,feat_in,feat_out,out_shape=None,scale=2):
        super().__init__()
        self.conv = nn.Conv2d(feat_in,feat_out,kernel_size=(3,3),stride=1,padding=1)
        self.out_shape,self.scale = out_shape,scale

    def forward(self,x):
        return self.conv(
            nn.functional.interpolate(
                x,size=self.out_shape,scale_factor=self.scale,mode='bilinear',align_corners=True))

def get_upSamp(feat_in,feat_out, out_shape=None, scale=2, act='relu'):

    upSamp = UpSample(feat_in,feat_out,out_shape=out_shape,scale=scale).cuda()

    layer = nn.Sequential(upSamp)

    if act == 'relu':
        act_f = nn.ReLU(inplace=True).cuda()
        bn = nn.BatchNorm2d(feat_out).cuda()
        layer.add_module('ReLU',act_f)
        layer.add_module('BN',bn)
    elif act == 'sig':
        act_f = nn.Sigmoid()
        layer.add_module('Sigmoid',act_f)
    return layer

def add_layer(m,feat_in,feat_out,name,out_shape=None,scale=2,act='relu'):
    upSamp = get_upSamp(feat_in,feat_out,out_shape=out_shape,scale=scale,act=act)
    m.add_module(name,upSamp)

add_layer(m,code_sz,256,'CodeOut')
m(t).size()
  > torch.Size([1, 256, 26, 26])

add_layer(m,256,128,'Upsample0',out_shape=(56,56),scale=None)
m(t).size()
  > torch.Size([1, 128, 56, 56])

add_layer(m,128,64,'Upsample1')
m(t).size()
  > torch.Size([1, 64, 112, 112])

add_layer(m,64,3,'Upsample2', act='sig')
m(t).size()
  > torch.Size([1, 3, 224, 224])

output = m(t)
print("out", output.size())
print("img", t.size())
  > out torch.Size([1, 3, 224, 224])
    img torch.Size([1, 3, 224, 224])

learn = Learner(dls_train,m,loss_func=F.mse_loss)

learn.lr_find(end_lr=10000000)

The code fails here with the following output:
/usr/local/lib/python3.6/dist-packages/fastai/learner.py:166: UserWarning: Using a target size (torch.Size([64])) that is different to the input size (torch.Size([64, 3, 224, 224])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
  if len(self.yb): self.loss = self.loss_func(self.pred, *self.yb)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-55-ab61ebca7e6a> in <module>()
----> 1 learn.lr_find(end_lr=10000000)

14 frames
/usr/local/lib/python3.6/dist-packages/torch/functional.py in broadcast_tensors(*tensors)
     63         if any(type(t) is not Tensor for t in tensors) and has_torch_function(tensors):
     64             return handle_torch_function(broadcast_tensors, tensors, *tensors)
---> 65     return _VF.broadcast_tensors(tensors)
     66
     67

RuntimeError: The size of tensor a (224) must match the size of tensor b (64) at non-singleton dimension 3

I don't really know what the initial warning means.
I'm a novice with Python, so I'm sure I'm missing something obvious here. I'm stuck and I don't really know how to fix this. Any help would be much appreciated!


title: Cannot assign BERT model in class field

body:
I have a class that extends the LightningModule class of Pytorch Lightning and has a field bert_model. I am trying to perform the following
self.bert_model = BertForTokenClassification.from_pretrained('bert-base-uncased')

However, after the execution of this line, the field bert_model is None. If I do this:
bert_model = BertForTokenClassification.from_pretrained('bert-base-uncased')
self.bert_model = bert_model

the variable bert_model is initialized correctly while the class field bert_model is still None.
Any suggestions on what could be the problem?


title: Using 128 bit floating point datatype with Pytorch (not a complex number)

body:
I am performing some operations in Pytorch that require a very high degree of precision. I am currently using torch.float64 as my default datatype. All my numbers are real. I would like to use a float128 datatype (since memory is not an issue for this simulation). However, the only datatypes I find in the documentation is torch.complex128 where the real and imaginary parts are both 64 bits. Is there a datatype or a way I can use all the 128 bits for my real numbers?
Thank you


title: How to import the SDN network environment created with mininet into pytorch for deep reinforcement learning

body: I want to study the congestion control of the SDN network based on deep reinforcement learning, but I don’t know how to import the SDN network environment built with mininet and Ryu controller into the Pytorch deep learning framework, or whether it is possible to add the SDN network environment to Gym first , and then import it into Pytorch


title: How to split Signature verification datsets into Genuine and Forgery inside the image Directory through pytorch?

body:
I am working on the Signature Verification project by using the PyTorch library. I want to split my dataset into genuine and forgery. Please, I wanna make sure I am not talking about data splitting into train and validation folders.
I have a lot of folders containing 55 images individually. Every dir has 1st 25 images genuine and the last 30 images forgery.
 sig_datasets
  001
  002
  003
   .
   .
   .
  4000

for example, 001 dir has 1st 25 images genuine and the last 30 images forgery.
"

The main thing which I wanna focus on is, my model is running on my datasets, but it is getting trained overall data and giving me the results like either this image is available in datasets or not. but I want this would show either this image is genuine or forgery. for that purpose, I want the model would train on genuine and forgery datasets.

Please help me to split data into genuine and forgery category.
Here is some code of datasets loading into the model.
please must reply how can I can split the datasets inside the folder of images?
learning_rate = 1e-3
transforms = transforms.Compose(
    [
        transforms.Resize((220, 220)),
        transforms.ToTensor()
    ])
root = './sig_datasets/'
train_dataset = datasets.ImageFolder(root=root+'Training', transform=transforms)
test_dataset = datasets.ImageFolder(root= root+'Test', transform=transforms)
train_dataloader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
print(train_dataloader)
test_dataloader = DataLoader(test_dataset, batch_size=batch_size, shuffle=True)
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')


title: Creating new channels from nearby pixels in pytorch

body:
Given a batch image tensor like B x C x W x H (batchSize,channels,width,height),
I would like to create a new tensor in which the new channels are the channels from nearby pixels (padded with 0s).
For instance, if I choose the nearby pixel size to be 3 x 3 (like a 3 x 3 filter) then there are 9 total nearby pixels and the final tensor size would be B x ( 9 * C ) x W x H.
Any recommendations on doing this, or do I just need to go the brute-force approach through iteration?


title: Visualize Tensor: RGB and axes issues

body:
I am creating a function that visualizes a Tensor, and encounter issues:

Get rid of the axes
Clipping input data to the valid range for imshow with RGB data ([0..1] for floats or [0..255] for integers). [I have seen recommendations to plot before normalizing but I cannot do that in my case]:

# Define transforms
transform = transforms.Compose(
    [transforms.ToPILImage(),
     transforms.ToTensor(),
     transforms.Normalize((0.5, ), (0.5, ))
])

def show_tensor_images(image_tensor, num_images=1, size=(1, 50, 300)):
    '''
    Function for visualizing images: Given a tensor of images, number of images, and
    size per image, plots and prints the images in a uniform grid.
    '''
    image_unflat = image_tensor.detach().cpu().view(-1, *size)
    Image = make_grid(image_unflat[:num_images], nrow=2)
    Image = Image/np.amax(Image)
    Image = np.clip(Image, 0, 1)
    plt.imshow(Image.permute(1, 2, 0).squeeze().reshape(100, 150, 3))
    plt.rcParams["axes.grid"] = False
    plt.show()


title: How to build PyTorch dataset with mutliple arrays for different channel

body:
I hope to know how to build up the dataset with multiple NumPy arrays as a data channel.
I have multiple arrays for 4-channel data as input and 1-channel data as an output.
For instance,
Example of Data Structure - Tensor Form
I think this form of data can be built with the below function,
from torch.utils.data.dataset import Dataset

But I am still having some issues to assign the tensor properly, specify the input, and output.
I hope to get some ideas or guides for this problem.
Thank you so much!


title: Create TabularDataSet but no Column names are given (Torchtext)

body:
I have a TSV File that looks like this:
3   It 's a lovely film with lovely performances by Buy and Accorsi .
2   No one goes unindicted here , which is probably for the best .
3   And if you 're not nearly moved to tears by a couple of scenes , you 've got ice water in your veins .
4   A warm , funny , engaging film .

There are no column names, but I want to create a TabularDataSet with the tsv file.
How would that be possible? In all the documentations and tutorials I saw, the column name of a file was needed (like at Point 3 in this Tutorial: http://mlexplained.com/2018/02/08/a-comprehensive-tutorial-to-torchtext/ ), or in this example in the "fields" variable:
import torch
import torch.nn as nn
import torch.optim as optim
from torchtext.data import Field, TabularDataset, BucketIterator

tokenize = lambda x: x.split()

text = Field(sequential=True, use_vocab=True, tokenize=tokenize, lower=True)
score = Field(sequential=False, use_vocab=False)

fields = {"quote": ("q", quote), "score": ("s", score)}

train_data, test_data, dev_data = TabularDataset.splits(
                                         path='mypath',
                                         train='sentiment.train.tsv',
                                         test= 'sentiment.test.tsv',
                                         dev = 'sentiment.dev.tsv',
                                         format='tsv',
                                         fields=fields)


title: How will filter(kernel) size affect the Transpose Convolution operation(ConvTranspose2d in Pytorch)?

body: Will larger kernel help upsample with better detail or a small sized kernel? And how would padding fit in this scenario?