title: Pytorch GAN model doesn't train: matrix multiplication error

body:
I'm trying to build a basic GAN to familiarise myself with Pytorch. I have some (limited) experience with Keras, but since I'm bound to do a larger project in Pytorch, I wanted to explore first using 'basic' networks.
I'm using Pytorch Lightning. I think I've added all necessary components. I tried passing some noise through the generator and the discriminator separately, and I think the output has the expected shape. Nonetheless, I get a runtime error when I try to train the GAN (full traceback below):
RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x9 and 25x1)
I noticed that 7 is the size of the batch (by printing out the batch dimensions), even though I specified batch_size to be 64. Other than that, quite honestly, I don't know where to begin: the error traceback doesn't help me.
Chances are, I made multiple mistakes. However, I'm hoping some of you will be able to spot the current error from the code, since the multiplication error seems to point towards a dimensionality problem somewhere. Here's the code.
import os

import pytorch_lightning as pl
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision.transforms as transforms
from skimage import io
from torch.utils.data import Dataset, DataLoader, random_split
from torchvision.utils import make_grid
from torchvision.transforms import Resize, ToTensor, ToPILImage, Normalize

class DoppelDataset(Dataset):
    """
    Dataset class for face data
    """

    def __init__(self, face_dir: str, transform=None):

        self.face_dir = face_dir
        self.face_paths = os.listdir(face_dir)
        self.transform = transform

    def __len__(self):

        return len(self.face_paths)

    def __getitem__(self, idx):

        if torch.is_tensor(idx):
            idx = idx.tolist()

        face_path = os.path.join(self.face_dir, self.face_paths[idx])
        face = io.imread(face_path)

        sample = {'image': face}

        if self.transform:
            sample = self.transform(sample['image'])

        return sample

class DoppelDataModule(pl.LightningDataModule):

    def __init__(self, data_dir='../data/faces', batch_size: int = 64, num_workers: int = 0):
        super().__init__()
        self.data_dir = data_dir
        self.batch_size = batch_size
        self.num_workers = num_workers

        self.transforms = transforms.Compose([
            ToTensor(),
            Resize(100),
            Normalize(mean=(123.26290927634774, 95.90498110733365, 86.03763122875182),
                      std=(63.20679012922922, 54.86211954409834, 52.31266645797249))
        ])

    def setup(self, stage=None):
        # Initialize dataset
        doppel_data = DoppelDataset(face_dir=self.data_dir, transform=self.transforms)

        # Train/val/test split
        n = len(doppel_data)
        train_size = int(.8 * n)
        val_size = int(.1 * n)
        test_size = n - (train_size + val_size)

        self.train_data, self.val_data, self.test_data = random_split(dataset=doppel_data,
                                                                      lengths=[train_size, val_size, test_size])

    def train_dataloader(self) -> DataLoader:
        return DataLoader(dataset=self.test_data, batch_size=self.batch_size, num_workers=self.num_workers)

    def val_dataloader(self) -> DataLoader:
        return DataLoader(dataset=self.val_data, batch_size=self.batch_size, num_workers=self.num_workers)

    def test_dataloader(self) -> DataLoader:
        return DataLoader(dataset=self.test_data, batch_size=self.batch_size, num_workers=self.num_workers)

class DoppelGenerator(nn.Sequential):
    """
    Generator network that produces images based on latent vector
    """

    def __init__(self, latent_dim: int):
        super().__init__()

        def block(in_channels: int, out_channels: int, padding: int = 1, stride: int = 2, bias=False):
            return nn.Sequential(
                nn.ConvTranspose2d(in_channels=in_channels, out_channels=out_channels, kernel_size=4, stride=stride,
                                   padding=padding, bias=bias),
                nn.BatchNorm2d(num_features=out_channels),
                nn.ReLU(True)
            )

        self.model = nn.Sequential(
            block(latent_dim, 512, padding=0, stride=1),
            block(512, 256),
            block(256, 128),
            block(128, 64),
            block(64, 32),
            nn.ConvTranspose2d(32, 3, kernel_size=4, stride=2, padding=1, bias=False),
            nn.Tanh()
        )

    def forward(self, input):
        return self.model(input)

class DoppelDiscriminator(nn.Sequential):
    """
    Discriminator network that classifies images in two categories
    """

    def __init__(self):
        super().__init__()

        def block(in_channels: int, out_channels: int):
            return nn.Sequential(
                nn.Conv2d(in_channels=in_channels, out_channels=out_channels, kernel_size=4, stride=2, padding=1,
                          bias=False),
                nn.BatchNorm2d(num_features=out_channels),
                nn.LeakyReLU(0.2, inplace=True),
            )

        self.model = nn.Sequential(
            block(3, 64),
            block(64, 128),
            block(128, 256),
            block(256, 512),
            nn.Conv2d(512, 1, kernel_size=4, stride=1, padding=0, bias=False),
            nn.Flatten(),
            nn.Linear(25, 1),
            nn.Sigmoid()
        )

    def forward(self, input):
        return self.model(input)

class DoppelGAN(pl.LightningModule):

    def __init__(self,
                 channels: int,
                 width: int,
                 height: int,
                 lr: float = 0.0002,
                 b1: float = 0.5,
                 b2: float = 0.999,
                 batch_size: int = 64,
                 **kwargs):

        super().__init__()

        # Save all keyword arguments as hyperparameters, accessible through self.hparams.X)
        self.save_hyperparameters()

        # Initialize networks
        # data_shape = (channels, width, height)
        self.generator = DoppelGenerator(latent_dim=self.hparams.latent_dim, )
        self.discriminator = DoppelDiscriminator()

        self.validation_z = torch.randn(8, self.hparams.latent_dim,1,1)

    def forward(self, input):
        return self.generator(input)

    def adversarial_loss(self, y_hat, y):
        return F.binary_cross_entropy(y_hat, y)

    def training_step(self, batch, batch_idx, optimizer_idx):
        images = batch

        # Sample noise (batch_size, latent_dim,1,1)
        z = torch.randn(images.size(0), self.hparams.latent_dim,1,1)

        # Train generator
        if optimizer_idx == 0:

            # Generate images (call generator -- see forward -- on latent vector)
            self.generated_images = self(z)

            # Log sampled images (visualize what the generator comes up with)
            sample_images = self.generated_images[:6]
            grid = make_grid(sample_images)
            self.logger.experiment.add_image('generated_images', grid, 0)

            # Ground truth result (ie: all fake)
            valid = torch.ones(images.size(0), 1)

            # Adversarial loss is binary cross-entropy
            generator_loss = self.adversarial_loss(self.discriminator(self(z)), valid)
            tqdm_dict = {'gen_loss': generator_loss}

            output = {
                'loss': generator_loss,
                'progress_bar': tqdm_dict,
                'log': tqdm_dict
            }
            return output

        # Train discriminator: classify real from generated samples
        if optimizer_idx == 1:

            # How well can it label as real?
            valid = torch.ones(images.size(0), 1)
            real_loss = self.adversarial_loss(self.discriminator(images), valid)

            # How well can it label as fake?
            fake = torch.zeros(images.size(0), 1)
            fake_loss = self.adversarial_loss(
                self.discriminator(self(z).detach()), fake)

            # Discriminator loss is the average of these
            discriminator_loss = (real_loss + fake_loss) / 2
            tqdm_dict = {'d_loss': discriminator_loss}
            output = {
                'loss': discriminator_loss,
                'progress_bar': tqdm_dict,
                'log': tqdm_dict
            }
            return output

    def configure_optimizers(self):
        lr = self.hparams.lr
        b1 = self.hparams.b1
        b2 = self.hparams.b2

        # Optimizers
        opt_g = torch.optim.Adam(self.generator.parameters(), lr=lr, betas=(b1, b2))
        opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2))

        # Return optimizers/schedulers (currently no scheduler)
        return [opt_g, opt_d], []

    def on_epoch_end(self):

        # Log sampled images
        sample_images = self(self.validation_z)
        grid = make_grid(sample_images)
        self.logger.experiment.add_image('generated_images', grid, self.current_epoch)

if __name__ == '__main__':

    # Global parameter
    image_dim = 128
    latent_dim = 100
    batch_size = 64

    # Initialize dataset
    tfs = transforms.Compose([
        ToPILImage(),
        Resize(image_dim),
        ToTensor()
    ])
    doppel_dataset = DoppelDataset(face_dir='../data/faces', transform=tfs)

    # Initialize data module
    doppel_data_module = DoppelDataModule(batch_size=batch_size)

    # Build models
    generator = DoppelGenerator(latent_dim=latent_dim)
    discriminator = DoppelDiscriminator()

    # Test generator
    x = torch.rand(batch_size, latent_dim, 1, 1)
    y = generator(x)
    print(f'Generator: x {x.size()} --> y {y.size()}')

    # Test discriminator
    x = torch.rand(batch_size, 3, 128, 128)
    y = discriminator(x)
    print(f'Discriminator: x {x.size()} --> y {y.size()}')

    # Build GAN
    doppelgan = DoppelGAN(batch_size=batch_size, channels=3, width=image_dim, height=image_dim, latent_dim=latent_dim)

    # Fit GAN
    trainer = pl.Trainer(gpus=0, max_epochs=5, progress_bar_refresh_rate=1)
    trainer.fit(model=doppelgan, datamodule=doppel_data_module)

Full traceback:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-28805d67d74b>", line 1, in <module>
    runfile('/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger/gan.py', wdir='/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger')
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger/gan.py", line 298, in <module>
    trainer.fit(model=doppelgan, datamodule=doppel_data_module)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 510, in fit
    results = self.accelerator_backend.train()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 57, in train
    return self.train_or_test()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 74, in train_or_test
    results = self.trainer.train()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 561, in train
    self.train_loop.run_training_epoch()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 550, in run_training_epoch
    batch_output = self.run_training_batch(batch, batch_idx, dataloader_idx)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 718, in run_training_batch
    self.optimizer_step(optimizer, opt_idx, batch_idx, train_step_and_backward_closure)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 485, in optimizer_step
    model_ref.optimizer_step(
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/core/lightning.py", line 1298, in optimizer_step
    optimizer.step(closure=optimizer_closure)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 286, in step
    self.__optimizer_step(*args, closure=closure, profiler_name=profiler_name, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/core/optimizer.py", line 144, in __optimizer_step
    optimizer.step(closure=closure, *args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/optim/adam.py", line 66, in step
    loss = closure()
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 708, in train_step_and_backward_closure
    result = self.training_step_and_backward(
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 806, in training_step_and_backward
    result = self.training_step(split_batch, batch_idx, opt_idx, hiddens)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/training_loop.py", line 319, in training_step
    training_step_output = self.trainer.accelerator_backend.training_step(args)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/cpu_accelerator.py", line 62, in training_step
    return self._step(self.trainer.model.training_step, args)
  File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/cpu_accelerator.py", line 58, in _step
    output = model_step(*args)
  File "/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger/gan.py", line 223, in training_step
    real_loss = self.adversarial_loss(self.discriminator(images), valid)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/wouter/Documents/OneDrive/Hardnose/Projects/Coding/0002_DoppelGANger/doppelganger/gan.py", line 154, in forward
    return self.model(input)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 93, in forward
    return F.linear(input, self.weight, self.bias)
  File "/usr/local/lib/python3.9/site-packages/torch/nn/functional.py", line 1690, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: mat1 and mat2 shapes cannot be multiplied (7x9 and 25x1)


title: LSTM-CNN to classify sequences of images

body:
I got an assignment and stuck with it while going down the rabbit hole of learning PyTorch, LSTM and cnn.
Provided the well known MNIST library I take combinations of 4 numbers and per combination it falls down into one of 7 labels.
eg:
1111 label 1 (follow a constant trend)
1234 label 2 increasing trend
4321 label 3 decreasing trend
...
7382 label 7 decreasing trend - increasing trend - decreasing trend
The shape of my tensor after loading of the tensor become (3,4,28,28) where the 28 comes from the MNIST image's width and height. 3 is the batch size and 4 is the channels (4 images).
I'm somewhat stuck with how to pass this into a PyTorch backed LSTM and CNN as basically all Google searches lead to articles where simply one image is passed in.
I was thinking of reshaping it to 1 long array of (pixel values) where I put all of the values of the first image row by row (28) after each other, then appended by the same approach for the second, third and fourth image. So that would make 4 * 28 * 28 = 3136.
Is my way of thinking on how to tackle this a correct one or should I rethink? I'm rather new to this all and looking for some guidance on how to go forward. I've been reading loads of articles, YT videos, ... but all seem to touch the basic stuff or alternatives of the same subject.
I have written some code but running it gives errors.
import numpy as np
import torch
import torch.nn as nn
from torch import optim, softmax
from sklearn.model_selection import train_test_split

#dataset = sequences of 4 MNIST images each
#datalabels =7

#Data
x_train, x_test, y_train, y_test = train_test_split(dataset.data, dataset.data_label, test_size=0.15,
                                                    random_state=42)
#model
class Mylstm(nn.Module):
    def __init__(self, input_size, hidden_size, n_layers, n_classes):
        super(Mylstm, self).__init__()
        self.input_size = input_size
        self.n_layers = n_layers
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size, hidden_size, n_layers, batch_first=True)
        # readout layer
        self.fc = nn.Linear(hidden_size, n_classes)

    def forward(self, x):
        # Initialize hidden state with zeros
        h0 = torch.zeros(self.n_layers, x.size(0), self.hidden_size).requires_grad_()
        # initialize the cell state:
        c0 = torch.zeros(self.n_layers, x.size(0), self.hidden_size).requires_grad_()
        out, (h_n, h_c) = self.lstm(x, (h0.detach(), c0.detach()))
        x = h_n[-1, :, 1]
        x = self.fc(x)
        x = softmax(x, dim=1)
        return x

#Hyperparameters
input_size = 28
hidden_size = 256
sequence_length = 28
n_layers = 2
n_classes = 7
learning_rate = 0.001
model = Mylstm(input_size, hidden_size, n_layers, n_classes)
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

#training
bs = 0
num_epochs = 5
batch_size=3

if np.mod(x_train.shape[0], batch_size) == 0.0:
    iter = int(x_train.shape[0] / batch_size)
else:
    iter = int(x_train.shape[0] / batch_size) + 1
bs = 0
for i in range(iter):
    sequences = x_test[bs:bs + batch_size, :]
    labels = y_test[bs:bs + batch_size]
    test_images = dataset.load_images(sequences)
    bs += batch_size

for epoch in range(num_epochs):
    for i in range(iter):
        sequences = x_train[bs:bs + batch_size, :]
        labels = y_train[bs:bs + batch_size]
        input_images = dataset.load_images(sequences)
        bs += batch_size
        images=(torch.from_numpy(input_images)).view(batch_size,4,-1)
        labels=torch.from_numpy(labels)
        optimizer.zero_grad()
        output = model(images)
        # calculate Loss
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

The error I'm currently getting is:

RuntimeError: input.size(-1) must be equal to input_size. Expected 28, got 784


title: Pytorch Tensor storages have the same id when calling the storage() method

body:
I'm learning about tensor storage through a blog (in my native language - Viet), and after experimenting with the examples, I found something that was difficult to understand. Given 3 tensors x, zzz, and x_t as below:
import torch

x = torch.tensor([[3, 1, 2],
                  [4, 1, 7]])
zzz = torch.tensor([1,2,3])
# Transpose of the tensor x
x_t = x.t()

When I set the storage of each tensor to the corresponding variable, then their ids are different from each other:
x_storage = x.storage()
x_t_storage = x_t.storage()
zzz_storage = zzz.storage()
print(id(x_storage), id(x_t_storage), id(zzz_storage))
print(x_storage.data_ptr())
print(x_t_storage.data_ptr())

Output:
140372837772176 140372837682304 140372837768560
94914110126336
94914110126336

But when I called the storage() method on each original tensor in the same print statement, the same outputs are observed from all tensors, no matter how many times I tried:
print(id(x.storage()), id(x_t.storage()), id(zzz.storage()))
# 140372837967904 140372837967904 140372837967904

The situation gets even weirder as I print them separately on different lines; sometimes their results are different and sometimes theirs are the same:
print(id(x.storage()))
print(id(x_t.storage()))
# Output:
# 140372837771776
# 140372837709856

So my question is, why are there differences between the id of the storages in the first case, and the same id is observed in the second? (and where did that id come from?). And what is happening in the third case?
Also, I want to ask about the method data_ptr(), as it was suggested to be used instead of id in one question I saw on Pytorch discuss, but the Docs in Pytorch just show no more detail. I would be glad if anyone can give me detailed answers to any/all of the questions.


title: How to continue training serialized AllenNLP model using `allennlp train`?

body:
Currently training models using AllenNLP 1.2:
allennlp train -f --include-package custom-exp /usr/training_config/mock_model_config.jsonnet -s test-mock-out

The config is very standard:
"dataset_reader" : {

        "reader": "params"

    },
    "data_loader": {
        "batch_size": 3,
        "num_workers": 1,
    },
    "trainer": {
        "trainer_params": "various"
    },

    "vocabulary": {
        "type": "from_files",
        "directory": vocab_folder,
        "oov_token": "[UNK]",
        "padding_token": "[PAD]",
    },

    "model": {
        "various params": ...
    }

and serializing them to  the test-mock-out directory (also have model.tar.gz).
Using the allennlp train command, is it possible to continue training? The documentation states Model.from_archive should be used, but it's unclear how the config should be adapted to use it.
http://docs.allennlp.org/v1.2.0/api/commands/train/


title: PyTorch - Replace a log_softmax funtion by F.cross_entropy with different result

body:
I'm working on the PyTorch tutorial "What is torch.nn really?". There a simple MNIST neural net is developed from scratch and then gets refactored piece for a piece with torch elements.
It is used a log_softmax function:
def log_softmax(x):
    return x - x.exp().sum(-1).log().unsqueeze(-1)

def model(xb):
    return log_softmax(xb @ weights + bias)

After finishing the simple neural net the first replacement is above function by (in a continuous Jupyter cell):
import torch.nn.functional as F
loss_func = F.cross_entropy

def model(xb):
    return xb @ weights + bias

The proof of the result is fine with a further call of
print(loss_func(model(xb), yb), accuracy(model(xb), yb))

which brings in both cases the same result, e. g.

loss_func 1:  tensor(0.0830, grad_fn=) tensor(1.)
vs.
loss_func 2:  tensor(0.0830, grad_fn=) tensor(1.)

Now I tried to make the replacement directly there where the definition of the log_softmax function is done, expecting a comparable result with the proof functions. But the result is totally different

loss_func 1:  tensor(-3578.4377, grad_fn=) tensor(0.5625)
vs.
loss_func 2:  tensor(310.3191, grad_fn=)
tensor(0.5625)

Can you please help me find my mistake?
Here is the complete code. It's close to the original, only the the comments are removed. The log_softmax func is outcommented and replaced by the F.cross_entropy.
from pathlib import Path
import requests

DATA_PATH = Path("data")
PATH = DATA_PATH / "mnist"

PATH.mkdir(parents=True, exist_ok=True)

URL = "https://github.com/pytorch/tutorials/raw/master/_static/"
FILENAME = "mnist.pkl.gz"

if not (PATH / FILENAME).exists():
        content = requests.get(URL + FILENAME).content
        (PATH / FILENAME).open("wb").write(content)

import pickle
import gzip

with gzip.open((PATH / FILENAME).as_posix(), "rb") as f:
        ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding="latin-1")

from matplotlib import pyplot
import numpy as np

pyplot.imshow(x_train[0].reshape((28, 28)), cmap="gray")
print(x_train.shape)

import torch

x_train, y_train, x_valid, y_valid = map(
    torch.tensor, (x_train, y_train, x_valid, y_valid)
)
n, c = x_train.shape
print(x_train, y_train)
print(x_train.shape)
print(y_train.min(), y_train.max())

import math

weights = torch.randn(784, 10) / math.sqrt(784)
weights.requires_grad_()
bias = torch.zeros(10, requires_grad=True)

"""
def log_softmax(x):
    return x - x.exp().sum(-1).log().unsqueeze(-1)

def model(xb):
    return log_softmax(xb @ weights + bias)
"""

import torch.nn.functional as F
loss_func = F.cross_entropy

def model(xb):
    return xb @ weights + bias

bs = 64  # batch size

xb = x_train[0:bs]  # a mini-batch from x
preds = model(xb)  # predictions
preds[0], preds.shape
print(preds[0], preds.shape)

def nll(input, target):
    return -input[range(target.shape[0]), target].mean()

loss_func = nll

yb = y_train[0:bs]
print(loss_func(preds, yb))

def accuracy(out, yb):
    preds = torch.argmax(out, dim=1)
    return (preds == yb).float().mean()

print(accuracy(preds, yb))

from IPython.core.debugger import set_trace

lr = 0.5  # learning rate
epochs = 2  # how many epochs to train for

for epoch in range(epochs):
    for i in range((n - 1) // bs + 1):
        #         set_trace()
        start_i = i * bs
        end_i = start_i + bs
        xb = x_train[start_i:end_i]
        yb = y_train[start_i:end_i]
        pred = model(xb)
        loss = loss_func(pred, yb)

        loss.backward()
        with torch.no_grad():
            weights -= weights.grad * lr
            bias -= bias.grad * lr
            weights.grad.zero_()
            bias.grad.zero_()

print("loss_func 1: ", loss_func(model(xb), yb), accuracy(model(xb), yb))

import torch.nn.functional as F
loss_func = F.cross_entropy

def model(xb):
    return xb @ weights + bias

print("loss_func 2: ", loss_func(model(xb), yb), accuracy(model(xb), yb))


title: Wrong image dimensions in OpenCV when converting color spaces

body:
I have 2 identical arrays, one obtained by simply reading the image and using
img = cv2.imread(path)
array_1 = cv2.cvtColor(img, cv2.COLOR_BGR2Lab)

and other by
from torchvision import transforms
transform = transforms.Compose([
    transforms.Lambda(lambda img: cv2.cvtColor(img, cv2.COLOR_BGR2Lab)),
    transforms.Lambda(lambda img: np.array(img)),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5]),
])

img_scaled = img * 1./255
img_lab_norm = transform(img).permute(1,2,0).numpy()
array_2 = (((img_lab_norm + 1) / 2)*255).astype(np.int)
img_rec_norm = cv2.cvtColor(array_2, cv2.COLOR_Lab2RGB)

For some reason in the cv2.cvtColor(array, cv2.COLOR_Lab2RGB) line array_1 works and array_2 throws this error
> Unsupported depth of input image:
>     'VDepth::contains(depth)'
> where
>     'depth' is 4 (CV_32S)

Doing (array_2 == array_1).all() returns True
I have no idea why OpenCV treats array_2 like it has 4 dimensions.


title: How to implement mini-batch descent in Pytorch for Seq2seq model

body:
I am a beginner and using the link https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html to implement a sequence to sequence model for the purpose of transliteration. It can be seen that the code in the function definition of train() looks something like this:
    for ei in range(input_length):
            encoder_output, encoder_hidden = encoder(
                   input_tensor[ei], encoder_hidden)
            encoder_outputs[ei] = encoder_output[0, 0]

It can be seen that the way this functions is by iterating over one text character by character. So the batch size is 1. So the output after the embedding layer is [1,1,embedding_layer_size].
I want to implement something like this:
  for i in batches:
        encoder_output, encoder_hidden = encoder(
               i, encoder_hidden)

So that I get [batch_size ,seq_length, embedding_layer_size]. And also decode the same way. Note that seq_length is different for each sequence in the batch
Please help me out, a pointer to resource will also suffice


title: Understanding torch.nn.Flatten

body:
I understand that Flatten removes all of the dimensions except for one. For example, I understand flatten():
> t = torch.ones(4, 3)
> t
tensor([[1., 1., 1.],
    [1., 1., 1.],
    [1., 1., 1.],
    [1., 1., 1.]])

> flatten(t)
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])

However, I dont get Flatten, especially I dont get meaning of this snippet from the doc:
>>> input = torch.randn(32, 1, 5, 5)
>>> m = nn.Sequential(
>>>     nn.Conv2d(1, 32, 5, 1, 1),
>>>     nn.Flatten()
>>> )
>>> output = m(input)
>>> output.size()
torch.Size([32, 288])

I felt the output should of size [160], because 32*5=160.
Q1. So why it outputted size [32,288]?
Q2. I also dont get meaning of shape information given in the doc:

Q3. And also meaning of parameters:


title: How to Predict K univariate time series with N-Beats algorithm and pytorch forecasting?

body:
I am trying to build K autoregressive model for time series prediction. In particular I am interested in the N-Beats algorithm (https://arxiv.org/abs/1905.10437).
The most promising implementation of it is available at: https://pytorch-forecasting.readthedocs.io/en/latest/api/pytorch_forecasting.models.nbeats.NBeats.html#pytorch_forecasting.models.nbeats.NBeats.from_dataset and it is the one I am trying to use.
I have tried several options but I  keep getting errors. As far as I understand the first step is to create a TimeSeriesDataset

from pytorch_forecasting.models.nbeats import NBeats
from pytorch_forecasting.data.timeseries import TimeSeriesDataSet
import pandas as pd

df_str= """',0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,time_idx\n0,13392.0,13382.0,13476.0,13478.0,13392.0,13382.0,13444.0,13447.0,13309.0,13306.0,13398.0,13399.0,13302.0,13300.0,13413.0,13412.0,0.0,0.0,0\n1,13465.0,13471.0,12813.0,12809.0,13474.0,13474.0,12809.0,12806.0,13343.0,13329.0,13340.0,13346.0,13366.0,13366.0,13345.0,13346.0,1.0,1.0,1\n2,13503.0,13472.0,12032.0,12015.0,13519.0,13523.0,12052.0,12054.0,13383.0,13373.0,13330.0,13328.0,13363.0,13368.0,13330.0,13319.0,2.0,2.0,2\n3,13502.0,13503.0,12061.0,12061.0,13517.0,13505.0,12071.0,12093.0,13339.0,13337.0,13376.0,13335.0,13336.0,13335.0,13324.0,13322.0,3.0,3.0,3\n4,13532.0,13536.0,12001.0,12001.0,13502.0,13498.0,12031.0,12023.0,13352.0,13350.0,13339.0,13380.0,13363.0,13369.0,13335.0,13341.0,4.0,4.0,4\n5,200.0,200.0,180.0,180.0,201.0,201.0,179.0,179.0,354.0,354.0,239.0,240.0,423.0,423.0,428.0,426.0,5.0,5.0,5\n6,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,6.0,6.0,6\n7,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,7.0,7.0,7\n8,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,8.0,8.0,8\n9,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,9.0,9.0,9\n10,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,10.0,10.0,10\n11,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,11.0,11.0,11\n12,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,12.0,12.0,12\n13,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,13.0,13.0,13\n14,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,14.0,14.0,14\n15,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,15.0,15.0,15\n16,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,16.0,16.0,16\n17,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,17.0,17.0,17\n18,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,18.0,18.0,18\n19,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,0.0,19.0,19.0,19\n'"""

df  = pd.read_csv(io.StringIO(df_str), index_col=0)

dataset = TimeSeriesDataSet(df_size, time_idx='time_idx',
                           target=[c for c in df_size.columns if c != 'time_idx'],
                            group_ids=[c for c in df_size.columns if c != 'time_idx']
                           )

This fails as:
ValueError                                Traceback (most recent call last)
<ipython-input-55-3736854a9150> in <module>
      3 df  = pd.read_csv(io.StringIO(df_str), index_col=0)
      4
----> 5 dataset = TimeSeriesDataSet(df_size, time_idx='time_idx',
      6                            target=[c for c in df_size.columns if c != 'time_idx' and c < 10],
      7                             group_ids=[c for c in df_size.columns if c != 'time_idx' and c >= 10]

~/n_beats_dnn/venv/lib/python3.8/site-packages/pytorch_forecasting/data/timeseries.py in __init__(self, data, time_idx, target, group_ids, weight, max_encoder_length, min_encoder_length, min_prediction_idx, min_prediction_length, max_prediction_length, static_categoricals, static_reals, time_varying_known_categoricals, time_varying_known_reals, time_varying_unknown_categoricals, time_varying_unknown_reals, variable_groups, dropout_categoricals, constant_fill_strategy, allow_missings, lags, add_relative_time_idx, add_target_scales, add_encoder_length, target_normalizer, categorical_encoders, scalers, randomize_length, predict_mode)
    353
    354         # validate
--> 355         self._validate_data(data)
    356         assert data.index.is_unique, "data index has to be unique"
    357

~/n_beats_dnn/venv/lib/python3.8/site-packages/pytorch_forecasting/data/timeseries.py in _validate_data(self, data)
    587                 )
    588         # check for "." in column names
--> 589         columns_with_dot = data.columns[data.columns.str.contains(r"\.")]
    590         if len(columns_with_dot) > 0:
    591             raise ValueError(

~/n_beats_dnn/venv/lib/python3.8/site-packages/pandas/core/indexes/base.py in __getitem__(self, key)
   4302             return promote(getitem(key))
   4303
-> 4304         if com.is_bool_indexer(key):
   4305             key = np.asarray(key, dtype=bool)
   4306

~/n_beats_dnn/venv/lib/python3.8/site-packages/pandas/core/common.py in is_bool_indexer(key)
    112                     # Don't raise on e.g. ["A", "B", np.nan], see
    113                     #  test_loc_getitem_list_of_labels_categoricalindex_with_na
--> 114                     raise ValueError(na_msg)
    115                 return False
    116             return True

ValueError: Cannot mask with non-boolean array containing NA / NaN values


I am new to pytorch, and i am working on a project,I wanna know how batch_sampler differs from sampler in pytorch dataloader modules, i have been used sampler parameter before where i just passed data indices in sampler parameters using SubsetRandomSampler. I read documentation where sampler and data_sampler both takes iterables as parameters, so whats the main differences between them and when we need to pass sampler or batch_sampler.


title: Remove zero dimension in pytorch

body:
I have torch.tensor which has dimension of 0 x 240 x 3 x 540 x 960
I just want to remove 0 dimension, so I want to have pytorch tensor
size of 240 x 3 x 540 x 960.
I used tensor= torch.squeeze(tensor) to try that, but zero dimension was not removed...
In my case, the size of tensor is variable so I can't hard code it to torch.squeeze..
Is there any simple way to remove undesired dimension in pytorch.tensor?


title: Optimizing with BCE is not working, nothing will change

body:
I have the following code:
import torch
import torch.nn as nn
import torch.nn.functional as F
from tqdm import tqdm
import matplotlib.pyplot as plt
import os
import keras
from random import choice
import sys
devicet = 'cuda' if torch.cuda.is_available() else 'cpu'
device = torch.device(devicet)
if devicet == 'cpu':
  print ('Using CPU')
else:
  print ('Using GPU')
cuda0 = torch.device('cuda:0')
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.step1 = nn.Linear(5, 25)
        self.step2 = nn.Linear(25, 50)
        self.step3 = nn.Linear(50, 100)
        self.step4 = nn.Linear(100, 100)
        self.step5 = nn.Linear(100, 10)
        self.step6 = nn.Linear(10, 1)
    def forward(self, x):
      x = F.relu(x)
      x = self.step1(x)
      x = F.relu(x)
      x = self.step2(x)
      x = F.relu(x)
      x = self.step3(x)
      x = F.relu(x)
      x = self.step4(x)
      x = F.relu(x)
      x = self.step5(x)
      x = F.relu(x)
      x = self.step6(x)
      x = F.relu(x)
      return (x)
net = Net()
x = torch.rand(10,5)
num = choice(range(10))
zero_tensor = torch.zeros(num, 1)
one_tensor = torch.ones(10-num, 1)
y = torch.cat((zero_tensor,one_tensor),0)
x.to(devicet)
y.to(devicet)
learning_rate = 1e-3
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
loss_fn = torch.nn.BCELoss()
acc_list = []
for i in tqdm(range(1000),desc='Training'):
    y_pred = net(x)
    loss = loss_fn(y_pred, y)
    loss.backward()
    optimizer.step()
    acc_list.append(abs(net(x).detach().numpy()[0]-y.detach().numpy()[0]))
    with torch.no_grad():
        for param in net.parameters():
            param -= learning_rate * param.grad
    optimizer.zero_grad()
print ('\nFinished training in {} epochs.'.format(len(acc_list)))
plt.plot(range(len(acc_list)),acc_list)
plt.show()
for i in range(10):
  print (str(net(x).detach().numpy()[i][0])+', '+str(y.detach().numpy()[i][0]))

When I run this, it consistently just prints out the following:
Image
Why won't it do any training? It works if I use MSE loss (actually, it only works sometimes with MSE loss, sometimes it does the same thing as in the image) , it's only when I use BCE that it stops working entirely.


title: Binary classification - BCELoss and model output size not corresponding

body:
I'm doing a binary classification, hence I used a binary cross entropy loss:
criterion = torch.nn.BCELoss()

However, I'm getting an error:
Using a target size (torch.Size([64, 1])) that is different to the input size (torch.Size([64, 2])) is deprecated. Please ensure they have the same size.

My model ends with:
    x = self.wave_block6(x)
    x = self.sigmoid(self.fc(x))
    return x.squeeze()

I tried removing the squeeze, but to no avail. My batch size is 64. It seems like I'm doing something simple wrong here. Is my model giving 1 output and BCE loss expecting 2 inputs? Which loss should I use then?


title: Why is saving state_dict getting slower as training progresses?

body:
I'm saving my model's and optimizer's state dict as follows:
if epoch % 50000 == 0:
  #checkpoint save every 50000 epochs
  print('\nSaving model... Loss is: ', loss)
  torch.save({
      'epoch': epoch,
      'model': self.state_dict(),
      'optimizer_state_dict': self.optimizer.state_dict(),
      'scheduler': self.scheduler.state_dict(),
      'loss': loss,
      'losses': self.losses,
      }, PATH)

When I first start the training it saves in less than 5 seconds. However, after a couple of hours of training it takes over a two minutes to save. The only reason I could think of is the list of losses. But I can't see how that would increase the time by that much.
Update 1:
I have my losses as:
self.losses = []

I'm appending the loss at each epoch to this list as follows:
    #... loss calculation
    loss.backward()
    self.optimizer.step()
    self.scheduler.step()

    self.losses.append(loss)


title: Find euclidean / cosine distance between a tensor and all tensors stored in a column of dataframe efficently

body:
I have a tensor 'input_sentence_embed' with shape torch.Size([1, 768])
There is a dataframe 'matched_df' which looks like
   INCIDENT_NUMBER           enc_rep
0  INC000030884498      [[tensor(-0.2556), tensor(0.0188), tensor(0.02...
1  INC000029956111      [[tensor(-0.3115), tensor(0.2535), tensor(0.20..
2  INC000029555353      [[tensor(-0.3082), tensor(0.2814), tensor(0.24...
3  INC000029555338      [[tensor(-0.2759), tensor(0.2604), tensor(0.21...

Shape of each tensor element in dataframe looks like
 matched_df['enc_rep'].iloc[0].size()
 torch.Size([1, 768])

I want to find euclidean / cosine similarity between 'input_sentence_embed' and each row of 'matched_df' efficently.
If they were scalar values, I could have easily broadcasted 'input_sentence_embed' as a new column in 'matched_df' and then find cosine similarity between two columns.
I am struggling with two problems

How to broadcast 'input_sentence_embed' as a new column to the
'matched_df'
How to find cosine similarity between tensors stored
in two column

May be someone can also suggest me other easier methods to achieve the end goal of finding similarity between a tensor value and all tensors stored in a column of dataframe efficently.


title: OOP structure of Pytorch

body:
I have started learning OOP for implementing my DL models in Pytorch. Where can I find the OOP structure of Pytorch. By structure I mean the class structures (i.e. inheritance structure of classes like Sequential container). I know Module class is base for most structures.
Also I was exploring Pytorch on github and I am confused how import statements in Pytorch works. For example, the Module class is defined in torch/nn/modules/module.py but we only import torch.nn and write nn.Module to represent the Module class. Should it not be nn.modules.module.Module by the way it is stored on github?
# Example of using Sequential
model = nn.Sequential(
nn.Conv2d(1,20,5),
nn.ReLU(),
nn.Conv2d(20,64,5),
nn.ReLU()
)

Sequential class in defined in /torch/nn/modules/container.py file but we just use nn.Sequential here?
Also I didn't find __call__ method in Sequential class then how the above code calling the class as a function.


title: How to re-use old weights in a slightly modified model?

body:
I have a CNN network built like this for a particular task.

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
        self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
        self.conv13 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

    def forward(self, x):
        in_size = x.size(0)

        x = F.relu(self.conv11(x))
        x = F.relu(self.conv12(x))
        x = F.relu(self.conv13(x))

        x = F.softmax(x, 2)

        return x

The model is stored using the torch built-in method like this.

net = Net()
optimizer = optim.SGD(net.parameters(), lr=1e-3)
state = {
    'state_dict': net.state_dict()
    'opt': optimizer.state_dict()
}
torch.save(state, 'model.pt')

I have increased a single layer in the network while the rest of the model was kept the same.

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
        self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
        self.conv13 = nn.Conv2d(256, 256, kernel_size=3, padding=1) # (new added)
        self.conv14 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

    def forward(self, x):
        in_size = x.size(0)

        x = F.relu(self.conv11(x))
        x = F.relu(self.conv12(x))
        x = F.relu(self.conv13(x)) (new added)
        x = F.relu(self.conv14(x))

        x = F.softmax(x, 2)

        return x

Since the other conv layers are kept the same, is there any way I can re-use the saved model to load the weights to conv11, conv12 and conv14 ? Instead of starting to train from beginning ?


title: How to deal with different categories in pytorch train, test, and holdout set

body:
I have a tabular pytorch model that takes in cities and zipcodes as a categorical embeddings.  However, I can't stratify effectively based on those columns.

How can I get pytorch to run if it's missing a categorical value in the test set that's not in the train set, or has a categorical value in the holdout set that was not in the train/test set?


title: Two almost exactly similar Pytorch neural network implementations but with consisently differing accuracies

body:
Here is the implementation of a Pytorch ResNet architecture on CIFAR10 Dataset

https://github.com/Paperspace/PyTorch-101-Tutorial-Series/blob/master/PyTorch%20101%20Part%202%20-%20Building%20Neural%20Networks%20With%20PyTorch.ipynb

And here is mine

https://github.com/VishakBharadwaj94/Resnets_from_scratch/blob/master/CIFAR_Dataset_with_ResNet_from_Scratch.ipynb

Despite repeatedly reintializing my weights and training it with the SAME hyperparamters, I consistently get a lower accuracy for mine.

I cannot see any difference in the code and I am unable to figure out what is causing it


title: Matrix Multiplication in 3,4 axes pytorch

body: I've two tensors of shape a(16,8,8,64) and b(64,64). Suppose, I extract last dimension of ainto another column vector c, I want to compute matmul(matmul(c.T, b), c). I want this to be done in each of the first 3 dimensions of a. That is the final product should be of shape (16,8,8,1). How can I achieve this in pytorch?


title: Image Generation using cGAN model

body:
I trained a cGAN model over the Flowers dataset, On Testing I could generate a Batch of images when a noise and embedding is fed as input. I am unable to generate a single image by feeding a noise and embedding instead of batch noise. On searching Internet I found out I shouldn't  use predict method since Batch normalising is done at each layers, Instead I should use eval(). eval() takes only one argument as input, But cGAN works only by feeding embedding and noise (2 arguments)

CODE FOR GENERATING BATCH Of IMAGES

  '''
   generator = torch.nn.DataParallel(gan_factory.generator_factory('gan').cuda())
   generator.load_state_dict(torch.load(pre_trained_gen))

 #loading state dict
'''

Next_batch = next(iter(data_loader))
embd = Next_batch['right_embed']
noise = Variable(torch.randn(64, 100)).cuda()
noise = noise.view(noise.size(0), 100, 1, 1)
fake_images = generator(embd, noise)

The above code works to generate batches of images. I made changes so to generate a single image

single = embd[0]
noise = Variable(torch.randn( 1,100)).cuda()
noise1 = noise1.view(noise1.size(0), 100, 1, 1)
new_shape = single.view(1,1024)

fake_images = generator(new_shape, noise1

got this error "Expected more than 1 value per channel when training, got input size torch.Size([1, 128])"


title: Metrics of validation set are weird

body:
I'm doing a project about image processing by deep learning, and the question I am facing is that metrics of validation data set are much different with those of training data set. The difference is shown as below.
image1
For the upper graph (training data set), the green, blue, red lines represent ms-ssim, loss of generator, and loss of discriminator respectively.
For the lower graph (validatation data set), the blue, gray, orange lines represent ms-ssim, loss of generator, and loss of discriminator respectively.
The only difference between these two data set is that validation set is unprocessed (1200*800) but training set is cropped (256*256).

Next, I attempt to crop validation set with the same setting. Just as what I guess, the metrics of two data set become similar. However, since the real images (testing data set) are not cropped, these metrics seem to be useless.

My question is :

Why 'cropping' can make such difference?
How can I make the metrics of two sets similar without cropping validation set?
Does it look like over-fitting?


title: Loss does not reduce with an auto-encoder like setup with PyTorch v1.4

body:
I am new to PyTorch and have written the following code. I am using one neural network for the encoding part followed by some processing of the encoded data and then using another network for the decoding this data.

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.optim.lr_scheduler import StepLR
import numpy as np
import itertools
import datetime

class Encoder(nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.fc1 = nn.Linear(4, 32, bias=False)
        self.fc2 = nn.Linear(32, 16, bias=False)
        self.fc3 = nn.Linear(16, 7, bias=False)

    def forward(self, x):
        x = self.fc1(x)
        x = torch.tanh(x)
        x = self.fc2(x)
        x = torch.tanh(x)
        x = self.fc3(x)
        output = torch.tanh(x)
        return output

def channel(codeword, snr_db, device):
    snr_value = 10 ** (snr_db / 10)
    h_real = torch.normal(mean=0, std=1, size=(codeword.shape[0], 1)) * torch.sqrt(torch.as_tensor(1/2))
    h_imag = torch.normal(mean=0, std=1, size=(codeword.shape[0], 1)) * torch.sqrt(torch.as_tensor(1/2))
    h_real_t = h_real.repeat(1, codeword.shape[1]).to(device)
    h_imag_t = h_imag.repeat(1, codeword.shape[1]).to(device)
    noise_real = torch.normal(mean=0, std=1, size=codeword.shape) * torch.sqrt(torch.as_tensor(1/(2*snr_value)))
    noise_imag = torch.normal(mean=0, std=1, size=codeword.shape) * torch.sqrt(torch.as_tensor(1/(2*snr_value)))
    noise_real = noise_real.to(device)
    noise_imag = noise_imag.to(device)
    faded_cw_real = torch.mul(h_real_t, codeword) + noise_real
    faded_cw_imag = torch.mul(h_imag_t, codeword) + noise_imag
    return torch.cat([faded_cw_real[:, :, None], faded_cw_imag[:, :, None]], dim=2), h_real, h_imag

class Decoder(nn.Module):
    def __init__(self):
        super(Decoder, self).__init__()
        self.bigru = nn.GRU(input_size=7, hidden_size=200, num_layers=20, bidirectional=True)
        self.fc0 = nn.Linear(4, 1)
        self.fc1 = nn.Linear(400, 4)

    def forward(self, x):
        x, states = self.bigru(x)
        output = torch.squeeze(self.fc0(torch.transpose(x, 2, 1)))
        output = torch.tanh(output)
        output = self.fc1(output)
        output = torch.tanh(output)
        # output = torch.softmax(output, dim=0)
        return output

def train(args, model1, model2, device, optimizer, epoch, snr):
    model1.train()
    model2.train()
    count = 1000
    for i in range(count):
        data = np.array([list(i) for i in itertools.product([-1, 1], repeat=4)])
        p = np.random.permutation(16)
        # p = np.random.randint(low=0, high=16, size=(16,))
        train_data = data[p]
        data_one_hot = np.eye(16)
        truth = data_one_hot[p]
        # truth = torch.as_tensor(truth).to(device).float()  # Uncomment this for BCE loss
        train_data = torch.as_tensor(train_data).float()
        train_data = train_data.to(device)
        # optimizer1.zero_grad()
        optimizer.zero_grad()
        output = model1(train_data)
        output = output.to(device)
        ch_out, h_r, h_i = channel(output, snr, device)
        h_r = torch.as_tensor(h_r[:, :, None].repeat(1, 7, 1)).to(device)
        h_i = torch.as_tensor(h_i[:, :, None].repeat(1, 7, 1)).to(device)
        dec_ip = torch.cat([ch_out, h_r, h_i], 2)
        dec_ip = torch.transpose(dec_ip, 2, 1)
        hat = model2(torch.as_tensor(dec_ip).float())
        loss_d = F.mse_loss(hat, train_data)
        # loss_d = F.binary_cross_entropy(hat, truth)
        loss_d.backward()
        optimizer.step()
        if i % 10 == 0:
            # print(f"Train epoch: {epoch}, Batch: {i}, Encoder Loss: {loss_e.item()}, SNR: {snr}")
            print(f"Train epoch: {epoch}, Batch: {i}, Decoder Loss: {loss_d.item()}, SNR: {snr}")

def main():
    epochs = 14
    learning_rate = 1
    learning_rate_step = 0.7
    no_cuda = False
    log_interval = 10
    use_cuda = not no_cuda and torch.cuda.is_available()
    device = torch.device("cuda" if use_cuda else "cpu")
    enc_model = Encoder().to(device)
    dec_model = Decoder().to(device)
    optimizer = optim.Adam(list(dec_model.parameters())+list(enc_model.parameters()), lr=learning_rate)
    scheduler = StepLR(optimizer, step_size=1, gamma=learning_rate_step)
    for epoch in range(1, epochs+1):
        snr = 20 - 20 * epoch / epochs
        train(log_interval, enc_model, dec_model, device, optimizer, epoch, snr)
        scheduler.step()

if __name__ == "__main__":
    main()

However when I run this, the output is:

Train epoch: x, Batch: y, Decoder Loss: 2.0, SNR: z

x, y and z values are dependent on the iteration. The decoder loss is stuck at 2.0. In fact, the loss starts at lesser than 2 and then gets stuck at 2.0.

Is there something simply wrong with the code?


title: Need pytorch help doing 2D convolutions of N images with N kernels all at once

body:
I have a torch tensor which is a stack of images. Let's say for kicks it is

im=th.arange(4*5*6,dtype=th.float32).view(4,5,6)

which is a 4x5x6 tensor, meaning four 5x6 images stacked vertically.

I want to convolve each layer with its own 2-D kernel so that

I_{out,j} = k_j*I_{in,j}, j=(1...4)

I can obviously do this with a for loop, but I'd like to take advantage of GPU acceleration and do all the convolutions at the same time. No matter what I try, I've only been able to use torch's conv2d or conv3d to produce a single output layer that is the sum of all the 2d convolutions. Or I can make 4 layers where each is the same sum of all the 2d convolutions. Here's a concrete example. Let's use im as defined above. Say that the kernel is defined by

k=th.zeros((4,3,3),dtype=th.float32)
n=-1
for i in range(2):
    for j in range(2):
        n+=1
        k[n,i,j]=1
        k[n,2,2]=1
print(k)

tensor([[[1., 0., 0.],
         [0., 0., 0.],
         [0., 0., 1.]],

        [[0., 1., 0.],
         [0., 0., 0.],
         [0., 0., 1.]],

        [[0., 0., 0.],
         [1., 0., 0.],
         [0., 0., 1.]],

        [[0., 0., 0.],
         [0., 1., 0.],
         [0., 0., 1.]]])

and from above, im is

tensor([[[  0.,   1.,   2.,   3.,   4.,   5.],
         [  6.,   7.,   8.,   9.,  10.,  11.],
         [ 12.,  13.,  14.,  15.,  16.,  17.],
         [ 18.,  19.,  20.,  21.,  22.,  23.],
         [ 24.,  25.,  26.,  27.,  28.,  29.]],

        [[ 30.,  31.,  32.,  33.,  34.,  35.],
         [ 36.,  37.,  38.,  39.,  40.,  41.],
         [ 42.,  43.,  44.,  45.,  46.,  47.],
         [ 48.,  49.,  50.,  51.,  52.,  53.],
         [ 54.,  55.,  56.,  57.,  58.,  59.]],

        [[ 60.,  61.,  62.,  63.,  64.,  65.],
         [ 66.,  67.,  68.,  69.,  70.,  71.],
         [ 72.,  73.,  74.,  75.,  76.,  77.],
         [ 78.,  79.,  80.,  81.,  82.,  83.],
         [ 84.,  85.,  86.,  87.,  88.,  89.]],

        [[ 90.,  91.,  92.,  93.,  94.,  95.],
         [ 96.,  97.,  98.,  99., 100., 101.],
         [102., 103., 104., 105., 106., 107.],
         [108., 109., 110., 111., 112., 113.],
         [114., 115., 116., 117., 118., 119.]]])

The right answer is easy if I do the for loop:

import torch.functional as F
for i in range(4):
    print(F.conv2d(im[i].expand(1,1,5,6),k[i].expand(1,1,3,3)))

tensor([[[[14., 16., 18., 20.],
          [26., 28., 30., 32.],
          [38., 40., 42., 44.]]]])
tensor([[[[ 75.,  77.,  79.,  81.],
          [ 87.,  89.,  91.,  93.],
          [ 99., 101., 103., 105.]]]])
tensor([[[[140., 142., 144., 146.],
          [152., 154., 156., 158.],
          [164., 166., 168., 170.]]]])
tensor([[[[201., 203., 205., 207.],
          [213., 215., 217., 219.],
          [225., 227., 229., 231.]]]])

As I noted earlier, the only thing I've been able to get is one sum of those four output images (or four copies of the same summed layer):

F.conv2d(im.expand(1,4,5,6),k.expand(1,4,3,3))

tensor([[[[430., 438., 446., 454.],
          [478., 486., 494., 502.],
          [526., 534., 542., 550.]]]])

I'm certain that what I want to do is possible, I just haven't been able to wrap my head around it yet. Does anyone have a solution to offer?


title: RNG implementation without unsigned int

body: I need implement random number generator by myself in Pytorch, and in my knowledge, all kinds of RNG need unsigned int in the implementation. However PyTorch only supports uint8 tensors but not uint32, uint64. Is there an RNG implementation that uses only int instead of uint?


title: Get the input channels for the conv2d from previous layer?

body:
I was wondering if there are many convolutional layers (conv1 --> conv2 ). How can we get the input channels parameter for the conv2 from the conv1 output channel?

class MyModel(nn.Module):
  def __init__(self, in_ch, num_features, out_ch2):
    super(MyModel, self).__init__()
    self.conv1 = nn.Conv2D(in_channels,num_features)
    self.conv2 = nn.Conv2D(in_channnels_from_out_channels_of_conv1,out_ch2)

Can I get the out_channels from the conv1 layer and use it as in_ch for conv2?


title: Index Array to one-hot ID matrix

body:
Assume I have an array [0, 2], I would like to output a matrix consisting of one-hot vector based on [0, 2] like
[ [1, 0, 0]
[0, 0, 1]]
(Note that the second dimension of output matrix is assumed to be 3 but it can be any number larger then argmax([0,2]) which is 2.

I can only think of this way achieves this function. Is there any simpler way.

t = torch.tensor([0,2])
dim2_size = 3
id_t = torch.zeros(t.shape[0], dim2_size)
row_idx = 0
for i in t:
  col_idx = i.item()
  id_t[row_idx, col_idx] = 1
  row_idx += 1
id_t


title: Remove LongTensor from a set of LongTensors in Python/Pytorch

body:
I have a set containing some LongTensors and I need to remove some of the LongTensors from the set, is there an efficient way of doing that in pytorch?

import torch
ks = {torch.LongTensor([1, 3]), torch.LongTensor([2, 3]), torch.LongTensor([3, 3])}
p = torch.LongTensor([1, 3])
ks.remove(p)

The method above returns

  KeyError: tensor([1, 3])

Is there an efficient way of removing the tensor??


title: How to calculate difference in y co-ordinate in percentage

body:
I am running a keypoint training and I wanted to create an accuracy parameter using points real and predicted values. Can someone help me please.

Say, I have an image of size (512,512) and I have annotated a point(real) at (x,y)=(100,100). My model predicts (X, Y) = (95,110), (X,Y) = (100,70), (X,Y) = (110,90)....
So I want to calculate how precise the model was with each value during training so that I can get the accuracy of the model.


title: How to build a model to predict a graph (not a image) in time series?

body: There is an adjacent matrix dataset that is based on time series. I would like to know if it is possible to build a neural network model to predict tn time point's matrix by using the previous time-series data. In my opinion, traditional models such as CNN may not fit for the sparse matrix graph.


title: I would like to know how to handle LSTM in Pytorch

body:
We are trying to reconstruct a model using LSTM.

LSTM Model Image

In Pytorch, LSTMs know that 3D data is entered.

From the picture above, I am currently going to put it in the following shape.

(batch, lstm_num, dv_batch, dvector)

lstm_num: Number of LSTMs used

I am wondering if there is another way to process the 4-dimensional data using lstm_num as a loop or a tensor itself.

The original processing method code is as follows.

In the original, we get a
(batch, dv_batch, dvector) in 3-dimensional data.

for epoch in range(init_epoch, max_epochs):
    for i, (dvec_batch, prob_batch) in enumerate(data_loader):
        dvec_batch = torch.reshape(dvec_batch,
                                   (-1, dvec_batch.size(2))).to(device)
        prob_batch = torch.reshape(prob_batch, (-1, )).to(device)
        outputs = model(dvec_batch).squeeze()
        loss = criterion(outputs, prob_batch)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        adjust_learning_rate(optimizer, epoch)


title: please explain why we called same class in the super method in fallowing code?

body:
I'm a beginner the field of machine learning and I was studying regression techniques using pytorch.  I'm having hard time understanding how the fallowing works? why we are calling same LogisticRegression in the super method of that same class?

class LogisticRegression(torch.nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LogisticRegression, self).__init__()
        self.linear = torch.nn.Linear(input_dim, output_dim)


title: Dimension error in implementation of a convolutional network

body:
I am trying to understand why my classifier has a dimension issue. Here is my code:

class convnet(nn.Module):

    def __init__(self, num_classes=1000):
        super(convnet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(kernel_size=2, stride = 2),
            nn.Conv2d(32, 32, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(kernel_size=2, stride = 2), #stride=2),
            nn.Conv2d(32, 64, kernel_size=3, stride=1),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(kernel_size=2, stride = 2),
        )

        self.classifier = nn.Sequential(
            nn.Linear(576, 128),
            nn.BatchNorm2d(128),
            nn.ReLU(inplace=True),
            nn.Linear(128, 64),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(64),
            nn.Linear(64,num_classes),
            nn.Softmax(),
       )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x,1) #x.view(x.size(0), 256 * 6 * 6)
        x = self.classifier(x)
        return x

def neuralnet(num_classes,**kwargs):
    model = convnet(**kwargs)
    return model

So here my issue is: expected 4D input (got 2D input)

I'm quite sure that the error arises from the flatten command, however I don't really understand why as the classifier has fully dense connections. If someone knows where I'm going wrong, that would be very helpful!

Thank you


title: How to print out and see Tensorflow model shapes/layers: Current error: AttributeError: '' object has no attribute 'summary'

body:
I was trying to see the details (including shapes of tensors, types/dimensions/attirbutes of different layers, total params, layer-by-layer summary) of a TF model https://github.com/Rayhane-mamah/Tacotron-2/blob/master/tacotron/synthesizer.py#L26.

I get this error:

(Pdb) p self.model
<tacotron.models.tacotron.Tacotron object at 0x7fc0a05d02b0>
(Pdb) p self.model.summary()
*** AttributeError: 'Tacotron' object has no attribute 'summary'

Why? Isn't model.summary() supposed to produce a detailed summary of the model and tensors?

How do I see all dimensions/shapes based summary/details of the model?


title: Saliency prediction with SalGAN

body:
I am studying informatics and I am participating in a course about computer vision where I currently have to work on a computer vision project of my choice. Therefore, I choose saliency prediction / eye fixation prediction as topic and I choose to reimplement the paper SalGAN: Visual Saliency Prediction with Generative Adversarial Networks (paper:https://arxiv.org/abs/1701.01081).

This is not my first course about machine learning & deep learning, but it is my first time implementing a CNN, GAN network. I choose pytorch as programming language.

Until recently, I thought my implementation is fine, but after training for 100 epochs I realized that the model is learning very very slow. (e.g. the training loss changes from 0.3478 to 0.3474 after one epoch). Since this is my first computer vision project, I am not sure if this is normal, or if this is due to a mistake in my loss calculation, or if this is just because of this really small learning rate of 0.0003...

Is it common in CV to have such small learning rates? I feel like even if I would train for 300 epochs (which would take like 2 days I guess) the model would not improve much because of these small updates.

You can take a look at my code here: https://github.com/bigabig/saliency/blob/master/train.py


title: PyTorch - Superior Model Performance by Misusing Loss Function (Negative Log Likelihood)?

body:
I misread PyTorch's NLLLoss() and accidentally passed my model's probabilities to the loss function instead of my model's log probabilities, which is what the function expects. However, when I train a model under this misused loss function, the model (a) learns faster, (b) learns more stably, (b) reaches a lower loss, and (d) performs better at the classification task.
I don't have a minimal working example, but I'm curious if anyone else has experienced this or knows why this is? Any possible hypotheses?
One hypothesis I have is that the gradient with respect to the misused loss function is more stable because the derivative isn't scaled by 1/model output probability.


title: Trying to overfit in GRU characters RNN

body:
I have a GRU network, which is manually built (i.e. no nn.GRU) with 2 vertical layers, 128-dim hidden layer, and sequence length of 64 chars.
I'm trying to overfit a small corpus taken out of Shakespeare:

I ran training for 500+ epochs, and after every 25 epochs I generate a sample with (very low temperature) by giving only the first letter "B". At first it's gibberish, but at the end it does get close to the actual text. The first version was:

This was without passing the hidden state between batches. I thought maybe this was my problem, so I passed the hidden state, detached, between batches. But I still don't get perfect overfit, and it actually seems to worsen it:

Here's how I implemented the GRU:
for i in range(self.n_layers):
    layer_input = layer_middle
    layer_middle = torch.zeros((batch_size, seq_len, self.h_dim))
    params = self.layer_params[i]
    for j in range(seq_len):
        x = layer_input[:, j, :].to(torch.device('cuda'))
        z = F.sigmoid(params['W1'](x) + params['W2'](layer_states[i]))
        r = F.sigmoid(params['W3'](x) + params['W4'](layer_states[i]))
        g = F.tanh(params['W5'](x) + params['W6'](r*layer_states[i]))
        layer_states[i] = z*layer_states[i] + (1-z)*g
        layer_middle[:, j, :] = layer_states[i]
    layer_middle = nn.Dropout(self.dropout)(layer_middle)
layer_output = torch.zeros((batch_size, seq_len, self.out_dim))
for j in range(seq_len):
    x = layer_middle[:, j, :].to(torch.device('cuda'))
    layer_output[:, j, :] = self.layer_params[-1]['W7'](x)

The params are nn.Linear with the correct shapes.
Any idea what might prevent me from overfitting to this tiny corpus??


title: What is the use of Batch Normalization Layer and Evolving normalization activation layers

body:
When to decide that we need a batch or evolving layer and how do we decide it?
I am currently using PyTorch and I want to understand how I can decide which layer to add?


title: Keras LSTM to Pytorch

body:
I am using the following code to apply sequential LSTM to time-series data with one value. It works fine with a Keras version. I am wondering how could I do the same using PyTorch?
import tensorflow
from tensorflow.keras import optimizers
from tensorflow.keras import losses
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, Input, Dropout, Embedding, LSTM
from tensorflow.keras.optimizers import RMSprop, Adam, Nadam
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.callbacks import TensorBoard

# training_dataset.shape = (303, 24, 1)
time_steps = 24
metric = 'mean_absolute_error'
model = Sequential()
model.add(LSTM(units=32, activation='tanh', input_shape=(time_steps, 1), return_sequences=True))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='adam', loss='mean_absolute_error', metrics=[metric])
print(model.summary())

batch_size=32
epochs=20

model.fit(x=training_dataset, y=training_dataset,
                       batch_size=batch_size, epochs=epochs,
                       verbose=1, validation_data=(training_dataset, training_dataset),
                       callbacks=[TensorBoard(log_dir='../logs/{0}'.format(tensorlog))])

testing_pred = model.predict(x=testing_dataset)


title: TensorboardX: writer.add_graph(model,inputs) show a lot of unknowing data and the code stops quickly

body:
When I use the writer.add_graph(model, inputs) to describe the structure of the net model, the screen shows a lot of data and the code stops quickly. If I remove the writer.add_graph(), the code works well. The showing data on the screen size is [8,2,32,128].  The inputs' size is [16,1,32,128], and the size of predicts=model(inputs) is [16,2,32,128]. The network is U-Net.
def train(train_loader, model, optimizer, scheduler, epoch, writer):
lock = threading.Lock()
threadlist = []
model.train()
end = time.time()
batch_time = func.AverageMeter()
losses = func.AverageMeter()
tbar = tqdm(train_loader)
for step, (inputs, targets) in enumerate(tbar):

    inputs, targets = inputs.to(device), targets.to(device)
    targets = targets.long()
    current_lr = scheduler.get_lr()[0]
    predicts = model(inputs)
    loss = criterion(predicts, targets.squeeze(1))
    losses.update(loss.item(), targets.size(0))
    # compute gradient
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    scheduler.step()
    tbar.set_description('Train {epoch:.5f} Loss {loss.val:.5f} {loss.avg:.5f} Lr {lr:.5f}'.format(epoch=epoch, loss=losses, lr=current_lr))
    # measure elapsed time
    batch_time.update(time.time() - end)
    end = time.time()
    writer.add_scalar('lr', current_lr, epoch*len(train_loader)+step)
    writer.add_scalar('train_loss', losses.avg, epoch)
    debug()
    writer.add_graph(model,inputs)

CLI output:


title: What is the difference between `.zero_grad()` and `.zero_grad`?

body: I am working on neural network and I find that, with *.grad_zero() I get loss function values properly and also converge to zero. Where, with *.grad_zero (with out bracket) gives loss function values in 5 digits. (13,564.23). So, what is the difference between them? Why "()" important in FPP. Thank you.


title: Get Predictions from Trained Pytorch Model

body:
I am using transfer learning to fine tune an inception_v3 model. After I train the model and store the best version off I am attempting to use it to generate predictions for my test set. Below is an example of my attempt on one image.
img_test=Image.open("img.png")

#Perform same transformations to image that the model used
transform_pipeline = transforms.Compose([
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])
img_test = transform_pipeline(img_test)

# I believe this is adding in the batch size of 1, but in looking around online it looked like I needed it
img = img_test.unsqueeze(0)
img = Variable(img)

model_ft(img)

When I do the above I get
RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

Which seems to imply my model weights are on my gpu and the variable is on the cpu, how do I move one or the other over so I can use it, or reference one that is on the opposite processor?


title: BertModel for 2 category classification - How to evaluate the performance

body:
I am building a fine-tuned BERT model for classification (with a linear layer in the end). The prediction should just be 1/0 (Yes, No).
When I am writing the evaluation part, I saw some people online did a F.log_softmax for the logits then use np.argmax to get the predicted label. However, I also saw people directly apply np.argmax on logits without the softmax activation. I am wondering which one should I follow and how to decide that?
Here's my model definition:
class ReviewClassification(BertPreTrainedModel):
def __init__(self, config):
    super().__init__(config)
    self.num_labels = 2

    self.bert = BertModel(config)
    self.dropout = nn.Dropout(config.hidden_dropout_prob)

    embedding_size = config.hidden_size

    self.classifier = nn.Linear(embedding_size, len(LABEL_NAME))
    self.init_weights()

def forward(
        self,
        review_input_ids=None,
        review_attention_mask=None,
        review_token_type_ids=None,
        agent_input_ids=None,
        agent_attention_mask=None,
        agent_token_type_ids=None,
        labels=None,
):

    review_outputs = self.bert(
        review_input_ids,
        attention_mask=review_attention_mask,
        token_type_ids=review_token_type_ids,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
    )

    feature = review_outputs[1]  # (batch_size, seq_len) -? Should it be (batch_size, hidden_size)

    # nn.CrossEntropyLoss applies F.log_softmax and nn.NLLLoss internally on your input,
    # so you should pass the raw logits to it.
    logits = self.classifier(feature)

    outputs = (logits,)  # + outputs[2:]  # add hidden states and attention if they are here

    if labels is not None:
        loss_fct = nn.CrossEntropyLoss()
        loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
        outputs = (loss,) + outputs

    return outputs  # (loss, logits, hidden_states, attentions)

Then this is my validation code
def model_validate(model, data_loader):
# Put the model in evaluation mode--the dropout layers behave differently
# during evaluation.
model.eval()
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)
if torch.cuda.device_count() > 1:
    model = nn.DataParallel(model)

label_prop = data_loader.dataset.dataset.label_prop()

total_valid_loss = 0

batch_size = data_loader.batch_size
num_batch = len(data_loader)

y_pred, y_true = [], []

# Evaluate data
for step, batch in tqdm(enumerate(data_loader), desc="Validation...", total=num_batch):
    b_review_input_ids = batch["review_input_ids"].to(device)
    b_review_attention_mask = batch["review_attention_mask"].to(device)
    b_review_token_type_ids = batch["review_token_type_ids"].to(device)

    b_binarized_label = batch["binarized_label"].to(device)

    # Tell pytorch not to bother with constructing the compute graph during
    # the forward pass, since this is only needed for backprop (training).
    with torch.no_grad():
        (loss, logits,) = model(review_input_ids=b_review_input_ids,
                                review_attention_mask=b_review_attention_mask,
                                review_token_type_ids=b_review_token_type_ids,

                                labels=b_binarized_label)

    total_valid_loss += loss.item()
    numpy_probas = logits.detach().cpu().numpy()
    y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
    y_true.extend(b_binarized_label.cpu().numpy())
# End of an epoch of validation

# put model to train mode again.
model.train()

ave_loss = total_valid_loss / (num_batch * batch_size)

# compute the various f1 score for each label
report = classification_report(y_true, y_pred, output_dict=True)
metrics_df = pd.DataFrame(report).transpose()

metrics_df = metrics_df.sort_index()

weighted_f1_score = metrics_df.loc['weighted avg', 'f1-score']
averaged_f1_score = metrics_df.loc['macro avg', 'f1-score']

return ave_loss, metrics_df, {
    "weighted": weighted_f1_score,
    "averaged": averaged_f1_score,
}

The other version I was trying is:
transfored_logits = F.log_softmax(logits,dim=1)
numpy_probas = transfored_logits.detach().cpu().numpy()
y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
y_true.extend(b_binarized_label.cpu().numpy())

The third version I was trying is:
transfored_logits = torch.sigmoid(logits)
numpy_probas = transfored_logits.detach().cpu().numpy()
y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
y_true.extend(b_binarized_label.cpu().numpy())

I also don't know how to understand the result. When I see online, people say if I set dim = 1 for log_softmax, the sum of probability for all feature (categories) should = 1. However, giving some examples below:
This is logits output: (for one batch - batch size = 16, num_labels = 2)
tensor([[ 1.1261, -1.8547],
    [ 0.6066, -1.1498],
    [ 1.3667, -2.0078],
    [ 2.0652, -2.6669],
    [ 1.0388, -1.7555],
    [ 0.6801, -1.1652],
    [ 0.8315, -1.3860],
    [ 1.5685, -2.2362],
    [ 0.1150, -0.3344],
    [ 2.0751, -2.6166],
    [ 1.5033, -2.1702],
    [ 0.1115, -0.3096],
    [ 0.8610, -1.4834],
    [ 1.5544, -2.2773],
    [ 2.1014, -2.6533],
    [ 0.7789, -1.3748]], device='cuda:0')

If I apply softmax first, F.log_softmax(logits,dim=1), I get:
tensor([[-0.0495, -3.0302],
    [-0.1593, -1.9157],
    [-0.0337, -3.4082],
    [-0.0088, -4.7409],
    [-0.0594, -2.8537],
    [-0.1467, -1.9920],
    [-0.1033, -2.3209],
    [-0.0220, -3.8267],
    [-0.4935, -0.9429],
    [-0.0091, -4.7008],
    [-0.0251, -3.6985],
    [-0.5046, -0.9257],
    [-0.0916, -2.4360],
    [-0.0214, -3.8531],
    [-0.0086, -4.7632],
    [-0.1098, -2.2635]], device='cuda:0')

The sum per row doesn't sum up to 1 and doesn't look like probability to me.
If I apply sigmoid, torch.sigmoid(logits)
tensor([[0.7551, 0.1353],
    [0.6472, 0.2405],
    [0.7969, 0.1184],
    [0.8875, 0.0650],
    [0.7386, 0.1474],
    [0.6638, 0.2377],
    [0.6967, 0.2000],
    [0.8276, 0.0965],
    [0.5287, 0.4172],
    [0.8885, 0.0681],
    [0.8181, 0.1025],
    [0.5278, 0.4232],
    [0.7029, 0.1849],
    [0.8255, 0.0930],
    [0.8910, 0.0658],
    [0.6854, 0.2018]], device='cuda:0')

It does look like probability more, although it still doesn't sum up to 1.
No matter which version I use, the predicted result is always the same in this case (since my 1 (Yes) Label has a really low incidence rate)
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])


title: How to fix connection issue when starting Jupyter kernel for PyTorch?

body:
I followed the steps in this link to install PyTorch from Conda (Anaconda) :
https://timoast.github.io/blog/installing-pytorch/
I then added the kernel for the virtualenv (cnn) as per the steps in the below link:
https://medium.com/@jeremy.from.earth/multiple-python-kernels-for-jupyter-lab-with-conda-c67e50de3aa3
When I start Jupyter and select the environment for the PyTorch kernel, I get connection failed error with below error messages. Any idea how I can get past this error to have a successful connection to the kernel from Jupyter ? .
[I 16:02:20.653 NotebookApp] Kernel started: 81b6a461-87a0-4542-a1c0-b564979c08f2
Traceback (most recent call last):
  File "C:\Users\mraja\.conda\envs\cnn\lib\runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\mraja\.conda\envs\cnn\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\ipykernel_launcher.py", line 15, in <module>
    from ipykernel import kernelapp as app
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\ipykernel\__init__.py", line 2, in <module>
    from .connect import *
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\ipykernel\connect.py", line 18, in <module>
    import jupyter_client
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\jupyter_client\__init__.py", line 4, in <module>
    from .connect import *
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\jupyter_client\connect.py", line 21, in <module>
    import zmq
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\zmq\__init__.py", line 50, in <module>
    from zmq import backend
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\zmq\backend\__init__.py", line 40, in <module>
    reraise(*exc_info)
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\zmq\utils\sixcerpt.py", line 34, in reraise
    raise value
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\zmq\backend\__init__.py", line 27, in <module>
    _ns = select_backend(first)
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\zmq\backend\select.py", line 28, in select_backend
    mod = __import__(name, fromlist=public_api)
  File "C:\Users\mraja\.conda\envs\cnn\lib\site-packages\zmq\backend\cython\__init__.py", line 6, in <module>
    from . import (constants, error, message, context,
ImportError: DLL load failed while importing error: The specified module could not be found.
[I 16:02:23.643 NotebookApp] KernelRestarter: restarting kernel (1/5), new random ports
Traceback (most recent call last):


title: Creating custom loss function for regression in CNN

body:
I am trying to make my CNN regression network run smoothly, but output is far from ground truth. The input is a binary image (600x600) which the background is black, and foreground is white. The ground truth associated with each input is an image with color range from 0 to 255 which is normalized between 0 and 1.
Below, you can see one sample of Input (x), ground truth(y) and predicted output (y_hat), respectively. The network can almost detect edges and background but in foreground all the predicted values are almost same (see predicted one).
I believe that since all foregrounds in input are 1, network cannot update weights correctly and it may need custom loss function to penalize loss function.
I am wondering whether my idea is right or not, if yes what kind of custom function I need?
I tried to penalize the foreground by custom loss function below, but it didn’t improve the result.
mse = nn.MSELoss(reduction=‘mean’)

def criterion(y, y_hat, x, loss, weight=0.1):
    y_hat_modified = torch.where(x[:,0:1]==1, weight*y_hat,y_hat)
    return loss(y,y_hat_modified)


title: Is a 0.0 loss rate on a Hybrid Quantum-Classical RNN possible?

body: I've been playing around with the hybrid quantum-classical RNN on the qiskit notebook using IBM Quantum Experience, and I managed to get a loss rate of 0.0001, however no more decimal places are calculated, even when changing the formatting. But seeing this made me wonder, "Is a zero loss rate possible?". At the time of writing this the program is on epoch 1450 out of 10000, so I'm not quite sure if I should even bother continuing this specific experiment. I am using pytorch and qiskit to run the program.


title: Problem running GRU model; missing argument for forward()

body:
I am working on a GRU and when I try to make predictions I get an error indicating that I need to define h for forward(). I have tried several things and ran out of patience after googling and scouring stack overflow for hours.
This is the class:
class GRUNet(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim, n_layers, drop_prob = 0.2):
        super(GRUNet, self).__init__()
        self.hidden_dim = hidden_dim
        self.n_layers = n_layers

        self.gru = nn.GRU(input_dim, hidden_dim, n_layers, batch_first=True, dropout=drop_prob)
        self.fc = nn.Linear(hidden_dim, output_dim)
        self.relu = nn.ReLU()

    def forward(self, x, h):
        out, h = self.gru(x,h)
        out = self.fc(self.relu(out[:,-1]))
        return out, h

    def init_hidden(self, batch_size):
        weight = next(self.parameters()).data
        hidden = weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().to(device)
        return hidden

and then this is where I load the model and try to make a prediction. Both of these are in the same script.
inputs = np.load('.//Pred//input_list.npy')
print(inputs.ndim, inputs.shape)
Gmodel = GRUNet(24,256,1,2)
Gmodel = torch.load('.//GRU//GRU_1028_48.pkl')
Gmodel.eval()
pred = Gmodel(inputs)

Without any other arguments to Gmodel I get the following:
Traceback (most recent call last):
  File ".\grunet.py", line 136, in <module>
    pred = Gmodel(inputs)
  File "C:\Users\ryang\Anaconda-3\envs\tf-gpu\lib\site-packages\torch\nn\modules\module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() missing 1 required positional argument: 'h'


title: What is the purpose of ListElementReference?

body:
I have a variable data which has type c10::List<c10::optional<c10::List<std::string>>>.
When I do the following for loop, the variable item seems to have type ListElementReference.
for (const auto& item: data ):
   ...

What is the purpose of the class ListElementReference in the torch code and how do I get the the underlying value it wraps?
https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/core/List.h#L57-L79


title: LU is not same as A when I used torch.solver

body:
let A=[[3,2],[1,-3]] and B=[[3],[-10]]
and solve equation AX=B by using torch.solve:
X, LU = torch.solve(B,A)

Then I got X=[[-1],[3]] and LU=[[3,2],[0.333,-3.666]].
According to definition of LU decompose, LU must be same as A, however they aren't same.
Can anyone explain this???
Thank you


title: Symmetrizing loss function in Pytorch for autoencoder

body:
I am trying to engineer an autoencoder for a complex image generation problem, and I want to try symmetrizing the loss function since both the target image and the network output are between 0 and 1. I am using the binary cross entropy loss, which is supposed to be good for image reconstruction. The way I currently handle the loss is:
loss = criterion(model(ipt), target)
optimizer.zero_grad()
loss.backward()
optimizer.step()

My thought for symmetrizing the loss would be:
loss = criterion(model(ipt), target)+criterion(target,model(ipt))
optimizer.zero_grad()
loss.backward()
optimizer.step()

but I'm not sure that this will work properly and respect the gradients correctly. Will this do what I expect? And more broadly, how would I know how to determine these things in general in the future?
Edit:
After trying this out, I get the error:
RuntimeError: the derivative for 'target' is not implemented

which makes sense. Is there a better way of symmetrizing this loss function?


title: When training a multi class CNN with PyTorch displays extraordinarily large loss

body:
I am currently trying train a CNN using PyTorch to predict a subject's age. The age group ranges from 0 to 116. I used the same model to train it on gender classification with two options: male or female.
I ported the same code for the age classification, I was getting errors. The error was due to our last fully connected layer not return a large enough output (in terms of matrix size, it was initially returning a 50 x 2 matrix due to our gender classifier but I switched it to 50 x 117 for the age classification based on the total age options.)
My issue now is that the training loop prints epochs with a huge loss (~3.5 while before, when training the gender classification, it was sub zero.)
Below is my code:
DataLoader class:
class MyDataset(Dataset):
    def __init__(self, root_directory, csv_file, image_path, transform = None):
        annotated_path = os.path.relpath(csv_file)   # Path to UTKFace Dataset and Annotations
        self.read_in_csv = pd.read_csv(annotated_path, index_col=False)
        self.image_path = os.path.join(root_directory, image_path)
        self.transform = transform
        self.labels = np.asarray(self.read_in_csv.loc[:,'age'])

    def __getitem__(self, index):
        attr = self.labels[index]
        image_name = str(self.read_in_csv.loc[index, 'file'])
        image = Image.open(image_name)

        if self.transform:
            image = self.transform(image)

        dict = {'image':image, 'label':attr}

        return dict

    def __len__(self):

        return len(self.read_in_csv.index)

CNN Architecture:
class ConvolutionalNN(nn.Module):
    def __init__(self):
        super(ConvolutionalNN,self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(3,96,kernel_size=7,stride=4),
            nn.BatchNorm2d(96), # Number of Features
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3,stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(96,256,kernel_size=5,padding=2),
            nn.BatchNorm2d(256),
            nn.ReLU(), # Default = False
            nn.MaxPool2d(kernel_size=3,stride=2))
        self.layer3 = nn.Sequential(
            nn.Conv2d(256,384,kernel_size=3,padding=1),
            nn.BatchNorm2d(384),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=3,stride=2))
        self.fc1 = nn.Linear(384*6*6,512)
        self.fc2 = nn.Linear(512,512)
        self.fc3 = nn.Linear(512,117)

    def forward(self,x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = out.view(out.size(0),-1)
        #print out.size()
        out = F.dropout(F.relu(self.fc1(out)))
        out = F.dropout(F.relu(self.fc2(out)))
        out = self.fc3(out)

        return out

Training Loop:
def training_loop(checkpoint = None, best=False):
    current_epoch = 1
    num_epochs = 50
    train_acc_history = []
    val_acc_history = []
    epoch_history = []
    learning_rate = 0.001
    best_val_acc = 0.0
    is_best = False
    criterion = nn.CrossEntropyLoss()
    ## Predict the Age and Gender of the Human in the Image
    optimizer = torch.optim.SGD(cnn.parameters(),lr=0.001,momentum=0.9)
    if checkpoint is not None:
        is_best = best
        current_epoch = checkpoint['epoch']
        train_acc_history = checkpoint['train_acc_history']
        val_acc_history = checkpoint['val_acc_history']
        best_val_acc = checkpoint['best_val_acc']
        optimizer.load_state_dict(checkpoint['optimizer'])
        epoch_history = checkpoint['epoch_history']

    print('Uploading our images now...')
    for epoch in range(current_epoch, num_epochs + current_epoch):
        print('Starting epoch %d / %d' % (epoch + 1, num_epochs + current_epoch))
        print('Learning Rate for this epoch: {}'.format(learning_rate))

        for i, batch in enumerate(train_loader):
            images, labels = batch['image'], batch['label']
            images = images.clone().detach()
            labels = labels.clone().detach()
            if use_gpu:
                images = images.cuda()
                labels = labels.cuda()

            optimizer.zero_grad()
            pred_labels = cnn(images)
            loss = criterion(pred_labels,labels)
            loss.backward()
            optimizer.step()

So this is my code. It does not seem to be training well.
Please let me know on what could be done to fix this.


title: How to specify the batch_size in PyTorch Sequantial model?

body:
I have a Sequential model in PyTorch:
model = nn.Sequential(
        nn.Embedding(alphabet_size, 64),
        nn.LSTM(64, ...),
        nn.Flatten(),
        nn.Linear(...),
        nn.Softmax()
    )

I would like to force the batch size to Embedding layer:
nn.Embedding(alphabet_size, 64)

as in Keras:
Embedding(alphabet_size, 64, batch_input_shape=(batch_size, time_steps)))

How to do it?


title: More than one GPU in Google Colab

body:
I was trying to run a Python code from GitHub on Google Colab, I need 2 GPUs to run the code, I installed Pytorch with this command: !conda install pytorch=0.4.1 cuda90 -c pytorch, When I try to run code, I get an error about cuda device:
Traceback (most recent call last):
  File "train.py", line 188, in <module>
    encoder.cuda(encoder_gpus[0])
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 258, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 185, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 185, in _apply
    module._apply(fn)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 191, in _apply
    param.data = fn(param.data)
  File "/usr/local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 258, in <lambda>
    return self._apply(lambda t: t.cuda(device))
RuntimeError: CUDA error (10): invalid device ordinal

Is that error because of the number of GPUs to run the code? how many GPUs are available in Google Colab?
How can we access more than one GPU in Google Colab?( is it possible to access more Gpus with "Colab Pro")


title: Visualize Autoencoder Output

body:
I come with a pretty noob question but I'm stuck...
I have created a Autoencoder with Pytorch and I trained it with the typical MNIST dataset and so on:
class Autoencoder(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.encoder_hidden_layer = nn.Linear(
            in_features=kwargs["input_shape"], out_features=kwargs["embedding_dim"]
        )
        self.encoder_output_layer = nn.Linear(
            in_features=kwargs["embedding_dim"], out_features=kwargs["embedding_dim"]
        )
        self.decoder_hidden_layer = nn.Linear(
            in_features=kwargs["embedding_dim"], out_features=kwargs["embedding_dim"]
        )
        self.decoder_output_layer = nn.Linear(
            in_features=kwargs["embedding_dim"], out_features=kwargs["input_shape"]
        )

    def forward(self, features):
        activation = self.encoder_hidden_layer(features)
        activation = torch.relu(activation)

        code = self.encoder_output_layer(activation)
        code = torch.relu(code)

        activation = self.decoder_hidden_layer(code)
        activation = torch.relu(activation)

        activation = self.decoder_output_layer(activation)
        reconstructed = torch.relu(activation)

        return reconstructed

model = Autoencoder(input_shape=784, embedding_dim=128)

criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001)

What I want now is to visualize the reconstructed images, but I don't know how to do it. I know it's quite simple but I cannot find a way. I know that the shape of the output is [128,784] because the batch_size is 128 and 784 is 28x28(x1channel).
Could anyone please tell me how could I get an image from my reconstructed tensor?
Thank you so much!


title: Decaying the learning rate from the 100th epoch

body:
Knowing that
learning_rate = 0.0004
optimizer = torch.optim.Adam(
    model.parameters(),
    lr=learning_rate, betas=(0.5, 0.999)
)

is there a way of decaying the learning rate from the 100th epoch?
Is this a good practice:
decayRate = 0.96
my_lr_scheduler = torch.optim.lr_scheduler.ExponentialLR(optimizer=my_optimizer, gamma=decayRate)


title: Why am I not getting an output with torch.save?

body:
I'm trying to save my trained model with torch.save:
torch.save(model, r'C:\Users\USER\AppData\Local\HDF_Group\HDFView\3.1.1\VAE2.hdf5')

but still not getting the output in my file .h5 (always empty)
Does someone know why?


title: convert [-1.0-1.0] float range to 24 bit RGB pixel

body:
I have 2D feature vector, want to convert these vector to 24 bit RGB pixel,
so want to get some abstract picture to train.
[ [-0.0069201705,0.051567946,0.04751903,0.032615487,0.004781969,...],
  ...
  [0.034763698,-0.010756443,-0.0738584,-0.14706105,-0.092746474,...]]

rgb = 65536 * r + 256 * g + b;

[ [rgb1_1, rgb2_1, rgb3_1, rgb4_1, rgb5_1,... ],
  ...
  [rgb1_64, rgb2_64, rgb3_64, rgb4_64, rgb5_64,... ]]

--
Okay, found some alternative answer myself
#im = Image.fromarray(s+200, mode='F')
from matplotlib import cm
s = (s + 200)/200.0 # input data should range from 0-1
im = Image.fromarray(cm.jet(s, bytes=True))
im.show()

now left part is to normalize [-1.0 - 1.0] to [0.0 - 1.0] without losing original feature vector information.


title: Where does the downloaded models in JupyterNotebook locat at?

body:
When I run this command, I can see that I download the model. So, where are these kind of downloaded model/data/tokenizers stored in my machine ?
Thanks !