title: Assertion Error assert len(mixture) ==len(enhanced) == len(clean)

body:
I am working on the Speech Enhancement model using PyTorch and when I write the training class as shown below I found that the len of the mixture (noisy signals) is not equal to clean signals and it raises an assertion error. I printed the length of these NumPy arrays and it is as follows:
len(clean) = 83560 len(enhanced) = 83560 len(mixture) = 93500

Any Idea how can I tackle this problem or how can I make these array equal in length

class Trainer(BaseTrainer):
    def __init__(
            self,
            config,
            resume: bool,
            model,
            loss_function,
            optimizer,
            train_dataloader,
            validation_dataloader,
    ):
        super(Trainer, self).__init__(config, resume, model, loss_function, optimizer)
        self.train_data_loader = train_dataloader
        self.validation_data_loader = validation_dataloader

    def _train_epoch(self, epoch):
        loss_total = 0.0

        for i, (mixture, clean, name) in enumerate(self.train_data_loader):
            mixture = mixture.to(self.device)
            clean = clean.to(self.device)

            self.optimizer.zero_grad()
            enhanced = self.model(mixture)
            loss = self.loss_function(clean, enhanced)
            loss.backward()
            self.optimizer.step()

            loss_total += loss.item()

        dl_len = len(self.train_data_loader)
        self.writer.add_scalar(f"Train/Loss", loss_total / dl_len, epoch)

    @torch.no_grad()
    def _validation_epoch(self, epoch):
        visualize_audio_limit = self.validation_custom_config["visualize_audio_limit"]
        visualize_waveform_limit = self.validation_custom_config["visualize_waveform_limit"]
        visualize_spectrogram_limit = self.validation_custom_config["visualize_spectrogram_limit"]

        sample_length = self.validation_custom_config["sample_length"]

        for i, (mixture, clean, name) in enumerate(self.validation_data_loader):
            assert len(name) == 1, "Only support batch size is 1 in enhancement stage."
            name = name[0]
            padded_length = 0

            mixture = mixture.to(self.device)  # [1, 1, T]

            # The input of the model should be fixed length.
            if mixture.size(-1) % sample_length != 0:
                padded_length = sample_length - (mixture.size(-1) % sample_length)
                mixture = torch.cat([mixture, torch.zeros(1, 1, padded_length, device=self.device)], dim=-1)

            assert mixture.size(-1) % sample_length == 0 and mixture.dim() == 3
            mixture_chunks = list(torch.split(mixture, sample_length, dim=-1))

            enhanced_chunks = []
            for chunk in mixture_chunks:
                enhanced_chunks.append(self.model(chunk).detach().cpu())

            enhanced = torch.cat(enhanced_chunks, dim=-1)  # [1, 1, T]
            enhanced = enhanced if padded_length == 0 else enhanced[:, :, :-padded_length]

            enhanced = enhanced.reshape(-1).numpy()
            clean = clean.numpy().reshape(-1)
            mixture = mixture.cpu().numpy().reshape(-1)
            assert len(mixture) ==len(enhanced) == len(clean)


title: Test Time Augmentation throwing a value error

body:
I am trying to use the Test Time Augmentation on my Classifier:

log_preds,y = learn.TTA(scale=1.1, ds_type=DatasetType.Valid, with_loss=True)

And this is the error that it threw:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-14-f33d9737819a> in <module>()
----> 1 log_preds,y = learn.TTA(scale=1.1, ds_type=DatasetType.Valid, with_loss=True)
      2 probs = np.mean(np.exp(log_preds),0)
      3
      4 accuracy(probs, y)

ValueError: too many values to unpack (expected 2)

Initially, I tried to find a way to use my custom test set in TTA but couldn’t figure out how to do it and DatasetType.Test was throwing an error so I decided to go with DatasetType.Valid and after running 8 epochs, I got the above error.


title: I get different inferences to result from my Image classifier. When I do inference on Pytorch as compared to fastai

body:
When I use pickle file of fastai and do prediction using learn.predict() method. And I use the same pickle file and convert that pickle file using this method to PyTorch. Below is the code snippet.

learn = load_learner(src_path,model_name)
var=learn.model
import pickle
new_str=f"/pytorch-weight/{model_name}"
file = open(new_str, 'wb')
pickle.dump(var, file)
# close the file
file.close()

And do the prediction using that generated pickle file in pytorch but I get different result as compared to fastai. What is the reason behind this I want to know.Am I doing the right way to convert that pickle file for pytorch inference?


title: how multiple underscores in a for loop work

body:
I am trying to understand the code from this website:
https://www.assemblyai.com/blog/end-to-end-speech-recognition-pytorch
for (waveform, _, utterance, _, _, _) in data:

Can someone please explain,what is the meaning of the above line and how it works ?
Like are we ignoring some values while unpacking the data or something else.
Complete code for the function:
def data_processing(data, data_type="train"):
    spectrograms = []
    labels = []
    input_lengths = []
    label_lengths = []
    for (waveform, _, utterance, _, _, _) in data:
        if data_type == 'train':
            spec = train_audio_transforms(waveform).squeeze(0).transpose(0, 1)
        elif data_type == 'valid':
            spec = valid_audio_transforms(waveform).squeeze(0).transpose(0, 1)
        else:
            raise Exception('data_type should be train or valid')
        spectrograms.append(spec)
        label = torch.Tensor(text_transform.text_to_int(utterance.lower()))
        labels.append(label)
        input_lengths.append(spec.shape[0]//2)
        label_lengths.append(len(label))

    spectrograms = nn.utils.rnn.pad_sequence(spectrograms, batch_first=True).unsqueeze(1).transpose(2, 3)
    labels = nn.utils.rnn.pad_sequence(labels, batch_first=True)

    return spectrograms, labels, input_lengths, label_lengths


title: Loss Analysis of Deep Learning

body:
I'm new to deep learning, and I have built a graph convolution network. I used 5 fold cross-validations. After plotting the mean train_loss (blue) and validate_loss (orange) together, I got this baby.
MSE loss
As you can see, from the curvilinear trend of validate_loss, it seems that my networks learn few things. (I guess data? GCN frameworks? learning rate?)
Could you guys specifically help me to figure out the bugs?
I would appreciate that! If you don't get my point, please let me know.
class Scorer(nn.Module):
    """
    Three conv_layers and two fc_layers with Dropout
    """
    def __init__(self):
        super(Scorer, self).__init__()
        self.conv_layer1 = GraphConvNet(5, 64)
        self.conv_layer2 = GraphConvNet(64, 128)
        self.conv_layer3 = GraphConvNet(128, 256) # (I have tried delete conv_layer3)
        self.fc_layer1 = nn.Linear(256, 128)
        self.drop_layer1 = nn.Dropout(0.5)
        self.fc_layer2 = nn.Linear(128, 64)
        self.drop_layer2 = nn.Dropout(0.5)
        self.out_layer = nn.Linear(64, 1)

    def forward(self, NormLap, feat):
        h = self.conv_layer1(NormLap, feat)
        h = F.leaky_relu(h)
        h = self.conv_layer2(NormLap, h)
        h = F.leaky_relu(h)
        h = self.conv_layer3(NormLap, h)
        h = F.leaky_relu(h)
        h = self.fc_layer1(h)
        h = self.drop_layer1(h)
        h = F.leaky_relu(h)
        h = self.fc_layer2(h)
        h = self.drop_layer2(h)
        h = F.leaky_relu(h)
        h = self.out_layer(h)
        h = F.leaky_relu(h)

Below is my networks and parameters:

    # parameter setting
    learning_rate = 0.001 # (I have tried 1e-1, 1e-2)
    weight_decay = 1e-3  # (I have tried 1e-4)
    epochs = 500
    batch_size = 50  # (I have tried 30)

    model = Scorer()
    loss_func = nn.MSELoss()
    optimizer = th.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)


title: How to understand the dimensions while converting a tensorflow generator model to pytorch

body:
def make_generator_model(n_classes=62):
    # Create class embedding channel
    input_label = tf.keras.layers.Input(shape=(1,))
    label_embedding = tf.keras.layers.Embedding(n_classes, 50)(input_label)
    upscaling = tf.keras.layers.Dense(7*7*1)(label_embedding)
    upscaling = tf.keras.layers.Reshape((7, 7, 1))(upscaling)

    # create seed encoding network
    seed_input = tf.keras.layers.Input(shape=(NOISE_DIM,))
    seed_fc = tf.keras.layers.Dense(7*7*256, use_bias=False)(seed_input)
    seed_fc = tf.keras.layers.BatchNormalization()(seed_fc)
    seed_fc = tf.keras.layers.LeakyReLU()(seed_fc)
    seed_fc = tf.keras.layers.Reshape((7, 7, 256))(seed_fc)

    # merge embedding with seed encoder
    merge = tf.keras.layers.Concatenate()([seed_fc, upscaling])
    assert tuple(merge.shape) == (None, 7, 7, 256+1)

    x = layers.Conv2DTranspose(128, (5, 5), strides=(1, 1), padding='same', use_bias=False)(merge)
    x = layers.BatchNormalization()(x)
    x = layers.LeakyReLU()(x)
    assert tuple(x.shape) == (None, 7, 7, 128)

    x = layers.Conv2DTranspose(64, (5, 5), strides=(2, 2), padding='same', use_bias=False)(x)
    x = layers.BatchNormalization()(x)
    x = layers.LeakyReLU()(x)
    assert tuple(x.shape) == (None, 14, 14, 64)

    # TODO tanh function with output between [-1, 1]
    output = layers.Conv2DTranspose(3, (5, 5), strides=(2, 2), padding='same', use_bias=False, activation='tanh')(x)
    model = tf.keras.Model([seed_input, input_label], output)
    assert model.output_shape == (None, 28, 28, 3)

    return model

While trying to convert the above code to PyTorch, I faced dimensionality issue. Not able to understand how to add that assert tuple code. I tried with view() function but it does not accept "None" as dimension. Here is what I tried later. Replaced 1 with none in view() function but then it gave an error in discriminator.
class Generator(nn.Module):

    def __init__(self):
        super(Generator,self).__init__()
        #claas = label
        #noise == seed == data
        self.linear_data = nn.Linear(100, 256, bias = False)
        self.BN_data  = nn.BatchNorm1d(256)

        self.label_embedding = nn.Embedding(62, 100)
        self.linear_label = nn.Linear(100, 1, bias = False)

        self.DCV1 = nn.ConvTranspose2d(257, 128, kernel_size =(4,4), stride = (2,2), padding = (1,1))
        self.BN1  = nn.BatchNorm2d(128)

        self.DCV2 = nn.ConvTranspose2d(128, 64, 4, 2, 1)
        self.BN2  = nn.BatchNorm2d(64)

        self.DCV3 = nn.ConvTranspose2d(64, 3, 4, 2, 1)

    def forward(self, inputs, labels):
        '''
        input size: (batch_size, 100, 1, 1)
        outputs size: (batch_size, 1, 32, 32)
        '''
        # linear 1 for data
        x = self.linear_data(inputs)
        x = F.leaky_relu(x, negative_slope=0.2)
        x = self.BN_data(x)
        x = x.view(1,256,4,4)
        #x = x.view(None, 4,4,256)
        # linear 1 for label
        y = self.label_embedding(labels)
        #print(y.shape)

        y = self.linear_label(y)
        #print(y.shape)
        y = y.view(1,1,4,4)
        #x = x.view(None, 4,4,1)
        #print(y.shape)
        #concat data and label
        x = torch.cat((x, y), dim=1)
        x = x.view(1,257,4,4)
        #print(x.shape)
        #assert tuple(x.shape) == (1, 4, 4, 256+1)
        # Deconv 1
        x = self.DCV1(x)
        x = self.BN1(x)
        x = F.leaky_relu(x, 0.2)

        # Deconv 2
        x = self.DCV2(x)
        x = self.BN2(x)
        x = F.leaky_relu(x, 0.2)

        # Deconv 3 + output
        x = self.DCV3(x)
        outputs = torch.tanh(x)
        return outputs

The batch size is 16 and dimension of input noise: 16,100
dimension of input label: 16,1


title: Pytorch, sample given batch logits

body:
Given logits like
# each row is a record of data
logits = np.array([ [0.1, 0.3, 0.5], [0.3, 0.1, 0.5], [0.1, 0.3, 0.0] ])

How can I use Pytorch to sample the index for the logits of each row? Current distribution APIs does not support such functions.
What I want is, for example
distribution = Categorical(logits=logits)
labels = distribution.sample(dim=1)


title: Conflict between shape of (-1, 784) with input size of 200

body:
There is an error with the shape of a tensor and it's input size, as it is conflicting. I have absolutely no idea what to do at all. Because I am still new to this topic, which is the reason why you may or may not spot lines of code that are not needed at all. Just a heads up. If needed, personally contact me and I will send you the .ipynb file
If it matters I am doing computer vision, with the MNIST handrawn numbers dataset.
import torch
import torchvision
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
from torchvision.datasets import MNIST
import torchvision.transforms as transforms
from torch.utils.data import SubsetRandomSampler, DataLoader

dataset = MNIST(root='data/', download=True)

def split_indices(dataset, rate):
    eval = int(dataset*rate/100)
    index = np.random.permutation(dataset)
    return index[eval:], index[:eval]

train_index, eval_index = split_indices(len(dataset), rate=20)

dataset = MNIST(root='data/', train=True, transform=transforms.ToTensor())

train_sampler = SubsetRandomSampler(train_index)
train_dl = DataLoader(dataset, batch_size=200, sampler=train_sampler)

val_sampler = SubsetRandomSampler(eval_index)
val_dl = DataLoader(dataset, batch_size=200, sampler=eval_sampler)

inputs = 28*28
nums = 10
model = nn.Linear(inputs, nums)

class MnistModel(nn.Module):
        def __init__(self):
            super().__init__()
            self.linear = nn.Linear(inputs, nums)

        def forward(self, xb):
            xb = xb.reshape(-1, 784)
            outputs = self.linear(xb)
            return outputs

        def accuracy(x, y):
            print(torch.sum(x == y).item()/len(x))

model = MnistModel()

for images, labels in train_dl:
        outputs = model(images)
        break

loss_fn = F.cross_entropy
loss = loss_fn(outputs, labels)
opt = torch.optim.Adam(model.parameters(), lr=7)

def loss_batch(model, loss_fn, xb, yb, opt=None, metric=None):
    preds = model(xb)
    loss = loss_fn(preds, yb)

    if opt is not None:
        loss.backward()
        opt.step()
        opt.zero_grad()

    metric_result = None
    if metric is not None:
        metric_result = metric(preds, yb)

    return loss.item(), len(xb), metric_result

def evaluate(model, loss_fn, valid_dl, metric=None):
    with torch.no_grad():
        results = [loss_batch(model, loss_fn, xb, yb, metric=metric) for xb, yb in valid_dl]

        losses, nums, metrics = zip(*results)
        total = np.sum(nums)
        avg_loss = np.sum(np.multiply(losses, nums)) / total

        avg_metric = None
        if metric is not None:
            avg_metric = np.sum(np.multiply(metrics, nums)) / total

        return avg_loss, total, avg_metric

def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.sum(preds == labels).item() / len(preds)

eval_loss, total, eval_acc = evaluate(model, loss_fn, val_dl, metric=accuracy)
print(f"loss: {eval_loss}, accuracy: {eval_acc*100}")

def fit(epochs, model, loss_fn, opt, train_dl, valid_dl, metric=None):
    for epoch in range(epochs):
        for xb, yb in train_dl:
            loss,_,_ = loss_batch(model, xb, yb, opt)

        result = evaluate(model, loss_fn, valid_dl, metric)
        eval_loss, total, eval_metric = result

        if metric is None:
            print(f"Epoch: {epoch+1}, loss: {loss.item()}")

        else:
            print(f"Epoch: {epoch+1}, loss: {loss.item()}, metric: {metric.__name__} {eval_metric}")

model = MnistModel()

opt = torch.optim.Adam(model.parameters(), lr=7)
fit(5, model, loss_fn, opt, train_dl, eval_dl, accuracy) #Error line

Error output:
RuntimeError                              Traceback (most recent call last)

<ipython-input-55-90c5585d3b40> in <module>()
      1 opt = torch.optim.Adam(model.parameters(), lr=7)
----> 2 fit(5, model, loss_fn, opt, train_dl, eval_dl, accuracy)

3 frames

<ipython-input-49-afd130f584e4> in forward(self, xb)
     18
     19     def forward(self, xb):
---> 20         xb = xb.reshape(-1, 784)
     21         outputs = self.linear(xb)
     22         return outputs

RuntimeError: shape '[-1, 784]' is invalid for input of size 200


title: CNN Gives Same Loss And Accuracy

body:
I am trying to create a CNN using pytorch. When I use 3 or fewer consecutive layers, it appears to work correctly, with the loss decreasing and the accuracy increasing. However, when I have 4 or more conv2d layers in a row, the loss and accuracy remain constant (accuracy=0.0961). Occasionally, the 4+ layer models work, but if I try training them again, they have the aforementioned problem. I am testing using the MNIST dataset.
My code is below:
from pathlib import Path
import requests
import pickle
import gzip
from matplotlib import pyplot
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import DataLoader, TensorDataset

DATA_PATH = Path("data")
PATH = DATA_PATH / "mnist"

PATH.mkdir(parents=True, exist_ok=True)

URL = "http://deeplearning.net/data/mnist/"
FILENAME = "mnist.pkl.gz"

if not (PATH / FILENAME).exists():
    content = requests.get(URL + FILENAME).content
    (PATH / FILENAME).open("wb").write(content)

with gzip.open((PATH / FILENAME).as_posix(), "rb") as f:
        ((x_train, y_train), (x_valid, y_valid), _) = pickle.load(f, encoding="latin-1")

x_train = torch.tensor(x_train)
y_train = torch.tensor(y_train).unsqueeze(1)

x_valid = torch.tensor(x_valid)
y_valid = torch.tensor(y_valid).unsqueeze(1)

train_ds = TensorDataset(x_train, y_train)
train_dl = DataLoader(train_ds, batch_size=64, shuffle=True)

valid_ds = TensorDataset(x_valid, y_valid)
valid_dl = DataLoader(valid_ds, batch_size=64)

class Lambda(nn.Module):
    def __init__(self, func):
        super().__init__()
        self.func = func

    def forward(self, x):
        return self.func(x)

def preprocess(x):
  return x.view(-1, 1, 28, 28)

model = nn.Sequential(
    Lambda(preprocess),
    nn.Conv2d(1, 1, kernel_size=(3,3), padding=2, padding_mode='zeros'),
    nn.ReLU(),
    nn.Conv2d(1, 1, kernel_size=(3,3), padding=2, padding_mode='zeros'),
    nn.ReLU(),
    nn.Conv2d(1, 1, kernel_size=(3,3), padding=2, padding_mode='zeros'),
    nn.ReLU(),
    nn.Conv2d(1, 1, kernel_size=(3,3), padding=2, padding_mode='zeros'),
    nn.ReLU(),
    nn.AdaptiveAvgPool2d((10, 1)),
    Lambda(lambda x: x.squeeze(1)),
    nn.Softmax(dim=1)
)

def loss_batch(model, loss_func, xb, yb, opt=None):
  pred = model(xb)
  loss = loss_func(pred, yb)

  if opt is not None:
    loss.backward()
    opt.step()
    opt.zero_grad()
    return loss.item(), len(xb)
  else:
    label = torch.argmax(pred, dim=1)
    correct = (label == yb).float().sum()
    accuracy = correct/len(xb)
    return loss.item(), len(xb), accuracy

def fit(epochs, model, loss_func, opt, train_dl, valid_dl):
  for epoch in range(epochs):
    model.train()
    losses = []
    nums = []
    for xb, yb in train_dl:
      loss, num = loss_batch(model, loss_func, xb, yb, opt)
      losses.append(loss)
      nums.append(num)
    train_loss = np.sum(np.multiply(losses, nums)) / np.sum(nums)

    model.eval()
    with torch.no_grad():
      losses = []
      nums = []
      accuracies = []
      for xb, yb in valid_dl:
        loss, num, accuracy = loss_batch(model, loss_func, xb, yb)
        losses.append(loss)
        nums.append(num)
        accuracies.append(accuracy)
      val_loss = np.sum(np.multiply(losses, nums)) / np.sum(nums)
      val_acc = np.sum(np.multiply(accuracies, nums)) / np.sum(nums)

      print("Epoch ", epoch+1, ": ", train_loss, val_loss, val_acc)

opt = torch.optim.Adam(model.parameters(), lr=0.01)
fit(20, model, F.cross_entropy, opt, train_dl, valid_dl)


title: forward() function in EncoderRNN DataParellel with batch_first = False

body:
I have been trying very hard to get this right but there seems to be no documentation from Pytorch to help me. I am trying for use an encoder RNN to perform DataParallel processing on my system that has 2 GPUs. So far I donot want to change the code so that it works with batch_first = True
The following is the code snippet from my Encoder RNN module.
def forward(self, input_seq, input_lengths, hidden=None):
        # Convert word indexes to embeddings
        embedded = self.embedding(input_seq)
        # Get the longest sequence size - We need this for multi GPU training.
        total_length = input_lengths.cpu().numpy()[0]
        # Pack padded batch of sequences for RNN module
        packed = torch.nn.utils.rnn.pack_padded_sequence(embedded, input_lengths,enforce_sorted=False)

        # Forward pass through GRU
        outputs, hidden = self.gru(packed, hidden)
        # Unpack padding
        outputs, _ = torch.nn.utils.rnn.pad_packed_sequence(outputs,total_length=total_length)
        # Sum bidirectional GRU outputs
        outputs = outputs[:, :, :self.hidden_size] + outputs[:, : ,self.hidden_size:]

        # Return output and final hidden state
        return outputs, hidden

Right now I have the largest sequence length from the line  total_length = input_lengths.cpu().numpy()[0] which happens to be equal to 5.
I just cant seem to get this work. The documentation from PyTorch says

Additionally, extra care needs to be taken when batch dimension is dim
1 (i.e., batch_first=False) with data parallelism. In this case, the
first argument of pack_padded_sequence padding_input will be of shape
[T x B x *] and should be scattered along dim 1, but the second
argument input_lengths will be of shape [B] and should be scattered
along dim 0. Extra code to manipulate the tensor shapes will be
needed.

So at the moment I am not sure what will be the correct Tensor operations/permutations to get this right
embedded has a shape of [5 32 36] with 5 being the max sequence length
Thanks


title: The GPU memory is not enough when computing feature centers using pytorch

body:
I want use the class feature centers in the loss function when using pytorch. But when I compute the centers, the GPU memory is not enough. How can I solve it? The code is as follows:
for epoch in range((args.start_epoch+1), args.epochs):
  Center= computer_Center(model,dataloader, classnum)
    for input, target in train_loader:
        target = target.cuda()
        input = input.cuda()
        input_var = torch.autograd.Variable(input)
        target_var = torch.autograd.Variable(target)
        outputs, feature = model(input_var)
        l = criterion(feature,target_var, Center) .forward()
        l.backward(retain_graph=True)

def computer_Center(model,dataloader, classnum):
    model.train()
    for i in range(classnum):
            j=0
            for input,target in dataloader:
                target=target.cuda()
                input = input.cuda()
                input_var = torch.autograd.Variable(input)
                target=torch.autograd.Variable(target)
                _, feature_ext = model(input_var)
                ind=torch.where(target==i)[0]
                if ind.shape[0]>0:
                    if j==0:
                        feature_mid = feature_ext[ind, :]
                        feature_sum_mid=feature_mid.sum(0)
                    else:
                        feature_mid = feature_ext[ind, :]
                        feature_sum_mid = feature_sum_mid+feature_mid.sum(0)
                    j=j+1

            feature_sum_mid=feature_sum_mid.unsqueeze(0)
            if i==0:
                feature_sum=feature_sum_mid
            else:
                feature_sum=torch.cat([feature_sum,feature_sum_mid],dim=0)

    Center=feature_sum
    for i in range(7):
            Center[i,:]=feature_sum[i,:]/ClaSamNum[i]

return Center


title: Some strange ImportError/PermissionError occur when two same script run simultaneously on GPU (pytorch DDP)

body:
I have a program that using distributeddataparallel with pytorch to perform muti-gpu parallelism.
The script can run correctly by CUDA_VISIBLE_DEVICES=0,1 python ddp_test.py , but the problem is, when I try to run the same script on the same node simultaneously by CUDA_VISIBLE_DEVICES=2,3 python ddp_test.py , the second script I run will occur ImportError/PermissionError.
Here is a minium example, ddp_test.py is a small training program, ddp_import.py and ddp_import_import.py is to simulate my real training program.
# ddp_test.py
import os
from datetime import datetime
import argparse
import torch.multiprocessing as mp
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torch.distributed as dist

from ddp_import import *

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-n', '--nodes', default=1, type=int, metavar='N',
                        help='number of data loading workers (default: 4)')
    parser.add_argument('-g', '--gpus', default=2, type=int,
                        help='number of gpus per node')
    parser.add_argument('-nr', '--nr', default=0, type=int,
                        help='ranking within the nodes')
    parser.add_argument('--epochs', default=50, type=int, metavar='N',
                        help='number of total epochs to run')
    args = parser.parse_args()
    args.world_size = args.gpus * args.nodes
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '1234'
    mp.spawn(train, nprocs=args.gpus, args=(args,))

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc = nn.Linear(7*7*32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

def train(gpu, args):
    rank = args.nr * args.gpus + gpu
    dist.init_process_group(backend='nccl', init_method='env://', world_size=args.world_size, rank=rank)
    torch.manual_seed(0)
    model = ConvNet()
    print("rank:{} gpu:{}".format(rank, gpu))
    torch.cuda.set_device(gpu)
    model.cuda(gpu)
    batch_size = 20
    # define loss function (criterion) and optimizer
    criterion = nn.CrossEntropyLoss().cuda(gpu)
    optimizer = torch.optim.SGD(model.parameters(), 1e-4)
    # Wrap the model
    model = nn.parallel.DistributedDataParallel(model, device_ids=[gpu])
    # Data loading code
    train_dataset = torchvision.datasets.MNIST(root='./data',
                                               train=True,
                                               transform=transforms.ToTensor(),
                                               download=True)
    train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset,
                                                                    num_replicas=args.world_size,
                                                                    rank=rank)
    train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                               batch_size=batch_size,
                                               shuffle=False,
                                               num_workers=0,
                                               pin_memory=True,
                                               sampler=train_sampler)

    start = datetime.now()
    total_step = len(train_loader)
    for epoch in range(args.epochs):
        for i, (images, labels) in enumerate(train_loader):
            images = images.cuda(non_blocking=True)
            labels = labels.cuda(non_blocking=True)
            # Forward pass
            outputs = model(images)
            loss = criterion(outputs, labels)

            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            if (i + 1) % 100 == 0 and gpu == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch + 1, args.epochs, i + 1, total_step,
                                                                         loss.item()))
    if gpu == 0:
        print("Training complete in: " + str(datetime.now() - start))

if __name__ == '__main__':
    main()

# ddp_import.py
from ddp_import_import import *

# ddp_import_import.py
import nltk
from pycocotools.coco import COCO

When I run the script by CUDA_VISIBLE_DEVICES=0,1 python ddp_test.py, the script work correctly. But if I run two this script simultaneously, one by CUDA_VISIBLE_DEVICES=0,1 python ddp_test.py, and another by CUDA_VISIBLE_DEVICES=2,3 python ddp_test.py(I have 4 gpus on the node), the error occurs.
PermissionError and sometimes ImportError.
I'm new to ddp, and this problem make me confused for several days. Can anyone tell me what the problem is and how to solve it. Thanks a lot!


title: How can I import my CNN-model in another code?

body:
This is my "model", I just added a piece of it here for understanding purpose. I want this model to be imported in another piece of code named as "trainer".
import torch.nn as nn
import torch

class Signet(nn.Module):
    def __init__(self, num_users=10):
        super(Signet, self).__init__()
        self.num_classes = num_users
        self.forward_util_layer = {}

This is another piece of code. Here "Model" is what I want to import like I mentioned above.
import model

class Trainer:
    def __init__(self, trainData, testData, num_users=10):
        self.trainData = trainData
        self.testData = testData

What are the ways to import it. Where to save my model.ipynb, so that my trainer.ipynb can import it. Or if there is any way to call it directly in my trainer.ipynb code?


title: pytorch slice 2nd dimension of 3-d tensor based on a list/array

body:
I'm working on a many-to-one RNN with variable length, and trying to extract the relevant output from the last non-padded unit. In particular, let's say I'm using a batch size of 3, with 16 hidden units, and the max-length in this batch is 6, so the "output" is of size 3-6-16, for example:
tensor([[[-0.0650, -0.0712, -0.1024, -0.0544,  0.1491, -0.0390, -0.0559,
           0.0675,  0.0298, -0.1015,  0.1557, -0.0176, -0.0348,  0.0585,
          -0.0673, -0.1097],
         [-0.0515, -0.1113, -0.1194, -0.0942,  0.2349, -0.0736, -0.0474,
           0.1339,  0.0362, -0.1515,  0.2344,  0.0125, -0.0403,  0.0810,
          -0.0817, -0.1919],
         [-0.0588, -0.1078, -0.1033, -0.0454,  0.2273, -0.0773, -0.0720,
           0.1857,  0.0817, -0.1805,  0.2329,  0.0146, -0.0437,  0.0905,
          -0.1100, -0.2268],
         [-0.0926, -0.1068, -0.1196, -0.0647,  0.2364, -0.1125, -0.0422,
           0.1958,  0.0841, -0.2176,  0.2558,  0.0397, -0.0856,  0.0867,
          -0.0862, -0.2232],
         [-0.0692, -0.1501, -0.1342, -0.0725,  0.2564, -0.1084, -0.0767,
           0.2042,  0.1136, -0.2037,  0.2773, -0.0236, -0.0786,  0.0889,
          -0.1053, -0.2285],
         [-0.0783, -0.1734, -0.1472, -0.1053,  0.2649, -0.0928, -0.0306,
           0.1727,  0.0962, -0.2102,  0.3104, -0.0102, -0.0566,  0.0878,
          -0.1159, -0.2546]],

        [[-0.0446, -0.0554, -0.1074, -0.0609,  0.1355, -0.0883, -0.0113,
           0.0571, -0.0149, -0.0931,  0.1628,  0.0269, -0.0660,  0.0590,
          -0.0089, -0.1344],
         [-0.0122, -0.1504, -0.1654, -0.0200,  0.2308, -0.0633,  0.0030,
           0.1265,  0.0164, -0.1145,  0.1606,  0.0188, -0.0546,  0.0469,
          -0.0717, -0.2503],
         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000]],

        [[-0.0213, -0.0946, -0.0691, -0.0673,  0.1766, -0.1065, -0.0824,
           0.0633, -0.0019, -0.1095,  0.2158, -0.0181, -0.0451,  0.0849,
          -0.0387, -0.1313],
         [ 0.0065, -0.1240, -0.1519, -0.0734,  0.2415, -0.1553, -0.0661,
           0.1188,  0.0530, -0.1496,  0.2765, -0.0181, -0.0736,  0.1144,
          -0.0671, -0.2182],
         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000],
         [ 0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,
           0.0000,  0.0000]]], device='cuda:0', grad_fn=<TransposeBackward0>)

I want to extract this (the last non-padded row in each item):
tensor([[-0.0783, -0.1734, -0.1472, -0.1053,  0.2649, -0.0928, -0.0306,  0.1727,
          0.0962, -0.2102,  0.3104, -0.0102, -0.0566,  0.0878, -0.1159, -0.2546],
        [-0.0122, -0.1504, -0.1654, -0.0200,  0.2308, -0.0633,  0.0030,  0.1265,
          0.0164, -0.1145,  0.1606,  0.0188, -0.0546,  0.0469, -0.0717, -0.2503],
        [ 0.0065, -0.1240, -0.1519, -0.0734,  0.2415, -0.1553, -0.0661,  0.1188,
          0.0530, -0.1496,  0.2765, -0.0181, -0.0736,  0.1144, -0.0671, -0.2182]])

with proper gradient info so the RNN can continue training.
What I've tried:
selectors = torch.tensor([5, 1, 1])
torch.index_select(sequence, 1, selectors)

but this returns 3 rows (the 5th, 1st, 1st (0-indexed)) for each item, while I wanted the 5th for the first item, 1st for the second item, and so forth.
I've also tried
arr = torch.zeros(len(sequence), 16, dtype=torch.float32)
    for i in range(len(sequence)):
        arr[i, :] = sequence[i, selectors[i], :]

This gives me something very close to what I want, but I think there are some issues with the gradient info..


title: Increase transfer speed from CPU to GPU for simple computations

body:
I have parameterized a filtering problem, so that I essentially only have to make a product and sum between two arrays. I want to do this on the GPU, using pytorch to accelerate it.
Right now my code is:
kernels_cuda = torch.cuda.FloatTensor(kernels) #(32,32,159,21,21,21) float32 array
sub_matrices_cuda = torch.cuda.FloatTensor(sub_matrices) #(32,32,159,21,21,21) float32 array
out = (sub_matrices_cuda*kernels_cuda).sum(dim=[3,4,5]).cpu().numpy()
torch.cuda.synchronize() #the synchronize might not be needed

This takes about 3.0s right now, and almost all the time is spent on sending the data to the GPU. This operation is supposed to be run 16 times, but with new slices of the data. Is there anyway I can speed up my code (mainly the CPU->GPU transfer)? Preferably with pytorch, but I'm open to alternatives.
PS My hardware is: Titan RTX, Intel Xeon W2133 and 2666 MHz DDR4 RAM. I have a feeling, that I should be able to exceed the observed CPU->GPU transfer speeds with this hardware?


title: How to optimise memory usage of looped loss function in PyTorch?

body:
I have a loss function that requires multiple internal passes:
def my_loss_func(logits, sigma, labels, num_passes):
    total_loss = 0
    img_batch_size = logits.shape[0]
    logits_shape = list(logits.shape)
    for fpass in range(num_passes):
        noise_array = torch.normal(mean=0.0, std=1.0, size=logits_shape, device=torch.device('cuda:0'))
        stochastic_output = logits + sigma * noise_array
        del noise_array
        exponent_B = torch.log(torch.sum(torch.exp(stochastic_output), dim=-1, keepdim=True))
        inner_logits = exponent_B - stochastic_output
        soft_inner_logits = labels * inner_logits
        total_loss += torch.exp(soft_inner_logits)
        del exponent_B, inner_logits, soft_inner_logits
    mean_loss = total_loss / num_passes
    actual_loss = torch.mean(torch.log(mean_loss))
    return actual_loss

Both logits and sigma are networks outputs and therefore have associated gradients. The bottleneck (expectedly) relates to the line total_loss += torch.exp(soft_inner_logits), since afaik a new computation graph is appended for subsequent passes. I've read that calling loss.backward() within the loop could help in similar circumstances, but unfortunately, I need to log the output after the loop and backdrop based on that, so this solution doesn't seem viable here.
To be more specific, I run into memory issues when num_passes exceeds 20, are there any other ways in which I could fully optimise memory allocation to allow for a greater number of passes? I’m not at all concerned with readability/ ugly solutions, any advice will be of great help.


title: PyTorch unfold vs as_stride

body:
It seems PyTorch unfold and as_stride are doing the same thing but for the former, you cannot control the tensor output size.
import torch
import torch.nn as nn

x = torch.arange(0, 10)

x1 = x.unfold(0, 3, 1)

x2 = x.as_strided((8,3), (1,1))

print(f'x1 = {x1}')
print(f'x2 = {x2}')

output:
x1 = tensor([[0, 1, 2],
        [1, 2, 3],
        [2, 3, 4],
        [3, 4, 5],
        [4, 5, 6],
        [5, 6, 7],
        [6, 7, 8],
        [7, 8, 9]])
x2 = tensor([[0, 1, 2],
        [1, 2, 3],
        [2, 3, 4],
        [3, 4, 5],
        [4, 5, 6],
        [5, 6, 7],
        [6, 7, 8],
        [7, 8, 9]])

Then is there any situation that you should use unfold instead of as_stride and vice versa?


title: graph classification using pytorch

body:
I am trying to perform graph classification. I have a list of DGL graphs which look like this
DGLGraph(num_nodes=64267, num_edges=155523,
         ndata_schemes={}
         edata_schemes={'norm': Scheme(shape=(), dtype=torch.float32), 'rel_type': Scheme(shape=(17,), dtype=torch.float64)

and its corresponding label is tensor([0, 1]). I was following an example as I am not familiar with PyTorch.
The model definition is as below
def gcn_message(edges):
    # The argument is a batch of edges.
    # This computes a (batch of) message called 'msg' using the source node's feature 'h'.
    return {'msg' : edges.src['h']}

def gcn_reduce(nodes):
    # The argument is a batch of nodes.
    # This computes the new 'h' features by summing received 'msg' in each node's mailbox.
    return {'h' : torch.sum(nodes.mailbox['msg'], dim=1)}

# Define the GCNLayer module
class GCNLayer(nn.Module):
    def __init__(self, in_feats, out_feats):
        super(GCNLayer, self).__init__()
        self.linear = nn.Linear(in_feats, out_feats)

    def forward(self, g, inputs):
        # g is the graph and the inputs is the input node features
        # first set the node features
        g.ndata['h'] = inputs
        # trigger message passing on all edges
        g.send(g.edges(), gcn_message)
        # trigger aggregation at all nodes
        g.recv(g.nodes(), gcn_reduce)
        # get the result node features
        h = g.ndata.pop('h')
        # perform linear transformation
        return self.linear(h)

class GCN(nn.Module):
    def __init__(self, in_feats, hidden_size, num_classes):
        super(GCN, self).__init__()
        self.gcn1 = GCNLayer(in_feats, hidden_size)
        self.gcn2 = GCNLayer(hidden_size, num_classes)

    def forward(self, g, inputs):
        h = self.gcn1(g, inputs)
        h = torch.relu(h)
        h = self.gcn2(g, h)
        return h
# The first layer transforms input features of size of 41 to a hidden size of 5.
# The second layer transforms the hidden layer and produces output features of
# size 2, corresponding to the two classification groups
net = GCN(7, 16, 2)

for epoch in range(epochs):
    epoch_loss = 0
    epoch_logits = []
    labs = []
    # Iterate over batches
    for i, (bg, labels) in enumerate(train_loader):
        logits = net(bg, bg.ndata['h'])
        # we save the logits for visualization later
        train_logits.append(logits.detach().numpy())
        epoch_logits.append(logits.detach().numpy())
        labs.append(labels.unsqueeze(1).detach().numpy())
        logp = F.softmax(logits, 1)
        loss = loss_fn(logp,labels)

Upon calling the loss_fn, i get an error ValueError: Expected input batch_size (64267) to match target batch_size (64).. For some reason, each node in my graph is treated as a different input value and the model returns a prediciton for it. The collate is defined as
def collate(samples):
    # The input `samples` is a list of pairs
    #  (graph, label).
    graphs, labels = map(list, zip(*samples))
    batched_graph = dgl.batch(graphs, node_attrs='h')
    batched_graph.set_n_initializer(dgl.init.zero_initializer)
    batched_graph.set_e_initializer(dgl.init.zero_initializer)
    return batched_graph, torch.stack(labels)

and i use this collate in another model in which I am using dgllife.model.model_zoo.GCNPredictor and it does not create any issue.


title: Pytorch build seq2seq MT model ,But how to get the translation results from the output tensor?

body:
I am try to implement my own MT engine, i am following the steps in https://github.com/bentrevett/pytorch-seq2seq/blob/master/1%20-%20Sequence%20to%20Sequence%20Learning%20with%20Neural%20Networks.ipynb
SRC = Field(tokenize=tokenize_en,
            init_token='<sos>',
            eos_token='<eos>',
            lower=True)

TRG = Field(tokenize=tokenize_de,
            init_token='<sos>',
            eos_token='<eos>',
            lower=True)

After training the model,the link only share a way to batch evaluate but i want to try single string and get the translation results. for example i want my model to translate the input "Boys" and get the German translations.
savedfilemodelpath='./pretrained_model/2020-09-27en-de.pth'
model.load_state_dict(torch.load(savedfilemodelpath))
model.eval()
inputstring = 'Boys'
processed=SRC.process([SRC.preprocess(inputstring)]).to(device)
output=model(processed,processed)
output_dim = output.shape[-1]
outputs = output[1:].view(-1, output_dim)
for item in outputs:
    print('item shape is {} and item.argmax is {}, and words is {}'.format(item.shape,item.argmax(),TRG.vocab.itos[item.argmax()]))

So my question is that it it right to get the translation results by:
First: convert the string to tensor
inputstring = 'Boys'
processed=SRC.process([SRC.preprocess(inputstring)]).to(device)

Second: send the tensor to the model. As the model have a TRG param.I have to give the tensor,am i able not given the TRG tensor?
output=model(processed,processed)
output_dim = output.shape[-1]
outputs = output[1:].view(-1, output_dim)

Third：through the  return tensor, i use the argmax to get the translation results? is it right?
Or how can i get the right translation results?
for item in outputs:
        print('item shape is {} and item.argmax is {}, and words is {}'.format(item.shape,item.argmax(),TRG.vocab.itos[item.argmax()+1]))


title: PyTorch model not converging

body:
I'm training a binary classification model on a series of images.
The model was derived from resnet18 in torchvision and I made the last FC as nn.Linear(512, 1)
The loss function is BCELoss
However, the model doesn't show any sign of converging even after 5000 iterations.
I'm suspecting I might do something wrong in the training stage? But I can't find where's the bug.
Here's my code:
Model:
## Model
import torch.nn as nn
import torch.nn.functional as F
import torchvision.models as models

resnet18 = models.resnet18(pretrained= True)
resnet18.fc = nn.Linear(512, 1)

Parameters, loss, optimizers:
## parameter
epochs = 200
learning_rate = 0.1
momen = 0.9
batch = 8
criterion = nn.BCELoss()
resnet18.to(device)
opt = optim.SGD(resnet18.parameters(), lr = learning_rate, momentum = momen)

Dataloaders:
# Generators
training_set = Dataset(X_train)
training_generator = torch.utils.data.DataLoader(training_set, batch_size= batch, shuffle=True)

validation_set = Dataset(X_test)
validation_generator = torch.utils.data.DataLoader(validation_set, batch_size=1, shuffle=False)

Training:
# training
history = []
for t in range(epochs):
  for i, data in enumerate(training_generator, 0):

    inputs, labels = data
    # check if input size == batch size #
    if inputs.shape[0] < batch:
      break

#    print("labels", labels, labels.dtype)

    # move data to GPU #
    inputs, labels = inputs.to(device), labels.to(device)

    opt.zero_grad()
    # Prediction #
    y_pred = resnet18(inputs).view(batch,)
    y_pred = (y_pred > 0).float().requires_grad_()
#    print("y_pred", y_pred, y_pred.dtype)

    # Calculating loss #
    loss = criterion(y_pred, labels.view(batch,))
    loss.backward()
    opt.step()

    if i % 10 == 0:
      history.append(loss.item())
      print("Epoch: {}, iter: {}, loss: {}".format(t, i, loss.item())

torch.save(resnet18, 'trained_resnet18.pt')

Edit:
The loss values are like this:
Epoch: 3, iter: 310, loss: 0.0
Epoch: 3, iter: 320, loss: 37.5
Epoch: 3, iter: 330, loss: 37.5
Epoch: 3, iter: 340, loss: 0.0
Epoch: 3, iter: 350, loss: 37.5
Epoch: 3, iter: 360, loss: 50.0
Epoch: 3, iter: 370, loss: 37.5
Epoch: 3, iter: 380, loss: 25.0
Epoch: 3, iter: 390, loss: 12.5


title: How to generate images from specific class using Conditional ProGAN?

body:
This is my first time using a GAN. I trained a ConditionalProGAN (implementation in this link : https://github.com/minxdragon/pro_gan_pytorch) using a dataset that contains 3 classes. I saved the model. Now, I would like to use it to generate images of a specific class. Please help.
Example Code for CIFAR-10 dataset to undrestand my question:
import torch as th
import torchvision as tv
import pro_gan_pytorch.PRO_GAN as pg

# select the device to be used for training
device = th.device("cuda" if th.cuda.is_available() else "cpu")
data_path = "cifar-10/"

def setup_data(download=False):
    """
    setup the CIFAR-10 dataset for training the CNN
    :param batch_size: batch_size for sgd
    :param num_workers: num_readers for data reading
    :param download: Boolean for whether to download the data
    :return: classes, trainloader, testloader => training and testing data loaders
    """
    # data setup:
    classes = ('plane', 'car', 'bird', 'cat', 'deer',
               'dog', 'frog', 'horse', 'ship', 'truck')

    transforms = tv.transforms.ToTensor()

    trainset = tv.datasets.CIFAR10(root=data_path,
                                   transform=transforms,
                                   download=download)

    testset = tv.datasets.CIFAR10(root=data_path,
                                  transform=transforms, train=False,
                                  download=False)

    return classes, trainset, testset

if __name__ == '__main__':

    # some parameters:
    depth = 4
    # hyper-parameters per depth (resolution)
    num_epochs = [10, 20, 20, 20]
    fade_ins = [50, 50, 50, 50]
    batch_sizes = [128, 128, 128, 128]
    latent_size = 128

    # get the data. Ignore the test data and their classes
    _, dataset, _ = setup_data(download=True)

    # ======================================================================
    # This line creates the PRO-GAN
    # ======================================================================
    pro_gan = pg.ConditionalProGAN(num_classes=10, depth=depth,
                                   latent_size=latent_size, device=device)
    # ======================================================================

    # ======================================================================
    # This line trains the PRO-GAN
    # ======================================================================
    pro_gan.train(
        dataset=dataset,
        epochs=num_epochs,
        fade_in_percentage=fade_ins,
        batch_sizes=batch_sizes
    )
    # ======================================================================

After Training, I loaded the model :
import torch as th
from pro_gan_pytorch import PRO_GAN as pg

device = th.device("cuda" if th.cuda.is_available() else "cpu")

gen = th.nn.DataParallel(pg.Generator(depth=4))
gen.load_state_dict(th.load("GAN_GEN_SHADOW_4.pth", map_location=str(device)))

Now how can I ask to generate images only for the class "plane" for exemple ?? I am sure I missed something in the way how conditional GAN work.


title: pythorch: take the derivative of network model output to input, the values are incorrect

body:
I build a simple neural network model to test how to take the higher-order derivative of output to input in pytorch. The data are quite simple, however, the results are quite different from expected. The model fit y (output) very well but not its derivatives (y_x,y_xx,etc.). Can anyone tell me what is the problem?
import scipy.io as sio
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as Fun
from torch.autograd import Variable
import matplotlib.pyplot as plt
import scipy.sparse as sparse
from torchvision import datasets
from pytorchtools import EarlyStopping
import time
start_time = time.time()
np.random.seed(seed=1)
torch.manual_seed(0)

class MyNet1(nn.Module):
    def __init__(self, D_in, H, D_out):
        super(MyNet1, self).__init__()
        self.fc1 = nn.Linear(D_in, H[0])
        self.fc2 = nn.Linear(H[0], H[1])
        self.fc3 = nn.Linear(H[1], H[2])
        self.fc4 = nn.Linear(H[2], D_out)
    def forward(self, x):
        f = nn.Tanh()(self.fc1(x))
        f = nn.Tanh()(self.fc2(f))
        f = nn.Tanh()(self.fc3(f))
        f = self.fc4(f)
        return f

Data = sio.loadmat('dataODE.mat')
x = np.linspace(0,1,1000)
y = x**3+0.5*x**2
x = np.reshape(x, (-1, 1))
y = np.reshape(y, (-1, 1))
y_x = 3*x**2+x
y_xx = 6*x+1
# Convert numpy arrays to torch Variables
X = Variable(torch.from_numpy(x).float(),requires_grad=True)
Y = Variable(torch.from_numpy(y).float(),requires_grad=True)

# D_in is input dimension;
# H is the dimension of the hidden layers
# D_out is output dimension.
D_in, H, D_out =  1, [50, 100, 500, 100, 50], 1

# Construct our model by instantiating the class defined above
model = MyNet1(D_in, H, D_out)
# Define loss function and optimizer
loss_fn = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr = 0.001)

def train_model(model, n_epochs):

    for epoch in range(1, n_epochs + 1):
        ###################
        # train the model #
        ###################
        model.train() # prep model for training
        # clear the gradients of all optimized variables
        optimizer.zero_grad()
        # forward pass: compute predicted outputs by passing inputs to the model
        Y_pred = model(X)
        # calculate the loss
        loss = loss_fn(Y_pred,Y)
        # backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # perform a single optimization step (parameter update)
        optimizer.step()
        # print loss value
        if epoch%1000 == 0:
            print('loss:',epoch,loss.item())

    return model

n_epochs = 100000
model = train_model(model,n_epochs)

model.eval()
Y_pred = model(X)
Y_x_pred = torch.autograd.grad(Y_pred,X,grad_outputs=torch.ones_like(X),create_graph = True,only_inputs=True,allow_unused=True)[0]
Y_xx_pred = torch.autograd.grad(Y_x_pred,X,grad_outputs=torch.ones_like(X),create_graph = True,only_inputs=True,allow_unused=True)[0]
y_pred = Y_pred.detach().numpy()
y_x_pred = Y_x_pred.detach().numpy()
y_xx_pred = Y_xx_pred.detach().numpy()

fig = plt.figure(figsize=(5,4))
plt.plot(x,y,'g-', linewidth=1)
plt.plot(x,y_pred,'r--', linewidth=2)
plt.show()

fig = plt.figure(figsize=(5,4))
plt.plot(x,y_x,'g-', linewidth=1)
plt.plot(x,y_x_pred,'r--', linewidth=2)
plt.show()

fig = plt.figure(figsize=(5,4))
# plt.plot(x,y_x,'g-', linewidth=1)
plt.plot(x,y_xx_pred,'r--', linewidth=2)
plt.show()


title: pytorch classifier for mnist data not work

body:
I'm new to pytorch, and I try to train a simple classifier with mnist data. However, my classifier's accuracy is about 10%, I tried several method to adjust the network, but failed, the classifier's output label is always the same, all 0, or all 7, or all 6. Please tell me what is wrong with the code.(I know I should use DataLoader, I will take a look at it later, now I just want to make the classifier's accuracy looks gook)
# coding=utf-8
# 数据为data中的handwritten_digit

import struct
import numpy as np
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import time

data_folder = '../data/handwritten_digit/'
dt = torch.get_default_dtype()
train_label_file = 'train-labels-idx1-ubyte'
train_img_file = 'train-images-idx3-ubyte'
test_img_file = 't10k-images-idx3-ubyte'
test_label_file = 't10k-labels-idx1-ubyte'
model_path = './handwritten_digit_recognition_net3.pth'

def timer(func):
    def cal_time(*args, **kw):
        start_time = time.time()
        out = func(*args, **kw)
        end_time = time.time()
        print('函数 ', func.__name__, ' 运行耗时', end_time-start_time, '秒', sep = '')
        return out
    return cal_time

def read_imgs(file):
    with open(data_folder+file, 'rb') as frb:
        # 先读取meta
        magic_num, img_num, row_num, col_num = struct.unpack('>IIII', frb.read(16))
        # print(magic_num, img_num, row_num, col_num)
        # img = np.fromfile(frb, dtype = np.uint8, count = row_num*col_num).reshape(row_num, col_num)
        # print(img, img.shape, 'img')
        imgs = np.fromfile(frb, dtype = np.uint8).reshape(img_num, row_num, col_num)
        # imgs = np.fromfile(frb, dtype = np.uint8, count = row_num*col_num*img_num).reshape(img_num, row_num, col_num)
    return torch.from_numpy(imgs).type(dt).unsqueeze(1).unsqueeze(1)

def read_labels(file):
    with open(data_folder+file, 'rb') as frb:
        # 先读取meta
        magic_num, label_num = struct.unpack('>II', frb.read(8))
        # print(magic_num, label_num)
        labels = np.fromfile(frb, dtype = np.uint8)
    return torch.from_numpy(labels).type(dt)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 12, 5)
        self.conv2 = nn.Conv2d(12, 12, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.linear1 = nn.Linear(12*16, 30)
        self.linear2 = nn.Linear(30, 10)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        x = F.relu(self.conv2(x))
        x = self.pool(x)
        x = x.view(-1, 12*16)
        # print(x.size(), 'x.size()')
        x = F.relu(self.linear1(x))
        x = self.linear2(x)
        return x

@timer
def train_and_save_net():
    train_imgs = read_imgs(train_img_file)
    train_labels = read_labels(train_label_file)
    test_imgs = read_imgs(test_img_file)
    test_labels = read_labels(test_label_file)

    # label = torch.zeros(1, 10)

    # label[0][int(train_labels[0])] = 1
    # print(label)
    # print(train_labels[0])
    # return

    net = Net()
    # criterion = nn.MSELoss()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr = 0.001, momentum = 0.9)

    print('Start Training')
    sum_loss = 0
    for i, img in enumerate(train_imgs):
        optimizer.zero_grad()
        predicted = net(img)
        # label = torch.zeros(1, 10)
        # label[0][int(train_labels[i])] = 1
        label = torch.tensor([train_labels[i]], dtype = torch.long)
        # print(predicted, predicted.size(), 'predicted')
        # print(label, label.size(), 'label')
        loss = criterion(predicted, label)
        loss.backward()
        optimizer.step()

        sum_loss += loss.item()
        if i % 2000 == 1999:
            print('已经训练了', i+1, '张图片，', '完成进度：', '%.2f'%((i+1)/len(train_labels)*100), '%', sep = '')
            print('loss为：', sum_loss/2000)
            sum_loss = 0
    print('End Training')

    torch.save(net.state_dict(), model_path)
    print('End Saving Net Parameters')

def load_net():
    net = Net()
    net.load_state_dict(torch.load(model_path))
    return net

@timer
def evaluate():

    train_imgs = read_imgs(train_img_file)
    train_labels = read_labels(train_label_file)
    test_imgs = read_imgs(test_img_file)
    test_labels = read_labels(test_label_file)

    net = load_net()

    # 直观感受
    for i in range(5):
        img = train_imgs[i]
        # plt.imshow(img.squeeze(), cmap = 'gray')
        # plt.show()
        predicted_vector = net(img)
        _, predicted = torch.max(predicted_vector, 1)
        predicted = predicted.item()
        print('预测的分类是：', predicted, '，实际的分类是：', int(train_labels[i].item()), sep = '')

    # 训练集精度
    total = len(train_labels)
    correct = 0
    for i in range(len(train_labels)):
        img = train_imgs[i]
        predicted_vector = net(img)
        _, predicted = torch.max(predicted_vector, 1)
        label = int(train_labels[i].item())
        if predicted == label:
            correct += 1
    print('训练集上的准确率为：', '%.2f'%(correct/total*100), '%', sep = '')

    total = len(test_labels)
    correct = 0
    pre_arr = []
    for i in range(len(test_labels)):
        img = test_imgs[i]
        predicted_vector = net(img)
        _, predicted = torch.max(predicted_vector, 1)
        label = int(test_labels[i].item())
        pre_arr.append(predicted)
        if predicted == label:
            correct += 1
    print('测试集上的准确率为：', '%.2f'%(correct/total*100), '%', sep = '')
    print('模型判断为0的个数/总判断数 为：', pre_arr.count(0), '/', len(pre_arr), sep = '')

@timer
def test():
    predicted_vector = torch.randn(1,10)
    _, predicted = torch.max(predicted_vector, 1)
    print(predicted.item())

if __name__ == '__main__':
    train_and_save_net()
    # test()
    evaluate()


title: Java bindings for PyTorch failed due to missing LIBTORCH_HOME

body:
I am trying to run the PyTorch java-demo:
https://github.com/pytorch/java-demo
When running:
java-demo $ ./gradlew run

It failed with this error message
LIBTORCH_HOME not present in environment

I used Conda install or pytorch with version 1.6.
conda install pytorch

That did not create anything that I could point LIBTORCH_HOME to.
I also tried a
pip3 install torch==1.6.0 torchvision==0.7.0

Do you need to install PyTorch from source to get java-demo working?


title: Pytorch accessing data points in a data batch

body:
I'm trying to run my model on selected data points in a data batch, but the model generates completely different outputs from directly running on the batch.
Here's an example:
I have a data batch (firstData) with shape torch.Size([128, 1, 28, 28]), which means it contains 128 data points.
firstData is retrieved by:
for _, (data, _) in enumerate(train_loader):
    firstData = data
    break

train_loader is a torch.utils.data.DataLoader on MNIST dataset
Then I run:
with torch.no_grad():
    print(cnn(firstData.cuda())[0:2]) # run on batch and get first 2 outputs

and it outputs:
tensor([[-0.5045, -0.8611, -1.0237, -2.2146,  1.6829, -0.1202,  8.2230, -2.9030,
         -0.6736, -1.6260],
        [-1.6367, -0.0683,  2.3553, -2.8480,  8.0057, -2.7570, -0.7046, -1.3720,
         -1.1558,  0.2683]], device='cuda:0')

When I run:
with torch.no_grad():
    print(cnn(firstData.cuda()[0:2])) # select first 2 data points and run the model

and it outputs:
tensor([[ 0.8256, -1.2180, -2.2164, -0.3754, -1.9170,  2.4913,  4.2119, -2.6611,
          1.1260, -0.2956],
        [-1.1582,  0.8964,  3.2793, -0.4405,  2.4863, -2.9732, -3.5990,  1.8273,
         -0.6883,  0.3878]], device='cuda:0')

Additionally, cnn is a resnet18 like structure and is already trained to 0.99 accuracy on the MNIST dataset. I also made cnn deterministic and it always returns the same results given the same inputs.
I'm new to Pytorch and I guess it is because I'm accessing the batched data in a wrong way. Can anyone give some suggestions to solve that?
Update
Thanks @kHarshit ! Calling cnn.eval() solved my problem.
In a PyTorch tutorial here:
"Remember that you must call model.eval() to set dropout and batch normalization layers to evaluation mode before running inference. Failing to do this will yield inconsistent inference results. If you wish to resuming training, call model.train() to ensure these layers are in training mode."
This makes sense because my cnn contains BatchNorm2d layers.
Official API can be found here


title: How to cache big data in memory (efficiently) in complex variables across executions of Python scripts?

body:
I am trying to call (from Java Spring beans) Python Pytorch scripts that contains trained neural networks for the use: my Pytorch neural networks are neural functions that accepts state, encodes it and returns the action, decodes all this is according to learned/trained policy.
So - each time when I am trying to invoke Python script I should construc torch.nn, load weights and biases from some external store (DB, file) and then execute this nn to get single answer, pretty expensive operation.
How can I keep torch.nn instance (with loaded weights and biases) in memory and make it available immediately for each execution of Python script?
Memcached is not the solution, because it can keep string or binary values only and it is quite expensive to serialized and deserialize torch.nn instance. One suggestion was Tensorflow Serving, I am currently researching it, so - I don't yet know whether this is the answer.
It is quite possible that Tensorflow has some caching technologies which I can use for the Pytroch as well?


title: Pytorch ignite create_supervised_evaluator not returning the Custom Metrics

body:
I am trying to use create_supervised_evaluator from ignite.engine with my custom Metrics
evaluator = create_supervised_evaluator(model, metrics={'perform': Perform()}, device=device)

But evaluator.state.metrics.keys() is returning empty state dict.
I am using PyTorch == 1.7.1 and pytorch-ignite 0.4.2 . Can any one please let me know where am I going wrong .


title: PyTorch: Higher order derivates of bivariate function

body:
Suppose I have a bivariate function f(x,y) with domain R^d x R^d, and I have two sets of inputs X = [x1,x2,...,xm]  and Y = [y1,y2,...,yn], where each xi and yj are d-dimensional vectors and I would like to compute the matrix with i,jth element [f(xi,yj)]_ij. I can do this in PyTorch by broadcasting and doing something like f(X.unsqueeze(1),Y).
What I would really like to compute is the following matrix:
[d/dx d/dy f(xi,yj)]_ij, where I have assumed each xi and yi is scalar valued. How would I do this in PyTorch?
I am aware that for single input functions and for first derivatives, I can do something like (using PyTorch's autograd.grad):
def derivative(x,f):
    return grad(f(x).sum(), x, create_graph=True)[0]

Or:
def derivative(x, f):
    return grad(f(x), x, grad_outputs=torch.ones(x.shape),create_graph=True)

However, I am unable to generalise to the two-dimensional and higher derivative output case! Any help would be appreciated.


title: Autoencoder - Reconstruction lacks color

body:
My goal is to find unsupervised full body landmarks. For that purpose I am using an autoencoder structure to disentangle shape and appearance of full body images (deep fashion dataset). The loss function contains a reconstruction loss, which is needed as a side task.
The detection of landmarks works quite okay, as can be seen in the image below. However, the reconstructed images strongly lack in color diversity.

On this picture, you can see two input images - the original image and an augmented version of the image. Both image should be reconstructed. You can see the images in the following order:
input image - spatial representation - detected keypoints - reconstruction - original image with landmarks - input image augmented - etc..
As you can see, the reconstruction of the augmented image totally lacks color. It has no green parts at all, even though the loss is calculated with respect to the augmented image. Also the reconstruction of the unaugmented image is very "dry".
As loss function for the reconstruction loss, I am using L1-Loss around the detected landmarks. I assume, that this might be suboptimal. Does anyone have ideas, how this could be improved? Are there more suitable loss functions for reconstruction? I have also tried L2-Loss, but there was no improvement.
Thanks a lot!


title: Pytorch GRU error RuntimeError : size mismatch, m1: [1600 x 3], m2: [50 x 20]

body:
Currently, I'm trying to make the training model for an LSTM and GRU. The LSTM is working perfectly,  but once I switched into GRU training, errors pop-out such as size mismatch error.
This is my code
    path = "new_z_axis"
    device = "cuda:0"
    in_size = 3
    h_size = 50
    n_layers = 3
    fc = 20
    out = 1
    batch_size = 16
    seq = 100
    epoch = 100
    learning_rate = 1e-3
    ratio = 0.8
    checkpoint = os.path.join("checkpoints","model_"+path+"_"+str(in_size)+".pth")

    class GRUNet(nn.Module):
        def __init__(self,in_size,h_size,n_layers,fc_out,out_size,dropout=0.5):
            super(GRUNet, self).__init__()
            self.gru = nn.GRU(input_size=in_size,hidden_size=h_size,num_layers=n_layers,dropout=dropout,bias=False)
            self.fc = nn.Linear(in_features=h_size,out_features=fc_out,bias=False)
            self.relu = nn.ReLU(inplace=True)
            self.out = nn.Linear(in_features=fc_out,out_features=out_size,bias=False)
            self.tanh = nn.Tanh()
        def forward(self, x, hidden):
            out, hidden = self.gru(x, hidden)
            x = self.fc(x)
            x = self.relu(x)
            x = self.out(x)
            x = self.tanh(x)
            return x, hidden

    class MyLstm(nn.Module):
        def __init__(self,in_size,h_size,n_layers,fc_out,out_size,dropout=0.5):
            super(MyLstm, self).__init__()
            self.lstm = nn.LSTM(input_size=in_size,hidden_size=h_size,num_layers=n_layers,dropout=dropout,bias=False)
            self.fc = nn.Linear(in_features=h_size,out_features=fc_out,bias=False)
            self.relu = nn.ReLU(inplace=True)
            self.out = nn.Linear(in_features=fc_out,out_features=out_size,bias=False)
            self.tanh = nn.Tanh()
        def forward(self,x,hidden):
            x, hidden = self.lstm(x,hidden)
    #         x = x[-1:]
            x = self.fc(x)
            x = self.relu(x)
            x = self.out(x)
            x = self.tanh(x)
            return x, hidden

    def train(model,train_list,val_list,path,seq,epoch,batch_size,criterion,optimizer,model_type):
        for e in range(epoch):
            train_data = load_data(train_list,batch_size)
            a_loss = 0
            a_size = 0
            model.train()
            for x,y in train_data:
                x,y = x.to(device),y.to(device)
                bs = x.size()[1]

    #             hidden = (hidden[0].detach(),hidden[1].detach())
    #             print(x.size(),hidden[0].size())
                if model_type == "GRU":
                    h1 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                    hidden = h1
                    hidden = hidden.data
                else:
                    h1 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                    h2 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                    hidden = (h1,h2)
                    hidden = tuple([e.data for e in hidden])

                model.zero_grad()
                print (len(hidden))
                pred,hidden = model(x,hidden)
                loss = criterion(pred,y)
                loss.backward()
                nn.utils.clip_grad_norm_(model.parameters(),5)
                optimizer.step()
                a_loss += loss.detach()
                a_size += bs
    #         print(e,a_loss/a_size*1e+6)
            model.eval()
            with torch.no_grad():
                val_data = load_data(val_list,batch_size)
                b_loss = 0
                b_size = 0
                for x,y in val_data:
                    x,y = x.to(device),y.to(device)
                    bs = x.size()[1]
                    if model_type == "GRU":
                        h1 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                        hidden = h1
                        hidden = hidden.data
                    else:
                        h1 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                        h2 = torch.zeros((n_layers,bs,h_size)).to("cuda:0")
                        hidden = (h1,h2)
                        hidden = tuple([e.data for e in hidden])
                    pred,hidden = model(x,hidden)
                    loss = criterion(pred,y)
                    b_loss += loss.detach()
                    b_size += bs
            print("epoch: {} - train_loss: {} - val_loss: {}".format(e+1,float(a_loss.item()/a_size*1e+6),b_loss.item()/b_size*1e+6))

train(modelGRU,train_list,val_list,path,seq,epoch,batch_size,criterionGRU,optimizerGRU,model_type="GRU")

This is the error i got

-------------------------------------------------- -------------------------  RuntimeError                               Traceback (most recent call last)   in

---- > 1  train ( modelGRU , train_list , val_list , path , seq , epoch , batch_size , criterionGRU , optimizerGRU , model_type = "GRU"
)
 in train (model, train_list, val_list,
path, seq, epoch, batch_size, criterion, optimizer, model_type)
61              model . zero_grad ( )
62              print ( len ( hidden ) )
---> 63              pred , hidden = model ( x , hidden )
64              loss = criterion ( pred , y )
65              loss .backward ( )
~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in call
(self, *input, **kwargs)
539              result = self . _slow_forward ( * input ,  ** kwargs )
540          else :
--> 541              result = self . forward ( * input ,  ** kwargs )
542          for hook in self . _forward_hooks . values ( ) :
543              hook_result = hook ( self , input , result )
 in forward (self, x, hidden)
11      def forward ( self , x , hidden ) :
12          out , hidden = self . gru ( x , hidden )
---> 13          x = self . fc ( x )
14          x = self . relu ( x )
15          x =self . out ( x )
~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in call
(self, *input, **kwargs)
539              result = self . _slow_forward ( * input ,  ** kwargs )
540          else :
--> 541              result = self . forward ( * input ,  ** kwargs )
542          for hook in self . _forward_hooks . values ( ) :
543              hook_result = hook ( self , input , result )
~ \ Anaconda3 \ lib \ site-packages \ torch \ nn \ modules \ linear.py
in forward (self, input)
85
86      def forward ( self , input ) :
---> 87 return F . Linear ( the Input , Self . Weight , Self . negative bias )      88      89 def extra_repr ( Self ) :
~\Anaconda3\lib\site-packages\torch\nn\functional.py in linear (input,
weight, bias)     1370          ret = torch . addmm ( bias , input ,
weight . t ( ) )     1371      else :
-> 1372          output = input . matmul ( weight . t ( ) )     1373          if bias is  not  None :     1374              output += bias
RuntimeError : size mismatch, m1: [1600 x 3], m2: [50 x 20] at
C:/w/1/s/tmp_conda_3.7_104508/conda/conda-bld/pytorch_1572950778684/work/aten/src\THC/
generic/THCTensorMathBlas.cu:290

Any advice?
Thank you


title: PyTorch training with Batches of different lenghts?

body: Is it possible to train  model with batches that have unequal lenght during an epoch? I am new to pytorch.


title: Pytorch Linear Regression with squared features

body:
I am new to PyTorch and I would like to implement linear regression partly with PyTorch and partly on my own. I want to use squared features for my regression:
import torch

# init
x = torch.tensor([1,2,3,4,5])
y = torch.tensor([[1],[4],[9],[16],[25]])
w = torch.tensor([[0.5], [0.5], [0.5]], requires_grad=True)

iterations = 30
alpha = 0.01

def forward(X):
    # feature transformation [1, x, x^2]
    psi = torch.tensor([[1.0, x[0], x[0]**2]])
    for i in range(1, len(X)):
        psi = torch.cat((psi, torch.tensor([[1.0, x[i], x[i]**2]])), 0)
    return torch.matmul(psi, w)

def loss(y, y_hat):
    return ((y-y_hat)**2).mean()

for i in range(iterations):

    y_hat = forward(x)

    l = loss(y, y_hat)
    l.backward()

    with torch.no_grad():
        w -= alpha * w.grad
    w.grad.zero_()

    if i%10 == 0:
        print(f'Iteration {i}: The weight is:\n{w.detach().numpy()}\nThe loss is:{l}\n')

When I execute my code, the regression doesn't learn the correct features and the loss increases permanently. The output is the following:
Iteration 0: The weight is:
[[0.57 ]
 [0.81 ]
 [1.898]]
The loss is:25.450000762939453

Iteration 10: The weight is:
[[ 5529.5835]
 [22452.398 ]
 [97326.12  ]]
The loss is:210414632960.0

Iteration 20: The weight is:
[[5.0884394e+08]
 [2.0662339e+09]
 [8.9567642e+09]]
The loss is:1.7820802835250162e+21

Does somebody know, why my model is not learning?
UPDATE
Is there a reason why it performs so poorly? I thought it's because of the low number of training data. But also with 10 data points, it is not performing well :


title: PyTorch cuda memory allocation, why do multiplication is more memory consuming than addition

body:
It looks like multiplication duplicates the original input while addition is performed in_place.
Can someone please help me understand why?
To replicate:
import torch
import torch.nn as nn

x = torch.rand(32, 256, 100, 100).cuda()
z = torch.rand(32, 40).cuda()

batchnorm = nn.BatchNorm2d(256).cuda()
gamma = nn.Linear(in_features=40, out_features=256, bias=False).cuda()
beta = nn.Linear(in_features=40, out_features=256, bias=False).cuda()

print(f"Allocated mem: {torch.cuda.memory_stats('cuda')['allocated_bytes.all.current']/1e9:.3f} GB")
x = batchnorm(x)
print(f"Allocated mem: {torch.cuda.memory_stats('cuda')['allocated_bytes.all.current']/1e9:.3f} GB")
x = x + beta(z).view((-1, x.shape[1], 1, 1))
print(f"Allocated mem: {torch.cuda.memory_stats('cuda')['allocated_bytes.all.current']/1e9:.3f} GB")
x = x * gamma(z).view((-1, x.shape[1], 1, 1))
print(f"Allocated mem: {torch.cuda.memory_stats('cuda')['allocated_bytes.all.current']/1e9:.3f} GB")

The order of the two operations is for sake of clarity alone. The results are the same when flipping the two operations.
The output:
Allocated mem: 0.328 GB
Allocated mem: 0.655 GB
Allocated mem: 0.655 GB
Allocated mem: 0.983 GB

Thanks


title: How to get the indexes of equal elements in two different size PyTorch tensors?

body:
Let's say I have two PyTorch tensors:
t_1d = torch.Tensor([6, 5, 1, 7, 8, 4, 7, 1, 0, 4, 11, 7, 4, 7, 4, 1])
t = torch.Tensor([4, 7])

I want to get the indices of exact match intersection between the sets for
the tensor t_1d with tensor t.
Desired output of t_1d and t: [5, 12] (first index of exact intersection)
Preferably on GPU for large Tensors, so no loops or Numpy casts.


title: Creating an annotation list using time stamps

body:
I have ECG data of length 230897 and sampling frequency=100hz. I would like to create an annotation list (of length 230897) corresponding to the timestamps and duration (in seconds) of the arrhythmia event (sequence of 1's):
timestamps                                  duration
2014-09-10T22:10:20.000000000                 3.5
2014-09-10T23:10:10.000000000                 4
2014-09-10T23:50:20.000000000                 6

FOr example: the annotation list should be of length 230897 and at '2014-09-10T22:10:20.000000000' there should be 1's for 3.5 seconds; at'2014-09-10T23:10:10.000000000' 1's for 4 seconds and so on. This is what I have tried so far and unfortunately, it is not working:
start_time= '2014-09-10T21:01:10.000000000'
data_length=230897
annot_list = np.zeros(data_length)

prev=0, fs=100
for t in timestamps:
        time_diff = t - prev
        seconds = time_diff.astype('timedelta64[s]').astype(np.int32)
        total_cov = seconds*fs
        i1 = i0 + total_cov
        i0, i1 = int(i0), int(i1)
        annot_list[i0:i1] = 1.0
        i0 = i1
        prev = t

Can someone please help me here?


title: Pytorch: split a tensor by column

body:
How can I split a tensor by column (axis = 1). For example
"""
input:              result:
tensor([[1, 1],     (tensor([1, 2, 3, 1, 2, 3]),
        [2, 1],      tensor([1, 1, 2, 2, 3, 3]))
        [3, 2],
        [1, 2],
        [2, 3],
        [3, 3]])
"""

The solution I came out with is first transpose the input tensor, split it and then flatten each of the split tensor. However, is there a simpler and more effective way on doing this? Thank you
import torch
x = torch.LongTensor([[1,1],[2,1],[3,2],[1,2],[2,3],[3,3]])
x1, x2 = torch.split(x.T, 1)
x1 = torch.flatten(x1)
x2 = torch.flatten(x2)
x1, x2 # output


title: pytorch tensorboard "add_embedding error"

body:
everyone.
I'm stuck in using tensorboard in pytorch.
The point is add_embedding method makes the error like below:
Traceback (most recent call last):
  File "test2.py", line 126, in <module>
    writer.add_embedding(features, metadata=class_labels, label_img = images.unsqueeze(1))
  File "/home/dgjung/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/utils/tensorboard/writer.py", line 798, in add_embedding
    fs = tf.io.gfile.get_filesystem(save_path)
AttributeError: module 'tensorflow._api.v2.io.gfile' has no attribute 'get_filesystem'

My code is pytorch tutorial.
# log embeddings
features = images.view(-1, 28 * 28)
writer.add_embedding(features,
                    metadata=class_labels,
                    label_img=images.unsqueeze(1))

My enviroment is :

PyTorch : '1.7.1'
Tensorflow : '2.4.1'
Python : 3.8.8

PLZ, help me!


The task at hand is to attach the Opacus privacy engine to the optimizer and train the following model to make it differentially private.
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    """Class used to initialize model of student/teacher"""

    def __init__(self):

        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5, 1)
        self.conv2 = nn.Conv2d(20, 50, 5, 1)
        self.fc1 = nn.Linear(4 * 4 * 50, 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):

        x = F.relu(self.conv1(x))
        x = F.max_pool2d(x, 2, 2)
        x = F.relu(self.conv2(x))
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 4 * 4 * 50)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        #return F.log_softmax(x, dim=1)
        return x

I followed the instructions given in Opacus's [Documentation][1] and read through their documentation of [privacy engine][2], but wasn't able to figure out the cause of the error.
Given below is the code
model = Model()
optimizer = optim.SGD(model.parameters(), lr = 0.001)
batch_size = sample_size = 32
privacy_engine = PrivacyEngine(
    model,
    batch_size,
    alphas=[10, 100],
    noise_multiplier= 9.7,
    max_grad_norm = 1.0,
)
privacy_engine.attach(optimizer)

Error
TypeError                                 Traceback (most recent call last)
<ipython-input-8-84493fad0b76> in <module>()
      7     alphas=[10, 100],
      8     noise_multiplier= 9.7,
----> 9     max_grad_norm = 1.0
     10 )
     11 privacy_engine.attach(optimizer)

TypeError: __init__() takes 2 positional arguments but 3 positional arguments (and 3 keyword-only arguments) were given


title: pytorch deep learning loading data sequentially and efficiently

body:
I have been doing neural network analysis on 20 thousand "images", each image represented in the form of the intensity of 100 * 100 * 100 neurons.
x = np.loadtxt('imgfile')
x = x.reshape(-1, img_channels, 100, 100, 100)
//similarly for target variable 'y'

Above, the first dimension of x will be the number of images. Am using DataLoader to get appropriate number of images for training during each iteration as shown below.
batch_size = 16
traindataset = TensorDataset(Tensor(x[:-testdatasize]), Tensor(y[:-testdatasize]) )
train_loader = DataLoader(dataset=traindataset, batch_size=batch_size, shuffle=True)

for epoch in range(num_epochs):
    for i, (data,targets) in enumerate(train_loader):
        ...

I hope to increase the number of images to 50k but am restricted by the computer memory  (imgfile is ~50 GB).
I was wondering if there is an efficient way to handle all the data? Like, rather than loading the whole imgfile, can we first divide them into sets, each with batch_size number of images, and load the sets periodically during training. I am not completely sure how to implement this.
I found some similar ideas using Keras here: https://machinelearningmastery.com/how-to-load-large-datasets-from-directories-for-deep-learning-with-keras/
Please point me towards any similar ideas implemented with pytorch or you have any ideas.


title: Confusion about the output channels of convolution neural network

body:
I'm confused about the multi-channel scenario in convolution neural network.
Say I have a 10(width) * 5(height) * 6(channels) image, and I feed it into a default 2-D convolution layer with stride=1 and padding=0 and expect the output to be 8(width) * 3(height) * 16(channels).
I know the size of the kernel is 3(width) * 3(height), but I don't know how many kernels are there exactly, and how the are applied to the input data to give the final 16 channels.
Someone can help me please.


title: How to get topk's values with its indices (2D)?

body:
I have two 3D tensor and I want to use one's top k indices get another top k.
For example for the following tensor
a = torch.tensor([[[1], [2], [3]],
                  [[4], [5], [6]]])

b = torch.tensor([[[7,1], [8,2], [9,3]],
                  [[10,4],[11,5],[12,6]]])

pytorch's topk function will give me the following.
top_tensor, indices = torch.topk(a, 2, dim=1)

# top_tensor: tensor([[[3], [2]],
#                    [[6],  [5]]])

# indices: tensor([[[2], [1]],
#                 [[2],  [1]]])

But I want to use the result of a, map to b.
# use indices to do something for b, get torch.tensor([[[8,2], [9,3]],
#                                                      [[11,5],[12,6]]])

In this case, I don't know the real values of b, so I can't use topk to b.
on the other word, I want to get a funtion foo_slice as following:
top_tensor, indices = torch.topk(a, 2, dim=1)
# top_tensor == foo_slice(a, indices)

Is there any approach to achieve this using pytorch?
Thanks!


title: when setting .eval() my model performs worse than when I set .train()

body:
During the training phase, I select the model parameters with the best performance metric.
if performance_metric.item()>max_performance:
        max_performance= performance_metric.item()
        torch.save(neural_net.state_dict(), PATH+'/best_model.pt')

This is the neural network model used:
class Neural_Net(nn.Module):
        def __init__(self, M,shape_input,batch_size):
            super(Neural_Net, self).__init__()
            self.lstm = nn.LSTM(shape_input,M)
            #self.dense1 = nn.Linear(shape_input,M)
            self.dense1 = nn.Linear(M,M) #Used with the LSTM
            torch.nn.init.xavier_uniform_(self.dense1.weight)
            self.dense2 = nn.Linear(M,M)
            torch.nn.init.xavier_uniform_(self.dense2.weight)
            self.dense3 = nn.Linear(M,1)
            torch.nn.init.xavier_uniform_(self.dense3.weight)
            self.drop   = nn.Dropout(0.7)
            self.bachnorm1 = nn.BatchNorm1d(M)
            self.relu = nn.ReLU()
            self.sigmoid = nn.Sigmoid()

            self.hidden_cell = (torch.zeros(1,batch_size,M),torch.zeros(1,batch_size,M))

        def forward(self, x):
            lstm_out, self.hidden_cell = self.lstm(x.view(1 ,len(x), -1), self.hidden_cell)
            x = self.drop(self.relu(self.dense1(self.bachnorm1(lstm_out.view(len(x), -1)))))
            x = self.drop(self.relu(self.dense2(x)))
            x = self.relu(self.dense3(x))
            return x

After that I load the model with the best parameters and set the evaluation mode:
neural_net.load_state_dict(torch.load(PATH+'/best_model.pt'))
neural_net.eval()

The results are completely random. When I set train() the performance is similar to the selected best model parameter.
There is an important aspect of the eval() that I am forgetting? Is the batch normalization correctly used? I am using a batch the same size as in the training phase for the test phase.


title: RuntimeError:shape ‘[4, 98304]’ is invalid for input of size 113216

body:
I am learning to train a basic nn model for image classification, the error happened when I was trying to feed in image data into the model. I understand that I should input correct size of image data. My image data is 128*256 with 3 channels,4 classes, and the batch size is 4. What I don't understand is where does the size 113216 come from? I checked all related parameters or image meta data, but didn't find a clue. Here is my code:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(3*128*256, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(4, 3*128*256)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
for epoch in range(2):  # loop over the dataset multiple times
    print('round start')
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        print(inputs.shape)
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

Thanks for your help!


title: Given groups=1, weight of size [32, 3, 3, 3], expected input[1, 1, 32, 340] to have 3 channels, but got 1 channels instead

body:
This is the question:
Before we define the model, we define the size of our alphabet. Our alphabet consists of lowercase English letters, and additionally a special character used for space between symbols or before and after the word. For the first part of this assignment, we don't need that extra character.
Our end goal is to learn to transcribe words of arbitrary length. However, first, we pre-train our simple convolutional neural net to recognize single characters. In order to be able to use the same model for one character and for entire words, we are going to design the model in a way that makes sure that the output size for one character (or when input image size is 32x18) is 1x27, and Kx27 whenever the input image is wider. K here will depend on particular architecture of the network, and is affected by strides, poolings, among other things. A little bit more formally, our model   , for an input image    gives output energies  =() . If  ∈ℝ32×18 , then  ∈ℝ1×27 . If  ∈ℝ32×100  for example, our model may output  ∈ℝ10×27 , where    corresponds to a particular window in   , for example from  0,9  to  32,9+18  (again, this will depend on the particular architecture).
The code:
# constants for number of classes in total, and for the special extra character for empty space
ALPHABET_SIZE = 27, # Extra character for space inbetween
BETWEEN = 26

print(alphabet.shape) # RETURNS: torch.Size([32, 340])

My CNN Block:
from torch import nn
import torch.nn.functional as F

"""

Remember basics:
1. Bigger strides = less overlap
2. More filters = More features

Image shape = 32, 18
Alphabet shape = 32, 340

"""

class SimpleNet(torch.nn.Module):

    def __init__(self):
        super().__init__()
        self.cnn_block = torch.nn.Sequential(
            nn.Conv2d(3, 32, 3),
            nn.BatchNorm2d(32),
            nn.Conv2d(32, 32, 3),
            nn.BatchNorm2d(32),
            nn.Conv2d(32, 32, 3),
            nn.BatchNorm2d(32),
            nn.MaxPool2d(2),
            nn.Conv2d(32, 64, 3),
            nn.BatchNorm2d(64),
            nn.Conv2d(64, 64, 3),
            nn.BatchNorm2d(64),
            nn.Conv2d(64, 64, 3),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(2)
          )

    def forward(self, x):
        x = self.cnn_block(x)
        # after applying cnn_block, x.shape should be:
        # batch_size, alphabet_size, 1, width
        return x[:, :, 0, :].permute(0, 2, 1)

model = SimpleNet()

alphabet_energies = model(alphabet.view(1, 1, *alphabet.shape))

def plot_energies(ce):
    fig=plt.figure(dpi=200)
    ax = plt.axes()
    im = ax.imshow(ce.cpu().T)

    ax.set_xlabel('window locations →')
    ax.set_ylabel('← classes')
    ax.xaxis.set_label_position('top')
    ax.set_xticks([])
    ax.set_yticks([])

    cax = fig.add_axes([ax.get_position().x1+0.01,ax.get_position().y0,0.02,ax.get_position().height])
    plt.colorbar(im, cax=cax)

plot_energies(alphabet_energies[0].detach())

I get the error in the title at alphabet_energies = model(alphabet.view(1, 1, *alphabet.shape))
Any help would be appreciated.


title: Dimensionality Issue on Pytorch Transformer

body:
I'm starting with Pytorch and I'm currently making an Attention model with a encoder-decored structure.
I am following this tutorial, and while I'm not getting any errors, there has to be an issue because I'm not getting the output tensor with the size needed, I have checked the code multiple times but I can't find the problem, maybe you can see something that I haven't seen.
This is the encoder:
import torch
import torch.nn.functional as F

class Encoder(torch.nn.Module):
    def __init__(self):
        super(Encoder, self).__init__()
        self.hidden_size = 32
        self.input_size = 16

        self.lstm = torch.nn.LSTM(self.input_size, self.hidden_size)

    def forward(self, inputs, hidden):
        output, hidden = self.lstm(inputs, hidden)
        return output, hidden

    def init_hidden(self):
        # (num_layers * num_directions, batch, hidden_size)
        return (torch.zeros(1*1, 10, self.hidden_size),
                torch.zeros(1*1, 10, self.hidden_size))

And this is the decoder:
class AttentionDecoder(torch.nn.Module):
    def __init__(self):
        super(AttentionDecoder, self).__init__()
        self.hidden_size = 20
        self.output_size = 20
        self.vocab_size = 16
        self.batch_size = 64

        self.attn = torch.nn.Linear(self.hidden_size + self.vocab_size, 16)
        self.lstm = torch.nn.LSTM(self.vocab_size, self.output_size, batch_first=True) #if we are using embedding hidden_size should be added with embedding of vocab size
        self.final = torch.nn.Linear(self.output_size, self.vocab_size)

    def init_hidden(self):
        # (num_layers * num_directions, batch, hidden_size)
        return (torch.zeros(1, self.batch_size, self.hidden_size),
                torch.zeros(1, self.batch_size, self.hidden_size))

    def forward(self, decoder_hidden, encoder_outputs, inputs):
        weights = []
        for i in range(len(encoder_outputs)):
            weights.append(self.attn(torch.cat((decoder_hidden[0][0],
                                              encoder_outputs[i]), dim=1)))
        normalized_weights = F.softmax(torch.cat(weights, 1), 1)
        attn_applied = torch.bmm(normalized_weights.view(-1, self.vocab_size, self.hidden_size),
                                 encoder_outputs.view(-1, self.hidden_size, self.vocab_size))

        input_lstm = torch.cat((attn_applied, inputs), dim=1) #if we are using embedding, use embedding of input here instead
        output, hidden = self.lstm(input_lstm, decoder_hidden)
        output = self.final(output)
        return output, hidden, normalized_weights

Then I create the model in a new class:
class Model(torch.nn.Module):
    def __init__(self, bidirectional=False):
        super().__init__()

        self.encoder = Encoder()
        self.decoder = AttentionDecoder()

    def forward(self, inputs):
        a, b = self.encoder.forward(x.float(), self.encoder.init_hidden())
        a = a.view(20, 64, 16)
        output, z, w = self.decoder.forward(self.decoder.init_hidden(), a, x.float())
        return output

When training the model, I have two variables, x and y, with dimensions:
# (batch, seq_len, vocab_size)
x = torch.zeros([64, 10, 16])
y = torch.zeros([64, 20, 16])

Then I pass x to the model:
model = Model()
output = model.forward(x)
print(output.size())
# torch.Size([64, 26, 16])

And I need the model to output size torch.Size([64, 26, 16]) instead, when do I have to pass the model the output seq_len (which is 20)?


I get the following error when i train my torch model with a custom dataset.
The dataset is text data transformed, tokenized and then made into a tensor of shape (100000 , 784)
I input the dataset into the following:
train_ds= data_utils.dataset.Dataset(MyData[:100000])

Im training a VAE
Later in the script, the following example function is used with '.view' where the error occurs:
 def loss(self, x):
    z_mu, z_std = self.encode(x.view(-1, 784))

The following error is thrown:
> > Traceback (most recent call last):   File "MyFile.py", line 234, in <module>
>     model.reconstruct(recon_dataset, device, epoch)   File "MyFile.py", line 157, in reconstruct
>     x = x.view(-1, 784).float().to(device) AttributeError: 'TensorDataset' object has no attribute 'view'

How to solve this?


title: Pytorch model training without using forward

body:
I'm working on training CLIP model.
Here's the source code of the model https://github.com/openai/CLIP/blob/main/clip/model.py
Basically the CLIP object is constructed like this :
class CLIP(nn.module):
   ...
   def encode_image(self, image):
     return self.visual(image.type(self.dtype))

   def encode_text(self, text):
    x = ...
    ...
    return x

   def forward(self, image, text):
     image_features = self.encode_image(image)
     text_features = self.encode_text(text)
     ...
     return logits_per_image, logits_per_text

The forward method except pair of image and text, since I want to repurpose CLIP for other task(text-text pairs),  I'm not using forward from CLIP, but I'm using others method defined inside CLIP. My training code look like this :
for k in range(epoch):
  for batch in dataloader :
    x,y = batch
    y1 = model.encode_text(x[first_text_part])
    y2 = model.encode_text(x[second_text_part])
    <calculate loss, backward, step, etc>

The problem is, after 1 epoch, all the gradients turn out to be nan even though the loss is not nan.
My suspicion is PyTorch only able to propagate the gradient through the forward method.
Some source says that forward is not that special (https://discuss.pytorch.org/t/must-use-forward-function-in-nn-module/50943/3), but other source say coding with torch must use the forward (https://stackoverflow.com/a/58660175/12082666).
The question is, can we train Pytorch network without using forward method?


title: Best way to debug or step over a sequential pytorch model

body:
I used to write the PyTorch model with nn.Module which included __init__ and forward so that I can step over my model to check how the variable dimension changes along the network.
However I have since realized that you can also do it with nn.Sequential which only requires an __init__, you don't need to write a forward function as below:

However, the problem is when I try to step over this network, it is not easy to check the variable any more. It just jumps to another place and back.
Does anyone know how to do step over in this situation?
P.S: I am using PyCharm.


title: Class weighting in multi-class semantic segmentation task

body:
I'm facing a semantic segmentation task using a neural network with U-Net architecture.
My dataset is Pannuke dataset, in which I have 6 different classes and I'm using a loss which calculates the dice loss for each separated channel and then returns the mean.
Since my dataset is very unbalanced, now I'd want to weight each channel with a different weight, related to class frequencies.
How can I calculate these weights?
Below the code of my "mean dice loss" function:
def mean_dice_loss(y_pr, y_gt):
        """
        y_pr shape: [batch_size, num_classes, H, W]
        y_gt shape: [batch_size, num_classes, H, W]
        """
        classes = set(range(y_pr.shape[1]))  # {0, 1, ..., num_classes - 1}
        loss = torch.zeros(len(classes)).to(device)

        for cl in range(len(classes)):
            ignore = classes - {cl}  # ignore all other classes
            loss[cl] = 1.0 - f_score(y_pr, y_gt, ignore_channels=ignore)

        ### here I want to do something like: loss *= weights

        return loss.mean()

P.S. I'm using Pytorch. f-score is calculated with this library: https://github.com/qubvel/segmentation_models.pytorch