title: Pytorch LSTM and cross entropy

body:
I am working on sentiment analysis, I want to classify the output into 4 classes. For loss I am using cross-entropy.
The problem is PyTorch cross-entropy needs the input of (batch_size, output) which is am having trouble with.
I am taking a batch size of 12 and sequence size is 32
import torch.nn as nn

class RNN(nn.Module):
    def __init__(self, hidden_dim = 256, input_size = 32 , num_layers = 1, num_classes=4, vocab_size = len(vocab_to_int)+1, embedding_dim=100):
        super().__init__()
        self.input_size = input_size
        self.hidden_dim = hidden_dim
        self.num_layers = num_layers
        self.num_classes = num_classes
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, num_layers)
        self.fc1 = nn.Linear(hidden_dim, 50)
        self.fc2 = nn.Linear(50, 4)

    def forward(self, x, hidden):
        x = self.embedding(x)
        x = x.view(32, 12, 100)
        x, hidden = self.lstm(x, hidden)
        x = x.contiguous().view(-1, 256)
        x = self.fc1(x) # output shape ([384, 50])
        x = self.fc2(x) # output shape [384, 4]
        return x, hidden

    def init_hidden(self, batch_size=12):
        weight = next(self.parameters()).data
        hidden = (weight.new(self.num_layers, 12, self.hidden_dim).zero_().cuda(), weight.new(self.num_layers, 12, self.hidden_dim).zero_().cuda())
        return hidden


title: Can't fix: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

body:
Currently, I’m trying to replicate a DeblurGanV2 network. At the moment, I’m working on performing the training. Here is my current status of my trainings pipeline:
from torch.autograd import Variable
torch.autograd.set_detect_anomaly(mode=True)
total_generator_loss = 0
total_discriminator_loss = 0
psnr_score = 0.0
used_loss_function = 'wgan_gp_loss'
for epoch in range(n_epochs):

      #set to train mode
      generator.train(); discriminator.train()
      tqdm_bar = tqdm(train_loader, desc=f'Training Epoch {epoch} ', total=int(len(train_loader)))
      for batch_idx, imgs in enumerate(tqdm_bar):

        #load imgs to cpu
        blurred_images = imgs["blurred"].cuda()
        sharped_images = imgs["sharp"].cuda()

        # generator output
        deblurred_img = generator(blurred_images)

        # denormalize
        with torch.no_grad():
          denormalized_blurred = denormalize(blurred_images)
          denormalized_sharp = denormalize(sharped_images)
          denormalized_deblurred = denormalize(deblurred_img)

        # get D's output
        sharp_discriminator_out = discriminator(sharped_images)
        deblurred_discriminator_out = discriminator(deblurred_img)

        # set critic_updates
        if used_loss_function== 'wgan_gp_loss':
          critic_updates = 5
        else:
            critic_updates = 1

        #train discriminator
        discriminator_loss = 0
        for i in range(critic_updates):
          discriminator_optimizer.zero_grad()
          # train discriminator on real and fake
          if used_loss_function== 'wgan_gp_loss':
            gp_lambda = 10
            alpha = random.random()
            interpolates = alpha * sharped_images + (1 - alpha) * deblurred_img
            interpolates_discriminator_out = discriminator(interpolates)
            kwargs = {'gp_lambda': gp_lambda,
                       'interpolates': interpolates,
                       'interpolates_discriminator_out': interpolates_discriminator_out,
                       'sharp_discriminator_out': sharp_discriminator_out,
                       'deblurred_discriminator_out': deblurred_discriminator_out
                        }
            wgan_loss_d, gp_d = wgan_gp_loss('D', **kwargs)
            discriminator_loss_per_update = wgan_loss_d + gp_d

          discriminator_loss_per_update.backward(retain_graph=True)
          discriminator_optimizer.step()
          discriminator_loss += discriminator_loss_per_update.item()

But when I run this code, I receive the following error message:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [1, 512, 4, 4]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

RuntimeError Traceback (most recent call last) in () 62 # discriminator_loss_per_update = gan_loss_d 63 —> 64
discriminator_loss_per_update.backward(retain_graph=True) 65
discriminator_optimizer.step() 66 discriminator_loss +=
discriminator_loss_per_update.item()
1 frames /usr/local/lib/python3.7/dist-packages/torch/tensor.py in
backward(self, gradient, retain_graph, create_graph, inputs) 243
create_graph=create_graph, 244 inputs=inputs) → 245
torch.autograd.backward(self, gradient, retain_graph, create_graph,
inputs=inputs) 246 247 def register_hook(self, hook):
/usr/local/lib/python3.7/dist-packages/torch/autograd/init.py in
backward(tensors, grad_tensors, retain_graph, create_graph,
grad_variables, inputs) 145 Variable.execution_engine.run_backward(
146 tensors, grad_tensors, retain_graph, create_graph, inputs, → 147
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag
148 149

Unfortunately, I can’t really trace the in-place operation that would cause this error. Does anybody maybe has an idea or advice for me? I would appreciate any input :slight_smile:


title: PyTorch Training an LSTM - AttributeError: 'NoneType' object has no attribute 'config'

body:
I am training an LSTM and doing some hyperparameter tuning using Ray Tune that is based very much off of the PyTorch Tutorial here: https://pytorch.org/tutorials/beginner/hyperparameter_tuning_tutorial.html.
I get the following error:
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-161-e7423f64e4d4> in <module>()
    108 best_trial = result.get_best_trial(metric = "loss", mode = "min")
    109
--> 110 print("Best trial config: {}".format(best_trial.config))
    111 print("Best trial final validation loss: {}".format(
    112     best_trial.last_result["loss"]))

AttributeError: 'NoneType' object has no attribute 'config'

My code is:
checkpoint_dir = '/content/gdrive/MyDrive/Checkpoint_dir'
data_dir = './data'

epochs=5

def custom_train_part(config, checkpoint_dir=None, data_dir=None):
    model = LSTM(len(True_IMF_df.T), config["Hidden"], config["Layers"], 1)

    device = "cpu"
    if torch.cuda.is_available():
        device = "cuda:0"
        if torch.cuda.device_count() > 1:
            model = nn.DataParallel(model)
    model.to(device)
    optimizer = torch.optim.Adam(model.parameters(), lr=config["lr"])
    criterion = nn.MSELoss()

    if checkpoint_dir:
        model_state, optimizer_state = torch.load(
            os.path.join(checkpoint_dir, "checkpoint"))
        model.load_state_dict(model_state)
        optimizer.load_state_dict(optimizer_state)

    for e in range(epochs):

        running_loss = 0.0
        epoch_steps = 0

        model.train()  # put model to training mode
        x = x_hht_train.to(device)
        y = y_hht_train.to(device)

        scores = model(x)
        loss = criterion(scores, y_hht_train)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
        print(f"Running loss: {running_loss}")
        epoch_steps += 1

        if e % 5 == 0:
            print(f'Epoch: {e}, loss = {loss.cpu().item()}')
            # check_accuracy(loader_val, model)
            print()

        # Validation loss
        val_loss = 0.0
        val_steps = 0
        total = 0
        correct = 0

        with torch.no_grad():
            x = x_hht_val.to(device)  # move to device, e.g. GPU
            y = y_hht_val.to(device)

            scores = model(x)
            scores = scores.cpu()
            y = y.cpu()
            correct += (np.sign(scores) == np.sign(y)).sum().item()
            print(f"Correct: {correct}")

            loss = criterion(scores, y)
            print(f"Val Loss: {loss.item()}")
            val_loss += loss.cpu()
            val_steps += 1

        with tune.checkpoint_dir(e) as checkpoint_dir:
            path = os.path.join(checkpoint_dir, "checkpoint")
            torch.save((model.state_dict(), optimizer.state_dict()), path)

        tune.report(loss=(val_loss / val_steps), accuracy = correct / len(y_hht_val))
    print("Finished Training")

config = {
    "lr": tune.loguniform(1e-6, 1e-1),
    "Layers": tune.sample_from(lambda _: np.random.randint(1, 10)),
    "Hidden" : tune.sample_from(lambda _: np.random.randint(2, 100))
}

scheduler = ASHAScheduler(
        metric="loss",
        mode="min",
        max_t=10,
        grace_period=1,
        reduction_factor=2)

reporter = CLIReporter(
    #parameter_columns=["lr", "Layers", "Hidden"],
    metric_columns=["loss", "accuracy", "training_iteration"]
)

result = tune.run(
    custom_train_part,
    resources_per_trial={"cpu": 4, "gpu": 1},
    config=config,
    num_samples=1,
    scheduler = scheduler,
    progress_reporter=reporter
)

print(f"Results: {result.get_best_config(metric = 'loss', mode = 'min')}")

best_trial = result.get_best_trial(metric = "loss", mode = "min")

print("Best trial config: {}".format(best_trial.config))
print("Best trial final validation loss: {}".format(
    best_trial.last_result["loss"]))
print("Best trial final validation accuracy: {}".format(
    best_trial.last_result["accuracy"]))

##############################################################
#                       AFTER TUNNING                       #
##############################################################

best_trained_model = LSTM(len(True_IMF_df.T), best_trial.config["Hidden"], best_trial.config["Layers"], 1)
best_checkpoint_dir = best_trial.checkpoint.value
model_state, optimizer_state = torch.load(os.path.join(
    best_checkpoint_dir, "checkpoint"))
best_trained_model.load_state_dict(model_state)

It is evident that the issue is with the tune.run() but I don't know why it returns a NoneType Object


title: Pytorch Model always outputs 0.5 for an unkown reason

body:
I have a pytorch model I'm trying to use to do facial recognition. I am using the same model structure, loss, and optimizer as a working code, but it seems like the backprop won't do anything, any output of the NN is just 0.5. Here is the code, any help/suggestions is/are appreciated.
import cv2
import numpy as np
import PIL
import torchvision.transforms as transforms
import torch
from tqdm import tqdm
import torch.nn as nn
import torch.nn.functional as F

devicet = 'cuda' if torch.cuda.is_available() else 'cpu'
device = torch.device(devicet)
if devicet == 'cpu':
  print ('Using CPU')
else:
  print ('Using GPU')
cuda0 = torch.device('cuda:0')

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.step1 = nn.Linear(10000, 200)
        self.step2 = nn.Linear(200, 200)
        self.step3 = nn.Linear(200, 1)
    def forward(self, x):
      x = F.relu(x)
      x = self.step1(x)
      x = F.relu(x)
      x = self.step2(x)
      x = F.relu(x)
      x = self.step3(x)
      x = F.relu(x)
      x = torch.sigmoid(x)
      return (x)
net = Net()

transformer = transforms.ToTensor()
original_image = cv2.imread('group.jpg', cv2.IMREAD_GRAYSCALE)
face_cascade = cv2.CascadeClassifier(cv2.data.haarcascades + "haarcascade_frontalface_default.xml")
detected_faces = face_cascade.detectMultiScale(image=original_image, scaleFactor=1.05, minNeighbors=2)
tensor_collection_faces = []
tensor_collection_output = []
for (x, y, width, height) in detected_faces:
    im_pil = PIL.Image.fromarray(original_image)
    im_pil = im_pil.crop((x,y,x+width,y+height))
    im_pil = im_pil.resize((100,100))
    im_pil = PIL.ImageOps.grayscale(im_pil)
    curr_image_tens = (transformer(im_pil)).reshape((10000)).numpy()
    tensor_collection_faces.append(curr_image_tens)
    display(im_pil)
    tensor_collection_output.append([float(input('Expected Result: '))])
input_tensor = torch.tensor(tensor_collection_faces)
output_tensor = torch.tensor(tensor_collection_output)
loss_fn = torch.nn.BCELoss()
optimizer = torch.optim.Adam(net.parameters(), lr=1e-3)
for i in tqdm(range(500), desc='Training'):
    y_pred = net(input_tensor)
    loss = loss_fn(y_pred, output_tensor)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
print (net(input_tensor))

The output of the code:
Training: 100%|██████████| 500/500 [00:11<00:00, 44.40it/s]tensor([[0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000],
        [0.5000]], grad_fn=<SigmoidBackward>)


am running this code am getting an pytorch error inside the model i dont know where am going wrong. can anybody help me out please.
    import os
    import re
    import sys
    import argparse
    import time
    import pdb
    import random
    from pytorch_nndct.apis import torch_quantizer, dump_xmodel
    import torch
    import torchvision
    import torchvision.transforms as transforms
    from torchvision.models.resnet import resnet18
    import argparse
    import os
    import random
    import torch
    import torch.nn as nn
    import torch.nn.parallel
    import torch.backends.cudnn as cudnn
    import torch.optim as optim
    import torch.utils.data
    from torch.autograd import Variable
    import torch.nn.functional as F
    import skimage
    import skimage.io
    import skimage.transform
    import numpy as np
    import time
    import math
    from utils import preprocess
    from models import *
    from tqdm import tqdm

    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    parser = argparse.ArgumentParser()

    parser.add_argument(
        '--data_dir',
        default="/path/to/imagenet/",
        help='Data set directory, when quant_mode=calib, it is for calibration, while quant_mode=test it is for evaluation')
    parser.add_argument(
        '--model_dir',
        default="/path/to/trained_model/",
        help='Trained model file path. Download pretrained model from the following url and put it in model_dir specified path: https://download.pytorch.org/models/resnet18-5c106cde.pth'
    )
    parser.add_argument(
        '--subset_len',
        default=None,
        type=int,
        help='subset_len to evaluate model, using the whole validation dataset if it is not set')
    parser.add_argument(
        '--batch_size',
        default=32,
        type=int,
        help='input data batch size to evaluate model')
    parser.add_argument('--quant_mode',
        default='calib',
        choices=['float', 'calib', 'test'],
        help='quantization mode. 0: no quantization, evaluate float model, calib: quantize, test: evaluate quantized model')
    parser.add_argument('--fast_finetune',
        dest='fast_finetune',
        action='store_true',
        help='fast finetune model before calibration')
    parser.add_argument('--deploy',
        dest='deploy',
        action='store_true',
        help='export xmodel for deployment')
    args, _ = parser.parse_known_args()

    def load_data(train=True,
                  data_dir='dataset/imagenet',
                  batch_size=128,
                  subset_len=None,
                  sample_method='random',
                  distributed=False,
                  model_name='resnet18',
                  **kwargs):

      traindir = data_dir + '/train'
      valdir = data_dir + '/val'
      train_sampler = None
      normalize = transforms.Normalize(
          mean=[(0.485), (0.456), (0.406)], std=[(0.229), (0.224), (0.225)])
      if model_name == 'inception_v3':
        size = 88
        resize = 299
      else:
        size = 352
        resize = 296
      if train:
        dataset = torchvision.datasets.ImageFolder(
            traindir,
            transforms.Compose([
                transforms.RandomResizedCrop(size),
                transforms.RandomHorizontalFlip(),
                transforms.ToTensor(),
                normalize,
            ]))
        if subset_len:
          assert subset_len <= len(dataset)
          if sample_method == 'random':
            dataset = torch.utils.data.Subset(
                dataset, random.sample(range(0, len(dataset)), subset_len))
          else:
            dataset = torch.utils.data.Subset(dataset, list(range(subset_len)))
        if distributed:
          train_sampler = torch.utils.data.distributed.DistributedSampler(dataset)
        data_loader = torch.utils.data.DataLoader(
            dataset,
            batch_size=batch_size,
            shuffle=(train_sampler is None),
            sampler=train_sampler,
            **kwargs)
      else:

        dataset = torchvision.datasets.ImageFolder(
           valdir,
            transforms.Compose([
                transforms.Resize(resize),
                transforms.CenterCrop(size),
                transforms.ToTensor(),
                normalize,
            ]))
        if subset_len:
          assert subset_len <= len(dataset)
          if sample_method == 'random':
            dataset = torch.utils.data.Subset(
                dataset, random.sample(range(0, len(dataset)), subset_len))
          else:
            dataset = torch.utils.data.Subset(dataset, list(range(subset_len)))
        data_loader = torch.utils.data.DataLoader(
            dataset, batch_size=batch_size, shuffle=False, **kwargs)
      return data_loader, train_sampler

    class AverageMeter(object):
      """Computes and stores the average and current value"""

      def __init__(self, name, fmt=':f'):
        self.name = name
        self.fmt = fmt
        self.reset()

      def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

      def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

      def __str__(self):
        fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})'
        return fmtstr.format(**self.__dict__)

    def accuracy(output, target, topk=(1,)):
      """Computes the accuracy over the k top predictions
        for the specified values of k"""
      with torch.no_grad():
        maxk = max(topk)
        batch_size = target.size(0)

        _, pred = output.topk(maxk, 1, True, True)
        pred = pred.t()
        correct = pred.eq(target.view(1, -1).expand_as(pred))

        res = []
        for k in topk:
          correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)
          res.append(correct_k.mul_(100.0 / batch_size))
        return res

    def evaluate(model, val_loader,gt_loader, loss_fn):

      model.eval()
      model = model.to(device)
      top1 = AverageMeter('Acc@1', ':6.2f')
      top5 = AverageMeter('Acc@5', ':6.2f')
      total = 0
      Loss = 0

      images=val_loader

      for interaction,(labels,labels3) in tqdm(enumerate(gt_loader),total=2):
        print("gt lenght!!!!!!")
        print(labels.size())
      for interaction,(images,labels3) in tqdm(enumerate(val_loader),total=2):
        print(images.size())

      print("label!@!")
      print(labels.size())

      for i in range(0,1):

        zero=torch.zeros(1,32,88,88)
        zero=zero.to(device)
        images2 = images.to(device)
        labels2 = labels[:,0,:,:].to(device)
        print("ground truth dsize !")
        print(labels2.size())

        print("iamges sizxe")

        #print(images2[i].unsqueeze(0).size())
        outputs = model(images2[i].unsqueeze(0),images2[i].unsqueeze(0),zero)
        print("----------------------------------atom------------")
        outputs=outputs[:,0,:,:]
        print(outputs.size())
        print(labels2.size())

        loss = F.l1_loss(outputs,labels2)
        Loss += loss.item()

        return Loss
    def quantization(title='optimize',
                     model_name='',
                     file_path=''):

      data_dir = args.data_dir
      print(data_dir)
      quant_mode = args.quant_mode
      finetune = args.fast_finetune
      deploy = args.deploy

      batch_size = args.batch_size

      subset_len = args.subset_len
      print(subset_len)

      if quant_mode != 'test' and deploy:
        deploy = False
        print(r'Warning: Exporting xmodel needs to be done in quantization test mode, turn off it in this running!')
      if deploy and (batch_size != 1 or subset_len != 1):
        print(r'Warning: Exporting xmodel needs batch size to be 1 and only 1 iteration of inference, change them automatically!')
        batch_size = 1
        subset_len = 1
      batch_size = 1
      subset_len = 1

      model = stackhourglass(96)
      model = nn.DataParallel(model, device_ids=[0])
      torch.cuda.set_device(0)
      model.cuda()
      state_dict = torch.load("./models/pretrained_sceneflow.tar")
      model.load_state_dict(state_dict['state_dict'],strict=False)

      #input = torch.randn([batch_size, 3,352,352])
      #input1 = torch.randn([1,3, 352,352]).cuda(0)
      input2 = torch.randn([1,3, 352,352]).cuda(0)
      zero=torch.zeros(1,32,88,88)
      zero=zero.to(device)

      model.eval()
      #tempout=model(input2,input2,zero)
      if quant_mode == 'float':
        quant_model = model
      else:

        quantizer = torch_quantizer(
            quant_mode, model.module,(input2,input2,zero), device=device)

        quant_model = quantizer.quant_model

      print("quantization complete")

      loss_fn = torch.nn.CrossEntropyLoss().to(device)

      val_loader, _ = load_data(
          subset_len=subset_len,
          train=False,
          batch_size=batch_size,
          sample_method='random',
          data_dir='./calib',
          model_name=model_name)
      gt_loader, _ = load_data(
          subset_len=subset_len,
          train=False,
          batch_size=batch_size,
          sample_method='random',
          data_dir='./calib/labels',
          model_name='inception_v3')

      # fast finetune model or load finetuned parameter before test
      if finetune == True:
          ft_loader, _ = load_data(
              subset_len=1024,
              train=False,
              batch_size=batch_size,
              sample_method=None,
              data_dir=data_dir,
              model_name=model_name)
          if quant_mode == 'calib':
            quantizer.fast_finetune(evaluate, (quant_model, ft_loader, loss_fn))
          elif quant_mode == 'test':
            quantizer.load_ft_param()

      acc_org1 = 0.0
      acc_org5 = 0.0
      loss_org = 0.0

      loss_gen = evaluate(quant_model, val_loader,gt_loader, loss_fn)

      print('loss: %g' % (loss_gen))

      if quant_mode == 'calib':
        quantizer.export_quant_config()
      if deploy:
        print("inside deploy")
        quantizer.export_xmodel(deploy_check=False)

      print("deploy")

    if __name__ == '__main__':

      model_name = 'psmnet'
      file_path = os.path.join(args.model_dir, model_name + '.pth')
      file_path="pretrained_sceneflow.tar"
      feature_test = ' float model evaluation'
      if args.quant_mode != 'float':
        feature_test = ' quantization'

        args.optimize = 1
        feature_test += ' with optimization'
      else:
        feature_test = ' float model evaluation'
      title = model_name + feature_test

      print("-------- Start {} test ".format(model_name))

      # calibration or evaluation
      quantization(
          title=title,
          model_name=model_name,
          file_path=file_path)

      print("-------- End of {} test ".format(model_name))

Am getting the following error when i run it
quantization complete
0%|                                                                                | 0/2 [00:00<?, ?it/s]gt lenght!!!!!!
torch.Size([1, 3, 88, 88])
50%|████████████████████████████████████████████████████████████████████████████████████                                                                                | 1/2 [00:00<00:00, 161.34it/s]
0%|                                                                                | 0/2 [00:00<?, ?it/s]torch.Size([1, 3, 352, 352])
50%|████████████████████████████████████████████████████████████████████████████████████▌                                                                                | 1/2 [00:00<00:00, 82.86it/s]
label!@!
torch.Size([1, 3, 88, 88])
ground truth dsize !
torch.Size([1, 88, 88])
iamges sizxe
Traceback (most recent call last):
File "psm_quant_3.py", line 352, in
file_path=file_path)
File "psm_quant_3.py", line 314, in quantization
  loss_gen = evaluate(quant_model, val_loader,gt_loader, loss_fn)
  File "psm_quant_3.py", line 213, in evaluate
    outputs = model(images2[i].unsqueeze(0),images2[i].unsqueeze(0),zero)
  File "/opt/vitis_ai/conda/envs/vitis-ai-pytorch/lib/python3.6/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  **File "quantize_result/PSMNet.py", line 28, in forward
    self.output_module_5[:,:,:] = self.output_module_6
IndexError: too many indices for tensor of dimension 2**


title: Suboptimal convergence in PyTorch compared to TensorFlow when using Adam optimizer

body:
My program for training a model in PyTorch converges worse than the TensorFlow implementation. When I switch to SGD instead of Adam, the losses are identical. With Adam, the losses are different starting at the very first epoch. I believe I'm using the same settings in both programs. Any thoughts on how to debug this would be helpful!
Losses using SGD
PyTorch
0.1504615843296051
0.10858417302370071
0.08603279292583466

TensorFlow
0.15046157
0.108584
0.08603277

Losses using Adam
PyTorch
0.0031117501202970743
0.0020642257295548916
0.0019268309697508812
0.0016333406092599034
0.0017334128497168422
0.0014430736191570759
0.0010424457723274827
0.0012145100627094507
0.0011195113183930516
0.0009501167223788798
0.0009987876983359456
0.0007953296881169081
0.00075263757025823
0.0008374055614694953
0.000735406531020999

TensorFlow:
0.0036667113
0.0032563617
0.0021536187
0.0015266595
0.0013580231
0.0013878695
0.0011856346
0.0011136091
0.00091276
0.000890126
0.00088381825
0.0007283067
0.00081382995
0.0006670901
0.00046282331

Adam optimizer settings
TF 1.15.3:
adam_optimizer = tf.train.AdamOptimizer(learning_rate=5e-5)

# default parameters from the documentation at https://github.com/tensorflow/tensorflow/blob/v1.15.0/tensorflow/python/training/adam.py#L32-L235:
# learning_rate=0.001, beta1=0.9, beta2=0.999, epsilon=1e-8, use_locking=False, name="Adam")

PyTorch
torch.optim.Adam(params=model.parameters(), lr=5e-5, betas=(0.9, 0.999), eps=1e-08, weight_decay=0.0)

Training

I loaded identical weights from file to initialize the two models.
I trained and tested on a single data sample, also loaded from file. I used 1000 iterations for training and 1 for test, batch size 1.

Prior debugging

As above, I used identical parameters and data
I ran a single forward-backward pass using the Adam optimizer and saved the data and gradients at each layer. I plotted the results. All looked the same and were within 1e-6 to 1e-10 of each other. The loss was also identical within rounding error.

Saving and loading the PyTorch model
def train(...):
    ...
    checkpoint = torch.load(checkpoint_file, map_location=device)
    model.load_state_dict(checkpoint['state_dict'])
    optimizer.load_state_dict(checkpoint['optimizer'])
    ...
    counter = 0
    while run:
            counter += 1
            if counter > 1000:
                break

            in = np.load("debug_data/in.npy")
            out1 = np.load("debug_data/out1.npy")
            out2 = np.load("debug_data/out2.npy")

            # adjust from TF
            in = in.squeeze(3)
            in = np.expand_dims(in, axis=0)
            ... do the same for out1 and out2

        in, out1, out2 = \
                torch.from_numpy(in).to(device), \
                torch.from_numpy(out1).to(device), \
                torch.from_numpy(out2).to(device)

        optimizer.zero_grad()
        out1_hat, out2_hat = model(in)

        train_loss = loss_fn(out1_hat, out1) + loss_fn(out2_hat, out2)
        train_loss.backward()

        optimizer.step()

    save_checkpoint({'state_dict': model.state_dict(),
                    'optimizer': optimizer.state_dict()},
                    latest_filename=latest_checkpoint_path)

Saving and loading the TensorFlow model
sess.run(tf.global_variables_initializer())
writer = tf.summary.FileWriter(my_path, graph=sess.graph)

restorer = tf.train.Saver(tf.global_variables(), write_version=tf.train.SaverDef.V2)
restorer.restore(sess, load_path)

saver = tf.train.Saver(tf.global_variables(), write_version=tf.train.SaverDef.V2)

counter = 0
while run:
    counter += 1
    if counter > 1000:
        break

    in = np.load("")
    out1 = np.load("")
    out2 = np.load("")
    out1 = out1[0, :, :, :]
    out1 = out1[:, :, :, np.newaxis]
    out2 = out2[0, :, :, :]
    out2 = out2[:, :, :, np.newaxis]
    in = in[0, :, :, :]
    in = in[:, :, :, np.newaxis]
    _, _loss = sess.run([optimizer, loss],
    feed_dict={in: in, out1: out1, out2: out2})

save_path = saver.save(sess, my_save_path, global_step=int(_global_step))

sess.close()
tf.reset_default_graph()


I am getting this error and I don’t know what to do could anyone resolve it ?
AttributeError: module ‘torch.nn’ has no attribute ‘module’
This is the code for which I am facing the above error
import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.module):
    def init(self):
        super().init()
        self.con1 = nn.Conv2d(1, 32, 5) # here 1=input, 32=output i.e. 32 conv-features, 5=kernel size
        self.con2 = nn.Conv2d(32, 64, 5)
        self.con3 = nn.Conv2d(64, 128, 5)
        x = torch.randn(50,50).view(-1,1,50,50)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 512)
        self.fc2 = nn.Linear(512, 2)
    def convs(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)),  (2,2))
        x = F.max_pool2d(F.relu(self.conv2(x)),  (2,2))
        x = F.max_pool2d(F.relu(self.conv3(x)),  (2,2))

        print(x[0].shape)

        if self._to_linear is None:
        self._to_linear = x[0].shape*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1,self._to_linear)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.softmax(x, dim=1)
net = Net()


title: set variable network layers based on parameters in pytorch

body:
I want to make the following network definition to a parametric one. The number of continuous and discrete columns varies for different data. I first pass the whole input data, which in this case is 110 dimensional , from a linear with a relu activation. The output of each categorical field of my data varies based on a previous one-hot encoding data transformation. I need to define a nn.Linear(110, number of encodings) for each of them.
class Generator(nn.Module):
  def __init__(self):
    super(Generator, self).__init__(110)
    self.lin1 = nn.Linear(110,110)
    self.lin_numerical = nn.Linear(110, 6)
    self.lin_cat_job = nn.Linear(110, 9)
    self.lin_cat_sex = nn.Linear(110, 2)
    self.lin_cat_incomeclass = nn.Linear(110, 7)

  def forward(self, x):
    x = torch.relu(self.lin1(x))
    x_numerical = f.leaky_relu(self.lin_numerical(x))

    x_cat1 = f.gumbel_softmax(self.lin_cat_job(x), tau=0.2)
    x_cat2 = f.gumbel_softmax(self.lin_cat_sex(x), tau=0.2)
    x_cat3 = f.gumbel_softmax(self.lin_cat_incomeclass(x), tau=0.2)

    x_final = torch.cat((x_numerical, x_cat1, x_cat2, x_cat3),1)
    return x_final

I have managed to change the init part, using discrete_columns input which is an ordereddict that has the name and number of one-hot-encoding of each categorical field of my data as key and values, and continuous_columns which is only a list with the names of the continuous columns. But I have no idea how to edit the forward part:
class Generator(nn.Module):
  def __init__(self, input_dim, continuous_columns, discrete_columns):
    super(Generator, self).__init__()
    self._input_dim = input_dim
    self._discrete_columns = discrete_columns
    self._num_continuous_columns = len(continuous_columns)

    self.lin1 = nn.Linear(self._input_dim, self._input_dim)
    self.lin_numerical = nn.Linear(self._input_dim, self._num_continuous_columns)

    for key, value in self._discrete_columns.items():
      setattr(self, "lin_cat_{}".format(key), nn.Linear(self._input_dim, value))

  def forward(self, x):
    x = torch.relu(self.lin1(x))
    x_numerical = f.leaky_relu(self.lin_numerical(x))
    ####
    This is the problematic part
    #####
    return x


title: Loss over pixels

body:
During backpropagation, will these cases have different effect:-

sum up loss over all pixels then backpropagate.
average loss over all pixels then backpropagate
backpropagate individuallyover all pixels.

My main doubts in regarding the numerical value but the effect all these would be having.


title: Esoteric tensor indexing operation

body:
I want to index a tensor like this:

  A = [[4,5],[6,7]], index = [[0,2],[2,1]] -(?)-> [[nan, nan, 6], [4, 7 ,5]].

Basically, the element in index is the column in the output array the corresponding element in A belongs to, while the row index is simply stacked based on the number of occurrences of the column number in index.

Can I convert this to numpy, do the transformation, and convert back to a tensor in the middle of my ML pipeline, as numpy has more sophisticated array indexing? Can I just do this using pytorch?


title: Should convolution with $\pm 1$ valued kernel be faster than with a regular kind?

body:
I apologize for a long question and if it seems a really strange thing to ask.

If I have an N by N input image and I convolve it with a kernel that consists of (+1,-1) values, should this operation be faster then if kernel had random numbers (say, from a standard normal distribution)? To me, it seems that during the convolution operation the multiplication of the image and kernel values leads to a simple sign change but it still counts as regular multiplication and hence does not really affect the operation count and overall complexity. However, my colleague tells me that I'm wrong and we should see an increase in speed, which makes me sort of doubt that I am implementing everything correctly.

So far I have tried using timing magic in Jupyter notebooks to measure the time per convolution with (+1,-1) kernels and with regular kinds. I saw no difference. My main doubt is: should there be any at all? Is there a way to speed this up for such a specific kernel?

from torch.nn import functional as f
import torch

kernel = torch.randn((255,3,3,3)).sign_().float()
image = torch.randn((1,3,224,224)).float()

%%timeit
f.conv2d(image, kernel, stride=1, padding=1,)

This gives me: 10 loops, best of 3: 78.3 ms per loop
while running without sign_() operation yields 78.9 per loop.

Should there be any significant difference?


title: Video Synthesis with Transfer Learning on Encoder

body:
I am running into a very odd error with Pytorch in which I am synthesizing video using the following model. I am trying to apply transfer learning to the encoder only. To do this I have frozen the weights of the the generator with requires_grad = False and have done the opposite for the encoder.

Here is my model:

autoencoder(
  (generator): VideoGenerator(
    (recurrent): GRUCell(10, 10)
    (main): Sequential(
      (0): ConvTranspose2d(64, 512, kernel_size=(4, 4), stride=(1, 1), bias=False)
      (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (2): ReLU(inplace)
      (3): ConvTranspose2d(512, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
      (4): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (5): ReLU(inplace)
      (6): ConvTranspose2d(256, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
      (7): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (8): ReLU(inplace)
      (9): ConvTranspose2d(128, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
      (10): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (11): ReLU(inplace)
      (12): ConvTranspose2d(64, 3, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
      (13): Tanh()
    )
  )
  (encoder): Sequential(
    (0): Conv2d(3, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (1): ReLU(inplace)
    (2): Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (3): ReLU(inplace)
    (4): Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (5): ReLU(inplace)
    (6): Conv2d(256, 512, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False)
    (7): ReLU(inplace)
    (8): Conv2d(512, 64, kernel_size=(4, 4), stride=(1, 1), bias=False)
  )
)

Here is the output when I cycle through the params.requires_grad

*** encoder ***
1 False
2 False
3 False
4 False
5 False
6 False
7 False
8 False
9 False
10 False
11 False
12 False
13 False
14 False
15 False
16 False
17 False
*** generator ***
18 True
19 True
20 True
21 True
22 True

But this gives a runtime error as follows:

RuntimeError                              Traceback (most recent call last)
<ipython-input-7-73fe2d39b929> in on_wl_clicked(b)
     87 #         print(model)
     88
---> 89         loss.backward()
     90
     91 #         show_state("AFTER BACKWARD STEP:", model)

/usr/lib/python3.7/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    100                 products. Defaults to ``False``.
    101         """
--> 102         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    103
    104     def register_hook(self, hook):

/usr/lib/python3.7/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---> 90         allow_unreachable=True)  # allow_unreachable flag
     91
     92

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Oddly, when I set the 0th tensor requires_grad = True (see below), it runs but fails to converge

*** encoder ***
1 True
2 False
3 False
4 False
5 False
6 False
7 False
8 False
9 False
10 False
11 False
12 False
13 False
14 False
15 False
16 False
17 False
*** generator ***
18 True
19 True
20 True
21 True
22 True

Any ideas of where this error is originating from or how to remediate it?


title: can someone help me solve this problem, look for many solutions but they have not worked for me

body:
error al entrenar
the images have a size of ([64, 3, 224, 224])
I tried to change the batch-size or image size but I still get errors

Epoch 1/30
----------
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-160-dbcdb17ea6ee> in <module>()
      1 epochs = 30
      2 net.to(device)
----> 3 net = train_model(net, criterion, optimizer, sched, epochs)

2 frames
<ipython-input-157-d34ea1683b12> in forward(self, x)
     12     x = self.pool(F.relu(self.conv1(x)))
     13     x = self.pool(F.relu(self.conv2(x)))
---> 14     x = x.view(x.size(0), 16 * 38 * 38)
     15     x = F.relu(self.fc1(x))
     16     x = F.relu(self.fc2(x))

RuntimeError: shape '[64, 23104]' is invalid for input of size 2876416


title: Is there a PyTorch tensor function that reduces dimension in regular patterns

body:
Sorry for the poor wording of the title. What I wanted to do is like this:

Matrix 1 is the original matrix, and matrix 2 is matrix 1 but with every even columns and rows taken out. Matrix 3 is matrix 1 but only has 1 (mod 3) columns and rows. Matrix 4 is the same, with 1 (mod 4) columns and rows. Matrix 5 has 1 (mod 2) columns and all rows.

Is there a PyTorch function that manipulates tensors in this way that is fast and can utilize the GPU? This is sort of like MaxPool2d, however I just need the first value and not the max. If there aren't any functions like that, is there a way to do it manually but still fast?


title: None input to Lstm encoder

body:
I am using LSTM encoder as part of natural language inference model. There are two encoders hwo their output forward to an encoder.
Now, I need one of the encoder to sometimes get as input a sentence and sometimes I want it to get as input nothing, I mean I want ignore its input because I want the classifier to create its output based on the second classifier only, is there any way to do so? I don't think I can just input None.
Thank you.


title: Call Statements return Different result

body:
In Python programming, which using Pytorch i got this error

  4 and >5 are printing the type of x to check the type of tensor, but returning different results

why does this happen?My Code of Execution

x = x.type(torch.int64)
print(x.type())
type(X)


title: Pytorch Dataloader not spliting data into batch

body:
I have dataset class like this:

class LoadDataset(Dataset):
    def __init__(self, data, label):
        self.data = data
        self.label = label
    def __len__(self):
        dlen = len(self.data)
        return dlen
    def __getitem__(self, index):
        return self.data, self.label

then i load my image dataset which have [485, 1, 32, 32] shape

train_dataset = LoadDataset(xtrain, ytrain)
print(len(train_dataset))
# output 485

then i load the data with DataLoader

train_loader = DataLoader(train_dataset, batch_size=32)

and then i iterate the data:

for epoch in range(num_epoch):
        for inputs, labels in train_loader:
            print(inputs.shape)

the output prints torch.Size([32, 485, 1, 32, 32]), it should be torch.Size([32, 1, 32, 32]),

Can anyone help me?


title: Have to run setup.py in order to see changes in directory

body: I have a flask API that has an endpoint called /train, which runs a deep learning training script for image classification. When I pass in a new/unrecognized image label, I add this to multiple files, and train the model from scratch. When I pass in a recognized image label, I am supposed to leverage my pre-trained model and only train on this new image. When I pass in a new image label, I see that I correctly train the model and add it to the necessary files. Passing the same call again, I expect to leverage this pre-trained model, only training on the new image. However, python does not seem to recognize the modified files unless I run "setup.py install", causing the system to train from scratch repeatedly. Is there any reason why python doesn't seem to pick up the modified files in the directory without this call?


title: How to prevent my model from outputting zero-vectors while training using one-hot encoded vectors?

body:
I have been training a model for a study on one-shot learning.
It has 19280 examples in the training dataset (basically the popular Omniglot dataset), and a 300-length vector for each data sample.
The model consists of the following architecture-
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1           [-1, 32, 50, 50]           1,600
       BatchNorm2d-2           [-1, 32, 50, 50]              64
              ReLU-3           [-1, 32, 50, 50]               0
            Conv2d-4           [-1, 32, 50, 50]           9,248
       BatchNorm2d-5           [-1, 32, 50, 50]              64
              ReLU-6           [-1, 32, 50, 50]               0
            Conv2d-7           [-1, 32, 50, 50]           9,248
       BatchNorm2d-8           [-1, 32, 50, 50]              64
              ReLU-9           [-1, 32, 50, 50]               0
           Conv2d-10           [-1, 64, 24, 24]          18,496
      BatchNorm2d-11           [-1, 64, 24, 24]             128
             ReLU-12           [-1, 64, 24, 24]               0
           Conv2d-13           [-1, 64, 24, 24]          36,928
      BatchNorm2d-14           [-1, 64, 24, 24]             128
             ReLU-15           [-1, 64, 24, 24]               0
           Conv2d-16          [-1, 256, 11, 11]         147,712
      BatchNorm2d-17          [-1, 256, 11, 11]             512
             ReLU-18          [-1, 256, 11, 11]               0
           Conv2d-19            [-1, 512, 5, 5]       1,180,160
      BatchNorm2d-20            [-1, 512, 5, 5]           1,024
             ReLU-21            [-1, 512, 5, 5]               0
           Conv2d-22           [-1, 1024, 2, 2]       4,719,616
      BatchNorm2d-23           [-1, 1024, 2, 2]           2,048
             ReLU-24           [-1, 1024, 2, 2]               0
           Linear-25                  [-1, 300]       1,229,100
================================================================
Total params: 7,356,140
Trainable params: 7,356,140
Non-trainable params: 0
----------------------------------------------------------------

Basically the input is a 105x105 image, single channel (grayscale). I have applied sigmoid on the output.
While I train the model, I use simple Mean-squared error. I use the Adam optimizer with the learning rate set to $10^{-5}$ , to help tuning.
The model gets struck at the same constant loss after 2 or 3 epochs.
On further investigating, I found out that the model generalizes and makes outputs a zero vector in each case. I am assuming it is stuck on the local minima, but how do I go about training my model successfully?
Also, the model architecture has been chosen randomly (not literally, but no particular logic behind the dimensionality retained at the end of each layer), so please advise me if you think there is any present irregularity that would better training after rectification, in terms of layers.
I would love to hear some tips :) .


title: pylint(not-callable) & pylint (no member) problem

body:
enter image description here
This problem occurred when I run the python file in terminal.


title: Unknown error for thresholds within audio classification

body:
I was just wondering if I could get some help with some code I'm implementing to preprocess audio files. I'm trying to split an audio file to separate the signal from background noise. The following is a line of code
that I'm utilising to determine if bird sounds are there or not.
def isBird(spec, raw_thresh=10., threshold=30.):
"""
Decide if given spectrum shows bird sounds or noise

Options:
- raw_thresh : float
  Threshold for the absolute value of signal before fancy processing
  Defaults to 10

- threshold : float
  Threshold value for separating real signal / noise. Defaults to 30.
  Changeable via command-line option (see above)
"""
## working copy
img = spec.copy()

### STEP 0: get rid of highest/lowest freq bins
#-- total of 128 bins, to integer vals due to JPEG format
img = img[20:100,:].astype(int).astype(np.float32)

bird, rthresh = False, 0. # if too weak, treat as no signal no matter what
if img.max() > raw_thresh: # absolute value of maximum signal
    ### STEP 1: normalize to [0,1] for processing
    img -= img.min()
    img /= img.max()

    ### STEP 2: Median blur
    img = cv2.medianBlur(img,5)

    ### STEP 3: Median threshold
    col_median = np.median(img, axis=0, keepdims=True)
    row_median = np.median(img, axis=1, keepdims=True)

    img[img < row_median * 3] = 0.
    img[img < col_median * 4] = 0.
    img[img > 0.] = 1.

    ### STEP 4: Morph Closing
    img = cv2.morphologyEx(img, cv2.MORPH_CLOSE, np.ones((5,5), np.float32))

    ### STEP 5: Count columns and rows with signal
    #-- (Note: We only use rows with the signal as a threshold -- time axis for fixed freqs)

    ##### row has signal?
    row_max = np.max(img, axis=1)
    row_max = ndimage.morphology.binary_dilation(row_max, iterations=2).astype(row_max.dtype)
    rthresh = row_max.sum()

    ### STEP 6: Apply threshold
    if rthresh >= threshold:
        bird = True

return bird, rthresh, img

However, on the implementation of this method in the following way, I receive the error.
lbt,true  = isBird(melspec[:,:int(melspec.shape[1]/8.)])
lbf,false = isBird(melspec[:,1080:1260])

The error:

ValueError                                Traceback (most recent call last)
 in ()
----> 1 lbt,true  = isBird(melspec[:,:int(melspec.shape[1]/8.)])
2 lbf,false = isBird(melspec[:,1080:1260])
ValueError: too many values to unpack (expected 2)
Would appreciate any support in diagnosing this problem?


title: Pytorch multiple gatherings

body:
torch.gather could help in the following case,
out[i][j][k] = input[index[i][j][k]][j][k]  # if dim == 0
out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1
out[i][j][k] = input[i][j][index[i][j][k]]  # if dim == 2

However, what we do when we have multiple index tensors?
source = torch.ones(dim1, dim2, dim3, dim4)
index1 = torch.ones(dim3, dim4)
index2 = torch.ones(dim3, dim4)

for i in range(dim3):
    for j in range(dim4):
        ret[i, j] = source[index1[i, j], index2[i, j],i, j]


title: Is there a way to calculate how different two 2d arrays are from one another?

body: I am attempting to make a sudoku solver using a CNN. I am wondering if there is a way I can  calculate the loss between  the output puzzle and the without using a fully connected layer? If yes how?


title: Classification of ResNet34 with SMOTE ten percent worse than without smote

body:
I trained two ResNets with Mel-Spectograms of speech data.
The first one I trained with unbalanced data of 25 % of class 1 and 75 % of class 2.
The accuracy with the first network was 57 %.
The second network I trained with balanced data due to oversampling with smote and undersampling as suggested.
Surprisingly the accuracy dropped from 57 to 47 % percent on the evaluation dataset.
Maybe I did something wrong.
At the beginning I have a list with tensors of length 22000 and each tensor has a shape of (128, 200)
At the end X has a shape of (22000, 1, 128, 200).
And y has a shape of (22000,)
    X = np.stack(X,axis=0).squeeze(X)
    nsamples, nx, ny = X.shape
    X = X.reshape((nsamples,nx*ny))
    y = np.array(y)
    over = SMOTE(sampling_strategy=0.9)
    under = RandomUnderSampler(sampling_strategy=1)
    steps = [('o', over), ('u', under)]
    pipeline = Pipeline(steps=steps)
    X,y = pipeline.fit_resample(X ,y)
    nSmSamples, nxny = X.shape
    X = X.reshape(nSmSamples,1, nx, ny)

Did I do something wrong? I'm not sure if my reshaping is correct.


title: How do you assign ressourses when using PyTorch on AWS Sagemaker?

body:
I wish to use AWS Sagemaker in to train a PyTorch model. I am wondering how do I assign resources to the task? if I had my own computer I would use:
device = torch.cuda("cuda0")

Is it the same for AWS Sagemaker


title: Extremely high error in the first epochs (Pytorch - image segmentation)

body:
Why is the error function of my model on the validation data extremely high in the first epochs when I set my net to eval.
If I use model.eval() the error is greater than 40-50k in the first 4-5 epochs and then drops rapidly to 3-4, but if I leave my net on model.train() the error is only 5-6.
def eval_model(DataLoader, model, criterion, device, withStat, withImage):
    model.eval()
    eval_epochen_loss = 0
    img = None
    n_eval = 0
    TP, TN, FP, FN = 0, 0, 0, 0
    stat = None

    for i, data in enumerate(DataLoader):
        dicoms, targets = data
        dicoms, targets = Variable(dicoms.to(device)), Variable(targets.to(device))

        assert targets.shape[1] - 1 == model.n_classes, \
            f'Network has been defined with {model.n_classes} output classes, ' \
            f'but loaded target have {targets.shape[1] - 1} channels. Please check the labeled data or adjust ' \
            f'the network classes. '

        preds = model(dicoms)
        loss = criterion(preds, targets[:, -1, :, :].long())

        eval_epochen_loss += loss.item()
        n_eval += dicoms.shape[0]

        if withStat:
            res = TP_TN_FP_FN_in_batch(targets[:, -1, :, :].cpu().detach().long(), preds.detach().cpu())
            TP += res[0]
            TN += res[1]
            FP += res[2]
            FN += res[3]

    if withStat:
        stat = Statistic(TP, TN, FP, FN)

    if withImage:
        img_np = np.array(draw_images((dicoms, targets[:, -1], preds), outline_bool=True)).transpose(0, 3, 1, 2)
        img = torch.from_numpy(img_np)

    return eval_epochen_loss / n_eval, stat, img


title: The parameter doesn't updated throughout optimization

body:
I want to optimize the parameters vector, gamma, and v using pytorch or tensorflow optimization algorithms like Adam. The code below is my cost function and I want to minimize it with respect to above-mentioned three parameters. Unfortunately, the gamma parameter doesn't change and update at all!
I think it is for utilizing Sign function, because when I change it with softsign or tanh functions, it will get updated. However, is it very essential for me to preserve the precision and strict clustering at point zero, and simultaneously make gamma to update. Could anybody please help me with the problem?
def objective_function(self, x):
    nvector = self.vector / torch.norm(self.vector)
    alpha = torch.matmul(x, nvector) + self.gamma
    q1 = torch.sign(alpha)
    yh= q1 * self.v
    error = x - torch.matmul(yh, nvector.T)
    log_error = torch.log2( torch.mean( torch.square( torch.norm(error) ) ) )
    return log_error


title: Multifeature RNN in Pytorch with multi ouputs

body:
I'm currently trying to figure out the basis of RNN using Pytorch and I understand the basics of e.g. LSTM cells theoretically. I'm having some trouble understanding RNN(or LSTM) In Pytorch.
I can't seem to figure out how to prepare the data..
I have a data set with many sequences of different length.
Each sequence have 3 attributes (corresponding to Latitude, longitude and speed) looking like this:
array([[ 6.92456324e+01, -5.35513514e+01,  2.66666667e-01],
   [ 6.92456324e+01, -5.35513514e+01,  2.66666667e-01],
   [ 6.92456650e+01, -5.35514333e+01,  0.00000000e+00],
   [ 6.92456733e+01, -5.35514242e+01,  0.00000000e+00],
   ...,...,...])

I then want to train a RNN to predict the latitude and longitude (two first features) different  steps into the future.
I.e. for the first row, I want to predict LAT and LON in step 2, step 11 and the last step. For the second row  I want to predict LAT and LON in step 3, step 12 and the last step ect.
I would like to try this with both expanding and sliding back testing ( I think it is called) to see the difference.
I have tried to look around, but most often, the focus is on making the network to me..


title: Rising "sub-losses" when optimizing sum of losses in Neural Network with PyTorch and Adam

body:
I am optimizing the ELBO as part of the variational inference using a neural network for a dynamic topic model.
I am minimizing the sum of three losses (lower plots in the photo). While the loss decreases overall (top plot), the two smaller losses (on the right) increase during training. This behavior depends seems to depend on the optimizer and learning rate that I use. Only Adam seems to give interpretable results in the end, but always shows this behavior no matter how I set parameters for Adam. I have tried dropout but it does not completely solve the issue.
How can I avoid this and make sure that all losses decrease? Any ideas?


title: RNN not learning

body:
No matter how long i train it, the accuracy is still pretty low and the loss isn't decreasing.
input_dim = len(vocab)
embedding_dim: i tried 100~500, not much improved
hidden_dim = 512
output_dim = 6
Every "inputs" to forward() is something like the tensor below, each element being the index of the word in the vocabulary.
tensor([[11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831,
         11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831,
         11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831,
         11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831,
         11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831,
         11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831, 11831,
         11831, 11831, 11831, 11831, 11831, 11831,  5010,  3771,  7949,  2125,
          8461,  1170,  5010,  3771,  7949,  2125,     0,  4937,  4939,  4281]],
       device='cuda:0')

Is there anything wrong in the model?

    class RNN(nn.Module):
            def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim):
                super(RNN, self).__init__()

            self.hidden_dim = hidden_dim

            self.embedding = nn.Embedding(input_dim, embedding_dim)
            self.rnn = nn.RNN(embedding_dim, hidden_dim, batch_first=True)
            self.W = nn.Linear(hidden_dim, output_dim)

            self.softmax = nn.LogSoftmax(dim=1)
            self.loss = nn.NLLLoss()

        def compute_Loss(self, predicted_vector, gold_label):
            return self.loss(predicted_vector, gold_label)

        def hidden0(self, inputs):
            batch_dim = inputs.size(0)
            h0 = torch.zeros(1, batch_dim, self.hidden_dim)
            return h0

        def forward(self, inputs):

            hidden = self.hidden0(inputs).to(get_device())

            inputs = inputs.reshape(-1)
            #print(inputs)

            embed = self.embedding(inputs).unsqueeze(0)

            out, hn = self.rnn(embed, hidden)
            output = out.view(-1, self.hidden_dim)

            z = self.W(output)

            predicted_vector = self.softmax(z)

            predicted_vector = predicted_vector[-1].view(1,6)

            return predicted_vector


title: Build KNN graph over some subset of Node features

body:
I have a point cloud that I want to use a graph neural network on. Each point in the point cloud is characterised by its positional coordinates as well as it's color. So a single node is (X, Y, Z, C).
Now I want to apply an Edge Convolution on this (as described in the DGL Edge-Conv example, and to do it I should build a Nearest Neighbors graph on (X, Y, Z) (And not on C), then use all the 4 properties as features for my neural network.
What would be a clean and efficient way to do this? (I have a lot of data so I want to batch and collate well)


title: Defining my dataset in Pytorch before defining a train & test nn

body:
So I have an assignment and I think I understand everything that's required, except for the initialisation of my data. I have a folder structure with lots of directories, each containing opseq codes. There is a Benign folder with 1000 samples and spread across all the other folders is another 1000 samples. How do I set up the data in Pytorch so that I have one dataset with the Opseq codes labelled as either Malware or Benign?
The code that I have been given to start with seems to just print a list of all the Opseq codes with a 1 appended to them if they don't reside in the Benign folder and a 0 appended to them if they do, but how do I consolidate this into one dataset with two labels, with each Opseq code being assigned one of the two labels?
import torch
import os
import math
import torch
import torch.nn as nn
from torch.autograd import Variable
import torch.optim as optim
from torch.utils.data import Dataset

import pdb
#import test_cases

def read_file(filename):
    opcode_seq = []
    opcode_count = 0
    with open(filename, mode='rt', encoding='utf8') as f:
        content = f.readlines()
    for line in content:
        for c in range(0,len(line) - 1,2):
            #print(line[c:(c+2)],int(line[c:(c+2)], 16))
            opcode_seq.append(int(line[c:(c+2)], 16) + 1) # add one here so that the zero'th embedding is reserved for 'blank' i.e. no instruction whatsoever not even no-op
            opcode_count += 1
            # to save training time we only read
            # the first opt.max_opcode_seq_len opcodes of each file
            if opcode_count >= max_opcode_seq_len:
                return opcode_seq

    #pad with zeros between methods
    if opcode_count < max_opcode_seq_len:
        for i in range (opcode_count+1, max_opcode_seq_len+1):
            opcode_seq.append(0)
    return opcode_seq

def read_dataset(dir, verbose = True):
    malware = []
    clean = []
    min_file_len = min_opcode_seq_len #ignore opcode seq files shorter than this
    for root, dirs, files in os.walk(dir, topdown=True):
        for name in files:
            if os.path.split(root)[-1] == 'Benign':
                if (name.endswith('.b') or name.endswith('.opseq')) and (not name.startswith('._')):
                    if verbose:
                        print('0',os.path.join(root,name))
                    #labels.append(1)
                    tmp = read_file(os.path.join(root,name))
                    if len(tmp) >= min_file_len:
                        clean.append(tmp)
            else:
                if verbose:
                    print('1',os.path.join(root,name))
                if name.endswith('.opseq') and (not name.startswith('._')):
                    #labels.append(0)
                    tmp = read_file(os.path.join(root,name))
                    if len(tmp) >= min_file_len:
                        malware.append(tmp)
    return malware,clean

max_opcode_seq_len=8192
min_opcode_seq_len=32
dataset_root = './malwarecw'
malware,clean = read_dataset(dataset_root)


title: Efficient pytorch broadcasting not found

body:
I have the following code snippet in my implemenatation. There is a nested for loop with 3 loops. In the main code the 3D coordinates of the original system is stacked as a 1D vector of constinous stacking of points as for a point with coordinate (x,y,z) a sample cells will look like
Predictions =[...x,y,z,...]

whereas for my calulation I need reshaped_prediction vector as a 2D matrix with prediction_reshaped[i][0]=x, prediction_reshaped[i][1]=y prediction_reshaped[i][2]=z
where i is any sample row in the matrix prediction_reshaped.
The following code shows the logic
prediction_reshaped=torch.zeros([batch,num_node,dimesion])
    for i in range(batch):
        for j in range(num_node):
            for k in range(dimesion):
                prediction_reshaped[i][j][k]=prediction[i][3*j+k]

is their any efficient broadcasting to avoid these three nested loop? it is slowing down my code. torch.reshape does not suit my purpose.
The code is implemented using pytorch with all matrices as pytorch tensor but any numpy solution will also help.


title: PyTorch + PySyft Training Teacher Models Enumerate(trainloader) issue

body:
I have been following along this article: https://blog.openmined.org/maintaining-privacy-in-medical-data-with-differential-privacy/
I am running into an issue when training the teacher models. Specifically on the "for batch_idx, (data, target) in enumerate(trainloader):" as seen in the code snippet below.
def train(n_epochs, trainloader, validloader, model, optimizer, criterion, use_cuda, save_path= None, is_not_teacher=False):
    """returns trained model"""
    # # initialize tracker for minimum validation loss
    valid_loss_min = np.Inf

    for epoch in range(1, n_epochs+1):
        # initialize variables to monitor training and validation loss
        train_loss = 0.0
        valid_loss = 0.0
        train_correct = 0.0
        train_total = 0.0
        valid_correct =0.0
        valid_total = 0.0
        # train the model #
        model.train()
        print(trainloader)
        for batch_idx, (data, target) in enumerate(trainloader):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            # initialize weights to zero
            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output, target)
            loss.backward()
            optimizer.step()
            train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

            # convert output probabilities to predicted class
            pred = output.data.max(1, keepdim=True)[1]
            # compare predictions to true label
            train_correct += np.sum(np.squeeze(pred.eq(target.data.view_as(pred))).cpu().numpy())
            train_total += data.size(0)
            train_acc = 100. * train_correct / train_total

The main function call is shown below:
# Training teachers
teacher_models = []
i = 1
for trainloader, validloader in zip(trainloaders, validloaders):
  print(" Training Teacher {}".format(i))
  teacher_model = train(epochs, trainloader, validloader, model, optimizer, criterion, True)
  teacher_models.append(teacher_model)
  i+=1
  print("="*40)

I believe "enumerate(trainloader)" to be the issue because when placed higher in the function, the same behavior arises. That behavior being the python script being started over. I will show the unmodified conda prompt output + error below.
(fedLearn) C:\workDir\PySyft>python main.py cpu

Number of Classes:  2 425 203 118

 Image Size torch.Size([16, 3, 224, 224])  Image Size torch.Size([3, 224, 224])

Number of TEACHER trainloaders and validloaders: 5 5

Number of STUDENT trainloaders and validloaders: 6 2

 Training Teacher 1 <torch.utils.data.dataloader.DataLoader object at 0x000001E826208748> cpu

Number of Classes:  2 425 203 118

 Image Size torch.Size([16, 3, 224, 224])  Image Size torch.Size([3, 224, 224])

Number of TEACHER trainloaders and validloaders: 5 5

Number of STUDENT trainloaders and validloaders: 6 2

 Training Teacher 1 <torch.utils.data.dataloader.DataLoader object at 0x000001A921A3EEC8> Traceback (most recent call last): Traceback (most recent call last):   File "<string>", line 1, in <module>   File "main.py", line 118, in <module>
    teacher_model = train(epochs, trainloader, validloader, model, optimizer, criterion, True)   File "C:\workDir\PySyft\my_classes.py", line 157, in train
    for batch_idx, (data, target) in enumerate(trainloader):

Again, all source code is available at https://blog.openmined.org/maintaining-privacy-in-medical-data-with-differential-privacy/. The only modification I made is fixing a typo in the teacher_dataloaders function :
def teacher_dataloaders(transet=trainset,


title: How to use multiprocessing to run two pre-trained PyTorch models simultaneously?

body:
I have two pre-trained models that I want to get predictions from using the same dataset.
Is it possible to parallelize the process using multiprocessing.
The code I'm working on at the moment:
from torch.multiprocessing import Pool, Process, set_start_method

set_start_method('spawn', force=True)

output1parallel = None
output2parallel = None

def getOutput1():
  print("IM HERE 1")
  global loader, resnet18, output1parallel
  for i,batch in enumerate(loader):
    currentBatch = batch.cuda()
    resnet18 = resnet18.cuda()
    output1parallel = resnet18(currentBatch).cpu()

    del currentBatch
def getOutput2():
  print("IM HERE 2")
  global loader, densenet,output, output2parallel
  for i,batch in enumerate(loader):
    currentBatch = batch.cuda()
    densenet = densenet.cuda()
    output2parallel = densenet(currentBatch).cpu()
    del currentBatch

start = time.time()
p1 = Process(target = getOutput1 )
p2 = Process(target = getOutput2 )

p1.start()
p2.start()

p1.join()
p2.join()

print("Time for parallel implementation: {}".format(time.time() - start))

The functions are not called here.


title: Azure MySQL username with @ sign breaking connection string as two @ signs

body:
Having some issues porting app over to Azure due to the way that their managed mysql service has the users setup with an @ in them.
This means that when i try and update my DSN in an environment variable for an pytorch based container it breaks as it hits the first @ sign and falls over.
Had to previously done similar with a postgres connection and used %40 successfully but that doesn't work in this case.
Here's the env variable using:
mysqlusernameun@mysqlhostname-eastus:mypassword@tcp(mysqlhostname-eastus.mysql.database.azure.com)/database_name?charset=utf8&collation=utf8_unicode_ci&parseTime=true
Error in the log is:
Error opening DB connection Error 9002: The server name you tried cannot be found: mysqlusernameun


title: Chose rows of 3d Tensor based on some repeated indices. Tricky Slicing

body:
So this is a tricky bit of Tensor slicing I'm trying.
I have a tensor A which is 3d
>> A.shape
torch.Size([60, 10, 16])

So this tensor is composed of 5 different data samples, where in dim=0 we have cuts at split_ids = [10, 14, 10, 12, 14] i.e first 10 elements belong to sample1; next 14 belong to sample2 & so on.. I can split the tensor in such a way:
>> torch.split(A, split_ids, dim=0)
(tensor([[[-0.3888, -...Backward>), tensor([[[ 2.6473e-0...Backward>), tensor([[[ 1.1621,  ...Backward>), tensor([[[ 0.1953, -...Backward>), tensor([[[ 8.1993e-0...Backward>))

This comprises of tuple of 5 elements (or 5 tensors), of shapes Size(10,10,16);Size(14,10,16); and so on for the splits we had.
Now, comes the tricky part - I have another indices mapping that I have derived some previous processing for each of these individual splits. Its a list of 1d tensors like this:
>> reverse_map
[tensor([1, 2, 2, 1, ...='cuda:0'), tensor([ 7,  7,  9, ...='cuda:0'), tensor([7, 7, 4, 3, ...='cuda:0'), tensor([ 9,  4,  9, ...='cuda:0'), tensor([ 0,  0,  0, ...='cuda:0')]
>> reverse_map[0]
tensor([1, 2, 2, 1, 1, 2, 0, 1, 2, 0, 0, 1, 6, 1, 7, 1, 2, 7, 4, 5, 3, 4, 9, 7,
        7, 3, 8, 7, 7, 7], device='cuda:0')

So I need to basically use these indices and pull the count of these indices from the above split tensors i.e For tensor.Size(10,10,16) I need to pull [0, 0:3, :] which is for index 0 in dim=0, I need to pull 0:3 in dim=1 because there are 3 0's in the indexing. Then at index 1 I need to pull first 4 vectors since there are 4 1's.. and so on.
Whats the best way to do this ? Does scatter_() help here ?


title: What is the link between the GFLOPs value and the GPU RAM value?

body: I have a code that measures the complexity value according to the input picture size of my Mask RCNN model using ResNet50 backbone. When the input size is 800x800, the Flops value is calculated as 180.68 GFLOPs and Params: 43.75M. When I select the input size larger than 800px I get an Out of Memory error (GPU Ram 8 GB). Params value is always fixed but I get this error because GFLOPs value is getting bigger. Can anyone explain the link between the RAM GB value and GFLOPs? Thank you...


title: Zeroing the diagonal of a matrix by multiplying by (1-I)

body:
I have a tensor, lets say like this:
tensor([[2.7183, 0.4005, 2.7183, 0.5236],
    [0.4005, 2.7183, 0.4004, 1.3469],
    [2.7183, 0.4004, 2.7183, 0.5239],
    [0.5236, 1.3469, 0.5239, 2.7183]])

And I want to zero its main diagonal by multiplying it by (1-I), meaning by 1 minus the
identity matrix.
How can I do this in pytorch?
Result of the example should be:
    tensor([[0.0000, 0.4005, 2.7183, 0.5236],
    [0.4005, 0.0000, 0.4004, 1.3469],
    [2.7183, 0.4004, 0.0000, 0.5239],
    [0.5236, 1.3469, 0.5239, 0.0000]])

I'm looking for a general case solution and not specific to the example I gave.
Thanks!


title: How to append predicted data from test loader every other one

body:
I want to appended every other predicted images and real one in CNN code, but I am not sure how to implement it.
The code is as below:
test_loader = DataLoader(test_data, batch_size=batch_size, shuffle=False,
                         num_workers=num_workers, pin_memory=True)
    test_pred=[]
    test_real=[]
    model.eval()
    with torch.no_grad():
        for data in test_loader:
            x_test_batch, y_test_batch = data[0].to(device,
                dtype=torch.float), data[1].to(device, dtype=torch.float)

            y_test_pred = model(x_test_batch)
            mse_val_loss = criterion(y_test_batch, y_test_pred, x_test_batch, mse)
            mae_val_loss = criterion(y_test_batch, y_test_pred, x_test_batch, l1loss)

            mse_val_losses.append(mse_val_loss.item())
            mae_val_losses.append(mae_val_loss.item())
            N_test.append(len(x_test_batch))

             test_pred.append(y_test_pred[::2])
             test_real.append(y_test_batch[::2])


title: make latent vector with 2 methods, are they same?

body:
First method uses numpy.random.normal and the second one uses from torch.distributions import MultivariateNormal, Normal, Independent, Uniform
Are they the same? (their shapes are different but my point is whether the data is from the same distribution)


title: My Automatic rotoscope tool gave out this error

body:
So I was about to use the automatic rotoscope tool that I made after following a tutorial on the internet when this error popped up.
 Traceback (most recent call last):
  File "C:\Users\ZeroO\Documents\Python Scripts\rotoboy\rotoboy_v001.py", line 20, in <module>
    fm.createMatte(sourcefile, fullPathMatteName, matteheight)
  File "C:\Users\ZeroO\Documents\Python Scripts\rotoboy\FindMattes.py", line 45, in createMatte
    inp = trf(img).unsqueeze(0)
  File "C:\Users\ZeroO\anaconda3\envs\rotoboy\lib\site-packages\torchvision\transforms\transforms.py", line 67, in __call__
    img = t(img)
  File "C:\Users\ZeroO\anaconda3\envs\rotoboy\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\ZeroO\anaconda3\envs\rotoboy\lib\site-packages\torchvision\transforms\transforms.py", line 226, in forward
    return F.normalize(tensor, self.mean, self.std, self.inplace)
  File "C:\Users\ZeroO\anaconda3\envs\rotoboy\lib\site-packages\torchvision\transforms\functional.py", line 284, in normalize
    tensor.sub_(mean).div_(std)
RuntimeError: The size of tensor a (4) must match the size of tensor b (3) at non-singleton dimension 0

And this is what my code looks like:
import FindMattes as fm
import tkinter as tk
from tkinter import filedialog
from tkinter import simpledialog as sd
from os import listdir

# Get the file directory
root = tk.Tk()
root.withdraw()
directoryName = filedialog.askdirectory(parent=root, initialdir="/", title = 'Select Directory Here')
matteheight = sd.askinteger("Set matte size", "Output matte verical resolution (256 recommended for quick tests)", parent=root,initialvalue=256)

listofflies = listdir(directoryName)

for currentfile in listofflies:
    sourcefile = directoryName + "/" + currentfile
    mainNameEnd = currentfile.find('.')
    nameForMatte = currentfile[:mainNameEnd] + "_matte" + currentfile[mainNameEnd:]
    fullPathMatteName = directoryName + "/" + nameForMatte
    fm.createMatte(sourcefile, fullPathMatteName, matteheight)
    print("Just Created: " + nameForMatte)

Any help fixing this?


title: attention calculation at matrix level

body:
I am confused with the calculation in self attention or attention only.
Lets talk about self-attention first and I have:
x -> [batch_size, query_len, embedding_size]

now query's shape = key's shape = value's shape in self-attention, so
query = key = value = [batch_size, query_len, embedding_size]

self-attention -> softmax((query x key)/ square_root(key_len)) x value

In coding, I am confused in which dimension to apply softmax and how to take care of multiplication of score with value ? I mean which dimension is multiplied with each other ?
def attention(self, query, key, value):
    N = query.shape[0] # batch size
    # context's shape -> [batch_size, 10, 768]
    query_len, key_len, value_len = query.shape[1], key.shape[1], value.shape[1]

    energy = torch.einsum("nqd,nkd->nqk", [query, key])
    # [b, 10, 768]* [b, 10, 768] -> [b, 10, 10]
    attention = torch.softmax(energy / (self.embed_dim ** (1/2)), dim= 2)

    out = torch.einsum("nql, nld-> nqd", [attention, value])

    return out, attention


title: Where can I find pretrained torch vision Faster-RCNN for ImageNet classes

body: I am trying to use torch vision API for faster RCNN and use an existing pretrained model trained on large number of general class labels such as ImageNet. But the official documentation page only provides pretrained models over small set of class labels.


title: How does one execute an individual patched module using the higher pytorch library?

body:
I was using the pytorch higher library and I wanted to compute intermediate activations of neural networks. I was particularly interested in the activations using the adapted weights (what I assume the repo calls "fast weights"). My solution seems very hacky and gives some sketchy coincidences that worry me (like sometimes the activations are similar when I would have expected them to not be similar at all). The code that does not work is:
def f(self, fmodel):
        modules = fmodel.named_children()
        out = X
        for (name, m) in modules:
            print(f'{(name, m)')
            out = m(out)
        return out

the one that does "work" is:
    def f(self, mdl1, mdl2, X,):
        modules = zip(mdl1.named_children(), mdl2.named_children())
        out1 = X
        out2 = X
        for (name1, m1), (name2, m2) in modules:
            print(f'{(name1, m1), (name2, m2)}')
            out1 = m1(out1)
            # m2.direct_submodule_call = False
            m1.load_state_dict(m2.state_dict())
            out2 = m1(out2)

but I get weird similarities that I don't think should exist:
out1
Out[5]:
tensor([[-0.1519, -0.6428,  1.3292,  ..., -0.2724, -0.6826,  0.0502],
        [-1.8627, -0.6212,  2.6815,  ..., -0.2642, -0.6025, -1.4473],
        [-6.3190, -0.5651,  6.2037,  ..., -0.2429, -0.3940, -5.3476],
        ...,
        [-6.4152, -0.5639,  6.2798,  ..., -0.2424, -0.3895, -5.4319],
        [-1.4413, -0.6265,  2.3484,  ..., -0.2662, -0.6223, -1.0785],
        [ 0.4883, -0.6508,  0.8232,  ..., -0.2755, -0.7125,  0.6105]],
       grad_fn=<AddmmBackward>)
out2
Out[6]:
tensor([[-0.1519, -0.5300,  1.3292,  ...,  0.2442, -0.7063,  0.0502],
        [-1.8628, -0.3043,  2.6815,  ...,  1.1868, -0.6691, -1.4473],
        [-6.3190,  0.2833,  6.2037,  ...,  3.6421, -0.5722, -5.3476],
        ...,
        [-6.4152,  0.2960,  6.2798,  ...,  3.6951, -0.5701, -5.4319],
        [-1.4413, -0.3599,  2.3484,  ...,  0.9547, -0.6782, -1.0785],
        [ 0.4883, -0.6144,  0.8232,  ..., -0.1085, -0.7202,  0.6105]],
       grad_fn=<AddmmBackward>)

or

(out1 == out2).any()
Out[3]: tensor(True)
(out1 == out2).all()
Out[4]: tensor(False)

anyone know how to fix this properly?
for full discussion see gitissue: https://github.com/facebookresearch/higher/issues/96  or https://www.reddit.com/r/pytorch/comments/lb7py6/how_does_one_execute_an_individual_patched_module/


title: Convert 8-bit image with values from 0 to 4 into 1-bit image with 5 channels (each channel represents a label) in python/MATLAB

body:
I have segmented/labeled image data that I would want to use as input data for a DL model. 5 labels are available in the images and the data-type is 8 bit. So, my files are saved as 8-bit images (actually ranging from 0-255), but the values for the labels have grayvalues 0, 1, 2, 3, and 4. I would want to input this in a DL model as an image with 5 labels. So, I aim to have 1-bit files with five channels, each channel representing a certain label. Do anybody have any tips for me how to convert my images to the five channels?
Thanks in advance!


title: Pytorch change format of tensors from BxWxH to B, N, 3

body:
I have a tensor A with the shape BxWxH (B=Batch size, W=Width, H=Height) and want to change it to a tensor B of shape BxNx3 (B=Batch size, N=Number of points=W*H).
Tensor A represents a depth map, e.g. tensor[0,1,2] => gives the depth value for the pixel (1,2) in batch 0.
Tensor B also represents a depth map but in a different format. Each point in tensor B has the following three dimensions: (x coord, y coord, depth value).
How can I transform tensor A into tensor B?


I am trying to run the custom yolo model on my data set in my local machine. I am following some reference code from the kaggle platform. Here first time I encounter the wandb frame work. while doing so I use the following to run the train.py  file in my jupyter lab.
!WANDB_MODE="dryrun" python train.py --img 640 --batch 16 --epochs 30 --data D:/Anil/Shawn_Research/Iamge_DataSet/VinBigData/New_Direct/vinbigdata.yaml --weights yolov5x.pt --cache

This work fine on the kaggle platform but in my local machine it shows following:
'WANDB_MODE' is not recognized as an internal or external command, operable program or batch file.

While reading the similar thread I realized I might making mistake related to path variable or Environment variable.
Even I tried to get solution from the official document but couldn't figure out.
Thanks in advance.


title: pytorch question about tensor shape. And how to reshape a tensor

body: When I do  print("action shape:, ", action.shape) for my tensor action I got (64,). It is the same as (1, 64)? And how do I reshape its size to (64,1)?


title: Pytorch crashes cuda on wrong line

body:
How to see which python line causes a cuda crash down the line in Pytorch, which executes asynchronous code outside of the GIL?
Here is a case where I had Pytorch crash cuda, running this code on this dataset and every run would crash with the debugger on a different python line, making it very difficult to debug.


title: 3D image registration with CNN in medical images

body:
Recently, I am studying 3D image registration in medical images. There are big 2 topics that I want to try, implicit and explicit image registration. I will register 5 images of different motion states to 1 image.
For implicit image registration, the input channel is 5 and the output channel is 1.
Let's say if I have 5 motion states, I want to register first 4 motion states images to the last motion state image. As I have the target image for the last bin separately to the input, and the last motion state image of the input is slightly different to the target image, I input 5 channels of input to the network, and trough the network, the network output 1 channel of output.
I use ResUNet for the network, and I have 65 dataset of 3D images. learning rate is 1e-2~1e-3 ( still finding the optimal learning rate) and the number of features is 32 or 64.
I use 2 loss functions which is pixelwise loss(L1) and pre-trained perceptual loss
The problem what I have is that, the output is not similar to the target, but the input
Here is image of the input, output and target with black lines
Here is my codes
Training part
def train(epoch, data_loader, model, perceptual_loss, pixel_loss, optimizer):
model.train()
epoch_loss = []
with tqdm(total=len(data_loader), desc=f'Epoch {epoch}/{args.epochs}', unit='img') as pbar:
    for i, (input_img, target, meta) in enumerate(data_loader):
        input_img, target = input_img.to(DEVICE), target.to(DEVICE)

        output = model(input_img)

        output_masked, target_masked = utils.empty_masking(output, target, DEVICE)
        loss = pixel_loss(output_masked, target_masked)
        output_reshaped = output_masked.permute(2, 0, 1, 3, 4).reshape(-1, 1, *args.image_size)
        target_reshaped = target_masked.permute(2, 0, 1, 3, 4).reshape(-1, 1, *args.image_size)
        if perceptual_loss is not None:
            loss += args.pt_balance * perceptual_loss.forward(output_reshaped, target_reshaped, normalize=True).mean()

        pbar.set_postfix(**{'loss (batch)': loss.item(), 'learning rate': optimizer.param_groups[0]['lr']})

        optimizer.zero_grad()
        loss.backward()
        nn.utils.clip_grad_value_(model.parameters(), 0.1)  # gradient clipping
        optimizer.step()

        pbar.update(input_img.shape[0])

        epoch_loss.append(loss.item())
        print('Epoch[{}]({}/{}): Loss: {:.4f}'.format(epoch, i, len(data_loader), loss.item()))

    avg_loss = np.mean(epoch_loss)
    print('Epoch {} complete. Training Loss: {:.4f}'.format(epoch, avg_loss))
return avg_loss

The method "utils.empty_masking()" controls not to update weights where black lines exist in the target image so that weights don't affected by zero values.
This is the code of ResUNet network.
class ResUNet(nn.Module):
def __init__(self, in_channels, out_channels, n_features, trilinear=True, kernel=3, pad=1):
    super(ResUNet, self).__init__()
    self.n_channels = in_channels
    self.n_classes = out_channels
    self.trilinear = trilinear
    factor = 2 if trilinear else 1

    self.inc = DoubleConv(in_channels, n_features, kernel=kernel, pad=pad)
    self.down1 = Down(n_features, n_features * 2, kernel=kernel, pad=pad) # 32 64
    self.down2 = Down(n_features * 2, n_features * 4, kernel=kernel, pad=pad) #64 128
    self.down3 = Down(n_features * 4, n_features * 8, kernel=kernel, pad=pad) # 128 256
    self.down4 = Down(n_features * 8, n_features * 16 // factor, kernel=kernel, pad=pad) # 256 512
    self.up1 = Up(n_features * 16, n_features * 8 // factor, trilinear, kernel=kernel, pad=pad) # 512 256
    self.up2 = Up(n_features * 8, n_features * 4 // factor, trilinear, kernel=kernel, pad=pad) # 256 128
    self.up3 = Up(n_features * 4, n_features * 2 // factor, trilinear, kernel=kernel, pad=pad) # 128 64
    self.up4 = Up(n_features * 2, n_features, trilinear, kernel=kernel, pad=pad) # 64 32
    self.outc = OutConv(n_features, out_channels) # 32 1

def forward(self, x):
    x1 = self.inc(x)
    x2 = self.down1(x1)
    x3 = self.down2(x2)
    x4 = self.down3(x3)
    x5 = self.down4(x4)
    x6 = self.up1(x5, x4)
    x7 = self.up2(x6, x3)
    x8 = self.up3(x7, x2)
    x9 = self.up4(x8, x1)
    x10 = self.outc(x9)
    logits = torch.clamp(x10, 0, 1)

    return logits

What should I do for registering images properly?
Output of the network should be similar to the target rather than input.
Thank you very much in advance :)


title: How to properly graph train/val on same graph in pytorch

body:
I have the following code that trains a model and stores logs in a results variable
import tqdm.notebook as tq
import sys

num_epochs = 10
results = {"train_loss": [], "val_loss": [], "train_acc": [], "val_acc": []}

for epoch in range(1, num_epochs+1):
  sys.stdout.write(f"---Epoch {epoch}/{num_epochs}: ")
  epoch_loss = {"train": [], "val": []}
  epoch_acc = {"train": [], "val": []}

  for phase in ['train', 'val']:
    if phase=="train":
      model.train(True)
    else:
      model.train(False)

    # most important thing I learned from this project was how to fix tqdm nastiness in colab
    for batch_idx, (x, y) in tq.tqdm(enumerate(dataloaders[phase]),
                                     total=len(dataloaders[phase]),
                                     leave=False):

      # put data to device and get output
      x, y = x.to(device), y.to(device)
      preds = model(x)

      # calc and log model loss
      batch_loss = criterion(preds, y)
      epoch_loss[phase].append(batch_loss.item())

      # calculate acc and extend to epoch_acc
      preds = torch.argmax(preds, dim=1)
      batch_acc = torch.sum(preds==y)/len(y)
      epoch_acc[phase].append(batch_acc)

      # zero the grad
      optimizer.zero_grad()

      # take a step if training mode is on
      if phase=="train":
        batch_loss.backward()
        optimizer.step()
        scheduler.step()

  # at the end of each epoch, calculate avg epoch train/val loss/accuracy
  train_loss = sum(epoch_loss["train"])/len(epoch_loss["train"])
  val_loss = sum(epoch_loss["val"])/len(epoch_loss["val"])
  train_acc = 100*sum(epoch_acc["train"])/len(epoch_acc["train"])
  val_acc = 100*sum(epoch_acc["val"])/len(epoch_acc["val"])

  # log losses and accs every epoch
  results['train_loss'].extend(epoch_loss['train'])
  results['train_acc'].extend(epoch_acc['train'])
  results['val_loss'].extend(epoch_loss['val'])
  results['val_acc'].extend(epoch_acc['val'])

  # and print it nicely
  sys.stdout.write("train_loss: {:.4f} train_acc: {:.2f}% ".format(train_loss, train_acc))
  sys.stdout.write("val_loss: {:.4f} val_acc: {:.2f}%\n".format(val_loss, val_acc))

I'm logging the avg accuracy and avg loss of every batch into separate training/validation loss/acc arrays. The problem is that I have more training batches so when I try to graph my training logs I get something like this:

Is there a workaround for this?


title: Transformation parameter as learning target

body:
Let's say, I am applyng random blur to torchvision MNIST image, and want to train the net to guess blur power. It looks natural to use standard transform, but in dataset class implementation transformations of data and target applyes independently:
    if self.transform is not None:
        img = self.transform(img)

    if self.target_transform is not None:
        target = self.target_transform(target)

So I can't see a way to pass blur value for insertion in target.
Well, I can rewrite dataset class, but may be there is more natural way to pass parameter from data transform to target_transform?


title: I try to implement vggnet, but it does not be trained well

body:
I am new at CNN.
I try to train vggnet.
class Net(nn.Module) :
def __init__(self) :
    super(Net, self).__init__()

    self.conv = nn.Sequential (
        #1
        #####
        nn.Conv2d(3,64,3, padding=1),  nn.ReLU(inplace=True),
        nn.Conv2d(64,64,3, padding=1),nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #2
        #####
        nn.Conv2d(64,128,3, padding=1), nn.ReLU(inplace=True),
        nn.Conv2d(128,128,3, padding=1),nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #####
        #3
        #####
        nn.Conv2d(128,256,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(256,256,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(256,256,3, padding=1),nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #4
        #####
        nn.Conv2d(256,512,3, padding=1), nn.ReLU(inplace=True),
        nn.Conv2d(512,512,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(512,512,3, padding=1), nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #5
        #####
        nn.Conv2d(512,512,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(512,512,3, padding=1),nn.ReLU(inplace=True),
        nn.Conv2d(512,512,3, padding=1),nn.ReLU(inplace=True),
        nn.MaxPool2d(2,2),
        #####
    )
    self.fc = nn.Sequential(
       nn.Linear(512 * 7 * 7, 4096), nn.ReLU(inplace=True), nn.Dropout(0.5),
       nn.Linear(4096, 4096), nn.ReLU(inplace=True), nn.Dropout(0.5),
       nn.Linear(4096, 1000)
    )

def forward(self, x):

    # diffrent depending on the model
    x = self.conv(x)
    # change dimension
    x = torch.flatten(x, 1)
    # same
    x = self.fc(x)

    return x

But the loss is near 6.9077.
After epoch 0, it rarely change.
Even if I change weight decay to 0 (not use L2 normalization), the loss is slightly change.
My optimizer and scheduler is
optimizer = torch.optim.SGD(net.parameters(), lr=0.1, weight_decay=5e-4)
scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, factor = 0.1 , patience= 2 ,mode='min')
What is the problem.
Sometimes, it print bytes but only got 0.
warnings.warn(str(msg))
That is related to my problem ?


title: Why the actor neural network is not learning?

body:
Good evening,
the code shown below represents the training function of an actor network. The idea is to use it in a cycle to control a system. All the control actions are generated by this
neural network and they should converge to the values proposed by the Cross Entropy Method, that minimizes a cost. The latter is approximated by a critic neural network that seems to work well.
While the system is runing, the Cross Entropy Method function returns convincing values but the actor continues to increase the control action, augmenting the error.
What could be the problem?
'''
Code
#Input data are normalized
def Actor_train(state_train, action_train, force_train, Actor_NN, Critic_NN, Model_NN, xdh_intro, Kh, xy_norm, Nh, learning_rate):
#action_mean_, action_std_ = action_norm
#fh_mean_, fh_std_ = fh_norm

Actor_NN_ = Actor_NN
Critic_NN_ = Critic_NN
Model_NN_ = Model_NN
Nh_ = Nh
learning_rate_ = learning_rate
xy_norm_ = xy_norm

x_mean_v_, x_std_v_, y_mean_v_, y_std_v_ = xy_norm_ #variables to normalize
state_not_norm_ = state_train*x_std_v_[0:2] + x_mean_v_[0:2] #denormalization

# normalized quantities
state_train_ = np.copy(state_train[0:Nh_,:])
action_train_ = Variable(torch.from_numpy(action_train[0:Nh_].squeeze(1)), requires_grad=True)
# action_train comes from a neural network
force_train_ = np.copy(force_train[0:Nh_])

Actor_NN_.to("cpu")  # putting the model into GPU (but not available now)
optimizer_A = torch.optim.Adam(Actor_NN_.get_parameters(), lr =learning_rate_)
errore_f = nn.MSELoss()

cache_loss_ = []
resultant_loss_ = []
iteration_ = 0

for j in range(0, Nh_-1):

    Actor_NN.train()
    optimizer_A.zero_grad()
    # Cross entropy method to estimate u minimizing the output of the critic network
    U_npiu1 = CEM_critic(state_not_norm_[j:j+1], 1, 5, 64, xy_norm_, Cost_norm, Model_NN,
                         critic, xdh_intro, Kh, num_ensembles_cem_= 5)[0]

    action_from_NN = torch.DoubleTensor(action_train_[j+1:j+2])
    action_from_CEM = torch.DoubleTensor(U_npiu1)[0]
    print("NN ", action_from_NN[0])
    print("CEM ", action_from_CEM)
    # Cost function
    Error_a = errore_f.forward(action_from_NN[0], action_from_CEM)
    Error_a.backward(retain_graph=True)                           # backprop
    optimizer_A.step()                             # updating the parameters
    Nh_loss_ = Error_a
    # torch.cuda.empty_cache()
    cache_loss_.append(Nh_loss_.item())
    iteration_ =+1

    if (j+1)%2 ==0:
        resultant_ = np.sum(cache_loss_)/iteration_
        resultant_loss_.append(resultant_)
        iteration_ = 0
        cache_loss_ =[]
        print("epoch:",j+1," Actor Training_loss:", resultant_)

plt.plot(resultant_loss_,'r')
plt.ylabel('Loss')
plt.xlabel('epochs')
plt.title('actor')
plt.grid(True)
plt.show()

print("Final training loss: ",resultant_loss_[-1])

return

'''


title: How to add an external library from CMake to a Gradle project (or vice-versa)?

body:
I am trying to link LibTorch (C++ PyTorch library) to a C++ codebase that uses Gradle as its build system. Unfortunately, it appears that the vast majority of documentation covers adding LibTorch to a CMake project, but not Gradle.
Hence, I am wondering if it is possible to create a parallel project using CMake, and import it as a library to my Gradle project.
For Android, this appears achievable using external native builds. I am wondering if it is possible to do this on Linux, and if not what the best course of action could be. Another option is to make a CMake configuration that will load all my Gradle projects.