title: Pytorch-Forecasting N-Beats model with SELU() activation function?

body:
I am working at timeseries forecasting, and I've using the PyTorch lib pytorch-forecasting lately. If you didn't know it, try it. It's great.
I am interested in SELU activation function for Self-Normalizing-Networks (SNNs, see, e.g., the docs). As I didn't find any N-Beats corrected to use SELU and its requirements (i.e. AlphaDropout, proper weights init), I made an implementation myself.
It would be great if any of you with experience with these concepts -NBeats architecture, pytorch-forecasting, or SELU()- could review whether everything is right in my implementation.
My implementation here: https://gist.github.com/pnmartinez/fef1f488497fa85a2cc1626af2a5b4bd


title: pytorch initialize two sub-modules with same weights?

body:
I was wondering if there's a typical way to initialize two parts of one network to same some initial values? Say I got two separate auto-encoders for both query and document, and I would like to initialize the weights of these two auto-encoders to same weights (not sharing the weights).
Thanks!


title: Reshape LSTM output to get desired output shape

body:
I have a noob question: I built a very simple LSTM which takes windows of time-series data as input, in the shape (batch_size, n_time_steps, n_features).
This is the LSTM:
class LSTM(nn.Module):
    def __init__(self, input_dim, hidden_dim, num_classes):
        super(LSTM, self).__init__()
        self.hidden_dim = hidden_dim
        self.lstm_layer = nn.LSTM(input_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, num_classes)

    def forward(self, x):
        out, _ = self.lstm_layer(x)
        x = self.fc(out.squeeze())
        x = nn.functional.softmax(x, dim=2)
        return x

So, I would like the output (x in the code) to be in the shape (batch_size, num_classes), because I would like to know the class for each window of data. However, right now the output shape is (batch_size, n_time_steps, num_classes). How can I fix this?


title: iteration and dataset length goes wrong. And data_time_interval is too large. How to solve these question?

body:
I have 4 GPUs and want to use Pytorch DDP for training. The data set I tested has 80 samples in train and 20 samples in val, and the set batch size is 16. I want to print the number of iterations i and the length of the training set during training as args.step_per_epoch = train_data.len(). Below is the code of train and val.
def train(model, local_rank):
    model_path = '/output/saved_model'
    step = 0
    nr_eval = 0
    dataset = VimeoDataset(mode = 'train')
    sampler = DistributedSampler(dataset)
    train_data = DataLoader(dataset, batch_size=args.batch_size, num_workers=4, pin_memory=True, drop_last=True, sampler=sampler)
    args.step_per_epoch = train_data.__len__()
    dataset_val = VimeoDataset(mode = 'val')
    val_data = DataLoader(dataset_val, batch_size=16, pin_memory=True, num_workers=4)
    evaluate(model, val_data, nr_eval, local_rank)
    epochlist.append(0)
    model.save_model(model_path, local_rank)
    print('training...')
    time_stamp = time.time()
    for epoch in range(args.epoch):
        sampler.set_epoch(epoch)
        for i, data in enumerate(train_data):
            data_time_interval = time.time() - time_stamp
            time_stamp = time.time()
            data_gpu, flow_gt = data
            data_gpu = data_gpu.to(device, non_blocking=True) / 255.
            flow_gt = flow_gt.to(device, non_blocking=True)
            imgs = data_gpu[:, :6]
            gt = data_gpu[:, 6:9]
            mul = np.cos(step / (args.epoch * args.step_per_epoch) * math.pi) * 0.5 + 0.5
            learning_rate = get_learning_rate(step)
            pred, merged_img, flow, loss_LPIPS, loss_flow, loss_cons, loss_ter, flow_mask = model.update(imgs, gt, learning_rate, mul, True, flow_gt)
            train_time_interval = time.time() - time_stamp
            time_stamp = time.time()
            if local_rank == 0:
                print('epoch:{} {}/{} time:{:.2f}+{:.2f} loss_LPIPS:{:.4e}'.format(epoch, i, args.step_per_epoch, data_time_interval, train_time_interval, loss_LPIPS))
            step += 1
        nr_eval += 1
        if nr_eval % 5 == 0:
            evaluate(model, val_data, step, local_rank)
            epochlist.append(nr_eval)
        model.save_model(model_path, local_rank)
        dist.barrier()

def evaluate(model, val_data, nr_eval, local_rank):
    psnr_list = []
    time_stamp = time.time()
    for i, data in enumerate(val_data):
        data_gpu, flow_gt = data
        data_gpu = data_gpu.to(device, non_blocking=True) / 255.
        flow_gt = flow_gt.to(device, non_blocking=True)
        imgs = data_gpu[:, :6]
        gt = data_gpu[:, 6:9]
        with torch.no_grad():
            pred, merged_img, flow, loss_LPIPS, loss_flow, loss_cons, loss_ter, flow_mask = model.update(imgs, gt, training=False)
        for j in range(gt.shape[0]):
            psnr = -10 * math.log10(torch.mean((gt[j] - pred[j]) * (gt[j] - pred[j])).cpu().data)
            psnr_list.append(psnr)

    eval_time_interval = time.time() - time_stamp
    if local_rank == 0:
        print('eval time: {}'.format(eval_time_interval))
        print('mean psnr: {}'.format(np.mean(psnr_list)))
        psnrlist.append(np.mean(psnr_list))

The following is part of task training log. You can see that the parameter values such as i and args.step_per_epoch are wrong, always 0/1. What is the cause and how to modify it. In addition, you can also see that data_time_interval is actually greater than train_time_interval. How to reduce data_time_interval to improve training efficiency.
eval time: 8.030341386795044
mean psnr: 24.4023611466035
training...
epoch:0 0/1 time:0.30+2.96 loss_LPIPS:3.4511e-01
epoch:1 0/1 time:0.93+0.41 loss_LPIPS:3.3067e-01
epoch:2 0/1 time:1.29+0.36 loss_LPIPS:3.3386e-01
epoch:3 0/1 time:4.61+0.37 loss_LPIPS:3.2475e-01
epoch:4 0/1 time:5.26+0.36 loss_LPIPS:3.1935e-01
eval time: 0.9092228412628174
mean psnr: 24.403575688421018
epoch:5 0/1 time:3.94+0.36 loss_LPIPS:3.5425e-01
epoch:6 0/1 time:4.75+0.36 loss_LPIPS:3.5130e-01
epoch:7 0/1 time:3.60+0.36 loss_LPIPS:3.2492e-01
epoch:8 0/1 time:1.40+0.37 loss_LPIPS:3.4967e-01
epoch:9 0/1 time:1.12+0.37 loss_LPIPS:3.4065e-01
...
...
epoch:90 0/1 time:7.11+0.38 loss_LPIPS:3.1828e-01
epoch:91 0/1 time:2.66+0.37 loss_LPIPS:2.7712e-01
epoch:92 0/1 time:6.57+0.36 loss_LPIPS:3.0946e-01
epoch:93 0/1 time:5.59+0.36 loss_LPIPS:2.5663e-01
epoch:94 0/1 time:0.76+0.36 loss_LPIPS:2.8386e-01
eval time: 0.8664124011993408
mean psnr: 24.744649141015156
epoch:95 0/1 time:1.89+0.35 loss_LPIPS:2.8509e-01
epoch:96 0/1 time:2.24+0.36 loss_LPIPS:3.0353e-01
epoch:97 0/1 time:1.49+0.37 loss_LPIPS:3.0354e-01
epoch:98 0/1 time:1.34+0.36 loss_LPIPS:2.9313e-01
epoch:99 0/1 time:1.27+0.36 loss_LPIPS:2.9234e-01
eval time: 0.8701093196868896
mean psnr: 24.784283493153136


title: How can I get the name of a chosen optimizer in PyTorch?

body:
I'm trying different optimizers to compare performance and as part of this, I'm logging the results using TensorBoard.
I've set up SummaryWriter output to include details such as the model used and learning rate.
writer = SummaryWriter(comment=f"_{base_model}_{lr}")

I would like it to also dynamically include the name of the optimizer, but there doesn't seem to be a built-in way to get the name of the selected optimizer.
Printing the optimizer does return its name. However, it's bundled with parameter information so not ideal for logging.
>>> print(optimizer)

Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    lr: 0.001
    weight_decay: 0
)

Is there a way I can get only the name of the optimizer without having to parse this output?


title: Pytorch BCE loss not decreasing for word sense disambiguation task

body:
I am performing word sense disambiguation and have created my own vocabulary of the top 300k most common English words. My model is very simple where each word in the sentences (their respective index value) is passed through an embedding layer which embeds the word and average the resulting embedding. The averaged embedding is then sent through a linear layer, as shown in the model below.
class TestingClassifier(nn.Module):
  def __init__(self, vocabSize, features, embeddingDim):
      super(TestingClassifier, self).__init__()
      self.embeddings = nn.Embedding(vocabSize, embeddingDim)
      self.linear = nn.Linear(features, 2)
      self.sigmoid = nn.Sigmoid()

  def forward(self, inputs):
      embeds = self.embeddings(inputs)
      avged = torch.mean(embeds, dim=-1)
      output = self.linear(avged)
      output = self.sigmoid(output)
      return output

I am running BCELoss as loss function and SGD as optimizer. My problem is that my loss barely decreases as training goes on, almost as if it converges with a very high loss. I have tried different learning rates (0.0001, 0.001, 0.01 and 0.1) but I get the same issue.
My training function is as follows:
def train_model(model,
                optimizer,
                lossFunction,
                batchSize,
                epochs,
                isRnnModel,
                trainDataLoader,
                validDataLoader,
                earlyStop = False,
                maxPatience = 1
):

  validationAcc = []
  patienceCounter = 0
  stopTraining = False
  model.train()

  # Train network
  for epoch in range(epochs):
    losses = []
    if(stopTraining):
      break

    for inputs, labels in tqdm(trainDataLoader, position=0, leave=True):

      optimizer.zero_grad()

      # Predict and calculate loss
      prediction = model(inputs)
      loss = lossFunction(prediction, labels)
      losses.append(loss)

      # Backward propagation
      loss.backward()

      # Readjust weights
      optimizer.step()

    print(sum(losses) / len(losses))
    curValidAcc = check_accuracy(validDataLoader, model, isRnnModel) # Check accuracy on validation set
    curTrainAcc = check_accuracy(trainDataLoader, model, isRnnModel)
    print("Epoch", epoch + 1, "Training accuracy", curTrainAcc, "Validation accuracy:", curValidAcc)

    # Control early stopping
    if(earlyStop):
      if(patienceCounter == 0):
        if(len(validationAcc) > 0 and curValidAcc < validationAcc[-1]):
          benchmark = validationAcc[-1]
          patienceCounter += 1
          print("Patience counter", patienceCounter)

      elif(patienceCounter == maxPatience):
        print("EARLY STOP. Patience level:", patienceCounter)
        stopTraining = True

      else:
        if(curValidAcc < benchmark):
          patienceCounter += 1
          print("Patience counter", patienceCounter)

        else:
          benchmark = curValidAcc
          patienceCounter = 0

      validationAcc.append(curValidAcc)

Batch size is 32 (training set contains 8000 rows), vocabulary size is 300k, embedding dimension is 24. I have tried adding more linear layers to the network, but it makes no difference. The prediction accuracy on the training and validation sets stays at around 50% (which is horrible) even after many epochs of training. Any help is much appreciated!


title: Calculation FLOPs of adaptive_avg_pool2d

body:
I found several definitions of FLOPs in countering the flops for adaptive_avg_pool2d:

From fvcore , it defines the FLOPs as

1 * prod(input)

which is , 1 x N x C_in x H_in x W_in.

Another definition is from the output perspective. I found one from here :

It first calculate the kernel size, say, (kx, ky)
Then compute the flops as
( kx*xy +1 ) * prod(output)

which is , (k_x x k_y + 1) x (N x C_out x H_out x W_out)
Which definition is correct? Is there any material of calculating FLOPs?


title: Competition for pipe resources while trying to receive PyTorch tensors

body:
I met a problem that sub-processes in python will compete for pipe resources when one subprocess tries to receive PyTorch tensors and another subprocess requests a different pipe connection.
Here is my code (run on Ubuntu 18.04):

import time
from multiprocessing import Process, Pipe
import torch

def send(conn):
    a, b = Pipe()
    data = [torch.rand(1) for _ in range(20)]
    print('start sending', time.time())
    conn.send(data)
    print('finish sending', time.time())

    while True:
        if a.poll():
            pass
        # else:
        #    time.sleep(0.1)

def recv(conn):
    while True:
        if conn.poll():
            print('start receiving', time.time())
            conn.recv()
            print('finish receiving', time.time())
            break
        else:
            time.sleep(0.1)

if __name__ == '__main__':
    recv_conn, send_conn = Pipe()

    send_proc = Process(target=send, args=(send_conn,))
    recv_proc = Process(target=recv, args=(recv_conn,))

    send_proc.start()
    recv_proc.start()

    recv_proc.join()
    send_proc.terminate()

This code needs several seconds to receive the torch tensors. But if I uncomment the last two lines in send method, the speed of receiving the torch tensors will be very quick.
And this problem only appears while sending torch tensors.
Does anyone know the reason for this problem?


title: How to use a model (model.pt) that was created by pytorch for transfer learning to a new model using keras to train a new dataset?

body:
I want to freeze layers and use the model for transfer learning to another dataset but How?
By the way this model (model.pt) was created by pytorch and I would like to use it for transfer learning to a new model using keras to train a new dataset
I tried pytorch_to_keras and I have many confused . can you help me to transfer learning to another dataset.
Model(
  (data_bn): BatchNorm1d(54, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (st_gcn_networks): ModuleList(
    (0): st_gcn(
      (gcn): ConvTemporalGraphical(
        (conv): Conv2d(3, 192, kernel_size=(1, 1), stride=(1, 1))
      )
      (tcn): Sequential(
        (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(64, 64, kernel_size=(9, 1), stride=(1, 1), padding=(4, 0))
        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): Dropout(p=0, inplace=True)
      )
      (relu): ReLU(inplace=True)
    )
    (1): st_gcn(
      (gcn): ConvTemporalGraphical(
        (conv): Conv2d(64, 192, kernel_size=(1, 1), stride=(1, 1))
      )
        (1): ReLU(inplace=True)
        (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(64, 64, kernel_size=(9, 1), stride=(1, 1), padding=(4, 0))
        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): Dropout(p=0, inplace=True)
      )
      (relu): ReLU(inplace=True)
    )
    (2): st_gcn(
      (gcn): ConvTemporalGraphical(
        (conv): Conv2d(64, 192, kernel_size=(1, 1), stride=(1, 1))
      )
      (tcn): Sequential(
        (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(64, 64, kernel_size=(9, 1), stride=(1, 1), padding=(4, 0))
        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): Dropout(p=0, inplace=True)
      )
      (relu): ReLU(inplace=True)
    )
    (3): st_gcn(
      (gcn): ConvTemporalGraphical(
        (conv): Conv2d(64, 192, kernel_size=(1, 1), stride=(1, 1))
      )
      (tcn): Sequential(
        (0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(64, 64, kernel_size=(9, 1), stride=(1, 1), padding=(4, 0))
        (3): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): Dropout(p=0, inplace=True)
      )
    )
    (4): st_gcn(
      (gcn): ConvTemporalGraphical(
        (conv): Conv2d(64, 384, kernel_size=(1, 1), stride=(1, 1))
      )
      (tcn): Sequential(
        (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(128, 128, kernel_size=(9, 1), stride=(2, 1), padding=(4, 0))
        (3): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): Dropout(p=0, inplace=True)
      )
      (residual): Sequential(
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 1))
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (relu): ReLU(inplace=True)
    )
    (5): st_gcn(
      (gcn): ConvTemporalGraphical(
        (conv): Conv2d(128, 384, kernel_size=(1, 1), stride=(1, 1))
      )
      (tcn): Sequential(
        (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(128, 128, kernel_size=(9, 1), stride=(1, 1), padding=(4, 0))
        (3): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (relu): ReLU(inplace=True)
    )
    (6): st_gcn(
      (gcn): ConvTemporalGraphical(
        (conv): Conv2d(128, 384, kernel_size=(1, 1), stride=(1, 1))
      )
      (tcn): Sequential(
        (0): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(128, 128, kernel_size=(9, 1), stride=(1, 1), padding=(4, 0))
        (3): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (relu): ReLU(inplace=True)
    )
    (7): st_gcn(
        (conv): Conv2d(128, 768, kernel_size=(1, 1), stride=(1, 1))
      )
      (tcn): Sequential(
        (0): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(256, 256, kernel_size=(9, 1), stride=(2, 1), padding=(4, 0))
        (3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): Dropout(p=0, inplace=True)
      )
      (residual): Sequential(
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 1))
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (8): st_gcn(
      (gcn): ConvTemporalGraphical(
      )
      (tcn): Sequential(
        (0): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(256, 256, kernel_size=(9, 1), stride=(1, 1), padding=(4, 0))
        (3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): Dropout(p=0, inplace=True)
      )
      (relu): ReLU(inplace=True)
    )
    (9): st_gcn(
      (gcn): ConvTemporalGraphical(
        (conv): Conv2d(256, 768, kernel_size=(1, 1), stride=(1, 1))
      )
      (tcn): Sequential(
        (0): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (1): ReLU(inplace=True)
        (2): Conv2d(256, 256, kernel_size=(9, 1), stride=(1, 1), padding=(4, 0))
        (3): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        (4): Dropout(p=0, inplace=True)
      )
      (relu): ReLU(inplace=True)
    )
  )
  (edge_importance): ParameterList(
      (0): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (1): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (2): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (3): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (4): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (5): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (6): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (7): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (8): Parameter containing: [torch.FloatTensor of size 3x18x18]
      (9): Parameter containing: [torch.FloatTensor of size 3x18x18]
  )
  (fcn): Conv2d(256, 400, kernel_size=(1, 1), stride=(1, 1))
)


title: How can I pass the gradient of a tensor as a parameter in the loss function in PyTorch?

body:
I am trying to implement a neural network in PyTorch to solve an ordinary differential equation (ODE). The network architecture is straight-forward. It is just a feed-forward neural network with n inputs and outputs and k layers.
class PINN(torch.nn.Module):
    def __init__(self,n):
        super().__init__()
        # Layers
        self.L1=torch.nn.Linear(1,n)
        self.L2=torch.nn.Linear(n,n)
        self.L3=torch.nn.Linear(n,1)
        # Activation functions
        self.t=torch.nn.Tanh()
        self.r=torch.nn.ReLU()
    def forward(self,x):
        a1=self.r(self.L1(x))
        a2=self.r(self.L2(a1))
        a3=self.r(self.L3(a2))+x
        return a3

I want to minimize the loss between the gradient of the output of my neural network and the right-hand side of the ODE. I have chosen to work with a mean-squared error loss. I know that PyTorch includes a built-in MSE loss. However, I defined my own loss function since I have to pass in the gradient of a tensor.
def ODELoss(x,y,x0):
    # Number of collocation points to sample
    n=len(x)
    # Initialize the loss to zero
    loss=torch.tensor(0.,requires_grad=True)
    # Loop over the "data". Technically, this is an unsupervised problem.
    # The "data" are points sampled on the domain which are then evaluated
    # according to the ODE.
    for (xx,yy) in zip(x,y):
        xx=torch.tensor([[xx]],requires_grad=True)
        yy=torch.tensor([[yy]])
        g(xx,x0).backward()
        dg=xx.grad.clone().requires_grad_(True)
        loss=loss+(dg-yy)**2
    loss=loss/n
    return loss

Here, g(x) is called the universal predictor. It is used in the literature to account for the initial condition(s).
def g(x,x0):
    return x*model(x)+x0

This doesn't seem to work because it seems like I am not passing in the gradient of the output correctly. Can anyone give me some guidance on how to do this?


title: what is the meaning of pytorch lstm weight?

body:
I have a simple lstm pytorch model, the model structure is :
LSTM(
  (lstm): LSTM(1, 2)
  (fc): Linear(in_features=2, out_features=1, bias=True)
)

this is a simple task, i want to predict the alphabet seqence.
which means: the x is [[[1, 2, 3]]] - > [4] (1-A, 2-B, 3-C), i want to input 'ABC', output 'D'
after training, i want to know more about my lstm layer. so i print the model parameters like this:
n [7]: l = a._modules['lstm']

In [8]: l.__dict__
Out[8]:
{'training': True,
 '_parameters': OrderedDict([('weight_ih_l0',
               Parameter containing:
               tensor([[ 0.2127],
                       [ 1.5807],
                       [-0.8566],
                       [ 1.0215],
                       [-0.7563],
                       [ 0.8248],
                       [-0.7307],
                       [ 1.4174]], dtype=torch.float64, requires_grad=True)),
              ('weight_hh_l0',
               Parameter containing:
               tensor([[ 0.0245, -0.5089],
                       [ 0.0338,  2.8269],
                       [-1.0781, -0.4691],
                       [-0.2368,  2.2788],
                       [-1.0743,  0.5130],
                       [ 0.8970, -0.0829],
                       [-0.7051, -4.8892],
                       [-0.5335,  1.8777]], dtype=torch.float64, requires_grad=True)),
              ('bias_ih_l0',
               Parameter containing:
               tensor([ 1.7890,  2.4625, -0.4471,  0.8364, -1.2260,  1.5116,  2.1067,  1.6485],
                      dtype=torch.float64, requires_grad=True)),
              ('bias_hh_l0',
               Parameter containing:
               tensor([ 1.5659,  2.6634, -0.2972,  0.6908, -1.1136,  0.8588,  1.4372,  1.6157],
                      dtype=torch.float64, requires_grad=True))]),
 '_buffers': OrderedDict(),
 '_non_persistent_buffers_set': set(),
 '_backward_hooks': OrderedDict(),
 '_is_full_backward_hook': None,
 '_forward_hooks': OrderedDict(),
 '_forward_pre_hooks': OrderedDict(),
 '_state_dict_hooks': OrderedDict(),
 '_load_state_dict_pre_hooks': OrderedDict(),
 '_modules': OrderedDict(),
 'mode': 'LSTM',
 'input_size': 1,
 'hidden_size': 2,
 'num_layers': 1,
 'bias': True,
 'batch_first': False,
 'dropout': 0.0,
 'bidirectional': False,
 'proj_size': 0,
 '_flat_weights_names': ['weight_ih_l0',
  'weight_hh_l0',
  'bias_ih_l0',
  'bias_hh_l0'],
 '_all_weights': [['weight_ih_l0',
   'weight_hh_l0',
   'bias_ih_l0',
   'bias_hh_l0']],
 '_flat_weights': [Parameter containing:
  tensor([[ 0.2127],
          [ 1.5807],
          [-0.8566],
          [ 1.0215],
          [-0.7563],
          [ 0.8248],
          [-0.7307],
          [ 1.4174]], dtype=torch.float64, requires_grad=True),
  Parameter containing:
  tensor([[ 0.0245, -0.5089],
          [ 0.0338,  2.8269],
          [-1.0781, -0.4691],
          [-0.2368,  2.2788],
          [-1.0743,  0.5130],
          [ 0.8970, -0.0829],
          [-0.7051, -4.8892],
          [-0.5335,  1.8777]], dtype=torch.float64, requires_grad=True),
  Parameter containing:
  tensor([ 1.7890,  2.4625, -0.4471,  0.8364, -1.2260,  1.5116,  2.1067,  1.6485],
         dtype=torch.float64, requires_grad=True),
  Parameter containing:
  tensor([ 1.5659,  2.6634, -0.2972,  0.6908, -1.1136,  0.8588,  1.4372,  1.6157],
         dtype=torch.float64, requires_grad=True)]}

can you help on understand the model output, what is the meaning of weight_ih_10, weight_hh_10, bias_hh_10 and other things?


title: supporting more ranks in a GPU (pytorch/openmpi)

body:
Currently I use Pytorch with OpenMPI backend for distributed training. I have 2 ranks and 2 GPUs where each rank is allocated to one GPU.
I want to increase the number of ranks to 10 using again two GPUs.
Is there a way to have more ranks in a single GPU (e.g. 0-4 ranks on GPU 0 and 5-9 on GPU 1)?


title: What is bias node in googlenet and how to remove it?

body: I am new to deep learning, i want to build a model that can identify similar images, i am reading classification is a Strong Baseline for Deep Metric Learning research paper. and here is they used the phrase: "remove the bias term inthe last linear layer". i have no idea what is bias term is and how to remove it from googlenet or other pretrained models. if someone help me out with this it would be great! :)


title: Customized model takes too much memory when training

body:
I'm re-implementing ResNet model with Pytorch. Everything works well except that the model takes to much memory when training and leads to OOM (Out Of Memory) error. I tried to debug and see that GPU memory increase (4Gb to 11Gb) after calling optimizer.step(). When I replaced my customized model with official ResNet model from Pytorch, the OOM error does not appear. My customized model even has less parameters than the official one (23M < 25M). If you have tried to re-implement models with Pytorch already, could you help me to find what wrongs I made, all my code to re-implement the model as below, appreciate for helps from you, thank you.
import torch.nn as nn
from torch.nn import Conv2d, BatchNorm2d, LeakyReLU, MaxPool2d, ModuleList

class ConvBlock(nn.Module):
    def __init__(self, in_channels, out_channels, down_sampler=None, stride=1):
        super(ConvBlock, self).__init__()
        self.conv1 = Conv2d(in_channels, out_channels // 4, kernel_size=1, stride=1)
        self.batch_norm1 = BatchNorm2d(out_channels // 4)
        self.relu1 = LeakyReLU()

        self.conv2 = Conv2d(out_channels // 4, out_channels // 4, kernel_size=3, stride=stride, padding=1)
        self.batch_norm2 = BatchNorm2d(out_channels // 4)
        self.relu2 = LeakyReLU()

        self.conv3 = Conv2d(out_channels // 4, out_channels, kernel_size=1, stride=1)
        self.batch_norm3 = BatchNorm2d(out_channels)
        self.relu3 = LeakyReLU()

        self.down_sampler = down_sampler

    def forward(self, x):
        identity = x
        x = self.relu1(self.batch_norm1(self.conv1(x)))
        x = self.relu2(self.batch_norm2(self.conv2(x)))
        x = self.batch_norm3(self.conv3(x))
        if self.down_sampler is not None:
            identity = self.down_sampler(identity)
        x = x + identity
        x = self.relu3(x)
        return x

class ConvGroup(nn.Module):
    def __init__(self, in_channels, out_channels, number_of_conv=1, stride=1):
        super(ConvGroup, self).__init__()
        layers = []
        down_sampler = nn.Sequential(Conv2d(in_channels, out_channels, kernel_size=1, stride=stride),
                                     nn.BatchNorm2d(out_channels))
        layers.append(ConvBlock(in_channels, out_channels, down_sampler, stride=stride))
        for i in range(number_of_conv - 1):
            layers.append(ConvBlock(out_channels, out_channels, down_sampler=None))
        self.conv_group = nn.Sequential(*layers)

    def forward(self, x):
        x = self.conv_group(x)
        return x

class ResNet(nn.Module):
    R_50 = [3, 4, 6, 3]
    R_101 = [3, 4, 23, 3]
    R_152 = [3, 8, 36, 3]
    STRIDES = [1, 2, 2, 2]

    def __init__(self, type=R_50):
        super(ResNet, self).__init__()
        self.layers = ModuleList()
        self.conv1 = Conv2d(in_channels=3, out_channels=64, kernel_size=7, stride=2, padding=3)
        self.max_pool = MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.relu = LeakyReLU()
        self.output_channels = 256
        in_channels = 64
        out_channels = 256
        for i in range(len(type)):
            self.layers.append(
                ConvGroup(in_channels=in_channels, out_channels=out_channels, number_of_conv=type[i],
                          stride=ResNet.STRIDES[i]))
            in_channels = out_channels
            out_channels = out_channels * 2

    def forward(self, x):
        x = self.conv1(x)
        x = self.max_pool(x)
        x = self.relu(x)
        c2 = self.layers[0](x)
        c3 = self.layers[1](c2)
        c4 = self.layers[2](c3)
        c5 = self.layers[3](c4)
        return c5, c4, c3, c2


title: "NotImplemented" error when loading pre-trained network Pytorch

body:
I have a simple neural-network

class NeuralNetwork(nn.Module):
    def __init__(self,n_inputs):
        super(NeuralNetwork, self).__init__()

        self.l1 = nn.Linear(n_inputs, 10)

    def forward(self,x):
         return self.l1(x)
net = NeuralNetwork(n_inputs=10)
net = train_network(net)

torch.save(net,"./pretrained_models/model_cnn.pt")

Then when I want to load it again in another script, I can understand that we need to instantiate a class with the same name before it works e.g
import torch

class NeuralNetwork(nn.Module):
      pass

net = torch.load("./pre_trained_models/model_cnn.pt")
net.eval()
X = load_data(data_path)
pred = net(X)# <---  NotImplementedError

Do I then need to re-write __init__ and forward again when loading? Im trying to load the entire model instead of the state_dict since I want to avoid having to re-write the network in production, when we change it in the development.


title: How to detect memory leakage in pytorch?

body:
I'm training a VGG11 from scratch, and I keep getting "Out of memory" in my GPU even though i clean the GPU entirely.
    self.criterion = nn.CrossEntropyLoss()
    lr_dict = {100: 0.01, 150: 0.001}
    self.optimizer = optim.SGD(self.model.parameters(), lr=0.1, momentum=0.9)
    result_per_epoc = list()
    train_acc = 0.0
    train_loss = 0.0
    for epoch in range(epochs):
        debug_memory(f"Train epoc {epoch} START", clear_mem=True)

        if epoch in lr_dict.keys():
            for gopt in self.optimizer.param_groups:
                gopt['lr'] = lr_dict[epoch]
        epoch_time_start = datetime.now()

        for batch_idx, (images, labels) in tqdm(enumerate(iter(traindata_loader)), total=len(traindata_loader)):
            images = images.to(self.device)
            labels = labels.to(self.device)
            self.optimizer.zero_grad()
            try:
                outputs = self.model(images)
            except Exception as e:
                debug_memory(f"On crash", clear_mem=True)
                print(f"Batch ID: {batch_idx}")
                raise e
            loss = self.criterion(outputs, labels)
            loss.backward()
            self.optimizer.step()

            train_accuracy = accuracy(outputs, labels)
            train_acc += float(train_accuracy)
            train_loss += float(loss)

            del images, labels, loss, outputs
            clear_memory()
        debug_memory(f"Train epoc {epoch} TRAIN", clear_mem=True)
        train_acc /= (batch_idx + 1)
        train_loss /= (batch_idx + 1)

Up to line "debug_memory(f"Train epoc {epoch} START", clear_mem=True)" I used 3,499/11,019 (31%) of my memory. But after a few batchs my code crashes on memory even though i delete everything i've added to the GPU and in "clear_memory" I did this:
torch.cuda.empty_cache()
gc.collect()
with torch.no_grad():
    torch.cuda.empty_cache()
gc.collect()

The model i'm using is "vgg11_bn" from torchvision.models
What else am I missing?


title: How to convert torchscript model in PyTorch to ordinary nn.Module?

body:
I am loading the torchscript model in the following way:
model = torch.jit.load("model.pt").to(device)

The children modules of this model are identified as RecursiveScriptModule. I would like to finetune the uploaded weights and in order to make it simplier and cast them to torch.float32 It is preferable to convert all this stuff to ordinary PyTorch nn.Module.
In the official docs https://pytorch.org/docs/stable/jit.html it is told how to convert nn.Module to torchscript, but I have not found any examples in doing this in the opposite direction. Is there a way to do this?
P.S the example of loading model pretrained model is given here:  https://github.com/openai/CLIP/blob/main/notebooks/Interacting_with_CLIP.ipynb


title: Empty parameters list for model

body:
I am trying to implement an autoencoder neural network based on convolutional layers and max pooling & unpooling to encode and decode images on greyscale (MNIST images). I have defined its class ConvAEUnpool as follows:
class ConvAEUnpool(nn.Module):
    def __init__(self):
        super(ConvAEUnpool, self).__init__()

        self.relu = nn.ReLU()
        self.sigmoid = nn.Sigmoid()

        #Coder
        self.conv1 = nn.Conv2d(1, 16, 3, stride=3, padding=1),  # b, 16, 10, 10
        self.maxPool1 = nn.MaxPool2d(2, stride=2, return_indices=True),  # b, 16, 5, 5
        self.conv2 = nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
        self.maxPool2 = nn.MaxPool2d(2, stride=1, return_indices=True)  # b, 8, 2, 2

        # Decoder
        self.maxUnpool1 = nn.MaxUnpool2d(2, stride=1) # b, 8, 3, 3
        self.deConv1 = nn.ConvTranspose2d( 8, 16, 3, stride=2),  # b, 16, 5, 5
        self.maxUnpool2 = nn.MaxUnpool2d(2, stride=2) # b, 16, 10, 10
        self.deConv2 = nn.ConvTranspose2d(16, 1, 3, stride=3, padding=1),  # b, 1, 28, 28

    def forward(self, x):
        # Code
        x = self.Conv1(x)
        x = self.relu(x)
        x, idx1 = self.MaxPool1(x)
        x = self.Conv2(x)
        x = self.relu(x)
        x, idx2 = self.MaxPool2(x)

        # Decode
        x = self.MaxUnpool1(x, idx2)
        x = self.DeConv1(x)
        x = self.MaxUnpool2(x, idx1)
        x = self.DeConv2(x)
        x = self.sigmoid(x)

        return x

For some reason, when I create an instance and try to select an optimizer for its parameters, as follows:
model = ConvAEUnpool().to(device)

# create an optimizer object
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

it raises the following error:
ValueError: optimizer got an empty parameter list
I've done some search and have tried by using a nn.ModuleList():
self.nn_layers = nn.ModuleList(modules=[self.Conv1, self.MaxPool1, self.Conv2, self.MaxPool2,
                                        self.MaxUnpool1, self.DeConv1, self.MaxUnpool2, self.DeConv2])

but I am not sure how to make it work with a MaxUnpool2d, as I need to pass the indices of the previuos MaxPool2d layer. Could anyone help me here?


title: What is the recommended way to sum a list in TorchScript

body:
Because Python builtin <built-in function sum> is currently not supported in Torchscript: I'm looking for the recommended way to do something like:
class Model(nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x):
        return sum(x.tolist())

model = Model()
model = torch.jit.script(model)
model(torch.arange(10))


title: Sample a tensor of probability distributions in pytorch

body: I want to sample a tensor of probability distributions with shape (N, C, H, W), where dimension 1 (size C) contains normalized probability distributions with ‘C’ possibilities. Is there a pytorch function to efficiently sample all the distributions in the tensor in parallel? I just need to sample each distribution once, so the result could either be a one-hot tensor with the same shape or a tensor of indices with shape (N, 1, H, W).


title: Image based attendence system dataset creation

body:
I have a Image based attendence system folder which has 14 subfolders and then many subfolders again. Folder has total around 4500 images. How do I create a image dataset with labels that can be used for PyTorch CNN Model?
I tried following code to read the few images and get pixel values. But in which format shall i store image arrays so that it can be used for PyTorch CNN model.

image_array = []
label = []

def image_vector(root_node):
    image = Image.open(root_node, 'r')
    width, height = image.size
    pixel = list(image.getdata())               #convert to pixel values
    pixel_final = np.array(pixel).reshape((width, height, -1))    #Resize
    return pixel_final

for path, _ , files in os.walk(root_node):
    for file in files:
       image = image_vector(os.path.join(path, file))
       image_array.append(image)
       label.append(os.path.splitext(file)[0])


title: Pytorch error: 'BiSeNet' object has no attribute 'module'

body:
I'm new to pytorch and I'm trying to study BiSeNet for image segmentation (code took from a github repo: https://github.com/ooooverflow/BiSeNet/blob/master/train.py). During the training phase, after some training epochs, the net performs validation and tryes to save the parameters of the model if the results of the val are better than the previous ones.
During this last operation, I get this error in line 109 (and also 102 during training):
AttributeError: 'BiSeNet' object has no attribute 'module'
I do not paste all the code here, but just the main steps.
First of all, they built the model like that:
os.environ['CUDA_VISIBLE_DEVICES'] = args.cuda
model = BiSeNet(args.num_classes, args.context_path)
if torch.cuda.is_available() and args.use_gpu:
        model = model.cuda()

So there exists a BiSeNet object created thanks to an imported module called "model" where there is a file named build_BiSeNet.py; In this script the class BiSeNet is defined and there is no attribute named module.
Looking at the pytorch documentation, seems like in the Model class there is an attribute called modules which contains the module I'd like to save.
In the docs, they suggest also to do torch.save(model.state_dict(), ...) in order to save the model, without calling the module attribute Ilike it is done in the line 109)
So, finally, my question is: In order to avoid the error that gives me, should I remove .module in line 109 (and so also in line 102) or maybe change this attribute to .modules?


title: Why libtorch tensor value assignment takes so much time?

body:
I just assign 10000 values to a tensor:
clock_t start = clock();
torch::Tensor transform_tensor = torch::zeros({ 10000 });
for (size_t m = 0; m < 10000 m++)
    transform_tensor[m] = int(m);
clock_t finish = clock();

And it takes 0.317s. If I assign 10,000 to an array or a vector, the time cost will be less.
Why tensor takes so much time? Can the time cost be decreased?


title: difference between Dataset and TensorDataset in pyTorch

body: what is the difference between "torch.utils.data.TensorDataset" and "torch.utils.data.Dataset" - the docs are not clear about that and I could not find any answers on google.


title: Training loss remains nearly unchanged while validation loss fluctuates around

body:
I am using a pretrained Vision Transformer model to train a new model in which input is images and output is predicted prices of those images.
As the Vision Transformer model is used for classifying 1024 classes, I changed its last layer into nn.Linear(1024,1) and I am using MSELoss instead of softmax. However, the training loss and validation loss keep being high. As in the picture below, training loss remains nearly unchanged while validation loss fluctuates a bit around 800000.
Do you have any ideas about what should I do to improve the learning?

[Update]
Here is part of the code
#Import model
pip install timm
import timm
model = timm.create_model('vit_large_patch16_224_in21k', pretrained=True)
for param in model.parameters():
    param.requires_grad = False

model.head = nn.Sequential(nn.Linear(1024, 10),
                           nn.ReLU(),
                           nn.Linear(10, 1))
model.cuda()

#Training setting
device = 'cuda'
learning_rate = 0.0005
gamma = 0.1
seed = 42
use_gpu_number = 0
epochs = 30
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
scheduler = StepLR(optimizer, step_size=7, gamma=gamma)

#Training
loss_list = []
val_loss_list = []
val_corr_list = []
for epoch in range(epochs):
    epoch_loss = 0
    epoch_accuracy = 0

    model.train()
    for images, price in tqdm(train_loader):
        images = images.to(device)
        price = price.to(device)

        optimizer.zero_grad()

        output = model(images)
        loss = criterion(output, price)

        loss.backward()
        optimizer.step()

        epoch_loss += loss / len(train_loader)

    loss_list.append(epoch_loss.float())

    model.eval()
    with torch.no_grad():
        epoch_val_accuracy = 0
        epoch_val_loss = 0
        for images, price in valid_loader:
            images = images.to(device)
            price = price.to(device)

            val_output = model(images)
            val_loss = criterion(val_output, price)

            epoch_val_loss += val_loss / len(valid_loader)
        _, _, _, val_corr = model_valid(valid_loader,
                              model,
                              use_gpu_number,
                              criterion)

        val_loss_list.append(epoch_val_loss.float())
        val_corr_list.append(val_corr)


title: How to use CUDA toolkit and drivers installed in the singularity container and not from the host OS?

body:
I am using base container from https://gitlab.com/nvidia/container-images/cuda/blob/master/doc/supported-tags.md
My definition file starts with
Bootstrap: docker
From: nvidia/cuda:10.0-cudnn7-devel-ubuntu18.04

My Host OS has CUDA version 11.0.
I am using the following commands to build my container
singularity build --sandbox mycontainer/ new.def
singularity shell --nv --writable mycontainer/
The --nv flag leads to using drivers from the host OS(cuda 11.0), but I require CUDA 10.0 inside the container.
Also, nvidia-smi doesnt work inside the container, although nvcc --version works.
How to fix this ?


title: Creating a custom BCE with logit loss function

body:
I want to create a custom loss function for multi-label classification. The idea is to weigh the positive and negative labels differently. For this, I am making use of this custom code implementation.
class WeightedBCEWithLogitLoss(nn.Module):
    def __init__(self, pos_weight, neg_weight):
        super(WeightedBCEWithLogitLoss, self).__init__()
        self.register_buffer('neg_weight', neg_weight)
        self.register_buffer('pos_weight', pos_weight)

    def forward(self, input, target):
        assert input.shape == target.shape, "The loss function received invalid input shapes"
        y_hat = torch.sigmoid(input + 1e-8)
        loss = -1.0 * (self.pos_weight * target * torch.log(y_hat + 1e-6) + self.neg_weight * (1 - target) * torch.log(1 - y_hat + 1e-6))
        # Account for 0 times inf which leads to nan
        loss[torch.isnan(loss)] = 0
        # We average across each of the extra attribute dimensions to generalize it
        loss = loss.mean(dim=1)
        # We use mean reduction for our task
        return loss.mean()

I started getting nan values which I realized happened because of 0 times inf multiplication. I handled it as shown in the figure. Next, I again saw getting inf as the error value and corrected it by adding 1e-6 to the log (I tried with 1e-8 but that still gave me inf error value).
It would be great if someone can take a look and suggest further improvements and rectify any more bugs visible here.


title: How to decompose deep learning trainable parameter into two parts for Pytorch?

body:
I got this problem when I read the paper 'Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learning'. I am quite wondering about the disjoint learning. The author said 'For a given model, it should decompose into two parts where model = a + b. When we train the model on dataset A, we only update a and hold b constant. When we train the model on dataset B, we only update b and hold a constant.' How can we do this with pytorch?
Thanks so much for your help!!!


title: Using weights in CrossEntropyLoss and BCELoss (PyTorch)

body:
I am training a PyTorch model to perform binary classification. My minority class makes up about 10% of the data, so I want to use a weighted loss function. The docs for BCELoss and CrossEntropyLoss say that I can use a 'weight' for each sample.
However, when I declare CE_loss = nn.BCELoss() or nn.CrossEntropyLoss() and then do CE_Loss(output, target, weight=batch_weights), where output, target, and batch_weights are Tensors of batch_size, I get the following error message:
forward() got an unexpected keyword argument 'weight'


title: KeyError: 'CUDA_VISIBLE_DEVICES'

body:
today I asked this question concerning problems with PyTorch running on my GPU. In response, someone kindly suggested running the following code:
import os
num_gpus = os.environ['CUDA_VISIBLE_DEVICES'].split(',').__len__()
os.environ['CUDA_VISIBLE_DEVICES'] = ','.join(f'{i}' for i in range(num_gpus))

However, after running that second line, unfortunately I get another error:
num_gpus = os.environ['CUDA_VISIBLE_DEVICES'].split(',').__len__()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/hbp/anaconda3/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'CUDA_VISIBLE_DEVICES'

I tried to look it up, but couldn't find anything helpful.
Does anyone have an idea what I could try? The question I linked to may have relevant info.


title: Setting a minimum learning rate on "Reduce On Plateau"

body:
I'm currently using PyTorch's ReduceLROnPlateau learning rate scheduler using:
learning_rate = 1e-3
optimizer = optim.Adam(model.params, lr = learning_rate)
model.optimizer = optimizer
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(model.optimizer, factor=0.9, patience = 5000, verbose=True)

My issue is that the loss will get really low within a few minutes, then jumps really high and start decreasing steadily. Hence, ReduceLROnPlateau will just continuously decrease the learning rate to a point that the model can't learn anything (because it takes the lowest loss into acount).
Is there a way to set a minimum learning rate using this scheduler so that it won't go to almost 0? Say to set it at minimum of 1e8


title: Pytorch Predictions Becomes Same

body:
I developing a categorical deep learning model and I'm getting this results while training:
tensor([[ 0.0141, -0.0279,  0.0623],
        [ 0.0141, -0.0279,  0.0623],
        [ 0.0141, -0.0279,  0.0623],
        [ 0.0141, -0.0279,  0.0623]], grad_fn=<AddmmBackward>) tensor([1., 0., 1., 2.])
epoch = 1/10, step = 100/595, loss = 1.1103
tensor([[ 0.0294, -0.0161,  0.0353],
        [ 0.0294, -0.0161,  0.0353],
        [ 0.0294, -0.0161,  0.0353],
        [ 0.0294, -0.0161,  0.0353]], grad_fn=<AddmmBackward>) tensor([0., 0., 0., 1.])
epoch = 1/10, step = 200/595, loss = 1.0971
tensor([[ 0.0315, -0.0199,  0.0369],
        [ 0.0315, -0.0199,  0.0369],
        [ 0.0315, -0.0199,  0.0369],
        [ 0.0315, -0.0199,  0.0369]], grad_fn=<AddmmBackward>) tensor([0., 0., 2., 1.])
epoch = 1/10, step = 300/595, loss = 1.0951
tensor([[ 0.0388, -0.0036,  0.0134],
        [ 0.0388, -0.0036,  0.0134],
        [ 0.0388, -0.0036,  0.0134],
        [ 0.0388, -0.0036,  0.0134]], grad_fn=<AddmmBackward>) tensor([1., 1., 1., 1.])
epoch = 1/10, step = 400/595, loss = 1.1186
tensor([[ 0.0476, -0.0057,  0.0066],
        [ 0.0476, -0.0057,  0.0066],
        [ 0.0476, -0.0057,  0.0066],
        [ 0.0476, -0.0057,  0.0066]], grad_fn=<AddmmBackward>) tensor([1., 1., 2., 0.])
epoch = 1/10, step = 500/595, loss = 1.1043
tensor([[-0.0150,  0.0246,  0.0390],
        [-0.0150,  0.0246,  0.0390],
        [-0.0150,  0.0246,  0.0390],
        [-0.0150,  0.0246,  0.0390]], grad_fn=<AddmmBackward>) tensor([2., 1., 2., 0.])
epoch = 2/10, step = 100/595, loss = 1.0932
tensor([[0.0168, 0.0212, 0.0106],
        [0.0168, 0.0212, 0.0106],
        [0.0168, 0.0212, 0.0106],
        [0.0168, 0.0212, 0.0106]], grad_fn=<AddmmBackward>) tensor([0., 1., 0., 0.])
epoch = 2/10, step = 200/595, loss = 1.0969
tensor([[ 0.0448, -0.0110,  0.0148],
        [ 0.0448, -0.0110,  0.0148],
        [ 0.0448, -0.0110,  0.0148],
        [ 0.0448, -0.0110,  0.0148]], grad_fn=<AddmmBackward>) tensor([0., 2., 2., 1.])
epoch = 2/10, step = 300/595, loss = 1.0992
tensor([[0.0310, 0.0007, 0.0168],
        [0.0310, 0.0007, 0.0168],
        [0.0310, 0.0007, 0.0168],
        [0.0310, 0.0007, 0.0168]], grad_fn=<AddmmBackward>) tensor([1., 1., 0., 0.])
epoch = 2/10, step = 400/595, loss = 1.0990
tensor([[ 0.0336,  0.0241, -0.0092],
        [ 0.0336,  0.0241, -0.0092],
        [ 0.0336,  0.0241, -0.0092],
        [ 0.0336,  0.0241, -0.0092]], grad_fn=<AddmmBackward>) tensor([0., 1., 1., 1.])
epoch = 2/10, step = 500/595, loss = 1.0885
tensor([[0.0288, 0.0057, 0.0140],
        [0.0288, 0.0057, 0.0140],
        [0.0288, 0.0057, 0.0140],
        [0.0288, 0.0057, 0.0140]], grad_fn=<AddmmBackward>) tensor([1., 0., 1., 1.])
epoch = 3/10, step = 100/595, loss = 1.1033
tensor([[0.0243, 0.0002, 0.0240],
        [0.0243, 0.0002, 0.0240],
        [0.0243, 0.0002, 0.0240],
        [0.0243, 0.0002, 0.0240]], grad_fn=<AddmmBackward>) tensor([2., 2., 1., 1.])
epoch = 3/10, step = 200/595, loss = 1.1027
tensor([[ 0.0512,  0.0081, -0.0107],
        [ 0.0512,  0.0081, -0.0107],
        [ 0.0512,  0.0081, -0.0107],
        [ 0.0512,  0.0081, -0.0107]], grad_fn=<AddmmBackward>) tensor([2., 0., 0., 1.])
epoch = 3/10, step = 300/595, loss = 1.0902
tensor([[ 0.0374, -0.0038,  0.0150],
        [ 0.0374, -0.0038,  0.0150],
        [ 0.0374, -0.0038,  0.0150],
        [ 0.0374, -0.0038,  0.0150]], grad_fn=<AddmmBackward>) tensor([0., 2., 0., 0.])
epoch = 3/10, step = 400/595, loss = 1.0831
tensor([[ 0.0325,  0.0225, -0.0065],
        [ 0.0325,  0.0225, -0.0065],
        [ 0.0325,  0.0225, -0.0065],
        [ 0.0325,  0.0225, -0.0065]], grad_fn=<AddmmBackward>) tensor([1., 2., 2., 1.])
epoch = 3/10, step = 500/595, loss = 1.1069
tensor([[0.0210, 0.0100, 0.0175],
        [0.0210, 0.0100, 0.0175],
        [0.0210, 0.0100, 0.0175],
        [0.0210, 0.0100, 0.0175]], grad_fn=<AddmmBackward>) tensor([0., 1., 1., 2.])
epoch = 4/10, step = 100/595, loss = 1.1002
tensor([[0.0189, 0.0051, 0.0245],
        [0.0189, 0.0051, 0.0245],
        [0.0189, 0.0051, 0.0245],
        [0.0189, 0.0051, 0.0245]], grad_fn=<AddmmBackward>) tensor([0., 2., 0., 2.])
epoch = 4/10, step = 200/595, loss = 1.0931
tensor([[0.0184, 0.0076, 0.0226],
        [0.0184, 0.0076, 0.0226],
        [0.0184, 0.0076, 0.0226],
        [0.0184, 0.0076, 0.0226]], grad_fn=<AddmmBackward>) tensor([1., 2., 1., 0.])
epoch = 4/10, step = 300/595, loss = 1.1008
tensor([[-0.0005,  0.0347,  0.0144],
        [-0.0005,  0.0347,  0.0144],
        [-0.0005,  0.0347,  0.0144],
        [-0.0005,  0.0347,  0.0144]], grad_fn=<AddmmBackward>) tensor([1., 1., 0., 1.])
epoch = 4/10, step = 400/595, loss = 1.0890
tensor([[0.0122, 0.0191, 0.0173],
        [0.0122, 0.0191, 0.0173],
        [0.0122, 0.0191, 0.0173],
        [0.0122, 0.0191, 0.0173]], grad_fn=<AddmmBackward>) tensor([2., 1., 2., 2.])
epoch = 4/10, step = 500/595, loss = 1.0971
tensor([[ 0.0323, -0.0013,  0.0175],
        [ 0.0323, -0.0013,  0.0175],
        [ 0.0323, -0.0013,  0.0175],
        [ 0.0323, -0.0013,  0.0175]], grad_fn=<AddmmBackward>) tensor([2., 0., 0., 2.])
epoch = 5/10, step = 100/595, loss = 1.0900
tensor([[ 0.0335,  0.0183, -0.0032],
        [ 0.0335,  0.0183, -0.0032],
        [ 0.0335,  0.0183, -0.0032],
        [ 0.0335,  0.0183, -0.0032]], grad_fn=<AddmmBackward>) tensor([1., 2., 1., 2.])
epoch = 5/10, step = 200/595, loss = 1.1074
tensor([[0.0239, 0.0134, 0.0112],
        [0.0239, 0.0134, 0.0112],
        [0.0239, 0.0134, 0.0112],
        [0.0239, 0.0134, 0.0112]], grad_fn=<AddmmBackward>) tensor([1., 2., 0., 2.])
epoch = 5/10, step = 300/595, loss = 1.0999
tensor([[ 0.0394,  0.0244, -0.0153],
        [ 0.0394,  0.0244, -0.0153],
        [ 0.0394,  0.0244, -0.0153],
        [ 0.0394,  0.0244, -0.0153]], grad_fn=<AddmmBackward>) tensor([1., 1., 2., 2.])
epoch = 5/10, step = 400/595, loss = 1.1105
tensor([[ 0.0529,  0.0281, -0.0324],
        [ 0.0529,  0.0281, -0.0324],
        [ 0.0529,  0.0281, -0.0324],
        [ 0.0529,  0.0281, -0.0324]], grad_fn=<AddmmBackward>) tensor([2., 2., 2., 2.])
epoch = 5/10, step = 500/595, loss = 1.1478
tensor([[0.0104, 0.0371, 0.0011],
        [0.0104, 0.0371, 0.0011],
        [0.0104, 0.0371, 0.0011],
        [0.0104, 0.0371, 0.0011]], grad_fn=<AddmmBackward>) tensor([2., 1., 1., 1.])
epoch = 6/10, step = 100/595, loss = 1.0868
tensor([[0.0038, 0.0232, 0.0215],
        [0.0038, 0.0232, 0.0215],
        [0.0038, 0.0232, 0.0215],
        [0.0038, 0.0232, 0.0215]], grad_fn=<AddmmBackward>) tensor([2., 0., 1., 2.])
epoch = 6/10, step = 200/595, loss = 1.0973
tensor([[0.0094, 0.0273, 0.0119],
        [0.0094, 0.0273, 0.0119],
        [0.0094, 0.0273, 0.0119],
        [0.0094, 0.0273, 0.0119]], grad_fn=<AddmmBackward>) tensor([1., 0., 0., 2.])
epoch = 6/10, step = 300/595, loss = 1.1003
tensor([[ 0.0397, -0.0106,  0.0194],
        [ 0.0397, -0.0106,  0.0194],
        [ 0.0397, -0.0106,  0.0194],
        [ 0.0397, -0.0106,  0.0194]], grad_fn=<AddmmBackward>) tensor([0., 2., 2., 1.])
epoch = 6/10, step = 400/595, loss = 1.0980
tensor([[ 0.0321, -0.0099,  0.0264],
        [ 0.0321, -0.0099,  0.0264],
        [ 0.0321, -0.0099,  0.0264],
        [ 0.0321, -0.0099,  0.0264]], grad_fn=<AddmmBackward>) tensor([1., 1., 2., 2.])
epoch = 6/10, step = 500/595, loss = 1.1067
tensor([[ 0.0487, -0.0066,  0.0064],
        [ 0.0487, -0.0066,  0.0064],
        [ 0.0487, -0.0066,  0.0064],
        [ 0.0487, -0.0066,  0.0064]], grad_fn=<AddmmBackward>) tensor([2., 0., 0., 1.])
epoch = 7/10, step = 100/595, loss = 1.0908
tensor([[ 0.0628,  0.0229, -0.0372],
        [ 0.0628,  0.0229, -0.0372],
        [ 0.0628,  0.0229, -0.0372],
        [ 0.0628,  0.0229, -0.0372]], grad_fn=<AddmmBackward>) tensor([2., 2., 0., 0.])
epoch = 7/10, step = 200/595, loss = 1.1028
tensor([[ 0.0379,  0.0228, -0.0121],
        [ 0.0379,  0.0228, -0.0121],
        [ 0.0379,  0.0228, -0.0121],
        [ 0.0379,  0.0228, -0.0121]], grad_fn=<AddmmBackward>) tensor([0., 0., 0., 0.])
epoch = 7/10, step = 300/595, loss = 1.0771
tensor([[ 0.0359,  0.0242, -0.0116],
        [ 0.0359,  0.0242, -0.0116],
        [ 0.0359,  0.0242, -0.0116],
        [ 0.0359,  0.0242, -0.0116]], grad_fn=<AddmmBackward>) tensor([1., 1., 0., 1.])
epoch = 7/10, step = 400/595, loss = 1.0879
tensor([[ 0.0295,  0.0306, -0.0115],
        [ 0.0295,  0.0306, -0.0115],
        [ 0.0295,  0.0306, -0.0115],
        [ 0.0295,  0.0306, -0.0115]], grad_fn=<AddmmBackward>) tensor([1., 0., 2., 1.])
epoch = 7/10, step = 500/595, loss = 1.0952
tensor([[0.0272, 0.0016, 0.0197],
        [0.0272, 0.0016, 0.0197],
        [0.0272, 0.0016, 0.0197],
        [0.0272, 0.0016, 0.0197]], grad_fn=<AddmmBackward>) tensor([2., 1., 2., 0.])
epoch = 8/10, step = 100/595, loss = 1.0978
tensor([[0.0273, 0.0187, 0.0025],
        [0.0273, 0.0187, 0.0025],
        [0.0273, 0.0187, 0.0025],
        [0.0273, 0.0187, 0.0025]], grad_fn=<AddmmBackward>) tensor([1., 2., 1., 0.])
epoch = 8/10, step = 200/595, loss = 1.0980
tensor([[ 0.0573,  0.0246, -0.0333],
        [ 0.0573,  0.0246, -0.0333],
        [ 0.0573,  0.0246, -0.0333],
        [ 0.0573,  0.0246, -0.0333]], grad_fn=<AddmmBackward>) tensor([1., 0., 0., 0.])
epoch = 8/10, step = 300/595, loss = 1.0664
tensor([[ 0.0169,  0.0369, -0.0052],
        [ 0.0169,  0.0369, -0.0052],
        [ 0.0169,  0.0369, -0.0052],
        [ 0.0169,  0.0369, -0.0052]], grad_fn=<AddmmBackward>) tensor([2., 2., 1., 2.])
epoch = 8/10, step = 400/595, loss = 1.1096
tensor([[0.0025, 0.0081, 0.0379],
        [0.0025, 0.0081, 0.0379],
        [0.0025, 0.0081, 0.0379],
        [0.0025, 0.0081, 0.0379]], grad_fn=<AddmmBackward>) tensor([0., 1., 1., 0.])
epoch = 8/10, step = 500/595, loss = 1.1096
tensor([[-0.0015, -0.0033,  0.0534],
        [-0.0015, -0.0033,  0.0534],
        [-0.0015, -0.0033,  0.0534],
        [-0.0015, -0.0033,  0.0534]], grad_fn=<AddmmBackward>) tensor([1., 0., 0., 1.])
epoch = 9/10, step = 100/595, loss = 1.1176
tensor([[0.0130, 0.0270, 0.0085],
        [0.0130, 0.0270, 0.0085],
        [0.0130, 0.0270, 0.0085],
        [0.0130, 0.0270, 0.0085]], grad_fn=<AddmmBackward>) tensor([0., 0., 0., 1.])
epoch = 9/10, step = 200/595, loss = 1.0983
tensor([[ 0.0625,  0.0357, -0.0496],
        [ 0.0625,  0.0357, -0.0496],
        [ 0.0625,  0.0357, -0.0496],
        [ 0.0625,  0.0357, -0.0496]], grad_fn=<AddmmBackward>) tensor([1., 2., 0., 1.])
epoch = 9/10, step = 300/595, loss = 1.0949
tensor([[ 0.0622, -0.0005, -0.0132],
        [ 0.0622, -0.0005, -0.0132],
        [ 0.0622, -0.0005, -0.0132],
        [ 0.0622, -0.0005, -0.0132]], grad_fn=<AddmmBackward>) tensor([1., 0., 0., 2.])
epoch = 9/10, step = 400/595, loss = 1.0876
tensor([[ 0.0217, -0.0025,  0.0293],
        [ 0.0217, -0.0025,  0.0293],
        [ 0.0217, -0.0025,  0.0293],
        [ 0.0217, -0.0025,  0.0293]], grad_fn=<AddmmBackward>) tensor([1., 0., 2., 0.])
epoch = 9/10, step = 500/595, loss = 1.0973
tensor([[ 0.0379, -0.0200,  0.0306],
        [ 0.0379, -0.0200,  0.0306],
        [ 0.0379, -0.0200,  0.0306],
        [ 0.0379, -0.0200,  0.0306]], grad_fn=<AddmmBackward>) tensor([1., 0., 2., 1.])
epoch = 10/10, step = 100/595, loss = 1.1080
tensor([[ 0.0466,  0.0045, -0.0025],
        [ 0.0466,  0.0045, -0.0025],
        [ 0.0466,  0.0045, -0.0025],
        [ 0.0466,  0.0045, -0.0025]], grad_fn=<AddmmBackward>) tensor([2., 2., 1., 1.])
epoch = 10/10, step = 200/595, loss = 1.1141
tensor([[0.0321, 0.0124, 0.0040],
        [0.0321, 0.0124, 0.0040],
        [0.0321, 0.0124, 0.0040],
        [0.0321, 0.0124, 0.0040]], grad_fn=<AddmmBackward>) tensor([2., 0., 1., 2.])
epoch = 10/10, step = 300/595, loss = 1.1017
tensor([[0.0204, 0.0120, 0.0162],
        [0.0204, 0.0120, 0.0162],
        [0.0204, 0.0120, 0.0162],
        [0.0204, 0.0120, 0.0162]], grad_fn=<AddmmBackward>) tensor([1., 2., 0., 0.])
epoch = 10/10, step = 400/595, loss = 1.0976
tensor([[0.0092, 0.0228, 0.0165],
        [0.0092, 0.0228, 0.0165],
        [0.0092, 0.0228, 0.0165],
        [0.0092, 0.0228, 0.0165]], grad_fn=<AddmmBackward>) tensor([1., 2., 0., 0.])
epoch = 10/10, step = 500/595, loss = 1.1004
acc = 34.22818791946309
Accuracy of 0: 0.0, correct predictions = 0
Accuracy of 1: 0.0, correct predictions = 0
Accuracy of 2: 100.0, correct predictions = 204

Process finished with exit code 0

It gives me the same prediction no matter what is the input
Here is the model:
import torch
import torchvision
from sklearn.model_selection import train_test_split
from torch.utils.data import Dataset
from torchvision.transforms import transforms
import torch.nn as nn
import pandas as pd
import numpy as np

num_classes = 3
batch_size = 4
hidden_size = 100
input_size = 196
learning_rate = 0.01
num_epochs = 10

class price_dataset():
    def __init__(self):
        xy = pd.read_csv('data_balanced.csv')

        self.n_samples = xy.shape[0]

        xy = xy.to_numpy()
        self.x_data = torch.from_numpy(xy[:, 7:203].astype(np.float32))
        self.y_data = torch.from_numpy(xy[:, 6].astype(np.float32))

    def __getitem__(self, index):
        return self.x_data[index], self.y_data[index]

    def __len__(self):
        return self.n_samples

dataset = price_dataset()

train_dataset, test_dataset = train_test_split(dataset, test_size=0.20, random_state=0)

# Data loader
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

# print(example_data)

class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        self.input_size = input_size
        self.l1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.l2 = nn.Linear(hidden_size, num_classes)

    def forward(self, inputs):
        out = self.l1(inputs)
        out = self.relu(out)
        out = self.l2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes)

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

n_total_steps = len(train_loader)
for epoch in range(num_epochs):
    for i, (input, labels) in enumerate(train_loader):

        y_pred = model(input)

        loss = criterion(y_pred, labels.long())

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # print(f'prediction = {y_pred[0]}, label = {labels[0]}')

        if (i + 1) % 100 == 0:
            print(y_pred, labels)
            print(f'epoch = {epoch + 1}/{num_epochs}, step = {i + 1}/{n_total_steps}, loss = {loss.item():.4f}')

with torch.no_grad():
    n_correct = 0
    n_samples = 0
    n_class_correct = [0 for i in range(3)]
    n_class_samples = [0 for i in range(3)]

    for input, labels in test_loader:
        labels = torch.flatten(labels)
        y_pred = model(input)

        _, prediction = torch.max(y_pred.data, 1)

        # print(f'prediction = {prediction[0]}, label = {labels[0]}')
        n_samples += labels.shape[0]
        # n_correct += (prediction == labels).sum().item()

        for i in range(batch_size):
            label = labels[i]
            pred = prediction[i]

            if (label==pred):
                n_class_correct[int(label)] +=1

            n_class_samples[int(label)] += 1

        for pred, label in zip(prediction, labels):
            # print(f'prediction = {pred}, label = {label}')
            if pred != 0 and label !=0 and pred == label:
                n_correct += (pred == label).item()

    acc = 100 * (n_correct / n_samples)
    print(f'acc = {acc}')

    for i in range(3):
        acc = 100 * n_class_correct[i]/n_class_samples[i]
        print(f'Accuracy of {i}: {acc}, correct predictions = {n_class_correct[i]}')

Is there are any problems with the model? I tried changing the optimizer and learning rate but nothing is changed I also tried changing hidden layer size but it didn't solve the issue also
Link to the data: https://docs.google.com/spreadsheets/d/12lFXSnA_mOxRX-JBoMGyMcEEedmFU0eIEMbf5ne2a14/edit?usp=sharing


title: Why paths of length return 0 using Pytorch?

body:
This function return error which is same as below while the paths is correct. Am I misunderstanding how paths works? (This snippet is from github-repo)

paths = paths[::skip_frame]

ValueError: slice step cannot be zero

Here is the main function:

 def make_gif(paths, gif_path, max_frame=50, prefix=""):
      import imagei

paths.sort()
skip_frame = len(paths) // max_frame

paths = paths[::skip_frame]

images = [imageio.imread(path) for path in paths]
max_h, max_w, max_c = np.max(
        np.array([image.shape for image in images]), 0)

Here is how function called:

parser = argparse.ArgumentParser()
parser.add_argument("--model_name", type=str)
parser.add_argument("--max_frame", type=int, default=50)
parser.add_argument("--output", type=str, default="sampe.gif")
parser.add_argument("--title", type=str, default="")

if __name__ == "__main__":
    args = parser.parse_args()

    paths = glob(f"./logs/{args.model_name}/networks/*.png")

    make_gif(paths, args.output,
            max_frame=args.max_frame,
            prefix=f"{args.title}\n" if args.title else "")


title: Finding cell state in OpenNMT-py

body: I can't seem to find a value for the cell state when loading a model trained with a network of LSTM neurons in OpenNMT-py. I can only find the weight matrices and biases under 'model' in the model loaded with torch.load. Does anyone know if the cell state is saved for the model and if so, where?


title: Compute a convolution with weights that where computed by another function

body:
I would like to perform a conv2d but a special one because I compute the weights of a conv2d layer in a function for each sample. Can I simply assign the weights of a given layer for each sample?

I am implementing the Spatially variant convolution of this paper : https://arxiv.org/abs/1804.00389

Thanks


title: cant create Dataloader for cycle gans with two csv input files

body: I have tried the code which is attached in the screen shot below. But I am not able to do it. How can i create Dataloader for cycle gans with this input data


title: Can i have mutliple labels for a single image in an image classification scenario

body:
If I am building a model where I need to predict the vehicle, color of it, and make of it, then can I use all the labels for a single image and build my model around it.

Like for a single image of a vehicle which is a car (car1.jpg) will have labels like - Sedan(Make), Blue(Color) and Car(Type of vehicle). Can I make a single model for this or I will have to make 3 separate models for this problem.


title: Understanding TensorboardX graph

body:
I'm inspecting my model in the tensorboardX. I'm using PyTorch. Following is the forward method:

    def forward(self, x):
        x = self.net(x)
        x = x.view(x.shape[0], -1)  # flat
        x = self.ext_net(x)
        x = F.log_softmax(x, dim=1)

        return x

In the graph, I don't understand what these values and nodes in blue and green boxes represent. Why is there a separate dataflow from input to ext_net (FCNCustom)? Shouldn't there be only one coming through net?


title: How to feed decimal_y_values for training where last layer of model has 10 neurons with softmax activation and loss is Earth Mover Loss

body:
What should be the format/data types of y-labels for training if the actual y-labels cab be any decimal number between 0-9 (4.1,8.5  etc) and the last output layer is defined as:

out = Dense(10,activation='softmax')(previous_layer)

and the loss fuction to use is:

def earth_mover_loss(y_true, y_pred):
    cdf_ytrue = K.cumsum(y_true, axis=-1)
    cdf_ypred = K.cumsum(y_pred, axis=-1)
    samplewise_emd = K.sqrt(K.mean(K.square(K.abs(cdf_ytrue - cdf_ypred)), axis=-1))
    return K.mean(samplewise_emd)

How should I feed the labels in my network. I want to mimic this model


title: Why use torch.sparse api

body:
Let's say, my input is a sparse vector of 5000 dim. If I declare a normal linear layer

L=nn.Linear(5000,500)
output=L(input_sparse_vector)

and optimize it as usual instead it using torch.sparse, what problem I might have?

will the gradient calculation in every step will be the same?


title: In a sense, is using sin as an activation function equivalent to using cos as an activation function?

body: In the DNN structure, we may use different kinds of activation functions. We can use cos or sin, but in a sense, is using sin as an activation function equivalent to using cos as an activation function?


title: How to input fractional score labels for each input image while training a CNN in pytorch for regression problem

body:
During a convolution neural network training, each input image is assigned to a score (floating point). How to feed that fraction value corresponding to each image as a label while training a cnn? Then I want to use the MSELoss.
For normal classification one have to create one folder for each classes then store the respective images to that class folder but for a continuous fraction score label how to do that and what torch API to use? Any suggestions on this will be highly helpful.


title: Setting batch size in a LSTM network

body:
I was referring to this implementation - https://colab.research.google.com/drive/1_J2MrBSvsJfOcVmYAN2-WSp36BtsFZCa#scrollTo=X_f1WaTJhiXy
It uses a batch-size of 1.
I was wondering if this is a convention for implementing LSTM networks with batch size of 1 as with each time step (each iteration in the training dataset), the LSTM cells look back at the previous output and with batches it will not be accomplished?
If a batch implementation of the above example exists, can some one kindly help me write the code modifications I would need based on the above notebook especially in the encoder and the decoder?


title: Incorrect output from Convolutional Neural Network (CNN) "Acceleration Data to Delta Velocity"

body:
I am trying to calculate delta-V from acceleration data input with CNN. But the answer has been incorrect. I have tried different learning rates, layers, and Activation methods (ReLU, ELU, Tanh) but none of them give me correct answer.
The code and data samples are in the following link:https://github.com/Jihang/CNN
Really appreciate if you can help!


title: How to prevent the initial pytorch variable from changing using a function?

body:
I want to apply a function to the variable x and saved as y. But why the x is also changed? How to prevent it?
import torch
def minus_min(raw):
  for col_i in range(len(raw[0])):
    new=raw
    new[:,col_i] = (raw[:,col_i] - raw[:,col_i].min())
  return new

x=torch.tensor([[0,1,2,3,4],
                [2,3,4,0,8],
                [0,1,2,3,4]])
y=minus_min(x)
print(y)
print(x)

output:
tensor([[0, 0, 0, 3, 0],
        [2, 2, 2, 0, 4],
        [0, 0, 0, 3, 0]])
tensor([[0, 0, 0, 3, 0],
        [2, 2, 2, 0, 4],
        [0, 0, 0, 3, 0]])


title: Any compelling reason not to calculate Non-Maximal Suppression (NMS) in half precision?

body: Looking at Non-Maximal Suppression codes in a number of different Deep Learning architectures, and I notice that when using automatic mixed precision (keras, pytorch), it seems these are computed in float32. If we are using NMS for boxes in an object detector (e.g. YOLO), the inputs are pixel coordinates, so I feel it should be totally legit to use half precision in these calculations (especially if these are just for the detection stage and we use no gradients). Anyone find otherwise, i.e., that you really do need to use float32?


title: tensor-board inconsistent line graph colour

body: I'm using tensorboard to analyze a number of experiments. Each time i am comparing two experiments. Tensorboard seems to assign orange and blue to each experiment. However for some reason when i compare different colour the order of the colour is assigned change so If A was blue and B orange. some experiment seems to chagne A to orange to B to Blue. I would like to keep the same colour so it easy to refer to them in my re-writing up. I have tried to rename the event files in order to change the colour but that didn’t work. Is there any way to change the order of the way tensorboard assign the colour to the graphs? or how does assign the colour? I hope this makes sense. Thanks in advance


title: How to get the information like the output of Model.get_config() in keras?

body:
I want to get some information like this.
The input layer of one layer and the output layer of one layer.


title: new to OOP confused about declaring variables

body:
building a neural network on pytorch and trying to get the fully connected layer to accept the dimension that comes out of a flatten layer i call before it.
so far i'm doing the following:
from torch import nn
from torch.nn import functional as F

class my_model(nn.Module):
    def __init__(self, channel=16, out_class=1, flat_dim=24):
        super(my_model, self).__init__()

        self.channel = channel
        n_class = out_class

        self.conv1 = nn.Conv3d(1, channel // 8, kernel_size=5, stride=2, padding=1)
        self.pool1 = nn.AvgPool3d(2)
        self.bn1 = nn.BatchNorm3d(channel //8)
        self.conv2 = nn.Conv3d(channel // 8, channel // 4, kernel_size=4, stride=2, padding=1)
        self.bn2 = nn.BatchNorm3d(channel // 4)
        self.pool2 = nn.AvgPool3d(2)
        self.fc1 = nn.Linear(flat_dim,500)
        self.fc2 = nn.Linear(500,100)

    def forward(self, x):
        batch_size, D, W, H, T = x.shape
        print('input to model', x.shape)
        x = x.permute(0,4,1,2,3)
        x = x.reshape(batch_size * T, 1, D, W, H)
        print('after reshape', x.shape)
        h1 = F.relu(self.bn1(self.pool1(self.conv1(x))))
        print('after first layer sequence', h1.shape)
        h2 = F.relu(self.bn2(self.pool2(self.conv2(h1))))
        print('output to model', h2.shape)
        h2, flat_dim = self.flatten_cube(h2)
        h3 = self.fc1(h2)
        h4 = self.fc2(h3)
        print(h4.shape)

        return h4

    def flatten_cube(self, x):
        super_batch, C, D, W, H = x.shape
        flat_dim = C * D * W * H
        y = x.view(super_batch, flat_dim)
        return y, flat_dim

is this the proper way to do so?


title: Specific Object Classification (from 2000 different objects) from a single-image training on a complete synthetic environment of only 3D Models

body:
I was wondering: in the case we have only 3D models in our Dataset. (for example only the 3d models from ShapeNET, Pix3D, Pascal or others), but with no real pictures of those objects, how can we recognize those models from real images.
Since we need to perform the training on a completely synthetic environment, it will be important to use differentiable rendering techniques. probably in the past we used Blender, but now it could be possible to have everything in the same pipeline, with Tensorflow Graphics, Pytorch3d and others. I was wondering if there is any code sample available regarding this task of specific object classification, single-image model retrieval.
Let's say I have 3 items that I want to 3d print:
Chair Specific Model BXshsda43
Table Specific Model Anca34
Computer Specific Model Calds54

After I 3D print those objects I want to be able to recognize those when I take a picture.
I/O like this:
Training Dataset: Only 3D Models
Inference: Image --> Correct 3D object "reconstruction" or better retrieval.

Is there someone else working on this task?
Is there any good example?

Best Regards
I saw these posts before

Making a trained model (machine learning) from 3D models
3D object pose from single image using object CAD


title: I just want to print out the value from the tensor value

body:
I using pytorch
data = tensor(55.4099, device='cuda:0')

I want to get a separate price like this.
data2 = 55.4099


title: Dynamic Data Iterator Value Change During Training in Pytorch

body:
My Data Loader :
LoadData(Dataset):
    def __init__(self, ..., ...., ...):
        self.ns = (640, 640)
        .....
        .....

   def __getitem__(index):
       img = resize(img[index], self.ns)
       .....
       .....
       return img

   def set_size(self, ns):
       self.ns = ns

And the training loop
for img in dataiterator:
    forward(img)
    backward()
    datatiterator.dataset.set_size(new_ns)

I would like to resize images dynamically after each iteration. When I do it as above, it does not work. Is there any simple trick to do that? Please help.
Best,


title: can i provide explicit gradients/hessians to pytorch/keras to accelerate learning?

body: Same as questions. I think my current pytorch/keras does not train mse/ other custom loss function well. Is there a way that I can input gradient/hessian (diagonal) explicitly. To make it train better?


title: Explanation needed on PyTorch 2D convolutional layer

body:
I need explanation on what exaclty goes on in a Conv2d operation in PyTorch.
If I run the following:
import torch
import torch.nn.functional as F

torch.manual_seed(0)

x = torch.randint(20, (1,3,4,4));
y = torch.randint(2, (1,3,2,2));
z = F.conv2d(x,y, stride=1);

print(f"Input: (shape={x.shape})\n")
print(x)
print("")
print(f"Filter/convolutional kernel: (shape={y.shape})\n")
print(y)
print("")
print(f"Feature Map: (shape={z.shape})\n")
print(z)

I get:

Shouldn't the top left entry in the feature map be:
19+19+6+16+8+18+17=103?
Thanks


title: Pytorch tensor indexing clarification

body:
Consider the following:
a = torch.rand(2, 3, 4)

tensor([[[0.2410, 0.3700, 0.9221, 0.5289],
         [0.8820, 0.2856, 0.4072, 0.6177],
         [0.4279, 0.9396, 0.7483, 0.0087]],

        [[0.7295, 0.4965, 0.9559, 0.0419],
         [0.7379, 0.5761, 0.3439, 0.8682],
         [0.3886, 0.2435, 0.4024, 0.2007]]])

a[[0], [0,2]][..., [0,3]]

tensor([[0.2410, 0.5289],
        [0.4279, 0.0087]])

So far so good but when I try to select the same subset directly on a I'm getting (for me) unexpected results.
a[[0], [0,2], [0,3]]

tensor([0.2410, 0.0087])

I tried to find an explanation to what exactly happening in the last case but didn't find any. I'll be happy to understand what is the semantic of this last form of indexing.
Thank you!


title: When creating a dataset to perform a NER task, should I label the entities which appear more than once?

body:
Suppose I want to use a BERT model, which takes in account the context of the words depending on the sentence, to perform a NER task.
Suppose I have the following sentence in my training dataset:

"I am able to use both Java and Python very well, however I am more
proficient using Java"

The label I want to predict is "hard skill".
In this sentence both Java and Python are hard skills, so I should associate to these the hard skill label.
However Java is repeated twice and for both occurrencies it has the same meaning.
We are not in the case like

"After stealing money from the bank vault, the bank robber was seen "

"fishing on the Mississippi river bank"

where bank has different meanings.
In my case I am not sure whether to label both "Java" words or just one of them, to obtain better performance.


title: Large Neural Network Pruning

body:
I have done some experiments on neural network pruning, but only on small models. I used to prune the relevant weights as follows (similarly as it is explained in the official tutorial https://pytorch.org/tutorials/intermediate/pruning_tutorial.html):
for name,module in model.named_modules():
    if 'layer' in name:
        parameters_to_prune.append((getattr(model, name),'weight'))

prune.global_unstructured(
    parameters_to_prune,
    pruning_method=prune.L1Unstructured,
    amount=sparsity_constant,
    )

The main problem in doing this, is that I have to define a list (or tuple) of layers to prune. This works when I define my model by hands and I know the name of different layers (for example, in the code provided, I was aware of the fact that all the fully connected layers, had the string "layer" in their name.
How can I avoid this process, and define a pruning method that prunes all the parameters of a given model, without having to call the layers by name?
All in all, I'm looking for a function that, given a model and a constant of sparsity, globally prunes the given model (by masking it):
model = models.ResNet18()
function_that_prunes(model, sparsity_constant)


title: How to check if a small tensor is inside anthor large tensor

body:
a = torch.tensor([[1,1],[2,2]])

I want to know if tensor([1,1]) is inside in a (return one bool)
a.eq(torch.tensor([1,1]))
tensor([[ True, True],
[False, False]])

–> which should return a True to my case.
a.eq(torch.tensor([1,2]))
tensor([[ True, False],
[False, True]])

–> which should return a false to my case.
Any suggestion?


title: Why is the perceptron trick not working in Udacity "Intro to deep learning with pytorch"?

body: I'm working on neural networks course from udacity. I got lost in this section. I did the math and feel like the trick is not working for me. Of course I should be doing something wrong. Can someone tell me what I'm doing wrong here https://www.youtube.com/watch?v=lif_qPmXvWA&feature=emb_logo


title: Output and Broadcast shape mismatch in MNIST, torchvision and getting dead kernel if i fix it by changing transform xx

body:
Please find my relevant code here
batchSize=1

transform = transforms.Compose([transforms.ToTensor(),
                            transforms.Normalize((0.5,), (0.5,))])

If I am using the transforms.Normalize as transforms.Normalize(0.5,),(0.5) i.e. using only the mean and as a tuple,  the kernel is getting dead
if I am using it As transforms.Normalize(0.5,0.5,0.5),(0.5,0.5,0.5) then it is throwing runtime error of output mismatch.


title: Changing batch,height,width,alpha to batch,alpha,height,width for pytorch

body:
I have batch of images like
torch.Size([10, 512, 512, 3])
I can loop to the images and can see 10 images. But to feed this batch to pytorch i have to convert it to
torch.Size([10, 3, 512, 512])
I tried lot of ways but unable to get the solution for this
How can we do that ?


title: How to create a MICR font dataset in Python for Custom model?

body:
I am working on some project for few days. In which I want to train a model that can detect MICR font that are written on the Bottom of the bank cheques. I have tried the Non-Model method, but I want to do it with a very efficient model. I would like to have some help and some materials too for my work.
Any thoughts ?


title: What is the difference between a pytorch model and its executable version?

body: I have converted my pytorch model and converted into an executable by using torch.jit.trace. I noticed that for same input, the intermediate outputs of the model and executable are different. Furthermore, I created two different executable models, one with single return value, and the other with multiple return values. Surprisingly, intermediate outputs are different for both executables too. What can be the reason behind this behavior?


title: Equivalent of PyTorch's LPPool1d in TF2.0

body:
Is there a built-in TF2.0 (or tf.keras) equivalent of PyTorch's nn.LPPool1d?

Applies a 1D power-average pooling over an input signal composed of several input planes.

Defined as:


title: tensor dimension ordering for efficient memory access

body:
I have a question about memory layout and access efficiency.
I will explain about the situation and let me clarify the question.
I have certain amount of nodes(points, vertices). Each point has its own embedding of specific length(like, 3~1024).
So, this x's dimension is [batch_size, embedding length, numbers of points];
(By the way, My batch size is always '1'.)
And i have an array which consists of random indices whose name is idx_p0 (like, [0, 71, 234, 22, 0 ..., 2].
Using this idx_p0, i do random access to this x
y = x[:,:,idx_p0]
Here comes the question.
As, i do random access to the x. Then i will cause memory access(toward register file of gpu).
But i want to do this in efficient way.
Is it wise to change the dimension of the x to this 'x_bar = x.transpose(2,1).contiguous()'.
Then the 'x_bar' becomes [batch_size, numbers of points, embedding length]. And do random access like this 'y = x[:,idx_p0,:]'.
I want to know how the pytorch save the 3 dimension tensor.
Oh! my batch_size is usually only 1.
Thank you.


title: Unsafely overlapping signatures with mypy and pytorch

body:
I've run into an issue using mypy and PyTorch and I'm not sure if it's a bug or something I don't understand. In this example, I am trying to write __mul__ and __rmul__ for a custom class that can be multiplied by a tensor. However, when I write the signature of __rmul__ I get an error warning from mypy: Signatures of "__rmul__" of "TestClass" and "__mul__" of "Tensor" are unsafely overlapping. Here is my MWE that shows this error:
from torch import Tensor

class TestClass:
    def __rmul__(self, other: Tensor) -> "TestClass"
        ...

My best guess is that Tensor.__mul__ has a signature of __mul__(self, other) -> Tensor and other is implicitly defined as Any. As a result, mypy can't tell whether tensor_var * test_class_var will result in an object of type Tensor or TestClass, even though I know that it will result in TestClass. Is this interpretation correct? If so, is there any way to suppress this output, or inform mypy that Tensor.__mul__ does not, in fact, work with instances of TestClass?
Interestingly, this only happens with __rmul__. Defining TestClass.__mul__ does not cause this issue.


title: How to call each other between Python and C++

body:
PyTorch calls programs written in C++ that cannot be called. The following error is displayed.
ModuleNotFoundError: No module named 'soft_renderer.cuda.load_textures'.

Here is the program written in C++：
#include <torch/torch.h>
// CUDA forward declarations

at::Tensor load_textures_cuda(`enter code here`
        at::Tensor image,
        at::Tensor faces,
        at::Tensor textures,
        at::Tensor is_update);

// C++ interface

#define CHECK_CUDA(x) AT_CHECK(x.type().is_cuda(), #x " must be a CUDA tensor")
#define CHECK_CONTIGUOUS(x) AT_CHECK(x.is_contiguous(), #x " must be contiguous")
#define CHECK_INPUT(x) CHECK_CUDA(x); CHECK_CONTIGUOUS(x)

at::Tensor load_textures(
        at::Tensor image,
        at::Tensor faces,
        at::Tensor textures,
        at::Tensor is_update) {

    CHECK_INPUT(image);
    CHECK_INPUT(faces);
    CHECK_INPUT(is_update);
    CHECK_INPUT(textures);

    return load_textures_cuda(image, faces, textures, is_update);

}

//PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
PYBIND11_MODULE(load_textures, m) {
    m.def("load_textures", &load_textures, "LOAD_TEXTURES (CUDA)");
}


title: parallel two encoder in Pytorch

body:
I have an encoder class that is defined in a class in PyTorch. I want to use this for two parallel models as follows.  could you help me to solve that?
class  Encoder(nn.Module):
....```

model 1
optimizer        = torch.optim.Adam(model1.parameters(), lr=6e-4)
losses = train(model1, device, 100, data1, optimizer)

model2
model2           = Encoder(100,  1,   2,  10, device).to(device)
optimizer        = torch.optim.Adam(model2.parameters(), lr=6e-4)
losses = train(model2, device, 100, data2, optimizer)

Here is the figure:
https://imgur.com/a/VB3RhKT


title: PyTorch tensors element differ from numpy arrays after conversion

body:
I'm working on a metric and i want the numbers to be identically the same, for example:
torch.manual_seed(12)
a = torch.tensor(size=(10,1), dtype=torch.double)
b = a.numpy()

a
tensor([[0.8919],
        [0.9859],
        [0.6348],
        [0.8811],
        [0.9391],
        [0.1173],
        [0.1342],
        [0.9405],
        [0.6803],
        [0.5556]], dtype=torch.float64)

b
array([[0.89187783],
       [0.98594254],
       [0.63482434],
       [0.88114518],
       [0.93908453],
       [0.11732954],
       [0.1342178 ],
       [0.94047487],
       [0.68025023],
       [0.55559111]])

I want to avoid this difference in the values, how can i do that?


title: Pytorch1.6 What is the actual learning rate during training?

body:
I'd like to know the actual learning rate during training, here is my code.
learning_rate = 0.001
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[1, 2], gamma=0.1)

def train(epoch):
    train_loss = 0

    for batch_idx, (input, target) in enumerate(train_loader):
        predict_label = net(input)
        loss = criterion(predict_label, target)
        train_loss += loss.item()

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print(optimizer.param_groups[0]['lr'])

    scheduler.step()
    print(scheduler.state_dict()['_last_lr'])
    print(optimizer.param_groups[0]['lr'])

the output is 0.001, 0.0001, 0.0001. So what is the actual lr during optimizer.step()? 0.001 or 0.0001? Thanks.


title: Win Error 5 has occurred when I tried to execute Pytorch code

body:
I'm new to Anaconda. I tried to execute Pytorch Adversarial Neural Network in anaconda. It shows some error that I have no clue. Here is the code that downloads dataset
# MNIST Test dataset and dataloader declaration
test_loader = torch.utils.data.DataLoader(datasets.MNIST('../data', train=False, download=True, transform=transforms.Compose([
            transforms.ToTensor(),])),batch_size=1, shuffle=True)

This is the error message I got :
     PermissionError                           Traceback (most recent call last)
     <ipython-input-4-59310f6a37f8> in <module>
 41
 42 # MNIST Test dataset and dataloader declaration
 43 test_loader = torch.utils.data.DataLoader(datasets.MNIST('../data', train=False, download=True, transform=transforms.Compose([transforms.ToTensor(),])),batch_size=1, shuffle=True)
 44
 45 # Define what device we are using

     ~\anaconda3\lib\site-packages\torchvision\datasets\mnist.py in __init__(self, root,
     train,
 transform, target_transform, download)
 77
 78         if download:
 79             self.download()
 80
 81         if not self._check_exists():

        ~\anaconda3\lib\site-packages\torchvision\datasets\mnist.py in download(self)
138             return
139
140         os.makedirs(self.raw_folder, exist_ok=True)
141         os.makedirs(self.processed_folder, exist_ok=True)
142

     ~\anaconda3\lib\os.py in makedirs(name, mode, exist_ok)
211     if head and tail and not path.exists(head):
212         try:
213             makedirs(head, exist_ok=exist_ok)
214         except FileExistsError:
215             # Defeats race condition when another thread created the path

      ~\anaconda3\lib\os.py in makedirs(name, mode, exist_ok)
211     if head and tail and not path.exists(head):
212         try:
213             makedirs(head, exist_ok=exist_ok)
214         except FileExistsError:
215             # Defeats race condition when another thread created the path

    ~\anaconda3\lib\os.py in makedirs(name, mode, exist_ok)
221             return
222     try:
223         mkdir(name, mode)
224     except OSError:
225         # Cannot rely on checking for EEXIST, since the operating system

PermissionError: [WinError 5] Access is denied: '../data' .


title: Actor-Critic Model: How to mach the sizes between model and the action batch

body:
I am new to PyTorch and Reinforcement Learning and because of that sorry if this message will sound stupid or the solution too simple but I have no idea how to fix this problem and I've spent already a few days
researching this and trying to find a way to solve this and I couldn't. I would really appreciate it if any of you could help me with this or at least give me some advice.
I am trying to build a model that is going to buy and sell stocks on the market, this model is going to have just 2 actions possible which are BUY and SELL.
Also, I am trying to implement an Actor-Critic model using 2 interconnected GRU models for the Actor and just some simple linear layers for the Critic model connected because I am trying to see how better or not is in comparison with a normal model in my case.

class ActorNN(nn.Module):
    def __init__(self, stock_env: StockEnv, conf: wandb):
        super(ActorNN, self).__init__()
        self.stock_env = stock_env
        self.input_size = 20
        self.hidden_size_1 = 235
        self.hidden_size_2 = 135
        self.num_layers_1 = 2
        self.num_layers_2 = 3
        self.batch_size = 350

        output_size = self.stock_env.action_space.n # (2 - BUY, SELL)
        self.lstm = nn.GRU(input_size=self.input_size, hidden_size=self.hidden_size_1,
                           num_layers=self.num_layers_1, dropout=conf.dropout_1)
        self.lstm_2 = nn.GRU(input_size=self.hidden_size_1, hidden_size=self.hidden_size_2,
                             num_layers=self.num_layers_2, dropout=conf.dropout_2)
        self.output_layer = nn.Linear(self.hidden_size_2, output_size)
        self.activation = nn.Tanh()

    def forward(self, x):
        x = self.activation(x.view(len(x), -1, self.input_size))
        out, new_hidden_1 = self.lstm(x)
        out = self.activation(new_hidden_1)
        out, _ = self.lstm_2(out)
        out = self.activation(out)
        out = self.output_layer(out)
        return out

class CriticNN(nn.Module):
    def __init__(self, stock_env: StockEnv):
        # stock_env.window_size = 300
        super(CriticNN, self).__init__()
        self.stock_env = stock_env
        self.l1 = nn.Linear(stock_env.window_size * 25, 128)
        self.l2 = nn.Linear(128, 256)
        self.l3 = nn.Linear(256, 1)
        self.activation = nn.ReLU()

    def forward(self, x):
        output = self.activation(self.l1(torch.flatten(x, start_dim=1)))
        output = self.activation(self.l2(output))
        output = self.l3(output)
        return output

Now my problem rise on the optimization function from the agent

def optimize(self):
    if len(self.memory) < self.config.batch_size:
        return

    state, action, new_state, reward, done = self.memory.sample(batch_size=self.config.batch_size)

    state = torch.Tensor(np.array(state)).to(device)
    new_state = torch.Tensor(np.array(new_state)).to(device)
    reward = torch.Tensor(reward).to(device)
    action = torch.LongTensor(action).to(device)
    done = torch.Tensor(done).to(device)
    dist = torch.distributions.Categorical(self.actor(state))
    advantage = reward + (1 - done) * self.config.gamma * self.critic(new_state).squeeze(1) - self.critic(state).squeeze(1)

    critic_loss = advantage.pow(2).mean()
    self.optimizer_critic.zero_grad()
    critic_loss.backward()
    self.optimizer_critic.step()

    actor_loss = -dist.log_prob(action) * advantage.detach()
    self.optimizer_actor.zero_grad()
    actor_loss.mean().backward()
    self.optimizer_actor.step()

When I am trying to initialize my dist variable it will be the shape [3(hidden layer), 300(windows_size), 2(nr of actions)] and my action is of shape [350 (batch_size)]
Now when I am trying to run dist.log_prob(action) I am getting an error that says:

The size of tensor a (350) must match the size of tensor b(300) at non-singleton dimension 1

This is because my dist is not the same shape as action, and here comes my question, how can I make them match? Can any of you help me with this? I've tried to use
multiple Linear layers to match their size but I couldn't make them math.


title: Image Classification model for two datasets

body:
My aim is to train a Convolutional Network model with two datasets:
1- Real Images
2- Fake images
I want to train a model that takes Images from dataset A as label(real) and from dataset B as label(fake).
In this way, if I need to test some images. It should provide me the label, if that image belongs to dataset A or B.
I would like some help on it.


title: PyTorch: LSTM error while trying to update the hidden state

body:
I am trying to train an LSTM while keeping its hidden state (LSTM stateful) until the moment when I am going to start a new epoch(episode). But here it's come an interesting situation because I am getting the following error while trying to do so:

RuntimeError: one of the variables needed for gradient computation has
been modified by an inplace operation: [torch.cuda.FloatTensor [705,
25]] is at version 3; expected version 2 instead. Hint: the backtrace
further above shows the operation that failed to compute its gradient.
The variable in question was changed in there or anywhere later. Good
luck!

And this is happening just if I am trying to keep my hidden state if, for example, I will use a thin line instead:
dist, _ = self.actor(state, h_out) and also removing retain_graph=True everything is going to work just fine.
Could any of you help me to understand what is going on here and how can I fix this, please?
I have my training loop here:
for ep in range(conf.num_episode):
    state = env.reset()
    step = 0

    qnet_agent.hidden = None
    qnet_agent.hidden_2 = None
    while True:
        step += 1
        frames_total += 1

        epsilon = calculate_epsilon(frames_total)

        action, smart_decision = qnet_agent.select_action(state, epsilon)

        new_state, reward, done, info = env.step(action)

        memory.push(state, action, new_state, reward, done)

        qnet_agent.optimize()
        state = new_state

        if done:
            steps_total.append(step)
            break

And here is my optimize function:
 def optimize(self):
    if len(self.memory) < self.config.batch_size:
        return

    state, action, new_state, reward, done = self.memory.sample(batch_size=self.config.batch_size)

    state = torch.Tensor(np.array(state)).to(device)
    new_state = torch.Tensor(np.array(new_state)).to(device)
    reward = torch.Tensor(reward).to(device)
    action = torch.LongTensor(action).to(device)
    done = torch.Tensor(done).to(device)

    h_out = self.hidden
    dist, self.hidden = self.actor(state, h_out)
    dist = torch.distributions.Categorical(dist)

    advantage = reward + (1 - done) * self.config.gamma * self.critic(new_state).squeeze(1) - self.critic(state).squeeze(1)

    critic_loss = advantage.pow(2).mean()
    self.optimizer_critic.zero_grad()
    critic_loss.backward()
    self.optimizer_critic.step()

    actor_loss = -dist.log_prob(action) * advantage.detach()
    self.optimizer_actor.zero_grad()
    actor_loss.mean().backward(retain_graph=True)
    self.optimizer_actor.step()