title: RuntimeError: Expected hidden[0] size (2, 1, 512), got [2, 128, 512] - Seq2Seq Model with PreTrained BERT Model

body:
I am trying to build a seq2seq model with pretraind bert model in pytorch.
During training my model I received the following run time error:
RuntimeError                              Traceback (most recent call last)

<ipython-input-63-472071541d41> in <module>()
      8     start_time = time.time()
      9
---> 10     train_loss = train(model, train_iterator, optimizer, criterion, CLIP)
     11     valid_loss = evaluate(model, valid_iterator, criterion)
     12

8 frames

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in check_hidden_size(self, hx, expected_hidden_size, msg)
    221                           msg: str = 'Expected hidden size {}, got {}') -> None:
    222         if hx.size() != expected_hidden_size:
--> 223             raise RuntimeError(msg.format(expected_hidden_size, list(hx.size())))
    224
    225     def check_forward_args(self, input: Tensor, hidden: Tensor, batch_sizes: Optional[Tensor]):

RuntimeError: Expected hidden[0] size (2, 1, 512), got [2, 128, 512]

I am not quite sure why this issue is coming while the expected dimension of the hidden layer is :
[n layers * n directions, batch size, hid dim]

And a batch size of 128 has been passed like below:
from torchtext.legacy.data import BucketIterator,TabularDataset

BATCH_SIZE = 128

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator = data.BucketIterator.splits(
    (train_data, valid_data),
    batch_size = BATCH_SIZE,
    device = device)

Not sure why during the training it is expecting the batch size to be 1?
Kindly help in resolving the error.
The detailed code can also be available here for your reference:
https://github.com/Ninja16180/BERT/blob/main/Training_Seq2Seq_Model_using_Pre-Trained_BERT_Model.ipynb

Giving below further detailed code snippet:
from torchtext.legacy.data import BucketIterator,TabularDataset

BATCH_SIZE = 128

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

train_iterator, valid_iterator = data.BucketIterator.splits(
    (train_data, valid_data),
    batch_size = BATCH_SIZE,
    device = device)

# Encoder

class Encoder(nn.Module):
    def __init__(self, bert, hid_dim, n_layers, dropout):
        super().__init__()

        self.hid_dim = hid_dim
        self.n_layers = n_layers

        self.bert = bert

        emb_dim = bert.config.to_dict()['hidden_size']

        self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, batch_first = True,dropout = dropout)

        self.dropout = nn.Dropout(dropout)

    def forward(self, sent1):

        #sent1 = [sent1 len, batch size]

        with torch.no_grad():
            embedded = self.bert(sent1)[0]

        #embedded = [sent1 len, batch size, emb dim]

        outputs, (hidden, cell) = self.rnn(embedded)

        #outputs = [sent1 len, batch size, hid dim * n directions]
        #hidden = [n layers * n directions, batch size, hid dim]
        #cell = [n layers * n directions, batch size, hid dim]

        #outputs are always from the top hidden layer

        return hidden, cell

# Decoder

class Decoder(nn.Module):
    def __init__(self, bert, hid_dim, output_dim, n_layers, dropout):
        super().__init__()

        self.output_dim = output_dim
        self.hid_dim = hid_dim
        self.n_layers = n_layers

        self.bert = bert
        emb_dim = bert.config.to_dict()['hidden_size']

        self.rnn = nn.LSTM(emb_dim, hid_dim, n_layers, batch_first = True, dropout = dropout)

        self.fc_out = nn.Linear(hid_dim, output_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, input, hidden, cell):

        #input = [batch size]
        #hidden = [n layers * n directions, batch size, hid dim]
        #cell = [n layers * n directions, batch size, hid dim]

        #n directions in the decoder will both always be 1, therefore:
        #hidden = [n layers, batch size, hid dim]
        #context = [n layers, batch size, hid dim]

        input = input.unsqueeze(0)

        #input = [1, batch size]

        with torch.no_grad():
            embedded = self.bert(input)[0]

        #embedded = [1, batch size, emb dim]

        output, (hidden, cell) = self.rnn(embedded, (hidden, cell))

        #output = [seq len, batch size, hid dim * n directions]
        #hidden = [n layers * n directions, batch size, hid dim]
        #cell = [n layers * n directions, batch size, hid dim]

        #seq len and n directions will always be 1 in the decoder, therefore:
        #output = [1, batch size, hid dim]
        #hidden = [n layers, batch size, hid dim]
        #cell = [n layers, batch size, hid dim]

        prediction = self.fc_out(output.squeeze(0))

        #prediction = [batch size, output dim]

        return prediction, hidden, cell

# Seq2seq

class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder, device):
        super().__init__()

        self.encoder = encoder
        self.decoder = decoder
        self.device = device

        assert encoder.hid_dim == decoder.hid_dim, \
            "Hidden dimensions of encoder and decoder must be equal!"
        assert encoder.n_layers == decoder.n_layers, \
            "Encoder and decoder must have equal number of layers!"

    def forward(self, sent1, sent2, teacher_forcing_ratio = 0.5):

        #sent1 = [sent1 len, batch size]
        #sent2 = [sent2 len, batch size]
        #teacher_forcing_ratio is probability to use teacher forcing
        #e.g. if teacher_forcing_ratio is 0.75 we use ground-truth inputs 75% of the time

        batch_size = sent2.shape[1]
        sent2_len = sent2.shape[0]
        sent2_vocab_size = self.decoder.output_dim

        #tensor to store decoder outputs
        outputs = torch.zeros(sent2_len, batch_size, sent2_vocab_size).to(self.device)

        #last hidden state of the encoder is used as the initial hidden state of the decoder
        hidden, cell = self.encoder(sent1)

        #first input to the decoder is the <sos> tokens
        input = sent2[0,:]

        for t in range(1, sent2_len):

            #insert input token embedding, previous hidden and previous cell states
            #receive output tensor (predictions) and new hidden and cell states
            output, hidden, cell = self.decoder(input, hidden, cell)

            #place predictions in a tensor holding predictions for each token
            outputs[t] = output

            #decide if we are going to use teacher forcing or not
            teacher_force = random.random() < teacher_forcing_ratio

            #get the highest predicted token from our predictions
            top1 = output.argmax(1)

            #if teacher forcing, use actual next token as next input
            #if not, use predicted token
            input = sent2[t] if teacher_force else top1

        return outputs

OUTPUT_DIM = len(sent2.vocab)
ENC_EMB_DIM = 256
DEC_EMB_DIM = 256
HID_DIM = 512
N_LAYERS = 2
ENC_DROPOUT = 0.5
DEC_DROPOUT = 0.5

enc = Encoder(bert, HID_DIM, N_LAYERS, ENC_DROPOUT)
dec = Decoder(bert, HID_DIM, OUTPUT_DIM, N_LAYERS, DEC_DROPOUT)

model = Seq2Seq(enc, dec, device).to(device)

def train(model, iterator, optimizer, criterion, clip):

    model.train()

    epoch_loss = 0

    for i, batch in enumerate(iterator):

        sent1 = batch.sent1
        sent2 = batch.sent2

        optimizer.zero_grad()

        output = model(sent1, sent2)

        #sent2 = [sent2 len, batch size]
        #output = [sent2 len, batch size, output dim]

        output_dim = output.shape[-1]

        output = output[1:].view(-1, output_dim)
        sent2 = sent2[1:].view(-1)

        #sent2 = [(sent2 len - 1) * batch size]
        #output = [(sent2 len - 1) * batch size, output dim]

        loss = criterion(output, sent2)

        loss.backward()

        torch.nn.utils.clip_grad_norm_(model.parameters(), clip)

        optimizer.step()

        epoch_loss += loss.item()

    return epoch_loss / len(iterator)

for epoch in range(N_EPOCHS):

    start_time = time.time()

    train_loss = train(model, train_iterator, optimizer, criterion, CLIP)


title: Performance in reading dicom files with SimpleITK and PyTorch

body:
I want to directly load image from memory to python in pytorch tensor format.
I modified GetArrayViewFromImage() function by replacing those lines:
image_memory_view = _GetMemoryViewFromImage(image)
array_view = numpy.asarray(image_memory_view).view(dtype = dtype)

by:
image_memory_view = _GetMemoryViewFromImage(image)
array_view = torch.as_tensor(image_memory_view, dtype = dtype)

in practise it is so slow I replaced it with:
image_memory_view = _GetMemoryViewFromImage(image)
array_view = numpy.asarray(image_memory_view).view(dtype = dtype)
array_view  = torch.as_tensor(array_view)

Now I have two questions:

it is much slower, and I don't really know why reading it with numpy and converting it is faster.
even though I add the dtype argument and it returns a tensor with a correct dtype it reads it wrong (ex. -1000 in numpy is read as 252 no matter what torch.dtype I choose) which is not a problem when reading with numpy and converting, why is that happening?


title: Transfer Learning Segmentation Model Perfoming Significantly Worse on Test Data

body:
I am quite new to the field of semantic segmentation and have recently tried to run the code provided on this paper: Transfer Learning for Brain Tumor Segmentation that was made available on GitHub. It is a semantic segmentation task that uses the BraTS2020 dataset, comprising of 4 modalities, T1, T1ce, T2 and FLAIR. The author utilised a transfer learning approach using Resnet34 weights.
Due to hardware constraints, I had to half the batch size from 24 to 12. However, after training the model, I noticed a significant drop in performance, with the Dice Score (higher is better) of the 3 classes being only around 5-19-11 as opposed to the reported result of 78-87-82 in the paper. The training and validation accuracies however, seem to be performing normally, just that the model does not perform well on test data, I selected the model that was produced before overfitting (validation loss starts increasing but training loss still decreasing) but yielded equally bad results.
So far I have tried:

Decreasing the learning rate from 1e-3 to 1e-4, yielded similar results
Increased the number of batches fed to the model per training epoch to 200 batches per epoch, to match the number of iterations ran in the paper since I effectively halved the batch size - (100 batches per epoch, batch size of 24)

I noticed that image augmentations were applied to the training and validation dataset to increase the robustness of the model training. Do these augmentations need to be performed on the test set in order to make predictions? There are no resizing transforms, transforms that are present are Gaussian Blur and Noise, change in brightness intensity, rotations, elastic deformation, and mirroring, all implemented using the example here.
I'd greatly appreciate help on these questions:

By doubling the number of batches per epoch, it effectively matches the number of iterations performed as in the original paper since the batch size is halved. Is this the correct approach?

Does the test set data need to be augmented similarly to the training data in order to perform predictions? (Note: no resizing transformations were performed)


title: how can I do for Aten operator torch.mv when convert pytprch to onnx

body:
background
torch.mv operator is include in my pytorch model, so when I convert that model to onnx, console print
RuntimeError: Exporting the operator mv to ONNX opset version 11 is not supported. Please feel free to request support or submit a pull request on Pytorch Github

So I have to do something for the torch.mv operator. However I can not understandand the description in the docs of torch.onnx.
question

I can not confirm whether the torch.mv is an aten operator, because I can not find the declaration about the mv function in torch/csrc/autograd/generated/VariableType.h, however somebody says it is. Below, it is the content in VariableType.h.

#pragma once
// @generated from tools/autograd/templates/VariableType.h
#include <ATen/ATen.h>
#include <c10/util/intrusive_ptr.h>
#include <torch/csrc/WindowsTorchApiMacro.h>
#include <cstdint> // for size_t
#include <functional> // for function
#include <memory> // for unique_ptr
#include <string>
#include <vector>
namespace at {
  struct Quantizer;
};
namespace torch { namespace autograd {
using Variable = at::Tensor;
using at::Context;
using at::Device;
using at::Dimname;
using at::DimnameList;
using at::Generator;
using at::IntArrayRef;
using at::MemoryFormat;
using at::QScheme;
using at::Scalar;
using at::ScalarType;
using at::Storage;
using at::Tensor;
using at::TensorList;
using at::TensorOptions;
using at::Quantizer;
// This is temporary typedef to enable Quantizer in aten native function API
// we'll remove them when we are actually exposing Quantizer class
// to frontend
using ConstQuantizerPtr = const c10::intrusive_ptr<Quantizer>&;
using c10::optional;
namespace VariableType {
  TORCH_API std::vector<at::DeprecatedTypeProperties*> allCUDATypes();
  TORCH_API std::vector<at::DeprecatedTypeProperties*> allCPUTypes();
  at::Tensor & unpack(Tensor & t, const char * name, int pos);
  const at::Tensor & unpack(const Tensor & t, const char * name, int pos);
  at::Tensor unpack_opt(const Tensor & t, const char * name, int pos);
  std::vector<at::Tensor> unpack(at::TensorList tl, const char *name, int pos);
};
}} // namespace torch::autograd

In the docs, it take elu for illustration, but elu is standardized in ONNX and mv is not, so how can I do for export the model to onnx including torch.mv
ps: I can not find the contents virtual Tensor elu(const Tensor & input, Scalar alpha, bool inplace) const override; in VariableType.h as that docs says, why? :(

code for reproduce error
from torch.autograd import Variable
import torch

class custom_net(torch.nn.Module):

    def __init__(self,):
        super(custom_net, self).__init__()
        self.sigmoid = torch.nn.Sigmoid()

    def forward(self, martix, vector):
        martix = self.sigmoid(martix)
        vector = self.sigmoid(vector)
        outputs = torch.mv(martix, vector)
        return outputs

if __name__ == "__main__":

    net = custom_net()
    net = net.eval().cuda()
    martix = Variable(torch.randn(2, 3)).type(torch.FloatTensor)
    vector = Variable(torch.randn(3)).type(torch.FloatTensor)

    martix = martix.cuda()
    vector = vector.cuda()

    input_name = ["martix", "vector"]
    output_name = ["output"]

    torch.onnx.export(net, (martix, vector), "net.onnx", input_names=input_name, output_names=output_name,
                      opset_version=11)
    print("done!")


title: How solve the error 'ValueError: cannot reshape array of size 10 into shape (6)'?

body:
I want to train FairMOT
https://github.com/ifzhang/FairMOT
and I have this error, Traceback is this.
Traceback (most recent call last):
  File "train.py", line 98, in <module>
    main(opt)
  File "train.py", line 69, in main
    log_dict_train, _ = trainer.train(epoch, train_loader)
  File "/home/sonic/sw/FairMOT/src/lib/trains/base_trainer.py", line 119, in train
    return self.run_epoch('train', epoch, data_loader)
  File "/home/sonic/sw/FairMOT/src/lib/trains/base_trainer.py", line 62, in run_epoch
    for iter_id, batch in enumerate(data_loader):
  File "/home/sonic/sw/FairMOT2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 819, in __next__
    return self._process_data(data)
  File "/home/sonic/sw/FairMOT2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 846, in _process_data
    data.reraise()
  File "/home/sonic/sw/FairMOT2/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
    raise self.exc_type(msg)
ValueError: Caught ValueError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/home/sonic/sw/FairMOT2/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/sonic/sw/FairMOT2/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/sonic/sw/FairMOT2/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/sonic/sw/FairMOT/src/lib/datasets/dataset/jde.py", line 421, in __getitem__
    imgs, labels, img_path, (input_h, input_w) = self.get_data(img_path, label_path)
  File "/home/sonic/sw/FairMOT/src/lib/datasets/dataset/jde.py", line 188, in get_data
    labels0 = np.loadtxt(label_path, dtype=np.float32).reshape(-1, 6)
ValueError: cannot reshape array of size 10 into shape (6)

the code is
    def get_data(self, img_path, label_path):
        ...
        # Load labels
        if os.path.isfile(label_path):
            labels0 = np.loadtxt(label_path, dtype=np.float32).reshape(-1, 6)

            # Normalized xywh to pixel xyxy format
            labels = labels0.copy()
            labels[:, 2] = ratio * w * (labels0[:, 2] - labels0[:, 4] / 2) + padw
            labels[:, 3] = ratio * h * (labels0[:, 3] - labels0[:, 5] / 2) + padh
            labels[:, 4] = ratio * w * (labels0[:, 2] + labels0[:, 4] / 2) + padw
            labels[:, 5] = ratio * h * (labels0[:, 3] + labels0[:, 5] / 2) + padh
        else:
            labels = np.array([])

        ...

I don't modify the FairMOT git code, I just use my own custom dataset. Thank you.


title: How to combine two image features in a custom pytorch dataset?

body:
I am trying to build a custom pytorch dataset with two images as features and one image as output.
Here is how the images are in the file system:
./feature1/image1.jpeg #64x64px
./feature2/image1.jpeg #64x64px
./output/image1.jpeg #64x64px

My objective is to create a neural network that will take as input
feature1 + feature2 => output.

What is the best way to create such a dataset ?


title: How do I predict using a PyTorch model?

body:
I created a pyTorch Model to classify images.
I saved it once via state_dict and the entire model like that:
torch.save(model.state_dict(), "model1_statedict")
torch.save(model, "model1_complete")

How can i use these models?
I'd like to check them with some images to see if they're good.
I am loading the model with:
model = torch.load(path_model)
model.eval()

This works alright, but i have no idea how to use it to predict on a new picture.


title: How to overwrite checkpoint of same global step for Tensorboard?

body:
If you resume training from a specific epoch (same global step) and run tensorboard, tensorboard (add_scalar) will end up plotting 2 points for that particular global step.
Example, I want trying to test whether changing the learning rate halfway through training will improve/worsen the accuracy:
Plot example
The plots for same time step are plotted twice (I resumed from +15 epochs behind the latest epoch).
Search up the web, cannot find any commands that can ask Tensorboard to just overwrite the previous saved checkpoint for the new one. My expectation is that Tensorboard would know to overwrite the same global step point but it is plotting it together.


title: RuntimeError in Graph Convolutional Network

body:
My objective is to use the edge attributes to train Graph Convolutional Network. An example similar to my task I found here and I use it as a reference.
The problem occurs only when I include the edge_weight as a feature to train the model, e.g. x = self.conv1(x, edge_index, edge_weight) ,as opposed to x = self.conv1(x, edge_index) given in the tutorial.
Minimum reproducible example
import torch
from torch_geometric.datasets import TUDataset

dataset = TUDataset(root='data/TUDataset', name='MUTAG')

print()
print(f'Dataset: {dataset}:')
print('====================')
print(f'Number of graphs: {len(dataset)}')
print(f'Number of features: {dataset.num_features}')
print(f'Number of classes: {dataset.num_classes}')

data = dataset[0]  # Get the first graph object.

print()
print(data)
print('=============================================================')

# Gather some statistics about the first graph.
print(f'Number of nodes: {data.num_nodes}')
print(f'Number of edges: {data.num_edges}')
print(f'Average node degree: {data.num_edges / data.num_nodes:.2f}')
print(f'Contains isolated nodes: {data.contains_isolated_nodes()}')
print(f'Contains self-loops: {data.contains_self_loops()}')
print(f'Is undirected: {data.is_undirected()}')

torch.manual_seed(12345)
dataset = dataset.shuffle()

train_dataset = dataset[:150]
test_dataset = dataset[150:]

print(f'Number of training graphs: {len(train_dataset)}')
print(f'Number of test graphs: {len(test_dataset)}')

from torch_geometric.data import DataLoader

train_loader = DataLoader(train_dataset, batch_size=64, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)

for step, data in enumerate(train_loader):
    print(f'Step {step + 1}:')
    print('=======')
    print(f'Number of graphs in the current batch: {data.num_graphs}')
    print(data)
    print()

from torch.nn import Linear
import torch.nn.functional as F
from torch_geometric.nn import GCNConv
from torch_geometric.nn import global_mean_pool

class GCN(torch.nn.Module):
    def __init__(self, hidden_channels):
        super(GCN, self).__init__()
        torch.manual_seed(12345)
        self.conv1 = GCNConv(dataset.num_node_features, hidden_channels)
        self.conv2 = GCNConv(hidden_channels, hidden_channels)
        self.conv3 = GCNConv(hidden_channels, hidden_channels)
        self.lin = Linear(hidden_channels, dataset.num_classes)

    def forward(self, x, edge_index, batch, edge_weight):
        # 1. Obtain node embeddings
        x = self.conv1(x, edge_index, edge_weight)
        x = x.relu()
        x = self.conv2(x, edge_index, edge_weight)
        x = x.relu()
        x = self.conv3(x, edge_index, edge_weight)

        # 2. Readout layer
        x = global_mean_pool(x, batch)  # [batch_size, hidden_channels]

        # 3. Apply a final classifier
        x = F.dropout(x, p=0.5, training=self.training)
        x = self.lin(x)

        return x

model = GCN(hidden_channels=64)
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
criterion = torch.nn.CrossEntropyLoss()

def train():
    model.train()

    for data in train_loader:  # Iterate in batches over the training dataset.
         out = model(data.x, data.edge_index, data.batch, data.edge_attr)  # Perform a single forward pass.
         loss = criterion(out, data.y)  # Compute the loss.
         loss.backward()  # Derive gradients.
         optimizer.step()  # Update parameters based on gradients.
         optimizer.zero_grad()  # Clear gradients.

def test(loader):
     model.eval()

     correct = 0
     for data in loader:  # Iterate in batches over the training/test dataset.
         out = model(data.x, data.edge_index, data.batch, data.edge_attr)
         pred = out.argmax(dim=1)  # Use the class with highest probability.
         correct += int((pred == data.y).sum())  # Check against ground-truth labels.
     return correct / len(loader.dataset)  # Derive ratio of correct predictions.

for epoch in range(1, 201):
    train()
    train_acc = test(train_loader)
    test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Train Acc: {train_acc:.4f}, Test Acc: {test_acc:.4f}')

and the error I get is:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-4f3bec69ef03> in <module>
     25
     26 for epoch in range(1, 201):
---> 27     train()
     28     train_acc = test(train_loader)
     29     test_acc = test(test_loader)

<ipython-input-12-4f3bec69ef03> in train()
      7
      8     for data in train_loader:  # Iterate in batches over the training dataset.
----> 9          out = model(data.x, data.edge_index, data.batch, data.edge_attr)  # Perform a single forward pass.
     10          loss = criterion(out, data.y)  # Compute the loss.
     11          loss.backward()  # Derive gradients.

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-11-8392f2cf3638> in forward(self, x, edge_index, batch, edge_weight)
     16     def forward(self, x, edge_index, batch, edge_weight):
     17         # 1. Obtain node embeddings
---> 18         x = self.conv1(x, edge_index, edge_weight)
     19         x = x.relu()
     20         x = self.conv2(x, edge_index, edge_weight)

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\anaconda3\lib\site-packages\torch_geometric\nn\conv\gcn_conv.py in forward(self, x, edge_index, edge_weight)
    157                 cache = self._cached_edge_index
    158                 if cache is None:
--> 159                     edge_index, edge_weight = gcn_norm(  # yapf: disable
    160                         edge_index, edge_weight, x.size(self.node_dim),
    161                         self.improved, self.add_self_loops, dtype=x.dtype)

~\anaconda3\lib\site-packages\torch_geometric\nn\conv\gcn_conv.py in gcn_norm(edge_index, edge_weight, num_nodes, improved, add_self_loops, dtype)
     54
     55         if add_self_loops:
---> 56             edge_index, tmp_edge_weight = add_remaining_self_loops(
     57                 edge_index, edge_weight, fill_value, num_nodes)
     58             assert tmp_edge_weight is not None

~\anaconda3\lib\site-packages\torch_geometric\utils\loop.py in add_remaining_self_loops(edge_index, edge_weight, fill_value, num_nodes)
    133         if remaining_edge_weight.numel() > 0:
    134             loop_weight[row[inv_mask]] = remaining_edge_weight
--> 135         edge_weight = torch.cat([edge_weight[mask], loop_weight], dim=0)
    136
    137     return edge_index, edge_weight

RuntimeError: Tensors must have same number of dimensions: got 2 and 1

From the error, I thought that the problem lies in the dimensionalities of x, edge_index, edge_attr, however all of them have the dimensionality of 2.
It is possible to solve this error by including normalize = False in the definitions of layers
        self.conv1 = GCNConv(dataset.num_node_features, hidden_channels, normalize = False)
        self.conv2 = GCNConv(hidden_channels, hidden_channels, normalize = False)
        self.conv3 = GCNConv(hidden_channels, hidden_channels, normalize = False)

however, new error pops up:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-25-4f3bec69ef03> in <module>
     25
     26 for epoch in range(1, 201):
---> 27     train()
     28     train_acc = test(train_loader)
     29     test_acc = test(test_loader)

<ipython-input-25-4f3bec69ef03> in train()
      7
      8     for data in train_loader:  # Iterate in batches over the training dataset.
----> 9          out = model(data.x, data.edge_index, data.batch, data.edge_attr)  # Perform a single forward pass.
     10          loss = criterion(out, data.y)  # Compute the loss.
     11          loss.backward()  # Derive gradients.

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-24-27d853e5b6f9> in forward(self, x, edge_index, batch, edge_weight)
     16     def forward(self, x, edge_index, batch, edge_weight):
     17         # 1. Obtain node embeddings
---> 18         x = self.conv1(x, edge_index, edge_weight)
     19         x = x.relu()
     20         x = self.conv2(x, edge_index, edge_weight)

~\anaconda3\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

~\anaconda3\lib\site-packages\torch_geometric\nn\conv\gcn_conv.py in forward(self, x, edge_index, edge_weight)
    179
    180         # propagate_type: (x: Tensor, edge_weight: OptTensor)
--> 181         out = self.propagate(edge_index, x=x, edge_weight=edge_weight,
    182                              size=None)
    183

~\anaconda3\lib\site-packages\torch_geometric\nn\conv\message_passing.py in propagate(self, edge_index, size, **kwargs)
    235
    236             msg_kwargs = self.inspector.distribute('message', coll_dict)
--> 237             out = self.message(**msg_kwargs)
    238
    239             # For `GNNExplainer`, we require a separate message and aggregate

~\anaconda3\lib\site-packages\torch_geometric\nn\conv\gcn_conv.py in message(self, x_j, edge_weight)
    191             return x_j
    192         else:
--> 193             return edge_weight.view(-1, 1) * x_j
    194
    195     def message_and_aggregate(self, adj_t: SparseTensor, x: Tensor) -> Tensor:

RuntimeError: The size of tensor a (10096) must match the size of tensor b (2524) at non-singleton dimension 0

I guess, it refers again to the edge_index and edge_attr, but I do not understand it.
Could somebody help me to solve it?


title: opennmt cant find GPU

body:
I can see pytorch can find GPU resources.
>>> torch.cuda.is_available()
True
>>> torch.cuda.current_device()
0
>>> torch.cuda.device(0)
<torch.cuda.device object at 0x7f8c0a3cec50>
>>> torch.cuda.device_count()
1
>>> torch.cuda.get_device_name(0)
'NVIDIA Tesla K80'
>>> torch.cuda.get_device_name(0)
'NVIDIA Tesla K80'

However, OpenNMT is not finding the GPU resource. What is going on here?
I am using OpenNMT from https://github.com/OpenNMT/OpenNMT-py/tree/1.2.0
Traceback (most recent call last):
  File "../../OpenNMT-py/train.py", line 6, in <module>
    main()
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/bin/train.py", line 197, in main
    train(opt)
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/bin/train.py", line 91, in train
    p.join()
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/multiprocessing/process.py", line 140, in join
    res = self._popen.wait(timeout)
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/multiprocessing/popen_fork.py", line 48, in wait
    return self.poll(os.WNOHANG if timeout == 0.0 else 0)
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/multiprocessing/popen_fork.py", line 28, in poll
    pid, sts = os.waitpid(self.pid, flag)
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/bin/train.py", line 181, in signal_handler
    raise Exception(msg)
Exception:

-- Tracebacks above this line can probably
                 be ignored --

Traceback (most recent call last):
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/bin/train.py", line 135, in run
    gpu_rank = onmt.utils.distributed.multi_init(opt, device_id)
  File "/root/work/context/huggingface-models/OpenNMT-py/onmt/utils/distributed.py", line 27, in multi_init
    world_size=dist_world_size, rank=opt.gpu_ranks[device_id])
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 510, in init_process_group
    timeout=timeout))
  File "/root/miniconda3/envs/open-nmt-env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 603, in _new_process_group_helper
    timeout)
RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!


title: What is the TensorFlow/Keras equivalent of PyTorch's `no_grad` function?

body:
When writing machine learning models, I find myself needing to compute metrics, or run additional forward-passes in callbacks for visualization purposes. In PyTorch, I do this with torch.no_grad(), and this prevents gradients from being computed and these operations, therefore, do not influence the optimization.

How does this mechanism work in TensorFlow/Keras?
Keras models are callable. So, something like model(x) is possible. But, it is also possible to say model.predict(x), which also seems to invoke the call. Is there a difference between the two?


title: Can writing to Tensorboard cause memory (RAM) OOM issues? (especially in PyTorch)

body:
I am running pytorch code that logs to tensorboard summary writers. But I am also getting memory issues (my jobs are being SIGKILLed). I am trying to figure out why. I am thus curious if it is the part of my code the writes to a tensorboard file? I am also doing other things like caching tensors which might be the source of the issue but I want to isolate this question to one single this:

Does tensorboard write to disk when logging each iteration? (or some other way tensorboard might cause a memory issue e.g. doing tb.add_scalar(tag1, loss, it) or does it keep everything in memory first and that's why maybe my memory error is occurring?

error messages:
From condor HPC:
000 (6466.000.000) 05/03 17:20:00 Job submitted from host: <130.126.112.32:9618?addrs=130.126.112.32-9618+[--1]-9618&noUDP&sock=1423_a35b_4>
...
001 (6466.000.000) 05/03 18:34:52 Job executing on host: <172.22.224.113:9618?addrs=172.22.224.113-9618+[--1]-9618&noUDP&sock=2060_912f_3>
...
006 (6466.000.000) 05/03 18:35:00 Image size of job updated: 13872
10 - MemoryUsage of job (MB)
9564 - ResidentSetSize of job (KB)
...
006 (6466.000.000) 05/03 18:40:00 Image size of job updated: 53368188
51096 - MemoryUsage of job (MB)
52322148 - ResidentSetSize of job (KB)
...
006 (6466.000.000) 05/03 18:45:01 Image size of job updated: 86845436
83789 - MemoryUsage of job (MB)
85799388 - ResidentSetSize of job (KB)
...
006 (6466.000.000) 05/03 18:50:02 Image size of job updated: 120496460
116651 - MemoryUsage of job (MB)
119450412 - ResidentSetSize of job (KB)
...
006 (6466.000.000) 05/03 18:55:02 Image size of job updated: 129339104
123232 - MemoryUsage of job (MB)
126189336 - ResidentSetSize of job (KB)
...
005 (6466.000.000) 05/03 18:56:10 Job terminated.
(1) Normal termination (return value 0)
Usr 0 00:57:39, Sys 0 00:04:24 - Run Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Run Local Usage
Usr 0 00:57:39, Sys 0 00:04:24 - Total Remote Usage
Usr 0 00:00:00, Sys 0 00:00:00 - Total Local Usage
0 - Run Bytes Sent By Job
0 - Run Bytes Received By Job
0 - Total Bytes Sent By Job
0 - Total Bytes Received By Job
Partitionable Resources : Usage Request Allocated
Cpus : 8 8
Disk (KB) : 2 2 1252638
Gpus : 4 4
Memory (MB) : 123232 1 1
...

from my code:
I get the following error:

Starting training...

n_epoch=0
Traceback (most recent call last):
File "/home/miranda9/ML4Coq/ml4coq-proj-src/embeddings_zoo/tree_nns/main_brando.py", line 282, in <module>
main_distributed()
File "/home/miranda9/ML4Coq/ml4coq-proj-src/embeddings_zoo/tree_nns/main_brando.py", line 193, in main_distributed
mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
File "/home/miranda9/miniconda3/envs/coq-tactician-graph/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
File "/home/miranda9/miniconda3/envs/coq-tactician-graph/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
while not context.join():
File "/home/miranda9/miniconda3/envs/coq-tactician-graph/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 105, in join
raise Exception(
Exception: process 1 terminated with signal SIGKILL
JOB ENDED

edit:
Training code as requested by the commentator:
    def train(self, n_epoch):
        self.tactic_predictor.train()
        avg_loss = AverageMeter('train loss')
        avg_acc = AverageMeter('train accuracy')

        bar = ProgressBar(max_value=len(self.dataloaders['train']))
        self.log(f"{len(self.dataloaders['train'].dataset)=}")
        self.log(f"{len(self.dataloaders['train'])=}")
        self.log(f"{len(self.dataloaders['train'].dataset)//self.opts.batch_size=}")
        self.log(f"{len(self.dataloaders['train'])*self.opts.world_size=}")
        for i, data_batch in enumerate(self.dataloaders['train']):
            # self.log(f'--- {i=}')
            # self.log(f'{data_batch[0][0].f}')
            data_batch = process_batch_ddp_tactic_prediction(self.opts, data_batch)
            # self.log(f'{type(data_batch)=}')
            loss, logits = self.tactic_predictor(data_batch)
            # self.log(f'{loss=}')
            acc = accuracy(output=logits, target=data_batch['tac_label'])
            avg_loss.update(loss.item(), self.opts.batch_size)
            avg_acc.update(acc.item(), self.opts.batch_size)

            self.optimizer.zero_grad()
            # self.log(f'done optimizer.zero_grad()')
            loss.backward()  # each process synchronizes it's gradients in the backward pass
            # self.log(f'done loss.backward()')
            self.optimizer.step()  # the right update is done since all procs have the right synced grads
            # self.log(f'done optimizer.step()')

            if self.is_lead_worker() and i % 10 == 0:
                bar.update(i) if self.is_lead_worker() else None
                val_loss, val_acc = self.valid(n_epoch, val_iterations=1)
                self.log_tb(it=i, tag1='train loss', loss=loss.item(), tag2='train acc', acc=acc.item())
                self.log_tb(it=i, tag1='val loss', loss=val_loss.item(), tag2='val acc', acc=val_acc.item())

                self.log(f"\n{i=}: {loss=} {acc=}")
                self.log(f"{i=}: {val_loss=} {val_acc=}")

            gc.collect()
            # self.log(f'just did gc.collect()')

        return avg_loss.item(), avg_acc.item()


title: How to save training weight checkpoint of model and continue training from last point in PyTorch?

body:
I'm trying to save checkpoint weights of the trained model after a certain number of epochs and continue to train from that last checkpoint to another number of epochs using PyTorch
To achieve this I've written a script like below
To train the model:
def create_model():
  # load model from package
  model = smp.Unet(
      encoder_name="resnet152",        # choose encoder, e.g. mobilenet_v2 or efficientnet-b7
      encoder_weights='imagenet',           # use `imagenet` pre-trained weights for encoder initialization
      in_channels=3,                  # model input channels (1 for gray-scale images, 3 for RGB, etc.)
      classes=2,                      # model output channels (number of classes in your dataset)
  )
  return model

model = create_model()
model.to(device)
learning_rate = 1e-3
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
epochs = 5

for epoch in range(epochs):
    print('Epoch: [{}/{}]'.format(epoch+1, epochs))

    # train set
    pbar = tqdm(train_loader)
    model.train()
    iou_logger = iouTracker()
    for batch in pbar:
        # load image and mask into device memory
        image = batch['image'].to(device)
        mask = batch['mask'].to(device)

        # pass images into model
        pred = model(image)
        # pred = checkpoint['model_state_dict']

        # get loss
        loss = criteria(pred, mask)

        # update the model
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # compute and display progress
        iou_logger.update(pred, mask)
        mIoU = iou_logger.get_mean()
        pbar.set_description('Loss: {0:1.4f} | mIoU {1:1.4f}'.format(loss.item(), mIoU))

    # development set
    pbar = tqdm(development_loader)

    model.eval()
    iou_logger = iouTracker()
    with torch.no_grad():
        for batch in pbar:
            # load image and mask into device memory
            image = batch['image'].to(device)
            mask = batch['mask'].to(device)

            # pass images into model
            pred = model(image)

            # get loss
            loss = criteria(pred, mask)

            # compute and display progress
            iou_logger.update(pred, mask)
            mIoU = iou_logger.get_mean()
            pbar.set_description('Loss: {0:1.4f} | mIoU {1:1.4f}'.format(loss.item(), mIoU))

# save model
torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,}, '/content/drive/MyDrive/checkpoint.pt')

from this, I can save the model checkpoint file as checkpoint.pt for 5 epochs
To continue the training using the saved checkpoint weight file for another I wrote below script:
epochs = 5
for epoch in range(epochs):
    print('Epoch: [{}/{}]'.format(epoch+1, epochs))

    # train set
    pbar = tqdm(train_loader)

    checkpoint = torch.load( '/content/drive/MyDrive/checkpoint.pt')
    print(checkpoint)

    model.load_state_dict(checkpoint['model_state_dict'])
    model.to(device)

    optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
    epoch = checkpoint['epoch']
    loss = checkpoint['loss']
    model.train()
    iou_logger = iouTracker()
    for batch in pbar:
        # load image and mask into device memory
        image = batch['image'].to(device)
        mask = batch['mask'].to(device)

        # pass images into model
        pred = model(image)
        # pred = checkpoint['model_state_dict']

        # get loss
        loss = criteria(pred, mask)

        # update the model
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # compute and display progress
        iou_logger.update(pred, mask)
        mIoU = iou_logger.get_mean()
        pbar.set_description('Loss: {0:1.4f} | mIoU {1:1.4f}'.format(loss.item(), mIoU))

    # development set
    pbar = tqdm(development_loader)

    model.eval()
    iou_logger = iouTracker()
    with torch.no_grad():
        for batch in pbar:
            # load image and mask into device memory
            image = batch['image'].to(device)
            mask = batch['mask'].to(device)

            # pass images into model
            pred = model(image)

            # get loss
            loss = criteria(pred, mask)

            # compute and display progress
            iou_logger.update(pred, mask)
            mIoU = iou_logger.get_mean()
            pbar.set_description('Loss: {0:1.4f} | mIoU {1:1.4f}'.format(loss.item(), mIoU))

# save model
torch.save({
            'epoch': epoch,
            'model_state_dict': model.state_dict(),'optimizer_state_dict': optimizer.state_dict(),
            'loss': loss,}, 'checkpoint.pt')

This throws error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-54f48c10531a> in <module>()

---> 14     model.load_state_dict(checkpoint['model_state_dict'])

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in load_state_dict(self, state_dict, strict)
   1222         if len(error_msgs) > 0:
   1223             raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
-> 1224                                self.__class__.__name__, "\n\t".join(error_msgs)))
   1225         return _IncompatibleKeys(missing_keys, unexpected_keys)
   1226

RuntimeError: Error(s) in loading state_dict for DataParallel:
    Missing key(s) in state_dict: "module.encoder.conv1.weight", "module.encoder.bn1.weight", "module.encoder.bn1.bias", "module.encoder.bn1.running_mean", "module.encoder.bn1.running_var", "module.encoder.layer1.0.conv1.weight", "module.encoder.layer1.0.bn1.weight", "module.encoder.layer1.0.bn1.bias", "module.encoder.layer1.0.bn1.running_mean", "module.encoder.layer1.0.bn1.running_var", "module.encoder.layer1.0.conv2.weight", "module.encoder.layer1.0.bn2.weight", "module.encoder.layer1.0.bn2.bias", "module.encoder.layer1.0.bn2.running_mean", "module.encoder.layer1.0.bn2.running_var", "module.encoder.layer1.0.conv3.weight", "module.encoder.layer1.0.bn3.weight", "module.encoder.layer1.0.bn3.bias", "module.encoder.layer1.0.bn3.running_mean", "module.encoder.layer1.0.bn3.running_var", "module.encoder.layer1.0.downsample.0.weight", "module.encoder.layer1.0.downsample.1.weight", "module.encoder.layer1.0.downsample.1.bias", "module.encoder.layer1.0.downsample.1.running_mean", "module.encoder.layer1.0.downsample.1.running_var", "module.encoder.layer1.1.conv1.weight", "module.encoder.layer1.1.bn1.weight", "module.encoder.layer1.1.bn1.bias", "module.encoder.layer1.1.bn1.running_mean", "module.encoder.layer1.1.bn1.running_var", "module.encoder.layer1.1.conv2.weight", "module.encoder.layer1.1.bn2.weight", "module.encoder.layer1.1.bn2.bias", "module.encoder.layer1.1.bn2.running_mean", "module.encoder.layer1.1.bn2.running_var", "module.encoder.layer1.1.conv3.weight", "module.encoder.layer...
    Unexpected key(s) in state_dict: "encoder.conv1.weight", "encoder.bn1.weight", "encoder.bn1.bias", "encoder.bn1.running_mean", "encoder.bn1.running_var", "encoder.bn1.num_batches_tracked", "encoder.layer1.0.conv1.weight", "encoder.layer1.0.bn1.weight", "encoder.layer1.0.bn1.bias", "encoder.layer1.0.bn1.running_mean", "encoder.layer1.0.bn1.running_var", "encoder.layer1.0.bn1.num_batches_tracked", "encoder.layer1.0.conv2.weight", "encoder.layer1.0.bn2.weight", "encoder.layer1.0.bn2.bias", "encoder.layer1.0.bn2.running_mean", "encoder.layer1.0.bn2.running_var", "encoder.layer1.0.bn2.num_batches_tracked", "encoder.layer1.1.conv1.weight", "encoder.layer1.1.bn1.weight", "encoder.layer1.1.bn1.bias", "encoder.layer1.1.bn1.running_mean", "encoder.layer1.1.bn1.running_var", "encoder.layer1.1.bn1.num_batches_tracked", "encoder.layer1.1.conv2.weight", "encoder.layer1.1.bn2.weight", "encoder.layer1.1.bn2.bias", "encoder.layer1.1.bn2.running_mean", "encoder.layer1.1.bn2.running_var", "encoder.layer1.1.bn2.num_batches_tracked", "encoder.layer1.2.conv1.weight", "encoder.layer1.2.bn1.weight", "encoder.layer1.2.bn1.bias", "encoder.layer1.2.bn1.running_mean", "encoder.layer1.2.bn1.running_var", "encoder.layer1.2.bn1.num_batches_tracked", "encoder.layer1.2.conv2.weight", "encoder.layer1.2.bn2.weight", "encoder.layer1.2.bn2.bias", "encoder.layer1.2.bn2.running_mean", "encoder.layer1.2.bn2.running_var", "encoder.layer1.2.bn2.num_batches_tracked", "encoder.layer2.0.conv1.weight", "encoder.layer...

What am I doing wrong? How can I fix this? Any help on this will be helpful.


title: Torch to tensorflow

body:
Is there a way to convert pytorch code to TensorFlow? While I am a little familiar with TensorFlow, totally new to pytorch. For example,
def get_variation_uncertainty(prediction_score_vectors: List[torch.tensor], matrix_size: Tuple) -> Tuple[np.ndarray, np.ndarray, np.ndarray]:

    prediction_score_vectors = torch.stack(tuple(prediction_score_vectors))

    wt_var = np.var(np.sum(prediction_score_vectors[:, :, 1:].cpu().numpy(), axis=2), axis=0).reshape(matrix_size) * 100
    tc_var = np.var(np.sum(prediction_score_vectors[:, :, [1, 3]].cpu().numpy(), axis=2), axis=0).reshape( matrix_size) *100
    et_var = np.var(prediction_score_vectors[:, :, 3].cpu().numpy(), axis=0).reshape(matrix_size) * 100

    return wt_var.astype(np.uint8), tc_var.astype(np.uint8), et_var.astype(np.uint8)

how can I get the TensorFlow equivalent to the above code?


title: Separate optimizer for discriminator and the rest of the model in adversarial setting

body:
I have a model with the following components.

embedding layer
encoder
generator
discriminator
feed-forward neural network

I want to define two optimizers. One for the discriminator only and one for the rest. I am doing the following.

optimizers = []
model_params = chain(model.embedding.parameters(), model.encoder.parameters(),
                            model.generator.parameters(), model.ffnn.parameters())
optimizers.append(optim.Adam(model_params, args.lr))
optimizers.append(optim.Adam(model.discriminator.parameters(), args.lr))

Is there any better way to do the same? For example, can I take the difference between model.parameters() and model.discriminator.parameters()? If yes, how can I do that?


title: How to transform a vector to a matrix with each line equal to the vector in pytorch?

body:
for exemple I have the vector [1, 2, 3]

I want to get a matrix like

[[1, 2, 3],
[1, 2, 3],
[1, 2, 3],
[1, 2, 3]]

How to do that efficiently?


title: How can I make this multiplication much simpler with pytorch?

body:
I want to multiply two matrix like this(scalar multiplication).
But this method looks dumb. How can I make this much simpler?

A.size # B * S * V
B.size # B

A * B.unsqueeze(1).unsqueeze(2).expand_as(A)


title: optimize ram usage in trainning gru

body:
Below code is a nested loop for training a gru in python 2.7; but it is a ram consuming one. feats_tensor and dec_padded_text are too large object and loading them at same time make me face out of memory error. any idea how to optimize this code for ram usage?

  for epoch in xrange(0, 13):
    print ("Starting New Epoch: %d" % epoch)
    np.random.shuffle(order)
    del feats_tensor, dec_text_tensor
    if cuda:
        torch.cuda.empty_cache()
    feats_tensor = torch.tensor(feats[order], requires_grad=False)
    dec_text_tensor = torch.tensor(dec_padded_text[order], requires_grad=False)
    if cuda:
       feats_tensor = feats_tensor.cuda(device=device)
       dec_text_tensor = dec_text_tensor.cuda(device=device)

    for i in xrange(num_batches):
        s = i * BATCH_SIZE
        e = (i+1) * BATCH_SIZE

        enc.zero_grad()
        dec.zero_grad()

        hid_enc = enc.forward(feats_tensor[s:e]).unsqueeze(0)
        out_dec, hid_dec = dec.forward(dec_text_tensor[s:e,:-1], hid_enc)

        out_perm = out_dec.permute(0, 2, 1)
        loss = lossfunc(out_perm, dec_text_tensor[s:e,1:])
        if sm_loss is None:
            sm_loss = loss.data
        else:
            sm_loss = sm_loss*0.95 + 0.05*loss.data

        loss.backward()
        enc_optim.step()
        dec_optim.step()

        if i % 100 == 0:
            print ("Epoch: %.3f" % (i/float(num_batches) + epoch,), "Loss:", sm_loss)
            #print ("GEN:", untokenize(torch.argmax(out_dec,dim=2)[0,:], dec_idx_to_word))
            #print ("GT:", untokenize(dec_text_tensor[s,:], dec_idx_to_word))
            print ("--------------")

    save_state(enc, dec, enc_optim, dec_optim, dec_idx_to_word, dec_word_to_idx, epoch)


title: Changing file that torch is looking at in atom

body:
I am trying to run some deep learning experiments to learn more about the subject and I am using atom as my editor. In the code I am trying to execute, I am loading some dataset from pytorch. For some reason, when I run the code I get the following error:

    File "D:\Julio\Documents\Michigan_v2\CS\EECS_598_Deep_Learning\HW\Homework2\blah.py", line 32, in <module>
    for x in ['train', 'val']}
  File "D:\Julio\Documents\Michigan_v2\CS\EECS_598_Deep_Learning\HW\Homework2\blah.py", line 32, in <dictcomp>
    for x in ['train', 'val']}
  File "C:\Users\Julio\.julia\conda\3\lib\site-packages\torchvision\datasets\folder.py", line 178, in __init__
    target_transform=target_transform)
  File "C:\Users\Julio\.julia\conda\3\lib\site-packages\torchvision\datasets\folder.py", line 75, in __init__
    classes, class_to_idx = find_classes(root)
  File "C:\Users\Julio\.julia\conda\3\lib\site-packages\torchvision\datasets\folder.py", line 23, in find_classes
    classes = [d for d in os.listdir(dir) if os.path.isdir(os.path.join(dir, d))]
FileNotFoundError: [WinError 3] The system cannot find the path specified: 'hymenoptera_data\\train'

It seems to me that atom is looking for the files to load in a julia folder, but I don't know why atom is doing that and I don't know how to change it. Can anybody give me any suggestions on how to fix this?
THanks for your help!


title: python - tensor : access a value

body:
Given below is the output of VGG16 model. The command VGG16.classifier[6] output shows Linear(in_features=25088, out_features=4096, bias=True) I'm not able to understand how this works. ALso,how can I print the values of linear

    VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)


title: Preserve dimension while reshaping

body:
Let's say we have a tensor of size B x C x W x H (as common for batches of images), and we want to reshape it to B x M where M = C*W*H. Is there a built in way to do so without explicitly mentioning B?

If we know B in advance we can do following, even without explicitly knowing any of the three C,W,H:

a = torch.randn(20,3,512,512)
b = a.reshape((20, -1)) #we can use -1 to infer the dimension `M`

But can we also do so without knowing B?

(I know we could obviously find B using B = a.shape[0], but my question is whether it is possible without knowing B either.)


title: Use class from file based of name in string form (Python)

body:
Currently, I am importing every class I have in a python file in my main "Runner" script. The goal is to, based on a string given, instantiate the class that has the same name as the string given.

For example:

If there is a class called 'Test':

class Test():

if the String is "Test", it will call that class.


title: How do I retrieve name of a layer inside hook function?

body:
I have a neural network

class ConvNet(nn.Module):
  def __init__(self):
    super().__init__()
    self.trunk = nn.ModuleList()
    self.trunk.add_module('conv1', nn.Conv2d(3, 10, 3))

    self.classifier = nn.Linear(30, 2)
  def forward(self, x):
    out = self.classifier(self.trunk.conv1(x))
    return out

model = ConvNet()

I registered forward hook

def hook(module, input, output):
    print(module, input[0].shape, output.shape)

x =  model.trunk.conv1.register_forward_hook(hook)

How do I retrieve name of this layer that is 'conv1' inside hook function, module._get_name returns Conv2d, module.__class__ returns <class 'torch.nn.modules.conv.Conv2d'>, how do I get just 'conv1'?


title: Converting matrix inconsistent shape of np.array to torch.tensor?

body:
For example, I have an np array containing elements with np. array

 (5, 10)
(10, 10)
(32, 10)
(2, 10)
(48, 10)
(72, 10)
(42, 10)

how can I convert this kind of array into torch.tensor?

TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, int64, int32, int16, int8, and uint8.


title: Why is my VGG-19 implementation not performing well?

body:
I have to implement VGG-19 manually in order to have the best accuracy rate on CIFAR-10 data test.

I read the paper about VGGNet and implemented the VGG-19 network but I have an accuracy rate of 10% on the test dataset... I tried changing Batch size, learning rate with no improvement.

class MyClassifier(nn.Module):
  def __init__(self):
        super(MyClassifier, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3, stride=1, padding=1)
        self.conv12 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1)

        self.conv2 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1)
        self.conv22 = nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1)

        self.conv3 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1)
        self.conv32 = nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, stride=1, padding=1)

        self.conv4 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1)
        self.conv42 = nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, stride=1, padding=1)

        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(kernel_size=2, stride=2)

        self.Dropout = nn.Dropout(p=0.5)
        self.fc1 = nn.Linear(in_features=512*1*1, out_features=4096)
        self.fc2 = nn.Linear(in_features=4096, out_features=4096)
        self.fc3 = nn.Linear(in_features=4096, out_features=output_dim)

  def forward(self, x):
      x = self.relu(self.conv1(x))
      x = self.pool(self.relu(self.conv12(x)))

      x = self.relu(self.conv2(x))
      x = self.pool(self.relu(self.conv22(x)))

      x = self.relu(self.conv3(x))
      x = self.pool(self.relu(self.conv32(x)))

      x = self.relu(self.conv4(x))
      x = self.relu(self.conv42(x))
      x = self.relu(self.conv42(x))
      x = self.pool(self.relu(self.conv42(x)))

      x = self.relu(self.conv42(x))
      x = self.relu(self.conv42(x))
      x = self.relu(self.conv42(x))
      x = self.pool(self.relu(self.conv42(x)))

      x = x.view(-1, 512*1*1)
      x = self.Dropout(self.relu(self.fc1(x)))
      x = self.Dropout(self.relu(self.fc2(x)))
      outputs = self.fc3(x)
      return outputs

Here are my results:

[epoch:0, iteration:2000] train loss : 2.5708 train accuracy : 0.0000
[epoch:0, iteration:4000] train loss : 2.4595 train accuracy : 0.0000
[epoch:0, iteration:6000] train loss : 2.2051 train accuracy : 0.0000
[epoch:0, iteration:8000] train loss : 2.4449 train accuracy : 0.0000
[epoch:0, iteration:10000] train loss : 2.3113 train accuracy : 0.2500
[epoch:0, iteration:12000] train loss : 2.3602 train accuracy : 0.0000
[epoch:0, iteration:12500] test_loss : 2.3092 test accuracy : 0.1000
checkpoint is saved !

My test accuracy is 0.1 (10%) at epoch:0 but stays 0.1 even after 10 epochs. I believed I could achieve at least 0.8 (80%) with VGG-19's code.
Do you see any problem with the code or is it something else ?


title: How do I have to multiply the following 4darrays to obtain a 2d array in Pytorch?

body:
I am using Pytorch. I have a 4d array with shape (C, H, W, K) and I want to multiply it to another 4d-array with shape (N, C, H, W).
I want a final 2-d matrix (N, K).
How should I implement this in Pytorch?
And, more generally, how does higher order matrix multiplication works? Are there any rules to keep in mind in general?


title: Pytorch variable changes numpy variable even though memory addresses are not same

body:
I have the following bit of code:

a = torch.ones(10); b = a.numpy()
a[0] += 1
print(a, b)

Both variables essentially hold the same values even though I only modified a. However, I checked the memory addresses of a and b using hex(id(a)) and they're different. So in this case, is b a pointer to a? What's going on?


title: Using pytorch, how to write a jacobian matrix in the loss function？

body:
In knowledge distillation，we usually use Jacobian matching like this：T and S are the logits produced by the teacher network and student network

Thank you very much！


title: Should this boxes for my custom Object Detector be smaller?

body:
So Im trying to make a Object Detector for this companys forms, and we have labelled the images as shown in the example image I uploaded, my question is: Should We make more accurate boxes or is OK as they are, since the written part that we are trying to detect could be bigger.

So, what im asking is: In the example image, the "Descripcion" part or Description, has just 2 lines of text, but it could be more, should we make the box to just select the Description title + the 2 lines or so we stick to what we are doing now title + the 2 lines + all blank space that could have been filled with lines


title: how to add tensor size pytorch

body:
I have a problem while working on PyTorch.

I'm trying to increase tensor size from [1,97,1] to [1,97,2]. How can I add "1" in tensor size?


title: Consideration of Two Steps of Up & Down sampling Operation

body:
nice to meet you!

I have a one question about Up & Down sampling in CNN.

For example,

Method 1.

conv1 = nn.Conv2Transpose2d(10, 10, kernel_size = 3, stride = 2, padding = 1 )
conv2 = nn.Conv2d(10, 5, kernel_size = 3, stride = 1, padding = 1 )

input = torch.randn(1,10,30,30)
h = conv1(input)   # h.shape >>> (1,10,60,60)
output = conv2(h)  # output.shape >>> (1,5,60,60)

Method 2.

conv1 = nn.Conv2Transpose2d(10, 5, kernel_size = 3, stride = 2, padding = 1 )
input = torch.randn(1,10,30,30)
output = conv1(input)  # output.shape >>> (1,5,60,60)

So, what is best method ??
To my understanding, Method 2 seems reasonable.


title: What gets printed when you print a object of some class in python?

body:
I want to ask about this specific example, taken from the official pytorch tutorial.

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 3x3 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 3)
        self.conv2 = nn.Conv2d(6, 16, 3)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 6 * 6, 120)  # 6*6 from image dimension
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

net = Net()
print(net)

And the output is

Net(
  (conv1): Conv2d(1, 6, kernel_size=(3, 3), stride=(1, 1))
  (conv2): Conv2d(6, 16, kernel_size=(3, 3), stride=(1, 1))
  (fc1): Linear(in_features=576, out_features=120, bias=True)
  (fc2): Linear(in_features=120, out_features=84, bias=True)
  (fc3): Linear(in_features=84, out_features=10, bias=True)
)

As I understand, this code defines a child class Net of nn.Module, and defines in its initializer the members conv1, conv2, etc. These members are printed when print(net) is called. Based on this obsevation, I thought that if I add the line self.x = 0 to the initializer of Net, there would be an extra line of output, something like: (x): 0. But that didn't happen. So who decides which part of Net gets printed?


title: Slicing torch tensors with list of indeces

body:
I'm doing a reinforcement learning project, and I'm trying to get a tensor that represents the expected reward of all the given actions. I have a long tensor of chosen actions of size batch with values of either zero or one (the two potential actions). I have a tensor of expected rewards for each action of size batch * action_size, and I want a tensor of size batch.

For example, if batch size was 4, then I have

action = tensor([1,0,0,1])
expectedReward = tensor([[3,7],[5,9],[-1,12],[0,1]])

and what I want is

rewardForActions = tensor([7,5,-1,1])

I thought this would answer my question, but it's not the same at all, because if I went with that solution, it would end up with a 4*4 tensor, selecting from each row 4 times, instead of once.

Any ideas?


title: About model comparison in terms of training/validation loss

body:
I'm comparing two models, and want to clarify the weird results.

Model 1 achieves lower training loss than model 2, but get higher validation loss.

Because over-fitting and under-fitting are determined by comparing training/validation loss of themselves, therefore, I think it's not an issue of over-fitting.

Precisely, I'm now training with point cloud classification tasks,

got model 1 training loss : 1.51, test loss : 1.56 /
    model 2 training loss : 1.37, test loss : 1.58.

All other conditions are the same.

So the question is, how can this happens, test loss is lower than training loss?

it will be grateful anyone can help our problems.


title: Pytorch tensor.method() vs torch.method(tensor)?

body:
I've noticed that you can often invoke a method working with tensors from the torch module itself, or call it as a bound method on an instance of a torch tensor.

For instance:

import torch
my_tens = torch.ones((3,2))
another_tens = torch.ones((3,2))
res_tens = my_tens==another_tens

# both are equivalent:
torch.all(res_tens, dim=1)
res_tens.all(dim=1)

Similarly, .sum() and other methods work the same way. Why is that? Are there any advantages to using one approach or the other?


title: Add tensor to another by repeating along axis

body:
I have a tensor out with shape:

torch.Size([4, 644, 25])

and another one x with shape:

torch.Size([4, 161, 25])

I want to add x to out 4 times, something like:

        out[:, 0:161] += x
        out[:, 161:322] += x
        out[:, 322:483] += x
        out[:, 483:644] += x

Is there some one-liner that I can use to do this?


title: How to get a value out of tensor?

body: My python function is returning tensor([102],cuda::0) like structure. How do I obtain the first value from this?


title: Is there a pytorch function to compute fourier integrals

body:
I have this monstrous function

$f(x,p,t) = \int_{-\infty}^{\infty} sech^{2}(x + y/2)sech^{2}(x - y/2) × [2 sinh(x + y/2) sinh(x − y/2) + \sqrt(2) sinh(x − y/2)exp(i3t/2) +\sqrt(2) sinh(x + y/2)exp(-i3t/2) + 1]exp(−ipy)dy$

This is essentially a Fourier transform but there is a shift involved. In any case, I know scipy_integrate can handle this integral. But my goal is to plug in tensors in this function W so that I can use the autograd module to compute partial derivatives. Is there some way in pytorch I can approximate this integral. I can write out a Simpson’s rule formula but wondering if there is a better approximation out there in pytorch before I write my substandard approximations.

Thank you very much for your help.


title: Size mismatch error in converting TensorFlow model to Pytorch

body:
I´m trying to convert a TensorFlow model to Pytorch but got stuck in this error. Can anyone help me?

#getting weights and biases from tensorflow model
weights, biases = model.layers[0].get_weights()
#[1] is the dropout layer
weights2, biases2 = model.layers[2].get_weights()

#initializing pytorch
class TwoLayerNet(torch.nn.Module):
def __init__(self, weights, biases, weights2, biases2):

    super(TwoLayerNet, self).__init__()
    #created the model in the same dimensions as tensorflow´s model
    self.linear1 = torch.nn.Linear(9, 2048)
    self.hidden1 = nn.Dropout(0.2)
    self.linear2 = torch.nn.Linear(2048,5)

    weights = torch.from_numpy(weights)
    biases = torch.from_numpy(biases)
    weights2 = torch.from_numpy(weights2)
    biases2 = torch.from_numpy(biases2)

    self.linear1.weight = torch.nn.Parameter(weights)
    self.linear1.bias = torch.nn.Parameter(biases)
    self.linear2.weight.data = weights2
    self.linear2.bias.data = biases2
    #in this print the dimensions are ok (Linear(in_features=9, out_features=2048, bias=True))
    print(self.linear1)

def forward(self, x):
  print(self.linear1)
  x = self.linear1(x)
  x = self.hidden1(x)
  x = self.linear2(x)
  return x

model_pytorch = TwoLayerNet(weights, biases, weights2, biases2)

model_pytorch.eval()
exemplo_input_torch = torch.from_numpy(exemplo_input)
exemplo_input_torch = exemplo_input_torch.float()
print(exemplo_input_torch)
result = model_pytorch(exemplo_input_torch)

The error is:

RuntimeError: size mismatch, m1: [1 x 9], m2: [2048 x 9] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:41


title: Running GPU on multiple PyTorch tensor operators

body:
I have the following PyTorch tensor:
X = np.array([[1, 3, 2, 3], [2, 3, 5, 6]])
X = torch.FloatTensor(X).cuda()

I was wondering if there is any difference (especially in the speed) between Scenario A or B below if I run the following multiple PyTorch operators in one line?
Scenario A:
X_sq_sum = (X**2).cuda().sum(dim = 1).cuda()

Scenario B:
X_sq_sum = (X**2).sum(dim = 1).cuda()

ie. Scenario A has two .cuda() whereas Scenario B only has one .cuda().
Many thanks in advance.


title: How to reduce PyTorch time in `to`?

body:
I profiled my PyTorch code and I noticed that it spends a lot of time in to, might it be because of data transfer between GPU and main memory?
-----------------------------------  ---------------
Name                                 Self CPU total %
-----------------------------------  ---------------
to                                             20%
mul                                            20%
add                                            10%


title: pytorch "trying to backward through the graph a second time" error with chracter level RNN

body:
I am training a character level GRU with pytorch, while dividing the text into batches of a certain chunk length.
This is the trainning loop:
for e in range(self.epochs):
  self.model.train()
  h = self.get_init_state(self.batch_size)

  for batch_num in range(self.num_batch_runs):
    batch = self.generate_batch(batch_num).to(device)

    inp_batch = batch[:-1,:]
    tar_batch = batch[1:,:]

    self.model.zero_grad()
    loss = 0

    for i in range(inp_batch.shape[0]):
      out, h = self.model(inp_batch[i:i+1,:],h)

      loss += loss_fn(out[0],tar_batch[i].view(-1))

    loss.backward()

    nn.utils.clip_grad_norm_(self.model.parameters(), 5.0)

    optimizer.step()

    if not (batch_num % 5):
      print("epoch: {}, loss: {}".format(e,loss.data.item()/inp_batch.shape[0]))

Still, I am getting this error after the first batch:
Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

Thanks in advance..


title: Model Training: naming scheme for saved models with different hyperparameters

body:
For a classification task, I've been experimenting with different neural network architectures and training methods: varying the number of hidden layers, activation functions, batch size, epochs, loss function, etc.
Sometimes, I'll want to compare the performance of different models. After training/validation, each model is saved in S3, with a name that indicates its parameters + hyperparameters (e.g. '100_1000_0.5_100_..._1.model')
This naming scheme is messy, and I'd like to find a better way to save/load models based on the (hyper)parameters. One potential alternative is to create nested folders for different arguments (e.g. 100 -> 1000 -> 0.5 -> 100 -> ... -> 1.model) But this convention is still brittle — if I change the number/order of hyperparameters, I've got to reorganize the entire folder structure in S3.
Any suggestions on how to store networks with 10-15 different parameters? (Also, is it even worth storing multiple copies of the same model, trained with different hyperparameters?)


title: Simultaneous reads of the same PyTorch torchvision.datasets object

body:
Consider the following piece of code to fetch a data set for training from torchvision.datasets and to create a DataLoader for it.
import torch
from torchvision import datasets, transforms

training_set_mnist = datasets.MNIST('./mnist_data', train=True, download=True)
train_loader_mnist = torch.utils.data.DataLoader(training_set_mnist, batch_size=128,
                                                 shuffle=True)

Assume that several Python processes have access to the folder ./mnist_data and execute the above piece of code simultaneously; in my case, each process is a different machine on a cluster and the data set is stored in an NFS location accessible by everyone. You may also assume that the data is already downloaded in this folder so download=True should have no effect. Moreover, each process may use a different seed, as set by torch.manual_seed().
I would like to know whether this scenario is allowed in PyTorch. My main concern is whether the above code can change the data folders or files in ./mnist_data such that if ran by multiple processes it can potentially lead to unexpected behavior or other issues. Also, given that shuffle=True I would expect that if 2 or more processes try to create the DataLoader each of them will get a different shuffling of the data assuming that the seeds are different. Is this true?


title: max() of pad_sequence in python3.7 . ValueError: max() arg is an empty sequence

body:
I have a list of tensors , feat_tensor_list
I just try to get the max dim of all tensors at dim 0
When I
input_feat_tensor = nn.utils.rnn.pad_sequence(feat_tensor_list)

I got Error
max_size = sequences[0].size()
list index out of range

I find it's error of max_len = max([s.size(0) for s in sequences])   in pad_sequence
Acutally my list isn't empty
 for idx, tensor_i in enumerate(feat_tensor_list):
        print('size(0): ', tensor_i.size(0))
        size_ten = feat_tensor_list[idx].size()
        print('size: ', size_ten)
        dim = size_ten[1:]
        dim1_ = dim
        print('trail_dim: ', dim)
        print(type(tensor_i))
        print(tensor_i.shape)
        print('=========================')
    # max_list = [s.size(0) for s in feat_tensor_list]  is  same error
    max_list = []
    for s in feat_tensor_list:
        s_tensor = s.size(0)
        max_list.append(s_tensor)
    max_len = max(max_list)
    print('maxlen: ', max_len)

it will show as below
size(0):  34
size:  torch.Size([34, 24])
trail_dim:  torch.Size([24])
<class 'torch.Tensor'>
torch.Size([34, 24])
=========================
size(0):  10
size:  torch.Size([10, 24])
trail_dim:  torch.Size([24])
<class 'torch.Tensor'>
torch.Size([10, 24])
=========================
size(0):  3
size:  torch.Size([3, 24])
trail_dim:  torch.Size([24])
<class 'torch.Tensor'>
torch.Size([3, 24])
=========================
size(0):  2
size:  torch.Size([2, 24])
trail_dim:  torch.Size([24])
<class 'torch.Tensor'>
torch.Size([2, 24])
=========================
size(0):  4
size:  torch.Size([4, 24])
trail_dim:  torch.Size([24])
<class 'torch.Tensor'>
torch.Size([4, 24])
=========================
size(0):  5
size:  torch.Size([5, 24])
trail_dim:  torch.Size([24])
<class 'torch.Tensor'>
torch.Size([5, 24])
=========================
size(0):  2
size:  torch.Size([2, 24])
trail_dim:  torch.Size([24])
<class 'torch.Tensor'>
torch.Size([2, 24])
=========================
maxlen:  34

But it can get output but also will get error
File "/home/yang/Documents/paper/code/data/dataset.py", line 684, in user_tensor_concate
max_len = max(max_list)
ValueError: max() arg is an empty sequence

What should I do?
By the way, once I do as this
max_len = max([s.size(0) for s in feat_tensor_list], default=0)
It not get any error.But why?
(If my noob english make you unwell, I apologize for it )


title: Access 3D tensor (image) using a 2d tensor of indices

body:
With the following 3D tensor representing an image
img.shape=[H,W,F]
And a tensor representing the indices to that img
indices.shape=[N,2]
E.g. if indices = [[0,1],[5,3],...]]
I would like to create a new tensor of shape new.shape=[N,F] where new[k] == img[indices[k][0],indices[k][1]]
Currently to solve this I flatten both tensors:
    idx_flattened = idx_flattened [:,0] * (idx_flattened [:,1].max()+1) + idx_flattened[:,1]
    img = img .reshape(-1,F)
    new = img[idx_flattened ]

But I'm certain there is a better way:)
Here's a full minimal example:
img = torch.arange(8*10*3).reshape(8,10,3)
indices = torch.tensor([[0,0],[3,0],[1,2]])
new = img[indices] <- This does not work
new = [[  0,   1,   2],[ 90,  91,  92],[ 36,  37,  38]]

Ideas?


title: Is there a many to many convolution in Pytorch? is this a thing?

body:
I have been thinking about convolutions recently. There are common 3by3 convs, where (3,3) kernel's information is weighted and aggregated to supply information to a single spatial point on the output. There are also 3 by 3 upconvs, where a single spatial point on the input supplies weighted information to a 3 by 3 output space.
The conv is a many to one relationship and the upconv is a one to many relationship.
I have however never heard of many to many conv? is there such a thing? For example, a 3by3 kernel supplying information to another 3by3 kernel. I would like to experiment with it in PyTorch. My internet searching has not revealed anything.


title: Generate random vectors with a given (numerical) distribution matrix

body:
I'm trying to come up with a fast and smart way of generating random vectors from a distribution matrix, much like what is being discussed here:
Generate random numbers with a given (numerical) distribution
But the key difference is that I have a distribution matrix, rather than just a single vector.
Now obviously I could just create a for loop and loop over each vector in my matrix, but that doesn't seem very pythonic, or fast, so I'm kinda hoping that there is a better way of doing it.
To give a quick example of what I want to do:
Given a one-probability matrix
p = [[0.2, 0.4, 0.4],[0.1, 0.7, 0.2],[0.44, 0.5, 0.06],...]

I wish to draw elements, where each element gets selected with the probability in the probability matrix. (Essentially I want to generate a one-hot encoding from my one-probability matrix).
Which could for instance look like this given the above probabilities:
t = [2,1,2,...]

I need to do this for long sequences, and I need to do it millions of times, but only 1 time for each sequence each time. (Data augmentation for deep learning)
Does anyone have a good way of doing this?


title: torch.einsum doesn't accept float tensors?

body:
I get the following error:

RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #3 'mat2' in call to _th_addmm_out

I use torch.einsum as follows:
mu = torch.einsum('ijl, akij -> akl', idxs, activation_map)
I don't understand this, as in the documentation they are using float tensors too (https://pytorch.org/docs/stable/generated/torch.einsum.html). Also choosing a long tensor is no option, as all values in activation_map are between 0 and 1.


title: Add a median filter to adversarial examples

body:
How do I to add a median filter to the examples after adding the FGSM in this function.

def fgsm_attack(image, epsilon, data_grad):
    sign_data_grad = data_grad.sign()
    perturbed_image = image + epsilon*sign_data_grad
    perturbed_image = torch.clamp(perturbed_image, 0, 1)
    return perturbed_image

I use this code.
https://pytorch.org/tutorials/beginner/fgsm_tutorial.html


title: Training Sparse Autoencoders

body:
My dataset consists of vectors that are massive.  The data points are all mostly zeros with ~3% of the features being 1.  Essentially my data is super sparse and I am attempting to train an autoencoder however my model is learning just to recreate vectors of all zeros.
Are there any techniques to prevent this?  I have tried replacing mean squared error with dice loss but it completely stopped learning.  My other thoughts would be to use a loss function that favors guessing 1s correctly rather than zeros.  I have also tried using a sigmoid and linear last activation with no clear winner.  Any ideas would be awesome.


title: How Pytorch calculate Gradient in linear regression loss function

body:
I have been following this tutorial on PyTorch linear regression. the loss function is torch.sum(diff * diff) / diff.numel() where diff is Target - predicted values. i tried making computation graphs but the gradient of loss with respect to w i am getting is (inputs @ w.t()+ b - target) @ inputs/5 which is not correct since shape of inputs @ w.t()+ b - target is 5x2 and inputs shape is 5x3 which means they can't get multiplied.
how gradients of loss with respect to w is really calculated in the above equation which gives output of tensor([[-17301.5684, -19452.9375, -11681.3174],[ -4972.6597,  -5340.1387,  -3330.2732]])


title: Optimized way to extract patches based on bounding box in PyTorch

body:
Assuming we got tensors: img and bounding boxes: bbox of shape:
[B, C, H, W], [B, xmin, ymin, xmax, ymax], respectively

Is there an optimized way to extract the corresponding patches from each tensor(each tensor has a different bounding box) via PyTorch or Python?
Due to the fact that I want to build a real-time application all my tries were really slow regarding this.


title: Add a index selected numpy array to another numpy array with overlapping indices

body:
I have two numpy arrays image and warped_image and indices arrays ix,iy. I need to add image to warped_image such that image[i,j] is added to warped_image[iy[i,j],ix[i,j]]. The below code works if the pairs (iy[i,j], ix[i,j]) are unique for all i,j. But when they are not unique i.e. when 2 elements from image need to be added to the same element in warped_image, only one of them gets added. How can I add both elements from image to the same element in warped_image?
Note that, I don't want to use any for loops. I want to keep this vectorized. I'm planning to convert the code to TensorFlow or PyTorch in the future to use GPU capabilities for this. That's because, I have hundreds of such images and each image is of full HD resolution.
import numpy
image = numpy.array([[246,  50, 101], [116,   1, 113], [187, 110,  64]])
iy = numpy.array([[1, 0, 2], [1, 1, 0], [2, 0, 2]])
ix = numpy.array([[0, 2, 1], [1, 2, 0], [0, 1, 2]])
warped_image = numpy.zeros(shape=image.shape)
warped_image[iy, ix] += image

>> warped_image
Out[31]:
array([[  113., 110.,  50.],
       [246., 116.,   1.],
       [187., 101.,  64.]])

For the above case, indices are unique and hence the output is as expected.
import numpy
image = numpy.array([[246,  50, 101], [116,   1, 113], [187, 110,  64]])
iy = numpy.array([[1, 0, 2], [1, 0, 2], [2, 2, 2]])
ix = numpy.array([[0, 2, 1], [1, 2, 0], [0, 1, 2]])
warped_image = numpy.zeros(shape=image.shape)
warped_image[iy, ix] += image

>> warped_image
Out[32]:
array([[  0.,   0.,   1.],
       [246., 116.,   0.],
       [187., 110.,  64.]])

Expected Output:
array([[  0.,   0.,   51.],
       [246., 116.,   0.],
       [300., 211.,  64.]])

In this case, there are 3 pairs of indices which overlap and hence it fails. E.g. image[0,1] and image[1,1] should gt added to warped_image[0,2] to give a value 51. However only one of them (image[1,1]) gets added to give a value 1.
Context:
I'm trying to do warp an image from view1 to view2. I've computed which pixel has to go where. In case of overlapping pixels, I need to take a weighted average of them. So, I need to achieve the above. More details here


title: CNN Pytorch: what parameters to use when creating my own model?

body:
I am new in this field; I have to create a CNN by pulling my model.
I want to develop a network that categorizes car images using this dataset (8144 train images, 8041 test images, 196 different classes). To start getting used to it, I began using ResNet18, and now I wanted to create my model.
I have followed what is written here and here, but I am having difficulty in choosing the correct parameters and hyperparameters anyway, I do not understand based on what these values  (YY in code) are determined. I resized all my images at 3x224x224.
This is my code so far:
import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
import torchvision.models as models

imagenet_stats = ([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

train_tfms = transforms.Compose([transforms.Resize((224, 224)),
                                 transforms.RandomHorizontalFlip(),
                                 transforms.RandomRotation(15),
                                 transforms.ToTensor(),
                                 transforms.Normalize(*imagenet_stats,inplace=True)
                                ])
train_dataset = torchvision.datasets.ImageFolder(train_O, transform=train_tfms)
trainloader = torch.utils.data.DataLoader(train_dataset, batch_size=4,
                                          shuffle=True, num_workers=2)

test_tfms = transforms.Compose([transforms.Resize((224, 224)),
                                transforms.ToTensor(),
                                transforms.Normalize(*imagenet_stats,inplace=True),
                               ])
test_dataset = torchvision.datasets.ImageFolder(test_O, transform=test_tfms)
testloader = torch.utils.data.DataLoader(test_dataset, batch_size=4,
                                         shuffle=False, num_workers=2)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv = nn.Conv2d(3, 224, kernel_size=15, padding=7)
        self.pool = nn.MaxPool2d(2,2)
        self.hidden= nn.Linear(224*112*112, 128)
        self.drop = nn.Dropout(0.2)
        self.out = nn.Linear(128, 196)
        self.act = nn.ReLU()

    def forward(self, x):
        x = self.act(self.conv(x))
        x = self.pool(x)
        x = x.view(x.size(0), -1)
        x = self.act(self.hidden(x))
        x = self.drop(x)
        x = self.out(x)
        return x

net_train = Net()


title: Model is not imported into core ml

body:
I am unable to save the trace model to a coreml file. My code:
input_shape = ct.Shape(shape=[1, 3, 256, 256])
converted = ct.convert(
    traced_model,
    inputs=[ct.TensorType(name='input', shape=input_shape)])

I get the following error:
KeyError                                  Traceback (most recent call last)
<ipython-input-55-6daf89c84466> in <module>()
      2 converted = ct.convert(
      3     traced_model,
----> 4     inputs=[ct.TensorType(name='input', shape=input_shape)])

C:\anaconda3\lib\site-packages\coremltools\converters\_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, convert_to, **kwargs)
    181         outputs=outputs,
    182         classifier_config=classifier_config,
--> 183         **kwargs
    184     )
    185

C:\anaconda3\lib\site-packages\coremltools\converters\mil\converter.py in mil_convert(model, convert_from, convert_to, **kwargs)
    127     """
    128     proto = mil_convert_to_proto(model, convert_from, convert_to,
--> 129         ConverterRegistry, **kwargs)
    130     if convert_to == 'mil':
    131         return proto

C:\anaconda3\lib\site-packages\coremltools\converters\mil\converter.py in mil_convert_to_proto(model, convert_from, convert_to, converter_registry, **kwargs)
    169     frontend_converter = frontend_converter_type()
    170
--> 171     prog = frontend_converter(model, **kwargs)
    172     common_pass(prog)
    173

C:\anaconda3\lib\site-packages\coremltools\converters\mil\converter.py in __call__(self, *args, **kwargs)
     83         from .frontend.torch import load
     84
---> 85         return load(*args, **kwargs)
     86
     87

C:\anaconda3\lib\site-packages\coremltools\converters\mil\frontend\torch\load.py in load(model_spec, debug, **kwargs)
     83         raise e
     84     except Exception as e:
---> 85         raise e
     86
     87     return prog

C:\anaconda3\lib\site-packages\coremltools\converters\mil\frontend\torch\load.py in load(model_spec, debug, **kwargs)
     73
     74     try:
---> 75         prog = converter.convert()
     76     except RuntimeError as e:
     77         if debug and "convert function" in str(e):

C:\anaconda3\lib\site-packages\coremltools\converters\mil\frontend\torch\converter.py in convert(self)
    222
    223             # Add the rest of the operations
--> 224             convert_nodes(self.context, self.graph)
    225
    226             graph_outputs = [self.context[name] for name in self.graph.outputs]

C:\anaconda3\lib\site-packages\coremltools\converters\mil\frontend\torch\ops.py in convert_nodes(context, graph)
     54             )
     55         else:
---> 56             _add_op(context, node)
     57
     58         # We've generated all the outputs the graph needs, terminate conversion.

C:\anaconda3\lib\site-packages\coremltools\converters\mil\frontend\torch\ops.py in to(context, node)
   1890         res = mb.const(mode="immediate_value", val=casted_input, name=node.name)
   1891     else:
-> 1892         res = mb.cast(x=_input, dtype=NUM_TO_DTYPE_STRING[dtype], name=node.name)
   1893     context.add(res)
   1894

KeyError: 5

My model (head):
Net(
  original_name=Net
  (Encoder): Sequential(
    original_name=Sequential
    (0): Conv2d(original_name=Conv2d)
    (1): BatchNorm2d(original_name=BatchNorm2d)
    (2): ReLU(original_name=ReLU)
    (3): MaxPool2d(original_name=MaxPool2d)
    (4): LambdaReduce(
      original_name=LambdaReduce
      (0): Lambda(original_name=Lambda)
      (1): Sequential(
        original_name=Sequential
        (0): BatchNorm2d(original_name=BatchNorm2d)
        (1): ReLU(original_name=ReLU)
        (2): Conv2d(original_name=Conv2d)
        (3): BatchNorm2d(original_name=BatchNorm2d)
        (4): ReLU(original_name=ReLU)
        (5): Conv2d(original_name=Conv2d)
      ) ....

pytorch 1.4
coremltools 4
And i a warning:
WARNING:root:Tuple detected at graph output. This will be flattened in the converted model.
Converting Frontend ==> MIL Ops:   0%|                                            | 5/5269 [00:00<00:03, 1645.73 ops/s]
How to solve the problem?


title: Multi GPUs pytorch prediction

body:
I am trying to use multiple GPUs for a code that is already running correctly for one GPU.
I did those modifications to make it able to di multiple GPUs:
classifier = AnnotationCNN.load_from_checkpoint(classifier_ckpt,normalizer_ckpt=normalizer_ckpt)
classifier=classifier.eval()

**device = torch.device("cuda" if torch.cuda.is_available() else "cpu")**
classifier = classifier.to(torch.float16).to(device)
**classifier= nn.DataParallel(classifier)**

dataset = InitImageTifDataset(
                data_root=data_root,
                level=level,
                region_width=region_width,
                region_height=region_height,
                crop_size=crop_size,
                image_dir=image_dir,
                image_name=image_name,
                channels=channels
            )

        dl = DataLoader(dataset, batch_size=5, shuffle=False, num_workers=5)
        for i,(img,mean_stddev,filename) in enumerate(dl):
            mask = mean_stddev > 20

            img=img[torch.nonzero(mask)]
            if img.nelement() != 0 :
                **tensor=img[:,0,0,:,:,:].to(device).to(torch.float16).to(device)**
                *_,preds = classifier(tensor)*

But I have the following error:
RuntimeError: Expected tensor for argument #1 'output' to have the same device as tensor for argument #5 'weight'; but device 0 does not equal 1 (while checking arguments for batch_norm_cuda)

I am pretty sure the error is coming from the DataLoader that has not being adapted to multi GPUs. How can I fixe it, Thank you.


title: Use multiple softmax in transformers output layer and calculate loss

body:
Can I use multiple softmax in the last output layer in transformers? If so, how can I calculate loss from that. I am working in pytorch.
And I am asking because my data is a sequence of tuples where, the elements have different dimensions. Like,
[(2,1), (3,1), (3,1), (2,1), (2,1), (3,1), (3,0), (4,1)]

The first element of tuples has a vocab of 5 and the second element of tuples has a vocab of 2.


title: Dimension mismatch in loading data

body:
I have this codes as follows.
import torch.nn.functional as f
train_on_gpu=True
class CnnLstm(nn.Module):
    def __init__(self):
        super(CnnLstm, self).__init__()
        self.cnn = CNN()
        self.rnn = nn.LSTM(
            input_size=180000,
            hidden_size=256,
            num_layers=2,
            batch_first=True)
        self.linear = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        print('before forward ')
        print(x.shape)
        batch_size, time_steps, channels, height, width = x.size()
        c_in = x.view(batch_size * time_steps, channels, height, width)
        _, c_out = self.cnn(c_in)
        r_in = c_out.view(batch_size, time_steps, -1)
        r_out, (_, _) = self.rnn(r_in)
        r_out2 = self.linear(r_out[:, -1, :])
        return f.log_softmax(r_out2, dim=1)

cnnlstm_model = CnnLstm().to(device)
optimizer = torch.optim.Adam(cnnlstm_model.parameters(), lr=learning_rate)
#optimizer = torch.optim.SGD(cnnlstm_model.parameters(), lr=learning_rate)
#criterion = nn.functional.nll_loss()
criterion = nn.CrossEntropyLoss()
# Train the model
n_total_steps = len(train_dl)
num_epochs = 20
for epoch in range(num_epochs):
    t_losses=[]
    for i, (images, labels) in enumerate(train_dl):
        # origin shape: [5, 3, 300, 300]
        # resized: [5, 300, 300]
        print('load data '+str(images.shape))
        images = np.expand_dims(images, axis=1)
        print('after expand ')
        print(images.shape)
        images = torch.FloatTensor(images)
        images, labels = images.cuda(), labels.cuda()
        images, labels = Variable(images), Variable(labels)
        optimizer.zero_grad()
        outputs = cnnlstm_model(images)
        loss = criterion(outputs, labels)
        t_losses.append(loss)
        loss.backward()
        optimizer.step()

Three places are printing out.
(1)
print('load data '+str(images.shape))

(2)
print('after expand ')
print(images.shape)

(3)
print('before forward ')
print(x.shape)

I have batch size of 5 images.
Loading 2629 batches and only last batch has issues.
Earlier batches of loading images has no issues and loaded as
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])
.
.
.
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])

At the last batch loading,
load data torch.Size([5, 3, 300, 300])
after expand
(5, 1, 3, 300, 300)
before forward
torch.Size([5, 1, 3, 300, 300])
before forward
torch.Size([15, 300, 300])

Why do I have the 'before forward' log printed twice? Futhermore, it's not same shape.
What could be wrong?
EDIT:
This is code for loading data.
inputH = input_size
inputW = input_size
#Data transform (normalization & data augmentation)
stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
train_resize_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
train_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.RandomHorizontalFlip(),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
valid_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
test_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                        tt.ToTensor(),
                        tt.Normalize(*stats)])

#Create dataset
train_ds = ImageFolder('./data/train', train_tfms)
valid_ds = ImageFolder('./data/valid', valid_tfms)
test_ds = ImageFolder('./data/test', test_tfms)

from torch.utils.data.dataloader import DataLoader
batch_size = 5

#Training data loader
train_dl = DataLoader(train_ds, batch_size, shuffle = True, num_workers = 8, pin_memory=True)
#Validation data loader
valid_dl = DataLoader(valid_ds, batch_size, shuffle = True, num_workers = 8, pin_memory=True)
#Test data loader
test_dl = DataLoader(test_ds, 1, shuffle = False, num_workers = 1, pin_memory=True)


title: Is it possible to get 98% f1-score on a binary classification task or am I messing up the calculations?

body:
I recently used BERT for sentence classification on two datasets GAD and EU-ADR. I trained the datasets on the train files splitting the train files into train and validation. Then I tested on the test files provided by these datasets. I'm getting up to 98% f1-score on the test datasets and I am not sure if this is possible or my calculations are incorrect as the paper that I am comparing with doesn't report nearly as good a f1-score.
Link to paper BioBERT: a pre-trained biomedical language representation model for biomedical text mining
It is worth noting that I am not using BioBERT (their pretrained BERT model) for this task. I am simply using BERT-base-uncased. The links to the github notebooks are below. Any insight would be helpful and I am open to provide any information required. Moreover, I have run the model several times and I still get really high scores.
BERT relation extraction on GAD and EU-ADR datasets
Thank you in advance.


title: How to use torch_geometric graph as input to PyTorch Dataset and DataLoader

body: How to use torch_geometric & torch together ? I am building a pytorch Dataset with feature input and graph input (graph classification task). However each sample graph are built using networkx and then using 'from_networkx(G)' converted to torch_geometric format. However, the pytorch DataLoader expects to have fixed dimensional batch of samples. How can I handle it ?