title: Mesh-R-CNN Data with Colab and Pytorch3D

body:
While using the Mesh-R-CNN demo on Google Colab:
https://github.com/facebookresearch/meshrcnn
on the demo.py file I get this message
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-24-202aa0a6d1de> in <module>()
     17
     18 # required so that .register() calls are executed in module scope
---> 19 import meshrcnn.data  # noqa
     20 import meshrcnn.modeling  # noqa
     21 import meshrcnn.utils  # noqa

ModuleNotFoundError: No module named 'meshrcnn.data'

what should I do to import successfully Meshrcnn.data?!
I also don't know how to work with the config setting present in the repo. Any suggestions?


title: Index out of range for Huggingface BART tranformer (bart-large-xsum)

body:
I am facing a problem summarizing long articles with larger vocab size.
Many of the models have a limitation of maximum input size.
So, it is not accepting and trying to execute results in error of index out of range.
I am particularly using "BART-large-xsum". Please suggest what is the correct way of using these models with long documents shall I finetuning to increase the vocabsize or do anything else.
A code snippet with an example of how to handle long documents with the "BART-large-xsum" would be perfect to start with!
Thanks in advance,
Teja
My Code :

Article_text = "Article of more than 10000 characters"

from transformers import AutoModelWithLMHead, AutoTokenizer
model = AutoModelWithLMHead.from_pretrained("bart-large-xsum")
tokenizer = AutoTokenizer.from_pretrained("bart-large-xsum")

inputs = tokenizer.encode("summarize: " + Article_text, return_tensors="pt", max_length=100000)
outputs = model.generate(inputs, max_length=350, min_length=40,use_first = False)
print(tokenizer.decode(outputs[0],skip_special_tokens=True, clean_up_tokenization_spaces=False))


title: How to customize the loss function using Wasserstein distance in Pytorch?

body: I am new to using Pytorch. I have two sets of observational data Y and X, probably having different dimensions. My task is to train a function g such that the distribution distance between g(X) and Y is the smallest. I would like to impose the Wasserstein distance as the loss function. Is it possible to customize the loss function using the Wasserstein distance in Pytorch? If not, is it possible to write in Keras? When both methods do not work, what should be the alternative solution?


title: Converting lists of uneven size into LSTM input tensor

body:
So I have a nested list of 1366 samples with 2 features each and varying sequence lengths that is supposed to be the input data for an LSTM. The labels are supposed to be a pair of values for each sequence, i.e. [-0.76797587, 0.0713816]. In essence the data looks like the following:
X = [[[-0.11675862, -0.5416186], [-0.76797587, 0.0713816]], [[-0.5115555, 0.25823522], [0.6099151999999999, 0.21718016], [-0.0022403747, 0.6470206999999999]]]

What I would like to do is convert this list into an input tensor. As I understand, LSTMs accept sequences of different lengths, so in this case the first sample has length 2 and the second has length 3.
Currently I'm trying to convert the list in the following way:
train_data = TensorDataset(torch.tensor(X, dtype=torch.float32), torch.tensor(Y, dtype=torch.float32))
train_dataloader = DataLoader(train_data, batch_size=batch_size, shuffle=True)

Though this produces the following error ValueError: expected sequence of length 5 at dim 1 (got 3)
I'm guessing this is because the first sequence has length five and the second length 3, which is not convertible?
How do I convert the given list into a tensor? Or am I thinking wrong about the way to train the LSTM?
Thanks for any help!


title: Conda install from channel using proxy

body:
I'm using conda behind a company proxy. Therefore I updated the .condarc file accordingly (as explained here)
Now executing conda update numpy works fine.
Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

But I also want to install PyTorch using conda install pytorch torchvision torchaudio cpuonly -c pytorch
 However, this returns
Collecting package metadata (current_repodata.json): failed

ProxyError: Conda cannot proceed due to an error in your proxy configuration.
Check for typos and other configuration errors in any '.netrc' file in your home directory,
any environment variables ending in '_PROXY', and any other system-wide proxy
configuration settings.

I think this happens because I added a channel for the download. How can I resolve this issue?


title: RuntimeError: input.size(-1) must be equal to input_size. Expected 200, got 0 ---- PySyft / PyTorch / Federated Learning

body:
does anyone know a solution for this Error? I am trying to switch my PyTorch network to an Federated Learning network but i always get this Error.
I'm using Google Colab an train on GPU. When I print the size of embeds I get 0, but I don't understand why the data is not used there.

RuntimeError                              Traceback (most recent call last)
<ipython-input-42-fd4a5223524b> in <module>()
----> 1 model, history = train_model(net, dataloaders_dict, criterion, optimizer, num_epochs=10)
      2 #model = train_model(net, dataloaders_dict, criterion, optimizer, num_epochs=10)

6 frames
<ipython-input-41-a386f044d41f> in train_model(model, dataloaders, criterion, optimizer, num_epochs, batch_size)
     68                         # detaching it from its history on the last instance.
     69
---> 70                         outputs = model(inputs)
     71
     72                         loss = criterion(outputs, labels)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

<ipython-input-36-64e9a7d68b11> in forward(self, sentence)
     11     def forward(self, sentence):
     12         embeds = self.word_embeddings(sentence)
---> 13         lstm_out, (h,t) = self.lstm(embeds)
     14         lstm_out = self.dropout(lstm_out)
     15         tag_space = self.output(lstm_out[:,-1,:])

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    530             result = self._slow_forward(*input, **kwargs)
    531         else:
--> 532             result = self.forward(*input, **kwargs)
    533         for hook in self._forward_hooks.values():
    534             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    554             hx = self.permute_hidden(hx, sorted_indices)
    555
--> 556         self.check_forward_args(input, hx, batch_sizes)
    557         if batch_sizes is None:
    558             result = _VF.lstm(input, hx, self._flat_weights, self.bias, self.num_layers,

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in check_forward_args(self, input, hidden, batch_sizes)
    506     def check_forward_args(self, input, hidden, batch_sizes):
    507         # type: (Tensor, Tuple[Tensor, Tensor], Optional[Tensor]) -> None
--> 508         self.check_input(input, batch_sizes)
    509         expected_hidden_size = self.get_expected_hidden_size(input, batch_sizes)
    510

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in check_input(self, input, batch_sizes)
    157             raise RuntimeError(
    158                 'input.size(-1) must be equal to input_size. Expected {}, got {}'.format(
--> 159                     self.input_size, input.size(-1)))
    160
    161     def get_expected_hidden_size(self, input, batch_sizes):

RuntimeError: input.size(-1) must be equal to input_size. Expected 200, got 0

class LSTM(nn.Module):

def __init__(self, embedding_dim, hidden_layers,vocab_size,num_layers,pretrained_weights):
    super(LSTM, self).__init__()

    self.word_embeddings = nn.Embedding(vocab_size, embedding_dim,_weight=pretrained_weights, padding_idx=0)
    self.lstm = nn.LSTM(embedding_dim, hidden_size=hidden_layers,num_layers=num_layers, batch_first=True)
    self.output = nn.Linear(hidden_layers, vocab_size, bias=False)
    self.dropout = nn.Dropout(0.1)

def forward(self, sentence):
    embeds = self.word_embeddings(sentence)
    lstm_out, (h,t) = self.lstm(embeds)
    lstm_out = self.dropout(lstm_out)
    tag_space = self.output(lstm_out[:,-1,:])

    return tag_space

The Error throws in this line: lstm_out, (h,t) = self.lstm(embeds)
  for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)
    train_loss = 0
    acc_score = 0

    valid_loss = 0
    acc_valid_score = 0

    #Variables to store the losses temporary
    train_loss_result = 0
    acc_score_result = 0
    valid_loss_result = 0
    acc_valid_score_result = 0

    valid_loss_not_decreased = 0

    if valid_loss_not_decreased == 5:
      break

    # Each epoch has a training and validation phase
    for phase in ['train', 'val']:
        if phase == 'train':
            model.train()  # Set model to training mode
        else:
            model.eval()   # Set model to evaluate mode

        for inputs,labels in dataloaders[phase]:

            # Location of current batch
            worker = inputs.location  # <---- Where will send the model to

            #model.to(device)

            model = model.send(worker)   # <---- for Federated Learning

            inputs, labels = inputs.to(device), labels.to(device)

            print("--------> INPUT: ",inputs)
            print("--------> LABEL: ",labels)

            # zero the parameter gradients
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'train'):
                # Get model outputs and calculate loss

                # backward + optimize only if in training phase
                if phase == 'train':
                    # we need to clear out the hidden state of the LSTM,
                    # detaching it from its history on the last instance.

                    outputs = model(inputs)

                    loss = criterion(outputs, labels)
                    acc = binary_accuracy(outputs,labels)
                    acc_score = acc_score + acc
                    train_loss = train_loss + loss.item()
                    loss.backward()

                    optimizer.step()


title: Extracting hidden features from Autoencoders using Pytorch

body:
Following the tutorials in this post, I am trying to train an autoencoder and extract the features from its hidden layer.
So here are my questions:

In the autoencoder class, there is a "forward" function. However, I cannot see anywhere in the code that this function is called. So how does it get trained?

My question above is because I feel if I want to extract the features, I should add another function (f"orward_hidden") in the autoencoder class:
 def forward(self, features):
     #print("in forward")
     #print(type(features))
     activation = self.encoder_hidden_layer(features)
     activation = torch.relu(activation)
     code = self.encoder_output_layer(activation)
     code = torch.relu(code)
     activation = self.decoder_hidden_layer(code)
     activation = torch.relu(activation)
     activation = self.decoder_output_layer(activation)
     reconstructed = torch.relu(activation)
     return reconstructed

 def forward_hidden(self, features):
     activation = self.encoder_hidden_layer(features)
     activation = torch.relu(activation)
     code = self.encoder_output_layer(activation)
     code = torch.relu(code)
     return code

Then, after training, which means after this line in the main code:
print("AE, epoch : {}/{}, loss = {:.6f}".format(epoch + 1, epochs_AE, loss))

I can put the following code to retrieve the features from the hidden layer:
hidden_features = model_AE.forward_hidden(my_input)

Is this way correct? Still, I am wondering how the "forward" function was used for training. Because I cannot see it anywhere in the code that is being called.


title: What kinds of optimization are used in PyTorch methods?

body: I'm using PyTorch to implement an intense sequence of matrix operations, using methods such as torch.mm or torch.dot. I was wondering if PyTorch uses multithreading or other optimization mechanisms to speed up the process. I am not utilizing a GPU. I appreciate if you could inform me of how fast these methods are and whether I need to take any actions to help the process.


title: .py not reading from the content folder

body:
|-content
  |-utils
    |- parse_config.py
  |-models.py

folder structure
This is my folder structure in google colab.

I have already installed Pytorch and all other requirments for the project.
Here models.py file is not able to access the files in utils folder.

In my models.py I'm importing utils.parse_config which is inside the utils folder but it shows the following error.

ImportErrorTraceback (most recent call last) <ipython-input-6-77f4a3369184> in <module>()
----> 1 from models import *
      2 from utils import *
      3
      4 import os, sys, time, datetime, random
      5 import torch

/content/models.py in <module>()
      9 from PIL import Image
     10
---> 11 from utils.parse_config import *
     12 from utils.utils import build_targets
     13 from collections import defaultdict

ImportError: No module named utils.parse_config

--------------------------------------------------------------------------- NOTE: If your import is failing due to a missing package, you can manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the "Open Examples" button below.
---------------------------------------------------------------------------

Error

How do I make models.py get access to files in utils folder?


title: spacy-pytorch-transformers: loss = 0 constantly during training

body:
I'm using spacy 2.1.8 and spacy-pytorch-transformers 0.4.0 to train a text classifier. My code is strongly inspired from their example but the model is not learning anything, which seems to be caused by the loss being 0 all the time. A minimum (not-)working example of my code is as follows:

nlp = spacy.load("en_pytt_xlnetbasecased_lg")
textcategorizer = nlp.create_pipe("pytt_textcat", config={"exclusive_classes": True, "architecture": "softmax_last_hidden"})

for label in labels:
    textcategorizer.add_label(label)
nlp.add_pipe(textcategorizer, last=True)

optimizer = nlp.resume_training()

for epoch in range(num_of_epochs):
    np.random.shuffle(train)
    losses = Counter()

    for step, batch in enumerate(minibatch(train, size=batch_size)):
        optimizer.pytt_lr = 0.005
        texts, cats = zip(*batch)
        _, cats = preprocessed_labels_to_categories_for_training_and_eval(cats)

        nlp.update(texts, cats, sgd=optimizer, losses=losses, drop=0.1)

I already double and triple checked that relevant variables, such as cats and texts, contain valid and correct values.

Is there anything I'm missing?


title: Accuracy differences when using custom activation functions in neural network

body:
Note: The logistic activation function already exists in tf.nn.logistic. However, I am trying to develop a custom activation which is on similar lines. Please refer to the code snippet below.

def custom_logistic(x):
    value= tf.to_float(tf.math.exp(-x))
    Activate_input = 1/(1+value)
    return Activate_input

Now, when I am calling the above function at the input layer of the code:

model.add(Dense(10, input_dim=3, activation=custom_logistic))

Note: Here, the model is a sequential model.

I get an accuracy that is lesser than what I get when I use activation = tf.nn.sigmoid

Is there anything else I need to define?. Please let me know.


title: How to index a 3-d tensor with 2-d tensor in pytorch?

body:
import torch
a = torch.rand(5,256,120)
min_values, indices = torch.min(a,dim=0)
aa = torch.zeros(256,120)
for i in range(256):
    for j in range(120):
        aa[i,j] = a[indices[i,j],i,j]

print((aa==min_values).sum()==256*120)

I want to know how to avoid to using the for-for loop to get the aa values? (I want to use the indices to select elements in another 3-d tensors so I can't use the values return by min directly)


title: saving model in pytorch with weight decay

body:
this is my model :

# basic LeNet5 network
class LeNet5_mode0 (nn.Module) :

  # constructor
  def __init__(self):
    super(LeNet5_mode0, self).__init__() # call to super constructor

    # define layers
    # 6 @ 28x28
    self.conv1 = nn.Sequential(
        # Lenet's first conv layer is 3x32x32, squeeze color channels into 1 and pad 2
        nn.Conv2d(in_channels = 1, out_channels = 6, kernel_size = 5, stride = 1, padding = 2),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size = 2, stride = 2)
        )

    # 16 @ 10x10
    self.conv2 = nn.Sequential(
        nn.Conv2d(in_channels = 6, out_channels = 16, kernel_size = 5, stride = 1, padding = 0),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size =2, stride = 2)
        )

    self.fc1 = nn.Sequential(
        nn.Linear(in_features = 16*5*5, out_features = 120),
        nn.ReLU()
        )

    self.fc2 = nn.Sequential(
        nn.Linear(in_features = 120, out_features = 84),
        nn.ReLU()
        )
    self.classifier = nn.Sequential(
        nn.Linear(in_features = 84,out_features = 10),
        nn.Softmax(dim = 1) # dim =1 meaning do softmax on the colums of 84x10
        )

  # define forward function
  def forward(self, x):
    x = self.conv1(x)
    x = self.conv2(x)
    x = x.view(-1, 16*5*5) # reshape the tensor to [-1,16*5*5]
    x = self.fc1(x)
    x = self.fc2(x)
    x = self.classifier(x)
    return x

and I train this model once with :

criterion = nn.CrossEntropyLoss() # aka, LogLoss
optimizer = torch.optim.Adam(model.parameters(), lr=0.0005)
scheduler = torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[5,10,15], gamma=0.5)

and then save with with

torch.save(model.state_dict(), savepath)

and load it with

model.load_state_dict(torch.load(loadpath))

so far no problem . but when i change the optimizer  a little to

optimizer = torch.optim.Adam(model.parameters(), lr=0.0005, weight_decay = 0.0005)

and use the same save & load method

I receive the following error:

in loading state_dict for LeNet5_mode0:
    Unexpected key(s) in state_dict: "conv1.1.weight", "conv1.1.bias", "conv1.1.running_mean", "conv1.1.running_var", "conv1.1.num_batches_tracked", "conv2.1.weight", "conv2.1.bias", "conv2.1.running_mean", "conv2.1.running_var", "conv2.1.num_batches_tracked".

how can it be fixed? why different optimizer have that effect on the saving of the trained  network?


title: How to implement a Sparse Embedding in Tensorflow 2 like Pytorch Embedding(sparse=True)?

body:
I have a network that has a lot of items that need to be embedded.
However, in each training batch, only a very small portion of the items will actually be used.
If I use the normal tf.keras.layers.Embedding layer, it will add all the items into the network parameter, thus consuming a lot of memory and decreasing speed in distributed training significantly since in each step all the unused grads are still synchronized.
What I want is, that in each training step only the actually used embedding weights are added into the graph and be computed and synchronized.
Pytorch already provides this functionality with torch.nn.Embedding(sparse=True).
How can I implement this in Tensorflow 2?


title: How to use Torchtext model in flask app with vocabulary and vectors?

body:
I have the the following code for my model which uses vectors and vocabulary from the IMDB set.
TEXT = data.Field(tokenize="spacy", include_lengths=True)
LABEL = data.LabelField(dtype=torch.float)
from torchtext import datasets
train_data, valid_data = train_data.split(random_state=random.seed(SEED))
train_data, test_data = datasets.IMDB.splits(TEXT, LABEL)

TEXT.build_vocab(train_data, vectors="glove.6B.100d", unk_init=torch.Tensor.normal_)

LABEL.build_vocab(train_data)

BATCH_SIZE = 64

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

train_iterator, valid_iterator, test_iterator = data.BucketIterator.splits(
    (train_data, valid_data, test_data),
    batch_size=BATCH_SIZE,
    sort_within_batch=True,
    device=device,
)

class RNN(nn.Module):
    def __init__(
        self,
        vocab_size,
        embedding_dim,
        hidden_dim,
        output_dim,
        n_layers,
        bidirectional,
        dropout,
        pad_idx,
    ):

        super().__init__()

        self.embedding = nn.Embedding(vocab_size, embedding_dim, padding_idx=pad_idx)

        self.rnn = nn.LSTM(
            embedding_dim,
            hidden_dim,
            num_layers=n_layers,
            bidirectional=bidirectional,
            dropout=dropout,
        )

        self.fc = nn.Linear(hidden_dim * 2, output_dim)

        self.dropout = nn.Dropout(dropout)

    def forward(self, text, text_lengths):

        # text = [sent len, batch size]

        embedded = self.dropout(self.embedding(text))

        # embedded = [sent len, batch size, emb dim]

        # pack sequence
        packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths)

        packed_output, (hidden, cell) = self.rnn(packed_embedded)

        # unpack sequence
        output, output_lengths = nn.utils.rnn.pad_packed_sequence(packed_output)

        # output = [sent len, batch size, hid dim * num directions]
        # output over padding tokens are zero tensors

        # hidden = [num layers * num directions, batch size, hid dim]
        # cell = [num layers * num directions, batch size, hid dim]

        # concat the final forward (hidden[-2,:,:]) and backward (hidden[-1,:,:]) hidden layers
        # and apply dropout

        hidden = self.dropout(torch.cat((hidden[-2, :, :], hidden[-1, :, :]), dim=1))

        # hidden = [batch size, hid dim * num directions]

        return self.fc(hidden)

INPUT_DIM = len(TEXT.vocab)
EMBEDDING_DIM = 100
HIDDEN_DIM = 256
OUTPUT_DIM = 1
N_LAYERS = 2
BIDIRECTIONAL = True
DROPOUT = 0.5
PAD_IDX = TEXT.vocab.stoi[TEXT.pad_token]

model = RNN(
    INPUT_DIM,
    EMBEDDING_DIM,
    HIDDEN_DIM,
    OUTPUT_DIM,
    N_LAYERS,
    BIDIRECTIONAL,
    DROPOUT,
    PAD_IDX,
)

pretrained_embeddings = TEXT.vocab.vectors

print(pretrained_embeddings.shape)

model.embedding.weight.data.copy_(pretrained_embeddings)

UNK_IDX = TEXT.vocab.stoi[TEXT.unk_token]

model.embedding.weight.data[UNK_IDX] = torch.zeros(EMBEDDING_DIM)
model.embedding.weight.data[PAD_IDX] = torch.zeros(EMBEDDING_DIM)

optimizer = optim.Adam(model.parameters())
criterion = nn.BCEWithLogitsLoss()

model = model.to(device)
criterion = criterion.to(device)

*training and evaluation functions and such*

nlp = spacy.load("en")

def predict_sentiment(model, sentence):
    model.eval()
    tokenized = [tok.text for tok in nlp.tokenizer(sentence)]
    indexed = [TEXT.vocab.stoi[t] for t in tokenized]
    length = [len(indexed)]
    tensor = torch.LongTensor(indexed).to(device)
    tensor = tensor.unsqueeze(1)
    length_tensor = torch.LongTensor(length)
    prediction = torch.sigmoid(model(tensor, length_tensor))
    return prediction.item()

saved and loaded in the same as this:
torch.save(model.state_dict(), "Finished Models/Pytorch/LSTM_w_vectors.pt")

model.load_state_dict(torch.load("Finished Models/Pytorch/LSTM_w_vectors.pt"))

How can I use import / deploy this model? Do I need to pickle the vocab and PAD_IDX? Or is there a function in Pytorch itself?
I used the load_state while copying the class RNN but that didn't work.


title: Pytorch RNN model not learning anything

body:
Task: Predicting whether provided disaster tweets are real or not. Have already converted my textual data into tensors and then into train_loader. All the required code is mentioned below.
My Model Architecture
class RealOrFakeLSTM(nn.Module):

    def __init__(self, input_size, output_size, embedding_dim, hidden_dim, n_layers, bidirec, drop_prob):
        super().__init__()
        self.output_size=output_size
        self.n_layers=n_layers
        self.hidden_dim=hidden_dim
        self.bidirec=True;
        self.embedding=nn.Embedding(vocab_size, embedding_dim)
        self.lstm1=nn.LSTM(embedding_dim, hidden_dim, n_layers, dropout=drop_prob, batch_first=True, bidirectional=bidirec)
        #self.lstm2=nn.LSTM(hidden_dim, hidden_dim, n_layers, dropout=drop_prob, batch_first=True)
        self.dropout=nn.Dropout(drop_prob)
        self.fc=nn.Linear(hidden_dim, output_size)
        self.sigmoid=nn.Sigmoid()

    def forward(self, x):
        batch=len(x)
        hidden1=self.init_hidden(batch)
        #hidden2=self.init_hidden(batch)
        embedd=self.embedding(x)
        lstm_out1, hidden1=self.lstm1(embedd, hidden1)
        #lstm_out2, hidden2=self.lstm2(lstm_out1, hidden2)
        lstm_out1=lstm_out1.contiguous().view(-1, self.hidden_dim) # make it lstm_out2, if you un comment the other lstm cell.
        out=self.dropout(lstm_out1)
        out=self.fc(out)
        sig_out=self.sigmoid(out)
        sig_out=sig_out.view(batch, -1)
        sig_out=sig_out[:, -1]
        return sig_out

    def init_hidden(self, batch):
        if (train_on_gpu):
          if self.bidirec==True:
            hidden=(torch.zeros(self.n_layers*2, batch, self.hidden_dim).cuda(),torch.zeros(self.n_layers*2, batch, self.hidden_dim).cuda())
          else:
            hidden=(torch.zeros(self.n_layers, batch, self.hidden_dim).cuda(),torch.zeros(self.n_layers, batch, self.hidden_dim).cuda())
        else:
          if self.bidirec==True:
            hidden=(torch.zeros(self.n_layers*2, batch, self.hidden_dim),torch.zeros(self.n_layers*2, batch, self.hidden_dim))
          else:
            hidden=(torch.zeros(self.n_layers, batch, self.hidden_dim),torch.zeros(self.n_layers, batch, self.hidden_dim))
        return hidden

Hyper parameters and training
learning_rate=0.005
epochs=50
vocab_size = len(vocab_to_int)+1 # +1 for the 0 padding
output_size = 2
embedding_dim = 300
hidden_dim = 256
n_layers = 2
batch_size=23
net=RealOrFakeLSTM(vocab_size, output_size, embedding_dim, hidden_dim, n_layers, True, 0.3)
net.to(device)
criterion=nn.BCELoss()
optimizer=torch.optim.Adam(net.parameters(),lr=learning_rate)
net.train()
loss_arr=np.array([])
lossPerEpoch=np.array([])
for i in range(epochs):
  total_loss=0;
  for input,label in train_loader:
    if train_on_gpu:
      input=input.to(device)
      label=label.to(device)
    optimizer.zero_grad()
    input=input.clone().detach().long()
    out=net(input)
    loss=criterion(out.squeeze(),label.float())
    loss_arr=np.append(loss_arr,loss.cpu().detach().numpy())
    loss.backward()
    optimizer.step()
    total_loss+=loss
  total_loss=total_loss/len(train_loader)
  lossPerEpoch=np.append(lossPerEpoch,total_loss.cpu().detach().numpy())
  print("Epoch ",i,": ",total_loss)
  torch.save(net.state_dict(), Path+"/RealOrFakeLSTM.pt")
  torch.save(net, Path+"/RealOrFakeLSTM.pth")
current_time=str(time.time())
torch.save(net.state_dict(), Path+"/pt/RealOrFakeLSTM"+'_pt_'+current_time+".pt")
torch.save(net, Path+"/pth/RealOrFakeLSTM"+'_pth_'+current_time+".pth")

The total loss values are all almost same, All the outcomes probabilities in the test dataset are exactly same. I am quite new to this, so hyper parameter tuning, i am kinda going with bruteforce, but nothing seems to work, I think my problem is not with the architecture but with the training part, as all the predictions are exactly same.


title: How to draw a scatter plot in Tensorboard Pytorch?

body:
Assuming I want a generic scatter plot drawn in TensorBoard that draws the 1st batch[:, 0], batch[:, 1] of every epoch.
How can that be done in TensorBoard?
An old similar question (2017 january) has a workaround, but I hope we now (2020 december) have the technology for a real solution.
Not enough is my attempt:
if self._current_epoch == 0:
    self.logger.experiment.add_scalars("epoch", {"batch": batch[:, 1]}, batch[:, 0])

Gives me the wonderful error

assert(scalar.squeeze().ndim == 0), 'scalar should be 0D'


title: Maximum Input Length of words/sentences of the Pegasus Model in the Transformers library

body: In the Transformers library, what is the maximum input length of words and/or sentences of the Pegasus model? I read in the Pegasus research paper that the max was 512 tokens, but how many words and/or sentences is that? Also, can you increase the maximum number of 512 tokens?


title: Is it possible to view the code of a torch pretrained network

body:
If you're thinking such a noob while reading the title - yes I am.

I've googled but didn't find a single guide that allowed me to view how a pre-trained torch neural network is designed/coded. I have already downloaded the pre-trained network (file format .t7) and I have torch installed. Can anyone help me view how it is coded (what size filters used, parameters used etc.)?

May be it's not on google because it's not possible? Will be happy to answer any additional questions you have or if anything isn't clear.

Thank You.


title: Import System Modules in a Conda Environment

body:
I have installed caffe and pytorch0.3 in my system path(environment), and there is a project which only works under caffe and history version of pytorch0.2. To solve that I install pytorch0.2 in a Conda environment and I wonder if there is any way can save me from installing a caffe again in this conda environment.

In other words, can I use pytorch0.2 in this Conda environment meanwhile import the system caffe? and how?


title: Some questions about Pytocrh DA-RNN

body:
I have two different kind of data,and I try to Processing the two different data separately,not all put into the model.
I use DA-RNN and have one Encoder&Decoder.I'm trying add new 'encoder_num_hidden' in Decoder to deal with the data,and merge the data from Decoders ,but I always get wrong.
How can I do?Or I just did it wrong?
This is code,it come from here

class Encoder(nn.Module):
"""encoder in DA_RNN."""

def __init__(self, T,
             input_size,
             encoder_num_hidden,
             parallel=False):
    """Initialize an encoder in DA_RNN."""
    super(Encoder, self).__init__()
    self.encoder_num_hidden = encoder_num_hidden
    self.input_size = input_size
    self.parallel = parallel
    self.T = T

    # Fig 1. Temporal Attention Mechanism: Encoder is LSTM
    self.encoder_lstm = nn.LSTM(
        input_size=self.input_size, hidden_size=self.encoder_num_hidden)

    # Construct Input Attention Mechanism via deterministic attention model
    # Eq. 8: W_e[h_{t-1}; s_{t-1}] + U_e * x^k
    self.encoder_attn = nn.Linear(
        in_features=2 * self.encoder_num_hidden + self.T - 1, out_features=1, bias=True)

def forward(self, X):
    """forward.
    Args:
        X
    """
    X_tilde = Variable(X.data.new(
        X.size(0), self.T - 1, self.input_size).zero_())
    X_encoded = Variable(X.data.new(
        X.size(0), self.T - 1, self.encoder_num_hidden).zero_())

    # Eq. 8, parameters not in nn.Linear but to be learnt
    # v_e = torch.nn.Parameter(data=torch.empty(
    #     self.input_size, self.T).uniform_(0, 1), requires_grad=True)
    # U_e = torch.nn.Parameter(data=torch.empty(
    #     self.T, self.T).uniform_(0, 1), requires_grad=True)

    # hidden, cell: initial states with dimention hidden_size
    h_n = self._init_states(X)
    s_n = self._init_states(X)

    for t in range(self.T - 1):
        # batch_size * input_size * (2*hidden_size + T - 1)
        x = torch.cat((h_n.repeat(self.input_size, 1, 1).permute(1, 0, 2),
                       s_n.repeat(self.input_size, 1, 1).permute(1, 0, 2),
                       X.permute(0, 2, 1)), dim=2)

        x = self.encoder_attn(
            x.view(-1, self.encoder_num_hidden * 2 + self.T - 1))

        # get weights by softmax
        alpha = F.softmax(x.view(-1, self.input_size))

        # get new input for LSTM
        x_tilde = torch.mul(alpha, X[:, t, :])

        # encoder LSTM
        self.encoder_lstm.flatten_parameters()
        _, final_state = self.encoder_lstm(
            x_tilde.unsqueeze(0), (h_n, s_n))
        h_n = final_state[0]
        s_n = final_state[1]

        X_tilde[:, t, :] = x_tilde
        X_encoded[:, t, :] = h_n

    return X_tilde, X_encoded

def _init_states(self, X):
    """Initialize all 0 hidden states and cell states for encoder.
    Args:
        X
    Returns:
        initial_hidden_states
    """
    # hidden state and cell state [num_layers*num_directions, batch_size, hidden_size]
    # https://pytorch.org/docs/master/nn.html?#lstm
    initial_states = Variable(X.data.new(
        1, X.size(0), self.encoder_num_hidden).zero_())
    return initial_states

class Decoder(nn.Module):
    """decoder in DA_RNN."""

def __init__(self, T, decoder_num_hidden, encoder_num_hidden):
    """Initialize a decoder in DA_RNN."""
    super(Decoder, self).__init__()
    self.decoder_num_hidden = decoder_num_hidden
    self.encoder_num_hidden = encoder_num_hidden
    self.T = T

    self.attn_layer = nn.Sequential(nn.Linear(2 * decoder_num_hidden + encoder_num_hidden, encoder_num_hidden),
                                    nn.Tanh(),
                                    nn.Linear(encoder_num_hidden, 1))
    self.lstm_layer = nn.LSTM(
        input_size=1, hidden_size=decoder_num_hidden)
    self.fc = nn.Linear(encoder_num_hidden + 1, 1)
    self.fc_final = nn.Linear(decoder_num_hidden + encoder_num_hidden, 1)

    self.fc.weight.data.normal_()

def forward(self, X_encoed, y_prev):
    """forward."""
    d_n = self._init_states(X_encoed)
    c_n = self._init_states(X_encoed)

    for t in range(self.T - 1):

        x = torch.cat((d_n.repeat(self.T - 1, 1, 1).permute(1, 0, 2),
                       c_n.repeat(self.T - 1, 1, 1).permute(1, 0, 2),
                       X_encoed), dim=2)

        beta = F.softmax(self.attn_layer(
            x.view(-1, 2 * self.decoder_num_hidden + self.encoder_num_hidden)).view(-1, self.T - 1))
        # Eqn. 14: compute context vector
        # batch_size * encoder_hidden_size
        context = torch.bmm(beta.unsqueeze(1), X_encoed)[:, 0, :]
        if t < self.T - 1:
            # Eqn. 15
            # batch_size * 1
            y_tilde = self.fc(
                torch.cat((context, y_prev[:, t].unsqueeze(1)), dim=1))
            # Eqn. 16: LSTM
            self.lstm_layer.flatten_parameters()
            _, final_states = self.lstm_layer(
                y_tilde.unsqueeze(0), (d_n, c_n))
            # 1 * batch_size * decoder_num_hidden
            d_n = final_states[0]
            # 1 * batch_size * decoder_num_hidden
            c_n = final_states[1]
    # Eqn. 22: final output
    y_pred = self.fc_final(torch.cat((d_n[0], context), dim=1))

    return y_pred

def _init_states(self, X):
    """Initialize all 0 hidden states and cell states for encoder.
    Args:
        X
    Returns:
        initial_hidden_states
    """
    # hidden state and cell state [num_layers*num_directions, batch_size, hidden_size]
    # https://pytorch.org/docs/master/nn.html?#lstm
    initial_states = Variable(X.data.new(
        1, X.size(0), self.decoder_num_hidden).zero_())
    return initial_states

class DA_rnn(nn.Module):
    """da_rnn."""

def __init__(self, X, y, T,
             encoder_num_hidden,
             decoder_num_hidden,
             batch_size,
             learning_rate,
             epochs,
             parallel=False):
    """da_rnn initialization."""
    super(DA_rnn, self).__init__()
    self.encoder_num_hidden = encoder_num_hidden
    self.decoder_num_hidden = decoder_num_hidden
    self.learning_rate = learning_rate
    self.batch_size = batch_size
    self.parallel = parallel
    self.shuffle = False
    self.epochs = epochs
    self.T = T
    self.X = X
    self.y = y

    self.Encoder = Encoder(input_size=X.shape[1],
                           encoder_num_hidden=encoder_num_hidden,
                           T=T)
    self.Decoder = Decoder(encoder_num_hidden=encoder_num_hidden,
                           decoder_num_hidden=decoder_num_hidden,
                           T=T)

    # Loss function
    self.criterion = nn.MSELoss()

    if self.parallel:
        self.encoder = nn.DataParallel(self.encoder)
        self.decoder = nn.DataParallel(self.decoder)

    self.encoder_optimizer = optim.Adam(params=filter(lambda p: p.requires_grad,
                                                      self.Encoder.parameters()),
                                        lr=self.learning_rate)
    self.decoder_optimizer = optim.Adam(params=filter(lambda p: p.requires_grad,
                                                      self.Decoder.parameters()),
                                        lr=self.learning_rate)

    # Training set
    self.train_timesteps = int(self.X.shape[0] * 0.7)
    self.input_size = self.X.shape[1]

def train(self):
    """training process."""
    iter_per_epoch = int(np.ceil(self.train_timesteps * 1. / self.batch_size))
    self.iter_losses = np.zeros(self.epochs * iter_per_epoch)
    self.epoch_losses = np.zeros(self.epochs)

    n_iter = 0

    for epoch in range(self.epochs):
        if self.shuffle:
            ref_idx = np.random.permutation(self.train_timesteps - self.T)
        else:
            ref_idx = np.array(range(self.train_timesteps - self.T))

        idx = 0

        while (idx < self.train_timesteps):
            # get the indices of X_train
            indices = ref_idx[idx:(idx + self.batch_size)]
            # x = np.zeros((self.T - 1, len(indices), self.input_size))
            x = np.zeros((len(indices), self.T - 1, self.input_size))
            y_prev = np.zeros((len(indices), self.T - 1))
            y_gt = self.y[indices + self.T]

            # format x into 3D tensor
            for bs in range(len(indices)):
                x[bs, :, :] = self.X[indices[bs]:(indices[bs] + self.T - 1), :]
                y_prev[bs, :] = self.y[indices[bs]:(indices[bs] + self.T - 1)]

            loss = self.train_forward(x, y_prev, y_gt)
            self.iter_losses[epoch * iter_per_epoch + idx / self.batch_size] = loss

            idx += self.batch_size
            n_iter += 1

            if n_iter % 50000 == 0 and n_iter != 0:
                for param_group in self.encoder_optimizer.param_groups:
                    param_group['lr'] = param_group['lr'] * 0.9
                for param_group in self.decoder_optimizer.param_groups:
                    param_group['lr'] = param_group['lr'] * 0.9

            self.epoch_losses[epoch] = np.mean(self.iter_losses[range(epoch * iter_per_epoch, (epoch + 1) * iter_per_epoch)])

        if epoch % 10 == 0:
            print "Epochs: ", epoch, " Iterations: ", n_iter, " Loss: ", self.epoch_losses[epoch]

        if epoch == self.epochs - 1:
            y_train_pred = self.test(on_train=True)
            y_test_pred = self.test(on_train=False)
            y_pred = np.concatenate((y_train_pred, y_test_pred))
            plt.ioff()
            plt.figure()
            plt.plot(range(1, 1 + len(self.y)),
                     self.y, label="True")
            plt.plot(range(self.T, len(y_train_pred) + self.T),
                     y_train_pred, label='Predicted - Train')
            plt.plot(range(self.T + len(y_train_pred), len(self.y) + 1),
                     y_test_pred, label='Predicted - Test')
            plt.legend(loc='upper left')
            plt.show()

        # Save files in last iterations
        # if epoch == self.epochs - 1:
        #     np.savetxt('../loss.txt', np.array(self.epoch_losses), delimiter=',')
        #     np.savetxt('../y_pred.txt',
        #                np.array(self.y_pred), delimiter=',')
        #     np.savetxt('../y_true.txt',
        #                np.array(self.y_true), delimiter=',')

def train_forward(self, X, y_prev, y_gt):
    # zero gradients
    self.encoder_optimizer.zero_grad()
    self.decoder_optimizer.zero_grad()

    input_weighted, input_encoded = self.Encoder(
        Variable(torch.from_numpy(X).type(torch.FloatTensor)))
    y_pred = self.Decoder(input_encoded, Variable(
        torch.from_numpy(y_prev).type(torch.FloatTensor)))

    y_true = Variable(torch.from_numpy(
        y_gt).type(torch.FloatTensor))

    y_true = y_true.view(-1, 1)
    loss = self.criterion(y_pred, y_true)
    loss.backward()

    self.encoder_optimizer.step()
    self.decoder_optimizer.step()

    return loss.item()

def val(self):
    """validation."""
    pass

def test(self, on_train=False):
    """test."""

    if on_train:
        y_pred = np.zeros(self.train_timesteps - self.T + 1)
    else:
        y_pred = np.zeros(self.X.shape[0] - self.train_timesteps)

    i = 0
    while i < len(y_pred):
        batch_idx = np.array(range(len(y_pred)))[i : (i + self.batch_size)]
        X = np.zeros((len(batch_idx), self.T - 1, self.X.shape[1]))
        y_history = np.zeros((len(batch_idx), self.T - 1))

        for j in range(len(batch_idx)):
            if on_train:
                X[j, :, :] = self.X[range(batch_idx[j], batch_idx[j] + self.T - 1), :]
                y_history[j, :] = self.y[range(batch_idx[j],  batch_idx[j]+ self.T - 1)]
            else:
                X[j, :, :] = self.X[range(batch_idx[j] + self.train_timesteps - self.T, batch_idx[j] + self.train_timesteps - 1), :]
                y_history[j, :] = self.y[range(batch_idx[j] + self.train_timesteps - self.T,  batch_idx[j]+ self.train_timesteps - 1)]

        y_history = Variable(torch.from_numpy(y_history).type(torch.FloatTensor))
        _, input_encoded = self.Encoder(Variable(torch.from_numpy(X).type(torch.FloatTensor)))
        y_pred[i:(i + self.batch_size)] = self.Decoder(input_encoded, y_history).cpu().data.numpy()[:, 0]
        i += self.batch_size

    return y_pred


title: Extremely large gradient in last layer and small in the rest

body:
I’m very new to training neural nets but foolishly tried to implement my own novel architecture. It’s very similar to a transformer, in that the pipeline takes in a “batch of articles headlines” tensor, passes it through the encoder half of the Transformer, then through several transformations and versions of the transformer encoder. The output is then summed into a single vector and softmaxed. I also wrote a custom loss function.

When I graph the average gradients per layer, it looks like this.

(The last layer’s gradients are huge and the rest are barely existent. This is after clipping gradient to 0.25; before that the last layer's gradient was at around 1e7)

Here’s my relevant code:

def forward(self, example):

        art_mask = Variable((example.Art[:,:,0] != 0).unsqueeze(-2)).cuda()
        Art = Variable(example.Art).cuda()

        # To give context to article embeddings, pass through Transformer Encoder block
        Art = self.encoder_block(self.position_layer(Art), art_mask).cuda()

        Tsf = example.Tsf.cuda().repeat((1,Art.shape[1],1)).cuda()
        # Concanetante Tsf to Art
        Art = torch.cat((Art, Tsf), dim=2).cuda()

        # Convert Art to Ent and construct ent_mask
        Ent = Art[example.EntArt[:,:,0], example.EntArt[:,:,1],:].cuda()
        ent_mask = (example.EntArt[:,:,0] == -1).unsqueeze(-2).cuda()

        # Pass to graph block, alternating layers of Relational Attn and Entity Self Attn
        Ent = self.graph_block(Ent, ent_mask, example.RelEnt).cuda()

        # Slice and reorder Ent into assets tensor
        A = len(self.assets_list)
        Ass = torch.full((A,1), -1, dtype=torch.long).cuda()
        for i,uri in enumerate(self.assets_list):
            if uri in example.AssetIndex:
                Ass[i] = example.AssetIndex[uri]

        Assets = Ent[Ass, :, :].squeeze(1).cuda()
        mask = Ass.unsqueeze(2).repeat(1,Assets.shape[1],Assets.shape[2]).cuda()
        Assets = Assets.masked_fill(mask == -1, -1e9).cuda()

        Assets = Assets.sum(dim = 1).squeeze(1)
        Assets = torch.matmul(Assets, self.W).cuda()

        bias = torch.zeros((1)).cuda()
        Assets = torch.cat((Assets, bias)).cuda()

        return self.softmax(Assets)

"""
prices[i] is normalized closing / opening
:param prices <torch.Tensor(batch_size, len(assets))>
"""
def loss_f(model, XY):
    examples, prices = XY
    portfolios = torch.stack([model.forward(ex) for ex in examples], dim=0)
    prices = Variable(prices)

    # safe asset (US Dollars) at prices[:,-1]
    prices = torch.cat((prices, torch.ones((prices.shape[0], 1), dtype=torch.float)), dim = 1).cuda()

    return -torch.sum(portfolios * prices) / 4 # batch size

The layer with the large gradients is the coefficients and biases for a LayerNorm layer at the end of the transformer-esque architecture


title: Define a custom variable pytorch tensor

body:
I have a problem similar to this one:

There is an input vector of the form (n1, n2), and I want a model where f(n1, n2) = f(n2, n1). I want this model to be linear. For instance a 2x2 matrix that I want to learn. The 2x2 matrix has 4 weights which the training will try to learn. However, the constriction allows a clever reduction of the problem.
f(nvec) = W*nvec, being nvec=(n1, n2).
So, let X = [[0, 1],[1, 0]] be a 2x2 matrix that flips vectors, then f(n2,n1) = W*X*(n1, n2) = f(n1, n2) = W (n1, n2).

Basically, the equality implies that W*X = W, or if W = [[w1, w2], [w3, w4]], then the equality implies W = [[w1, w1], [w3, w3]], i.e. the matrix has only 2 free parameters. How can I define a model like this one in pytorch?

Thanks.


title: Pytorch transfer learning image classification model not converging

body:
I can't figure out if I am calculating the loss incorrectly, but
my training_loss is not converging; I am using the ants and bees dataset from Pytorch site:

https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html (just scroll down a bit)

root_dir = "main_dir"

image_transforms = transforms.Compose([
         transforms.Resize((224,224)),
         transforms.ToTensor(),
         transforms.Normalize(
             mean=[0.485, 0.456, 0.406],
             std=[0.229, 0.224, 0.225]
         )])

# First we create train_data and val_data
train = ImageFolder(root = os.path.join(root_dir, 'train'),
                         transform=image_transforms)
#valid data
valid = ImageFolder(root = os.path.join(root_dir, 'val'),
                       transform=image_transforms)

train_dataloader = DataLoader(train, batch_size=4,
                        shuffle=True)

valid_dataloader = DataLoader(valid, batch_size=4,
                        shuffle=True)

optimizer = optim.SGD(model.parameters(), lr=0.0001, momentum=0.9)
criterion = nn.CrossEntropyLoss() #loss function we want to use

def train_model(optimizer, criterion, model, epochs, train_loader, valid_loader):

    for epoch in range(epochs):
            print('Epoch {}/{}'.format(epoch, epochs - 1))
            print('-' * 10)
            train_loss = 0.0
            model.train()
            for inputs, labels in train_loader:
                inputs = inputs.to(device)
                labels = labels.to(device)

               #allow backprop
                with torch.set_grad_enabled(True):
                    optimizer.zero_grad()
                    outputs = model(inputs)
                    loss = criterion(outputs, labels) #uses softmax -> cross entropy

                    loss.backward()  # calculate the gradients
                    optimizer.step() # apply gradients to parameters

                    # statistics
                    train_loss += loss.item() * inputs.size(0) # total loss instead of average loss

            print('total training loss {:.3f}'.format(train_loss))
            print('training_loss per sample {:.5f}'.format(train_loss/len(train_loader.dataset)))

            model.eval()
            valid_corrects = 0.0
            total_valid_loss = 0.0
            for inputs, labels in valid_loader:

                inputs = inputs.to(device)
                labels = labels.to(device)
               #disallow backprop
                with torch.set_grad_enabled(False):
                    outputs = model(inputs)
                    loss = criterion(outputs, labels)
                    _, preds = torch.max(F.softmax(outputs), dim = 1) #values and indicies

                    total_valid_loss += loss.item() * inputs.size(0) # total loss instead of average loss
                    valid_corrects += torch.sum(preds == labels.data)

            valid_loss_per_sample = total_valid_loss/len(valid_loader.dataset)
            accuracy = valid_corrects.item()/len(valid_loader.dataset)

            print('total valid loss {:.3f}'.format(total_valid_loss))
            print('valid_loss per sample {:.3f}'.format(valid_loss_per_sample))
            print('total accuracy {:.3f}'.format(accuracy))

if __name__ == "__main__":
    epochs = 25
    train_model(optimizer, criterion, model, epochs, train_dataloader, valid_dataloader)

Here is how my loss is behaving for 24 epochs (as you can see
it doesn't seem to converge. I tried to change around the learning rates, but it doesn't seem to help much)

Epoch 5/24
----------
total training loss 35.477
training_loss per sample 0.14540
total valid loss 46.130
valid_loss per sample 0.302
total accuracy 0.850

Epoch 6/24
----------
total training loss 57.476
training_loss per sample 0.23556
total valid loss 55.434
valid_loss per sample 0.362
total accuracy 0.837

Epoch 7/24
----------
total training loss 66.397
training_loss per sample 0.27212
total valid loss 41.781
valid_loss per sample 0.273
total accuracy 0.869
Epoch 8/24
----------
total training loss 61.362
training_loss per sample 0.25148
total valid loss 45.502

Epoch 21/24
----------
total training loss 47.409
training_loss per sample 0.19430
total valid loss 50.288
valid_loss per sample 0.329
total accuracy 0.837
Epoch 22/24
----------
total training loss 35.873
training_loss per sample 0.14702
total valid loss 49.597
valid_loss per sample 0.324
total accuracy 0.863
Epoch 23/24
----------
total training loss 30.655
training_loss per sample 0.12563
total valid loss 44.393
valid_loss per sample 0.290
total accuracy 0.869
Epoch 24/24
----------
total training loss 54.373
training_loss per sample 0.22284
total valid loss 60.207
valid_loss per sample 0.394
total accuracy 0.850


title: Beta-Variational AutoEncoder can't disentangle

body:
I am working on a dummy example with generated heartbeats, and want to first use a VAE to encode the heartbeats and afterwards a simple classifier.

Problem is when i increase the beta above 0.01, the reconstructions become nonsense (see the first image).
And when the beta is low i get a normal autoencoder output with no disentanglement (second image).

I believe the problem may be in my KL divergence or VAE loss function, but i can't seem to find it.
In my encoder i do the reparameterization as such:

enc = self.encoder(x,batch_size, x_lenghts)
mu = self.enc2mean(enc)
logv = self.enc2logv(enc)
std = torch.exp(0.5*logv)
z = torch.randn([batch_size,1, self.encoder_hidden_sizes[-1] * (int(self.bidirectional)+1)]).to(self.device)
z = z * std + mu

And i define the VAE loss as:

def VAE_loss(x, reconstruction, mu, logvar, batch_size, latent_dim, beta=0):
    mse = F.mse_loss(x, reconstruction)
    KLD = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    KLD /= (batch_size * latent_dim)
    return mse + beta*KLD

Full standalone code to reproduce the results is here.

Any insights are appreciated!


title: Have you encountered the similar problem like loss jitter during training?

body:
Background: It's about loss jittering which generates at the beginning stage of every training epoch. When the dataloader loads the first batch data to feed into the network, the loss value always rises suddenly, then returns to normal from the second batch and continues to decline. The curve is so strange. I need your help!

    for epoch in range(begin_epoch, end_epoch):
        print('PROGRESS: %.2f%%' % (100.0 * epoch / end_epoch))

        # set epoch as random seed of sampler while distributed training
        if train_sampler is not None and hasattr(train_sampler, 'set_epoch'):
            train_sampler.set_epoch(epoch)

        # reset metrics
        metrics.reset()

        # set net to train mode
        net.train()

        # clear the paramter gradients
        # optimizer.zero_grad()

        # init end time
        end_time = time.time()

        if isinstance(lr_scheduler, torch.optim.lr_scheduler.ReduceLROnPlateau):
            name, value = validation_monitor.metrics.get()
            val = value[name.index(validation_monitor.host_metric_name)]
            lr_scheduler.step(val, epoch)

        # training
        train_loader_iter = iter(train_loader)
        for nbatch in range(total_size):
            try:
                batch = next(train_loader_iter)
            except StopIteration:
                print('reset loader .. ')
                train_loader_iter = iter(train_loader)
                batch = next(train_loader_iter)
            global_steps = total_size * epoch + nbatch

            os.environ['global_steps'] = str(global_steps)

            # record time
            data_in_time = time.time() - end_time

            # transfer data to GPU
            data_transfer_time = time.time()
            batch = to_cuda(batch)
            data_transfer_time = time.time() - data_transfer_time

            # forward
            forward_time = time.time()
            outputs, loss = net(*batch)
            loss = loss.mean()
            if gradient_accumulate_steps > 1:
                loss = loss / gradient_accumulate_steps
            forward_time = time.time() - forward_time

            # backward
            backward_time = time.time()
            if fp16:
                with amp.scale_loss(loss, optimizer) as scaled_loss:
                    scaled_loss.backward()
            else:
                loss.backward()
            backward_time = time.time() - backward_time

            optimizer_time = time.time()
            if (global_steps + 1) % gradient_accumulate_steps == 0:

                # clip gradient
                if clip_grad_norm > 0:
                    if fp16:
                        total_norm = torch.nn.utils.clip_grad_norm_(amp.master_params(optimizer),
                                                                    clip_grad_norm)
                    else:
                        total_norm = torch.nn.utils.clip_grad_norm_(net.parameters(),
                                                                    clip_grad_norm)
                    if writer is not None:
                        writer.add_scalar(tag='grad-para/Total-Norm',
                                        scalar_value=float(total_norm),
                                        global_step=global_steps)

                optimizer.step()

                # step LR scheduler
                if lr_scheduler is not None and not isinstance(lr_scheduler,
                                                            torch.optim.lr_scheduler.ReduceLROnPlateau):
                    lr_scheduler.step()

                # clear the parameter gradients
                optimizer.zero_grad()
            optimizer_time = time.time() - optimizer_time

            # update metric
            metric_time = time.time()
            metrics.update(outputs)
            if writer is not None and nbatch % 50 == 0:
                with torch.no_grad():
                    for group_i, param_group in enumerate(optimizer.param_groups):
                        writer.add_scalar(tag='Initial-LR/Group_{}'.format(group_i),
                                        scalar_value=param_group['initial_lr'],
                                        global_step=global_steps)
                        writer.add_scalar(tag='LR/Group_{}'.format(group_i),
                                        scalar_value=param_group['lr'],
                                        global_step=global_steps)
                    writer.add_scalar(tag='Train-Loss',
                                    scalar_value=float(loss.item()),
                                    global_step=global_steps)
                    name, value = metrics.get()
                    for n, v in zip(name, value):
                        if 'Logits' in n:
                            writer.add_scalar(tag='Train-Logits/' + n,
                                            scalar_value=v,
                                            global_step=global_steps)
                        else:
                            writer.add_scalar(tag='Train-' + n,
                                            scalar_value=v,
                                            global_step=global_steps)
                    for k, v in outputs.items():
                        if 'score' in k:
                            writer.add_histogram(tag=k,
                                                 values=v,
                                                 global_step=global_steps)

            metric_time = time.time() - metric_time


title: Sagemaker Neo Compilation witn Dynamic Input Data Size

body:
I'm trying to compile a PyTorch model to a raspberry pie 3. The model that I am using does not have any limitation in regards to the width and the height. Sagemaker Neo takes in a Data input configuration parameter which looks like it needs to be fixed from the documentation and the examples.

Data Input Configuration:
Amazon SageMaker needs to know the what the shape of the data matrix is
{"data": [1, 3, 224, 224]}

The above example can be found here.

I need something like the following:

Data Input Configuration:
Amazon SageMaker needs to know the what the shape of the data matrix is
{"data": [1, 3, *, *]}

entering the above yields the following error message:

JSON.parse: unexpected character at line 1 column 17 of the JSON data

and fixing the height and the width yields the following message:

ClientError: InputConfiguration: Invalid PyTorch model or input-shape mismatch. Make sure that inputs are lexically ordered and of the correct dimensionality.

Is there anyway to specify that I want 1 images, with 3 channels, and variable height and width?


title: Why my spatial transformer network does not let gradients flow in pytorch?

body:
I build a spatial transformer network using an example from the documentation https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html
from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import torchvision
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import numpy as np
class STN(nn.Module):
    def __init__(self):
        super(STN, self).__init__()
        self.conv1 = nn.Conv2d(3, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(180180, 50)
        self.fc2 = nn.Linear(50, 10)

        # Spatial transformer localization-network
        self.localization = nn.Sequential(
            nn.Conv2d(3, 8, kernel_size=7),
            nn.MaxPool2d(2, stride=2),
            nn.ReLU(True),
            nn.Conv2d(8, 10, kernel_size=5),
            nn.MaxPool2d(2, stride=2),
            nn.ReLU(True)
        )

        # Regressor for the 3 * 2 affine matrix
        self.fc_loc = nn.Sequential(
            nn.Linear(114080, 32),
            nn.ReLU(True),
            nn.Linear(32, 3 * 2)
        )

        # Initialize the weights/bias with identity transformation
        self.fc_loc[2].weight.data.zero_()
        self.fc_loc[2].bias.data.copy_(torch.tensor([1, 0, 0, 0, 1, 0], dtype=torch.float))

    # Spatial transformer network forward function
    def stn(self, x):
        xs = self.localization(x)
        xs = xs.view(-1, 114080)
        theta = self.fc_loc(xs)
        theta = theta.view(-1, 2, 3)

        grid = F.affine_grid(theta, x.size(), align_corners=True)
        x = F.grid_sample(x, grid, align_corners=True)

        return x

    def forward(self, x):
        # transform the input
        x = self.stn(x)
        return x

Then i want to train it end to end with a segmentation unet network.
for epoch in range(200):
    for i, (tensor_image, target_tensor) in enumerate(farfetchloader):
        tensor_image = tensor_image.to(device)
        target_tensor = target_tensor.to(device)
        res = unet(tensor_image)
        warped = tensor_image * res
        res2 = stn1.forward(warped)

        optimizer.zero_grad()
        optimizer2.zero_grad()
        mse = nn.MSELoss()
        loss = mse(res2, target_tensor)
        for param in stn1.parameters():
            print(param.grad)

        print('----------------------')
        loss.backward()

        for param in stn1.parameters():
            print(param.grad)

However, all the gradients despite the last layer are zeros!


title: How to add new sample to CIFAR10 torchvision?

body:
Hi I want to add my own images to the CIFAR10 dataset in torchvision, how can I do that?
train_data = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=train_transform)
train_data.add # or a workaround!

thanks


title: Three dimensional autoencoder has low MSE loss but gives noisy reconstruction image

body:
I am re-constructing spatio-temporal cuboids of 3-dimensional size with width and height equal to 32 and depth equal to 20. I am using Conv3d layers in my autoencoder architecture.
So my input shape is 32x32x20, which I am reducing to size 2048, and then reconstructing it back to 32x32x20.
The MSE loss of the model has nice convergence even though the reconstruction is just noise.
My encoder architecture:
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv3d-1       [-1, 64, 20, 32, 32]           5,248
         AvgPool3d-2       [-1, 64, 10, 16, 16]               0
            Conv3d-3      [-1, 128, 10, 16, 16]         221,312
         AvgPool3d-4         [-1, 128, 5, 8, 8]               0
            Conv3d-5         [-1, 256, 5, 8, 8]         884,992
         AvgPool3d-6         [-1, 256, 2, 4, 4]               0
            Conv3d-7         [-1, 512, 2, 4, 4]       3,539,456
         AvgPool3d-8         [-1, 512, 1, 2, 2]               0

My decoder architecture:
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
       Interpolate-1         [-1, 512, 2, 4, 4]               0
            Conv3d-2         [-1, 256, 4, 4, 4]       3,539,200
       Interpolate-3         [-1, 256, 8, 8, 8]               0
            Conv3d-4        [-1, 128, 10, 8, 8]         884,864
       Interpolate-5      [-1, 128, 20, 16, 16]               0
            Conv3d-6       [-1, 64, 20, 16, 16]         221,248
       Interpolate-7       [-1, 64, 40, 32, 32]               0
            Conv3d-8        [-1, 3, 40, 32, 32]           5,187
            Conv3d-9        [-1, 3, 20, 32, 32]             246

My reconstruction code:
from mpl_toolkits.axes_grid1 import ImageGrid
# recon_batch is the last batch of the autoencoder output.
# recon_batch has shape (batch_size, 3, 20, 32, 32)
recon_batch = recon_batch.permute(0, 2, 3, 4, 1) # new shape = (batch_size, 20, 32, 32, 3)
recon_batch = recon_batch.detach().cpu()
recon_batch_numpy = recon_batch.detach().cpu().numpy()
for k in range(3):  # K=3 because I want to display 3 frames
    fig = plt.figure(figsize=(4., 4.))
    grid = ImageGrid(fig, 111,
                      nrows_ncols=(4, 4),
                      axes_pad=0.1,
                      )
    images = [recon_batch_numpy[i][k] for i in range(16)]  # 16 because Original Image is of size (128x128), when I make 32x32 patches, 16 sub frames are formed
    for ax, im in zip(grid, images):
        ax.imshow((rgb2gray(im) * 255).astype(np.uint8), cmap='gray', vmin=0, vmax=255)
    name = str(num_epochs) + 'th_figure_' + str(k)
    plt.savefig(name)

Original input image(actual cuboids of size 32x32x20, the image is reconstructed by taking the first frame for depth 0 of each such cuboid):

The reconstructed output image:

If needed, I can also add a loss plot, the loss starts around 10,000 and converges around 200.


title: pytorch model saved from TPU run on CPU

body:
I found interesting model - question generator, but can't run it. I got an error:
Traceback (most recent call last):
  File "qg.py", line 5, in <module>
    model = AutoModelWithLMHead.from_pretrained("/home/user/ml-experiments/gamesgen/t5-base-finetuned-question-generation-ap/")
  File "/home/user/.virtualenvs/hugging/lib/python3.7/site-packages/transformers/modeling_auto.py", line 806, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/home/user/.virtualenvs/hugging/lib/python3.7/site-packages/transformers/modeling_utils.py", line 798, in from_pretrained
    import torch_xla.core.xla_model as xm
ModuleNotFoundError: No module named 'torch_xla'

I briefly googled and found that "torch_xla" is a something that is used to train pytorch model on TPU. But I would like to run it localy on cpu (for inference, of course) and got this error when pytorch tried to load tpu-bound tensors.
How can I fix it?
this is model, which I tried: https://huggingface.co/mrm8488/t5-base-finetuned-question-generation-ap


title: pytorch: how to calculate the loss by for-loop

body:
I want to calculate my-self loss. I want to predict N point by deep learning. Thus, the output of network is N point (N*3).
The numpy calculation should be:
import numpy as np

point1 = np.random.random(size=[10, 30, 3])
point2 = np.random.random(size=[10, 30, 3])

losses = []
for s in range(10):
    loss = 0
    for p in range(30):
        p1 = point1[s, p, :]
        dis = p1 - point2[s, :, :]
        dis = np.linalg.norm(dis, axis=1)
        loss += dis.min()
    losses.append(loss)
print(loss)

In pytorch, the point should be:
point1 = np.random.random(size=[10, 30, 3])
point2 = np.random.random(size=[10, 30, 3])

point1 = torch.from_numpy(point1)
point2 = torch.from_numpy(point2)

How can I calculate the loss in pytorch?
Any suggestion is appreciated！


title: Reading test images for resnet18

body:
I am trying to read an image file and classify and image.
My model is resnet18 I trained it previously and planning to use a different .py script to classify a list of images. This is my network:
PATH = './net.pth'
model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 16)
model_ft.load_state_dict(torch.load(PATH))
model_ft.eval()

And I am trying to read Images this way:
imsize = 256
loader = transforms.Compose([transforms.Scale(imsize), transforms.ToTensor()])
def image_loader(image_name):
    #load image, returns cuda tensor
    image = Image.open(image_name)
    image = loader(image).float()
    image = Variable(image, requires_grad=True)
    image = image.unsqueeze(0)
    return image.cuda()

image = image_loader("dataset/test/9673.png")

model_ft(image)

I am getting this error:
 "Given groups=1, weight of size [64, 3, 7, 7], expected input[1, 4, 676, 256] to have 3 channels, but got 4 channels instead"

I've got recommended to remove the unsqueeze for resnet18, doing that I got the following error:
"Expected 4-dimensional input for 4-dimensional weight [64, 3, 7, 7], but got 3-dimensional input of size [4, 676, 256] instead"

I do not quite understand the problem I am dealing with, how should I read my test set? I'll need to write the class ID-s and the file names into a .txt afterwards.


title: libpytorch macos build: static library eigen_blas_LIBRARY-NOTFOUND not found

body:
When building a static libpytorch on macos, cmake configuration for find_package reports the following warning:
[cmake] CMake Warning at libtorch_from_source/pytorch-build/TorchConfig.cmake:22 (message):
[cmake]   static library eigen_blas_LIBRARY-NOTFOUND not found.
[cmake] Call Stack (most recent call first):
[cmake]   libtorch_from_source/pytorch-build/TorchConfig.cmake:113 (append_torchlib_if_found)
[cmake]   src/CMakeLists.txt:4 (find_package)

And the build later fails with link errors that look related to the missing library.
I build libpytorch with the following command (from the source directory):
mkdir pytorch-build && cd pytorch-build && cmake -DBUILD_SHARED_LIBS=OFF -DUSE_CUDA=OFF -DCMAKE_BUILD_TYPE:STRING=MinSizeRel -DPYTHON_EXECUTABLE:PATH=`which python3` -DCMAKE_INSTALL_PREFIX:PATH=../pytorch-build ../pytorch && cmake --build . --target install

And this is the CMakeLists.txt that contains the find_package:
set(Torch_DIR ${CMAKE_SOURCE_DIR}/libtorch_from_source/pytorch-build)
set(ENV{TORCH_INSTALL_PREFIX} ${CMAKE_SOURCE_DIR}/libtorch_from_source/pytorch-build)
find_package(Torch 1.8.0 EXACT REQUIRED)

add_executable(myexec main.cpp)
target_link_libraries(myexec  PRIVATE "${TORCH_LIBRARIES}")
target_include_directories(myexec PRIVATE "${TORCH_INCLUDE_DIRS}")


title: How to import deep learning models from MATLAB to PyTorch?

body:
I’m trying to import a DNN trained model from MATLAB to PyTorch.
I’ve found solutions for the opposite case (from PyTorch to MATLAB), but no proposed solutions on how to import a trained model from MATLAB to PyTorch.
Any ideas, please?


title: How to include a OneHot in an ONNX coming from PyTorch

body:
I'm using PyTorch to train neural-net and output them into ONNX. I use these models in a Vespa index, which loads ONNXs through TensorRT. I need one-hot-encoding for some features but this is really hard to achieve within the Vespa framework.
Is it possible to embed a one-hot-encoding for some given features inside my ONNX net (e.g. before the network's representation) ?
If so, how should I achieve this based on a PyTorch model ?
I already noticed two things:

ONNX format includes the OneHot operator : see ONNX doc
PyTorch built-in ONNX exporting system not not support OneHot operator : see torch.onnx doc

EDIT 2021/03/11:
Here is my workflow:

training learning-to-rank models via PyTorch
exporting them as ONNX
importing these ONNX into my Vespa index in order to rank any query's results thanks to the ONNX model. Under the hood, Vespa uses TensorRT for inference (so I use Vespa's ONNX model evaluation)


title: encounter error when using custom filter in CNN: RuntimeError: expected backend CUDA and dtype Float but got backend CPU and dtype Float

body:
I want to use a custom filter in CNN. The filter has size 5*5 and each entry is a function of three variables: theta, Lambda, psi. There are two convolutional layers followed by two fully connected layers. I tested my filter on MNIST dataset. But when I run it on GPU, I encounter the error: RuntimeError: expected backend CUDA and dtype Float but got backend CPU and dtype Float. I guess it might be due to how I generate the filter box, but I cannot find where I made mistake exactly. Basically I modified this example code, I only modified the network structure with my custom filter. Training and test remained unchanged. I attach my code here. Thank you!

from __future__ import print_function
import argparse
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
from torchvision import datasets, transforms

class Net(nn.Module):
    def __init__(self, kernel_size, in_channels, channel1, channel2):
        super(Net, self).__init__()
        self.theta1, self.Lambda1, self.psi1, self.bias1 = self.generate_parameters(channel1, in_channels)
        self.filter1 = self.whole_filter(in_channels, channel1, kernel_size, self.theta1, self.Lambda1, self.psi1).cuda()

        self.theta2, self.Lambda2, self.psi2, self.bias2 = self.generate_parameters(channel2, channel1)
        self.filter2 = self.whole_filter(channel1, channel2, kernel_size, self.theta2, self.Lambda2, self.psi2).cuda()
        self.fc1 = nn.Linear(4*4*50, 500)
        self.fc2 = nn.Linear(500, 10)

    def forward(self, x):
        x = F.conv2d(x, self.filter1, bias=self.bias1)
        x = F.max_pool2d(x, 2, 2)
        x = F.conv2d(x, self.filter2, bias=self.bias2)
        x = F.max_pool2d(x, 2, 2)
        x = x.view(-1, 4*4*50)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)

    def generate_parameters(self, dim_out, dim_in):
        theta = nn.Parameter(torch.randn(dim_out, dim_in))
        Lambda = nn.Parameter(torch.randn(dim_out, dim_in))
        psi = nn.Parameter(torch.randn(dim_out, dim_in))
        bias = nn.Parameter(torch.randn(dim_out))
        return theta, Lambda, psi, bias

    def whole_filter(self, in_channels, out_channels, kernel_size, theta_column, Lambda_column, psi_column):
        result = torch.zeros(out_channels, in_channels, kernel_size, kernel_size) # \text{out\_channels} , \frac{\text{in\_channels}}{\text{groups}} , kH , kW
        for i in range(out_channels):
            result[i] = self.one_filter(in_channels, kernel_size, theta_column[i], Lambda_column[i], psi_column[i])
        return result

    def one_filter(self, in_channels, kernel_size, theta, Lambda, psi):
        result = torch.zeros(in_channels, kernel_size, kernel_size)
        for i in range(in_channels):
            result[i] = self.filter_fn(theta[i], Lambda[i], psi[i], kernel_size)
        return result

    def filter_fn(self, theta, Lambda, psi, kernel_size):
        # Bounding box
        half_size = (kernel_size - 1) // 2
        ymin, xmin = -half_size, -half_size
        ymax, xmax = half_size, half_size
        (y, x) = np.meshgrid(np.arange(ymin, ymax + 1), np.arange(xmin, xmax + 1))
        y = torch.FloatTensor(y)
        x = torch.FloatTensor(x)

        # Rotation
        x_theta = x * torch.cos(theta) + y * torch.sin(theta)
        y_theta = -x * torch.sin(theta) + y * torch.cos(theta)

        box = torch.cos(y_theta) * torch.sin(2 * np.pi / Lambda * x_theta + psi)
        return box

def train(args, model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)
        loss.backward(retain_graph=True)
        optimizer.step()
        if batch_idx % args.log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

def test(args, model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:
            data, target = data.to(device), target.to(device)
            output = model(data)
            test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss
            pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

def main():
    # Training settings
    parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
    parser.add_argument('--batch-size', type=int, default=64, metavar='N',
                        help='input batch size for training (default: 64)')
    parser.add_argument('--test-batch-size', type=int, default=1000, metavar='N',
                        help='input batch size for testing (default: 1000)')
    parser.add_argument('--epochs', type=int, default=10, metavar='N',
                        help='number of epochs to train (default: 10)')
    parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                        help='learning rate (default: 0.01)')
    parser.add_argument('--momentum', type=float, default=0.5, metavar='M',
                        help='SGD momentum (default: 0.5)')
    parser.add_argument('--no-cuda', action='store_true', default=False,
                        help='disables CUDA training')
    parser.add_argument('--seed', type=int, default=1, metavar='S',
                        help='random seed (default: 1)')
    parser.add_argument('--log-interval', type=int, default=10, metavar='N',
                        help='how many batches to wait before logging training status')
    parser.add_argument('--save-model', action='store_true', default=False,
                        help='For Saving the current Model')
    args = parser.parse_args()
    use_cuda = not args.no_cuda and torch.cuda.is_available()

    torch.manual_seed(args.seed)

    device = torch.device("cuda" if use_cuda else "cpu")

    kwargs = {'num_workers': 1, 'pin_memory': True} if use_cuda else {}
    train_loader = torch.utils.data.DataLoader(
        datasets.MNIST('../data', train=True, download=True,
                       transform=transforms.Compose([
                           transforms.ToTensor(),
                           transforms.Normalize((0.1307,), (0.3081,))
                       ])),
        batch_size=args.batch_size, shuffle=True, **kwargs)
    test_loader = torch.utils.data.DataLoader(
        datasets.MNIST('../data', train=False, transform=transforms.Compose([
                           transforms.ToTensor(),
                           transforms.Normalize((0.1307,), (0.3081,))
                       ])),
        batch_size=args.test_batch_size, shuffle=True, **kwargs)

    model = Net(5, 1, 20, 50).to(device)
    optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)

    for param in model.parameters():
        print(type(param.data), param.size())

    for epoch in range(1, args.epochs + 1):
        train(args, model, device, train_loader, optimizer, epoch)
        test(args, model, device, test_loader)

    if (args.save_model):
        torch.save(model.state_dict(),"mnist_cnn.pt")

if __name__ == '__main__':
    main()


title: How to reshape an Image in pytorch

body:
I have an image of shape (32, 3, 32, 32). I know it's of the form (batch_size, Channel, Height, Width).

Q. How do I convert it to be (32, 32, 32) overriding the Channel?


title: Couldn't close file error when using pytorch enumerate

body:
I seem to get an error when calling the pytorch enumerate method. From googling just the error message it seems to be a problem with my operating system, macOS catalina. Is there a solution to this that I just couldn't find and/or is there an alternative to using enumerate? (I am running scripts someone else wrote and am not familiar with DataLoaders in pytorch and couldn't find much information about the enumerate method). Here is the full error message:

libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
Traceback (most recent call last):
  File "background_analysis.py", line 313, in <module>
    validation_classifier(fit_tree, "../events/" + data + "/dataframes/bbbb_SR.h5", data_name=data, method_name=method, region='SR', epochs=15)
  File "/Users/Trevor/Desktop/Research/toy4b/python/validation.py", line 18, in validation_classifier
    model.runEpoch(print_all_epochs=True, fit_validation=True, data_name=data_name, method_name=method_name, region=region)
  File "fvt_scripts/model_train.py", line 1270, in runEpoch
    self.validate()
  File "fvt_scripts/model_train.py", line 1132, in validate
    self.evaluate(self.validation, doROC)
  File "fvt_scripts/model_train.py", line 1111, in evaluate
    for i, (J, O, D, Q, y, w) in enumerate(results.evalLoader):
  File "/Users/Trevor/Desktop/Research/myenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "/Users/Trevor/Desktop/Research/myenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 746, in __init__
    self._try_put_index()
  File "/Users/Trevor/Desktop/Research/myenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 861, in _try_put_index
    index = self._next_index()
  File "/Users/Trevor/Desktop/Research/myenv/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 339, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "/Users/Trevor/Desktop/Research/myenv/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 202, in __iter__
    if len(batch) == self.batch_size:
  File "/Users/Trevor/Desktop/Research/myenv/lib/python3.7/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 6750) is killed by signal: Unknown signal: 0.
libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: Couldn't close file
Abort trap: 6


title: Does tensorflow has similar op like torch.scatter_add so we can do batch aware unsorted_segment_sum?

body:
I'd like to use a batch aware version of tf.math.unsorted_segment_sum but this op only has 1D version.
In order to do unsorted_segment_sum for batch data(data shape like [batch_size, 200, 32], segment_ids shape like [batch_size, 200]) I need to do something like below
def unsorted_segment_sum(data, segment_ids, num_segments):
  num_rows = tf.shape(segment_ids)[0]
  rows_idx = tf.range(num_rows)
  rows_idx = tf.cast(rows_idx, segment_ids.dtype)
  segment_ids_per_row = segment_ids + num_segments * tf.expand_dims(rows_idx, axis=1)
  num_segments_ = tf.cast(num_segments * num_rows, segment_ids.dtype)
  seg_sums = tf.math.unsorted_segment_sum(data, segment_ids_per_row, num_segments_)
  result = tf.reshape(seg_sums, [-1, num_segments, get_shape(data, -1)])
  return result

But doing this is slow for it's complexity is proportional to batch size.
For torch we can use scatter_add which is much faster for large batch size.
def unsorted_segment_sum(data, segment_ids, num_segments):
  segment_ids = torch.repeat_interleave(segment_ids.unsqueeze(-1), repeats=data.shape[-1], dim=-1)
  shape = [data.shape[0], num_segments] + list(data.shape[2:])
  tensor = torch.zeros(*shape).scatter_add(1, segment_ids, data)
  return tensor

So can we using similar op like torch.scatter_add in tf ?


title: Max-pooling with complex masks in PyTorch

body:
Suppose I have a matrix src with shape (5, 3) and a boolean matrix adj with shape  (5, 5) as follow,
src = tensor(
       [[ 0,  1,  2],
        [ 3,  4,  5],
        [ 6,  7,  8],
        [ 9, 10, 11],
        [12, 13, 14]]
)

and
adj = tensor(
       [[1, 0, 1, 1, 0],
        [0, 1, 1, 1, 0],
        [1, 1, 0, 1, 1],
        [1, 1, 1, 0, 0],
        [0, 0, 1, 0, 1]])

we can take each row in src as one node embedding, and regard each row in adj as the indicator of which nodes are the neighborhood.
My goal is to operate a max-pooling among all neighborhood node embeddings for each node in src.
For example, as the neighborhood nodes (including itself) for the 0-th node is 0, 2, 3, thus we compute a max-pooling on [0, 1, 2], [6, 7,8], [ 9, 10, 11] and lead an updated embedding [ 9, 10, 11] to update 0-th node in src_update.
A simple solution I wrote is
src_update = torch.zeros_like(src)
for index in range(bool_adj.size(0)):
    list_of_non_zero = bool_adj[index].nonzero().view(-1)
    mat_non_zero = torch.index_select(src, 0, list_of_non_zero)
    output[index] = torch.sum(mat_non_zero, dim=0)
print (output)

with src_update is outputted as
src_update = tensor([[ 9, 10, 11],
        [ 9, 10, 11],
        [12, 13, 14],
        [ 6,  7,  8],
        [12, 13, 14]])

Although it works, but it runs very slowly and looks not elegant!
Any suggestions to improve it for better efficiency?
In addition, if both src and adj are appended with batches ((batch, 5, 3), (batch, 5, 5)), how to make it works?
Thanks a lot!!!


title: Extracting blocks from block diagonal PyTorch tensor

body:
I have a tensor of shape (m*n, m*n) and I want to extract a tensor of size (n, m*n) containing the m blocks of size n*n that are on the diagonal. For example:
>>> a
tensor([[1, 2, 0, 0],
        [3, 4, 0, 0],
        [0, 0, 5, 6],
        [0, 0, 7, 8]])

I want to have a function extract(a, m, n) that will output:
>>> extract(a, 2, 2)
tensor([[1, 2, 5, 6],
        [3, 4, 7, 8]])

I've thought of using some kind of slicing, because the blocks can be expressed by:
>>> for i in range(m):
...     print(a[i*m: i*m + n, i*m: i*m + n])
tensor([[1, 2],
        [3, 4]])
tensor([[5, 6],
        [7, 8]])


title: Increase of GPU memory usage during training

body:
I was training the network on usual MNIST dataset, and encountered the next problem:
when i start to add valid_metrics to a loss_list and accuracy_list, amount of GPU memory that is being used starts increasing with every 1 or 2 epochs.
This is the code of train_loop:
def train_model(model: torch.nn.Module,
                train_dataset: torch.utils.data.Dataset,
                valid_dataset: torch.utils.data.Dataset,
                loss_function: torch.nn.Module = torch.nn.CrossEntropyLoss(),
                optimizer_class: Type[torch.optim.Optimizer] = torch.optim,
                optimizer_params: Dict = {},
                initial_lr = 0.01,
                lr_scheduler_class: Any = torch.optim.lr_scheduler.ReduceLROnPlateau,
                lr_scheduler_params: Dict = {},
                batch_size = 64,
                max_epochs = 1000,
                early_stopping_patience = 20):

    optimizer = torch.optim.Adam(model.parameters(), lr=initial_lr, **optimizer_params)
    lr_scheduler = lr_scheduler_class(optimizer, **lr_scheduler_params)

    train_loader = torch.utils.data.DataLoader(train_dataset, shuffle=True, batch_size=batch_size)
    valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=batch_size)

    best_valid_loss = None
    best_epoch = None
    loss_list = list()
    accuracy_list = list()

    for epoch in range(max_epochs):

        print(f'Epoch {epoch}')

        start = timer()

        train_single_epoch(model, optimizer, loss_function, train_loader)

        valid_metrics = validate_single_epoch(model, loss_function, valid_loader)
        loss_list.append(valid_metrics['loss'])
        accuracy_list.append(valid_metrics['accuracy'])

        print('time:', timer() - start)
        print(f'Validation metrics: \n{valid_metrics}')

        lr_scheduler.step(valid_metrics['loss'])

        if best_valid_loss is None or best_valid_loss > valid_metrics['loss']:
            print(f'Best model yet, saving')
            best_valid_loss = valid_metrics['loss']
            best_epoch = epoch
            torch.save(model, './best_model.pth')

        if epoch - best_epoch > early_stopping_patience:
            print('Early stopping triggered')
            return loss_list, accuracy_list

and the code of validate_single_epoch:
def validate_single_epoch(model: torch.nn.Module,
                          loss_function: torch.nn.Module,
                          data_loader: torch.utils.data.DataLoader):

    loss_total = 0
    accuracy_total = 0

    for data in data_loader:

        X, y = data
        X, y = X.view(-1, 784), y.to(device)
        X = X.to(device)

        output = model(X)

        loss = loss_function(output, y)
        loss_total += loss

        y_pred = output.argmax(dim = 1, keepdim=True).to(device)
        accuracy_total += y_pred.eq(y.view_as(y_pred)).sum().item()

    loss_avg = loss_total / len(data_loader.dataset)
    accuracy_avg = 100.0 * accuracy_total / len(data_loader.dataset)

    return {'loss' : loss_avg, 'accuracy' : accuracy_avg}

I use GeForce MX250 as GPU


title: Implementing a simple optimization algorithm in PyTorch

body:
I'm currently learning PyTorch in order to utilize its open source autograd feature, and as an exercise for myself, I want to implement a simple optimization algorithm that I've already implemented in MATLAB. As a simple example, say I'm trying to solve the problem min_x 1/2 x'Ax - b'x, i.e. find the vector x which minimizes the quantity x'Ax - b'x, given both A and b. A simple gradient descent algorithm with exact line search in MATLAB may look something like this:
% initialize x = zeros(n, 1) where n is the length of b
while residual > tolerance
    grad = A*x - b; % compute the gradient of the objective
    alpha = norm(grad)^2/(grad'*A*grad); % compute step-size alpha by exact line search
    x = x - alpha*grad; % do a gradient step
    residual = norm(grad); % compute residual
    objective = x'*A*x - b'*x; % compute objective value at current iteration

How would I implement this optimization algorithm in PyTorch? Specifically, I would want to do an identical optimization loop, where I replace my own computation of the gradient with Torch's autograd feature. In other words, I want to perform the exact same algorithm as above in PyTorch, except instead of computing the gradient myself, I simply use PyTorch's autograd feature to compute the gradient. In this way, I don't want to call any given optimizer (like SGD or Adam)--just write the algorithm myself with the only difference being that the gradient is computed by PyTorch. I plan to compare the results of the above numpy/MATLAB implementation with the explicit gradient computation vs the PyTorch implementation with what I assume is a numerical approximation of the gradient.
Thank you very much!


How can I crop away a tensor’s constant value padding (padding height and width are the same) with an unknown value and size?
I would think that because the padding surrounding my tensor has a constant value, and the same height / width, that it should be possible to know where to crop the tensor to remove the padding.
import torch

# Test tensor with NCHW dimensions
a = torch.randn(1,4,5,5) # Can have any H & W size

# Create padding for testing
b = torch.nn.functional.pad(a, (2,2,2,2), 'constant', value=1.2) # Value can be any number

c = # equal to a, without being able to use the variables a or b (or their argument values)

NumPy solutions are acceptable as I can easily convert them to PyTorch.
Edit:
pad = torch.where(b[0, 0] - b[0, 0, 0, 0] != 0)[0][0]

x_pad, y_pad = 0, 0
if (b.size(3) % 2) == 0:
    x_pad = 1
if (b.size(2) % 2) == 0:
    y_pad = 1
c = b[:, :, pad : -(pad + y_pad), pad : -(pad + x_pad)]

assert c == a


title: How to perform the Text Similarity using BERT on 10M+ corpus? Using LSH/ ANNOY/ fiass or sklearn?

body:
My idea is to extract the CLS token for all the text in the DB and save it in CSV or somewhere else. So when a new text comes in, instead of using the Cosine Similarity/JAccard/MAnhattan/Euclidean or other distances, I have to use some approximation like LSH, ANN (ANNOY, sklearn.neighbor) or the one given here faiss . How can that be done? I have my code as:
PyTorch:
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, I am a text")).unsqueeze(0)  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

Using Tensorflow:
import tensorflow as tf
from transformers import BertTokenizer, TFBertModel

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = TFBertModel.from_pretrained('bert-base-uncased')
input_ids = tf.constant(tokenizer.encode("Hello, my dog is cute"))[None, :]  # Batch size 1
outputs = model(input_ids)
last_hidden_states = outputs[0]  # The last hidden-state is the first element of the output tuple

and I  think can get the CLS token as: (Please correct if wrong)
last_hidden_states = outputs[0]
cls_embedding = last_hidden_states[0][0]

Please tell me if it's the right way to use and how can I use any of the LSH, ANNOT, faiss or something like that?
So for every text, there'll a 768 length vector and we can create a N(No of texts 10M)x768 matrix, how can I find the Index  of top-5 data points (texts) which are most similar to the given image/embedding/data point?


title: What does the difference between 'torch.backends.cudnn.deterministic=True' and 'torch.set_deterministic(True)'?

body:
My network includes 'torch.nn.MaxPool3d' which throw a RuntimeError when cudnn deterministic flag is on according to the PyTorch docs (version 1.7 - https://pytorch.org/docs/stable/generated/torch.set_deterministic.html#torch.set_deterministic), however, when I inserted the code 'torch.backends.cudnn.deterministic=True' at the beginning of my code, there was no RuntimeError. Why doesn't that code throw a RuntimeError?
I wonder whether that code guarantees the deterministic computation of my training process.


title: Upload data from DICOM files in Torchvision Model

body:
I'm sorry if the question is too basic, but I am just getting started with PyTorch (and Python).
I was trying to follow step by step the instructions here:
https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html
However,I am working with have some DICOM files, that I kept into two directories (CANCER/NOCANCER). I split them with split-folders, to have it structured to be used with the ImageFolder dataset (as done in the tutorial).
I am aware that I only need to load the pixel_arrays extracted from the DICOM files, and I wrote some helper functions to:

read all paths of the .dcm files;
read them and extract the pixel_array;
do a little preprocessing.
Here are the outlines of the helper functions:

import os
import pydicom
import cv2
import numpy as np

def createListFiles(dirName):
   print("Fetching all the files in the data directory...")
   lstFilesDCM =[]
   for root, dir, fileList in os.walk(dirName):
       for filename in fileList:
            if ".dcm" in filename.lower():
               lstFilesDCM.append(os.path.join( root , filename))
   return lstFilesDCM

def castHeight(list):
   lstHeight = []
   min_height = 0
   for filenameDCM in list:
       readfile = pydicom.read_file(filenameDCM)
       lstHeight.append(readfile.pixel_array.shape[0])
       min_height = np.min(lstHeight)
   return  min_height

def castWidth(list):
   lstWidth = []
   min_Width = 0
   for filenameDCM in list:
       readfile = pydicom.read_file(filenameDCM)
       lstWidth.append(readfile.pixel_array.shape[1])
       min_Width = np.min(lstWidth)
   return  min_Width

def Preproc1(listDCM):
   new_height, new_width = castHeight(listDCM), castWidth(listDCM)
   ConstPixelDims = (len(listDCM), int(new_height), int(new_width))

   ArrayDCM = np.zeros(ConstPixelDims, dtype=np.float32)

   ## loop through all the DICOM files
   for filenameDCM in listDCM:
       ## read the file
       ds = pydicom.read_file(filenameDCM)

       mx0 = ds.pixel_array

       ## Standardisation
       imgb = mx0.astype('float32')
       imgb_stand = (imgb - imgb.mean(axis=(0, 1), keepdims=True)) / imgb.std(axis=(0, 1), keepdims=True)

       ## Normalisation
       imgb_norm = cv2.normalize(imgb_stand, None, 0, 1, cv2.NORM_MINMAX)

       ## we make sure that data is saved as a data_array as a numpy array
       data = np.array(imgb_norm)

       ## we save it into ArrayDicom and resize it based 'ConstPixelDims'
       ArrayDCM[listDCM.index(filenameDCM), :, :] =  cv2.resize(data, (int(new_width), int(new_height)), interpolation = cv2.INTER_CUBIC)

   return ArrayDCM

So, now, how do I tell the dataloader to load the data, considering the structure it's in for labelling purposes, but only after doing this extraction and preprocessing on it?
I'm referencing the "Loading data" part of the tutorial in the documentation, that goes:
# Create training and validation datasets
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x]) for x in ['train', 'val']}
# Create training and validation dataloaders
dataloaders_dict = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=batch_size, shuffle=True, num_workers=4) for x in ['train', 'val']}

If it makes any sense, is it possible to do something on the lines of
image_datasets = {x: datasets.ImageFolder(Preproc1(os.path.join(data_dir, x)), data_transforms[x]) for x in ['train', 'val']}

?
Also, another question I have is: is it worth to do a normalisation step in my preprocessing when the tutorial suggests to do a transforms.Normalize ?
I'm really sorry this sounds so vague, I've been trying to solve this for weeks now, but I can't manage.


title: Reshaping 2 numpy dimensions into one with zipping?

body:
I tried searching for this question but I couldn't find anything relevant.

The quickest way to describe the problem is with a simple example:
lets say I have a 2D numpy arrayl like this:

[[0, 1, 2, 3],
[10, 11, 12, 13],
[20, 21, 22, 23]]

So it has the shape [3,6]
I want to reshape it into a 1D array that looks like this:

[0, 10 ,20 ,1 ,11 ,21 ,2 ,12 ,22 ,3 ,13 ,23 ]

Unlike the array we get with reshape:

[ 0,  1,  2,  3, 10, 11, 12, 13, 20, 21, 22, 23]

Now, to the actual problem I have...
I actually have a 3D array and I want to reshape it into 2D array, and I want to do so with the method described above.
And another example would be:

import numpy
a = numpy.array([[[0,1,2],[10,11,12],[20,21,22]],
[[100,101,102],[110,111,112],[120,121,122]],
 [[200,201,202],[210,211,212],[220,221,222]]])
a.shape
a.reshape(3,9)
OUTPUT: array([[  0,   1,   2,  10,  11,  12,  20,  21,  22],
       [100, 101, 102, 110, 111, 112, 120, 121, 122],
       [200, 201, 202, 210, 211, 212, 220, 221, 222]])

And again, I want my output to look like this instead:

[[  0,  10,  20,   1,  11,  21,   2,  12,  22],
 [100, 110, 120, 101, 111, ..................],
 [...........................................]]

EDIT: just for the sake of people who google this problem, I add some search terms that some people might search for:

Interweave dimensions numpy array

Zip dimensions numpy array

Numpy reshape ordered dimensions

Tensor reshape dimensions timesteps


title: Updating parameters of neural network manually in PyTorch

body:
Let's say I have a neural network N where after training it, I want to manually update its parameters. For example, I would like to zero out elements of its N.parameters() which their absolute value is less than some threshold. How can I do that?
If I do

N.parameters()

It gives me a generator which I cannot iterate over its elements. Is there any easier way to access the parameters in PyTorch?