title: XLM/BERT sequence outputs to pooled output with weighted average pooling

body:
Let's say I have a tokenized sentence of length 10, and I pass it to a BERT model.
bert_out = bert(**bert_inp)
hidden_states = bert_out[0]
hidden_states.shape
>>>torch.Size([1, 10, 768])

This returns me a tensor of shape: [batch_size, seq_length, d_model] where each word in sequence is encoded as a 768-dimentional vector
In TensorFlow BERT also returns a so called pooled output which corresponds to a vector representation of a whole sentence.
I want to obtain it by taking a weighted average of sequence vectors and the way I do it is:
hidden_states.view(-1, 10).shape
>>> torch.Size([768, 10])

pooled = nn.Linear(10, 1)(hidden_states.view(-1, 10))
pooled.shape
>>> torch.Size([768, 1])

Is it the right way to proceed, or should I just flatten the whole thing and then apply linear?
Any other ways to obtain a good sentence representation?


title: How does this function traverse the token through slicing, Can you please explain how it selects the elements in the list?

body:
The code I'm referring to:
predicted_index = torch.argmax(predictions[0, -1, :]).item()


title: adding convolutional layer on bert

body:
I’m trying to fine-tune a pre-trainted BERT model by inserting a CNN layer.
but having trouble controlling the size of convolution layer's input.
Here is my bert classifier code
class BERTClassifier(nn.Module):
    def __init__(self,
                 bert,
                 hidden_size = 768,
                 kernel_size = 3,
                 num_classes=2,
                 dr_rate=None,
                 params=None):
        super(BERTClassifier, self).__init__()
        self.bert = bert
        self.dr_rate = dr_rate

        self.conv1 = nn.Sequential(
            nn.Conv1d(hidden_size, hidden_size, kernel_size),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size)
        )

        self.classifier = nn.Linear(hidden_size , num_classes)
        if dr_rate:
            self.dropout = nn.Dropout(p=dr_rate)

    def gen_attention_mask(self, token_ids, valid_length):
        attention_mask = torch.zeros_like(token_ids)
        for i, v in enumerate(valid_length):
            attention_mask[i][:v] = 1
        return attention_mask.float()

    def forward(self, token_ids, valid_length, segment_ids):
        attention_mask = self.gen_attention_mask(token_ids, valid_length)

        _, pooler = self.bert(input_ids = token_ids, token_type_ids = segment_ids.long(), attention_mask = attention_mask.float().to(token_ids.device))
        if self.dr_rate:
            out1 = self.dropout(pooler)
        out2=self.conv1(out1)
        return self.classifier(out2)

and this is the error message
RuntimeError: Expected 3-dimensional input for 3-dimensional weight [768, 768, 3], but got 2-dimensional input of size [64, 768] instead


title: OverflowError: (34, 'Numerical result out of range') using PyTorch learning rate scheduler

body:
I'm using PyTorch's learning rate scheduler CyclicLR with:
scheduler = torch.optim.lr_scheduler.CyclicLR(model.optimizer, base_lr=1e-6, max_lr=1e-3, step_size_up=1000, mode="triangular2", cycle_momentum=False)

However, I sometimes get this error that makes me have to retrain my entire model:
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py in _triangular2_scale_fn(self, x)
    876
    877     def _triangular2_scale_fn(self, x):
--> 878         return 1 / (2. ** (x - 1))
    879
    880     def _exp_range_scale_fn(self, x):

OverflowError: (34, 'Numerical result out of range')

Is there a way to fix this?


title: Batchnormalization over which dimension?

body:
Over which dimension do we calculate the mean and std? Is it over the hidden dimensions of the NN Layer, or over all the samples in the batch for every hidden dimension separately?
In the paper it says we normalize over the batch.
In torch.nn.BatchNorm1d however the input argument is num_features, which makes no sense to me.
Why does pytorch not follow the original paper on Batchnormalization?


title: How to track gradient graph, pytorch

body: I'm running into some memory leak errors in my training loop which make no sense and I think it's due to the fact I'm doing some stuff with the gradient which isn't a very common use case for the gradient. Is there anyway to view the graph generated from tensor operations visually or otherwise?


title: Pytorch Policy iteration agent not improving

body:
I am trying to implement an approach similar to this one: https://arxiv.org/abs/1912.01603
In the policy generation part, the agent tries to generate a policy by the returns of the environment model and the estimated values of a value network, which in turn computes the value of states based on the policy. However, my agent does not seem to be improving and its loss increases steadily because the value network assigns lower and lower values to the states. It seems like the agent converges to always output an action of 1.
Here is the important part of my code:
def agent_training(epochs):
   for p in gen.parameters():
   p.requires_grad_(False)
   for epoch in range(epochs):
    #load samples from training data
    sample= data_agent.__next__().to(device)
    try:
        #check that it is the correct shape
        sample = sample.view(agent_batches,d_state+d_action+d_state+3)
    except:
        continue
    sample = sample[:,:d_state+1]
    states = compute_transitions(sample)
    value_estimates = compute_value_estimates(states)
    train_action_model(value_estimates)
    states = states[:,:,:d_state+1]
    train_value_model(states.detach(),value_estimates.detach())
     #plot losses
    if epoch%20 == 0:
        print(value_estimates[0,0],values(states)[0,0])
        fig, ax1 = plt.subplots()
        ax2 = ax1.twinx()
        ax1.plot(range(len(v_losses)), v_losses, 'g', label = 'Value Loss')
        ax2.plot(range(len(a_losses)), a_losses,'r', label="Agent Loss")
        ax1.set_xlabel("Epoch")
        ax1.set_ylabel("Value Loss")
        ax2.set_ylabel("Agent Loss")
        plt.legend()
        plt.show()
    del states
    del value_estimates

def compute_transitions(states):
   state_tensor = torch.empty(agent_batches,t,d_state+2).to(device)
   s = states
   for i in range(t):
      n_batch = noise(agent_batches).to(device)
      action = agent(s)
      g_out = gen(n_batch,s,action)
      s = g_out[:,:d_state+1].to(device)
      state_tensor[:,i,:] = g_out[:,:d_state+2]
return state_tensor`

def compute_value_estimates(states):
   value_tensor = torch.empty(agent_batches,t,1).to(device)
   for i in range(t):
      value_tensor[:,i,:] = value_estimates(states,i)
   return value_tensor`

def train_action_model(value_estimates):
   a_optimizer.zero_grad()
   #compute loss
   loss = torch.mean(-torch.sum(value_estimates.squeeze(),dim=1))
   a_losses.append(loss)
   loss.backward()
   torch.nn.utils.clip_grad_norm_(agent.parameters(),100)
   a_optimizer.step() `

def train_value_model(states,value_estimates):
   v_optimizer.zero_grad()
   output = values(states)
   loss_fn = torch.nn.MSELoss()
   loss = loss_fn(value_estimates.squeeze(),output.squeeze())
   v_losses.append(loss)
   loss.backward()
   torch.nn.utils.clip_grad_norm_(values.parameters(),100)
   v_optimizer.step()`

def value_estimates(sample,steps_in_future):
   v = torch.zeros(agent_batches,1).to(device)
   for i in range(1,t):
      v += ((1-lambda_v)*(lambda_v**(i-1))*value_estimates_k(sample,i,steps_in_future))
   v += lambda_v**(t-1)*value_estimates_k(sample,t,t)
   return v

def value_estimates_k(sample,k,steps_in_future):
   v = torch.zeros(agent_batches,1).to(device)
   h = min(k,t-steps_in_future)
   for j in range(h):
      reward = (sample[:,j,d_state+1]*(0.99**j))
      v += reward.unsqueeze(dim=1)
   estimate = values(sample[:,h,:d_state+1])
   v = v+ (0.99**h)*estimate
   return v

I hope anyone can help my figure out why my code is not working.


title: How do I predict using using model trained on custom data using harvard pystruct?

body:
I am very new in pytorch. I was trying to train a model using custom data, using the following resources
https://github.com/harvardnlp/pytorch-struct/blob/master/notebooks/BertDependencies.ipynb
I was expecting torch.save, and model.load.... like function calls.
Unfortunately, I couldn't figure it out how, at which point the code is saving a model. And once saved, how can I use that model to predict some task with any example sentence? Please put some lights here? Thank you in advance.


title: How to control number of graphs displayed in a row

body: I would like to display only two graphs in a row in tensorboard. I am using pytorch. Is there any way to control the layout to display the required number of graphs in a row just like in matplotlib subplots? Thanks in advance


title: Pytorch model load requires the model imports at the calling directory

body:
Project
|- models
|-- simple_model.py
|- application
|-- infer_and_do_something.py
|- main.py

Main is where the program runs at the very beginning or could be replaced as jupyter notebook. All implementation of loading and using the trained model occurs in 'infer_and_do_something.py' file.
So I thought the import of the model structure to 'infer_and_do_something.py' is good enough but it seems like it requires the import of the model structure to 'main.py', when I run 'main.py' which calls what is implemented in 'infer_and_do_something.py'. Otherwise, it raises an error like this;
763     unpickler = pickle_module.Unpickler(f, **pickle_load_args)
    764     unpickler.persistent_load = persistent_load
--> 765     result = unpickler.load()
    766
    767     deserialized_storage_keys = pickle_module.load(f, **pickle_load_args)

AttributeError: Can't get attribute 'SimpleModel' on <module '__main__'>

Is there a way that the structure import is limited to only where it is actually used? say, in this case 'infer_and_do_something.py'


title: How to add output of Body Part Regressor to training of segmentation U-Net?

body:
I have finished writing my code to train and validate segmentation architectures (like U-Net, PIPO-Net, DFCN). Currently I am trying to improve my multi-segmentation net with a Body Part Regressor (BPR - much like this one https://arxiv.org/pdf/1707.03891.pdf 1) which maps a CT image-slice to its corresponding location in the body, i.e. outputs a score which represents the height in the body where the input slice is positioned.
So my BPR can tell me exactly what body part/area my volume (or sub-volume) shows. This means I could use partial evidence for my segmentation net, much like telling the net: “don´t look for liver or kidney while segmentating, since the BPR says the volume is only in the upper lung area”. But I do not want to set those partial labels manually. The more advanced use might be that the segmentation-net not only learns what organs are expected from the score but how much of that organ can be expected.
I want to combine the scores (from the trained BPR net) and the segmentation net while training, making this partial evidence a learnable parameter.
Does somebody know how to achieve this?
Thanks!


title: Activate pytorch network nodes without variable assignments?

body:
In tutorials for PyTorch networks, we typically see an implementation such as:
from torch.nn.functional import hardtanh, sigmoid
import torch.nn as nn

class great_network(nn.Module):
    def __init__(self):
        super(great_network, self).__init__()
        self.layer1 = nn.Conv2d(2, 2, 3)
        self.pool_1 = nn.MaxPool2d(1, 1)
        self.layer3 = nn.ConvTranspose2d(2, 2, 3)
        self.out_layer = nn.Conv2d(1, 1, 3)

    def forward(self, x):
        x = hardtanh(self.layer1(x))
        x = self.pool_1(x)
        x = hardtanh(self.layer3(x))
        x = sigmoid(self.out_layer(x))
        return x

net = great_network()
print(net)

great_network(
  (layer1): Conv2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
  (pool_1): MaxPool2d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)
  (layer3): ConvTranspose2d(2, 2, kernel_size=(3, 3), stride=(1, 1))
  (out_layer): Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1))
)

If I were to dynamically change the size of this network to run multiple experiments, I'd have to emulate the above code (resembling a data clump code bloat) without multiple assignments.
Something like this might do:
from torch.nn.functional import hardtanh, sigmoid
import torch.nn as nn
import numpy as np

class not_so_great_network(nn.Module):
    def __init__(self, n):
        super(not_so_great_network, self).__init__()
        self.pre_layers = self.generate_pre_layers(n)
        self.post_layers = self.generate_post_layers(n)
        self.pool = nn.MaxPool2d(1, 1)
        self.out = nn.Conv2d(1, 1, 3)

    def generate_pre_layers(self, layer_num):
        layers = np.empty(layer_num, dtype = object)
        for lay in range(0, len(layers)):
            layers[lay] = nn.Conv2d(2, 2, 3)
        return layers

    def generate_post_layers(self, layer_num):
        layers = np.empty(layer_num, dtype = object)
        for lay in range(0, len(layers)):
            layers[lay] = nn.Conv2d(2, 2, 3)
        return layers

    def forward(self, x):
        for pre in self.pre_layers:
            x = hardtanh(pre(x))
            x = self.pool(x)

        for post in self.post_layers:
            x = hardtanh(post(x))

        x = sigmoid(self.out(x))

        return x

However, not all layers are there:
if __name__ == '__main__':
    layer_num = 5
    net = not_so_great_network(layer_num)
    print(net)

not_so_great_network(
  (pool): MaxPool2d(kernel_size=1, stride=1, padding=0, dilation=1, ceil_mode=False)
  (out): Conv2d(1, 1, kernel_size=(3, 3), stride=(1, 1))
)

I am not assigning the variables because this could be more powerful if I can generate networks of varying sizes without copying and pasting. How can I emulate the output so that I may activate nodes with activation functions later?


title: Error occurred when pytorch reproduced yolo-v1 and loaded voc2012 `__call__() takes 2 positional arguments but 3 were given`

body:
I tried to replicate the code in yolo-v1 with pytorch, but I reported an error when I loaded the voc2012 dataset.
import torch

from torchvision import datasets
from torchvision import transforms

dataTransforms = transforms.Compose([transforms.ToTensor()])
VocTrainSet = datasets.VOCDetection("./pascal-voc-2012", year="2012", image_set="train", transforms=dataTransforms)
trainLoader = torch.utils.data.DataLoader(VocTrainSet)

Error when I execute the following code:
batch_iterator = iter(trainLoader)
images, targets = next(batch_iterator)

The contents of the error report are as follows:
TypeError: __call__() takes 2 positional arguments but 3 were given

enter image description here


title: Using embedding for convolutional features and low dimensional output space

body:
say I have some convolutional features that I wanna pass through an encoder-decoder model and map pixels to a certain vocabulary. In the context of language translation models I saw that the best practice is to map your vocabulary to a lower dimensional space using Embedding layers (I am using PyTorch). Now, in my case the vocabularies are 1. extracted features through convolutions and 2. a set of characters with cardinality of 30. My questions are:

do I need to use embeddings for the first vocabulary, although the these features are continuous and not one hot encodings with respect to a vocabulary's dimensionality?

do I need to use embeddings for the output vocabulary which is low-dimensional?

I have to mention that I trained a model without embeddings to the encoder input (conv features) and mapped the decoder input to a higher dimension (50) and got fairly good results, but I just want to see if my method is correct or there are better practices.
Thanks in advance


title: How to create a slicable torch.Dataset with an underlying pd.Dataframe?

body:
This is a bit long, but is the shortest I could do to describe this problem.
The code I provide is for conceptual understanding, and I request a conceptual answer. Providing full code makes no sense here.

Problem statement:
I have ~100 dataframes, each with ~100000 rows. Each describe a time series.
I want to create a Dataset called FullDataset for all of these Dataframes together, such that as a whole, I will be able to easily slice it, and sample from it by some global indexing. I also need to be able to split it to train, val, test.
Example usages:

torch.utils.data.DataLoader(full_dataset, batch_size=8, num_workers=1)
Split to train, val, test WITH NO PYTHONIC LOOP FOR THE ROWS.

What I have now uses a Pythonic loop for slicing, which makes it indefeasibly slow [not posing full code for question simplicity]:
def __getitem__(self, index: Union[int, slice]):
    if isinstance(index, int):
        if index < 0: index += len(self._in_features_df)
        if index < 0 or index > len(self): raise IndexError(f"The index {index} is out of range")

        data_item = self._in_features_df.iloc[index * self.frames_per_pd_sample: (index + 1) * self.frames_per_pd_sample, :].values.real
        data_item = data_item.T

        label_item = self._out_features_df.iloc[index * self.frames_per_pd_sample: (index + 1) * self.frames_per_pd_sample, :]

        return data_item, label_item  # channels as dim 0
    elif isinstance(index, slice):
        return [self[ii] for ii in range(*index.indices(len(self)))]

When using this mechanism for splitting to train, val, test, this is too slow:
train = full_dataset[:train_size]  # this is a very long python loop
all_val = full_dataset[train_size:]   # this is a very long python loop

Notice this data splitting scheme is easy to explain, but I want to be able to do more complicated ones, based on different slicing.
For example, If I wanted to slice the start and end of each of the 100 Dataframes, each sample having more than a single row (say 10 rows).
Example code for this use case:
train = []
all_val = []

case_lengths = full_dataset.case_lengths
case_start_ind = 0
case_end_ind = 0  # non inclusive
for case_len in case_lengths:
    case_end_ind += case_len

    case_data = full_dataset[case_start_ind:case_end_ind]
    case_train_size = int(self._config.train_size * case_len)
    case_train = case_data[:case_train_size]  # slow
    case_all_val = case_data[case_train_size:]  # slow

    train.extend(case_train) # slow
    all_val.extend(case_all_val) # slow

train_loader = DataLoader(train, batch_size=8, num_workers=1)

Summary

I am looking for a way to keep the slicing functionality, but without
the list comprehension part which is slow.
Keeping everything as Dataframes doesn't work, because Dataloader
doesn't know how to work with pd.Dataframe directly, and because I
have to help it anyway in case samples are not single rows, and this
is best done in the single Dataset's __getitem__
Maybe there is a way to split a dataset without explicitly sampling
it beforehand, and passing THAT to a Dataloader?

This is an ongoing discussion on the topic, but I wonder how this should be solved before torch version 3.0


title: How to run old python and some old libraries(Python2, PyTorch1.0.1...) on DM server?

body: I am currently doing some studies on machine learning. I found a paper related to my research, and I am trying to run the code. I have to run the code in the DM server, which I use ssh to control. It is currently running on Python3, but the code I found runs with Python2 and some old libraries from 2-3 years back. I thought of using Docker, but I also heard that it will be hard to install/run Docker on the server. Please let me know if there are any ideas.


title: Nested computational graphs during forward for videos - PyTorch

body:
I am using pre-trained models on imagenet to build custom video classification models.
In the first phase, we compute the forward on every frame:
outputs = []
for i in range(n_frames):
     outputs.append(pretrained_model(video[i])

Given that “outputs” is updated at each iteration, this loop results in a nested computational graph because the graphs are retained at every iteration causing a cuda out of memory for big models.
Is there a way to do this better ?
Thanks


title: Correct way of loading torchtext model + vocab for inference

body:
I trained a Transformer model for translation. The saved model achieves ~27 BLEU on my test set. When I reload the model in a new session the BLEU score dips to ~0.5. I managed to find out that the issue is probably the fact that I build my vocab anew everytime I start a session instead of loading it from memory, and the itos mapping is changing everytime (I randomly split loaded data into train,dev,test and build vocab from train).
That got me thinking: is there a correct way of loading a vocab from memory so that my inference results are consistent? Minding that I also need to continue training the model so the vocab needs to be part of the TabularDataset.


title: Plotting a BertModel

body: Hello is there an easy way to plot the architecture of an BertModel (Pytorch not Tensorflow)? I would like to get something similar to tf.keras.utils.plot_model


title: Hyperparameter filtering for scalar output tensorboard

body: I am using tensorboard along with pytorch in order to monitor the training metrics of my model: when it comes to hyperparameter optimization, I end up having quite a few scalar events (training loss) with various parameter combinations. Is it possible to filter some of these graphs out based on the value of the hyperparameter? It seems like the hparam feature only lets you record the final metric: ideally I don't want either to use the comment parameter of SummaryWriter  to do so and filter it with a regex as it offers a limited number of characters.


title: Discriminator and Generator loss not computed

body:
I am trying to compute the loss of a discriminator and generator bit for some reason it show this error. RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! How do I resolve it ? Please have a look at my code
    def test_discriminator_loss(logits_real, logits_fake, d_loss_true):
      d_loss = discriminator_loss(logits_real, logits_fake).to(device='cuda')
      print("Maximum error in d_loss: %g"%rel_error(d_loss_true, d_loss))
    test_discriminator_loss(answers['logits_real'], answers['logits_fake'],
                            answers['d_loss_true'])

def test_generator_loss(logits_fake, g_loss_true):
  g_loss = generator_loss(logits_fake)
  print("Maximum error in g_loss: %g"%rel_error(g_loss_true, g_loss))

test_generator_loss(answers['logits_fake'], answers['g_loss_true'])


title: Is it just me or is MobileBERT on huggingface so slow compared to DistilBERT?

body: When I train on DistilBERT I get 14 it/sec whereas I train on MobileBERT I get like 3.8 it/sec on the same gpu. Am I missing something? On the paper it says that MobileBERT is faster than DistilBERT


title: How to implement Value at Risk in Tensorflow1?

body:
I need to implement a VaR/CVaR class in tensorflow1. I've already implemented it in pytorch, but I am not familiar with tensorflow1 and I'd like to seek some help. Could someone familiar with tf1 please help me with this？Below is my implementation of VaR/CVaR in pytorch. Thank you in advance.
    class CVaR(nn.Module):
    def __init__(self, alpha, learning_rate=1e-2):
        super(CVaR, self).__init__()

        self._alpha = alpha
        self._var = nn.Parameter(torch.tensor(1.0), requires_grad=True)

        self._optimizer = torch.optim.Adam([{"params": self._var, "lr": learning_rate}])

    def forward(self, losses):
        return self._var + 1 / self._alpha * torch.relu(losses - self._var)

    def zero_grad(self):
        """Reset CVaR internal optimizer."""
        self._optimizer.zero_grad()

    def step(self):
        """Execute a step of the internal CVaR optimizer."""
        self._optimizer.step()

    @property
    def var(self):
        """Get estimated Value-at-Risk."""
        return self._var

class SoftCVaR(CVaR):
    def __init__(self, alpha, learning_rate=1e-3, temperature=1):
        super(SoftCVaR, self).__init__(alpha, learning_rate=learning_rate)

        self._temperature = temperature

    def forward(self, losses):
        return self._var + self._temperature / self._alpha * torch.log(
            1 + torch.exp((losses - self._var) / self._temperature)
        )


title: How can I comprehend the function of torch_geometric,which is called dense_to_sparse()？

body:
The official documentation describes the function as follows:
"Converts a dense adjacency matrix to a sparse adjacency matrix defined
by edge indices and edge attributes."

    assert tensor.dim() == 2
    index = tensor.nonzero().t().contiguous()
    value = tensor[index[0], index[1]]
    return index, value

So I want to know what is the meaning of converting a dense adjacency matrix to a sparse adjacency matrix?


title: Accuracy differs in each process during pytorch distributed train

body:
I am using torch.distributed.launch and DistributedDataParrallel with 4 GPUs to train various tasks in computer vision. Accuracy and losses are outputed in each process. But I found that the accuracy and losses are different in each process's output.
I wonder if it is normal in distributed train. I followed several tutorials, but got the same results. If I was wrong in something, please tell me where I was wrong.
By the way, I was using SyncBatchNorm.
Thanks in advance.


title: LSTM model doesn't seem to learn from training

body:
I was tasked with an assignment: To write an LSTM acceptor that, given a sequence of chars, will determine whether the sequence is positive ([1-9]+a+[1-9]+b+[1-9]+[1-9]+c+[1-9]+[1-9]+d+[1-9]) or negative (same but with b and c swapped). I am new to LSTMs and how they work, so I based my model of this tutorial by PyTorch: https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/02-intermediate/recurrent_neural_network/main.py
I indexed each char in every sequence and made a tensor out of the indexed sequence. Then I sent it as input to my model below:
class RNN(nn.Module):
    def __init__(self, vocab_size, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.E = nn.Embedding(vocab_size, input_size)
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc1 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):

        x = self.E(x)
        x = x.view(batch_size, sequence_length, -1)
        # Forward propagate LSTM
        out, _ = self.lstm(x)  # out: tensor of shape (batch_size, seq_length, hidden_size)

        # Decode the hidden state of the last time step
        out = torch.sigmoid(self.fc1(out[:, -1, :]))
        return F.softmax(out, dim=1)

However, when I train the model in this way:
model = RNN(len(indexed_vocab), input_size, hidden_size, num_layers, num_classes).to(device)
for param in model.parameters():
    param.requires_grad = True

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (chars, labels) in enumerate(train_loader, 0):
        chars = chars.reshape(-1, sequence_length, input_size).to(device)
        labels = labels.to(device)
        model.train()
        # Forward pass
        outputs = model(chars)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i + 1) % 100 == 0:
            print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                  .format(epoch + 1, num_epochs, i + 1, total_step, loss.item()))

The train loss does not decrease, and I get a validation accuracy of 50%. I would appreciate your help very much, even if it's just pinpointing things I've implemented wrong.


title: Training via Supervised Learning a Neural Network with Multiple Output Layers

body:
I have a neural network that has multiple output layers for softmax probability depending on the state my agent is in. Here is an example of my network:
class policy(nn.Module):
   def __init__(self):
      hidden_layer = 32
      super(policy, self).__init__()
      self.affine1 = nn.Linear(3, hidden_layer)
      self.affine2 = nn.Linear(hidden_layer, hidden_layer)
      self.output1 = nn.Linear(hidden_layer, 10)
      self.output2 = nn.Linear(hidden_layer, 5)
      self.output3 = nn.Linear(hidden_layer, 3)

   def forward(self, x):
      x = torch.nn.functional.relu(self.affine1(x))
      x = torch.nn.functional.relu(self.affine2(x))
      outputprobs1 = torch.nn.functional.softmax(self.output1(x), dim=-1)
      outputprobs2 = torch.nn.functional.softmax(self.output2(x), dim=-1)
      outputprobs3 = torch.nn.functional.softmax(self.output3(x), dim=-1)
      return outputprobs1, outputprobs2, outputprobs3

The softmax probabilities indicate a certain action my agent will perform, but the agent has different actions based on different states. Because I know which actions my agent should perform, I want to train the policy through supervised learning. I am planning to use torch.nn.CrossEntropyLoss(), as it is a multi-classification problem. Moreover, during each episode, the model chooses from the output probabilities multiple times.
For example, let's say that there are 3 states my agent can be in, A, B, and C. In state A, the agent uses output1, in state B, the agent uses output2, and in state C, the agent uses output 3. As a result, one example through an episode might be:

Agent starts in state A: chooses action 9
Now agent is in state C: chooses action 2
Now agent is in state B: chooses action 4
Now agent is in state A: chooses action 5

Here are my following questions:
How would I train this policy through supervised learning? The website for CrossEntropyLoss says I need a input of shape (N, C). So my C here would be 10, 5, and 3 respectively. My thinking is that I would need 3 of these inputs, one for each output layer. As a result, would I need a separate loss function for each output layer, or can I just use one loss function for all of them? Would propagating the loss for one of the output layers affect the others negatively through supervised learning?
Moreover, I would like more clarification on how to obtain the logits needed for the input to CrossEntropyLoss. In order to obtain this, would I use torch.logit(outputprobs1)?
Any help with this is much appreciated, thank you!


title: Slurm cannot call Pytorch

body:
So, I notice when I tried to run slurm by: srun -N 2 python test.py,
I am getting the following output:
[1 2 3]
Traceback (most recent call last):
  File "test.py", line 2, in <module>
    import torch
ImportError: No module named torch
srun: error: clientNode: task 0: Exited with exit code 1

And this is my test.py:
import numpy as np
import torch
if __name__ == "__main__":
    arr = np.array([1,2,3]);
    print(arr)

It looks like torch is imported successfully from my master node, but not on my client node. And I notice from other post like Cannot call Keras from Slurm where people recommended using module load inside a batch file. But is there anyway that I can make slurm recognize the location of pytorch?


title: When using deep learning to do video frame interpolation, I encountered the problem that the loss does not change

body:
There is no change from beginning to end, even the 7 digits after the decimal point.
here is my network :
enter image description here
here is my code

#实例化网络
mynet=Network()
#损失函数
criterion = nn.MSELoss()
#梯度下降
optimizer = optim.SGD(mynet.parameters(), lr=0.001, momentum=0.9)#学习率0.001 动量0.9

#定义训练几轮
epochs=60

dir_path = os.listdir(arguments_strVideo)
#开始训练
for epoch in range(epochs):
    for i,path in tqdm(enumerate(train_videos)):
        trueFrames = [ None, None,None ]
        tempFrames = [ None, None,None ]
        myloss = 0
        zs_all = 1
        in_movie = VideoFileClip(path)
        print("第{0}个视频，该视频帧数为{1}".format(i,in_movie.reader.nframes))
        for intFrame, npyFrame in enumerate(npyFrame[:, :, ::-1] for npyFrame in moviepy.editor.VideoFileClip(filename=path).iter_frames()):
            if intFrame%50==0:
                print("已训练{0}帧,loss为{1}".format(intFrame,myloss*3/zs_all))
            if intFrame%3==0 and intFrame!=0:
                tempFrames[0] = trueFrames[0]
                tempFrames[2] = trueFrames[2]
                tempFrames[1] = estimate(tempFrames[0], tempFrames[2])

                optimizer.zero_grad() #清空梯度
                loss=criterion(tempFrames[1],trueFrames[1])
                loss.backward(retain_graph=True)#反向传播
                optimizer.step()#梯度下降
                myloss+=loss.item()
                zs_all = intFrame

            trueFrames[intFrame%3] = torch.FloatTensor(numpy.ascontiguousarray(npyFrame.transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))
        print(i,"loss:{0}".format(myloss*3/zs_all))
        myloss=0

            # tenFrames[4] = torch.FloatTensor(numpy.ascontiguousarray(npyFrame.transpose(2, 0, 1).astype(numpy.float32) * (1.0 / 255.0)))

hope someone can help me.wuwuwuwu.....


title: Hidden states of a stacked RNN in Tensorflow/Keras

body:
I am trying to implement a stacked GRU using TensorFlow and Keras, similar to the PyTorch implementation torch.nn.GRU().
The PyTorch implementation looks like this:
import torch.nn
import torch

class stacked_GRU(torch.Model):
  def __init__(self, inp_dimension, gru_layers, gru_neurons):
    super(stacked_GRU, self).__init__()
    self.inp_dimension = inp_dimension

    self.rnn = nn.GRU(
      input_size= inp_dimension,
      hidden_size = gru_neurons,
      num_layers = gru_layers
    )

  def forward(self, x, hidden):
    output, hidden = self.rnn(x, hidden)
    return output, hidden

Here, we stack gru_layers of GRU on top of each other. If we have an input of seq_len time steps, and batch size batch, then the output will be the shape of (seq_len, batch, gru_neurons). The hidden state is the last hidden output of each stacked layer, so it has the shape of (gru_layers, batch, gru_neurons).
Now if I want to do the same in Keras, we can use return_state=True and return_sequences=True. This is what I have so far
import tensorflow as tf

input = tf.keras.Input((inp_dimension))
hidden_states = [None for _ in range(gru_layers)]

x = tf.keras.layers.GRU(
  gru_neurons, return_sequences=True, return_state=True)(input, initial_state=hidden_states[0])
for _ in range(gru_layers-2):
  x, hidden_states[_+1] = tf.keras.layers.GRU(
    gru_neurons, return_sequences=True)(x, initial_state=hidden_states[_+1])

x, hidden_states[-1] = tf.keras.layers.GRU(
  gru_neurons, return_sequences=True, return_state=True)(x, initial_state=hidden_states[-1])
model = tf.keras.Model(input, [x, hidden_states])

I am able to get a stacked GRU, but return_state=True only returns the hidden state of a single GRU layer, so I save all the hidden states from each layer in a list. I also want to input the hidden states back into the model like in PyTorch. But for some reason, this model is highly inefficient. A forward and backpass takes almost double the time compared to PyTorch.
Is there a more efficient way to implement a stacked GRU in Keras, like how it is done in PyTorch with the hidden states? What am I doing wrong here? Maybe I am not using the functional API correctly?


title: Is there an Implementation of Pulse-coupled Neural Networks in Python?

body:
I am currently trying to implement a PCNN for defect segmentation in fabrics.
Sadly there is no Implementation I know of in tensorflow or pytorch.
Does anyone know if the developers will implement this type of neural network in the future?
The only Implementation I found was this https://github.com/Stephan-kashkarov/PCNN
Thank you Stephan kashkarov for your work, but I am still trying to find a Tensorflow Implementation.
Maybe someone can help.


title: Pass Value to next Forward computation

body:
I am currently building my own autograd function in pytorch (for scientific purposes).
For what Iam trying to achieve I would like to pass a value to the next forward call from within my forward. So in essence I am searching for a similar functionality like ctx.save_for_backward() but instead of saving a value for the backward pass, I would like to save a value for the next forward pass, to that my forward pass can use a value that the previous forward pass computed.
Is there a functionality like this in pytorch or how would you approach this ?
Thanks for any help regarding this question


title: beam search language translation pytorch

body:
I want beam search code for making language translation PyTorch, I want to use it with sample code in this link.
https://pytorch.org/tutorials/beginner/translation_transformer.html
Because this link is only presented how to greedy decode in the translation process.


title: PyTorch binary label preprocessing

body:
I'm trying to preprocess my label in PyTorch. So, my questions are:

What could be the best way to encode binary labels in PyTorch? Should I convert them to [0, 1], or [[1 0], [0 1]], or [False, True], or either format is fine?

If I'm going to define labels as [0, 1] or [[1 0], [0 1]], do I need to specify a certain dtype?

P.s. my environment is Google Colab and I expect to use GPU. Not sure if that affects the dtype
Thank you for your help.


title: Pytorch: Changing number of neuron of FC layer according to input sentence length

body:
I have a dataset of various sentences length. I tokenize and batch them and send them to a RNN.
For each batch, each sentence array length is the length of the longest sentence in that batch. Thus different batch have different input shape. (The input shape is [batch_size, sentence_length, embedding_size])
class Text_RNN(nn.Module):
  def __init__():
    super(Text_RNN, self).__init__()
    self.rnn = nn.RNN(input_size=embedding_dim,hidden_size=n_hidden,num_layers=n_layers)
    # shape now become ([batch_size, sentence_length, n_hidden])
    self.flatten = nn.Flatten(start_dim=1, end_dim=2)
    self.fc = nn.Linear(sentence_length*n_hidden,1)

  def forward(self,input):
     output, hidden = self.rnn(input)
     output = self.flatten(output)
     len = print(output.shape[1])
     finalout = self.fc(len)(output)
     return finalout

Ideally, the logic of function should go like above. But in reallity, Pytorch do not allow me to do so. What should be the right approach without modifying the input? (e.g. change all sentence in batch to same length?)


title: Why google drive folders are not visible in the google colab project?

body:
I have cloned git respiratory to the google drive folder using google colab but it constantly gives me error like the directory does not exist in the folder tho it is there and is seen when using !ls
The command I am using:
%cd gdrive/My Drive/project_folder/CSRNet-pytorch

The error:
[Errno 2] No such file or directory: 'gdrive/My Drive/project_folder/CSRNet-pytorch'
/content/gdrive/My Drive/project_folder/CSRNet-pytorch

Tho sometimes it works sometimes it doesn't that is why I am confused. I have to mount my drive everytime using this command:
from google.colab import drive
drive.mount('/content/gdrive')

Maybe I am not seeing something but I am not sure where to look to solve this error, I can't access my datasets folder because of this and it can't read the images


title: Evaluate and change the neural achitecture using in RL training using Soft-Actor Critic (SAC)

body:
I am current training a custom environmnet used SAC from StableBaselines3. I am using MlpPolicy  and from what I understand it is a 2 layer 256 neuron neural network by default. I am interested in evaluating the training performance for different neural architectures but the best I could come up with after a lot of googling on how to change the neural architecture is shown in the code snippet below.
policy_kwargs = dict(net_arch=[128, 128, 128])
model = SAC(MlpPolicy, env1, policy_kwargs=policy_kwargs, verbose=1)

The problem is I don't know how to see the neural architecture of an already trained model to see if the neural net actually changed. Is there a way I can find the neural achitecture used from the model.zip file that is created after a training session? During a discussing recently, I was told that there was a way to see the percentage of neuron used during a training to evaluate if a model needs to be trained more and also decide if increasing/decreasing neural layers or neurons can be beneficial but I haven't been able to find any resources on how I could implement it in code.
I am new to RL training and any resources or code samples to achieve the task above would be greatly appreciated.


title: Is there a library that implements image regularization loss on the output of a Pytorch model?

body:
I am doing some sparse CT denoising with UNets right now, where the input is a sparse reconstruction and the output (hopefully) has the streak artifacts removed. I'm computing the loss of the network using MSELoss or SmoothL1Loss in reference to the fully-sampled reconstruction.
Now, what I want to compute besides the conventional loss (where you compute e.g. the l2 norm of the differences between the output and a label) is an additional loss based on the l2 norm of the actual output, similar to the l2 regularization parameter in an iterative reconstruction technique like the regualarized least squares problem
1/2 |Ax - b|^2_2 + beta * |x|^2_2

Do I have to write it myself, or does Pytorch have some functionality to implement a "unary" loss instead of a "binary" loss operation?
(I don't want weight_decay solutions, I'm not interested in regularizing the weights in the network. I'm strictly interested in regularizing the output of the network)


title: What is the difference between baiduCTC and CTCLoss in PyTorch

body:
I trained the CLovaAI's model from scratch. However I could not reproduce the results on the paper. Authors said it's because of implementation of Loss function. Authors used baiudu-CTC (on that time PyTorch did not have CTC Loss function). Now it's available but the results are lower than the results on the paper.

My question is how baiudu-CTC and CTCLoss(PyTorch) may produce
different results while they are the same algorithm.

The row OURS is achieved after retrain the network with CTCLoss
On paper they used baiudu-CTC Loss. Paper


title: Why loading without extension does load but, decreases in performance?

body:
I tried to save and load a torch model
Saved with

    def save(self, path='checkpoint.pth.tar'):
        print('save')
        torch.save({
            'state_dict': self.qNetwork.state_dict(),
            'optimizer_state_dict': self.optimizer.state_dict()
        }, path)

Loaded with
    def load(self, path='checkpoint.pth.tar'):
        print('load:', path)
        checkpoint = torch.load(path)
        self.qNetwork.load_state_dict(checkpoint['state_dict'])
        self.targetNetwork.load_state_dict(checkpoint['state_dict'])
        self.optimizer.load_state_dict(checkpoint['optimizer_state_dict'])

Problem is that I did not know that "pth.tar" extension was there for a reason. I just removed the extension and gave the path arguments checkpointfor both save and load. Now it did work.It loaded and trained again from there. But the problem was that the accuracy of my model decreased dramatically from before saving after loading. Doing Flappy Bird the score 500 and above became under 10. But more weirdly inference worked okay 500 and above. What is the cause of this behavior?


title: Using adversarial example to magnify the difference between two classes

body:
I have a dataset of images of cats and I split the dataset in two. I classify one set of imagee as 1 and the other as 0. This is very difficult to learn. I am thinking to increase the accuracy of my model by increasing the difference between the cats  using an adversarial example. For instance once I place the same adversarial transform on all the images, my algorithm should learn better by magnify the small difference between the classes. Better yet I want to learn an adversarial example that will improve my accuracy. I am using a library called foolbox with pytorch. I know how to create an adversarial example.Using another library is an option if you suggest.
import foolbox as fb

model = Net()
fmodel = fb.PyTorchModel(model, bounds=(0, 1))

attack = fb.attacks.LinfPGD()
epsilons = [0.0, 0.001, 0.01, 0.03, 0.1, 0.3, 0.5, 1.0]
_, advs, success = attack(fmodel, images, labels, epsilons=epsilons)

But i am at a loss to how to learn one adversarial example that will pull one class left and the other right.Help is appreciated.


title: LINK : fatal error LNK1181 while building pointnet++ module

body:
I want to build pointnet++ module but I get the following error message:
(torch) C:\Users\49152\Desktop\point-normals-upsampling-master>python setup.py build_ext --inplace
running build_ext
building 'pointnet2._ext' extension
Emitting ninja build file C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
1.10.2
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Users\49152\anaconda3\envs\torch\lib\site-packages\torch\lib "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib/x64" /LIBPATH:C:\Users\49152\anaconda3\envs\torch\libs /LIBPATH:C:\Users\49152\anaconda3\envs\torch\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.16.27023\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.17763.0\um\x64" c10.lib torch.lib torch_cpu.lib torch_python.lib cudart.lib c10_cuda.lib torch_cuda.lib /EXPORT:PyInit__ext C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\ball_query.obj C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\bindings.obj C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\group_points.obj C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\interpolate.obj C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\sampling.obj C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\ball_query_gpu.obj C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\group_points_gpu.obj C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\interpolate_gpu.obj C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\sampling_gpu.obj /OUT:build\lib.win-amd64-3.7\pointnet2\_ext.cp37-win_amd64.pyd /IMPLIB:C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2/_ext-src/src\_ext.cp37-win_amd64.lib
LINK : fatal error LNK1181: cannot open input file 'C:\Users\49152\Desktop\point-normals-upsampling-master\build\temp.win-amd64-3.7\Release\pointnet2\_ext-src\src\ball_query.obj'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Community\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX86\\x64\\link.exe' failed with exit status 1181

I am using cuda v10.1 with visual studio 2017.
I would appreciate very mush if somebody can guide me to where the problem might be.


title: Error mistmatch m1 and m2, what is the error on my conv layers calculations?

body:
I have the following architecture:
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        # first convolutional layer
        self.conv1 = torch.nn.Conv2d(1,32,5) # (32,220,220) output tensor # (W-F)/S + 1 = (224-5)/1 + 1 = 220

        # first Max-pooling layer
        self.pool1 = torch.nn.MaxPool2d(2,2) # (32,110,110) output tensor

        # second convolutional layer
        self.conv2 = torch.nn.Conv2d(32,64,5) # (64,106,106) output tensor # (W-F)/S + 1 = (110-5)/1 + 1 = 106

        # second Max-pooling layer
        self.pool2 = torch.nn.MaxPool2d(2,2) # (64,53,53) output tensor

        # Fully connected layer
        self.fc1 = torch.nn.Linear(64*53*53, 1000)
        self.fc2 = torch.nn.Linear(1000, 500)
        self.fc3 = torch.nn.Linear(500, 136)
        self.drop1 = nn.Dropout(p=0.4)

The images are of 224 x 224 dimensions. I'm getting the error:

RuntimeError: size mismatch, m1: [10 x 173056], m2: [179776 x 1000] at
/opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/TH/generic/THTensorMath.c:2033

I cannot see where my error is, can someone help me please?


title: How can I get the calibration.txt

body:
def generate_depth_map(calib_dir, velo_filename, cam=2, vel_depth=False):
"""Generate a depth map from velodyne data
"""
# load calibration files
cam2cam = read_calib_file(os.path.join(calib_dir, 'calib_cam_to_cam.txt'))
velo2cam = read_calib_file(os.path.join(calib_dir, 'calib_velo_to_cam.txt'))
velo2cam = np.hstack((velo2cam['R'].reshape(3, 3), velo2cam['T'][..., np.newaxis]))
velo2cam = np.vstack((velo2cam, np.array([0, 0, 0, 1.0])))

# get image shape
im_shape = cam2cam["S_rect_02"][::-1].astype(np.int32)

I　can't download these calibration text file KITTI　dataset official website，so how can Ｉ　get this files.


title: How to resume optimiser in PyTorch with parameter change

body:
I have a NN with 4 heads that I want to train in PyTorch one after the other.
After having trained 1 head, I save the net and I then want to resume the model, change the requires_grad of the already trained head and start training the next head but I get this error:
ValueError: loaded state dict contains a parameter group that doesn’t match the size of optimizer’s group
I get the same error even if the requires_grad is changed after having loaded the model.
I do not get any error if I do not make any change to requires_grad. Meaning that resume works fine.
The problem arises when I load the optimiser. The model can be loaded without problems.
Observe that the the Net structure never changes in time. It is the trainability of some layers that are switched.
How can I solve this problem?


title: pytorch torch/autograd.py error: RuntimeError: gradient computation has been modified by an inplace operation

body:
I'm running this project https://github.com/ShawK91/Evolutionary-Reinforcement-Learning  main.py.
Then got following error:
D:\anaconda\envs\snakes\python.exe C:/Users/lenovo/Desktop/Evolutionary-Reinforcement-Learning-master/main.py
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
D:\anaconda\envs\snakes\lib\site-packages\gym\logger.py:30: UserWarning: WARN: Box bound precision lowered by casting to float32
  warnings.warn(colorize('%s: %s'%('WARN', msg % args), 'yellow'))
Gen/Frames: 1 / 3000  Gen_max_score: -1032.61  Champ_len 200.00  Test_score u/std -1530.89 189.39  Rollout_u/std: -1322.02 134.85  Rollout_mean_eplen: 200.00
Gen/Frames: 2 / 6000  Gen_max_score: -947.52  Champ_len 200.00  Test_score u/std -1269.33 242.65  Rollout_u/std: -1305.89 191.56  Rollout_mean_eplen: 200.00
Traceback (most recent call last):
  File "C:/Users/lenovo/Desktop/Evolutionary-Reinforcement-Learning-master/main.py", line 54, in <module>
    ai.train(args.total_steps)
  File "C:\Users\lenovo\Desktop\Evolutionary-Reinforcement-Learning-master\algos\erl_trainer.py", line 183, in train
    max_fitness, champ_len, all_eplens, test_mean, test_std, rollout_fitness, rollout_eplens = self.forward_generation(gen, test_tracker)
  File "C:\Users\lenovo\Desktop\Evolutionary-Reinforcement-Learning-master\algos\erl_trainer.py", line 103, in forward_generation
    self.learner.update_parameters(s, ns, a, r, done)
  File "C:\Users\lenovo\Desktop\Evolutionary-Reinforcement-Learning-master\algos\sac.py", line 75, in update_parameters
    policy_loss.backward()
  File "D:\anaconda\envs\snakes\lib\site-packages\torch\tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "D:\anaconda\envs\snakes\lib\site-packages\torch\autograd\__init__.py", line 125, in backward
    Variable._execution_engine.run_backward(
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [256, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

Process finished with exit code -1

My system environment:

Anaconda Python version: 3.8.8
System: Win10
packages:

pytorch: 1.6.0
numpy: 1.19.2
Gym: 0.16.0
mujoco-py: 2.0.2.10
tensorboard: 2.5.0


title: How can/should we weight classes in HuggingFace token classification (entity recognition)?

body:
I'm training a token classification (AKA named entity recognition) model with the HuggingFace Transformers library, with a customized data loader.
Like most NER datasets (I'd imagine?) there's a pretty significant class imbalance: A large majority of tokens are other - i.e. not an entity - and of course there's a little variation between the different entity classes themselves.
As we might expect, my "accuracy" metrics are getting distorted quite a lot by this: It's no great achievement to get 80% token classification accuracy if 90% of your tokens are other... A trivial model could have done better!
I can calculate some additional and more insightful evaluation metrics - but it got me wondering... Can/should we somehow incorporate these weights into the training loss? How would this be done using a typical *ForTokenClassification model e.g. BERTForTokenClassification?


title: Looking for a strategy than can be used to approximate the surface used by each class using Deep Learning

body:
I regularly use CNN (such as ResnetXX) or object detection for various projects, but here I'd like to tackle a slightly different problem.
I have pictures containing about 100 objects and illustrating 4 to 6 classes (among 10 classes in total) and these objects fill the whole picture (no background).
I would like to know the ratio of pixels belonging to each class.
Input :

Image

Output :

Class1: 0.12 ( 12% of the surface of the original image belong to class 1

Class2: 0.22

Class3: 0

Class4: 0.13%

...
I do have 3 possible ideas in mind, but for some of them the annotation effort will be quite consequent (especially if I start with an "object detection" approach where I will segment by hand each mask of each object).
I'll be happy to see if any of you have some nice ideas for this slightly different problem (I couldn't find any research papers on the subject but I might have missed some keywords).
Thank you in advance
Dava


title: How to implement a GAN model on Predetor Prey dataset?

body:
I'm trying to use GAN architecture to generate a Predator-Prey dataset.
Obiviosly, the two have a correlation (some mathematical function).
I need to generate their numbers using GAN, without finding the parameters of the equation.
*I'm relatively new in GANs
For example:
Pray     = [50.   50.  50. ...  88.  88.  89.]  # length 10000
Predator = [100.  99.  99. ... 462. 462. 462.]  # length 10000

How would u approch this scenario?
Will I even be able to solve this using only Dense/Linear(+Relu) Layers?
I tried this: (while sampling 1000 values from the original data)
class Generator_distribution(nn.Module):
    def __init__(self):
        # initialize nn Module
        super().__init__()

        self.layers = nn.ModuleList()

        # architecture
        self.layers.append(nn.Linear(1000,64))
        self.layers.append(nn.LeakyReLU())
        self.layers.append(nn.Linear(64,32))
        self.layers.append(nn.LeakyReLU())
        self.layers.append(nn.Linear(32,1))

    def forward(self,input_tensor):
        x = input_tensor
        for l in self.layers:
            x = l(x)
        return x

class Discriminator_distribution(nn.Module):
    def __init__(self):
        super().__init__()

        self.layers = nn.ModuleList()

        self.layers.append(nn.Linear(1000,32))
        self.layers.append(nn.LeakyReLU())
        self.layers.append(nn.Linear(32,16))
        self.layers.append(nn.LeakyReLU())
        self.layers.append(nn.Linear(16,8))
        self.layers.append(nn.LeakyReLU())
        self.layers.append(nn.Linear(8,1))
        self.layers.append(nn.Sigmoid())

    def forward(self,input_tensor):
        x = input_tensor
        for l in self.layers:
            print(x.shape)
            x = l(x)
        return x

# later used :  nn.BCELoss() & Adam()

But It did not work well:


title: Can we do autoregressive using pad_packed_sequence?

body:
I’m curious about if we want to do the autoregressive manner. Is it possible to do with implemented using pad_packed_sequence and pack_padded_sequence input to some recurrent network? Because we need to wait for the output of timestep t. Given this scenario we cannot gather all prediction and packed them. Then what should I do with this. Is there any way to do this without looping to each time step? Also If i want to do it in batch processing is it possible?
Thank you


title: Duplicates in training data of pytorch-biggraph

body:
Should we keep duplicate values while training pytorch-biggraph graph embeddings model?
For example, If I have a value "A -> B" which came in training data 100 times. Should I keep it 100 times or take unique value and give it only once?


title: Data_loader results in "set" calling error

body:
train_data_path = '/content/gdrive/MyDrive/dataset/train'

train_data = datasets.ImageFolder(root=train_data_path, transform=data_transforms)
train_data_loader = torch.utils.data.DataLoader(train_data, batch_size=8, shuffle=True)

for xs, ts in train_data_loader:
  print(xs)
  print(ts)

I'm currently trying to shuffle and transform images in a folder to a dataset of tensors. One issue I'm facing is that this results in a set callable error, and I'm very confused as to why this is happening. The path specified contains a subfolder of images.


title: number of sequence mean in lstm

body: What does the number of sequence mean in lstm? I am working on time series data , stock market monthly data. if I give 1 or more than 1 then what it means? Any lstm detailed where i can see all parameters like


title: Train TFRobertaModel in PyTorch

body:
I want to create a model in PyTorch, where first I use ROBERTA to create the contexualized representation of the input sentence, and then I want to use it and some more inputs (which aren't words) for a prediction task.
That's why I want to use the basic TFRobertaModel to get the contexualized embeddings, but I would also like to train it together with my model (using GPU).
This is my model definition:
roberta_model = TFRobertaModel.from_pretrained("roberta-base")

However, roberta_model doesn't have cuda() or train() or eval() methods (but roberta_model.trainable == True).
How can I train it? What am I missing?
Thanks!


title: Damaged archive after download/PermissionError

body:
When I download the zip archive manually - it works well. However, when I use the download_url method from torch_geometric.data module:
from torch_geometric.data import download_url

download_url(url=url, folder=raw_dir)

my archive is downloaded, but it is damaged. When I try to open it, I get an error: The archive is either unknown format or damaged.
When I use different modules to download it, e.g.:
import requests

r = requests.get(url)

with open(raw_dir, 'wb') as f:
    f.write(r.content)

I receive an error either. Traceback:
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-2-263f5909aa61> in <module>
      3 r = requests.get(url)
      4
----> 5 with open(raw_dir, 'wb') as f:
      6     f.write(r.content)

PermissionError: [Errno 13] Permission denied:

Does anybody understand the problem and knows how to solve this?
Thanks a lot!


title: Add an RNN layer on top of BERT

body:
My BERT class model is the following
class BertModel(nn.Module):
    def __init__(self, num_labels):
        super().__init__()
        self._bert = BertModel.from_pretrained('bert-base-cased')

        for param in self._bert.parameters():
           param.requires_grad = False

        self._head = nn.Linear(768, num_labels, batch_first=True)

    def forward(self, batch, mask):
        b = self._bert(input_ids=batch["input_ids"], attention_mask=batch["attention_mask"])
        pooler = b.last_hidden_state[:, mask]
        return self._head(pooler)

I wanted to add an RNN layer before the last linear one. What changes do I have to make? Is it enough that I feed the last hidden state to an nn.RNN layer?


title: Modify traced model using Pytorch JIT

body:
I have a set of pytorch models that are simple fully connected neural nets and that all have the same architecture.
One of them is the master model and is trained through backpropagation. The others are workers that are only used for inference.
Every now and then, the master model is copied into the workers models so that they are up to date with the master model (which evolves constantly through backpropagation).
Everything works fine using standard pytorch, but I think that "compiling" all models would allow performance gains.

Is it possible to do it using JIT trace considering my usecase ?
How can I deal with the periodic copy from the master to the workers ? (i.e. I should be able to get and set params from compiled models)


title: Can you control the outputs when converting pytorch to CoreML?

body:
❓Question
I'm trying to convert a pytorch model to coreml. The model was based on yolov5.
Here is a netron view our our exported coreml model.

Currently, the architecture has 3 outputs.  You can see one of the outputs in the screenshot, number '740'.
However, we want a different output from coreml.  We need  to get the output before the reshapeStatic and transpose layers. So in this image you can see that we need the last convolution layer instead of 740.
Those reshapeStatic and transpose layers were added by the process which convert the net to coreML, they are not organic layers of yolov5.

Is there any way we can do the conversion to coreml differently in order to have more control over which layers are output?  For example, can we have control over the output layers in the sample code below:

model = ct.convert(
    traced_model,
    inputs=[ct.ImageType(name="input_1", shape=example_input.shape)],
    classifier_config = ct.ClassifierConfig(class_labels) )

Alternatively, is there a way where we can choose at runtime which values to pull out of the coreml model?  For example, is there a way to specify in the code below which layers we want to output?

img = load_image(img_path, resize_to=(img_size, img_size))
//can we specify here which layers to output?
coreml_out_dict = model.predict({'image': img})

Thanks!


title: Python: Exception in device=TPU:0: mean() received an invalid combination of arguments

body:
I am writing the code for a bert model to be run on a TPU but I come across an error. The error occured in the average pooling layer of the model.
Model
class BERTBaseUncased(nn.Module):
    def __init__(self, bert_path):
        super(BERTBaseUncased, self).__init__()
        self.bert_path = bert_path
        self.bert = transformers.BertModel.from_pretrained(self.bert_path)
        self.bert_drop = nn.Dropout(0.3)
        self.out = nn.Linear(768 * 2, 1)

    def forward(
            self,
            ids,
            mask,
            token_type_ids
    ):
        o1, o2 = self.bert(
            ids,
            attention_mask=mask,
            token_type_ids=token_type_ids)

        apool = torch.mean(o1, 1)  # Here is the error
        mpool, _ = torch.max(o1, 1)
        cat = torch.cat((apool, mpool), 1)

        bo = self.bert_drop(cat)
        p2 = self.out(bo)
        return p2

Error
Exception in device=TPU:0: mean() received an invalid combination of arguments - got (str, int), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch_xla/distributed/xla_multiprocessing.py", line 228, in _start_fn
    fn(gindex, *args)
  File "<ipython-input-15-94e926c1f4df>", line 4, in _mp_fn
    a = _run()
  File "<ipython-input-13-986128302ab6>", line 143, in _run
    train_loop_fn(para_loader.per_device_loader(device), model, optimizer, device, scheduler=scheduler)
  File "<ipython-input-13-986128302ab6>", line 22, in train_loop_fn
    token_type_ids=token_type_ids
  File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 577, in __call__
    result = self.forward(*input, **kwargs)
  File "<ipython-input-9-efc64cc6147b>", line 71, in forward
    apool = torch.mean(o1, 1)
TypeError: mean() received an invalid combination of arguments - got (str, int), but expected one of:
 * (Tensor input, *, torch.dtype dtype)
 * (Tensor input, tuple of names dim, bool keepdim, *, torch.dtype dtype, Tensor out)
 * (Tensor input, tuple of ints dim, bool keepdim, *, torch.dtype dtype, Tensor out)

An exception has occurred, use %tb to see the full traceback.

SystemExit: 17

How to fix this?


title: Reducing PyTorch LSTM output to prediction of a smaller (but greater than 1) timespan, while maintaining batches

body:
I’m trying to use 365 days to predict the next 30, for stock prediction, with 5 features (open, high, close, low, volume). I’m using a batch size of 64.
My network looks like this (I'm using batch-first inputs and ouputs for the LSTM):

Input > (64, 365, 5)
LSTM > (64, 365, 200)
Linear > (64, 365, 100)
Permute to (64, 100, 365) to fit into batch normalization
Batched Normalization > (64, 100, 365)
Permute to (64, 365, 100) to fit into ReLU
ReLU > (64, 365, 100)
Linear > (64, 365, 30)

Going roughly based on this for the structure.
The output I want is (64, 30, 5). I'm using batches so that I can normalize the input and output data based on 64-day timespans, as stocks can grow immensely over the years.
All the sources I looked at for this kind of thing do something like lstm_out.view(-1, hidden_dim) with the LSTM output which gives me (23360, 100) and gets rid of the batch organization, which makes me think they are assuming a batch size of 1. I could do (64, 36500) but that’s nothing like what any of the articles I read did, it’s a ridiculous amount of hidden units, and I don’t know how I would get back to (64, 30, 5).


title: Getting TypeError: dict object is not callable while loading yolo pt model model

body:
need small help here.
I am trying to load pytorch model and print results as it was given in `YOLO v5 tutorial.
https://github.com/ultralytics/yolov5/issues/36
So far I did this:
import cv2
import torch
from PIL import Image
import glob

#model
path = "./"
model = torch.load('./last.pt')
CUDA_VISIBLE_DEVICES = "0"

dataset_name = 'test_1'
test_img_path = './' + dataset_name + '/*.png'

test_imgs = sorted(glob.glob(test_img_path))
print(len(test_imgs))

for img in test_imgs:
    #print(img)
    file_name = img.split('/')[-1]
    image = cv2.imread(img)
    img1 = Image.open(img)
    #print(img)
    img2 = cv2.imread(img)[:, :, ::-1]
    imgs = [img2]
    #print(img2)
    results = model(imgs, size=640)
    results.print()
    results.xyxy[0]
    results.pandas().xyxy[0]

I think I missed some thing,
I am getting following error:
TypeError                                 Traceback (most recent call last)
<ipython-input-31-57941c797589> in <module>()
     29     imgs = [img2]
     30     #print(img2)
---> 31     results = model(imgs, size=640)
     32     results.print()
     33     results.xyxy[0]

TypeError: 'dict' object is not callable

It would be a great help if I get some suggestion here.
Thank you
My results.pandas().xyxy[0]
should print something like this
results.pandas().xyxy[0]  # img1 predictions (pandas)
#      xmin    ymin    xmax   ymax  confidence  class    name
# 0  749.50   43.50  1148.0  704.5    0.874023      0  person
# 1  433.50  433.50   517.5  714.5    0.687988     27     tie
# 2  114.75  195.75  1095.0  708.0    0.624512      0  person
# 3  986.00  304.00  1028.0  420.0    0.286865     27     tie


title: Is there a significant speed improvement when using transformers tokenizer over batch compared to per item?

body:
is calling tokenizer on a batch significantly faster than on calling it on each item in a batch? e.g.
encodings = tokenizer(sentences)
# vs
encodings = [tokenizer(x) for x in sentences]


title: Pytorch's Equivalent of keras train_on_batch()

body:
acc_loss = model.train_on_batch([support, smask, query], qmask)

This is a part of code that I want to convert to pytorch but I'm stuck to find out if there's any Pytorch's function as train_on_batch() or any custom function in Pytorch that can serve the purpose.
support,smask,query,qmask ## these are all rank-4 tensors having (batch_size,channel,height,width)

The model is a custom vgg16 encoder and a decoder for segmentation.


title: Understanding Tensor Assignments

body:
Say I create a placeholder for a batch of 3 images -
batch = torch.zeros(3, 3, 256, 256, dtype=torch.uint8)

I have my dummy image -
image = torch.randint(size = (3,256,256), low=0, high=256)

I then do -
batch[0] = image

I am unable to understand the following outputs -
1)
  id(batch[0]) == id(image)
  out: False

Should this not be true as both hold references to the same Tensor object ‘image’?
2)
  id(batch.storage()) == id(image.storage())
  out: True

  batch[0][0][0][0] = 5
  print(image[0][0][0])
  out: 171

Since both ‘batch’ and ‘image’ share the same underlying storage, why does the change in batch[0][0][0][0] not reflect when I print the corresponding element of ‘image’ ?
Thank you!


title: (Libtorch)How to use packed_accessor64 to access tensor elements in CUDA?

body:
The tutorial gives an example about using packed_accessor64 to access tensor elements efficiently as follows. However, I still do not know how to use packed_accessor64. Can anyone give me a more specific example? Thanks.
__global__ void packed_accessor_kernel(
    PackedTensorAccessor64<float, 2> foo,
    float* trace) {
  int i=threadIdx.x
  gpuAtomicAdd(trace, foo[i][i])
}

torch::Tensor foo = torch::rand({12, 12});

// assert foo is 2-dimensional and holds floats.
auto foo_a = foo.packed_accessor64<float,2>();
float trace = 0;

packed_accessor_kernel<<<1, 12>>>(foo_a, &trace);


title: The parameters of a CNN model is returning None

body:
i create a CNN model like this global_model = CNNMnist(args=args). Then i send it to device, set it to train. Then i train my local models, collect the local_weights and the average them to get updated global_model.
Now i am trying to get the items from the .parameters() function but all i get is None as item.grad. When i do the same thing for the local_models, i get the desired output. What am i doing wrong?
global_model.to(device)
global_model.train()
...................
global_weights = average_weights(local_weights)
global_model.load_state_dict(global_weights)
last_update = []
for item in global_model.parameters():
    last_update.append(copy.deepcopy(item.grad))
    print(item.grad)

Output: None None None None None None None None

Any help would be appreciated.


title: Data Structure coding for GDAS NAS

body:
https://github.com/D-X-Y/AutoDL-Projects/issues/99#issuecomment-835713058
how to code the following graph structure ?
which kind of data structure is suitable in this case especially for both forward inference and backward propagation ?
Would https://networkx.org/documentation/stable/tutorial.html#multigraphs be suitable ?
https://gist.github.com/promach/b6f526c56e20f029d68e6f9041c3f5c0#file-gdas-py-L106-L107


title: Correct loss function when using local reparameterization trick on bayes by backprop

body:
I'm trying to implement a simple linear layer according to the bayes by backprop-approach given in Blundell et al.: Weight Uncertainty in Neural Networks.
First, I used regular reparameterization trick where you sample over the weights' distributions and then calculate the loss according to the formular (like stated in the paper):
loss = 1/M [log(q(w|theta)-log(P(w))] - log(P(Di|Wi))

where
M = number of batches
log(q(w|theta)) = log of the variational posterior
log(P(w)) = log of the prior
log(P(Di|wi)) = log of likelihood

My implementation - which orients itself on shishir13sharma - therefore looks as follows:
# calculate gaussian distribution needed to compute variational posterior
weight_log_gaussian_distr = (-math.log(math.sqrt(2 * math.pi)) - torch.log(self.weight_sigma) - ((self.weight - self.weight_mu) ** 2) / (2 * self.weight_sigma ** 2)).sum()
bias_log_gaussian_distr = (-math.log(math.sqrt(2 * math.pi)) - torch.log(self.bias_sigma) - ((self.bias - self.bias_mu) ** 2) / (2 * self.bias_sigma ** 2)).sum()
# calculate variational posterior
self.log_variational_posterior = weight_log_gaussian_distr + bias_log_gaussian_distr
# calculate gaussian prior using scale mixture priors (implemented separately)
self.log_prior = self.weight_prior.log_mixed_prior(self.weight) + self.bias_prior.log_mixed_prior(self.bias)

However, it is recommended to use the local reparameterization trick instead of regular reparameterization as (if I remember it right) it leads to a more stable and accurate optimization process. Local reparameterization means that instead of sampling over the weights, the activation value is computed using the mean-values of the weights and then sampling is performed over the activation's distribution.
My question is:
How do I have to modify the loss function due to the slightly different sampling process? I mean, I'm actually not working with weigths-distributions anymore but with activation-distribution, right? And the loss-function as stated above tries to optimize the weights' distributions which would then not make sense anymore, or does it? Or doesn't it make a difference and I should use exactly the loss function above?
I've noticed that apparently other implementations (like e. g. kumar-shridhar or ThirstyScholar) used totally different loss-functions (or I didn't understand them right).


title: Load pytorch model from S3 bucket

body:
I want to load a pytorch model (model.pt) from a S3 bucket. I wrote the following code:
from smart_open import open as smart_open
import io

load_path = "s3://serial-no-images/yolo-models/model4/model.pt"
with smart_open(load_path) as f:
    buffer = io.BytesIO(f.read())
    model.load_state_dict(torch.load(buffer))

This results in the following error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

One solution would be to download the model locally, but I want to avoid this and load the model directly from S3. Unfortunately, I couldn't find a good solution for that online. Can someone help me out here?


title: AWS sagemaker local IDE + pytorch estimator training manage conda environnement

body:
I try to configure a conda environnement with sagemaker. I won't save a persistent environment on AWS and use it when I run the training.
Is there a solution to this issue?
Thanks


when I use pytorch，I meet with this warning:
/home/yzx/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/functional.py:580: UserWarning: stft will soon require the return_complex parameter be given for real inputs, and will further require that return_complex=True in a future PyTorch release. (Triggered internally at  /opt/conda/conda-bld/pytorch_1616554788289/work/aten/src/ATen/native/SpectralOps.cpp:639.)

Then it stay this without any results.Could you help me?


title: Attach attributes table to fastai image classifier

body:
I’ve been able to apply fastai to benthic foraminifera taxonomy Background information on foraminifera 2… The results i’m getting is simply amazing. However, I remember in one of the lecture in part 1, someone asked the question about the possibility of adding attribute table from a spreadsheet before classification and @jeremy said it was possible in part two of the course.
I’m at that stage now when that knowledge will be helpful…
I’ll appriciate any pointer on how to do this or the video in part 2 that explains the concepts.


title: How to resolve Target out of bounds error in pytorch?

body:
Model and training :
I have used pretrained mobile net and cross entropy loss, I read some answers that said class labels need to start from 0 which I have confirmed .
   ...
    MobileNet = models.mobilenet_v2(pretrained = True)
    for param in MobileNet.parameters():
      param.requires_grad = False
      torch.manual_seed(50)

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(MobileNet.classifier.parameters(), lr=0.001)

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(MobileNet.classifier.parameters(), lr=0.001)
    ...
...
    epochs = 10
    train_losses = []
    test_losses = []
    train_correct= []
    test_correct = []
    start_time = time.time()
    #Labs = []
    for i in range(epochs):
      trn_corr = 0
      tst_corr = 0

      for b, (images, labels) in enumerate(train_loader):
        torch.sub(labels,1)
        if torch.cuda.is_available():
          images = images.cuda()
          labels = labels.cuda()

        b+=1

        y_pred = MobileNet(images)
        #print(labels)
        #break
        #Labs.append(labels)
        #continue
        print(labels)
        print(y_pred)
        loss = criterion(y_pred, labels) # here is the out of bounds on 52 for first batch output below
        #print(y_pred)
        #print(labels)
        break
        predicted = torch.max(y_pred.data, 1)[1]
        batch_corr = (predicted == labels).sum()
        trn_corr += batch.corr

        accuracy = trn_corr.item()*100/(b*batch)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    ...

I have substracted 1 from labels and the labels run from 0 to 100 for 101 classes in the train_loader which I confirmed .
Below is the output
Output :
...

tensor([29, 33, 39, 85, 85, 39,  0, 23, 23, 76, 70,  2,  3, 57, 84, 11, 89, 31,
        59, 46, 83, 46, 52, 82, 44, 19, 52, 25, 33, 89, 77, 56, 22, 66, 45, 60,
        10, 71, 72, 16, 61,  4, 63, 15, 15, 57, 92, 17, 41, 80, 50, 56, 73, 80,
        68, 34, 47, 33, 31, 62, 68, 17, 47, 11])
tensor([[-0.9616, -1.1671, -1.1827],
        [-0.9615, -1.0641, -1.2995],
        [-0.9577, -1.4886, -0.9403],
        [-1.0647, -1.1032, -1.1289],
        [-1.1586, -1.0393, -1.1015],
        [-1.3143, -0.8895, -1.1379],
        [-0.9678, -0.8286, -1.6960],
        [-1.2635, -0.9934, -1.0583],
        [-0.8513, -1.3184, -1.1856],
        [-1.0806, -1.1071, -1.1084],
        [-1.1126, -1.1289, -1.0559],
        [-0.9698, -1.1128, -1.2303],
        [-1.0450, -1.2471, -1.0190],
        [-1.0669, -1.2297, -1.0119],
        [-1.0175, -1.0630, -1.2272],
        [-1.3280, -0.9455, -1.0598],
        [-0.9930, -1.2005, -1.1132],
        [-1.0558, -1.0148, -1.2392],
        [-1.1386, -1.1681, -0.9976],
        [-0.8060, -1.3254, -1.2460],
        [-0.8056, -1.1826, -1.3995],
        [-1.1848, -1.1338, -0.9879],
        [-0.9672, -1.1772, -1.1657],
        [-1.0675, -1.1348, -1.0946],
        [-1.1790, -1.0338, -1.0884],
        [-0.6344, -1.4177, -1.4806],
        [-1.1404, -0.9266, -1.2573],
        [-1.0258, -1.3760, -0.9445],
        [-0.8296, -1.2309, -1.3030],
        [-1.1527, -0.9346, -1.2328],
        [-0.9683, -1.2902, -1.0640],
        [-0.8827, -1.2443, -1.2100],
        [-0.8772, -1.0590, -1.4386],
        [-1.2291, -0.9050, -1.1944],
        [-0.9446, -1.0091, -1.3998],
        [-1.1590, -0.9005, -1.2736],
        [-0.8921, -1.4032, -1.0659],
        [-1.1511, -1.0787, -1.0681],
        [-1.1140, -1.2735, -0.9368],
        [-0.9793, -1.1024, -1.2299],
        [-0.9485, -1.2577, -1.1136],
        [-1.3164, -1.1032, -0.9160],
        [-1.3043, -1.2827, -0.7955],
        [-1.0703, -1.0073, -1.2314],
        [-1.2843, -1.1376, -0.9099],
        [-0.9937, -0.9318, -1.4441],
        [-1.1619, -1.2057, -0.9477],
        [-0.8990, -1.0851, -1.3659],
        [-1.3180, -0.7940, -1.2719],
        [-1.3387, -1.0477, -0.9492],
        [-0.8630, -1.2431, -1.2392],
        [-1.4398, -0.7549, -1.2277],
        [-1.0610, -1.1391, -1.0972],
        [-1.1071, -0.9328, -1.2872],
        [-1.1587, -1.2632, -0.9080],
        [-1.0558, -1.0159, -1.2378],
        [-1.1243, -1.0349, -1.1399],
        [-0.9771, -1.1900, -1.1414],
        [-1.0418, -1.1889, -1.0711],
        [-1.2620, -0.7551, -1.3985],
        [-1.0239, -0.9320, -1.3982],
        [-0.7640, -1.3455, -1.2953],
        [-1.2208, -0.9823, -1.1069],
        [-1.0224, -1.0562, -1.2293]])
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-71-e133409211d4> in <module>()
     25     print(labels)
     26     print(y_pred)
---> 27     loss = criterion(y_pred, labels)
     28     #print(y_pred)
     29     #print(labels)
...

Please help


title: Instance Segmentation with intertwined classes

body:
I've built a simple detectron2 based "wood/plank" instance segmentation model that detects in an image 1) the location of the wooden plank(s) and type (ex: Oak, Walnut, Pine) and 2) it's state (wet, dry, molded) (and which part of the plank is wet).
I've done this by 2x instance segmentation models(based on type & state), yet I feel that I'm approaching this in the wrong manner, and using supercategories will not be adequate neither. Thus, I'm wondering how should one go about this? Any tips or guidance would be highly appreciated.
To illustrate the matter:
A plank that is Oak, has similar visual characteristics to other Oak types, independent whether its is in a wet, dry state or has mold on it. Similarly, a plank that is wet, has similar visual characteristics to other wet planks, independent on whether the type is Oak, Walnut or Pine.
Thanks!


title: Strange error saving and loading PyTorch model from jupyter notebook

body:
I am trying to train a ConvNet using PyTorch. I would like to save and reload my model. I am writing my code in a Jupyter notebook.
I am encountering a very strange error that I have not been able to find online.
When I save and load my model using
torch.save({
            'epoch': epoch + 1,
            'state_dict': model.state_dict(),
            'optimizer': optimizer.state_dict(),
        }, 'checkpoint.pth.tar')

and
checkpoint = torch.load('checkpoint.pth.tar')

everything works fine.
However, if I try to save  or load to a different directory, for example
torch.save({
            'epoch': epoch + 1,
            'state_dict': model.state_dict(),
            'optimizer': optimizer.state_dict(),
        }, 'model_checkpoints/checkpoint.pth.tar')

or
checkpoint = torch.load('model_checkpoints/checkpoint.pth.tar')

I get the following error message:
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-137-4afed4ac9dc5> in <module>
----> 1 checkpoint = torch.load('model_checkpoints/checkpoint.pth.tar')

~\Anaconda3\Anaconda3\envs\streetview\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    569         pickle_load_args['encoding'] = 'utf-8'
    570
--> 571     with _open_file_like(f, 'rb') as opened_file:
    572         if _is_zipfile(opened_file):
    573             # The zipfile reader is going to advance the current file position.

~\Anaconda3\Anaconda3\envs\streetview\lib\site-packages\torch\serialization.py in _open_file_like(name_or_buffer, mode)
    227 def _open_file_like(name_or_buffer, mode):
    228     if _is_path(name_or_buffer):
--> 229         return _open_file(name_or_buffer, mode)
    230     else:
    231         if 'w' in mode:

~\Anaconda3\Anaconda3\envs\streetview\lib\site-packages\torch\serialization.py in __init__(self, name, mode)
    208 class _open_file(_opener):
    209     def __init__(self, name, mode):
--> 210         super(_open_file, self).__init__(open(name, mode))
    211
    212     def __exit__(self, *args):

FileNotFoundError: [Errno 2] No such file or directory: 'model_checkpoints/checkpoint.pth.tar'

This problem is occurring even though such a directory and file do exist! For example, I have double and triple checked that I am not misspelling the name of model_checkpoints, and I saved a checkpoint using the method that works and then manually moved it into model_checkpoints.
I am at my wits ends and would be very grateful for any assistance.


title: How doe some transforms a Tree Neural Net to a Dag Neural Net in Pytorch efficiently and without backward passes errors?

body:
I want to transform my tree neural net to a dag neural net. The forward pass is simple because I can simply detect if I've gone through that node/subtree already by caching a unique name for the node and the tensor. However, in the backward pass it is an issue because even if I reuse those tensors the auto grad engine complains that I've already done the backward pass through those tensors (and correctly so). Is there a way to modify the backward pass in a simple way so to re-use the already computed gradients?
Or is it just easier to abandon my old code and implement these models in Pytorch Geometric (PyG) or with the Deep Graph Library (dgl)?
For the sake of completeness here is the error message:
    train_loss, train_acc = agent.train(n_epoch)
  File "/home/miranda9/ML4Coq/ml4coq-proj-src/agents.py", line 102, in train
    loss.backward()  # each process synchronizes it's gradients in the backward pass
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/tensor.py", line 221, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/miranda9/miniconda3/envs/metalearningpy1.7.1c10.2/lib/python3.8/site-packages/torch/autograd/__init__.py", line 130, in backward
    Variable._execution_engine.run_backward(
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed. Specify retain_graph=True when calling backward the first time.

cross posted:

https://discuss.pytorch.org/t/transforming-a-tree-to-a-dag-in-pytorch-efficiently-and-without-backward-passes-errors/120866


title: Installing Alphapose with pytorch-rocm

body:
I want to use the AlphaPose tool on my AMD GPU Computer and heard that despite nativley only supporting the NVIDA CUDA it can be modified to use pytorch-rocm instead in order to work with an AMD GPU. Does anybody know what modifications are required for it work ?
https://github.com/MVIG-SJTU/AlphaPose


def save_experiment_config(self):
    with open(os.path.join(self.experiment_dir, 'parameters.txt'), 'w') as file:
        config_dict = vars(self.args)
        for k in vars(self.args):
            file.write(f"{k}={config_dict[k]} \n")
        repo = Repository('.')
        file.write(f"git-repo={repo.head.shorthand} \n")

I run the code,but occur  a error "_pygit2.GitError: Repository not found at .",I don't know how can I deal with it?Thank  for you help


title: RuntimeError : Could not allocate bytes object! in torch.save

body:
I'm now facing a problem during saving tensor data with torch.
My teammate provided me the code and I tried to run it with Jupyter Notebook.
sample_data = trainDataset(dir_data = train_path, mode_test = False)
X = list(map(lambda x: x[0], sample_data))
y = list(map(lambda x: x[1], sample_data))

b, c, w, h = X[0].shape

X_tensor = torch.Tensor(len(X), c, w, h)
torch.cat(X, out=X_tensor)
X_tensor.float()
y_tensor = torch.FloatTensor(y)

print("save Y")
torch.save(y_tensor, y_train_path)
print("save X")
torch.save(X_tensor, x_train_path)

According to the teammate, data for X is about 12GB. But before saving X data, runtime error occurs. Is there anybody who knows solution for this problem?


title: Running BERT SQUAD model on GPU

body:
I am using the BERT Squad model to ask the same question on a collection of documents (>20,000). The model currently runs on my CPU and it takes around a minute to process a single document - which means that I'll need several days to complete the program.
I was wondering if I could speed this up by running the model on a GPU. However, I am new to GPUs and I don't know how to send these inputs and the model to the device (Titan xp).
The code is borrowed from Chris McChormick.
import torch
import tensorflow as tf
from transformers import BertForQuestionAnswering
from transformers import BertTokenizer

model = BertForQuestionAnswering.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased-whole-word-masking-finetuned-squad')

'question' and 'answer_text' are the question and the context string respectively.
    input_ids = tokenizer.encode(question, answer_text)

    # ======== Set Segment IDs ========
    # Search the input_ids for the first instance of the `[SEP]` token.
    sep_index = input_ids.index(tokenizer.sep_token_id)

    if len(input_ids)>512:
        input_ids=input_ids[:512]

    num_seg_a = sep_index + 1
    num_seg_b = len(input_ids) - num_seg_a

    # Construct the list of 0s and 1s.
    segment_ids = [0]*num_seg_a + [1]*num_seg_b

    # There should be a segment_id for every input token.
    assert len(segment_ids) == len(input_ids)

    # ======== Evaluate ========
    # Run our example through the model.
    outputs = model(torch.tensor([input_ids]), # The tokens representing our input text.
                    token_type_ids=torch.tensor([segment_ids]), # The segment IDs to differentiate question from answer_text
                    return_dict=True)

    start_scores = outputs.start_logits
    end_scores = outputs.end_logits

I know that I can send the model to the GPU using model.tocuda(). But how do I send the inputs, train the model, and the retreive output from the GPU?


title: NLP: Error while using own pre-trained model

body:
I'm working on NLP task using RoBERTa model. As training last very long I saved my model, but now for some reason, part of my code doesn't work with this pre-trained model (getting an error), and before saving and uploading newly it, the code worked fine with the trained model. My code:
model = model.from_pretrained("/content/drive/MyDrive/model")
model = model.cuda()
model.eval()

def to_list(tensor):
    return tensor.detach().cpu().tolist()

def run_prediction(question_texts, context_text):
    """Setup function to compute predictions"""
    examples = []

    for i, question_text in enumerate(question_texts):
        example = SquadExample(
            qas_id=str(i),
            question_text=question_text,
            context_text=context_text,
            answer_text=None,
            start_position_character=None,
            title="Predict",
            is_impossible=False,
            answers=None,
        )

        examples.append(example)

    features, dataset = squad_convert_examples_to_features(
        examples=examples,
        tokenizer=tokenizer,
        max_seq_length=384,
        doc_stride=128,
        max_query_length=64,
        is_training=False,
        return_dataset="pt",
      #  threads=1,
    )

    eval_sampler = SequentialSampler(dev_dataset)
    eval_dataloader = DataLoader(dev_dataset, sampler=eval_sampler, batch_size=10)

    all_results = []

    for batch in eval_dataloader:
        model.eval()
        batch = tuple(t.to(device) for t in batch)

        with torch.no_grad():
            inputs = {
                "input_ids": batch[0],
                "attention_mask": batch[1],
                "token_type_ids": batch[2],
            }

            example_indices = batch[3]

            outputs = model(**inputs, return_dict=False)

            for i, example_index in enumerate(example_indices):
                eval_feature = features[example_index.item()]
                unique_id = int(eval_feature.unique_id)

                output = [to_list(output[i]) for output in outputs]

                start_logits, end_logits = output
                result = SquadResult(unique_id, start_logits, end_logits)
                all_results.append(result)

    output_prediction_file = "predictions.json"
    output_nbest_file = "nbest_predictions.json"
    output_null_log_odds_file = "null_predictions.json"

    predictions = compute_predictions_logits(
        examples,
        features,
        all_results,
        n_best_size,
        max_answer_length,
        do_lower_case,
        output_prediction_file,
        output_nbest_file,
        output_null_log_odds_file,
        False,  # verbose_logging
        True,  # version_2_with_negative
        null_score_diff_threshold,
        tokenizer,
    )

    return predictions

context = "New Zealand (Māori: Aotearoa) is a sovereign island country in the southwestern Pacific Ocean. It has a total land area of 268,000 square kilometres (103,500 sq mi), and a population of 4.9 million. New Zealand's capital city is Wellington, and its most populous city is Auckland."
questions = ["How many people live in New Zealand?",
             "What's the largest city?"]

predictions = run_prediction(questions, context)

# Print results
for key in predictions.keys():
  print(predictions[key])

The received error:

Any ideas how to solve this issue?
(this code part mostly is based on this notebook:
https://colab.research.google.com/github/spark-ming/albert-qa-demo/blob/master/Question_Answering_with_ALBERT.ipynb)