title: TorchScript requires source access in order to carry out compilation for collections.deque

body:
I'm trying to convert PyTorch FOMM model to TorchScript. As soon as I started to annotate some classes with @torch.jit.script I've got an error:
OSError: Can't get source for <class 'collections.deque'>. TorchScript requires source access in order to carry out compilation, make sure original .py files are available.
As I understand that class implemented in CPython therefore cannot be read by TorchScript compiler. I failed to find any pure-Python implementation. How can I overcome this issue?
Here is the class I'm trying to annotate:
import queue
import collections
import threading
import torch

@torch.jit.script
class SyncMaster(object):
    """An abstract `SyncMaster` object.

    - During the replication, as the data parallel will trigger an callback of each module, all slave devices should
    call `register(id)` and obtain an `SlavePipe` to communicate with the master.
    - During the forward pass, master device invokes `run_master`, all messages from slave devices will be collected,
    and passed to a registered callback.
    - After receiving the messages, the master device should gather the information and determine to message passed
    back to each slave devices.
    """

    def __init__(self, master_callback):
        """

        Args:
            master_callback: a callback to be invoked after having collected messages from slave devices.
        """
        self._master_callback = master_callback
        self._queue = queue.Queue()
        self._registry = collections.OrderedDict()
        self._activated = False

    def __getstate__(self):
        return {'master_callback': self._master_callback}

    def __setstate__(self, state):
        self.__init__(state['master_callback'])

    def register_slave(self, identifier):
        """
        Register an slave device.

        Args:
            identifier: an identifier, usually is the device id.

        Returns: a `SlavePipe` object which can be used to communicate with the master device.

        """
        if self._activated:
            assert self._queue.empty(), 'Queue is not clean before next initialization.'
            self._activated = False
            self._registry.clear()
        future = FutureResult()
        self._registry[identifier] = _MasterRegistry(future)
        return SlavePipe(identifier, self._queue, future)

    def run_master(self, master_msg):
        """
        Main entry for the master device in each forward pass.
        The messages were first collected from each devices (including the master device), and then
        an callback will be invoked to compute the message to be sent back to each devices
        (including the master device).

        Args:
            master_msg: the message that the master want to send to itself. This will be placed as the first
            message when calling `master_callback`. For detailed usage, see `_SynchronizedBatchNorm` for an example.

        Returns: the message to be sent back to the master device.

        """
        self._activated = True

        intermediates = [(0, master_msg)]
        for i in range(self.nr_slaves):
            intermediates.append(self._queue.get())

        results = self._master_callback(intermediates)
        assert results[0][0] == 0, 'The first result should belongs to the master.'

        for i, res in results:
            if i == 0:
                continue
            self._registry[i].result.put(res)

        for i in range(self.nr_slaves):
            assert self._queue.get() is True

        return results[0][1]

    @property
    def nr_slaves(self):
        return len(self._registry)


title: How to find the best value for mean and STD of Normalize in torchvision.transforms

body:
I have started working with PyTorch and cannot figure it how I am supposed to find mean and std as the input parameters of normalise.
I have seen this
transforms.Normalize((0.485, 0.456, 0.406), (0.229, 0.224, 0.225)) #https://pytorch.org/vision/stable/transforms.html#

and in another example:
transformation = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5,0.5,0.5],std=[0.5,0.5,0.5])
])  #https://github.com/MicrosoftDocs/ml-basics/blob/master/challenges/05%20%20-%20Safari%20CNN%20Solution%20(PyTorch).ipynb

so, how am I supposed to know or get these values if I have a set of images?
Are these three parameters are related to R G B also?


title: Understanding pos_weight argument in BCEWithLogitsLoss

body:
I am trying to understand how the pos_weight argument is being used in BCEWithLogitsLoss in order to be able to correctly define the pos_weight Tensor. The documentation only mentions: "a weight of positive examples. Must be a vector with length equal to the number of classes.". As I was not able to get a sufficient understanding by looking at the code (the actual code is hidden behind multiple function loaders and I didn't even manage to reach a point where pos_weight is used), I have several questions regarding the pos_weight argument:

Is the weight for negative samples always 1?
If the weight for negative samples is always 1 and assuming I want every sample to contribute equally to the loss I would do something like the following. Let l = [100, 10, 5, 15] where l[0] is the amount of negative samples and l[1:] is the amount of positive samples from each label. In pseudo code this looks like:

l = [100, 10, 5, 15]
lcm = LCM(l)  # 300
weights = lcm / l  # weights = [3, 30, 60, 20]
weights = weights / l[0]  # weights = [1, 10, 20, 6.6667]
positive_weights = weights[1:]  # [10, 20, 6.66667]
criterion = nn.BCEWithLogitsLoss(pos_weight=positive_weights)

Can someone please confirm if my understanding of how pos_weight should be used is correct?


title: Define manually sorted MNIST dataset with batch size = 1 in PyTorch

body:
[] : this indicates a batch. For example, if the batch size is 5, then the batch will look something like this [1,4,7,4,2]. The length of [] indicates the batch size.
What I want to make a training set something looks like this:
[1] -> [1] -> [1] -> [1] -> [1] -> [7] -> [7] -> [7] -> [7] -> [7] -> [3] -> [3] -> [3] -> [3] -> [3] -> ... and so on
Which means that firstly five 1s (batch size = 1), secondly five 7s (batch size = 1), thirdly five 3s (batch size = 1) and so on...
Can someone please provide me an idea?
It will be very helpful if someone can explain how to implement this with codes.
Thank you! :)


title: Model not saved after training in PyTorch

body:
I encounter the following problem.
I perform an increasing cross-validation; I have 20 subjects in my dataset and try to classify images. I start with 3 subjects and perform a cross-validation with k=3; that is I train 3 different models and validate on the subject left out. And this is what I do for 4, 5, ..., 20 drivers. Hence, I have a lot of models trained.
Now I wanted to check the performance of all models on another dataset, but for some reason the accuracy is the same for all models, which must be a bug somewhere.
I already use copy.deepcopy(), so I must have an error somewhere else.
I'm open for any hints!
Here is the code for the training function:
def train_model(model, num_classes, dirname, trainloader, valloader, trainset_size, valset_size, criterion, optimizer, scheduler, patience, min_delta, num_epochs, fold):

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
since = time.time()

train_loss, train_acc, val_loss, val_acc = [], [], [], []

best_model_wts = copy.deepcopy(model.state_dict())
best_acc = 0.0
early_stopping = False
counter = 0
last_train_epoch = 0

for epoch in range(num_epochs):

    if early_stopping:
        print('\nEarly Stopping')
        break

    print('Epoch {}/{}'.format(epoch+1, num_epochs))
    print('-' * 10)

    # Each epoch has a training and validation phase
    for phase in ['train', 'val']:
        if phase == 'train':
            model.train()  # Set model to training mode
            dataloader = trainloader
            dataset_size = trainset_size
        else:
            model.eval()   # Set model to evaluate mode
            dataloader = valloader
            dataset_size = valset_size

        running_loss = 0.0
        running_corrects = 0

        # Iterate over data.
        for inputs, labels in dataloader:
            model = model.to(device)
            inputs = inputs.to(device)
            #labels = labels.long().to(device)
            labels = labels.to(device) #test_tensor.type(torch.FloatTensor)

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'train'):
                # zero the parameter gradients
                optimizer.zero_grad()
                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)
                loss = criterion(outputs, labels)

                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward()
                    optimizer.step()

            # statistics
            running_loss += loss.item() * inputs.size(0)
            running_corrects += torch.sum(preds == labels.data)

        if phase == 'train':
            scheduler.step()

        epoch_loss = running_loss / dataset_size
        epoch_acc = running_corrects.double() / dataset_size

        if phase == 'train':
            train_loss.append(epoch_loss)
            train_acc.append(epoch_acc)
        else:
            val_loss.append(epoch_loss)
            val_acc.append(epoch_acc)

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(
            phase, epoch_loss, epoch_acc))

        # early stopping
        if phase == 'val':
            if counter == patience:
                early_stopping = True
                break

            if epoch == 0:
                best_loss = epoch_loss
            else:
                if best_loss >= epoch_loss + min_delta:
                    print('Validation loss decreased ({:.4f} --> {:.4f}).  Saving model ...'.format(best_loss,epoch_loss))
                    best_model_wts = copy.deepcopy(model.state_dict())
                    torch.save(model.state_dict(), '{}/weights/model_fold_{}.pth'.format(dirname,fold))
                    last_train_epoch = epoch + 1
                    best_loss = epoch_loss
                    counter = 0
                else:
                    counter += 1

time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(
    time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))

# load best model weights
model.load_state_dict(best_model_wts)
# save best model

return model, train_acc, train_loss, val_acc, val_loss, last_train_epoch

Here is how I call the function:
    model = net
    model = model.to(device)

    # train
    [model, train_acc, train_loss, val_acc, val_loss, last_train_epoch] = train_model(model, num_classes, dirname,\
                                                                                trainloader, valloader,\
                                                                                trainset_size, valset_size,\
                                                                                criterion, optimizer, \
                                                                                exp_lr_scheduler, patience, \
                                                                                min_delta, num_epochs, fold=val_index)

    # test model
    [preds_val, labels_val, idx_false_val, pred_time_val_fold] = test(model, valloader)
    [preds_tr, labels_tr, idx_false_train, pred_time_train_fold] = test(model, trainloader)
    [preds_all, labels_all, idx_false_all, pred_time_all_fold] = test(model, allloader)
    print('Accuracy on all data: ', accuracy_score(labels_all, preds_all))

and for the sake of completeness, this is what the test() function looks like:
def test(model, dataloader):
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
#device = torch.device("cpu")

pred_labels, gt_labels, idx_false, pred_time = [], [], [], []

was_training = model.training
model.eval()

with torch.no_grad():
    for i, (inputs, labels) in enumerate(dataloader):
        inputs = inputs.to(device)
        labels = labels.to(device)

        start_pred = time.clock()
        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)
        end_pred = time.clock()

        pred_time.append(end_pred-start_pred)

        for j in range(inputs.size()[0]):
            pred_labels.append(preds[j].item())
            gt_labels.append(labels[j].item())

    for i in range(len(pred_labels)):
        if pred_labels[i] != gt_labels[i]:
            idx_false.append(i)

    model.train(mode=was_training)

return pred_labels, gt_labels, idx_false, pred_time

Edit: It looks as if it always saves the same models even though I try to make sure that only the updated weights of the best model are saved.


title: RuntimeError: tensors must be 2-D

body:
I was running a simple MLP network with customized learning algorithms. It worked fine on the training set, but I got this error when I entered additional code to check the test accuracy. How can I fix it?
Test Accuracy code
epochs = 1
for epcoh in range(epochs):
    model_bp.eval()
    model_fa.eval()
    test_loss_bp = 0
    correct_bp = 0
    test_loss_fa = 0
    correct_fa = 0
    with torch.no_grad():
        for idx_batch, (inputs, targets) in enumerate(test_loader):

            output_bp = model_bp(inputs)
            output_fa = model_fa(inputs)
            # sum up batch loss
            test_loss_bp += loss_crossentropy(output_bp, targets).item()
            test_loss_bp += loss_crossentropy(output_fa, targets).item()
            # get the index of the max log-probability
            ## predict_bp = outputs_bp.argmax(dim=1, keepdim=True)
            predict_bp = torch.max(output_bp.data,1)[1]
            correct_bp += predict_bp.eq(targets.view_as(predict_bp)).sum().item()

            predict_fa = torch.max(output_fa.data,1)[1]
            correct_fa += predict_fa.eq(targets.view_as(predict_fa)).sum().item()

    print('Test set: BP Average loss: {:.4f}, Accuracy: {}/{} ({:.4f}%)\n'.format(test_loss_bp, correct_bp, len(test_loader.dataset),
            100. * correct_bp / len(test_loader.dataset)))
    print('Test set: FA Average loss: {:.4f}, Accuracy: {}/{} ({:.4f}%)\n'.format(test_loss_fa, correct_fa, len(test_loader.dataset),
            100. * correct_fa / len(test_loader.dataset)))

Error
I'm curious about the meaning of 'RuntimeError: tensors must be 2-D'. We would appreciate it if you could tell us why it happened and where you made the mistake.
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-9-9b8b6f683e59> in <module>
     16             #targets = targets.to(device)
     17
---> 18             output_bp = model_bp(inputs)
     19             output_fa = model_fa(inputs)
     20             # sum up batch loss

~\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

c:\Users\bclab\Desktop\feedback-alignment-pytorch-master\lib\linear.py in forward(self, inputs)
    102         """
    103         # first layer
--> 104         linear1 = F.relu(self.linear[0](inputs))
    105
    106         linear2 = self.linear[1](linear1)

~\anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

c:\Users\bclab\Desktop\feedback-alignment-pytorch-master\lib\linear.py in forward(self, input)
     69     def forward(self, input):
     70         # See the autograd section for explanation of what happens here.
---> 71         return LinearFunction.apply(input, self.weight, self.bias)
     72
     73

c:\Users\bclab\Desktop\feedback-alignment-pytorch-master\lib\linear.py in forward(ctx, input, weight, bias)
     11     def forward(ctx, input, weight, bias=None):
     12         ctx.save_for_backward(input, weight, bias)
---> 13         output = input.mm(weight.t())
     14         if bias is not None:
     15             output += bias.unsqueeze(0).expand_as(output)

RuntimeError: tensors must be 2-D

This is my model. And fa_linear, linear : customize network
# load feedforward dfa model
model_fa = fa_linear.LinearFANetwork(in_features=784, num_layers=2, num_hidden_list=[1000, 10]).to(device)

# load reference linear model
model_bp = linear.LinearNetwork(in_features=784, num_layers=2, num_hidden_list=[1000, 10]).to(device)

# optimizers
optimizer_fa = torch.optim.SGD(model_fa.parameters(),
                            lr=1e-4, momentum=0.9, weight_decay=0.001, nesterov=True)
optimizer_bp = torch.optim.SGD(model_bp.parameters(),
                            lr=1e-4, momentum=0.9, weight_decay=0.001, nesterov=True)

loss_crossentropy = torch.nn.CrossEntropyLoss()

# make log file
results_path = 'bp_vs_fa_'
logger_train = open(results_path + 'train_log2.txt', 'w')

linear
from torch.autograd import Function
from torch import nn
import torch
import torch.nn.functional as F

# Inherit from Function
class LinearFunction(Function):
    # Note that both forward and backward are @staticmethods
    @staticmethod
    # bias is an optional argument
    def forward(ctx, input, weight, bias=None):
        ctx.save_for_backward(input, weight, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output

    # This function has only a single output, so it gets only one gradient
    @staticmethod
    def backward(ctx, grad_output):
        # This is a pattern that is very convenient - at the top of backward
        # unpack saved_tensors and initialize all gradients w.r.t. inputs to
        # None. Thanks to the fact               that additional trailing Nones are
        # ignored, the return statement is simple even when the function has
        # optional inputs.
        input, weight, bias = ctx.saved_variables
        grad_input = grad_weight = grad_bias = None
        # These needs_input_grad checks are optional and there only to
        # improve efficiency. If you want to make your code simpler, you can
        # skip them. Returning gradients for inputs that don't require it is
        # not an error.

        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(weight)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[2]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_bias

class Linear(nn.Module):
    def __init__(self, input_features, output_features, bias=True):
        super(Linear, self).__init__()
        self.input_features = input_features
        self.output_features = output_features

        # nn.Parameter is a special kind of Variable, that will get
        # automatically registered as Module's parameter once it's assigned
        # as an attribute. Parameters and buffers need to be registered, or
        # they won't appear in .parameters() (doesn't apply to buffers), and
        # won't be converted when e.g. .cuda() is called. You can use
        # .register_buffer() to register buffers.
        # nn.Parameters can never be volatile and, different than Variables,
        # they require gradients by default.
        self.weight = nn.Parameter(torch.Tensor(output_features, input_features))
        if bias:
            self.bias = nn.Parameter(torch.Tensor(output_features))
        else:
            # You should always register all possible parameters, but the
            # optional ones can be None if you want.
            self.register_parameter('bias', None)

        # weight initialization
        torch.nn.init.kaiming_uniform(self.weight)
        torch.nn.init.constant(self.bias, 1)

    def forward(self, input):
        # See the autograd section for explanation of what happens here.
        return LinearFunction.apply(input, self.weight, self.bias)

class LinearNetwork(nn.Module):
    def __init__(self, in_features, num_layers, num_hidden_list):
        """
        :param in_features: dimension of input features (784 for MNIST)
        :param num_layers: number of layers for feed-forward net
        :param num_hidden_list: list of integers indicating hidden nodes of each layer
        """
        super(LinearNetwork, self).__init__()
        self.in_features = in_features
        self.num_layers = num_layers
        self.num_hidden_list = num_hidden_list

        # create list of linear layers
        # first hidden layer
        self.linear = [Linear(self.in_features, self.num_hidden_list[0])]
        # append additional hidden layers to list
        for idx in range(self.num_layers - 1):
            self.linear.append(Linear(self.num_hidden_list[idx], self.num_hidden_list[idx+1]))

        # create ModuleList to make list of layers work
        self.linear = nn.ModuleList(self.linear)

    def forward(self, inputs):
        """
        forward pass, which is same for conventional feed-forward net
        :param inputs: inputs with shape [batch_size, in_features]
        :return: logit outputs from the network
        """
        # first layer
        linear1 = F.relu(self.linear[0](inputs))

        linear2 = self.linear[1](linear1)

        return linear2

fa_linear
import torch
import torch.nn.functional as F
import torch.nn as nn
from torch import autograd
from torch.autograd import Variable

class LinearFANetwork(nn.Module):
    """
    Linear feed-forward networks with feedback alignment learning
    Does NOT perform non-linear activation after each layer
    """
    def __init__(self, in_features, num_layers, num_hidden_list):
        """
        :param in_features: dimension of input features (784 for MNIST)
        :param num_layers: number of layers for feed-forward net
        :param num_hidden_list: list of integers indicating hidden nodes of each layer
        """
        super(LinearFANetwork, self).__init__()
        self.in_features = in_features
        self.num_layers = num_layers
        self.num_hidden_list = num_hidden_list

        # create list of linear layers
        # first hidden layer
        self.linear = [LinearFAModule(self.in_features, self.num_hidden_list[0])]
        # append additional hidden layers to list
        for idx in range(self.num_layers - 1):
            self.linear.append(LinearFAModule(self.num_hidden_list[idx], self.num_hidden_list[idx+1]))

        # create ModuleList to make list of layers work
        self.linear = nn.ModuleList(self.linear)

    def forward(self, inputs):
        """
        forward pass, which is same for conventional feed-forward net
        :param inputs: inputs with shape [batch_size, in_features]
        :return: logit outputs from the network
        """

        # first layer
        linear1 = self.linear[0](inputs)

        # second layer
        linear2 = self.linear[1](linear1)

        return linear2

class LinearFAFunction(autograd.Function):

    @staticmethod
    # same as reference linear function, but with additional fa tensor for backward
    def forward(context, input, weight, weight_fa, bias=None):
        context.save_for_backward(input, weight, weight_fa, bias)
        output = input.mm(weight.t())
        if bias is not None:
            output += bias.unsqueeze(0).expand_as(output)
        return output

    @staticmethod
    def backward(context, grad_output):
        input, weight, weight_fa, bias = context.saved_variables
        grad_input = grad_weight = grad_weight_fa = grad_bias = None

        if context.needs_input_grad[0]:
            # all of the logic of FA resides in this one line
            # calculate the gradient of input with fixed fa tensor, rather than the "correct" model weight
            grad_input = grad_output.mm(weight_fa)
        if context.needs_input_grad[1]:
            # grad for weight with FA'ed grad_output from downstream layer
            # it is same with original linear function
            grad_weight = grad_output.t().mm(input)
        if bias is not None and context.needs_input_grad[3]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, grad_weight_fa, grad_bias

class LinearFAModule(nn.Module):

    def __init__(self, input_features, output_features, bias=True):
        super(LinearFAModule, self).__init__()
        self.input_features = input_features
        self.output_features = output_features

        # weight and bias for forward pass
        # weight has transposed form; more efficient (so i heard) (transposed at forward pass)
        self.weight = nn.Parameter(torch.Tensor(output_features, input_features))
        if bias:
            self.bias = nn.Parameter(torch.Tensor(output_features))
        else:
            self.register_parameter('bias', None)

        # fixed random weight and bias for FA backward pass
        # does not need gradient
        self.weight_fa = nn.Parameter(Variable(torch.FloatTensor(output_features, input_features), requires_grad=False))

        # weight initialization
        torch.nn.init.kaiming_uniform(self.weight)
        torch.nn.init.kaiming_uniform(self.weight_fa)
        torch.nn.init.constant(self.bias, 1)

    def forward(self, input):
        return LinearFAFunction.apply(input, self.weight, self.weight_fa, self.bias)


title: Focal Loss + Label Smoothing

body:
I'm trying to implement focal loss with label smoothing, I used this implementation kornia and tried to plugin the label smoothing based on this implementation with Cross-Entropy Cross entropy + label smoothing but the loss yielded doesn't make sense.
Focal loss + LS (My implementation): Train loss 2.9761913128770314 accuracy 0.40519300987212814

Focal loss(Kornia implementation): Train loss 0.0602325857395604 accuracy 0.8621959099036829


title: ValueError: only one element tensors can be converted to Python scalars when using torch.Tensor on list of tensors

body:
I have a list of tensors:
object_ids = [tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.]), tensor([2., 3.])]

Intuitively, it seems like I should be able to create a new tensor from this:
torch.as_tensor(object_ids, dtype=torch.float32)

But this does NOT work. Apparently, torch.as_tensor and torch.Tensor can only turn lists of scalars into new tensors. it cannot turn a list of d-dim tensors into a d+1 dim tensor.


title: pytorch multiply 4*1 matrix and 1 size variable occur error

body:
import torch
from torch.autograd import Variable
import numpy as np

x = np.transpose(np.array([[1, 2, 3, 4]]))
a = Variable(torch.rand(1), requires_grad=True)

print(a * x) # error!

I want result like x = [[2][4][6][8]] if a = 2

is there any solution?


title: Theoretical underpinning behind Hardmax operator

body:
In the tensor flow Github repository, in the file attentionwrapper.py, hardmax operator has been defined. On the docs, it has been mentioned tf.contrib.seq2seq.hardmax

I want to know what's the theoretical underpinning behind providing this functionality for hardmax operator. Prima facie google searches for past few weeks haven't led me to concrete understanding of the concept.

If softmax is differentiable (soft), why would hardmax be ever used? If it can't be used in back propagation (due to non-differentiability required in gradient calculation), where else can it be used?
Reinforcement learning literature talks about Soft vs Hard attention. However I couldn't see concrete examples nor explanations of where the tf.contrib.seq2seq.hardmax can be actually used in some RL model.
By the looks of it, since it is mentioned in seq2seq, it should be obviously having some application in Natural Language Processing. But exactly where? There are tonnes of NLP tasks. Couldn't find any direct task SOTA algorithm which uses hardmax.


title: Fully convolutional neural net in PyTorch will minimize loss but all predictions are the same?

body:
Edit: The issue was a careless error on my part and not incorrect structure. Or rather, the structure WAS incorrect, but only because of a typo of sorts. I used ReLU activation on each layer before the output layer, but I also did ReLU separately on the input to the pooling layers. So I was doing ReLU twice each time. I'm still having issues with the model but at the very least I get varying output.

I'm putting together a fully convolutional neural network in PyTorch and I'm having a bit of trouble getting results after the model has been trained. Now, the training portion is not nearly perfect, but for now it will minimize loss significantly, to the point where my results should look like something at least. However, when I make test predictions on the training data, the result is the same for every sample.

To give a bit of background on the purpose and structure, the input data are images and the result should be a single value (regression). As mentioned, it is fully convolutional and the structure is Conv/ReLU>pooling 3 times and then the last layer is just 1x1x1 conv with no activation. The loss function is Mean Squared Error, and I am using SGD as the optimizer.

My code is a bit lengthy, a bit ugly, and I'm not sure what specific portion is causing the error. Thus, for now I will refrain from posting any here, but if someone has a hunch as to the general culprit I can post any requested portion.

I'm simply hoping that this is an issue that would give a more experienced person a quick idea of what overall structure has been set up incorrectly. To me it seems like it is most likely the network itself, because as I said it appears to train just fine, but no matter the input image, the output is always the same single floating point number. Perhaps it is the activation functions (or lack thereof on the last layer)? Or maybe it actually is the training step and I'm doing something to mess up the end result? I should note that it looks like the weight value for that very last layer is the value of every prediction.

The input is normalized (i.e. subtracted mean and divide by stddev for each image), while the targets just have subtracted mean and are in the scale -1 to 1 (the original was 0 to 1).


title: Slicing uneven columns from tensor array

body:
I have an array like so:

([[[ 0,  1,  2],
 [ 3,  4,  5]],

[[ 6,  7,  8],
[ 9, 10, 11]],

[[12, 13, 14],
[15, 16, 17]]])

If i want to slice the numbers 12 to 17 i would use:

arr[2, 0:2, 0:3]

but how would i go about slicing the array to get 12 to 16?


title: object detection inference for high resolution images is taking huge time on cpu

body:
I have trained the ML model on pascal dataset with the image size of 224 but when inferencing on new images (some are of high resolution and some are of slightly higher resolution than the pascal images), I get the error in pil2tensor()

@app.route('/analyze', methods=['POST'])
async def analyze(request):
    data = await request.form()
    img_bytes = await (data['file'].read())
    img = open_image(BytesIO(img_bytes))

    t_img= PIL.Image.open(BytesIO(img_bytes)).convert('RGB')
    t_img = pil2tensor(t_img, np.float32)
    t_img = t_img.div_(255)
    with torch.no_grad():
        # test_output = learn.model.eval()(t_img.unsqueeze_(0).cuda())
        test_output = learn.model.eval()(t_img.unsqueeze_(0))

for small image sizes (like some low-resolution images from google), ML model is correctly able to make inference within seconds but for even slightly higher resolution images, it's taking around 20-40 minutes !!!


title: How to merge Hyperopt and Sacred together?

body:
I am trying to use omniboard to keep a track of all the experiments hyperopt library is doing. The link below is from sacred library documentation and I am not able to use it to merge hyperopt and sacred together. Right now when I am defining an objective function and putting it in fmin(), I am getting a TypeError saying 'NoneType object is not callable'. Can somebody please help me figure out a way around this.

https://github.com/yuvalatzmon/SACRED_HyperOpt_v2


title: How to loop a dataframe in Pytorch?

body:
I have a problem when I want to loop my DataFrame created before so I can pass it into my classifier, but I can't neither loop my DataFrame nor pass a filename to the classifier. What can I do?

dataset=pd.DataFrame({
                      'filename':train,
                      'category':categories
                    })
images=dataset.iloc[:,0]
labels=dataset.iloc[:,-1]

from torch import nn,optim
class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1=nn.Linear(49152,256)
        self.fc2=nn.Linear(256,128)
        self.fc3=nn.Linear(128,64)
        self.fc4=nn.Linear(64,2)
        # add the dropout
        self.dropout=nn.Dropout(p=0.2)
    def forward(self,x):
        #flatten the input
        x=x.view(x.shape[0],-1)
        x=self.dropout(F.relu(self.fc1(x)))
        x=self.dropout(F.relu(self.fc2(x)))
        x=self.dropout(F.relu(self.fc3(x)))
        x=F.log_softmax(self.fc4(x),dim=1)
        return x
model=Classifier()
criterion=nn.NLLLoss()
optimizer=optim.Adam(model.parameters(),lr=0.005)
epochs=20
steps=0
for epoch in range(epochs):
    running_loss=0
    for filename,label in dataset:
        image=cv2.imread(filename)
        image=cv2.resize(image,dsize=(128,128),interpolation=cv2.INTER_CUBIC)
        output=model.forward(image)
        loss=criterion(output,label)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        running_loss+=loss.item()
    print(running_loss)


title: pytorch MNIST example not converge

body:
I'm new to pytorch and I'm writing a toy example performing the MNIST classification. Here is the full code of my example:

import matplotlib
matplotlib.use("Agg")
import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader

import torchvision.transforms as transforms
import torchvision.datasets as datasets

import matplotlib.pyplot as plt
import os
from os import system, listdir
from os.path import join, isfile, isdir, dirname

def img_transform(image):
    transform=transforms.Compose([
        # transforms.ToTensor(),
        transforms.Normalize((0.1307,), (0.3081,))])
    return transform(image)

def normalize_output(img):
    img = img - img.min()
    img = img / img.max()
    return img

def save_checkpoint(state, filename='checkpoint.pth.tar'):
    torch.save(state, filename)

class Net(nn.Module):
    """docstring for Net"""
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 3, 1)
        self.conv2 = nn.Conv2d(32, 64, 3, 1)
        self.fc1 = nn.Linear(9216, 128)
        self.fc2 = nn.Linear(128, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.conv2(x)
        x = F.max_pool2d(x, 2)
        x = torch.flatten(x, 1)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        output = F.log_softmax(x, dim=1)
        return output

os.environ['CUDA_VISIBLE_DEVICES'] = '0'
data_images, data_labels = torch.load("./PATH/MNIST/processed/training.pt")
model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-2)
epochs = 5
batch_size = 30
num_batch = int(data_images.shape[0] / batch_size)
for epoch in range(epochs):
    for batch_idx in range(num_batch):
        data = data_images[ batch_idx*batch_size : (batch_idx+1)*batch_size ].float()
        label = data_labels[ batch_idx*batch_size : (batch_idx+1)*batch_size ]
        data = img_transform(data)
        data = data.unsqueeze_(1)
        pred_score = model(data)
        loss = criterion(pred_score, label)
        loss.backward()
        optimizer.step()
        if batch_idx % 200 == 0:
            print('epoch', epoch, batch_idx, '/', num_batch, 'loss', loss.item())
            _, pred = pred_score.topk(1)
            pred = pred.t().squeeze()
            correct = pred.eq(label)
            num_correct = correct.sum(0).item()
            print('acc=', num_correct/batch_size)

dict_to_save = {
    'epoch': epochs,
    'state_dict': model.state_dict(),
    'optimizer' : optimizer.state_dict(),
    }
ckpt_file = 'a.pth.tar'
save_checkpoint(dict_to_save, ckpt_file)
print('save to ckpt_file', ckpt_file)
exit()

The code is executable with MNIST dataset saved in the path ./PATH/MNIST/processed/training.pt

However, the training process does not converge, with the training accuracy always lower than 0.2. What's wrong with my implementation? I have tried different learning rates and batch size. It doesn't work.

Is there any other problem in my code?

Thank you all for helping me !!!

Here are some of the training logs

epoch 0 0 / 2000 loss 27.2023868560791
acc= 0.1
epoch 0 200 / 2000 loss 2.3346288204193115
acc= 0.13333333333333333
epoch 0 400 / 2000 loss 2.691042900085449
acc= 0.13333333333333333
epoch 0 600 / 2000 loss 2.6452369689941406
acc= 0.06666666666666667
epoch 0 800 / 2000 loss 2.7910964488983154
acc= 0.13333333333333333
epoch 0 1000 / 2000 loss 2.966330051422119
acc= 0.1
epoch 0 1200 / 2000 loss 3.111387014389038
acc= 0.06666666666666667
epoch 0 1400 / 2000 loss 3.1988155841827393
acc= 0.03333333333333333


title: Is there a way to use torch.autograd.gradient in parallel in Pytorch?

body:
I am trying to train some network where the loss is not only a function of the output but also the derivative of the output w.r.t. the input. The problem is that while computing the batch output can be done in parallel with the modules with Pytorh, I can't find a way to do the derivative in parallel. Here's the best I can do in serial:

import torch
x=torch.rand(300,1)
dydx=torch.zeros_like(x)
fc=torch.nn.Linear(1,1)
x.requires_grad=True
for ii in range(x.size(0)):
    xi=x[ii,0:]
    yi=torch.tanh(fc(xi))
    dydx[ii]=torch.autograd.grad(yi,xi,create_graph=True)[0]
dydxsum=(dydx**2).sum()
dydxsum.backward()

In the code above, x is split to save memory and time. However, when the size of x becomes large, parallelization (in CUDA) is still necessary. If it has to be implemented by tinkering Pytorch, a hint to where to start will be appreciated.


title: Getting SIGKILL when running study optimization with Optuna on PyCharm

body:
I am trying to run a study, using the optimize function with the default sampler and Median pruner.
every run crashes, sometimes after 1 succefull trial sometimes without completing any.
The crash message is: Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
Expected behavior
running a study
Environment

Optuna version:
2.0.0
Python version: 3.8
OS:QubeOS with debian 10 VM
(Optional) Other libraries and their versions:
Pytorch '1.5.0+cpu'

Error messages, stack traces, or logs
Process finished with exit code 137 (interrupted by signal 9: SIGKILL)
what cause such an error?


title: Given a set t of tuples containing elements from the set S, what is the most efficient way to build another set whose members are not contained in t?

body:
For example, suppose I had an (n,2) dimensional tensor t whose elements are all from the set S containing random integers. I want to build another tensor d with size (m,2) where individual elements in each tuple are from S, but the whole tuples do not occur in t.
E.g.
S = [0,1,2,3,7]
t = [[0,1],
     [7,3],
     [3,1]]

d = some_algorithm(S,t)

/*
d =[[2,1],
    [3,2],
    [7,4]]
*/

What is the most efficient way to do this in python? Preferably with pytorch or numpy, but I can work around general solutions.
In my naive attempt, I just use
d = np.random.choice(S,(m,2))
non_dupes = [i not in t for i in d]
d = d[non_dupes]

But both t and S are incredibly large, and this takes an enormous amount of time (not to mention, rarely results in a (m,2) array). I feel like there has to be some fancy tensor thing I can do to achieve this, or maybe making a large hash map of the values in t so checking for membership in t is O(1), but this produces the same issue just with memory. Is there a more efficient way?
An approximate solution is also okay.


title: Getting variance values for random samples generated from a standard normal distribution using numpy

body:
I have a function that gives me probability distributions for each class, in terms of a matrix corresponding to mean values and another matrix corresponding to variance values. For example, if I had four classes then I would have the following outputs:
y_means = [1,2,3,4]
y_variance = [0.01,0.02,0.03,0.04]

I need to do the following calculation to the mean values to continue with the rest of my program:
y_means = np.array(y_means)
y_means = np.reshape(y_means,(y_means.size,1))
A = np.random.randn(10,y_means.size)
y_means = np.matmul(A,y_means)

Here, I have used the numpy.random.randn function to generate random samples from a standard normal distribution, and then multiply this with the matrix with the mean value to obtain a new output matrix. The dimension of the output matrix would then be of the size (10 x 1).
I need to do a similar calculation such that my output_variances will also be a (10 x 1) matrix. But it is not meaningful to multiply the variances in the same way with random samples from a standard normal distribution, because this would result in negative values as well. This is undesirable because my ultimate aim would be to create a normal distribution with these mean values and their corresponding variance values using:
torch.distributions.normal.Normal(loc=y_means, scale=y_variance)

So my question is if there is any method by which I get a variance value for each random sample generated by numpy.random.randn? Because then the multplication of such a matrix would make more sense with output_variance.
Or if there is any other strategy for this that I might be unaware of, please let me know.


title: Understanding image size changes in DCGAN

body:
I have been studying and trying to implement Generative Adversarial Networks using PyTorch. More precisely I tried to replicate the DCGAN PyTorch Tutorial tutorial using some custom dataset.
My code worked well and I was able to get some nice results, but when I look at the architecture of both the generator and the discriminator I struggle to understand how the image sizes change as it goes through the different convolutional layers. To make my question a bit more clear, I do not understand the process to go from the original noise vector of size 100 to the 64X64X3 output image in the below architecture:
class Generator(nn.Module):
    def __init__(self, ngpu):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)

I checked some online resources as well as the PyTorch documentation, and I found some different formulas to calculate the output size of convolutional layers. However none of them were enough to explain the big transformations that occur in this particular architecture.
I hope someone here can help me understand this!


title: Pytorch tensor representing a 3D grid with color values

body:
Given a list of density values (scalars) that represent the density of an X,Y,Z coordinate on a 3D grid, how would I create a single tensor that can store this information?
i.e. a tensor that has dimensions of 1x20x20x20 for example would represent a 20x20x20 grid such that:
print(tensor[:,x1,y1,z1])
0.6

print(tensor[:,x2,y2,z2])
0.4


title: How to find intersection of two sets of 2D tensors (points on a 2D plane) in Pytorch

body:
I have two lists of Pytorch 2D tensors, which are points on a plane:
ListA = tensor([ [1.0,2.0], [1.0,3.0], [4.0,8.0] ], device='cuda:0')
ListB = tensor([ [5.0,7.0], [1.0,2.0], [4.0,8.0] ], device='cuda:0')

How to compute ?
Desired output = tensor([ [1.0,2.0] ,  [4.0,8.0] ], device='cuda:0')

I would like to find the Intersection between two lists ListA and ListB.
Note : Computation should be carried out only on CUDA.


title: Using Pytorch in AWS lambda Error- "Unable to import module 'lambda_function': No module named 'torch._C'",

body:
I am trying to use pytorch in AWS lambda but I keep getting this error.
I am using arn:aws:lambda:us-east-1:934676248949:layer:pytorchv1-py36:2 as the layer for pytorch
and this is my handler function
import unzip_requirements
import torch

def lambda_handler(event, context):

    return "Hello"

As in the function I am just only importing torch and it is giving me this error.
{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'torch._C'",
  "errorType": "Runtime.ImportModuleError"
}


title: Translating 3D CNN from Keras to Pytorch

body:
I'm trying to translate the below 3D CNN architecture from keras to pytorch. The 3D images all have the following dimensions: 193 x 229 x 193.
Network architecture in Keras:
def test_model(size):
    model = Sequential()
    model.add(Conv3D(filters=8,
                     kernel_size=(3, 3, 3),
                     activation='relu',
                     input_shape=size,
                     name="conv_1_1"))
    model.add(Conv3D(filters=8,
                     kernel_size=(3, 3, 3),
                     activation='relu',
                     name="conv_1_2"))
    model.add(MaxPooling3D(pool_size=(2, 2, 2),
                           strides=(2, 2, 2)))
    model.add(Conv3D(filters=16,
                     kernel_size=(3, 3, 3),
                     activation='relu',
                     input_shape=size,
                     name="conv_2_1"))
    model.add(Conv3D(filters=16,
                     kernel_size=(3, 3, 3),
                     activation='relu',
                     name="conv_2_2"))
    model.add(MaxPooling3D(pool_size=(2, 2, 2),
                           strides=(2, 2, 2)))
    model.add(Flatten())
    model.add(Dense(units=1,
                    name="d_2"))
    return model

My attempt at translating this to Pytorch:
class Model(torch.nn.Module):

    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv3d(input_channel=1, output_channel=8, kernel_size=3)
        self.conv2 = nn.Conv3d(input_channel=8, output_channel=8, kernel_size=3)
        self.conv3 = nn.Conv3d(input_channel=8, output_channel=16, kernel_size=3)
        self.conv4 = nn.Conv3d(input_channel=16, output_channel=16, kernel_size=3)
        self.fc1 = nn.Linear( ???? , 1)

     def forward (self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(F.max_pool3d(self.conv2(x), kernel_size=2, stride=2))
        x = F.relu(self.conv3(x))
        x = F.relu(F.max_pool3d(self.conv4(x), kernel_size=2, stride=2))
        x = self.fc1

        return x

net = Model()

Please could you let me know where I've made mistakes and also clarify how to determine the input for the nn.Linear( ???? , 2) layer? Thank you for your help!


title: PyTorch model + HTTP API = very slow execution

body:
I have:

ML model (PyTorch) that vectorizes data and makes a prediction in ~3.5ms (median ≈ mean)
HTTP API (FastAPI + uvicorn) that serves simple requests in ~2ms

But when I combine them, the median response time becomes almost 200ms.
What can be the reason for such degradation?

Note that:

I also tried aiohttp alone, aiohttp + gunicorn and Flask development server for serving - same result
I tried to send 2, 20 and 100 requests per second - same result
I do realize that parallel requests can lead to decreased latency, but not 30 times!
CPU load is only ~7%

Here's how I measured model performance (I measured the median time separately, it's nearly the same as the mean time):
def predict_all(predictor, data):
    for i in range(len(data)):
        predictor(data[i])

data = load_random_data()
predictor = load_predictor()
%timeit predict_all(predictor, data)
# manually divide total time by number of records in data

Here's FastAPI version:
from fastapi import FastAPI
from starlette.requests import Request
from my_code import load_predictor

app = FastAPI()

app.predictor = load_predictor()

@app.post("/")
async def root(request: Request):
    predictor = request.app.predictor
    data = await request.json()
    return predictor(data)

HTTP performance test:
wrk2 -t2 -c50 -d30s -R100 --latency -s post.lua http://localhost:8000/

EDIT.
Here's a slightly modified version which I tried with and without async:
@app.post("/")
# async def root(request: Request, user_dict: dict):
def root(request: Request, user_dict: dict):
    predictor = request.app.predictor
    start_time = time.time()
    y = predictor(user_dict)
    finish_time = time.time()
    logging.info(f"user {user_dict['user_id']}: "
                 "prediction made in {:.2f}ms".format((finish_time - start_time) * 1000))
    return y

So I just added logging of prediction time.
Log for async version:
2021-02-03 11:14:31,822: user 12345678-1234-1234-1234-123456789123: prediction made in 2.87ms
INFO:     127.0.0.1:49284 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,329: user 12345678-1234-1234-1234-123456789123: prediction made in 3.93ms
INFO:     127.0.0.1:49286 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,345: user 12345678-1234-1234-1234-123456789123: prediction made in 15.06ms
INFO:     127.0.0.1:49287 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,351: user 12345678-1234-1234-1234-123456789123: prediction made in 4.78ms
INFO:     127.0.0.1:49288 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,358: user 12345678-1234-1234-1234-123456789123: prediction made in 6.85ms
INFO:     127.0.0.1:49289 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,363: user 12345678-1234-1234-1234-123456789123: prediction made in 3.71ms
INFO:     127.0.0.1:49290 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,369: user 12345678-1234-1234-1234-123456789123: prediction made in 5.49ms
INFO:     127.0.0.1:49291 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:14:56,374: user 12345678-1234-1234-1234-123456789123: prediction made in 5.00ms

So prediction is fast, less than 10ms on average, but whole request takes 200ms.
Log for sync version:
2021-02-03 11:17:58,332: user 12345678-1234-1234-1234-123456789123: prediction made in 65.49ms
2021-02-03 11:17:58,334: user 12345678-1234-1234-1234-123456789123: prediction made in 23.05ms
INFO:     127.0.0.1:49481 - "POST / HTTP/1.1" 200 OK
INFO:     127.0.0.1:49482 - "POST / HTTP/1.1" 200 OK
2021-02-03 11:17:58,338: user 12345678-1234-1234-1234-123456789123: prediction made in 72.39ms
2021-02-03 11:17:58,341: user 12345678-1234-1234-1234-123456789123: prediction made in 78.66ms
2021-02-03 11:17:58,341: user 12345678-1234-1234-1234-123456789123: prediction made in 85.74ms

Now prediction takes long! For whatever reason, exactly the same call, but made in synchronous context, started to take ~30 times longer. But the whole request takes approximately the same time - 160-200ms.


title: Questions about batch_size in pytorch

body:
I tried out PyTorch and wanted to write a program for PHM 2010. But, the result of program is not good. There is a problem  that the prediction becomes worse as the batch_size increasing.
The dataset is PHM2010.There are 315 sets of samples, and the input of each set of samples has seven signals and the output is the wear value.There are six tool wear values in the data set, which are C1,C2,C3,C4,C5 and C6.Among them, 1,4 and 6 have real wear values as supervised learning.This code takes the C1 dataset.
The neural network uses the convolutional neural network.
When the batch_size is 1, the predicted value is close to that in the Keras framework. However, the predicted value will drift away from the real value with the batch_size increasing.
Code in github: https://github.com/charmerphil/CNN-keras-and-pytorch.git
As shown in the figure below.
Figure 1, batch_size = 1  pytorch
Figure 2, batch_size = 2  pytorch
Figure 3, batch_size = 3  pytorch
Figure 4, batch_size = 5  pytorch
Figure 5, batch_size = 10 pytorch
Figure 6, batch_size = 10 keras


title: Get some layers in a pytorch model that is not defined by nn.Sequential

body:
I have a network defined below.
class model_dnn_2(nn.Module):
    def __init__(self):
        super(model_dnn_2, self).__init__()
        self.flatten = Flatten()
        self.fc1 = nn.Linear(784, 200)
        self.fc2 = nn.Linear(200, 100)
        self.fc3 = nn.Linear(100, 100)
        self.fc4 = nn.Linear(100, 10)

    def forward(self, x):
        x = self.flatten(x)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        x = F.relu(x)
        x = self.fc4(x)

I would like to take the last two layers along with the relu functions. Using children method I get the following
>>> new_model = nn.Sequential(*list(model.children())[-2:])
>>> new_model
Sequential(
  (0): Linear(in_features=100, out_features=100, bias=True)
  (1): Linear(in_features=100, out_features=10, bias=True)
)

But I would like to have the Relu function present in between the layers-just like the original model, i.e the new model should be like:
>>> new_model
Sequential(
  (0): Linear(in_features=100, out_features=100, bias=True)
  (1): Relu()
  (2): Linear(in_features=100, out_features=10, bias=True)
)

I think the children method of the model is using the class initialization to create the model and thus the problem arises.
How can I obtain the model?


title: furrr with rTorch in multisession

body:
I want to use rTorch in a furrr "loop". A minimal example seems to be:
library(rTorch)
torch_it <- function(i) {
  #.libPaths("/User/homes/mreichstein/.R_libs_macadamia4.0/")
  #require(rTorch)
  cat("torch is: "); print(torch)
  out  <- torch$tensor(1)
  out
}

library(furrr)
plan(multisession, workers=8) # this works with sequential or multicore
result <- future_map(1:5, torch_it)

I get as output:
torch is: <environment: 0x556b57f07990>
attr(,"class")
[1] "python.builtin.module" "python.builtin.object"
Error in torch$tensor(1) : attempt to apply non-function

When I use multicore or sequential I get the expected output:
torch is: Module(torch)
torch is: Module(torch)
torch is: Module(torch)
torch is: Module(torch)
torch is: Module(torch)

... and result is the expected list of tensors.
Uncommenting the first lines in the function, assuming the new sessions "need this" did not help.
Update: I added torch <- import("torch") in the torch_it function. Then it runs without error, but the result I get is a list of empty pointers (?):
> result
[[1]]
<pointer: 0x0>
[[2]]
<pointer: 0x0>
[[3]]
<pointer: 0x0>
[[4]]
<pointer: 0x0>
[[5]]
<pointer: 0x0>

So, how do I properly link rTorch to each of the multisessions?
Thanks in advance!
Info:
> R.version
               _
platform       x86_64-redhat-linux-gnu
arch           x86_64
os             linux-gnu
system         x86_64, linux-gnu
status
major          4
minor          0.3
year           2020
month          10
day            10
svn rev        79318
language       R
version.string R version 4.0.3 (2020-10-10)
nickname       Bunny-Wunnies Freak Out


title: Which format is preferable for tflite model NCHW or NHWC?

body:
For gpus nchw mode is preferrable due to speed benefits, but what is the preferable mode in terms of mobile perfomance for tflite model?
Now converting from pytorch to tflite yields a working nchw model but is this mode optimal?


I'm a newbie in PyTorch and I got the following error from my cnn layer: "RuntimeError: expected scalar type Double but found Float". I converted each element into .astype(np.double) but the error message remains. Then after converting Tensor tried to use .double() and again the error message remains.
Here is my code for a better understanding:
import torch.nn as nn
class CNN(nn.Module):

    # Contructor
    def __init__(self, shape):
        super(CNN, self).__init__()
        self.cnn1 = nn.Conv1d(in_channels=shape, out_channels=32, kernel_size=3)
        self.act1 = torch.nn.ReLU()
    # Prediction
    def forward(self, x):
        x = self.cnn1(x)
        x = self.act1(x)
    return x

    X_train_reshaped = np.zeros([X_train.shape[0],int(X_train.shape[1]/depth),depth])

    for i in range(X_train.shape[0]):
        for j in range(X_train.shape[1]):
            X_train_reshaped[i][int(j/3)][j%3] = X_train[i][j].astype(np.double)

    X_train = torch.tensor(X_train_reshaped)
    y_train = torch.tensor(y_train)

    # Dataset w/o any tranformations
    train_dataset_normal = CustomTensorDataset(tensors=(X_train, y_train), transform=None)
    train_loader = torch.utils.data.DataLoader(train_dataset_normal, shuffle=True, batch_size=16)

    model = CNN(X_train.shape[1]).to(device)

    # Loss and optimizer
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(model.parameters())

    # Train the model
    #how to implement batch_size??
    for epoch in range(epochno):
        #for i, (dataX, labels) in enumerate(X_train_reshaped,y_train):
        for i, (dataX, labels) in enumerate(train_loader):
            dataX = dataX.to(device)
            labels = labels.to(device)

            # Forward pass
            outputs = model(dataX)
            loss = criterion(outputs, labels)

            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if (i+1) % 100 == 0:
                print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                       .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

And following is the error I received:
RuntimeError                              Traceback (most recent call last)
<ipython-input-39-d99b62b3a231> in <module>
     14
     15         # Forward pass
---> 16         outputs = model(dataX.double())
     17         loss = criterion(outputs, labels)
     18

~\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

<ipython-input-27-7510ac2f1f42> in forward(self, x)
     22     # Prediction
     23     def forward(self, x):
---> 24         x = self.cnn1(x)
     25         x = self.act1(x)

~\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~\torch\nn\modules\conv.py in forward(self, input)
    261
    262     def forward(self, input: Tensor) -> Tensor:
--> 263         return self._conv_forward(input, self.weight, self.bias)
    264
    265

~\torch\nn\modules\conv.py in _conv_forward(self, input, weight, bias)
    257                             weight, bias, self.stride,
    258                             _single(0), self.dilation, self.groups)
--> 259         return F.conv1d(input, weight, bias, self.stride,
    260                         self.padding, self.dilation, self.groups)
    261

RuntimeError: expected scalar type Double but found Float


title: How to load a learning rate scheduler state dict?

body:
I have a model and a learning rate scheduler. I'm saving the model and optimizer using the state dict method that is shown here.
import torch
import torch.nn as nn
import torch.optim as optim

class net_x(nn.Module):
        def __init__(self):
            super(net_x, self).__init__()
            self.fc1=nn.Linear(2, 20)
            self.fc2=nn.Linear(20, 20)
            self.out=nn.Linear(20, 4)

        def forward(self, x):
            x=self.fc1(x)
            x=self.fc2(x)
            x=self.out(x)
            return x

nx = net_x()

r = torch.tensor([1.0,2.0])
optimizer = optim.Adam(nx.parameters(), lr = 0.1)
scheduler = torch.optim.lr_scheduler.CyclicLR(optimizer, base_lr=1e-2, max_lr=0.1, step_size_up=1, mode="triangular2", cycle_momentum=False)

path = 'opt.pt'
for epoch in range(10):
    optimizer.zero_grad()
    net_predictions = nx(r)
    loss = torch.sum(torch.randint(0,10,(4,)) - net_predictions)
    loss.backward()
    optimizer.step()
    scheduler.step()
    print('loss:' , loss)
    torch.save({    'epoch': epoch,
                    'net_x_state_dict': nx.state_dict(),
                    'optimizer_state_dict': optimizer.state_dict(),
                    'scheduler': scheduler,
                    }, path)

PATH = control_path
checkpoint = torch.load(path)
nx.load_state_dict(checkpoint['net_x_state_dict'])
optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
scheduler.load_state_dict(checkpoint['scheduler'])

The code runs just fine without the part that I'm loading the scheduler state dict, so I'm not sure what I'm doing wrong. I'm trying to load the state dict as mentioned here, but I'm getting this error:
TypeError                                 Traceback (most recent call last)
<ipython-input-7-e3217d6dd870> in <module>
     42 nx.load_state_dict(checkpoint['net_x_state_dict'])
     43 optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
---> 44 scheduler.load_state_dict(checkpoint['scheduler'])

~/anaconda3/lib/python3.7/site-packages/torch/optim/lr_scheduler.py in load_state_dict(self, state_dict)
     92                 from a call to :meth:`state_dict`.
     93         """
---> 94         self.__dict__.update(state_dict)
     95
     96     def get_last_lr(self):

TypeError: 'CyclicLR' object is not iterable


title: How to set and get confidence threshold from custom YOLOv5 model?

body:
I am trying to perform inference on my custom YOLOv5 model. The official documentation uses the default detect.py script for inference.
Example: python detect.py --source data/images --weights yolov5s.pt --conf 0.25
I have written my own python script but I can neither set the confidence threshold during initialisation nor retrieve it from the predictions of the model. I am only able to get the labels and bounding box coordinates. Here is my code:
import torch

model = torch.hub.load('ultralytics/yolov5', 'custom', path_or_model='best.pt')
results = model("my_image.png")
labels, cord_thres = results.xyxyn[0][:, -1].numpy(), results.xyxyn[0][:, :-1].numpy()


title: What does x[x!=x] mean?

body:
I don't understand this line:
lprobs[lprobs != lprobs] = torch.tensor(-math.inf).to(lprobs)

There is no comment, so is it some well-known Python (or PyTorch?) idiom? Could someone explain what it means, or show a different way that makes the intent clearer?
lprobs is a pytorch Tensor, and it could contain any size float type (I doubt this code is intended to support int or complex types). As far as I know, the Tensor classes don't override the __ne__ function.


title: What can make loss stop going down?

body:
I'm training my network on a very small dataset (size = 2) over and over again. It looks fine on the beginning but after Epoch 34 the loss stop going down.

I though it could be caused by the Dying Relu problem, so I replaced all my relu activations with Softplus. However, the problem is still the same.

I also tried to add some Linear hidden layers combined with Softplus with no results.

If i try with a dataset with only one example, then loss goes to 0 as expected.

What can cause this behavior and how to avoid it?

Epoch 0
Expected: [243.0, 0.0]; Predicted: [-2367.9, 3.8]; Loss: 6059457.6000
Expected: [178.0, 32.0]; Predicted: [-1731.4, 10.9]; Loss: 3241238.0000

Epoch 1
Expected: [243.0, 0.0]; Predicted: [-1237.8, 8.1]; Loss: 1949257.6000
Expected: [178.0, 32.0]; Predicted: [-883.9, 14.1]; Loss: 1002567.6000

Epoch 2
Expected: [243.0, 0.0]; Predicted: [-602.2, 10.6]; Loss: 635017.6000
Expected: [178.0, 32.0]; Predicted: [-407.1, 15.9]; Loss: 304548.4500

...

Epoch 12
Expected: [243.0, 0.0]; Predicted: [212.6, 13.7]; Loss: 991.0653
Expected: [178.0, 32.0]; Predicted: [203.9, 18.3]; Loss: 764.2527

Epoch 13
Expected: [243.0, 0.0]; Predicted: [213.7, 13.7]; Loss: 930.9330
Expected: [178.0, 32.0]; Predicted: [204.8, 18.3]; Loss: 803.9944

...

Epoch 32
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9812
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9783

Epoch 33
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9806
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9789

Epoch 34
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Epoch 35
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Epoch 36
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Epoch 37
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Epoch 38
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Epoch 39
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Epoch 40
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Epoch 41
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Epoch 42
Expected: [243.0, 0.0]; Predicted: [215.1, 13.7]; Loss: 856.9800
Expected: [178.0, 32.0]; Predicted: [205.9, 18.3]; Loss: 856.9795

Here's the code:

class NN(Module):
    def __init__(self, vocab_size, *_, hidden_size=200, text_max_len=200, rnn_num_layers=1, dropout=0.2):
        super(NN, self).__init__()

        self.rnn_layer_range = list(range(rnn_num_layers))
        self.hidden_size = hidden_size
        self.text_max_len = text_max_len
        self.softplus = Softplus()

        # text:
        self.text_embed = Embedding(vocab_size, hidden_size, padding_idx=0)
        self.text_conv1 = Conv2d( 1,  6, 5)
        self.text_conv2 = Conv2d( 6, 12, 5)
        self.text_conv3 = Conv2d(12, 24, 5)

        self.text_lin1  = Linear(4536, hidden_size)

        # image:
        self.img_conv1 = Conv2d( 3,  6, 5)
        self.img_conv2 = Conv2d( 6, 16, 5)
        self.img_conv3 = Conv2d(16, 20, 5)

        self.img_lin1 = Linear(2420, hidden_size)

        # union:
        self.u_size = 3*hidden_size
        self.u_linear_augment = Linear(9, hidden_size)
        self.u_gru = GRU(input_size=self.u_size, hidden_size=self.u_size, dropout=dropout)
        self.u_linear_reduce1 = Linear(self.u_size, self.u_size // 2)
        self.u_linear_reduce2 = Linear(self.u_size // 2, 2)

    def initHidden(self):
        return Variable(zeros(1, 1, self.hidden_size)), Variable(zeros(1, 1, self.u_size))

    def forward(self, text, img, data, *_, text_hidden=None, u_hidden=None):
        text_hidden, u_hidden = self.initHidden()

        # encode text
        max_len = self.text_max_len
        if len(text) < max_len:
            text = cat((text, Variable(LongTensor(max_len - len(text)).zero_())))
        text = self.text_embed(text)
        text = text.view(1, 1, max_len, self.hidden_size)
        text = max_pool2d(self.softplus(self.text_conv1(text)), 2)
        text = max_pool2d(self.softplus(self.text_conv2(text)), 2)
        text = max_pool2d(self.softplus(self.text_conv3(text)), 2)

        text = text.view(1, -1)
        text = self.softplus(self.text_lin1(text))
        text = text.view(-1, 1)

        # encode image
        img = max_pool2d(self.softplus(self.img_conv1(img)), 2)
        img = max_pool2d(self.softplus(self.img_conv2(img)), 2)
        img = max_pool2d(self.softplus(self.img_conv3(img)), 2)

        img = img.view(1, -1)
        img = self.softplus(self.img_lin1(img))
        img = img.view(-1, 1)

        # join
        data = self.softplus(self.u_linear_augment(data))
        vector = text.view(1, 1, -1)
        vector = cat((data, text, img)).view(1, 1, -1)
        for _ in self.rnn_layer_range:
            vector, u_hidden = self.u_gru(vector, u_hidden)
        vector = self.softplus(self.u_linear_reduce1(vector))
        vector = self.u_linear_reduce2(vector)
        return vector.view(-1, 1)

and

def train(neuron_network, optimizer, criterion, text, img, data, target, *_, loop_size=5):
    optimizer.zero_grad()

    loss = 0
    for _ in range(loop_size):
        output = neuron_network(text, img, data)
        loss += criterion(output, target)

    loss.backward()
    optimizer.step()

    return loss.data[0]/loop_size

And here is how i'm training it:

neural = NN(vocab_size=len(letter_dict)+1, dropout=0, rnn_num_layers=1, hidden_size=100)
optimizer = optim.SGD(neural.parameters(), lr=0.01)
criterion = nn.MSELoss()

for epoch in range(500):
    for item in dataset:
        loss = train(neural, optimizer, criterion, item["text"], item["img"], item["data"], item["target"])


title: where is the definition of _cudnn_convolution_full_forward in pytorch

body:
as the title mentioned, I want to find the definition of _cudnn_convolution_full_forward, but I search through all the project
in pytorch and failed. And I cannot find and doc about this function.

any one can help me?


title: How do I install and run pytorch in MSVS2017 (to avoid "module not found" error on "import torch" statement)?

body: I'm trying to use pytorch in MSVS2017.  I started a pytorch project, have anaconda environment set using python3.6, but when I run the debugger, I get a "module not found" error on the first import statement "import torch".  I've tried various methods for installing pytorch in a way that allows MSVS2017 to use it, including command line and Anaconda command line installations (using tips from other closely related StackOverflow questions), but I cannot clear the error.  This is a native MSVS2017 project type that came with their AI Tools module.  What am I doing wrong?


title: pytorch passing architecture type with argprse

body:
Using Pytorch. When passing architecture type by using the following code:

parser.add_argument('-arch', action='store',
                    dest='arch',
                    default= str('vgg16'))

When using the name of the architecture with the following code:

model = models.__dict__['{!r}'.format(results.arch)](pretrained=True)

I get the following error:

  model = models.dict'{!r}'.format(results.arch)
  KeyError: "'vgg16'"

What am I doing wrong?


title: vgg pytorch is probability distribution supposed to add up to 1?

body:
I've trained a vgg16 model to predict 102 classes of flowers.
It works however now that I'm trying to understand one of it's predictions I feel it's not acting normally.

model layout

# Imports here
import os
import numpy as np
import torch

import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import json
from pprint import pprint
from scipy import misc

%matplotlib inline

data_dir = 'flower_data'
train_dir = data_dir + '/train'
test_dir = data_dir + '/valid'

json_data=open('cat_to_name.json').read()
main_classes = json.loads(json_data)
main_classes = {int(k):v for k,v in classes.items()}

train_transform_2 = transforms.Compose([transforms.RandomResizedCrop(224),
                                    transforms.RandomRotation(30),
                                    transforms.RandomHorizontalFlip(),
                                    transforms.ToTensor()])

test_transform_2= transforms.Compose([transforms.RandomResizedCrop(224),
                                    transforms.ToTensor()])

# TODO: Load the datasets with ImageFolder
train_data = datasets.ImageFolder(train_dir, transform=train_transform_2)
test_data = datasets.ImageFolder(test_dir, transform=test_transform_2)

# define dataloader parameters
batch_size = 20
num_workers=0

# TODO: Using the image datasets and the trainforms, define the dataloaders
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size,
                                        num_workers=num_workers, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size,
                                        num_workers=num_workers, shuffle=True)

vgg16 = models.vgg16(pretrained=True)

# Freeze training for all "features" layers
for param in vgg16.features.parameters():
    param.requires_grad = False

import torch.nn as nn

n_inputs = vgg16.classifier[6].in_features

# add last linear layer (n_inputs -> 102 flower classes)
# new layers automatically have requires_grad = True
last_layer = nn.Linear(n_inputs, len(classes))

vgg16.classifier[6] = last_layer

import torch.optim as optim

# specify loss function (categorical cross-entropy)
criterion = nn.CrossEntropyLoss()

# specify optimizer (stochastic gradient descent) and learning rate = 0.001
optimizer = optim.SGD(vgg16.classifier.parameters(), lr=0.001)

pre_trained_model=torch.load("model.pt")
new=list(pre_trained_model.items())

my_model_kvpair=vgg16.state_dict()
count=0

for key,value in my_model_kvpair.items():
    layer_name, weights = new[count]
    my_model_kvpair[key] = weights
    count+=1

# number of epochs to train the model
n_epochs = 6

# initialize tracker for minimum validation loss
valid_loss_min = np.Inf # set initial "min" to infinity

for epoch in range(1, n_epochs+1):

    # keep track of training and validation loss
    train_loss = 0.0
    valid_loss = 0.0

    ###################
    # train the model #
    ###################
    # model by default is set to train
    vgg16.train()

    for batch_i, (data, target) in enumerate(train_loader):
        # clear the gradients of all optimized variables
        optimizer.zero_grad()
        # forward pass: compute predicted outputs by passing inputs to the model
        output = vgg16(data)
        # calculate the batch loss
        loss = criterion(output, target)
        # backward pass: compute gradient of the loss with respect to model parameters
        loss.backward()
        # perform a single optimization step (parameter update)
        optimizer.step()
        # update training loss
        train_loss += loss.item()

        if batch_i % 20 == 19:    # print training loss every specified number of mini-batches
            print('Epoch %d, Batch %d loss: %.16f' %
                (epoch, batch_i + 1, train_loss / 20))
            train_loss = 0.0

    ######################
    # validate the model #
    ######################
    vgg16.eval() # prep model for evaluation
    for data, target in test_loader:
        # forward pass: compute predicted outputs by passing inputs to the model
        output = vgg16(data)
        # calculate the loss
        loss = criterion(output, target)
        # update running validation loss
        valid_loss += loss.item()

    # print training/validation statistics
    # calculate average loss over an epoch
    train_loss = train_loss/len(train_loader.dataset)
    valid_loss = valid_loss/len(test_loader.dataset)

    print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
        epoch+1,
        train_loss,
        valid_loss
        ))

    # save model if validation loss has decreased
    if valid_loss <= valid_loss_min:
        print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(
        valid_loss_min,
        valid_loss))
        torch.save(vgg16.state_dict(), 'model.pt')
        valid_loss_min = valid_loss

testing on a single image

tensor = torch.from_numpy(test_image)
reshaped = tensor.permute(2, 0, 1).unsqueeze(0)
floatified = reshaped.to(torch.float32) / 255
vgg16(floatified)
>>>
tensor([[ 2.5686, -1.1964, -0.0872, -1.7010, -1.6669, -1.0638,  0.4515,  0.1124,
          0.0166,  0.3156,  1.1699,  1.5374,  1.8720,  2.5184,  2.9046, -0.8241,
         -1.1949, -0.5700,  0.8692, -1.0485,  0.0390, -1.3783, -3.4632, -0.0143,
          1.0986,  0.2667, -1.1127, -0.8515,  0.7759, -0.7528,  1.6366, -0.1170,
         -0.4983, -2.6970,  0.7545,  0.0188,  0.1094,  0.5002,  0.8838, -0.0006,
         -1.7993, -1.3706,  0.4964, -0.3251, -1.7313,  1.8731,  2.4963,  1.1713,
         -1.5726,  1.5476,  3.9576,  0.7388,  0.0228,  0.3947, -1.7237, -1.8350,
         -2.0297,  1.4088, -1.3469,  1.6128, -1.0851,  2.0257,  0.5881,  0.7498,
          0.0738,  2.0592,  1.8034, -0.5468,  1.9512,  0.4534,  0.7746, -1.0465,
         -0.7254,  0.3333, -1.6506, -0.4242,  1.9529, -0.4542,  0.2396, -1.6804,
         -2.7987, -0.6367, -0.3599,  1.0102,  2.6319,  0.8305, -1.4333,  3.3043,
         -0.4021, -0.4877,  0.9125,  0.0607, -1.0326,  1.3186, -2.5861,  0.1211,
         -2.3177, -1.5040,  1.0416,  1.4008,  1.4225, -2.7291]],
       grad_fn=<ThAddmmBackward>)

sum([ 2.5686, -1.1964, -0.0872, -1.7010, -1.6669, -1.0638,  0.4515,  0.1124,
          0.0166,  0.3156,  1.1699,  1.5374,  1.8720,  2.5184,  2.9046, -0.8241,
         -1.1949, -0.5700,  0.8692, -1.0485,  0.0390, -1.3783, -3.4632, -0.0143,
          1.0986,  0.2667, -1.1127, -0.8515,  0.7759, -0.7528,  1.6366, -0.1170,
         -0.4983, -2.6970,  0.7545,  0.0188,  0.1094,  0.5002,  0.8838, -0.0006,
         -1.7993, -1.3706,  0.4964, -0.3251, -1.7313,  1.8731,  2.4963,  1.1713,
         -1.5726,  1.5476,  3.9576,  0.7388,  0.0228,  0.3947, -1.7237, -1.8350,
         -2.0297,  1.4088, -1.3469,  1.6128, -1.0851,  2.0257,  0.5881,  0.7498,
          0.0738,  2.0592,  1.8034, -0.5468,  1.9512,  0.4534,  0.7746, -1.0465,
         -0.7254,  0.3333, -1.6506, -0.4242,  1.9529, -0.4542,  0.2396, -1.6804,
         -2.7987, -0.6367, -0.3599,  1.0102,  2.6319,  0.8305, -1.4333,  3.3043,
         -0.4021, -0.4877,  0.9125,  0.0607, -1.0326,  1.3186, -2.5861,  0.1211,
         -2.3177, -1.5040,  1.0416,  1.4008,  1.4225, -2.7291])
>>>
5.325799999999998

given this as how I test it on a single image (and the model as usual is trained and tested on batches it returns a prediction matrix that doesn't seem to be normalized or add up to 1.

Is this normal?


title: Pytorch - Use parts or segments of existing models

body:
I have a "how to do" question about Pytorch.
Suppose I have two models in Pytorch, and I want to input in the second model an output that comes out of the middle of the first model. But the first model is not sequential, it's very complex and may have inner branches. Toy example:

How can I make a version/subclass/copy of model 1 that outputs both the final and the middle tensors?

What I could do so far?
So far, the only way I found was to know the entire model and all its forward connections, so I replicate the forward method exactly and make it output the desired middle tensor.
But this is very troublesome, as I don't really want to rebuild entire models with high complexity. This is too much prone to errors.
Is there any easier way, like it's done in Keras, for instance?
See how easily it's done in Keras
Just get the middle layer's output and put it as an output of a new model based on model1:
middleTensor = model1.get_layer(layer_name).output #or model1.layers[i]
newModel1 = Model(model1.input, [middleTensor,model1.output])

Now I have a version of model 1 that outputs both the final and the middle tensors without any effort to deeply understand or replicate it.

Is there any similar approach in pytorch?


title: Pytorch: How to backward only a part of a recursive tree?

body:
Assume I have

result = mynet1(input)

calculated_value1 = result
calculated_value2 = calculated_value1 * result

and I want to only derive result using backward, in the calculated_value2 expression, but not the inner result which comes from calculated_value1

Meaning, I want

calculated_value2.backward()

to refer to calculated_value1 as a constant, even though it depends on result, so that the gradient would only depend on result once, rather that on result^2.

Obviously, result.detach() won't do the trick, because then the entire expression would have no parameters to optimize.

How can I achieve the described behavior?

Clarification of intent:

I intend to implement section 13.5 of this book

In which delta depends on V, but is not being derived, while V itself IS being derived.

Also, if there is a more suitable name for this question, please tell me so that I could change it.


title: Equal output values given for Multiclass Classification

body:
I'm trying to build a CNN for predicting the number of fingers in an image, using PyTorch. The network:

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()

        self.Layer1 = nn.Sequential(
        nn.Conv2d(in_channels=3, out_channels=16, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=256, out_channels=16, kernel_size=(1, 1)),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=256, out_channels=16, kernel_size=(1, 1)),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=128, out_channels=16, kernel_size=(1, 1)),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size=(2, 2)),
        nn.Conv2d(in_channels=128, out_channels=16, kernel_size=(1, 1)),
        nn.Conv2d(in_channels=16, out_channels=32, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=32, out_channels=64, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=(3, 3)),
        nn.ReLU(),
        nn.Conv2d(in_channels=128, out_channels=256, kernel_size=(3, 3)),
        nn.ReLU(),
        )

        self.Layer2 = nn.Sequential(
        nn.Linear(1536, 100),
        nn.Tanh(),
        nn.Linear(100, 6),
        nn.Softmax()
        )
        self.optimizer = optimizers.Adadelta(self.parameters())

    def forward(self, X):
        X = self.Layer1(X)
        print(X.shape)
        X = self.Layer2(X.reshape(1, 1536))
        X = X.squeeze()

        return X

    def calc_loss(self, X, num):
        out = self.forward(X).unsqueeze(dim=0)
        print("Output: "+str(out))
        target = torch.tensor([num], dtype=torch.int64).cuda()
        criterion = nn.CrossEntropyLoss()
        loss = criterion(out, target)
        return loss

    def train_step(self, X, Y):
        loss = self.calc_loss(X, Y)
        print(loss)
        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

However, after the training is complete, all predictions have almost the same values (around 0.15 ~ 0.18).

It appears as though the network averages out the output probabilities to minimize loss, instead of learning the actual values.

I get the same result whether I use Softmax for the last layer with cross-entropy loss, or Sigmoid with binary cross-entropy, MSE, or SmoothL1Loss.

In case of using Adam optimizer, I get similar results, only in the range of 1e-12 ~ 1e-14.

What am I missing?


title: What are the differences between type of layer and its activation function in PyTorch?

body:
I am trying to write some simple neural network using pytorch. I am new to this library. I faced with two ways of implementing the same idea: a layer with some fixed activation function (e.g. tanh).

The first way to implement it:

l1 = nn.Tanh(n_in, n_out)

The second way:

l2 = nn.Linear(n_in, n_out) # linear layer, that do nothing with its input except summation

but in forward propagation use:

import torch.nn.functional as F
x = F.tanh(l2(x)) # x - value that propagates from layer to layer

What are the differences between those mechanisms? Which one is better for which purposes?


title: How the parenthesis are used right after the object's name?

body:
I want to know about the parenthesis after the Object's name.

I am learning AI and building the AI model, now in the Tutorial's code the author has written a line which is containing the Parenthesis right after the object's name which is : self.model(...)

Where self.model is the Object of the Network class.
How the objects are having parenthesis being an object, not a function?
Now I want to know about the parenthesis after the Object's name.

class Network(nn.Module):

    def __init__(self, input_size, nb_action):
        super(Network, self).__init__()
        self.input_size = input_size
        self.nb_action = nb_action
        self.fc1 = nn.Linear(input_size, 30)
        self.fc2 = nn.Linear(30, nb_action)

    def forward(self, state):
        x = F.relu(self.fc1(state))
        q_values = self.fc2(x)
        return q_values

class Dqn():

    def __init__(self, input_size, nb_action, gamma):
        self.gamma = gamma
        self.reward_window = []
        self.model = Network(input_size, nb_action)
        self.memory = ReplayMemory(100000)
        self.optimizer = optim.Adam(self.model.parameters(), lr = 0.001)
        self.last_state = torch.Tensor(input_size).unsqueeze(0)
        self.last_action = 0
        self.last_reward = 0

    def select_action(self, state):
        probs = F.softmax(self.model(Variable(state, volatile = True))*100) # <-- The problem is here where the self.model object is CALLED with Parenthesis.
        action = probs.multinomial(10)
        return action.data[0,0]


title: compute yolo v1 loss when output w,h is negative

body: Then I start to train my model, the output w and h is a negative value, I don't know how to compute the loss in this situation. I can't find any description in the paper.


title: named entity recognition using seq2seq model such as transformer?

body: Can we use neural machine translation (like seq2seq) for named entity recognition?such as  USING TRANSFORMER nerual network FOR NER TASK. SOURCE IS WORD SEQUENCE, TARGET IS TAG sequence LIKE "o o o PERSON o o o location"，is it possible?


title: totally clear GPU memory

body:
I can't seem to clear the GPU memory after sending a single variable to the GPU.

import torch
tm = torch.Tensor([1,2]).to("cuda")
!nvidia-smi

|===============================+======================+======================|
|   0  GeForce RTX 208...  On   | 00000000:3D:00.0 Off |                  N/A |
|  0%   37C    P2    52W / 250W |    730MiB / 10989MiB |      0%      Default

So I use 730MiB...
Now no matter what I try I can not make the 730MiB go to zero:

del tm
torch.cuda.empty_cache()
import sys;sys.modules[__name__].__dict__.clear()
%reset
Once deleted, variables cannot be recovered. Proceed (y/[n])? y
!nvidia-smi
|   0  GeForce RTX 208...  On   | 00000000:3D:00.0 Off |                  N/A |
|  0%   35C    P8     1W / 250W |    728MiB / 10989MiB |      0%      Default |

I would be happy to hear any suggestions,
Thanks


title: How to improve object detection model pytorch

body:
i have 433 image dataset which consists of class (multiple class can be inside one image):
'a': 634,
'b': 501,
'c': 590,
'd': 293,
'e': 524,
'f': 262

i know my data is too small so i added several image augmentation technique and i get 2598 images without original image:
'a': 3804,
'b': 3006,
'c': 3540,
'd': 1757,
'e': 3142,
'f': 1569

i'm using SSD as detector and mobilenet v1 as backbone based on this repo:
https://github.com/qfgaohao/pytorch-ssd
in ssd there are 2 loss function , regression with smoothL1 and classification with  cross entropy,
i trained with my dataset with 80% training and 20% testing, i forgot to plot the training and validation loss but i'm sure i don't overfit my model, i stopped training when the loss function increase, i get around 0.5 for regression loss and 1.5 for classification loss, i trained without pretrained model because that repo only provide voc pretrained model.
when i try to test my model with live cam the accuracy of detection is not good but can detect object with probability below 0.5, and cannot detect with little oclussion like when i grab the object with my hand.
here AP result for my model:
mAP 33.68%:
'a': 0.38,
'b': 0.45,
'c': 0.27,
'd': 0.65,
'e': 0.16,
'f': 0.11

i want to ask what should i do to improve my model, did i wrong with handling the dataset? thank you


title: How to understand "-input[range(target.shape[0]), target]"?

body:
In the official example of PyTorch, it gives a loss function as follows.
def nll(input, target):
    return -input[range(target.shape[0]), target].mean()

loss_func = nll

How to understand the grammar of "input[range(target.shape[0]), target]" in the above function?
"input" has a torch.Size([64, 10]) and "target" has a torch.Size([64]). Why use "range" function here?


title: Why can't PyCharm show PyTorch Module object attributes in debug mode

body:
I define a sub-class of PyTorch's Module in PyCharm and create an instance a:
from torch.nn import Module

class AModule(Module):
    def __init__(self):
        self.something = 10

    def __repr__(self):
        return "AModule"

a = AModule()

If I run the debugger and examine a, I can't see its attributes:

I checked and Module is written in Python (as opposed to being implemented in C), so why is that?


title: Getting an error in svi step due to to a multiclass distribution in sample using pyro and pytorch

body:
I'm working on a causal variational autoencoder which works with class segmentation masks, class labels and causality(0 or 1) as the inputs.
I'm getting an error when working with batch sizes more than 1 due to the svi step. I'm using a bernoulling function because I want it to learn the probability distribution for multiple classes in an image. I think that the Categorical distribution also fits the bill here, but I get the same error with it too.
When I tried narrowing down the code lines which create the problem, I think it's in the model function:
one_vec2 = torch.ones([batch_size, self.lbl_shape[0]], **options)
class_labels = pyro.sample('class_labels', dist.Bernoulli(one_vec2*0.5), obs = lbls)

The error:
ValueError                                Traceback (most recent call last)
<ipython-input-19-8cbc046dd2c1> in <module>()
      6 vae = Vae_Model1(lbl_sz, ch, img_sz).to(device)
      7 svi = SVI(vae.model, vae.guide, optimizer, loss = Trace_ELBO())
----> 8 train(svi, train_loader, USE_CUDA)

6 frames
/usr/local/lib/python3.6/dist-packages/pyro/util.py in check_site_shape(site, max_plate_nesting)
    320                 '- enclose the batched tensor in a with plate(...): context',
    321                 '- .to_event(...) the distribution being sampled',
--> 322                 '- .permute() data dimensions']))
    323
    324     # Check parallel dimensions on the left of max_plate_nesting.

ValueError: at site "class_labels", invalid log_prob shape
  Expected [-1], actual [32, 21]
  Try one of the following fixes:
  - enclose the batched tensor in a with plate(...): context
  - .to_event(...) the distribution being sampled
  - .permute() data dimensions

Currently the batch size is 32 and the lbl_shape[0] is 21 (VOC Dataset (background and other labels))
Could someone help me with this? It'll be very much appreciated.
Thank you


title: Pytorch equivalent of TensorFlow

body:
I am trying to follow up on this code in Pytorch. I have been trying for days but reading tensor flow docs then PyTorch docs has made me totally confused.
input_data = Input(shape=(256, 64, 1), name=‘input’)

inner = Conv2D(32, (3, 3), padding=‘same’, name=‘conv1’, kernel_initializer=‘he_normal’)(input_data)
inner = BatchNormalization()(inner)
inner = Activation(‘relu’)(inner)
inner = MaxPooling2D(pool_size=(2, 2), name=‘max1’)(inner)

inner = Conv2D(64, (3, 3), padding=‘same’, name=‘conv2’, kernel_initializer=‘he_normal’)(inner)
inner = BatchNormalization()(inner)
inner = Activation(‘relu’)(inner)
inner = MaxPooling2D(pool_size=(2, 2), name=‘max2’)(inner)
inner = Dropout(0.3)(inner)

inner = Conv2D(128, (3, 3), padding=‘same’, name=‘conv3’, kernel_initializer=‘he_normal’)(inner)
inner = BatchNormalization()(inner)
inner = Activation(‘relu’)(inner)
inner = MaxPooling2D(pool_size=(1, 2), name=‘max3’)(inner)
inner = Dropout(0.3)(inner)

CNN to RNN
inner = Reshape(target_shape=((64, 1024)), name=‘reshape’)(inner)
inner = Dense(64, activation=‘relu’, kernel_initializer=‘he_normal’, name=‘dense1’)(inner)

RNN
inner = Bidirectional(LSTM(256, return_sequences=True), name = ‘lstm1’)(inner)
inner = Bidirectional(LSTM(256, return_sequences=True), name = ‘lstm2’)(inner)

OUTPUT
inner = Dense(num_of_characters, kernel_initializer=‘he_normal’,name=‘dense2’)(inner)
y_pred = Activation(‘softmax’, name=‘softmax’)(inner)

model = Model(inputs=input_data, outputs=y_pred)

I tried following up it in Pytorch, bit by bit
class Net(nn.Module):
def init(self):
super(Net,self).init()
self.input_data = input_size
self.conv1 = nn.Conv2d(32,3,3)
self.conv2 = nn.Conv2d(64,3,3)
self.conv3 = nn.Conv2d(128,3,3)

self.dropout = nn.Dropout(0.3)
self.maxp = torch.nn.MaxPool2d((2,2))

#CNN to RNN
self.linear1 = nn.Linear(256*62*62,64)

#RNN
self.lstm = torch.nn.LSTM(256, 10,bidirectional = True)

#output
self.linear2 = nn.Linear(64,num_of_chars)

def forward(self,x,input_size):

    x = self.conv1(input_size)
    x = nn.BatchNorm2d(x)
    x = F.relu(x)
    x = self.maxp(x)
    x = self.conv2(x)
    x = nn.BatchNorm2d(x)
    x = F.relu(x)
    x = self.maxp(x)
    x = self.dropout(x)
    x = self.conv3(x)
    x = nn.BatchNorm2d(x)
    x = F.relu(x)
    x = self.maxp(x)
    x = self.dropout(x)
    x = x.view((64,1024))
    x = self.linear1(x)

    x = self.lstm(x)
    x = self.lstm(x)

    x = self.linear2(x)
    x = nn.Softmax(x,dim=1)
    return x

But the model summary is not at all same. I am very confused by the parameters. Any help would be appreciated. Tell me if you need anything. Thanks


title: How to check if an object is a certain PyTorch optimizer?

body:
I want to check the class of objects like this:
from torch.optim import Adam, SGD, AdamW

adam_range = (0.8, 1.0)
adamw_range = (0.6, 0.7)
sgd_range = (0.0, 0.5)

targets = []
for cfg in configs:
    if isinstance(cfg["optimizer"], Adam):
        sample = np.random.uniform(low=adam_range[0], high=adam_range[1], size=1)
    elif isinstance(cfg["optimizer"], AdamW):
        sample = np.random.uniform(low=adamw_range[0], high=adamw_range[1], size=1)
    elif isinstance(cfg["optimizer"], SGD):
        sample = np.random.uniform(low=sgd_range[0], high=sgd_range[1], size=1)

where configs is a list of dicts and cfg["optimizer"] are uninitialized references of from torch.optim import Adam, SGD, AdamW. However, none of the if/elif statements evaluate to True.
How do I properly check the type of these references? Is there a better way than checking cfg["optimizer"].__name__? The latter seems inelegant.
Thank you.


title: how to find out the method definition of torch.nn.Module.parameters()

body:
I am following this notebook:
One of the method:
def init_hidden(self, batch_size):
    ''' Initializes hidden state '''
    # Create two new tensors with sizes n_layers x batch_size x n_hidden,
    # initialized to zero, for hidden state and cell state of LSTM
    weight = next(self.parameters()).data

    if (train_on_gpu):
        hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
              weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
    else:
        hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                  weight.new(self.n_layers, batch_size, self.n_hidden).zero_())

    return hidden

I would like to see what type of weight is and how to use new() method, so I was trying to find out the parameters() method as the data attribute comes from paramerters() method.
Surprisingly, I cannot find where it comes from after reading the source code of nn module in PyTorch.
How do you figure out where to see the definition of methods you saw from PyTorch?