title: PyTorch and TensorFlow object detection - evaluate - object of type  cannot be safely interpreted as an integer

body:
I'm attempting to get this PyTorch person detection example running:

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I'm using Ubuntu 18.04. Here is a summary of the steps I've performed:

1) Stock Ubuntu 18.04 install on a Lenovo ThinkPad X1 Extreme Gen 2 with a GTX 1650 GPU.

2) Perform a standard CUDA 10.0 / cuDNN 7.4 install. I'd rather not restate all the steps as this post is going to be more than long enough already. This is a standard procedure, pretty much any link found via googling is what I followed.

3) Install torch and torchvision

4) From this link on the PyTorch site:

https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I saved the source available from the link at the bottom:

https://pytorch.org/tutorials/_static/tv-training-code.py

To a directory I made, PennFudanExample

5) I did the following (found at the top of the above linked notebook):

Install the CoCo API into Python:

cd ~
git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI

open Makefile in gedit, change the two instances of "python" to "python3", then:

python3 setup.py build_ext --inplace
sudo python3 setup.py install

Get the necessary files the above linked files need to run:

cd ~
git clone https://github.com/pytorch/vision.git
cd vision
git checkout v0.5.0

from ~/vision/references/detection, copy coco_eval.py, coco_utils.py, engine.py, transforms.py, and utils.py to directory PennFudanExample.

6) Download the Penn Fudan Pedestrian dataset from the link on the above page:

https://www.cis.upenn.edu/~jshi/ped_html/PennFudanPed.zip

then unzip and put in directory PennFudanExample

7) The only change I made to tv-training-code.py was to change the training batch size from 2 to 1 to prevent a GPU out of memory crash, see this other post I made here:

PyTorch Object Detection with GPU on Ubuntu 18.04 - RuntimeError: CUDA out of memory. Tried to allocate xx.xx MiB

Here is tv-training-code.py as I'm running it with the slight batch size edit I mentioned:

# Sample code from the TorchVision 0.3 Object Detection Finetuning Tutorial
# http://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

import os
import numpy as np
import torch
from PIL import Image

import torchvision
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

from engine import train_one_epoch, evaluate
import utils
import transforms as T

class PennFudanDataset(object):
    def __init__(self, root, transforms):
        self.root = root
        self.transforms = transforms
        # load all image files, sorting them to
        # ensure that they are aligned
        self.imgs = list(sorted(os.listdir(os.path.join(root, "PNGImages"))))
        self.masks = list(sorted(os.listdir(os.path.join(root, "PedMasks"))))

    def __getitem__(self, idx):
        # load images ad masks
        img_path = os.path.join(self.root, "PNGImages", self.imgs[idx])
        mask_path = os.path.join(self.root, "PedMasks", self.masks[idx])
        img = Image.open(img_path).convert("RGB")
        # note that we haven't converted the mask to RGB,
        # because each color corresponds to a different instance
        # with 0 being background
        mask = Image.open(mask_path)

        mask = np.array(mask)
        # instances are encoded as different colors
        obj_ids = np.unique(mask)
        # first id is the background, so remove it
        obj_ids = obj_ids[1:]

        # split the color-encoded mask into a set
        # of binary masks
        masks = mask == obj_ids[:, None, None]

        # get bounding box coordinates for each mask
        num_objs = len(obj_ids)
        boxes = []
        for i in range(num_objs):
            pos = np.where(masks[i])
            xmin = np.min(pos[1])
            xmax = np.max(pos[1])
            ymin = np.min(pos[0])
            ymax = np.max(pos[0])
            boxes.append([xmin, ymin, xmax, ymax])

        boxes = torch.as_tensor(boxes, dtype=torch.float32)
        # there is only one class
        labels = torch.ones((num_objs,), dtype=torch.int64)
        masks = torch.as_tensor(masks, dtype=torch.uint8)

        image_id = torch.tensor([idx])
        area = (boxes[:, 3] - boxes[:, 1]) * (boxes[:, 2] - boxes[:, 0])
        # suppose all instances are not crowd
        iscrowd = torch.zeros((num_objs,), dtype=torch.int64)

        target = {}
        target["boxes"] = boxes
        target["labels"] = labels
        target["masks"] = masks
        target["image_id"] = image_id
        target["area"] = area
        target["iscrowd"] = iscrowd

        if self.transforms is not None:
            img, target = self.transforms(img, target)

        return img, target

    def __len__(self):
        return len(self.imgs)

def get_model_instance_segmentation(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask,
                                                       hidden_layer,
                                                       num_classes)

    return model

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def main():
    # train on the GPU or on the CPU, if a GPU is not available
    device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

    # our dataset has two classes only - background and person
    num_classes = 2
    # use our dataset and defined transformations
    dataset = PennFudanDataset('PennFudanPed', get_transform(train=True))
    dataset_test = PennFudanDataset('PennFudanPed', get_transform(train=False))

    # split the dataset in train and test set
    indices = torch.randperm(len(dataset)).tolist()
    dataset = torch.utils.data.Subset(dataset, indices[:-50])
    dataset_test = torch.utils.data.Subset(dataset_test, indices[-50:])

    # define training and validation data loaders
    # !!!! CHANGE HERE !!!! For this function call, I changed the batch_size param value from 2 to 1, otherwise this file is exactly as provided from the PyTorch website !!!!
    data_loader = torch.utils.data.DataLoader(
        dataset, batch_size=1, shuffle=True, num_workers=4,
        collate_fn=utils.collate_fn)

    data_loader_test = torch.utils.data.DataLoader(
        dataset_test, batch_size=1, shuffle=False, num_workers=4,
        collate_fn=utils.collate_fn)

    # get the model using our helper function
    model = get_model_instance_segmentation(num_classes)

    # move model to the right device
    model.to(device)

    # construct an optimizer
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.SGD(params, lr=0.005,
                                momentum=0.9, weight_decay=0.0005)
    # and a learning rate scheduler
    lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
                                                   step_size=3,
                                                   gamma=0.1)

    # let's train it for 10 epochs
    num_epochs = 10

    for epoch in range(num_epochs):
        # train for one epoch, printing every 10 iterations
        train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)
        # update the learning rate
        lr_scheduler.step()
        # evaluate on the test dataset
        evaluate(model, data_loader_test, device=device)

    print("That's it!")

if __name__ == "__main__":
    main()

Here is the full text output including error I'm getting currently:

Epoch: [0]  [  0/120]  eta: 0:01:41  lr: 0.000047  loss: 7.3028 (7.3028)  loss_classifier: 1.0316 (1.0316)  loss_box_reg: 0.0827 (0.0827)  loss_mask: 6.1742 (6.1742)  loss_objectness: 0.0097 (0.0097)  loss_rpn_box_reg: 0.0046 (0.0046)  time: 0.8468  data: 0.0803  max mem: 1067
Epoch: [0]  [ 10/120]  eta: 0:01:02  lr: 0.000467  loss: 2.0995 (3.5058)  loss_classifier: 0.6684 (0.6453)  loss_box_reg: 0.0999 (0.1244)  loss_mask: 1.2471 (2.7069)  loss_objectness: 0.0187 (0.0235)  loss_rpn_box_reg: 0.0060 (0.0057)  time: 0.5645  data: 0.0089  max mem: 1499
Epoch: [0]  [ 20/120]  eta: 0:00:56  lr: 0.000886  loss: 1.0166 (2.1789)  loss_classifier: 0.2844 (0.4347)  loss_box_reg: 0.1631 (0.1540)  loss_mask: 0.4710 (1.5562)  loss_objectness: 0.0187 (0.0242)  loss_rpn_box_reg: 0.0082 (0.0099)  time: 0.5524  data: 0.0020  max mem: 1704
Epoch: [0]  [ 30/120]  eta: 0:00:50  lr: 0.001306  loss: 0.5554 (1.6488)  loss_classifier: 0.1258 (0.3350)  loss_box_reg: 0.1356 (0.1488)  loss_mask: 0.2355 (1.1285)  loss_objectness: 0.0142 (0.0224)  loss_rpn_box_reg: 0.0127 (0.0142)  time: 0.5653  data: 0.0023  max mem: 1756
Epoch: [0]  [ 40/120]  eta: 0:00:45  lr: 0.001726  loss: 0.4520 (1.3614)  loss_classifier: 0.1055 (0.2773)  loss_box_reg: 0.1101 (0.1530)  loss_mask: 0.1984 (0.8981)  loss_objectness: 0.0063 (0.0189)  loss_rpn_box_reg: 0.0139 (0.0140)  time: 0.5621  data: 0.0023  max mem: 1776
Epoch: [0]  [ 50/120]  eta: 0:00:39  lr: 0.002146  loss: 0.3448 (1.1635)  loss_classifier: 0.0622 (0.2346)  loss_box_reg: 0.1004 (0.1438)  loss_mask: 0.1650 (0.7547)  loss_objectness: 0.0033 (0.0172)  loss_rpn_box_reg: 0.0069 (0.0131)  time: 0.5535  data: 0.0022  max mem: 1776
Epoch: [0]  [ 60/120]  eta: 0:00:33  lr: 0.002565  loss: 0.3292 (1.0543)  loss_classifier: 0.0549 (0.2101)  loss_box_reg: 0.1113 (0.1486)  loss_mask: 0.1596 (0.6668)  loss_objectness: 0.0017 (0.0148)  loss_rpn_box_reg: 0.0082 (0.0140)  time: 0.5590  data: 0.0022  max mem: 1776
Epoch: [0]  [ 70/120]  eta: 0:00:28  lr: 0.002985  loss: 0.4105 (0.9581)  loss_classifier: 0.0534 (0.1877)  loss_box_reg: 0.1049 (0.1438)  loss_mask: 0.1709 (0.5995)  loss_objectness: 0.0015 (0.0132)  loss_rpn_box_reg: 0.0133 (0.0138)  time: 0.5884  data: 0.0023  max mem: 1783
Epoch: [0]  [ 80/120]  eta: 0:00:22  lr: 0.003405  loss: 0.3080 (0.8817)  loss_classifier: 0.0441 (0.1706)  loss_box_reg: 0.0875 (0.1343)  loss_mask: 0.1960 (0.5510)  loss_objectness: 0.0015 (0.0122)  loss_rpn_box_reg: 0.0071 (0.0137)  time: 0.5812  data: 0.0023  max mem: 1783
Epoch: [0]  [ 90/120]  eta: 0:00:17  lr: 0.003825  loss: 0.2817 (0.8171)  loss_classifier: 0.0397 (0.1570)  loss_box_reg: 0.0499 (0.1257)  loss_mask: 0.1777 (0.5098)  loss_objectness: 0.0008 (0.0111)  loss_rpn_box_reg: 0.0068 (0.0136)  time: 0.5644  data: 0.0022  max mem: 1794
Epoch: [0]  [100/120]  eta: 0:00:11  lr: 0.004244  loss: 0.2139 (0.7569)  loss_classifier: 0.0310 (0.1446)  loss_box_reg: 0.0327 (0.1163)  loss_mask: 0.1573 (0.4731)  loss_objectness: 0.0003 (0.0101)  loss_rpn_box_reg: 0.0050 (0.0128)  time: 0.5685  data: 0.0022  max mem: 1794
Epoch: [0]  [110/120]  eta: 0:00:05  lr: 0.004664  loss: 0.2139 (0.7160)  loss_classifier: 0.0325 (0.1358)  loss_box_reg: 0.0327 (0.1105)  loss_mask: 0.1572 (0.4477)  loss_objectness: 0.0003 (0.0093)  loss_rpn_box_reg: 0.0047 (0.0128)  time: 0.5775  data: 0.0022  max mem: 1794
Epoch: [0]  [119/120]  eta: 0:00:00  lr: 0.005000  loss: 0.2486 (0.6830)  loss_classifier: 0.0330 (0.1282)  loss_box_reg: 0.0360 (0.1051)  loss_mask: 0.1686 (0.4284)  loss_objectness: 0.0003 (0.0086)  loss_rpn_box_reg: 0.0074 (0.0125)  time: 0.5655  data: 0.0022  max mem: 1794
Epoch: [0] Total time: 0:01:08 (0.5676 s / it)
creating index...
index created!
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 117, in linspace
    num = operator.index(num)
TypeError: 'numpy.float64' object cannot be interpreted as an integer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/cdahms/workspace-apps/PennFudanExample/tv-training-code.py", line 166, in <module>
    main()
  File "/home/cdahms/workspace-apps/PennFudanExample/tv-training-code.py", line 161, in main
    evaluate(model, data_loader_test, device=device)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "/home/cdahms/workspace-apps/PennFudanExample/engine.py", line 80, in evaluate
    coco_evaluator = CocoEvaluator(coco, iou_types)
  File "/home/cdahms/workspace-apps/PennFudanExample/coco_eval.py", line 28, in __init__
    self.coco_eval[iou_type] = COCOeval(coco_gt, iouType=iou_type)
  File "/home/cdahms/models/research/pycocotools/cocoeval.py", line 75, in __init__
    self.params = Params(iouType=iouType) # parameters
  File "/home/cdahms/models/research/pycocotools/cocoeval.py", line 527, in __init__
    self.setDetParams()
  File "/home/cdahms/models/research/pycocotools/cocoeval.py", line 506, in setDetParams
    self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)
  File "<__array_function__ internals>", line 6, in linspace
  File "/usr/local/lib/python3.6/dist-packages/numpy/core/function_base.py", line 121, in linspace
    .format(type(num)))
TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.

Process finished with exit code 1

The really strange thing is after I resolved the above-mentioned GPU error this was working for about 1/2 a day and now I'm getting this error, and I could swear I didn't change anything.

I've tried uninstalling and reinstalling torch, torchvision, pycocotools, and for copying the files coco_eval.py, coco_utils.py, engine.py, transforms.py, and utils.py, I've tried checking out torchvision v0.5.0, v0.4.2, and using the latest commit, all produce the same error.

Also, I was working from home yesterday (Christmas) and this error does not happen on my home computer, which is also Ubuntu 18.04 with an NVIDIA GPU.

In Googling for this error one suggestion that is relatively common is to backdate numpy to 1.11.0, but that version is really old now and therefore this would likely cause problems with other packages.

Also in Googleing for this error it seems the general fix is to add a cast to int somewhere or to change a divide by / to // but I'm really hesitant to make changes internal to pycocotools or worse yet inside numpy.  Also since error was not occurring previously and is not occurring on another computer I don't suspect this is a good idea anyway.

Fortunately I can comment out the line

evaluate(model, data_loader_test, device=device)

For now and the training will complete, although I don't get the evaluation data (Mean Average Precision, etc.)

About the only thing left I can think of at this point is to format the HD and reinstall Ubuntu 18.04 and everything else, but this will take at least a day, and if this ever happens again I'd really like to know what may be causing it.

Ideas?  Suggestions?  Additional stuff I should check?

-- EDIT --

After re-testing on the same computer experiencing the concern, I found this same error occurs with the evaluation step when using the TensorFlow object detection API.


title: What is a buffer in Pytorch?

body:
I understand what register_buffer does and the difference between register_buffer and register_parameters.

But what is the precise definition of a buffer in PyTorch?


While trying to import warmup_linear, I'm getting this error

ImportError: cannot import name 'warmup_linear'

Import -

from pytorch_pretrained_bert.optimization import BertAdam, warmup_linear

Requirements file

boto3==1.9.198
botocore==1.12.198
certifi==2019.6.16
chardet==3.0.4
docutils==0.14
h5py==2.9.0
idna==2.8
jmespath==0.9.4
Keras==2.2.4
Keras-Applications==1.0.8
Keras-Preprocessing==1.1.0
numpy==1.17.0
Pillow==6.1.0
python-dateutil==2.8.0
pytorch-pretrained-bert==0.6.2
PyYAML==5.1.1
regex==2019.6.8
requests==2.22.0
s3transfer==0.2.1
scipy==1.3.0
seqeval==0.0.12
six==1.12.0
torch==1.1.0
torchvision==0.3.0
tqdm==4.32.2
urllib3==1.25.3

What needs to be done to import 'warmup_linear'?


title: PyTorch Confusion Matrix Plot

body:
I have met a problem for plotting a confusion matrix. The upper and lower lines are put incorrectly.
When I plot it, it looks like this.

I think there is nothing wrong with my code, since I took it from this YouTube exactly.

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

# Graphical analytics
cm = confusion_matrix(train_set.targets, train_preds.argmax(dim=1))
names = ('T-shirt/top','Trouser','Pullover','Dress','Coat','Sandal','Shirt','Sneaker','Bag','Ankle boot')
plt.figure(figsize=(10,10))
plot_confusion_matrix(cm, names)


title: How to do a "element by element in-place inverse" with pytorch?

body:
Given is an array a:

a = np.arange(1, 11, dtype = 'float32')

With numpy, I can do the following:

np.divide(1.0, a, out = a)

Resulting in:

array([1.        , 0.5       , 0.33333334, 0.25      , 0.2       ,
       0.16666667, 0.14285715, 0.125     , 0.11111111, 0.1       ],
      dtype=float32)

Assuming that a is instead a pytorch tensor, the following operation fails:

torch.div(1.0, a, out = a)

The first parameter of div is expected to be a tensor of matching length/shape.

If I substitute 1.0 with an array b filled with ones, its length equal to the length of a, it works. The downside is that I have to allocate memory for b. I can also do something like a = 1.0 / a which will yet again allocate extra (temporary) memory.

How can I do this operation efficiently "in-place" (without the allocation of extra memory), ideally with broadcasting?


title: Batch normalization seems to not work same in keras and pytorch

body:
I am having a simple model and trying out how batch normalization works, applying after linear layer. It seem to not normalize at all, as by default it is initialized to identity in keras. After checking with the same weights in pytorch, its batch normalization does make changes. Please see below. Why and what is wrong in the model?

Edit: self-contained example which prints results both from keras and pytorch model for visual comparison. To use batchnormalization layer, please uncomment few lines that are pointing it out, then compare the results again.

import tensorflow as tf
import numpy as np
from collections import OrderedDict

from tensorflow.python.keras import layers
from tensorflow.python.keras import models

import torch
from torch import nn
from torch.nn import functional as F

from tensorflow.contrib import eager as tfe
tfe.enable_eager_execution()

class PytorchModel(nn.Module):
    def __init__(self,
                 in_channels,
                 out_channels):
        super().__init__()

        self.linear = nn.Linear(in_channels, out_channels, bias=True)
        self.norm = nn.BatchNorm1d(out_channels, eps=1e-3, momentum=0.01)

    def forward(self, inputs):
        x = self.linear(inputs)
        ## uncomment for batch normalization
        # x = self.norm(x.permute(0, 2, 1).contiguous()).permute(0, 2, 1).contiguous()
        x = F.relu(x)
        return x

class KerasModel(models.Model):
    def __init__(self,
                 num_filters):
        super(KerasModel, self).__init__()

        my_layers = []
        BN = layers.BatchNormalization(name='my_bn', momentum=0.01, epsilon=1e-3)
        LIN = layers.Dense(num_filters, name='my_linear', activation=None, use_bias=True)
        my_layers.append([LIN, BN])
        self.my_layers = my_layers

    def call(self, ins):
        x = self.my_layers[0][0](tf.convert_to_tensor(ins))
        ## uncomment for batch normalization
        # x = self.my_layers[0][1](x)
        x = tf.nn.relu(x)
        return x

if __name__ == '__main__':

    # create dummy input
    np.random.seed(0)
    input_np = np.random.rand(4,5,6)
    filters = 8

    keras_l = KerasModel(num_filters=filters)

    tf_features = keras_l(tf.convert_to_tensor(input_np))

    pytorch_l = PytorchModel(in_channels=6,
                             out_channels=filters)

    # copy weights from keras model to pytorch model
    new_state_dict = OrderedDict()
    new_state_dict['linear.weight'] = torch.from_numpy(np.transpose(keras_l.layers[0].weights[0].numpy(), (1, 0)))
    new_state_dict['linear.bias'] = torch.from_numpy(keras_l.layers[0].bias.numpy())
    ## uncomment for batch normalization
    # new_state_dict['norm.weight'] = torch.from_numpy(keras_l.layers[1].weights[0].numpy())  # gamma
    # new_state_dict['norm.bias'] = torch.from_numpy(keras_l.layers[1].weights[1].numpy())  # bias
    # new_state_dict['norm.running_mean'] = torch.from_numpy(keras_l.layers[1].weights[2].numpy())
    # new_state_dict['norm.running_var'] = torch.from_numpy(keras_l.layers[1].weights[3].numpy())
    pytorch_l.load_state_dict(new_state_dict, strict=False)

    batch_input_voxels_np = torch.from_numpy(input_np).float()
    batch_pytorch_features = pytorch_l.forward(batch_input_voxels_np)

    # => check how results differ, when batch normalization is applied.
    print(tf_features[0, 0, :])
    print(pytorch_features[0, 0, :])


title: Cant install pytorch - ModuleNotFoundError: No module named 'tools.nnwrap'

body:
I cannot install PyTorch on python 3.7. This error occurs for others but the suggested fixes did not work. I have tried instaling wheel, and importing tools but neither work.
Got error: ModuleNotFoundError: No module named 'tools.nnwrap'
ran command pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu102/torch_nightly.html
full output: Looking in links: https://download.pytorch.org/whl/nightly/cu102/torch_nightly.html Collecting torch Using cached torch-0.1.2.post2.tar.gz (128 kB) Collecting torchvision Using cached torchvision-0.2.2.post3-py2.py3-none-any.whl (64 kB) Requirement already satisfied: pyyaml in c:\users\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\lib\site-packages (from torch) (5.3.1) Collecting pillow>=4.1.1 Using cached Pillow-7.2.0-cp37-cp37m-win32.whl (1.8 MB) Collecting numpy Using cached numpy-1.19.1-cp37-cp37m-win32.whl (10.9 MB) Collecting six Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) Using legacy 'setup.py install' for torch, since package 'wheel' is not installed. Installing collected packages: torch, pillow, numpy, six, torchvision Running setup.py install for torch ... error ERROR: Command errored out with exit status 1: command: 'c:\users\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\scripts\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[ 0] = '"'"'C:\\Users\\LV7682\\AppData\\Local\\Temp\\pip-install-4b80gt33\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\LV7682\\AppData\\Local\\Temp\\pip-install-4b80gt33\\torch\\setup .py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --reco rd 'C:\Users\LV7682\AppData\Local\Temp\pip-record-z5_fy7cc\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\lv7682\documents\repos\security _patch\spms\webapp\securitypatchmanagementsystem\venv\venv\include\site\python3.7\torch' cwd: C:\Users\LV7682\AppData\Local\Temp\pip-install-4b80gt33\torch\ Complete output (23 lines): running install running build_deps Traceback (most recent call last): File "<string>", line 1, in <module> File "C:\Users\LV7682\AppData\Local\Temp\pip-install-4b80gt33\torch\setup.py", line 265, in <module> description="Tensors and Dynamic neural networks in Python with strong GPU acceleration", File "c:\users\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\lib\site-packages\setuptools\__init__.py", line 163, in setup return distutils.core.setup(**attrs) File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\core.py", line 148, in setup dist.run_commands() File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist.py", line 966, in run_commands self.run_command(cmd) File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "C:\Users\LV7682\AppData\Local\Temp\pip-install-4b80gt33\torch\setup.py", line 99, in run self.run_command('build_deps') File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\cmd.py", line 313, in run_command self.distribution.run_command(command) File "C:\Users\LV7682\AppData\Local\Programs\Python\Python37-32\lib\distutils\dist.py", line 985, in run_command cmd_obj.run() File "C:\Users\LV7682\AppData\Local\Temp\pip-install-4b80gt33\torch\setup.py", line 51, in run from tools.nnwrap import generate_wrappers as generate_nn_wrappers ModuleNotFoundError: No module named 'tools.nnwrap' ---------------------------------------- ERROR: Command errored out with exit status 1: 'c:\users\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\scripts\python.exe' -u -c 'import sys , setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\LV7682\\AppData\\Local\\Temp\\pip-install-4b80gt33\\torch\\setup.py'"'"'; __file__='"'"'C:\\Users\\LV7682\\AppData\\Local\\Temp\\p ip-install-4b80gt33\\torch\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\LV7682\AppData\Local\Temp\pip-record-z5_fy7cc\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\user s\lv7682\documents\repos\security_patch\spms\webapp\securitypatchmanagementsystem\venv\venv\include\site\python3.7\torch' Check the logs for full command output.


When I load the BERT pretrained model online I get this error OSError: Error no file named ['pytorch_model.bin', 'tf_model.h5', 'model.ckpt.index'] found in directory uncased_L-12_H-768_A-12 or 'from_tf' set to False what should I do?


title: Map function along arbitrary dimension with pytorch? numpy?

body:
I wonder why I find no utility to map custom pytorch or numpy transformations along any dimensions of complicated tensors/arrays/matrices.
I think I remember that such a thing was available in R. With this fantasy tch.map utility you could do:

>>> import torch as tch # or numpy

>>> # one torch tensor
>>> a = tch.tensor([0, 1, 2, 3, 4])
>>> # one torch function (dummy) returning 2 values
>>> f = lambda x: tch.tensor((x + 1, x * 2))
>>> # map f along dimension 0 of a, expecting 2 outputs
>>> res = tch.map(f, a, 0, 2) # fantasy, optimized on CPU/GPU..
>>> res
tensor([[1, 0],
        [2, 2],
        [3, 4],
        [4, 6],
        [5, 8]])
>>> res.shape
torch.Size([5, 2])

>>> # another tensor
>>> a = tch.tensor(list(range(24))).reshape(2, 3, 4).type(tch.double)
>>> # another function (dummy) returning 2 values
>>> f = lambda x: tch.tensor((tch.mean(x), tch.std(x)))
>>> # map f along dimension 2 of a, expecting 2 outputs
>>> res = tch.map(f, a, 2, 2) # fantasy, optimized on CPU/GPU..
tensor([[[ 1.5000,  1.2910],
         [ 5.5000,  1.2910],
         [ 9.5000,  1.2910]],

        [[13.5000,  1.2910],
         [17.5000,  1.2910],
         [21.5000,  1.2910]]])
>>> res.shape
torch.Size([2, 3, 2])

>>> # yet another tensor
>>> a = tch.tensor(list(range(12))).reshape(3, 4)
>>> # another function (dummy) returning 2x2 values
>>> f = lambda x: x + tch.rand(2, 2)
>>> # map f along all values of a, expecting 2x2 outputs
>>> res = tch.map(f, a, -1, (2, 2)) # fantasy, optimized on CPU/GPU..
>>> print(res)
tensor([[[[ 0.4827,  0.3043],
          [ 0.8619,  0.0505]],

         [[ 1.4670,  1.5715],
          [ 1.1270,  1.7752]],

         [[ 2.9364,  2.0268],
          [ 2.2420,  2.1239]],

         [[ 3.9343,  3.6059],
          [ 3.3736,  3.5178]]],

        [[[ 4.2063,  4.9981],
          [ 4.3817,  4.4109]],

         [[ 5.3864,  5.3826],
          [ 5.3614,  5.1666]],

         [[ 6.6926,  6.2469],
          [ 6.7888,  6.6803]],

         [[ 7.2493,  7.5727],
          [ 7.6129,  7.1039]]],

        [[[ 8.3171,  8.9037],
          [ 8.0520,  8.9587]],

         [[ 9.5006,  9.1297],
          [ 9.2620,  9.8371]],

         [[10.4955, 10.5853],
          [10.9939, 10.0271]],

         [[11.3905, 11.9326],
          [11.9376, 11.6408]]]])
>>> res.shape
torch.Size([3, 4, 2, 2])

Instead, I keep finding myself messing around with complicated tch.stack, tch.squeeze, tch.reshape, tch.permute, etc., counting dimensions on my fingers not to get lost.

Does such a utility exist and I have missed it for some reason?
Is such a utility impossible to implement for some reason?


title: Invalid device id when using pytorch dataparallel！

body:
Environment：

Win10
Pytorch 1.3.0
python3.7

Problem：

I am using dataparallel in Pytorch to use the two 2080Ti GPUs. Code are like below:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = Darknet(opt.model_def)
model.apply(weights_init_normal)

model = nn.DataParallel(model, device_ids=[0, 1]).to(device)

But when run this code, I encounter errors below:

Traceback (most recent call last):
  File "C:/Users/Administrator/Desktop/PyTorch-YOLOv3-master/train.py", line 74, in <module>
    model = nn.DataParallel(model, device_ids=[0, 1]).to(device)
  File "C:\Users\Administrator\Anaconda3\envs\py37_torch1.3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 133, in __init__
    _check_balance(self.device_ids)
  File "C:\Users\Administrator\Anaconda3\envs\py37_torch1.3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 19, in _check_balance
    dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
  File "C:\Users\Administrator\Anaconda3\envs\py37_torch1.3\lib\site-packages\torch\nn\parallel\data_parallel.py", line 19, in <listcomp>
    dev_props = [torch.cuda.get_device_properties(i) for i in device_ids]
  File "C:\Users\Administrator\Anaconda3\envs\py37_torch1.3\lib\site-packages\torch\cuda\__init__.py", line 337, in get_device_properties
    raise AssertionError("Invalid device id")
AssertionError: Invalid device id

When I debug into it, I find the function device_count() in get_device_properties() returns 1 while I have 2 GPU on my machine. And torch._C._cuda_getDeviceCount() returns 2 in Anaconda Prompt. What is wrong?

Qustion:

How to solve this problem?
How can I manage to use the two GPUs using dataparallel?
Thank you guys!


title: Pytorch save embeddings as part of encoder class or not

body:
So I'm using pytorch for the first time. I'm trying to save weights to a file. I'm using a Encoder class that has a GRU and a embedding component. I want to make sure when I save the Encoder values that I will get the embedding values. Initially my code uses state_dict() to copy values to a dictionary of my own which I pass to torch.save(). Should I be looking for some way to save this embedding component or is it part of the larger encoder? The Encoder is a subclass of nn.Module . here's a link:

http://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html#sphx-glr-intermediate-seq2seq-translation-tutorial-py

def make_state(self, converted=False):
    if not converted:
        z = [
            {
                'epoch':0,
                'arch': None,
                'state_dict': self.model_1.state_dict(),
                'best_prec1': None,
                'optimizer': self.opt_1.state_dict(),
                'best_loss': self.best_loss
            },
            {
                'epoch':0,
                'arch':None,
                'state_dict':self.model_2.state_dict(),
                'best_prec1':None,
                'optimizer': self.opt_2.state_dict(),
                'best_loss': self.best_loss
            }
        ]
    else:
        z = [
            {
                'epoch': 0,
                'arch': None,
                'state_dict': self.model_1.state_dict(),
                'best_prec1': None,
                'optimizer': None , # self.opt_1.state_dict(),
                'best_loss': self.best_loss
            },
            {
                'epoch': 0,
                'arch': None,
                'state_dict': self.model_2.state_dict(),
                'best_prec1': None,
                'optimizer': None, # self.opt_2.state_dict(),
                'best_loss': self.best_loss
            }
        ]
    #print(z)
    return z
    pass

def save_checkpoint(self, state=None, is_best=True, num=0, converted=False):
    if state is None:
        state = self.make_state(converted=converted)
        if converted: print(converted, 'is converted.')
    basename = hparams['save_dir'] + hparams['base_filename']
    torch.save(state, basename + '.' + str(num)+ '.pth.tar')
    if is_best:
        os.system('cp '+ basename + '.' + str(num) + '.pth.tar' + ' '  +
                  basename + '.best.pth.tar')

https://discuss.pytorch.org/t/saving-and-loading-a-model-in-pytorch/2610/3

Here is another link


title: 'None' gradients in pytorch

body:
I am trying to implement a simple MDN that predicts the parameters of a distribution over a target variable instead of a point value, and then assigns probabilities to discrete bins of the point value. Narrowing down the issue, the code from which the 'None' springs is:

import torch

# params
tte_bins = np.linspace(
    start=0,
    stop=399,
    num=400,
    dtype='float32'
).reshape(1, 1, -1)
bins = torch.tensor(tte_bins, dtype=torch.float32)
x_train = np.random.randn(1, 1024, 3)
y_labels = np.random.randint(low=0, high=399, size=(1, 1024))
y_train = np.eye(400)[y_labels]

# data
in_train = torch.tensor(x_train[0:1, :, :], dtype=torch.float)
in_train = (in_train - torch.mean(in_train)) / torch.std(in_train)
out_train = torch.tensor(y_train[0:1, :, :], dtype=torch.float)

# model
linear = torch.nn.Linear(in_features=3, out_features=2)
lin = linear(in_train)
preds = torch.exp(lin)

# intermediate values
alpha = torch.clamp(preds[0:1, :, 0:1], 0, 500)
beta = torch.clamp(preds[0:1, :, 1:2], 0, 100)

# probs
p1 = torch.exp(-torch.pow(bins / alpha, beta))
p2 = torch.exp(-torch.pow((bins + 1.0) / alpha, beta))
probs = p1 - p2

# loss
loss = torch.mean(torch.pow(out_train - probs, 2))

# gradients
loss.backward()
for p in linear.parameters():
    print(p.grad, 'gradient')

in_train has shape: [1, 1024, 3], out_train has shape: [1, 1024, 400], bins has shape: [1, 1, 400]. All the broadcasting etc.. appears find, the resulting matrices (like alpha/beta/loss) are the right shape and have the right values - there's simply no gradients

edit: added loss.backward() and x_train/y_train, now I have nans


title: Build Pytorch from source

body:
I'm trying to install Pytorch from source on my MacOS (version 10.14.3) to use GPU. I have follow the documentation from this link. When I launch in my terminal the MACOSX_DEPLOYMENT_TARGET=10.9 CC=clang CXX=clang++ python setup.py install I'm getting the following error in my terminal:

[ 69%] Built target caffe2_observers make: *** [all] Error 2 Traceback (most recent
call last): File "setup.py", line 710, in <module>
build_deps()
File "setup.py", line 282, in build_deps
build_dir='build')
File "/Users/Desktop/pytorch/tools/build_pytorch_libs.py", line 259, in build_caffe2
check_call(['make', '-j', str(max_jobs), 'install'], cwd=build_dir, env=my_env)
File "/Users/anaconda3/lib/python3.6/subprocess.py", line 291, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j', '4', 'install']'
returned non-zero exit status 2.

I tried to upgrade pip and reinstall anaconda and cuda without success.

Here you can find the error just belong :

[ 69%] Building CXX object modules/observers/CMakeFiles/caffe2_observers.dir/net_observer_reporter_print.cc.o
In file included from <built-in>:1:
In file included from /usr/local/cuda/include/cuda_runtime.h:115:
In file included from /usr/local/cuda/include/crt/common_functions.h:77:
/Library/Developer/CommandLineTools/usr/include/c++/v1/string.h:61:15: fatal error: 'string.h' file not found
#include_next <string.h>
          ^~~~~~~~~~
1 error generated.
CMake Error at caffe2_gpu_generated_THCReduceApplyUtils.cu.o.Release.cmake:219 (message):
    Error generating
/Users/Desktop/pytorch/build/caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/./caffe2_gpu_generated_THCReduceApplyUtils.cu.o

make[2]: *** [caffe2/CMakeFiles/caffe2_gpu.dir/__/aten/src/THC/caffe2_gpu_generated_THCReduceApplyUtils.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
Scanning dependencies of target torch_shm_manager
1 error generated.


title: How to modify path where Torch Hub models are downloaded

body:
When I download models through Torch Hub, models are automatically downloaded in /home/me/.cache/torch.

How can I modify this behavior ?


title: Multi class classification - RuntimeError: 1D target tensor expected, multi-target not supported

body:
My goal is to build a multi-class image classifier using Pytorch and based on the EMNIST dataset (black and white pictures of letters).
The shape of my training data X_train is (124800, 28, 28).
The shape of the original target variables y_train is (124800, 1), however I created a one-hot encoding so that now the shape is (124800, 26).
The model that I am building should have 26 output variables, each representing the probability of one letter.
I read in my data as follows:
import scipy .io
emnist = scipy.io.loadmat(DATA_DIR + '/emnist-letters.mat')
data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
X_train = X_train.reshape((-1,28,28), order='F')

y_train = data ['train'][0, 0]['labels'][0, 0]

Then, I created a one-hot-encoding as follows:
y_train_one_hot = np.zeros([len(y_train), 27])

for i in range (0, len(y_train)):
    y_train_one_hot[i, y_train[i][0]] = 1

y_train_one_hot = np.delete(y_train_one_hot, 0, 1)

I create the dataset with:
train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train_one_hot))

batch_size = 128
n_iters = 3000
num_epochs = n_iters / (len(train_dataset) / batch_size)
num_epochs = int(num_epochs)

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

And then I build my model as follows:
class CNNModel(nn.Module):

    def __init__(self):
        super(CNNModel, self).__init__()

        # Convolution 1
        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=0)
        self.relu1 = nn.ReLU()

        # Max pool 1
        self.maxpool1 = nn.MaxPool2d(2,2)

        # Convolution 2
        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=0)
        self.relu2 = nn.ReLU()

        # Max pool 2
        self.maxpool2 = nn.MaxPool2d(kernel_size=2)

        # Fully connected 1 (readout)
        self.fc1 = nn.Linear(32 * 4 * 4, 26)

    def forward(self, x):
        # Convolution 1
        out = self.cnn1(x.float())
        out = self.relu1(out)

        # Max pool 1
        out = self.maxpool1(out)

        # Convolution 2
        out = self.cnn2(out)
        out = self.relu2(out)

        # Max pool 2
        out = self.maxpool2(out)

        # Resize
        # Original size: (100, 32, 7, 7)
        # out.size(0): 100
        # New out size: (100, 32*7*7)
        out = out.view(out.size(0), -1)

        # Linear function (readout)
        out = self.fc1(out)

        return out

model = CNNModel()

criterion = nn.CrossEntropyLoss()

learning_rate = 0.01

optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)

And then I train the model as follows:
iter = 0
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

        # Add a single channel dimension
        # From: [batch_size, height, width]
        # To: [batch_size, 1, height, width]
        images = images.unsqueeze(1)

        # Forward pass to get output/logits
        outputs = model(images)

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(images)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for images, labels in test_loader:

                images = images.unsqueeze(1)

                # Forward pass only to get logits/output
                outputs = model(images)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                correct += (predicted == labels).sum()

            accuracy = 100 * correct / total

            # Print Loss
            print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))

However, when I run this, I get the following error:
    ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-c26c43bbc32e> in <module>()
     21
     22         # Calculate Loss: softmax --> cross entropy loss
---> 23         loss = criterion(outputs, labels)
     24
     25         # Getting gradients w.r.t. parameters

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    930     def forward(self, input, target):
    931         return F.cross_entropy(input, target, weight=self.weight,
--> 932                                ignore_index=self.ignore_index, reduction=self.reduction)
    933
    934

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2315     if size_average is not None or reduce is not None:
   2316         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2317     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2318
   2319

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2113                          .format(input.size(0), target.size(0)))
   2114     if dim == 2:
-> 2115         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2116     elif dim == 4:
   2117         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

I expect that I do something wrong when I initialize/use my loss function. What can I do so that I can start training my model?


title: Why am I getting a divide by zero error here?

body:
So I'm following along this tutorial in the docs on custom datasets. I'm using the MNIST dataset instead of the fancy one in the tutorial. This is the extension of the Dataset class I wrote:
class KaggleMNIST(Dataset):

    def __init__(self, csv_file, transform=None):
        self.pixel_frame = pd.read_csv(csv_file)
        self.transform = transform

    def __len__(self):
        return len(self.pixel_frame)

    def __getitem__(self, index):
        if torch.is_tensor(index):
            index = index.tolist()

        image = self.pixel_frame.iloc[index, 1:]
        image = np.array([image])

        if self.transform:
            image = self.transform(image)

        return image

It works, until I try to use a transform on it:
tsf = transforms.Compose([transforms.ToTensor(),
                          transforms.Normalize((0.5,), (0.5,))
                         ])

trainset = KaggleMNIST('train/train.csv', transform=tsf)

image0 = trainset[0]

I've looked at the stack trace, and it seems like the normalization is happening in this line of code:
c:\program files\python38\lib\site-packages\torchvision\transforms\functional.py in normalize(tensor, mean, std, inplace)
--> 218     tensor.sub_(mean[:, None, None]).div_(std[:, None, None])

So I don't get why there is divide by zero since std should be 0.5, nowhere remotely close to a small value.
Thanks for your help!
EDIT:
This does not answer my question, but I discovered that if I change these lines of code:
image = self.pixel_frame.iloc[index, 1:]
image = np.array([image])

to
image = self.pixel_frame.iloc[index, 1:].to_numpy(dtype='float64').reshape(1, -1)

Essentially, making sure the datatype was float64 fixed the problem. I'm still not sure why the problem existed in the first place, so I'd still be happy for a well-explained answer!


title: ModuleNotFoundError: No module named 'tools.nnwrap' (macOS)

body:
When installing PyTorch using pip install torch I get the following:

from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'

I'm on macOS 10.14.4, Python 3.8.

I read through the thread here but it didn't help.

UPDATE 13/11/19

The accepted answer works.

As a side note, I finally setup my local environment using Anaconda, despite the meaty installation creating an enviroment has been easy:

#create the env
conda create -n mytensorflowenv python=3.7

#activate it
conda activate mytensorflowenv

then run the PyTorch installation for Conda:

conda install pytorch torchvision -c pytorch

All smooth now.


title: How to update a Tensor?

body:
I've had some troubles updating a tensor using a previous one.
My problem: let's suppose that I have a tensor x1 [Nx1] and a new one calculated through the previous, x2 [Nx1]. Now I want to update the elements of x2 that are less than x1. I'm using dtype=torch.cuda.FloatTensor.

  This is the straight code in Python:

import numpy as np
...
index     = np.where(x1 > x2)
x2[index] = x1[index]

  Why can I do this using PyTorch with dtype=torch.cuda.FloatTensor?
  And if the x1 change to [NxD]

Thank you!


title: pyTorch DataLoader killed

body:
I'm trying to run a pyTorch pipeline with the ResNet architecture.
Everithing is run on a CPU with the pyTorch official Docker on a Mac laptop.

Everything seems fine until I get this error:

Batch 23/51
Batch 24/51
Traceback (most recent call last): File "model.py", line 251, in <module>
  loss.backward()
File "/opt/conda/lib/python3.6/site-packages/torch/tensor.py", line 93, in backward
  torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/opt/conda/lib/python3.6/site-packages/torch/autograd/__init__.py", line 89, in backward
  allow_unreachable=True)  # allow_unreachable flag
File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 178, in handler
  _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 397) is killed by signal: Killed.

It seems to work properly until the 24th batch of the first epoch.
The dataset is using the TensorDataset from the standard library.

Can anyone give me a hint on what is happening?
Multiple tries give different batch number (even without shuffle) and the laptop resources seem fine during the entire run.

If I try with workers=0 for the loader I just get:

Batch 24/51
Batch 25/51
Batch 26/51
Batch 27/51
Batch 28/51
Killed

Thank you very much.


title: How to test one single image in pytorch

body:
I created my model in pytorch and is working really good, but when i want to test just one image batch_size=1 always return the second class (in this case a dog).

I tried to test with batch > 1 and in all cases this works!

The architecture:

model = models.densenet121(pretrained=True)

for param in model.parameters():
param.requires_grad = False
from collections import OrderedDict
classifier = nn.Sequential(OrderedDict([
                          ('fc1', nn.Linear(1024, 500)),
                          ('relu', nn.ReLU()),
                          ('fc2', nn.Linear(500, 2)),
                          ('output', nn.LogSoftmax(dim=1))
                          ]))

model.classifier = classifier

so my tensors are [batch, 3, 224, 224]

i have tried with:

resize
reshape
unsqueeze(0)

the response when is one image is always [[0.4741, 0.5259]]

My Test Code

from PIL import *
msize = 256
loader = transforms.Compose([transforms.Scale(imsize), transforms.ToTensor()])

def image_loader(image_name):
    """load image, returns cuda tensor"""
    image = Image.open(image_name)
    image = loader(image).float()
    image = image.unsqueeze(0)
    return image.cuda()
image = image_loader('Cat_Dog_data/test/cat/cat.16.jpg')
with torch.no_grad():
    logits = model.forward(image)
ps = torch.exp(logits)
_, predTest = torch.max(ps,1)
print(ps) ## same value in all cases
imagen_mostrar = images[ii].to('cpu')
helper.imshow(imagen_mostrar,title=clas_perro_gato(predTest), normalize=True)

Second Test Code

andrea_data = datasets.ImageFolder(data_dir + '/andrea', transform=test_transforms)
andrealoader = torch.utils.data.DataLoader(andrea_data, batch_size=1, shuffle=True)
dataiter = iter(andrealoader)
images, labels = dataiter.next()
images, labels = images.to(device), labels.to(device)
ps = torch.exp(model.forward(images))
_, predTest = torch.max(ps,1)
print(ps.float())

if i changed my batch_size to 1 always returned a tensor who say that is a dog [0.43,0.57] for example.

Thanks!


title: outputs are different between ONNX and pytorch

body:
I try to convert my pytorch Resnet50 model to ONNX and do inference. The conversion procedural makes no errors, but the final result of onnx model from onnxruntime has large gaps with the result of origin model from pytorch.
What is possible solution ?

Version of ONNX: 1.5.0
Version of pytorch: 1.1.0
CUDA: 9.0
System: Ubuntu 18.06
Python: 3.5

Here is the code of conversion

import torch
import models
from collections import OrderedDict
state_dict = "/home/yx-wan/newhome/workspace/filter-pruning-geometric-median/scripts/snapshots/resnet50-rate-0.7/best.resnet50.GM_0.7_76.82.pth.tar"
arch = 'resnet50'

def import_sparse(model,state_dict):
    new_state_dict = OrderedDict()
    for k, v in state_dict.items():
        name = k[7:]  # remove `module.`
        new_state_dict[name] = v
    model.load_state_dict(new_state_dict)
    print("sparse_model_loaded")
    return model

# initialize model
model = models.__dict__[arch](pretrained=False).cuda()

checkpoint = torch.load(state_dict)
model = import_sparse(model, checkpoint['state_dict'])
print("Top 1 precise of model: {}".format(checkpoint['best_prec1']))
dummy_input =torch.randn(1, 3, 224, 224).cuda()
torch.onnx.export(model, dummy_input, "{}.onnx".format(arch), verbose=True)

Here is the result checking code

import sys

from onnxruntime.datasets import get_example
import onnxruntime
import cv2
import numpy as np
import torch
import models
import onnxruntime
from collections import  OrderedDict
from my_tools import resize_img

def import_sparse(model,checkpoint):
    new_state_dict = OrderedDict()
    for k, v in checkpoint['state_dict'].items():
        name = k[7:]  # remove `module.`
        new_state_dict[name] = v
    model.load_state_dict(new_state_dict)
    return model

image_path = "./img652.jpg"
onnx_model_path = "/workplace/workspace/filter-pruning-geometric-median/resnet50.onnx"
ckpt="./scripts/snapshots/resnet50-rate-0.7/best.resnet50.GM_0.7_76.82.pth.tar"

img_ori = cv2.imread(image_path)  # BGR
img = cv2.cvtColor(img_ori, cv2.COLOR_BGR2RGB)
img, ratio_h, ratio_w = resize_img(img,224,224)
img = img - np.array([123.68, 116.78, 103.94],dtype=np.float32)
img_batch = np.expand_dims(img, 0)

# NHWC -> NCHW
img_batch = np.transpose(img_batch,[0,3,1,2])

example_model = get_example(onnx_model_path)
sess = onnxruntime.InferenceSession(example_model)
input_name = sess.get_inputs()[0].name
print("Input name  :", input_name)
input_shape = sess.get_inputs()[0].shape
print("Input shape :", input_shape)
input_type = sess.get_inputs()[0].type
print("Input type  :", input_type)
output_name = sess.get_outputs()[0].name
print("Output name  :", output_name)
output_shape = sess.get_outputs()[0].shape
print("Output shape :", output_shape)
output_type = sess.get_outputs()[0].type
print("Output type  :", output_type)

print("Input data shape{}".format(img_batch.shape))
assert(list(input_shape) == list(img_batch.shape))
result_onnx = sess.run([output_name], {input_name: img_batch})

# initialize model
model = models.__dict__["resnet50"]()
checkpoint = torch.load(ckpt,map_location='cpu')
best_prec1 = checkpoint['best_prec1']
model = import_sparse(model,checkpoint)
img_batch = torch.FloatTensor(img_batch)
with torch.no_grad():
    result_torch = model(img_batch)
    result_torch = result_torch.numpy()

print("max onnx-torch:{}".format(np.max(result_onnx-result_torch)))

And the output (with some warning but I think is doesn't matter) of checking code is

2019-08-09 02:59:21.378599853 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.2.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378654931 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.2.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378665235 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.2.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378675069 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.1.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378686874 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378698995 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378718700 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.5.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378729567 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.4.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378739657 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.4.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378752091 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.3.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378762533 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.3.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378771168 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.2.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378781705 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.2.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378792325 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.4.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378802071 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.1.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378812061 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.0.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378822884 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378834198 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378845176 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.2.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378859324 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378869709 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378883281 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.5.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378893302 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.3.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378904876 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378915507 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.0.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378926638 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378938115 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378948686 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378958670 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.2.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378969125 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378979556 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.378990553 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.2.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379001126 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.2.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379011508 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379021900 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379033504 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.2.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379044076 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.2.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379064049 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379076654 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.0.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379089769 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.1.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379102140 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.0.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379114598 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.3.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379133520 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.2.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379144015 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.3.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379155771 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.1.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379167084 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.3.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379178303 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.0.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379189605 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer4.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379199974 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.1.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379211042 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.0.downsample.1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379221800 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer3.5.bn2.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379232566 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer1.0.bn1.num_batches_tracked'. It is not used by any node and should be removed from the model.
2019-08-09 02:59:21.379243442 [W:onnxruntime:Default, graph.cc:2263 CleanUnusedInitializers] Removing initializer 'layer2.1.bn3.num_batches_tracked'. It is not used by any node and should be removed from the model.
Input name  : 0
Input shape : [1, 3, 224, 224]
Input type  : tensor(float)
Output name  : 503
Output shape : [1, 1000]
Output type  : tensor(float)
Input data shape(1, 3, 224, 224)
max onnx-torch:104.89282989501953


title: Why am I getting calculated padding input size per channel smaller than kernel size?

body:
I have the following model but its returning an error. Not sure why. I have tried googling but not found anything so far. My input is an numpy array of 6 by 6.
class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 16, kernel_size=(3,3), stride=1, padding=0)
        self.conv2 = nn.Conv2d(16, 32, kernel_size=(3,3), stride=1, padding=0)
        self.conv3 = nn.Conv2d(32, 64, kernel_size=(3,3), stride=1, padding=0)

        self.fc1 = nn.Linear(64*4*4, 320)
        self.fc2 = nn.Linear(320, 160)
        self.out = nn.Linear(160, 2)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=2, stride=2)

        x = self.conv2(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=2, stride=2)

        x = self.conv3(x)
        x = F.relu(x)
        x = F.max_pool2d(x, kernel_size=2, stride=2)

        x = x.reshape(-1, 64*4*4)
        #x = torch.flatten(x)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.out(x)
        return F.softmax(x, dim=1)

My input is a 6x6 numpy array and I get the following error, any idea why?
RuntimeError: Calculated padded input size per channel: (2 x 2). Kernel size: (3 x 3). Kernel size can't be greater than actual input size


title: How to change activation layer in Pytorch pretrained module?

body:
How to change the activation layer of a Pytorch pretrained network?
Here is my code :

print("All modules")
for child in net.children():
    if isinstance(child,nn.ReLU) or isinstance(child,nn.SELU):
        print(child)

print('Before changing activation')
for child in net.children():
    if isinstance(child,nn.ReLU) or isinstance(child,nn.SELU):
        print(child)
        child=nn.SELU()
        print(child)
print('after changing activation')
for child in net.children():
    if isinstance(child,nn.ReLU) or isinstance(child,nn.SELU):
        print(child)

Here is my output:

All modules
ReLU(inplace=True)
Before changing activation
ReLU(inplace=True)
SELU()
after changing activation
ReLU(inplace=True)


title: Load pytorch model from different working directory

body:
I want to load and access a pretrained model from the directory outside where all files are saved. Here is how directory structure is:

-MyProject
----Model_checkpoint_and_scripts
------access_model.py
--run_model.py
--other files

When run model calls access_model.py it looks for model.py in current working directory and does not find it. As suggested here, I could use

the_model = TheModelClass(*args, **kwargs)
the_model.load_state_dict(torch.load(PATH))

But in this case what is a good way to save all the arguments to initialize the model? I am thinking to pickle the command line args but there are some arguments like vocab size which are calculated.

Thanks


title: Index a torch tensor with an array

body:
I have the following torch tensor:

tensor([[-0.2,  0.3],
    [-0.5,  0.1],
    [-0.4,  0.2]])

and the following numpy array: (I can convert it to something else if necessary)

[1 0 1]

I want to get the following tensor:

tensor([0.3, -0.5, 0.2])

i.e. I want the numpy array to index each sub-element of my tensor. Preferably without using a loop.

Thanks in advance


title: Pytorch nn.functional.batch_norm for 2D input

body:
I am currently implementing a model on which I need to change the running mean and standard deviation during test time. As such, I assume the nn.functional.batch_norm would be a better choice than the nn.BatchNorm2d

However, I have batches of images as input, and am currently not sure how to take in the images. How would I apply nn.functional.batch_norm on batches of 2D images?

The current code I have is this, I post this even though this is not correct:

mu = torch.mean(inp[0])
stddev = torch.std(inp[0])
x = nn.functional.batch_norm(inp[0], mu, stddev, training=True, momentum=0.9)


title: Can numpy arrays run in GPUs?

body:
I am using PyTorch. I have the following code:
import numpy as np
import torch

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X).cuda()

X_split = np.array_split(X.numpy(),
                         indices_or_sections = 2,
                         axis = 0)
X_split

but I am getting this error:
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-121-870b5d3f67b6> in <module>()
----> 1 X_prime_class_split = np.array_split(X_prime_class.numpy(),
      2                                      indices_or_sections = 2,
      3                                      axis = 0)
      4 X_prime_class_split

TypeError: can't convert cuda:0 device type tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first.

The error message is clear and I know how to fix this error by just including .cpu(), ie. X_prime_class.cpu().numpy(). I am just curious to know if this confirms that numpy arrays cannot run in GPUs/Cuda?


title: PyTorch: RuntimeError: Function MulBackward0 returned an invalid gradient at index 0 - expected type torch.cuda.FloatTensor but got torch.FloatTensor

body:
I don't understand what this error is telling me. In a different post the same problem was also addressed but there was no useful solution for this.

Traceback (most recent call last):
  File "train.py", line 252, in <module>
    main()
  File "train.py", line 231, in main
    train(net, training_dataset, targets, device, criterion, optimizer, epoch, args.epochs)
  File "train.py", line 103, in train
    loss.backward()
  File "/home/hb119056/.local/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/hb119056/.local/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function MulBackward0 returned an invalid gradient at index 0 - expected type torch.cuda.FloatTensor but got torch.FloatTensor

This is the corresponding segment from my code.

outputs = net(x, indices)
outputs = outputs.transpose(0, 1)
prob = F.normalize(outputs, p=1, dim=1).detach()
target = torch.from_numpy(np.load(file_dir + '/points/points{:03}.npy'.format(i))).to(device)
rv = torch.zeros(12 * outputs.shape[0])

for j in [x for x in range(10) if x != i]:
    source = torch.from_numpy(np.load(file_dir + '/points/points{:03}.npy'.format(j))).to(device)
    rv = factor.ransac(source, target, prob, n_iter, tol, device) # self-written

predicted = factor.predict(source, rv, outputs, device) # self-written
loss = criterion(predicted, target.type(torch.FloatTensor).to(device))
loss.backward() # error occurs here
optimizer.step()

Any help is greatly appreciated, thank you in advance!


Hello below is the pytorch model I am trying to run. But getting error. I have posted the error trace as well. It was running very well unless I added convolution layers. I am still new to deep learning and Pytorch. So I apologize if this is silly question. I am using conv1d so why should conv1d expect 3 dimensional input and it is also getting a 2d input which is also odd.
class Net(nn.Module):
        def __init__(self):
            super().__init__()
            self.fc1 = nn.Linear(CROP_SIZE*CROP_SIZE*3, 512)
            self.conv1d1 = nn.Conv1d(in_channels=512, out_channels=64, kernel_size=1, stride=2)
            self.fc2 = nn.Linear(64, 128)
            self.conv1d2 = nn.Conv1d(in_channels=128, out_channels=64, kernel_size=1, stride=2)
            self.fc3 = nn.Linear(64, 256)
            self.conv1d3 = nn.Conv1d(in_channels=256, out_channels=64, kernel_size=1, stride=2)
            self.fc4 = nn.Linear(64, 256)
            self.fc4 = nn.Linear(256, 128)
            self.fc5 = nn.Linear(128, 64)
            self.fc6 = nn.Linear(64, 32)
            self.fc7 = nn.Linear(32, 64)

     self.fc8 = nn.Linear(64, frame['landmark_id'].nunique())

    def forward(self, x):
        x = F.relu(self.conv1d1(self.fc1(x)))
        x = F.relu(self.conv1d2(self.fc2(x)))
        x = F.relu(self.conv1d3(self.fc3(x)))
        x = F.relu(self.fc4(x))
        x = F.relu(self.fc5(x))
        x = F.relu(self.fc6(x))
        x = F.relu(self.fc7(x))
        x = self.fc8(x)
        return F.log_softmax(x, dim=1)

net = Net()

import torch.optim as optim

loss_function = nn.CrossEntropyLoss()
net.to(torch.device('cuda:0'))
for epoch in range(3): # 3 full passes over the data
    optimizer = optim.Adam(net.parameters(), lr=0.001)
    for data in tqdm(train_loader):  # `data` is a batch of data
        X = data['image'].to(device)  # X is the batch of features
        y = data['landmarks'].to(device) # y is the batch of targets.
        optimizer.zero_grad()  # sets gradients to 0 before loss calc. You will do this likely every step.
        output = net(X.view(-1,CROP_SIZE*CROP_SIZE*3))  # pass in the reshaped batch
#         print(np.argmax(output))
#         print(y)
        loss = F.nll_loss(output, y)  # calc and grab the loss value
        loss.backward()  # apply this loss backwards thru the network's parameters
        optimizer.step()  # attempt to optimize weights to account for loss/gradients

    print(loss)  # print loss. We hope loss (a measure of wrong-ness) declines!

Error trace
    ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-42-f5ed7999ce57> in <module>
      5         y = data['landmarks'].to(device) # y is the batch of targets.
      6         optimizer.zero_grad()  # sets gradients to 0 before loss calc. You will do this likely every step.
----> 7         output = net(X.view(-1,CROP_SIZE*CROP_SIZE*3))  # pass in the reshaped batch
      8 #         print(np.argmax(output))
      9 #         print(y)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-37-6d3e34d425a0> in forward(self, x)
     16
     17     def forward(self, x):
---> 18         x = F.relu(self.conv1d1(self.fc1(x)))
     19         x = F.relu(self.conv1d2(self.fc2(x)))
     20         x = F.relu(self.conv1d3(self.fc3(x)))

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
    210                             _single(0), self.dilation, self.groups)
    211         return F.conv1d(input, self.weight, self.bias, self.stride,
--> 212                         self.padding, self.dilation, self.groups)
    213
    214

RuntimeError: Expected 3-dimensional input for 3-dimensional weight [64, 512, 1], but got 2-dimensional input of size [4, 512] instead


title: How to fix this error: list indices must be integers or slices, not tuple

body:
Suppose I have a list of tensors called outputs

>> outputs[2][0][0,:,:]
Out[20]:
tensor([[ 14.0448,  -5.1494,  -0.1780,  ...,  10.1937,  -8.9158,  -5.3964],
        [ 32.0382,  -0.5201,  29.9942,  ..., -18.8268, -23.1068,  23.9745],
        [-24.5911,  14.7233,  -6.3053,  ...,  -5.8131,  -3.3088,   0.5685],
        [  0.8842, -14.8318,   6.7204,  ...,  17.7127,   7.3332,   3.7249],
        [  2.0654, -16.5236,  38.3582,  ..., -23.1663,  -5.1202,  13.6506]],
       grad_fn=<SliceBackward>)

>> outputs[2][1][0,:,:]
Out[24]:
tensor([[-0.1260,  0.0463, -0.3362,  ...,  0.1089, -0.2454,  0.0140],
        [ 0.5050, -0.0750, -0.1639,  ..., -0.0020, -0.0521, -0.3224],
        [-0.5311,  0.4526,  0.0079,  ..., -0.0654, -0.1255, -0.0012],
        [ 0.0728, -0.1219,  0.0905,  ...,  0.1354,  0.2730, -0.1186],
        [-0.0680, -0.5570,  0.0295,  ..., -0.2411, -0.1690,  0.0331]],
       grad_fn=<SliceBackward>)

When I try to do:

>> outputs[2][0:1][0,:,:]

python generates an error, and the error message is

TypeError: list indices must be integers or slices, not tuple

How can I fix this error?

Thank you,


title: packed_padded_sequence gives error when used with GPU

body:
I am trying to set up an RNN capable of utilizing a GPU but packed_padded_sequence gives me a

RuntimeError: 'lengths' argument should be a 1D CPU int64 tensor

here is how I direct gpu computing

parser = argparse.ArgumentParser(description='Trainer')
parser.add_argument('--disable-cuda', action='store_true',
                    help='Disable CUDA')
args = parser.parse_args()
args.device = None
if not args.disable_cuda and torch.cuda.is_available():
    args.device = torch.device('cuda')
    torch.set_default_tensor_type(torch.cuda.FloatTensor)
else:
    args.device = torch.device('cpu')

here is the relevant part of the code.

def Tensor_length(track):
    """Finds the length of the non zero tensor"""
    return int(torch.nonzero(track).shape[0] / track.shape[1])
.
.
.
def forward(self, tracks, leptons):
        self.rnn.flatten_parameters()
        # list of event lengths
        n_tracks = torch.tensor([Tensor_length(tracks[i])
                                 for i in range(len(tracks))])
        sorted_n, indices = torch.sort(n_tracks, descending=True)
        sorted_tracks = tracks[indices].to(args.device)
        sorted_leptons = leptons[indices].to(args.device)
        # import pdb; pdb.set_trace()
        output, hidden = self.rnn(pack_padded_sequence(sorted_tracks,
                                                       lengths=sorted_n.cpu().numpy(),
                                                       batch_first=True)) # this gives the error

        combined_out = torch.cat((sorted_leptons, hidden[-1]), dim=1)
        out = self.fc(combined_out)  # add lepton data to the matrix
        out = self.softmax(out)
        return out, indices  # passing indices for reorganizing truth

I have tried everything from casting sorted_n to a long tensor to having it be a list, but i aways seem to get the same error.
I have not worked with pytorch on gpu before and any advice will be greatly appreciated.

Thanks!


title: PyTorch: create non-fully-connected layer / concatenate output of hidden layers

body:
In PyTorch, I want to create a hidden layer whose neurons are not fully connected to the output layer. I try to concatenate the output of two linear layers but run into the following error:

  RuntimeError: size mismatch, m1: [2 x 2], m2: [4 x 4]

my current code:

class NeuralNet2(nn.Module):
    def __init__(self):
        super(NeuralNet2, self).__init__()

        self.input = nn.Linear(2, 40)
        self.hiddenLeft = nn.Linear(40, 2)
        self.hiddenRight = nn.Linear(40, 2)
        self.out = nn.Linear(4, 4)

    def forward(self, x):
        x = self.input(x)
        xLeft, xRight = torch.sigmoid(self.hiddenLeft(x)), torch.sigmoid(self.hiddenRight(x))
        x = torch.cat((xLeft, xRight))
        x = self.out(x)

        return x

I don't get why there is a size mismatch? Is there an alternative way to implement non-fully-connected layers in pytorch?


title: How to speed up the "ImageFolder" for ImageNet

body:
I am in an university, and all the file system are in a remote system, wherever I log in with my account, I could aways access my home directory. even though I log into the GPU servers through SSH command. This is the condition where I employ the GPU servers to read data.

Currently, I use the PyTorch to train ResNet from scratch on ImageNet, my codes only use all the GPUs in the same computer, I found that the "torchvision.datasets.ImageFolder" will take almost two hours.

Would you please provide some experiences in how to speed up "torchvision.datasets.ImageFolder"? Thanks very much.


title: Is there a tensor.item() equivalent for a tensor containing a list in pytorch?

body:
In pytorch, if I define a one-element tensor as follows:

>>> import torch
>>> target1 = torch.tensor([5])

I'm able to pull out the value of its one element like this:

>>> target1.item()
5

What I would like to know is if when my tensor is defined as:

target2 = torch.tensor([[5], [5], [5], [5]])

Is there some way (similar or not to .item() above) to pull out all of its entries into a list like:

>>> target2.(something)
[5, 5, 5, 5]

I can't seem to find any function in the documentation that supports an operation like this.


title: Pytorch BiLSTM POS Tagging Issue: RuntimeError: input.size(-1) must be equal to input_size. Expected 6, got 12

body:
I have a nlp dataset, and according to the Pytorch official tutorial, I change the dataset to the word_to_idx and tag_to_idx, like:

word_to_idx =  {'I': 0, 'have': 1, 'used': 2, 'transfers': 3, 'on': 4, 'three': 5, 'occasions': 6, 'now': 7, 'and': 8, 'each': 9, 'time': 10}
tag_to_idx =  {'PRON': 0, 'VERB': 1, 'NOUN': 2, 'ADP': 3, 'NUM': 4, 'ADV': 5, 'CONJ': 6, 'DET': 7, 'ADJ': 8, 'PRT': 9, '.': 10, 'X': 11}

I want to complete the POS-Tagging task with BiLSTM. Here is my BiLSTM code:

class LSTMTagger(nn.Module):

    def __init__(self, embedding_dim, hidden_dim, vocab_size, tagset_size):
        super(LSTMTagger, self).__init__()
        self.hidden_dim = hidden_dim
        self.word_embeddings = nn.Embedding(vocab_size, tagset_size)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, bidirectional=True)

        # The linear layer that maps from hidden state space to tag space
        self.hidden2tag = nn.Linear(in_features=hidden_dim * 2, out_features=tagset_size)

    def forward(self, sentence):
        embeds = self.word_embeddings(sentence)
        lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
        tag_space = self.hidden2tag(lstm_out.view(len(sentence), -1))

        # tag_scores = F.softmax(tag_space, dim=1)
        tag_scores = F.log_softmax(tag_space, dim=1)
        return tag_scores

Then I run the training code in Pycharm, like:

EMBEDDING_DIM = 6
HIDDEN_DIM = 6
NUM_EPOCHS = 3

model = LSTMTagger(embedding_dim=EMBEDDING_DIM,
                   hidden_dim=HIDDEN_DIM,
                   vocab_size=len(word_to_idx),
                   tagset_size=len(tag_to_idx))

loss_function = nn.NLLLoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

# See what the scores are before training
with torch.no_grad():
    inputs = prepare_sequence(training_data[0][0], word_to_idx)
    tag_scores = model(inputs)
    print(tag_scores)
    print(tag_scores.size())

However, it shows error with line tag_scores = model(inputs) and line lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1)).
The error is:

Traceback (most recent call last):
  line 140, in <module>
    tag_scores = model(inputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  line 115, in forward
    lstm_out, _ = self.lstm(embeds.view(len(sentence), 1, -1))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 559, in forward
    return self.forward_tensor(input, hx)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 539, in forward_tensor
    output, hidden = self.forward_impl(input, hx, batch_sizes, max_batch_size, sorted_indices)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 519, in forward_impl
    self.check_forward_args(input, hx, batch_sizes)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 490, in check_forward_args
    self.check_input(input, batch_sizes)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 153, in check_input
    self.input_size, input.size(-1)))
RuntimeError: input.size(-1) must be equal to input_size. Expected 6, got 12

I don't know how to debug with it. Could somebody help me fix this issue? Thanks in advance!


title: How can I install torch and torchvision with pip and python 3.8.3 on windows in an virtual environment?

body:
I created a virtual environment for my project and I'm trying to install torch via pip.
pip install torch===1.5.1 torchvision===0.6.1 -f https://download.pytorch.org/whl/torch_stable.html

This gives me the error -
ERROR: Could not find a version that satisfies the requirement torch===1.5.1 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch===1.5.1

Some solutions recommend the use of conda, is it necessary? Can I not do it through just pip


title: Can neuroevolution of augmenting topologies (NEAT) neural networks be built in TensorFlow?

body: I am making a machine learning program for time series data analysis and using NEAT could help the work. I started to learn TensorFlow not long ago but it seems that the computational graphs in TensorFlow are usually fixed. Is there tools in TensorFlow to help build a dynamically evolving neural network? Or something like Pytorch would be a better alternative? Thanks.


title: How to fix 'RuntimeError: Address already in use' in PyTorch?

body:
I am trying to run a distributive application with PyTorch distributive trainer. I thought I would first try the example they have, found here. I set up two AWS EC2 instances and configured them according to the description in the link, but when I try to run the code I get two different errors: in the first terminal window for node0 I get the error message: RuntimeError: Address already in use

Under the other three windows I get the same error message:

  RuntimeError: NCCL error in:
  /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:272, unhandled system
  error

I followed the code in the link, and terminated the instances an redid but it didn't help/

This is using python 3.6 with the nightly build Cuda 9.0. I tried changing the MASTER_ADDR to the ip for node0 on both nodes, as well as using the same MASTER_PORT (which is an available, unused port). However I still get the same error message.

After running this, my goal is to the adjust this StyleGan implementation so that I can train it across multiple GPUs in two different nodes.


title: Error " 'Softmax' object has no attribute 'log_softmax' " while training Neural Network with PyTorch

body:
I am working on a classifier for MNIST dataset. When I am running the code below, I am getting the error " 'Softmax' object has no attribute 'log_softmax' " at line loss = loss_function(output, y). I have not managed to find a solution to the problem. I will appreciate if you can advise on how the issue can be resolved. Thank you.
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset, TensorDataset
import torchvision
import torchvision.transforms as transforms
import numpy as np

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")
batch_size = 512

# Image transformations of Torchvision will convert to the images to tensor and normalise with mean and standard deviation
transformer = transforms.Compose([transforms.ToTensor(), transforms.Normalize(mean=(0.1307,), std=(0.3081,))])

data_train = DataLoader(torchvision.datasets.MNIST('Data/data/mnist', download=True, train=True, transform=transformer),
                batch_size=batch_size, drop_last=False, shuffle=True)
data_test = DataLoader(torchvision.datasets.MNIST('Data/data/mnist', download=True, train=False, transform=transformer),
                batch_size=batch_size, drop_last=False, shuffle=True)

class neural_nw(nn.Module):

    def __init__(self):
        super(neural_nw, self).__init__()
        self.fc1 = nn.Linear(784, 128, True)
        self.fc2 = nn.Linear(128, 128, True)
        self.fc3 = nn.Linear(128, 10, True)

    def forward(self, x):
        output = torch.sigmoid(self.fc1(x))
        output = torch.sigmoid(self.fc2(output))
        output = nn.Softmax(self.fc3(output))
        return output

MLP = neural_nw()

loss_function = nn.CrossEntropyLoss()
optimiser = optim.Adam(MLP.parameters(), lr = 0.01)

Epochs = 50

for epoch in range(Epochs):
    for X, y in data_train:
        X = X.view(X.shape[0], -1)
        optimiser.zero_grad()
        output = MLP.forward(X)
        loss = loss_function(output, y)
        loss.backward()
        optimiser.step()


title: Fastai on google colab

body:
Two days back I ran my model using fastai 0.7.0 on google colab.
And for two days I got busy and now if I am trying to run it , its throwing me an error, on the execution of the line
*"from fastai.transforms import ’ .
the error is AttributeError: module ‘torch’ has no attribute ‘float32’.


title: In pytorch, how to fill a tensor with another tensor?

body:
I'm looking for a way to expand the size of an image by adding 0 values to the right & lower edges of it. My initial plan is to use nn.padding to add the edge, until I encounter this error:

  File "/home/shared/virtualenv/dl-torch/lib/python3.7/site-packages/torch/nn/functional.py", line 2796, in pad
    assert len(pad) % 2 == 0, 'Padding length must be divisible by 2'
AssertionError: Padding length must be divisible by 2

It appears that torch tries to pad the image from both side! Is there an easy way to override this and fill the tensor into the upper-left side of another image?


title: How to move a variable from GPU to CPU in c++ pytorch frontend API?

body:
I'm writing an inference code to load a converted pytorch model (a tagging model from imagenet) in C++. I used c++ pytorch frontend API. My code works correctly on CPU but does not work on GPU.
The problem is that when I want to print the final results, I get the Segmentation fault (core dumped) error.
I have to transfer the "top_scores_a" and "top_idx_a" variables to the CPU but I don't know how to do so.

I load the model and the input image on GPU.
The error is occurred in the following part:

for (int i = 0; i < 2; ++i)
    {
        // int idx = top_idxs_a[i];
        std::cout << "top-" << i+1 << " label: ";
        // std::cout << labels[idx] << ", score: " << top_scores_a[i] << std::endl;
    }

The complete code is available here:

#include "torch/script.h"
#include <torch/script.h>
#include <torch/torch.h>
#include <ATen/Tensor.h>
#include <opencv2/opencv.hpp>
#include <opencv2/imgproc/imgproc.hpp>
#include <time.h>

#include <iostream>
#include <memory>
#include <cuda.h>
#include <cuda_runtime_api.h>

using namespace std;

// __global__
int main(int argc, const char* argv[]) {

    //// asign gpu
    torch::Device device(torch::kCPU);
    clock_t tStart = clock();

    //// check cuda visibility
    if (torch::cuda::is_available())
    {
        std::cout << "CUDA is available! Run on GPU." << std::endl;
        device = torch::kCUDA;

    }

    if (argc != 4) {
        cout << "ptcpp path/to/scripts/model.pt path/to/image.jpg path/to/label.txt\n";
        return -1;
    }

    cout << "Will load from " << argv[1] << endl;
    shared_ptr<torch::jit::script::Module> module = torch::jit::load(argv[1]);
    module->to(device); // on gpu

    if (module == nullptr) {
        cerr << "model load error from " << argv[1] << endl;
    }
    cout << "Model load ok.\n";

    // load image and transform
    cv::Mat image;
    image = cv::imread(argv[2], 1);

    cv::Mat image_rgb;
    cv::cvtColor(image, image_rgb, CV_BGR2RGB);

    cv::Mat image_resized;
    cv::resize(image_rgb, image_resized, cv::Size(224, 224));

    cv::Mat image_resized_float;
    image_resized.convertTo(image_resized_float, CV_32F, 1.0/255);

    auto img_tensor = torch::CPU(torch::kFloat32).tensorFromBlob(image_resized_float.data, {1, 224, 224, 3}).to(device); // work correctly

    cout << "img tensor loaded..\n";
    img_tensor = img_tensor.permute({0, 3, 1, 2});
    img_tensor[0][0] = img_tensor[0][0].sub(0.485).div(0.229);
    img_tensor[0][1] = img_tensor[0][1].sub(0.456).div(0.224);
    img_tensor[0][2] = img_tensor[0][2].sub(0.406).div(0.225);

    auto img_var = torch::autograd::make_variable(img_tensor, false);

    vector<torch::jit::IValue> inputs;
    inputs.push_back(img_var);
    torch::Tensor out_tensor = module->forward(inputs).toTensor();

    // load labels
    vector<string> labels;
    ifstream ins;
    ins.open(argv[3]);
    string line;
    while (getline(ins, line))
    {
        labels.push_back(line);
    }

    std::tuple<torch::Tensor,torch::Tensor> result = out_tensor.sort(-1, true); //-1
    torch::Tensor top_scores = std::get<0>(result)[0];
    torch::Tensor top_idxs = std::get<1>(result)[0].toType(torch::kInt32);

    auto top_scores_a = top_scores.accessor<float,1>();
    auto top_idxs_a = top_idxs.accessor<int,1>();

    for (int i = 0; i < 2; ++i)
    {
        int idx = top_idxs_a[i];
        std::cout << "top-" << i+1 << " label: ";
        std::cout << labels[idx] << ", score: " << top_scores_a[i] << std::endl;
    }

    float tend = clock();
    printf("Time taken: %.2fs\n", (double)(tend - tStart)/CLOCKS_PER_SEC);

    return 0;
}


title: PyTorch error with expected object of scaler float but instead got object of scaler Long

body:
So I am following a course on Udemy (artificial intelligence a-z)and I have run into an error where it says the following:
return map(lambda x: Variable(torch.cat(x, 0)), samples)
 RuntimeError: Expected object of scalar type Float but got scalar type Long for sequence element 1 in sequence argument at position #1 'tensors'

The element also seems to change each time so I don't know where the problem could be
can someone please help me solve this?

I have already tried to modify the previous line (samples) to see if that's what I need to convert to Long but it doesn't work, unless in doing it wrong

Included lines 43-45 of ai.py

def sample(self, batch_size):
    samples = zip(*random.sample(self.memory, batch_size))
    return map(lambda x: Variable(torch.cat(x, 0)), samples)

Included lines 74-87 of ai.py (whats calling the sample method)

def update(self, reward, new_signal):
    new_state = torch.Tensor(new_signal).float().unsqueeze(0)
    self.memory.push((self.last_state, new_state,
    torch.LongTensor([int(self.last_action)]),
    torch.tensor([self.last_reward])))
    action = self.select_action(new_state)
    if len(self.memory.memory) > 100:
        batch_state, batch_next_state, batch_reward, batch_action =
        self.memory.sample(100)
    self.learn(batch_state, batch_next_state, batch_reward, batch_action)
    self.last_action = action
    self.last_state = new_state
    self.last_reward = reward
    self.reward_window.append(reward)
    if len(self.reward_window) > 1000:
        del self.reward_window[0]
    return action

To be honest, I expect the rest of the program to run, but it always seems to crash here, sorry I can't be more specific on what I expect, that's because I don't know it's my first time making a proper AI


title: Pytorch: saving model or state_dict gives different on-disk space occupation

body:
I was playing around with the function torch.save and I noticed something curious, let's say i load a model from torchvision repository:

model = torchvision.models.mobilenet_v2()

if i save the model in this way:

torch.save(model,'model.pth')

I get a 14MB file, while if i do:

torch.save(model.state_dict(),'state_dict.pth')

The file size blow to ~500MB. Since i didn't find any reference on this behaviour I was wondering what does cause the increment in size. Is it something related to compression? does saving the whole state_dict stores extra stuff like uninitialized gradients?

P.S. the same happens for other model like vgg16