title: How to swap from gpu to a cpu only?

body:
Hi I was wondering how can run a machine learning code onto my CPU instead of a GPU?

I have tried making the GPU false on the setting file but it hasn't been able to fix it.

### global settings

GPU = False                                                                 # running on GPU is highly suggested
CLEAN = False                                                                # set to "True" if you want to clean the temporary large files after generating result
APP = "classification"                                                       # Do not change! mode choide: "classification", "imagecap", "vqa". Currently "imagecap" and "vqa" are not supported.
CATAGORIES = ["object", "part"]                                              # Do not change! concept categories that are chosen to detect: "object", "part", "scene", "material", "texture", "color"
map_location='cpu'

CAM_THRESHOLD = 0.5                                                          # the threshold used for CAM visualization
FONT_PATH = "components/font.ttc"                                            # font file path
FONT_SIZE = 26                                                               # font size
SEG_RESOLUTION = 7                                                           # the resolution of cam map
BASIS_NUM = 7

Traceback (most recent call last):
  File "test.py", line 22, in <module>
    model = loadmodel()
  File "/home/joshuayun/Desktop/IBD/loader/model_loader.py", line 44, in loadmodel
    checkpoint = torch.load(settings.MODEL_FILE)
  File "/home/joshuayun/.local/lib/python3.6/site-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/home/joshuayun/.local/lib/python3.6/site-packages/torch/serialization.py", line 574, in _load
    result = unpickler.load()
  File "/home/joshuayun/.local/lib/python3.6/site-packages/torch/serialization.py", line 537, in persistent_load
    deserialized_objects[root_key] = restore_location(obj, location)
  File "/home/joshuayun/.local/lib/python3.6/site-packages/torch/serialization.py", line 119, in default_restore_location
    result = fn(storage, location)
  File "/home/joshuayun/.local/lib/python3.6/site-packages/torch/serialization.py", line 95, in _cuda_deserialize
    device = validate_cuda_device(location)
  File "/home/joshuayun/.local/lib/python3.6/site-packages/torch/serialization.py", line 79, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location='cpu' to map your storages to the CPU.


title: Facing "Torch not complied with CUDA enabled" when using DeOldify

body:
I have been trying to use DeOldify and I'm following the steps from here.
Manual

Im facing two issues I'm unable to find the Runtime settings and running from fasterai.visualize import * yields the following error.

-----------------------------------------------------------------------    ----
AssertionError                            Traceback (most recent call last)
<ipython-input-24-ec3d448c437f> in <module>
----> 1 from fasterai.visualize import *

~/DeOldify/fasterai/visualize.py in <module>
  5 from matplotlib.backends.backend_agg import FigureCanvasAgg as FigureCanvas
  6 from .filters import IFilter, MasterFilter, ColorizerFilter
----> 7 from .generators import gen_inference_deep, gen_inference_wide
  8 from tensorboardX import SummaryWriter
  9 from scipy import misc

~/DeOldify/fasterai/generators.py in <module>
 14       return learn
 15
---> 16 def gen_learner_wide(data:ImageDataBunch, gen_loss=FeatureLoss(), arch=models.resnet101, nf_factor:int=2)->Learner:
 17     return unet_learner_wide(data, arch=arch, wd=1e-3, blur=True, norm_type=NormType.Spectral,
 18                         self_attention=True, y_range=(-3.,3.), loss_func=gen_loss, nf_factor=nf_factor)

~/DeOldify/fasterai/loss.py in __init__(self, layer_wgts)
 10         super().__init__()
 11
---> 12         self.m_feat = models.vgg16_bn(True).features.cuda().eval()
 13         requires_grad(self.m_feat, False)
 14         blocks = [i-1 for i,o in enumerate(children(self.m_feat)) if isinstance(o,nn.MaxPool2d)]

/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in cuda(self, device)
263             Module: self
264         """
--> 265         return self._apply(lambda t: t.cuda(device))
266
267     def cpu(self):

/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _apply(self, fn)
191     def _apply(self, fn):
192         for module in self.children():
--> 193             module._apply(fn)
194
195         for param in self._parameters.values():

/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _apply(self, fn)
197                 # Tensors stored in modules are graph leaves, and we don't
198                 # want to create copy nodes, so we have to unpack the data.
--> 199                 param.data = fn(param.data)
200                 if param._grad is not None:
201                     param._grad.data = fn(param._grad.data)

/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in <lambda>(t)
263             Module: self
264         """
--> 265         return self._apply(lambda t: t.cuda(device))
266
267     def cpu(self):

/anaconda3/lib/python3.7/site-packages/torch/cuda/__init__.py in _lazy_init()
160         raise RuntimeError(
161             "Cannot re-initialize CUDA in forked subprocess. " + msg)
--> 162     _check_driver()
163     torch._C._cuda_init()
164     _cudart = _load_cudart()

/anaconda3/lib/python3.7/site-packages/torch/cuda/__init__.py in _check_driver()
 73 def _check_driver():
 74     if not hasattr(torch._C, '_cuda_isDriverSufficient'):
---> 75         raise AssertionError("Torch not compiled with CUDA enabled")
 76     if not torch._C._cuda_isDriverSufficient():
 77         if torch._C._cuda_getDriverVersion() == 0:

AssertionError: Torch not compiled with CUDA enabled


title: Object Detection inference using multi-gpu & multi threading, Pytorch

body:
I am trying to detect objects in a video using multiple GPUs. I want to distribute frames to GPUs for inference to increase total process time. I succeeded running inference in single gpu, but failed to run on multiple GPUs.

I thought dividing frames per number of gpus and processing inference would decrease the time. If there is another way I can decrease running time, I would be glad to receive suggestions.

I am using pre-trained model provided by Pytorch. What I tried is as follows:
1. I read the video and divide frames by number of gpus I have(currently two NVIDIA GeForce GTX 1080 Ti)
2. Then, I distributed frames to gpus and process object detection inference.
(Later I planned to use multi-threads to dynamically distribute frames per number of gpus, but currently I made it static)

The same method I tried worked well in Tensorflow using with tf.device() and I am trying to make it possible in Pytorch as well.

pytorch_multithread.py

...
    def detection_gpu(frame_list, device_name, device, detect, model):
        model.to(device)
        model.eval()

        for frame in frame_list:
            start = time.time()
            detect.bounding_box_rcnn(frame, model=model)
            end = time.time()

            cv2.putText(frame, '{:.2f}ms'.format((end - start) * 1000), (40, 40), cv2.FONT_HERSHEY_SIMPLEX, 0.75,
                        (255, 0, 0),
                        2)

            cv2.imshow(str(device_name), frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

    def main():
        args = arg_parse()

        VIDEO_PATH = args.video

        print("Loading network.....")
        model = models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
        print("Network successfully loaded")

        num_gpus = torch.cuda.device_count()
        if torch.cuda.is_available() and num_gpus > 1:
            device = ["cuda:{}".format(i) for i in range(num_gpus)]
        elif num_gpus == 1:
            device = "cuda"
        else:
            device = "cpu"
        # class names ex) person, car, truck, and etc.
        PATH_TO_LABELS = "labels/mscoco_labels.names"

        # load detection class, default confidence threshold is 0.5
        if num_gpus>1:
            detect = [DetectBoxes(PATH_TO_LABELS, device[i], conf_threshold=args.confidence) for i in range(num_gpus)]
        else:
            detect = [DetectBoxes(PATH_TO_LABELS, device, conf_threshold=args.confidence) for i in range(1)]

        cap = cv2.VideoCapture(VIDEO_PATH)

        # find number of gpus that is available
        frame_length = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

        # TODO: CPU환경 고려하기
        # divide frames of video by number of gpus
        div = frame_length // num_gpus
        divide_point = [i for i in range(frame_length) if i != 0 and i % div == 0]
        divide_point.pop()

        frame_list = []
        fragments = []
        count = 0
        while cap.isOpened():
            hasFrame, frame = cap.read()
            if not hasFrame:
                frame_list.append(fragments)
                break
            if count in divide_point:
                frame_list.append(fragments)
                fragments = []
            fragments.append(frame)
            count += 1
        cap.release()

        detection_gpu(frame_list[0], 0, device[0], detect[0], model)
        detection_gpu(frame_list[1], 1, device[1], detect[1], model)
        # Process object detection using threading
        # thread_detection = [ThreadWithReturnValue(target=detection_gpu,
        #                                           args=(frame_list[i], i, detect, model))
        #                     for i in range(num_gpus)]
        #
        #
        # final_list = []
        # # Begin operating threads
        # for th in thread_detection:
        #     th.start()
        #
        # # Once tasks are completed get return value (frames) and put to new list
        # for th in thread_detection:
        #     final_list.extend(th.join())
        cv2.destroyAllWindows()

detection_boxes_pytorch.py

    def bounding_box_rcnn(self, frame, model):
            print(self.device)
            # Image is converted to image Tensor
            transform = transforms.Compose([transforms.ToTensor()])
            img = transform(frame).to(self.device)
            with torch.no_grad():
                # The image is passed through model to get predictions
                pred = model([img])

            # classes, bounding boxes, confidence scores are gained
            # only classes and bounding boxes > confThershold are passed to draw_boxes
            pred_class = [self.classes[i] for i in list(pred[0]['labels'].cpu().clone().numpy())]
            pred_boxes = [[(i[0], i[1]), (i[2], i[3])] for i in list(pred[0]['boxes'].detach().cpu().clone().numpy())]
            pred_score = list(pred[0]['scores'].detach().cpu().clone().numpy())
            pred_t = [pred_score.index(x) for x in pred_score if x > self.confThreshold][-1]
            pred_colors = [i for i in list(pred[0]['labels'].cpu().clone().numpy())]
            pred_boxes = pred_boxes[:pred_t + 1]
            pred_class = pred_class[:pred_t + 1]

            for i in range(len(pred_boxes)):
                left = int(pred_boxes[i][0][0])
                top = int(pred_boxes[i][0][1])
                right = int(pred_boxes[i][1][0])
                bottom = int(pred_boxes[i][1][1])

                color = STANDARD_COLORS[pred_colors[i] % len(STANDARD_COLORS)]

                self.draw_boxes(frame, pred_class[i], pred_score[i], left, top, right, bottom, color)

The error I get is as follows:

    Traceback (most recent call last):
      File "C:/Users/username/Desktop/Object_Detection_Video_AllInOne/pytorch_multithread.py", line 133, in <module>
        main()
      File "C:/Users/username/Desktop/Object_Detection_Video_AllInOne/pytorch_multithread.py", line 113, in main
        detection_gpu(frame_list[1], 1, device[1], detect[1], model)
      File "C:/Users/username/Desktop/Object_Detection_Video_AllInOne/pytorch_multithread.py", line 39, in detection_gpu
        detect.bounding_box_rcnn(frame, model=model)
      File "C:\Users\username\Desktop\Object_Detection_Video_AllInOne\p_utils\detection_boxes_pytorch.py", line 64, in bounding_box_rcnn
        pred = model([img])
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\models\detection\generalized_rcnn.py", line 51, in forward
        proposals, proposal_losses = self.rpn(images, features, targets)
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 493, in __call__
        result = self.forward(*input, **kwargs)
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\models\detection\rpn.py", line 409, in forward
        proposals = self.box_coder.decode(pred_bbox_deltas.detach(), anchors)
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\models\detection\_utils.py", line 168, in decode
        rel_codes.reshape(sum(boxes_per_image), -1), concat_boxes
      File "C:\Users\username\AppData\Local\Programs\Python\Python37\lib\site-packages\torchvision\models\detection\_utils.py", line 199, in decode_single
        pred_ctr_x = dx * widths[:, None] + ctr_x[:, None]
    RuntimeError: binary_op(): expected both inputs to be on same device, but input a is on cuda:1 and input b is on cuda:0


title: Convolutional encoder error - 'RuntimeError: input and target shapes do not match'

body:
In below code three images are created, saved and a convolutional auto-encoder attempts to encode them to a lower dimensional representation.

%reset -f

import torch.utils.data as data_utils
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from matplotlib import pyplot as plt
from sklearn import metrics
import datetime
from sklearn.preprocessing import MultiLabelBinarizer
import seaborn as sns
sns.set_style("darkgrid")
from ast import literal_eval
import numpy as np
from sklearn.preprocessing import scale
import seaborn as sns
sns.set_style("darkgrid")
import torch
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable
from os import listdir
import cv2
import torch.nn.functional as F
import numpy as np
from numpy.polynomial.polynomial import polyfit
import matplotlib.pyplot as plt

number_channels = 3

%matplotlib inline

x = np.arange(10)
m = 1
b = 2
y = x * x
plt.plot(x, y)
plt.axis('off')
plt.savefig('1-increasing.jpg')

x = np.arange(10)
m = 0.01
b = 2
y = x * x * x
plt.plot(x, y)
plt.axis('off')
plt.savefig('2-increasing.jpg')

x = np.arange(10)
m = 0
b = 2
y = (m*x)+b
plt.plot(x, y)
plt.axis('off')
plt.savefig('constant.jpg')

batch_size_value = 2

train_image = []

train_image.append(cv2.imread('1-increasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('2-increasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('decreasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('constant.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))

data_loader = data_utils.DataLoader(train_image, batch_size=batch_size_value, shuffle=False,drop_last=True)

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.utils import save_image
from torchvision.datasets import MNIST
import os

def to_img(x):
    x = 0.5 * (x + 1)
    x = x.clamp(0, 1)
    x = x.view(x.size(0), 1, 28, 28)
    return x

num_epochs = 100
# batch_size = 128
batch_size = 2

learning_rate = 1e-3
dataloader = data_loader

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
#         torch.nn.Conv2d(in_channels, out_channels, kernel_size, stride=1, padding=0, dilation=1, groups=1, bias=True)
        self.encoder = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=16, kernel_size=3, stride=3, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2),  # b, 16, 5, 5
            nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
            nn.ReLU(True),
            nn.MaxPool2d(3, stride=1)  # b, 8, 2, 2
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(8, 16, 2, stride=1),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 8, 3, stride=3, padding=1),  # b, 8, 15, 15
            nn.ReLU(True),
            nn.ConvTranspose2d(8, 3, 2, stride=2, padding=1),  # b, 1, 28, 28
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = autoencoder().cuda().double()

criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,
                             weight_decay=1e-5)

for epoch in range(num_epochs):
    for data in dataloader:
        img, _ = data
        img = img.double()
        img = Variable(img).cuda()
        img = img.unsqueeze_(0)

        # ===================forward=====================
        output = model(img)
        loss = criterion(output, img)
        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    # ===================log=================to_img=======
    print('epoch [{}/{}], loss:{:.4f}'
          .format(epoch+1, num_epochs, loss.data[0]))

torch.save(model.state_dict(), './conv_autoencoder.pth')

But error is returned :

RuntimeError: input and target shapes do not match: input [1 x 3 x 132 x 204], target [1 x 3 x 288 x 432] at /pytorch/aten/src/THCUNN/generic/MSECriterion.cu:15

The shape of the images are (3, 288, 432) . How to change the configuration of the model to allow [1 x 3 x 288 x 432] instead of [1 x 3 x 132 x 204] ?

Update:

I changed

nn.ConvTranspose2d(8, 3, 2, stride=2, padding=1)

to :

nn.ConvTranspose2d(8, 3, 3, stride=4, padding=2)

Which results in closer dimensional output but not exact so error is now :

RuntimeError: input and target shapes do not match: input [1 x 3 x 263 x 407], target [1 x 3 x 288 x 432] at /pytorch/aten/src/THCUNN/generic/MSECriterion.cu:12

How should the output decoder dimensions be calculated in order to produce the correct dimension ?


title: How can I do to evaluate mean and std for a dataset?

body:
I am using pytorch and the dataset fashion MNIST but I do not know how can I do to evaluate the mean and the std for this dataset. Here is my code :

import torch
from torchvision import datasets, transforms
import torch.nn.functional as F

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((mean), (std))])
batch_size = 32
train_loader = torch.utils.data.DataLoader(datasets.MNIST(
'../data', train=True, download=True, transform=transform)
, batch_size=batch_size, shuffle=True)

Could you help me please ?

Thank you very much !


title: How to use a numpy function as the loss function in PyTorch and avoid getting errors during run time?

body:
For my task, I do not need to compute gradients. I am simply replacing nn.L1Loss with a numpy function (corrcoef) in my loss evaluation but I get the following error:

RuntimeError: Can’t call numpy() on Variable that requires grad. Use var.detach().numpy() instead.

I couldn’t figure out how exactly I should detach the graph (I tried torch.Tensor.detach(np.corrcoef(x, y)) but I still get the same error. I eventually wrapped everything using with torch.no_grad as follow:

with torch.no_grad():
    predFeats = self.forward(x)
    targetFeats = self.forward(target)
    loss = torch.from_numpy(np.corrcoef(predFeats.cpu().numpy().astype(np.float32), targetFeats.cpu().numpy().astype(np.float32))[1][1])

But this time I get the following error:

TypeError: expected np.ndarray (got numpy.float64)

I wonder, what am I doing wrong?


title: Proper Usage of PyTorch's non_blocking=True for Data Prefetching

body:
I am looking into prefetching data into the GPU from the CPU when the model is being trained on the GPU. Overlapping CPU-to-GPU data transfer with GPU model training appears to require both

Transferring data to GPU using data = data.cuda(non_blocking=True)
Pin data to CPU memory using train_loader = DataLoader(..., pin_memory=True)

However, I cannot understand how non-blocking transfer is being performed in this official PyTorch example, specifically this code block:
for i, (images, target) in enumerate(train_loader):
        # measure data loading time
        data_time.update(time.time() - end)

        if args.gpu is not None:
            images = images.cuda(args.gpu, non_blocking=True)
        if torch.cuda.is_available():
            target = target.cuda(args.gpu, non_blocking=True)

        # compute output
        output = model(images)
        loss = criterion(output, target)

Won't images.cuda(non_blocking=True) and target.cuda(non_blocking=True) have to be completed before output = model(images) is executed. Since this is a synchronization point, images must be first fully transferred to the CUDA device, so the data transfer steps are effectively no longer non-blocking.
Since output = model(images) is blocking, images.cuda() and target.cuda() in the next i iteration of the for loop will not occur until the model output is computed, meaning no prefetching in the next loop iteration.
If this is correct, what is the correct way to perform data prefetching to the GPU?


title: Load a plain text file into PyTorch

body: I have two separate files, one is a text file, with each line being a single text. The other file contains the class label of that corresponding line. How do I load this into PyTorch and carry out further tokenization, embedding, etc?


title: How make custom dataset for classification task when the class is folder name using Pytorch?

body:
The problem is dataloader is returning the wrong class for correspond image?
for example if I print the class_to_idx from the train_loader, when batch size is 1I was expecting to get one class per batch, but currently it’s returning all the classes which is 15 classes per image.

In this case, the classes are folder class (all images exist in one folder belongs to one class)

snippet is here:(this is a function to return the class from the folder name dir)

import os
def find_classes(dir):   # Finds the class folders in a dataset, dir (string): Root directory path.

        classes = [d.name for d in os.scandir(dir) if d.is_dir()]
        classes.sort()
        class_to_idx = {classes[i]: i for i in range(len(classes))}
        return classes, class_to_idx

here is the main snippet for create a custom dataset and dataloder
def main():

class CustomDataset(Dataset):

    def __init__(self, image_paths, classes, class_to_id):

        self.image_paths = image_paths
        self.transforms = transforms.ToTensor()
        classes, class_to_id = find_classes('D:/Neda/Echo_View_Classification/avi_images/')
        self.classes = classes
        self.class_to_idx = class_to_idx

    def __getitem__(self, index):

        image = Image.open(self.image_paths[index])
        t_image = image.convert('L')
        t_image = self.transforms(t_image)

        class_to_idx = self.class_to_idx

        return t_image, class_to_idx, self.image_paths[index]

    def __len__(self):

        return len(self.image_paths)

folder_data = glob.glob("D:\\Neda\\Echo_View_Classification\\avi_images\\*\\*.png") # no augmnetation
#numpy.savetxt('distribution_class.csv', numpy.c_[folder_data], fmt=['%s'], comments='', delimiter = ",")

 #split these path using a certain percentage
len_data = len(folder_data)
print("count of dataset: ", len_data)

split_1 = int(0.6 * len(folder_data))
split_2 = int(0.8 * len(folder_data))

folder_data.sort()

train_image_paths = folder_data[:split_1]
print("count of train images is: ", len(train_image_paths))
numpy.savetxt('im_training_path_1.csv', numpy.c_[train_image_paths], fmt=['%s'], comments='', delimiter = ",")

valid_image_paths = folder_data[split_1:split_2]
print("count of validation image is: ", len(valid_image_paths))
numpy.savetxt('im_valid_path_1.csv', numpy.c_[valid_image_paths], fmt=['%s'], comments='', delimiter = ",")

test_image_paths = folder_data[split_2:]
print("count of test images is: ", len(test_image_paths))
numpy.savetxt('im_testing_path_1.csv', numpy.c_[test_image_paths], fmt=['%s'], comments='', delimiter = ",")

classes = ['1_PLAX_1_PLAX_full',
  '1_PLAX_2_PLAX_valves',
  '1_PLAX_4_PLAX_TV',
  '2_PSAX_1_PSAX_AV',
  '2_PSAX_2_PSAX_LV',
  '3_Apical_1_MV_LA_IAS',
  '3_Apical_2_A2CH',
  '3_Apical_3_A3CH',
  '3_Apical_5_A5CH',
  '4_A4CH_1_A4CH_LV',
  '4_A4CH_2_A4CH_RV',
  '4_Subcostal_1_Subcostal_heart',
  '4_Subcostal_2_Subcostal_IVC',
  'root_5_Suprasternal',
  'root_6_OTHER']

class_to_idx = {'1_PLAX_1_PLAX_full': 0,
  '1_PLAX_2_PLAX_valves': 1,
  '1_PLAX_4_PLAX_TV': 2,
  '2_PSAX_1_PSAX_AV': 3,
  '2_PSAX_2_PSAX_LV': 4,
  '3_Apical_1_MV_LA_IAS': 5,
  '3_Apical_2_A2CH': 6,
  '3_Apical_3_A3CH': 7,
  '3_Apical_5_A5CH': 8,
  '4_A4CH_1_A4CH_LV': 9,
  '4_A4CH_2_A4CH_RV': 10,
  '4_Subcostal_1_Subcostal_heart': 11,
  '4_Subcostal_2_Subcostal_IVC': 12,
  'root_5_Suprasternal': 13,
  'root_6_OTHER': 14}

train_dataset = CustomDataset(train_image_paths, class_to_idx, classes)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=1, shuffle=False, num_workers=0)

valid_dataset = CustomDataset(valid_image_paths,  class_to_idx, classes)
valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=1, shuffle=False, num_workers=0)

test_dataset = CustomDataset(test_image_paths, class_to_idx, classes)
test_loader = torch.utils.data.DataLoader(test_dataset, batch_size=1, shuffle=False, num_workers=0)

dataLoaders = {
        'train': train_loader,
        'valid': valid_loader,
         'test': test_loader,
        }


title: How to create upper triangular matrix in Pytorch?

body: Simple question, but is there a native way to create an upper triangular matrix from an existing matrix in Pytorch? I was thinking of using a mask, but even that requires creating the upper triangular matrix.


title: How to train BART for text summarization using custom datset?

body:
I have prepared a custom dataset for training my own custom model for text summarization. I wish to use BART as it is the state of art now. I am using Transformer Library of HuggingFace using pytorch.

But I don't know how to train a model on my own.I tried running run_train.sh on the following link:
https://github.com/huggingface/transformers/tree/master/examples/summarization/bart

I have loaded the CNN data into Google Colab. Still it is showing errors.

Can someone help me with how to do custom text summarization using transfer learning?
Any other models are also helpful.


title: PyTorch LSTM has nan for MSELoss

body:
My model is:

class BaselineModel(nn.Module):
    def __init__(self, feature_dim=5, hidden_size=5, num_layers=2, batch_size=32):
        super(BaselineModel, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size=feature_dim,
                            hidden_size=hidden_size, num_layers=num_layers)

    def forward(self, x, hidden):
        lstm_out, hidden = self.lstm(x, hidden)
        return lstm_out, hidden

    def init_hidden(self, batch_size):
        hidden = Variable(next(self.parameters()).data.new(
            self.num_layers, batch_size, self.hidden_size))
        cell = Variable(next(self.parameters()).data.new(
            self.num_layers, batch_size, self.hidden_size))
        return (hidden, cell)

Training looks like:

train_loader = torch.utils.data.DataLoader(
    train_set, batch_size=BATCH_SIZE, shuffle=True, **params)

model = BaselineModel(batch_size=BATCH_SIZE)
optimizer = optim.Adam(model.parameters(), lr=0.01, weight_decay=0.0001)
loss_fn = torch.nn.MSELoss(reduction='sum')

for epoch in range(250):

    # hidden = (torch.zeros(2, 13, 5),
    #           torch.zeros(2, 13, 5))
    # model.hidden = hidden
    for i, data in enumerate(train_loader):
        hidden = model.init_hidden(13)
        inputs = data[0]
        outputs = data[1]

        print('inputs',  inputs.size())
        # print('outputs', outputs.size())

        # optimizer.zero_grad()
        model.zero_grad()

        # print('inputs', inputs)
        pred, hidden = model(inputs, hidden)

        loss = loss_fn(pred, outputs)

        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()

        print('Epoch: ', epoch, '\ti: ', i, '\tLoss: ', loss)

I have gradient clipping set already, which seems to be the recommended solution. But after even the first step, I get:

  Epoch:  0       i:  0   Loss:  tensor(nan, grad_fn=)


title: What exactly does the forward function output in Pytorch?

body:
This example is taken verbatim from the PyTorch Documentation. Now I do have some background on Deep Learning in general and know that it should be obvious that the forward call represents a forward pass, passing through different layers and finally reaching the end, with 10 outputs in this case, then you take the output of the forward pass and compute the loss using the loss function one defined. Now, I forgot what exactly the output from the forward() pass yields me in this scenario... I thought that the last layer in a Neural Network should be some sort of activation function like sigmoid() or softmax(), but I did not see these being defined anywhere, furthermore, when I was doing a project now, I found out that softmax() is called later on. So I just want to clarify what exactly is the outputs = net(inputs) giving me, from this link, it seems to me by default the output of a PyTorch model's forward pass is logits? Do help me to clear up some air, if my question is bad, please let me know and point me to the right direction :)
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

import torch.optim as optim

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        print(outputs)
        break
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')


title: LSTM model implementation in PyTorch

body: I was trying to implement CNN+LSTM model in PyTorch, but I have problem with LSTM part (I never used LSTM before). Could you write Many-to-one-LSTM model class (Image-link: https://i.ibb.co/SRGWT5j/lstm.png  )...


title: How does pytorch compute the gradients for a simple linear regression model?

body:
I am using pytorch and trying to understand how a simple linear regression model works.

I'm using a simple LinearRegressionModel class:

class LinearRegressionModel(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(LinearRegressionModel, self).__init__()
        self.linear = nn.Linear(input_dim, output_dim)

    def forward(self, x):
        out = self.linear(x)
        return out

model = LinearRegressionModel(1, 1)

Next I instantiate a loss criterion and an optimizer

criterion = nn.MSELoss()

optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Finally to train the model I use the following code:

for epoch in range(epochs):
    if torch.cuda.is_available():
        inputs = Variable(torch.from_numpy(x_train).cuda())

    if torch.cuda.is_available():
        labels = Variable(torch.from_numpy(y_train).cuda())

    # Clear gradients w.r.t. parameters
    optimizer.zero_grad()

    # Forward to get output
    outputs = model(inputs)

    # Calculate Loss
    loss = criterion(outputs, labels)

    # Getting gradients w.r.t. parameters
    loss.backward()

    # Updating parameters
    optimizer.step()

My question is how does the optimizer get the loss gradient, computed by loss.backward(), to update the parameters using the step() method? How are the model, the loss criterion and the optimizer tied together?


title: NotImplementedError: when i try to create a DataLoader object in Pytorch

body:
I created a custom Dataset class that inherits from PyTorch's Dataset class, in order to handle my custom dataset which i already preprocessed.

When i try to create a DataLoader object, i get this error:

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __init__(self, dataset, batch_size, shuffle, sampler, batch_sampler, num_workers, collate_fn, pin_memory, drop_last, timeout, worker_init_fn)
    174             if sampler is None:
    175                 if shuffle:
--> 176                     sampler = RandomSampler(dataset)
    177                 else:
    178                     sampler = SequentialSampler(dataset)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py in __init__(self, data_source, replacement, num_samples)
     62                              "since a random permute will be performed.")
     63
---> 64         if not isinstance(self.num_samples, int) or self.num_samples <= 0:
     65             raise ValueError("num_samples should be a positive integer "
     66                              "value, but got num_samples={}".format(self.num_samples))

/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py in num_samples(self)
     70         # dataset size might change at runtime
     71         if self._num_samples is None:
---> 72             return len(self.data_source)
     73         return self._num_samples
     74

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in __len__(self)
     18
     19     def __len__(self):
---> 20         raise NotImplementedError
     21
     22     def __add__(self, other):

NotImplementedError:

So, the error message regards to the not implementation of the len() function in the dataset.py, right?  But i did implement it and the getitem(), init() as well .

How can i overcome this?
Thank you


title: What is the fastest Mask R-CNN implementation available

body:
I'm running a Mask R-CNN model on an edge device (with an NVIDIA GTX 1080). I am currently using the Detectron2 Mask R-CNN implementation and I archieve an inference speed of around 5 FPS.

To speed this up I looked at other inference engines and model implementations. For example ONNX, but I'm not able to gain a faster inference speed.

TensorRT looks very promising to me but I did not found a ready "out-of-the-box" implementation for it.

Are there any other mature and fast inference engines or other techniques to speed up the inference?


title: gaussianblur transform not found in torchvision.transforms

body:
I have written the following data augmentation pipeline for Pytorch:
transform = transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.GaussianBlur(11, sigma=(0.1, 2.0)),
            transforms.ToTensor(),
            transforms.Normalize(mean, std)
        ])

But running the above code gives me the following error:
AttributeError: module 'torchvision.transforms' has no attribute 'GaussianBlur'

Is GaussianBlur a new feature that has not been included in torchvision yet? Or is it just my torchvision version that is too old?
I found it in the following documentation page: torchvision.transforms
Here are my packages versions:
torch                    1.6.0+cu101
torchvision              0.7.0+cu101


title: how to efficiently make a mini-batch of images in pytorch?

body:
I am trying to calculate a forward pass using pre-trained ResNet model in pytorch. I am having trouble creating a 4-d Tensor of mini-batches. Can someone please tell what is the proper way to do that?

EDIT: I changed the code and it works now. However, I still think there should a more efficient way of doing this.

Here's my code:

import pickle
import json
import shutil
import Image
import torchvision.models as models
import torchvision.transformers as transformers
from torch.autograd import Variable
from torch import Tensor
import glob
import torch

batch_size = 128
im_size = 299

normalize = transforms.Normalize(
   mean=[0.485, 0.456, 0.406],
   std=[0.229, 0.224, 0.225]
)
preprocess = transforms.Compose([
   transforms.Scale(im_size),
   transforms.CenterCrop(im_size),
   transforms.ToTensor(),
   normalize
])

model = models.resnet50(pretrained=True)

d_batch = make_batch(imgs, batch_size)

dtype = torch.FloatTensor
tmp = Variable(torch.randn(batch_size, 3, im_size, im_size).type(dtype), requires_grad=False)

for batch in tqdm(batches):
        try:
                data = [Image.open(img) for img in batch]
                for idx, item in enumerate(data):
                        tmp[idx] = preprocess(item)
                batch_result = model(tmp)
        except Exception,x:
                print x


title: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation?

body:
I am using pytorch-1.5 to do some gan test. My code is very simple gan code which just fit the sin(x) function:

import torch
import torch.nn as nn
import numpy as np
import matplotlib.pyplot as plt

# Hyper Parameters
BATCH_SIZE = 64
LR_G = 0.0001
LR_D = 0.0001
N_IDEAS = 5
ART_COMPONENTS = 15
PAINT_POINTS = np.vstack([np.linspace(-1, 1, ART_COMPONENTS) for _ in range(BATCH_SIZE)])

def artist_works():  # painting from the famous artist (real target)
    r = 0.02 * np.random.randn(1, ART_COMPONENTS)
    paintings = np.sin(PAINT_POINTS * np.pi) + r
    paintings = torch.from_numpy(paintings).float()
    return paintings

G = nn.Sequential(  # Generator
    nn.Linear(N_IDEAS, 128),  # random ideas (could from normal distribution)
    nn.ReLU(),
    nn.Linear(128, ART_COMPONENTS),  # making a painting from these random ideas
)

D = nn.Sequential(  # Discriminator
    nn.Linear(ART_COMPONENTS, 128),  # receive art work either from the famous artist or a newbie like G
    nn.ReLU(),
    nn.Linear(128, 1),
    nn.Sigmoid(),  # tell the probability that the art work is made by artist
)

opt_D = torch.optim.Adam(D.parameters(), lr=LR_D)
opt_G = torch.optim.Adam(G.parameters(), lr=LR_G)

for step in range(10000):
    artist_paintings = artist_works()  # real painting from artist
    G_ideas = torch.randn(BATCH_SIZE, N_IDEAS)  # random ideas
    G_paintings = G(G_ideas)  # fake painting from G (random ideas)

    prob_artist0 = D(artist_paintings)  # D try to increase this prob
    prob_artist1 = D(G_paintings)  # D try to reduce this prob

    D_loss = - torch.mean(torch.log(prob_artist0) + torch.log(1. - prob_artist1))
    G_loss = torch.mean(torch.log(1. - prob_artist1))

    opt_D.zero_grad()
    D_loss.backward(retain_graph=True)  # reusing computational graph
    opt_D.step()

    opt_G.zero_grad()
    G_loss.backward()
    opt_G.step()

But when i runing it got this error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [128, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Is there something wrong with my code?


title: Fine tuning of Bert word embeddings

body:
I would like to load a pre-trained Bert model and to fine-tune it and particularly the word embeddings of the model using a custom dataset.
The task is to use the word embeddings of chosen words for further analysis.
It is important to mention that the dataset consists of tweets and there are no labels.
Therefore, I used the BertForMaskedLM model.
Is it OK for this task to use the input ids (the tokenized tweets) as the labels?
I have no labels. There are just tweets in randomized order.
From this point, I present the code I wrote:
First, I cleaned the dataset from emojis, non-ASCII characters, etc as described in the following link (2.3 Section):
https://www.kaggle.com/jaskaransingh/bert-fine-tuning-with-pytorch
Second, the code of the fine tuning process:
import torch

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

model = BertForMaskedLM.from_pretrained('bert-base-uncased')

model.to(device)
model.train()

lr = 1e-2

optimizer = AdamW(model.parameters(), lr=lr, correct_bias=False)
max_len = 82
chunk_size = 20
epochs = 20

for epoch in range(epochs):
    epoch_losses = []
    for j, batch in enumerate(pd.read_csv(path + file_name, chunksize=chunk_size)):
        tweets = batch['content_cleaned'].tolist()

        encoded_dict = tokenizer.batch_encode_plus(
                            tweets,                      # Sentence to encode.
                            add_special_tokens = True, # Add '[CLS]' and '[SEP]'
                            max_length = max_len,           # Pad & truncate all sentences.
                            pad_to_max_length = True,
                            truncation=True,
                            return_attention_mask = True,   # Construct attn. masks.
                            return_tensors = 'pt',     # Return pytorch tensors.
                       )
        input_ids = encoded_dict['input_ids'].to(device)

        # Is it correct? or should I train it in another way?
        loss, _ = model(input_ids, labels=input_ids)
        loss_score = loss.item()

        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), max_grad_norm)
        optimizer.step()
        optimizer.zero_grad()
model.save_pretrained(path + "Fine_Tuned_BertForMaskedLM")

The loss starts from 50 and reduced until 2.3.


title: How to use PyTorch to calculate partial derivatives?

body:
I want to use PyTorch to get the partial derivatives between output and input. Suppose I have a function Y = 5*x1^4 + 3*x2^3 + 7*x1^2 + 9*x2 - 5, and I train a network to replace this function, then I use autograd to calculate dYdx1, dYdx2:

net = torch.load('net_723.pkl')
x = torch.tensor([[1,-1]],requires_grad=True).type(torch.FloatTensor)
y = net(x)
grad_c = torch.autograd.grad(y,x,create_graph=True,retain_graph=True)[0]

Then I get a wrong derivative as:

>>>tensor([[ 7.5583, -5.3173]])

but when I use function to calculate, I get the right answer:

Y = 5*x[0,0]**4 + 3*x[0,1]**3 + 7*x[0,0]**2 + 9*x[0,1] - 5
grad_c = torch.autograd.grad(Y,x,create_graph=True,retain_graph=True)[0]
>>>tensor([[ 34.,  18.]])

Why does this happen?


title: Implementing a custom dataset with PyTorch

body:
I'm attempting to modify this feedforward network taken from https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/01-basics/feedforward_neural_network/main.py
to utilize my own dataset.

I define a custom dataset of two 1 dim arrays as input and two scalars the corresponding output :

x = torch.tensor([[5.5, 3,3,4] , [1 , 2,3,4], [9 , 2,3,4]])
print(x)

y = torch.tensor([1,2,3])
print(y)

import torch.utils.data as data_utils

my_train = data_utils.TensorDataset(x, y)
my_train_loader = data_utils.DataLoader(my_train, batch_size=50, shuffle=True)

I've updated the hyperparameters to match new input_size (2) & num_classes (3).

I've also changed images = images.reshape(-1, 28*28).to(device) to images = images.reshape(-1, 4).to(device)

As the training set is minimal I've changed the batch_size to 1.

Upon making these modifications I receive error when attempting to train :

  RuntimeError                              Traceback (most recent call
  last)  in ()
       51
       52         # Forward pass
  ---> 53         outputs = model(images)
       54         loss = criterion(outputs, labels)
       55

  /home/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
      489             result = self._slow_forward(*input, **kwargs)
      490         else:
  --> 491             result = self.forward(*input, **kwargs)
      492         for hook in self._forward_hooks.values():
      493             hook_result = hook(self, input, result)

   in forward(self, x)
       31
       32     def forward(self, x):
  ---> 33         out = self.fc1(x)
       34         out = self.relu(out)
       35         out = self.fc2(out)

  /home/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in call(self, *input, **kwargs)
      489             result = self._slow_forward(*input, **kwargs)
      490         else:
  --> 491             result = self.forward(*input, **kwargs)
      492         for hook in self._forward_hooks.values():
      493             hook_result = hook(self, input, result)

  /home/.local/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
       53
       54     def forward(self, input):
  ---> 55         return F.linear(input, self.weight, self.bias)
       56
       57     def extra_repr(self):

  /home/.local/lib/python3.6/site-packages/torch/nn/functional.py
  in linear(input, weight, bias)
      990     if input.dim() == 2 and bias is not None:
      991         # fused op is marginally faster
  --> 992         return torch.addmm(bias, input, weight.t())
      993
      994     output = input.matmul(weight.t())

  RuntimeError: size mismatch, m1: [3 x 4], m2: [2 x 3] at
  /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:249

How to amend code to match expected dimensionality ? I'm unsure what code to change as I've changed all parameters that require updating ?

Source prior to changes :

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms

# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Hyper-parameters
input_size = 784
hidden_size = 500
num_classes = 10
num_epochs = 5
batch_size = 100
learning_rate = 0.001

# MNIST dataset
train_dataset = torchvision.datasets.MNIST(root='../../data',
                                           train=True,
                                           transform=transforms.ToTensor(),
                                           download=True)

test_dataset = torchvision.datasets.MNIST(root='../../data',
                                          train=False,
                                          transform=transforms.ToTensor())

# Data loader
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
                                          batch_size=batch_size,
                                          shuffle=False)

# Fully connected neural network with one hidden layer
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
        images = images.reshape(-1, 28*28).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

# Test the model
# In test phase, we don't need to compute gradients (for memory efficiency)
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.reshape(-1, 28*28).to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

    print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))

# Save the model checkpoint
torch.save(model.state_dict(), 'model.ckpt')

Source post changes :

x = torch.tensor([[5.5, 3,3,4] , [1 , 2,3,4], [9 , 2,3,4]])
print(x)

y = torch.tensor([1,2,3])
print(y)

import torch.utils.data as data_utils

my_train = data_utils.TensorDataset(x, y)
my_train_loader = data_utils.DataLoader(my_train, batch_size=50, shuffle=True)

print(my_train)

print(my_train_loader)

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms

# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Hyper-parameters
input_size = 2
hidden_size = 3
num_classes = 3
num_epochs = 5
batch_size = 1
learning_rate = 0.001

# MNIST dataset
train_dataset = my_train

# Data loader
train_loader = my_train_loader

# Fully connected neural network with one hidden layer
class NeuralNet(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

model = NeuralNet(input_size, hidden_size, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Move tensors to the configured device
        images = images.reshape(-1, 4).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 100 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

# Test the model
# In test phase, we don't need to compute gradients (for memory efficiency)
with torch.no_grad():
    correct = 0
    total = 0
    for images, labels in test_loader:
        images = images.reshape(-1, 4).to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

    print('Accuracy of the network on the 10000 test images: {} %'.format(100 * correct / total))

# Save the model checkpoint
torch.save(model.state_dict(), 'model.ckpt')


title: Getting the accuracy from classification_report back into a list

body:
I am using Sklean's classification_report to summarize my train and test epochs.

sklearn.metrics.classification_report

I'm getting kind of this back for each epoch:

>>> from sklearn.metrics import classification_report
>>> y_true
>>> y_pred
>>> target_names = ['class 0', 'class 1', 'class 2']
>>> print(classification_report(y_true, y_pred, target_names=target_names))
              precision    recall  f1-score   support

     class 0       0.50      1.00      0.67         1
     class 1       0.00      0.00      0.00         1
     class 2       1.00      0.67      0.80         3

    accuracy                           0.60         5
   macro avg       0.50      0.56      0.49         5
weighted avg       0.70      0.60      0.61         5

(e.g. from sklearn script)

Now I am searching for a way, to get those accuracy for each epoch in a list to calculate the mean and std of all accuracy.

This question seems to be pretty trivial but as you can see from my questions before I am pretty new to Python/Machine Learning.

Thanks for your help

Leo


title: How to select specific labels in pytorch MNIST dataset

body:
I'm trying to create dataloaders using only a specific digit from PyTorch Mnist dataset

I already tried to create my own Sampler but it doesn't work and I'm not sure I'm using correctly the mask.

class YourSampler(torch.utils.data.sampler.Sampler):

    def __init__(self, mask):

        self.mask = mask

    def __iter__(self):

        return (self.indices[i] for i in torch.nonzero(self.mask))

    def __len__(self):

        return len(self.mask)

mnist = datasets.MNIST(root=dataroot, train=True, download=True, transform = transform)

mask = [True if mnist[i][1] == 5 else False for i in range(len(mnist))]

mask = torch.tensor(mask)

sampler = YourSampler(mask)

trainloader = torch.utils.data.DataLoader(mnist, batch_size=4, sampler = sampler, shuffle=False, num_workers=2)

So far I had many different types of errors. For this implementation, it's "Stop Iteration".
I feel like this is very easy/stupid but I can't find a simple way to do it.
Thank you for your help!


title: Pytorch: Learnable threshold for clipping activations

body:
What is the proper way to clip ReLU activations with a learnable threshold? Here's how I implemented it, however I'm not sure if this is correct:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.act_max = nn.Parameter(torch.Tensor([0]), requires_grad=True)

        self.conv1 = nn.Conv2d(3, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5)
        self.pool = nn.MaxPool2d(2, 2)
        self.relu = nn.ReLU()
        self.linear = nn.Linear(64 * 5 * 5, 10)

    def forward(self, input):
        conv1 = self.conv1(input)
        pool1 = self.pool(conv1)
        relu1 = self.relu(pool1)

        relu1[relu1 > self.act_max] = self.act_max

        conv2 = self.conv2(relu1)
        pool2 = self.pool(conv2)
        relu2 = self.relu(pool2)
        relu2 = relu2.view(relu2.size(0), -1)
        linear = self.linear(relu2)
        return linear

model = Net()
torch.nn.init.kaiming_normal_(model.parameters)
nn.init.constant(model.act_max, 1.0)
model = model.cuda()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
for epoch in range(100):
    for i in range(1000):
        output = model(input)
        loss = nn.CrossEntropyLoss()(output, label)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        model.act_max.data = model.act_max.data - 0.001 * model.act_max.grad.data

I had to add the last line because without it the value would not update for some reason.

UPDATE: I am now trying a method to compute the uppper bound (act_max) based on the gradients for activations:

For all activations above the threshold (relu1[relu1 > self.act_max]), look at their gradients: compute the average direction all these gradients point to.
For all positive activations below the threshold, compute the average gradient of which direction they want to change to.
The sum of these average gradients determines the direction and magnitude of the change for act_max.


title: RuntimeError 'DivBackward0' nan values in its 0th output, but works when tensors loaded from disk?

body:
I'm trying to implement a particular loss function in PyTorch called SMAPE (commonly used in time series forecasting). I have two variables, model_outputs and target_outputs, and the formula for computing the element-wise SMAPE is straight-forward:

numerator = torch.abs(model_outputs - target_outputs)
denominator = torch.abs(model_outputs) + torch.abs(target_outputs)
elementwise_smape = torch.div(numerator, denominator)
nan_mask = torch.isnan(elementwise_smape)
loss = elementwise_smape[~nan_mask].mean()
assert ~torch.isnan(loss)  # loss = 0.023207199
loss.backward()

But when I ran my code, I received the following error: RuntimeError: Function 'DivBackward0' returned nan values in its 0th output.

I used autograd.detect_anomaly() to track down the error as stemming from elementwise_smape = torch.div(numerator, denominator). I tried the suggested solutions I could find online (checking for NaN values, adding hooks, printing gradients) and nothing helped. So I decided to write the tensors to disk and write an independent script:

import torch

model_outputs = torch.load('model_outputs.pt')
target_outputs = torch.load('target_outputs.pt')
print(model_outputs)
print(target_outputs)

model_outputs.requires_grad = True
target_outputs.requires_grad = False

numerator = torch.abs(model_outputs - target_outputs)
denominator = torch.abs(model_outputs) + torch.abs(target_outputs)
elementwise_smape = torch.div(numerator, denominator)
nan_mask = torch.isnan(elementwise_smape)
loss = elementwise_smape[~nan_mask].mean()
assert ~torch.isnan(loss)
print(loss.detach().numpy())
loss.backward()
print('Success!')

This worked completely fine! To test that the problem doesn't originate from any of the variables elementwise_smape depends on (model_outputs, numerator, denominator), I tried replacing the loss function with each of the dependent variables, but all worked!

nan_mask = torch.isnan(elementwise_smape)
loss = model_outputs[~nan_mask].mean()
loss.backward()  # succeeds

nan_mask = torch.isnan(elementwise_smape)
loss = numerator[~nan_mask].mean()
loss.backward()  # succeeds

nan_mask = torch.isnan(elementwise_smape)
loss = denominator[~nan_mask].mean()
loss.backward()  # succeeds

So what the hell is going on with the element-wise division that causes a problem when model_outputs are generated by the model, but not when model_outputs are loaded from disk?

Update #1: Adding complete error:

Traceback of forward call that caused the error:
  File "/home/rylan/tsml/main.py", line 116, in <module>
    main(experiment_number)
  File "/home/rylan/tsml/main.py", line 23, in main
    loss_per_model_per_step=loss_per_model_per_step)
  File "/home/rylan/tsml/main.py", line 47, in run
    sequence_lengths)
  File "/home/rylan/tsml/main.py", line 80, in step
    sequence_lengths=sequence_lengths)
  File "/home/rylan/tsml/models.py", line 153, in loss
    model_output_log_probs=None)
  File "/home/rylan/tsml/losses.py", line 96, in calculate_mean_smape
    target_outputs=target_outputs)
  File "/home/rylan/tsml/losses.py", line 117, in calculate_elementwise_smape
    elementwise_smape = torch.div(numerator, denominator)

Traceback (most recent call last):
  File "/home/rylan/tsml/main.py", line 116, in <module>
    main(experiment_number)
  File "/home/rylan/tsml/main.py", line 23, in main
    loss_per_model_per_step=loss_per_model_per_step)
  File "/home/rylan/tsml/main.py", line 47, in run
    sequence_lengths)
  File "/home/rylan/tsml/main.py", line 89, in step
    total_loss.backward()
  File "/home/rylan/.local/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/rylan/.local/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Function 'DivBackward0' returned nan values in its 0th output.

Process finished with exit code 1

Update #2: Finomnis suggested writing the tensors to disk, reading them back and comparing them to the original:

torch.save(model_outputs, 'model_outputs.pt')
torch.save(target_outputs, 'target_outputs.pt')
model_outputs_2 = torch.load('model_outputs.pt')
target_outputs_2 = torch.load('target_outputs.pt')
(model_outputs == model_outputs_2).all()  # tensor(1, dtype=torch.uint8)
(target_outputs == target_outputs_2).all()  # tensor(0, dtype=torch.uint8)

I suspected that (target_outputs == target_outputs_2).all() evaluates to False due to NaNs. This is correct:

(torch.isnan(target_outputs) == torch.isnan(target_outputs_2)).all()  # tensor(1, dtype=torch.uint8)
(target_outputs[~torch.isnan(target_outputs)] == target_outputs_2[~torch.isnan(target_outputs_2)]).all()  # tensor(1, dtype=torch.uint8)

Update #3: I tried the next suggestion of detaching model_outputs to ensure that the problem isn't originating from elsewhere:

def calculate_elementwise_smape(model_outputs,
                                target_outputs):
    model_outputs = model_outputs.detach()
    model_outputs.requires_grad = True

    target_outputs = target_outputs.detach()

    numerator = torch.abs(model_outputs - target_outputs)
    denominator = torch.abs(model_outputs) + torch.abs(target_outputs)
    elementwise_smape = torch.div(numerator, denominator)
    return elementwise_smape

but this produced the same error RuntimeError: Function 'DivBackward0' returned nan values in its 0th output. at the same location elementwise_smape = torch.div(numerator, denominator)

To reiterate what I said earlier, this problem does not occur if I load the tensors from disk and make the exact same sequence of operations.


I'm trying to build a Model using EfficientNet-B0.
The details of the Model are shown in the code below.
I got the following error when I tried to learn.

TypeError                                 Traceback (most recent call last)
'''
<ipython-input-17-fb3850894108> in forward(self, *x)
     24         #x: bs*N x 3 x 128 x 128
     25         print(x.shape)     #([384, 3, 224, 224])
---> 26         x = self.enc(x)
     27         #x: bs*N x C x 4 x 4
     28         shape = x.shape

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
     98     def forward(self, input):
     99         for module in self:
--> 100             input = module(input)
    101         return input
    102

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() takes 1 positional argument but 2 were given

I suspect that m.children() may have an effect.
If anyone knows the cause of this error, please let me know.
Thank you.

class Model(nn.Module):
    def __init__(self, arch='efficientnet-b0', n=6, pre=True):
        super().__init__()
        m = EfficientNet.from_pretrained('efficientnet-b0')
        #print(*list(m.children())
        nc = m._fc.in_features
        print(nc)
        self.enc = nn.Sequential(*list(m.children())[:-2])
        #nc = list(m.children())[-2].in_features
        self.head = nn.Sequential(*list(m.children())[-2:])
        self.head._fc = nn.Linear(nc, n)
        #self.enc = nn.Sequential(*list(m.children()))
        #print('fc_infeatures : {}'.format(nc))
        #self.head = nn.Sequential(Flatten(),nn.Linear(nc,512),
        #                    relu(),nn.BatchNorm1d(512), nn.Dropout(0.5),nn.Linear(512,n))

    def forward(self, *x):
        print(x[0].shape)  #([32, 3, 224, 224])
        shape = x[0].shape
        n = len(x)           # n = 12
        #torch.stack直後では32*12*3*224*224(bs x N x 3 x 224 x 224)
        x = torch.stack(x,1).view(-1,shape[-3],shape[-2],shape[-1])
        #x: bs*N x 3 x 128 x 128
        print(x.shape)     #([384, 3, 224, 224])
        x = self.enc(x)
        #x: bs*N x C x 4 x 4
        shape = x.shape
        #concatenate the output for tiles into a single map
        x = x.view(-1,n,shape[1],shape[2],shape[3]).permute(0,2,1,3,4).contiguous()\
          .view(-1,shape[1],shape[2]*n,shape[3])
        #x: bs x C x N*4 x 4
        x = self.head(x)
        return x


title: pytorch gradient / derivative / difference along axis like numpy.diff

body:
I have been struggling with this for quite some time. All I want is a torch.diff() function. However, many matrix operations do not appear to be easily compatible with tensor operations.

I have tried an enormous amount of various pytorch operation combinations, yet none of them work.

Due to the fact that pytorch hasn't implemented this basic feature, I started by simply trying to subtract the element i+1 from element i along a specific axis.

However, you can't simply do this element-wise (due to the tensor limitations), so I tried to construct another tensor, with the elements shifted along one axis:

ix_plus_one = [0]+list(range(0,prediction.size(1)-1))
ix_differential_tensor = torch.LongTensor(ix_plus_one)
diff_one_tensor = prediction[:,ix_differential_tensor]

But now we have a different problem - indexing doesn't really work to mimic numpy in pytorch as it advertises, so you can't index with a "list-like" Tensor like this.  I also tried using the tensor scatter functions

So I'm still stuck with this simple problem of trying to get a gradient on a pytoch tensor.

All of my searching leads to the marvelous capabilities of pytorchs' "autograd" function - which has nothing to do with this problem.


I get:

  RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes'
  failed.  at
  /opt/conda/conda-bld/pytorch_1550796191843/work/aten/src/THNN/generic/ClassNLLCriterion.c:93

When running this code:

    criterion = nn.CrossEntropyLoss()
    #Define the optimizer
    optimizer=optim.SGD(net.parameters(),lr=0.01,momentum=0.9)
    epochs=20
    for epoch in range(epochs):
        print ("epoch #", epoch)
        running_loss=0.0
        for i, data in enumerate(train_loader,0):
            inputs,labels=data
            inputs,labels= inputs.to(device),labels.to(device)
            optimizer.zero_grad()
            #train
            output=net(inputs)
            loss=criterion(output,labels)

    print ("loss: ", loss.item())
    running_loss+=loss.item()
    loss.backward()
    optimizer.step()
    print ('Finished Training')


title: Implementation of Focal loss for multi label classification

body:
trying to write focal loss for multi-label classification

class FocalLoss(nn.Module):
    def __init__(self, gamma=2, alpha=0.25):
        self._gamma = gamma
        self._alpha = alpha

    def forward(self, y_true, y_pred):
        cross_entropy_loss = torch.nn.BCELoss(y_true, y_pred)
        p_t = ((y_true * y_pred) +
               ((1 - y_true) * (1 - y_pred)))
        modulating_factor = 1.0
        if self._gamma:
            modulating_factor = torch.pow(1.0 - p_t, self._gamma)
        alpha_weight_factor = 1.0
        if self._alpha is not None:
            alpha_weight_factor = (y_true * self._alpha +
                                   (1 - y_true) * (1 - self._alpha))
        focal_cross_entropy_loss = (modulating_factor * alpha_weight_factor *
                                    cross_entropy_loss)
        return focal_cross_entropy_loss.mean()

But when i run this i get

  File "train.py", line 82, in <module>
    loss = loss_fn(output, target)
  File "/home/bubbles/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 538, in __call__
    for hook in self._forward_pre_hooks.values():
  File "/home/bubbles/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 591, in __getattr__
    type(self).__name__, name))
AttributeError: 'FocalLoss' object has no attribute '_forward_pre_hooks'

Any suggestions would be really helpful, Thanks in advance.


title: 'tuple' object has no attribute 'cuda'

body:
It's my first time to use stack overflow. Thanks for your help!
Recently, I'm learning Deep-Learning with Pytorch. When I finished the DataSet and DataLoader, I got some trouble in writing train function. It show me that:

<ipython-input-16-e211160f5ce8> in <module>()
      5 total = 0
      6 for id, (inputs, labels) in enumerate(train_loader):
----> 7   inputs, labels = inputs.cuda(), labels.cuda()
      8   optimizer.zero_grad()
      9   outputs = net(inputs)

AttributeError: 'tuple' object has no attribute 'cuda'

My code is here:

for id, (inputs, labels) in enumerate(train_loader):
  inputs, labels = inputs.cuda(), labels.cuda()
  optimizer.zero_grad()
  outputs = net(inputs)
  loss = self.criterion(outputs, labels)
  loss.backward()
  optimizer.step()

The Dataloader return image and labels, the image's type is numpy.array and the labels' type is tuple. I can not do anything to the parameters inputs and labels. I don't find answer in Google. Thanks for help and endure my English and code.

For more details:
My dataset is Stanford Cars, and I use it to achieve image classification.
It just a simple code, I don't know what's worry with it. Or you can give me some advice to learn Pytorch, Thanks:)
torch                    1.5.0+cu101
torchvision              0.6.0+cu101


title: Exploding loss in pyTorch

body:
I am trying to train a latent space model in pytorch. The model is relatively simple and just requires me to minimize my loss function but I am getting an odd error. After running for a short while the loss suddenly explodes upwards.

import numpy as np
import scipy.sparse.csgraph as csg
import torch
from torch.autograd import Variable
import torch.autograd as autograd
import matplotlib.pyplot as plt
%matplotlib inline

def cmdscale(D):
    # Number of points
    n = len(D)

    # Centering matrix
    H = np.eye(n) - np.ones((n, n))/n

    # YY^T
    B = -H.dot(D**2).dot(H)/2

    # Diagonalize
    evals, evecs = np.linalg.eigh(B)

    # Sort by eigenvalue in descending order
    idx   = np.argsort(evals)[::-1]
    evals = evals[idx]
    evecs = evecs[:,idx]

    # Compute the coordinates using positive-eigenvalued components only
    w, = np.where(evals > 0)
    L  = np.diag(np.sqrt(evals[w]))
    V  = evecs[:,w]
    Y  = V.dot(L)

    return Y, evals

Y = np.array([[0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 0., 1., 1., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0.],
              [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 1., 0., 0., 1., 0.],
              [0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 1., 0.],
              [0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
              [0., 1., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 1.],
              [0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0.],
              [1., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 1.],
              [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0.],
              [0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 0., 0., 0., 1., 0.],
              [0., 0., 0., 0., 0., 0., 0., 0., 1., 0., 0., 0., 0., 1., 1.],
              [0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0., 0., 0.],
              [0., 0., 0., 1., 1., 0., 0., 0., 0., 0., 1., 1., 0., 0., 0.],
              [0., 0., 0., 0., 0., 0., 1., 0., 1., 0., 0., 1., 0., 0., 0.]])

temp = Y[~np.all(Y == 0, axis=1)]
temp = temp[:,~np.all(Y == 0, axis=1)]
Y = temp

n = np.shape(Y)[0]
k = 2

D = csg.shortest_path(Y, directed=True)
Z = cmdscale(D)[0][:,0:k]
Z = Z - Z.mean(axis=0, keepdims=True)

tZ = autograd.Variable(torch.Tensor(Z), requires_grad=True)
B = autograd.Variable(torch.Tensor([0]), requires_grad=True)
tY = torch.autograd.Variable(torch.Tensor(Y), requires_grad=False)

#calculating pairwise euclidean distance
def distMatrix(m):
    n = m.size(0)
    d = m.size(1)
    x = m.unsqueeze(1).expand(n, n, d)
    y = m.unsqueeze(0).expand(n, n, d)
    return torch.sqrt(torch.pow(x - y, 2).sum(2) + 1e-4)

def loss(tY):
    d = -distMatrix(tZ)+B
    sigmoidD = torch.sigmoid(d)
    #removing diagonal
    reduce = tY*torch.log(sigmoidD)+(1-tY)*torch.log(1-sigmoidD)
    reduce[torch.eye(n).byte()] = 0
    return -reduce.sum()

losses = []
learning_rate = 1e-4
l = loss(tY)
stepSize = 1000

for i in range(stepSize):
    l.backward(retain_graph=True)
    losses.append(float(loss(tY)))
    tZ.data = tZ.data - learning_rate * tZ.grad.data
    B.data = B.data - learning_rate * B.grad.data

    tZ.grad.data.zero_()
    B.grad.data.zero_()

plt.subplot(122)
plt.plot(losses)
plt.title('Loss')
plt.xlabel('Iteration')
plt.ylabel('loss')

plt.show()

shouldnt the loss keep going down? or atleast converge to some point? I must've done something wrong, I am new to pytorch, any hints or nudges in the right direction would be highly appreciated!


title: Is there an Real-Time Object Detection algorithm for CPU?

body:
I need to use some object detection algorithms in real time (will detect stuff while driving).
So far I've managed to use YOLO-v3 with pytorch on a GPU pc with great results.

Is there an algorithm I can use on an ~1.6GHz CPU on a laptop?
My current net is detecting on the CPU with ~5 second per frame which is too slow for realtime...

Thanks in advance


title: Load a single layer weights from a pretrained model

body:
I want to specifically add the pretrained model parameters of some layers to my new network .
For Linear Layer i just did :

model_enc.linear_3d.weight = model_trained.linear_3d.weight
model_enc.linear_3d.bias = model_trained.linear_3d.bias

Will this suffice or are there any other parameters that I need to load or is there a easy way than this . My module is already trained and I just want to load params for few layers .

Thank you


title: Splitting a directory with images into sub folders using Pytorch or Python

body:
I have a directory with two sub directories in it. Each sub-dir has bunch of images. These sub-dir also specify the two classes of images.

I want to have 3 directories (train, validation, test) and within each of these 3 sub dir, I want to 2 sub directory of each class respectively with images.

I want to split the no.of images into train, val, test directories by random sampling. SO some 60% images go to train, 20, val and 20 in test.

Initial Structure:

Main_folder
 - Good
 - Bad

What I want:

Main_folder:
 - Train
  - Good
  - Bad
 - Val
  - Good
  - Bad
 - Test
  - Good
  - Bad

I want to split each sub directories into two directories within it with random assignment of images (


title: HuggingFace Transformers For Text Generation with CTRL with Google Colab's free GPU

body:
I wanted to test TextGeneration with CTRL using PyTorch-Transformers, before using it for fine-tuning. But it doesn't prompt anything like it does with GPT-2 and other similar language generation models. I'm very new for this and am stuck and can't figure out what's going on.

This is the procedure I followed in my Colab notebook,

!pip install transformers

!git clone https://github.com/huggingface/pytorch-transformers.git

!python pytorch-transformers/examples/run_generation.py \
    --model_type=ctrl \
    --length=100 \
    --model_name_or_path=ctrl \
    --temperature=0.2 \
    --repetition_penalty=1.2 \

And this is what I get after running the script

02/10/2020 01:02:31 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-vocab.json from cache at /root/.cache/torch/transformers/a858ad854d3847b02da3aac63555142de6a05f2a26d928bb49e881970514e186.285c96a541cf6719677cfb634929022b56b76a0c9a540186ba3d8bbdf02bca42
02/10/2020 01:02:31 - INFO - transformers.tokenization_utils -   loading file https://raw.githubusercontent.com/salesforce/ctrl/master/ctrl-merges.txt from cache at /root/.cache/torch/transformers/aa2c569e6648690484ade28535a8157aa415f15202e84a62e82cc36ea0c20fa9.26153bf569b71aaf15ae54be4c1b9254dbeff58ca6fc3e29468c4eed078ac142
02/10/2020 01:02:31 - INFO - transformers.configuration_utils -   loading configuration file https://storage.googleapis.com/sf-ctrl/pytorch/ctrl-config.json from cache at /root/.cache/torch/transformers/d6492ca334c2a4e079f43df30956acf935134081b2b3844dc97457be69b623d0.1ebc47eb44e70492e0c20494a084f108332d20fea7fe5ad408ef5e7a8f2baef4
02/10/2020 01:02:31 - INFO - transformers.configuration_utils -   Model config CTRLConfig {
  "architectures": null,
  "attn_pdrop": 0.1,
  "bos_token_id": 0,
  "dff": 8192,
  "do_sample": false,
  "embd_pdrop": 0.1,
  "eos_token_ids": 0,
  "finetuning_task": null,
  "from_tf": false,
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_range": 0.02,
  "is_decoder": false,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_epsilon": 1e-06,
  "length_penalty": 1.0,
  "max_length": 20,
  "model_type": "ctrl",
  "n_ctx": 512,
  "n_embd": 1280,
  "n_head": 16,
  "n_layer": 48,
  "n_positions": 50000,
  "num_beams": 1,
  "num_labels": 1,
  "num_return_sequences": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pad_token_id": 0,
  "pruned_heads": {},
  "repetition_penalty": 1.0,
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "temperature": 1.0,
  "top_k": 50,
  "top_p": 1.0,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 246534
}

02/10/2020 01:02:31 - INFO - transformers.modeling_utils -   loading weights file https://storage.googleapis.com/sf-ctrl/pytorch/seqlen256_v1.bin from cache at /root/.cache/torch/transformers/c146cc96724f27295a0c3ada1fbb3632074adf87e9aef8269e44c9208787f8c8.b986347cbab65fa276683efbb9c2f7ee22552277bcf6e1f1166557ed0852fdf0
tcmalloc: large alloc 1262256128 bytes == 0x38b92000 @  0x7fe1900bdb6b 0x7fe1900dd379 0x7fe139843b4a 0x7fe1398455fa 0x7fe13bb7578a 0x7fe13bdbe30b 0x7fe13be05b37 0x7fe184c8cad5 0x7fe184c8d17b 0x7fe184c91160 0x7fe184ade496 0x551b15 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x5096b7 0x595311 0x54a6ff 0x551b81 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x509642 0x595311 0x54a6ff 0x551b81 0x5aa6ec 0x50abb3 0x50c5b9
tcmalloc: large alloc 1262256128 bytes == 0x19fdda000 @  0x7fe1900bdb6b 0x7fe1900dd379 0x7fe139843b4a 0x7fe1398455fa 0x7fe13bb7578a 0x7fe13bdbe30b 0x7fe13be05b37 0x7fe184c8cad5 0x7fe184c8d17b 0x7fe184c91160 0x7fe184ade496 0x551b15 0x5aa6ec 0x50abb3 0x50c5b9 0x508245 0x509642 0x595311 0x54a6ff 0x551b81 0x5aa6ec 0x50abb3 0x50d390 0x508245 0x509642 0x595311 0x54a6ff 0x551b81 0x5a067e 0x50d966 0x508245
^C

and then terminates. Could this be because of a GPU problem?

Any sort of help is appreciated.


title: How to set gradients to Zero without optimizer?

body:
Between mutliple .backward() passes I'd like to set the gradients to zero. Right now I have to do this for every component seperately  (here these are x and t), is there a way to do this "globally" for all affected variables? (I imagine something like z.set_all_gradients_to_zero().)

I know there is optimizer.zero_grad() if you use an optimizer, but is there also a direct way without using an optimizer?

import torch

x = torch.randn(3, requires_grad = True)
t = torch.randn(3, requires_grad = True)
y = x + t
z = y + y.flip(0)

z.backward(torch.tensor([1., 0., 0.]), retain_graph = True)
print(x.grad)
print(t.grad)
x.grad.data.zero_()  # both gradients need to be set to zero
t.grad.data.zero_()
z.backward(torch.tensor([0., 1., 0.]), retain_graph = True)
print(x.grad)
print(t.grad)


title: How Batch learning in Pytorch is performed?

body:
When you look at how network architecture is built inside the pytorch code, we need to extend the torch.nn.Module and inside __init__, we define the module of networks and pytorch is going to track the gradients of parameters of these modules. Then inside the forward function, we define how the forward pass should be done for our network.

The thing I do not understand here is how the batch learning is going to occur. In none of the definition above including the forward function, we do not care about the dimension of batch of the input to our network. The only thing we need to set to perform batch learning is to add an extra dimension to the input which corresponds to the batch size but nothing inside the network definition is going to be changed if we are working with batch learning. At least, this is the thing I have seen in the codes here.

So, if all the things I have explained so far is correct (I would really appreciate if you let me know if I have misunderstood something), how batch learning is performed if nothing is declared regarding the batch size inside the definition of our network class (the class that inherits torch.nn.Module)? Specifically, I am interested to know how batch gradient descent algorithm is implemented in pytorch when we just set nn.MSELoss with batch dimension.


title: What is pixel-wise softmax loss?

body: what is the pixel-wise softmax loss? In my understanding, it's just a cross-entropy loss, but I didn't find the formula. Can someone help me? It's better to have the pytorch code.


title: In PyTorch, what makes a tensor have non-contiguous memory?

body:
According to this SO and this PyTorch discussion, PyTorch's view function works only on contiguous memory, while reshape does not. In the second link, the author even claims:

  [view] will raise an error on a non-contiguous tensor.

But when does a tensor have non-contiguous memory?


title: in-place shuffle torch.Tensor in the order of a numpy.ndarray

body:
I want to change the order of elements of a torch.Tensor from default to a numpy.ndarray. In other words, I want to shuffle it so that the order of its elements be specified with a numpy array; the important thing about this problem is that I don't want any third object to be created (because of memory limits)
Is there something like below code in python 2.7?

torch_tensor.shuffle(order)


title: Torchtext 0.7 shows Field is being deprecated. What is the alternative?

body:
Looks like the previous paradigm of declaring Fields, Examples and using BucketIterator is deprecated and will move to legacy in 0.8. However, I don't seem to be able to find an example of the new paradigm for custom datasets (as in, not the ones included in torch.datasets) that doesn't use Field. Can anyone point me at an up-to-date example?
Reference for deprecation:
https://github.com/pytorch/text/releases


title: What does the .numpy() function do?

body:
I tried searching for the documentation online but I can't find anything that gives me an answer. What does .numpy() function do? The example code given is:
y_true = []
for X_batch, y_batch in mnist_test:
    y_true.append(y_batch.numpy()[0].tolist())


title: Axes don't match array error in pytorch

body:
I was trying to run the vnet implementation in pytorch
(https://github.com/mattmacy/vnet.pytorch) and after normalising the scans with

x_max = 512
y_max = 512
z_max = 500
voxspacing = 0.7

when I call the tran function on the line where the for loop is enumerating through the data loader I get a

  axes don't match array error.

I don't understand how to fix this. Any help would be really helpful. I am a newbie in machine learning


title: How to do softmax for pixelwise classification

body:
My goal is to do grey scale image segmentation using pixelwise classification. So I have two labels 0 and 1. I made a network in pytorch which looks like the following.

class Net(nn.Module):

def __init__(self):
    super(Net, self).__init__()

    self.up = nn.Upsample(scale_factor=2, mode='nearest')

    self.conv11 = nn.Conv2d(1, 128, kernel_size=3, padding=1)
    self.conv12 = nn.Conv2d(128, 256, kernel_size=3, padding=1)
    self.conv13 = nn.Conv2d(256, 2, kernel_size=3, padding=1)

def forward(self, x):
    in_size = x.size(0)

    x = F.relu(self.conv11(x))
    x = F.relu(self.conv12(x))
    x = F.relu(self.conv13(x))

    x = F.softmax(x, 2)

    return x

In the last layer I designed the conv13 in such that it produces 2 channels one for each class.

Since I was using the softmax I was expecting that summation of value of same index on 2 separate channel would equal to 1.

For example assume the output image is ( 2{channel}, 4, 4). So I was expecting that

  image[ channel 1 ][0][0] + image[ channel 2 ][0][0] = 1

But the output I get is 0.0015 which is not even close to 1. How can i use the softmax to predict channelwise ?

To check this I used the following code

for batch, data in enumerate(trainloader, 0):
    inputs , labels = data
    inputs, labels = inputs.to(device), labels.to(device)

    optimizer.zero_grad()
    outputs = net(inputs)
    loss = rmse(outputs, labels)
    loss.backward()
    optimizer.step()
    running_loss += loss.item()

    predicted = outputs.data
    predicted = predicted.to('cpu')
    predicted_img = predicted.numpy()

    predicted_img = np.reshape(predicted_img,(2, 4, 4))

    print(predicted_img[0])
    print(predicted_img[1])

Those prints showed this

[[**0.2762002**  0.13305853 0.2510342  0.23114938]
 [0.26812425 0.28500515 0.05682982 0.15851443]
 [0.1640967  0.5409352  0.43547812 0.44782472]
 [0.29157883 0.0410011  0.2566578  0.16251141]]

[[**0.23052207** 0.868455   0.43436486 0.0684725 ]
 [0.18001427 0.02341573 0.0727293  0.2525512 ]
 [0.06587404 0.04974682 0.3773188  0.6559266 ]
 [0.5235896  0.05838248 0.11558701 0.02304965]]

It is clear that the corresponding elements are not summing up to 1 like

  0.2762002 (index 0, 0) + 0.23052207 (index 0, 0) != 1

How can I fix it ?


title: Training multiple neural networks asynchronously in parallel

body:
The problem
I am currently working on a project that I sadly can't share with you. The project is about hyper-parameter optimization for neural networks, and it requires that I train multiple neural network models (more than I can store on my GPU) in parallel. The network architectures stay the same, but the network parameters and hyper-parameters are subjected to change between each training interval. I am currently achieving this using PyTorch on a linux environment in order to allow my NVIDIA GTX 1660 (6GB RAM) to use the multiprocessing feature that PyTorch provides.
Code (simplified):
def training_function(checkpoint):
   load(checkpoint)
   train(checkpoint)
   unload(checkpoint)

for step in range(training_steps):
   trained_checkpoints = list()
   for trained_checkpoint in pool.imap_unordered(training_function, checkpoints):
      trained_checkpoints.append(trained_checkpoint)
   for optimized_checkpoint in optimize(trained_checkpoints):
      checkpoints.update(optimized_checkpoint)

I currently test with a population of 30 neural networks (i.e. 30 checkpoints) with the MNIST and FashionMNIST datasets which consists of 70 000 (50k training, 10k validation, 10k testing) 28x28 images with 1 channel each respectively. The network I train is a simple Lenet5 implementation.
I use a torch.multiprocessing pool and allow 7 processes to be spawned. Each process uses some of the GPU memory available just to initialize the CUDA environment in each process. After training, the checkpoints are adapted with my hyper-parameter optimization technique.
The load function in the training_function loads the model- and optimizer state (holds the network parameter tensors) from a local file into GPU memory using torch.load. The unload saves the newly trained states back to file using torch.save and deletes them from memory. I do this because PyTorch will only detach GPU tensors when no variable is referencing them. I have to do this because I have limited GPU memory.
The current setup works, but each CUDA initialization occupies over 700MB of GPU RAM, and so I am interested if there are other ways I could do this that may use less memory without a penalty to efficiency.
My attempts
I suspected I could use a thread pool in order to save some memory, and it did. By spawning 7 threads instead of 7 processes, CUDA was only initialized once, which saved almost half of my memory. However, this lead to a new problem in which the GPU only utilized approx. 30% utilization according to nvidia-smi that I am monitoring in a separate linux terminal. Without threads, I get around 85-90% utilization.
I also messed around with torch.multiprocessing.set_sharing_strategy which is currently set to 'file_descriptor', but with no luck.
My questions

Is there a better way to work with multiple model- and optimizer states without saving and loading them to files while training? I have tried to move the model to CPU using model.cpu() before saving the state_dict, but this did not work in my implementation (memory leaks).
Is there an efficient way I can train multiple neural networks at the same time that uses less GPU memory? When searching the web, I only find references to nn.DataParallel which trains the same model over multiple GPUs by copying it to each GPU. This does not work for my problem.

I will soon have access to multiple, more powerful GPUs with more memory, and I suspect this problem will be less annoying then, but I wouldn't be surprised if there is a better solution I am not getting.
Update (09.03.2020)
For any future readers, if you set out to do something similar to the pseudo code displayed above, and you plan on using multiple GPUs, please make sure to create one multiprocessing pool for each GPU device. Pools don't execute functions in order with the underlying processes it contains, and so you will end up initializing CUDA multiple times on the same process, wasting memory.
Another important note is that while you may be passing the device (e.g. 'cuda:1') to every torch.cuda-function, you may discover that torch does something with the default cuda device 'cuda:0' somewhere in the code, initializing CUDA on that device for every process, which wastes memory on an unwanted and non-needed CUDA initialization. I fixed this issue by using with torch.cuda.device(device_id) that encapsulate the entire training_function.
I ended up not using multiprocessing pools and instead defined my own custom process class that holds the device and training function. This means I have to maintain in-queues for each device-process, but they all share the same out-queue, meaning I can retrieve the results the moment they are available. I figured writing a custom process class was simpler than writing a custom pool class. I desperately tried to keep using pools as they are easily maintained, but I had to use multiple imap-functions, and so the results were not obtainable one at a time, which lead to a less efficient training-loop.
I am now successfully training on multiple GPUs, but my questions posted above still remains unanswered.

Update (10.03.2020)
I have implemented another way to store model- and optimizer statedicts outside of GPU RAM. I have written function that replaces every tensor in the dicts with it's .to('cpu') equivalent. This costs me some CPU memory, but it is more reliable than storing local files.
Update (11.06.2020)
I have still not found a different approach that leads to fewer CUDA initializations while maintaining the same processing speed. From what I've read and come to understand, PyTorch does not infer too much with how CUDA is operating, and leaves that up to NVIDIA.
I have ended up using a pool of custom, device-specific processes, called Workers, that is maintained by my custom pool class (more about this above). In addition, I let each of these Workers take in one or more checkpoints as well as the function that processes them (training, testing, hp optimization) via a Queue. These checkpoints are then processed simultaneously via a python multiprocessing ThreadPool in each Worker and the results are then returned one by one via the return Queue the moment they are ready.
This gives me the parallel procedure I was needing, but the memory issue is still there. Due to time constraints, I have come to terms with it for now.


title: A vector and matrix rows cosine similarity in pytorch

body: In pytorch, I have multiple (scale of hundred thousand) 300 dim vectors (which I think I should upload in a matrix), I want to sort them by their cosine similarity with another vector and extract the top-1000. I want to avoid for loop as it is time consuming. I was looking for an efficient solution.


title: CUDA vs. DataParallel: Why the difference?

body:
I have a simple neural network model and I apply either cuda() or DataParallel() on the model like following.

model = torch.nn.DataParallel(model).cuda()

OR,

model = model.cuda()

When I don't use DataParallel, rather simply transform my model to cuda(), I need to explicitly convert the batch inputs to cuda() and then give it to the model, otherwise it returns the following error.

  torch.index_select received an invalid combination of arguments - got (torch.cuda.FloatTensor, int, torch.LongTensor)

But with DataParallel, the code works fine. Rest of the other things are same. Why this happens? Why when I use DataParallel, I don't need to transform the batch inputs explicitly to cuda()?


title: pytorch can not get the device name

body:
torch.cuda._initialized=True
torch.cuda.is_available()

#return True

torch.cuda.set_device(0)
#i have only one gpu card

torch.cuda.current_device()
#return 0

#now comes to the error
torch.cuda.get_device_name(0)

  return NameError: name '_get_device_properties' is not defined

sometimes i could get the correct device name，but more often the NameError above was given.

if the error was given，I choose to continue to run dnn on gpu，the anaconda will report 'kernel died'

please tell me how to solve this error，thanks a lot!!


title: Using automatic differentiation libraries to compute partial derivatives of an arbitrary tensor

body:
(Note: this is not a question about back-propagation.)
I am trying so solve on a GPU a non-linear PDE using PyTorch tensors in place of Numpy arrays. I want to calculate the partial derivatives of an arbitrary tensor, akin to the action of the center finite-difference numpy.gradient function. I have other ways around this problem, but since I am already using PyTorch, I'm wondering if it is possible use the autograd module (or, in general, any other autodifferentiation module) to perform this action.

I have created a tensor-compatible version of the numpy.gradient function - which runs a lot faster. But perhaps there is a more elegant way of doing this. I can't find any other sources that address this question, either to show that it's possible or impossible; perhaps this reflects my ignorance with the autodifferentiation algorithms.


title: FileNotFoundError: Could not find module 'caffe2_nvrtc.dll' (or one of its dependencies). Try using the full path with constructor syntax

body:
I am using anaconda and created one virtual environment named myenv1. After activating my virtual environment, I have installed pytorch using following command.
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

But if I am trying to import torch in a filename.py it's giving following error.
 Could not find module 'C:\ProgramData\anaconda3\envs\myenv1\lib\site-packages\torch\lib\caffe2_nvrtc.dll' (or one of its dependencies). Try using the full path with constructor syntax.

After navigating to the above path which I am getting in the error console,I am able to see this caffe2_nvrtc.dll .


title: How to modify Pretrained Torchvision Models in PyTorch to return two outputs for multilabel Image Classification

body:
Input: Set of ten "Vowels", set of ten "Consonents", Image dataset where in every image both, one vowel and one consonent, are written.

Task: To identify the vowel and consonent from given image.

Approach: First apply CNN hidden layers on image, then apply two parallel fully connected/dense layers where one will classify vowel in image and other will classify consonent in image.

Problem: I am taking Pretrained Model like VGG or GoogleNet. How to modify that pretrained model to apply two parallel dense layers and return two outputs.

I have tried two different models but my query is can we modify pretrained model for this task.

Right now my model is having only one "fc" layer. I have modified number of neurons in final "fc" layer, like this

final_in_features = googlenet.fc.in_features

googlenet.fc = nn.Linear(final_in_features, 10)

But I need to add one more fc layer so that both "fc" layers are connecting with hidden layers parallel.

Right now model is returning only one output.

outputs1 = googlenet(inputs)

Task is to return two outputs from both "fc" layers, so that it should be look like this

outputs1, outputs2 = googlenet(inputs)