title: CUDA runtime unknown error, maybe a driver problem? CUDA can't see my gpu

body:
My code is very simple for now:

import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.cuda.current_device()

    ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-3380d2c12118> in <module>
----> 1 torch.cuda.current_device()

~/.conda/envs/tensorflow/lib/python3.6/site-packages/torch/cuda/__init__.py in current_device()
    349 def current_device():
    350     r"""Returns the index of a currently selected device."""
--> 351     _lazy_init()
    352     return torch._C._cuda_getDevice()
    353

~/.conda/envs/tensorflow/lib/python3.6/site-packages/torch/cuda/__init__.py in _lazy_init()
    161             "Cannot re-initialize CUDA in forked subprocess. " + msg)
    162     _check_driver()
--> 163     torch._C._cuda_init()
    164     _cudart = _load_cudart()
    165     _cudart.cudaGetErrorName.restype = ctypes.c_char_p

RuntimeError: cuda runtime error (30) : unknown error at /opt/conda/conda-bld/pytorch_1556653099582/work/aten/src/THC/THCGeneral.cpp:51

Looking in the internet it looks like it is a version problem, but I swear I tried all combinations of drivers from CUDA 10.0, 10.1, tensorflow-gpu 13, 12, etc. and nothing seems to work.

NVIDIA driver: nvidia-smi:

    +-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.14       Driver Version: 430.14       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 930MX       Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   36C    P8    N/A /  N/A |    139MiB /  2004MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0       986      G   /usr/lib/xorg/Xorg                            64MiB |
|    0      1242      G   /usr/bin/gnome-shell                          72MiB |
+-----------------------------------------------------------------------------+

CUDA VERSION nvcc --version :

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

tensorflow-gpu version: pip list | grep tensorflow:

tensorflow                         1.13.1
tensorflow-estimator               1.13.0

pytorch version pip list | grep torch

pytorch-pretrained-bert            0.6.2
torch                              1.1.0
torchvision                        0.3.0

Can anyone see a problem of compatibility and explain why and how I can fix it?


title: How can I cast a tensor to the complex type in Pytorch?

body: I want to do some quantum mechanics calculations with Pytorch, where the quantities are sometimes complex. I would like to know how can I cast an existing real tensor to the complex type.


title: How to fix 'Expected object of scalar type Float but got scalar type Double for argument #4 'mat1''?

body:
I am trying to build an lstm model. My model code is below.

My input has 4 features, Sequence length of 5 and batch size of 32.

        class RNN(nn.Module):

            def __init__(self, feature_dim, output_size, hidden_dim, n_layers, dropout=0.5):
                """
                Initialize the PyTorch RNN Module
                :param feature_dim: The number of input dimensions of the neural network
                :param output_size: The number of output dimensions of the neural network
                :param hidden_dim: The size of the hidden layer outputs
                :param dropout: dropout to add in between LSTM/GRU layers
                """
                super(RNN, self).__init__()

                # set class variables
                self.output_size = output_size
                self.n_layers = n_layers
                self.hidden_dim = hidden_dim

                # define model layers
                self.lstm = nn.LSTM(feature_dim, hidden_dim, n_layers, batch_first=True)

                self.fc = nn.Linear(hidden_dim, output_size)

                self.dropout = nn.Dropout(dropout)

            def forward(self, nn_input, hidden):
                """
                Forward propagation of the neural network
                :param nn_input: The input to the neural network
                :param hidden: The hidden state
                :return: Two Tensors, the output of the neural network and the latest hidden state
                """

                # Get Batch Size
                batch_size = nn_input.size(0)

                # Pass through LSTM layer
                lstm_out, hidden = self.lstm(nn_input, hidden)
                # Stack up LSTM outputs
                lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)
                # Add dropout and pass through fully connected layer
                x = self.dropout(lstm_out)
                x = self.fc(lstm_out)

                # reshape to be batch_size first
                output = x.view(batch_size, -1, self.output_size)
                # get last batch of labels
                out = output[:, -1]

                # return one batch of output word scores and the hidden state
                return out, hidden

            def init_hidden(self, batch_size):
                '''
                Initialize the hidden state of an LSTM/GRU
                :param batch_size: The batch_size of the hidden state
                :return: hidden state of dims (n_layers, batch_size, hidden_dim)
                '''
                # Implement function

                # initialize   state with zero weights, and move to GPU if available
                weight = next(self.parameters()).data

                if is_gpu_available:
                    hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().to(device),
                          weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().to(device))
                else:
                    hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                              weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())
                return hidden

When I train, I got the error

RuntimeError                              Traceback (most recent call last)
    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in <module>
          3
          4 # training the model
    ----> 5 trained_rnn = train_rnn(rnn, batch_size, optimizer, num_epochs, show_every_n_batches)
          6
          7 # saving the trained model

    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in train_rnn(rnn, batch_size, optimizer, n_epochs, show_every_n_batches)
         18
         19             # forward, back prop
    ---> 20             loss, hidden = forward_back_prop(rnn, optimizer, inputs, labels, hidden)
         21             # record loss
         22             batch_losses.append(loss)

    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in forward_back_prop(rnn, optimizer, inp, target, hidden)
         22
         23     # get the output from the model
    ---> 24     output, h = rnn(inp, h)
         25
         26    # calculate the loss and perform backprop

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
        491             result = self._slow_forward(*input, **kwargs)
        492         else:
    --> 493             result = self.forward(*input, **kwargs)
        494         for hook in self._forward_hooks.values():
        495             hook_result = hook(self, input, result)

    /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in forward(self, nn_input, hidden)
         36
         37         # Pass through LSTM layer
    ---> 38         lstm_out, hidden = self.lstm(nn_input, hidden)
         39         # Stack up LSTM outputs
         40         lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
        491             result = self._slow_forward(*input, **kwargs)
        492         else:
    --> 493             result = self.forward(*input, **kwargs)
        494         for hook in self._forward_hooks.values():
        495             hook_result = hook(self, input, result)

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
        557             return self.forward_packed(input, hx)
        558         else:
    --> 559             return self.forward_tensor(input, hx)
        560
        561

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in forward_tensor(self, input, hx)
        537         unsorted_indices = None
        538
    --> 539         output, hidden = self.forward_impl(input, hx, batch_sizes, max_batch_size, sorted_indices)
        540
        541         return output, self.permute_hidden(hidden, unsorted_indices)

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in forward_impl(self, input, hx, batch_sizes, max_batch_size, sorted_indices)
        520         if batch_sizes is None:
        521             result = _VF.lstm(input, hx, self._get_flat_weights(), self.bias, self.num_layers,
    --> 522                               self.dropout, self.training, self.bidirectional, self.batch_first)
        523         else:
        524             result = _VF.lstm(input, batch_sizes, hx, self._get_flat_weights(), self.bias,

    RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

I am not able to figure the cause of this error. How to fix it? Please help.

Also, is it the correct way of implementing the LSTM or is there a better way to achieve the same?


title: Filling torch tensor with zeros after certain index

body:
Given a 3d tenzor, say:
batch x sentence length x embedding dim

a = torch.rand((10, 1000, 96))

and an array(or tensor) of actual lengths for each sentence

lengths =  torch .randint(1000,(10,))

outputs tensor([ 370., 502., 652., 859., 545., 964., 566., 576.,1000., 803.])

How to fill tensor ‘a’ with zeros after certain index along dimension 1 (sentence length) according to tensor ‘lengths’ ?

I want smth like that :

a[ : , lengths : , : ]  = 0

One way of doing it (slow if batch size is big enough):

for i_batch in range(10):
    a[ i_batch  , lengths[i_batch ] : , : ]  = 0


title: How to do parallel processing in pytorch

body:
I am working on a deep learning problem. I am solving it using pytorch. I have two GPU's which are on the same machine (16273MiB,12193MiB). I want to use both the GPU's for my training (video dataset).

I get a warning:

There is an imbalance between your GPUs. You may want to exclude GPU 1 which
    has less than 75% of the memory or cores of GPU 0. You can do so by setting
    the device_ids argument to DataParallel, or by setting the CUDA_VISIBLE_DEVICES
    environment variable.
  warnings.warn(imbalance_warn.format(device_ids[min_pos], device_ids[max_pos]))

I also get an error:

raise TypeError('Broadcast function not implemented for CPU tensors')
TypeError: Broadcast function not implemented for CPU tensors

if __name__ == '__main__':

    opt.scales = [opt.initial_scale]
    for i in range(1, opt.n_scales):
        opt.scales.append(opt.scales[-1] * opt.scale_step)
    opt.arch = '{}-{}'.format(opt.model, opt.model_depth)
    opt.mean = get_mean(opt.norm_value)
    opt.std = get_std(opt.norm_value)
    print("opt",opt)
    with open(os.path.join(opt.result_path, 'opts.json'), 'w') as opt_file:
        json.dump(vars(opt), opt_file)

    torch.manual_seed(opt.manual_seed)

    model, parameters = generate_model(opt)
    #print(model)

    pytorch_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
    print("Total number of trainable parameters: ", pytorch_total_params)

    # Define Class weights
    if opt.weighted:
        print("Weighted Loss is created")
        if opt.n_finetune_classes == 2:
            weight = torch.tensor([1.0, 3.0])
        else:
            weight = torch.ones(opt.n_finetune_classes)
    else:
        weight = None

    criterion = nn.CrossEntropyLoss()
    if not opt.no_cuda:

        criterion = nn.DataParallel(criterion.cuda())

    if opt.no_mean_norm and not opt.std_norm:
        norm_method = Normalize([0, 0, 0], [1, 1, 1])
    elif not opt.std_norm:
        norm_method = Normalize(opt.mean, [1, 1, 1])
    else:
        norm_method = Normalize(opt.mean, opt.std)

        train_loader = torch.utils.data.DataLoader(
            training_data,
            batch_size=opt.batch_size,
            shuffle=True,
            num_workers=opt.n_threads,
            pin_memory=True)
        train_logger = Logger(
            os.path.join(opt.result_path, 'train.log'),
            ['epoch', 'loss', 'acc', 'precision','recall','lr'])
        train_batch_logger = Logger(
            os.path.join(opt.result_path, 'train_batch.log'),
            ['epoch', 'batch', 'iter', 'loss', 'acc', 'precision', 'recall', 'lr'])

        if opt.nesterov:
            dampening = 0
        else:
            dampening = opt.dampening
        optimizer = optim.SGD(
            parameters,
            lr=opt.learning_rate,
            momentum=opt.momentum,
            dampening=dampening,
            weight_decay=opt.weight_decay,
            nesterov=opt.nesterov)
        # scheduler = lr_scheduler.ReduceLROnPlateau(
        #     optimizer, 'min', patience=opt.lr_patience)
    if not opt.no_val:
        spatial_transform = Compose([
            Scale(opt.sample_size),
            CenterCrop(opt.sample_size),
            ToTensor(opt.norm_value), norm_method
        ])

    print('run')
    for i in range(opt.begin_epoch, opt.n_epochs + 1):
        if not opt.no_train:
            adjust_learning_rate(optimizer, i, opt.lr_steps)
            train_epoch(i, train_loader, model, criterion, optimizer, opt,
                        train_logger, train_batch_logger)

I have also made changes in my train file:

      model = nn.DataParallel(model(),device_ids=[0,1]).cuda()
        outputs = model(inputs)

It does not seem to work properly and is giving error. Please advice, I am new to pytorch.

Thanks


title: Why not super().__init__(Model,self) in Pytorch

body:
For torch.nn.Module()

According to the official documentation:
Base class for all neural network modules.
Your models should also subclass this class.
Modules can also contain other Modules, allowing to nest them in a tree structure. You can assign the submodules as regular attributes.

import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.conv1 = nn.Conv2d(1, 20, 5)
        self.conv2 = nn.Conv2d(20, 20, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        return F.relu(self.conv2(x))

It used super(Model, self).__init__()
Why not super().__init__(Model, self)


title: How to make a class in pytorch use GPU

body:
So I am running some code and getting the following error in Pytorch: "RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same"

From what I understand, this means that my model may not be pushed to the GPU, while the input data already is using the GPU. I can share my code if that would help (I am refraining from doing it right now since it is longer than a small code snippet).

I am aware that I can do something like

    myModel=Model()
    myModel.cuda()

However, I am making a class as part of a custom module that will be added to a Sequential wrapper. So, I can't really make an object out of it (I'm not good with OOP terminology, so I apologize for any technical writing mistakes). I was wondering if there is a way to get around this issue, and make the class always use the GPU, even though I never explicitly define an object?

If this was not clear enough, I can post my code, but as previously warned it may take some time to go through (not too long, but not very convenient either).

Any help is much appreciated.

Edit: Here is the code, I presume the issue is in the RLSTM class, since there was not an error before I added this.

class VGG(nn.Module):
'''
VGG model
'''
def __init__(self, features): # features represents the layers array
    super(VGG, self).__init__()
    self.features = features
    self.classifier = nn.Sequential(
        nn.Dropout(),
        nn.Linear(512,512),
        nn.ReLU(True),
        nn.Dropout(),
        nn.Linear(512, 512),
        nn.ReLU(True),
        nn.Linear(512, 10),
    )
     # Initialize weights
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
            m.weight.data.normal_(0, math.sqrt(2. / n))
            m.bias.data.zero_()

def forward(self, x): # x is the image, we run x through the layers
    print(x.size())
    x = self.features(x) # runs through all features, where each feature is a function
    x = x.view(x.size(0), -1)
    # after running through features, does sequential steps to finally classify
    x = self.classifier(x)
    # print(x)
    return x

def make_layers(cfg, batch_norm=False):
   # print("Making layers!")
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = v
            layers+=[RLSTM()]

    return nn.Sequential(*layers)

class RLSTM(nn.Module):
def __init__(self):
    super(RLSTM,self).__init__()

def forward(self, image):
    print("going in rowlstm")
    global current
    global _layer
    global isgates
    size = image.size()
    b = size[0]
    indvs = list(image.split(1,0)) # split up the batch into individual images
    #print(indvs[0].size())
    tensor_array = []
    for i in range(b):
        current = 0
        _layer = []
        isgates = []
        tensor_array.append(self.RowLSTM(indvs[i]))

    seq=tuple(tensor_array)
    trans = torch.cat(seq,0)
    return trans.cuda() # trying to make floattensor error go away
def RowLSTM(self, image):
    global current
    global _layer
    global isgates

    # input-to-state (K_is * x_i) : 3x1 convolution. generate 4h x n x n tensor. 4hxnxn tensor contains all i -> s info

# the input to state convolution should only be computed one time
    if current==0:
        n = image.size()[2]
        ch=image.size()[1]
        input_to_state = torch.nn.Conv2d(ch,4*ch,kernel_size=(1,3),padding=(0,1))
        isgates = self.splitIS(input_to_state(image)) # convolve, then split into gates (4 per row)
        cell=RowLSTMCell(0,torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1),torch.randn(ch,n,1))
        # now have dummy, learnable variables for first row
        _layer.append(cell)

    else:
        Cell_prev = _layer[current-1] # access previous row
        hidPrev = Cell_prev.getHiddenState()
        ch = image.size()[1]
    #   print("about to apply conv1d")
        state_to_state = torch.nn.Conv2d(ch,4*ch,kernel_size=(1,3),padding=(0,1)) # error is here: hidPrev is an array - not a valid number of input channel
    #   print("applied conv1d")
        prevHid=Cell_prev.getHiddenState()
        ssgates = self.splitSS(state_to_state(prevHid.unsqueeze(0))) #need to unsqueeze (Ex: currently 16x5, need to make 1x16x5)
        gates = self.addGates(isgates,ssgates,current)
        # split gates
        ig, og, fg, gg = gates[0], gates[1], gates[2], gates[3] # into four, ADD SIGMOID!
        cell = RowLSTMCell(Cell_prev,ig,og,fg,gg,0,0)
        cell.compute()
        _layer.append(cell)
    # attempting to eliminate requirement of getting size

    #print(current)
    try:

        current+=1
        y=(isgates[0][0][1][current])
        return self.RowLSTM(image)
    except Exception as error:
        concats=[]
        for cell in _layer:
            tensor=torch.unsqueeze(cell.h,0)

            concats.append(tensor)
        seq=tuple(concats)
        tensor=torch.cat(seq,3)
        return tensor

def splitIS(tensor): #always going to be splitting into 4 pieces, so no need to add extra parameters
    inputStateGates={}
    size=tensor.size() # 1 x 4h x n x n
    out_ft=size[1] # get 4h for the nxnx4h tensor
    num=size[2] # get n for the nxn image
    hh=out_ft/4 # we want to split the tensor into 4, for the gates
    tensor = torch.squeeze(tensor) # 4h x n x n

    # First, split by row: Creates n tensors of 4h x n x 1
    rows = list(tensor.split(1,2))

    for i in range(num):
        # Each row is a tensor of 4h x n x 1, split it into 4 of h x n x 1
        row=rows[i]
        inputStateGates[i]=list(row.split(hh,0))

    return inputStateGates

def splitSS(tensor): # 1 x 4h x n x 1, create 4 of 1 x h x n x 1
    size=tensor.size()
    out_ft=size[1] # get 4h for the 1x4hxn tensor
    num=size[2] # get n for the 1xhxn row
    hh=out_ft/4 # we want to split the tensor into 4, for the gates
    tensor = tensor.squeeze(0) # 4h x n x 1
    splitted=list(tensor.split(hh,0))
    return splitted

def addGates(i2s,s2s,key):
    """ these dictionaries are of form {key : [[i], [o], [f], [g]]}
        we want to add pairwise elemeents """

    # i2s is of form key: [[i], [o], [f], [g]] where each gate is hxn
    # s2s is of form [[h,n],[h,n],[h,n], [h,n]]
    gateSum = []
    for i in range(4): # always of length 4, representing the gates
        gateSum.append(torch.sigmoid(i2s[key][i] + s2s[i]))

    return gateSum


title: Visualizing Neural Network Layer Activation

body:
Feature visualizing in tensor flow or keras is easy and can be found here. https://machinelearningmastery.com/how-to-visualize-filters-and-feature-maps-in-convolutional-neural-networks/ or Convolutional Neural Network visualization - weights or activations?

how to do this in pytorch?

I am using PyTorch with pretrained resnet18 model. All i need to input the image and get activation for specific layer(e.g. Layer2.0.conv2). Layer2.0.conv2 is specified in the pretrained model.

In simple words; how to convert link one code to PyTorch? how to get the specific layers in resnet18 PyTorch and how to get the activation for input image.
I tried this in tensorflow and it worked but not PyTorch.


title: Custom cross-entropy loss in pytorch

body:
I have done a custom implementation of the pytorch cross-entropy loss function (as I need more flexibility to be introduced later). The model I intend to train with this will need a considerable amount of time to train and the resources available can't be used to merely test if the function is correct implementation. I have implemented vectorized implementation as it will be quicker to run.

Following is my code for the same:

def custom_cross(my_pred,true,batch_size=BATCH_SIZE):
    loss= -torch.mean(torch.sum(true.view(batch_size, -1) * torch.log(my_pred.view(batch_size, -1)), dim=1))
    return loss

I will really appreciate if you can suggest a more optimized implementation of the same or if I am making a mistake in the present one. The model will use a Nvidia Tesla K-80 to train.


I am trying to run a PyTorch implementation of a code, which is supposed to work on SBD dataset.

The training labels are originally available in .bin file, which are then converted to HDF5 (.h5) files.

Upon running the algorithm, I get an error as: " TypeError: h5py objects cannot be pickled "

I think the error is stemming from torch.utils.data.DataLoader.

Any idea if I am missing any concept here? I read that pickling is generally not preferred but as of now, my dataset is in HDF5 format only.

For your reference, the error's stack trace is as follows:

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\main.py", line 130, in <module>
    main()

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\main.py", line 85, in main
    win_feats5, win_fusion, viz, global_step)

  File "G:\My Drive\Debvrat - shared\Codes\CASENet PyTorch Implementations\SBD-lijiaman\train_val\model_play.py", line 31, in train
    for i, (img, target) in enumerate(train_loader):

  File "C:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 819, in __iter__
    return _DataLoaderIter(self)

  File "C:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 560, in __init__
    w.start()

  File "C:\Anaconda3\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)

  File "C:\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)

  File "C:\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)

  File "C:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)

  File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)

  File "C:\Anaconda3\lib\site-packages\h5py\_hl\base.py", line 308, in __getnewargs__
    raise TypeError("h5py objects cannot be pickled")

  TypeError: h5py objects cannot be pickled

I am using Conda version 4.8.2, Python 3.7.4, PyTorch 1.0.0 with Cuda 10.2.89

Thanks,


title: Difference between src_mask and src_key_padding_mask

body:
I am having a difficult time in understanding transformers. Everything is getting clear bit by bit but one thing that makes my head scratch is
what is the difference between src_mask and src_key_padding_mask which is passed as an argument in forward function in both encoder layer and decoder layer.

https://pytorch.org/docs/master/_modules/torch/nn/modules/transformer.html#Transformer


title: how to modify resnet 50 with 4 channels as input using pre-trained weights in Pytorch?

body:
I would like to change the resnet50 so that I can switch to 4 channel input,
use the same weights for the rgb channels and initialize the last channel with a normal with mean 0 and variance 0.01.
here is my code:
import torch.nn as nn
import torch
from torchvision import models

from misc.layer import Conv2d, FC

import torch.nn.functional as F
from misc.utils import *

import pdb

class Res50(nn.Module):
    def __init__(self,  pretrained=True):
        super(Res50, self).__init__()

        self.de_pred = nn.Sequential(Conv2d(1024, 128, 1, same_padding=True, NL='relu'),
                                     Conv2d(128, 1, 1, same_padding=True, NL='relu'))

        self._initialize_weights()

        res = models.resnet50(pretrained=pretrained)
        pretrained_weights = res.conv1.weight

        res.conv1 = nn.Conv2d(4, 64, kernel_size=7, stride=2, padding=3,bias=False)

        res.conv1.weight[:,:3,:,:] = pretrained_weights
        res.conv1.weight[:,3,:,:].data.normal_(0.0, std=0.01)

        self.frontend = nn.Sequential(
            res.conv1, res.bn1, res.relu, res.maxpool, res.layer1, res.layer2
        )

        self.own_reslayer_3 = make_res_layer(Bottleneck, 256, 6, stride=1)
        self.own_reslayer_3.load_state_dict(res.layer3.state_dict())

    def forward(self,x):
        x = self.frontend(x)
        x = self.own_reslayer_3(x)
        x = self.de_pred(x)
        x = F.upsample(x,scale_factor=8)
        return x

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                m.weight.data.normal_(0.0, std=0.01)
                if m.bias is not None:
                    m.bias.data.fill_(0)
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.fill_(1)
                m.bias.data.fill_(0)

but it produces the following error, does anyone have any advice?
/usr/local/lib/python3.6/dist-packages/torch/tensor.py:746: UserWarning: The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad attribute won't be populated during autograd.backward(). If you indeed want the gradient for a non-leaf Tensor, use .retain_grad() on the non-leaf Tensor. If you access the non-leaf Tensor by mistake, make sure you access the leaf Tensor instead. See github.com/pytorch/pytorch/pull/30531 for more informations.
  warnings.warn("The .grad attribute of a Tensor that is not a leaf Tensor is being accessed. Its .grad "
Traceback (most recent call last):
  File "train.py", line 62, in <module>
    cc_trainer = Trainer(loading_data,cfg_data,pwd)
  File "/content/drive/My Drive/Folder/Code/trainer.py", line 28, in __init__
    self.optimizer = optim.Adam(self.net.CCN.parameters(), lr=cfg.LR, weight_decay=1e-4) #remenber was 1e-4
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/adam.py", line 44, in __init__
    super(Adam, self).__init__(params, defaults)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 51, in __init__
    self.add_param_group(param_group)
  File "/usr/local/lib/python3.6/dist-packages/torch/optim/optimizer.py", line 206, in add_param_group
    raise ValueError("can't optimize a non-leaf Tensor")
ValueError: can't optimize a non-leaf Tensor


title: Implementing a many-to-many regression task

body:
Sorry if I present my problem not clearly, English is not my first language

Problem

Short description:

I want to train a model which map input x (with shape of [n_sample, timestamp, feature]) to an output y (with exact same shape). It's like mapping 2 space

Longer version:

I have 2 float ndarrays of shape [n_sample, timestamp, feature], representing MFCC feature of n_sample audio file. These 2 ndarray are 2 speakers' speech of the same corpus, which was aligned by DTW.  Lets name these 2 arrays x and y. I want to train a model, which predict y[k] given x[k]. It's like mapping from space x to space y, and the output must be exact same shape as the input

What I've tried

It's time-series problem so I decide to use RNN approach. Here is my code in PyTorch (I put comment along the code. I removed the calculation of average loss for simplicity). Note that I've tried many option for learning rate, the behavior still the same

Class define

class Net(nn.Module):
    def __init__(self, in_size, hidden_size, out_size, nb_lstm_layers):
        super().__init__()
        self.in_size = in_size
        self.hidden_size = hidden_size
        self.out_size = out_size
        self.nb_lstm_layers = nb_lstm_layers

        # self.fc1 = nn.Linear()
        self.lstm = nn.LSTM(input_size=self.in_size, hidden_size=self.hidden_size, num_layers=self.nb_lstm_layers, batch_first=True, bias=True)
        # self.fc = nn.Linear(self.hidden_size, self.out_size)
        self.fc1 = nn.Linear(self.hidden_size, 128)
        self.fc2 = nn.Linear(128, 128)
        self.fc3 = nn.Linear(128, self.out_size)

    def forward(self, x, h_state):
        out, h_state = self.lstm(x, h_state)
        output_fc = []

        for frame in out:
            output_fc.append(self.fc3(torch.tanh(self.fc1(frame)))) # I added fully connected layer to each frame, to make an output with same shape as input

        return torch.stack(output_fc), h_state

    def hidden_init(self):
        if use_cuda:
            h_state = torch.stack([torch.zeros(nb_lstm_layers, batch_size, 20) for _ in range(2)]).cuda()
        else:
            h_state = torch.stack([torch.zeros(nb_lstm_layers, batch_size, 20) for _ in range(2)])

        return h_state

Training step:

net = Net(20, 20, 20, nb_lstm_layers)
optimizer = optim.Adam(net.parameters(), lr=0.0001, weight_decay=0.0001)
criterion = nn.MSELoss()

for epoch in range(nb_epoch):
    count = 0
    loss_sum = 0

    batch_x = None
    for i in (range(len(data))):
    # data is my entire data, which contain A and B i specify above.
        temp_x = torch.tensor(data[i][0])
        temp_y = torch.tensor(data[i][1])

        for ii in range(0, data[i][0].shape[0] - nb_frame_in_batch*2 + 1): # Create batches
            batch_x, batch_y = get_batches(temp_x, temp_y, ii, batch_size, nb_frame_in_batch)
            # this will return 2 tensor of shape (batch_size, nb_frame_in_batch, 20),
            # with `batch_size` is the number of sample each time I feed to the net,
            # nb_frame_in_batch is the number of frame in each sample
            optimizer.zero_grad()

            h_state = net.hidden_init()

            prediction, h_state = net(batch_x.float(), h_state)
            loss = criterion(prediction.float(), batch_y.float())

            h_state = (h_state[0].detach(), h_state[1].detach())

            loss.backward()
            optimizer.step()

Problem is, the loss seems not to decrease but fluctuate a lot, without a clear behaviour

Please help me. Any suggestion will be greatly appreciated. If somebody can inspect my code and provide some comment, that would be so kind.
Thanks in advance!


title: How to use tensor.item() ? IndexError: invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number

body:
Im pretty new to Siamese Neural Networks and recently found this example and Colab notebook.

When running the code i get the following error:

  IndexError: invalid index of a 0-dim tensor. Use tensor.item() to
  convert a 0-dim tensor to a Python number

on the line:

result=torch.max(res,1)[1][0][0][0].data[0].tolist()

I found something about the tensor.item() but i really just don't know how to use it here.

EDIT:

test_dataloader = DataLoader(test_dataset,num_workers=6,batch_size=1,shuffle=True)
accuracy=0
counter=0
correct=0
for i, data in enumerate(test_dataloader,0):
x0, x1 , label = data
# onehsot applies in the output of 128 dense vectors which is then  converted to 2 dense vectors
output1,output2 = model(x0.to(device),x1.to(device))
res=torch.abs(output1.cuda() - output2.cuda())
label=label[0].tolist()
label=int(label[0])
result=torch.max(res,1)[1][0][0][0].data.item().tolist()
if label == result:
correct=correct+1
counter=counter+1
#   if counter ==20:
#      break

accuracy=(correct/len(test_dataloader))*100
print("Accuracy:{}%".format(accuracy))

Thats the code i get the error on.


title: How to construct a 3D Tensor where every 2D sub tensor is a diagonal matrix in PyTorch?

body:
Consider I have 2D Tensor, index_in_batch * diag_ele.
How can I get a 3D Tensor index_in_batch * Matrix (who is a diagonal matrix, construct by drag_ele)?

The torch.diag() construct diagonal matrix only when input is 1D, and return diagonal element when input is 2D.


title: How can I convert pytorch cpu-based transformation to cuda-based?

body:
The original issue for the code is availablehere.

I am using this repository for a line segmentation project and I developed this code to get an input (whether image or video) and draw road lines on it and give it in output:

import argparse
import sys
from time import time, clock
from os.path import splitext, basename, exists

from model import SCNN
from utils.check_extension import is_video, is_image
from utils.transforms import *
# I will put all the necessary code for utils.transforms after this

# ------------------------------------------------  SCNN parameters
time1 = time()
net = SCNN(input_size=(800, 288), pretrained=False)
mean = (0.3598, 0.3653, 0.3662)  # CULane mean, std
std = (0.2573, 0.2663, 0.2756)
transform_img = Resize((800, 288))
transform_to_net = Compose(ToTensor(), Normalize(mean=mean, std=std))

# ------------------------------------------------  Arguments

def parse_args():
    parser = argparse.ArgumentParser()

    parser.add_argument('--weights', type=str,
                        default='models/vgg_SCNN_DULR_w9.pth',
                        help='path to vgg models')

    parser.add_argument('--input', type=str, default='demo/line_3.mp4',
                        help='path to image file')

    parser.add_argument('--output', type=str, default='public/',
                        help='path to the output directory')
    args = parser.parse_args()

    return args

def main():
    args = parse_args()
    filename, extension = splitext(basename(args.input))

    print("Loading file [{}] ....".format(filename))
    if not exists(args.input):
        print("file [{}] is not recognized".format(args.input))
        sys.exit()

    if is_video(extension):
        video_capture = cv2.VideoCapture()
        fourcc = cv2.VideoWriter_fourcc(*'XVID')
        output = args.output + filename + '.avi'

        if video_capture.open(args.input):
            property_id = int(cv2.CAP_PROP_FRAME_COUNT)
            total_frames = int(cv2.VideoCapture.get(video_capture, property_id))
            frame_no = 1
            width, height = int(video_capture.get(cv2.CAP_PROP_FRAME_WIDTH)), \
                            int(video_capture.get(cv2.CAP_PROP_FRAME_HEIGHT))
            fps = video_capture.get(cv2.CAP_PROP_FPS)
            device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
            save_dict = torch.load(args.weights, map_location=device)
            net.load_state_dict(save_dict['net'])
            net.eval()

            # can't write out mp4, so try to write into an AVI file
            video_writer = cv2.VideoWriter(output, fourcc, fps, (width, height))
            while video_capture.isOpened():
                start = time()
                ret, frame = video_capture.read()
                if not ret:
                    break
                frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
                frame = transform_img({'img': frame})['img']
                x = transform_to_net({'img': frame})['img']
                x.unsqueeze_(0)

                stop1 = time()
                print('stop1: ', stop1 - start)

                seg_pred, exist_pred = net(x)[:2]
                seg_pred = seg_pred.detach().cpu().numpy()
                exist_pred = exist_pred.detach().cpu().numpy()
                seg_pred = seg_pred[0]

                stop2 = time()
                print('stop2: ', stop2 - stop1)

                frame = cv2.cvtColor(frame, cv2.COLOR_RGB2BGR)
                lane_img = np.zeros_like(frame)
                color = np.array([[255, 125, 0], [0, 255, 0], [0, 0, 255], [0, 255, 255]], dtype='uint8')
                coord_mask = np.argmax(seg_pred, axis=0)
                for i in range(0, 4):
                    if exist_pred[0, i] > 0.5:
                        lane_img[coord_mask == (i + 1)] = color[i]
                img = cv2.addWeighted(src1=lane_img, alpha=0.8, src2=frame, beta=1., gamma=0.)
                img = cv2.resize(img, (width, height))

                stop3 = time()
                print('stop3: ', stop3 - stop2)

                # if frame_no % 20 == 0:
                #     print('# {}/{} frames processed!'.format(frame_no, total_frames))
                frame_no += 1
                video_writer.write(img)
                end = time()
                print('Whole loop: {} seconds'.format(end - start))
                print('------------')
                print('------------')

            print('# All frames processed ')

            video_capture.release()
            video_writer.release()

    elif is_image(extension):
        img = cv2.imread(args.input)
        height, width, _ = img.shape
        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        img = transform_img({'img': img})['img']
        x = transform_to_net({'img': img})['img']
        x.unsqueeze_(0)

        save_dict = torch.load(args.weights, map_location='cpu')
        net.load_state_dict(save_dict['net'])
        net.eval()

        seg_pred, exist_pred = net(x)[:2]
        seg_pred = seg_pred.detach().cpu().numpy()
        exist_pred = exist_pred.detach().cpu().numpy()
        seg_pred = seg_pred[0]

        img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
        lane_img = np.zeros_like(img)
        color = np.array([[255, 125, 0], [0, 255, 0], [0, 0, 255], [0, 255, 255]], dtype='uint8')
        coord_mask = np.argmax(seg_pred, axis=0)
        for i in range(0, 4):
            if exist_pred[0, i] > 0.5:
                lane_img[coord_mask == (i + 1)] = color[i]
        img = cv2.addWeighted(src1=lane_img, alpha=0.8, src2=img, beta=1., gamma=0.)
        img = cv2.resize(img, (width, height))
        output = args.output + filename + '.jpg'

        cv2.imwrite(output, img)

    else:
        print("file format [{}] is not supported".format(args.input))
        sys.exit()

if __name__ == '__main__':
    main()

The code which belong to Resize, ToTensor, Normalize, Compose are here:

class Compose(CustomTransform):
    """
    All transform in Compose should be able to accept two non None variable, img and boxes
    """
    def __init__(self, *transforms):
        self.transforms = [*transforms]

    def __call__(self, sample):
        for t in self.transforms:
            sample = t(sample)
        return sample

    def __iter__(self):
        return iter(self.transforms)

    def modules(self):
        yield self
        for t in self.transforms:
            if isinstance(t, Compose):
                for _t in t.modules():
                    yield _t
            else:
                yield t

class Normalize(CustomTransform):
    def __init__(self, mean, std):
        self.transform = Normalize_th(mean, std)

    def __call__(self, sample):
        img = sample.get('img')

        img = self.transform(img)

        _sample = sample.copy()
        _sample['img'] = img
        return _sample

class ToTensor(CustomTransform):
    def __init__(self, dtype=torch.float):
        self.dtype=dtype

    def __call__(self, sample):
        img = sample.get('img')
        segLabel = sample.get('segLabel', None)
        exist = sample.get('exist', None)

        img = img.transpose(2, 0, 1)
        img = torch.from_numpy(img).type(self.dtype) / 255.
        if segLabel is not None:
            segLabel = torch.from_numpy(segLabel).type(torch.long)
        if exist is not None:
            exist = torch.from_numpy(exist).type(torch.float32)  # BCEloss requires float tensor

        _sample = sample.copy()
        _sample['img'] = img
        _sample['segLabel'] = segLabel
        _sample['exist'] = exist
        return _sample

class Resize(CustomTransform):
    def __init__(self, size):
        if isinstance(size, int):
            size = (size, size)
        self.size = size  #(W, H)

    def __call__(self, sample):
        img = sample.get('img')
        segLabel = sample.get('segLabel', None)

        img = cv2.resize(img, self.size, interpolation=cv2.INTER_CUBIC)
        if segLabel is not None:
            segLabel = cv2.resize(segLabel, self.size, interpolation=cv2.INTER_NEAREST)

        _sample = sample.copy()
        _sample['img'] = img
        _sample['segLabel'] = segLabel
        return _sample

    def reset_size(self, size):
        if isinstance(size, int):
            size = (size, size)
        self.size = size

The code works fine but I found out that its too slow for testing in real-time application. I added some time measurement to see if I can find out the bottlenecks and this is the output for one loop:

------------
stop1:  0.002989053726196289
stop2:  1.4032211303710938
stop3:  0.004946708679199219
Whole loop: 1.41636061668396 seconds

These lines happened to be the most computationally expensive lines:

seg_pred, exist_pred = net(x)[:2]
seg_pred = seg_pred.detach().cpu().numpy()
exist_pred = exist_pred.detach().cpu().numpy()
seg_pred = seg_pred[0]

Now I am stuck with this issue that how I can modify the code to improve the computation speed.

Initially I thought of modifying the code to allow cuda computation. I asked the main author how I can modify the code for cuda version in here and he pointed out to these lines:

frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frame = transform_img({'img': frame})['img']
x = transform_to_net({'img': frame})['img']
x.unsqueeze_(0)

Unfortunately my experience with pytorch is not much, so I am asking for help now.

I hope the information I shared suffices for the readers. Any help would be appreciated

Thanks


title: Flatten Tensor in Pytorch Convolutional Neural Network (size mismatch error)

body:
I made a reproducible example with random pixels. I'm trying to flatten the tensors for the dense layers after the convolutional layers. The problem is at the intersection of the convolutional layers and the dense layers. I don't know how to put the right number of neurons.

tl;dr I'm looking for the manual equivalent of keras.layers.Flatten() since it doesn't exist in pytorch.

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader

x = np.random.rand(1_00, 3, 100, 100)
y = np.random.randint(0, 2, 1_00)

if torch.cuda.is_available():
    x = torch.from_numpy(x.astype('float32')).cuda()
    y = torch.from_numpy(y.astype('float32')).cuda()

class ConvNet(nn.Module):

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(3, 32, 3)
        self.conv2 = nn.Conv2d(32, 64, 3)
        self.conv3 = nn.Conv2d(64, 128, 3)

        self.fc1 = nn.Linear(128, 1024) # 128 is wrong here
        self.fc2 = nn.Linear(1024, 1)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = torch.sigmoid(self.fc2(x))
        return x

net = ConvNet()
net.cuda()
optimizer = optim.Adam(net.parameters(), lr=0.03)
loss_function = nn.BCELoss()

class Train:

    def __init__(self):
        self.len = x.shape[0]
        self.x_train = x
        self.y_train = y

    def __getitem__(self, index):
        return x[index], y[index].unsqueeze(0)

    def __len__(self):
        return self.len

train = Train()
train_loader = DataLoader(dataset=train, batch_size=64, shuffle=True)

epochs = 1
train_losses = list()
for e in range(epochs):
    running_loss = 0
    for images, labels in train_loader:
        optimizer.zero_grad()
        log_ps = net(images)
        loss = loss_function(log_ps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    print('It\'s working.')


title: How to add a Validation and Test Set in Pytorch Model?

body:
I have Non Linear Regression Model ANN( X = [1000,3] , Y = [1000,8] ) with One hidden Layer(Nh = 6).

How to add a Validation(10% Dataset) and Test Set(10% Dataset) in this model ?

Model :

N, D_in, H, D_out = x.shape[0], x.shape[1], 6, y.shape[1]

model = nn.Sequential(OrderedDict([ ('fc1', nn.Linear(D_in, H)),
                                    #('Sig', nn.Sigmoid()),
                                    ('ISRU', ISRU()), # Add ISRU
                                    ('fc2', nn.Linear(H, D_out))]))

# Error -----
loss_fn = torch.nn.L1Loss(reduction='mean')

# Train -----
optimizer = torch.optim.Adam(model.parameters(), lr=1,eps=2**(-EPS))
epoch = 250
for t in range(epoch):
    # Forward pass: compute predicted y by passing x to the model.
    clear_output(wait=True)
    y_pred = model(X)

    # Compute and print loss.
    loss = loss_fn(y_pred, Y)
    if t % 100 == 99:
        print(t, loss.item())

    optimizer.zero_grad() ;
    loss.backward() ;
    optimizer.step() ;
if loss.item() < diff : lista = np.vstack((lista, [loss.item(),2,EPS])) ; diff = loss.item()


UPDATE: after looking back on this question, most of the code was unnecessary. To make a long story short, the hidden layer of a Pytorch RNN needs to be a torch tensor. When I posted the question, the hidden layer was a tuple.
Below are my data loader and model.
from torch.utils.data import TensorDataset, DataLoader

def batch_data(log_returns, sequence_length, batch_size):
    """
    Batch the neural network data using DataLoader
    :param log_returns: asset's daily log returns
    :param sequence_length: The sequence length of each batch
    :param batch_size: The size of each batch; the number of sequences in a batch
    :return: DataLoader with batched data
    """

    # total number of batches we can make
    n_batches = len(log_returns)//batch_size

    # Keep only enough characters to make full batches
    log_returns = log_returns[:n_batches * batch_size]

    y_len = len(log_returns) - sequence_length

    x, y = [], []
    for idx in range(0, y_len):
        idx_end = sequence_length + idx
        x_batch = log_returns[idx:idx_end]
        x.append(x_batch)
        # only making predictions after the last word in the batch
        batch_y = log_returns[idx_end]
        y.append(batch_y)

    # create tensor datasets
    x_tensor = torch.from_numpy(np.asarray(x))
    y_tensor = torch.from_numpy(np.asarray(y))

    # make x_tensor 3-d instead of 2-d
    x_tensor = x_tensor.unsqueeze(-1)

    data = TensorDataset(x_tensor, y_tensor)

    data_loader = DataLoader(data, shuffle=False, batch_size=batch_size)

    # return a dataloader
    return data_loader

    def init_hidden(self, batch_size):
        ''' Initializes hidden state '''
        # Create two new tensors with sizes n_layers x batch_size x n_hidden,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data

        if (train_on_gpu):
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda(),
                      weight.new(self.n_layers, batch_size, self.n_hidden).zero_().cuda())
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_(),
                      weight.new(self.n_layers, batch_size, self.n_hidden).zero_())

        return hidden

I don't know what is wrong. When I try to start training the model, I am getting the error message:
AttributeError: 'tuple' object has no attribute 'size'


title: RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.DoubleTensor for argument #2 'weight'

body:
I have been trying to retrain a model but unfortunately the last 2 days I keep getting the same error.

Could you please help a little bit with this one?

Initial work:

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
import numpy as np
import time

import torch
from torch import nn
from torch import optim
import torch.nn.functional as F
from torch.autograd import Variable
from torchvision import datasets, transforms
import torchvision.models as models
from collections import OrderedDict

Datasets:

data_dir = 'flowers'
train_dir = data_dir + '/train'

data_dir = 'flowers'

train_transforms = transforms.Compose([transforms.Resize(224),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomRotation(45),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406],
                                                            [0.229, 0.224, 0.225])])

train_data = datasets.ImageFolder(train_dir, transform=train_transforms)

trainloader = torch.utils.data.DataLoader(train_data, batch_size=32, shuffle=True)

import json

with open('cat_to_name.json', 'r') as f:
    cat_to_name = json.load(f)

Tried to use a pretrained model and train only the classifier:

# Load a pretrained model
model = models.vgg16(pretrained=True)

# Keep the parameters the same
for param in model.parameters():
    param.requires_grad = False

# and final output 102, since tht we have 102 flowers.
classifier = nn.Sequential(OrderedDict([
                          ('fc1', nn.Linear(25088, 4096)),
                          ('relu', nn.ReLU()),
                          ('fc3', nn.Linear(4096, 102)),
                          ('output', nn.LogSoftmax(dim=1))
                          ]))

# Replace model's old classifier with the new classifier
model.classifier = classifier

# Calculate the loss
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

model.to('cuda')

epochs = 1
print_every = 40
steps = 0

for e in range(epochs):
    running_loss = 0
    model.train()
   # model = model.double()
    for images, labels in iter(trainloader):
        steps += 1

        images.resize_(32, 3, 224, 224)

        inputs = Variable(images.to('cuda'))
        targets = Variable(labels.to('cuda'))

        optimizer.zero_grad()

        # Forward and backward passes
        output = model.forward(images)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()

        #running_loss += loss.data[0]
        running_loss += loss.item()

        if steps % print_every == 0:
            print("Epoch: {}/{}... ".format(e+1, epochs),
                  "Loss: {:.4f}".format(running_loss/print_every))

Error message:

  RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.DoubleTensor for argument #2 weight


title: Saving and reload huggingface fine-tuned transformer

body:
I am trying to reload a fine-tuned DistilBertForTokenClassification model. I am using transformers 3.4.0 and pytorch version 1.6.0+cu101. After using the Trainer to train the downloaded model, I save the model with trainer.save_model() and in my trouble shooting I save in a different directory via model.save_pretrained(). I am using Google Colab and saving the model to my Google drive. After testing the model I also evaluated the model on my test getting great results, however, when I return to the notebook (or Factory restart the colab notebook) and try to reload the model, the predictions are terrible. Upon checking the directories, the config.json file is there as is the pytorch_mode.bin. Below is the full code.
from transformers import DistilBertForTokenClassification

# load the pretrained model from huggingface
#model = DistilBertForTokenClassification.from_pretrained('distilbert-base-cased', num_labels=len(uniq_labels))
model = DistilBertForTokenClassification.from_pretrained('distilbert-base-uncased', num_labels=len(uniq_labels))

model.to('cuda');

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir = model_dir +  'mitmovie_pt_distilbert_uncased/results',          # output directory
    #overwrite_output_dir = True,
    evaluation_strategy='epoch',
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir = model_dir +  'mitmovie_pt_distilbert_uncased/logs',            # directory for storing logs
    logging_steps=10,
    load_best_model_at_end = True
)

trainer = Trainer(
    model = model,                         # the instantiated  Transformers model to be trained
    args = training_args,                  # training arguments, defined above
    train_dataset = train_dataset,         # training dataset
    eval_dataset = test_dataset             # evaluation dataset
)

trainer.train()

trainer.evaluate()

model_dir = '/content/drive/My Drive/Colab Notebooks/models/'
trainer.save_model(model_dir + 'mitmovie_pt_distilbert_uncased/model')

# alternative saving method and folder
model.save_pretrained(model_dir + 'distilbert_testing')

Coming back to the notebook after restarting...
from transformers import DistilBertForTokenClassification, DistilBertConfig, AutoModelForTokenClassification

# retreive the saved model
model = DistilBertForTokenClassification.from_pretrained(model_dir + 'mitmovie_pt_distilbert_uncased/model',
                                                        local_files_only=True)

model.to('cuda')

Model predictions are terrible now from either directory, however, the model does work and outputs the number of classes I would expect, it appears that the actual trained weights have not been saved or are somehow not getting loaded.


title: mat1 dim 1 must match mat2 dim 0 - PyTorch

body:
Im new to PyTorch and I keep getting the error mat1 dim1 must match mat1 dim0
this is my code for the network
class Net(Module):
    def __init__(self):
        super(Net, self).__init__()

        self.cnn_layers = Sequential(
            Conv1d(4,4,kernel_size=2, stride=1, padding=1),
            BatchNorm1d(4),
            ReLU(inplace=True),
            MaxPool1d(kernel_size=2,stride=1),
        )

        self.linear_layers = Sequential(
            Linear(8267*4,2)
        )

    def forward(self, x):
        print(x.shape)
        x = self.cnn_layers(x)
        print(x.shape)
        x = self.linear_layers(x)
        return x

and where the print statements are:
torch.Size([8267, 4, 1])
torch.Size([8267, 4, 1])

Any help/advice?


title: 'no module named cv2' import error on pytorch

body:
I am currently studying Pytorch and trying to use the cv2 module. I am using Jupyter notebook and Windows.

I have installed opencv like this:

!pip install opencv-python

When I choose the kernel (by change kernel option) Python3 and import cv2 then there is no problem.

But when I choose the kernel Pytorch and import cv2, then there is an error:

ModuleNotFoundError: No module named 'cv2'

This must be a basic question but I can't find out what the problem is.


title: How can I apply a transformation to a torch tensor

body:
I have a Torch Tensor z and I would like to apply a transformation matrix mat to z and have the output be exactly the same size as z. Here is the code I am running:
def trans(z):
    print(z)
    mat = transforms.Compose([transforms.ToPILImage(),transforms.RandomRotation(90),transforms.ToTensor()])
    z = Variable(mat(z.cpu()).cuda())
    z = nnf.interpolate(z, size=(28, 28), mode='linear', align_corners=False)
    return z
z = trans(z)

However, I get this error:
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-e2fc36889ba5> in <module>()
      3 inputs,targs=next(iter(tst_loader))
      4 recon, mean, var = vae.predict(model, inputs[img_idx])
----> 5 out = vae.generate(model, mean, var)

4 frames
/content/vae.py in generate(model, mean, var)
     90     z = trans(z)
     91     z = Variable(z.cpu().cuda())
---> 92     out = model.decode(z)
     93     return out.data.cpu()
     94

/content/vae.py in decode(self, z)
     56
     57     def decode(self, z):
---> 58         out = self.z_develop(z)
     59         out = out.view(z.size(0), 64, self.z_dim, self.z_dim)
     60         out = self.decoder(out)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/linear.py in forward(self, input)
     89
     90     def forward(self, input: Tensor) -> Tensor:
---> 91         return F.linear(input, self.weight, self.bias)
     92
     93     def extra_repr(self) -> str:

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1674         ret = torch.addmm(bias, input, weight.t())
   1675     else:
-> 1676         output = input.matmul(weight.t())
   1677         if bias is not None:
   1678             output += bias

RuntimeError: mat1 dim 1 must match mat2 dim 0

How can I successfully apply this rotation transform mat and not get any errors doing so?
Thanks,
Vinny


title: pytorch save a minibatch of 4D tensor as images

body:
I have set of 8 tensors in 4D tensor of shape (B, C, H, W). More specifically, the shape of my tensor is (8,3,64,64) i.e 8 images with [3x64x64] format.

I would like to save these in 1.png, 2.png ..., 8.png etc. When I try torchvision.utils.save_images(my_tensor), this is working fine. However, that is saving the images in a single grid instead of individual images.

How can I fix this issue?

I tried below code but that did not work.

for i in range(tensor.size(0)):

        np_data = tensor.cpu().numpy()
        imgplot = plt.imshow(np_data)


title: Windows 10,CUDA 9,: CUDA driver version is insufficient for CUDA runtime version at ..\src\THC\THCG

body:
My env configuration:

python:3.6
tensorflow-GPU 1.3
CUDA:9.0
VS:2013
torch:0.4.0

run CUDA 9.0 sample:success

but when I run the pytorch code,I get the error info as follows:

File "D:\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 249, in
    return self._apply(lambda t: t.cuda(device))
RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at ..\src\THC\THCGeneral.cpp:70

I have tried reinstall CUDA,but the error still exists


title: Why does PyTorch require to retain graph?

body:
I train my model like this:

for i in range(5):
  optimizer.zero_grad()
  y = next_input()
  loss = model(y)
  loss.backward()
  optimizer.step()

and get this error

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

Why does it require me to retain the graph? It could just recalculate derivatives if they are freed. To prove it consider the code:

for i in range(5):
  optimizer.zero_grad()
  model.zero_grad() # drop derivatives
  y = next_input()
  loss = model(y)
  loss.backward(retain_graph=True)
  optimizer.step()

In this case the derivatives from the previous iteration are also zeroed, but Torch doesn't care because the flag retain_graph=True is set.

Am I right that model.zero_grad() cancels the effect of retain_graph=True (i.e. drops the retained derivatives)?


title: PyTorch: get input layer size

body:
I want to programatically find the size of my input layer.

If my first layer is called fc1, how do I find out its input?


title: How to convert a tensor of booleans to ints in PyTorch?

body:
Suppose, we have a tensor

t = torch.tensor([True, False, True, False])

How do we convert it to an integer tensor with values [1, 0, 1, 0]?


title: Linear Regression with CNN using Pytorch: input and target shapes do not match: input [400 x 1], target [200 x 1]

body:
Let me explain the objective first. Let's say I have 1000 images each with an associated quality score [in range of 0-10]. Now, I am trying to perform the image quality assessment using CNN with regression(in PyTorch). I have divided the images into equal size patches. Now, I have created a CNN network in order to perform the linear regression.

Following is the code:

class MultiLabelNN(nn.Module):
    def __init__(self):
        super(MultiLabelNN, self).__init__()
        self.conv1 = nn.Conv2d(1, 32, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(32, 64, 5)
        self.fc1 = nn.Linear(3200,1024)
        self.fc2 = nn.Linear(1024, 512)
        self.fc3 = nn.Linear(512, 1)

    def forward(self, x):
        x = self.conv1(x)
        x = F.relu(x)
        x = self.pool(x)
        x = self.conv2(x)
        x = F.relu(x)
        x = x.view(-1, 3200)
        x = self.fc1(x)
        x = F.relu(x)
        x = self.fc2(x)
        x = F.relu(x)
        x = self.fc3(x)
        return x

While running this code of network I am getting following error

input and target shapes do not match: input [400 x 1], target [200 x 1]

the target shape is [200x1] is because I have taken the batch size of 200. I found the solution that if I change "self.fc1 = nn.Linear(3200,1024)" and "x = x.view(-1, 3200)" here from 3200 to 6400 my code runs without any error.

Similarly, It will throw an error input and target shapes do not match: input [100 x 1], target [200 x 1] if I put 12800 instead of 6400

Now my doubt is that I am not able to understand the reason behind this. If I am giving 200 images as input to my network then why the input shape is getting affected while changing the parameters when I move from convolutional layer to fully connected layer. I hope I have clearly mentioned my doubt. Even though I anybody has any doubt please ask me. It will be a great help. Thanks in advance.


title: Pytorch-1.2.0 Error on CrossEntropyLoss: only batches of spatial targets supported (3D tensors) but got targets of dimension

body:
Just resurfaced an old template of mine for basic MLP's and I am a bit rusted. I use CELoss on MNIST (transformed images in tensors).

I tried to unsqueeze to factor in batch size, but it really insists on dim=3 for target tensor.
If anyone could also explain why it now requires this it'd be appreciated.

PS. Seems related to post below, which was not answered here.

My code

train_dataset = datasets.MNIST('../', download=True, transform=transforms.ToTensor())
test_dataset = datasets.MNIST('../', train=False, download=True, transform=transforms.ToTensor())

indices = list(range(len(train_dataset.data)))
indices = list(range(len(test_dataset.data)))

random.shuffle(indices)

train_loader = torch.utils.data.DataLoader(train_dataset,
                                          batch_size=batchSize,
                                          num_workers=1)

test_loader = torch.utils.data.DataLoader(test_dataset,
                                          batch_size=batchSize,
                                          num_workers=1)

# Surrogate loss used for training
loss_fn = nn.CrossEntropyLoss()
test_loss_fn = nn.CrossEntropyLoss(reduction='sum')

optimizer = optim.Adam(model.parameters(), lr=lr)
#optimizer = optim.SGD(model.parameters(), lr=lr ,weight_decay=weight_decay)

print('Training beginning...')
start_time = time.time()

for epoch in range(1, nbr_epochs+1):
    print('Epoch ', epoch, ':')
    train(model, train_loader, optimizer, epoch,loss_fn)
    loss, acc = test(model, test_loader)

    test_accuracy.append(acc)
    train_loss.append(loss)

def train(model,train_loader, optimizer, epoch, loss_fn):
    model.train()

    for batch_idx, (inputs, target) in enumerate(train_loader):

        #inputs = inputs.view(batchSize, 1,100,100)
        inputs, target = inputs.to(device), target.to(device)

        optimizer.zero_grad()
        output = model(inputs)
        loss = loss_fn(output, target.unsqueeze(1))

        # Backprop
        loss.backward()
        optimizer.step()

And the Error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-c6a8cb2b8f18> in <module>
     22 for epoch in range(1, nbr_epochs+1):
     23     print('Epoch ', epoch, ':')
---> 24     train(model, train_loader, optimizer, epoch,loss_fn)
     25     loss, acc = test(model, test_loader)
     26

<ipython-input-29-d78d7fdaeb4d> in train(model, train_loader, optimizer, epoch, loss_fn)
     10         optimizer.zero_grad()
     11         output = model(inputs)
---> 12         loss = loss_fn(output, target.unsqueeze(1))
     13
     14         # Backprop

~/.local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    545             result = self._slow_forward(*input, **kwargs)
    546         else:
--> 547             result = self.forward(*input, **kwargs)
    548         for hook in self._forward_hooks.values():
    549             hook_result = hook(self, input, result)

~/.local/lib/python3.6/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    914     def forward(self, input, target):
    915         return F.cross_entropy(input, target, weight=self.weight,
--> 916                                ignore_index=self.ignore_index, reduction=self.reduction)
    917
    918

~/.local/lib/python3.6/site-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   1993     if size_average is not None or reduce is not None:
   1994         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 1995     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   1996
   1997

~/.local/lib/python3.6/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1824         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1825     elif dim == 4:
-> 1826         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1827     else:
   1828         # dim == 3 or dim > 4

RuntimeError: invalid argument 3: only batches of spatial targets supported (3D tensors) but got targets of dimension: 2 at /pytorch/aten/src/THNN/generic/SpatialClassNLLCriterion.c:61

Where inputs in train() for loop is shape (32,1,28,28) and problematic target is (32,1)


title: Install Pytorch on Windows

body:
I am trying to install Pytorch on Windows8.1. I am using Python 3.6.4 and no GPU. I've tried already the Anaconda package provided by peterjc123 by running conda install -c peterjc123 pytorch_legacy cuda80 using a virtual environment. While the installation goes smooth (without errors), after import torch I get the following error.

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\myuser\AppData\Local\conda\conda\envs\myenv\lib\site-packages\torch\__init__.py", line 76, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

Can somebody help me to install it?


title: How can I use Numba for Pytorch tensors?

body:
I am new to Numba and I need to use Numba to speed up some Pytorch functions. But I find even a very simple function does not work :(
import torch
import numba

@numba.njit()
def vec_add_odd_pos(a, b):
    res = 0.
    for pos in range(len(a)):
        if pos % 2 == 0:
            res += a[pos] + b[pos]
    return res

x = torch.tensor([3, 4, 5.])
y = torch.tensor([-2, 0, 1.])
z = vec_add_odd_pos(x, y)

But the following error appears
def vec_add_odd_pos(a, b):
res = 0.
^
This error may have been caused by the following argument(s):

argument 0: cannot determine Numba type of <class 'torch.Tensor'>
argument 1: cannot determine Numba type of <class 'torch.Tensor'>

Can anyone help me? A link with more examples would be also appreciated. Thanks.


title: Pytorch Torch.save FileNotFoundError

body:
When I try to call "torch.save" to save my model in a "tmp_file", it rises a FileNotFoundError. the trace back is as follow:

  Traceback (most recent call last):
  File “C:/Users/Haoran/Documents/GitHub/dose-response/python/simulations/hdr.py”, line 234, in
  test_hdr_continuous()
  File “C:/Users/Haoran/Documents/GitHub/dose-response/python/simulations/hdr.py”, line 195, in test_hdr_continuous
  model = fit_mdn(X[:split], y[:split], nepochs=20)
  File “C:\Users\Haoran\Documents\GitHub\dose-response\python\simulations\continuous.py”, line 192, in fit_mdn
  torch.save(model, tmp_file)
  File “C:\Users\Haoran\Documents\GitHub\dose-response\python\venv\lib\site-packages\torch\serialization.py”, line 161, in save
  return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File “C:\Users\Haoran\Documents\GitHub\dose-response\python\venv\lib\site-packages\torch\serialization.py”, line 116, in _with_file_like
  f = open(f, mode)
  FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/tmp_file_4358f298-a1d9-4c81-9e44-db4d8f1b4319’

It is weird that everything works perfectly on my mac, but I got this error on my Windows desktop.


title: How can I change the padded input size per channel in Pytorch?

body:
I am trying to set up an image classifier using Pytorch. My sample images have 4 channels and are 28x28 pixels in size. I am trying to use the built-in torchvision.models.inception_v3() as my model. Whenever I try to run my code, I get this error:

  RuntimeError: Calculated padded input size per channel: (1 x 1).
  Kernel size: (3 x 3). Kernel size can't greater than actual input size
  at
  /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THNN/generic/SpatialConvolutionMM.c:48

I can't find how to change the padded input size per channel or quite figure out what the error means. I figure that I must modify the padded input size per channel since I can't edit the Kernel size in the pre-made model.

I have tried padding, but it didn't help.
Here is a shortened part of my code that throws the error when I call train():

import torch
import torchvision as tv
import torch.optim as optim
from torch import nn
from torch.utils.data import DataLoader

model = tv.models.inception_v3()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001, weight_decay=0)
lr_scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=4, gamma=0.9)

trn_dataset = tv.datasets.ImageFolder(
    "D:/tests/classification_test_data/trn",
    transform=tv.transforms.Compose([tv.transforms.RandomRotation((0,275)), tv.transforms.RandomHorizontalFlip(),
                                  tv.transforms.ToTensor()]))
trn_dataloader = DataLoader(trn_dataset, batch_size=32, num_workers=4, shuffle=True)

for epoch in range(0, 10):
    train(trn_dataloader, model, criterion, optimizer, lr_scheduler, 6, 32)
print("End of training")

def train(train_loader, model, criterion, optimizer, scheduler, num_classes, batch_size):
    model.train()
    scheduler.step()

    for index, data in enumerate(train_loader):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = model(inputs)
        outputs_flatten = flatten_outputs(outputs, num_classes)
        loss = criterion(outputs_flatten, labels)
        loss.backward()
        optimizer.step()

def flatten_outputs(predictions, number_of_classes):
    logits_permuted = predictions.permute(0, 2, 3, 1)
    logits_permuted_cont = logits_permuted.contiguous()
    outputs_flatten = logits_permuted_cont.view(-1, number_of_classes)
    return outputs_flatten


title: TracerWarning: Converting a tensor to a Python index might cause the trace to be incorrect

body:
I am trying to create trace file. Related code part is here:

def center_crop(self, layer, target_size):
    _, _, layer_height, layer_width = layer.size()

    diff_y = (layer_height - target_size[0]) // 2
    diff_x = (layer_width - target_size[1]) // 2
    return layer[:, :, diff_y:(diff_y + target_size[0]), diff_x:(diff_x + target_size[1])]

def forward(self, x, bridge):

    up = self.up(x)
    crop1 = self.center_crop(bridge, up.shape[2:])
    out = torch.cat([up, crop1], 1)
    out = self.conv_block(out)

It gives the warning at this line:

return layer[:, :, diff_y:(diff_y + target_size[0]), diff_x:(diff_x + target_size[1])]

Then, I loaded this trace file in C++ and made prediction for a test image. Then I compared the result with the output in python. I observed that results are very different. I guess that the reason originates from that warning.


title: Handling C++ arrays in Cython (with numpy and pytorch)

body:
I am trying to use cython to wrap a C++ library (fastText, if its relevant). The C++ library classes load a very large array from disk. My wrapper instantiates a class from the C++ library to load the array, then uses cython memory views and numpy.asarray to turn the array into a numpy array, then calls torch.from_numpy to create a tensor.

The problem arising is how to handle deallocating the memory for the array.

Right now, I get pointer being freed was not allocated when the program exits. This is, I expect, because both the C++ code and numpy/pytorch are trying to manage the same chunk of RAM.

I could simply comment out the destructor in the C++ library, but that feels like its going to cause me a different problem down the road.

How should I approach the issue?  Is there any kind of best practices documentation somewhere on how to handle memory sharing with C++ and cython?

If I modify the C++ library to wrap the array in a shared_ptr, will cython (and numpy, pytorch, etc.) share the shared_ptr properly?

I apologize if the question is naive; Python garbage collection is very mysterious to me.

Any advice is appreciated.


title: PyTorch element-wise filter layer

body:
Hi, I want to add element-wise multiplication layer to duplicate the input to multi-channels like this figure. (So, the input size M x N and multiplication filter size M x N is same), as illustrated in this figure

I want to add custom initialization value to filter, and also want them to get gradient while training. However, I can't find element-wise filter layer in PyTorch. Can I make it? Or is it just impossible in PyTorch?


title: Trouble crating Dataset and Dataloader for HDF5 file in Pytorch: not enough values to unpack (expected 2, got 1)

body:
After using datasets from Torchvision, I am trying to load in Pytorch a HDF5 file instead, with no success. I read I should define my own Dataset and Dataloader classes containing getitem to enable indexing and len to return the length of the dataset. Plus, that I should define transform because the default option of pytorch expect PIL images. I tried doing that, but I get the error "ValueError: not enough values to unpack (expected 2, got 1)" What am I doing wrong?

#PyTorch packages
import torch
from torch.utils.data import Dataset, DataLoader
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from torchvision import transforms, datasets
from torch import optim
from torch.autograd import Variable
from torch.utils import data

import h5py
import numpy as np
import matplotlib.pyplot as plt

torch.manual_seed(0)

#open training file
with h5py.File('train_catvnoncat.h5', 'r') as hdf:
    ls = list(hdf.keys())
    print('List of datasets in this file: \n', ls)
    data = hdf.get('dataset')
    dataset1 = np.array(data)
    print('Shape of dataset1: \n', dataset1.shape)
    length = len(h5py.File('train_catvnoncat.h5', 'r'))
    print(length)

#image size (64,64,3) 64*64*3=12,288.
#209 training examples
#50 test examples

# Example of a picture

#image size (64,64,3) 64*64*3=12,288.
#209 training examples
#50 test examples

#Def the dataloader for h5 files:

class HDF5Dataset(Dataset):
    def __init__(self, h5_path):
        self.h5_path = '/Users/teff/Downloads/'
        self.train = train_catvnoncat.h5(h5_path, 'r')
        self.train = test_catvnoncat.h5(h5_path, 'r')
        self.length = len(h5py.File(h5_path, 'r'))
        # self.transform = transform_hdf5    #I need to define the "transformToTensor"

    def __getitem__(self, index): #to enable indexing
        record = self.train[str(index)]
        return (
                record['X'].value,
                record['y'].value,
        )

    def __len__(self): #returns the lenght of the dataset
        return self.length

train_loader = torch.utils.data.DataLoader('train_catvnoncat.h5', shuffle=True)
test_loader = torch.utils.data.DataLoader('test_catvnoncat.h5', shuffle=True)


title: Pytorch: Getting the correct dimensions for final layer

body:
Pytorch newbie here! I am trying to fine-tune a VGG16 model to predict 3 different classes. Part of my work involves converting FC layers to CONV layers. However, the values of my predictions don't fall between 0 to 2 (the 3 classes).

Can someone point me to a good resource on how to compute the correct dimensions for the final layer?

Here are the original fC layers of VGG16:

(classifier): Sequential(
    (0): Linear(in_features=25088, out_features=4096, bias=True)
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Linear(in_features=4096, out_features=4096, bias=True)
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )

My code for converting FC layers to CONV:

 def convert_fc_to_conv(self, fc_layers):
        # Replace first FC layer with CONV layer
        fc = fc_layers[0].state_dict()
        in_ch = 512
        out_ch = fc["weight"].size(0)
        first_conv = nn.Conv2d(512, out_ch, kernel_size=(1, 1), stride=(1, 1))

        conv_list = [first_conv]
        for idx, layer in enumerate(fc_layers[1:]):
            if isinstance(layer, nn.Linear):
                fc = layer.state_dict()
                in_ch = fc["weight"].size(1)
                out_ch = fc["weight"].size(0)
                if idx == len(fc_layers)-4:
                    in_ch = 3
                conv = nn.Conv2d(out_ch, in_ch, kernel_size=(1, 1), stride=(1, 1))
                conv_list += [conv]
            else:
                conv_list += [layer]
            gc.collect()

        avg_pool = nn.AvgPool2d(kernel_size=2, stride=1, ceil_mode=False)
        conv_list += [avg_pool, nn.Softmax()]
        top_layers = nn.Sequential(*conv_list)
        return top_layers

Final model architecture:

    Model(
    (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False))

    (classifier): Sequential(
    (0): Conv2d(512, 4096, kernel_size=(1, 1), stride=(1, 1))
    (1): ReLU(inplace)
    (2): Dropout(p=0.5)
    (3): Conv2d(4096, 3, kernel_size=(1, 1), stride=(1, 1))
    (4): ReLU(inplace)
    (5): Dropout(p=0.5)
    (6): AvgPool2d(kernel_size=2, stride=1, padding=0)
    (7): Softmax()
  )
)

Summary of the model:

            Conv2d-1         [-1, 64, 224, 224]           1,792
              ReLU-2         [-1, 64, 224, 224]               0
            Conv2d-3         [-1, 64, 224, 224]          36,928
              ReLU-4         [-1, 64, 224, 224]               0
         MaxPool2d-5         [-1, 64, 112, 112]               0
            Conv2d-6        [-1, 128, 112, 112]          73,856
              ReLU-7        [-1, 128, 112, 112]               0
            Conv2d-8        [-1, 128, 112, 112]         147,584
              ReLU-9        [-1, 128, 112, 112]               0
        MaxPool2d-10          [-1, 128, 56, 56]               0
           Conv2d-11          [-1, 256, 56, 56]         295,168
             ReLU-12          [-1, 256, 56, 56]               0
           Conv2d-13          [-1, 256, 56, 56]         590,080
             ReLU-14          [-1, 256, 56, 56]               0
           Conv2d-15          [-1, 256, 56, 56]         590,080
             ReLU-16          [-1, 256, 56, 56]               0
        MaxPool2d-17          [-1, 256, 28, 28]               0
           Conv2d-18          [-1, 512, 28, 28]       1,180,160
             ReLU-19          [-1, 512, 28, 28]               0
           Conv2d-20          [-1, 512, 28, 28]       2,359,808
             ReLU-21          [-1, 512, 28, 28]               0
           Conv2d-22          [-1, 512, 28, 28]       2,359,808
             ReLU-23          [-1, 512, 28, 28]               0
        MaxPool2d-24          [-1, 512, 14, 14]               0
           Conv2d-25          [-1, 512, 14, 14]       2,359,808
             ReLU-26          [-1, 512, 14, 14]               0
           Conv2d-27          [-1, 512, 14, 14]       2,359,808
             ReLU-28          [-1, 512, 14, 14]               0
           Conv2d-29          [-1, 512, 14, 14]       2,359,808
             ReLU-30          [-1, 512, 14, 14]               0
        MaxPool2d-31            [-1, 512, 7, 7]               0
           Conv2d-32           [-1, 4096, 7, 7]       2,101,248
             ReLU-33           [-1, 4096, 7, 7]               0
          Dropout-34           [-1, 4096, 7, 7]               0
           Conv2d-35              [-1, 3, 7, 7]          12,291
             ReLU-36              [-1, 3, 7, 7]               0
          Dropout-37              [-1, 3, 7, 7]               0
        AvgPool2d-38              [-1, 3, 6, 6]               0
          Softmax-39              [-1, 3, 6, 6]               0


title: How to calculate roc auc score for the whole epoch like avg accuracy?

body:
I am implementing a training loop in PyTorch and for metrics, I want to use ROC AUC score using sklearn.metrics.roc_auc_score.

I can use sklearn's implementation for calculating the score for a single prediction but have a little trouble imagining how to use it to calculate the average score for the whole epoch. Can anyone push me in the right direction?


title: Is it possible to train pytorch and tensorflow model together on one GPU?

body:
I have a pytorch model and a tensorflow model, I want to train them together  on one GPU, following the process bellow: input --> pytorch model--> output_pytorch --> tensorflow model --> output_tensorflow --> pytorch model.

Is is possible to do this? If answer is yes, is there any problem which I will encounter?

Thanks in advance.


title: PyTorch Cuda with anaconda not available

body:
I'm using anaconda to regulate my environment,
for a project i have to use my GPU for network training.
I use pytorch for my project and i'm trying to get CUDA working.

I installed cudatoolkit, numba, cudnn

still, when i try this command:

torch.cuda.is_available()

I get "False" as output.
This is my environment:

# Name                    Version                   Build  Channel
blas                      1.0                         mkl
bzip2                     1.0.6                h470a237_2    conda-forge
ca-certificates           2018.03.07                    0
cairo                     1.14.12              he6fea26_5    conda-forge
certifi                   2018.8.24                py35_1
cffi                      1.11.5           py35he75722e_1
cloudpickle               0.5.5                    py35_0
cudatoolkit               9.2                           0    anaconda
cudnn                     7.2.1                 cuda9.2_0    anaconda
cycler                    0.10.0                     py_1    conda-forge
cython                    0.28.5           py35hf484d3e_0    anaconda
dask-core                 0.19.2                   py35_0
dbus                      1.13.0               h3a4f0e9_0    conda-forge
decorator                 4.3.0                    py35_0
expat                     2.2.5                hfc679d8_2    conda-forge
ffmpeg                    4.0.2                ha0c5888_1    conda-forge
fontconfig                2.13.1               h65d0f4c_0    conda-forge
freetype                  2.9.1                h6debe1e_4    conda-forge
gettext                   0.19.8.1             h5e8e0c9_1    conda-forge
giflib                    5.1.4                h470a237_1    conda-forge
glib                      2.55.0               h464dc38_2    conda-forge
gmp                       6.1.2                hfc679d8_0    conda-forge
gnutls                    3.5.19               h2a4e5f8_1    conda-forge
graphite2                 1.3.12               hfc679d8_1    conda-forge
gst-plugins-base          1.12.5               hde13a9d_0    conda-forge
gstreamer                 1.12.5               h61a6719_0    conda-forge
harfbuzz                  1.9.0                h08d66d9_0    conda-forge
hdf5                      1.10.2               hc401514_2    conda-forge
icu                       58.2                 hfc679d8_0    conda-forge
imageio                   2.4.1                    py35_0
intel-openmp              2019.0                      118
jasper                    1.900.1              hff1ad4c_5    conda-forge
jpeg                      9c                   h470a237_1    conda-forge
kiwisolver                1.0.1            py35h2d50403_2    conda-forge
libedit                   3.1.20170329         h6b74fdf_2
libffi                    3.2.1                hd88cf55_4
libgcc-ng                 8.2.0                hdf63c60_1
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.3.0                hdf63c60_0
libiconv                  1.15                 h470a237_3    conda-forge
libopenblas               0.3.3                h5a2b251_3
libpng                    1.6.35               ha92aebf_2    conda-forge
libstdcxx-ng              8.2.0                hdf63c60_1
libtiff                   4.0.9                he6b73bb_2    conda-forge
libuuid                   2.32.1               h470a237_2    conda-forge
libwebp                   0.5.2                         7    conda-forge
libxcb                    1.13                 h470a237_2    conda-forge
libxml2                   2.9.8                h422b904_5    conda-forge
llvmlite                  0.24.0           py35hdbcaa40_0
matplotlib                3.0.0            py35h0b34cb6_1    conda-forge
mkl                       2019.0                      118
mkl_fft                   1.0.6                    py35_0    conda-forge
mkl_random                1.0.1                    py35_0    conda-forge
ncurses                   6.1                  hf484d3e_0
nettle                    3.3                           0    conda-forge
networkx                  2.1                      py35_0
ninja                     1.8.2            py35h6bb024c_1
numba                     0.39.0           py35h04863e7_0
numpy                     1.15.2           py35h1d66e8a_0
numpy-base                1.15.2           py35h81de0dd_0
olefile                   0.46                     py35_0
openblas                  0.2.20                        8    conda-forge
opencv                    3.4.1            py35h6fd60c2_1
opencv-python             3.4.3.18                  <pip>
openh264                  1.7.0                         0    conda-forge
openssl                   1.0.2p               h14c3975_0
pandas                    0.23.4           py35h04863e7_0
pcre                      8.41                 hfc679d8_3    conda-forge
pillow                    5.2.0            py35heded4f4_0
Pillow                    5.3.0                     <pip>
pip                       10.0.1                   py35_0
pixman                    0.34.0               h470a237_3    conda-forge
pthread-stubs             0.4                  h470a237_1    conda-forge
pycparser                 2.19                     py35_0
pyparsing                 2.2.2                      py_0    conda-forge
pyqt                      5.6.0            py35h8210e8a_7    conda-forge
python                    3.5.6                hc3d631a_0
python-dateutil           2.7.3                      py_0    conda-forge
pytorch                   0.4.1           py35_py27__9.0.176_7.1.2_2    pytorch
pytz                      2018.5                   py35_0
pywavelets                1.0.0            py35hdd07704_0
qt                        5.6.2                hf70d934_9    conda-forge
readline                  7.0                  h7b6447c_5
scikit-image              0.14.0           py35hf484d3e_1
scipy                     1.1.0            py35hfa4b5c9_1
setuptools                40.2.0                   py35_0
sip                       4.18.1           py35hfc679d8_0    conda-forge
six                       1.11.0                   py35_1    conda-forge
sqlite                    3.25.2               h7b6447c_0
tk                        8.6.8                hbc83047_0
toolz                     0.9.0                    py35_0
torchvision               0.1.9            py35h72e4c6f_1    soumith
tornado                   5.1.1            py35h470a237_0    conda-forge
wheel                     0.31.1                   py35_0
x264                      1!152.20180717       h470a237_1    conda-forge
xorg-kbproto              1.0.7                h470a237_2    conda-forge
xorg-libice               1.0.9                h470a237_4    conda-forge
xorg-libsm                1.2.2                h8c8a85c_6    conda-forge
xorg-libx11               1.6.6                h470a237_0    conda-forge
xorg-libxau               1.0.8                h470a237_6    conda-forge
xorg-libxdmcp             1.1.2                h470a237_7    conda-forge
xorg-libxext              1.3.3                h470a237_4    conda-forge
xorg-libxrender           0.9.10               h470a237_2    conda-forge
xorg-renderproto          0.11.1               h470a237_2    conda-forge
xorg-xextproto            7.3.0                h470a237_2    conda-forge
xorg-xproto               7.0.31               h470a237_7    conda-forge
xz                        5.2.4                h14c3975_4
zlib                      1.2.11               ha838bed_2

My desktop has a NVIDIA GeForce GTX 970 (so it is cuda available)
Also for some reason, as you can see here:

My graphics card doesn't show, however when using the

lspci -v

command, i can see my graphics card there. Don't know if that has something to do with it. Anyone knows how i can fix this?


title: I fine tuned a pre-trained BERT for sentence classification, but i cant get it to predict for new sentences

body:
below is the result of my fine-tuning.

Training Loss   Valid. Loss Valid. Accur.   Training Time   Validation Time
epoch
1   0.16    0.11    0.96    0:02:11 0:00:05
2   0.07    0.13    0.96    0:02:19 0:00:05
3   0.03    0.14    0.97    0:02:22 0:00:05
4   0.02    0.16    0.96    0:02:21 0:00:05

next i tried to use the model to predict labels from a csv file. i created a label column, set the type to int64 and run the prediction.

print('Predicting labels for {:,} test sentences...'.format(len(input_ids)))
model.eval()
# Tracking variables
predictions , true_labels = [], []
# Predict
for batch in prediction_dataloader:
  # Add batch to GPU
  batch = tuple(t.to(device) for t in batch)

  # Unpack the inputs from our dataloader
  b_input_ids, b_input_mask, b_labels = batch

  # Telling the model not to compute or store gradients, saving memory and
  # speeding up prediction
  with torch.no_grad():
      # Forward pass, calculate logit predictions
      outputs = model(b_input_ids, token_type_ids=None,
                      attention_mask=b_input_mask)

  logits = outputs[0]

  # Move logits and labels to CPU
  logits = logits.detach().cpu().numpy()
  label_ids = b_labels.to('cpu').numpy()

  # Store predictions and true labels
  predictions.append(logits)
  true_labels.append(label_ids)

however, while i am able to print out the predictions[4.235, -4.805] etc, and the true_labels[NaN,NaN.....], i am unable to actually get the predicted labels{0 or 1}. Am i missing something here?


title: Convert torch t7 model to pytorch

body:
I have a torch t7 model which I want to convert it to a pytorch model. I used this method:

model = load_lua('xxx.t7', unknown_classes=True)

However, I get the following error:

AttributeError: type object 'torch.cuda.FloatStorage' has no attribute 'from_buffer'

Any idea how to solve it?