title: PyTorch: while loading batched data using Dataloader, how to transfer the data to GPU automatically

body:
If we use a combination of the Dataset and Dataloader classes (as shown below), I have to explicitly load the data onto the GPU using .to() or .cuda(). Is there a way to instruct the dataloader to do it automatically/implicitly?
Code to understand/reproduce the scenario:
from torch.utils.data import Dataset, DataLoader
import numpy as np

class DemoData(Dataset):
    def __init__(self, limit):
        super(DemoData, self).__init__()
        self.data = np.arange(limit)

    def __len__(self):
        return self.data.shape[0]

    def __getitem__(self, idx):
        return (self.data[idx], self.data[idx]*100)

demo = DemoData(100)

loader = DataLoader(demo, batch_size=50, shuffle=True)

for i, (i1, i2) in enumerate(loader):
    print('Batch Index: {}'.format(i))
    print('Shape of data item 1: {}; shape of data item 2: {}'.format(i1.shape, i2.shape))
    # i1, i2 = i1.to('cuda:0'), i2.to('cuda:0')
    print('Device of data item 1: {}; device of data item 2: {}\n'.format(i1.device, i2.device))

Which will output the following; note - without explicit device transfer instruction, the data is loaded onto CPU:
Batch Index: 0
Shape of data item 1: torch.Size([50]); shape of data item 2: torch.Size([50])
Device of data item 1: cpu; device of data item 2: cpu

Batch Index: 1
Shape of data item 1: torch.Size([50]); shape of data item 2: torch.Size([50])
Device of data item 1: cpu; device of data item 2: cpu

A possible solution is at this PyTorch GitHub repo. Issue(still open at the time this question was posted), but, I am unable to make it to work when the dataloader has to return multiple data-items!


title: Issues importing pytorch with conda

body:
My system is:

 x86_64
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=17.1
DISTRIB_CODENAME=rebecca
DISTRIB_DESCRIPTION="Linux Mint 17.1 Rebecca"
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
cat: /etc/upstream-release: Is a directory

I have recently installed pytorch since it is required for the use of the Python causal discovery toolbox, and I have installed it using Anaconda without the use of CUDA, like so:

conda install pytorch-cpu torchvision-cpu -c pytorch

This installation was successful, but when I try to run

import torch

in IDLE, it is not able to find the module. I realize that this question may be a redundancy, but I have searched through similar issues without finding anything that worked, and I am sort on time. Also, I am unfamiliar with Anaconda, so I am not sure how to get packages installed through it to work with the rest of Python. Thank you in advance for any help.


title: How to vectorize custom algorithms in numpy or pytorch?

body:
Suppose I have two matrices:

A: size k x m

B: size m x n

Using a custom operation, my output will be k x n.

This custom operation is not a dot product between the rows of A and columns of B. Suppose this custom operation is defined as:

For the Ith row of A and Jth column of B, the i,j element of the output is:

sum( (a[i] + b[j]) ^20 ), i loop over I, j loops over J

The only way I can see to implement this is to expand this equation, calculate each term, them sum them.

Is there a way in numpy or pytorch to do this without expanding the equation?


title: torch.nn.conv2d does not give the same result as torch.nn.functional.conv2d

body:
This is my code:

l1 = nn.Conv2d(3, 2, kernel_size=3, stride=2).double() #Layer
l1wt = l1.weight.data #filter
inputs = np.random.rand(3, 3, 5, 5) #input
it = torch.from_numpy(inputs) #input tensor
output1 = l1(it) #output
output2 = torch.nn.functional.conv2d(it, l1wt, stride=2) #output
print(output1)
print(output2)

I would expect to get the same result for output1 and output2, but they are not.
Am I doing something wrong are do nn and nn.functional work different?


title: error in BatchNorm2d in pytorch CNN model

body:
my database has grayscale images of size 128 * 128* 1   each  with batch size =10
i am using cnn model but I got this error in BatchNorm2d
expected 4D input (got 2D input)
I posted the way which i used to transform my image (gray scale - tensor - normalize ) and divide it to batches
data_transforms = {
    'train': transforms.Compose([
        transforms.Grayscale(num_output_channels=1),
        transforms.Resize(128),
        transforms.CenterCrop(128),
        transforms.ToTensor(),
        transforms.Normalize([0.5], [0.5])
    ]),
    'val': transforms.Compose([
        transforms.Grayscale(num_output_channels=1),
        transforms.Resize(128),
        transforms.CenterCrop(128),
        transforms.ToTensor(),
        transforms.Normalize([0.5], [0.5])
    ]),
}

data_dir = '/content/drive/My Drive/Colab Notebooks/pytorch'
dsets = {x: datasets.ImageFolder(os.path.join(data_dir, x), data_transforms[x])
         for x in ['train', 'val']}
dset_loaders = {x: torch.utils.data.DataLoader(dsets[x], batch_size=10,
                                               shuffle=True, num_workers=25)
                for x in ['train', 'val']}
dset_sizes = {x: len(dsets[x]) for x in ['train', 'val']}
dset_classes = dsets['train'].classes

I used this model
class HeartNet(nn.Module):
    def __init__(self, num_classes=7):

        super(HeartNet, self).__init__()

        self.features = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(64),
            nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(64),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(128),
            nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(128),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(256),
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(256),
            nn.MaxPool2d(kernel_size=2, stride=2)
            )

        self.classifier = nn.Sequential(
            nn.Dropout(0.5),
            nn.Linear(16*16*256, 2048),
            nn.ELU(inplace=True),
            nn.BatchNorm2d(2048),
            nn.Linear(2048, num_classes)
            )

        nn.init.xavier_uniform_(self.classifier[1].weight)
        nn.init.xavier_uniform_(self.classifier[4].weight)

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), 16 * 16 * 256)
        x = self.classifier(x)
        return x

How can I solve this problem ?


title: requires_grad = False seems not working in my case

body:
I received a Cannot insert a Tensor that requires grad as a constant. Consider making it a parameter or input, or detaching the gradient error with tensor W.
W has the size of (10,10) and grad_fn=<DivBackward0>. The error happens at the second line
def muy(self, x):
    V = torch.tensor(self.W - self.lambda_ * torch.eye(self.ENCODING_DIM), requires_grad=False)
    return -0.5 * V.inverse().mm(self.b + self.lambda_ * x[:, None])

Other vars, values taken at the time of the error

self.lambda_: 1.0
self.ENCODING_DIM: 10
self.b: torch.Size([10, 1]), requires_grad=True
x: torch.Size([3, 1, 10]), grad_fn=<MulBackward0>

How could I set the result of muy as just an ingredient of the leaf node, so grad through V is required?
I tried this monstrosity, to no avail
def muy(self, x):
    V_inv = np.linalg.inv(self.V.detach().numpy())
    x_numpy = x[:, None].detach().numpy()
    temp= -0.5 * np.matmul(V_inv, self.b.detach().numpy() + self.lambda_ * x_numpy)
    return temp

Why I cared about this JIT:
I wanted to use tensorboard to visualize my model, if I understand the error messages right, the visualzing models use Tracer
EDIT
This still gives the same error, W or W.detach()
with torch.no_grad():
    V = self.W - self.lambda_ * torch.eye(self.ENCODING_DIM)
    return -0.5 * V.inverse().mm(self.b + self.lambda_ * x[:, None])


title: How to load a dataset of images starting from list of images Pytorch

body:
I have a service that receives images in a binary format from another service (let's call it service B):
from PIL import Image

img_list = []
img_bin = get_image_from_service_B()
image = Image.open(io.BytesIO(img_bin)) # Convert bytes to image using PIL

When an image is successfully converted thanks to PIL it is also appended to a list of images.
img_list.append(image)

When I've enough images I want to load my list of images using Pytorch as if it was a dataset
if img_list.__len__() == 500:
     ### Load dataset and do a transform operation on the data

In a previous version of the software the requirement was simply to retrieve the images from a folder, so it was quite simple to load all the images
my_dataset = datasets.ImageFolder("path/to/images/folder/", transform=transform)
dataset_iterator = DataLoader(my_dataset, batch_size=1)

Now my issue is how to perform the transform and load the dataset from a list.


I know my images have only 1 channel so the first conv layer is (1,16,3,1) , but I have no idea why I got such an error.
Here is my code (I post only the related part).
    org_x = train_csv.drop(['id', 'digit', 'letter'], axis=1).values
    org_x = org_x.reshape(-1, 28, 28, 1)
    org_x = org_x/255
    org_x = np.array(org_x)
    org_x = org_x.reshape(-1, 1, 28, 28)
    org_x = torch.Tensor(org_x).float()

    x_test = test_csv.drop(['id','letter'], axis=1).values
    x_test = x_test.reshape(-1, 28, 28, 1)
    x_test = x_test/255
    x_test = np.array(x_test)
    x_test = x_test.reshape(-1, 1, 28, 28)
    x_test = torch.Tensor(x_test).float()

    y = train_csv['digit']
    y = list(y)
    print(len(y))
    org_y = np.zeros([len(y), 1])
    for i in range(len(y)):
        org_y[i] = y[i]
    org_y = np.array(org_y)
    org_y = torch.Tensor(org_y).float()

    from sklearn.model_selection import train_test_split
    x_train, x_valid, y_train, y_valid = train_test_split(
        org_x, org_y, test_size=0.2, random_state=42)

I checked the x_train shape is [1638, 1, 28, 28] and the x_valid shape is [410, 1, 28, 28].
    transform = transforms.Compose([transforms.ToPILImage(),
                            transforms.ToTensor(),
                            transforms.Normalize((0.5, ), (0.5, )) ])

    class kmnistDataset(data.Dataset):
        def __init__(self, images, labels, transforms=None):
            self.x = images
            self.y = labels
            self.transforms = transforms

        def __len__(self):
            return (len(self.x))

        def __getitem__(self, idx):
            data = np.asarray(self.x[idx][0:]).astype(np.uint8)

            if self.transforms:
                data = self.transforms(data)

            if self.y is not None:
                return (data, self.y[idx])
            else:
                return data

    train_data = kmnistDataset(x_train, y_train, transforms=transform)
    valid_data = kmnistDataset(x_valid, y_valid, transforms=transform)

    # dataloaders
    train_loader = DataLoader(train_data, batch_size=16, shuffle=True)
    valid_loader = DataLoader(valid_data, batch_size=16, shuffle = False)

And here is my model
    class Net(nn.Module):
      def __init__(self):
            super(Net, self).__init__()

            self.conv1 = nn.Conv2d(1, 16, 3, padding=1)
            self.conv2 = nn.Conv2d(16, 32, 3, padding=1)
            self.conv3 = nn.Conv2d(32, 64, 3, padding=1)

            self.bn1 = nn.BatchNorm2d(16)
            self.pool = nn.MaxPool2d(2, 2)

            unit = 64 * 14 * 14
            self.fc1 = nn.Linear(unit, 500)
            self.fc2 = nn.Linear(500, 10)

        def forward(self, x):
            x = self.pool(F.relu(self.bn1(self.conv1(x))))
            x = F.relu(self.conv2(x))
            x = F.relu(self.conv3(x))
            x = x.view(-1, 128 * 28 * 28)
            x = F.relu(self.fc1(x))
            x = self.fc2(x)
            return x

    model = Net()
    print(model)

Lastly,
    n_epochs = 30

    valid_loss_min = np.Inf

    for epoch in range(1, n_epochs+1):
        train_loss = 0
        valid_loss = 0

        ###################
        # train the model #
        ###################
        model.train()
        for data in train_loader:
            inputs, labels = data[0], data[1]
            optimizer.zero_grad()
            output = model(inputs)
            loss = criterion(output, labels)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()*data.size(0)

        #####################
        # validate the model#
        #####################
        model.eval()
        for data in valid_loader:
            inputs, labels = data[0], data[1]
            output = model(inputs)
            loss = criterion(output, labels)
            valid_loss += loss.item()*data.size(0)

        train_loss = train_loss/ len(train_loader.dataset)
        valid_loss = valid_loss / len(valid_loader.dataset)

        print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
            epoch, train_loss, valid_loss))

When I run it, I got this error message
RuntimeError: Given groups=1, weight of size [16, 1, 3, 3], expected input[16, 3, 1, 28] to have 1 channels, but got 3 channels instead
To be specific,
    ---------------------------------------------------------------------------
    RuntimeError                              Traceback (most recent call last)
    <ipython-input-14-b8783819421f> in <module>
         14         inputs, labels = data[0], data[1]
         15         optimizer.zero_grad()
    ---> 16         output = model(inputs)
         17         loss = criterion(output, labels)
         18         loss.backward()

    /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in   _call_impl(self, *input, **kwargs)
        725             result = self._slow_forward(*input, **kwargs)
        726         else:
    --> 727             result = self.forward(*input, **kwargs)
        728         for hook in itertools.chain(
        729                 _global_forward_hooks.values(),

    <ipython-input-12-500e34c49306> in forward(self, x)
         26
         27     def forward(self, x):
    ---> 28         x = self.pool(F.relu(self.bn1(self.conv1(x))))
         29         x = F.relu(self.conv2(x))
         30         x = F.relu(self.conv3(x))

    /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in         _call_impl(self, *input, **kwargs)
        725             result = self._slow_forward(*input, **kwargs)
        726         else:
    --> 727             result = self.forward(*input, **kwargs)
        728         for hook in itertools.chain(
        729                 _global_forward_hooks.values(),

    /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in forward(self, input)
        421
        422     def forward(self, input: Tensor) -> Tensor:
    --> 423         return self._conv_forward(input, self.weight)
        424
        425 class Conv3d(_ConvNd):

    /opt/anaconda3/lib/python3.7/site-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight)
        418                             _pair(0), self.dilation, self.groups)
        419         return F.conv2d(input, weight, self.bias, self.stride,
    --> 420                         self.padding, self.dilation, self.groups)
        421
        422     def forward(self, input: Tensor) -> Tensor:

    RuntimeError: Given groups=1, weight of size [16, 1, 3, 3], expected input[16, 3, 1, 28]         to have 1 channels, but got 3 channels instead


I am trying to check the GPU device name but after executing this code. I got this unknown runtime error. please help me to solve this problem and give complete instructions to solve this error. Thanks.

        (base) kumar@kumar:~$ conda activate pytorch
        (pytorch) kumar@kumar:~$ python
        Python 3.8.5 (default, Sep  4 2020, 07:30:14)
        [GCC 7.3.0] :: Anaconda, Inc. on linux
        Type "help", "copyright", "credits" or "license" for more information.
        >>> import torch
        >>> print(torch.__version__)
        1.9.0a0+gitb39eeb0
        >>> print(torch.version.cuda)
        11.2
        >>> print(torch.cuda.current_device())
        Traceback (most recent call last):
          File "<stdin>", line 1, in <module>
          File "/home/kumar/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/cuda/__init__.py", line 430, in current_device
            _lazy_init()
          File "/home/kumar/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/cuda/__init__.py", line 170, in _lazy_init
            torch._C._cuda_init()
        RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.
        >>> exit()

Here is the output of nvcc -V
  (pytorch) kumar@kumar:~$ nvcc -V
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2021 NVIDIA Corporation
    Built on Sun_Feb_14_21:12:58_PST_2021
    Cuda compilation tools, release 11.2, V11.2.152
    Build cuda_11.2.r11.2/compiler.29618528_0
    (pytorch) kumar@kumar:~$

here is the output of nvidia-smi
    Thu Apr  8 15:04:49 2021
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 460.67       Driver Version: 460.39       CUDA Version: 11.2     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr: Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  GeForce RTX 3070    Off  | 00000000:01:00.0  On |                  N/A |
    |  0%   38C    P8    10W / 220W |    525MiB /  7982MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |    0   N/A  N/A      1015      G   /usr/lib/xorg/Xorg                 70MiB |
    |    0   N/A  N/A      1542      G   /usr/lib/xorg/Xorg                257MiB |
    |    0   N/A  N/A      1675      G   /usr/bin/gnome-shell               89MiB |
    |    0   N/A  N/A      3560      G   ...AAAAAAAAA= --shared-files       94MiB |
    +-----------------------------------------------------------------------------+

However, when I try to run
print(torch.cuda.current_device())

I get the following error:
Traceback (most recent call last):
              File "<stdin>", line 1, in <module>
              File "/home/kumar/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/cuda/__init__.py", line 430, in current_device
                _lazy_init()
              File "/home/kumar/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/cuda/__init__.py", line 170, in _lazy_init
                torch._C._cuda_init()
            RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.


title: what does THCudaTensor_data ( and THC in general ) do?

body:
The program I am inspecting uses pytorch to load weights and cuda code to do the computations with the weights. My understanding of THC library is how tensors are implemented in the backend of pytorch ( and torch, maybe? ).

how is THC implemented ( I would really appreiciate some details if possible )?
what does THCudaTensor_data( THC_state, THCudaTensor* ) do? ( from the way it is used in the code, it seems like it is used to convert pytorch's tensor to an array in cuda. if this is the case, then would the function preserve all elements and the length of the array?)


title: Training loss explodes from first training example and then outputs nan

body:
I'm a novice at deep learning, I have built some basic CNNs, but this time I'm trying to build FCN(Fully Convolutional Network) similar to Yolo3. my network contains 32 layers with LeakyRelu as the activation function and adam optimizer. there are 680 data samples and the input image size is 416x416 same as yolo model. I have given some snipes of my code below.

I'm using Pytorch 1.1 and cuda 9 version.
I have tried different learning rates like 0.0001, 0.000001, 0.0000001 as suggested at in many blogs, and also have tried different betas like (0.9,0.999), (0.5,0.999) and so on. I also have tried to train it long, up to 200 epochs.

Output

Running loss :  59102027776.0
  0%|          | 1/630 [00:19<3:24:03, 19.46s/it]Running loss :  nan
  0%|          | 2/630 [00:23<2:34:23, 14.75s/it]Running loss :  nan
  0%|          | 3/630 [00:25<1:53:32, 10.87s/it]Running loss :  nan

Loss Function formula

Please consider eq: 7 and 8 only in picture.
click the link for formula picture

Loss Function Code

masked_pose_loss = torch.mean(
        torch.sum(mask * torch.sum(torch.mul(pred - true, pred - true), dim=[1, 2]), dim=[1, 2, 3]))

FCN

    self.relu1_1 = nn.LeakyReLU(inplace=True)
    self.conv1_2 = nn.Conv2d(32, 64, 3, stride=1,padding=1)
    self.relu1_2 = nn.LeakyReLU(inplace=True)
    self.pool1 = nn.MaxPool2d(2, stride=2, ceil_mode=True)

    # conv2

    self.conv2_1 = nn.Conv2d(64, 128, 3, stride=1, padding=1)
    self.relu2_1 = nn.LeakyReLU(inplace=True)
    self.conv2_2 = nn.Conv2d(128, 64, 1, stride=1)
    self.relu2_2 = nn.LeakyReLU(inplace=True)
    self.pool2 = nn.MaxPool2d(2, stride=2, ceil_mode=True)

    # conv3
    self.conv3_1 = nn.Conv2d(64, 128, 3, stride=1, padding=1)
    self.relu3_1 = nn.LeakyReLU(inplace=True)
    self.conv3_2 = nn.Conv2d(128, 256, 3, stride=1, padding=1)
    self.relu3_2 = nn.LeakyReLU(inplace=True)
    self.conv3_3 = nn.Conv2d(256, 128, 1, stride=1)
    self.relu3_3 = nn.LeakyReLU(inplace=True)
    self.pool3 = nn.MaxPool2d(2, stride=2, ceil_mode=True)

    # conv4
    self.conv4_1 = nn.Conv2d(128, 256, 3, stride=1, padding=1)
    self.relu4_1 = nn.LeakyReLU(inplace=True)
    self.conv4_2 = nn.Conv2d(256, 512, 3, stride=1, padding=1)
    self.relu4_2 = nn.LeakyReLU(inplace=True)
    self.conv4_3 = nn.Conv2d(512, 256, 1, stride=1)
    self.relu4_3 = nn.LeakyReLU(inplace=True)
    self.pool4 = nn.MaxPool2d(2, stride=2, ceil_mode=True)

    # conv5
    self.conv5_1 = nn.Conv2d(256, 512, 3, stride=1, padding=1)
    self.relu5_1 = nn.LeakyReLU(inplace=True)
    self.conv5_2 = nn.Conv2d(512, 256, 1, stride=1)
    self.relu5_2 = nn.LeakyReLU(inplace=True)
    self.conv5_3 = nn.Conv2d(256, 512, 3, stride=1, padding=1)
    self.relu5_3 = nn.LeakyReLU(inplace=True)
    self.pool5 = nn.MaxPool2d(2, stride=2, ceil_mode=True)

    # fc6

    self.conv6_1 = nn.Conv2d(512, 1024, 3, stride=1, padding=1)
    self.relu6_1 = nn.LeakyReLU(inplace=True)
    self.conv6_2 = nn.Conv2d(1024, 512, 1, stride=1)
    self.relu6_2 = nn.LeakyReLU(inplace=True)
    self.conv6_3 = nn.Conv2d(512, 1024, 3, stride=1, padding=1)
    self.relu6_3 = nn.LeakyReLU(inplace=True)
    self.conv7_1 = nn.Conv2d(1024, 512, 1, stride=1)
    self.relu7_1 = nn.LeakyReLU(inplace=True)
    self.conv7_2 = nn.Conv2d(512, 1024, 3, stride=1, padding=1)
    self.relu7_2 = nn.LeakyReLU(inplace=True)
    self.conv7_3 = nn.Conv2d(1024, 1024, 3, stride=1, padding=1)
    self.relu7_3 = nn.LeakyReLU(inplace=True)
    self.conv8_1 = nn.Conv2d(1024, 1024, 3, stride=1, padding=1)
    self.relu8_1 = nn.LeakyReLU(inplace=True)
    self.conv8_2 = nn.Conv2d(1024, 1024, 3, stride=1)
    self.conv8_2 = nn.ReLU(inplace=True)
    self.conv_rout16 = nn.Conv2d(512, 64, 1, stride=1)
    self.relu_rout16 = nn.ReLU(inplace=True)enter code here

    # Resuming comments: I was implementing last two layer of network:
    self.convf_1 = nn.Conv2d(1280, 1024, 3, stride=1, padding=1)
    self.reluf_1 = nn.LeakyReLU(inplace=True)

    self.convf_2 = nn.Conv2d(1024, self.target_channel_size, 1, stride=1)

    self.reluf_2 = nn.LeakyReLU(inplace=True)

your response is highly appreciated.

Note: this is the first time that I'm posting. If I have missed some necessary information please let me know.


title: PureFrameworkTensorFoundError, Runtime error -FedeartedLearning

body:
I am trying a Linear Regression algorithm with Federated learning using Pytorch and I face the following error. I am implementing it on Colab. According to me this error might be due to some code line in the train() function. Kindly help is you have worked with Pysyft and have faced such error before.

RuntimeError: invalid argument 8: lda should be at least max(1, 0), but have 0 at /pytorch/aten/src/TH/generic/THBlas.cpp:363

And the following is the code:

#import the necessasry packages
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import syft as sy

#create target and data variables as tensors
x_data=Variable(torch.Tensor([[1.0],[0.0],[1.0],[0.0]]))
y_data=Variable(torch.Tensor([[0.0],[0.0],[1.0],[1.0]]))

#Create virtual Workers
hook = sy.TorchHook(torch)
bob = sy.VirtualWorker(hook, id="bob")
alice = sy.VirtualWorker(hook, id="alice")

data_bob = x_data[0:2]
target_bob = y_data[0:2]
data_alice = x_data[2:0]
target_alice = y_data[2:0]

#creating a class that does Linear Regression
class LinearRegression (nn.Module):

  def __init__(self):
    super(LinearRegression,self). __init__ ()
    self.linear = torch.nn.Linear(1,1)

  def forward(self, x):
    y_pred = self.linear(x)
    return y_pred

#assign the function to the variable name 'Model'
model=LinearRegression()

#send the data to the virtual worker pointers
data_bob = data_bob.send(bob)
data_alice = data_alice.send(alice)

target_bob = target_bob.send(bob)
target_alice = target_alice.send(alice)

# organize pointers into a list
datasets = [(data_bob,target_bob),(data_alice,target_alice)]

#create optimizer and calculate the loss
opt = torch.optim.SGD(params=model.parameters(),lr=0.1)
criterion = torch.nn.MSELoss(size_average=False)

def train():
  opt = torch.optim.SGD(params=model.parameters(),lr=0.1)
  for epoch in range (20):
    model.train()
    print("Training started..")

    for x_data,y_data in datasets:

      model.send(x_data.location)

      opt.zero_grad()

       #forwardpass
       #the model here is the linear regression model
      y_pred = model(x_data)

      #ComputeLoss
      loss=criterion(y_pred,y_data)

      #BackwardPass
      loss.backward()

      opt.step()

      model.get()

      print(loss.get())

train()


title: pytorch nn.L1Loss vs. Sklearn’s l1 loss - very different optimization results?

body:
I was implementing L1 regularization with pytorch for feature selection and found that I have different results compared to Sklearn or cvxpy. Perhaps I am implementing nn.L1Loss incorrectly or maybe there is a better way to optimize (I tried both Adam and SGD with a few different lr)?

import numpy as np
from tqdm import tqdm_notebook
import matplotlib.pyplot as plt

import cvxpy as cp

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
from sklearn.linear_model import Lasso

import torch
import torch.nn as nn
import torch.optim as optim

# generate data
X,y, coef_true = make_regression(n_samples=200, n_features=10000, n_informative=10,
                                 coef = True, random_state = 123)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.25)

print(np.where(coef_true != 0)[0])
# [ 893 4422 4428 5284 5632 5975 6388 7586 8270 9597]

Using sklearn, I get the correct answer:

# sklearn lasso
lasso_sklearn = Lasso(alpha = 0.2, warm_start = True)
lasso_sklearn.coef_ = np.zeros(X_train.shape[1])
lasso_sklearn.fit(X_train, y_train)
coef_sklearn = lasso_sklearn.coef_
print(np.where(lasso_sklearn.coef_ != 0)[0])
# [ 893 4422 4428 5284 5632 5975 6388 7586 8270 9597]

Using pytorch, I get this answer:

# pytorch lasso
class lasso(nn.Module):
    def __init__(self, in_dim,):
        super(lasso, self).__init__()
        self.linear = nn.Linear(in_dim,1)
    def forward(self, X):
        return self.linear(X)

def weights_init(m):
    if isinstance(m, nn.Linear):
        torch.nn.init.zeros_(m.weight)

lasso_pytorch = lasso(X_train.shape[1])
lasso_pytorch.apply(weights_init)

l1_loss = nn.L1Loss(reduction = 'sum')
mse_loss = nn.MSELoss()
optimizer = optim.Adam(lasso_pytorch.parameters(), lr = 0.0001)
alpha = 0.20
n_epoch = 5000
loss_history = []

lasso_pytorch.train()
for epoch in tqdm_notebook(range(n_epoch)):
    optimizer.zero_grad()
    outputs = lasso_pytorch(torch.from_numpy(X_train).float(),)
    loss = 0.5 * mse_loss(outputs, torch.from_numpy(y_train.reshape(-1,1)).float())
    p = 0
    for param in lasso_pytorch.parameters():
        loss += alpha * l1_loss(param, torch.zeros_like(param))

    loss_history.append(loss)
    loss.backward()
    optimizer.step()

coef_pytorch = np.array(lasso_pytorch.linear.weight.data).squeeze()
print(np.argsort(-np.abs(coef_pytorch))[:10])
# [5141 2251  902 2848 5002 8925 9328 8084 1888 2208]

Using cvxpy, I also get the correct answer:

# cvxpy lasso
def loss_fn(X, Y, beta):
    return cp.norm2(cp.matmul(X, beta) - Y)**2

def regularizer(beta,):
    return cp.norm1(beta)

def objective_fn(X, Y, beta, alpha):
    return 0.5/(len(X)) * loss_fn(X, Y, beta) + alpha * regularizer(beta) # from sklearn lasso

coef = cp.Variable(X_train.shape[1])
coef.value = np.zeros(X_train.shape[1])
alpha = cp.Parameter(nonneg=True)
alpha.value = 0.2
problem = cp.Problem(cp.Minimize(objective_fn(X, y, coef, alpha,)))
problem.solve(solver = cp.ECOS, warm_start = True,)
coef_cvxpy = coef.value
print(np.where(coef.value > 0.01)[0])
# [ 893, 4422, 4428, 5284, 5632, 5975, 6388, 7586, 8270, 9597]


title: How do I get a tiny subset of a image dataset with only a few classes?

body:
I'm trying to test my custom CNN model, so I want to train my model on a tiny subset and see if it can overfit.My question is as follows: How can I get only a few classes(say 5) for training?

Here's my dataloader:

train_data = CUB200.CUB200(transform=transform_train, train=True)
test_data = CUB200.CUB200(transform=transform_test, train=False)

trainloader = torch.utils.data.DataLoader(dataset=train_data, batch_size=32, shuffle=True, num_workers=0, drop_last=True)
testloader = torch.utils.data.DataLoader(dataset=test_data, batch_size=32, shuffle=True, num_workers=0)

custom data:

import os
import pandas as pd
from torchvision.datasets.folder import default_loader
from torch.utils.data import Dataset

base_folder = '../../data/CUB_200_2011/'

class CUB200(Dataset):
    def __init__(self, base_folder=base_folder, transform=None, train=True):
        self.base_folder = base_folder
        self.transform = transform
        self.train = train
        images = pd.read_csv(os.path.join(base_folder, 'images.txt'), sep=' ',
                        names=['img_id', 'filepath'])
        image_class_labels = pd.read_csv(os.path.join(base_folder, 'image_class_labels.txt'),
                                    sep=' ', names=['img_id', 'target'])
        train_test_split = pd.read_csv(os.path.join(base_folder, 'train_test_split.txt'),
                                        sep=' ', names=['img_id', 'is_training_img'])
        data = images.merge(image_class_labels, on='img_id')
        data = data.merge(train_test_split, on='img_id')
        if self.train:
            self.data = data[data.is_training_img == 1]
        else:
            self.data = data[data.is_training_img == 0]

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        sample = self.data.iloc[index]
        path = os.path.join(base_folder, 'images', sample.filepath)
        img = default_loader(path)
        # Target start from 1 in data, so shift to 0
        label = sample.target - 1
        if self.transform is not None:
            img = self.transform(img)
        return img, label

image folder:

I know we can use torch.utils.data.Subset in pytorch for doing this. But if I do this way, I must count the indices of the first 5 classes. Is there any more convient way to do this?


title: How to run a custom Pytorch object detection model in browser using ONNX.js?

body:
I am using this repo: https://github.com/microsoft/onnxjs-demo
It allows me to run Yolo object detection in my browser.
When I run the demo on my computer, it works just fine.  However, when I replace their yolo.onnx model with my own custom-trained model (yolo v5), the program gives me an error saying I do not have the correct backend.  What is the deal, and how can I fix this?
Thank you in advance for your help!


title: PyTorch: How to multiply via broadcasting of two tensors with different shapes

body:
I have the following two PyTorch tensors A and B.
A = torch.tensor(np.array([40, 42, 38]), dtype = torch.float64)

tensor([40., 42., 38.], dtype=torch.float64)

B = torch.tensor(np.array([[[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5],[1,2,3,4,5]], [[4,5,6,7,8],[4,5,6,7,8],[4,5,6,7,8],[4,5,6,7,8],[4,5,6,7,8]], [[7,8,9,10,11],[7,8,9,10,11],[7,8,9,10,11],[7,8,9,10,11],[7,8,9,10,11]]]), dtype = torch.float64)

tensor([[[ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.]],

        [[ 4.,  5.,  6.,  7.,  8.],
         [ 4.,  5.,  6.,  7.,  8.],
         [ 4.,  5.,  6.,  7.,  8.],
         [ 4.,  5.,  6.,  7.,  8.],
         [ 4.,  5.,  6.,  7.,  8.]],

        [[ 7.,  8.,  9., 10., 11.],
         [ 7.,  8.,  9., 10., 11.],
         [ 7.,  8.,  9., 10., 11.],
         [ 7.,  8.,  9., 10., 11.],
         [ 7.,  8.,  9., 10., 11.]]], dtype=torch.float64)

Tensor A is of shape:
torch.Size([3])

Tensor B is of shape:
torch.Size([3, 5, 5])

How do I multiply tensor A with tensor B (using broadcasting) in such a way for eg. the first value in tensor A (ie. 40.) is multiplied with all the values in the first 'nested' tensor in tensor B, ie.
tensor([[[ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.],
         [ 1.,  2.,  3.,  4.,  5.]],

and so on for the other 2 values in tensor A and the other two nested tensors in tensor B, respectively.
I could do this multiplication (via broadcasting) with numpy arrays if A and B are arrays of both shape (3,) - ie. A*B - but I can't seem to figure out a counterpart of this with PyTorch tensors. Any help would really be appreciated.


title: PyTorch says NO cuda GPU found on Colab however torch.cuda.is_available() is True

body:
Is there any way around to get GPU on colab
import torch
print(torch.version)
print(torch.cuda.is_available())

Error:
RuntimeError: No CUDA GPUs are available

Any suggestion would be appreciated


title: WARNING: The following packages were previously imported in this runtime: [PIL] You must restart the runtime in order to use newly installed versions

body:
This is a question from an absolute beginner of programing.
I tried the following codes on Google Colaboratory, and received the warning as written on the title of this post.
I restarted the runtime (which I believe worked successfully), but then when I retried the codes (by hitting 'run' button), I got the same warning.
Do you have any suggestion on how to fix this?
Thank you in advance.
!pip install torch torchvision
import torch
print('torch vision: ', torch.__version__)

!pip uninstall -y Pillow
!pip install Pillow==5.3.0
import PIL
print('pillow version: ', PIL.PILLOW_VERSION)
like so


title: How can I decrease Dedicated GPU memory usage and use Shared GPU memory for CUDA and Pytorch

body:
I'm getting following error when I try to use one of the huggingface models for sentimental analysis:
RuntimeError: CUDA out of memory. Tried to allocate 72.00 MiB (GPU 0; 3.00 GiB total capacity; 1.84 GiB already allocated; 5.45 MiB free; 2.04 GiB reserved in total by PyTorch)

Although I'm not using the CUDA memory it is still staying on the same level.
I tried to use torch.cuda.empty_cache() however it didn't affect the problem. When I closed the jupyter notebook it decreases to 0. So I am well sure that it is something with pytorch and python.
Here is my code:
import joblib
import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification,pipeline
import torch.nn.functional as F
from torch.utils.data import DataLoader
import pandas as pd
import numpy as np
from tqdm import tqdm

tokenizer = AutoTokenizer.from_pretrained("savasy/bert-base-turkish-sentiment-cased")
model = AutoModelForSequenceClassification.from_pretrained("savasy/bert-base-turkish-sentiment-cased")

sa= pipeline("sentiment-analysis", tokenizer=tokenizer, model=model,device=0)
batcher = DataLoader(dataset=comments,
                      batch_size=100,
                      shuffle=True,
                      pin_memory=True)
predictions= []
for batch in tqdm(batcher):
     p = sa(batch)
     predictions.append(p)

I have a GTX 1060, python 3.8 and torch==1.7.1 and my os is Windows 10.
And the count of comments are 187K. I would like to know if there is any work around for this memory issue. Maybe holding tensors somehow on CPU and only use batch on GPU. After using and getting this error the memory usage still continues. When I close my jupyter notebook it goes away. Is there any way that I can clear this memory ? Is there any way I can utilize Shared GPU memory ?


title: Error in training PyTorch classifier from the 60 minute blitz in GPU

body:
I've started to learn pytorch with their official 60 minute blitz tutorial in a jupyter lab (using their .ipynb file, link to the tutorial), and have completed it successfully until the conversion and training of the classifier using the gpu. I think that I have managed to change the device for the net, inputs and labels according to these results:

net=net.to(device)
net.fc1.weight.type()

With output:

'torch.cuda.FloatTensor'

And:

inputs, labels = inputs.to(device), labels.to(device)
inputs.type(),labels.type()

With output:

('torch.cuda.FloatTensor', 'torch.cuda.LongTensor')

After running these cells, I ran the cell for training the model, containing this code:

for epoch in range(2):  # loop over the dataset multiple times

running_loss = 0.0
for i, data in enumerate(trainloader, 0):
    # get the inputs
    inputs, labels = data

    # zero the parameter gradients
    optimizer.zero_grad()

    # forward + backward + optimize
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

    # print statistics
    running_loss += loss.item()
    if i % 2000 == 1999:    # print every 2000 mini-batches
        print('[%d, %5d] loss: %.3f' %
              (epoch + 1, i + 1, running_loss / 2000))
        running_loss = 0.0

print('Finished Training')

And received this error:

RuntimeError                              Traceback (most recent call last)
<ipython-input-55-fe85c778b0e6> in <module>()
     10
     11         # forward + backward + optimize
---> 12         outputs = net(inputs)
     13         loss = criterion(outputs, labels)
     14         loss.backward()

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self,
*input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

<ipython-input-52-725d44154459> in forward(self, x)
    14
    15     def forward(self, x):
--->16         x=self.conv1(x)
    17         x = self.pool(F.relu(x))
    18         x = self.pool(F.relu(self.conv2(x)))

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self,
*input, **kwargs)
    475             result = self._slow_forward(*input, **kwargs)
    476         else:
--> 477             result = self.forward(*input, **kwargs)
    478         for hook in self._forward_hooks.values():
    479             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torch\nn\modules\conv.py in forward(self,
input)
    299     def forward(self, input):
    300         return F.conv2d(input, self.weight, self.bias, self.stride,
--> 301                         self.padding, self.dilation, self.groups)
    302
    303

RuntimeError: Expected object of type torch.FloatTensor but found type
torch.cuda.FloatTensor for argument #2 'weight'

Why did I receive this error and how can I fix it?


title: Dict[str, Any] or Dict[str, Field] in pytext

body:
I'm reading the document of pytext (NLP modeling framework built on PyTorch) and this simple method from_config, a factory method to create a component from a config, has lines like Dict[str, Field] = {ExtraField.TOKEN_RANGE: RawField()}.

@classmethod
def from_config(cls, config: Config, model_input_config, target_config, **kwargs):
    model_input_fields: Dict[str, Field] = create_fields(
      model_input_config,
        {
            ModelInput.WORD_FEAT: TextFeatureField,
            ModelInput.DICT_FEAT: DictFeatureField,
            ModelInput.CHAR_FEAT: CharFeatureField,
        },
    )
    target_fields: Dict[str, Field] = {WordLabelConfig._name: WordLabelField.from_config(target_config)}
    extra_fields: Dict[str, Field] = {ExtraField.TOKEN_RANGE: RawField()}
    kwargs.update(config.items())
    return cls(
        raw_columns=config.columns_to_read,
        targets=target_fields,
        features=model_input_fields,
        extra_fields=extra_fields,
        **kwargs,
    )

and

    def preprocess(self, data: List[Dict[str, Any]]):
        tokens = []
        for row in data:
            tokens.extend(self.preprocess_row(row))
        return [{"text": tokens}]

How can a dictionary have keys with 2 items? What exactly is this?

I would appreciate any pointer!


title: PyTorch does not converge when approximating square function with linear model

body:
I'm trying to learn some PyTorch and am referencing this discussion here

The author provides a minimum working piece of code that illustrates how you can use PyTorch to solve for an unknown linear function that has been polluted with random noise.

This code runs fine for me.

However, when I change the function such that I want t = X^2, the parameter does not seem to converge.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.autograd import Variable

# Let's make some data for a linear regression.
A = 3.1415926
b = 2.7189351
error = 0.1
N = 100 # number of data points

# Data
X = Variable(torch.randn(N, 1))

# (noisy) Target values that we want to learn.
t = X * X + Variable(torch.randn(N, 1) * error)

# Creating a model, making the optimizer, defining loss
model = nn.Linear(1, 1)
optimizer = optim.SGD(model.parameters(), lr=0.05)
loss_fn = nn.MSELoss()

# Run training
niter = 50
for _ in range(0, niter):
    optimizer.zero_grad()
    predictions = model(X)
    loss = loss_fn(predictions, t)
    loss.backward()
    optimizer.step()

    print("-" * 50)
    print("error = {}".format(loss.data[0]))
    print("learned A = {}".format(list(model.parameters())[0].data[0, 0]))
    print("learned b = {}".format(list(model.parameters())[1].data[0]))

When I execute this code, the new A and b parameters are seemingly random thus it does not converge. I think this should converge because you can approximate any function with a slope and offset function. My theory is that I'm using PyTorch incorrectly.

Can any identify a problem with my t = X * X + Variable(torch.randn(N, 1) * error) line of code?


title: In PyTorch/Numpy, how to multiply rows of a matrix with "matrices" in a 3-D tensor?

body:
For example, a = torch.Tensor([[1,2],[3,4]]) (for numpy it is just a = np.array([[1,2],[3,4]])), and b = torch.ones((2,2,2)),

I would like to product every row of a with the two 2x2 matrices, and get a new matrix [[3,3],[7,7]] (i.e. [1,2]*[[1,1],[1,1]]=[3,3], [3,4]*[[1,1],[1,1]]=[7,7]). Is it possible to achieve this? Thanks!


title: flat_hash_map.h : error C3203: 'templated_iterator': unspecialized class template

body:
I've tried to build dcn but I got these errors:

 /torch/include\c10/util/order_preserving_flat_hash_map.h(1499): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1557): note: see reference to class template instantiation 'ska_ordered::order_preserving_flat_hash_map<K,V,H,E,A>' being compiled
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1503): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1511): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1593): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1630): note: see reference to class template instantiation 'ska_ordered::flat_hash_set<T,H,E,A>' being compiled
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1598): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1602): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1606): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/order_preserving_flat_hash_map.h(1610): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/flat_hash_map.h(1380): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/flat_hash_map.h(1438): note: see reference to class template instantiation 'ska::flat_hash_map<K,V,H,E,A>' being compiled
 /torch/include\c10/util/flat_hash_map.h(1384): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/flat_hash_map.h(1392): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/flat_hash_map.h(1474): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/flat_hash_map.h(1511): note: see reference to class template instantiation 'ska::flat_hash_set<T,H,E,A>' being compiled
 /torch/include\c10/util/flat_hash_map.h(1479): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/flat_hash_map.h(1483): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/flat_hash_map.h(1487): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type
 /torch/include\c10/util/flat_hash_map.h(1491): error C3203: 'templated_iterator': unspecialized class template can't be used as a template argument for template parameter '_Ty1', expected a real type

and this is command script that setup.py run it :

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\nvcc.exe -c src/deform_conv_cuda_kernel.cu -o build\temp.win-amd64-3.5\Release\src/deform_conv_cuda_kernel.obj -IC:\Users\127051\AppData\Local\Programs\Python\Python35\lib\site-packages\torch\include -IC:\Users\127051\AppData\Local\Programs\Python\Python35\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\127051\AppData\Local\Programs\Python\Python35\lib\site-packages\torch\include\TH -IC:\Users\127051\AppData\Local\Programs\Python\Python35\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include" -IC:\Users\127051\AppData\Local\Programs\Python\Python35\include -IC:\Users\127051\AppData\Local\Programs\Python\Python35\include "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\INCLUDE" "-IC:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\ATLMFC\INCLUDE" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.6.1\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" -Xcompiler /EHsc -Xcompiler /wd4819 -Xcompiler /MD -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=deform_conv_cuda -D_GLIBCXX_USE_CXX11_ABI=0 -gencode=arch=compute_50,code=sm_50
deform_conv_cuda_kernel.cu

I think the reason might be for my visual studio version that is VS 2015 but also I've install VS 2019 but i don't know how to force python setuptools to use VS2019, It default use VS2015.

thanks.


title: Obtaining computation graph structure of a forward pass in pytorch

body:
Is there any way that we can automatically extract a computation graph that represents the forward pass in a series of pytorch computations? We would like the nodes in the graph to have access to the functions that make up the forward pass, so that we can call them individually. We would also like to have access to the values of intermediate outputs of each function.
This would be something different from the computation graph constructed by autograd, which generally only represents the backward pass. Its nodes only contain a reference to a grad_fn, which is the derivative of the forward function.
For instance, if we provide the following code:
import torch

x = torch.randn(10)
M = torch.randn(10, 10)
h = torch.matmul(M, x)
h = torch.add(h, x)
h = torch.tanh(h)
y = torch.sum(h)

We would like to automatically extract a graph structure like
     y
     |
   [sum]
     |
   [tanh]
     |
   [add]
  /     \
  | [matmul]
  | /      \
  x         M

Where each bracketed node could be an object (call Node, for instance) that stores a reference to its corresponding function as an attribute, and stores the output of the function during the forward pass.
It would be great if anyone could point to any pytorch APIs that enables some kind of functionality like this!


title: Batch size keeps on changin, throwing `Pytorch Value Error Expected: input batch size does not match target batch size`

body:
I am working on a multilabel text classification task with Bert.
The following is the code for generating an iterable Dataset.
from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler

train_set = TensorDataset(X_train_id,X_train_attention, y_train)
test_set = TensorDataset(X_test_id,X_test_attention,y_test)

train_dataloader = DataLoader(
    train_set,
    sampler = RandomSampler(train_set),
    drop_last=True,
    batch_size=13
)

test_dataloader = DataLoader(
    test_set,
    sampler = SequentialSampler(test_set),
    drop_last=True,
    batch_size=13
)

The following are the the dimensions of the training set:
In[]
print(X_train_id.shape)
print(X_train_attention.shape)
print(y_train.shape)

Out[]
torch.Size([262754, 512])
torch.Size([262754, 512])
torch.Size([262754, 34])

There should be 262754 rows each with 512 columns. The output should predict the values from 34 possible labels. I am breaking them down into batches of 13.
Training code
optimizer = AdamW(model.parameters(), lr=2e-5)
# Training
def train(model):
    model.train()
    train_loss = 0
    for batch in train_dataloader:
        b_input_ids = batch[0].to(device)
        b_input_mask = batch[1].to(device)
        b_labels = batch[2].to(device)
        optimizer.zero_grad()
        loss, logits = model(b_input_ids,
                             token_type_ids=None,
                             attention_mask=b_input_mask,
                             labels=b_labels)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
        optimizer.step()
        train_loss += loss.item()
    return train_loss

# Testing
def test(model):
    model.eval()
    val_loss = 0
    with torch.no_grad():
        for batch in test_dataloader:
            b_input_ids = batch[0].to(device)
            b_input_mask = batch[1].to(device)
            b_labels = batch[2].to(device)
            with torch.no_grad():
                (loss, logits) = model(b_input_ids,
                                    token_type_ids=None,
                                    attention_mask=b_input_mask,
                                    labels=b_labels)
            val_loss += loss.item()
    return val_loss

# Train task
max_epoch = 1
train_loss_ = []
test_loss_ = []

for epoch in range(max_epoch):
    train_ = train(model)
    test_ = test(model)
    train_loss_.append(train_)
    test_loss_.append(test_)

Out[]
Expected input batch_size (13) to match target batch_size (442).

This is the description of my model:
from transformers import BertForSequenceClassification, AdamW, BertConfig

model = BertForSequenceClassification.from_pretrained(
    "cl-tohoku/bert-base-japanese-whole-word-masking", # 日本語Pre trainedモデル
    num_labels = 34,
    output_attentions = False,
    output_hidden_states = False,
)

I have clearly stated that I want the batch size to be 13. However, during the training process pytorch is throwing a Runtime Error
Where is the number 442 even coming from? I have clearly stated that I want each batch to have a size of 13 rows.
I have already confirmed that each batch has input_id with dimensions [13,512], attention tensor with dimensions [13,512], and labels with dimensions [13,34].
I have tried caving in and using a batch size of 442 when initializing the DataLoader, but after a single batch iteration, it throws another Pytorch Value Error Expected:  input batch size does not match target batch size, this time showing:
ValueError: Expected input batch_size (442) to match target batch_size (15028).

Why does the batch size keep on changing? Where is the number 15028 even coming from?
The following are some of the answers I have looked through, but had no luck on applying to my source code:
https://discuss.pytorch.org/t/valueerror-expected-input-batch-size-324-to-match-target-batch-size-4/24498
https://discuss.pytorch.org/t/valueerror-expected-input-batch-size-1-to-match-target-batch-size-64/43071
Pytorch CNN error: Expected input batch_size (4) to match target batch_size (64)
Thanks in advance. Your support is truly appreciated :)


title: How to solve the contradiction between Expected all tensors to be on the same device and can’t convert CUDA tensor to numpy

body:
I'm trying yolov3 with multi GPUs...

    def evaluate(self):
        self.models.eval()

        labels = []
        sample_metrics = []  # List of tuples (TP, confs, pred)
        for batch_i, (_, imgs, targets) in enumerate(tqdm.tqdm(self.valid_dataloader, desc="Detecting objects")):
            # Extract labels
            labels += targets[:, 1].tolist()
            # Rescale target
            targets[:, 2:] = xywh2xyxy(targets[:, 2:])
            targets[:, 2:] *= self.img_size

            #targets = targets.cuda()

            #imgs = Variable(imgs.type(Tensor), requires_grad=False)

            imgs = imgs.cuda()

            with torch.no_grad():
                outputs = self.models(imgs)
                outputs = non_max_suppression(outputs, conf_thres=self.conf_thres, nms_thres=self.nms_thres)

            sample_metrics += get_batch_statistics(outputs, targets, iou_threshold=self.iou_thres)

        # Concatenate sample statistics
        true_positives, pred_scores, pred_labels = [np.concatenate(x, 0) for x in list(zip(*sample_metrics))]
        precision, recall, AP, f1, ap_class = ap_per_class(true_positives, pred_scores, pred_labels, labels)

        return precision, recall, AP, f1, ap_class

If I use those two commented lines, I will face

TypeError: can’t convert CUDA tensor to numpy. Use Tensor.cpu() to copy the tensor to host memory first

But if I don't, I will face

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!

How to solve?
Trace:
    Traceback (most recent call last):
    File "D:/Code/DeblurGANv2-master/train.py", line 415, in <module>
        trainer.train()
    File "D:/Code/DeblurGANv2-master/train.py", line 253, in train
        self._run_epoch(epoch)
    File "D:/Code/DeblurGANv2-master/train.py", line 287, in _run_epoch
        loss_detect = self.calculate()
    File "D:/Code/DeblurGANv2-master/train.py", line 224, in calculate
        precision, recall, AP, f1, ap_class = self.evaluate()
    File "D:/Code/DeblurGANv2-master/train.py", line 160, in evaluate
        sample_metrics += get_batch_statistics(outputs, targets,
    iou_threshold=self.iou_thres)
    File "D:\Code\DeblurGANv2-master\util\utils.py", line 172, in
    get_batch_statistics
        if pred_label not in target_labels:
    File "D:\soft\Anaconda3\envs\DeblurGANv2-master\lib\site-
    packages\torch\tensor.py", line 659, in __contains__
        return (element == self).any().item()  # type: ignore[union-attr]
    File "D:\soft\Anaconda3\envs\DeblurGANv2-master\lib\site-
   packages\torch\tensor.py", line 27, in wrapped
        return f(*args, **kwargs)
    RuntimeError: Expected all tensors to be on the same device, but found
    at least two devices, cuda:0 and cpu!

I assume the problem is about the sample_metrics, no matter put this tensor on GPU or CPU, there are always problems...


title: Using pytorch cuda for RNNs on google colaboratory

body:
I have a code (a code we saw in a class) of a recurrent neural network that reads a given text and tries to produce its own text similar to the example. The code is written in python and uses the pytorch library. I wanted to modify to see whether I could increase its speed by using GPU instead of CPU and I made some tests on google collaboratory. The GPU version of the code runs fine but is about three times slower than the CPU version. I do not know the details of GPU architecture so I can not really understand why it is slower. I know that GPUs can do more arithmetic operations per cycle but have more limited memory so I am curious if I am having a memory issue. I also tried using CUDA with a generative adversarial network and in this case it was almost ten times faster. Any tips on this would be welcome.

The code (CUDA version) is below. I am new at this stuff so sorry if some of the terminology is not correct.
The architecture is input->encoder->recursive network->decoder->output.

import torch
import time
import numpy as np
from torch.autograd import Variable
import matplotlib.pyplot as plt
from google.colab import files

#uploding text on google collab
uploaded = files.upload()

for fn in uploaded.keys():
  print('User uploaded file "{name}" with length {length} bytes'.format(
      name=fn, length=len(uploaded[fn])))

#data preprocessing

with open('text.txt','r') as file:
#with open closes the file after we are done with it
    rawtxt=file.read()

rawtxt = rawtxt.lower()

#a function that assigns a number to each unique character in the text
def create_map(rawtxt):
    letters = list(set(rawtxt))
    lettermap = dict(enumerate(letters)) #gives each letter in the list a number
    return lettermap

num_to_let = create_map(rawtxt)
#inverse to num_to_let
let_to_num =dict(zip(num_to_let.values(), num_to_let.keys()))

print(num_to_let)

#turns a text of characters into text of numbers using the mapping
#given by the input mapdict
def maparray(txt, mapdict):

    txt = list(txt)

    for k, letter in enumerate(txt):
        txt[k]=mapdict[letter]

    txt=np.array(txt)
    return txt

X=maparray(rawtxt, let_to_num) #the data text in numeric format
Y= np.roll(X, -1, axis=0) #shifted data text in numeric format

X=torch.LongTensor(X)
Y=torch.LongTensor(Y)

#up to here we are done with data preprocessing

#return a random batch for training
#this reads a random piece inside data text
#with the size chunk_size
def random_chunk(chunk_size):
    k=np.random.randint(0,len(X)-chunk_size)
    return X[k:k+chunk_size], Y[k:k+chunk_size]

nchars = len(num_to_let)

#define the recursive neural network class
class rnn(torch.nn.Module):
    def __init__(self,input_size,hidden_size,output_size, n_layers=1):
        super().__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.n_layers= n_layers

        self.encoder = torch.nn.Embedding (input_size, hidden_size)
        self.rnn =  torch.nn.RNN(hidden_size, hidden_size, n_layers, batch_first=True)
        self.decoder = torch.nn.Linear (hidden_size, output_size)

    def forward (self,x,hidden):
        x=self.encoder(x.view(1,-1))

        output, hidden = self.rnn(x.view(1,1,-1), hidden)
        output = self.decoder(output.view(1,-1))

        return output, hidden

    def init_hidden(self):
        return Variable(torch.zeros(self.n_layers , 1 , self.hidden_size)).cuda()

#hyper-params
lr = 0.009
no_epochs = 50
chunk_size = 150

myrnn = rnn(nchars, 150, nchars,1)
myrnn.cuda()

criterion = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(myrnn.parameters(), lr=lr)

t0 = time.time()

for epoch in range(no_epochs):
    totcost=0
    generated = ''

    for _ in range(len(X)//chunk_size):
        h=myrnn.init_hidden()
        cost = 0
        x, y=random_chunk(chunk_size)
        x, y= Variable(x).cuda(), Variable(y).cuda()

        for i in range(chunk_size):
            out, h = myrnn.forward(x[i],h)

            _, outl = out.data.max(1)
            letter = num_to_let[outl[0]]
            generated+=letter

            cost += criterion(out, y[i])

        optimizer.zero_grad()
        cost.backward()
        optimizer.step()
        totcost+=cost
        totcost/=len(X)//chunk_size

    print('Epoch', epoch, 'Avg cost/chunk: ', totcost)
    print(generated[0:750],'\n\n\n')

t1 = time.time()
total = t1-t0

print('total',total)

#we encode each word into a vector of fixed size


title: Pytorch nn.Parameter only updates for first epoch

body:
I am trying to implement a custom Module (this is simplified version), and my inspecting the variable self.param during each iteration the value doesn't change after the first iteration even though the gradient has a value. Does anyone familiar with pytorch have any idea why this is happening?

class Custom_RNN(nn.Module):
    def __init__(self):
        super(Custom_RNN, self).__init__()
        self.hl1 = nn.Linear(2, 1024)
        self.hl2 = nn.Linear(1024, 512)
        self.out = nn.Linear(512, 1)
        self.param = nn.Parameter(torch.Tensor([250]), requires_grad=True)

    def forward(self, state_init, input, output):

        combined = torch.cat((state_init, input[0]), 1)
        out = torch.sigmoid(self.hl1(combined))
        out = torch.sigmoid(self.hl2(out))
        out = self.out(out)

        for t in range(input.shape[1]):

            # Predict SoC
            shifted_out = out + input[t]/self.param

            combined = torch.cat((shifted_out, input[t]), 1)
            out = torch.sigmoid(self.hl1(combined))
            out = torch.sigmoid(self.hl2(out))
            out = self.out(out)
            if first:
                loss = torch.pow(out - output[t], 2.0)
            else:
                loss = loss + torch.pow(out - output[t], 2.0)

        return loss

rnn = Custom_RNN()
optimiser = optim.Adam(rnn.parameters())

for epoch in range(epochs):
    optimiser.zero_grad()
    loss = rnn(initial_cond, data_in, data_out)
    rnn_param_before = rnn.param.item()
    loss.backward()
    optimiser.step()
    rnn_param_after = rnn.param.item()
    print(rnn_param_before - rnn_param_after)
    print(rnn.param.item(), rnn.param.grad)

The first epoch the first print gets a number that is not zeros, then every other epoch it is 0.0 and the second print statement shows that the value remains the same and the grad is non zero always.


title: How to discover torch model name from the model itself?

body:
I am creating a built in torch model in this way:

m = torchvision.models.resnet50()

and I want to discover the model's name (resnet50 in this case) using the variable m afterwards.
When calling type(m) I get torchvision.models.resnet.ResNet which is almost the answer but the depth is important to me as well. I could try to iterate m.modules (which is 151 modules long in this way) and infer the resblocks depth but that would work entirely different in other models (like inception, alexnet and so on).


title: nvprof warning on CUDA_VISIBLE_DEVICES

body:
When I use os.environ['CUDA_VISIBLE_DEVICES'] in pytorch, I get the following message

Warning: Device on which events/metrics are configured are different than the device on which it is being profiled. One of the possible reason is setting CUDA_VISIBLE_DEVICES inside the application.

What does this actually mean? How can I avoid this by using 'CUDA_VISIBLE_DEVICES' (not torch.cuda.set_device())?

Here is the code in pytorch test.py

import torch
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '1'
g = 1
c1 = 512
c2 = 512
input = torch.randn(64, c1, 28, 28).cuda()
model = nn.Sequential(
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
      nn.Conv2d(c1,c2,1,groups=g),
      nn.ReLU(),
    ).cuda()
out = model(input)

and the command:

nvprof --analysis-metrics -o metrics python test.py


title: How to create a train-val split in custom image datasets using PyTorch?

body:
I want to create a train+val set from my original trainset. The directory is split into train and test. I load the original train set and want to split it into train and val sets so I can evaulate validation loss during training using the train_loader and val_loader.

There's not a lot of documentation about this which explains things clearly.


title: Understanding tf.nn.depthwise_conv2d

body:
From
https://www.tensorflow.org/api_docs/python/tf/nn/depthwise_conv2d

  Given a 4D input tensor ('NHWC' or 'NCHW' data formats) and a filter
  tensor of shape [filter_height, filter_width, in_channels,
  channel_multiplier] containing in_channels convolutional filters of
  depth 1, depthwise_conv2d applies a different filter to each input
  channel (expanding from 1 channel to channel_multiplier channels for
  each), then concatenates the results together. The output has
  in_channels * channel_multiplier channels

What does it mean "expanding from 1 channel to channel_multiplier channels for each" ?
Is it possible to have out_channels < in_channels?
Is it possible to divide input tensor to groups like in Pytorch https://pytorch.org/docs/stable/nn.html#conv2d?

Example:

import tensorflow as tf
import numpy as np
import os

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '1'
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)

np.random.seed(2020)

print('tf.__version__', tf.__version__)

def get_data_batch():
    bs = 2
    h = 3
    w = 3
    c = 4

    x_np = np.random.rand(bs, h, w, c)
    x_np = x_np.astype(np.float32)
    print('x_np.shape', x_np.shape)

    return x_np

def run_conv_dw():
    print('='*60)
    x_np = get_data_batch()
    in_channels = x_np.shape[-1]
    kernel_size = 3
    channel_multiplier = 1
    with tf.Session() as sess:
        x_tf = tf.convert_to_tensor(x_np)
        filter = tf.get_variable('w1', [kernel_size, kernel_size, in_channels, channel_multiplier],
                                 initializer=tf.contrib.layers.xavier_initializer())
        z_tf = tf.nn.depthwise_conv2d(x_tf, filter=filter, strides=[1, 1, 1, 1], padding='SAME')

        sess.run(tf.global_variables_initializer())
        z_np = sess.run(fetches=[z_tf], feed_dict={x_tf: x_np})[0]
        print('z_np.shape', z_np.shape)

if '__main__' == __name__:
    run_conv_dw()

Channel multiplier can't be float:

If channel_multiplier = 1:

x_np.shape (2, 3, 3, 4)
z_np.shape (2, 3, 3, 4)

If channel_multiplier = 2:

x_np.shape (2, 3, 3, 4)
z_np.shape (2, 3, 3, 8)


title: transformers: how to use hugging face EncoderDecoderModel to do machine translation task?

body:
I have trained a EncoderDecoderModel from huggging face to do english-German translation task. I tried to overfit a small dataset (100 parallel sentences), and use model.generate() then tokenizer.decode() to perform the translation. However, the output seems to be proper German sentences, but it is definitely not the correct translation.
Here are the code for building the model
encoder_config = BertConfig()
decoder_config = BertConfig()
config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config, decoder_config)
model = EncoderDecoderModel(config=config)

Here are the code for testing the model
model.eval()
input_ids = torch.tensor(tokenizer.encode(input_text)).unsqueeze(0)
output_ids = model.generate(input_ids.to('cuda'), decoder_start_token_id=model.config.decoder.pad_token_id)
output_text = tokenizer.decode(output_ids[0])

Example input: "iron cement is a ready for use paste which is laid as a fillet by putty knife or finger in the mould edges ( corners ) of the steel ingot mould ."
Ground truth translation: "iron cement ist eine gebrauchs ##AT##-##AT## fertige Paste , die mit einem Spachtel oder den Fingern als Hohlkehle in die Formecken ( Winkel ) der Stahlguss -Kokille aufgetragen wird ."
What the model outputs after trained 100 epochs: "[S] wenn sie den unten stehenden link anklicken, sehen sie ein video uber die erstellung ansprechender illustrationen in quarkxpress" which is totally nonesense.
Where is the problem?


title: Get values of tensors in loss function

body:
I would like to get the values of the y_pred and y_true tensors of this keras backend function. I need this to be able to perform some custom calculations and change the loss, these calculations are just possible with the real array values.

def mean_squared_error(y_true, y_pred):
    #some code here
    return K.mean(K.square(y_pred - y_true), axis=-1)

There is a way to do this in keras? Or in any other ML framework (tf, pytorch, theano)?


title: Unexpected increase in validation error in MNIST Pytorch

body:
I'm a bit new to the whole field and thus decided to work on the MNIST dataset. I pretty much adapted the whole code from https://github.com/pytorch/examples/blob/master/mnist/main.py, with only one significant change: Data Loading. I didn't want to use the pre-loaded dataset within Torchvision. So I used MNIST in CSV.

I loaded the data from CSV file by inheriting from Dataset and making a new dataloader.
Here's the relevant code:

mean = 33.318421449829934
sd = 78.56749081851163
# mean = 0.1307
# sd = 0.3081
import numpy as np
from torch.utils.data import Dataset, DataLoader

class dataset(Dataset):
    def __init__(self, csv, transform=None):
        data = pd.read_csv(csv, header=None)
        self.X = np.array(data.iloc[:, 1:]).reshape(-1, 28, 28, 1).astype('float32')
        self.Y = np.array(data.iloc[:, 0])

        del data
        self.transform = transform

    def __len__(self):
        return len(self.X)

    def __getitem__(self, idx):
        item = self.X[idx]
        label = self.Y[idx]

        if self.transform:
            item = self.transform(item)

        return (item, label)

import torchvision.transforms as transforms
trainData = dataset('mnist_train.csv', transform=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((mean,), (sd,))
]))
testData = dataset('mnist_test.csv', transform=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((mean,), (sd,))
]))

train_loader = DataLoader(dataset=trainData,
                         batch_size=10,
                         shuffle=True,
                         )
test_loader = DataLoader(dataset=testData,
                        batch_size=10,
                        shuffle=True,
                        )

However this code gives me the absolutely weird training error graph that you see in the picture, and a final validation error of 11% because it classifies everything as a '7'.

I managed to track the problem down to how I normalize the data and if I use the values given in the example code (0.1307, and 0.3081) for transforms.Normalize, along with reading the data as type 'uint8' it works perfectly.
Note that there is very minimal difference in the data which is provided in these two cases. Normalizing by 0.1307 and 0.3081 on values from 0 to 1 has the same effect as normalizing by 33.31 and 78.56 on values from 0 to 255. The values are even mostly the same (A black pixel corresponds to -0.4241 in the first case and -0.4242 in the second).

If you would like to see a IPython Notebook where this problem is seen clearly, please check out https://colab.research.google.com/drive/1W1qx7IADpnn5e5w97IcxVvmZAaMK9vL3

I am unable to understand what has caused such a huge difference in behaviour within these two slightly different ways of loading data. Any help would be massively appreciated.


title: using torchtext with sequences of text as input

body:
I'm trying to use torchtext for sequence labeling where the sequences are sentences. So each Example is a list of sentences and their tags:

[
  {
    'texts': ["An example.", "It's the data.", "I'm talking about it."],
    'labels': [0,2,0]
  },
  { 'texts': ["This is another example", "This is also data."],
    'labels': [0,1]
  }
]

Of course I still want tokenization and vocabulary at the word level, but
there doesn't seem to be an obvious way to do this. For example, TabularDataset expects a list of Fields that apply to an element of each example, but I want to map the text field across the list of sentences that constitutes the input.

I thought maybe this is what NestedField was for, but it seems to be specifically geared towards character embeddings.

Am I missing something? Is this possible (without too much trouble) with torchtext? I'm perfectly willing to accept no as an answer :)


title: Can't install Caffe2 on Ubuntu 18.04

body:
I am trying to install Facebook's Detectron and so I need to install Caffe2. I followed this instruction.
Basically, I clone the PyTorch repo with
git clone https://github.com/pytorch/pytorch.git && cd pytorch  git submodule update --init --recursive  and when I run the following command  python setup.py install
it goes to 84-86% and I get this error.
[ 84%] Built target roi_align_op_gpu_test
Makefile:140: recipe for target 'all' failed
make: *** [all] Error 2
Traceback (most recent call last):
  File "setup.py", line 756, in <module>
    build_deps()
  File "setup.py", line 321, in build_deps
    cmake=cmake)
  File "/home/milos/pytorch/tools/build_pytorch_libs.py", line 63, in build_caffe2
    cmake.build(my_env)
  File "/home/milos/pytorch/tools/setup_helpers/cmake.py", line 330, in build
    self.run(build_args, my_env)
  File "/home/milos/pytorch/tools/setup_helpers/cmake.py", line 143, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '12']' returned non-zero exit status 2
Segmentation fault (core dumped)

System info:

Ubuntu 18.04 LTS (Bionic Beaver)
Python 3.6.8 and Python 2.7.15
OpenCV 3.4.0 for both python versions
CUDA 10.2 (Nvidia 430.26)

I've managed to get this working before, but can't remember how.


title: Understanding Memory Usage by PyTorch DataLoader Workers

body:
When running a PyTorch training program with num_workers=32 for DataLoader, htop shows 33 python process each with 32 GB  of VIRT and 15 GB of RES.
Does this mean that the PyTorch training is using 33 processes X 15 GB = 495 GB of memory? htop shows only about 50 GB of RAM and 20 GB of swap is being used on the entire machine with 128 GB of RAM. So, how do we explain the discrepancy?
Is there a more accurate way of calculating the total amount of RAM being used by the main PyTorch program and all its child DataLoader worker processes?
Thank you


title: PyTorch Tutorial freeze_support() issue

body:
I tried following the tutorial from PyTorch here: https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py.
Full code is here:
import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

# Loading and normalizing CIFAR10
transform = transforms.Compose(
    [transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

# Shows training images, DOESN'T WORK

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()

# show images
imshow(torchvision.utils.make_grid(images))
# print labels
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

# define a convolutional neural network
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()

# Define a loss function and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

# Train the network
for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0

    # DOESN'T WORK
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels = data

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

# save trained model
PATH = './cifar_net.pth'
torch.save(net.state_dict(), PATH)

# test the network on the test data
dataiter = iter(testloader)
images, labels = dataiter.next()

# print images
dataiter = iter(testloader)
images, labels = dataiter.next()
imshow(torchvision.utils.make_grid(images))
print('GroundTruth: ', ' '.join('%5s' % classes[labels[j]] for j in range(4)))

# load back saved model
net = Net()
net.load_state_dict(torch.load(PATH))

# see what the nueral network thinks these examples above are:
ouputs = net(images)

# index of the highest energy
_, predicted = torch.max(outputs, 1)

print('Predicted: ', ' '.join('%5s' % classes[predicted[j]]
                              for j in range(4)))

# accuracy on the whole dataset
correct = 0
total = 0
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))

# classes that perfomed well vs classes that didn't perform well
class_correct = list(0. for i in range(10))
class_total = list(0. for i in range(10))
with torch.no_grad():
    for data in testloader:
        images, labels = data
        outputs = net(images)
        _, predicted = torch.max(outputs, 1)
        c = (predicted == labels).squeeze()
        for i in range(4):
            label = labels[i]
            class_correct[label] += c[i].item()
            class_total[label] += 1

for i in range(10):
    print('Accuracy of %5s : %2d %%' % (
        classes[i], 100 * class_correct[i] / class_total[i]))

if __name__ == '__main__':
    torch.multiprocessing.freeze_support()

However I got this issue:
An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

I'm just trying to run this in a regular python file. When I added
if __name__ == '__main__':
                freeze_support()

to the end of my file, I still get the error.


title: TypeError: forward() missing 1 required positional argument: 'hidden' - LSTM Model

body:
I'm a beginner to the LSTM and PyTorch. I try to create a model for a bursty traffic prediction scenario. This is kind of a overfit model. First, It matches all the next values using the x_data as inputs and targets. You can identify the shape of the x_data by looking at the main. Then try to predict the whole traffic shape using the first 100 seed values. But this model gives me an error when running "outputs, hidden = model(inputs)". please help me to solve this error.
Here is my complete code,
import numpy as np
import torch
import torch,torch.nn as nn
from torch import Tensor
from torch.autograd import Variable
from torch import optim
import torch.nn.functional as F
from scipy.signal import savgol_filter

is_cuda = torch.cuda.is_available()

if is_cuda:
    device = torch.device("cuda")
    print("GPU is available")
else:
    device = torch.device("cpu")
    print("GPU not available, CPU used")

class FPredRNN(nn.Module):
  def __init__(self, input_size, hidden_size, num_layers, dropout_val = 0.1):
    super(FPredRNN, self).__init__()
    self.input_size = input_size
    self.nh = hidden_size
    self.nl = num_layers

    self.lstm = nn.LSTM(self.input_size, self.nh, self.nl, dropout = dropout_val)
    self.dropout = nn.Dropout(dropout_val)
    self.linear = nn.Linear(self.nh, 1)

  def forward(self, x, hidden, steps = 1000, eval = False):
    predictions = []
    batch_size = x.size(0)
    if(hidden.size(0) != batch_size):
      self.init_hidden(batch_size)

    l_out, hidden = self.lstm(x, hidden)
    l_out = l_out.contiguous().view(-1, self.nh)

    out = self.dropout(l_out)
    out = self.linear(out)
    #out = out.view(batch_size, -1)
    #out = out[:,-1]

    if(eval):
      eval_input = out[-1:]
      for i in range(steps):
        lstm_out, hidden = self.lstm(eval_input, hidden)
        linear_out = self.linear(lstm_out)
        predictions += [linear_out]
        eval_input = linear_out
      out = torch.stack(predictions).squeeze()

    return out, hidden

  def init_hidden(self, batch_size):
    weight = next(self.parameters()).data
    hidden = (weight.new(self.nl, batch_size, self.nh).zero_().to(device), weight.new(self.nl, batch_size, self.nh).zero_().to(device))
    #hidden = ((self.nl, batch_size, self.nh).zero_().to(device), (self.nl, batch_size, self.nh).zero_().to(device))
    return hidden

if __name__ == "__main__":
  x_data = np.empty((1, 2000))
  y_data = np.empty((1, 1))

  for n in [30000]:
    traffic_generator = GenerateTraffic()
    bursty_traffic, a_t = traffic_generator.create_bursty_traffic(n_d=n)
    detected, attempted = traffic_generator.simulate_bursty_traffic_arrivals(bursty_traffic, backoff_bool= True)
    smooth_x = savgol_filter(detected, 97, 2)
    x_data[(n//10000)-3] = smooth_x

  inputs = x_data[:, :1999]
  targets = x_data[:, 1:2000]

  inputs = torch.from_numpy(inputs)
  targets = torch.from_numpy(targets)

  print(inputs.size(1))
  print(inputs.size(0))
  #print(hidden.size(0))

  model = FPredRNN(input_size = inputs.size(1), hidden_size = 1100, num_layers = 2, dropout_val = 0.1)
  model.to(device)

  criterion = nn.MSELoss()
  optimizer = torch.optim.Adam(model.parameters(), lr = 0.0001)

  # Train Model
  n_epochs = 2
  for epoch in range(1, n_epochs + 1):
    optimizer.zero_grad()
    inputs.to(device)
    outputs, hidden = model(inputs)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()

# Test Model
  seed_lenght = 100
  seed = inputs[:seed_lenght]
  outt = model(seed, steps=1000, eval = True)
  test_out = torch.cat((seed.squeeze(), outt))

Error code
TypeError                                 Traceback (most recent call last)
<ipython-input-2-0e63f8c64103> in <module>()
   101     optimizer.zero_grad()
   102     inputs.to(device)
--> 103     outputs, hidden = model(inputs)
   104     loss = criterion(outputs, targets)
   105     loss.backward()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   725             result = self._slow_forward(*input, **kwargs)
   726         else:
--> 727             result = self.forward(*input, **kwargs)
   728         for hook in itertools.chain(
   729                 _global_forward_hooks.values(),

TypeError: forward() missing 1 required positional argument: 'hidden'


title: Unable to download saved model from online resource, pickle error

body:
I am unable to download and use a model I saved earlier from a online-repository. Here's the code:

model = Model().double()   # Model is defined in another class
state_dict = torch.hub.load_state_dict_from_url(r'https://filebin.net/j2977ux7kts41aft/checkpoint_best.pt?t=wjbujfoo')
model.load_state_dict(state_dict)
model.eval()

Which gives me the following error:

Traceback (most recent call last):
  File "/path/file.py", line 47, in <module>
    state_dict = torch.hub.load_state_dict_from_url(r'https://filebin.net/j2977ux7kts41aft/checkpoint_best.pt?t=wjbujfoo')
  File "anaconda3/envs/torch_env/lib/python3.6/site-packages/torch/hub.py", line 466, in load_state_dict_from_url
    return torch.load(cached_file, map_location=map_location)
  File "/anaconda3/envs/torch_env/lib/python3.6/site-packages/torch/serialization.py", line 386, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "anaconda3/envs/torch_env/lib/python3.6/site-packages/torch/serialization.py", line 563, in _load
    magic_number = pickle_module.load(f, **pickle_load_args)
_pickle.UnpicklingError: invalid load key, '\x0a'.

The model resides in:
https://filebin.net/j2977ux7kts41aft/checkpoint_best.pt?t=wjbujfoo

Note that I can perfectly download it manually, and then use torch.load(path) to load it without errors, but I need to do it from code! Could it be that the serializing when downloading from url somehow messes up the pickle encoding?

Edit: I don't have to use filebin, any online-storage that supports what I try to do will suffice.


title: PyTorch BatchNorm1D, 2D, 3D and TensorFlow/Keras BatchNormalization

body: I want to use BatchNorm1D like in PyTorch in TensorFlow. I notice that BatchNormalization() in TF has axis=-1 as default. Which axis is the correct one for BatchNorm1D, BatchNorm2D, BatchNorm3D as in PyTorch?


title: How to solve "RuntimeError: expected scalar type Double but found Float" when loading torchscript model in C++

body:
I try to load a torchscript model in C++, but got an error

RuntimeError: expected scalar type Double but found Float.

The full output is
$ ./example-app ../turb_nn.pt
1 2 3 4 5
 1  2  3  4  5
[ CPUDoubleType{1,5} ]
ok
run forward
terminate called after throwing an instance of 'std::runtime_error'
  what():  The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
  File "<string>", line 3, in forward

      def addmm(self: Tensor, mat1: Tensor, mat2: Tensor, beta: number = 1.0, alpha: number = 1.0):
          return self + mat1.mm(mat2)
                        ~~~~~~~ <--- HERE

      def batch_norm(input : Tensor, running_mean : Optional[Tensor], running_var : Optional[Tensor], training : bool, momentum : float, eps : float) -> Tensor:
RuntimeError: expected scalar type Double but found Float

Aborted (core dumped)

The input data is actually double type. I tried to use .double() following here but cannot run make.
My code:
#include <torch/script.h> // One-stop header.

#include <iostream>
#include <memory>

int main(int argc, const char* argv[]) {
  std::vector<double> inputdata({ 1.0, 2.0, 3.0, 4.0, 5.0 });
  std::cout << inputdata << std::endl;

  auto opts = torch::TensorOptions().dtype(torch::kDouble);
  torch::Tensor input = torch::from_blob(inputdata.data(), {1, (int)inputdata.size()}, opts).to(torch::kDouble);
  std::cout << input << std::endl;

  if (argc != 2) {
    std::cerr << "usage: example-app <path-to-exported-script-module>\n";
    return -1;
  }

  torch::jit::script::Module module;
  try {
    // Deserialize the ScriptModule from a file using torch::jit::load().
    module = torch::jit::load(argv[1]);
  }
  catch (const c10::Error& e) {
    std::cerr << "error loading the model\n";
    return -1;
  }

  std::cout << "ok" << std::endl;

  std::vector<torch::jit::IValue> input_invar;
  input_invar.push_back(input);
  std::cout << "run forward" << std::endl;
  at::Tensor output = module.forward(input_invar).toTensor();

  std::vector<double> out_vect(output.data_ptr<double>(), output.data_ptr<double>() + output.numel());

  std::cout << out_vect << std::endl;
}

Could you give me some suggestions, please ?


title: Prediction from Already Saved Weights and Configuration file using Detectron2

body:
I have weights in .pth file and confgiguarion file in YAML format, i want to perform prediction (Detection of Building) on test images and utilizing the following code for this purpose:
from detectron2.engine import DefaultPredictor
import torch
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog
import cv2

# get image
!wget http://images.cocodataset.org/val2017/000000439715.jpg -O input.jpg
 im = cv2.imread("./input.jpg")

 # Create config
 cfg = get_cfg()
 cfg.merge_from_file("./detectron2_repo/configs/COCO-Detectio/faster_rcnn_R_101_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5  # set threshold for this model
cfg.MODEL.WEIGHTS = "detectron2://COCO-Detection/faster_rcnn_R_101_FPN_3x/137851257/model_final_f6e8b1.pkl"

# Create predictor
predictor = DefaultPredictor(cfg)

# Make prediction
outputs = predictor(im)

but when i load my own configuration fle, it is giving this following error


title: In google-cloud-ml, setup.py error during setting up PYTORCH

body:
i write setup.py like below to setup pytorch in google-cloud-ml engine

from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['torchvision']
DEPENDENCY_LINKS =['http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl']
setup(
    name='trainer',
    version='0.1',
    dependency_links=DEPENDENCY_LINKS,
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My pytorch trainer application package.'
)

error message

"Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-_ZQ7aQ/torch/"

i don't know what it happen....
when i searched about that problem... the answers are just upgrade the setuptools... but i don't know how to upgrade setuptools in ml-engine...

so please help me...!
i want to run pytorch code in ml engine


title: How to crop features outside an image region using pytorch?

body:
We can use ROI-Pool/ROI-Align to crop the sub-features inside an image region (which is a rectangle).

I was wondering how to crop features outside this region.
In other words, how to set values (of a feature map) inside a rectangle region to zero, but values outside the region remains unchanged.


title: Computing Output Pixel-wise Gradient Norm in PyTorch

body:
I am looking for an efficient way of computing \hat{x} of dimensions (b x c x h x w) defined per sample as:

where x is the output of the same dimensions generated by a model with parameters \theta, and

i,j: index the height and width of the 2D output feature map

k: indexes the channel dimension

l: indexes the parameters.

How do I accomplish this with x.backward()? Is there an efficient way of doing this?

If I did x.backward(torch.ones_like(x)), I would compute the sum of the gradients in the above equation in the place of sum of their absolute values.

Brute-force way to do this would be to loop over each pixel in the output map and compute .backward(). But, that would be incredibly expensive. Is there an efficient way which avoids the loop?


title: TypeError: object() takes no parameters is encountered when practicing basic pytorch dataloader

body:
Dear all pro:

I tried the tutorial in the link of https://pythonprogramming.net/training-deep-learning-neural-network-pytorch/?completed=/building-deep-learning-neural-network-pytorch/ and played around with the code on the google colab notebook.

The part of importing and dataloading the MNIST dataset is fine. But when i tried to iterate through the dataset , it complained the error showed in the picture. I do not understand why this happend.


title: Operator translate error occurs when I try to convert onnx file to caffe2

body:
I train a boject detection model on pytorch, and I have exported to onnx file.

And I want to convert it to caffe2 model :

import onnx
import caffe2.python.onnx.backend as onnx_caffe2_backend

# Load the ONNX ModelProto object. model is a standard Python protobuf object
model = onnx.load("CPU4export.onnx")

# prepare the caffe2 backend for executing the model this converts the ONNX model into a
# Caffe2 NetDef that can execute it. Other ONNX backends, like one for CNTK will be
# availiable soon.
prepared_backend = onnx_caffe2_backend.prepare(model)

# run the model in Caffe2

# Construct a map from input names to Tensor data.
# The graph of the model itself contains inputs for all weight parameters, after the input image.
# Since the weights are already embedded, we just need to pass the input image.
# Set the first input.
W = {model.graph.input[0].name: x.data.numpy()}

# Run the Caffe2 net:
c2_out = prepared_backend.run(W)[0]

# Verify the numerical correctness upto 3 decimal places
np.testing.assert_almost_equal(torch_out.data.cpu().numpy(), c2_out, decimal=3)

print("Exported model has been executed on Caffe2 backend, and the result looks good!")

I always got this error :

RuntimeError: ONNX conversion failed, encountered 1 errors:

Error while processing node: input: "90"
input: "91"
output: "92"
op_type: "Resize"
attribute {
  name: "mode"
  s: "nearest"
  type: STRING
}
. Exception: Don't know how to translate op Resize

How can I solve it ?