title: Iterating through DataLoader (PyTorch): RuntimeError: Expected object of scalar type unsigned char but got scalar type float for sequence element 9

body:
I am new to PyTorch and am running into an expected error. The overall context is trying to build a building segmentation model off of Spacenet imagery. I am forked off of this repo from someone at Microsoft AI who built a segmentation model, and I am just trying to re-run her training scripts.
I've been able to download the data, and do the pre-processing. My issue comes when trying to actually train the model, I am trying to iterate through my DataLoader, and I get the following error message:

RuntimeError: Expected object of scalar type unsigned char but got
scalar type float for sequence element 9.

Snippets of code that are useful:
I have a dataset.py that creates the SpaceNetDataset class and looks like:
import os
# Ignore warnings
import warnings

import numpy as np
from PIL import Image
import torch
from torch.utils.data import Dataset

warnings.filterwarnings('ignore')

class SpaceNetDataset(Dataset):
    """Class representing a SpaceNet dataset, such as a training set."""

    def __init__(self, root_dir, splits=['trainval', 'test'], transform=None):
        """
        Args:
            root_dir (string): Directory containing folder annotations and .txt files with the
            train/val/test splits
            splits: ['trainval', 'test'] - the SpaceNet utilities code would create these two
                splits while converting the labels from polygons to mask annotations. The two
                splits are created after chipping larger images into the required input size with
                some overlaps. Thus to have splits that do not have overlapping areas, we manually
                split the images (not chips) into train/val/test using utils/split_train_val_test.py,
                followed by using the SpaceNet utilities to annotate each folder, and combine the
                trainval and test splits it creates inside each folder.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.root_dir = root_dir
        self.transform = transform
        self.image_list = []
        self.xml_list = []

        data_files = []
        for split in splits:
            with open(os.path.join(root_dir, split + '.txt')) as f:
                data_files.extend(f.read().splitlines())

        for line in data_files:
            line = line.split(' ')

            image_name = line[0].split('/')[-1]
            xml_name = line[1].split('/')[-1]

            self.image_list.append(image_name)
            self.xml_list.append(xml_name)

    def __len__(self):
        return len(self.image_list)

    def __getitem__(self, idx):
        img_path = os.path.join(self.root_dir, 'RGB-PanSharpen', self.image_list[idx])
        target_path = os.path.join(self.root_dir, 'annotations', self.image_list[idx].replace('.tif', 'segcls.tif'))

        image = np.array(Image.open(img_path))
        target = np.array(Image.open(target_path))
        target[target == 100] = 1  # building interior
        target[target == 255] = 2  # border

        sample = {'image': image, 'target': target, 'image_name': self.image_list[idx]}

        if self.transform:
            sample = self.transform(sample)

        return sample

To create the DataLoader, I have something like:
dset_train = SpaceNetDataset(data_path_train, split_tags, transform=T.Compose([ToTensor()]))
loader_train = DataLoader(dset_train, batch_size=train_batch_size, shuffle=True,
                          num_workers=num_workers)

I then iterate over the data loader by doing something like:
    for batch in loader_train:
        image_tensors = batch['image']
        images = batch['image'].cpu().numpy()
        break  # take the first shuffled batch

but then I get the error:
Traceback (most recent call last):

File "training/train_aml.py", line 137, in <module> sample_images_train, sample_images_train_tensors = get_sample_images(which_set='train')
File "training/train_aml.py", line 123, in get_sample_images for i, batch in enumerate(loader):
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 345, in __next__ data = self._next_data() File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 856, in _next_data return self._process_data(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 881, in _process_data data.reraise()
File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise raise self.exc_type(msg)

RuntimeError: Caught RuntimeError in DataLoader worker process 0. Original Traceback (most recent call last):

File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop data = fetcher.fetch(index)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch return self.collate_fn(data)
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in default_collate return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 74, in <dictcomp> return {key: default_collate([d[key] for d in batch]) for key in elem}
File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 55, in default_collate return torch.stack(batch, 0, out=out)

RuntimeError: Expected object of scalar type unsigned char but got scalar type float for sequence element 9.

The error seems quite similar to this one, although I did try a similar solution by casting:
    dtype = torch.cuda.CharTensor if torch.cuda.is_available() else torch.CharTensor
    for batch in loader:
        batch['image'] = batch['image'].type(dtype)
        batch['target'] = batch['target'].type(dtype)

but I end up with the same error.
A couple of other things that are weird:

This seems to be non-deterministic. Most of the time I get this error, but some times the code keeps running (not sure why)
The "Sequence Element" number at the end of the error message keeps changing. In this case it was "sequence element 9" sometimes it's "sequence element 2", etc. Not sure why.


title: How to access individual data points of MNIST data and check their size, shape etc from trainset object using torchvision's import dataset

body:
the following code is
    from torchvision import datasets, transforms
    trainset = datasets.MNIST('./data/', download=True, train=True, transform=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))]))

I would like to visualize the first data point in the trainset variable above.
I want to have a look at the pixel values of the first data point by doing something like print(trainset[0]) or check the size by doing print(trainset[0].size) or check the shape by doing print(trainset[0].shape) etc.


title: cyclic learning rate with multiple learning rates

body:
I am trying to use the OneCycleLR or atleast the cyclicLR in torch.optim.lr_scheduler.
Suppose I have the following:
param_list = []
for lr, block in zip(lrs, blocks):
    param_list.extend([{'params':p ,'lr':lr} for n,p in model.named_parameters() if n.startswith(block)])
optimizer = torch.optim.Adam(param_list)

where blocks = ["base", "fc"] (in my use case there is ~20 blocks) and lrs=[1e-4, 1e-3].
It is easy enough to control the learning rates manually by using a function, eg:
lr_sched = lambda batch: 1.1**batch
scheduler = LambdaLR(optimizer, lr_lambda=[lr_sched]*len(param_list))

The above example increases the learning rate.
However, what I would like to do is change the learning rate as well as the momentum parameters as offered in OneCycleLR. So my question is:

Is it possible?
If not, is there a way to manipulate the momentum while training, and I can write a function for the cyclic learning rate myself.
Is it possible to use a list of optimizers instead of one, if so is that slower?

Minimal example:
import torch
import torch.nn as nn
from torch.optim.lr_scheduler import LambdaLR

class Model(nn.Module):
    def __init__(self):
        self.base = nn.Linear(10, 5)
        self.fc = nn.Linear(5, 1)
        self.relu = nn.ReLU()

    def forward(self, x):
        return self.fc(self.relu(self.base(x)))

model = Model()

param_list = []
for lr, block in zip(lrs, blocks):
    param_list.extend([{'params':p ,'lr':lr} for n,p in model.named_parameters() if n.startswith(block)])
optimizer = torch.optim.Adam(param_list)

lr_sched = lambda batch: 1.1**batch
scheduler = LambdaLR(optimizer, lr_lambda=[lr_sched]*len(param_list))


title: why skimage.io.imread warns “corrupt exif data” but cv2.imread doesn't?

body:
I am using pytorch to build a neural network classifier.
I read somewhere that many python packages use the same c++ api when reading a picture, so I thought skimage.io.imread and cv2.imread would be the same except channel order. However I got warning like this in skimage,
/xxx/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:766: UserWarning: Possibly corrupt EXIF data.  Expecting to read 1245184 bytes but only got 0. Skipping tag 0
  " Skipping tag %s" % (size, len(data), tag)
/xxx/lib/python3.6/site-packages/PIL/TiffImagePlugin.py:785: UserWarning: Corrupt EXIF data.  Expecting to read 12 bytes but only got 11.
  warnings.warn(str(msg))

but cv2 works without warnings.
My code goes like this:

in Dataset

    ...

    def __getitem__(self, index):
        file = os.path.join(self.root_dir, self.files[index])
        image = io.imread(file)
        # image = cv2.imread(file)
        if self.transform:
            image = self.transform(image)
        return image

transforms:

transform = transforms.Compose([
    transforms.ToPILImage(),
    transforms.Resize((128, 128)),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor()
])


title: How to split tensor to small chunk for matrix multiplication to solve memory issue

body:
I have two matrix:
torch.Size([1, 8, 50176, 64]) torch.Size([1, 8, 64, 50176])
I need to perform matrix mutiplicatio to get torch size [1, 8, 50176, 50176], but I got cuda out of memory error and requires 75GB GPU memory, I cannot afford that much GPU, my one is 8GB, I am wondering how to spit the matrix into small chunk and perform mutiplication and finally concat resuts in one torch tensor?
I have tried:
y = torch.matmul(a, b)
y = torch.bmm(a.view(-1, 50176, 64), b.view(-1, 64, 50176)).view(1, 8, 50176, 50176)
y = torch.einsum('bdij, bdjk -> bdik', (a, b))

none of the above work, both got cuda out of memory error.
Thank you for helping.
Ling


title: How do I use two loss for two dataset in one pytorch nn?

body:
I am quite new to pytorch and deep learning. Here is my question. I have two different datasets with the same feature domain sharing one neural network for a regression problem. The input is the features and the output is the target value. The first dataset uses a normal loss while the second dataset, I am trying to create a new loss for it.
I have searched multi-loss problems, people usually have two loss summed up for the backward process. But I want to use the loss in turn. (When I train the first dataset, the nn uses the first loss and when I train the second dataset, the nn uses the other loss)
Is this possible to do? Appreciate if anyone has some idea.


title: torch.mul causes param.grad to be NoneType

body:
class Net(torch.nn.Module):
    def __init__(self, D_u, D_i, D_t, D_m):
        super(Net, self).__init__()
        self.lin_u = nn.Linear(D_u, 1)
        self.lin_i = nn.Linear(D_i, 1)
        self.lin_t = nn.Linear(D_t, 1)
        self.lin_m = nn.Linear(D_m, 1)

        self.output = nn.Linear(4, 1)

    def forward(self, args):
        (u, i, t, m) = args
        u = F.relu(self.lin_u(u))
        i = F.relu(self.lin_i(i))
        t = F.relu(self.lin_t(t))
        m = F.relu(self.lin_m(m))
        out = torch.mul(u, i)
        out = torch.mul(out, t)
        out = torch.mul(out, m)
        return out

I have this simple model class which has four inputs, each with its own linear layer. I want the output to be the product of the four nodes, but for some reason no matter how I multiply them (with torch.mul or *), the grad is always Nonetype:
model = Net(N, 3, T, 1)
u_block, i_block, t_block, m_block, y_block = get_data_new(data)

loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-4
for t in range(5000):
    y_pred = model((u_block, i_block, t_block, m_block))

    loss = loss_fn(y_pred, y_block)
    if t % 100 == 99:
        print(t, loss.item())

    model.zero_grad()

    loss.backward()

    with torch.no_grad():
        for param in model.parameters():
          param -= learning_rate * param.grad

TypeError
--->   param -= learning_rate * param.grad

TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

I've set the inputs to requires_grad=True, I think the issue is that out is not a leaf and therefore has no gradient, but I don't know how to fix this.
Edit:
The data u_block, i_block, t_block, m_block, y_block are shown below. u_block, i_block, and t_block are one-hot vectors.
u_block:  tensor([[1., 0., 0.,  ..., 0., 0., 0.],
        [1., 0., 0.,  ..., 0., 0., 0.],
        [1., 0., 0.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 0., 0., 1.],
        [0., 0., 0.,  ..., 0., 0., 1.],
        [0., 0., 0.,  ..., 0., 0., 1.]], requires_grad=True)
i_block:  tensor([[1., 0., 0.],
        [1., 0., 0.],
        [1., 0., 0.],
        ...,
        [0., 1., 0.],
        [0., 1., 0.],
        [0., 1., 0.]], requires_grad=True)
t_block:  tensor([[1., 0., 0.,  ..., 0., 0., 0.],
        [0., 1., 0.,  ..., 0., 0., 0.],
        [0., 0., 1.,  ..., 0., 0., 0.],
        ...,
        [0., 0., 0.,  ..., 1., 0., 0.],
        [0., 0., 0.,  ..., 0., 1., 0.],
        [0., 0., 0.,  ..., 0., 0., 1.]], requires_grad=True)
m_block:  tensor([[ 0.0335],
        [ 0.0000],
        [ 0.0000],
        ...,
        [ 0.1515],
        [-0.2261],
        [-0.0402]], requires_grad=True)
y_block:  tensor([[ 0.0000],
        [ 0.0000],
        [ 0.0000],
        ...,
        [-0.2261],
        [-0.0402],
        [-0.1318]], requires_grad=True)```


title: How can I convert PyTorch Tensor to ndarray of Numpy?

body:
The shape of my tensor is torch.Size([3, 320, 480])
Tensor is
tensor([[[0.2980, 0.4353, 0.6431,  ..., 0.2196, 0.2196, 0.2157],
         [0.4235, 0.4275, 0.5569,  ..., 0.2353, 0.2235, 0.2078],
         [0.5608, 0.5961, 0.5882,  ..., 0.2314, 0.2471, 0.2588],
         ...,

         ...,
         [0.0588, 0.0471, 0.0784,  ..., 0.0392, 0.0471, 0.0745],
         [0.0275, 0.1020, 0.1882,  ..., 0.0196, 0.0157, 0.0471],
         [0.1569, 0.2353, 0.2471,  ..., 0.0549, 0.0549, 0.0627]]])

I need something of shape 320, 480, 3 I guess
So, the tensor should look like this
array([[[0.29803923, 0.22352941, 0.10980392],
        [0.43529412, 0.34117648, 0.20784314],
        [0.6431373 , 0.5254902 , 0.3764706 ],
        ...,

        ...,
        [0.21960784, 0.13333334, 0.05490196],
        [0.23529412, 0.14509805, 0.05490196],
        [0.2627451 , 0.1764706 , 0.0627451 ]]], dtype=float32)


title: Pytorch BiDirectional RNN not working: RuntimeError: Expected hidden[0] size (2, 76, 6), got (2, 500, 6)

body:
class BiLSTMnetwork(nn.Module):
    def __init__(self, in_features, hidden_sz=6, p=0.5, out_features=2):
    #def __init__(self, embedding_dim, hidden_dim, vocab_size, target_size):
        super(BiLSTMnetwork, self).__init__()
        self.hidden_sz = hidden_sz

        self.in_features = in_features

        #Define the initial linear hidden layer
        #self.init_linear = nn.Linear(self.in_features, self.in_features)

        # The LSTM takes word embeddings as inputs, and outputs hidden states
        # with dimensionality hidden_dim.
        self.lstm = nn.LSTM(self.in_features, hidden_sz,batch_first=True,bidirectional=True)

        # The linear layer that maps from hidden state space to tag space
        self.linear = nn.Linear(self.hidden_sz*2, out_features)

        # Initialize h0 and c0:
        self.hidden = (torch.zeros(2,500,self.hidden_sz),
                       torch.zeros(2,500,self.hidden_sz))

    def forward(self,x):
        lstm_out, self.hidden = self.lstm(
            x.view(len(x), 1, -1), self.hidden)

   # Can pass on the entirety of lstm_out to the next layer if it is a seq2seq prediction
    pred = self.linear(lstm_out.view(len(x),-1))
    y_val = F.log_softmax(pred, dim=1)
    return y_val

for _ in range(epochs):
    # using data_loader
    for i, (data, labels) in enumerate(trainloader):
        # Forward and get a prediction
        # x is the training data which is X_train
        if name.lower() == "rnn":
            model.hidden = (torch.zeros(1,1,model.hidden_sz),
                torch.zeros(1,1,model.hidden_sz))
        if name.lower() == "birnn":
            model.hidden = (torch.zeros(2,500,model.hidden_sz),
                torch.zeros(2,500,model.hidden_sz))

        y_pred = model.forward(data)

        # compute loss/error by comparing predicted out vs acutal labels
        loss = criterion(y_pred, labels)
        #losses.append(loss)

        if i%10==0:  # print out loss at every 10 epoch
            print(f'epoch {i} and loss is: {loss}')

        #Backpropagation
        optimizer.zero_grad()
        loss.backward(retain_graph=False)
        optimizer.step()

hi all I am doing a BiDirectional RNN as seen above, as u can see the hidden state as been defined as (2,500,self.hidden_sz) , where self.hidden_sz = 6.
I ran the code and it managed to pass for the first 2 epoch, during the 3rd epoch and error popped up below. Really need ur help , my idea is this could be because of the bidirectional nature of BiRNN and hidden states cannot just be solely defined for 2 direction. Correct me if I am wrong , and help me with the code, thanks!
Running birnn classifier on dcuf.csv.gz features...
Evaluation 1
epoch 0 and loss is: 0.6595679521560669
epoch 10 and loss is: 0.6663855910301208
Traceback (most recent call last):
  File "dltanh.py", line 496, in <module>
    n_instances, n_features, scores = run_analysis()
  File "dltanh.py", line 361, in run_analysis
    y_pred = model.forward(data)
  File "dltanh.py", line 258, in forward
    x.view(len(x), 1, -1), self.hidden)
  File "/home/shar/.local/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/shar/.local/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 564, in forward
    return self.forward_tensor(input, hx)
  File "/home/shar/.local/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 543, in forward_tensor
    output, hidden = self.forward_impl(input, hx, batch_sizes, max_batch_size, sorted_indices)
  File "/home/shar/.local/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 523, in forward_impl
    self.check_forward_args(input, hx, batch_sizes)
  File "/home/shar/.local/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 500, in check_forward_args
    'Expected hidden[0] size {}, got {}')
  File "/home/shar/.local/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 166, in check_hidden_size
    raise RuntimeError(msg.format(expected_hidden_size, tuple(hx.size())))
RuntimeError: Expected hidden[0] size (2, 76, 6), got (2, 500, 6)
(base) shar@smag3:~/msoftx/scriptx$


title: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: PyTorch error

body:
I am trying to run some code in PyTorch but I got stacked at this point:
At first iteration, both backward operations, for Discriminator and Generator are running well
....

self.G_loss.backward(retain_graph=True)

self.D_loss.backward()

...

At the second iteration, when self.G_loss.backward(retain_graph=True) executes, I get this error:
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8192, 512]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

According to torch.autograd.set_detect_anomaly, the last of the following lines in the Discriminator network, is responsible for this:
    bottleneck = bottleneck[:-1]
    self.embedding = x.view(x.size(0), -1)
    self.logit = self.layers[-1](self.embedding)

The strange thing is that I have used that network architecture in other code where it worked properly. Any suggestions?
The full error:
    site-packages\torch\autograd\__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [8192, 512]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


title: use opencv to draw my bounding boxes in the image, but the bbox never shows

body:
This is part of my code. Basically, I trained my own model, and I want to predict where the boxes are and draw them, and output the images with the bbox. But the bbox never shows...
def get_transform(train):
    transforms = []
    transforms.append(Transforms.ToTensor())
    if train:
        transforms.append(Transforms.RandomHorizontalFlip(0.5))
    return Transforms.Compose(transforms)

def showbbox(model, image):

    model.eval()
    image = ToTensor()(image).unsqueeze(0)
    image = Variable(image)
    with torch.no_grad():
        prediction = model(image)

    print(prediction)

    image = image.permute(0, 2,3,1)
    image = (image * 255).byte().data.cpu()
    image =np.array(image)

    for i in range(prediction[0]['boxes'].cpu().shape[0]):
        xmin = round(prediction[0]['boxes'][i][0].item())
        ymin = round(prediction[0]['boxes'][i][1].item())
        xmax = round(prediction[0]['boxes'][i][2].item())
        ymax = round(prediction[0]['boxes'][i][3].item())

        label = prediction[0]['labels'][i].item()

        if label == 1:
            image = np.ascontiguousarray(image)
            cv2.rectangle(image, (xmin, ymin), (xmax, ymax), (255, 0, 0), 2)
            cv2.putText(image, 'mark_type_1', (xmin, ymin), cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 0, 0),
                        thickness=2)

cv2.imwrite("test.png", image)
image_name = 'image'
cv2.namedWindow(image_name, cv2.WINDOW_NORMAL)

I also attach my log here: it shows the coordinates of the predicted bboxes:
[{'boxes': tensor([[ 205.3901,  874.5291, 2172.6992, 3289.5364],
        [  10.5118,  837.6425,  977.6479, 3137.6086],
        [   0.0000,   45.9050, 1582.4817, 2750.2344],
        [ 558.1544,  133.1127, 2530.4922, 2671.6353],
        [ 917.5895, 1817.0457, 2532.4092, 3297.6543],
        [1203.6671,  660.2758, 2538.5496, 3149.8977],
        [ 205.8660,   14.9926, 2237.4023, 1690.5226],
        [1062.8268,    0.0000, 2541.5366, 1408.9767]]), 'labels': tensor([1, 1, 1, 1, 1, 1, 1, 1]), 'scores': tensor([0.2706, 0.2349, 0.2335, 0.2269, 0.2250, 0.2125, 0.2105, 0.1781])}]


title: TypeError: a bytes-like object is required, not 'Tensor' in TensorRT

body:
I am trying to use tensorrt to speed up my model inference.
Now, I have got the onnx file of my model, and I also have got the engine of my model.
This is the input of my model in the onnx file.
a = torch.ones(64, 30, 25, 3).cuda()

x=torch.onnx.export(model,  # 待转换的网络模型和参数
                (a, a, a, a), # 虚拟的输入，用于确定输入尺寸和推理计算图每个节点的尺寸
                'ConVNet.onnx',  # 输出文件的名称
                verbose=False,      # 是否以字符串的形式显示计算图
                input_names=input_name,  # 输入节点的名称，这里也可以给一个list，list中名称分别对应每一层可学习的参数，便于后续查询
                output_names=["output"], # 输出节点的名称
                opset_version=10,   # onnx 支持采用的operator set, 应该和pytorch版本相关，目前我这里最高支持10
                do_constant_folding=True, # 是否压缩常量
                dynamic_axes={"input":{0: "batch_size", 1: "30个骨架",2:"维度为3",3:"骨架"}, "output":{0: "batch_size"},} #设置动态维度，此处指明input节点的第0维度可变，命名为batch_size
                )

I get this error when I trying to transform onnx to tensorrt.
Reading engine from file: ./ConVNet.engine
the bindding's name: frame_1
bingding shape:  (64, 30, 25, 3)
the bindding's name: frame_2
bingding shape:  (64, 30, 25, 3)
the bindding's name: d_1
bingding shape:  (64, 30, 25, 3)
the bindding's name: d_2
bingding shape:  (64, 30, 25, 3)
the bindding's name: 140
bingding shape:  (64, 4096)
the bindding's name: output
bingding shape:  (64, 60)

Traceback (most recent call last):
  File "onnx2rt.py", line 196, in <module>
    trt_outputs = do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)  # numpy data
  File "onnx2rt.py", line 159, in do_inference
    [cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs]
  File "onnx2rt.py", line 159, in <listcomp>
    [cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs]
TypeError: a bytes-like object is required, not 'tuple'

and this is how can I try to do inference.
a = torch.ones(64, 30, 25, 3).cuda()
inputs[0].host = (a, a, a, a)
trt_outputs = do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)

If anyone can help me I will be grateful!
===============================================
I have resloved this question. just use like this：
a = np.ones((64, 30, 25, 3),dtype=np.float32)
inputs[0].host = (a.reshape(-1))
inputs[1].host = (a.reshape(-1))
inputs[2].host = (a.reshape(-1))
inputs[3].host = (a.reshape(-1))


title: variational auto encoder loss goes down but does not reconstruct input. out of debugging ideas. works on mnist but not on other data

body:
My variational autoencoder seems to work for MNIST, but fails on slightly "harder" data.
By "fails" I mean there are at least two apparent problems:

Very poor reconstruction, for example sample reconstructions from the last epoch on validation set

without any regularization at all.
The last reported losses from console are val_loss=9.57e-5, train_loss=9.83e-5 which I thought would imply exact reconstructions.
validation loss is low (which does not seem to reflect the reconstruction), and always lower than training loss which is very suspicious.

For MNIST everything looks fine (with less layers!).

I will give as much nformation as I can, since I am not sure what I should provide to help anyone help me.

Firstly, here is the full code
You will notice loss calculation and logging is very simple and straight forward and I can't seem to find what's wrong.
import torch
from torch import nn
import torch.nn.functional as F
from typing import List, Optional, Any
from pytorch_lightning.core.lightning import LightningModule
from Testing.Research.config.ConfigProvider import ConfigProvider
from pytorch_lightning import Trainer, seed_everything
from torch import optim
import os
from pytorch_lightning.loggers import TensorBoardLogger
# import tfmpl
import matplotlib.pyplot as plt
import matplotlib
from Testing.Research.data_modules.MyDataModule import MyDataModule
from Testing.Research.data_modules.MNISTDataModule import MNISTDataModule
from Testing.Research.data_modules.CaseDataModule import CaseDataModule
import torchvision
from Testing.Research.config.paths import tb_logs_folder
from Testing.Research.config.paths import vae_checkpoints_path
from pytorch_lightning.callbacks.model_checkpoint import ModelCheckpoint

class VAEFC(LightningModule):
    # see https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73
    # for possible upgrades, see https://arxiv.org/pdf/1602.02282.pdf
    # https://stats.stackexchange.com/questions/332179/how-to-weight-kld-loss-vs-reconstruction-loss-in-variational
    # -auto-encoder
    def __init__(self, encoder_layer_sizes: List, decoder_layer_sizes: List, config):
        super(VAEFC, self).__init__()
        self._config = config
        self.logger: Optional[TensorBoardLogger] = None
        self.save_hyperparameters()

        assert len(encoder_layer_sizes) >= 3, "must have at least 3 layers (2 hidden)"
        # encoder layers
        self._encoder_layers = nn.ModuleList()
        for i in range(1, len(encoder_layer_sizes) - 1):
            enc_layer = nn.Linear(encoder_layer_sizes[i - 1], encoder_layer_sizes[i])
            self._encoder_layers.append(enc_layer)

        # predict mean and covariance vectors
        self._mean_layer = nn.Linear(encoder_layer_sizes[
                                         len(encoder_layer_sizes) - 2],
                                     encoder_layer_sizes[len(encoder_layer_sizes) - 1])
        self._logvar_layer = nn.Linear(encoder_layer_sizes[
                                           len(encoder_layer_sizes) - 2],
                                       encoder_layer_sizes[len(encoder_layer_sizes) - 1])

        # decoder layers
        self._decoder_layers = nn.ModuleList()
        for i in range(1, len(decoder_layer_sizes)):
            dec_layer = nn.Linear(decoder_layer_sizes[i - 1], decoder_layer_sizes[i])
            self._decoder_layers.append(dec_layer)

        self._recon_function = nn.MSELoss(reduction='mean')
        self._last_val_batch = {}

    def _encode(self, x):
        for i in range(len(self._encoder_layers)):
            layer = self._encoder_layers[i]
            x = F.relu(layer(x))

        mean_output = self._mean_layer(x)
        logvar_output = self._logvar_layer(x)
        return mean_output, logvar_output

    def _reparametrize(self, mu, logvar):
        if not self.training:
            return mu
        std = logvar.mul(0.5).exp_()
        if std.is_cuda:
            eps = torch.FloatTensor(std.size()).cuda().normal_()
        else:
            eps = torch.FloatTensor(std.size()).normal_()
        reparameterized = eps.mul(std).add_(mu)
        return reparameterized

    def _decode(self, z):
        for i in range(len(self._decoder_layers) - 1):
            layer = self._decoder_layers[i]
            z = F.relu((layer(z)))

        decoded = self._decoder_layers[len(self._decoder_layers) - 1](z)
        # decoded = F.sigmoid(self._decoder_layers[len(self._decoder_layers)-1](z))
        return decoded

    def _loss_function(self, recon_x, x, mu, logvar, reconstruction_function):
        """
        recon_x: generating images
        x: origin images
        mu: latent mean
        logvar: latent log variance
        """
        binary_cross_entropy = reconstruction_function(recon_x, x)  # mse loss TODO see if mse or cross entropy
        # loss = 0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
        kld_element = mu.pow(2).add_(logvar.exp()).mul_(-1).add_(1).add_(logvar)
        kld = torch.sum(kld_element).mul_(-0.5)
        # KL divergence Kullback–Leibler divergence, regularization term for VAE
        # It is a measure of how different two probability distributions are different from each other.
        # We are trying to force the distributions closer while keeping the reconstruction loss low.
        # see https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73

        # read on weighting the regularization term here:
        # https://stats.stackexchange.com/questions/332179/how-to-weight-kld-loss-vs-reconstruction-loss-in-variational
        # -auto-encoder
        return binary_cross_entropy + kld * self._config.regularization_factor

    def _parse_batch_by_dataset(self, batch, batch_index):
        if self._config.dataset == "toy":
            (orig_batch, noisy_batch), label_batch = batch
            # TODO put in the noise here and not in the dataset?
        elif self._config.dataset == "mnist":
            orig_batch, label_batch = batch
            orig_batch = orig_batch.reshape(-1, 28 * 28)
            noisy_batch = orig_batch
        elif self._config.dataset == "case":
            orig_batch, label_batch = batch

            orig_batch = orig_batch.float().reshape(
                    -1,
                    len(self._config.case.feature_list) * self._config.case.frames_per_pd_sample
            )
            noisy_batch = orig_batch
        else:
            raise ValueError("invalid dataset")
        noisy_batch = noisy_batch.view(noisy_batch.size(0), -1)

        return orig_batch, noisy_batch, label_batch

    def training_step(self, batch, batch_idx):
        orig_batch, noisy_batch, label_batch = self._parse_batch_by_dataset(batch, batch_idx)

        recon_batch, mu, logvar = self.forward(noisy_batch)

        loss = self._loss_function(
                recon_batch,
                orig_batch, mu, logvar,
                reconstruction_function=self._recon_function
        )
        # self.logger.experiment.add_scalars("losses", {"train_loss": loss})
        tb = self.logger.experiment
        tb.add_scalars("losses", {"train_loss": loss}, global_step=self.current_epoch)
        # self.logger.experiment.add_scalar("train_loss", loss, self.current_epoch)
        if batch_idx == len(self.train_dataloader()) - 2:
            # https://pytorch.org/docs/stable/_modules/torch/utils/tensorboard/writer.html#SummaryWriter.add_embedding
            # noisy_batch = noisy_batch.detach()
            # recon_batch = recon_batch.detach()
            # last_batch_plt = matplotlib.figure.Figure()  # read https://github.com/wookayin/tensorflow-plot
            # ax = last_batch_plt.add_subplot(1, 1, 1)
            # ax.scatter(orig_batch[:, 0], orig_batch[:, 1], label="original")
            # ax.scatter(noisy_batch[:, 0], noisy_batch[:, 1], label="noisy")
            # ax.scatter(recon_batch[:, 0], recon_batch[:, 1], label="reconstructed")
            # ax.legend(loc="upper left")
            # self.logger.experiment.add_figure(f"original last batch, epoch {self.current_epoch}", last_batch_plt)
            # tb.add_embedding(orig_batch, global_step=self.current_epoch, metadata=label_batch)
            pass
        self.logger.experiment.flush()
        self.log("train_loss", loss, prog_bar=True, on_step=False, on_epoch=True)
        return loss

    def _plot_batches(self, orig_batch, noisy_batch, label_batch, batch_idx, recon_batch, mu, logvar):
        # orig_batch_view = orig_batch.reshape(-1, self._config.case.frames_per_pd_sample,
        # len(self._config.case.feature_list))
        #
        # plt.figure()
        # plt.plot(orig_batch_view[11, :, 0].detach().cpu().numpy(), label="feature 0")
        # plt.legend(loc="upper left")
        # plt.show()

        tb = self.logger.experiment
        if self._config.dataset == "mnist":
            orig_batch -= orig_batch.min()
            orig_batch /= orig_batch.max()
            recon_batch -= recon_batch.min()
            recon_batch /= recon_batch.max()

            orig_grid = torchvision.utils.make_grid(orig_batch.view(-1, 1, 28, 28))
            val_recon_grid = torchvision.utils.make_grid(recon_batch.view(-1, 1, 28, 28))

            tb.add_image("original_val", orig_grid, global_step=self.current_epoch)
            tb.add_image("reconstruction_val", val_recon_grid, global_step=self.current_epoch)

            label_img = orig_batch.view(-1, 1, 28, 28)
            pass
        elif self._config.dataset == "case":
            orig_batch_view = orig_batch.reshape(-1, self._config.case.frames_per_pd_sample,
                                                 len(self._config.case.feature_list)).transpose(1, 2)
            recon_batch_view = recon_batch.reshape(-1, self._config.case.frames_per_pd_sample,
                                                   len(self._config.case.feature_list)).transpose(1, 2)

            # plt.figure()
            # plt.plot(orig_batch_view[11, 0, :].detach().cpu().numpy())
            # plt.show()
            # pass

            n_samples = orig_batch_view.shape[0]
            n_plots = min(n_samples, 4)
            first_sample_idx = 0

            # TODO either plotting or data problem
            fig, axs = plt.subplots(n_plots, 1)
            for sample_idx in range(n_plots):
                for feature_idx, (orig_feature, recon_feature) in enumerate(
                        zip(orig_batch_view[sample_idx + first_sample_idx, :, :],
                            recon_batch_view[sample_idx + first_sample_idx, :, :])):
                    i = feature_idx
                    if i > 0: continue  # or scale issues don't allow informative plotting

                    # plt.figure()
                    # plt.plot(orig_feature.detach().cpu().numpy(), label=f'orig{i}, sample{sample_idx}')
                    # plt.legend(loc='upper left')
                    # pass

                    axs[sample_idx].plot(orig_feature.detach().cpu().numpy(), label=f'orig{i}, sample{sample_idx}')
                    axs[sample_idx].plot(recon_feature.detach().cpu().numpy(), label=f'recon{i}, sample{sample_idx}')
                    # sample{sample_idx}')
                    axs[sample_idx].legend(loc='upper left')
                pass
            # plt.show()

            tb.add_figure("recon_vs_orig", fig, global_step=self.current_epoch, close=True)

    def validation_step(self, batch, batch_idx):
        orig_batch, noisy_batch, label_batch = self._parse_batch_by_dataset(batch, batch_idx)

        recon_batch, mu, logvar = self.forward(noisy_batch)

        loss = self._loss_function(
                recon_batch,
                orig_batch, mu, logvar,
                reconstruction_function=self._recon_function
        )

        tb = self.logger.experiment
        # can probably speed up training by waiting for epoch end for data copy from gpu
        # see https://sagivtech.com/2017/09/19/optimizing-pytorch-training-code/
        tb.add_scalars("losses", {"val_loss": loss}, global_step=self.current_epoch)

        label_img = None
        if len(orig_batch) > 2:
            self._last_val_batch = {
                "orig_batch": orig_batch,
                "noisy_batch": noisy_batch,
                "label_batch": label_batch,
                "batch_idx": batch_idx,
                "recon_batch": recon_batch,
                "mu": mu,
                "logvar": logvar
            }
        # self._plot_batches(orig_batch, noisy_batch, label_batch, batch_idx, recon_batch, mu, logvar)

        outputs = {"val_loss":  loss, "recon_batch": recon_batch, "label_batch": label_batch,
                   "label_img": label_img}
        self.log("val_loss", loss, prog_bar=True, on_step=False, on_epoch=True)
        return outputs

    def validation_epoch_end(self, outputs: List[Any]) -> None:
        first_batch_dict = outputs[-1]

        self._plot_batches(
                self._last_val_batch["orig_batch"],
                self._last_val_batch["noisy_batch"],
                self._last_val_batch["label_batch"],
                self._last_val_batch["batch_idx"],
                self._last_val_batch["recon_batch"],
                self._last_val_batch["mu"],
                self._last_val_batch["logvar"]
        )
        self.log(name="VAEFC_val_loss_epoch_end", value={"val_loss": first_batch_dict["val_loss"]})

    def test_step(self, batch, batch_idx):
        orig_batch, noisy_batch, label_batch = self._parse_batch_by_dataset(batch, batch_idx)

        recon_batch, mu, logvar = self.forward(noisy_batch)

        loss = self._loss_function(
                recon_batch,
                orig_batch, mu, logvar,
                reconstruction_function=self._recon_function
        )

        tb = self.logger.experiment
        tb.add_scalars("losses", {"test_loss": loss}, global_step=self.global_step)

        return {"test_loss": loss, "mus": mu, "labels": label_batch, "images": orig_batch}

    def test_epoch_end(self, outputs: List):
        tb = self.logger.experiment

        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()
        self.log(name="test_epoch_end", value={"test_loss_avg": avg_loss})

        if self._config.dataset == "mnist":
            tb.add_embedding(
                    mat=torch.cat([o["mus"] for o in outputs]),
                    metadata=torch.cat([o["labels"] for o in outputs]).detach().cpu().numpy(),
                    label_img=torch.cat([o["images"] for o in outputs]).view(-1, 1, 28, 28),
                    global_step=self.global_step,
            )

    def configure_optimizers(self):
        optimizer = optim.Adam(self.parameters(), lr=self._config.learning_rate)
        return optimizer

    def forward(self, x):
        mu, logvar = self._encode(x)
        z = self._reparametrize(mu, logvar)
        decoded = self._decode(z)
        return decoded, mu, logvar

def train_vae(config, datamodule, latent_dim, dec_layer_sizes, enc_layer_sizes):
    model = VAEFC(config=config, encoder_layer_sizes=enc_layer_sizes, decoder_layer_sizes=dec_layer_sizes)

    logger = TensorBoardLogger(save_dir=tb_logs_folder, name='VAEFC', default_hp_metric=False)
    logger.hparams = config

    checkpoint_callback = ModelCheckpoint(dirpath=vae_checkpoints_path)
    trainer = Trainer(deterministic=config.is_deterministic,
                      # auto_lr_find=config.auto_lr_find,
                      # log_gpu_memory='all',
                      # min_epochs=99999,
                      max_epochs=config.num_epochs,
                      default_root_dir=vae_checkpoints_path,
                      logger=logger,
                      callbacks=[checkpoint_callback],
                      gpus=1
                      )
    # trainer.tune(model)
    trainer.fit(model, datamodule=datamodule)
    best_model_path = checkpoint_callback.best_model_path
    print("done training vae with lightning")
    print(f"best model path = {best_model_path}")
    return trainer

def run_trained_vae(trainer):
    # https://pytorch-lightning.readthedocs.io/en/latest/test_set.html
    # (1) load the best checkpoint automatically (lightning tracks this for you)
    trainer.test()

    # (2) don't load a checkpoint, instead use the model with the latest weights
    # trainer.test(ckpt_path=None)

    # (3) test using a specific checkpoint
    # trainer.test(ckpt_path='/path/to/my_checkpoint.ckpt')

    # (4) test with an explicit model (will use this model and not load a checkpoint)
    # trainer.test(model)

Parameters
I am getting very similar results for any combination of parameters I am (manually) using. Maybe I didn't try something.
num_epochs: 40
batch_size: 32
learning_rate: 0.0001
auto_lr_find: False

noise_factor: 0.1
regularization_factor: 0.0

train_size: 0.8
val_size: 0.1
num_workers: 1

dataset: "case" # toy, mnnist, case
mnist:
  enc_layer_sizes: [784, 512,]
  dec_layer_sizes: [512, 784]
  latent_dim: 25
  n_classes: 10
  classifier_layers: [20, 10]
toy:
  enc_layer_sizes: [2, 200, 200, 200]
  dec_layer_sizes: [200, 200, 200, 2]
  latent_dim: 8
  centers_radius: 4.0
  n_clusters: 10
  cluster_size: 5000
case:
  #enc_layer_sizes: [ 1800, 600, 300, 100 ]
  #dec_layer_sizes: [ 100, 300, 600, 1800 ]
  #frames_per_pd_sample: 600

  enc_layer_sizes: [ 10, 600, 300, 300 ]
  dec_layer_sizes: [ 600, 300, 300, 10 ]
  frames_per_pd_sample: 10

  latent_dim: 300
  n_classes: 10
  classifier_layers: [ 20, 10 ] # unused right now.

  feature_list:
    #- V_0_0 # 0, X
    #- V_0_1 # 0, Y
    #- V_0_2 # 0, Z
    - pads_0
  enc_kernel_sizes: [] # for conv
  end_strides: []
  dec_kernel_sizes: []
  dec_strides: []

is_deterministic: False

real_data_pd_dir: "D:/pressure_pd"
case_dir: "real_case_20_min"
case_file: "pressure_data_0.pkl"

Data
For Mnist everything works fine.
When changing to my specific data, results are as above.
The data is a time series, of several features. To dumb this down even more, I am feeding just a single feature, sliced to equal-length chunks, and fed into the input layer as a vector.
The fact that the data is a time series could maybe help modeling in the future, but for now I want to just refer to it as chunks of data, which I believe I am doing.
code:
from torch.utils.data import Dataset
import matplotlib.pyplot as plt
import torch
from Testing.Research.config.ConfigProvider import ConfigProvider
import os
import pickle
import pandas as pd
from typing import Tuple
import numpy as np

class CaseDataset(Dataset):
    def __init__(self, path):
        super(CaseDataset, self).__init__()
        self._path = path

        self._config = ConfigProvider.get_config()
        self.frames_per_pd_sample = self._config.case.frames_per_pd_sample
        self._load_case_from_pkl()
        self.__len = len(self._full) // self.frames_per_pd_sample  # discard last non full batch

    def _load_case_from_pkl(self):
        assert os.path.isfile(self._path)
        with open(self._path, "rb") as f:
            p = pickle.load(f)

        self._full: pd.DataFrame = p["full"]
        self._subsampled: pd.DataFrame = p["subsampled"]
        self._misc: pd.DataFrame = p["misc"]

        feature_list = self._config.case.feature_list
        self._features_df = self._full[feature_list].copy()

        # normalize from -1 to 1
        features_to_normalize = self._features_df.columns
        self._features_df[features_to_normalize] = \
            self._features_df[features_to_normalize].apply(lambda x: (((x - x.min()) / (x.max() - x.min())) * 2) - 1)

        pass

    def __len__(self):
        # number of samples in the dataset
        return self.__len

    def __getitem__(self, index: int) -> Tuple[np.array, np.array]:
        data_item = self._features_df.iloc[index * self.frames_per_pd_sample: (index + 1) * self.frames_per_pd_sample, :].values
        label = 0.0
        # plt.figure()
        # plt.plot(data_item[:, 0], label="feature 0")
        # plt.legend(loc="upper left")
        # plt.show()
        return data_item, label

The amount of time-steps per batch does not seem to affect convergence.
Train test val split
is done like so:
import os
from pytorch_lightning import LightningDataModule
import torchvision.datasets as datasets
from torchvision.transforms import transforms
import torch
from torch.utils.data import DataLoader
from torch.utils.data import Subset
from Testing.Research.config.paths import mnist_data_download_folder
from Testing.Research.datasets.real_cases.CaseDataset import CaseDataset
from typing import Optional

class CaseDataModule(LightningDataModule):
    def __init__(self, config, path):
        super().__init__()
        self._config = config
        self._path = path

        self._train_dataset: Optional[Subset] = None
        self._val_dataset: Optional[Subset] = None
        self._test_dataset: Optional[Subset] = None

    def prepare_data(self):
        pass

    def setup(self, stage):
        # transform
        transform = transforms.Compose([transforms.ToTensor()])
        full_dataset = CaseDataset(self._path)

        train_size = int(self._config.train_size * len(full_dataset))
        val_size = int(self._config.val_size * len(full_dataset))
        test_size = len(full_dataset) - train_size - val_size
        train, val, test = torch.utils.data.random_split(full_dataset, [train_size, val_size, test_size])

        # assign to use in dataloaders
        self._full_dataset = full_dataset
        self._train_dataset = train
        self._val_dataset = val
        self._test_dataset = test

    def train_dataloader(self):
        return DataLoader(self._train_dataset, batch_size=self._config.batch_size, num_workers=self._config.num_workers)

    def val_dataloader(self):
        return DataLoader(self._val_dataset, batch_size=self._config.batch_size, num_workers=self._config.num_workers)

    def test_dataloader(self):
        return DataLoader(self._test_dataset, batch_size=self._config.batch_size, num_workers=self._config.num_workers)

Questions

I believe having the validation loss consistently lower than the train loss shows something is very wrong here, but I can't put my finger on what, or come up with how to verify this.
How can I just make the model auto-encode the data correctly? Basicaly, I would want it to learn the identity function, and for the loss to reflect that.
The loss does not seem to reflect the reconstruction. I think this is probably the most fundamental issue

My thoughts

Try a convolutional net instead of FC? maybe it would be able to better learn features?
Out of ideas :(

Will provide any lacking information.


title: GCP AI Platform: Error when creating a custom predictor model version ( trained model Pytorch model + torchvision.transform)

body:
Am currently trying to deploy a custom model to AI platform by following https://cloud.google.com/ai-platform/prediction/docs/deploying-models#gcloud_1. which is based on a combination of the pre-trained model from 'Pytorch' and 'torchvision.transform'. Currently, I keep getting below error which happens to be related to 500MB constraint on custom prediction.
ERROR: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error: Model requires more memory than allowed. Please try to decrease the model size and re-deploy. If you continue to experience errors, please contact support.
Setup.py
from setuptools import setup
from pathlib import Path

base = Path(__file__).parent
REQUIRED_PACKAGES = [line.strip() for line in open(base/"requirements.txt")]
print(f"\nPackages: {REQUIRED_PACKAGES}\n\n")

# [torch==1.3.0,torchvision==0.4.1, ImageHash==4.2.0
# Pillow==6.2.1,pyvis==0.1.8.2] installs 800mb worth of files

setup(description="Extract features of a image",
      author='Amrit',
      name='test',
      version='0.1',
      install_requires=REQUIRED_PACKAGES,
      project_urls={
                    'Documentation':'https://cloud.google.com/ai-platform/prediction/docs/custom-prediction-routines#tensorflow',
                    'Deploy':'https://cloud.google.com/ai-platform/prediction/docs/deploying-models#gcloud_1',
                    'Ai_platform troubleshooting':'https://cloud.google.com/ai-platform/training/docs/troubleshooting',
                    'Say Thanks!': 'https://medium.com/searce/deploy-your-own-custom-model-on-gcps-ai-platform-
 7e42a5721b43',
                    'google Torch wheels':"http://storage.googleapis.com/cloud-ai-pytorch/readme.txt",
                    'Torch & torchvision wheels':"https://download.pytorch.org/whl/torch_stable.html "
                    },
    python_requires='~=3.7',
    scripts=['predictor.py', 'preproc.py'])

Steps taken:
Tried adding ‘torch’ and torchvision directly to ‘REQUIRED_PACKAGES’ list in setup.py file in order to provide PyTorch + torchvision as a dependency to be installed while deployment.  I am guessing, Internally Ai platform downloads PyPI package for PyTorch which is +500 MB, this results in the failure of our model deployment. If I just deploy the model with 'torch' only and it seems to be working (of course throws error for not able to find library 'torchvision')
File size

pytorch (torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl about 111MB)
torchvision (torchvision-0.4.1+cpu-cp37-cp37m-linux_x86_64.whl about 46MB) from https://download.pytorch.org/whl/torch_stable.html and stored it on GKS.
The zipped predictor model  file (.tar.gz format) which is the output of  setup.py (5kb )
A trained PyTorch model (size 44MB)

In total, the model dependencies should be less than 250MB but still, keep getting this error. Have also tried to use the torch and torchvision provided from Google mirrored packages  http://storage.googleapis.com/cloud-ai-pytorch/readme.txt, but same memory issue persists. AI platform is quite new for us and would like some input from professional’s.
MORE INFO:
GCP CLI Input:
My environment variable:
BUCKET_NAME= “something”
MODEL_DIR="gs://$BUCKET_NAME/"
VERSION_NAME='v6'
MODEL_NAME="something_model"
STAGING_BUCKET=$MODEL_DIR"staging_area/"
# TORCH_PACKAGE=$MODEL_DIR"package/torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl"
# TORCHVISION_PACKAGE=$MODEL_DIR"package/torchvision-0.4.1+cpu-cp37-cp37m-linux_x86_64.whl"
TORCH_PACKAGE="gs://cloud-ai-pytorch/torch-1.3.1+cpu-cp37-cp37m-linux_x86_64.whl"
TORCHVISION_PACKAGE="gs://cloud-ai-pytorch/torchvision-0.4.1+cpu-cp37-cp37m-linux_x86_64.whl"
CUSTOM_CODE_PATH=$STAGING_BUCKET"imt_ai_predict-0.1.tar.gz"
PREDICTOR_CLASS="predictor.MyPredictor"
REGION='europe-west1'
MACHINE_TYPE='mls1-c4-m2'

gcloud beta ai-platform versions create $VERSION_NAME   \
--model=$MODEL_NAME   \
--origin=$MODEL_DIR  \
 --runtime-version=2.3  \
 --python-version=3.7   \
--machine-type=$MACHINE_TYPE  \
 --package-uris=$CUSTOM_CODE_PATH,$TORCH_PACKAGE,$TORCHVISION_PACKAGE   \
--prediction-class=$PREDICTOR_CLASS \

GCP CLI Output:
 **[1] global**
 [2] asia-east1
 [3] asia-northeast1
 [4] asia-southeast1
 [5] australia-southeast1
 [6] europe-west1
 [7] europe-west2
 [8] europe-west3
 [9] europe-west4
 [10] northamerica-northeast1
 [11] us-central1
 [12] us-east1
 [13] us-east4
 [14] us-west1
 [15] cancel
Please enter your numeric choice:  1

To make this the default region, run `gcloud config set ai_platform/region global`.

Using endpoint [https://ml.googleapis.com/]
Creating version (this might take a few minutes)......failed.
ERROR: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error: **Model requires more memory than allowed. Please try to decrease the model size and re-deploy. If you continue to experience errors, please contact support.**

My finding:
Have found articles of people struggling in same ways for PyTorch package and made it work by installing torch wheels on the GCS (https://medium.com/searce/deploy-your-own-custom-model-on-gcps-ai-platform-
7e42a5721b43).
Have tried the same approach with torch and torchvision but no luck till now and waiting response from "cloudml-feedback@google.com cloudml-feedback@google.com". Any help on getting custom  torch_torchvision based custom predictor working on AI platform that will be great.


title: RuntimeError: shape '[-1, 1031]' is invalid for input of size 900

body:
I am trying to build a CNN but I get this error:
shape '[-1, 1031]' is invalid for input of size 900

My code is below:
class CPillModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(CPillModel, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu1 = nn.ReLU()

        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.relu2 = nn.ReLU()

        self.fc3 = nn.Linear(hidden_dim, hidden_dim)
        self.relu3 = nn.ReLU()

        self.fc4 = nn.Linear(hidden_dim, output_dim)
        self.act4 = Sigmoid()

    def forward(self, x):

       ....
       ....
       return x

# instantiate ANN
input_dim = 1031
hidden_dim = 150
output_dim = 10

# Create ANN
model = CPillModel(input_dim, hidden_dim, output_dim)

# Cross Entropy Loss
error = nn.CrossEntropyLoss()

# SGD Optimizer
learning_rate = 0.02
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

# ANN model training
count = 0
loss_list = []
iteration_list = []
accuracy_list = []
for epoch in range(num_epochs):
# enumerate mini batches
    for i, (data_a, data_b) in enumerate(train_loader):
        train = Variable(data_a.view(-1, 1031))
        labels = Variable(data_b)
        # clear the gradients
        optimizer.zero_grad()
        optimizer.zero_grad()
        # compute the model output
        outputs = model(data_a)
        # calculate loss
        loss = error(outputs, data_b)
        # credit assignment
        loss.backward()
        # update model weights
        optimizer.step()
        count += 1

I think the issue I have is with my train variable line: train = Variable(data_a.view(-1, 1031))
I have 1031 rows of features training data but they are not being converted.
Do I split these 1031 data points into a format such as: a x b to work?
Would data reshape work?
The print out of data_a is: data_a is torch.Size([100, 9]


title: Detectron2 Speed up inference instance segmentation

body:
I have working instance segmentation, I'm using "mask_rcnn_R_101_FPN_3x" model. When I inference image it takes about 3 second / image on GPU. How can I speed up it faster ?
I code in Google Colab
This is my setup config:
cfg = get_cfg()

cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))

cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1

cfg.OUTPUT_DIR = "/content/drive/MyDrive/TEAM/save/"

cfg.DATASETS.TRAIN = (train_name,)
cfg.DATASETS.TEST = (test_name, )
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")

This is inference:
torch.backends.cudnn.benchmark = True
start = time.time()

predictor = DefaultPredictor(cfg)

im = cv2.imread("/content/drive/MyDrive/TEAM/mcocr_val_145114ixmyt.jpg")

outputs = predictor(im)

print(f"Inference time per image is : {(time.time() - start)} s")

Return time:
Inference time per image is : 2.7835421562194824 s
Image I inference size 1024 x 1024 pixel. I have change different size but it still inference 3 second / image. Am I missing anything about Detectron2 ?
More information GPU
enter image description here


title: Can't Backpropagate in Pytorch module with .backwards(); weights are not updated

body:
I'm willing to create a network estimating the next sample of the signal. I've statred with a simple sin signal. But when I run the code I got noise as an output. Then checked the layer weights and figured aout that they are not updating. I can't find the mistake here.

class Model(nn.Module):
def __init__(self,in_dim,hidden_dim,num_classes):
    super(Model, self).__init__()
    self.layer1 = nn.Linear(in_dim,hidden_dim)
    self.layer2 = nn.Linear(hidden_dim,hidden_dim)
    self.layer3 = nn.Linear(hidden_dim,num_classes)
    self.relu = nn.ReLU()

def forward(self,x):
    a = self.relu(self.layer1(x))
    a = self.relu(self.layer2(a))
    return self.relu(self.layer3(a))

train:

def train(epoch,L,depth):
    criteria = nn.MSELoss()
    learning_rate = 1e-3
    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    t = np.linspace(0,2,L+2)
    fs = L+2

    trn_loss = list()

   for f in range(0,epoch):
       phase = f/np.pi
       x = np.sin(2*np.pi*t*fs+phase)
       x = torch.from_numpy(x).detach().float()

       optimizer.zero_grad()

       x_hat = model(x[:-2])

       currentCost = criteria(x_hat,x[-2])
       trn_loss.append(currentCost.item())
       print(model.layer1.weight.data.clone())
       currentCost.backward()
       optimizer.step()
       print(model.layer1.weight.data.clone())
       sys.exit('DEBUG')

output:

tensor([[-0.1715, -0.1696,  0.0424,  ...,  0.0154,  0.1450, -0.0544],
    [ 0.0368,  0.1427, -0.1419,  ...,  0.0966,  0.0298, -0.0659],
    [-0.1641, -0.1551,  0.0570,  ..., -0.0227, -0.1426, -0.0648],
    ...,
    [-0.0684, -0.1707, -0.0711,  ...,  0.0788,  0.1386,  0.1546],
    [ 0.1401, -0.0922, -0.0104,  ..., -0.0490,  0.0404,  0.1038],
    [-0.0604, -0.0517,  0.0715,  ..., -0.1200,  0.0014,  0.0215]])
tensor([[-0.1715, -0.1696,  0.0424,  ...,  0.0154,  0.1450, -0.0544],
    [ 0.0368,  0.1427, -0.1419,  ...,  0.0966,  0.0298, -0.0659],
    [-0.1641, -0.1551,  0.0570,  ..., -0.0227, -0.1426, -0.0648],
    ...,
    [-0.0684, -0.1707, -0.0711,  ...,  0.0788,  0.1386,  0.1546],
    [ 0.1401, -0.0922, -0.0104,  ..., -0.0490,  0.0404,  0.1038],
    [-0.0604, -0.0517,  0.0715,  ..., -0.1200,  0.0014,  0.0215]])


title: How can I compute the mean of values selected from a vector A from an indexing vector B?

body:
I have a vector of values, for example:

import torch
v = torch.rand(6)

tensor([0.0811, 0.9658, 0.1901, 0.0872, 0.8895, 0.9647])

and an index to select values from v:

index = torch.tensor([0, 1, 0, 2, 0, 2])

I want to produce a vector mean which would compute the mean values of v grouped by indexes from index.

In this example:

mean = torch.tensor([(0.0811 + 0.1901 + 0.8895) / 3, 0.9658, (0.0872 + 0.9647) / 2, 0, 0, 0])

tensor([0.3869, 0.9658, 0.5260, 0.0000, 0.0000, 0.0000])


title: Neural net not training at all

body:
my neural net accuracy and test loss are stuck while building a model on HAR. The model was built using only time series variables(expert features were not used)

I have tried many learning rates, and optimizers, even then from epoch 1 to 200 they are stuck at 18% accuracy

The architecture used was from here
And I couldn't figure out if architecture is really bad or my code has a bug in it.

class Net(nn.Module):

  def __init__(self):
    super().__init__()
    self.model = nn.ModuleDict({
            'lstm': nn.LSTM(
                input_size=64,
                hidden_size=64,batch_first=True
            ),
            'linear': nn.Linear(
                in_features=128,
                out_features=6),
            'dropout': nn.Dropout(0.7),
            'lstm2': nn.LSTM(
                input_size=64,
                hidden_size=128,batch_first=True),
            'linear2': nn.Linear(
                in_features=9,
                out_features=64)})

  def forward(self,x):
    x = x.permute(1,0,2)
    x = x.contiguous().view(-1,9)
    x = self.model['linear2'](x)

    x = x.contiguous().view(-1,128,64)
    x, _ = self.model['lstm'](x)
    x = self.model['dropout'](x)
    x, _ = self.model['lstm2'](x)
    x = self.model['dropout'](x)
    x = self.model['linear'](x[:,-1,:])
    return x

net=Net().to(device)

def train( model, device, train_loader, optimizer, epoch,):
    model.train()
    train_loss=0
    for batch_idx, (data, target) in enumerate(train_loader):
        data, target = data.to(device), target.view(-1).to(device)
        optimizer.zero_grad()

        output = model(data.float())

        train_loss+=F.cross_entropy(output, target).item()
        loss = F.cross_entropy(output, target)
        loss.backward()
        optimizer.step()

    train_loss /= np.ceil(len(train_loader.dataset)/bs)
    print('Train Epoch: {} \tLoss per Batch: {:.6f}'.format(epoch,train_loss))

def test( model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for batch_idx,(data, target) in enumerate(test_loader):
            data, target = data.to(device), target.to(device)
            output = model(data.float())
            test_loss += F.cross_entropy(output, target).item() # sum up batch loss
            pred = output.argmax(dim=1, keepdim=True) # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()
    test_loss /= np.ceil(len(test_loader.dataset)/bs)

    print('\nTest set: Loss per Batch: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

import torch.optim as optim
optimizer = optim.Adam(net.parameters(), lr=.01)

By using Adam with lr=.01, I got the following results

loss per batch and accuracy of Test set have remained constant during the whole training period of 70 epochs

Train Epoch: 0 Loss per Batch: 1.788564

Test set: Loss per Batch: 1.7989, Accuracy: 533/2947 (18%)

Train Epoch: 1  Loss per Batch: 1.786124

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 2  Loss per Batch: 1.785584

Test set: Loss per Batch: 1.7947, Accuracy: 537/2947 (18%)

Train Epoch: 3  Loss per Batch: 1.785016

Test set: Loss per Batch: 1.8117, Accuracy: 537/2947 (18%)

Train Epoch: 4  Loss per Batch: 1.786215

Test set: Loss per Batch: 1.7921, Accuracy: 537/2947 (18%)

Train Epoch: 5  Loss per Batch: 1.785138

Test set: Loss per Batch: 1.7939, Accuracy: 537/2947 (18%)

Train Epoch: 6  Loss per Batch: 1.784805

Test set: Loss per Batch: 1.7932, Accuracy: 537/2947 (18%)

Train Epoch: 7  Loss per Batch: 1.784633

Test set: Loss per Batch: 1.7932, Accuracy: 537/2947 (18%)

Train Epoch: 8  Loss per Batch: 1.784855

Test set: Loss per Batch: 1.7938, Accuracy: 537/2947 (18%)

Train Epoch: 9  Loss per Batch: 1.785028

Test set: Loss per Batch: 1.7936, Accuracy: 537/2947 (18%)

Train Epoch: 10     Loss per Batch: 1.784683

Test set: Loss per Batch: 1.7930, Accuracy: 537/2947 (18%)

Train Epoch: 11     Loss per Batch: 1.785211

Test set: Loss per Batch: 1.7930, Accuracy: 527/2947 (18%)

Train Epoch: 12     Loss per Batch: 1.784648

Test set: Loss per Batch: 1.7912, Accuracy: 514/2947 (17%)

Train Epoch: 13     Loss per Batch: 1.784625

Test set: Loss per Batch: 1.7909, Accuracy: 537/2947 (18%)

Train Epoch: 14     Loss per Batch: 1.785082

Test set: Loss per Batch: 1.7921, Accuracy: 537/2947 (18%)

Train Epoch: 15     Loss per Batch: 1.785162

Test set: Loss per Batch: 1.7931, Accuracy: 509/2947 (17%)

Train Epoch: 16     Loss per Batch: 1.784732

Test set: Loss per Batch: 1.7932, Accuracy: 532/2947 (18%)

Train Epoch: 17     Loss per Batch: 1.784819

Test set: Loss per Batch: 1.7949, Accuracy: 537/2947 (18%)

Train Epoch: 18     Loss per Batch: 1.785135

Test set: Loss per Batch: 1.7927, Accuracy: 537/2947 (18%)

Train Epoch: 19     Loss per Batch: 1.784840

Test set: Loss per Batch: 1.7933, Accuracy: 532/2947 (18%)

Train Epoch: 20     Loss per Batch: 1.784750

Test set: Loss per Batch: 1.7939, Accuracy: 532/2947 (18%)

Train Epoch: 21     Loss per Batch: 1.784810

Test set: Loss per Batch: 1.7940, Accuracy: 517/2947 (18%)

Train Epoch: 22     Loss per Batch: 1.784939

Test set: Loss per Batch: 1.7872, Accuracy: 537/2947 (18%)

Train Epoch: 23     Loss per Batch: 1.785099

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 24     Loss per Batch: 1.784571

Test set: Loss per Batch: 1.7918, Accuracy: 537/2947 (18%)

Train Epoch: 25     Loss per Batch: 1.784607

Test set: Loss per Batch: 1.7922, Accuracy: 532/2947 (18%)

Train Epoch: 26     Loss per Batch: 1.784505

Test set: Loss per Batch: 1.7902, Accuracy: 514/2947 (17%)

Train Epoch: 27     Loss per Batch: 1.785079

Test set: Loss per Batch: 1.7932, Accuracy: 532/2947 (18%)

Train Epoch: 28     Loss per Batch: 1.784903

Test set: Loss per Batch: 1.7933, Accuracy: 532/2947 (18%)

Train Epoch: 29     Loss per Batch: 1.784797

Test set: Loss per Batch: 1.7938, Accuracy: 537/2947 (18%)

Train Epoch: 30     Loss per Batch: 1.784668

Test set: Loss per Batch: 1.7940, Accuracy: 537/2947 (18%)

Train Epoch: 31     Loss per Batch: 1.784762

Test set: Loss per Batch: 1.7934, Accuracy: 537/2947 (18%)

Train Epoch: 32     Loss per Batch: 1.784556

Test set: Loss per Batch: 1.7947, Accuracy: 532/2947 (18%)

Train Epoch: 33     Loss per Batch: 1.784983

Test set: Loss per Batch: 1.7935, Accuracy: 532/2947 (18%)

Train Epoch: 34     Loss per Batch: 1.785202

Test set: Loss per Batch: 1.7945, Accuracy: 532/2947 (18%)

Train Epoch: 35     Loss per Batch: 1.785080

Test set: Loss per Batch: 1.7921, Accuracy: 532/2947 (18%)

Train Epoch: 36     Loss per Batch: 1.784834

Test set: Loss per Batch: 1.7925, Accuracy: 532/2947 (18%)

Train Epoch: 37     Loss per Batch: 1.784711

Test set: Loss per Batch: 1.7938, Accuracy: 532/2947 (18%)

Train Epoch: 38     Loss per Batch: 1.784832

Test set: Loss per Batch: 1.7922, Accuracy: 537/2947 (18%)

Train Epoch: 39     Loss per Batch: 1.784580

Test set: Loss per Batch: 1.7931, Accuracy: 532/2947 (18%)

Train Epoch: 40     Loss per Batch: 1.784810

Test set: Loss per Batch: 1.7928, Accuracy: 537/2947 (18%)

Train Epoch: 41     Loss per Batch: 1.784549

Test set: Loss per Batch: 1.7934, Accuracy: 532/2947 (18%)

Train Epoch: 42     Loss per Batch: 1.784458

Test set: Loss per Batch: 1.7920, Accuracy: 537/2947 (18%)

Train Epoch: 43     Loss per Batch: 1.785136

Test set: Loss per Batch: 1.7974, Accuracy: 537/2947 (18%)

Train Epoch: 44     Loss per Batch: 1.784797

Test set: Loss per Batch: 1.7762, Accuracy: 548/2947 (19%)

Train Epoch: 45     Loss per Batch: 1.785898

Test set: Loss per Batch: 1.7944, Accuracy: 532/2947 (18%)

Train Epoch: 46     Loss per Batch: 1.784899

Test set: Loss per Batch: 1.7951, Accuracy: 537/2947 (18%)

Train Epoch: 47     Loss per Batch: 1.784710

Test set: Loss per Batch: 1.7934, Accuracy: 537/2947 (18%)

Train Epoch: 48     Loss per Batch: 1.785024

Test set: Loss per Batch: 1.7946, Accuracy: 532/2947 (18%)

Train Epoch: 49     Loss per Batch: 1.784930

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 50     Loss per Batch: 1.784564

Test set: Loss per Batch: 1.7913, Accuracy: 532/2947 (18%)

Train Epoch: 51     Loss per Batch: 1.784940

Test set: Loss per Batch: 1.7936, Accuracy: 532/2947 (18%)

Train Epoch: 52     Loss per Batch: 1.785000

Test set: Loss per Batch: 1.7921, Accuracy: 537/2947 (18%)

Train Epoch: 53     Loss per Batch: 1.784505

Test set: Loss per Batch: 1.7947, Accuracy: 532/2947 (18%)

Train Epoch: 54     Loss per Batch: 1.784886

Test set: Loss per Batch: 1.7914, Accuracy: 532/2947 (18%)

Train Epoch: 55     Loss per Batch: 1.784894

Test set: Loss per Batch: 1.7920, Accuracy: 537/2947 (18%)

Train Epoch: 56     Loss per Batch: 1.784917

Test set: Loss per Batch: 1.7920, Accuracy: 537/2947 (18%)

Train Epoch: 57     Loss per Batch: 1.784813

Test set: Loss per Batch: 1.7941, Accuracy: 537/2947 (18%)

Train Epoch: 58     Loss per Batch: 1.784905

Test set: Loss per Batch: 1.7933, Accuracy: 537/2947 (18%)

Train Epoch: 59     Loss per Batch: 1.784620

Test set: Loss per Batch: 1.7914, Accuracy: 537/2947 (18%)

Train Epoch: 60     Loss per Batch: 1.784722

Test set: Loss per Batch: 1.7934, Accuracy: 532/2947 (18%)

Train Epoch: 61     Loss per Batch: 1.784789

Test set: Loss per Batch: 1.7929, Accuracy: 537/2947 (18%)

Train Epoch: 62     Loss per Batch: 1.784626

Test set: Loss per Batch: 1.7933, Accuracy: 537/2947 (18%)

Train Epoch: 63     Loss per Batch: 1.784967

Test set: Loss per Batch: 1.7932, Accuracy: 537/2947 (18%)

Train Epoch: 64     Loss per Batch: 1.784695

Test set: Loss per Batch: 1.7918, Accuracy: 537/2947 (18%)

Train Epoch: 65     Loss per Batch: 1.784872

Test set: Loss per Batch: 1.7939, Accuracy: 537/2947 (18%)

Train Epoch: 66     Loss per Batch: 1.784757

Test set: Loss per Batch: 1.7925, Accuracy: 537/2947 (18%)

Train Epoch: 67     Loss per Batch: 1.784619

Test set: Loss per Batch: 1.7912, Accuracy: 537/2947 (18%)

Train Epoch: 68     Loss per Batch: 1.784919

Test set: Loss per Batch: 1.7927, Accuracy: 537/2947 (18%)

Train Epoch: 69     Loss per Batch: 1.784839

Test set: Loss per Batch: 1.7927, Accuracy: 537/2947 (18%)

Train Epoch: 70     Loss per Batch: 1.784806

Test set: Loss per Batch: 1.7937, Accuracy: 537/2947 (18%)

Train Epoch: 71     Loss per Batch: 1.784729

Test set: Loss per Batch: 1.7910, Accuracy: 537/2947 (18%)

Train Epoch: 72     Loss per Batch: 1.784705

Test set: Loss per Batch: 1.7937, Accuracy: 537/2947 (18%)

Train Epoch: 73     Loss per Batch: 1.785041

Test set: Loss per Batch: 1.7936, Accuracy: 537/2947 (18%)

Train Epoch: 74     Loss per Batch: 1.784568

Test set: Loss per Batch: 1.7919, Accuracy: 537/2947 (18%)

Train Epoch: 75     Loss per Batch: 1.784705

Test set: Loss per Batch: 1.7946, Accuracy: 537/2947 (18%)

Train Epoch: 76     Loss per Batch: 1.784397

Test set: Loss per Batch: 1.7896, Accuracy: 537/2947 (18%)

Train Epoch: 77     Loss per Batch: 1.785445

Test set: Loss per Batch: 1.7932, Accuracy: 532/2947 (18%)

Train Epoch: 78     Loss per Batch: 1.785322

Test set: Loss per Batch: 1.7935, Accuracy: 532/2947 (18%)

Train Epoch: 79     Loss per Batch: 1.784972

Test set: Loss per Batch: 1.7929, Accuracy: 537/2947 (18%)

Train Epoch: 80     Loss per Batch: 1.785531

Test set: Loss per Batch: 1.7935, Accuracy: 532/2947 (18%)

Train Epoch: 81     Loss per Batch: 1.784897

Test set: Loss per Batch: 1.7930, Accuracy: 537/2947 (18%)

Train Epoch: 82     Loss per Batch: 1.784814

Test set: Loss per Batch: 1.7935, Accuracy: 536/2947 (18%)

Train Epoch: 83     Loss per Batch: 1.784614

Test set: Loss per Batch: 1.7960, Accuracy: 532/2947 (18%)

Train Epoch: 84     Loss per Batch: 1.785118

Test set: Loss per Batch: 1.7930, Accuracy: 537/2947 (18%)

Train Epoch: 85     Loss per Batch: 1.785127

Test set: Loss per Batch: 1.7919, Accuracy: 532/2947 (18%)

Train Epoch: 86     Loss per Batch: 1.784900

Test set: Loss per Batch: 1.7934, Accuracy: 537/2947 (18%)

Train Epoch: 87     Loss per Batch: 1.784775

Test set: Loss per Batch: 1.7954, Accuracy: 537/2947 (18%)

Train Epoch: 88     Loss per Batch: 1.784831

Test set: Loss per Batch: 1.7931, Accuracy: 537/2947 (18%)

Train Epoch: 89     Loss per Batch: 1.784919

Test set: Loss per Batch: 1.7937, Accuracy: 537/2947 (18%)

Train Epoch: 90     Loss per Batch: 1.784544

Test set: Loss per Batch: 1.7935, Accuracy: 532/2947 (18%)

Train Epoch: 91     Loss per Batch: 1.784984

Test set: Loss per Batch: 1.7938, Accuracy: 537/2947 (18%)

Train Epoch: 92     Loss per Batch: 1.784681

Test set: Loss per Batch: 1.7919, Accuracy: 537/2947 (18%)

Train Epoch: 93     Loss per Batch: 1.784792

Test set: Loss per Batch: 1.7928, Accuracy: 537/2947 (18%)

Train Epoch: 94     Loss per Batch: 1.785022

Test set: Loss per Batch: 1.7918, Accuracy: 532/2947 (18%)

Train Epoch: 95     Loss per Batch: 1.785049

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 96     Loss per Batch: 1.784717

Test set: Loss per Batch: 1.7946, Accuracy: 537/2947 (18%)

Train Epoch: 97     Loss per Batch: 1.784922

Test set: Loss per Batch: 1.7912, Accuracy: 537/2947 (18%)

Train Epoch: 98     Loss per Batch: 1.784669

Test set: Loss per Batch: 1.7917, Accuracy: 532/2947 (18%)

Train Epoch: 99     Loss per Batch: 1.784664

Test set: Loss per Batch: 1.7927, Accuracy: 537/2947 (18%)

Train Epoch: 100    Loss per Batch: 1.784869

Test set: Loss per Batch: 1.7937, Accuracy: 537/2947 (18%)

Train Epoch: 101    Loss per Batch: 1.784915

Test set: Loss per Batch: 1.7954, Accuracy: 537/2947 (18%)

Train Epoch: 102    Loss per Batch: 1.784592

Test set: Loss per Batch: 1.7913, Accuracy: 532/2947 (18%)

Train Epoch: 103    Loss per Batch: 1.784817

Test set: Loss per Batch: 1.7920, Accuracy: 537/2947 (18%)

Train Epoch: 104    Loss per Batch: 1.784869

Test set: Loss per Batch: 1.7941, Accuracy: 532/2947 (18%)

Train Epoch: 105    Loss per Batch: 1.787049

Test set: Loss per Batch: 1.7915, Accuracy: 537/2947 (18%)

Train Epoch: 106    Loss per Batch: 1.785012

Test set: Loss per Batch: 1.7919, Accuracy: 537/2947 (18%)

Train Epoch: 107    Loss per Batch: 1.784757

Test set: Loss per Batch: 1.7927, Accuracy: 532/2947 (18%)

Train Epoch: 108    Loss per Batch: 1.784881

Test set: Loss per Batch: 1.7939, Accuracy: 537/2947 (18%)

Train Epoch: 109    Loss per Batch: 1.785723

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 110    Loss per Batch: 1.784988

Test set: Loss per Batch: 1.7914, Accuracy: 532/2947 (18%)

Train Epoch: 111    Loss per Batch: 1.785197

Test set: Loss per Batch: 1.7928, Accuracy: 532/2947 (18%)

Train Epoch: 112    Loss per Batch: 1.784856

Test set: Loss per Batch: 1.7937, Accuracy: 537/2947 (18%)

Train Epoch: 113    Loss per Batch: 1.785120

Test set: Loss per Batch: 1.7932, Accuracy: 537/2947 (18%)

Train Epoch: 114    Loss per Batch: 1.784842

Test set: Loss per Batch: 1.7906, Accuracy: 537/2947 (18%)

Train Epoch: 115    Loss per Batch: 1.785119

Test set: Loss per Batch: 1.7918, Accuracy: 537/2947 (18%)

Train Epoch: 116    Loss per Batch: 1.784935

Test set: Loss per Batch: 1.7917, Accuracy: 537/2947 (18%)

Train Epoch: 117    Loss per Batch: 1.784734

Test set: Loss per Batch: 1.7931, Accuracy: 537/2947 (18%)

Train Epoch: 118    Loss per Batch: 1.784930

Test set: Loss per Batch: 1.7916, Accuracy: 532/2947 (18%)

Train Epoch: 119    Loss per Batch: 1.785058

Test set: Loss per Batch: 1.7931, Accuracy: 537/2947 (18%)

Train Epoch: 120    Loss per Batch: 1.784399

Test set: Loss per Batch: 1.7955, Accuracy: 532/2947 (18%)

Train Epoch: 121    Loss per Batch: 1.784580

Test set: Loss per Batch: 1.7921, Accuracy: 537/2947 (18%)

Train Epoch: 122    Loss per Batch: 1.784730

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 123    Loss per Batch: 1.784921

Test set: Loss per Batch: 1.7946, Accuracy: 537/2947 (18%)

Train Epoch: 124    Loss per Batch: 1.785217

Test set: Loss per Batch: 1.7918, Accuracy: 537/2947 (18%)

Train Epoch: 125    Loss per Batch: 1.784670

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 126    Loss per Batch: 1.784825

Test set: Loss per Batch: 1.7935, Accuracy: 537/2947 (18%)

Train Epoch: 127    Loss per Batch: 1.785658

Test set: Loss per Batch: 1.7926, Accuracy: 532/2947 (18%)

Train Epoch: 128    Loss per Batch: 1.785025

Test set: Loss per Batch: 1.7924, Accuracy: 537/2947 (18%)

Train Epoch: 129    Loss per Batch: 1.784739

Test set: Loss per Batch: 1.7949, Accuracy: 532/2947 (18%)

Train Epoch: 130    Loss per Batch: 1.785050

Test set: Loss per Batch: 1.7932, Accuracy: 532/2947 (18%)

Train Epoch: 131    Loss per Batch: 1.784816

Test set: Loss per Batch: 1.7909, Accuracy: 537/2947 (18%)

Train Epoch: 132    Loss per Batch: 1.784678

Test set: Loss per Batch: 1.7943, Accuracy: 537/2947 (18%)

Train Epoch: 133    Loss per Batch: 1.784638

Test set: Loss per Batch: 1.7934, Accuracy: 532/2947 (18%)

Train Epoch: 134    Loss per Batch: 1.784962

Test set: Loss per Batch: 1.7941, Accuracy: 537/2947 (18%)

Train Epoch: 135    Loss per Batch: 1.784798

Test set: Loss per Batch: 1.7941, Accuracy: 537/2947 (18%)

Train Epoch: 136    Loss per Batch: 1.784776

Test set: Loss per Batch: 1.7932, Accuracy: 532/2947 (18%)

Train Epoch: 137    Loss per Batch: 1.785209

Test set: Loss per Batch: 1.7908, Accuracy: 537/2947 (18%)

Train Epoch: 138    Loss per Batch: 1.784884

Test set: Loss per Batch: 1.7926, Accuracy: 532/2947 (18%)

Train Epoch: 139    Loss per Batch: 1.784842

Test set: Loss per Batch: 1.7931, Accuracy: 532/2947 (18%)

Train Epoch: 140    Loss per Batch: 1.784622

Test set: Loss per Batch: 1.7919, Accuracy: 537/2947 (18%)

Train Epoch: 141    Loss per Batch: 1.785023

Test set: Loss per Batch: 1.7925, Accuracy: 537/2947 (18%)

Train Epoch: 142    Loss per Batch: 1.784689

Test set: Loss per Batch: 1.7944, Accuracy: 537/2947 (18%)

Train Epoch: 143    Loss per Batch: 1.784810

Test set: Loss per Batch: 1.7931, Accuracy: 537/2947 (18%)

Train Epoch: 144    Loss per Batch: 1.784905

Test set: Loss per Batch: 1.7920, Accuracy: 532/2947 (18%)

Train Epoch: 145    Loss per Batch: 1.784574

Test set: Loss per Batch: 1.7946, Accuracy: 537/2947 (18%)

Train Epoch: 146    Loss per Batch: 1.784829

Test set: Loss per Batch: 1.7930, Accuracy: 537/2947 (18%)

Train Epoch: 147    Loss per Batch: 1.784832

Test set: Loss per Batch: 1.7933, Accuracy: 537/2947 (18%)

Train Epoch: 148    Loss per Batch: 1.784582

Test set: Loss per Batch: 1.7941, Accuracy: 537/2947 (18%)

Train Epoch: 149    Loss per Batch: 1.784732

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 150    Loss per Batch: 1.784652

Test set: Loss per Batch: 1.7947, Accuracy: 537/2947 (18%)

Train Epoch: 151    Loss per Batch: 1.784722

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 152    Loss per Batch: 1.784994

Test set: Loss per Batch: 1.7951, Accuracy: 532/2947 (18%)

Train Epoch: 153    Loss per Batch: 1.784773

Test set: Loss per Batch: 1.7938, Accuracy: 532/2947 (18%)

Train Epoch: 154    Loss per Batch: 1.784962

Test set: Loss per Batch: 1.7934, Accuracy: 537/2947 (18%)

Train Epoch: 155    Loss per Batch: 1.784522

Test set: Loss per Batch: 1.7934, Accuracy: 537/2947 (18%)

Train Epoch: 156    Loss per Batch: 1.784805

Test set: Loss per Batch: 1.7921, Accuracy: 537/2947 (18%)

Train Epoch: 157    Loss per Batch: 1.785455

Test set: Loss per Batch: 1.7944, Accuracy: 532/2947 (18%)

Train Epoch: 158    Loss per Batch: 1.784509

Test set: Loss per Batch: 1.7943, Accuracy: 537/2947 (18%)

Train Epoch: 159    Loss per Batch: 1.784982

Test set: Loss per Batch: 1.7926, Accuracy: 537/2947 (18%)

Train Epoch: 160    Loss per Batch: 1.784593

Test set: Loss per Batch: 1.7938, Accuracy: 537/2947 (18%)

Train Epoch: 161    Loss per Batch: 1.785101

Test set: Loss per Batch: 1.7930, Accuracy: 537/2947 (18%)

Train Epoch: 162    Loss per Batch: 1.785277

Test set: Loss per Batch: 1.7939, Accuracy: 537/2947 (18%)

Train Epoch: 163    Loss per Batch: 1.784970

Test set: Loss per Batch: 1.7935, Accuracy: 532/2947 (18%)

Train Epoch: 164    Loss per Batch: 1.784984

Test set: Loss per Batch: 1.7943, Accuracy: 537/2947 (18%)

Train Epoch: 165    Loss per Batch: 1.784928

Test set: Loss per Batch: 1.7931, Accuracy: 537/2947 (18%)

Train Epoch: 166    Loss per Batch: 1.784774

Test set: Loss per Batch: 1.7923, Accuracy: 537/2947 (18%)

Train Epoch: 167    Loss per Batch: 1.785107

Test set: Loss per Batch: 1.7935, Accuracy: 537/2947 (18%)

Train Epoch: 168    Loss per Batch: 1.784620

Test set: Loss per Batch: 1.7941, Accuracy: 537/2947 (18%)

Train Epoch: 169    Loss per Batch: 1.784884

Test set: Loss per Batch: 1.7931, Accuracy: 537/2947 (18%)

Train Epoch: 170    Loss per Batch: 1.784452

Test set: Loss per Batch: 1.7923, Accuracy: 537/2947 (18%)

Train Epoch: 171    Loss per Batch: 1.784965

Test set: Loss per Batch: 1.7935, Accuracy: 537/2947 (18%)

Train Epoch: 172    Loss per Batch: 1.784781

Test set: Loss per Batch: 1.7924, Accuracy: 537/2947 (18%)

Train Epoch: 173    Loss per Batch: 1.784724

Test set: Loss per Batch: 1.7941, Accuracy: 537/2947 (18%)

Train Epoch: 174    Loss per Batch: 1.784639

Test set: Loss per Batch: 1.7912, Accuracy: 532/2947 (18%)

Train Epoch: 175    Loss per Batch: 1.784592

Test set: Loss per Batch: 1.7936, Accuracy: 537/2947 (18%)

Train Epoch: 176    Loss per Batch: 1.784642

Test set: Loss per Batch: 1.7922, Accuracy: 532/2947 (18%)

Train Epoch: 177    Loss per Batch: 1.785098

Test set: Loss per Batch: 1.7931, Accuracy: 537/2947 (18%)

Train Epoch: 178    Loss per Batch: 1.784758

Test set: Loss per Batch: 1.7938, Accuracy: 537/2947 (18%)

Train Epoch: 179    Loss per Batch: 1.784484

Test set: Loss per Batch: 1.7910, Accuracy: 532/2947 (18%)

Train Epoch: 180    Loss per Batch: 1.784845

Test set: Loss per Batch: 1.7949, Accuracy: 537/2947 (18%)

Train Epoch: 181    Loss per Batch: 1.784970

Test set: Loss per Batch: 1.7926, Accuracy: 532/2947 (18%)

Train Epoch: 182    Loss per Batch: 1.785243

Test set: Loss per Batch: 1.7917, Accuracy: 532/2947 (18%)

Train Epoch: 183    Loss per Batch: 1.784563

Test set: Loss per Batch: 1.7932, Accuracy: 537/2947 (18%)

Train Epoch: 184    Loss per Batch: 1.784880

Test set: Loss per Batch: 1.7936, Accuracy: 537/2947 (18%)

Train Epoch: 185    Loss per Batch: 1.784850

Test set: Loss per Batch: 1.7961, Accuracy: 537/2947 (18%)

Train Epoch: 186    Loss per Batch: 1.784834

Test set: Loss per Batch: 1.7921, Accuracy: 532/2947 (18%)

Train Epoch: 187    Loss per Batch: 1.784880

Test set: Loss per Batch: 1.7917, Accuracy: 532/2947 (18%)

Train Epoch: 188    Loss per Batch: 1.784855

Test set: Loss per Batch: 1.7919, Accuracy: 532/2947 (18%)

edit1:
Now i have tried to check if params are updating at all as suggested by Ahmad.

       x1=net.model.linear2.weight.data # weights at 150 epoch
       x3=net.model.linear2.weight.data # weights at 152 epoch
       x1-x3
tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0.]], device='cuda:0')```

So,now i tried to check grad at 152 epoch
net.model.linear2.weight.grad

tensor([[-5.2324e-14, -1.8329e-14, -4.5901e-15,  3.9225e-14, -4.8923e-14,
         -3.7663e-14, -6.6461e-14, -1.3109e-12, -1.4465e-12],
        [ 3.0959e-14,  2.7681e-14, -1.2809e-14, -2.1927e-14,  4.5212e-14,
          3.6410e-14, -1.3647e-13, -1.2680e-12, -1.4908e-12],
        [-7.7439e-15, -1.8330e-14,  1.2772e-14,  4.4520e-15, -2.0769e-14,
         -1.6550e-14,  1.3497e-13,  1.6057e-12,  1.8513e-12],
        [-7.0256e-15, -3.3731e-14,  2.6417e-14,  1.2869e-14, -4.3155e-14,
         -3.2898e-14,  3.1167e-13,  3.5165e-12,  4.0472e-12],
        [ 4.1122e-14,  1.7677e-14,  4.3402e-16, -2.3387e-14,  3.5274e-14,
          2.8663e-14,  3.6638e-14,  6.8818e-13,  7.4608e-13],
        [-7.3279e-14, -3.9648e-14,  6.5658e-15,  4.4829e-14, -7.3947e-14,
         -5.9679e-14,  2.8081e-14, -2.1435e-13, -1.6659e-13],
        [ 3.9702e-15,  1.8115e-14, -1.4382e-14, -1.4078e-15,  1.8547e-14,
          1.5202e-14, -1.5490e-13, -1.8519e-12, -2.1300e-12],
        [ 1.9415e-14,  1.8774e-14, -8.1929e-15, -1.9029e-14,  3.2933e-14,
          2.4555e-14, -9.4528e-14, -8.9978e-13, -1.0555e-12],
        [-9.5713e-15, -3.9856e-14,  3.1216e-14,  1.8188e-14, -5.4189e-14,
         -4.1472e-14,  3.7540e-13,  4.1245e-12,  4.7497e-12],
        [ 6.3520e-15, -6.6954e-15,  7.5276e-15, -1.1499e-16, -5.6589e-15,
         -3.1609e-15,  9.2859e-14,  1.1613e-12,  1.3267e-12],
        [ 2.0287e-14,  1.1300e-14, -2.4096e-15, -1.6815e-14,  2.5231e-14,
          1.9853e-14, -2.9268e-14, -4.7256e-14, -7.8026e-14],
        [-7.6004e-14, -4.6867e-14,  1.1355e-14,  5.1983e-14, -8.6401e-14,
         -6.8230e-14,  9.0062e-14,  4.3427e-13,  5.8239e-13],
        [ 3.0111e-14,  3.5032e-14, -1.8398e-14, -2.6007e-14,  5.4304e-14,
          4.1803e-14, -2.0308e-13, -2.1058e-12, -2.4519e12]],device='cuda:0')

Now i know that this is the case of vanishing gradient.
How should i proceed to remedy vanishing gradients for this architecture??


title: How to train separate attention weight for each label in pytorch?

body:
I am new to PyTorch. I am trying to train a multilabel LSTM-attention model where each label has its own attention weight.
I would like to ask is there a way like tf.map_fn in Keras that I can train separates attention weight using the same LSTM hidden state?

I've referred to https://github.com/iliaschalkidis/lmtc-eurlex57k. Seems like there is no similar function like tf.map_fn in PyTorch so I need to use for loop. However, I'm not quite familiar with how PyTorch saves the weight and how can I make sure that each attention weight depends on its own label.

def call(self, x, mask=None):

        a = dot_product(x, self.Wa)

        def label_wise_attention(values):
            doc_repi, ai = values
            ai = K.softmax(K.transpose(ai))
            label_aware_doc_rep = K.dot(ai, doc_repi)
            if self.return_attention:
                return [label_aware_doc_rep, ai]
            else:
                return [label_aware_doc_rep, label_aware_doc_rep]

        label_aware_doc_reprs, attention_scores = K.tf.map_fn(label_wise_attention, [x, a])

        # Compute label-scores
        label_aware_doc_reprs = K.sum(label_aware_doc_reprs * self.Wo, axis=-1) + self.bo
        label_aware_doc_reprs = K.sigmoid(label_aware_doc_reprs)

        if self.return_attention:
            return [label_aware_doc_reprs, attention_scores]

        return label_aware_doc_reprs


title: Multiprocessing inside pytorch forward function

body:
I am trying to execute a fully connected layer to each node embedding after a graph constitutional layer.
It works but is extremely slow, so I am trying to paralyze it.

Basically I have this code for the normal version:

class Net(nn.Module):
def __init__(self):
    super(Net, self).__init__()
    # first 3 is the feature size
    self.gcn = nn.ModuleList([
        GCN(3, 32, F.leaky_relu),
        GCN(32, 32, F.leaky_relu),
        GCN(32, 32, F.leaky_relu),
        GCN(32, 32, F.leaky_relu),
        GCN(32, 32, F.leaky_relu),
        GCN(32, 32, F.leaky_relu),
        GCN(32, 32, F.leaky_relu),
        GCN(32, 32, F.leaky_relu),
        GCN(32, 32, F.leaky_relu),
        GCN(32, 32, None)])

    self.fc = nn.ModuleList([
        nn.Linear(32, 32, bias=True),
        nn.Linear(32, 32, bias=True),
        nn.Linear(32, 1, bias=True)])

def forward(self, g, features):

    ret = th.zeros(len(features), 1, dtype=features.dtype, device=features.device)

    for layer in self.gcn:
        features = layer(g, features)

    for i, f in enumerate(features):
        for l in self.fc:
            f = l(f)
        ret[i][0] = f

    return ret

in theory the second for should be easily parallelizable. The problem is I haven't been able to do it. I tried torch.multiprocessing with something like

def tail(layers, i, f, dictionary):
    for l in layers:
        f = l(f)
    dictionary[i]: f

def forward(self, g, features):

    ret = th.zeros(len(features), 1, dtype=features.dtype, device=features.device)

    for layer in self.gcn:
        features = layer(g, features)

    manager = mp.Manager()
    return_dict = manager.dict()

    proc = []
    for i, f in enumerate(features):
        p = mp.Process(target=tail, args=(self.fc, i, f, return_dict))
        p.start()
        proc.append(p)

    for p in proc:
        p.join()

    for i in return_dict.keys():
        ret[i] = return_dict[i]

    return ret

But no luck. Any ideas?


title: How can i convert an ndarray shape (32 by xxx) into a single column dataframe, while discarding the last ten objects of this array

body:
After running a binary classification NLP model over a dataset with batch size of 32, i have an nparray of predictions of size 32 by 300 and the last batch is of size 24. i am trying to rearrange these values in a dataframe.

predictions.append(logits.argmax(1))

[array([0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0,
       0, 0, 0, 1, 0, 0, 0, 0, 0, 0]), array([0, 1, 0, 0, 0, 0, 0, 0............

however, when i try to convert it to a dataframe,

df.labels = pd.DataFrame(predictions)
df.labels.head()

this is what i get

0    0.0
1    0.0
2    0.0
3    0.0
4    0.0
Name: labels, dtype: float64

i want to have the values as 1 or 0 integers, not floats
the final data batch of 32 has NaN from value 24 to 32, instead of being blank


title: Replacing layer weights with torch.sparse tensors

body:
Let's say I have a layer defined C = nn.Conv2d(1,3,3, bias=False), ie, 1 input channel, 3 output channels and a kernel size of 3x3. The internal weight of this layer is thus a tensor of shape (3,1,3,3); I can access this with C.weight.data. Now suppose that this internal weight is very sparse; it's full of zeros and has only a few nonzero values. I can easily construct a sparse tensor from the weight by:

idx = C.weight.data.nonzero().T
values = C.weight.data[C.weight.data!=0]
sp_T = torch.sparse.FloatTensor(idx, values, C.weight.data.size())

Is it possible to store the conv layer's weights as this sparse tensor somehow? I tried simply doing C.weight.data = sp_T but it throws an error. It would be pretty convenient if we could store all the weights in a model in this sparsified way.


title: How to write a scikit-learn estimator in PyTorch

body:
I had developed an estimator in Scikit-learn but because of performance issues (both speed and memory usage) I am thinking of making the estimator to run using GPU.

One way I can think of to do this is to write the estimator in PyTorch (so I can use GPU processing) and then use Google Colab to leverage on their cloud GPUs and memory capacity.

What would be the best way to write an estimator which is already scikit-learn compatible in PyTorch?

Any pointers or hints pointing to the right direction would really be appreciated. Many thanks in advance.


title: Converting this pytorch network in tensorflow 2.0

body:
I have this network:
(ngf = 160, nc=3,nz=100)
class Generator(nn.Module):
    def __init__(self, ngpu):
        super(Generator, self).__init__()
        self.ngpu = ngpu
        self.main = nn.Sequential(  # non usa il dense ma noise Size([16, 100, 1, 1]) che crea sotto
            # input is Z, going into a convolution
            nn.ConvTranspose2d(nz, ngf * 16, 4, bias=False),
            nn.BatchNorm2d(ngf * 16),
            nn.ReLU(True),
            # state size. (ngf*16) x 4 x 4
            nn.ConvTranspose2d(ngf * 16, ngf * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 8 x 8
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 16 x 16
            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 32 x 32
            nn.ConvTranspose2d(ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 64 x 64
            nn.ConvTranspose2d(ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 128 x 128
        )

    def forward(self, input):
        return self.main(input)

and I would like to convert it into tensorflow 2.0. So far this was my attempt:
(ngf = 160, nc=3,nz=100)

def make_generator_model():
    model = tf.keras.Sequential()

    N = 4
    model.add(layers.Input((100,1,1)))
    model.add(layers.Conv2DTranspose(nz, (4, 4), padding='same',use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    #assert model.output_shape == (None, N, N, ngf*16)

    model.add(layers.Conv2DTranspose(ngf*16, (4, 4), strides=(2, 2), padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
   # model.add(layers.Reshape((N*2, N*2, ngf*8)))
   # assert model.output_shape == (None, N*2, N*2, ngf*8)

    model.add(layers.Conv2DTranspose(ngf*8, (4, 4), strides=(2, 2), padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
  #  model.add(layers.Reshape((N*4, N*4, ngf*4)))
   # assert model.output_shape == (None, N*4, N*4, ngf*4)

    model.add(layers.Conv2DTranspose(ngf*4, (4, 4), strides=(2, 2), padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
   # model.add(layers.Reshape((N*8, N*8, ngf*2)))
   # assert model.output_shape == (None, N*8, N*8, ngf*2)

    model.add(layers.Conv2DTranspose(ngf*2, (4, 4), strides=(2, 2), padding='same', use_bias=False))
    model.add(layers.BatchNormalization())
    model.add(layers.ReLU())
    #model.add(layers.Reshape((N*16, N*16, ngf)))
   # assert model.output_shape == (None, N*16, N*16, ngf)

    model.add(layers.Conv2DTranspose(3, (4, 4), strides=(2, 2), padding='same', use_bias=False, activation='tanh'))
   # model.add(layers.Reshape((N*32, N*32, nc)))
  #  assert model.output_shape == (None, N*32, N*32, nc)

    return model

But sadly this is not working. So far the problem is that the shapes are not correct,and for sure the error has to be on the feeding of the input to the conv2d(that's my guess,at least). The main thing that is giving me trouble is that in pytorch I can specify the output dimension from a conv2dTranspose while I can't in tensorflow. There has to be something that I'm missing. I also thought that maybe I could have added a dense layer and then apply a reshape layer before the first conv2dtranspose,but am I changing the shape of the network/am I icnreasing the number of parameters? I had such doubt so I avoided to do such things.