title: sklearn & pytorch: Train test split for neural net training in pipeline for a grid search

body:
I am working on a pretty big dataset, which we decided to groupKfold (as we got measurements in the dataset which shouldn't get split, but folded i k folds).
We then are grid searching sklearn models with the groupkfolded dataset and either RandomizedGridSearch or BayesenGridSearch. To use neural nets in this pipeline we decied to fit pytorch in the sklearn interface. For that we used from sklearn.base import BaseEstimator, ClassifierMixin.
Then we are setting up a pipeline:
class Neural_Net_Interface(ClassifierMixin, BaseEstimator):
    def __init__(self, X_test, y_test, Max_num_epochs, Early_Stopping, and so on...):
     self.....

    def fit(self, X_train, y_train):
       ...

    def predict(self, X):
       ...

pipeline_nn = Pipeline([('std', StandardScaler()),
                        ('splitter', train_test_split(X, y, test_size=0.2, random_state=69)),
                        ('nn', Neural_Net_Interface(X_test=X,
                                                    y_test=y,
                                                    Max_num_epochs=3,
                                                    Early_Stopping=True,
                                                    ... (20 more parameters))])

cv_object = GroupKFold(n_splits=np.max(group_vector) + 1)

model_grid_cv = BayesSearchCV(estimator=pipeline_nn,
                              search_spaces=search_space,
                              scoring=my_scorer,
                              optimizer_kwargs={'base_estimator': 'NN', 'n_initial_points': 20},
                              cv=cv_object,
                              n_jobs=N_JOBS,
                              verbose=100,
                              n_iter=N_ITER,
                              n_points=N_POINTS,
                              iid=False,
                              random_state=69)

model_grid_cv.fit(X, y, groups=groups)

And here comes the problem:
As you can see above the NeuralNetInterface (sklearn classifier) awaits an input for a test X & y. This is because after each training epoch we need to evaluate the NN accuracy. I can't train test split the dataset once in the beginning, as this would undermine the sense of a kFold. So what I am trying to do is to define the pipeline in a way that the output of the train test split is passed to the neural net interface. This is not working.
Besides my real question is:
-The groupKFold folds 4 groups 4 times, taking 3 parts for training and one part for the score estimation. --> How can I adjust the pipeline in a way that the 1 out of 4 parts of the kFold is passed to the NeuralNetInterface so that this part is used for the NN evaluation? Do I need to adjust the NeuralNetInterface not taking a test set??
-Or is that not possible and I need to train test split the data in the GridSearch always passing one part to the NeuralNetInterface? How do I get that working?
I hope I described my question well enough to understand.
Thanks for your help in advance!
Best regards


title: warnings when using python setup.py develop

body:
I'm trying to use mmskeleton from https://github.com/open-mmlab/mmskeleton/blob/master/doc/GETTING_STARTED.md
and I have encountered an error while following their instructions.
The steps I took are:

Download Anaconda and create a new environment (named env6)
Activate the environment
clone the repos from github and cd into the directory of the setup.py files
run conda install pytorch==1.2.0 torchvision==0.4.0 cudatoolkit=10.0 -c pytorch

note: the computer I'm using has CUDA 11.2 (checked using nvidia-smi and nvcc -V)

Got an error saying ModuleNotFoundError: No module names 'Cython' so downloaded cython using conda install -c anaconda cython
Tried to run python setup.py develop and got the error below

C:\Users\flyingalpaca\Anaconda3\envs\env6\lib\site-packages\setuptools\dist.py:452: UserWarning: Normalizing '0.7.rc1+unknown' to '0.7rc1+unknown'
  warnings.warn(tmpl.format(**locals()))
running develop
running egg_info
writing mmskeleton.egg-info\PKG-INFO
writing dependency_links to mmskeleton.egg-info\dependency_links.txt
writing requirements to mmskeleton.egg-info\requires.txt
writing top-level names to mmskeleton.egg-info\top_level.txt
reading manifest file 'mmskeleton.egg-info\SOURCES.txt'
writing manifest file 'mmskeleton.egg-info\SOURCES.txt'
running build_ext
C:\Users\flyingalpaca\Anaconda3\envs\env6\lib\site-packages\torch\utils\cpp_extension.py:189: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
  warnings.warn('Error checking compiler version for {}: {}'.format(compiler, error))
building 'mmskeleton.ops.nms.gpu_nms' extension
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -DWITH_CUDA -Immskeleton\ops\nms -IC:\Users\flyingalpaca\Anaconda3\envs\env6\lib\site-packages\numpy\core\include -IC:\Users\flyingalpaca\Anaconda3\envs\env6\lib\site-packages\torch\include -IC:\Users\flyingalpaca\Anaconda3\envs\env6\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\flyingalpaca\Anaconda3\envs\env6\lib\site-packages\torch\include\TH -IC:\Users\flyingalpaca\Anaconda3\envs\env6\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\include" -IC:\Users\flyingalpaca\Anaconda3\envs\env6\include -IC:\Users\flyingalpaca\Anaconda3\envs\env6\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\ATLMFC\include" "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.28.29333\include" "-IC:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\include\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt" /EHsc /Tpmmskeleton\ops\nms\gpu_nms.cpp /Fobuild\temp.win-amd64-3.7\Release\mmskeleton\ops\nms\gpu_nms.obj /MD /wd4819 /EHsc -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=gpu_nms -D_GLIBCXX_USE_CXX11_ABI=0
gpu_nms.cpp
C:\Users\flyingalpaca\Anaconda3\envs\env6\lib\site-packages\numpy\core\include\numpy\npy_1_7_deprecated_api.h(14) : Warning Msg: Using deprecated NumPy API, disable it with #define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION
mmskeleton\ops\nms\gpu_nms.cpp(2005): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
mmskeleton\ops\nms\gpu_nms.cpp(2014): warning C4244: '=': conversion from 'npy_intp' to 'int', possible loss of data
mmskeleton\ops\nms\gpu_nms.cpp(2216): error C2664: 'void _nms(int *,int *,const float *,int,int,float,int)': cannot convert argument 1 from '__pyx_t_5numpy_int32_t *' to 'int *'
mmskeleton\ops\nms\gpu_nms.cpp(2216): error C2664: 'void _nms(int *,int *,const float *,int,int,float,int)': cannot convert argument 1 from '__pyx_t_5numpy_int32_t *' to 'int *'
mmskeleton\ops\nms\gpu_nms.cpp(2216): note: Types pointed to are unrelated; conversion requires reinterpret_cast, C-style cast or function-style cast
C:\Users\flyingalpaca\Documents\mmskeleton\mmskeleton\ops\nms\gpu_nms.hpp(1): note: see declaration of '_nms'
error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29333\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2

Could I get guidance on how I could solve this problem?
Thanks in advance


title: How to get the class with highest confidence from a tensor?

body:
I want to obtain the class with the highest confidence.
Heres the code that performs classification task:
names = ['class A', 'class B', 'class C']

def classify_face(image):
    device = torch.device("cpu")
    img = process_image(image)
    print('Image processed')
    # img = image.unsqueeze_(0)
    # img = image.float()

    pred = model(img)[0]

    # Apply NMS
    pred = non_max_suppression(pred, 0.4, 0.5, classes = [0, 1, 2], agnostic = None )
    if classify:
        pred = apply_classifier(pred, modelc, img, im0s)
    #print(pred)

    model.eval()
    model.cpu()

    print(pred)

    # output = non_max_suppression(output, 0.4, 0.5, classes = class_names, agnostic = False)

    #_, predicted = torch.max(output[0], 1)
    #print(predicted.data[0], "predicted")

    classification = torch.cat(pred)[:, -1]
    index = int(classification)
    print(names[index])
    return names[index]

The above code works perfectly when the pred has a 1D tensor. But I get an error if the tensor size is more than that.
There are 5 elements: x1, y1, x2, y2, confidence, and class.
For example:
pred = [torch.tensor([[212.38568, 117.47020, 339.35773, 266.00513, 0.74144, 2.00000],
                      [214.60651, 118.50694, 339.90192, 265.91696, 0.94277, 0.00000]])]

Error:
Traceback (most recent call last):
  File "WEBCAM_DETECT.py", line 172, in <module>
    label = classify_face(frame)
  File "WEBCAM_DETECT.py", line 154, in classify_face
    index = int(classification)
ValueError: only one element tensors can be converted to Python scalars

So I want to access the class with highest confidence. Please let me know how to do that or if there is a better way to do.


title: ValueError: Expected x_max for bbox (0.65, 0.51, 1.12, 0.64, 3) to be in the range [0.0, 1.0], got 1.1234809015877545

body:
I want to apply data augmentations from PyTorch's Albumentations to images with bounding boxes.
When I apply the HorizontalFlip Transformation, I receive this error ValueError: Expected x_max for bbox (0.6505353259854019, 0.517013871576637, 1.1234809015877545, 0.6447916687466204, 3) to be in the range [0.0, 1.0], got 1.1234809015877545.
I use the following code
A.Compose([
           A.HorizontalFlip(p=1),
           ToTensorV2(p=1.0)],
           p=1.0,
           bbox_params=A.BboxParams(format='coco',min_area=0, min_visibility=0,label_fields=['labels'])
                  )

When I apply the Cutout transformation, I do not have any error regarding the bounding boxes
A.Compose([
          A.Cutout(num_holes=10, max_h_size=32, max_w_size=32, fill_value=0, p=0.5),
          ToTensorV2(p=1.0)],
          p=1.0,
          bbox_params=A.BboxParams(format='coco',min_area=0, min_visibility=0,label_fields=['labels'])
                  )


title: Is there an easy way to time pytorch code running in GPU?

body: Are there any easy ways to time pytorch code (python) while running in GPU ( not in CPU)? I can simply use the time module builtin in python to time code in CPU. However are there any easy way to use functions to time code for GPU? Thanks.


title: My LSTM code give flat prediction and don't know what is wrong..Anyone can help take a look?

body:
Here is the code. I think the class mylstm has problem but I can not find it... The input is simple, which is just 7 columns data.

I tried to print out all the tensors but did not find what was wrong. Thanks for help!

class mylstm(nn.Module):
    def __init__(self, input_size, hidden_size, T, logger):
        super(mylstm, self).__init__()

        self.T = T
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.logger = logger

        self.lstm_layer = nn.LSTM(input_size = 7, hidden_size = hidden_size)
        self.fc = nn.Linear(hidden_size, 1)
        #self.fc.weight.data.normal_()

    def forward(self, input_data):

        hidden = self.init_hidden(input_data)
        cell = self.init_hidden(input_data)
        for t in range(self.T - 1):
            if t < self.T - 1:
                self.lstm_layer.flatten_parameters()
                _, lstm_output = self.lstm_layer(input_data[:,t,:].unsqueeze(0), (hidden, cell))
                hidden = lstm_output[0]
                cell = lstm_output[1]
        y_pred = self.fc(hidden[0])
        return y_pred

    def init_hidden(self, x):
        return Variable(x.data.new(1, x.size(0), self.hidden_size).zero_())

# Train the model
class rnn:
    def __init__(self, file_data, logger, input_size = 7, hidden_size = 64, T = 10,
                 learning_rate = 0.01, batch_size = 128, parallel = True, debug = False):
        self.T = T
        dat = pd.read_csv(file_data, nrows = 100 if debug else None)
        self.logger = logger
        self.logger.info("Shape of data: %s.\nMissing in data: %s.", dat.shape, dat.isnull().sum().sum())

        self.X = dat.loc[:, [x for x in dat.columns.tolist()]].values
        self.y = np.array(dat.rtm_spp)
        self.batch_size = batch_size

        self.lstm1 = mylstm(input_size = input_size,
                               hidden_size = hidden_size,
                               T = T, logger = logger)

        if parallel:
            self.lstm1 = nn.DataParallel(self.lstm1)

        self.lstm1_optimizer = optim.Adam(params = filter(lambda p: p.requires_grad, self.lstm1.parameters()),
                                           lr = learning_rate)

        self.train_size = 20000
        self.y = self.y - np.mean(self.y[:self.train_size]) # Question: why Adam requires data to be normalized?
        self.logger.info("Training size: %d.", self.train_size)

    def train(self, n_epochs = 10):
        iter_per_epoch = int(np.ceil(self.train_size * 1. / self.batch_size))
        logger.info("Iterations per epoch: %3.3f ~ %d.", self.train_size * 1. / self.batch_size, iter_per_epoch)
        self.iter_losses = np.zeros(n_epochs * iter_per_epoch)
        self.epoch_losses = np.zeros(n_epochs)

        self.loss_func = nn.MSELoss()

        n_iter = 0

        learning_rate = 1.

        for i in range(n_epochs):
            perm_idx = np.random.permutation(self.train_size - self.T-1)
            j = 0
            while j < self.train_size:
                batch_idx = perm_idx[j:(j + self.batch_size)]
                X = np.zeros((len(batch_idx), self.T - 1, self.X.shape[1]))
                #y_history = np.zeros((len(batch_idx), self.T - 1))
                y_target = self.y[batch_idx + self.T]

                for k in range(len(batch_idx)):
                    X[k, :, :] = self.X[batch_idx[k] : (batch_idx[k] + self.T - 1), :]

                loss = self.train_iteration(X, y_target)
                self.iter_losses[i * iter_per_epoch + j // self.batch_size] = loss
                #if (j / self.batch_size) % 50 == 0:

                j += self.batch_size
                n_iter += 1

                if n_iter % 10000 == 0 and n_iter > 0:
                    for param_group in self.lstm1_optimizer.param_groups:
                        param_group['lr'] = param_group['lr'] * 0.9

            self.epoch_losses[i] = np.mean(self.iter_losses[range(i * iter_per_epoch, (i + 1) * iter_per_epoch)])
            if i % 10 == 0:
                self.logger.info("Epoch %d, loss: %3.3f.", i, self.epoch_losses[i])
                y_train_pred = self.predict(on_train = True)
                y_test_pred = self.predict(on_train = False)

    def train_iteration(self, X,y_target):
        self.lstm1_optimizer.zero_grad()
        y_pred = self.lstm1(Variable(torch.from_numpy(X).type(torch.FloatTensor)))
        y_true = Variable(torch.from_numpy(y_target).type(torch.FloatTensor))
        y_true = y_true.view(y_true.shape[0],1)
        y_pred=y_pred.squeeze(0)
        print(y_pred)
        loss = self.loss_func(y_pred, y_true)
        loss.backward()
        self.lstm1_optimizer.step()
        return loss.data[0]

    def predict(self, on_train = False):
        if on_train:
             y_pred = np.zeros(self.train_size - self.T +1)
        else:
             y_pred = np.zeros(self.X.shape[0] - self.train_size)

        i = 0
        while i < len(y_pred):
            batch_idx = np.array(range(len(y_pred)))[i : (i + self.batch_size)]
            X = np.zeros((len(batch_idx), self.T - 1, self.X.shape[1]))
            #y_history = np.zeros((len(batch_idx), self.T - 1))
            for j in range(len(batch_idx)):
                if on_train:
                    X[j, :, :] = self.X[range(batch_idx[j], batch_idx[j] + self.T - 1), :]

                else:
                    X[j, :, :] = self.X[range(batch_idx[j] + self.train_size - self.T, batch_idx[j] + self.train_size - 1), :]

            input_data = Variable(torch.from_numpy(X).type(torch.FloatTensor))
           # print(self.lstm1(torch.randn(128,9,7)))
            #print(self.lstm1(X).data.numpy())
            y_pred[i:(i + self.batch_size)] = self.lstm1(input_data).data.numpy()[:,0]
            i += self.batch_size
        return y_pred

model = rnn(file_data = 'L.csv', logger = logger, parallel = False,
              learning_rate = .001)

model.train(n_epochs = 1000)

y_pred = model.predict()


title: Why does my PyTorch installed with anaconda need root privilege to run on cuda GPU?

body:
I use a 1080Ti and set up my environment as follows:

Installed ubuntu 16.04, Nvidia driver 384.130, CUDA-8.0
Installed anaconda
Created an environment and installed PyTorch with conda install pytorch==0.4.0 torchvision cudatoolkit=8.0 -c pytorch

When I run a PyTorch code on cuda:0, a cuda runtime error (30) error occurs. I did some searching and it returns that I need to run with sudo. [link to the solution] It worked when I ran the code with root privileges.

But is there any other solution? I cannot always do this on a server where I do not have root access.


title: PyTorch: index 2D tensor with 2D tensor of row indices

body:
I have a torch tensor a of shape (x, n) and another tensor b of shape (y, n) where y <= x. every column of b contains a sequence of row indices for a and what I would like to be able to do is to somehow index a with b such that I obtain a tensor of shape (y, n) in which the ith column contains a[:, i][b[:, i]] (not quite sure if that's the correct way to express it).

Here's an example (where x = 5, y = 3 and n = 4):

import torch

a = torch.Tensor(
    [[0.1, 0.2, 0.3, 0.4],
     [0.6, 0.7, 0.8, 0.9],
     [1.1, 1.2, 1.3, 1.4],
     [1.6, 1.7, 1.8, 1.9],
     [2.1, 2.2, 2.3, 2.4]]
)

b = torch.LongTensor(
    [[0, 3, 1, 2],
     [2, 2, 2, 0],
     [1, 1, 0, 4]]
)

# How do I get from a and b to c
# (so that I can also assign to those elements in a)?

c = torch.Tensor(
    [[0.1, 1.7, 0.8, 1.4],
     [1.1, 1.2, 1.3, 0.4],
     [0.6, 0.7, 0.3, 2.4]]
)

I can't get my head around this. What I'm looking for is a method that will not yield the tensor c but also let me assign a tensor of the same shape as c to the elements of a which c is made up of.


title: How to customize pytorch data

body:
I am trying to make a customized Dataloader using pytorch.

I've seen some codes like (omitted the class sorry.)

def __init__(self, data_root, transform=None, training=True, return_id=False):
    super().__init__()
    self.mode = 'train' if training else 'test'

    self.data_root = Path(data_root)
    csv_fname = 'train.csv' if training else 'sample_submission.csv'
    self.csv_file = pd.read_csv(self.data_root / csv_fname)
    self.transform = transform
    self.return_id = return_id
def __getitem__():
    """ TODO
    """
def __len__():
    """ TODO
    """

The problem here is that the datas I've dealt with before contains all the training data in one csv file, and all the testing data in the other csv file, with total 2 csv files for training and testing. (For example like in MNIST, the last column is the labeling, and the all the previous columns are each different features.)

However, the problem I've been facing is that I've got very many, (about 200,000) csv files for training, each one smaller than 60,000 sized MNIST, but still quite big. All these csv files contain different number of rows.

To inherit torch.util.data, how can I make customized class? MNIST dataset is quite small, so can be uploaded on RAM at once. However, the data I'm dealing with is super big, so I need some help.

Any ideas? Thank you in advance.


title: Python, Pytorch, having trouble using cuda

body:
OS: Windows 10

Python 3.7.4 (Conda)

GPU: GTX 980

So I installed CUDA toolkit v.10.1, got the matching cuDNN files, installed the cuda 10.1 enabled pytorch version and in addition to that i updated my gpu drivers. but if i now try to check wether my gpu is available with torch.cuda.is_available() it still get a False. Any Ideas??


title: How Pytorch DataParallel affects learning rate for same batch size?

body:
Given that PyTorch DataParallel split data across GPUs, is this process expected to affect the Learning rate parameter?
Let's say I was able to use a batch_size=200 with lr=0.01 on a 16GB GPU instance. Now is switch to 4x8GB instance and choose to use a batch_size=200. PyTorch DataParallel will split this batch to 50 mini_batch for each GPU and perform forward ad backward pass. As my effective batch_size is reduced did i suppose to decrease the lr?


title: Transfer Learning With Resnet18 on CIFAR10: Poor Training Accuracy

body:
I am playing around with the Pytorch library and trying to use Transfer Learning.

My code is as follows:

# get the model with pre-trained weights
resnet18 = models.resnet18(pretrained=True)

# freeze all the layers
for param in resnet18.parameters():
    param.requires_grad = False

# print and check what the last FC layer is:
# Linear(in_features=512, out_features=1000, bias=True)
print(resnet18)

# set the final FC layer to what we require for our problem
resnet18.fc = nn.Linear(512, 10)

# unfreeze the last layer so it learns on our dataset
for param in resnet18.fc.parameters():
  param.requires_grad = True

I set the optimizer as:

# set optimizer
lr = 1e-2
optimizer = torch.optim.SGD(resnet18.parameters(), lr=lr, momentum=0.5)

Training this model on CIFAR10 gives me a very poor training accuracy of 44%.

Am I doing transfer learning correctly here? I would have expected much better results. If someone with more experience could kindly explain how they would approach this it would be much appreciated.

I followed the tutorial here:

https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html


title: numpy array type not supported?

body:
I'm trying to copy a model I was able to follow and run through a tutorial, but this time with my own data.
I was able to convert my own MRI images to numpy arrays in the same dimensions as the arrays the tutorial data is.
I tried replacing the numpy arrays in my tutorial with my own arrays and writing my own fictional csv file for normal or abnormal (case, not case).
However when I run it, I get:
(Pytorch) C:\Users\GlaDOS\PythonProjects\dicomnpy>python train.py  -t acl -p sagittal --epochs=10 --prefix_name hue
Traceback (most recent call last):
  File "train.py", line 277, in <module>
    run(args)
  File "train.py", line 214, in run
    mrnet, train_loader, epoch, num_epochs, optimizer, writer, current_lr, log_every)
  File "train.py", line 34, in train_model
    for i, (image, label, weight) in enumerate(train_loader):
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "C:\Users\GlaDOS\PythonProjects\dicomnpy\dataloader.py", line 56, in __getitem__
    array = self.transform(array)
  File "c:\users\glados\src\torchsample\torchsample\transforms\tensor_transforms.py", line 32, in __call__
    inputs = transform(*inputs)
  File "C:\Users\GlaDOS\anaconda3\envs\Pytorch\lib\site-packages\torchvision\transforms\transforms.py", line 313, in __call__
    return self.lambd(img)
  File "train.py", line 167, in <lambda>
    transforms.Lambda(lambda x: torch.Tensor(x)),
TypeError: can't convert np.ndarray of type numpy.uint16. The only supported types are: float64, float32, float16, int64, int32, int16, int8, uint8, and bool.

Now I'm wondering if this error means I somehow didn't convert my MRI's to the "correct" numpy array type? And if so, how do I go about changing them to the correct type?


title: Is there a power function for tensors in pytorch?

body:
Is there an equivalent of numpy.power in pytorch?
The function basically raises each element in the first tensor to the power represented in each corresponding element in the second tensor.


title: torch Adamoptimizer produce cuda error in optimizer.step()

body:
after add my custom loss function using 3 Torch Linear layer,
i get an cuda error
class KLDLoss(nn.Module):
  def __init__(self, reduction='sum'):
      super(KLDLoss, self).__init__()
      self.reduction = reduction

  def forward(self, mean, logvar):
    # KLD loss
      kld_loss = -0.5 * torch.sum(1 + logvar - mean.pow(2) - logvar.exp(), 1)
    # Size average
      if self.reduction == 'mean':
        kld_loss = torch.mean(kld_loss)
      elif self.reduction == 'sum':
        kld_loss = torch.sum(kld_loss)
      return kld_loss

class Latent_Classifier(nn.Module):
    def __init__(self):
        super(Latent_Classifier, self).__init__()
        layers = []
        layers += [nn.Linear(128, 750)]
        layers += [nn.Linear(750, 750)]
        layers += [nn.Linear(750, 1)]

        self.seq = nn.Sequential(*layers)
  def forward(self, latent_z):
    x = self.seq(latent_z)

    return -torch.mean(torch.log(x)) - torch.mean(torch.log(1 - x))

KLDLoss has no error but Latent Classifier has an error after some training phase in optimizer.step()
105                     denom = (max_exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(group['eps'])

106                 else:

--> 107                     denom = (exp_avg_sq.sqrt() / math.sqrt(bias_correction2)).add_(group['eps'])

108

109                 step_size = group['lr'] / bias_correction1

RuntimeError: CUDA error: device-side assert triggered

is an error exist in my Latent Classifier code??
optimizer is AdamOptimizer and args are 0.0002 lr, (0.5, 0.999)betas


title: How does one install torchmeta for a ppc64le architecture in pytorch?

body:
I was trying to use torchmeta in a ppc64le architecture. Unfortunately it's not been easy to install since ppc64le requires special binaries to work.
I eventually managed to get the right binaries for pytorch and torchvision by following these instructions (that prepend the right ibm channel with the conda binaries, plus installs all the required files too):
conda config --prepend channels https://public.dhe.ibm.com/ibmdl/export/pub/software/server/ibm-ai/conda/
conda create -n my_new_env python=3.7 powerai=1.7.0
conda activate my_new_env

after that I proceeded to install the right version of torchmeta, which was 1.3.1 since ppc64le only has pytorch 1.3.1 and torchvision 0.4.2. So I did:
pip install torchmeta==1.3.1

but now I have a new error that it cannot find the right version of h5py compatible with what I want to do. The error message is to large to paste but I will paste what I hope are useful part of it:
(my_new_env) [miranda9@hal-login ~]$ pip install torchmeta==1.3.1
Collecting torchmeta==1.3.1
  Using cached torchmeta-1.3.1-py3-none-any.whl (144 kB)
Requirement already satisfied: requests in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (2.22.0)
Requirement already satisfied: torchvision<0.6.0,>=0.4.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (0.4.2)
Requirement already satisfied: torch<1.5.0,>=1.3.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.3.1)
Processing ./.cache/pip/wheels/87/f5/ad/9f04a48453875e8054c19f9fe3f50cbbe0c09b956835555019/Pillow-6.2.2-cp37-cp37m-linux_ppc64le.whl
Requirement already satisfied: numpy>=1.14.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (1.17.4)
Requirement already satisfied: tqdm>=4.0.0 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchmeta==1.3.1) (4.36.1)
Collecting h5py~=2.9.0
  Using cached h5py-2.9.0.tar.gz (287 kB)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (3.0.4)
Requirement already satisfied: certifi>=2017.4.17 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2020.6.20)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (1.25.10)
Requirement already satisfied: idna<2.9,>=2.5 in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from requests->torchmeta==1.3.1) (2.8)
Requirement already satisfied: six in ./.conda/envs/my_new_env/lib/python3.7/site-packages (from torchvision<0.6.0,>=0.4.0->torchmeta==1.3.1) (1.13.0)
Building wheels for collected packages: h5py
  Building wheel for h5py (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ccg1oj0n
       cwd: /tmp/pip-install-bpmeop26/h5py/
  Complete output (1321 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-ppc64le-3.7
  creating build/lib.linux-ppc64le-3.7/h5py
  copying h5py/__init__.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/h5py_warnings.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/highlevel.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/ipy_completer.py -> build/lib.linux-ppc64le-3.7/h5py
  copying h5py/version.py -> build/lib.linux-ppc64le-3.7/h5py
  creating build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/attrs.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/base.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/compat.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/dataset.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/datatype.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/dims.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/files.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/filters.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/group.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/selections.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/selections2.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  copying h5py/_hl/vds.py -> build/lib.linux-ppc64le-3.7/h5py/_hl
  creating build/lib.linux-ppc64le-3.7/h5py/tests
  copying h5py/tests/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests
  copying h5py/tests/common.py -> build/lib.linux-ppc64le-3.7/h5py/tests
  creating build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_attrs.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_attrs_data.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_base.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_dataset.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_dimension_scales.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_file_image.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_group.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5d_direct_chunk_write.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5f.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5p.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_h5t.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_objects.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_selections.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  copying h5py/tests/old/test_slicing.py -> build/lib.linux-ppc64le-3.7/h5py/tests/old
  creating build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_attribute_create.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dataset_getitem.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dataset_swmr.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_datatype.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_deprecation.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_dims_dimensionproxy.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_file.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_filters.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  copying h5py/tests/hl/test_threads.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl
  creating build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/__init__.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_highlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
  running build_ext
  Autodetected HDF5 1.10.2
  ********************************************************************************
                         Summary of the h5py configuration

      Path to HDF5: None
      HDF5 Version: '1.10.2'
       MPI Enabled: False
  Rebuild Required: True

  ********************************************************************************
  Executing api_gen rebuild of defs
  Executing cythonize()
  [ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx
  /tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd
    tree = Parsing.p_module(s, pxd, full_module_name)

...

  /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   #warning "Using deprecated NumPy API, disable it with " \
    ^
  In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0:
  /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
   #include "hdf5.h"
                    ^
  compilation terminated.
  error: command 'gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for h5py
  Running setup.py clean for h5py
Failed to build h5py
DEPRECATION: Could not build wheels for h5py which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Installing collected packages: Pillow, h5py, torchmeta
  Attempting uninstall: Pillow
    Found existing installation: Pillow 7.1.2
    Uninstalling Pillow-7.1.2:
      Successfully uninstalled Pillow-7.1.2
  Attempting uninstall: h5py
    Found existing installation: h5py 2.8.0
    Uninstalling h5py-2.8.0:
      Successfully uninstalled h5py-2.8.0
    Running setup.py install for h5py ... error
    ERROR: Command errored out with exit status 1:
     command: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py

...

    copying h5py/tests/hl/test_vds/test_lowlevel_vds.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
    copying h5py/tests/hl/test_vds/test_virtual_source.py -> build/lib.linux-ppc64le-3.7/h5py/tests/hl/test_vds
    running build_ext
    Autodetected HDF5 1.10.2
    ********************************************************************************
                           Summary of the h5py configuration

        Path to HDF5: None
        HDF5 Version: '1.10.2'
         MPI Enabled: False
    Rebuild Required: True

    ********************************************************************************
    Executing cythonize()
    [ 1/22] Cythonizing /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pyx
    /tmp/pip-install-bpmeop26/h5py/.eggs/Cython-0.29.21-py3.7.egg/Cython/Compiler/Main.py:369: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/pip-install-bpmeop26/h5py/h5py/_conv.pxd

...

    warning: h5py/api_types_hdf5.pxd:730:6: 'H5Z_ERROR_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:731:6: 'H5Z_DISABLE_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:732:6: 'H5Z_ENABLE_EDC' redeclared
    warning: h5py/api_types_hdf5.pxd:733:6: 'H5Z_NO_EDC' redeclared
    building 'h5py.defs' extension
    creating build/temp.linux-ppc64le-3.7
    creating build/temp.linux-ppc64le-3.7/tmp
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py
    creating build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py
    gcc -pthread -B /home/miranda9/.conda/envs/my_new_env/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -DH5_USE_16_API -I./h5py -I/tmp/pip-install-bpmeop26/h5py/lzf -I/opt/local/include -I/usr/local/include -I/home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include -I/home/miranda9/.conda/envs/my_new_env/include/python3.7m -c /tmp/pip-install-bpmeop26/h5py/h5py/defs.c -o build/temp.linux-ppc64le-3.7/tmp/pip-install-bpmeop26/h5py/h5py/defs.o
    In file included from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h:1830:0,
                     from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                     from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                     from /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:26,
                     from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:
    /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
     #warning "Using deprecated NumPy API, disable it with " \
      ^
    In file included from /tmp/pip-install-bpmeop26/h5py/h5py/defs.c:654:0:
    /tmp/pip-install-bpmeop26/h5py/h5py/api_compat.h:27:18: fatal error: hdf5.h: No such file or directory
     #include "hdf5.h"
                      ^
    compilation terminated.
    error: command 'gcc' failed with exit status 1
    ----------------------------------------
  Rolling back uninstall of h5py
  Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py
   from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py
  Moving to /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/h5py-2.8.0-py3.7.egg-info
   from /home/miranda9/.conda/envs/my_new_env/lib/python3.7/site-packages/~5py-2.8.0-py3.7.egg-info
ERROR: Command errored out with exit status 1: /home/miranda9/.conda/envs/my_new_env/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"'; __file__='"'"'/tmp/pip-install-bpmeop26/h5py/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-hlwpfooj/install-record.txt --single-version-externally-managed --compile --install-headers /home/miranda9/.conda/envs/my_new_env/include/python3.7m/h5py Check the logs for full command output.

anyone know how I can successfully install a working torchmeta version in a ppc64le (using wmcle 1.7.0)?

related:

gitissue for torchmeta: https://github.com/tristandeleu/pytorch-meta/issues/95

IBM gitissue for torchmeta support: https://github.com/IBM/powerai/issues/269

h5py gitissue for torchmeta: https://github.com/h5py/h5py/issues/1678

IBM h5py support for torchmeta: https://github.com/IBM/powerai/issues/270


title: Using Adam to find the minimum of the Rosenbrock function using Pytorch

body:
I am comparing the Adam - Algorithm to SGD with Momentum. I realised that the convergence rate of Adam is way worse than the convergence rate of SGD with Momentum if applied to the Rosenbrock function. This finding is in contrast to this visualisation. You can read the underlying code here.
Too ensure that I did not have an implementation error I compared the results of my algorithm to the Pytorch implementation. Pytorch and my implementation return the same result.
Therefore either Pytorch and my implementation is incorrect or the implementation in the link is incorrect. If you check out the code from the link above you will find that the Bias correction step is missing. After adapting my code in the same way the results did not significantly improve.
So my question is why does it work in the linked scenario but not in my/Pytorch implementation? Even though all of the three should return the same result.
import numpy as np
import torch

# Rosenbrock function
class Rosenbrock:
    a_f = 1.
    b_f = 2.
    # The minimum is at (a_f, a_f**2)
class Adam_para:
    beta1 = 0.9 # 0.7 # modified because of github: https://gist.github.com/EmilienDupont/f97a3902f4f3a98f350500a3a00371db
    beta2 = 0.999
    eps = 1e-8
    lr = 2e-2
    iterations = 100

def f(x,y):
    return ( Rosenbrock.a_f - x ) ** 2 + Rosenbrock.b_f * (y - x ** 2 ) ** 2

def grad_f(x,y):
    grad_x = - 1. * 2 * (Rosenbrock.a_f - x) + Rosenbrock.b_f * (- 2 * x) * 2 * ( y - x ** 2 )
    grad_y = Rosenbrock.b_f * ( 1. ) * 2 * (y - x ** 2)
    return np.array([grad_x, grad_y])

def adam_inner(p: np.ndarray,t,exp_avg,exp_avg_sqr, lr):
    # inner loop of adam algorithm
    # p current point
    # exp_avg first moment estimate
    # exp_avg_sqr second moment estimate
    # lr learning rate
    # the following values are taken from the ADAM Paper
    beta1 = Adam_para.beta1
    beta2 = Adam_para.beta2
    eps = Adam_para.eps
    t = t+1
    g = grad_f(*p)
    exp_avg = beta1 * exp_avg + ( 1 - beta1 ) * g
    exp_avg_sqr = beta2 * exp_avg_sqr + ( 1 - beta2 ) * np.square(g)
    bias_corr_1 =  1 - beta1 ** t
    bias_corr_2 =  1 - beta2 ** t
    exp_avg_hat = exp_avg / bias_corr_1
    exp_avg_sqr_hat = exp_avg_sqr / bias_corr_2
    denom = np.sqrt(exp_avg_sqr_hat) + eps
    p = p - lr * exp_avg_hat / denom
    return {'p': p, 'first_mom': exp_avg, 'second_mom': exp_avg_sqr}

def adam(p, it, lr=0.001):
    # it number of iterations
    # m first moment estimate
    # v second moment estimate
    # init
    m = 0
    v = 0
    p_list = [p]
    for i in range(it):
        tmp = adam_inner(p_list[-1],i,m,v,lr)
        p_list.append(tmp['p'])
        m = tmp['first_mom']
        v = tmp['second_mom']
    return np.asarray(p_list)

x0 = np.array([3.,3.])
t = adam(x0,Adam_para.iterations,Adam_para.lr)
x0_torch = torch.tensor(x0, requires_grad=True)
f_torch = f(x0_torch[0],x0_torch[1])
optimizer = torch.optim.Adam([x0_torch], lr = Adam_para.lr, betas=(Adam_para.beta1,Adam_para.beta2))
for i in range(Adam_para.iterations):
    optimizer.zero_grad()
    f_torch = f(x0_torch[0],x0_torch[1])
    f_torch.backward()
    optimizer.step()

print("pytorch result:", x0_torch)
print("my result:", t[-1])


title: What is the function of FrozenBatchNorm2d in “maskrcnn_benchmark”?

body:
"maskrcnn_benchmark"s github
Here is the source code for "FrozenBatchNorm2d"
import torch
from torch import nn
class FrozenBatchNorm2d(nn.Module):
    def __init__(self, n):
        super(FrozenBatchNorm2d, self).__init__()
        self.register_buffer("weight", torch.ones(n))
        self.register_buffer("bias", torch.zeros(n))
        self.register_buffer("running_mean", torch.zeros(n))
        self.register_buffer("running_var", torch.ones(n))

    def forward(self, x):
        scale = self.weight * self.running_var.rsqrt()
        bias = self.bias - self.running_mean * scale
        scale = scale.reshape(1, -1, 1, 1)
        bias = bias.reshape(1, -1, 1, 1)
        return x * scale + bias

When I put this function in my script, I found that this function had almost no effect.
Here is my usage
import torch.nn as nn
import torch
class FrozenBatchNorm2d(nn.Module):
    """
    BatchNorm2d where the batch statistics and the affine parameters
    are fixed
    """

    def __init__(self, n):
        super(FrozenBatchNorm2d, self).__init__()
        self.register_buffer("weight", torch.ones(n))
        self.register_buffer("bias", torch.zeros(n))
        self.register_buffer("running_mean", torch.zeros(n))
        self.register_buffer("running_var", torch.ones(n))

    def forward(self, x):
        scale = self.weight * self.running_var.rsqrt()
        bias = self.bias - self.running_mean * scale
        scale = scale.reshape(1, -1, 1, 1)
        bias = bias.reshape(1, -1, 1, 1)
        print(scale.shape,bias.shape)
        return x * scale + bias

a=FrozenBatchNorm2d((1,2))
a(torch.tensor([1,2,3]))

The running result is different from what I thought.
So can someone tell me what this function exactly does?
I will appreciate it if someone could help me.


title: HTTPS POST to query FastAPI using python requests

body:
I am trying to serve a Neural Network using FastAPI.
from fastapi import Depends, FastAPI
from pydantic import BaseModel
from typing import Dict

class iRequest(BaseModel):
    arg1: str
    arg2: str

class iResponse(BaseModel):
    pred: str
    probs: Dict[str, float]

@app.post("/predict", response_model=iResponse)
def predict(request: iRequest, model: Model = Depends(get_model)):
    pred, probs = model.predict(request.arg1, request.arg2)
    return iResponse(pred = pred, probs = probs)

The manual site http://localhost:8000/docs#/default/predict_predict_post works fine and translates into the following curl command:
curl -X POST "http://localhost:8000/predict" -H  "accept: application/json" -H  "Content-Type: application/json" -d "{\"arg1\":\"I am the King\",\"arg2\":\"You are not my King\"}"

which also works.
When I try to query the API using python requests:
import requests
data = {"arg1": "I am the King",
        "arg2": "You are not my King"}
r = requests.post("http://localhost:8000/predict", data=data)

I only get the "422 Unprocessable Entity" Errors. Where am I going wrong here?


title: Need help understanding the label input in a CGAN

body:
I am trying to implement a CGAN. I understand that in convolutional generator and discriminator models, you add volume to the inputs by adding depth that represents the label. So if you have 10 classes in your data, your generator and discriminator would both have the base depth + 10 as its input volume.

However, I am reading various implementations online and I can't seem to find where they are actually acquiring this label. Surely CGANs can't be unsupervised because you need to obtain the label to input. e.g. in cifar10 if you are training the discriminator on a real image of a frog, you would need the 'frog' annotation.

Here is one of the pieces of code I am studying:

class CGAN(object):
def __init__(self, args):
    # parameters
    self.epoch = args.epoch
    self.batch_size = args.batch_size
    self.save_dir = args.save_dir
    self.result_dir = args.result_dir
    self.dataset = args.dataset
    self.log_dir = args.log_dir
    self.gpu_mode = args.gpu_mode
    self.model_name = args.gan_type
    self.input_size = args.input_size
    self.z_dim = 62
    self.class_num = 10
    self.sample_num = self.class_num ** 2

    # load dataset
    self.data_loader = dataloader(self.dataset, self.input_size, self.batch_size)
    data = self.data_loader.__iter__().__next__()[0]

    # networks init
    self.G = generator(input_dim=self.z_dim, output_dim=data.shape[1], input_size=self.input_size, class_num=self.class_num)
    self.D = discriminator(input_dim=data.shape[1], output_dim=1, input_size=self.input_size, class_num=self.class_num)
    self.G_optimizer = optim.Adam(self.G.parameters(), lr=args.lrG, betas=(args.beta1, args.beta2))
    self.D_optimizer = optim.Adam(self.D.parameters(), lr=args.lrD, betas=(args.beta1, args.beta2))

    if self.gpu_mode:
        self.G.cuda()
        self.D.cuda()
        self.BCE_loss = nn.BCELoss().cuda()
    else:
        self.BCE_loss = nn.BCELoss()

    print('---------- Networks architecture -------------')
    utils.print_network(self.G)
    utils.print_network(self.D)
    print('-----------------------------------------------')

    # fixed noise & condition
    self.sample_z_ = torch.zeros((self.sample_num, self.z_dim))
    for i in range(self.class_num):
        self.sample_z_[i*self.class_num] = torch.rand(1, self.z_dim)
        for j in range(1, self.class_num):
            self.sample_z_[i*self.class_num + j] = self.sample_z_[i*self.class_num]

    temp = torch.zeros((self.class_num, 1))
    for i in range(self.class_num):
        temp[i, 0] = i

    temp_y = torch.zeros((self.sample_num, 1))
    for i in range(self.class_num):
        temp_y[i*self.class_num: (i+1)*self.class_num] = temp

    self.sample_y_ = torch.zeros((self.sample_num, self.class_num)).scatter_(1, temp_y.type(torch.LongTensor), 1)
    if self.gpu_mode:
        self.sample_z_, self.sample_y_ = self.sample_z_.cuda(), self.sample_y_.cuda()

def train(self):
    self.train_hist = {}
    self.train_hist['D_loss'] = []
    self.train_hist['G_loss'] = []
    self.train_hist['per_epoch_time'] = []
    self.train_hist['total_time'] = []

    self.y_real_, self.y_fake_ = torch.ones(self.batch_size, 1), torch.zeros(self.batch_size, 1)
    if self.gpu_mode:
        self.y_real_, self.y_fake_ = self.y_real_.cuda(), self.y_fake_.cuda()

    self.D.train()
    print('training start!!')
    start_time = time.time()
    for epoch in range(self.epoch):
        self.G.train()
        epoch_start_time = time.time()
        for iter, (x_, y_) in enumerate(self.data_loader):
            if iter == self.data_loader.dataset.__len__() // self.batch_size:
                break

            z_ = torch.rand((self.batch_size, self.z_dim))
            y_vec_ = torch.zeros((self.batch_size, self.class_num)).scatter_(1, y_.type(torch.LongTensor).unsqueeze(1), 1)
            y_fill_ = y_vec_.unsqueeze(2).unsqueeze(3).expand(self.batch_size, self.class_num, self.input_size, self.input_size)
            if self.gpu_mode:
                x_, z_, y_vec_, y_fill_ = x_.cuda(), z_.cuda(), y_vec_.cuda(), y_fill_.cuda()

            # update D network
            self.D_optimizer.zero_grad()

            D_real = self.D(x_, y_fill_)
            D_real_loss = self.BCE_loss(D_real, self.y_real_)

            G_ = self.G(z_, y_vec_)
            D_fake = self.D(G_, y_fill_)
            D_fake_loss = self.BCE_loss(D_fake, self.y_fake_)

            D_loss = D_real_loss + D_fake_loss
            self.train_hist['D_loss'].append(D_loss.item())

            D_loss.backward()
            self.D_optimizer.step()

            # update G network
            self.G_optimizer.zero_grad()

            G_ = self.G(z_, y_vec_)
            D_fake = self.D(G_, y_fill_)
            G_loss = self.BCE_loss(D_fake, self.y_real_)
            self.train_hist['G_loss'].append(G_loss.item())

            G_loss.backward()
            self.G_optimizer.step()

It seems as if y_vec_ and y_fill_ are the labels for the images, but in the instance of y_fill_ which is used to label real images for the discriminator, it equals y_fill_ = y_vec_.unsqueeze(2).unsqueeze(3).expand(self.batch_size, self.class_num, self.input_size, self.input_size)

It doesn't seem like its getting any information on the label from the dataset?
How is it giving the discriminator the correct label?

Thanks!


title: ValueError: operands could not be broadcast together with shapes (50,50,512) (3,) (50,50,512) while converting tensor to image in pytorch

body:
I'm doing a neural style transfer. I'm trying to reconstruct the output of the convolutional layer conv4_2 of the VGG19 network.

def get_features(image, model):

    layers = {'0': 'conv1_1', '5': 'conv2_1',  '10': 'conv3_1',
              '19': 'conv4_1', '21': 'conv4_2', '28': 'conv5_1'}
    x = image
    features = {}

    for name, layer in model._modules.items():
        x = layer(x)

        if name in layers:
            features[layers[name]] = x

    return features

content_img_features = get_features(content_img, vgg)
style_img_features   = get_features(style_img, vgg)

target_content = content_img_features['conv4_2']

content_img_features is a dict that contains the output of every layer.
target_content is a tensor of shape torch.Size([1, 512, 50, 50])

This is the method I use to plot the image using the tensor. It works fine for the input image as well as the final output.

def tensor_to_image(tensor):
    image = tensor.clone().detach()
    image = image.numpy().squeeze()
    image = image.transpose(1, 2, 0)
    image *= np.array((0.22, 0.22, 0.22))+ np.array((0.44, 0.44, 0.44))
    image = image.clip(0, 1)
    return image

image = tensor_to_image(target_content)
fig = plt.figure()
plt.imshow(image)

But this throws the error,

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-188-a75a5f0743bb> in <module>()
      1
----> 2 image = tensor_to_image(target_content)
      3 fig = plt.figure()
      4 plt.imshow(image)

<ipython-input-186-e9385dbc4a85> in tensor_to_image(tensor)
      3     image = image.numpy().squeeze()
      4     image = image.transpose(1, 2, 0)
----> 5     image *= np.array((0.22, 0.22, 0.22))+ np.array((0.44, 0.44, 0.44))
      6     image = image.clip(0, 1)
      7     return image

ValueError: operands could not be broadcast together with shapes (50,50,512) (3,) (50,50,512)

This is the initial transformation I apply to the image before passing to the cnn layers,

def transformation(img):

    tasks = tf.Compose([tf.Resize(400), tf.ToTensor(),
            tf.Normalize((0.44,0.44,0.44),(0.22,0.22,0.22))])
    img = tasks(img)[:3,:,:].unsqueeze(0)

    return img

How do I fix this? Is there another way to reconstruct the image from the convolution layer?


title: View on portion of tensor

body:
I have a multiple dimensions tensor, let's take this simple one as example:

out = torch.Tensor(3, 4, 5)

I have to get a portion/subpart of this tensor out[:,0,:] and then apply the method view(-1), but it's not possible:

 out[:,0,:].view(-1)

RuntimeError: invalid argument 2: view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Call .contiguous() before .view(). at ../aten/src/TH/generic/THTensor.cpp:203

A solution is to clone the subpart:

 out[:,0,:].clone().view(-1)

Is there a better/faster solution than cloning?


title: How to get embedding from bert finetuned model?

body:
I have finedtuned 'bert-base-uncased' model using transformer and torch which gave me pytorch_model.bin, vocab.txt and other files as output.
After loading the model how to I get embedding for complete vocab, like a matrix which maps every word to its embedding vector


title: Loss function negative log likelihood giving loss despite perfect accuracy

body:
I am debugging a sequence-to-sequence model and purposely tried to perfectly overfit a small dataset of ~200 samples (sentence pairs of length between 5-50). I am using negative log-likelihood loss in pytorch. I get low loss (~1e^-5), but the accuracy on the same dataset is only 33%.
I trained the model on 3 samples as well and obtained 100% accuracy, yet during training I had loss. I was under the impression that negative log-likelihood only gives loss (loss is in the same region of ~1e^-5) if there is a mismatch between predicted and target label?
Is a bug in my code likely?


title: Calculating determinants via Cholesky decomposition in PyTorch

body:
I've been trying to calculate the determinant of a 2x2 matrix via Cholesky decomposition in PyTorch and it won't give the same number as Numpy and I'm not sure why. From my understanding, you can calculate the determinant of a square positive-definite matrix via decomposing it into a lower triangular matrix and its transpose, i.e. M = LL^T.
Then by the law of determinants, the determinant of M is equal to the determinant of L multiplied by the determinant of L^T. Which, in the case of lower triangular matrices, is just the product of the diagonal. So, M would be equal to product of the diagonal of L multiplied by the product of the diagonal of L^T.
However, when I implement this in PyTorch, I get the wrong value. I've copied an example code below.
import torch
import numpy as np

matrix = torch.Tensor(2,2).uniform_()
print("Matrix: \n", matrix.detach().numpy(), "\n")

print("Positive-definite?: ", np.all(np.linalg.eigvals(matrix.detach().numpy()) > 0))
det_np = np.linalg.det(matrix.detach().numpy())

det_tor = torch.cholesky(matrix, upper=False).diag().prod()**2

print("determinant (numpy) %8.4f" % (det_np))
print("determinant (torch) %8.4f" % (det_tor))

An example output would be something like this,
Matrix:
 [[0.5305128  0.2795679 ]
 [0.41778737 0.40350497]]

Positive-definite?:  True
determinant (numpy)   0.0973
determinant (torch)   0.0395

What is it that is wrong? Why is there a difference between these two methods?


title: torch.save() gives : RuntimeError: CUDA error: no CUDA-capable device is detected

body:
I was training a neural network model on GPU but I get the above mentioned error when I use torch.save() to save checkpoints. My question is even though I have a CUDA device why am I getting the mentioned error? My model was running okay on the GPU please see bellow the Output of: nvidia-smi command.
$ nvidia-smi
Sat Aug 15 09:51:58 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 2060    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   55C    P3    33W /  N/A |   4774MiB /  5934MiB |     97%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      7080      C   python3                                     4763MiB |
+-----------------------------------------------------------------------------+

$ python --version
Python 3.8.2

$ python -c "import torch; print(torch.__version__)"
1.5.1

$ python -c "import torchvision as torch; print(torch.__version__)"
0.6.1

I have even tried the following:

os.environ["CUDA_VISIBLE_DEVICES"] = '0'

torch.save({
                'epoch': epoch + 1,
                'metrics': metrics,
                'model': model.state_dict(),
                'optimizer' : optimizer.state_dict(),
            }, name)

But nothing worked. I am new to deep learning and still learning PyTorch. Please do pardon my ignorance.


title: Perform Delta Function between elements in PyTorch tensor

body:
I have a 1 dimensional pyTorch tensor (dtype: int32) and was wondering if there was a way to perform a Dirac Delta function on the elements in this tensor, i.e:
f = tensor[1, 0, 0, 0, 1, 1, 0, 1, 1, 0, 1, 1]
f_after_dirac_delta = tensor[0, 1, 1, 0, 1, 0, 0, 1, 0, 0, 1]

Thanks for any help in advance!
EDIT: as @GirishDattatrayHegde mentioned, the term Dirac-Delta was misleading. The correct term should have been a Kronecker-Delta. My apologies.


title: How can we check if a matrix is PSD is PyTorch?

body: There is a poste on checking if a matrix is PSD in Python. I am wondering how we can check it in PyTorch? is there a function for that?


title: How can I unroll a PyTorch Tensor?

body:
I have a tensor:
t1 = torch.randn(564, 400)

I want to unroll it to a 1-d tensor that's 225600 long.
How can I do this?


title: Problem with forward step in pytorch model

body:
Traceback saying that:
* Epoch 1/20
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-24-4f0f868c6227> in <module>()
      1 max_epochs = 20
      2 optim = torch.optim.Adam(model.parameters(), lr=1e-3)
----> 3 train(model, optim, bce_loss, max_epochs, data_tr, data_val)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

TypeError: forward() takes 2 positional arguments but 3 were given

But in train section of my net I did not pass more than two arguments(including self argument)
this is link to my code
Maybe problem not in train


title: PyTorch mini batch, when to call optimizer.zero_grad()

body:
When we use mini batch, should I call optimizer.zero_grad() before starting the iteration? Or inside the iteration? I think the second code is correct, but I'm not sure.
nb_epochs = 20
    for epoch in range(nb_epochs + 1):
      optimizer.zero_grad() # THIS PART!!
      for batch_idx, samples in enumerate(dataloader):

        x_train, y_train = samples

        prediction = model(x_train)

        cost = F.mse_loss(prediction, y_train)

        cost.backward()
        optimizer.step()

        print('Epoch {:4d}/{} Batch {}/{} Cost: {:.6f}'.format(
            epoch, nb_epochs, batch_idx+1, len(dataloader),
            cost.item()
            ))

or
nb_epochs = 20
    for epoch in range(nb_epochs + 1):

      for batch_idx, samples in enumerate(dataloader):

        x_train, y_train = samples

        prediction = model(x_train)

        optimizer.zero_grad() #THIS PART!!
        cost = F.mse_loss(prediction, y_train)

        cost.backward()
        optimizer.step()

        print('Epoch {:4d}/{} Batch {}/{} Cost: {:.6f}'.format(
            epoch, nb_epochs, batch_idx+1, len(dataloader),
            cost.item()
            ))

Which one is correct? The only difference is location of optimizer.zero_grad().


title: Cannot import "BasicBlock" from torchvision.models.resnet

body:
I am trying to import the class BasicBlock from torchvision.models.resnet by doing this
from torchvision.models.resnet import *

It is giving no error but when I am trying to use BasicBlock class in my code (which is supposed to be imported already), I am getting the error that
NameError: name 'BasicBlock' is not defined
even though BasicBlock is present in torchvision.models.resnet
But it gives no error when I import like this
from torchvision.models.resnet import BasicBlock
and then use it in my code
Why am I getting this error?


title: Huggingface BERT NER Example Batch_Size error

body:
I am trying to implement the NER example using BERT and pytorch from the huggingface guide (https://huggingface.co/transformers/custom_datasets.html#ft-trainer). Reading in the data works fine, but when I want to start the training, I get the error
Expected input batch_size (16) to match target batch_size (4000)

When I try to change the input batch_size, the target batch_size is always the input batch_size*250. I would greatly appreciate, if someone could look over the following code and sees my mistakes.
from pathlib import Path
import re

def read_data(file_path):
    file_path = Path(file_path)

    raw_text = file_path.read_text().strip()
    raw_docs = re.split(r'\n\t?\n', raw_text)
    token_docs = []
    tag_docs = []
    for doc in raw_docs:
        tokens = []
        tags = []
        for line in doc.split('\n'):
            token, tag = line.split('\t')
            tokens.append(token)
            tags.append(tag)
        token_docs.append(tokens)
        tag_docs.append(tags)

    return token_docs, tag_docs

train_texts, train_tags = read_data('data/train.tsv')
val_texts, val_tags = read_data('data/test.tsv')

unique_tags = set(tag for doc in tags for tag in doc)
tag2id = {tag: id for id, tag in enumerate(unique_tags)}
id2tag = {id: tag for tag, id in tag2id.items()}

from transformers import DistilBertTokenizerFast
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-cased')
train_encodings = tokenizer(train_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)
val_encodings = tokenizer(val_texts, is_split_into_words=True, return_offsets_mapping=True, padding=True, truncation=True)

import numpy as np

def encode_tags(tags, encodings):
    labels = [[tag2id[tag] for tag in doc] for doc in tags]
    encoded_labels = []
    for doc_labels, doc_offset in zip(labels, encodings.offset_mapping):
        # create an empty array of -100
        doc_enc_labels = np.ones(len(doc_offset),dtype=int) * -100
        arr_offset = np.array(doc_offset)

        # set labels whose first offset position is 0 and the second is not 0
        doc_enc_labels[(arr_offset[:,0] == 0) & (arr_offset[:,1] != 0)] = doc_labels
        encoded_labels.append(doc_enc_labels.tolist())

    return encoded_labels

train_labels = encode_tags(train_tags, train_encodings)
val_labels = encode_tags(val_tags, val_encodings)

import torch

class TestDataset(torch.utils.data.Dataset):
    def __init__(self, encodings, labels):
        self.encodings = encodings
        self.labels = labels

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        item['labels'] = torch.tensor(self.labels[idx])
        return item

    def __len__(self):
        return len(self.labels)

train_encodings.pop("offset_mapping") # we don't want to pass this to the model
val_encodings.pop("offset_mapping")
train_dataset = TestDataset(train_encodings, train_labels)
val_dataset = TestDataset(val_encodings, val_labels)

from transformers import DistilBertForTokenClassification
model = DistilBertForTokenClassification.from_pretrained('distilbert-base-cased', num_labels=len(unique_tags))

from transformers import DistilBertForSequenceClassification, Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir='./results',          # output directory
    num_train_epochs=3,              # total number of training epochs
    per_device_train_batch_size=16,  # batch size per device during training
    per_device_eval_batch_size=64,   # batch size for evaluation
    warmup_steps=500,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='./logs',            # directory for storing logs
    logging_steps=10,
)

model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-cased")

trainer = Trainer(
    model=model,                         # the instantiated  Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_dataset,         # training dataset
    eval_dataset=val_dataset             # evaluation dataset
)

trainer.train()


title: Pytorch crop images from Top Left Corner in Transforms

body:
I'm using Pytorch's transforms.Compose and in my dataset I have 1200x1600 (Height x Width) images.
I want to crop the images starting from the Top Left Corner (0,0) so that I can have 800x800 images.
I was looking in Pytorch documentation but I didn't find anything to solve my problem, so I copied the source code of center_crop in my project and modified it as follows:
def center_crop(img: Tensor, output_size: List[int]):
    # .... Other stuff of Pytorch

    # ....
    # Original Pytorch Code (that I commented)
    crop_top = int((image_height - crop_height + 1) * 0.5)
    crop_left = int((image_width - crop_width + 1) * 0.5)

    # ----
    # My modifications:
    crop_top = crop_left = 0

    return crop(img, crop_top, crop_left, crop_height, crop_width)

But basically I think this is quite an overkill, if it's possible I'd like to avoid to copy their code and modify it.
Isn't there anything that already implements the desired behaviour by default, is there?


title: How to create own dataset in c++ pyTorch

body:
I'm try to do some machine learning with pyTorch C++. I got own dataset in .csv files and after loading them I receive eg. [N,M] 2D std::vector with N samples and M features and [N,] 1D std::vector with N labels.

My question is, what should I do to create proper dataset to use torch::data::make_data_loader method (or other) and learn my NN.

Thanks in advance!


title: Is concatenated matrix multiplication faster than multiple non-concatenated matmul? If so, why?

body:
The definition of the LSTM cell involves 4 matrix multiplications with the input, and 4 matrix multiplications with the output. We can simplify the expression by using a single matrix multiply by concatenating 4 small matrices (now the matrix are 4 times larger).

My question is: does this improve the efficiency of the matrix multiplication? If so, why? Because we can put them in continuous memory? Or is it because of the conciseness of the code?

The number of items that we multipy doesn't change whether or not we concatenate the matrices. (Therefore complexity shouldn't change.) So I'm wondering why we would do this..

Here is an excerpt from pytorch doc of torch.nn.LSTM(*args, **kwargs). W_ii, W_if, W_ig, W_io are concatenated.

weight_ih_l[k] – the learnable input-hidden weights of the \text{k}^{th}k
th
  layer (W_ii|W_if|W_ig|W_io), of shape (4*hidden_size x input_size)

weight_hh_l[k] – the learnable hidden-hidden weights of the \text{k}^{th}k
th
  layer (W_hi|W_hf|W_hg|W_ho), of shape (4*hidden_size x hidden_size)

bias_ih_l[k] – the learnable input-hidden bias of the \text{k}^{th}k
th
  layer (b_ii|b_if|b_ig|b_io), of shape (4*hidden_size)

bias_hh_l[k] – the learnable hidden-hidden bias of the \text{k}^{th}k
th
  layer (b_hi|b_hf|b_hg|b_ho), of shape (4*hidden_size)


title: Why do we need padding in seq2seq network

body:
To handle sequences of different, I would like to know.

Why do we need padding the sequence the word to the same length?
If the answer is "Yes, you need padding.". Can I set the padding in other indexes? For example, if I have an index word like this:

{0:"<s>,1:"<e>",2:"AAA",3:"BBB",.......,500:"zzz"}

Where <s> is starting word of the sentence and  is the ending word of the sentence.

Can I set the padding flag to the last index?

{0:"<s>,1:"<e>",2:"AAA",3:"BBB",.......,500:"zzz",501:"<pad>"}


title: Libtorch: cannot load traced lstm scriptmodel

body:
I save a pytorch ScriptModule and load it using libtorch. However I encountered the following problem

I use linux subsystem under win10 and I use pytorch 1.2.

To reproduce my problem, you could run this piece of python code to save a pt model

import torch
import torch.nn as nn

# TODO: https://github.com/pytorch/pytorch/issues/23930
class test(torch.jit.ScriptModule):

    def __init__(self, vocab_size=10, rnn_dims=512):
        super().__init__()
        self.word_embeds = nn.Embedding(vocab_size, rnn_dims)
        self.emb_drop = nn.Dropout(0.1)
        self.rnn = nn.LSTM(input_size=rnn_dims, hidden_size=rnn_dims, batch_first=True,
                           num_layers=2, dropout=0.1)
        # delattr(self.rnn, 'forward_packed')

    @torch.jit.script_method
    def forward(self, x):
        h1 = (torch.zeros(2, 1, 512), torch.zeros(2, 1, 512))
        embeds = self.emb_drop(self.word_embeds(x))
        out, h1 = self.rnn(embeds, h1)

        return h1

model = test()

input = torch.ones((1,3)).long()
output = model(input)
print('output', output)

# torch.onnx.export(model,  # model being run
#                   input,
#                   'test.onnx',
#                   example_outputs=output)
#torch.jit.trace(model, (torch.ones((1,3)).long(), torch.ones((3,1))), check_trace=False)
model.save('lstm_test.pt')

Then load the model in libtorch.

I don't know why I have this error. I don't use PackedSequence at all. Hope some could help me out.


title: Indexing Pytorch tensor

body:
I have a Pytorch code which generates a Pytorch tensor in each iteration of for loop, all of the same size. I want to assign each of those tensors to a row of new tensor, which will include all the tensors at the end. In other works something like this

for i=1:N:
  X = torch.Tensor([[1,2,3], [3,2,5]])
  #Y is a pytorch tensor
  Y[i] = X

I wonder how I can implement this with Pytorch.


title: pytorch Crossentropy results in unmatched batch size

body:
I was loading a image folder using dataloader

The image folder consists of three categories(labels) ie

'/root/ant/dsd.png'

'/root/ant/sfds.png'

...

....

'/root/bee/dsf.png'

....

..

'/root/whey/sfd.png'

Here there are three classes ant,bee,whey

By executing the above code i getting an error of unmatched bacth size of output and target at criterion

Error:Expected input batch_size (3) to match target batch_size (1).

I thought that the error might be in trainloader because extracting labels of different unmatched shape

data_transform = transforms.Compose([
        transforms.Resize(size=28),
        transforms.ToTensor()
    ])
     kumda_dataset = datasets.ImageFolder(root='/content/gdrive/My Drive/Colab Notebooks/images',
                                           transform=data_transform)
#train & test
train_size = int(0.8 * len(kumda_dataset))
test_size = len(kumda_dataset) - train_size

#splitting
train_dataset, test_dataset = torch.utils.data.random_split(kumda_dataset, [train_size, test_size])

trainloader = torch.utils.data.DataLoader(train_dataset , batch_size = 1, shuffle = True)
testloader = torch.utils.data.DataLoader(test_dataset , batch_size = 4, shuffle = False )

model=nn.Linear(784,1)
criterion=nn.CrossEntropyLoss()
optimizer=optim.SGD(model.parameters(),lr=0.01)

num=10
for epoch in range(num):
  for i,(images,labels) in enumerate(trainloader):
    images=images.reshape(-1,28*28)
    output=model(images)
    loss=criterion(output,labels)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    if(i+1%70==0):
        print("Epoch: {}/{}, \tIteration: {}/{}, \tLoss: {}".format(epoch + 1, num, i + 1,len(dataset_loader), loss.item()))

Thank you in advance for solving


title: bias dimension definition in coding neural network

body:
In the following Figure showing the code for defining the dimension of the bias b1 term, I wonder why the first dimension of bias b1 is not the batch size? Does it mean then it just assumes this bias is applied to all batches then?

If I specify the bias b1 dimension to be (batch_size, 256) then does it mean i am applying a different b1 to different batch? But theoretically it should still work right? Also what is the difference between tensor (256), (256,) and (256,1)...?

Figure: dimension definition of nn


title: Pytorch: List of Coordinates to Bounded Shape

body:
Suppose I have a PyTorch tensor that represents a list of 1000 coordinates, where indices 0-2 are independent x-y-z co-ordinates and index 3 is a value associated with the coordinate.

I want to produce a 'graph' of those co-ordinates: an I x J x K grid, with the value from the coordinates at each index.

For example:

batch_size=10
channels = 2
n_coords = 1000

len_coords = 4
low = 0
high=10
some_limit = 5

coords = torch.zeros(batch_size, channels, n_coords, len_coords).random_(low, high) # a set of coordinates
#SOME OPERATION YIELDS ->
image = torch.zeros(batch_size, channels, high, high, high) #The 'space' I want to embed the coo-ordinates in

I have a solution involving loops that I have developed that produces the desired result:

coords_shape = torch.Tensor([*data.shape][0:2]).int()
axis_shape = torch.Tensor([high,high,high]).int()
image_shape = torch.cat((coords_shape, axis_shape), axis=0)
image = torch.zeros(*image_shape)

for b in range(coords.shape[0]):
    for c in range(coords.shape[1]):
        for v in range(coords.shape[2]):
                elem = coords[b, c, v]
                if(elem[3]>some_limit):
                    image[batch_elem, channel, int(elem[0]), int(elem[1]), int(elem[2])]=elem[3]

I know this is suboptimal because of the loops.

Is there a smarter way of implementing this?


title: if statement within __init__ python class is not working

body:
This is maybe a really dumb question, but I am pretty frustrated, so bear with me. I am writing a class to perform neural net analysis using pytorch, and I have some "if" statements within the init definition.
class RBM(nn.Module):
    '''
    This class defines all the functions needed for an BinaryRBN model
    where the visible and hidden units are both considered binary
    '''

    def __init__(self,
                visible_units=256,
                hidden_units = 64,
                k=2,
                learning_rate=1e-5,
                learning_rate_decay = False,
                xavier_init = False,
                increase_to_cd_k = False,
                use_gpu = False
                ):

        super(RBM,self).__init__()
        self.desc = "RBM"

        self.visible_units = visible_units
        self.hidden_units = hidden_units
        self.k = k
        self.learning_rate = learning_rate
        self.learning_rate_decay = learning_rate_decay
        self.xavier_init = xavier_init
        self.increase_to_cd_k = increase_to_cd_k
        self.use_gpu = use_gpu
        self.batch_size = 16

        # Initialization
        if not self.xavier_init:
            print("I am here")
            self.W = torch.randn(self.visible_units,self.hidden_units) * 0.01 #weights
        else:
            self.xavier_value = torch.sqrt(torch.FloatTensor([1.0 /
                                                              (self.visible_units + self.hidden_units)]))
            self.W = -self.xavier_value + torch.rand(self.visible_units,
                                                     self.hidden_units) * (2 * self.xavier_value)
        self.h_bias = torch.zeros(self.hidden_units) #hidden layer bias
        self.v_bias = torch.zeros(self.visible_units) #visible layer bias

        if self.use_gpu:
            self.W = self.W.cuda()
            self.v_bias = self.v_bias.cuda()
            self.h_bias = self.h_bias.cuda()
            if self.xavier_init:
                self.xavier_value = self.xavier_value.cuda()

This class fails whenever I specify use_gpu=True, but is fine if I specify use_gpu=False. For example:
 rbm = RBM(use_gpu=1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/Deep-Belief-Network-pytorch-master/RBM.py", line 64, in __init__
    self.v_bias = self.v_bias.cuda()
  File "/home/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'RBM' object has no attribute 'xavier_value'

So the specific error I'm getting has to do with the variable self.xavier_value, but this variable should never get called given the if self.xavier_init call. What am I doing wrong here?


title: Getting an error "UnpicklingError: invalid load key, 'v'." in Pytorch model deploying in Streamlit

body:
I am trying to deploy my wheat detection model of Pytorch in Streamlit.
But, I am getting an error as
UnpicklingError: invalid load key, 'v'.
Traceback:
File "/usr/local/lib/python3.7/site-packages/streamlit/script_runner.py", line 332, in _run_script
    exec(code, module.__dict__)
File "/app/streamlit/streamlit.py", line 84, in <module>
    model.load_state_dict(torch.load(weights_file, map_location=device))
File "/home/appuser/.local/lib/python3.7/site-packages/torch/serialization.py", line 529, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
File "/home/appuser/.local/lib/python3.7/site-packages/torch/serialization.py", line 692, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)

My Github repo is in https://github.com/Anubhav1107/streamlit
PLs write what you want. Thanks for helping !


I am trying to extract features from an intermediate of ResNet18 in PyTorch using forward hooks
class CCLModel(nn.Module):
    def __init__(self,output_layer,*args):
        self.output_layer = output_layer
        super().__init__(*args)

        self.output_layer = output_layer
        #PRETRAINED MODEL
        self.pretrained = models.resnet18(pretrained=True)

        #TAKING OUTPUT FROM AN INTERMEDIATE LAYER

        #self._layers = []
        for l in list(self.pretrained._modules.keys()):
            #self._layers.append(l)
            if l == self.output_layer:
                handle = getattr(self.pretrained,l).register_forward_hook(self.hook)

    def hook(self,input,output):
        return output

    def _forward_impl(self, x):
        x = self.pretrained(x)
        return x

    def forward(self, x):
        return self._forward_impl(x)

I also want the predictions alongside the feature outputs from layer 4
But I am getting the TypeError: hook() takes 3 positional arguments but 4 were given
The full error message is this
TypeError                                 Traceback (most recent call last)
<ipython-input-66-18c4a0f917f2> in <module>()
----> 1 out = model(x.to('cuda:0').float())

6 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-61-71fe0d1420a6> in forward(self, x)
     78
     79     def forward(self, x):
---> 80         return self._forward_impl(x)
     81
     82     '''def forward(self,x):

<ipython-input-61-71fe0d1420a6> in _forward_impl(self, x)
     73         #x = torch.flatten(x, 1)
     74         #x = self.fc(x)
---> 75         x = self.pretrained(x)
     76
     77         return x

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torchvision/models/resnet.py in forward(self, x)
    218
    219     def forward(self, x):
--> 220         return self._forward_impl(x)
    221
    222

/usr/local/lib/python3.6/dist-packages/torchvision/models/resnet.py in _forward_impl(self, x)
    209         x = self.layer2(x)
    210         x = self.layer3(x)
--> 211         x = self.layer4(x)
    212
    213         x = self.avgpool(x)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    729                 _global_forward_hooks.values(),
    730                 self._forward_hooks.values()):
--> 731             hook_result = hook(self, input, result)
    732             if hook_result is not None:
    733                 result = hook_result

TypeError: hook() takes 3 positional arguments but 4 were given

Why is the hook not working, although in various forums I see that this is the way to do it?


title: test accuracy of vgg is too low(cifar-100)

body:
I'm new with deep learning and I was making VGG 16(cifar-100) with pytorch.
I thought every thing was fine but when I test it, the test loss(accuracy) is very bad.
red=test, blue=train, left=loss, right=acc
I think this is overfit problem, so I searched to solve it.
I tried changing epochs, batch size, learning rate, weight decay, image preprocessing, and adding dropout, batch normalization.
Also tried to change the architecture of VGG.
But almost nothing improved the performance(rather the performance was worse)
My code is below and is there any solution to increase the performance? Thank you
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms

import matplotlib.pyplot as plt

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5071, 0.4867, 0.4408), (0.2675, 0.2565, 0.2761))])

train = torchvision.datasets.CIFAR100(root='CIFAR-100/',
                                      train=True, transform=transform,
                                      download=True)
test = torchvision.datasets.CIFAR100(root='CIFAR-100/',
                                     train=False, transform=transform,
                                     download=True)

train_loader = torch.utils.data.DataLoader(dataset=train, batch_size=256, shuffle=True)
test_loader = torch.utils.data.DataLoader(dataset=test, batch_size=256, shuffle=True)

cuda = torch.device('cuda')

class VGG(nn.Module):
    def __init__(self):
        super(VGG, self).__init__()
        # 16
        self.layer_1 = nn.Sequential(nn.Conv2d(in_channels=3, out_channels=64, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(64),
                                       nn.ReLU(),
                                       nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(64),
                                       nn.ReLU(),
                                       nn.MaxPool2d(kernel_size=2, stride=2)
                                       )
        self.layer_2 = nn.Sequential(nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(128),
                                       nn.ReLU(),
                                       nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(128),
                                       nn.ReLU(),
                                       nn.MaxPool2d(kernel_size=2, stride=2)
                                       )
        # 8
        self.layer_3 = nn.Sequential(nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(256),
                                       nn.ReLU(),
                                       nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(256),
                                       nn.ReLU(),
                                       nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(256),
                                       nn.ReLU(),
                                       nn.MaxPool2d(kernel_size=2, stride=2)
                                       )
        self.layer_4 = nn.Sequential(nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(512),
                                       nn.ReLU(),
                                       nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(512),
                                       nn.ReLU(),
                                       nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(512),
                                       nn.ReLU(),
                                       nn.MaxPool2d(kernel_size=2, stride=2)
                                       )
        # 4
        self.layer_5 = nn.Sequential(nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(512),
                                       nn.ReLU(),
                                       nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(512),
                                       nn.ReLU(),
                                       nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3,
                                                 stride=1, padding=1),
                                       nn.BatchNorm2d(512),
                                       nn.ReLU(),
                                     nn.MaxPool2d(kernel_size=2, stride=2)
                                       )
        self.layer_6 = nn.Sequential(nn.Flatten(),
                                     nn.Linear(512, 100, bias=True)
                                     )

    def forward(self, x):
        x = self.layer_1(x)
        x = self.layer_2(x)
        x = self.layer_3(x)
        x = self.layer_4(x)
        x = self.layer_5(x)
        x = self.layer_6(x)
        return x

model = VGG()
model = model.cuda()

loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.00001, weight_decay=5e-4)
cost = 0

iterations = []
train_losses = []
test_losses = []
train_acc = []
test_acc = []

for epoch in range(100):
    model.train()
    correct = 0
    for X, Y in train_loader:
        X = X.to(cuda)
        Y = Y.to(cuda)
        optimizer.zero_grad()
        hypo = model(X)
        cost = loss(hypo, Y)
        cost.backward()
        optimizer.step()
        prediction = hypo.data.max(1)[1]
        correct += prediction.eq(Y.data).sum()

    model.eval()
    correct2 = 0
    for data, target in test_loader:
        data = data.to(cuda)
        target = target.to(cuda)
        output = model(data)
        cost2 = loss(output, target)
        prediction = output.data.max(1)[1]
        correct2 += prediction.eq(target.data).sum()

    print("Epoch : {:>4} / cost : {:>.9}".format(epoch + 1, cost))
    iterations.append(epoch)
    train_losses.append(cost.tolist())
    test_losses.append(cost2.tolist())
    train_acc.append((100*correct/len(train_loader.dataset)).tolist())
    test_acc.append((100*correct2/len(test_loader.dataset)).tolist())

# del train_loader
# torch.cuda.empty_cache()

model.eval()
correct = 0
for data, target in test_loader:
    data = data.to(cuda)
    target = target.to(cuda)
    output = model(data)
    prediction = output.data.max(1)[1]
    correct += prediction.eq(target.data).sum()

print('Test set: Accuracy: {:.2f}%'.format(100. * correct / len(test_loader.dataset)))

plt.subplot(121)
plt.plot(range(1, len(iterations)+1), train_losses, 'b--')
plt.plot(range(1, len(iterations)+1), test_losses, 'r--')
plt.subplot(122)
plt.plot(range(1, len(iterations)+1), train_acc, 'b-')
plt.plot(range(1, len(iterations)+1), test_acc, 'r-')
plt.title('loss and accuracy')
plt.show()


title: Rewriting PyTorch model as Keras model and transferring the weights with incorrect shape?

body:
I have a PyTorch model that I am attempting to rewrite in Keras which I have minimal experience with. This is the PyTorch model
class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.hidden_x1 = nn.Sequential(
            nn.Conv1d(in_channels=2, out_channels=8, kernel_size=7),
            nn.SELU(),
            nn.Conv1d(in_channels=8, out_channels=6, kernel_size=7),
            nn.SELU(),
            nn.Conv1d(in_channels=6, out_channels=4, kernel_size=5),
            nn.SELU(),
        )
        self.hidden_xfc = nn.Sequential(
            nn.Linear(256, 20),
            nn.SELU(),
            nn.Linear(20, 20),
            nn.SELU(),
        )
        self.hidden_x2 = nn.Sequential(
            nn.MaxPool1d(kernel_size=2),
            nn.Conv1d(in_channels=2, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.Conv1d(in_channels=4, out_channels=4, kernel_size=5),
            nn.SELU(),
            nn.AvgPool1d(kernel_size=2),
            nn.Conv1d(in_channels=4, out_channels=2, kernel_size=3),
            nn.SELU(),
            nn.AvgPool1d(kernel_size=2),
            nn.Conv1d(in_channels=2, out_channels=2, kernel_size=3),
            nn.SELU(),
            nn.AvgPool1d(kernel_size=2),
        )
        self.hidden_encoded_1 = nn.Flatten()
        self.hidden_embedding = nn.Sequential(
            nn.Linear(26, 16),
            nn.SELU(),
            nn.Linear(16, 8),
            nn.SELU(),
            nn.Linear(8, 4),
        )

    def forward(self, x, n=-1):
      x = torch.transpose(x, 1, 2)
      x = self.hidden_x1(x)
      xfc = torch.reshape(x, (n, 256))
      xfc = self.hidden_xfc(xfc)
      x = torch.reshape(x, (n, 2, 128))
      x = self.hidden_x2(x)
      encoded = self.hidden_encoded_1(x)
      encoded = torch.cat((encoded, xfc), 1)
      embedding = self.hidden_embedding(encoded)

      return embedding

This is the translated Keras model which I'm not sure is right or not at the moment.
class sho_Model(tf.keras.Model):
    def __init__(self):
        super(sho_Model, self).__init__()
        self.hidden_x1 = keras.models.Sequential(layers=[

            Conv1D(filters=8, kernel_size=7, activation=selu, input_shape=(1, 80, 2)),
            Conv1D(filters=6, kernel_size=7, activation=selu),
            Conv1D(filters=4, kernel_size=5, activation=selu)
        ])

        self.hidden_xfc = keras.models.Sequential(layers=[
            Dense(20, activation=selu),
            Dense(20, activation=selu)
        ])

        self.hidden_x2 = keras.models.Sequential(layers=[

            MaxPool1D(pool_size=2),
            Conv1D(filters=4, kernel_size=5, activation=selu),
            Conv1D(filters=4, kernel_size=5, activation=selu),
            Conv1D(filters=4, kernel_size=5, activation=selu),
            Conv1D(filters=4, kernel_size=5, activation=selu),
            Conv1D(filters=4, kernel_size=5, activation=selu),
            Conv1D(filters=4, kernel_size=5, activation=selu),
            AveragePooling1D(pool_size=2),
            Conv1D(filters=2, kernel_size=3, activation=selu),
            AveragePooling1D(pool_size=2),
            Conv1D(filters=2, kernel_size=3, activation=selu),
            AveragePooling1D(pool_size=2)
        ])

        self.hidden_encoded = Flatten()

        self.hidden_embedding = keras.models.Sequential(layers=[
            Dense(16, activation=selu),
            Dense(8, activation=selu),
            Dense(4)
        ])

    def call(self, inputs, n=-1):
        x = K.permute_dimensions(inputs, (2, 1))
        x = self.hidden_x1(x)
        xfc = K.reshape(x, (n, 256))
        xfc = self.hidden_xfc(xfc)
        x = K.reshape(x, (n, 2, 128))
        x = self.hidden_x2(x)
        encoded = self.hidden_encoded(x)
        encoded = K.concatenate((encoded, xfc), 1)
        embedding = self.hidden_embedding(encoded)
        return embedding

To transfer the weights from the PyTorch model to the Keras model I'm using
curr_layer = 0
for layer in keras_model.layers:
    layer.set_weights(list(pytorch_model.parameters())[curr_layer].data.cpu().numpy())
    curr_layer+=1

But it gives the error
ValueError: You called `set_weights(weights)` on layer "sequential" with a weight list of length 8, but the layer was expecting 6 weights. Provided weights: [[[ 0.0098705   0.04642887 -0.02304325 -0.15255862...

When I list the layers in the keras model with
for layer in keras_model.layers:
    print(layer.name)

I just get six layers, how could I call each individual layer to transfer the weights?
sequential
sequential_1
sequential_2
flatten
sequential_3