I have a problem when converting onnx model to trt. My Environment:

ONNX IR version:  0.0.4
Opset version:    9
Producer version: 1.3

Converting a model leads to

[2020-01-16 07:04:54 WARNING] Setting layouts of network and plugin input/output tensors to linear, as 3D operators are found and 3D non-linear IO formats are not supported, yet.
[2020-01-16 07:04:54   ERROR] (Unnamed Layer* 314) [Shuffle]: IShuffleLayer applied to shape tensor must have 0 or 1 reshape dimensions: dimensions were [1,4]
[2020-01-16 07:04:54   ERROR] Builder failed while analyzing shapes.
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to create object```


title: saliency map for multi-class multi-label classification?

body:
I have found ways to do CAM/saliency map for multi class, but not multi label multi class. Do you know of any resources I can use to do it so I don't reinvent the wheel, or rather do you have advice for implementing it?

My specific use case is that I have a transfer learned ResNet that outputs a binary 1x11 vector. Each entry corresponds to presence of a certain feature in the input image. I want to be able to get a saliency map for each feature, so I can know what the network was looking at when deciding if each image has each of those features.


title: Concatenate with respect to 1st dimension

body:
In the following code what does torch.cat really do. I know it concatenates the batch which is contained in the sample but why do we have to do that and what does concatenate really mean.

 # memory is just a list of events
 def sample(self, batch_size):
    samples = zip(*random.sample(self.memory, batch_size))
    return map(lambda x: Variable(torch.cat(x,0)))


title: how visualize multi channel of feature from PyTorch?

body:
I'm almost newbie at PyTorch

One of my output size from conv is [1, 25, 8, 32]
(25=channel, 8=height, 32=width)

I can use squeeze and make it to [25, 8, 32].

But I'm confused with 25 channel.

When I want to visualize sum of 25 channel and make to one GRAYorRGB image(1or3x8x32),How can i deal with in code??

I can use matplot or tensorboardX for visualizing..


title: Making a PyTorch LSTM bidirectional=True causes: The size of tensor a (322) must match the size of tensor b (161) at non-singleton dimension 2

body:
My model is:

class BaselineModel(nn.Module):
    def __init__(self, feature_dim=5, hidden_size=5, num_layers=2, seq_length=1, dropout=0.1):
        super(BaselineModel, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        self.seq_length = seq_length

        self.lstm = nn.LSTM(input_size=feature_dim,
                            hidden_size=hidden_size, num_layers=num_layers, dropout=0.1, bidirectional=False)

    def forward(self, x, hidden=None):
        lstm_out, hidden = self.lstm(x, hidden)
        return lstm_out, hidden

    def init_hidden(self, batch_size):
        hidden = torch.zeros(
            self.num_layers, self.seq_length, self.hidden_size)
        cell = torch.zeros(self.num_layers, self.seq_length,
                           self.hidden_size)
        return (hidden.float(), cell.float())

Everything works fine. My input and output have dimension torch.Size([32, 15, 161]) and my hidden and cell have dimension torch.Size([2, 15, 161]).

But when I set bidirectional=True, the first thing I have to do is in my init_hidden, change self.num_layers to self.num_layers * 2. Which is fine. But then when I execute my training loop I get:

The size of tensor a (322) must match the size of tensor b (161) at non-singleton dimension 2

It's referring to the line that has loss = loss_fn(pred, outputs). I'm using loss_fn = torch.nn.MSELoss(reduction='sum') if that matters. So what am I doing wrong?


title: Create slice mask in pytorch?

body:
Is there a way to specify a mask based on a slice operation?
For example

A = torch.arange(6).view((2,3))
# A = [[0,1,2], [3,4,5]]

mask_slice = torch.mask_slice(A[:,1:])
# mask_slice = [[0,1,1],[0,1,1]]


title: why cannot cuda model be initialized under the __init__ method in a class that inherits multiprocessing.process?

body:
Here is my code:

from MyDetector import Helmet_Detector
from multiprocessing import Process

class Processor(Process):
    def __init__(self):
        super().__init__()
        self.helmet_detector = Helmet_Detector()

    def run(self):
         print(111)

    if __name__ == '__main__':
         p=Processor()
         p.start()

As you can see, the class 'Processor' inherits multiprocessing.Process, and Helmet_Detector is a YOLO model using cuda. But when I ran it, the error occurred as follow:

THCudaCheck FAIL file=C:\w\1\s\tmp_conda_3.7_075911\conda\conda-bld\pytorch_1579075223148\work\torch/csrc/generic/StorageSharing.cpp line=245 error=71 : operation not supported
Traceback (most recent call last):
  File "E:/python-tasks/WHU-CSTECH/Processor.py", line 17, in <module>
    p.start()
  File "C:\Anaconda\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Anaconda\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Anaconda\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Anaconda\lib\multiprocessing\popen_spawn_win32.py", line 89, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Anaconda\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Anaconda\lib\site-packages\torch\multiprocessing\reductions.py", line 242, in reduce_tensor
    event_sync_required) = storage._share_cuda_()
RuntimeError: cuda runtime error (71) : operation not supported at C:\w\1\s\tmp_conda_3.7_075911\conda\conda-bld\pytorch_1579075223148\work\torch/csrc/generic/StorageSharing.cpp:245

then I tried to intialize the Helmet_Detector in run method:

def run(self):
     print(111)
     self.helmet_detector = Helmet_Detector()

No error occurred. Could anyone please tell me the reason for this and how could I solve this problem? Thank you!


title: How do I convert 'the array saved as string to csv' back to float array?

body:
I had to merge a lot of files (containing word embeddings and othe real valued vectors) based on some common attributes so I used Pandas DataFrame and saved the intermediate files as csv.

Currently I have a dataframe whose columns look something like this:

I want to merge all last 4 columns (t-1embedding1a,t-1embedding7b,t-2embedding1a,t-2embedding7b) into a single vector to pass to neural network.

I planned to iterate over the current dataframe and take 4 temporary tensors with value of each column and concatenate and write to new dataframe.
However torch.tensor doesn't work as it says:

torch_tensor = torch.tensor(final['t-1embedding1a'].astype(float).values)

could not convert string to float: '[-6.12873614e-01 -5.58319509e-01 -9.73452032e-01  3.66993636e-01\n

I also tried np.fromstring() but the original values are lost in this case.

Sorry, if the question is unnecessarily complicated, I am a newbie to pytorch. Any help is appreciated!


title: Local Inference of saved PyTorch models

body:
I am interested in performing local (no server or cloud) inference of saved PyTorch models that I can "deploy" (for example, using PyInstaller) to machines that do not have any dependencies. Can someone point me in the correct direction for this?

Thanks in advance.

EDIT:

My goal is to be able create a command-line executable that I can share with collaborators who do not have any idea about Python/programming/Docker/containers and would simply like to infer on the models I send (with the correct data, of course).

EDIT-2:

Requirements from end product:

No "cloud" or remote processing - data privacy is a critical requirement and processing needs to be local on Windows and/or Linux machines using CPU (GPU would be a huge plus but not required)
No need for user to install anything (no Python/Docker engine/server/etc.) as this needs to be catered to a non-tech audience

Please comment if further clarifications are needed.


title: What is the least total batch size for SyncBatchNorm

body:
For the normal BatchNorm, the least batch size per GPU is 2.

I wonder if I use the SyncBatchNorm, can I use batch_size=1 for every GPU with more than a single GPU?

I.e, the total_batch_size is more than 1 but batch_size_per_gpu is 1.

I would appreciate answers for any deep learning framework, pytorch, tensorflow, mxnet, etc


title: How to do feature extraction on a graph using Pytorch?

body:
I'm trying to implement a GraphConv network whose goal is to adapt the graph it's being fed to a more suitable one for the target task. To do so, I feed it a deformed grid, compute prediction and compare the result with the a target graph. Which I know is a good graph.

Basically, the code looks like this:

import torch.nn as nn
import torch.nn.functional as F
from pygcn.layers import GraphConvolution

class GCN(nn.Module):
    def __init__(self, nfeat, nhid, nclass, dropout):
        super(GCN, self).__init__()

        self.gc1 = GraphConvolution(nfeat, nhid)
        self.gc2 = GraphConvolution(nhid, nclass)
        self.dropout = dropout

    def forward(self, x, adj):
        x = F.relu(self.gc1(x, adj))
        x = F.dropout(x, self.dropout, training=self.training)
        x = self.gc2(x, adj)
        return F.log_softmax(x, dim=1)

However I noticed that most GCN in litterature requires a lot of features on every nodes. And I only have  a few (4 features). So is there a way to extract features on a graph as we would do with convolutions and pooling for a CNN on an image ?


title: How can I load a partially trained model in PyTorch?

body: Let's suppose that the model I'm loading has 4 layers (layer0, layer1, layer2, layer3) for simplicity. If I only wanted that model to be pretrained for say layer0 and layer1, but have randomly initialized parameters for layer2 and layer3, how would I be able to do that?


title: Can I change volatile = False to torch.set_grad_enabled(True)??(in Pytorch)

body:
I have old python codes, so I need to change some parts.
next_q_values.volatile=False

I have this code, and next_q_values is 'torch.Tensor'
when I run this code:

error occured: "volatile was removed and now has no effect use with
torch no_grad instead"

After search, I know that volatile = True is same as torch.no_grad(),
but I want to use volatile = False, So I can't use torch.no_grad().
Can I change volatile = False to torch.set_grad_enabled(True)?


title: How to install torch 0.4 in cuda 10.2 environment?

body:
I want to install torch 0.4 to avoid version problem.
How to install torch 0.4 in Anaconda?
I can't found it.
Thank you,


title: ValueError: tile cannot extend outside image pytorch

body:
I am loading NumPy arrays as images in PyTorch while training the model, it’s giving me this error, I tried everything but couldn’t figure out pls help…, I am training a classifier model.......................................................................................................................................................................................

ValueError                                Traceback (most recent call last)
<ipython-input-12-b4d3f7be01c1> in <module>
      1 # training
----> 2 trained_model = train(n_epochs, np.Inf, loaders, model, optimizer, criterion)

<ipython-input-10-b4d180a2c041> in train(n_epochs, valid_loss_min_input, loaders, model, optimizer, criterion, device, checkpoint_path, best_model_path)
     29         ###################
     30         model.train()
---> 31         for batch_idx, (data, target) in enumerate(loaders['train']):
     32             # move to gpu
     33             data, target = data.to(device), target.to(device)

G:\anaconda3\envs\data_env\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

G:\anaconda3\envs\data_env\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

G:\anaconda3\envs\data_env\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

G:\anaconda3\envs\data_env\lib\site-packages\torch\utils\data\_utils\fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

G:\anaconda3\envs\data_env\lib\site-packages\torchvision\datasets\folder.py in __getitem__(self, index)
    135         sample = self.loader(path)
    136         if self.transform is not None:
--> 137             sample = self.transform(sample)
    138         if self.target_transform is not None:
    139             target = self.target_transform(target)

G:\anaconda3\envs\data_env\lib\site-packages\torchvision\transforms\transforms.py in __call__(self, img)
     59     def __call__(self, img):
     60         for t in self.transforms:
---> 61             img = t(img)
     62         return img
     63

<ipython-input-3-88cdee8f0d6c> in __call__(self, img)
      7         im = np.asarray(img)
      8         im = detect(im)
----> 9         img = Image.fromarray(im)
     10         img = img.resize(size=(128, 128))
     11         return img

G:\anaconda3\envs\data_env\lib\site-packages\PIL\Image.py in fromarray(obj, mode)
   2768             obj = obj.tostring()
   2769
-> 2770     return frombuffer(mode, size, obj, "raw", rawmode, 0, 1)
   2771
   2772

G:\anaconda3\envs\data_env\lib\site-packages\PIL\Image.py in frombuffer(mode, size, data, decoder_name, *args)
   2708             return im
   2709
-> 2710     return frombytes(mode, size, data, decoder_name, args)
   2711
   2712

G:\anaconda3\envs\data_env\lib\site-packages\PIL\Image.py in frombytes(mode, size, data, decoder_name, *args)
   2648
   2649     im = new(mode, size)
-> 2650     im.frombytes(data, decoder_name, args)
   2651     return im
   2652

G:\anaconda3\envs\data_env\lib\site-packages\PIL\Image.py in frombytes(self, data, decoder_name, *args)
    795         # unpack data
    796         d = _getdecoder(self.mode, decoder_name, args)
--> 797         d.setimage(self.im)
    798         s = d.decode(data)
    799

ValueError: tile cannot extend outside the image

In the custom function, I am trying to preprocess the image and then calling it
It works fine when I test out the code loading a small batch for display


title: How to create custom dataset for TextGAN-PyTorch

body:
Whenever I am training TextGAN-PyTorch  with my own custom dataset with seqGAN, I write each one sentence in each line, but this error is coming,
Starting Generator MLE Training...
Traceback (most recent call last):
  File "main.py", line 169, in <module>
    inst._run()
  File "/content/drive/My Drive/Colab Notebooks/2/TextGAN-PyTorch/instructor/real_data/seqgan_instructor.py", line 41, in _run
    self.pretrain_generator(cfg.MLE_train_epoch)
  File "/content/drive/My Drive/Colab Notebooks/2/TextGAN-PyTorch/instructor/real_data/seqgan_instructor.py", line 85, in pretrain_generator
    pre_loss = self.train_gen_epoch(self.gen, self.train_data.loader, self.mle_criterion, self.gen_opt)
  File "/content/drive/My Drive/Colab Notebooks/2/TextGAN-PyTorch/instructor/real_data/instructor.py", line 107, in train_gen_epoch
    return total_loss / len(data_loader)
ZeroDivisionError: division by zero

I am using google colab for training the GAN.
How to make my own dataset.txt file?


title: Pytorch memory profiler returns high CPU memory for empty

body:
When I run the pytorch memory profiler it returns the following:
Name                       Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CPU Mem          Self CPU Mem     CUDA Mem         Self CUDA Mem    Number of Calls
-------------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------
empty                      1.11%            443.155ms        100.19%          39.822s          1.851ms          25.95 Gb         7.98 Gb          0 b              0 b

This ´empty´ is taking up a lot of memory (more than anything else in the network). I wonder whether this is expected behaviour, or a sign that there are a lot of empty tensors taking up memory, without being useful?
If so, where should I start looking for the culprits?


title: Is there a better way to calculate loss for multi-task DNN modeling?

body:
Suppose there are over one thousand tasks in the multi-task deep learning. More than a thousand columns of labels. Each task (column) has a specific weight in this case. It would take such long time to loop over each task to calculate the sum of loss using the following code snippet.
criterion = nn.MSELoss()

outputs = model(inputs)
loss = torch.tensor(0.0).to(device)
for j, w in enumerate(weights):
    # mask keeping labeled molecules for each task
    mask = labels[:, j] >= 0.0
    if len(labels[:, j][mask]) != 0:
        # the loss is the sum of each task/target loss.
        # there are labeled samples for this task, so we add it's loss
        loss += criterion(outputs[j][mask], labels[:, j][mask].view(-1, 1)) * w

This dataset was quite small. The dataset has 10K rows and 1024 columns and the labels are a 10K * 160 sparse matrix. Each of those 160 columns is one task. Batch size is 32.  Below are the shapes of outputs, labels, weights:
len(outputs[0]), len(outputs)
(32, 160)
weights.shape
torch.Size([160])
labels.shape
torch.Size([32, 160])

But what I really want to try is one dataset which has over 1M rows and 1024 features and over 10K labels. The labels are sparse of course.
**update**
Thanks for you suggestions and code, Shai. I modified the code a little bit as follows, but the loss was the same as your code.

all_out = torch.cat(outputs).view(len(outputs), -1).T
all_mask = labels != -100.0
err = (all_out - labels) ** 2  # raw L2
err = all_mask * err  # mask only the relevant entries in the err
mask_nums = all_mask.sum(axis=0)
err = err * weights[None, :]  # weight each task
err = err / mask_nums[None, :]
err[err != err] = torch.tensor([0.0], requires_grad=True).to(device) # replace nan to 0.0
loss = err.sum()

A newly raised question is the loss can't get back propagated. Only the loss of the first batch was calculated. The following batches got a loss of 0.0.

Epoch: [1/20], Step: [1/316], Loss: 4.702103614807129
Epoch: [1/20], Step: [2/316], Loss: 0.0
Epoch: [1/20], Step: [3/316], Loss: 0.0
Epoch: [1/20], Step: [4/316], Loss: 0.0
Epoch: [1/20], Step: [5/316], Loss: 0.0
The loss was 0 and outputs was 32* 160 of nan after the first batch.


title: PyTorch convolutional block - CIFAR10 - RuntimeError

body:
I am using PyTorch 1.7 and Python 3.8 with CIFAR-10 dataset. I am trying to create a block with: conv -> conv -> pool -> fc. Fully connected layer (fc) has 256 neurons. The code for this is as follows:
# Testing-
conv1 = nn.Conv2d(
    in_channels = 3, out_channels = 64,
    kernel_size = 3, stride = 1,
    padding = 1, bias = True
    )
conv2 = nn.Conv2d(
    in_channels = 64, out_channels = 64,
    kernel_size = 3, stride = 1,
    padding = 1, bias = True
    )
pool = nn.MaxPool2d(
    kernel_size = 2, stride = 2
    )
fc1 = nn.Linear(
    in_features = 64 * 16 * 16, out_features = 256
    bias = True
)

images.shape
# torch.Size([32, 3, 32, 32])

x = conv1(images)
x.shape
# torch.Size([32, 64, 32, 32])

x = conv2(x)
x.shape
# torch.Size([32, 64, 32, 32])

x = pool(x)
x.shape
# torch.Size([32, 64, 16, 16])

# This line of code gives error-
x = fc1(x)

RuntimeError: mat1 and mat2 shapes cannot be multiplied (32768x16 and
16384x256)

What is going wrong?


title: Applying a simple transformation to get a binary image using pytorch

body:
I'd like to binarize image before passing it to the dataloader, I have created a dataset class which works well. but in the __getitem__() method  I'd like to threshold the image:
    def __getitem__(self, idx):
        # Open image, apply transforms and return with label
        img_path = os.path.join(self.dir, self.filelist[filename"])
        image = Image.open(img_path)
        label = self.x_data.iloc[idx]["label"]

        # Applying transformation to the image
        if self.transforms is not None:
           image = self.transforms(image)

        # applying threshold here:
        my_threshold = 240
        image = image.point(lambda p: p < my_threshold and 255)
        image = torch.tensor(image)

        return image, label

And then I tried to invoke the dataset:
    data_transformer = transforms.Compose([
        transforms.Resize((10, 10)),
        transforms.Grayscale()
        //transforms.ToTensor()
    ])

train_set = MyNewDataset(data_path, data_transformer, rows_train)

Since I have applied the threshold on a PIL object I need to apply afterwards a conversion to a tensor object , but for some reason it crashes. can somebody please assist me?


title: Best way to cut a pytorch tensor into overlapping chunks?

body:
If for instance I have:
eg6 = torch.tensor([
    [ 1.,  7., 13., 19.],
    [ 2.,  8., 14., 20.],
    [ 3.,  9., 15., 21.],
    [ 4., 10., 16., 22.],
    [ 5., 11., 17., 23.],
    [ 6., 12., 18., 24.]])
batch1 = eg6
batch2 = -eg6
x = torch.cat((batch1,batch2)).view(2,6,4)

And then I want to slice it up into overlapping chunks, like a sliding window function, and have the chunks be batch-processable. For example, and just looking at the first dimension, I want 1,2,3, 3,4,5, 5,6 (or 5,6,0).
It seems unfold() kind of does what I want. It transposes the last two dimensions for some reason, but that is easy enough to repair. Changing it from shape [2,3,3,4] to [6,3,4] requires a memory copy, but I believe that is unavoidable?
SZ=3
x2 = x.unfold(1,SZ,2).transpose(2,3).reshape(-1,SZ,4)

This works perfectly when x is of shape [2,7,4]. But with only 6 rows, it throws away the final row.
Is there a version of unfold() that can be told to use all data, ideally taking a pad character?
Or do I need to pad x before calling unfold()? What is the best way to do that? I'm wondering if "pad" is the wrong word, as I'm only finding functions that want to put padding characters at both ends, with convolutions in mind.

Aside: Looking at the source of unfold, it seems the strange transpose is there deliberately and explicitly?! For that reason, and the undesired chop behaviour, it made me think the correct answer to my question might be write a new low-level function. But that is too much effort for me, at least for today... (I think a second function for the backwards pass also needs to be written.)


title: How perform unsupervised clustering on numbers in an Array using PyTorch

body:
I got this array and I want to cluster/group the numbers into similar values.
An example of input array:
array([ 57, 58, 59, 60, 61, 78, 79, 80, 81, 82, 83, 101, 102, 103, 104, 105, 106]

expected result :
array([57,58,59,60,61]), ([78,79,80,81,82,83]), ([101,102,103,104,105,106])

I tried to use clustering but I don't think it's gonna work if I don't know how many I'm going to split up.
true = np.where(array>=1)
-> (array([ 57,  58,  59,  60,  61,  78,  79,  80,  81,  82,  83, 101, 102,
    103, 104, 105, 106], dtype=int64),)


title: loss.backward() one of the variables needed for gradient computation has been modified by an inplace operation error

body:
I can't figure out the error, I went exactly according to the pytorch tutorials but loss.backward() is giving an error.
 def forward(self, x):
        hidden_inputs = self.hidden(x)
        hidden_outputs = self.sigmoid(hidden_inputs)
        final_inputs = self.output(hidden_outputs)
        final_outputs = self.softmax(final_inputs)
        return final_outputs

def run_model(self):
        self.preprocessing()
        # create your optimizer
        optimizer = torch.optim.SGD(self.parameters(), lr = self.lr)
        accuracy = 0
        for epoch in range(self.num_epochs):

            torch.autograd.set_detect_anomaly(True)
            label_predict = self.forward(self.features)

            print(label_predict.shape, label_target.shape)
            loss = self.loss(label_predict.float(), label_target.float())
            print(loss)

            optimizer.zero_grad()   # zero the gradient buffers
            # Backward pass: compute gradient of the loss with respect to model
            # parameters
            loss.backward()

            # Calling the step function on an Optimizer makes an update to its
            # parameters
            optimizer.step()

errors:
UserWarning: Error detected in SoftmaxBackward. Traceback of forward call that caused the error
untimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [225741, 2]], which is output 0 of SoftmaxBackward, is at version 225741; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!


title: How to make manual batch in pytorch with MNIST dataset

body:
I am new to pytorch.
I am trying to use a different approach when training MNIST dataset in pytorch.
Normally, we put all shuffled numbers for each batch.
What I want to do is (assuming by batch size is 20 and there are 1000 training examples).
Make a batch look this this (I wrote only labels for simplicity)

batch_1 = [0, 0, 1, 0, ... ] only 0 and 1 for this batch
batch_2 = [6, 6, 7, 7, ... ] only 6 and 7 for this batch
batch_3 = [2, 3, 2, 2, ... ] only 2 and 3 for this batch
batch_4 = [4, 5, 5, 4, ... ] only 4 and 5 for this batch
batch_5 = [6, 6, 7, 7, ... ] only 6 and 7 for this batch
batch_6 = [8, 9, 9, 9, ... ] only 8 and 9 for this batch
batch_7 = [2, 3, 2, 2, ... ] only 2 and 3 for this batch
batch_8 = [4, 5, 5, 4, ... ] only 4 and 5 for this batch
...
batch_50 = [4, 5, 5, 4, ... ] only 4 and 5 for this batch

How can I do this in pytorch??
Thank you

So far this is my implementation for training / testing.
I will expand/manipulate this code below to develop the above.
import torch
from torch.utils.data import DataLoader, Subset
import torchvision.datasets as datasets
from torchvision import transforms
import numpy as np

mnist_trainset = datasets.MNIST(root='./data', train=True,
                                download=True,
                                transform=transforms.Compose([transforms.ToTensor()]))

mnist_testset  = datasets.MNIST(root='./data',
                                train=False,
                                download=True,
                                transform=transforms.Compose([transforms.ToTensor()]))

class_inds = [torch.where(mnist_trainset.targets == class_idx)[0]
              for class_idx in mnist_trainset.class_to_idx.values()]

train_dataloaders = [
                     DataLoader(dataset=Subset(mnist_trainset, inds),
                                batch_size=50,
                                shuffle=True,
                                drop_last=False
                     )
                     for inds in class_inds
]

test_dataloader  = torch.utils.data.DataLoader(mnist_testset,
                                               batch_size=50,
                                               shuffle=False)

class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear_1 = torch.nn.Linear(784, 256)
        self.linear_2 = torch.nn.Linear(256, 10)
        self.sigmoid  = torch.nn.Sigmoid()

    def forward(self, x):
        x = x.reshape(x.size(0), -1)
        x = self.linear_1(x)
        x = self.sigmoid(x)
        pred = self.linear_2(x)

        return pred

model = Model()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
epochs = 20

criterion = torch.nn.CrossEntropyLoss()

test_acc   = list()

for epoch in range(epochs):

    print(f"epoch {epoch} started")

    model.train()
    itr = 0
    iterators = list(map(iter, train_dataloaders))
    while iterators:
        itr = itr + 1
        iterator = np.random.choice(iterators)
        try:
            image, label = next(iterator)

            optimizer.zero_grad()

            pred = model(image)

            loss = criterion(pred, label)

            loss.backward()
            optimizer.step()

        except StopIteration:
            iterators.remove(iterator)

    model.eval()
    total = 0
    for itr, (image, label) in enumerate(test_dataloader):

        pred = model(image)

        loss = criterion(pred, label)

        # we now need softmax because we are testing.
        pred = torch.nn.functional.softmax(pred, dim=1)
        for i, p in enumerate(pred):
            if label[i] == torch.max(p.data, 0)[1]:
                total = total + 1

    accuracy = total / len(mnist_testset)

    # append accuracy here
    test_acc.append(accuracy)


I am new to deep learning and have the project on detecting traffic lights in university where we can use open-source code.
So, I tried to run the code on kaggle https://www.kaggle.com/endoruk1234/trafficlightdetection-fasterrcnn-pytorch/log
However on the stage of testing the saved model on a video, I got this mistake 'In training mode, targets should be passed'.
I am not sure why do I need to pass targets on the testing stage. I don't understand is it a problem with the initial model, or video capture part is written with mistakes.
The model

    from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
    def _get_instance_segmentation_model(num_classes):
        model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
        in_features = model.roi_heads.box_predictor.cls_score.in_features
        model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
        return model
    model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
    N_CLASS = 4
    INP_FEATURES = model.roi_heads.box_predictor.cls_score.in_features
    model.roi_heads.box_predictor = FastRCNNPredictor(INP_FEATURES, N_CLASS)
    model.to(device)
    params = [p for p in model.parameters() if p.requires_grad]
    optimizer = torch.optim.Adam(params)
    lr_scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer)

Training

    lossHist = LossAverager()
    valLossHist = LossAverager()

    for epoch in range(EPOCHS):

        start_time = time()
        model.train()
        lossHist.reset()

        for images, targets, image_ids in tqdm(trainDataLoader):
            #bbox = check_bbox(bbox)
            images = torch.stack(images).to(device)
            targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

            bs = images.shape[0]

            loss_dict = model(images, targets)

            totalLoss = sum(loss for loss in loss_dict.values())
            lossValue = totalLoss.item()

            lossHist.update(lossValue,bs)

            optimizer.zero_grad()
            totalLoss.backward()
            optimizer.step()

        if lr_scheduler is not None:
            lr_scheduler.step(totalLoss)

        print(f"[{str(datetime.timedelta(seconds = time() - start_time))[2:7]}]")
        print(f"Epoch {epoch}/{EPOCHS}")
        print(f"Train loss: {lossHist.avg}")

        if(epoch == 10):
            torch.save(model.state_dict(), 'fasterrcnn_resnet{}_fpn.pth'.format(epoch))

    torch.save(model.state_dict(), 'fasterrcnn_resnet{}_fpn.pth'.format(epoch))

Testing on a video with a mistake

    while(True):
        ret, input = cap.read()
        image = input.copy()
        input = preprocess(input).float()
        input = input.unsqueeze_(0)
        input = input.type(torch.cuda.FloatTensor)

        print(input)

        result = model(input)

        boxes = result[0]['boxes'].type(torch.cuda.FloatTensor)
        scores = result[0]['scores'].type(torch.cuda.FloatTensor)
        labels = result[0]['labels'].type(torch.cuda.FloatTensor)

        mask = nms(boxes,scores,0.3)
        boxes = boxes[mask]
        scores = scores[mask]
        labels = labels[mask]

        boxes = boxes.data.cpu().numpy().astype(np.int32)
        scores = scores.data.cpu().numpy()
        labels = labels.data.cpu().numpy()

        mask = scores >= 0.5
        boxes = boxes[mask]
        scores = scores[mask]
        labels = labels[mask]

        colors = {1:(0,255,0), 2:(255,255,0), 3:(255,0,0)}

        for box,label in zip(boxes,labels):
            image = cv2.rectangle(image,
                              (box[0], box[1]),
                              (box[2], box[3]),
                              (0,0,255), 1)

        cv2.imshow("image", image)

        if cv2.waitKey(0):
            break

    ValueError                                Traceback (most recent call last)
    <ipython-input-84-e32f9d25d942> in <module>()
          8     print(input)
          9
    ---> 10     result = model(input)
         11
         12     boxes = result[0]['boxes'].type(torch.cuda.FloatTensor)

    /usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
        725             result = self._slow_forward(*input, **kwargs)
        726         else:
    --> 727             result = self.forward(*input, **kwargs)
        728         for hook in itertools.chain(
        729                 _global_forward_hooks.values(),

    /usr/local/lib/python3.7/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
         58         """
         59         if self.training and targets is None:
    ---> 60             raise ValueError("In training mode, targets should be passed")
         61         if self.training:
         62             assert targets is not None

    ValueError: In training mode, targets should be passed

Thank you in advance! If you can tell me how to correct the model or videocapture code I will be very grateful.


title: Cross entropy IndexError Dimension out of range

body:
I'm trying to train a GAN in some images, I followed the tutorial on pytorch's page and got to the following code, but when the crossentropy function is applyed during the training it returns the error below the code:
import random
import torch.nn as nn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as torch_dataset
import torchvision.transforms as transforms
import torchvision.utils as vutils
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
from IPython.display import HTML

seed = 1
print("Random Seed: ", seed)
random.seed(seed)
torch.manual_seed(seed)
images_folder_path = "./images/"

batch_size = 128
image_size = 256
n_channels = 1
z_vector = 100
n_features_generator = 64
n_features_discriminator = 64
num_epochs = 5
lr = 0.0002
beta1 = 0.5

dataset = torch_dataset.ImageFolder(
    root=images_folder_path, transform=transforms.Compose(
        [
            transforms.Grayscale(num_output_channels=1),
            transforms.Resize(image_size),
            transforms.CenterCrop(image_size),
            transforms.ToTensor(),
            transforms.Normalize(0.5, 0.5)
         ]
    )
)

dataloader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=True, num_workers=0)

device = torch.device("cuda:0" if (torch.cuda.is_available()) else "cpu")

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.main = nn.Sequential(
            nn.ConvTranspose2d(z_vector, n_features_generator * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(n_features_generator * 8),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 8, n_features_generator * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator * 4),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 4, n_features_generator * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator * 2),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator * 2, n_features_generator, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_generator),
            nn.ReLU(True),
            nn.ConvTranspose2d(n_features_generator, n_channels, 4, 2, 1, bias=False),
            nn.Tanh()
        )

    def forward(self, inputs):
        return self.main(inputs)

class Discriminator(nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        self.main = nn.Sequential(
            nn.Conv2d(n_channels, n_features_discriminator, 4, 2, 1, bias=False),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator, n_features_discriminator * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 2),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 2, n_features_discriminator * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 4),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 4, n_features_discriminator * 8, 4, 2, 1, bias=False),
            nn.BatchNorm2d(n_features_discriminator * 8),
            nn.LeakyReLU(0.2, inplace=True),
            nn.Conv2d(n_features_discriminator * 8, 1, 4, 1, 0, bias=False),
            nn.Sigmoid()
        )

    def forward(self, inputs):
        return self.main(inputs)

netG = Generator().to(device)
if device.type == 'cuda':
    netG = nn.DataParallel(netG)
netG.apply(weights_init)
print(netG)

netD = Discriminator().to(device)
if device.type == 'cuda':
    netD = nn.DataParallel(netD)
netD.apply(weights_init)
print(netD)

criterion = nn.CrossEntropyLoss()

fixed_noise = torch.randn(64, z_vector, 1, 1, device=device)

real_label = 1.
fake_label = 0.

optimizerD = optim.Adam(netD.parameters(), lr=lr, betas=(beta1, 0.999))
optimizerG = optim.Adam(netG.parameters(), lr=lr, betas=(beta1, 0.999))

img_list = []
G_losses = []
D_losses = []
iters = 0

print("Starting Training Loop...")
for epoch in range(num_epochs):
    for i, data in enumerate(dataloader, 0):
        netD.zero_grad()
        real_cpu = data[0].to(device)
        b_size = real_cpu.size(0)
        label = torch.full((b_size,), real_label, dtype=torch.float, device=device)
        output = netD(real_cpu).view(-1)
# ----------------------------------------------------------------------------------
        errD_real = criterion(output, label) # ERROR HAPPENS HERE
# ----------------------------------------------------------------------------------
        errD_real.backward()
        D_x = output.mean().item()

        noise = torch.randn(b_size, z_vector, 1, 1, device=device)
        fake = netG(noise)
        label.fill_(fake_label)
        output = netD(fake.detach()).view(-1)
        errD_fake = criterion(output, label)
        errD_fake.backward()
        D_G_z1 = output.mean().item()
        errD = errD_real + errD_fake
        optimizerD.step()

        netG.zero_grad()
        label.fill_(real_label)
        output = netD(fake).view(-1)
        errG = criterion(output, label)
        errG.backward()
        D_G_z2 = output.mean().item()
        optimizerG.step()

        if i % 50 == 0:
            print('[%d/%d][%d/%d]\tLoss_D: %.4f\tLoss_G: %.4f\tD(x): %.4f\tD(G(z)): %.4f / %.4f'
                  % (epoch, num_epochs, i, len(dataloader),
                     errD.item(), errG.item(), D_x, D_G_z1, D_G_z2))

        G_losses.append(errG.item())
        D_losses.append(errD.item())

        if (iters % 500 == 0) or ((epoch == num_epochs-1) and (i == len(dataloader)-1)):
            with torch.no_grad():
                fake = netG(fixed_noise).detach().cpu()
            img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

        iters += 1

Error:
Traceback (most recent call last):
File "G:/Pastas Estruturadas/Conhecimento/CEFET/IA/SpectroGAN/dcgan.py", line 137, in
errD_real = criterion(output, label)
File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
result = self.forward(*input, **kwargs)
File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\modules\loss.py", line 948, in forward
ignore_index=self.ignore_index, reduction=self.reduction)
File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\functional.py", line 2422, in cross_entropy
return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
File "C:\Users\Ramon\anaconda3\envs\vision\lib\site-packages\torch\nn\functional.py", line 1591, in log_softmax
ret = input.log_softmax(dim)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
Process finished with exit code 1


title: got "Could not export Python function call '_ScaleGradient'" when export detectron2 model to torchscript

body:
After trained a module by Detectron2, I tried to export the model to TorchScript,
Then I got the following errors:

Could not export Python function call '_ScaleGradient'. Remove calls to Python functions > before export. Did you forget to add @script or @script_method annotation? If this is a  > nn.ModuleList, add it to  __constants__

I found code is in detectron2/modeling/roi_heads/cascade_rcnn.py
class _ScaleGradient(Function):
    @staticmethod
    def forward(ctx, input, scale):
        ctx.scale = scale
        return input

    @staticmethod
    def backward(ctx, grad_output):
        return grad_output * ctx.scale, None

So I change @statcmethod annos to @torch.jit.script_method, after that, I got a "'ScriptMethodStub' object is not callable" error.
I'm not familar with torchscript, how to fix this problem?
Thanks in advance.


title: Using pytorch cuda in AWS sagemaker notebook instance

body:
In colab, whenever we need GPU, we simply click change runtime type and change hardware accelarator to GPU
and cuda becomes available, torch.cuda.is_available() is True
How to do this is AWS sagemaker, i.e. turning on cuda.
I am new to AWS and trying to train model using pytorch in aws sagemaker, where Pytorch code is first tested in colab environment.
my sagemaker notebook insatnce is ml.t2.medium


title: image translation in Pytorch, using affine_grid & grid_sample functions

body:
I am going to move the image for 1 or 2 pixels, as I specified a small number (1.25 , 1.9) in the affine matrix.
BUT, the image is moved far far away, like hundreds of pixels:

( my input image is fully filled with yellow pineapples)
Below is a working example.
import torch
import numpy as np
import matplotlib.pyplot as plt
from torchvision import datasets, transforms
import torch.nn.functional as F

rotation_simple = np.array([[1,0, 1.25],
                           [ 0,1, 1.9]])

#load image
transform = transforms.Compose([transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor()])
dataloader = torch.utils.data.DataLoader(datasets.ImageFolder('/home/Pictures',transform=transform,), shuffle=True)
dtype =  torch.FloatTensor

i = 0
while i<3:
    img, labels = next(iter(dataloader))
    img = img#.double() # 有时候要转为double有时候不用转

    rotation_simple = torch.as_tensor(rotation_simple)[None]

    grid = F.affine_grid(rotation_simple, img.size()).type(dtype)
    x = F.grid_sample(img, grid)

    plt.imshow(x[0].permute(1, 2, 0))
    plt.show()
    i+=1

I wonder why does the function move the the image so far away instead of moving it for just 1 pixel in x and y direction.
Ps. Setting "align_corners=True" didn't help for this case.
Pps. My pytorch version is 1.4.0+cu100


title: Multi-output regression using skorch & sklearn pipeline gives runtime error due to dtype

body:
I want to use skorch to do multi-output regression. I've created a small toy example as can be seen below. In the example, the NN should predict 5 outputs. I also want to use a preprocessing step that is incorporated using sklearn pipelines (in this example PCA is used, but it could be any other preprocessor). When  executing this example I get the following error in the Variable._execution_engine.run_backward step of torch:
RuntimeError: Found dtype Double but expected Float

Am I forgetting something? I suspect, somewhere something has to be cast, but as skorch handles a lot of the pytorch stuff, I don't see what and where.
Example:
import torch
import skorch
from sklearn.datasets import make_classification, make_regression
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.decomposition import PCA

X, y = make_regression(n_samples=1000, n_features=40, n_targets=5)
X = X.astype('float32')

class RegressionModule(torch.nn.Module):
    def __init__(self, input_dim=80):
        super().__init__()
        self.l0 = torch.nn.Linear(input_dim, 10)
        self.l1 = torch.nn.Linear(10, 5)

    def forward(self, X):
        y = self.l0(X)
        y = self.l1(y)
        return y

class InputShapeSetter(skorch.callbacks.Callback):
    def on_train_begin(self, net, X, y):
        net.set_params(module__input_dim=X.shape[-1])

net = skorch.NeuralNetRegressor(
    RegressionModule,
    callbacks=[InputShapeSetter()],
)

pipe = make_pipeline(PCA(n_components=10), net)

pipe.fit(X, y)
print(pipe.predict(X))

Edit 1:
Casting X to float32 at the start won't work for every preprocessor as can be seen from this example:
import torch
import skorch
from sklearn.datasets import make_classification, make_regression
from sklearn.pipeline import Pipeline
from sklearn.decomposition import PCA
from category_encoders import OneHotEncoder

X, y = make_regression(n_samples=1000, n_features=40, n_targets=5)
X = pd.DataFrame(X,columns=[f'feature_{i}' for i in range(X.shape[1])])
X['feature_1'] = pd.qcut(X['feature_1'], 3, labels=["good", "medium", "bad"])

y = y.astype('float32')

class RegressionModule(torch.nn.Module):
    def __init__(self, input_dim=80):
        super().__init__()
        self.l0 = torch.nn.Linear(input_dim, 10)
        self.l1 = torch.nn.Linear(10, 5)

    def forward(self, X):
        y = self.l0(X)
        y = self.l1(y)
        return y

class InputShapeSetter(skorch.callbacks.Callback):
    def on_train_begin(self, net, X, y):
        net.set_params(module__input_dim=X.shape[-1])

net = skorch.NeuralNetRegressor(
    RegressionModule,
    callbacks=[InputShapeSetter()],
)

pipe = make_pipeline(OneHotEncoder(cols=['feature_1'], return_df=False), net)

pipe.fit(X, y)
print(pipe.predict(X))


title: how to create shared weights layer in multiple input models with no grads

body:
I want to create a model with 2 inputs x and y. And I want make loss function only concerned about x. So the model can optimize former layer only with x. But now even the loss is only concerned about x, the optimazition will still calculate x and y in the former layer.

I have tried to make y to y.detach() to make grads stop. But that won't work. And I also want to try to create new shared weights layer, but I don't know how to do that.

def forward(self, x, y=None):
    x = self.conv1(x)
    x = self.bn1(x)
    x = self.maxpool(x)
    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)
    y = self.conv1(y)
    y = self.bn1(y)
    y = self.maxpool(y)
    y = self.layer1(y)
    return x, y


title: Is it possible to use a deep learning library (Tensorflow, PyTorch, MXNet) to train models for Botframework?

body: If I recall correctly, you need to train models on LUIS.ai to use with Botframework-Python. I want to train my own models with my own architecture using a deep learning library. So the question is, is it possible to train models using DL frameworks?


title: Would it possible to use all memory of GPUs with one model?

body:
There is a model and two GPUs. I put the model on GPU with model.cuda(). If I passed a big image to the model, it allocated all memory of GPU0 and then it raised CUDA out of memory error without allocating any memory of GPU1.

Because there is only one image every forward(), I can not use such torch.nn.DataParallel things to split input.

Is there any way to use all the memory of GPUs when passing a image to the model?

I'm using Python3.7 and Pytorch1.1.


title: Pytorch :Why my dataset variance do not get the correct result?

body:
Here is the function I wrote:

def channel_var(image_dataset):
    res = image_dataset[0]
    for image in image_dataset[1:]:
        res += image
    return tuple(map(lambda x: x/len(image_dataset),
     (torch.var(res[0]),
      torch.var(res[1]),
      torch.var(res[2]))))

then I tested it with a Normal distribution :

m = normal.Normal(0, 3)
m.sample((1, 3, 32, 32))

And I get this wrong result :

channel_var(list_test)

>>(tensor(0.0338), tensor(0.0352), tensor(0.0365))

Thank you


title: Fitting the MNIST dataset to neural network giving “NotImplementedError”

body:
I'm new to PyTorch and I'm using the classic MNIST dataset for image classification.While fitting the model, I'm getting the error :

NotImplementedError: uint8

I'm using fastai's library's class as a wrapper for all training and validation data and a very basic single-layer neural network. The code I'm using is as follows :

from keras.datasets import mnist

import matplotlib.pyplot as plt

from fastai.metrics import *

from fastai.model import *

from fastai.dataset import *

import torch.nn as nn

(x_train, y_train), (x_valid, y_valid) = mnist.load_data()

net = nn.Sequential(
  nn.Linear(784,10),
  nn.Softmax()).cuda()

md = ImageClassifierData.from_arrays('/data/mnist',
(x_train,y_train),
(x_valid, y_valid))

loss = nn.NLLLoss()
metrics = [accuracy]
opt=optim.SGD(net.parameters(), 1e-1, momentum=0.9, weight_decay=1e-3)

fit(net, md, n_epochs=3, crit=loss, opt=opt, metrics=metrics)

Can someone tell what is this error about and it's solution?


title: GCP DL Image: Base m36 (with CUDA 10.0). How to enable GPU in PyTorch?

body:
I am trying to run DL PyTorch jobs on a GCP box. Steps I take:

Create a VM with m36 deep learning image and K80 attached. There is a question re CUDA 10 driver installation at first login, I choose 'y' and the driver gets set up.
install pyenv
pyenv install 3.7.5
pyenv virtualenv 3.7.5 myenv
pyenv activate myenv
pip install fastai
Start python console and run:

Code:

import torch
torch.cuda.is_available()

It returns False. What am I supposed to do to enable GPU in PyTorch?


title: CUDA error while passing hidden states to convLSTM

body:
I'm getting the following error, here's the traceback:

Traceback (most recent call last):
  File "train.py", line 136, in <module>
    train(epoch)
  File "train.py", line 112, in train
    output = model(data, hc)             # Get outputs of LSTM
  File "/home/ama1128/.conda/envs/matrix/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/scratch/ama1128/convLSTM/model.py", line 118, in forward
    hc = self.cell_list[t](input=x[0], prev_state=[h_0, c_0])
  File "/home/ama1128/.conda/envs/matrix/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/scratch/ama1128/convLSTM/model.py", line 50, in forward
    combined = torch.cat((input, h_prev), dim=1) # concatenate along channel axis
RuntimeError: Expected object of backend CUDA but got backend CPU for sequence element 1 in sequence argument at position #1 'tensors'

Training Loop:

def train(epoch):
    model.train()
    hc = model.init_hidden(batch_size=1)
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        target = data[1:]                   # Set target, images 2 to 18
        if gpu:
            data = data.cuda()
            target = target.cuda()
        output = model(data, hc)            # outputs of LSTM
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

convLSTMCell forward:

def forward(self, input, prev_state):
    h_prev, c_prev = prev_state
    combined = torch.cat((input, h_prev), dim=1) # concatenate along channel axis

    combined_conv = self.conv(combined)
    cc_i, cc_f, cc_o, cc_g = torch.split(combined_conv, self.hidden_dim, dim=1)

    i = torch.sigmoid(cc_i)
    f = torch.sigmoid(cc_f)
    o = torch.sigmoid(cc_o)
    g = torch.tanh(cc_g)

    c_cur = f * c_prev + i * g
    h_cur = o * torch.tanh(c_cur)

    return h_cur, c_cur

convLSTM Class:
Initialize states:

def init_hidden(self, batch_size):
        hidden = torch.zeros(batch_size, self.hidden_dim[0], self.height, self.width)
        cell = torch.zeros(batch_size, self.hidden_dim[0], self.height, self.width)
        if gpu:
            hidden.cuda()
            cell.cuda()
        return hidden, cell

convLSTM forward:

def forward(self, x, hc):
    outputs = []
    states = []
    x = torch.unsqueeze(x, 1)   # change shape from (18,3,128,128) to (18,1,3,128,128)
    h_0, c_0 = hc

    for t in range(0, self.num_layers):
        if t==0:
            hc = self.cell_list[t](input=x[0], prev_state=[h_0, c_0])
                states.append(hc)
            hidden, cell = hc
            outputs.append(hidden.view(3,128,128))
        else:
            h, c = states[t-1]                              # unpack previous states
            if gpu:
                h.cuda()
                c.cuda()
            hc = self.cell_list[t](input=x[t], prev_state=[h,c]) # current states
            states.append(hc)                               # store current states for next cell
            hidden, cell = hc                               # unpack current states
            outputs.append(hidden.view(3,128,128))

outputs is reformatted using torch.stack before passing it to the loss function.
I've tried to cast the hidden states to gpu wherever I can and the error still persists. What's going on?


title: Boundary Detection trained for Specific Boundary through Deep Learning

body:
I am currently looking for the best way to complete a project of mine. This entails a network trained to detect a particular boundary in an image. I have attached an example image below that shows such a boundary (the tree ring). In this case, of course, an edge detector would find many other edges in the image in addition to the one I want. The end goal is to detect the boundary and essentially draw an approximate line at the ring.

I am some experience with PyTorch and have implemented some simple object detection networks, but in my search for a project similar to this, all I have found is doing edge detection through deep learning, but this will detect all edges. I have thought of maybe doing the detection in steps, starting with edge detection and training a CNN on the straight edges, but that does not seem like it will be accurate.

I also thought of trying to train a CNN on images that are centered on a point on the boundary. Thus, every good result from the CNN should be essentially a point on a line, I can then just run some simple linear regression on it to draw. However, I am not sure if any of these ideas will work out. Of course, I can try them out but was hoping for some insight from the experienced.

Example image of tree core. Focus on dark blue marks, they mark the ring. Ignore the lines connecting the marks, and the light blue/white marks.


title: Considering transforms in torchaudio, is window length etc. in audio frames or milliseconds

body: I am preprocessing audio recordings using torchaudio. The documentation lists win_length (int) – Window size. (Default: n_fft) Does this value, which defaults to 400, represent the window length in frames or milliseconds?


title: PyTorch: why the difference between dir(nn.Module()) and dir(nn.Module)

body:
Tried

e = dir(nn.Module())
f = dir(nn.Module)
print([item for item in e if item not in f])

It gives

['_backward_hooks', '_buffers', '_forward_hooks', '_forward_pre_hooks', '_load_state_dict_pre_hooks', '_modules', '_parameters', '_state_dict_hooks', 'training']

why these are only available for the object not class ?
What's special about these attributes?


title: Pytorch: Increase indice of tensor to specific size

body:
I have a tensor of [20, 3, 32, 32]
I want to increase the 3 to 64 ([20,64,32,32], where 20 is the batch size). I tried the repeat function. But that only gives me 63 or 66, because with repeat you can only tile (multiply) indices.
How can I solve this?
Thanks!


title: Linear relation between batch size and inference time per batch

body:
I made the following observation which I can not follow. I am using the deeplabv3_resnet50 model from torchvision and run it in eval mode with different batch sizes. The runtime including a torch.synchronize() are almost linear with the batch size. That means that the rate of images/second is almost constant. I also tried out different settings of cudnn.benchnmark = True/False. I was hoping to see a constant inference time per batch over all batch sizes because they are running in parallel on the gpu. Am I getting anything wrong here?

2 - 0.02s

4 - 0.031s

8 - 0.05s

16 - 0.094s

32 - 0.178s

relevant code:

           with torch.no_grad():
                inputs, side_data = self.create_input(sample) # inputs.to(device) happens in here
                torch.cuda.synchronize()
                labels = sample["label"]
                start = time.time()
                outputs = self.model(inputs)
                torch.cuda.synchronize()
                print(time.time() - start)
                preds = torch.argmax(outputs, 1)

Thanks for your help.
Markus

torch=1.4.0, CUDA Version: 10.0, Tesla V100


title: SSD’s loss not decreasing in PyTorch

body:
I am implementing SSD(Single shot detector) to study in PyTorch.
However, my custom training loss didn't decrease...
I've searched and tried various solution for week, but problem is still remaining.

What should I do?
My loss function is incorrect?

Here is my SSD300 model

SSD300(
  (feature_layers): ModuleDict(
    (conv1_1): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1_1): ReLU()
    (conv1_2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu1_2): ReLU()
    (pool1): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
    (conv2_1): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu2_1): ReLU()
    (conv2_2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu2_2): ReLU()
    (pool2): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
    (conv3_1): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3_1): ReLU()
    (conv3_2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3_2): ReLU()
    (conv3_3): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu3_3): ReLU()
    (pool3): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=True)
    (conv4_1): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu4_1): ReLU()
    (conv4_2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu4_2): ReLU()
    (conv4_3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu4_3): ReLU()
    (pool4): MaxPool2d(kernel_size=(2, 2), stride=(2, 2), padding=0, dilation=1, ceil_mode=False)
    (conv5_1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5_1): ReLU()
    (conv5_2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5_2): ReLU()
    (conv5_3): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (relu5_3): ReLU()
    (pool5): MaxPool2d(kernel_size=(3, 3), stride=(1, 1), padding=1, dilation=1, ceil_mode=False)
    (conv6): Conv2d(512, 1024, kernel_size=(3, 3), stride=(1, 1), padding=(6, 6), dilation=(6, 6))
    (relu6): ReLU()
    (conv7): Conv2d(1024, 1024, kernel_size=(1, 1), stride=(1, 1))
    (relu7): ReLU()
    (conv8_1): Conv2d(1024, 256, kernel_size=(1, 1), stride=(1, 1))
    (relu8_1): ReLU()
    (conv8_2): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
    (relu8_2): ReLU()
    (conv9_1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1))
    (relu9_1): ReLU()
    (conv9_2): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
    (relu9_2): ReLU()
    (conv10_1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
    (relu10_1): ReLU()
    (conv10_2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1))
    (relu10_2): ReLU()
    (conv11_1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1))
    (relu11_1): ReLU()
    (conv11_2): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1))
    (relu11_2): ReLU()
  )
  (localization_layers): ModuleDict(
    (loc1): Sequential(
      (l2norm_loc1): L2Normalization()
      (conv_loc1): Conv2d(512, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc1): ReLU()
    )
    (loc2): Sequential(
      (conv_loc2): Conv2d(1024, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc2): ReLU()
    )
    (loc3): Sequential(
      (conv_loc3): Conv2d(512, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc3): ReLU()
    )
    (loc4): Sequential(
      (conv_loc4): Conv2d(256, 24, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc4): ReLU()
    )
    (loc5): Sequential(
      (conv_loc5): Conv2d(256, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc5): ReLU()
    )
    (loc6): Sequential(
      (conv_loc6): Conv2d(256, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_loc6): ReLU()
    )
  )
  (confidence_layers): ModuleDict(
    (conf1): Sequential(
      (l2norm_conf1): L2Normalization()
      (conv_conf1): Conv2d(512, 84, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf1): ReLU()
    )
    (conf2): Sequential(
      (conv_conf2): Conv2d(1024, 126, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf2): ReLU()
    )
    (conf3): Sequential(
      (conv_conf3): Conv2d(512, 126, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf3): ReLU()
    )
    (conf4): Sequential(
      (conv_conf4): Conv2d(256, 126, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf4): ReLU()
    )
    (conf5): Sequential(
      (conv_conf5): Conv2d(256, 84, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf5): ReLU()
    )
    (conf6): Sequential(
      (conv_conf6): Conv2d(256, 84, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
      (relu_conf6): ReLU()
    )
  )
  (predictor): Predictor()
)

My loss function is defined as;

class SSDLoss(nn.Module):
    def __init__(self, alpha=1, matching_func=None, loc_loss=None, conf_loss=None):
        super().__init__()

        self.alpha = alpha
        self.matching_strategy = matching_strategy if matching_func is None else matching_func
        self.loc_loss = LocalizationLoss() if loc_loss is None else loc_loss
        self.conf_loss = ConfidenceLoss() if conf_loss is None else conf_loss

    def forward(self, predicts, gts, dboxes):
        """
        :param predicts: Tensor, shape is (batch, total_dbox_nums, 4+class_nums=(cx, cy, w, h, p_class,...)
        :param gts: Tensor, shape is (batch*bbox_nums(batch), 1+4+class_nums) = [[img's_ind, cx, cy, w, h, p_class,...],..
        :param dboxes: Tensor, shape is (total_dbox_nums, 4=(cx,cy,w,h))
        :return:
            loss: float
        """
        # get predict's localization and confidence
        pred_loc, pred_conf = predicts[:, :, :4], predicts[:, :, 4:]

        # matching
        pos_indicator, gt_loc, gt_conf = self.matching_strategy(gts, dboxes, batch_num=predicts.shape[0], threshold=0.5)

        # calculate ground truth value considering default boxes
        gt_loc = gt_loc_converter(gt_loc, dboxes)

        # Localization loss
        loc_loss = self.loc_loss(pos_indicator, pred_loc, gt_loc)

        # Confidence loss
        conf_loss = self.conf_loss(pos_indicator, pred_conf, gt_conf)

        return conf_loss + self.alpha * loc_loss

class LocalizationLoss(nn.Module):
    def __init__(self):
        super().__init__()
        self.smoothL1Loss = nn.SmoothL1Loss(reduction='none')

    def forward(self, pos_indicator, predicts, gts):
        N = pos_indicator.sum()

        total_loss = self.smoothL1Loss(predicts, gts).sum(dim=-1) # shape = (batch num, dboxes num)
        loss = total_loss.masked_select(pos_indicator)

        return loss.sum() / N

class ConfidenceLoss(nn.Module):
    def __init__(self, neg_factor=3):
        """
        :param neg_factor: int, the ratio(1(pos): neg_factor) to learn pos and neg for hard negative mining
        """
        super().__init__()
        self.logsoftmax = nn.LogSoftmax(dim=-1)
        self._neg_factor = neg_factor

    def forward(self, pos_indicator, predicts, gts):

        loss = (-gts * self.logsoftmax(predicts)).sum(dim=-1) # shape = (batch num, dboxes num)

        N = pos_indicator.sum()
        neg_indicator = torch.logical_not(pos_indicator)

        pos_loss = loss.masked_select(pos_indicator)
        neg_loss = loss.masked_select(neg_indicator)

        neg_num = neg_loss.shape[0]
        neg_num = min(neg_num, self._neg_factor * N)

        _, topk_indices = torch.topk(neg_loss, neg_num)
        neg_loss = neg_loss.index_select(dim=0, index=topk_indices)

        return (pos_loss.sum() + neg_loss.sum()) / N

loss output is below;

Training... Epoch: 1, Iter: 1,   [32/21503   (0%)]  Loss: 28.804445
Training... Epoch: 1, Iter: 10,  [320/21503  (1%)]  Loss: 12.880742
Training... Epoch: 1, Iter: 20,  [640/21503  (3%)]  Loss: 15.932519
Training... Epoch: 1, Iter: 30,  [960/21503  (4%)]  Loss: 14.624641
Training... Epoch: 1, Iter: 40,  [1280/21503     (6%)]  Loss: 16.301014
Training... Epoch: 1, Iter: 50,  [1600/21503     (7%)]  Loss: 15.710087
Training... Epoch: 1, Iter: 60,  [1920/21503     (9%)]  Loss: 12.441727
Training... Epoch: 1, Iter: 70,  [2240/21503     (10%)] Loss: 12.283393
Training... Epoch: 1, Iter: 80,  [2560/21503     (12%)] Loss: 12.272835
Training... Epoch: 1, Iter: 90,  [2880/21503     (13%)] Loss: 12.273635
Training... Epoch: 1, Iter: 100,     [3200/21503     (15%)] Loss: 12.273409
Training... Epoch: 1, Iter: 110,     [3520/21503     (16%)] Loss: 12.266172
Training... Epoch: 1, Iter: 120,     [3840/21503     (18%)] Loss: 12.272820
Training... Epoch: 1, Iter: 130,     [4160/21503     (19%)] Loss: 12.274920
Training... Epoch: 1, Iter: 140,     [4480/21503     (21%)] Loss: 12.275247
Training... Epoch: 1, Iter: 150,     [4800/21503     (22%)] Loss: 12.273258
Training... Epoch: 1, Iter: 160,     [5120/21503     (24%)] Loss: 12.277486
Training... Epoch: 1, Iter: 170,     [5440/21503     (25%)] Loss: 12.266512
Training... Epoch: 1, Iter: 180,     [5760/21503     (27%)] Loss: 12.265674
Training... Epoch: 1, Iter: 190,     [6080/21503     (28%)] Loss: 12.265306
Training... Epoch: 1, Iter: 200,     [6400/21503     (30%)] Loss: 12.269717
Training... Epoch: 1, Iter: 210,     [6720/21503     (31%)] Loss: 12.274122
Training... Epoch: 1, Iter: 220,     [7040/21503     (33%)] Loss: 12.263970
Training... Epoch: 1, Iter: 230,     [7360/21503     (34%)] Loss: 12.267252


title: Pytorch Multiclass Logistic Regression Type Errors

body:
I'm new to ML and even more naive with Pytorch. Here's the problem. (I've skipped certain parts like the random_split() which seem to work just fine)

I've to predict wine quality (red) which from the dataset is the last column with 6 classes

That's what my dataset looks like

The link to the dataset (winequality-red.csv)

features = df.drop(['quality'], axis = 1)
targets = df.iloc[:, -1] # theres 6 classes

dataset = TensorDataset(torch.Tensor(np.array(features)).float(), torch.Tensor(targets).float())
# here's where I think the error might be, but I might be wrong

batch_size = 8
# Dataloader

train_loader = DataLoader(train_ds, batch_size, shuffle = True)
val_loader = DataLoader(val_ds, batch_size)
test_ds = DataLoader(test_ds, batch_size)

input_size = len(df.columns) - 1
output_size = 6
threshold = .5

class WineModel(nn.Module):

  def __init__(self):
    super().__init__()
    self.linear = nn.Linear(input_size, output_size)

  def forward(self, xb):
    out = self.linear(xb)
    return out

model = WineModel()
n_iters = 2000
num_epochs = n_iters / (len(train_ds) / batch_size)
num_epochs = int(num_epochs)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2)

# the part below returns the error on running
iter = 0
for epoch in range(num_epochs):
  for i, (x, y) in enumerate(train_loader):
    optimizer.zero_grad()
    outputs = model(x)
    loss = criterion(outputs, y)
    loss.backward()
    optimizer.step()

RuntimeError: expected scalar type Long but found Float

Hopefully that is sufficient info


title: I am trying to implement LSTM in pytorch to predict coronavirus

body:
I am trying to implement LSTM bases corona virus predictor for Pakistan using countries with same spread trend as Pakistan to train my model. But my model is showing poor output. I have tried changing trainwindow size, hidden layer size but nothing is helping.

#defining LSTM
    class LSTM(nn.Module):
        def __init__(self, input_size=1, hidden_layer_size=100, output_size=1):
            super().__init__()
            self.hidden_layer_size = hidden_layer_size

            self.lstm = nn.LSTM(input_size, hidden_layer_size)

            self.linear = nn.Linear(hidden_layer_size, output_size)

            self.hidden_cell = (torch.zeros(1,1,self.hidden_layer_size),
                                torch.zeros(1,1,self.hidden_layer_size))

        def forward(self, input_seq):
            lstm_out, self.hidden_cell = self.lstm(input_seq.view(len(input_seq) ,1, -1), self.hidden_cell)
            predictions = self.linear(lstm_out.view(len(input_seq), -1))
            return predictions[-1]
    #model training function
    def train(train_data_normalized, model, optimizer, train_window):

      epochs = 150
      train_inout_seq = create_inout_sequences(train_data_normalized, train_window)
      for i in range(epochs):
          for seq, labels in train_inout_seq:
              optimizer.zero_grad()
              model.hidden_cell = (torch.zeros(1, 1, model.hidden_layer_size),
                              torch.zeros(1, 1, model.hidden_layer_size))

              y_pred = model(seq)

              single_loss = loss_function(y_pred, labels)
              single_loss.backward()
              optimizer.step()

          if i%25 == 1:
              print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

      print(f'epoch: {i:3} loss: {single_loss.item():10.10f}')
      return model

    #setting up model
    model = LSTM()
    loss_function = nn.MSELoss()
    optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

    #training model with data
    for x in country:
        print(x)
        X = torch.FloatTensor(train_data_scaled[x].values).view(-1)
        model = train(X, model, optimizer, train_window)

My test_data_scaled contains following data. 'country' array has column names of test_data_scaled.[Screenshot of the output is also shared.]1


title: The DNN network requires the input data is 2D, but my training data is rgb (3D)

body:
I'm new to the deep learning. I'm trying to train an online-setting deep neural network on ucf101 dataset. Before the training, I read the usage example given by author. The dnn network requires the training data in 2D. But the video frames extracted from the video data is 3D and they are rgb images. For that requirement, I reshape each image to tensors in dimension (1, WxHxC), but the training accuracy is too low. what should I do?
The first code cell is usage example. The second is mine and the third is training loss. The training data in usage example is (15000,10), which means dimension of each sample is (1,10). My training data's dimension is (160,160,10)
onn_network = ONN(features_size=10, max_num_hidden_layers=5,
                  qtd_neuron_per_hidden_layer=40, n_classes=10)
X, Y = make_classification(n_samples=50000, n_features=10, n_informative=4, n_redundant=0, n_classes=10,
                           n_clusters_per_class=1, class_sep=3)
X_train, X_test, y_train, y_test = train_test_split(X,Y, test_size=0.3,
                                                    random_state=42, shuffle=True)

for i in range(len(X_train)):
  onn_network.partial_fit(np.asarray([X_train[i, :]]), np.asarray([y_train[i]]))

  if i % 1 == 0:
    predictions = onn_network.predict(X_test)
    print(" Online Accuracy: {}".format(balanced_accuracy_score(y_test, predictions)))

X_train=np.load('ucf101_X_train_3channels.npy')
X_test=np.load('ucf101_X_test_3channels.npy')
y_train=np.load('ucf101_y_train_3channels.npy')
y_test=np.load('ucf101_y_test_3channels.npy')
X_test=X_test.reshape(7385,160*160*3)
onn_network = ONN(features_size=160*160*3, max_num_hidden_layers=20, qtd_neuron_per_hidden_layer=100,
                  n_classes=101)
for i in range(len(X_train)):

    onn_network.partial_fit(np.asarray([X_train[i]]).reshape(1,160*160*3),
                            np.asarray(y_train[i]).reshape(1,))

    if i % 100 == 0:
        predictions = onn_network.predict(X_test)
        print("Online Accuracy: {}".format(balanced_accuracy_score(y_test, predictions)))

Online Accuracy: 0.0244914025343677
Online Accuracy: 0.02107729631350257
Online Accuracy: 0.02098160986186016
Online Accuracy: 0.025212566301405722
Online Accuracy: 0.02635510928009669
Online Accuracy: 0.025764109717447806
Online Accuracy: 0.026325141135435725
Online Accuracy: 0.018179492570127884
Online Accuracy: 0.025331778179893114
Online Accuracy: 0.02743639553656709
WARNING: Set 'show_loss' to 'False' when not debugging. It will deteriorate the fitting performance.
Alpha:[0.18638033 0.07175495 0.04563371 ... 0.04181847 0.04256647 0.03721857]
Training Loss: 4.5786605
Online Accuracy: 0.028941511549095487
Online Accuracy: 0.031018418116356354
Online Accuracy: 0.01979826579435767
Online Accuracy: 0.03229507089186072
Online Accuracy: 0.024989337301466186
Online Accuracy: 0.03374924564377245


title: How to deploy a Pytorch model in IBM Watson Machine learning using Python Client

body:
I am not able to find any good reference for Deployment of a PyTorch Model in IBM Watson .
I have created Bert Model from Hugging face transformers library and implemented using Pytorch . Now i need to deploy the PyTorch model in IBM watson for real time prediction .
I have searched a lot but couldn't find any good reference of Deployment Steps to follow .


title: Pytorch Neural Network Errors

body:
I am trying to compute the loss and accuracy of a certain  machine learning model by using Pytorch and I am having trouble initializing the dataset so that it can run. Using the Moon dataset, I am getting a few errors when I run the code. I first initialize the dataset:
X, y = make_moons(200, noise=0.2, random_state=42)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=1, stratify = y)
x, y = Variable (torch.from_numpy(X_train)).float(), Variable(torch.from_numpy(y_train)).float()

and then when I run the Neural Network:
    def __init__(self):
        super(SoftmaxRegression, self).__init__()
        self.fc = nn.Linear(200, 1)
        self.softmax = nn.Softmax()

    def forward(self, x):
        x = self.fc(x)
        x = self.softmax(x)
        return x

I get the following errors:
serWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
x = F.softmax(self.layer(x))
ret = torch._C._nn.nll_loss(input, target, weight,  _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 1 is out of bounds.
How can I fix this so that it can run the dataset and output the loss and accuracy?


title: How to slice Kinetics400 training dataset? (pytorch)

body:
I am trying to run the official script for video classification.
I want to tweak some functions and running through all examples would cost me too much time.
I wonder how can I slice the training kinetics dataset based on that script.
This is the code I added before
train_sampler = RandomClipSampler(dataset.video_clips, args.clips_per_video)
in the script: (let's say I just want to run 100 examples.)
tr_split_len = 100
dataset = torch.utils.data.random_split(dataset, [tr_split_len, len(dataset)-tr_split_len])[0]

Then when hitting train_sampler = RandomClipSampler(dataset.video_clips, args.clips_per_video)
, it pops out the error:
AttributeError: 'Subset' object has no attribute 'video_clips'

Yeah, so the type of dataset converts from torchvision.datasets.kinetics.Kinetics400 to torch.utils.data.dataset.Subset.
I understand. So how can I do it? (hopefully not the way using break in the dataloader loop).
Thanks.


title: Implementations and strategies for fast 2D interpolation from irregularly spaced points

body:
Given a large (~10 million) number of irregularly spaced points in two dimensions, where each point has some intensity ("weight") associated with it, what existing python implementations are there for interpolating the value at:

a specific point at some random position (i.e. point = (0.5, 0.8))
a large number of points at random positions (i.e. points = np.random.random((1_000_000, 2)))
a regular grid at integer positions (i.e. np.indices((1000, 1000)).T)

I am aware that Delaunay triangulation is often used for this purpose. Are there alternatives to doing it this way?
Do any solutions take advantage of multiple CPU cores or GPUs?
As an example, here is an approach using scipy's LinearNDInterpolator. It does not appear to use more than one CPU core.
There are also other options in scipy, but with this question I am especially interested in hearing about other solutions than the ones in scipy.
# The %time tags are IPython magic functions that time that specific line
dimension_shape = (1000, 1000) # we spread the random [0-1] over [0-1000] to avoid floating point errors
N_points = dimension_shape[0] * dimension_shape[1]

known_points = np.random.random((N_points, 2)) * dimension_shape
known_weights = np.random.random((N_points,))

unknown_point = (0.5, 0.8)
unknown_points = np.random.random((N_points, 2)) * dimension_shape
unknown_grid = np.indices(dimension_shape, dtype=float).T.reshape((-1, 2)) # reshape to a list of 2D points

%time tesselation = Delaunay(known_points) # create grid to know neighbours # 6 sec
%time interp_func = LinearNDInterpolator(tesselation, known_weights) # 1 ms
%time interp_func(unknown_point) # 2 sec # run it once because the scipy function needs to compile

%time interp_func(unknown_point) # ~ns
%time interp_func(unknown_grid) # 400 ms
%time interp_func(unknown_points) # 1 min 13 sec

# Below I sort the above `unknown_points` array, and try again
%time ind = np.lexsort(np.transpose(unknown_points)[::-1]) # 306 ms
unknown_points_sorted = unknown_points[ind].copy()

%time interp_func(unknown_points_sorted) # 19 sec <- much less than 1 min!

In the above code, things that take an appreciable amount of time are the construction of the Delaunay grid, and interpolation on a non-regular grid of points. Note that sorting the non-regular points first results in a significant speed improvement!
Do not feel the need to give a complete answer from the start. Tackling any aspect of the above is welcome.


title: compute accuracy of Band RNN

body:
So I am trying to figure out how to compute the accuracy of a BandRNN.
BandRnn is a diagonalRNN model with a different number of connections per neuron. For example:

here C is the number of connections per neuron.
My current model training is as follows:
model = ModelLSTM(m, k).to(device)

model.train()

opt = torch.optim.Adam(model.parameters(), lr=args.lr)

best_test = 1e7
best_validation = 1e7

for ep in range(1, args.epochs + 1):

init_time = datetime.now()
processed = 0
step = 1

for batch_idx, (batch_x, batch_y, len_batch) in enumerate(train_loader):
    batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)

    opt.zero_grad()

    logits = model(batch_x)

    loss = model.loss(logits, batch_y, len_batch)

    acc = sum(logits == batch_y) * 1.0 / len(logits)
    print(acc)

    loss.backward()

    if args.clip > 0:
        nn.utils.clip_grad_norm_(model.parameters(), args.clip)

    opt.step()

    processed += len(batch_x)
    step += 1
    print("   batch_idx {}\tLoss: {:.2f} ".format(batch_idx, loss))

print("Epoch {}, LR {:.5f} \tLoss: {:.2f} ".format(ep, opt.param_groups[0]['lr'], loss))

And my model test is as follows:
model.eval()
with torch.no_grad():

for batch_x, batch_y, len_batch in test_loader:
    batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)
    logits = model(batch_x)
    loss_test = model.loss(logits, batch_y, len_batch)

    acc = sum(logits == batch_y) * 1.0 / len(logits)

for batch_x, batch_y, len_batch in val_loader:
    batch_x, batch_y, len_batch = batch_x.to(device), batch_y.to(device), len_batch.to(device)
    logits = model(batch_x)
    loss_val = model.loss(logits, batch_y, len_batch)

if loss_val < best_validation:
    best_validation = loss_val.item()
    best_test = loss_test.item()

print()
print("Val:  Loss: {:.2f}\tBest: {:.2f}".format(loss_val, best_validation))
print("Test: Loss: {:.2f}\tBest: {:.2f}".format(loss_test, best_test))
print()

model.train()

I am struggling with thinking about a way to compute the accuracy of this model and I would like to receive some suggestions about a way to do so.
Thank you.