title: Replacing the model in PreSumm with LegalBert

body:
I have replaced the LegalBert model in the code of Presumm app and when I run the code the summaries that are produced are empty.
When I am installing the PResumm app from https://github.com/nlpyang/PreSumm when using the torch==1.1.0 I get the error below. I can provide more information if necessary.
Can you point me to what could be the issue?
[2021-01-28 16:52:31,419 INFO] https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt not found in cache or force_download set to True, downloading to /tmp/tmp1cak34np
100% 231508/231508 [00:00<00:00, 1209050.34B/s]
[2021-01-28 16:52:31,922 INFO] copying /tmp/tmp1cak34np to cache at ../temp/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
[2021-01-28 16:52:31,922 INFO] creating metadata file for ../temp/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
[2021-01-28 16:52:31,922 INFO] removing temp file /tmp/tmp1cak34np
[2021-01-28 16:52:31,923 INFO] loading file https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt from cache at ../temp/26bc1ad6c0ac742e9b52263248f6d0f00068293b33709fae12320c0e35ccfbbb.542ce4285a40d23a559526243235df47c5f75c197f04f37d1a0c124c32c9a084
gpu_rank 0
[2021-01-28 16:52:31,966 INFO] * number of parameters: 180222522
[2021-01-28 16:52:31,966 INFO] Start training...
Traceback (most recent call last):
  File "/usr/lib/python3.6/tarfile.py", line 188, in nti
    s = nts(s, "ascii", "strict")
  File "/usr/lib/python3.6/tarfile.py", line 172, in nts
    return s.decode(encoding, errors)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb in position 1: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/tarfile.py", line 2299, in next
    tarinfo = self.tarinfo.fromtarfile(self)
  File "/usr/lib/python3.6/tarfile.py", line 1093, in fromtarfile
    obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
  File "/usr/lib/python3.6/tarfile.py", line 1035, in frombuf
    chksum = nti(buf[148:156])
  File "/usr/lib/python3.6/tarfile.py", line 191, in nti
    raise InvalidHeaderError("invalid header")
tarfile.InvalidHeaderError: invalid header

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 556, in _load
    return legacy_load(f)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 467, in legacy_load
    with closing(tarfile.open(fileobj=f, mode='r:', format=tarfile.PAX_FORMAT)) as tar, \
  File "/usr/lib/python3.6/tarfile.py", line 1591, in open
    return func(name, filemode, fileobj, **kwargs)
  File "/usr/lib/python3.6/tarfile.py", line 1621, in taropen
    return cls(name, mode, fileobj, **kwargs)
  File "/usr/lib/python3.6/tarfile.py", line 1484, in __init__
    self.firstmember = self.next()
  File "/usr/lib/python3.6/tarfile.py", line 2311, in next
    raise ReadError(str(e))
tarfile.ReadError: invalid header

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 122, in <module>
    train_abs(args, device_id)
  File "/src/train_abstractive.py", line 275, in train_abs
    train_abs_single(args, device_id)
  File "/src/train_abstractive.py", line 336, in train_abs_single
    trainer.train(train_iter_fct, args.train_steps)
  File "/src/models/trainer.py", line 133, in train
    train_iter = train_iter_fct()
  File "/src/train_abstractive.py", line 315, in train_iter_fct
    shuffle=True, is_test=False)
  File "/src/models/data_loader.py", line 136, in __init__
    self.cur_iter = self._next_dataset_iterator(datasets)
  File "/src/models/data_loader.py", line 156, in _next_dataset_iterator
    self.cur_dataset = next(dataset_iter)
  File "/src/models/data_loader.py", line 90, in load_dataset
    yield _lazy_dataset_loader(pt, corpus_type)
  File "/src/models/data_loader.py", line 78, in _lazy_dataset_loader
    dataset = torch.load(pt_file)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)
  File "/usr/local/lib/python3.6/dist-packages/torch/serialization.py", line 560, in _load
    raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name))
RuntimeError: /pretrained_files/bert/temp.train.0.bert.pt is a zip archive (did you mean to use torch.jit.load()?)


title: Getting a package not missing in current linux64 channels

body:
I am trying to install mkl-include, pytorch and cuda-toolkit using the following -
conda install -y mkl-include
conda install -y pytorch=1.0.1 torchvision cudatoolkit=10.0 -c pytorch

I am on Ubuntu 18.04.5 LTS and here's the output of my conda info -
Current conda install:

               platform : linux-64
          conda version : 4.3.16
       conda is private : False
      conda-env version : 4.3.16
    conda-build version : 3.17.8
         python version : 2.7.16.final.0
       requests version : 2.25.1
       root environment : /home/keshav/anaconda2  (writable)
    default environment : /home/keshav/anaconda2
       envs directories : /home/keshav/anaconda2/envs
                          /home/keshav/.conda/envs
          package cache : /home/keshav/anaconda2/pkgs
                          /home/keshav/.conda/pkgs
           channel URLs : https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : None
           offline mode : False
             user-agent : conda/4.3.16 requests/2.25.1 CPython/2.7.16 Linux/5.4.0-65-generic debian/buster/sid glibc/2.27
                UID:GID : 1000:1000

The error that I am getting with all these installations is this -
PackageNotFoundError: Package missing in current linux-64 channels:

How I can rectify this. I tried uninstalling and reinstalling everything, including Ubuntu.


title: How to do elementwise comparison sum without for loop

body:
The for loop makes my program very slow. I would've used np.sum(target==output) but I need the argmax value for each row in the output. How can I speed this up?
The output is a tensor data type
for i, x in enumerate(target):
  if target[i] == torch.argmax(output[i]):
    correct_class += 1


title: Problem with computing loss in feed forward NN

body:
I'm trying to create a simple feed forward NN, but when computing the loss it is returning NaN. Any advice?
Here is my architecture:
class Net(nn.Module):
    def __init__(self,k):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(k, 5) # 1st hidden layer takes an input of size k
        self.fc2 = nn.Linear(5, 3) # Output layer has a size of 3 neurons

    def forward(self, x):
        x = F.relu(self.fc1(x)) # ReLu activation for 1st hidden layer
        x = F.softmax(self.fc2(x), dim=1) # Softmax activation for output layer
        return x

model = Net(300) # Create model for an embedding of size k=300

Here is my optimizer and loss fn:
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)
loss_fn = nn.CrossEntropyLoss()

I was running a check over a single epoch to see what was happening and this is what happened:
y_pred = model(x_train) # Create model using training data
loss = loss_fn(y_pred, y_train) # Compute loss on training data

print(y_pred)
print(y_train)
print(loss_fn(y_pred, y_train))

The printing returns the following output:
tensor([[0.3597, 0.2954, 0.3449],         [0.3615, 0.2955, 0.3430],         [0.3600, 0.2954, 0.3446],         ...,         [0.3590, 0.2953, 0.3457],         [0.3603, 0.2955, 0.3442],         [0.3605, 0.2955, 0.3441]], grad_fn=<SoftmaxBackward>)
tensor([0, 0, 0,  ..., 0, 2, 0])
tensor(nan, grad_fn=<NllLossBackward>)

My Tensors are of length 300 as I am passing in word embeddings with 300 dimensions.


title: torch : Dijkstra's algorithm

body:
I am working on 3D point clouds. I have the SPARSE MATRIX representation of the graph structure of the point cloud (like csr_matrix in scipy.sparse). I want to club together the points that are within certain threshold of the Geodesic distance (approximated by the path length in the graph) and process them together. TO FIND such points, I need to run some shortest path finding algorithm like Dijkstra's. In a nutshell, my idea is like this

Sample K points out of N points (that I could do using Furthest Point Sampling)
Find the nearest Geodesic neighbours (using BackProp supported algorithm) for each of K points
Process the neighbours for each point using some Neural Network

This will go in my forward function.
Is there a way to implement Dijkstra’s in my functionality?
Or any other idea that I can implement?
Thank you very much!


title: Backwards Propagation Error of Inplace Operation in PyTorch

body:
Preliminary
In my preliminary model, I use an RNN Cell to learn user embedding.
I take the embedding of time t-1 as RNN input,
and output the embedding of time t.
Then I can use the embedding of time t to learn the embedding of time t+1, and so on.
For each batch, I will get a tensor rnn_output, which includes the embedding of unique users.
Hereafter, I want to add an attention mechanism to my model.
But I don't want to limit t.
So, I use a list of tensors rnn_history to remember the embedding history of each user in the beginning of each epoch,
rnn_history = [torch.zeros((0, args.embedding_dim)).cuda()] * user_num

and use these tensors to compute attention in each batch.
First, I added the last output of embedding into rnn_history.
Then I took the overall history of the user to compute attention.
Finally, I put the attention embedding back to the tensor rnn_output.
for e in range(len(rnn_output)):
    rnn_history[batch_id[e]] = torch.cat([rnn_history[batch_id[e]], rnn_output)
    rnn_output[e,:] = model.attention(rnn_history[batch_id[e]])

And here is how attention works in my model.
def attention(self, embeddings):
    q = torch.mm(embeddings[-1:], self.w_q)
    k = torch.mm(embeddings, self.w_k)
    v = torch.mm(embeddings, self.w_v)
    alpha = torch.softmax(torch.matmul(q, k.T) / self.embedding_dim, dim=1)
    return torch.sum((v.T * alpha).T, dim=0, keepdim=True)

Problem
I got the error below after the attention mechanism was added.
RunTimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [301, 128]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

I thought the error that happened may be due to this line.
rnn_output[e,:] = model.attention(rnn_history[batch_id[e]])

It might be an inplace operation. Therefore, I rewrite my code.
attention_output = torch.zeros(0, args.embedding_dim)).cuda()
for e in range(len(rnn_output)):
    rnn_history[batch_id[e]] = torch.cat([rnn_history[batch_id[e]], rnn_output)
    attention_output = torch.cat([attention_output, model.attention(rnn_history[batch_id[e]])])

I use a new tensor attention_output to catch the attention output.
In this way, I can replace the inplace operation with the concatenate operation.
However, the error still exists.
Am I misunderstanding the inplace operation?
How can I rewrite my code to solve this problem?


title: How to calculate in_features for every layer in multilayer perceptron?

body:
What ratio between in_features of different layers is appropriate? Should it be integer?
For example, it is a part of the fully connected network for solving CIFAR10:
self.fc1 = nn.Linear(32*32*3, 256)
self.bn1 = nn.BatchNorm1d(256)
self.fc2 = nn.Linear(256, 64)
self.bn2 = nn.BatchNorm1d(64)
self.fc3 = nn.Linear(64, 10)

Why does an author in the tutorial uses 256 nodes in the second layer and 64 in the third? Or there could be absolutely different numbers?


title: How can I solve the shpe and reconstract CNN for this project?

body:
when I train this network on medical images data
-train
-benign
-normal
-cancer
-test
-benign
-normal
-cancer
-valid
-benign
-normal
-cancer
I get an error when I do training
this is data loading.
import os
import torch
from torchvision import datasets, transforms
### TODO: Write data loaders for training, validation, and test sets
## Specify appropriate transforms, and batch_sizes
from PIL import ImageFile
ImageFile.LOAD_TRUNCATED_IMAGES = True

# number of subprocesses to use for data loading
num_workers = 0
# how many samples per batch to load
batch_size = 32

data_transform_train = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])
data_transform_test = transforms.Compose([
    transforms.Resize(234),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
    ])

data_dir = '/content/drive/MyDrive/COVID-19  Database/COVID'
train_dir = os.path.join(data_dir, 'train')
valid_dir = os.path.join(data_dir, 'valid')
test_dir = os.path.join(data_dir, 'test')

train_data = datasets.ImageFolder(train_dir, transform=data_transform_train)
valid_data = datasets.ImageFolder(valid_dir, transform=data_transform_test)
test_data = datasets.ImageFolder(test_dir, transform=data_transform_test)

train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, num_workers=num_workers, shuffle=True)
valid_loader = torch.utils.data.DataLoader(valid_data, batch_size=batch_size, num_workers=num_workers, shuffle=True)
test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, num_workers=num_workers, shuffle=True)

loaders_scratch = {
    'train' : train_loader,
    'valid' : valid_loader,
    'test'  : test_loader
}

make a model here from scratch
import torch.nn as nn
import torch.nn.functional as F

# define the CNN architecture
class Net(nn.Module):
    ### TODO: choose an architecture, and complete the class
    def __init__(self):
        super(Net, self).__init__()
        ## Define layers of a CNN
        self.conv1 = nn.Conv2d(1, 128, 3) #(224-3)/1+1= 222
        self.conv2 = nn.Conv2d(128, 64, 3) #110 after pooling with (2,2) ==>(110-3)/1+1=108
        self.conv3 = nn.Conv2d(64, 64, 3) # 54 after pooling with (2,2) ==> 110/2=54 ==>(54-3)/1+1=52
        self.conv4 = nn.Conv2d(64, 32, 3) # 26 after pooling with (2,2) ==> 52/2=26  ==>(26-3)/1+1=24
        self.conv5 = nn.Conv2d(32, 16, 3) # 12 after pooling with (2,2) ==> 24/2=12 ==> (12-3)/1+1=10
        self.conv6 = nn.Conv2d(16, 8, 3) # 5 after pooling with (2,2) ==> 10/2=2
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(8 * 5 * 5, 160) #8 is a out_channel(number of filter) of last conv layer and 5 is the output of last conv layer after pooling(200 input to fc1)
        self.fc2 = nn.Linear(160, 3) #166 is the output of the fc1 as input to fc2 and 133 output classes
        self.dropout25 = nn.Dropout(p=0.5) # 50% dropout of nodes
        self.softmax = nn.Softmax(dim = 1)

    def forward(self, x):
        ## Define forward behavior
        x = F.relu(self.conv1(x))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = self.pool(F.relu(self.conv4(x)))
        x = self.pool(F.relu(self.conv5(x)))
        x = self.pool(F.relu(self.conv6(x)))
        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = self.dropout25(x)
        x = self.fc2(x)
        x = self.softmax(x)

        return x

#-#-# You so NOT have to modify the code below this line. #-#-#

# instantiate the CNN
model_scratch = Net()

use_cuda = torch.cuda.is_available()

# move tensors to GPU if CUDA is available
if use_cuda:
    model_scratch.cuda()
print(model_scratch)

here I define loss and optimizer
import torch.optim as optim

### TODO: select loss function
criterion_scratch = nn.CrossEntropyLoss()

### TODO: select optimizer
optimizer_scratch = optim.Adam(model_scratch.parameters(), lr = 0.001)

make a training and the error i appear here
import numpy as np
def train(n_epochs, loaders, model, optimizer, criterion,use_cuda,save_path):
  """returns trained model"""
  # initialize tracker for maxi validation loss
  valid_loss_min = np.Inf

    for epoch in range(1, n_epochs+1):
        # initialize variables to monitor training and validation loss
        train_loss = 0.0
        valid_loss = 0.0

        ###################
        # train the model #
        ###################
        model.train()
        for batch_idx, (data, target) in enumerate(loaders['train']):
           # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            ## find the loss and update the model parameters accordingly
            ## record the average training loss, using something like
            ## train_loss = train_loss + ((1 / (batch_idx + 1)) * (loss.data - train_loss))

            optimizer.zero_grad()
            output = model(data)
            loss = criterion(output,target)
            loss.backward()
            optimizer.step()
            train_loss += loss.item()*data.size(0)
        ######################
        # validate the model #
        ######################
        model.eval()
        for batch_idx, (data, target) in enumerate(loaders['valid']):
            # move to GPU
            if use_cuda:
                data, target = data.cuda(), target.cuda()
            ## update the average validation loss

            output = model(data)
            loss = criterion(output,target)

            valid_loss += loss.item()*data.size(0)

        train_loss = train_loss/len(loaders['train'].dataset)
        valid_loss = valid_loss/len(loaders['valid'].dataset)

        # print training/validation statistics
        print('Epoch: {} \tTraining Loss: {:.6f} \tValidation Loss: {:.6f}'.format(
            epoch,
            train_loss,
            valid_loss
            ))

        ## TODO: save the model if validation loss has decreased
        if valid_loss <= valid_loss_min:
            print('Validation loss decreased ({:.6f} --> {:.6f}).  Saving model ...'.format(
            valid_loss_min,
            valid_loss))
            torch.save(model.state_dict(),save_path)
            valid_loss_min = valid_loss
            # return trained model
    return model

# train the model
model_scratch = train(15, loaders_scratch, model_scratch, optimizer_scratch,
                      criterion_scratch, use_cuda, 'model_scratch.pt')

# load the model that got the best validation accuracy
model_scratch.load_state_dict(torch.load('model_scratch.pt'))

and this is an error
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-63f181ccccc5> in <module>()
     66 # train the model
     67 model_scratch = train(15, loaders_scratch, model_scratch, optimizer_scratch,
---> 68                       criterion_scratch, use_cuda, 'model_scratch.pt')
     69
     70 # load the model that got the best validation accuracy

5 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/conv.py in _conv_forward(self, input, weight, bias)
    394                             _pair(0), self.dilation, self.groups)
    395         return F.conv2d(input, weight, bias, self.stride,
--> 396                         self.padding, self.dilation, self.groups)
    397
    398     def forward(self, input: Tensor) -> Tensor:

RuntimeError: Given groups=1, weight of size [128, 1, 3, 3], expected input[32, 3, 224, 224] to have 1 channels, but got 3 channels instead


title: GPU memory usage accumulating when calling function even though tensor isn't stored

body:
Hey I have a probably very simple question, but I couldn't find answers to it when searching. I've been testing how PyTorch manages GPU memory, and found a memory leak that I am not sure how to resolve. Consider this piece of code
a = torch.randn(1000000,10)
a = a.to(device=device) #device here is is my GPU
b = torch.randn(1000000,10)
b = b.to(device= device)

def sum2(c,d):
    return c+d

Now the GPU's memory usage as measured by nvidia-smi and torch.cuda.memory_allocated() increases by 80 MiB (40 MiB each for a and b)
Now if I were to repeatedly call  sum2(a,b), without storing the result (I am doing this in a jupyter notebook, not sure if that is relevant), the GPU memory usage keeps increasing by 40 MiB with every call, indicating that the result somehow is stored in GPU even though the result tensor isn't really referenced and the result is "lost".
This behavior persists even when I wrap the return c+d in a with torch.no_grad():
Any ideas what is going on here and how to make sure that memory doesn't baloon up in this way?


title: Design of the feed forward neural network with multiclass output and low number of inputs

body:
I have a set of data items Xij each having 20 columns  (categorical data) and there are 30 different types of Xij
As an input to NN, I want to provide a combination of these items (8 items = 20*8 = 160 is the size of X vector)(Xi). As a result, I want a multi-class output (Yi). (Yi) is a vector of length 670 (this number may change).
X1 will be labeled as 000000000000000000100000000000000000000000010000000000001 (a total of 670 elements in a vector).
X2 will be labeled as another combination of 0's and 1's with the same length (these points out an index for another dataset with length 670)
In order to train the model, I will provide sample sets of Xij values. I decided 12 X items (12 different combinations of 30 unique items= an input of 12*160). I know this is a low number, but manually labeling is not easy, so I want to keep the number small. If you suggest to use a higher number or some other techniques I would love to listen?
In summary:
Xi (one input sample) has 160 columns consisting of 8 Xij. This may not be relevant to the network design, but I am giving this as an additional information).
Yi (corresponding label for Xi) is a vector of size 670 (may be lower or higher) consisting of 0's and 1's.
My training data would have a size 12 (questionable), that is, I will provide a 12 arrays each having a size 160.
What layers should I put in the NN? I want help in the design of the NN? I have access to pytorch library. I feel that a convolution layer would also be useful, cause not only the selection of Xiis but also the order of Xiis (The places) would change in each Xi.


title: Use PyTorch (android Library) as a dependency in pure Kotlin library

body:
I've created a Kotlin library for segmentation (let's call it MySeg). It should use PyTorch as a dependency.
This MySegshould run on both an Android device, on on a pure Kotlin/Java Desktop application for test purposes.
I've added the PyTorch dependency but the compiler can find references to PyTorch.
This is what my build.gradle dependencies look like:
dependencies {
    implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"
    implementation 'org.pytorch:pytorch_android:1.4.0'
    implementation 'org.pytorch:pytorch_android_torchvision:1.4.0'
    implementation project(path: ':opencv', configuration: 'default')
}

Is there any way I can create this MySeg library, with Torch as a dependency, so that MySeg can be imported both by an Android App and a pure Kotlin Desktop app?


title: Concatination on a 3 dimensional tensor (Tensor Re-Shaping)

body:
I have 2 tensors,
their format currently is [13, 2] respectively. I am trying to combine both into a 3 dimensional tensor with dimensions [2, 13, 2] such that they stack on top of each other, however are separated as batches.
here is an example of one of the tensors in format [13, 2]:
tensor([[[-1.8588,  0.3776],
         [ 0.1683,  0.2457],
         [-1.2740,  0.5683],
         [-1.7262,  0.4350],
         [-1.7262,  0.4350],
         [ 0.1683,  0.2457],
         [-1.0160,  0.5940],
         [-1.3354,  0.5565],
         [-0.7497,  0.5792],
         [-0.2024,  0.4251],
         [ 1.0791, -0.2770],
         [ 0.3032,  0.1706],
         [ 0.8681, -0.1607]])

I would like to maintain the shape, but have them in 2 groups in the same tensor. below is an example of the format I am after:
tensor([[[-1.8588,  0.3776],
         [ 0.1683,  0.2457],
         [-1.2740,  0.5683],
         [-1.7262,  0.4350],
         [-1.7262,  0.4350],
         [ 0.1683,  0.2457],
         [-1.0160,  0.5940],
         [-1.3354,  0.5565],
         [-0.7497,  0.5792],
         [-0.2024,  0.4251],
         [ 1.0791, -0.2770],
         [ 0.3032,  0.1706],
         [ 0.8681, -0.1607]],

         [[-1.8588,  0.3776],
         [ 0.1683,  0.2457],
         [-1.2740,  0.5683],
         [-1.7262,  0.4350],
         [-1.7262,  0.4350],
         [ 0.1683,  0.2457],
         [-1.0160,  0.5940],
         [-1.3354,  0.5565],
         [-0.7497,  0.5792],
         [-0.2024,  0.4251],
         [ 1.0791, -0.2770],
         [ 0.3032,  0.1706],
         [ 0.8681, -0.1607]]])

Anyone have any ideas on how to do this using concatination? I have tried using .unsqueeze when using torch.cat((a, b.unsqueeze(0)), dim=-1) however it changed the format to [13, 4, 1] which is not the format I am after.
solution below works, however, my idea was that I would keep stacking to y via a loop without being restricted by the shape. Sorry for not projecting my idea clearly enough.
They will all be of size [13,2] so it will go up in the form of [1,13,2], [2,13,2], [3,13,2], [4,13,2] etc...


title: How to achieve removing/pruning the near-zero parameters in neural network?

body:
I need to remove the near-zero weights of the Neural network so that the distribution of parameters is far away from the zero point.
The distribution of weights after removing nearzero weights and weight-scaling
I met the problem from this paper: https://ieeexplore.ieee.org/document/7544366
I wonder how can I achieve this in my PyTorch/TensorFlow program, such as use a customized activation layer? Or Define a loss function that punishes the near-zero weight?
Thank you if you can provide any help.


title: Python TypeError: 'Tensor' object is not callable when sorting dictionary

body:
Here is my code. The packages imported are not shown. I am trying to feed the CIFAR-10 test data into alexnet. The dictionary at the end needs to be sorted so I can find the most common classification. Please help, I have tried everything!
............................................................................................................................................................................................................................................................................................................
alexnet = models.alexnet(pretrained=True)

transform = transforms.Compose([            #[1]
 transforms.Resize(256),                    #[2]
 transforms.CenterCrop(224),                #[3]
 transforms.ToTensor(),                     #[4]
 transforms.Normalize(                      #[5]
 mean=[0.485, 0.456, 0.406],                #[6]
 std=[0.229, 0.224, 0.225]                  #[7]
 )])

# Getting the CIFAR-10 dataset
dataset = CIFAR10(root='data/', download=True, transform=transform)
test_dataset = CIFAR10(root='data/', train=False, transform=transform)

classes = dataset.classes
#print(classes)

torch.manual_seed(43)
val_size = 10000
train_size = len(dataset) - val_size

train_ds, val_ds = random_split(dataset, [train_size, val_size])

#print(len(train_ds), len(val_ds))

batch_size=100

train_loader = DataLoader(train_ds, batch_size, shuffle=True, num_workers=8, pin_memory=True)
val_loader = DataLoader(val_ds, batch_size, num_workers=8, pin_memory=True)
test_loader = DataLoader(test_dataset, batch_size, num_workers=8, pin_memory=True)

with open("/home/shaan/Computer Science/CS4442/Ass4/imagenet_classes.txt") as f:
    classes = eval(f.read())

holder = []
dic = {}

current = ''
#data_iter = iter(test_loader)
#images,labels = data_iter.next()

#alexnet.eval()
with torch.no_grad():
    for data in test_loader:
        images, labels = data
        out = alexnet(images)
        #print(out.shape)

        for j in range(0,batch_size):
            sorted, indices = torch.sort(out,descending=True)
            percentage = F.softmax(out,dim=1)[j]*100
            results = [(classes[i.item()],percentage[i].item()) for i in indices[j][:5]]
            holder.append(results[0][0])

holder.sort()
for z in holder:
    if current != z:
        count = 1
        dic[z] = count
        current = z

    else:
        count = count + 1
        dic[z] = count
        current = z

This is where im getting the error:
for w in sorted(dic, key=dic.get, reverse=True):
    print(w, dic[w])


title: PyTorch - Change weights of Conv2d

body:
For some reason, I cannot seem to assign all the weights of a Conv2d layer in PyTorch - I have to do it in two steps. Can anyone help me with what I am doing wrong?
layer = torch.nn.Conv2d(in_channels=1, out_channels=2, kernel_size=(2,2), stride=(2,2))
layer.state_dict()['weight']

gives me a tensor of size (2,1,2,2)
tensor([[[[ 0.4738, -0.2197],
      [-0.3436, -0.0754]]],

    [[[ 0.1662,  0.4098],
      [-0.4306, -0.4828]]]])

When I try to assign weights like so
layer.state_dict()['weight'] = torch.tensor([
  [[[ 1, 2],
    [3, 4]]],

  [[[-1, -2],
    [-3, -4]]]
])

the weights don't change. However, if I do something like this
layer.state_dict()['weight'][0] = torch.tensor([
  [[[1, 2],
    [3, 4]]],
])
layer.state_dict()['weight'][1] = torch.tensor([
  [[[-1, -2],
    [-3, -4]]],
])

The weights change. Why is this so?


title: Facing problem with PyTroch Installation with Anaconda environment

body:
I am trying to install pytorch over my system (Alienware, with 32 GB RAM and GTX1080TI) with Anaconda installation.
I am using this command from the pytorch website

conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0
cudatoolkit=11.1 -c pytorch -c conda-forge

But it's throwing this error-

ERROR conda.core.link:_execute(698): An error occurred while
installing package 'conda-forge::cudatoolkit-10.2.89-hb195166_8'.

Can someone help me fix this issue. I uninstalled all the previous versions of Cuda toolkit as well, but still getting the same error message.


title: Dimensionality of tensor from my WaveNet incompatible with PyTorch cross_entropy function

body:
I've been doing a project regarding making my own WaveNet implementation as Deepmind delivered early in the 2016's in Python.

Preprocessing includes mu law encoding, and one hot encoding. The model itself functions well, my problem lies in the loss function torch.nn.functional.cross_entropy used during training, found here: https://pytorch.org/docs/stable/nn.functional.html
Particularly, the relation between my output and my target tensors, namely
input_tensor.shape = tensor([1, 256, 225332]) # [batch_size, sample_size, audio_length]
output.shape = tensor([1, 256, 225332])

According to F.cross_entropy, I must have output = (N, C) and target = input_tensor = (N).
My supervisor told me to do the following:
output = output.T.reshape(-1, 256) = tensor([225332, 256])
target = input_tensor.T.long() = tensor([225332, 256, 1]) # This needs to be 1-dimensional, help?

For anyone interested in the explicit code, below:
NOTE - the receptive field is not padded, so just for debugging purposes I have subtracted it, while I do know this is not natural.
>>> output.T.reshape(-1, 256).shape
torch.Size([225332, 256])
>>> input_tensor[:, :, model.input_size - model.output_size:].T.shape
torch.Size([225332, 256, 1])
>>> loss = F.cross_entropy(output.T.reshape(-1, 256), input_tensor[:, :, model.input_size - model.output_size:].T.long().to(device))

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.3\plugins\python-ce\helpers\pydev\_pydevd_bundle\pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "C:\Users\JaQtae\anaconda3\envs\CortiGit\lib\site-packages\torch\nn\functional.py", line 2693, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "C:\Users\JaQtae\anaconda3\envs\CortiGit\lib\site-packages\torch\nn\functional.py", line 2388, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: 1D target tensor expected, multi-target not supported

Somewhat of a novice-in-training with ML and AI, particularly the PyTorch library.
Would appreciate any advice regarding how I should tackle this issue.
The training:
model = Wavenet(layers=3,blocks=2,output_size=32).to(device)
model.apply(initialize) # Initialize causalconv1d() with xavier_uniform_ weights and bias of 0.
model.train()

optimizer = optim.Adam(model.parameters(), lr=0.0003)
for i, batch in tqdm(enumerate(train_loader)):
    mu_enc_my_x = encode_mu_law(x=batch, mu=256)
    input_tensor = one_hot_encoding(mu_enc_my_x)

    input_tensor = input_tensor.to(device)
    output = model(input_tensor)
    # TODO: Inspect input/output formats, maybe something wrong....
    loss = F.cross_entropy(output.T.reshape(-1, 256), input_tensor[:,:,model.input_size - model.output_size:].long().to(device)) # subtract receptive field instead of pad it, workaround for quick debugging of loss-issue.
    print("\nLoss:", loss.item())
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if i % 1000 == 0:
        print("\nSaving model")
        torch.save(model.state_dict(), "wavenet.pt")

The purpose is to get my loss function to work properly, so that I can generate sound files. The current ones with my bad loss function obviously return pure noise.
My full model if any help.
"""
Wavenet model

Sources:
https://github.com/kan-bayashi/PytorchWaveNetVocoder/blob/master/wavenet_vocoder/nets/wavenet.py
https://github.com/r9y9/wavenet_vocoder/blob/master/wavenet_vocoder/wavenet.py
https://github.com/Dankrushen/Wavenet-PyTorch/blob/master/wavenet/models.py
https://github.com/vincentherrmann/pytorch-wavenet
"""

from torch import nn
import torch

#TODO: Add local and global conditioning

def initialize(m):
    """
    Initialize CNN with Xavier_uniform weight and 0 bias.
    """
    if isinstance(m, torch.nn.Conv1d):
        nn.init.xavier_uniform_(m.weight)
        nn.init.constant_(m.bias, 0.0)

class CausalConv1d(torch.nn.Module):
    """
    Causal Convolution for WaveNet
    Causality can be introduced with padding as (kernel_size - 1) * dilation (see Keras documentation)
    or it can be introduced as follows according to Golbin.
    https://github.com/golbin/WaveNet/blob/05545339096c3a1d9909d96fb19da4fbae28d8c6/wavenet/networks.py#L38

    Else, look at the following article, several ways to implement it using PyTorch:
    https://github.com/pytorch/pytorch/issues/1333

    - Jakob
    """

    def __init__(self, in_channels, out_channels, kernel_size, dilation = 1, bias = True):
        super(CausalConv1d, self).__init__()
        # padding=1 for same size(length) between input and output for causal convolution
        self.dilation = dilation
        self.kernel_size = kernel_size
        self.in_channels = in_channels
        self.out_channels = out_channels
        self.padding = padding = (kernel_size-1) * dilation # kernelsize = 2, -1 * dilation = 1, = 1. - Jakob.
        self.conv = torch.nn.Conv1d(in_channels, out_channels,
                                    kernel_size, padding=padding, dilation=dilation,
                                    bias=bias)  # Fixed for WaveNet but not sure

    def forward(self, x):
        output = self.conv(x)
        if self.padding != 0:
            output = output[:, :, :-self.padding]
        return output

class Wavenet(nn.Module):

    def __init__(self,
                 layers=3,
                 blocks=2,
                 dilation_channels=32,
                 residual_block_channels=512,
                 skip_connection_channels=512,
                 output_channels=256,
                 output_size=32,
                 kernel_size=3
                 ):

        super(Wavenet, self).__init__()

        self.layers = layers
        self.blocks = blocks
        self.dilation_channels = dilation_channels
        self.residual_block_channels = residual_block_channels
        self.skip_connection_channels = skip_connection_channels
        self.output_channels = output_channels
        self.kernel_size = kernel_size
        self.output_size = output_size

        # initialize dilation variables
        receptive_field = 1
        init_dilation = 1

        # List of layers and connections
        self.dilations = []
        self.residual_convs = nn.ModuleList()
        self.filter_conv_layers = nn.ModuleList()
        self.gate_conv_layers = nn.ModuleList()
        self.skip_convs = nn.ModuleList()

        # First convolutional layer
        self.first_conv = CausalConv1d(in_channels=self.output_channels,
                                    out_channels=residual_block_channels,
                                    kernel_size = 2)

        # Building the Modulelists for the residual blocks
        for b in range(blocks):
            additional_scope = kernel_size - 1
            new_dilation = 1
            for i in range(layers):
                # dilations of this layer
                self.dilations.append((new_dilation, init_dilation))

                # dilated convolutions
                self.filter_conv_layers.append(nn.Conv1d(in_channels=residual_block_channels, out_channels=dilation_channels, kernel_size=kernel_size, dilation=new_dilation))

                self.gate_conv_layers.append(nn.Conv1d(in_channels=residual_block_channels, out_channels=dilation_channels, kernel_size=kernel_size, dilation=new_dilation))

                # 1x1 convolution for residual connection
                self.residual_convs.append(nn.Conv1d(in_channels=dilation_channels, out_channels=residual_block_channels, kernel_size=1))

                # 1x1 convolution for skip connection
                self.skip_convs.append(nn.Conv1d(in_channels=dilation_channels,
                                                 out_channels=skip_connection_channels,
                                                 kernel_size=1))

                # Update receptive field and dilation
                receptive_field += additional_scope
                additional_scope *= 2
                init_dilation = new_dilation
                new_dilation *= 2

        # Last two convolutional layers
        self.last_conv_1 = nn.Conv1d(in_channels=skip_connection_channels,
                                  out_channels=skip_connection_channels,
                                  kernel_size=1)

        self.last_conv_2 = nn.Conv1d(in_channels=skip_connection_channels,
                                    out_channels=output_channels,
                                    kernel_size=1)

        #Calculate model receptive field and the required input size for the given output size
        self.receptive_field = receptive_field
        self.input_size = receptive_field + output_size - 1

    def forward(self, input):

        # Feed first convolutional layer with input
        x = self.first_conv(input)

        # Initialize skip connection
        skip = 0

        # Residual block
        for i in range(self.blocks * self.layers):

            (dilation, init_dilation) = self.dilations[i]

            # Residual connection bypassing dilated convolution block
            residual = x

            # input to dilated convolution block
            filter = self.filter_conv_layers[i](x)
            filter = torch.tanh(filter)
            gate = self.gate_conv_layers[i](x)
            gate = torch.sigmoid(gate)
            x = filter * gate

            # Feed into 1x1 convolution for skip connection
            s = self.skip_convs[i](x)

            #Adding skip & Match size with decreasing dimensionality of x
            if skip is not 0:
                skip = skip[:, :, -s.size(2):]
            skip = s + skip # Sum all skip connections

            # Feed into 1x1 convolution for residual connection
            x = self.residual_convs[i](x)
            #Adding Residual & Match size with decreasing dimensionality of x
            x = x + residual[:, :, dilation * (self.kernel_size - 1):]

            # print(x.shape)

        x = torch.relu(skip)
        #Last conv layers
        x = torch.relu(self.last_conv_1(x))
        x = self.last_conv_2(x)
        soft = torch.nn.Softmax(dim=1)
        x = soft(x)
        return x

EDIT: added code snippet of train for clarity, and full model


title: Inference Discrepancy between PyTorch and DJL implementation via Kotlin

body:
I have a PyTorch model trained on the 17flowers dataset, and converted via PyTorch's tracing to a JIT model. I have tested the inference output for the PyTorch model and the JIT converted model, and the results are equivalent there. This leads me to believe there is an issue with my implementation of the DJL framework.
There is an issue when I attempt to utilized DJL for inference utilizing the converted JIT model, which is necessary for DJL. I am not getting 100% match, which I expected.
The Kotlin implementation for djl.ai is straightforward and essentially follows the instructions here.
I have a sanitized version of the Kotlin code below:
@Throws(IOException::class, ModelException::class, TranslateException::class)
internal fun main(args: Array<String>) {
    val artifactId = "ai.djl.localmodelzoo:torchscript_17flowers"
    val pipeline = Pipeline()
    pipeline.add(CenterCrop(224, 224))
        .add(Resize(224, 224))
        .add(ToTensor())
        .add(Normalize(floatArrayOf(0.485f, 0.456f, 0.406f), floatArrayOf(0.229f, 0.224f, 0.225f)))
    val translator = ImageClassificationTranslator.builder()
        .setPipeline(pipeline)
        .optSynsetArtifactName("synset.txt")
        .optApplySoftmax(true)
        .build();

    System.setProperty("ai.djl.repository.zoo.location","build/pytorch_models/torchscript_17flowers")

    val criteria = Criteria.builder()
        .setTypes(Image::class.java, Classifications::class.java) // defines input and output data type
        .optTranslator(translator)
        .optArtifactId(artifactId) // defines which model to load
        .optProgress(ProgressBar())
        .build()

    val model = ModelZoo.loadModel(criteria)

    // single image test
    var img = ImageFactory.getInstance().fromUrl("https://image.jpg");
    img.getWrappedImage()

    val predictor: Predictor<Image, Classifications> = model.newPredictor()
    val classifications: Classifications = predictor.predict(img)
    val best = classifications.best<Classifications.Classification>()
}

My issue isn't getting things to run so much as it is getting the inference results to match. It is my understanding that they should match, and that Kotlin should work fine as DJL is meant to work for Java. I'm curious if there are any thoughts surrounding this encountered issue.


title: How to create a PyTorch mutable tensor?

body:
I'm trying to create a copy of a tensor that will change if the original changes.
r = torch.tensor(1.0, requires_grad=True)
p = r.clone()
print('before')
print(r)
print(p)
r = r*5
print('after')
print(r)
print(p)

>>>
before
tensor(1., requires_grad=True)
tensor(1.)
after
tensor(5., grad_fn=<MulBackward0>)
tensor(1.)

I tried with clone(), detach(), and even simply p=r, nothing worked.
Update.
Tried view:
r = torch.tensor(1.0, requires_grad=True)
p = r.view(1)
print('before')
print(r)
print(p)
r = r*5
print('after')
print(r)
print(p)

>>>before
tensor(1., requires_grad=True)
tensor([1.], grad_fn=<ViewBackward>)
after
tensor(5., grad_fn=<MulBackward0>)
tensor([1.], grad_fn=<ViewBackward>)


title: Differentiable convolution between two 1d signals in pytorch

body:
i need to implement a convolution between a signal and a window in pytorch and i want it to be differentiable. Since i couldn't find an already existing function for tensors (i could only find the ones with learnable parameters) i wrote one myself but, i'm unable to make it work without breaking the computation graph. How could i do it? The function i made is:
def Convolve(a, b):
  conv=torch.zeros(a.shape[0], a.shape[1], requires_grad=True).clone()
  l=b.shape[0]
  r=int((l-1)/2)
  l2=a.shape[1]
  for x in range(a.shape[0]):#for evry signal
    for x2 in range(a.shape[1]):#for every time instant
      for x4 in range(l):#compute the convolution (for every window value)
        if (x2-r+x4<0 or x2-r+x4>=l2):#if the index is out of bonds the result is 0 (to avoid zero padding)
           conv[x][x2]+=0
        else:
          conv[x][x2-r+x4]+=a[x][x2-r+x4]*b[x4]#otherwise is window*signal
  return conv

Where 'a' is a two dimensional tensor (signal index, time) and 'b' is an Hann window.
The lenght of the window is odd.


title: Is .contiguous().flatten() the same as .view(-1) in PyTorch?

body:
Are these exactly the same?
myTensor.contiguous().flatten()
myTensor.view(-1)

Will they return the same auto grad function etc?


title: Effecient Way to Access an Element of a PyTorch Tensor?

body: I want to extract only the first element of a very large pytorch tensor. I've seen posts talking about options like my_tensor.numpy()[0] or my_tensor.detach().numpy()[0] if I'm using requires_grad. This seems really inefficient just to access one element, especially if my tensor is big. Is there any way around this?


title: How to translate a conv2D in keras or tensorflow which is already implemented in PyTorch?

body:
I have the following function in pytorch implementation for replacing a conv2D layer with 3 different layers:
first_layer = torch.nn.Conv2d(in_channels=3, \
            out_channels=3, kernel_size=1,
            stride=1, padding=0, dilation = (1,1), bias=False)

core_layer = torch.nn.Conv2d(in_channels=3, \
            out_channels=16, kernel_size=(3,3),
            stride=(1,1), padding=(1,1), dilation=(1,1),
            bias=False)

last_layer = torch.nn.Conv2d(in_channels=16, \
                                 out_channels=64], kernel_size=1, stride=1,
                                 padding=0, dilation=(1,1), bias=True)

last_layer.bias.data = layer.bias.data

first_layer.weight.data = \
        torch.transpose(first, 1, 0).unsqueeze(-1).unsqueeze(-1)
last_layer.weight.data = last.unsqueeze(-1).unsqueeze(-1)
core_layer.weight.data = core

new_layers = [first_layer, core_layer, last_layer]
y = nn.Sequential(*new_layers)

where, 'first' represents a random 3 by 3 matrix.
'core' represents a tensor of shape [16,3,3,3]
'last' represents another random matrix of  size (64,16).
When I tried to translate this into keras, I have the following :
first_layer = tf.keras.layers.SeparableConv2D(3, kernel_size=1, strides = (1,1), padding = 'same',  dilation_rate = (1,1), use_bias = False )

core_layer = tf.keras.layers.Conv2D(16, kernel_size=3, strides = (1,1), padding = (1,1), dilation_rate = (1,1), use_bias = False)

last_layer = tf.keras.layers.SeparableConv2D(64, kernel_size=1, strides = (1,1), \
                           padding = 'same', dilation_rate = (1,1), use_bias =True )

first_layer = tf.expand_dims(tf.expand_dims(tf.transpose(first, perm = [1,0]),0),0)

last_layer = tf.expand_dims(tf.expand_dims(last, 0),0)
core_layer = core
new_layers = [first_layer, core_layer, last_layer]

when I tried to get back the weights of the model in keras, I am getting a list with no weights at all. The convolution is not performed. Any idea on how to proceed further/ any other approached of transforming the above pytorch implementation to keras or tensorflow?


title: tf.nn.depth_to_space vs pixel_shuffle

body:
I want to implement a function like tf.nn.depth_to_space, but in Pytorch. Function pixel_shuffle should do the same thing, but I had an error.
x = tf.constant([[[[1, 2, 3, 15], [4, 5, 6, 16]],
      [[7, 8, 9, 17], [10, 11, 12, 13]]]])
y = tf.nn.depth_to_space(x,2)
output:
array([[[[ 1], [ 2], [ 4],[ 5]],
        [[ 3], [15], [ 6], [16]],
        [[ 7], [ 8], [10], [11]],
        [[ 9], [17], [12], [13]]]], dtype=int32)>

Pytorch:
x = torch.Tensor([[[[1, 2, 3, 15], [4, 5, 6, 16]],
      [[7, 8, 9, 17], [10, 11, 12, 13]]]])
pixel_shuffle = torch.nn.PixelShuffle(2)
y = pixel_shuffle(x)

RuntimeError: pixel_shuffle expects its input's 'channel' dimension to be divisible by the square of upscale_factor, but input.size(-3)=2 is not divisible by 4

Where can be a mistake or how can I make the same function in Torch as in tf?
Thanks!


title: PyTorch: sparse softmax as functional?

body:
I have a torch tensor of shape (batch_size, N). I want to apply functional softmax with dim 1 to this tensor, but I also want it to ignore zeros in the tensor and only apply it to non-zero values (the non-zeros in the tensor are positive numbers). I think what I am looking for is the sparse softmax.
I came up with this code: GitHub, but seems like it uses nn.Module instead of functional.
How can I use that sparse softmax as a functional one?


title: How to get dataset from prepare_data() to setup() in PyTorch Lightning

body:
I made my own dataset using NumPy in the prepare_data() methods using the DataModules method of PyTorch Lightning. Now, I want to pass the data into the setup() method to split into training and validation.
import numpy as np
import pytorch_lightning as pl
from torch.utils.data import random_split, DataLoader, TensorDataset
import torch
from torch.autograd import Variable
from torchvision import transforms

np.random.seed(42)

device = 'cuda' if torch.cuda.is_available() else 'cpu'

class DataModuleClass(pl.LightningDataModule):
    def __init__(self):
        super().__init__()
        self.constant = 2
        self.batch_size = 10

    def prepare_data(self):
        a = np.random.uniform(0, 500, 500)
        b = np.random.normal(0, self.constant, len(a))

        c = a + b
        X = np.transpose(np.array([a, b]))

        # Converting numpy array to Tensor
        self.x_train_tensor = torch.from_numpy(X).float().to(device)
        self.y_train_tensor = torch.from_numpy(c).float().to(device)

        training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)

        return training_dataset

    def setup(self):
        data = # What I have to write to get the data from prepare_data()
        self.train_data, self.val_data = random_split(data, [400, 100])

    def train_dataloader(self):
        training_dataloader = setup() # Need to get the training data
        return DataLoader(self.training_dataloader)

    def val_dataloader(self):
        validation_dataloader = prepare_data() # Need to get the validation data
        return DataLoader(self.validation_dataloader)

obj = DataModuleClass()
print(obj.prepare_data())


I am trying to learn PyTorch Lightning by writing a very simple DataModuleClass. After prepare_data(), and setup() I am trying to check these functions are working or not. So, I am trying to get the training and validation dataset from setup(). But I am getting an error
AttributeError: 'DataModuleClass' object has no attribute 'training_dataset'

Code
def prepare_data(self):
    x = np.random.uniform(0, 10, 10)
    e = np.random.normal(0, self.sigma, len(x))

    # Making target or labels
    y = x + e

    # Marging x and e for 2 features
    X = np.transpose(np.array([x, e]))

    # Converting numpy array to Tensor
    self.x_train_tensor = torch.from_numpy(X).float().to(device)
    self.y_train_tensor = torch.from_numpy(y).float().to(device)

    training_dataset = TensorDataset(self.x_train_tensor, self.y_train_tensor)

    self.training_dataset = training_dataset

def setup(self):
    data = self.training_dataset
    self.train_data, self.val_data = random_split(data, [8, 2])
    return self.train_data, self.val_data

def train_dataloader(self):
    return DataLoader(self.train_data)

def val_dataloader(self):
    return DataLoader(self.val_data)

obj = DataModuleClass()
print(obj.setup())

Could you tell me why I am getting this error?


title: Why does Dropout not change my input tensor?

body:
Please see the following code associated with output,
import torch
import torch.nn as nn
inputTensor = torch.tensor([1.0, 2.0, 3, 4, 5])

outplace_dropout = nn.Dropout(p=0.4)
print(inputTensor)
output_afterDropout = outplace_dropout(inputTensor)
print(output_afterDropout)
print(inputTensor)

The output is:
tensor([1., 2., 3., 4., 5.])
tensor([1.6667, 3.3333, 0.0000, 6.6667, 0.0000])
tensor([1., 2., 3., 4., 5.])

Could you please elaborate why the input tensor values are still unchanged?


title: Given multiple prediction vectors, how to efficiently obtain the label with most votes (in numpy/pytorch)?

body:
I have 3 vectors representing the 3 different predictions of labels for the same data:
P1=[31, 22, 11, 10,  9, 9, 0, 0, 23 ....]  # length over 1M
P2=[31, 22, 12, 10,  8, 9, 0, 0, 30 ....]  # length over 1M
P3=[30, 22, 12, 11,  8, 9, 0, 1, 31 ....]  # length over 1M

Ans= [31, 22, 12, 10, 8, 9, 0, 0, 23, ....]

The basic idea is if a prediction has the highest vote count (e.g. "31" has count 2 in the first column), we pick it, but if all candidates have a different vote (e.g. "23", "30", "31" in the last column), we can pick any one of them.
These vectors may be numpy array, list or pytorch tensor. Considering the length of such vector is over 1000,000, what's the most efficient way (mainly runtime) to find Ans?


title: AssertionError while training RNN using PyTorch-ligtning

body:
I'm new to PyTorch so I'm using PyTorch-Lightning to train simple (Vanilla) RNN:
1.Data preparation
import torch
from torch import nn
from torch.utils.data import DataLoader,TensorDataset
import matplotlib.pyplot as plt
import pytorch_lightning as pl
import numpy as np
import pandas as pd

#...
#X_train,Y_train are np arrs with shape (n,t,d)
X_train_tensors = torch.Tensor(X_train).to(device)
Y_train_tensors = torch.Tensor(Y_train).to(device)
#create train dataset
train = TensorDataset(X_train_tensors, Y_train_tensors)
trainloader = DataLoader(train, batch_size=32, shuffle=False)

2.Create Learner class
#use pl to create learner
class Learner(pl.LightningModule):
    def __init__(self, model:nn.Module):
        super().__init__()
        self.model = model

    def training_step(self, batch, batch_idx):
        x, y = batch
        y_hat = self.model(x)
        loss = nn.MSELoss()(y_hat, y)
        logs = {'train_loss': loss}
        return {'loss': loss, 'log': logs}

    def configure_optimizers(self):
        return torch.optim.Adam(self.model.parameters(), lr=0.001)

3.create a model and use trainer
NN = nn.Sequential(
            nn.RNN(1,3,nonlinearity='tanh',batch_first=True),
            nn.RNN(3,5,nonlinearity='tanh',batch_first=True),
            nn.Linear(5, 1)
        )

model = Learner(NN)
trainer = pl.Trainer(max_epochs=100, weights_summary='full')
trainer.fit(model, train_dataloader=trainloader)

I had this assertion error:
AssertionError                            Traceback (most recent call last)
<ipython-input-29-781e293c05ed> in <module>()
     10 model = Learner(NN)
     11 trainer = pl.Trainer(max_epochs=100, weights_summary='full')
---> 12 trainer.fit(model, train_dataloader=trainloader)
     13 #only when called it uses the test loop
     14 trainer.test(model, testloader)

16 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    242             max_batch_size = int(batch_sizes[0])
    243         else:
--> 244             assert isinstance(input, Tensor)
    245             batch_sizes = None
    246             max_batch_size = input.size(0) if self.batch_first else input.size(1)

when I checked the baseRNN in github https://github.com/pytorch/pytorch/blob/d09abf004cc16f8fd5f320e3d5d07c383c174ea7/torch/nn/modules/rnn.py#L247 I found no assert!
Could you pls help?


title: PyTorch DataLoader Error: object of type 'type' has no len()

body:
I'm quite new to programming and have now clue where my error comes from.
I got the following code to set up my dataset for training my classifier:
class cows_train(Dataset):

    def __init__(self, folder_path):
        self.image_list = glob.glob(folder_path+'/content/cows/train')
        self.data_len = len(self.image_list)

    def __getitem__(self, index):
        single_image_path = self.image_list[index]
        im_as_im = Image.open(single_image_path)
        im_as_np = np.asarray(im_as_im)/255
        im_as_np = np.expand_dims(im_as_np, 0)
        im_as_ten = torch.from_numpy(im_as_np).float()
        class_indicator_location = single_image_path.rfind('/content/cows/train/_annotations.csv')
        label = int(single_image_path[class_indicator_location+2:class_indicator_location+3])
        return (im_as_ten, label)

    def __len__(self):
        return self.data_len

And this for the DataLoader:
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

batch_size = 4

trainset = cows_train
trainloader =    torch.utils.data.DataLoader(dataset = trainset, batch_size=10,

shuffle=True, num_workers=2)

classes = ('cow_left', 'cow_other')

As Output I receive:
TypeError Traceback (most recent call last)
<ipython-input-6-54702f98a725> in <module>()
      6
      7 trainset = cows_train
----> 8 trainloader = torch.utils.data.DataLoader(dataset = trainset, batch_size=10, shuffle=True, num_workers=2)
      9
     10 testset = cows_test

2 frames
/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py in __init__(self, dataset, batch_size, shuffle, sampler, batch_sampler, num_workers, collate_fn, pin_memory, drop_last, timeout, worker_init_fn, multiprocessing_context, generator, prefetch_factor, persistent_workers)
    264                     # Cannot statically verify that dataset is Sized
    265                     # Somewhat related: see NOTE [ Lack of Default `__len__` in Python Abstract Base Classes ]
--> 266                     sampler = RandomSampler(dataset, generator=generator)  # type: ignore
    267                 else:
    268                     sampler = SequentialSampler(dataset)

/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py in __init__(self, data_source, replacement, num_samples, generator)
    100                              "since a random permute will be performed.")
    101
--> 102         if not isinstance(self.num_samples, int) or self.num_samples <= 0:
    103             raise ValueError("num_samples should be a positive integer "
    104                              "value, but got num_samples={}".format(self.num_samples))

/usr/local/lib/python3.7/dist-packages/torch/utils/data/sampler.py in num_samples(self)
    108         # dataset size might change at runtime
    109         if self._num_samples is None:
--> 110             return len(self.data_source)
    111         return self._num_samples
    112

TypeError: object of type 'type' has no len()

Problem is: I don't understand why typ has no length, in my eyes it's defined... Someone please help?
Add: This is where "return len(self.data_source)" shows up in the code

def num_samples(self) -> int:
    if self._num_samples is None:
        return len(self.data_source)
    return self._num_samples


title: How can I change the result of torch.cuda.is_available() to True GTX 1050ti

body:
hello i have recently started using pytorch but now i need to use my GPU which is a Nvidia GTX 1050ti to process some data but unfortunately torch.cuda.is_available() is returning False i have tried uninstaling cudatoolkit 11.3 and downgrade it to 11.1 and also deleating and reinstaling pytorch using conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia -c conda-forge, here's some information about my local configuration :
OS : windows 10
graphical card : nvidia 1050ti
current nvidia driver : 465.89

>>> torch.version.cuda
'11.1'
>>> torch.cuda.is_available()
False
>>> torch.backends.cudnn.enabled
True
>>> torch.__version__
'1.8.1'

   ->   nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2020 NVIDIA Corporation
Built on Tue_Sep_15_19:12:04_Pacific_Daylight_Time_2020
Cuda compilation tools, release 11.1, V11.1.74
Build cuda_11.1.relgpu_drvr455TC455_06.29069683_0


title: Assertion error "Pytorch not compiled with cuda eneabled"

body:
I am trying to run a code in this repository. I have tried many methods I've found online and none of them worked for me. My GPU is AMD Radeon(TM) graphics and Nvidia GeForce GTX 1650 Ti respectively, with the latter in the 'Display 2' tab in the 'dxdiag' tool. The full error message is:
 Traceback (most recent call last):
  File "C:\Users\XXX\Documents\Python\Generator\DameDaneGenerator\main.py", line 50, in <module>
    generator, kp_detector = load_checkpoints(config_path='firstordermodel/config/vox-256.yaml',
  File "C:\Users\XXX\Documents\Python\Generator\DameDaneGenerator\firstordermodel\demo.py", line 33, in load_checkpoints
    generator.cuda()
  File "C:\Users\XXX\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 491, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "C:\Users\XXX\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 387, in _apply
    module._apply(fn)
  File "C:\Users\XXX\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 387, in _apply
    module._apply(fn)
  File "C:\Users\XXX\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 387, in _apply
    module._apply(fn)
  [Previous line repeated 3 more times]
  File "C:\Users\XXX\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 409, in _apply
    param_applied = fn(param)
  File "C:\Users\XXX\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\nn\modules\module.py", line 491, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "C:\Users\XXX\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\cuda\__init__.py", line 164, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

I've installed Nvidia drivers, pytorch and cuda but so far I haven't have any progress, thanks in advance.


title: Tensor of 1 in nn.BCEWithLogitsLoss() in PyTorch - cuda out of memory?

body:
For the purpose of the custom loss function in PyTorch, I use nn.BCEWithLogitsLoss().
The first parameter is the probability I get from the model (after some math, of course) and the second parameter is 1 (100%).
My code looks like this one:
criterion = nn.BCEWithLogitsLoss()
loss =criterion(my_probability, torch.tensor(1.).cuda())

This part:
torch.tensor(1.).cuda() causes Cuda out of memory.
My doubts are:

Is there a better way (some other loss function?) to compare my_probability with 100% target value?
Is there a better way to declare and use a tensor of 1 on cuda?

Thank you!


title: pytorch : LSTM inputs and outputs dimensions and training loop

body:
I want to write a program that predictes the value of Bitcoin, but i keep getting errors i don't know what i did wrong.
I got the prepared and processed the datasets and made my neuron network. But i was having a hard time undrestanding the shape of LSTMs' inputs and outputs and matching my features and labels' shapes with them, so i think that's where i messed up but i don't know how to fix it. It might also be the training loop.
The errors i'm getting :

IndexError: index 652 is out of bounds for dimension 0 with size 1

My code :
import torch
import numpy as np
from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader
import torch.nn as nn
import pandas as pd
from sklearn.preprocessing import MinMaxScaler

df = pd.read_csv("C:/Users/HP/Documents/MASTER GL CLOUD/Python/coin_Bitcoin.csv",index_col = "Date", parse_dates=True)
df=df.iloc[:,6:7]

scaler =  MinMaxScaler(feature_range=(0,1))
data = scaler.fit_transform(np.array(df).reshape(-1,1))

train_data = data[:2000,:]
test_data = data[200:,:]

def creat_dataset(dataset,time_step):
    data_x,data_y = [] , []
    for i in range(len(dataset) - time_step -1):
        a = dataset[i:(i+time_step), 0]
        data_x.append(a)
        data_y.append(dataset[i+time_step, 0])

    return np.array(data_x), np.array(data_y)

time_step = 100
x_train, y_train = creat_dataset(train_data,time_step)
x_test, y_test = creat_dataset(test_data,time_step)

x_train = Variable(torch.Tensor(x_train))
y_train = Variable(torch.Tensor(y_train))

x_test = Variable(torch.Tensor(x_test))
y_test = Variable(torch.Tensor(y_test))

class MyDataset(Dataset):
    def __init__(self,x_train,y_train):
        self.len = x_train.shape[0]
        self.x_data = torch.reshape(x_train, (-1,x_train.shape[0], x_train.shape[2]))
        self.y_data = y_train
    def __getitem__(self,index):
        return self.x_data[index], self.y_data[index]
    def __len__(self):
        return self.len

train_data = MyDataset(x_train,y_train)
train_loader = DataLoader(dataset = train_data, batch_size = 256, shuffle = True)

test_data = MyDataset(x_test,y_test)
test_loader = DataLoader(dataset = test_data, batch_size = 256, shuffle = True)

class PredNet(nn.Module):
        def __init__(self,seq_len, n_layers=2, hidden_dim=200, input_dim=100, output_size=1):
            super(PredNet, self).__init__()
            self.hidden_dim = hidden_dim
            self.n_layers = n_layers
            self.hidden_cell = (torch.zeros(n_layers,seq_len,hidden_dim), torch.zeros(n_layers,seq_len,hidden_dim))
            self.lstm = nn.LSTM(input_dim, hidden_dim,n_layers, batch_first=True)
            self.fc =  nn.Linear(hidden_dim, output_size)
            self.relu = nn.ReLU()

        def forward(self, x ):
            lstm_out, _= self.lstm(x, self.hidden_cell)
            lstm_out = lstm_out[:,-1,:]
            out = self.relu(lstm_out)
            output = self.relu(self.fc(out))
            return output

prednet = PredNet(dataset.__len__())

criterion = torch.nn.MSELoss(size_average = False)
optimizer = torch.optim.SGD(prednet.parameters(), lr=0.01 )

for epoch in range(10):
    for i,data in enumerate(train_loader,0):
       inputs, labels = data
       outputs = prednet(inputs)
       optimizer.zero_grad()
       loss = criterion(outputs,labels)
       loss.backward()
       optimizer.step()
       print("Epoch: %d, loss: %1.5f" % (epoch, loss.item()))


title: Implementing a 3D gaussian blur using separable 2D convolutions in pytorch

body:
I'm trying to implement a gaussian-like blurring of a 3D volume in pytorch. I can do a 2D blur of a 2D image by convolving with a 2D gaussian kernel easy enough, and the same approach seems to work for 3D with a 3D gaussian kernel. However, it is very slow in 3D (especially with larger sigmas/kernel sizes). I understand this can also be done instead by convolving 3 times with the 2D kernel which should be much faster, but I can't get this to work. My test case is below.
import torch
import torch.nn.functional as F

VOL_SIZE = 21

def make_gaussian_kernel(sigma):
    ks = int(sigma * 5)
    if ks % 2 == 0:
        ks += 1
    ts = torch.linspace(-ks // 2, ks // 2 + 1, ks)
    gauss = torch.exp((-(ts / sigma)**2 / 2))
    kernel = gauss / gauss.sum()

    return kernel

def test_3d_gaussian_blur(blur_sigma=2):
    # Make a test volume
    vol = torch.zeros([VOL_SIZE] * 3)
    vol[VOL_SIZE // 2, VOL_SIZE // 2, VOL_SIZE // 2] = 1

    # 3D convolution
    vol_in = vol.reshape(1, 1, *vol.shape)
    k = make_gaussian_kernel(blur_sigma)
    k3d = torch.einsum('i,j,k->ijk', k, k, k)
    k3d = k3d / k3d.sum()
    vol_3d = F.conv3d(vol_in, k3d.reshape(1, 1, *k3d.shape), stride=1, padding=len(k) // 2)

    # Separable 2D convolution
    vol_in = vol.reshape(1, *vol.shape)
    k2d = torch.einsum('i,j->ij', k, k)
    k2d = k2d / k2d.sum()
    k2d = k2d.expand(VOL_SIZE, 1, *k2d.shape)
    for i in range(3):
        vol_in = vol_in.permute(0, 3, 1, 2)
        vol_in = F.conv2d(vol_in, k2d, stride=1, padding=len(k) // 2, groups=VOL_SIZE)
    vol_3d_sep = vol_in

    torch.allclose(vol_3d, vol_3d_sep)  # --> False

Any help would be very much appreciated!


title: How to check that all gradients weights are zeros in PyTorch?

body:
I want to know how to check all PyTorch neural network gradient weights to see if they are zero or not whether to continue training or not.
This may seem to be like PyTorch: How to check if some weights are not changed during training? but it's actually a different approach toward problem getting stuck in local minima. I don't have a clue what code I should add cause it seems a general question and I even don't know how to access gradient tensor in PyTorch.


title: How to check the output gradient by each layer in pytorch in my code?

body:
I am working on the pytorch to learn.
And  There is a question how to check the output gradient by each layer in my code.
My code is below
#import the nescessary libs
import numpy as np
import torch
import time

# Loading the Fashion-MNIST dataset
from torchvision import datasets, transforms

# Get GPU Device

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
                                    transforms.Normalize((0.5,), (0.5,))
                                                                   ])
# Download and load the training data
trainset = datasets.FashionMNIST('MNIST_data/', download = True, train = True, transform = transform)
testset = datasets.FashionMNIST('MNIST_data/', download = True, train = False, transform = transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size = 32, shuffle = True, num_workers=4)
testloader = torch.utils.data.DataLoader(testset, batch_size = 32, shuffle = True, num_workers=4)

# Examine a sample
dataiter = iter(trainloader)
images, labels = dataiter.next()

# Define the network architecture
from torch import nn, optim
import torch.nn.functional as F

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 10),
                      nn.LogSoftmax(dim = 1)
                     )
model.to(device)

# Define the loss
criterion = nn.CrossEntropyLoss()

# Define the optimizer
optimizer = optim.Adam(model.parameters(), lr = 0.001)

# Define the epochs
epochs = 5

train_losses, test_losses = [], []

# start = time.time()
for e in range(epochs):
    running_loss = 0

    for images, labels in trainloader:
    # Flatten Fashion-MNIST images into a 784 long vector
        images = images.to(device)
        labels = labels.to(device)
        images = images.view(images.shape[0], -1)

    # Training pass
        optimizer.zero_grad()

        output = model.forward(images)
        loss = criterion(output, labels)

        loss.backward()

#         print(loss.grad)

        optimizer.step()

        running_loss += loss.item()

    else:
        print(model[0].grad)

If I print model[0].grad after back-propagation, Is it going to be the output gradient by each layer for every epoches?
Or, If I want to know the output gradient by each layer, where and what am I should print?
Thank you!!
Thank you for reading


title: Re-encoding a context vector after every step of decoding?

body:
I am currently working on a sequence model which aims to predict the head orientation of somebody watching VR for an arbitrary amount of frames in the future.

Using an encoder-decoder paradigm, a person's previous 100 frames of head orientation and the video's previous 100 frames of data are fed into an encoder to create a context.  (100 is arbitrary here, but for sake of writing I will use 100 frames, so the previous sequence's data will be  [ T(-100) - T0 ]   )

The context is then fed into the decoder, and frame by frame outputting a probability distribution of where the viewer will be looking at T1 for the first iteration of decoding, T2 for the second iteration, etc.

The context vector therefore has a lot of weight on its shoulders, encoding data from the entire previous sequence.  So in order to help with the decoder's output, at each step of decoding I want to let the decoder have access to the video's future 100 frames of data.
So at time  T2, the decoder will be on its second iteration and have access to the 102nd video frame to use for convolution, saliency, etc.

The way we were thinking to accomplish this is to have the encoder re-encode the context vector on each iteration of decoding the context, and replacing the encoder's input to contain the future video frame data.

So for example on the second iteration of decoding, we would re-encode the context, but replace the data for T(-100) with the decoded output for T1, and the video frame for T(-100) with the video frame for T2.  We would continue to iterate through decoding in this way.

The thought is this would be a way to provide the decoder with information about the upcoming video, and use its previous predictions to enforce the integrity of future predictions of movement.  This idea of re-encoding the context was also presented by Johannes Baptist from the University of Amsterdam in this paper, though I am an undergraduate student and still new to this field, so the paper was a bit challenging to grasp.

If anybody has any intuition on what re-encoding the context at every step of decoding will do to an encoder-decoder model, and can provide any insight or resources to continue looking into, that would be great, thank you very much.


title: Why there is diiference between the matrix after loading .mat file from Lua torch and pytorch using scipy?

body:
I have the output matrix from a .mat file using Lua torch 7 and pytorch with scipy both. I am getting the single bit different matrix in using pytorch . Please some one explain me why ? In pytorch matric is in float point, but in Lua torch it has fixe point. How do I fix the point in pytorch.

Here is the matrix.

From Scipy and pytorch Result:

tensor([[[112.,  -1.,  -1.,  -1.,  -1.,  -1.],
     [ -1.,  -1., 130., 138.,  -1.,  -1.]],

    [[ 57.,  -1.,  -1.,  -1.,  -1.,  -1.],
     [ -1.,  -1.,  11.,  53.,  -1.,  -1.]],

    [[ 58.,  -1.,  -1.,  -1.,  -1.,  -1.],
     [ -1.,  -1.,  31.,  28.,  -1.,  -1.]],

    [[ 46.,  -1.,  -1.,  -1.,  -1.,  -1.],
     [ -1.,  -1.,  24.,  28.,  -1.,  -1.]]])

From Lua torch7 result

[[[113   -1   -1   -1   -1   -1]
   [-1   -1  131  139   -1   -1]],
   [[57   -1   -1   -1   -1   -1],
   [-1   -1   11   53   -1   -1]],
   [[58   -1   -1   -1   -1   -1],
   [-1   -1   31   29   -1   -1]],
   [[47   -1   -1   -1   -1   -1],
   -[1   -1   24   28   -1   -1]]]

you can see the first  element 112 in pytorch but 113 in lua torch. sSimilary 130 & 131 , 138 & 139 , 28 & 29 and 46 and 47. I want the result similar to Lua torch which is the second matrix. Is there any method to get the fix point in pytorch also.  Please help me out. Thanks


title: In multitasking, cannot create a new calculation graph after the backward propagation

body:
Background：I use DQN and DDPG to solve two tasks simultaneously. The state(input) of DQN and DDPG are both two parts. One part is the states of the environment, and the other one is the states abstracted from the environment by CNN+LSTM. The two parts are concatenate in forward_dqn(), forward_actor() and forward_critic() respectively.

Question1：
I backward propagate the loss_dqn, loss_ddpg_actor, and loss_ddpg_critic in sequence and get the error "Trying to backward through the graph a second time, but the buffers have already been freed." in the backward propagation of loss_ddpg_actor. Since after the backward propagation of loss_dqn, the computational graph has been freed, so I have forward propagated CNN+LSTM again to calculate the loss_ddpg_actor. Why the computational graph cannot be created again? Thanks.

Model: (env: environment)

output_cnnlstm = cnnlstm.forward(env)
DQN_output = dqn.forward(cat(output_cnnlstm, state_env))
Actor_output = actor.forward(cat(output_cnnlstm, state_env))
Critic_output = critic.forward(cat(output_cnnlstm, state_env))

Code 1 (Q1):

    # dqn
    # forward: cnnlstm
    s_cnnlstm_out, _, _ = self.model.forward_cnnlstm(s_cnnlstm, flag_optim=True)
    # forward: dqn
    q_eval_dqn = self.model.forward_dqn_eval(s_dqn, s_cnnlstm_out).gather(1, a_dqn)
    q_next_dqn = self.model.forward_dqn_target(s_dqn_next, s_cnnlstm_out).detach()
    q_target_dqn = r + GAMMA_DQN * q_next_dqn.max(dim=1)[0].reshape(SIZE_BATCH * SIZE_TRANSACTION, 1)
    # optimzie: dqn
    loss_dqn = self.loss_dqn(q_eval_dqn, q_target_dqn)
    self.optimizer_cnnlstm.zero_grad()
    self.optimizer_dqn.zero_grad()
    loss_dqn.backward()
    self.optimizer_cnnlstm.step()
    self.optimizer_dqn.step()
    loss_dqn = loss_dqn.detach().numpy()
    # ddpg
    # actor
    # forward: cnnlstm
    s_cnnlstm_out, _, _ = self.model.forward_cnnlstm(s_cnnlstm, flag_optim=True)
    # forward: ddpg: actor
    a_eval_ddpg = self.model.forward_actor_eval(s_ddpg, s_cnnlstm_out)
    # optimze: ddpg: cnnlstm + actor
    loss_ddpg_actor = - self.model.forward_cirtic_eval(s_ddpg, a_eval_ddpg, s_cnnlstm_out).mean()
    self.optimizer_cnnlstm.zero_grad()
    self.optimizer_actor.zero_grad()
    loss_ddpg_actor.backward()
    self.optimizer_cnnlstm.step()
    self.optimizer_actor.step()
    loss_ddpg_actor = loss_ddpg_actor.detach().numpy()

Question2:
I write a demo to test the propagation process and the demo seems to work well since the loss descends normally and the test error is low. So I want to ask the difference between the two codes and models.

Model:

output_model1 = model1.forward(x)
output_model21 = model21.forward(cat(output_model1, x1))
output_model22 = model221.forward(cat(output_model1, x2))

compared with the model of Q1, output_model1 ~ cnnlstm, output_model21 ~ DQN, output_model22 ~ Actor

Question3：
I set breakpoint in the demo after loss1.backward() and before optimizer1.step(). However, on the one hand, the weight of the linear layer of Model21 changes with the optimization. On the other hand, x._grad is a gradient value tensor, while x1._grad is None. So I wonder the parameters of Model21 are optimized whether or not and why x1._grad is None.

Code 2 (Q2 and Q3):

for i in range(NUM_OPTIM):
    # optimize task 1
    y1_pred = self.model.forward_task1(x, x1)
    loss1 = self.loss_21(y1_pred, y1)
    self.optimizer1.zero_grad()
    self.optimizer21.zero_grad()
    loss1.backward()
    self.optimizer1.step()
    self.optimizer21.step(
    # optimze task 2
    y2_pred = self.model.forward_task2(x, x2)
    loss2 = self.loss_22(y2_pred, y2)
    self.optimizer1.zero_grad()
    self.optimizer22.zero_grad()
    loss2.backward()
    self.optimizer1.step()
    self.optimizer22.step()


title: Pytorch model uses GPU memory but volatility at 0

body:
I have a model which takes up gpu memory, but most of the time the GPU is idle and it reaches 90 volatility and then goes back to 0.

How can i maintain volatility to be at 100?

class KaggleAmazonDataset(Dataset):
    """Dataset wrapping images and target labels for Kaggle - Planet Amazon from Space competition
    Arguments:
        A CSV file path
        Path to image folder
        Extension of images
        PIL transforms
    """

    def __init__(self, csv_path, img_path, img_ext, transform=None):
        # import pdb;pdb.set_trace()
        tmp_df = pd.read_csv(csv_path)
        # assert tmp_df['Left-Fundus'].apply(lambda x: os.path.isfile(img_path + x + img_ext)).all(), \
# "Some images referenced in the CSV file were not found"

        # self.mlb = MultiLabelBinarizer()
        self.img_path = img_path
        self.img_ext = img_ext
        self.transform = transform

        self.X_train = tmp_df['Left-Fundus']
        self.y_train = tmp_df.iloc[:, 7:16].values
        # self.y_train = self.mlb.fit_transform(tmp_df['tags'].str.split()).astype(np.float32)

    def __getitem__(self, index):
        img = Image.open(self.img_path + self.X_train[index] + self.img_ext)
        img = img.convert('RGB')
        if self.transform is not None:
            img = self.transform(img)

        label = torch.from_numpy(self.y_train[index])
        return img, label

    def __len__(self):
        return len(self.X_train.index)

#
transformations = transforms.Compose([transforms.Resize((224,224)),transforms.ToTensor()])

dset_train = KaggleAmazonDataset(TRAIN_DATA,IMG_PATH,IMG_EXT,transformations)
train_loader = DataLoader(dset_train,
                          batch_size=256,
                          shuffle=True,
                          num_workers=12, # 1 for CUDA
                         pin_memory=True)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(2304, 256)
        self.fc2 = nn.Linear(256, 8)

    def forward(self, x):

        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(x.size(0), -1) # Flatten layer
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return torch.sigmoid(x)

model = Net().cuda()
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

Is somthing wrong here, Any suggestions will be really helpful.Thanks in advance.


title: PyTorch dot product does not work for some cases

body:
I am struggling with an issue I cannot solve in PyTorch code.  My task is simple: just get the inner product of two tensors.  However, the output becomes zero in certain cases instead of returning the correct value.

For example, the code below demonstrates the issue.

a = torch.tensor([3,1])
b = torch.tensor([3.,1.])
a.dot(a)  # this returns 10, which is correct
b.dot(b)  # this returns 0., which is not correct

I am using PyTorch version 1.2 and this has never happened to me before...  Is there something simple that I am missing around the way I define the tensors?

Thank you for your time!


title: Why my deep learning model cannot be trained in 10 iterations?

body:
My deep learning model is used to segment the CT images. But the loss value is always very high. To debug the program, I used the same input images to observe the output and the loss value of the 10 iteration are all the same, which means the network's parameter didn't change at all. Could you please tell me why my model cannot be trained ? Is the reason that I didn't use the DataFrame method?
The code shows as follows:

for ite in range(iteration):
    print("=====iteration", ite, "=======")
    weights = few_shot_model.conditioner(condition_input)
    output = few_shot_model.segmentor(query_input, weights)
    # out = F.softmax(output, dim=1)
    print("out.shape, y2.shape:", output.shape, y2.shape)
    print("y2.unique:", torch.unique(y2))
    loss = loss_func(output, y2, binary=True)
    optim_s.zero_grad()  # clear gradients for this training step
    optim_c.zero_grad()  #
    loss.backward()  # backpropagation, compute gradients
    if phase == 'train':
        if epoch <= warm_up_epoch:
            optim_s.step()  # apply gradients
            optim_c.step()
            scheduler_c.step()
            scheduler_s.step()
        elif epoch > warm_up_epoch and change_model:
            optim_n.step()
        print("loss.item", loss.item())


title: I want to show accuracy under each picture output shown in Image Classification with Transfer Learning in PyTorch

body:
I'm follow this link : https://stackabuse.com/image-classification-with-transfer-learning-and-pytorch/#settingupapretrainedmodel

but I'm new with coding skill. please tell me how to show accuracy value under image.


title: Use pytorch to bulid multi-linear,but the result i get is not what i want?

body:
import sklearn
    from sklearn.datasets import load_boston
    import torch
    import torch.nn as nn

    boston = load_boston()

    num_epochs = 10

    linear_model = nn.Linear(13,1,bias = True)
    criterion = nn.MSELoss()
    optimizer = torch.optim.SGD(linear_model.parameters(), lr=0.01)

    for epoch in range(num_epochs):

        inputs = torch.from_numpy(boston.data)
        targets = torch.from_numpy(boston.target)
        targets = targets.float()
        inputs = inputs.float()

        outputs = linear_model(inputs)
        loss = criterion(outputs,targets)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        print('epoch:{}/{} ...... loss:{:.4f}'.format(epoch,num_epochs,loss.item()))


title: How to prepare my own data set for the DCP training?

body:
I have ply files to be used for training in DCP(deep closest point). Data format is hdf5 for DCP Training.  The Data Format contains the following information: Data, FaceId, Normal, Label... But Ply file just  have Datas(x y z).
how can i do this transformation?


title: PyTorch custom DataLoader dimension issues for CNN

body:
I have written a custom Dataset and DataLoader for a PyTorch CNN project. Here is the relevant code for the dataset

class MyDataset(Dataset):

  def __init__(self):
    pass

  def __len__(self):
    return COUNT

  def __getitem__(self, idx):
    x, y = X[idx], Y[idx]
    x = image_augment(x)  # custom func to resize image to 32x32
    return x, y

The shape of each training x is [4, 32, 32, 3].

And here is my Net code, taken directly from this PyTorch example.

class Net(nn.Module):

    def __init__(self, nc):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, nc)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

When I try to train this net on my data from my DataLoader, I get the error statement Given groups=1, weight of size [6, 3, 5, 5], expected input[4, 32, 32, 3] to have 3 channels, but got 200 channels instead. It seems to me my issue is with the shape of my data coming from my DataLoader using x.view(4, 3, 32, 32), but then I got an error saying I couldn't use Conv2D on a ByteTensor. I'm a little lost here and would really appreciate any help. Thanks!


title: app crash when the execution of model.forward on android code ,i don't know why?

body:
When I execute this code precisely at the line which corresponds to module.forward my apps restarts. Code attached below:

@WorkerThread
@Nullable
public IValue translation(String input){
     // loading serialized torchscript module from packaged into app android asset model.pt,
     Module module = Module.load(assetFilePath(this.context,"model_seq2seq_22_04_2020_13.pt"));

    inputTensor = generateTensor(lists,input);
    Log.d("tensor shape", Arrays.toString(inputTensor.shape()));

    long[] shape = new long[]{1,1};
    long[] start = new long[1];
    start[0]=0;

    Tensor trgStart = Tensor.fromBlob(start,shape);
    Log.d("tensor shape", Arrays.toString(trgStart.shape()));
    IValue output = module.forward(IValue.from(inputTensor),IValue.from(trgStart));
}


title: Latency issue with Tensorflow cuDNN model execution

body:
I am having problems with a cuDNN RNN model I am trying to train on a set of natural language explanations and embeddings for the semantic parsing of texts. Here is what my RNN model architecture looks like on a simplified level:

class Cudnn_RNN:

    def __init__(self, num_layers, num_units, mode="lstm", keep_prob=1.0, is_train=None, scope="cudnn_rnn"):
        self.num_layers = num_layers
        self.rnns = []
        self.mode = mode
        if mode == "gru":
            rnn = tf.contrib.cudnn_rnn.CudnnGRU
        elif mode == "lstm":
            rnn = tf.contrib.cudnn_rnn.CudnnLSTM
        else:
            raise Exception("Unknown mode for rnn")
        for layer in range(num_layers):
            rnn_fw = rnn(1, num_units)
            rnn_bw = rnn(1, num_units)
            self.rnns.append((rnn_fw, rnn_bw, ))

    def __call__(self, inputs, seq_len, keep_prob=1.0, is_train=None, concat_layers=True):
        outputs = [tf.transpose(inputs, [1, 0, 2])]
        for layer in range(self.num_layers):
            rnn_fw, rnn_bw = self.rnns[layer]
            output = dropout(outputs[-1], keep_prob=keep_prob, is_train=is_train)
            with tf.variable_scope("fw_{}".format(layer)):
                out_fw, state_fw = rnn_fw(output)
            with tf.variable_scope("bw_{}".format(layer)):
                inputs_bw = tf.reverse_sequence(output, seq_lengths=seq_len, seq_axis=0, batch_axis=1)
                out_bw, state_bw = rnn_bw(inputs_bw)
                out_bw = tf.reverse_sequence(out_bw, seq_lengths=seq_len, seq_axis=0, batch_axis=1)
            outputs.append(tf.concat([out_fw, out_bw], axis=2))
        if concat_layers is True:
            res = tf.concat(outputs[1:], axis=2)
        else:
            res = outputs[-1]
        res = tf.transpose(res, [1, 0, 2])
        state_fw = tf.squeeze(state_fw[0], [0])
        state_bw = tf.squeeze(state_bw[0], [0])
        state = tf.concat([state_fw, state_bw], axis=1)
        return res, state

The model is set up such that after data is loaded, it goes through pretraining, training, and then evaluation. For some reason the data is being loaded with no issues, but as soon as the model starts running it gets stuck, not even making it to the pretraining phase. Here is the data loading and execution code (model executes up until just before print('---Pretrain-----'):

def pseudo_labeling(config, data):
    word2idx_dict, fixed_emb, traiable_emb, train_data, dev_data, test_data,pretrain_data,pretrain_data2 = data

    pretrain_test_data = (pretrain_data[0][:config.pretrain_test_size],pretrain_data[1][:config.pretrain_test_size],pretrain_data[2][:config.pretrain_test_size,:])
    pretrain_data = (pretrain_data[0][config.pretrain_test_size:config.pretrain_test_size+config.pretrain_train_size],pretrain_data[1][config.pretrain_test_size:config.pretrain_test_size+config.pretrain_train_size],pretrain_data[2][config.pretrain_test_size:config.pretrain_test_size+config.pretrain_train_size,:])

    lfs = get_lfs(config, word2idx_dict)
    identifier = "_{}".format(config.tag)

    with tf.variable_scope("models", reuse=tf.AUTO_REUSE):
        regex = Pat_Match(config)
        match = Soft_Match(config,lfs['lfs'],np.array(lfs['rels'],np.float32),lfs['keywords'],lfs['keywords_rels'], lfs['raw_keywords'],mat=((fixed_emb, traiable_emb, )), word2idx_dict=word2idx_dict, pseudo=True)

    sess_config = tf.ConfigProto(allow_soft_placement=True)
    sess_config.gpu_options.allow_growth = True
    if os.path.exists('labeled_data.pkl'):
        with open('labeled_data.pkl', 'rb') as f:
            labeled_data = pickle.load(f)
        with open('unlabeled_data.pkl', 'rb') as f:
            unlabeled_data = pickle.load(f)
        with open('weights.pkl', 'rb') as f:
            lfs["weights"] = pickle.load(f)
    else:
        with open('exp2pat.json','r') as f:
            exp2pat = json.load(f)
        exp2pat = {int(key):val for key,val in exp2pat.items()}
        lab_d = []
        unlab_d = []

        tacred_labeled = []
        tacred_unlabeled = []
        labeled_data = []
        unlabeled_data = []
        idxx = -1

        idx2rel = {val:key for key,val in constant.LABEL_TO_ID.items()}

        for x in tqdm(train_data):
            idxx+=1
            batch = [x["phrase"]]
            res, pred = regex.match(batch)
            lfs["weights"] += res[0]
            new_dict = {}
            if np.amax(res) > 0:

                x["rel"] = pred.tolist()[0]
                x["logic_form"] = np.argmax(res, axis=1).tolist()[0]
                new_dict['tokens'] = x['phrase'].token
                new_dict['start'] = min(x['phrase'].subj_posi,x['phrase'].obj_posi)+1
                new_dict['end'] = max(x['phrase'].subj_posi,x['phrase'].obj_posi)-1
                new_dict['rel'] = pred.tolist()[0]
                try:
                    new_dict['pat'] = exp2pat[np.argmax(res, axis=1).tolist()[0]]
                    lab_d.append(new_dict)
                except:
                    new_dict['pat'] = -1
                    unlab_d.append(new_dict)
                tacred_labeled.append((idxx,idx2rel[x['rel']]))
                labeled_data.append(x)
            else:
                tacred_unlabeled.append(idxx)
                new_dict['tokens'] = x['phrase'].token
                new_dict['start'] = min(x['phrase'].subj_posi,x['phrase'].obj_posi)+1
                new_dict['end'] = max(x['phrase'].subj_posi,x['phrase'].obj_posi)-1
                new_dict['rel'] = pred.tolist()[0]
                new_dict['pat']=-1
                x["rel"] = 0
                unlab_d.append(new_dict)
                unlabeled_data.append(x)

        new_weight = np.array([elem for i, elem in enumerate(list(lfs['weights'])) if i in exp2pat],np.float32)
        new_weight = new_weight/np.sum(new_weight)
        lfs["weights"] = lfs["weights"] / np.sum(lfs["weights"])

        with open('tacred_labeled.json','w') as f:
            json.dump(tacred_labeled,f)

        with open('tacred_unlabeled.json','w') as f:
            json.dump(tacred_unlabeled,f)

        with open('labeled_data.pkl','wb') as f:
            pickle.dump(labeled_data,f)
        with open('unlabeled_data.pkl','wb') as f:
            pickle.dump(unlabeled_data,f)
        with open('weights.pkl', 'wb') as f:
            pickle.dump(lfs["weights"], f)

        with open('lab_d.pkl','wb') as f:
            pickle.dump(lab_d,f)
        with open('unlab_d.pkl','wb') as f:
            pickle.dump(unlab_d,f)
        with open('weights_d.pkl','wb') as f:
            pickle.dump(new_weight,f)

    random.shuffle(unlabeled_data)

    print('unlabdel data:',str(len(unlabeled_data)),'labeled data:',str(len(labeled_data)))

    dev_history, test_history = [], []
    dev_history2, test_history2 = [], []

    with tf.Session(config=sess_config) as sess:

        lr = float(config.init_lr)
        writer = tf.summary.FileWriter(config.log_dir + identifier)
        sess.run(tf.global_variables_initializer())

        print('---Pretrain-----')
        for epoch in range(config.pretrain_epoch):
            loss_list,pretrain_loss_lis,sim_loss_lis = [],[],[]
            for batch in get_pretrain_batch(config, pretrain_data, word2idx_dict):
                pretrain_loss_prt,sim_loss_prt,loss,_ = sess.run([match.pretrain_loss,match.sim_loss,match.pretrain_loss_v2,match.pre_train_op],feed_dict={match.pretrain_sents: batch['sents'], match.pretrain_pats: batch['pats'],match.pretrain_labels: batch['labels'],match.is_train:True})
                loss_list.append(loss)
                pretrain_loss_lis.append(pretrain_loss_prt)
                sim_loss_lis.append(sim_loss_prt)
            print("{} epoch:".format(str(epoch)))
            print("loss:{} pretrain_loss:{} sim_loss:{}".format(str(np.mean(loss_list)),str(np.mean(pretrain_loss_lis)),str(np.mean(sim_loss_lis))))
            pred_labels = []
            goldens = []
            prt_id = 0
            for batch in get_pretrain_batch(config,pretrain_data2,word2idx_dict,shuffle=False):
                prt_id+=1
                pp,ppp,pred_label = sess.run([match.prt_loss,match.prt_pred,match.pretrain_pred_labels],feed_dict={match.pretrain_sents: batch['sents'], match.pretrain_pats: batch['pats'],match.is_train:False,match.pretrain_labels: batch['labels']})
                pred_label = list(pred_label)
                golden = list(np.reshape(batch['labels'],[-1]))
                assert len(golden)==len(pred_label)
                pred_labels.extend(pred_label)
                goldens.extend(golden)
            p,r,f = f_score(pred_labels,goldens)
            print('PRF:',(p,r,f))
            if p>0.9 and r>0.9:
                break
            print('\n')

Here are my system specifications:
Tensorflow version: 1.14.0 (w/ GPU support)
Operating System: Linux 9.12
OS Distribution: Debian
OS Architecture: x86_64
Python version: 3.7.6
NLTK version: 3.4.5
CUDA version: 10.0
cuDNN version: 7.4.2
NVIDIA graphics card: Tesla T4
NVIDIA driver version: 410.104
Compiler version: GCC 6.3.0

If anyone would like to share their thoughts on why my model is unable to properly execute to the pretraining phase and beyond, I would greatly appreciate it. Thank you.


title: regarding one code segment in computing log_sum_exp

body:
In this tutorial on using Pytorch to implement BiLSTM-CRF, author implements the following function. In specific, I am not quite understand what does max_score_broadcast = max_score.view(1, -1).expand(1, vec.size()[1]) try to do?, or which kind of math formula it corresponds to?

# Compute log sum exp in a numerically stable way for the forward algorithm
def log_sum_exp(vec):
    max_score = vec[0, argmax(vec)]
    max_score_broadcast = max_score.view(1, -1).expand(1, vec.size()[1])
    return max_score + \
        torch.log(torch.sum(torch.exp(vec - max_score_broadcast)))