title: How to reset a CUDA GPU in python

body:
Is there a way to restart CUDA GPU within python ? or release it so that another script can get access to it ?

I sometimes get an error using the GPU in python, and the only solution to get access to the GPU again is to restart my Jupyter notebook.

PS: I am using the GPU for some computation using pytorch.

the particular error I get is:

RuntimeError: CUDA error: device-side assert triggered

I have seen https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3 and many others discussing this problem. My main issue is how to continue using the GPU session without restarting the Jupyter kernel.

i.e. I would like to catch this error which I do with a try/except and move on since I am training multiple models one after the other so i cannot restart to solve my problem.

EDIT restarting the kernel seems to be the accepted naive solution see:

-https://forums.fast.ai/t/how-to-free-up-gpu-memory-in-pytorch-0-2-x/9256/2
-https://towardsdatascience.com/cuda-error-device-side-assert-triggered-c6ae1c8fa4c3 where she mentions "Working on Kaggle? Here’s why even after following the above steps, you are still struggling"


title: Error in converting tensorflow BERT checkpoints to pytorch

body:
I downloaded tensorflow checkpoints for domain specific bert model and extracted the zip file into the folder pretrained_bert which contains the following the three files

model.ckpt.data-00000-of-00001
model.ckpt.index
model.ckpt.meta

I used the following code to convert tensorflow checkpoints to pytorch
import torch

from pytorch_transformers.modeling_bert import BertConfig, BertForPreTraining, load_tf_weights_in_bert

tf_checkpoint_path="pretrained_bert/model.ckpt"
bert_config_file = "bert-base-cased-config.json"
pytorch_dump_path="pytorch_bert"

config = BertConfig.from_json_file(bert_config_file)
print("Building PyTorch model from configuration: {}".format(str(config)))
model = BertForPreTraining(config)

# Load weights from tf checkpoint
load_tf_weights_in_bert(model, config, tf_checkpoint_path)

# Save pytorch-model
print("Save PyTorch model to {}".format(pytorch_dump_path))
torch.save(model.state_dict(), pytorch_dump_path)

I got this error when I ran the above code

NotFoundError: Unsuccessful TensorSliceReader constructor: Failed to
find any matching files for pretrained_bert/model.ckpt

Any help is really appreciated............


title: From Coco annotation json to semantic segmentation image like VOC's .png in pytorch

body:
I am trying to use COCO 2014 data for semantic segmentation training in PyTorch.
I have a PSPNet model with a Cross Entropy loss function that worked perfectly on PASCAL VOC dataset from 2012.
Now I am trying to use a portion of COCO pictures to do the same process. But Coco has json data instead of .png images for annotation and I somehow have to covert one to the other. I have noticed that there is annToMask in cocotools, but I cannot quiet figure out how to use that function in my case.
This is kind of what my dataloader's pull item looks like

def pull_item(self, index):

        I DONT KNOW WHAT TO DO HERE

        raw_img = self.transform(raw_img)
        anns_img = self.transform(anns_img)

        return raw_img, anns_img

Below is what my training function that uses data from dataloaders looks like.

 for images, labels in dataloaders_dict[phase]:

                images = images.to(device)

                labels = torch.squeeze(labels)
                labels = labels.to(device)

                with torch.set_grad_enabled(phase == 'train'):
                    outputs = net(images)

                    loss = criterion(outputs, labels.long())


title: How to free GPU memory for a specific tensor in PyTorch?

body:
I’m currently running a deep learning program using PyTorch and wanted to free the GPU memory for a specific tensor.

I’ve thought of methods like del and torch.cuda.empty_cache(), but del doesn’t seem to work properly (I’m not even sure if it frees memory at all) and torch.cuda.empty_cache() seems to free all unused memory, but I want to free memory for just a specific tensor.

Is there any functionality in PyTorch that provides this?

Thanks in advance.


title: How to set target in cross entropy loss for pytorch multi-class problem

body:
Problem Statement: I have an image and a pixel of the image can belong to only(either) one of Band5','Band6', 'Band7' (see below for details). Hence, I have a pytorch multi-class problem but I am unable to understand how to set the targets which needs to be in form [batch, w, h]

My dataloader return two values:

x = chips.loc[:, :, :, self.input_bands]
y = chips.loc[:, :, :, self.output_bands]
x = x.transpose('chip','channel','x','y')
y_ohe = y.transpose('chip','channel','x','y')

Also, I have defined:

input_bands = ['Band1','Band2', 'Band3', 'Band3', 'Band4']  # input classes
output_bands = ['Band5','Band6', 'Band7'] #target classes

model = ModelName(num_classes = 3, depth=default_depth, in_channels=5, merge_mode='concat').to(device)
loss_new = nn.CrossEntropyLoss()

In my training function:

        #get values from dataloader
        X = normalize_zero_to_one(X) #input
        y = normalize_zero_to_one(y) #target

        images = Variable(torch.from_numpy(X)).to(device) # [batch, channel, H, W]
        masks = Variable(torch.from_numpy(y)).to(device)
        optim.zero_grad()
        outputs = model(images)

        loss = loss_new(outputs, masks) # (preds, target)
        loss.backward()
        optim.step() # Update weights

I know the the target (here masks) should be [batch_size, w, h]. However, it is currently [batch_size, channels, w, h].

I read a lot of posts including 1, 2 and they say the target should only contain the target class indices. I don't understand how can I concatenate indices of three classes and still set target as [batch_size, w, h].

Right now, I get the error:

RuntimeError: only batches of spatial targets supported (3D tensors) but got targets of dimension: 4

To the best of my understanding, I don't need to do any one hot encoding. Similar errors and explanation I found on the internet are here:'

Reference 1
Reference 2
Reference 3
Reference 4

Any help will be appreciated! Thank you.


title: trying to import png images to torchvision

body:
I am attempting to import images for use with torch and torchvision. But I am receiving this error:
TypeError: Caught TypeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\worker.py", line 178, in _worker_loop
    data = fetcher.fetch(index)
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "c:\python38\lib\site-packages\torch\utils\data\_utils\collate.py", line 81, in default_collate
    raise TypeError(default_collate_err_msg_format.format(elem_type))
TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found <class 'PIL.Image.Image'>

Based on this post, I am converting them to Tensor:
https://discuss.pytorch.org/t/typeerror-default-collate-batch-must-contain-tensors-numpy-arrays-numbers-dicts-or-lists-found-class-imageio-core-util-array/62667
Here is my code:
import torch
import torchvision
import torchvision.transforms
from torchvision import datasets, transforms

transform = transforms.Compose([
    transforms.Resize(256),
    transforms.ToTensor()
])

dataset = torchvision.datasets.ImageFolder('datasets')

dataloader = torch.utils.data.DataLoader(dataset,
                                          batch_size=16,
                                          shuffle=True,
                                          num_workers=12)

tensor_dataset = []

for i, data in enumerate(dataloader, 0):
    Tensor = torch.tensor(data)
    tensor_dataset.append(Tensor.flatten)

The first last part is from https://github.com/TerragonDE/PyTorch but I have had no success.  The data I am trying to load is from here:
http://www.cvlibs.net/datasets/kitti/
How can I solve this?
UPDATE:
Thanks @trialNerror, but now I am getting this error:
ValueError                                Traceback (most recent call last)
<ipython-input-6-aa72392b67e8> in <module>
      1 for i, data in enumerate(dataloader, 0):
----> 2     Tensor = torch.tensor(data)
      3     tensor_dataset.append(Tensor.flatten)

ValueError: only one element tensors can be converted to Python scalars

This is what I have found so far but am not sure how to apply it:
https://discuss.pytorch.org/t/pytorch-autograd-grad-only-one-element-tensors-can-be-converted-to-python-scalars/56681
UPDATE 2:
The reason why I didn't end up using the dataloader is because I end up getting this error:
num_epochs = 10
loss_values = list()

for epoch in range(1, num_epochs):
    for i, data in enumerate(train_array, 0):
        outputs = model(data.unsqueeze(0))
        loss = criterion(outputs,data.unsqueeze(0))

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        print('Epoch - %d, loss - %0.5f '%(epoch, loss.item()))
        loss_values.append(loss.item())

torch.Size([1, 16, 198, 660])
torch.Size([1, 32, 97, 328])
torch.Size([1, 1018112])

RuntimeError                              Traceback (most recent call last)
<ipython-input-106-5e6fa86df079> in <module>
      4 for epoch in range(1, num_epochs):
      5     for i, data in enumerate(train_array, 0):
----> 6         outputs = model(data.unsqueeze(0))
      7         loss = criterion(outputs,data.unsqueeze(0))
      8

c:\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

<ipython-input-90-467a3f84a03f> in forward(self, x)
     29         print(out.shape)
     30
---> 31         out = self.fc(out)
     32         print(out.shape)
     33

c:\python38\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

c:\python38\lib\site-packages\torch\nn\modules\linear.py in forward(self, input)
     85
     86     def forward(self, input):
---> 87         return F.linear(input, self.weight, self.bias)
     88
     89     def extra_repr(self):

c:\python38\lib\site-packages\torch\nn\functional.py in linear(input, weight, bias)
   1608     if input.dim() == 2 and bias is not None:
   1609         # fused op is marginally faster
-> 1610         ret = torch.addmm(bias, input, weight.t())
   1611     else:
   1612         output = input.matmul(weight.t())

RuntimeError: size mismatch, m1: [1 x 1018112], m2: [512 x 10] at C:\w\b\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:41

I realize that if you have m1: [a * b] and m2: [c * d] then b and c have to be the same value, but I am not sure, what is the best way to resize my images?


title: PyTorch TypeError: 'ToTensor' object is not iterable

body:
I'm trying to print an image name at each iteration. However, I'm getting an error TypeError: 'ToTensor' object is not iterable. Please could some advise where I am going wring please? Many Thanks
from torchvision import datasets
import torch.utils.data
from torch.utils.data import DataLoader
from torchvision import transforms
from dataset2 import CellsDataset
from torchvision import datasets
import torch
import torchvision
import torchvision.transforms as transforms

class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

# override the __getitem__ method. this is the method that dataloader calls
def __getitem__(self, index):
    # this is what ImageFolder normally returns
    original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
    # the image file path
    path = self.imgs[index][0]
    # make a new tuple that includes original and the path
    tuple_with_path = (original_tuple + (path,))
    return tuple_with_path

# EXAMPLE USAGE:
# instantiate the dataset and dataloader
data_dir = "/Users/nubstech/Documents/GitHub/CellCountingDirectCount/Eddata/"
dataset = ImageFolderWithPaths(data_dir) # our custom dataset
#dataloader = DataLoader(dataset)
transform = transforms.Compose([
    # you can add other transformations in this list
    transforms.ToTensor()
])
dataset = DataLoader(data_dir, transforms.Compose(transforms.ToTensor()))
dataloader = torch.utils.DataLoader(dataset)

# iterate over data
for inputs, labels, paths in dataloader:
    # use the above variables freely
   print(inputs, labels, paths)

Traceback Message:
Traceback (most recent call last):
  File "file_location2.py", line 37, in <module>
    dataset = DataLoader(data_dir, transforms.Compose(transforms.ToTensor()))
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 219, in __init__
    batch_sampler = BatchSampler(sampler, batch_size, drop_last)
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 190, in __init__
    "but got batch_size={}".format(batch_size))
  File "/Users/nubstech/opt/anaconda3/envs/Cells_Counting/lib/python3.7/site-packages/torchvision/transforms/transforms.py", line 66, in __repr__
    for t in self.transforms:
TypeError: 'ToTensor' object is not iterable


title: PyTorch nn.CrossEntropyLoss runtime dimension out of range error

body:
I'm currently implementing the continuous bag-of-words (CBOW) model using PyTorch. I'm facing some problems when implementing the cross entropy loss, though. Here's the portion of code that's causing the problem:

for idx, sample in enumerate(self.train_data):
    x = torch.tensor(sample[0], dtype=torch.long)
    y = np.zeros(shape=(self.vocab_size)) # self.vocab_size = 85,000
    y[int(sample[1])] = np.float64(1)
    y = torch.tensor(y, dtype=torch.long)

    if torch.cuda.is_available():
        x = x.cuda()
        y = y.cuda()

    optimizer.zero_grad()

    output = self.model(x) # output's shape is the same as self.vocab_size
    loss = criterion(output, y)
    loss.backward()
    optimizer.step()

To briefly explain my code, the model that I've implemented basically outputs the averaged embedding values of a context array and performs a linear projection to project them into a shape that's identical to the size of the vocabulary. Then we run this array through a softmax function.

The contents of self.train_data are basically (context, target_word) pairs. y is a one-hot encoded array of the token.

I'm aware that the second input to nn.CrossEntropyLoss is C = # of classes, but I'm not sure where my code went wrong. The vocabulary size is 85,000 and so aren't the number of class 85,000?

If I change the input to

loss = criterion(output, 85000)

I get the same error:

*** RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

What am I doing wrong, and how should I understand the input to PyTorch's cross entropy loss?

Thanks.


title: What is the first parameter (gradients) of the backward method, in pytorch?

body:
We have the following code from the pytorch documentation:

x = torch.randn(3)
x = Variable(x, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
    y = y * 2

gradients = torch.FloatTensor([0.1, 1.0, 0.0001])
y.backward(gradients)

What exactly is the gradients parameter that we pass into the backward method and based on what do we initialize it?


title: How to use pytorch DataLoader with a 3-D matrix for LSTM input?

body:
I have a dataset of 3-D(time_stepinputsizetotal_num) matrix which is a .mat file. I want to use DataLoader to get a input dataset for LSTM which batch_size is 5. My code is as following:

file_path = "…/database/frameLength100/notOverlap/a.mat"
mat_data = s.loadmat(file_path)
tensor_data = torch.from_numpy(mat_data[‘a’]) #Tensor

class CustomDataset(Dataset):

def __init__(self, tensor_data):
    self.tensor_data = tensor_data

def __getitem__(self, index):
    data = self.tensor_data[index]
    label = 1;
    return data, label

def __len__(self):
    return len(self.tensor_data)
custom_dataset = CustomDataset(tensor_data=tensor_data)
train_loader = DataLoader(dataset=custom_dataset, batch_size=5, shuffle=True)

I think the code is wrong but I have no idea how to correct it. What makes me confused is how how can I make DataLoader know which dimension is ‘total_num’ so that I get the dataset which batch size is 5.


title: Could I use BERT to Cluster phrases with pre-trained model

body:
I found it was a failure that I had used Gensim with GoogleNews pre-trained model to cluster phrases like:

knitting
knit loom
loom knitting
weaving loom
rainbow loom
home decoration accessories
loom knit/knitting loom
...

I am advised that GoogleNews model does't have the phrases in it. The phrases I have are a little specific to GoogleNews model while I don't have corpus to train a new model. I have only the phrases. And now I am considering to turn to BERT. But could BERT do that as I expected as above? Thank you.


title: PyTorch - Custom ReLU squared Implementation

body:
I work on a project and I want to implement the ReLU squared activation function (max{0,x^2}). Is it ok to call it like:

# example code
    def forward(self, x):
        s = torch.relu(x**2)
        return s

Or should I implement the activation function on my own? In the second case could you please provide me an example on how to do so?

Thanks a lot!


title: index selection in case of conflict in pytorch Argmax

body:
I have been trying to learn tensor operations and this one has thrown me for a loop.
Let us say I have one tensor t:

    t = torch.tensor([
        [1,0,0,2],
        [0,3,3,0],
        [4,0,0,5]
    ], dtype  = torch.float32)

Now this is a rank 2 tensor and we can apply argmax for each rank/dimension.
let us say we apply it for dim = 1

t.max(dim = 1)
(tensor([2., 3., 5.]), tensor([3, 2, 3]))

Now we can see that the result is as expected the tensor along dim =1 has 2,3,and 5 as the max elements. But there is a conflict on 3. There are two values that are exactly similar.
How is it resolved? is it arbitrarily chosen? Is there an order for selecting like L-R, higher index value?
I'd appreciate any insights into how this is resolved!


title: Type mismatch in pytorch

body:
I am trying my hands on PyTorch.
I am getting this error:

RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'weight' in call to _thnn_conv2d_forward

This is my code(shamelessly copied from a online tutorial):

class Net(Module):
    def __init__(self):
        super(Net,self).__init__()

        self.cnn_layers = Sequential(
            Conv2d(1,4,kernel_size=3,stride=1,padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2,stride=2),

            Conv2d(4,4,kernel_size=3,stride=1,padding=1),
            BatchNorm2d(4),
            ReLU(inplace=True),
            MaxPool2d(kernel_size=2,stride=2)
        )
        self.linear_layers = Sequential(
            Linear(900,10)
        )

    def forward(self,x):
#         self.weights = self.weights.double()
        x = self.cnn_layers(x)
        x = x.view(x.size(0),-1)
        x = self.linear_layers(x)
        return x

tdata = dt.Data("train")
train_x = torch.from_numpy(tdata.get_train()[0].reshape(925,1,300,300))
train_y = torch.from_numpy(tdata.get_train()[1].astype(int))
val_x = torch.from_numpy(tdata.get_test()[0].reshape(102,1,300,300))
val_y = torch.from_numpy(tdata.get_test()[1].astype(int))
print(val_y.shape)
plt.imshow(tdata.get_train()[0][100],cmap='gray')
plt.show()

model = Net()
# defining the optimizer
optimizer = Adam(model.parameters(), lr=0.07)
# defining the loss function
criterion = CrossEntropyLoss()
# checking if GPU is available
if torch.cuda.is_available():
    model = model.cuda()
    criterion = criterion.cuda()

print(model)

def train(epoch):
    model.train()
    tr_loss = 0
    # getting the training set
    x_train, y_train = Variable(train_x.double()), Variable(train_y.double())
    # getting the validation set
    x_val, y_val = Variable(val_x), Variable(val_y)
    # converting the data into GPU format
    if torch.cuda.is_available():
        x_train = x_train.cuda()
        y_train = y_train.cuda()
        x_val = x_val.cuda()
        y_val = y_val.cuda()

    # clearing the Gradients of the model parameters
    optimizer.zero_grad()

    # prediction for training and validation set
    output_train = model(x_train.double())
    output_val = model(x_val)

    # computing the training and validation loss
    loss_train = criterion(output_train, y_train)
    loss_val = criterion(output_val, y_val)
    train_losses.append(loss_train)
    val_losses.append(loss_val)

    # computing the updated weights of all the model parameters
    loss_train.backward()
    optimizer.step()
    tr_loss = loss_train.item()
    if epoch%2 == 0:
        # printing the validation loss
        print('Epoch : ',epoch+1, '\t', 'loss :', loss_val)

n_epochs = 25
# empty list to store training losses
train_losses = []
# empty list to store validation losses
val_losses = []
# training the model
for epoch in range(n_epochs):
    train(epoch)

tdata.get_train() and tdata.get_test() returns a tuple (numpy(dtype='double'),numpy(dtype='int') )
I think weights are an internal data structure. So, its type should be adjusted by PyTorch itself. What is the problem here?


title: Compute cross entropy loss for classification in pytorch

body:
I am trying to build two neural network for classification. One for Binary and the second is for multi-class classification. I am trying to use the torch.nn.CrossEntropyLoss() as a loss function, but I try to train my first neural network I get the following error:

multi-target not supported at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THNN/generic/ClassNLLCriterion.c:22

From my analysis, I found that the my dataset has two problems that caused the error.

My data set is one hot encoded. I used one hot encoding to pre processes my dataset. The first target Y_binary variable has the shape of torch.Size([125973, 1]) full of 0s and 1 indicating classes 'No' and 'Yes'.
My data has the wrong dimensions? I found that I can't use a simple vector with the cross entropy loss function. Some people used the following code to reshape their target vector before feeding to the loss function.

out = out.permute(0, 2, 3, 1).contiguous().view(-1, class_number)

But I didn't really understand the reasoning behind this code. But it seems for my that I need to keep track of the following variables: Class_Number, Batch_size, Dimension_Output. For my code here are the dimensions

X_train.shape: (125973, 122)
Y_train2.shape: (125973, 1)
batch_size = 64
K = len(set(Y_train2)) # Binary classification For multi class classification use K = len(set(Y_train5))

Should the target value be one hot encoded? If not, how I can feed a nominal feature to the loss function?
If I use reshape the output, can you help me do this for my code ?

I am trying to use this loss function for both my neural networks.

Thank you in advance,


title: Cuda error: device side assert triggered - only after certain number of batches

body:
I am trying to put a dataset through a neural network. It is running on a Google Cloud virtual machine using a Tesla V100 GPU. However, before I can finish training a single epoch, I get an error message: "Cuda error: device side assert triggered". I think the problem may be in my data, but I have no idea where and I'm not sure what the problem is exactly (but I tested the code with a different dataset and it ran fine).

The thing that is odd is that the network actually runs for some time before triggering the error. I had it print every time it finished a batch and sometimes it finishes 60+ batches, sometimes 80+, I've even gotten it to finish as many as 140 batches (given the size of my data and my batches, there are 200 batches in each epoch). No matter how many it finishes, it eventually triggers this error and has not completed an epoch.

I tried setting CUDA_LAUNCH_BLOCKING = 1 and did not get any better error message. I of course made sure the neural network has the right number of input and output parameters (this is given because it works for the first however many batches). I also standardized the inputs. Some were really large and some were closes to zero so I normalized them to all fall in the range [-1,1]. Certainly the network should be able to handle that but it still causes a problem.

Here is my training loop which WORKS with a different data set. It is always the line "loss.backward()" that eventually triggers the error message.

CUDA_LAUNCH_BLOCKING = 1

start = time.time()
for epoch in range(1,6):

    # Decrease learning rate at epoch 3 and 5
    if epoch == 3 or epoch == 5:
        lr = lr/3

    # Setup optimizer
    optimizer = optim.SGD(net.parameters(), lr=lr)

    # Initialize stats to zeros to track network's progress
    running_loss = 0
    running_error = 0
    num_batches = 0

    # Shuffle indices to train randomly
    shuffled_indices = torch.randperm(50000)

    for count in range(0, 50000, bs):

        # Clear gradient before each iteration
        optimizer.zero_grad()

        # Setup indices for minibatch
        if (count + bs > 50000):
            indices_list = shuffled_indices[count : ].tolist() + shuffled_indices[ : (count + bs) - 50000].tolist()
            indices = torch.Tensor(indices_list)
        else:
            indices = shuffled_indices[count : count + bs]

        # Create minibatch
        minibatch_data = train_data[indices]
        minibatch_label = train_label[indices]

        # Send minibatch to gpu for training
        minibatch_data = minibatch_data.to(device)
        minibatch_label = minibatch_label.to(device)
        temp = minibatch_data - mean

        # Standardize entries with mean and std
        inputs = ((minibatch_data - mean) / std).view(bs, 33)

        # Begin tracking changes
        inputs.requires_grad_()

        # Forward inputs through the network
        scores = net(inputs)

        print(scores[:2])
        print(minibatch_label)

        # Compute loss
        loss = criterion(scores, minibatch_label)

        # Back propogate neural network
        loss.backward()

        # Do one step of stochastic gradient descent
        optimizer.step()

        # Update summary statistics
        with torch.no_grad():
            num_batches += 1
            error = get_error(scores, minibatch_label)
            running_error += error
            running_loss += loss.item()

        print("success: ", num_batches)

    # At the end of each epoch, compute and print summary statistics
    total_error = running_error / num_batches
    avg_loss = running_loss / num_batches
    print('Epoch: ', epoch)
    print('Time: ', time.time(), '\t Loss: ', avg_loss, '\t Error (%): ', total_error * 100)

Here is my dataset formatting and normalizing:

train_list_updated = []
train_label_list = []
for entry in train_list[1:]:
    entry[0] = string_to_int(entry[0])
    entry[1] = handedness[entry[1]]
    entry[2] = string_to_int(entry[2])
    entry[3] = handedness[entry[3]]
    entry[4] = string_to_int(entry[4])
    entry[5] = string_to_int(entry[5])
    entry[6] = string_to_int(entry[6])
    entry[17] = entry[17].replace(':','')
    entry[-3] = pitch_types[entry[-3]]
    entry[-2] = pitch_outcomes[entry[-2]]
    train_label_list.append(entry[-2])
    del entry[-1]
    del entry[-1]
    del entry[-3]
    train_list_updated.append(entry)

final_train_list = []
for entry in train_list_updated:
    for index in range(len(entry)):
        try:
            entry[index] = float(entry[index])
        except:
            entry[index] = 0.
    final_train_list.append(entry)

# Do the same for the test data
test_list_updated = []
for entry in test_list[1:]:
    entry[0] = string_to_int(entry[0])
    entry[1] = handedness[entry[1]]
    entry[2] = string_to_int(entry[2])
    entry[3] = handedness[entry[3]]
    entry[4] = string_to_int(entry[4])
    entry[5] = string_to_int(entry[5])
    entry[6] = string_to_int(entry[6])
    entry[17] = entry[17].replace(':','')
    entry[-3] = pitch_types[entry[-3]]
    del entry[-1]
    del entry[-1]
    del entry[-3]
    test_list_updated.append(entry)

final_test_list = []
for entry in test_list_updated:
    for index in range(len(entry)):
        try:
            entry[index] = float(entry[index])
        except:
            entry[index] = 0.
    final_test_list.append(entry)

# Create tensors of test and train data
train_data = torch.tensor(final_train_list)
train_label = torch.tensor(train_label_list)
test_data = torch.tensor(final_test_list)

And normalizing:

max_indices = torch.argmax(train_data, dim = 0)
min_indices = torch.argmin(train_data, dim = 0)

max_values = []
min_values = []
for i in range(33):
    max_idx = max_indices[i].item()
    min_idx = min_indices[i].item()
    max_val = train_data[max_idx][i]
    min_val = train_data[min_idx][i]
    max_values.append(max_val)
    min_values.append(min_val)

max_values = torch.Tensor(max_values)
min_values = torch.Tensor(min_values)
ranges = max_values - min_values

min_values = min_values.view(1, 33)
min_values = torch.repeat_interleave(min_values, 582205, dim = 0)
ranges = ranges.view(1, 33)
ranges = torch.repeat_interleave(ranges, 582205, dim = 0)

train_data = train_data - min_values
train_data = 2 * (train_data / ranges)
train_data = train_data - 1

And here's my net (a lot is commented out since I thought maybe there was an issue with the gradient zeroing or something. A five layer neural network should definitely not cause a problem though):

"""
DEFINING A NEURAL NETWORK
"""

# Define a fifteen layer artificial neural network
class fifteen_layer_net(nn.Module):
    def __init__(self):
        super().__init__()

        self.linear1 = nn.Linear(33, 200)
        self.linear2 = nn.Linear(200, 250)
        self.linear3 = nn.Linear(250, 300)
        self.linear4 = nn.Linear(300, 350)
        self.linear5 = nn.Linear(350, 7)
#         self.linear6 = nn.Linear(400, 450)
#         self.linear7 = nn.Linear(450, 500)
#         self.linear8 = nn.Linear(500, 450)
#         self.linear9 = nn.Linear(450, 400)
#         self.linear10 = nn.Linear(400, 350)
#         self.linear11 = nn.Linear(350, 300)
#         self.linear12 = nn.Linear(300, 250)
#         self.linear13 = nn.Linear(250, 200)
#         self.linear14 = nn.Linear(200, 150)
#         self.linear15 = nn.Linear(150, 7)

    def forward(self, x):
        x = self.linear1(x)
        x = F.relu(x)
        x = self.linear2(x)
        x = F.relu(x)
        x = self.linear3(x)
        x = F.relu(x)
        x = self.linear4(x)
        x = F.relu(x)
        scores = self.linear5(x)
#         x = F.relu(x)
#         x = self.linear6(x)
#         x = F.relu(x)
#         x = self.linear7(x)
#         x = F.relu(x)
#         x = self.linear8(x)
#         x = F.relu(x)
#         x = self.linear9(x)
#         x = F.relu(x)
#         x = self.linear10(x)
#         x = F.relu(x)
#         x = self.linear11(x)
#         x = F.relu(x)
#         x = self.linear12(x)
#         x = F.relu(x)
#         x = self.linear13(x)
#         x = F.relu(x)
#         x = self.linear14(x)
#         x = F.relu(x)
#         scores = self.linear15(x)

        return scores

Network should output scores, compute a loss using cross entropy loss criterion, and then do one step of stochastic gradient descent. This works for awhile and then mysteriously breaks. I have no idea why.

Any help is greatly appreciated.

Thanks in advance.


title: Something wrong with my checkpoint file when using torch.load()

body:
When I use torch.load to load one checkpoint:
torch.load('./latest_net_G.pth', map_location='cpu')

I got the runtime error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/opt/conda/lib/python3.7/tarfile.py in nti(s)
    186             s = nts(s, "ascii", "strict")
--> 187             n = int(s.strip() or "0", 8)
    188         except ValueError:

ValueError: invalid literal for int() with base 8: '_v2\nq\x03(('

During handling of the above exception, another exception occurred:

InvalidHeaderError                        Traceback (most recent call last)
/opt/conda/lib/python3.7/tarfile.py in next(self)
   2288             try:
-> 2289                 tarinfo = self.tarinfo.fromtarfile(self)
   2290             except EOFHeaderError as e:

/opt/conda/lib/python3.7/tarfile.py in fromtarfile(cls, tarfile)
   1094         buf = tarfile.fileobj.read(BLOCKSIZE)
-> 1095         obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)
   1096         obj.offset = tarfile.fileobj.tell() - BLOCKSIZE

/opt/conda/lib/python3.7/tarfile.py in frombuf(cls, buf, encoding, errors)
   1036
-> 1037         chksum = nti(buf[148:156])
   1038         if chksum not in calc_chksums(buf):

/opt/conda/lib/python3.7/tarfile.py in nti(s)
    188         except ValueError:
--> 189             raise InvalidHeaderError("invalid header")
    190     return n

InvalidHeaderError: invalid header

During handling of the above exception, another exception occurred:

ReadError                                 Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    555         try:
--> 556             return legacy_load(f)
    557         except tarfile.TarError:

/opt/conda/lib/python3.7/site-packages/torch/serialization.py in legacy_load(f)
    466
--> 467         with closing(tarfile.open(fileobj=f, mode='r:', format=tarfile.PAX_FORMAT)) as tar, \
    468                 mkdtemp() as tmpdir:

/opt/conda/lib/python3.7/tarfile.py in open(cls, name, mode, fileobj, bufsize, **kwargs)
   1590                 raise CompressionError("unknown compression type %r" % comptype)
-> 1591             return func(name, filemode, fileobj, **kwargs)
   1592

/opt/conda/lib/python3.7/tarfile.py in taropen(cls, name, mode, fileobj, **kwargs)
   1620             raise ValueError("mode must be 'r', 'a', 'w' or 'x'")
-> 1621         return cls(name, mode, fileobj, **kwargs)
   1622

/opt/conda/lib/python3.7/tarfile.py in __init__(self, name, mode, fileobj, format, tarinfo, dereference, ignore_zeros, encoding, errors, pax_headers, debug, errorlevel, copybufsize)
   1483                 self.firstmember = None
-> 1484                 self.firstmember = self.next()
   1485

/opt/conda/lib/python3.7/tarfile.py in next(self)
   2300                 elif self.offset == 0:
-> 2301                     raise ReadError(str(e))
   2302             except EmptyHeaderError:

ReadError: invalid header

During handling of the above exception, another exception occurred:

RuntimeError                              Traceback (most recent call last)
<ipython-input-15-2abbf3aab3ae> in <module>
----> 1 torch.load('multi_task/checkpoints/latest_pet/latest_net_G.pth.tar', map_location='cpu')

/opt/conda/lib/python3.7/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    385         f = f.open('rb')
    386     try:
--> 387         return _load(f, map_location, pickle_module, **pickle_load_args)
    388     finally:
    389         if new_fd:

/opt/conda/lib/python3.7/site-packages/torch/serialization.py in _load(f, map_location, pickle_module, **pickle_load_args)
    558             if zipfile.is_zipfile(f):
    559                 # .zip is used for torch.jit.save and will throw an un-pickling error here
--> 560                 raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name))
    561             # if not a tarfile, reset file offset and proceed
    562             f.seek(0)

RuntimeError: multi_task/checkpoints/latest_pet/latest_net_G.pth.tar is a zip archive (did you mean to use torch.jit.load()?)

And here's how I save the model:
    def save_networks(self, epoch):
        """Save all the networks to the disk.

        Parameters:
            epoch (int) -- current epoch; used in the file name '%s_net_%s.pth' % (epoch, name)
        """
        for name in self.model_names:
            if isinstance(name, str):
                save_filename = '%s_net_%s.pth' % (epoch, name)
                save_path = os.path.join(self.save_dir, save_filename)
                net = getattr(self, 'net' + name)

                if len(self.gpu_ids) > 0 and torch.cuda.is_available():
                    if name == 'Rgr':
                        torch.save(net.state_dict(), save_path)
                    else:
                        torch.save(net.module.cpu().state_dict(), save_path)
                        net.cuda(self.gpu_ids[0])
                else:
                    if name == 'Rgr':
                        torch.save(net.state_dict(), save_path)
                    else:
                        torch.save(net.cpu().state_dict(), save_path)

I don't know what's wrong with my checkpoint file. Because I can actually load my other checkpoint files successfully. Plus, my pytorch version is 1.1.0. Could you help me with this problem?
Thank you.


title: How to merge two learning rate schedulers in PyTorch?

body:
I want to use OneCycleLR for warming up the LR from a low value, then decreasing the LR from peak with ReduceLROnPlateau.

I understand that I could simulate ReduceLROnPlateau manually after increasing LR with OneCycleLR, but is there a simpler way to use these two schedulers together directly?

Thanks in advance.


title: tensorflow's Timedistributed equivalent in pyTorch

body:
Is there any equivalent implementation of tensorflow.keras.layers.Timedistributed for pytorch?
I am trying to build something like
Timedistributed(Resnet50()).


title: Remove certain elements of all the tensors in a list of dictionary of tensors in pytorch

body:
I am new to pytorch, and trying to learn by playing with simple codes. What I have is a list of length zero, populated by a dictionary, where values in this dictionary are tensors. This is what this list looks like:

A = [{'boxes': tensor([[ 142.1232,  142.9373, 1106.0452,  971.3792],
    [ 259.1277,  618.4834, 1100.1293, 1028.8989],
    [ 232.1346,  692.5888,  763.3408, 1028.6766],
    [ 206.8070,  312.2080, 1137.1434, 1013.4373],
    [ 495.9471,  675.7287,  978.5932, 1012.7568]], grad_fn=<StackBackward>), 'labels': tensor([16,  1,  1,  1,  1]), 'scores': tensor([0.9988, 0.9489, 0.5228, 0.3500, 0.0639], grad_fn=<IndexBackward>), 'masks': tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]]], grad_fn=<UnsqueezeBackward0>)}]

This list is the output of Mask-RCNN and I want to remove certain element of all the tensors in the nested dictionary. In this case, I only want to keep information related with class label "1". Information related to each class ('boxes', 'labels', 'scores', and 'masks') are all in the same location(s) (index) in each of the tensors. So, I find the index of all the "1"s in the tensor with key "labels":

idxOfClass = [i for i, x in enumerate(list(pred[0]['labels'])) if x == 1]

which gives me: [1, 2, 3, 4].
Then, I want to keep all the values located at indices in idxOfClass in all of the tensors in the nested dictionary. If I do something like this:

Anew = [{pred[0]['boxes'][idxOfClass],pred[0]['labels'][idxOfClass],pred[0]['masks'][idxOfClass],pred[0]['scores'][idxOfClass]}]

I get:

[{tensor([[ 259.1277,  618.4834, 1100.1293, 1028.8989],
    [ 232.1346,  692.5888,  763.3408, 1028.6766],
    [ 206.8070,  312.2080, 1137.1434, 1013.4373],
    [ 495.9471,  675.7287,  978.5932, 1012.7568]], grad_fn=
    <IndexBackward>), tensor([0.9489, 0.5228, 0.3500, 0.0639],
    grad_fn=
    <IndexBackward>), tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]],

    [[[0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      ...,
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.],
      [0., 0., 0.,  ..., 0., 0., 0.]]]], grad_fn=<IndexBackward>),
    tensor([1, 1, 1, 1])}]

But, this is not a list of dictionary with tensors as the values of the dictionary. This is a list of tensors, with no key-value structure of the nested dictionary. My question is, "is there any way to keep the original structure of the list when I remove certain elements of all the tensors using indices of specific elements?".


title: Why does by torch.optim.SGD method learning rate change?

body:
With SGD learning rate should not be changed during epochs but it is. Help me understand why it happens please and how to prevent this LR changing?

import torch
params = [torch.nn.Parameter(torch.randn(1, 1))]
optimizer = torch.optim.SGD(params, lr=0.9)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9)
for epoch in range(5):
    print(scheduler.get_lr())
    scheduler.step()

Output is:

[0.9]
[0.7290000000000001]
[0.6561000000000001]
[0.5904900000000002]
[0.5314410000000002]

My torch version is 1.4.0


title: High CPU consumption - PyTorch

body:
Although I saw several questions/answers about my problem, I could not solve it yet. I am trying to run a basic code from GitHub for training GAN. Although the code is working on GPU, the CPU usage is 100% (even more) during training.

https://github.com/eriklindernoren/PyTorch-GAN/blob/master/implementations/wgan_gp/wgan_gp.py

In order to use my data, I added the following data loading code that I checked on PyTorch official documentation.

def Read_LMDB(root, classes):
    d1 = torchvision.datasets.LSUN(root=root, classes=classes, transform=transforms.Compose(
                [transforms.Resize((opt.img_size, opt.img_size)), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]))
    dataloader = torch.utils.data.DataLoader(d1, batch_size=opt.batch_size, shuffle=True)
    return dataloader

def Read_from_folder(folder_name):
    imagefolder = datasets.ImageFolder(folder_name,transform=transforms.Compose(
        [transforms.Resize(opt.img_size), transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))]))
    dataloader = torch.utils.data.DataLoader(imagefolder,batch_size=opt.batch_size,shuffle=True)
    return dataloader

However, still the same problem. It does not matter which functions I used, the CPU usage is 100%. Using such high CPU is not normal since the training is performed on GPU (and there is no problem with GPU usage. The GPU memory usage, utility, and the training speed are okay).

Could anyone have any idea? Thanks!

Best.


title: PyTorch naive single label classification with embedding layer fails at random

body:
I am new to PyTorch and I am trying out the Embedding Layer.

I wrote a naive classification task, where all the inputs are the equal and all the labels are set to 1.0. I hence expect the model to learn quickly to predict 1.0.

The input is always 0, which is fed into a nn.Embedding(1,32) layer, followed by nn.Linear(32,1) and nn.Relu().

However, an unexpected and undesired behavior occurs: training outcome is different for different times I run the code.
For example,

setting the random seed to 10, model converges: loss decreases and model always predicts 1.0
setting the random seed to 1111, model doesn't converge: loss doesn't decrease and model always predicts 0.5. In those cases the parameters are not updated

Here is the minimal, replicable code:

from torch.nn import BCEWithLogitsLoss
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torch.autograd import Variable
from torch.utils.data import Dataset
import torch

class MyModel(nn.Module):

    def __init__(self):
        super(MyModel, self).__init__()
        self.vgg_fc = nn.Linear(32, 1)
        self.relu = nn.ReLU()
        self.embeddings = nn.Embedding(1, 32)

    def forward(self, data):
        emb = self.embeddings(data['index'])
        return self.relu(self.vgg_fc(emb))

class MyDataset(Dataset):

    def __init__(self):
        pass
    def __len__(self):
        return 1000
    def __getitem__(self, idx):
        return {'label': 1.0, 'index': 0}

def train():
    model = MyModel()
    db = MyDataset()
    dataloader = DataLoader(db, batch_size=256, shuffle=True, num_workers=16)

    loss_function = BCEWithLogitsLoss()
    optimizer_rel = optim.SGD(model.parameters(), lr=0.1)

    for epoch in range(50):
        for i_batch, sample_batched in enumerate(dataloader):

            model.zero_grad()
            out = model({'index': Variable(sample_batched['index'])})

            labels = Variable(sample_batched['label'].type(torch.FloatTensor).view(sample_batched['label'].shape[0], 1))

            loss = loss_function(out, labels)
            loss.backward()
            optimizer_rel.step()
            print 'Epoch:', epoch, 'batch', i_batch, 'Tr_Loss:', loss.data[0]
    return model

if __name__ == '__main__':

    # please, try seed 10 (converge) and seed 1111 (fails)
    torch.manual_seed(10)
    train()

Without specifying the random seed, different runs have different outcome.

Why is, in those cases, the model unable to learn such a easy task?
Is there any mistake in the way I use nn.Embedding layer?

Thank you


title: Torch.cat memory explode

body:
I tried to replace Conv2d with this module on ResNet50.

class SubtractedConv(nn.Module):
    def __init__(self, input_ch, output_ch, kernels, stride=1):
        super().__init__()
        self.point_wise = nn.Conv2d(input_ch, output_ch//2, 1, bias=False, stride=stride)
        self.depth_wise = nn.Conv2d(output_ch // 2, output_ch // 2, kernels, groups=output_ch // 2, bias=False, padding=kernels // 2)
        self.low_pass = nn.Conv2d(output_ch // 2, output_ch // 2, kernels, bias=False, padding=kernels // 2)
    def forward(self, x):
        p = self.point_wise(x)
        d = self.depth_wise(p)
        d -= p
        l = self.low_pass(p)
        x = torch.cat((d, l), 1)
        return x

The expected outputs should have same channel with normal Conv2d but I got cuda out of memory at torch.cat().
I wonder why? And how to deal with this?

I use Pytorch 0.4.0 and run on Tesla P100, image size 224*224, batch size 16.

In fact, something like this works on Keras and it has less parameters (16M in ResNet50 compare to 25M with normal Conv2D).

def subtractedconv(input_tensor, kernel_size, filters, stride=1):
    p = kl.Conv2D(filters//2, (1, 1), use_bias=False, strides=stride, padding='same')(input_tensor)
    d = DepthwiseConv2D(kernel_size, use_bias=False, padding='same')(p)
    d = kl.subtract([d, p])
    l = kl.Conv2D(filters//2, kernel_size, use_bias=False, padding='same')(p)
    x = kl.Concatenate(axis=-1)([d, l])
    return x


title: Pytorch summary only works for one specific input size for U-Net

body:
I am trying to implement the UNet architecture in Pytorch. When I print the model using print(model) I get the correct architecture:

but when I try to print the summary using (or any other input size for that matter):

from torchsummary import summary
summary(model, input_size=(13, 572, 572))

I get an error:

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 70 and 71 in dimension 2 at /Users/distiller/project/conda/conda-bld/pytorch_1579022061893/work/aten/src/TH/generic/THTensor.cpp:612

However, it works perfectly if I give the input_size as input_size=(3, 224, 224))( like it worked for this person here). I am so baffled.

Can someone help me what's wrong?

Edit: I have used the model architecture from here.


Warning (from warnings module):
File "C:\Users\A\AppData\Local\Programs\Python\Python36\lib\site-packages\torchaudio\extension\extension.py", line 14
warnings.warn('torchaudio C++ extension is not available.')
UserWarning: torchaudio C++ extension is not available.
Can anyone help me with this...


title: PyTorch 0.4 LSTM: Why does each epoch get slower?

body:
I have a toy model of a PyTorch 0.4 LSTM on a GPU.  The overall idea of the toy problem is that I define a single 3-vector as an input, and define a rotation matrix R.  The ground truth targets are then a sequence of vectors:  At T0, the input vector; at T1 the input vector rotated by R; at T2 the input rotated by R twice, etc.  (The input is padded the output length with zero-inputs after T1)

The loss is the average L2 difference between ground truth and outputs.  The rotation matrix, construction of the input/output data, and loss functions are probably not of interest, and not shown here.

Never mind that the results are pretty terrible: Why does this become successively slower with each passing epoch?!

I've shown on-GPU information below, but this happens on the CPU as well (only with larger times.)  The time to execute ten epochs of this silly little thing grows rapidly.  It's quite noticeable just watching the numbers scroll by.

epoch:   0,     loss: 0.1753,   time previous: 33:28.616360 time now: 33:28.622033  time delta: 0:00:00.005673
epoch:  10,     loss: 0.2568,   time previous: 33:28.622033 time now: 33:28.830665  time delta: 0:00:00.208632
epoch:  20,     loss: 0.2092,   time previous: 33:28.830665 time now: 33:29.324966  time delta: 0:00:00.494301
epoch:  30,     loss: 0.2663,   time previous: 33:29.324966 time now: 33:30.109241  time delta: 0:00:00.784275
epoch:  40,     loss: 0.1965,   time previous: 33:30.109241 time now: 33:31.184024  time delta: 0:00:01.074783
epoch:  50,     loss: 0.2232,   time previous: 33:31.184024 time now: 33:32.556106  time delta: 0:00:01.372082
epoch:  60,     loss: 0.1258,   time previous: 33:32.556106 time now: 33:34.215477  time delta: 0:00:01.659371
epoch:  70,     loss: 0.2237,   time previous: 33:34.215477 time now: 33:36.173928  time delta: 0:00:01.958451
epoch:  80,     loss: 0.1076,   time previous: 33:36.173928 time now: 33:38.436041  time delta: 0:00:02.262113
epoch:  90,     loss: 0.1194,   time previous: 33:38.436041 time now: 33:40.978748  time delta: 0:00:02.542707
epoch: 100,     loss: 0.2099,   time previous: 33:40.978748 time now: 33:43.844310  time delta: 0:00:02.865562

The model:

class Sequence(torch.nn.Module):
def __init__ (self):
    super(Sequence, self).__init__()

    self.lstm1 = nn.LSTM(3,30)
    self.lstm2 = nn.LSTM(30,300)
    self.lstm3 = nn.LSTM(300,30)
    self.lstm4 = nn.LSTM(30,3)

    self.hidden1 = self.init_hidden(dim=30)
    self.hidden2 = self.init_hidden(dim=300)
    self.hidden3 = self.init_hidden(dim=30)
    self.hidden4 = self.init_hidden(dim=3)

    self.dense   = torch.nn.Linear(30, 3)
    self.relu    = nn.LeakyReLU()

def init_hidden(self, dim):
    return (torch.zeros(1, 1, dim).to(device)  ,torch.zeros(1, 1, dim).to(device)  )

def forward(self, inputs):
    out1, self.hidden1 = self.lstm1(inputs, self.hidden1)
    out2, self.hidden2 = self.lstm2(out1,   self.hidden2)
    out3, self.hidden3 = self.lstm3(out2,   self.hidden3)
    #out4, self.hidden4 = self.lstm4(out3,   self.hidden4)

    # This is intended to act as a dense layer on the output of the LSTM
    out4               = self.relu(self.dense(out3))

    return out4

The training loop:

sequence = Sequence().to(device)

criterion = L2_Loss()
optimizer = torch.optim.Adam(sequence.parameters())
_, _, _, R = getRotation(np.pi/27, np.pi/26, np.pi/25)

losses = []
date1 = datetime.datetime.now()
for epoch in range(1001):
    # Define input as a Variable-- each row of 3 is a vector, a distinct input
    # Define target directly from input by applicatin of rotation vector
    # Define predictions by running input through model

    inputs       = getInput(25)
    targets      = getOutput(inputs, R)

    inputs       = torch.cat(inputs).view(len(inputs), 1, -1).to(device)
    targets      = torch.cat(targets).view(len(targets), 1, -1).to(device)

    target_preds = sequence(inputs)
    target_preds = target_preds.view(len(target_preds), 1, -1)
    loss = criterion(targets, target_preds).to(device)

    losses.append(loss.data[0])
    if (epoch % 10 == 0):
        date2 = datetime.datetime.now()
        print("epoch: %3d, \tloss: %6.4f, \ttime previous: %s\ttime now: %s\ttime delta: %s" % (epoch, loss.data[0], date1.strftime("%M:%S.%f"), date2.strftime("%M:%S.%f"), date2 - date1))
        date1 = date2
    # Zero out the grads, run the loss backward, and optimize on the grads
    optimizer.zero_grad()
    loss.backward(retain_graph=True)
    optimizer.step()


title: Pytorch geometric: Having issues with tensor sizes

body:
This is the first time I'm using Pytorch and Pytorch geometric. I'm trying to create a simple Graph Neural Network with Pytorch Geometric. I'm creating a custom dataset by following the Pytorch Geometric documentations and extending the InMemoryDataset. After that I split the dataset into training, validation and test dataset which sizes (3496, 437, 439) respectively. These are the number of graphs in each dataset. Here is my simple Neural Network
class Net(torch.nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = GCNConv(dataset.num_node_features, 10)
    self.conv2 = GCNConv(10, dataset.num_classes)

def forward(self, data):
    x, edge_index, batch = data.x, data.edge_index, data.batch
    x = self.conv1(x, edge_index)
    x = F.relu(x)
    x = F.dropout(x, training=self.training)
    x = self.conv2(x, edge_index)

    return F.log_softmax(x, dim=1)

I get this error while training my model, which suggest that there's some issue with my input dimensions. Maybe the reason is behind my batch sizes?
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
File "E:\Users\abc\Anaconda3\lib\site-packages\torch_scatter\scatter.py", line 22, in scatter_add
        size[dim] = int(index.max()) + 1
    out = torch.zeros(size, dtype=src.dtype, device=src.device)
    return out.scatter_add_(dim, index, src)
           ~~~~~~~~~~~~~~~~ <--- HERE
else:
    return out.scatter_add_(dim, index, src)
RuntimeError: index 13654 is out of bounds for dimension 0 with size 678

The error happens specifically on this line of code in the Neural Network,
x = self.conv1(x, edge_index)

EDIT: Added more information about edge_index and explained in more detail about the data that I'm using.
Here are the shapes of the variables that I'm trying to pass
x: torch.Size([678, 43])
edge_index: torch.Size([2, 668])
torch.max(edge_index): tensor(541690)
torch.min(edge_index): tensor(1920)

I'm using a datalist which contains Data(x=node_features, edge_index=edge_index, y=labels) objects. When I'm splitting the dataset into training, validation and test datasets, I get (3496, 437, 439) graphs in each dataset respectively. Originally I tried to create one single graph from my dataset, but I'm not sure how it would work with Dataloader and minibatches.
train_loader = DataLoader(train_dataset, batch_size=batch_size)
val_loader = DataLoader(val_dataset, batch_size=batch_size)
test_loader = DataLoader(test_dataset, batch_size=batch_size)

Here's the code that generates the graph from dataframe. I've tried to create an simple graph where there are just some amount of vertices with some amount of edges connecting them. I've probably overlooked something and that's why I have this issue. I've tried to follow the Pytorch geometric documentation when creating this graph (Pytorch Geometric: Creating your own dataset)
def process(self):
        data_list = []

        grouped = df.groupby('EntityId')
        for id, group in grouped:
            node_features = torch.tensor(group.drop(['Labels'], axis=1).values)
            source_nodes = group.index[1:].values
            target_nodes = group.index[:-1].values
            labels = torch.tensor(group.Labels.values)
            edge_index = torch.tensor([source_nodes, target_nodes])

            data = Data(x=node_features, edge_index=edge_index, y=labels)
            data_list.append(data)

        if self.pre_filter is not None:
            data_list = [data for data in data_list if self.pre_filter(data)]

        if self.pre_transform is not None:
            data_list = [self.pre_transform(data) for data in data_list]

        data, slices = self.collate(data_list)
        torch.save((data, slices), self.processed_paths[0])

If someone could help me with the process of creating a graph on any kind of data and using it with GCNConv, I would appreciate it.


title: A proper way to adjust input size of CNN (e.g. VGG)

body: I want to train VGG on 128x128-sized images. I don't want to rescale them to 224x224 to save GPU-memory and training time. What would be the proper way to do so?


title: How to find built-in function source code in pytorch

body:
I am trying to do research on batch normalization, and had to make some modifications for the pytorch BN code. I dig into the pytorch code and got stuck with torch.nn.functional.batch_norm, which references torch.batch_norm.

The problem is that torch.batch_norm cannot be further found in the torch library. Is there any way I can find the source code of this built-in function and re-implement it? Thanks!


title: How to train a simple linear regression model with SGD in pytorch successfully?

body:
I was trying to train a simple polynomial linear regression model in pytorch with SGD. I wrote some self contained (what I thought would be extremely simple code), however, for some reason my model does not train as I thought it should.

I have 5 points sampled from a sine curve and try to fit it with a polynomial of degree 4. This is a convex problem so GD or SGD should find a solution with zero train error eventually as long as we have enough iterations and small enough step size. For some reason however my model does not train well (even though it seems that it is changing the parameters of the model. Anyone have an idea why? Here is the code (I tried making it self contained and minimal):

import numpy as np
from sklearn.preprocessing import PolynomialFeatures

import torch
from torch.autograd import Variable

from maps import NamedDict

from plotting_utils import *

def index_batch(X,batch_indices,dtype):
    '''
    returns the batch indexed/sliced batch
    '''
    if len(X.shape) == 1: # i.e. dimension (M,) just a vector
        batch_xs = torch.FloatTensor(X[batch_indices]).type(dtype)
    else:
        batch_xs = torch.FloatTensor(X[batch_indices,:]).type(dtype)
    return batch_xs

def get_batch2(X,Y,M,dtype):
    '''
    get batch for pytorch model
    '''
    # TODO fix and make it nicer, there is pytorch forum question
    X,Y = X.data.numpy(), Y.data.numpy()
    N = len(Y)
    valid_indices = np.array( range(N) )
    batch_indices = np.random.choice(valid_indices,size=M,replace=False)
    batch_xs = index_batch(X,batch_indices,dtype)
    batch_ys = index_batch(Y,batch_indices,dtype)
    return Variable(batch_xs, requires_grad=False), Variable(batch_ys, requires_grad=False)

def get_sequential_lifted_mdl(nb_monomials,D_out, bias=False):
    return torch.nn.Sequential(torch.nn.Linear(nb_monomials,D_out,bias=bias))

def train_SGD(mdl, M,eta,nb_iter,logging_freq ,dtype, X_train,Y_train):
    ##
    N_train,_ = tuple( X_train.size() )
    #print(N_train)
    for i in range(nb_iter):
        # Forward pass: compute predicted Y using operations on Variables
        batch_xs, batch_ys = get_batch2(X_train,Y_train,M,dtype) # [M, D], [M, 1]
        ## FORWARD PASS
        y_pred = mdl.forward(batch_xs)
        ## LOSS + Regularization
        batch_loss = (1/M)*(y_pred - batch_ys).pow(2).sum()
        ## BACKARD PASS
        batch_loss.backward() # Use autograd to compute the backward pass. Now w will have gradients
        ## SGD update
        for W in mdl.parameters():
            delta = eta*W.grad.data
            W.data.copy_(W.data - delta)
        ## train stats
        if i % (nb_iter/10) == 0 or i == 0:
            current_train_loss = (1/N_train)*(mdl.forward(X_train) - Y_train).pow(2).sum().data.numpy()
            print('i = {}, current_loss = {}'.format(i, current_train_loss ) )
        ## Manually zero the gradients after updating weights
        mdl.zero_grad()
##
logging_freq = 100
dtype = torch.FloatTensor
## SGD params
M = 3
eta = 0.0002
nb_iter = 20*1000
##
lb,ub = 0,1
f_target = lambda x: np.sin(2*np.pi*x)
N_train = 5
X_train = np.linspace(lb,ub,N_train)
Y_train = f_target(X_train)
## degree of mdl
Degree_mdl = 4
## pseudo-inverse solution
c_pinv = np.polyfit( X_train, Y_train , Degree_mdl )[::-1]
## linear mdl to train with SGD
nb_terms = c_pinv.shape[0]
mdl_sgd = get_sequential_lifted_mdl(nb_monomials=nb_terms,D_out=1, bias=False)
## Make polynomial Kernel
poly_feat = PolynomialFeatures(degree=Degree_mdl)
Kern_train = poly_feat.fit_transform(X_train.reshape(N_train,1))
Kern_train_pt, Y_train_pt = Variable(torch.FloatTensor(Kern_train).type(dtype), requires_grad=False), Variable(torch.FloatTensor(Y_train).type(dtype), requires_grad=False)
train_SGD(mdl_sgd, M,eta,nb_iter,logging_freq ,dtype, Kern_train_pt,Y_train_pt)

the error seems to hover on 2ish:

i = 0, current_loss = [ 2.08996224]
i = 2000, current_loss = [ 2.03536892]
i = 4000, current_loss = [ 2.02014995]
i = 6000, current_loss = [ 2.01307297]
i = 8000, current_loss = [ 2.01300406]
i = 10000, current_loss = [ 2.01125693]
i = 12000, current_loss = [ 2.01162267]
i = 14000, current_loss = [ 2.01296973]
i = 16000, current_loss = [ 2.00951076]
i = 18000, current_loss = [ 2.00967121]

which is weird cuz it should be able to reach zero.

I also plotted the learned function:

the code for the plotting:

##
x_horizontal = np.linspace(lb,ub,1000).reshape(1000,1)
X_plot = poly_feat.fit_transform(x_horizontal)
X_plot_pytorch = Variable( torch.FloatTensor(X_plot), requires_grad=False)
##
fig1 = plt.figure()
#plots objs
p_sgd, = plt.plot(x_horizontal, [ float(f_val) for f_val in mdl_sgd.forward(X_plot_pytorch).data.numpy() ])
p_pinv, = plt.plot(x_horizontal, np.dot(X_plot,c_pinv))
p_data, = plt.plot(X_train,Y_train,'ro')
## legend
nb_terms = c_pinv.shape[0]
legend_mdl = f'SGD solution standard parametrization, number of monomials={nb_terms}, batch-size={M}, iterations={nb_iter}, step size={eta}'
plt.legend(
        [p_sgd,p_pinv,p_data],
        [legend_mdl,f'linear algebra soln, number of monomials={nb_terms}',f'data points = {N_train}']
    )
##
plt.xlabel('x'), plt.ylabel('f(x)')
plt.show()

I actually went ahead and implemented a TensorFlow version. That one does seem to train the model. I tried having both of them match by giving them the same initialization:

mdl_sgd[0].weight.data.fill_(0)

but that still didn't work. Tensorflow code:

graph = tf.Graph()
with graph.as_default():
    X = tf.placeholder(tf.float32, [None, nb_terms])
    Y = tf.placeholder(tf.float32, [None,1])
    w = tf.Variable( tf.zeros([nb_terms,1]) )
    #w = tf.Variable( tf.truncated_normal([Degree_mdl,1],mean=0.0,stddev=1.0) )
    #w = tf.Variable( 1000*tf.ones([Degree_mdl,1]) )
    ##
    f = tf.matmul(X,w) # [N,1] = [N,D] x [D,1]
    #loss = tf.reduce_sum(tf.square(Y - f))
    loss = tf.reduce_sum( tf.reduce_mean(tf.square(Y-f), 0))
    l2loss_tf = (1/N_train)*2*tf.nn.l2_loss(Y-f)
    ##
    learning_rate = eta
    #global_step = tf.Variable(0, trainable=False)
    #learning_rate = tf.train.exponential_decay(learning_rate=eta, global_step=global_step,decay_steps=nb_iter/2, decay_rate=1, staircase=True)
    train_step = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(loss)
    with tf.Session(graph=graph) as sess:
        Y_train = Y_train.reshape(N_train,1)
        tf.global_variables_initializer().run()
        # Train
        for i in range(nb_iter):
            #if i % (nb_iter/10) == 0:
            if i % (nb_iter/10) == 0 or i == 0:
                current_loss = sess.run(fetches=loss, feed_dict={X: Kern_train, Y: Y_train})
                print(f'i = {i}, current_loss = {current_loss}')
            ## train
            batch_xs, batch_ys = get_batch(Kern_train,Y_train,M)
            sess.run(train_step, feed_dict={X: batch_xs, Y: batch_ys})

I also tried changing the initialization but it didn't change anything, which makes sense cuz it shouldn't make a big difference:

mdl_sgd[0].weight.data.normal_(mean=0,std=0.001)

Original post:

https://discuss.pytorch.org/t/how-to-train-a-simple-linear-regression-model-with-sgd-in-pytorch-successfully/9620

This is how it should look like:

SOLUTION:

it seems that there is an issue with the result being returned as a vector instead of a number causing the issue. i.e. the following code fixed things:

    y_pred = model(batch_xs).view(-1) # change this to "y_pred = model(batch_xs)" to get the incorrect results
    loss = (y_pred - batch_ys).pow(2).mean()

which seems completely mysterious to me. Does someone know why this fixed the issue? it just seems like magic.


title: Unexpected and missing keys in state_dict when converting pytorch to onnx

body:
When I convert a '.pth' model from PyTorch to ONNX, an error like Unexpected keys and missing keys occur.

This is my model:

    1 import torch
    2 import torch.onnx
    3 from mmcv import runner
    4 import torch.`enter code here`nn as nn
    5 from mobilenet import MobileNet
    6 # A model class instance (class not shown)
    7 md=MobileNet(1,2)
    8 model = md
    9 device_ids = [0,2,6,7,8]
    10 model = nn.DataParallel(model,device_ids)
    11 #torch.backends.cudnn.benchmark = True
    12 # Load the weights from a file (.pth usually)
    13 runner.load_checkpoint(model,'../mmdetection-        master/work_dmobile/faster_rcnn_r50_fpn_1x/epoch_60.pth')
    14 #model = MMDataParallel(model, device_ids=[0])
    15 #state_dict=torch.load('../mmdetection-master/r.pkl.json')
    16 # Load the weights now into a model net architecture defined by our class
    17 #model.load_state_dict(state_dict)
    18 #model = runner.load_state_dict(state_dict)
    19 model=runner.load_state_dict({k.replace('module.',' '):v for k,v in state_dict['state_dict'].items()})
    20 # Create the right input shape (e.g. for an image)
    21 dummy_input = torch.randn(1, 64, 512, 256)
    22
    23 torch.onnx.export(model, dummy_input, "onnx_model_name.onnx")

And this is the error:

unexpected key in source state_dict: backbone.stem.0.conv.weight, backbone.stem.0.bn.weight, backbone.stem.0.bn.bias, backbone.stem.0.bn.running_mean, backbone.stem.0.bn.running_var, backbone.stem.0.bn.num_batches_tracked, backbone.stem.1.depthwise.0.weight, backbone.stem.1.depthwise.1.weight, backbone.stem.1.depthwise.1.bias, backbone.stem.1.depthwise.1.running_mean, backbone.stem.1.depthwise.1.running_var, backbone.stem.1.depthwise.1.num_batches_tracked, backbone.stem.1.pointwise.0.weight, backbone.stem.1.pointwise.0.bias, backbone.stem.1.pointwise.1.weight, backbone.stem.1.pointwise.1.bias, backbone.stem.1.pointwise.1.running_mean, backbone.stem.1.pointwise.1.running_var, backbone.stem.1.pointwise.1.num_batches_tracked, backbone.conv1.0.depthwise.0.weight, backbone.conv1.0.depthwise.1.weight, backbone.conv1.0.depthwise.1.bias, backbone.conv1.0.depthwise.1.running_mean, backbone.conv1.0.depthwise.1.running_var, backbone.conv1.0.depthwise.1.num_batches_tracked, backbone.conv1.0.pointwise.0.weight, backbone.conv1.0.pointwise.0.bias, backbone.conv1.0.pointwise.1.weight, backbone.conv1.0.pointwise.1.bias, backbone.conv1.0.pointwise.1.running_mean, backbone.conv1.0.pointwise.1.running_var, backbone.conv1.0.pointwise.1.num_batches_tracked, backbone.conv1.1.depthwise.0.weight, backbone.conv1.1.depthwise.1.weight, backbone.conv1.1.depthwise.1.bias, backbone.conv1.1.depthwise.1.running_mean, backbone.conv1.1.depthwise.1.running_var, backbone.conv1.1.depthwise.1.num_batches_tracked, backbone.conv1.1.pointwise.0.weight, backbone.conv1.1.pointwise.0.bias, backbone.conv1.1.pointwise.1.weight, backbone.conv1.1.pointwise.1.bias, backbone.conv1.1.pointwise.1.running_mean, backbone.conv1.1.pointwise.1.running_var, backbone.conv1.1.pointwise.1.num_batches_tracked, backbone.conv2.0.depthwise.0.weight, backbone.conv2.0.depthwise.1.weight, backbone.conv2.0.depthwise.1.bias, backbone.conv2.0.depthwise.1.running_mean, backbone.conv2.0.depthwise.1.running_var, backbone.conv2.0.depthwise.1.num_batches_tracked, backbone.conv2.0.pointwise.0.weight, backbone.conv2.0.pointwise.0.bias, backbone.conv2.0.pointwise.1.weight, backbone.conv2.0.pointwise.1.bias, backbone.conv2.0.pointwise.1.running_mean, backbone.conv2.0.pointwise.1.running_var, backbone.conv2.0.pointwise.1.num_batches_tracked, backbone.conv2.1.depthwise.0.weight, backbone.conv2.1.depthwise.1.weight, backbone.conv2.1.depthwise.1.bias, backbone.conv2.1.depthwise.1.running_mean, backbone.conv2.1.depthwise.1.running_var, backbone.conv2.1.depthwise.1.num_batches_tracked, backbone.conv2.1.pointwise.0.weight, backbone.conv2.1.pointwise.0.bias, backbone.conv2.1.pointwise.1.weight, backbone.conv2.1.pointwise.1.bias, backbone.conv2.1.pointwise.1.running_mean, backbone.conv2.1.pointwise.1.running_var, backbone.conv2.1.pointwise.1.num_batches_tracked, backbone.conv3.0.depthwise.0.weight, backbone.conv3.0.depthwise.1.weight, backbone.conv3.0.depthwise.1.bias, backbone.conv3.0.depthwise.1.running_mean, backbone.conv3.0.depthwise.1.running_var, backbone.conv3.0.depthwise.1.num_batches_tracked, backbone.conv3.0.pointwise.0.weight, backbone.conv3.0.pointwise.0.bias, backbone.conv3.0.pointwise.1.weight, backbone.conv3.0.pointwise.1.bias, backbone.conv3.0.pointwise.1.running_mean, backbone.conv3.0.pointwise.1.running_var, backbone.conv3.0.pointwise.1.num_batches_tracked, backbone.conv3.1.depthwise.0.weight, backbone.conv3.1.depthwise.1.weight, backbone.conv3.1.depthwise.1.bias, backbone.conv3.1.depthwise.1.running_mean, backbone.conv3.1.depthwise.1.running_var, backbone.conv3.1.depthwise.1.num_batches_tracked, backbone.conv3.1.pointwise.0.weight, backbone.conv3.1.pointwise.0.bias, backbone.conv3.1.pointwise.1.weight, backbone.conv3.1.pointwise.1.bias, backbone.conv3.1.pointwise.1.running_mean, backbone.conv3.1.pointwise.1.running_var, backbone.conv3.1.pointwise.1.num_batches_tracked, backbone.conv3.2.depthwise.0.weight, backbone.conv3.2.depthwise.1.weight, backbone.conv3.2.depthwise.1.bias, backbone.conv3.2.depthwise.1.running_mean, backbone.conv3.2.depthwise.1.running_var, backbone.conv3.2.depthwise.1.num_batches_tracked, backbone.conv3.2.pointwise.0.weight, backbone.conv3.2.pointwise.0.bias, backbone.conv3.2.pointwise.1.weight, backbone.conv3.2.pointwise.1.bias, backbone.conv3.2.pointwise.1.running_mean, backbone.conv3.2.pointwise.1.running_var, backbone.conv3.2.pointwise.1.num_batches_tracked, backbone.conv3.3.depthwise.0.weight, backbone.conv3.3.depthwise.1.weight, backbone.conv3.3.depthwise.1.bias, backbone.conv3.3.depthwise.1.running_mean, backbone.conv3.3.depthwise.1.running_var, backbone.conv3.3.depthwise.1.num_batches_tracked, backbone.conv3.3.pointwise.0.weight, backbone.conv3.3.pointwise.0.bias, backbone.conv3.3.pointwise.1.weight, backbone.conv3.3.pointwise.1.bias, backbone.conv3.3.pointwise.1.running_mean, backbone.conv3.3.pointwise.1.running_var, backbone.conv3.3.pointwise.1.num_batches_tracked, backbone.conv3.4.depthwise.0.weight, backbone.conv3.4.depthwise.1.weight, backbone.conv3.4.depthwise.1.bias, backbone.conv3.4.depthwise.1.running_mean, backbone.conv3.4.depthwise.1.running_var, backbone.conv3.4.depthwise.1.num_batches_tracked, backbone.conv3.4.pointwise.0.weight, backbone.conv3.4.pointwise.0.bias, backbone.conv3.4.pointwise.1.weight, backbone.conv3.4.pointwise.1.bias, backbone.conv3.4.pointwise.1.running_mean, backbone.conv3.4.pointwise.1.running_var, backbone.conv3.4.pointwise.1.num_batches_tracked, backbone.conv3.5.depthwise.0.weight, backbone.conv3.5.depthwise.1.weight, backbone.conv3.5.depthwise.1.bias, backbone.conv3.5.depthwise.1.running_mean, backbone.conv3.5.depthwise.1.running_var, backbone.conv3.5.depthwise.1.num_batches_tracked, backbone.conv3.5.pointwise.0.weight, backbone.conv3.5.pointwise.0.bias, backbone.conv3.5.pointwise.1.weight, backbone.conv3.5.pointwise.1.bias, backbone.conv3.5.pointwise.1.running_mean, backbone.conv3.5.pointwise.1.running_var, backbone.conv3.5.pointwise.1.num_batches_tracked, backbone.conv4.0.depthwise.0.weight, backbone.conv4.0.depthwise.1.weight, backbone.conv4.0.depthwise.1.bias, backbone.conv4.0.depthwise.1.running_mean, backbone.conv4.0.depthwise.1.running_var, backbone.conv4.0.depthwise.1.num_batches_tracked, backbone.conv4.0.pointwise.0.weight, backbone.conv4.0.pointwise.0.bias, backbone.conv4.0.pointwise.1.weight, backbone.conv4.0.pointwise.1.bias, backbone.conv4.0.pointwise.1.running_mean, backbone.conv4.0.pointwise.1.running_var, backbone.conv4.0.pointwise.1.num_batches_tracked, backbone.conv4.1.depthwise.0.weight, backbone.conv4.1.depthwise.1.weight, backbone.conv4.1.depthwise.1.bias, backbone.conv4.1.depthwise.1.running_mean, backbone.conv4.1.depthwise.1.running_var, backbone.conv4.1.depthwise.1.num_batches_tracked, backbone.conv4.1.pointwise.0.weight, backbone.conv4.1.pointwise.0.bias, backbone.conv4.1.pointwise.1.weight, backbone.conv4.1.pointwise.1.bias, backbone.conv4.1.pointwise.1.running_mean, backbone.conv4.1.pointwise.1.running_var, backbone.conv4.1.pointwise.1.num_batches_tracked, neck.lateral_convs.0.conv.weight, neck.lateral_convs.0.conv.bias, neck.lateral_convs.1.conv.weight, neck.lateral_convs.1.conv.bias, neck.lateral_convs.2.conv.weight, neck.lateral_convs.2.conv.bias, neck.fpn_convs.0.conv.weight, neck.fpn_convs.0.conv.bias, neck.fpn_convs.1.conv.weight, neck.fpn_convs.1.conv.bias, neck.fpn_convs.2.conv.weight, neck.fpn_convs.2.conv.bias, rpn_head.rpn_conv.weight, rpn_head.rpn_conv.bias, rpn_head.rpn_cls.weight, rpn_head.rpn_cls.bias, rpn_head.rpn_reg.weight, rpn_head.rpn_reg.bias, bbox_head.fc_cls.weight, bbox_head.fc_cls.bias, bbox_head.fc_reg.weight, bbox_head.fc_reg.bias, bbox_head.shared_fcs.0.weight, bbox_head.shared_fcs.0.bias, bbox_head.shared_fcs.1.weight, bbox_head.shared_fcs.1.bias

missing keys in source state_dict: conv2.1.depthwise.1.weight, conv4.0.depthwise.0.weight, conv4.1.pointwise.1.weight, conv3.2.depthwise.0.weight, conv3.1.pointwise.0.weight, conv3.4.pointwise.1.bias, conv3.5.depthwise.1.bias, conv2.1.pointwise.1.weight, stem.1.pointwise.1.running_mean, conv3.3.pointwise.1.weight, conv3.3.depthwise.1.running_mean, conv3.1.depthwise.1.num_batches_tracked, conv3.0.depthwise.1.num_batches_tracked, conv2.1.depthwise.1.running_var, conv1.0.depthwise.1.weight, conv3.5.depthwise.1.running_var, stem.0.bn.bias, conv3.2.depthwise.1.num_batches_tracked, conv2.0.depthwise.0.weight, conv2.1.pointwise.0.bias, conv3.1.pointwise.1.bias, conv3.2.pointwise.1.bias, conv2.0.pointwise.1.num_batches_tracked, stem.1.pointwise.0.weight, conv2.0.depthwise.1.weight, stem.1.depthwise.0.weight, conv1.1.pointwise.1.weight, conv3.5.pointwise.0.weight, conv3.4.depthwise.1.running_var, conv1.0.pointwise.0.bias, conv3.3.depthwise.1.running_var, conv3.0.pointwise.1.weight, conv4.0.pointwise.1.num_batches_tracked, conv4.1.depthwise.1.running_var, stem.1.depthwise.1.running_var, conv3.0.pointwise.1.running_var, conv3.4.depthwise.0.weight, conv3.4.pointwise.1.num_batches_tracked, conv4.0.depthwise.1.num_batches_tracked, conv3.0.depthwise.1.weight, conv3.3.pointwise.0.bias, conv3.0.depthwise.1.running_mean, conv3.2.pointwise.1.running_mean, conv3.1.pointwise.0.bias, conv3.5.depthwise.1.num_batches_tracked, conv3.5.pointwise.1.running_mean, conv3.1.pointwise.1.running_var, conv1.0.depthwise.1.running_mean, stem.1.pointwise.1.bias, conv1.0.depthwise.0.weight, conv3.2.pointwise.0.weight, conv4.0.pointwise.1.running_mean, conv2.1.pointwise.1.running_mean, stem.1.pointwise.1.weight, conv4.1.depthwise.1.weight, conv4.0.pointwise.0.weight, conv1.1.depthwise.1.bias, conv3.2.pointwise.1.num_batches_tracked, conv4.1.depthwise.0.weight, conv3.4.depthwise.1.running_mean, conv1.0.depthwise.1.bias, conv2.0.pointwise.0.bias, conv3.4.depthwise.1.num_batches_tracked, conv4.1.pointwise.1.running_mean, conv2.1.depthwise.1.bias, conv3.2.depthwise.1.weight, conv2.0.pointwise.1.weight, conv1.0.pointwise.0.weight, conv3.1.depthwise.1.running_var, conv2.0.pointwise.1.bias, conv4.0.depthwise.1.bias, conv3.3.pointwise.1.running_var, conv3.4.pointwise.1.weight, conv4.0.pointwise.0.bias, conv3.4.depthwise.1.bias, conv4.1.depthwise.1.num_batches_tracked, conv2.0.pointwise.1.running_mean, conv1.1.depthwise.1.weight, conv2.0.pointwise.1.running_var, stem.1.depthwise.1.running_mean, conv3.4.pointwise.1.running_var, stem.1.depthwise.1.num_batches_tracked, conv3.3.depthwise.1.weight, stem.1.pointwise.1.running_var, conv4.1.depthwise.1.bias, conv3.0.pointwise.1.bias, conv2.0.depthwise.1.running_mean, conv1.1.pointwise.1.bias, conv4.1.pointwise.0.bias, conv3.2.pointwise.0.bias, conv1.1.pointwise.0.weight, conv1.0.pointwise.1.weight, conv1.0.pointwise.1.running_mean, stem.0.conv.weight, stem.1.depthwise.1.bias, conv3.3.depthwise.0.weight, conv1.1.depthwise.1.num_batches_tracked, conv3.3.pointwise.1.num_batches_tracked, conv3.2.pointwise.1.running_var, conv3.2.depthwise.1.running_mean, conv3.3.depthwise.1.bias, conv4.1.pointwise.1.num_batches_tracked, conv2.0.depthwise.1.num_batches_tracked, conv3.0.pointwise.0.bias, conv3.1.depthwise.1.running_mean, conv3.1.depthwise.1.weight, conv3.0.pointwise.1.num_batches_tracked, conv3.1.pointwise.1.weight, conv4.0.pointwise.1.bias, conv3.3.depthwise.1.num_batches_tracked, conv3.4.pointwise.0.weight, stem.1.pointwise.0.bias, conv3.0.depthwise.1.bias, conv1.1.pointwise.0.bias, conv4.0.pointwise.1.running_var, stem.0.bn.weight, conv1.0.pointwise.1.num_batches_tracked, conv2.1.depthwise.1.running_mean, conv4.1.depthwise.1.running_mean, conv1.1.pointwise.1.running_var, conv2.1.pointwise.1.num_batches_tracked, conv2.0.depthwise.1.running_var, conv3.5.depthwise.1.weight, conv3.0.depthwise.0.weight, conv4.0.depthwise.1.running_mean, stem.0.bn.num_batches_tracked, conv3.3.pointwise.1.running_mean, conv2.1.pointwise.1.running_var, conv3.0.pointwise.1.running_mean, conv1.1.depthwise.1.running_var, conv3.0.depthwise.1.running_var, conv1.0.depthwise.1.running_var, stem.1.pointwise.1.num_batches_tracked, conv4.0.pointwise.1.weight, conv1.1.pointwise.1.running_mean, conv2.1.depthwise.0.weight, conv1.0.depthwise.1.num_batches_tracked, conv1.0.pointwise.1.running_var, conv3.5.pointwise.1.weight, conv3.5.depthwise.1.running_mean, conv3.1.depthwise.1.bias, conv3.1.depthwise.0.weight, conv1.1.depthwise.1.running_mean, conv2.0.pointwise.0.weight, conv4.1.pointwise.1.bias, conv3.2.depthwise.1.running_var, conv3.5.pointwise.0.bias, conv3.4.depthwise.1.weight, conv3.2.depthwise.1.bias, stem.0.bn.running_mean, conv4.0.depthwise.1.running_var, conv1.1.depthwise.0.weight, stem.0.bn.running_var, conv4.1.pointwise.0.weight, conv2.1.pointwise.1.bias, conv3.4.pointwise.0.bias, conv1.0.pointwise.1.bias, conv3.5.pointwise.1.running_var, conv1.1.pointwise.1.num_batches_tracked, conv3.1.pointwise.1.running_mean, conv2.1.depthwise.1.num_batches_tracked, conv2.1.pointwise.0.weight, stem.1.depthwise.1.weight, conv3.5.pointwise.1.bias, conv3.5.pointwise.1.num_batches_tracked, conv3.1.pointwise.1.num_batches_tracked, conv3.2.pointwise.1.weight, conv3.5.depthwise.0.weight, conv3.3.pointwise.0.weight, conv2.0.depthwise.1.bias, conv3.0.pointwise.0.weight, conv3.3.pointwise.1.bias, conv3.4.pointwise.1.running_mean, conv4.0.depthwise.1.weight, conv4.1.pointwise.1.running_var


title: How does torch.nn.DataParallel change the output size?

body:
It looks like using torch.nn.DataParallel changes the output size.
Though in official docs https://pytorch.org/docs/stable/nn.html#torch.nn.DataParallel
all information about size changing is as follows:

  When module returns a scalar (i.e., 0-dimensional tensor) in forward(), this wrapper will return a vector of length equal to number of devices used in data parallelism, containing the result from each device.

My module returns tensor of 10 coordinates, and I have 2 GPU's where I want to run the code. The last layer of my CNN is nn.Linear(500, 10).

import torch
import torch.nn as nn

net = LeNet()    #CNN-class written above
device = torch.device("cuda:0")
net.to(device)
net = nn.DataParallel(net)

#skipped some code, where inputs and targets are loaded from files

output = net(input)
criterion = nn.SmoothL1Loss()
loss = criterion(output, target)

Note that without calling DataParallel this piece of code works okay. With DataParallel the runtime error occurs when trying to calculate loss.

RuntimeError: The size of tensor a (20) must match the size of tensor b (10) at non-singleton dimension 0

Seems like output size for each GPU separately is 10 as stated, but afterwards the two outputs are joined and that's where size 20 come from.

When changing output size in the CNN-class from 10 to 5 it started to work again, but I'm not sure it's right solution and the CNN will work properly.


title: How does loss.backward() relate to the appropriate parameters of the model?

body:
I'm new in PyTorch and I'm having trouble understanding how loss knows to compute the gradients through loss.backward()?

Sure, I understand that the parameters need to have requires_grad=True and I understand that it sets x.grad to the appropriate gradient only for the optimizer later to perform the gradient update.

The optimizer is linked to the model parameters when it's instantiated, but the loss is never linked to the model.

I've been going through this thread, but I don't think anyone answered it clearly and the person that started the thread seems to have the same issue as I do.

What happens when I have two different networks with two different loss functions and two different optimizers. I will easily link the optimizers to each of the networks, but how will the loss functions know how to compute the gradients for each of their appropriate network if I never link them together?


title: Faster method of computing confusion matrix?

body:
I am computing my confusion matrix as shown below for image semantic segmentation which is a pretty verbose approach:

def confusion_matrix(preds, labels, conf_m, sample_size):
    preds = normalize(preds,0.9) # returns [0,1] tensor
    preds = preds.flatten()
    labels = labels.flatten()
    for i in range(len(preds)):
        if preds[i]==1 and labels[i]==1:
            conf_m[0,0] += 1/(len(preds)*sample_size) # TP
        elif preds[i]==1 and labels[i]==0:
            conf_m[0,1] += 1/(len(preds)*sample_size) # FP
        elif preds[i]==0 and labels[i]==0:
            conf_m[1,0] += 1/(len(preds)*sample_size) # TN
        elif preds[i]==0 and labels[i]==1:
            conf_m[1,1] += 1/(len(preds)*sample_size) # FN
    return conf_m

In the prediction loop:

conf_m = torch.zeros(2,2) # two classes (object or no-object)
for img,label in enumerate(data):
    ...
    out = Net(img)
    conf_m = confusion_matrix(out, label, len(data))
    ...

Is there a faster approach (in PyTorch) to efficiently calculate the confusion matrix for a sample of inputs for image semantic segmentation?


title: Inference pytorch C++ with alexnet and cv::imread image

body:
I am trying to infer with a C++ application an image classification task using an alexnet pre-trained net.I have successfully inferred a dog image loading the net with python:

alexnet = torchvision.models.alexnet(pretrained=True)
img = Image.open("dog.jpg")
transform = transforms.Compose([
 transforms.Resize(256),
 transforms.CenterCrop(224),
 transforms.ToTensor(),
 transforms.Normalize(
 mean=[0.485, 0.456, 0.406],
 std=[0.229, 0.224, 0.225]
 )])
img_t = transform(img)
batch_t = torch.unsqueeze(img_t, 0)
alexnet.forward(batch_t)
_, index = torch.max(out, 1)

Result index is 208, Labrador_retriever, that looks good.
Then I save the net to be loaded from a C++ application

example = torch.rand(1, 3, 224, 224)
traced_script_module_alex = torch.jit.trace(alexnet, example)
traced_script_module.save("alexnet.pt")

When I  load to C++, I get the wrong result:

cv::Mat img = cv::imread("dog.jpg");
cv::resize(img, img, cv::Size(224, 224), cv::INTER_CUBIC);

// Convert the image and label to a tensor.
torch::Tensor img_tensor = torch::from_blob(img.data, { 1, img.rows, img.cols, 3 }, torch::kByte);
img_tensor = img_tensor.permute({ 0, 3, 1, 2 }); // convert to CxHxW
img_tensor = img_tensor.to(torch::kFloat);
std::vector<torch::jit::IValue> input;
input.push_back(img_tensor);
torch::jit::script::Module  module = torch::jit::load("alexnet.pt");
at::Tensor output = module.forward(input).toTensor();
std::cout << output.argmax(1) << '\n';

the argmax is 463, bucket.
I think I am not looking at the same image; what am I missing...?


title: Pytorch torch.cholesky ignoring exception

body:
For some matrices on my batch I'm having an exception due the matrix being singular.

L = th.cholesky(Xt.bmm(X))

  cholesky_cpu: For batch 51100: U(22,22) is zero, singular U

Since they are few for my use case I would like to ignore the exception and further deal with them. I will set the resulting calculation as nan is it possible somehow?

Actually if I catch the exception and use continue still it doesn’t finish the calculation of the rest of the batch.

The same happens in C++ with Pytorch libtorch.


title: pytorch - Where is “conv1d” implemented?

body:
I wanted to see how the conv1d module is implemented
https://pytorch.org/docs/stable/_modules/torch/nn/modules/conv.html#Conv1d. So I looked at functional.py but still couldn’t find the looping and cross-correlation computation.

Then I searched Github by keyword ‘conv1d’, checked conv.cpp https://github.com/pytorch/pytorch/blob/eb5d28ecefb9d78d4fff5fac099e70e5eb3fbe2e/torch/csrc/api/src/nn/modules/conv.cpp 1 but still couldn’t locate where the computation is happening.

My question is two-fold.

Where is the source code that "conv1d” is implemented?
In general, if I want to check how the modules are implemented, where is the best place to find? Any pointer to the documentation will be appreciated. Thank you.


title: How to feed input into a pytorch lstm layer

body:
I am new to PyTorch, and don't understand how to fit a network using this framework.

I have a simple model in Keras:

model = Sequential()
model.add(Embedding(vocab_size, embedding_size, input_length=55, weights=[pretrained_weights]))
model.add(Bidirectional(LSTM(units=len(X_train))))
model.add(Dense(n_classes, activation='softmax'))
model.compile(loss='categorical_crossentropy',
          optimizer = RMSprop(lr=0.0005),
          metrics=['accuracy'])

model.fit(np.array(X_train), np.array(y_train), epochs=100, validation_data=(np.array(X_val), np.array(y_val)))

In Keras I can just feed the network with my X_train (a 2D array containing indices) and my y_train(a 2D array containing a single index for each input).

Now, in order to feed my PyTorch model I transformed my matrix into a tensor like this:

M = torch.tensor(X_train)

And defined my network:

# Bidirectional recurrent neural network (many-to-one)
class BiRNN(nn.Module):
  def __init__(self, input_size, hidden_size, num_layers, num_classes):
    super(BiRNN, self).__init__()
    self.hidden_size = hidden_size
    self.num_layers = num_layers
    self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True, bidirectional=True)
    self.fc = nn.Linear(hidden_size*2, num_classes)  # 2 for bidirection

  def forward(self, x):
    # Set initial states
    h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device) # 2 for bidirection
    c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device)

    # Forward propagate LSTM
    out, _ = self.lstm(x, (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size*2)

    # Decode the hidden state of the last time step
    out = self.fc(out[:, -1, :])
    return out

model = BiRNN(input_size, hidden_size, num_layers, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

But what I don't understand is how to call the function and use my own data for the predictions.


title: Local fully connected layer - Pytorch

body:
Assume we have a feature representation with kN neurons before the classification layer. Now, the classification layer produces an output layer of size N with only local connections.

That is, the kth neuron at the output is computed using input neurons at locations from kN to kN+N. Hence, every N locations in the input layer (with stride N) give single neuron value at the output.

This is done using conv1dlocal in Keras, however, the PyTorch does not seem to have this.

Weight matrix in standard linear layer: kNxN = kN^2 variables

Weight matrix with local linear layer: (kx1)@N times = NK variables


title: spyder doesn't launch after installing pytorch

body:
I installed pytorch but after that Spyder can no longer be launched. Here are the terminal info:

  conda install pytorch torchvision -c pytorch Solving environment: done

  ==> WARNING: A newer version of conda exists. <==   current version: 4.5.10   latest version: 4.5.11

  Please update conda by running

$ conda update -n base -c defaults conda

  Package Plan

  environment location: /anaconda3/envs/base_py36

  added / updated specs:
      - pytorch
      - torchvision

  The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
torchvision-0.2.1          |           py36_1         123 KB  pytorch
scipy-1.1.0                |   py36hf1f7d93_0        15.4 MB
scikit-learn-0.20.0        |   py36h4f467ca_1         5.4 MB
numpy-base-1.15.2          |   py36h8a80b8c_1         4.1 MB
numpy-1.11.3               |   py36heee0a97_5         3.4 MB
ninja-1.8.2                |   py36h04f5b5a_1          93 KB
pytorch-0.4.1              |py36_cuda0.0_cudnn0.0_1        10.0 MB  pytorch
------------------------------------------------------------
                                       Total:        38.5 MB

  The following NEW packages will be INSTALLED:

ninja:        1.8.2-py36h04f5b5a_1
pytorch:      0.4.1-py36_cuda0.0_cudnn0.0_1 pytorch
torchvision:  0.2.1-py36_1                  pytorch

  The following packages will be REMOVED:

accelerate:   2.3.1-np111py36_0

  The following packages will be UPDATED:

mkl:          11.3.3-0                              --> 2019.0-118
numexpr:      2.6.7-py36hde7755b_0                  --> 2.6.8-py36h1dc9127_0
numpy:        1.11.3-py36_nomklh8ecaf62_5           --> 1.11.3-py36heee0a97_5
numpy-base:   1.15.0-py36he97cb71_0                 --> 1.15.2-py36h8a80b8c_1
scikit-learn: 0.19.1-py36_nomklhde7755b_0           --> 0.20.0-py36h4f467ca_1
scipy:        1.1.0-py36_nomklh7cd7d8e_0            --> 1.1.0-py36hf1f7d93_0

  The following packages will be DOWNGRADED:

blas:         1.0-openblas                          --> 1.0-mkl

  Proceed ([y]/n)? y

  Downloading and Extracting Packages torchvision-0.2.1    | 123 KB    |

  ############################### | 100%  scipy-1.1.0          | 15.4 MB   | ##################################### | 100%  scikit-learn-0.20.0  | 5.4 MB    |

  ############################### | 100%  numpy-base-1.15.2    | 4.1 MB    | ##################################### | 100%  numpy-1.11.3         | 3.4 MB    | ##################################### | 100%

  ninja-1.8.2          | 93 KB     |

  ############################### | 100%  pytorch-0.4.1        | 10.0 MB   | ##################################### | 100%  Preparing transaction: done


title: Preventing PyTorch Dataset iteration from exceeding length of dataset

body:
I am using a custom PyTorch Dataset with the following:

class ImageDataset(Dataset):
    def __init__(self, input_dir, input_num, input_format, transform=None):
        self.input_num = input_num
        # etc
    def __len__ (self):
        return self.input_num
    def __getitem__(self,idx):
        targetnum = idx % self.input_num
        # etc

However, when I iterate over this dataset, iteration loops back to the start of the dataset instead of terminating at the end of the dataset. This effectively becomes an infinite loop in the iterator, with the epoch print statement never occurring for subsequent epochs.

train_dataset=ImageDataset(input_dir = 'path/to/directory',
                           input_num = 300, input_format = "mask") # Size 300
num_epochs = 10
for epoch in range(num_epochs):
    print("EPOCH " + str(epoch+1) + "\n")
    num = 0
    for data in train_dataset:
        print(num, end=" ")
        num += 1
        # etc

Print output (... for values in between):

EPOCH 1
0 1 2 3 4 5 6 7 ... 298 299 300 301 302 303 304 305 ... 597 598 599 600 601 602 603 604 ...

Why is the basic iteration over the Dataset continuing past the defined __len__ of the DataSet, and how can I ensure that iteration over the dataset terminates after hitting the length of the dataset when using this method (or is manually iterating over the range of the dataset length the only solution)?

Thank you.


title: Non-squared image as input of YOLOv3

body: I am trying to detect road objects (that are very small) using yolov3. For that, my dataset is composed of images of size 3840x400 px. The network implementation I am currently using (pytorch implementation by ultralytics 1) takes as input squared images. If, for instance, I pass through it a 3840x400 px image and I set the img_size to 1000 px, it resizes the image to 1000x400 px and then it squares it up to 1000x1000 px (letterboxing these 600 remaining pixels with black values in order to not change the height aspect). Most implementations of CNNs out there seems to work with squared images as the input of the network architecture. My question is: Will yolo still work if I manually change the network architecture to a non-squared size (e.g. 1000x400 px). Why most of the CNNs in the art work with squared-images?


title: Can torchtext's BucketIterator pad all batches to the same length?

body:
I recently started using torchtext to replace my glue code and I'm running into an issue where I'd like to use an attention layer in my architecture. In order to do this, I need to know the maximum sequence length of my training data.

The problem is that torchtext.data.BucketIterator does padding on a per-batch basis:

# All 4 examples in the batch will be padded to maxlen in the batch
train_iter = torchtext.data.BucketIterator(dataset=train, batch_size=4)

Is there some way to ensure that all training examples are padded to the same length; i.e., the maxlen in training?


title: Bidirectional RNN cells - shared or not?

body: Should I use the same weights to compute forward and backward passes in a bidirectional RNN, or should those weights be learned independently?