title: Kernel size can't be greater than actual input size

body:
I have a data with depth = 3 and I want to pass it through 3 convolution layers with 3x3x3 kernels each.
My current code is below. The first input is
[batch_size=10, in_channels=1, depth=3, height=128, width=256]
and I notice after the first conv3d layer the output is [10,8,1,126,254]. Obviously it has now depth 1 and doesn't accept it for another 3x3x3 layer. How can I achieve this?

class CNet(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.conv1 = nn.Conv3d(1, 8, kernel_size=3, stride=1, padding=0)
        self.conv2 = nn.Conv3d(8, 16, kernel_size=3, stride=1, padding=0)
        self.conv3 = nn.Conv3d(16, 32, kernel_size=3, stride=1, padding=0)
        self.fc1 = nn.Linear(value, 2)

    def forward(self, X):
        X = F.relu(self.conv1(X))
        X = F.relu(self.conv2(X))
        X = F.max_pool2d(X,2)
        X = self.conv3(X)
        X = F.max_pool2d(X,2)
        X = self.fc1(X)
        return F.softmax(X,dim =1)


title: PyTorch - better way to get back original tensor order after torch.sort

body:
I want to get back the original tensor order after a torch.sort operation and some other modifications to the sorted tensor, so that the tensor is not anymore sorted. It is better to explain this with an example:

x = torch.tensor([30., 40., 20.])
ordered, indices = torch.sort(x)
# ordered is [20., 30., 40.]
# indices is [2, 0, 1]
ordered = torch.tanh(ordered) # it doesn't matter what operation is
final = original_order(ordered, indices)
# final must be equal to torch.tanh(x)

I have implemented the function in this way:

def original_order(ordered, indices):
    z = torch.empty_like(ordered)
    for i in range(ordered.size(0)):
        z[indices[i]] = ordered[i]
    return z

Is there a better way to do this? In particular, it is possible to avoid the loop and compute the operation more efficiently?

In my case I have a tensor of size torch.Size([B, N]) and I sort each of the B rows separately with a single call of torch.sort. So, I have to call original_order B times with another loop.

Any, more pytorch-ic, ideas?

EDIT 1 - Get rid of inner loop

I solved part of the problem by simply indexing z with indices in this way:

def original_order(ordered, indices):
    z = torch.empty_like(ordered)
    z[indices] = ordered
    return z

Now, I just have to understand how to avoid the outer loop on B dimension.

EDIT 2 - Get rid of outer loop

def original_order(ordered, indices, batch_size):
    # produce a vector to shift indices by lenght of the vector
    # times the batch position
    add = torch.linspace(0, batch_size-1, batch_size) * indices.size(1)

    indices = indices + add.long().view(-1,1)

    # reduce tensor to single dimension.
    # Now the indices take in consideration the new length
    long_ordered = ordered.view(-1)
    long_indices = indices.view(-1)

    # we are in the previous case with one dimensional vector
    z = torch.zeros_like(long_ordered).float()
    z[long_indices] = long_ordered

    # reshape to get back to the correct dimension
    return z.view(batch_size, -1)


title: Pytorch/torchvision - modify images and labels of a Dataset object

body:
So I have this line of code to load a dataset of images from two classes called "0" and "1" for simplicity:

train_data = torchvision.datasets.ImageFolder(os.path.join(TRAIN_DATA_DIR), train_transform)

and then I prepare the loader to be used with my model in this way:

train_loader = torch.utils.data.DataLoader(train_data, TRAIN_BATCH_SIZE, shuffle=True)

So for now each image is associated to a class, what I want to do is take each image and apply a transformation to it between those two lines of code, let's say a rotation of one of four degrees: 0, 90, 180, 270, and add that info as an additional label of four classes: 0, 1, 2, 3. In the end I want the dataset to contain the rotated images and as their labels a list of two values: the class of the image and the applied rotation.

I tried this and there is no error, but the dataset remains unchanged if then I try to print the labels:

for idx,label in enumerate(train_data.targets):
    train_data.targets[idx] = [label, 1]

Is there a nice way to do it by modifying directly train_data without requiring a custom dataset?


title: PyTorch model input shape

body:
I loaded a custom PyTorch model and I want to find out its input shape. Something like this:
model.input_shape

Is it possible to get this information?

Update: print() and summary() don't show this model's input shape, so they are not what I'm looking for.


title: How to generate accurate masks for an image from Mask R-CNN prediction in PyTorch?

body:
I have trained a Mask RCNN network for instance segmentation of apples.  I am able to load the weights and generate predictions for my test images. The masks being generated seem to be in the correct location, but the mask itself has no real form.. it just looks like a bunch of pixels

Training is done based on the dataset from this paper, and here is the github link to code being used to train and generate weights

code for prediction is as follows. (i have omitted the parts where i create path variables and assign the paths)

import os
import glob
import numpy as np
import pandas as pd
import cv2 as cv
import fileinput

import torch
import torch.utils.data
import torchvision

from data.apple_dataset import AppleDataset
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor

import utility.utils as utils
import utility.transforms as T

from PIL import Image
from matplotlib import pyplot as plt
%matplotlib inline

def get_transform(train):
    transforms = []
    transforms.append(T.ToTensor())
    if train:
        transforms.append(T.RandomHorizontalFlip(0.5))
    return T.Compose(transforms)

def get_maskrcnn_model_instance(num_classes):
    # load an instance segmentation model pre-trained pre-trained on COCO
    model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=False)

    # get number of input features for the classifier
    in_features = model.roi_heads.box_predictor.cls_score.in_features
    # replace the pre-trained head with a new one
    model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)

    # now get the number of input features for the mask classifier
    in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
    hidden_layer = 256
    # and replace the mask predictor with a new one
    model.roi_heads.mask_predictor = MaskRCNNPredictor(in_features_mask, hidden_layer, num_classes)
    return model

num_classes = 2
device = torch.device('cpu')

model = get_maskrcnn_model_instance(num_classes)
checkpoint = torch.load('model_49.pth', map_location=device)
model.load_state_dict(checkpoint['model'], strict=False)

dataset_test = AppleDataset(test_image_files_path, get_transform(train=False))
img, _ = dataset_test[1]
model.eval()

with torch.no_grad():
    prediction = model([img.to(device)])

prediction

Image.fromarray(img.mul(255).permute(1, 2, 0).byte().numpy())

(unable to load image here since its over 2MB.

Image.fromarray(prediction[0]['masks'][0, 0].mul(255).byte().cpu().numpy())

Here is an Imgur link to the original image.. below is the predicted mask for one of the instances

Also, could you please help me understand the data structure of the generated prediction matrix shown below.. How do i access the masks so as to generate a single image with all masks displayed???

[{'boxes': tensor([[ 966.8143, 1633.7491, 1106.7389, 1787.6367],
          [1418.7872, 1467.0619, 1732.0828, 1796.1527],
          [1608.0396, 2064.6482, 1710.7534, 2206.5535],
          [2326.3750, 1690.3418, 2542.2112, 1883.2626],
          [2213.2024, 1864.3657, 2299.8933, 1963.0178],
          [1112.9083, 1732.5953, 1236.7600, 1823.0170],
          [1150.8256,  614.0334, 1218.8584,  711.4094],
          [ 942.7086,  794.6043, 1138.2318, 1008.0430],
          [1065.4371,  723.0493, 1192.7570,  870.3763],
          [1002.3103,  883.4616, 1146.9994, 1006.6841],
          [1315.2816, 1680.8625, 1531.3210, 1989.3317],
          [1244.5769, 1925.0903, 1459.5417, 2175.3252],
          [1725.2191, 2082.6187, 1934.0227, 2274.2952],
          [ 936.3065, 1554.3765, 1014.2722, 1659.4229],
          [ 934.8851, 1541.3331, 1090.4736, 1657.3751],
          [2486.0120,  776.4577, 2547.2329,  847.9725],
          [2336.1675,  698.6327, 2508.6492,  921.4550],
          [2368.4077, 1954.1102, 2448.4004, 2049.5796],
          [1899.1403, 1775.2371, 2035.7561, 1962.6923],
          [2176.0664, 1075.1553, 2398.6084, 1267.2555],
          [2274.8899,  641.6769, 2395.9634,  791.3353],
          [2535.1580,  874.4780, 2642.8213,  966.4614],
          [2183.4236,  619.9688, 2288.5676,  758.6825],
          [2183.9832, 1122.9382, 2334.9583, 1263.3226],
          [1135.7822,  779.0529, 1225.9871,  890.0135],
          [ 317.3954, 1328.6995,  397.3900, 1467.7740],
          [ 945.4811, 1833.3708,  997.2318, 1878.8607],
          [1992.4447,  679.4969, 2134.6667,  835.8701],
          [1098.5416, 1452.7799, 1429.1808, 1771.4460],
          [1657.3193, 1405.5405, 1781.6273, 1574.6780],
          [1443.8911, 1747.1544, 1739.0361, 2076.9724],
          [1092.6003, 1165.3340, 1206.0881, 1383.8314],
          [2466.4170, 1945.5931, 2555.1931, 2039.8368],
          [2561.8508, 1616.2659, 2672.1033, 1742.2332],
          [1894.4806,  907.9214, 2097.1875, 1182.6473],
          [2321.5005, 1701.3344, 2368.3699, 1865.3914],
          [2180.0781,  567.5969, 2344.6357,  763.4360],
          [1845.7612,  668.6808, 2045.2688,  899.8501],
          [1858.9216, 2145.7097, 1961.8870, 2273.5088],
          [ 261.4607, 1314.0154,  396.9288, 1486.9498],
          [2488.1682, 1585.2357, 2669.0178, 1794.9926],
          [2696.9548,  936.0087, 2802.7961, 1025.2294],
          [1593.6837, 1489.8641, 1720.3124, 1627.8135],
          [2517.9468,  857.1713, 2567.1125,  929.4335],
          [1943.2167,  636.3422, 2151.4419,  853.8924],
          [2143.5664, 1100.0521, 2308.1570, 1290.7125],
          [2140.9231, 1947.9692, 2238.6956, 2000.6249],
          [1461.6316, 2105.2593, 1559.7675, 2189.0264],
          [2114.0781,  374.8153, 2222.8838,  559.9851],
          [2350.5320,  726.5779, 2466.8140,  878.2617]]),
  'labels': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
          1, 1]),
  'scores': tensor([0.9916, 0.9841, 0.9669, 0.9337, 0.9118, 0.7729, 0.7202, 0.7193, 0.6928,
          0.6872, 0.6690, 0.5913, 0.4877, 0.4683, 0.3781, 0.3327, 0.3164, 0.2364,
          0.1696, 0.1692, 0.1502, 0.1365, 0.1316, 0.1171, 0.1119, 0.1094, 0.1041,
          0.0865, 0.0853, 0.0835, 0.0822, 0.0816, 0.0797, 0.0796, 0.0788, 0.0780,
          0.0757, 0.0736, 0.0736, 0.0689, 0.0681, 0.0644, 0.0642, 0.0630, 0.0612,
          0.0598, 0.0563, 0.0531, 0.0525, 0.0522]),
  'masks': tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          ...,

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]],

          [[[0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            ...,
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.],
            [0., 0., 0.,  ..., 0., 0., 0.]]]])}]


title: Pytorch - How to add a self-attention to another architecture

body:
I'm a beginner with pytorch framework and I'm trying to add a multiheaded self attention on top of another architecture (BERT) (this is a simple question but I'm not familiar with PyTorch):

UPDATE 1

import math
class PositionalEncoding(nn.Module):
    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)
        self.d_model = d_model

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x, seq_len = 768, mask = None):
        pos_emb = self.pe[:, :seq_len]
        x = x * mask[:, :, None].float()
        x = x + pos_emb
        return x

The problem in how to add the transformer is in the following class:

class CamemBERTQA(nn.Module):
   def __init__(self,bert_type, hidden_size, num_labels, num_inter_layers=1, heads = 12, do_lower_case = True):
       super(CamemBERTQA, self).__init__()
       self.do_lower_case = do_lower_case
       self.bert_type = bert_type
       self.hidden_size = hidden_size
       self.num_labels = num_labels
       self.num_inter_layers = num_inter_layers
       self.camembert = CamembertModel.from_pretrained(self.bert_type)

       # ---------------- Transformer ------------------------------------------
       self.d_model = self.hidden_size # 768
       dropout = 0.1
       self.pos_emb = PositionalEncoding(d_model = self.d_model, dropout = dropout)
       self.transformer_inter = nn.ModuleList(
           [nn.TransformerEncoderLayer(d_model = self.d_model, nhead = heads, dim_feedforward = 2048, dropout = dropout)
            for _ in range(num_inter_layers)])
       # ---------------- Transformer ------------------------------------------

       self.qa_outputs = nn.Linear(self.hidden_size, self.num_labels)

   def forward(self, input_ids, mask=None):
       bert_output = self.camembert(input_ids = input_ids) # input_ids is a tensor

       # ---------------- Transformer ------------------------------------------
       seq_len = self.hidden_size
       x = self.pos_emb(x = bert_output, seq_len = seq_len, mask = None)

       for i in range(self.num_inter_layers):
           x = self.transformer_inter[i](i, x, x, 1 - mask)  # all_tokens * max_tokens * dim
       output = self.layer_norm(x)
       # ---------------- Transformer ------------------------------------------

       sequence_output = output[0]
       logits = self.qa_outputs(sequence_output)
       start_logits, end_logits = logits.split(1, dim=-1)
       start_logits = start_logits.squeeze(-1)
       end_logits = end_logits.squeeze(-1)
       outputs = (start_logits, end_logits,)
       return x

Thank you so much.


title: How is the output h_n of an RNN (nn.LSTM, nn.GRU, etc.) in PyTorch structured?

body:
The docs say

  h_n of shape (num_layers * num_directions, batch, hidden_size): tensor containing the hidden state for t = seq_len

Now, the batch and hidden_size dimensions are pretty much self-explanatory. The first dimension remains a mystery, though.

I assume, that the hidden states of all "last cells" of all layers are included in this output. But then what is the index of, for example, the hidden state of the "last cell" in the "uppermost layer"? h_n[-1]? h_n[0]?

Is the output affected by the batch_first option?


title: Loss of Multi-Output Model in Pytorch

body:
I have a multi-output model in PyTorch when I train them using the same loss and then to backpropagate I combine the loss of both the output but when one output loss decreases others increase and so on. How can I fix the problem?

def forward(self, x):
    #neural network arch. forward pass
    x = dense1(x)
    x1 = dense2(x)
    x2 = dense2(x)
    x1 = F.log_softmax(x1)
    x2 = F.log_softmax(x2)
    return x1, x2

out1, out2 = model(data)
loss1 = NLLL(out1, target1)
loss2 = NLLL(out2, target2)
loss = loss1 + loss2
loss.backward()

When loss1 decrease loss2 increase and when loss2 decrease loss1 increase how can I fix the issue.
Can any other operator other than '+' be used to combine the loss or should I put weights to the different loss?


title: How to get final hidden state of bidirectional 2-layers GRU in pytorch

body:
I am struggling with understanding how to get hidden layers and concatenate them.

I am using the following code as an example:

class classifier(nn.Module):

#define all the layers used in model
def __init__(self, vocab_size, embedding_dim, hidden_dim, output_dim, n_layers,
             bidirectional, dropout):

    #Constructor
    super().__init__()

    self.batch = BATCH_SIZE
    self.hidden = hidden_dim
    self.layers = n_layers
    if(bidirectional):
        self.directions = 2
    else:
        self.directions = 1

    #embedding layer
    self.embedding = nn.Embedding(vocab_size, embedding_dim)

    #lstm layer
    self.gru = nn.GRU(embedding_dim,
                       hidden_dim,
                       num_layers=n_layers,
                       bidirectional=bidirectional,
                       dropout=dropout,
                       batch_first=True)

    #dense layer
    self.fc = nn.Linear(hidden_dim * 2, output_dim)

    #activation function
    self.act = nn.Sigmoid()

def forward(self, text, text_lengths):

    #text = [batch size,sent_length]
    embedded = self.embedding(text)
    #embedded = [batch size, sent_len, emb dim]

    #packed sequence
    packed_embedded = nn.utils.rnn.pack_padded_sequence(embedded, text_lengths,batch_first=True)

    packed_output, (hidden, cell) = self.lstm(packed_embedded)
    #hidden = [batch size, num layers * num directions,hid dim]
    #cell = [batch size, num layers * num directions,hid dim]

    #concat the final forward and backward hidden state
    hidden = torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1)

    #hidden = [batch size, hid dim * num directions]
    dense_outputs=self.fc(hidden)

    #Final activation function
    outputs=self.act(dense_outputs)

    return outputs

The line hidden = torch.cat((hidden[-2,:,:], hidden[-1,:,:]), dim = 1) I did not get it.

As per my understanding, I was doing this which did not work.

hidden2 = hidden.view(batch_size,self.layers,self.directions,self.hidden)
hidden2 = torch.cat((hidden2[:,:,0,:],hidden2[:,:,1,:]),dim=1)
dense_outputs=self.fc(hidden2)

Can somebody please explain. I went through PyTorch documentation but did not get.


title: Calling super's forward() method

body:
What is the most appropriate way to call the forward() method of a parent Module? For example, if I subclass the nn.Linear module, I might do the following

class LinearWithOtherStuff(nn.Linear):
    def forward(self, x):
        y = super(Linear, self).forward(x)
        z = do_other_stuff(y)
        return z

However, the docs say not to call the forward() method directly:

  Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

which makes me think super(Linear, self).forward(x) could result in some unexpected errors. Is this true or am I misunderstanding inheritance?


title: PyTorch Tutorial Error Training a Classifier

body:
I just started the PyTorch-Tutorial Deep Learning with PyTorch: A 60 Minute Blitz and I should add, that I haven't programmed any python (but other languages like Java) before.

Right now, my Code looks like

import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np

print("\n-------------------Backpropagation-------------------\n")
transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,download=True, transform=transform)

trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)

testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

dataiter = iter(trainloader)
images, labels = dataiter.next()

def imshow(img):
    img = img / 2 + 0.5
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

imshow(torchvision.utils.make_grid(images))

print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

which should be consistent with the tutorial.
If I execute this, I'll get the following error:

"C:\Program Files\Anaconda3\python.exe" C:/MA/pytorch/deepLearningWithPytorchTutorial/trainingClassifier.py

-------------------Backpropagation-------------------

Files already downloaded and verified
Files already downloaded and verified

-------------------Backpropagation-------------------

Files already downloaded and verified
Files already downloaded and verified
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Program Files\Anaconda3\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Program Files\Anaconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Program Files\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\MA\pytorch\deepLearningWithPytorchTutorial\trainingClassifier.py", line 23, in <module>
    dataiter = iter(trainloader)
  File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
  File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
  File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "C:\Program Files\Anaconda3\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "C:/MA/pytorch/deepLearningWithPytorchTutorial/trainingClassifier.py", line 23, in <module>
    dataiter = iter(trainloader)
  File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 451, in __iter__
    return _DataLoaderIter(self)
  File "C:\Program Files\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 239, in __init__
    w.start()
  File "C:\Program Files\Anaconda3\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Program Files\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
BrokenPipeError: [Errno 32] Broken pipe

Process finished with exit code 1

I already downloaded the *.py and *.ipynb.
Running the *.ipynb with jupyter works fine (but I don't want to programm in the juniper web-interface, I prefer pyCharm) while the *.py in the console (Anaconda prompt and cmd) fails with the same error.

Does anyone know how to fix this?
(I'm using Python 3.6.5 (from Anaconda) and pyCharm, OS: Win10 64-bit)

Thanks!
Bene

Update:
If it is relevant, I just set num_workers=2 to num_workers=0 (both) and then it'll work.. .


title: Pytorch error "RuntimeError: index out of range: Tried to access index 512 out of table with 511 rows"

body:
I have sentences that I vectorize using sentence_vector() method of BiobertEmbedding python module (https://pypi.org/project/biobert-embedding/). For some group of sentences I have no problem but for some others I have the following error message :

File
"/home/nobunaga/.local/lib/python3.6/site-packages/biobert_embedding/embedding.py",
line 133, in sentence_vector
encoded_layers = self.eval_fwdprop_biobert(tokenized_text)   File "/home/nobunaga/.local/lib/python3.6/site-packages/biobert_embedding/embedding.py",
line 82, in eval_fwdprop_biobert
encoded_layers, _ = self.model(tokens_tensor, segments_tensors)   File
"/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/modules/module.py",
line 547, in __call__
result = self.forward(*input, **kwargs)   File "/home/nobunaga/.local/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py",
line 730, in forward
embedding_output = self.embeddings(input_ids, token_type_ids)   File
"/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/modules/module.py",
line 547, in __call__
result = self.forward(*input, **kwargs)   File "/home/nobunaga/.local/lib/python3.6/site-packages/pytorch_pretrained_bert/modeling.py",
line 268, in forward
position_embeddings = self.position_embeddings(position_ids)   File
"/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/modules/module.py",
line 547, in __call__
result = self.forward(*input, **kwargs)   File "/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/modules/sparse.py",
line 114, in forward
self.norm_type, self.scale_grad_by_freq, self.sparse)   File "/home/nobunaga/.local/lib/python3.6/site-packages/torch/nn/functional.py",
line 1467, in embedding
return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse) RuntimeError: index out of range: Tried to
access index 512 out of table with 511 rows. at
/pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:237

I discovered that for some group of sentences, the problem was related to tags like <tb> for instance. But for others, even when tags are removed, the error message is still there.
(Unfortunately I can't share the code for confidentiality reasons)
Do you have any ideas of what could be the problem?
Thank you by advance
EDIT : you are right cronoik, it will be better with an example.
Example :
sentences = ["This is the first sentence.", "This is the second sentence.", "This is the third sentence."

biobert = BiobertEmbedding(model_path='./biobert_v1.1_pubmed_pytorch_model')

vectors = [biobert.sentence_vector(doc) for doc in sentences]

This last line of code is what caused the error message in my opinion.


title: How to create a custom data loader in Pytorch?

body:
I have a file containing paths to images I would like to load into Pytorch, while utilizing the built-in dataloader features (multiprocess loading pipeline, data augmentations, and so on).
def create_links():
    data_dir = "/myfolder"

    full_path_list = []
    assert os.path.isdir(data_dir)
    for _, _, filenames in os.walk(data_dir):
        for filename in filenames:
            full_path_list.append(os.path.join(data_dir, filename))

    with open(config.data.links_file, 'w+') as links_file:
        for full_path in full_path_list:
            links_file.write(f"{full_path}\n")

def read_links_file_to_list():
    config = ConfigProvider.config()
    links_file_path = config.data.links_file
    if not os.path.isfile(links_file_path):
        raise RuntimeError("did you forget to create a file with links to images? Try using 'create_links()'")
    with open(links_file_path, 'r') as links_file:
        return links_file.readlines()

So I have a list of files (or a generator, or whatever works), file_list = read_links_file_to_list().
How can I build a Pytorch dataloader around it, and how would I use it?


title: What would be the equivalent of keras.layers.Masking in pytorch?

body:
I have time-series sequences which I needed to keep the length of sequences fixed to a number by padding zeroes into matrix and using keras.layers.Masking in keras I could neglect those padded zeros for further computations, I am wondering how could it be done in Pytorch?

Either I need to do the padding in pytroch and pytorch can't handle the sequences with varying lengths what is the equivalent to Masking layer of keras in pytorch, or if pytorch handles the sequences with varying lengths, how could it be done?


title: Loading .npy files as dataset for pytorch

body: I have preprocessed data in .npy files, let's call it X.npy for raw data and Y.npy for labels. They're organized to match every element from both files (first element from X has first label from Y etc.). How can I load it as dataset using torch.utils.data.DataLoader? I'm very new to pytorch, and any help will be useful.


title: RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle)` with GPU only

body:
I'm working on the CNN with one-dimensional signal. It works totally fine with CPU device. However, when I training model in GPU, CUDA error occurred. I set os.environ['CUDA_LAUNCH_BLOCKING'] = "1" command after I got RuntimeError: CUDA error: CUBLAS_STATUS_ALLOC_FAILED when calling cublasCreate(handle). With doing this, a cublasSgemm error occurred instead of cublasCreate error.
Though the nvidia document doubt the hardware problem, I can training other CNN with images without any error. Below is my code for the data loading and set data in training model.
    idx = np.arange(len(dataset))  # dataset & label shuffle in once
    np.random.shuffle(idx)

    dataset = dataset[idx]
    sdnn = np.array(sdnn)[idx.astype(int)]

    train_data, val_data = dataset[:int(0.8 * len(dataset))], dataset[int(0.8 * len(dataset)):]
    train_label, val_label = sdnn[:int(0.8 * len(sdnn))], sdnn[int(0.8 * len(sdnn)):]
    train_set = DataLoader(dataset=train_data, batch_size=opt.batch_size, num_workers=opt.workers)

    for i, data in enumerate(train_set, 0):  # data.shape = [batch_size, 3000(len(signal)), 1(channel)] tensor

        x = data.transpose(1, 2)
        label = torch.Tensor(train_label[i * opt.batch_size:i * opt.batch_size + opt.batch_size])
        x = x.to(device, non_blocking=True)
        label = label.to(device, non_blocking=True) # [batch size]
        label = label.view([len(label), 1])
        optim.zero_grad()

        # Feature of signal extract
        y_predict = model(x) # [batch size, fc3 output] # Error occurred HERE
        loss = mse(y_predict, label)

Below is the error message from this code.
File C:/Users/Me/Desktop/Me/Study/Project/Analysis/Regression/main.py", line 217, in Processing
    y_predict = model(x) # [batch size, fc3 output]
  File "C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Users\ME\Desktop\ME\Study\Project\Analysis\Regression\cnn.py", line 104, in forward
    x = self.fc1(x)
  File "C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\Anaconda\envs\torch\lib\site-packages\torch\nn\modules\linear.py", line 91, in forward
    return F.linear(input, self.weight, self.bias)
  File "C:\Anaconda\envs\torch\lib\site-packages\torch\nn\functional.py", line 1674, in linear
    ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

I've tried to solve this error for weeks but can't find the solution. If you can see anything wrong here, please let me know.


title: TypeError: object of type has no len() when using Pandas Dataframe

body:
I tried the following code

import pandas as pd
import os
from torch.utils.data import Dataset
from torch.utils.data import Dataset, DataLoader

class ArticleDataset(Dataset):
   def __init__(self):
    #  super().__init__()
     df= pd.read_csv('https://drive.google.com/uc?export?format=csv&=download&id=*********', sep = "delimiter", index_col=0, header=None, engine='python')

     self.articles_list=df

     def __len__(self):
        return len(self.articles_list)

     def __getitem__(self, item):
        return self.articles_list.index[item]

The data frame has data and prints data when df.head() is executed.
But it gives the error

TypeError                                 Traceback (most recent call last)
<ipython-input-74-08e914e52560> in <module>()
      1 dataset =ArticleDataset()
----> 2 article_loader=DataLoader(dataset, batch_size=1, shuffle=True)

2 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/sampler.py in num_samples(self)
     98         # dataset size might change at runtime
     99         if self._num_samples is None:
--> 100             return len(self.data_source)
    101         return self._num_samples
    102

TypeError: object of type 'ArticleDataset' has no len()

when

dataset =ArticleDataset()
article_loader=DataLoader(dataset, batch_size=1, shuffle=True)

is executed.

But according to the correct answer found in here, len(DataFrame.index) gives me the expected no. of rows in the Dataframe as well.
And df.index1 prints the first row of the Dataframe, but cannot figure out what is wrong.

I tried turning the data frame to list as well, but the same error was given.
May be there is some error in the code that I'm not seeing and I'm new to python.
Any sort of help is appreciated.

Edit (After Unindenting once):

import pandas as pd
import os
from torch.utils.data import Dataset
from torch.utils.data import Dataset, DataLoader

class ArticleDataset(Dataset):
   def __init__(self):
    #  super().__init__()
     df= pd.read_csv('https://drive.google.com/uc?export?format=csv&=download&id=*******', sep = "delimiter", index_col=0, header=None, engine='python')

     self.articles_list=df

    def __len__(self):
        return len(self.articles_list)

    def __getitem__(self, item):
        return self.articles_list.index[item]

this will give the following error

File "<ipython-input-104-165bbc37826a>", line 13
    def __len__(self):
                      ^
IndentationError: unindent does not match any outer indentation level


title: list of fastai models

body:
FastAI uses AWD-LSTM for text processing. They provide pretrained models with get_language_model(). But I can't find proper documentation on what's available.

Their github example page is really a moving target. Model names such as lstm_wt103 and WT103_1 are used. In the forums I found wt103RNN.

Where can I find an updated list of pretrained models and their download URLs ?


title: Pytorch: Gradient of output w.r.t parameters

body:
I'm interested in Finding the Gradient of Neural Network output with respect to the parameters (weights and biases).

More specifically, assume I have the following Neural Network Structure [6,4,3,1]. The input samples size is 20. What I'm interested in, is finding the gradient of Neural Network output w.r.t the weights (and biases), which if I'm not mistaken, in this case would be 47. In Literature , this gradient is sometimes known as Weight_Jacobian.

I'm using Pytorch version 0.4.0 on Python 3.6 on Jupyter Notebook.

The Code that I have produced is this:

def init_params(layer_sizes, scale=0.1, rs=npr.RandomState(0)):
    return [(rs.randn(insize, outsize) * scale,   # weight matrix
                 rs.randn(outsize) * scale)           # bias vector
                 for insize, outsize in
                 zip(layer_sizes[:-1],layer_sizes[1:])]
layers = [6, 4, 3, 1]
w = init_params(layers)
first_layer_w = Variable(torch.tensor(w[0][0],requires_grad=True))
first_layer_bias = Variable(torch.tensor(w[0][1],requires_grad=True))
second_layer_w = Variable(torch.tensor(w[1][0],requires_grad=True))
second_layer_bias = Variable(torch.tensor(w[1][1],requires_grad=True))
third_layer_w = Variable(torch.tensor(w[2][0],requires_grad=True))
third_layer_bias = Variable(torch.tensor(w[2][1],requires_grad=True))
X = Variable(torch.tensor(X_batch),requires_grad=True)
output=torch.tanh(torch.mm(torch.tanh(torch.mm(torch.tanh(torch.mm(X,first_layer_w)+first_layer_bias),second_layer_w)+second_layer_bias),third_layer_w)+third_layer_bias)
output.backward()

As it is obvious from the code, I'm using hyperbolic tangent as the Non Linearity. The code produces the output vector with length 20. Now, I'm interested in finding the Gradient of this Output vector w.r.t all the weights (all 47 of them). I have read the documentation of Pytorch at here. I have also seen similar questions for example,  here. However, I have failed to find the gradient of output vector w.r.t parameters.
If I use the Pytorch function backward(), it generates an error as

RuntimeError: grad can be implicitly created only for scalar outputs

My Question is, is there a way to calculate the gradient of output vector w.r.t parameters, which could essentially be represented as a 20*47 matrix as I have the size of output vector to be 20 and size of parameter vector to be 47? If so, how ? Is there anything wrong in my code ? You can take any example of X as long as its dimension is 20*6.


title: NumPyro vs Pyro: Why is former 100x faster and when should I use the latter?

body:
From Pytorch-Pyro's website:

  We’re excited to announce the release of NumPyro, a NumPy-backed Pyro using JAX for automatic differentiation and JIT compilation, with over 100x speedup for HMC and NUTS!

My questions:

Where is the performance gain (which is sometimes 340x or 2X) of NumPyro (over Pyro) coming from exactly?
And more importantly, why (rather, where) would I continue to use Pyro?

Extra:

How should I view the performance and features of NumPyro compared to Tensorflow Probability, in deciding which to use where?


title: Prunning model doesn't improve inference speed or reduce model size

body:
I'm trying to prune my model in PyTorch with torch.nn.utils.prune, which provides 2 tensors,

one is the original weight and
the other is a mask contain 0s and 1s that help us close certain connections in the network.

I have tried both of the solutions, but none improve the inference speed:

Use the network after pruning to infer which will first close some connections with the mask and then run inference.
Zeros out the original weights with the mask and then remove the mask from the state_dict to infer.

Is there a way to improve the speed with the model tensor and the mask? Doesn't multiply with a non-zero float number with 0 will faster than multiply 2 floats with each other?
Here is my prune function and the pruning speed calculating procedure:

def prune_net(net):
    """Prune 20% net's weights that have abs(value) approx. 0
    Function that will be use when an iteration is reach
    Args:

    Return:
        newnet (nn.Module): a newnet contain mask that help prune network's weight
    """
    if not isinstance(net,nn.Module):
        print('Invalid input. Must be nn.Module')
        return
    newnet = copy.copy(net)
    modules_list = []

    for name, module in newnet.named_modules():
        if isinstance(module, torch.nn.Conv2d):
            modules_list += [(module,'weight'),(module,'bias')]
        if isinstance(module, torch.nn.Linear):
            modules_list += [(module,'weight'),(module,'bias')]

    prune.global_unstructured(
        modules_list,
        pruning_method=prune.L1Unstructured,
        amount=0.2,)
    return newnet

Test inference speed 1st case:

import torch
from torch import nn
import torch.nn.utils.prune as prune
import torch.nn.functional as F
import time
from torch.autograd import Variable

torch.set_default_tensor_type('torch.cuda.FloatTensor')
old_net = init_your_net()

new_net = prune_net(old_net)
new_net = prune_net(new_net)

old_net.eval()
new_net.eval()

old_net = old_net.cuda()
new_net = new_net.cuda()
dataset = load_your_dataset()

for i in range(100):
    x = dataset[i]
    x = x.cuda()
    y = x.cuda()

    #new infer
    start_time = time.perf_counter()
    detections = new_net(x).data
    time_new += time.perf_counter() - start_time

    #old infer
    start_time = time.perf_counter()
    detections = old_net(y).data
    time_old += time.perf_counter() - start_time
print('old ',time_old)
print('new ', time_new)

Test inference speed 2nd case:

import torch
from torch import nn
import torch.nn.utils.prune as prune
import torch.nn.functional as F
import time
from torch.autograd import Variable

torch.set_default_tensor_type('torch.cuda.FloatTensor')
old_net = init_your_net()

new_net = prune_net(old_net)
new_net = prune_net(new_net)
# Apply mask to model tensor and remove mask from state_dict
for name, module in new_net.named_modules():
    if isinstance(module, torch.nn.Conv2d):
        prune.remove(module,'weight')
        prune.remove(module,'bias')
    if isinstance(module, torch.nn.Linear):
        prune.remove(module,'weight')
        prune.remove(module,'bias')

old_net.eval()
new_net.eval()

old_net = old_net.cuda()
new_net = new_net.cuda()
dataset = load_your_dataset()

for i in range(100):
    x = dataset[i]
    x = x.cuda()
    y = x.cuda()

    #new infer
    start_time = time.perf_counter()
    detections = new_net(x).data
    time_new += time.perf_counter() - start_time

    #old infer
    start_time = time.perf_counter()
    detections = old_net(y).data
    time_old += time.perf_counter() - start_time
print('old ',time_old)
print('new ', time_new)

UPDATE
I found torch have a sparse module that can reduce memory usage if we prune enough parameter but it hasn't support nn.Module yet, only Tensor object. Here are some useful link:
https://github.com/pytorch/pytorch/issues/36214#issuecomment-619586452
https://pytorch.org/docs/stable/sparse.html


title: In Pytorch, how to test simple image with my loaded model?

body:
I made a alphabet classification CNN model using Pytorch, and then use that model to test it with a single image that I've never seen before. I extracted a bounding box in my handwriting image with opencv, but I don't know how to apply it to the model.

bounded my_image

this is custom dataset

class CustomDatasetFromCSV(Dataset):
    def __init__(self, csv_path, height, width, transforms=None):
        """
        Args:
            csv_path (string): path to csv file
            height (int): image height
            width (int): image width
            transform: pytorch transforms for transforms and tensor conversion
        """
        self.data = pd.read_csv(csv_path)
        self.labels = np.asarray(self.data.iloc[:, 0])
        self.height = height
        self.width = width
        self.transforms = transforms

    def __getitem__(self, index):
        single_image_label = self.labels[index]
        # Read each 784 pixels and reshape the 1D array ([784]) to 2D array ([28,28])
        img_as_np = np.asarray(self.data.iloc[index][1:]).reshape(28,28).astype('uint8')
        # Convert image from numpy array to PIL image, mode 'L' is for grayscale
        img_as_img = Image.fromarray(img_as_np)
        img_as_img = img_as_img.convert('L')
        # Transform image to tensor
        if self.transforms is not None:
            img_as_tensor = self.transforms(img_as_img)
        # Return image and the label
        return (img_as_tensor, single_image_label)

    def __len__(self):
        return len(self.data.index)

    transformations = transforms.Compose([
                            transforms.ToTensor()
                        ])

    alphabet_from_csv = CustomDatasetFromCSV("/content/drive/My Drive/A_Z Handwritten Data.csv",
                                                28, 28, transformations)

random_seed = 50
data_size = len(alphabet_from_csv)
indices = list(range(data_size))
split = int(np.floor(0.2 * data_size))
if True:
    np.random.seed(random_seed)
    np.random.shuffle(indices)
train_indices, test_indices = indices[split:], indices[:split]
train_dataset = SubsetRandomSampler(train_indices)
test_dataset = SubsetRandomSampler(test_indices)

train_loader = torch.utils.data.DataLoader(dataset = alphabet_from_csv,
                                           batch_size = batch_size,
                                           sampler = train_dataset)
test_loader = torch.utils.data.DataLoader(dataset = alphabet_from_csv,
                                          batch_size = batch_size,
                                          sampler = test_dataset)

this is my model

class ConvNet3(nn.Module):
    def __init__(self, num_classes=26):
        super().__init__()

        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 28, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(28),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )

        self.layer2 = nn.Sequential(
            nn.Conv2d(28, 56, kernel_size=3, stride=1, padding=1),
            nn.BatchNorm2d(56),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )

        self.fc = nn.Sequential(
            nn.Dropout(p = 0.5),
            nn.Linear(56 * 7 * 7, 512),
            nn.BatchNorm1d(512),
            nn.ReLU(),
            nn.Dropout(p = 0.5),
            nn.Linear(512, 26),
        )

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out
model = ConvNet3(num_classes).to(device)
loss_func = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
def train():
    # train phase
    model.train()

    # create a progress bar
    batch_loss_list = []
    progress = ProgressMonitor(length=len(train_dataset))

    for batch, target in train_loader:
        # Move the training data to the GPU
        batch, target = batch.to(device), target.to(device)

        # forward propagation
        output = model( batch )

        # calculate the loss
        loss = loss_func( output, target )

        # clear previous gradient computation
        optimizer.zero_grad()

        # backpropagate to compute gradients
        loss.backward()

        # update model weights
        optimizer.step()

        # update progress bar
        batch_loss_list.append(loss.item())
        progress.update(batch.shape[0], sum(batch_loss_list)/len(batch_loss_list) )

def test():
    # test phase
    model.eval()

    correct = 0

    # We don't need gradients for test, so wrap in
    # no_grad to save memory
    with torch.no_grad():
        for batch, target in test_loader:
            # Move the training batch to the GPU
            batch, target = batch.to(device), target.to(device)

            # forward propagation
            output = model( batch )

            # get prediction
            output = torch.argmax(output, 1)

            # accumulate correct number
            correct += (output == target).sum().item()

    # Calculate test accuracy
    acc = 100 * float(correct) / len(test_dataset)
    print( 'Test accuracy: {}/{} ({:.2f}%)'.format( correct, len(test_dataset), acc ) )

for epoch in range(num_epochs):

    print("{}'s try".format(int(epoch)+1))
    train()
    test()
    print("-----------------------------------------------------------------------------")

this is my image to bound

import cv2
import matplotlib.image as mpimg
im = cv2.imread('/content/drive/My Drive/my_handwritten.jpg')

gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (5, 5), 0)
thresh = cv2.adaptiveThreshold(blur, 255, 1, 1, 11, 2)

contours = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)[1]
rects=[]

for cnt in contours:
  x, y, w, h = cv2.boundingRect(cnt)
  if h < 20: continue
  red = (0, 0, 255)
  cv2.rectangle(im, (x, y), (x+w, y+h), red, 2)
  rects.append((x,y,w,h))

cv2.imwrite('my_handwritten_bounding.png', im)

img_result = []
img_for_class = im.copy()

margin_pixel = 60

for rect in rects:
    #[y:y+h, x:x+w]
    img_result.append(
        img_for_class[rect[1]-margin_pixel : rect[1]+rect[3]+margin_pixel,
                      rect[0]-margin_pixel : rect[0]+rect[2]+margin_pixel])

    # Draw the rectangles
    cv2.rectangle(im, (rect[0], rect[1]),
                  (rect[0] + rect[2], rect[1] + rect[3]), (0, 0, 255), 2)

count = 0
nrows = 4
ncols = 7

plt.figure(figsize=(12,8))

for n in img_result:
    count += 1
    plt.subplot(nrows, ncols, count)
    plt.imshow(cv2.resize(n,(28,28)), cmap='Greys', interpolation='nearest')

plt.tight_layout()
plt.show()


title: RuntimeError in Pytorch when increasing batch size to more than 1

body:
This code for my custom data loader runs smoothly with batch_size=1, but when I increase batch size I get the following Error:

RuntimeError: Expected object of scalar type Double but got scalar type Long for sequence element 1 in sequence argument at position #1 'tensors'

import numpy as np
import matplotlib.pyplot as plt
import matplotlib
matplotlib.use("TkAgg")
import os, h5py
import PIL
#------------------------------
import torch
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms
#------------------------------
from data_augmentation import *
#------------------------------
dtype = torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor

class NiftiDataset(Dataset):

    def __init__(self,transformation_params,data_path, mode='train',transforms=None ):
        """
        Parameters:
            data_path (string): Root directory of the preprocessed dataset.
            mode (string, optional): Select the image_set to use, ``train``, ``valid``
            transforms (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.data_path = data_path
        self.mode = mode
        self.images = []
        self.labels = []
        self.W_maps = []
        self.centers = []
        self.radiuss = []
        self.pixel_spacings = []
        self.transformation_params = transformation_params
        self.transforms = transforms
        #-------------------------------------------------------------------------------------
        if self.mode == 'train':
            self.data_path = os.path.join(self.data_path,'train_set')
        elif self.mode == 'valid':
            self.data_path = os.path.join(self.data_path,'validation_set')
        #-------------------------------------------------------------------------------------
        for _, _, f in os.walk(self.data_path):
            for file in f:
                hdf_file = os.path.join(self.data_path,file)
                data = h5py.File(hdf_file,'r') # Dictionary
                # Preprocessing of Input Image and Label
                patch_img, patch_gt, patch_wmap = PreProcessData(file, data, self.mode, self.transformation_params)
                #print(type(data))
                self.images.append(patch_img) # 2D image
                #print('image shape is : ',patch_img.shape)
                self.labels.append(patch_gt) # 2D label
                #print('label shape is : ',patch_img.shape)
                self.W_maps.append(patch_wmap) # Weight_Map
                # self.centers.append(data['roi_center'][:]) # [x,y]
                # self.radiuss.append(data['roi_radii'][:]) # [R_min,R_max]
                # self.pixel_spacings.append(data['pixel_spacing'][:]) # [x , y , z]

    def __len__(self):
        return len(self.images)

    def __getitem__(self, index):
        image = self.images[index]
        label = self.labels[index]
        W_map = self.W_maps[index]
        if self.transforms is not None:
            image, label, W_maps = self.transforms(image, label, W_map)

        return image, label, W_map
#=================================================================================================

if __name__ == '__main__':
    # Test Routinue to check your threaded dataloader
    # ACDC dataset has 4 labels
    n_labels = 4
    path = './hdf5_files'
    batch_size = 1
    # Data Augmentation Parameters
    # Set patch extraction parameters
    size1 = (128, 128)
    patch_size = size1
    mm_patch_size = size1
    max_size = size1

    train_transformation_params = {
        'patch_size': patch_size,
        'mm_patch_size': mm_patch_size,
        'add_noise': ['gauss', 'none1', 'none2'],
        'rotation_range': (-5, 5),
        'translation_range_x': (-5, 5),
        'translation_range_y': (-5, 5),
        'zoom_range': (0.8, 1.2),
        'do_flip': (False, False),
        }

    valid_transformation_params = {
        'patch_size': patch_size,
        'mm_patch_size': mm_patch_size}

    transformation_params = { 'train': train_transformation_params,
                              'valid': valid_transformation_params,
                              'n_labels': 4,
                              'data_augmentation': True,
                              'full_image': False,
                              'data_deformation': False,
                              'data_crop_pad': max_size}
#====================================================================
dataset = NiftiDataset(transformation_params=transformation_params,data_path=path,mode='train')
dataloader = DataLoader(dataset=dataset,batch_size=2,shuffle=True,num_workers=0)

dataiter = iter(dataloader)
data = dataiter.next()
images, labels,W_map = data

#===============================================================================
# Data Visualization
#===============================================================================
print('image: ',images.shape,images.type(),'label: ',labels.shape,labels.type(),
    'W_map: ',W_map.shape,W_map.type())

img   = transforms.ToPILImage()(images[0,0,:,:,0].float())
lbl   = transforms.ToPILImage()(labels[0,0,:,:].float())
W_mp  = transforms.ToPILImage()(W_map [0,0,:,:].float())

plt.subplot(1,3,1)
plt.imshow(img,cmap='gray',interpolation=None)
plt.title('image')
plt.subplot(1,3,2)
plt.imshow(lbl,cmap='gray',interpolation=None)
plt.title('label')
plt.subplot(1,3,3)
plt.imshow(W_mp,cmap='gray',interpolation=None)
plt.title('Weight Map')
plt.show()

I have noticed some strange things such as Tensor types are different even though images and labels and weight maps are images with same type and size.
The Error Traceback:

Traceback (most recent call last):
  File "D:\Saudi_CV\Vibot\Smester_2\2_Medical Image analysis\Project_2020\OUR_Project\data_loader.py", line 118, in <module>
    data = dataiter.next()
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 385, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 79, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 64, in default_collate
    return default_collate([torch.as_tensor(b) for b in batch])
  File "F:\Download_2019\Anaconda3\lib\site-packages\torch\utils\data\_utils\collate.py", line 55, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: Expected object of scalar type Double but got scalar type Long for sequence element 1 in sequence argument at position #1 'tensors'
[Finished in 19.9s with exit code 1]


title: Pytorch in-place concatenation and conversion to tensor from numpy

body:
Suppose I have a list of tensors of the same size which could be concatenated along a dimension, say 0. Do any of the commands torch.cat or torch.stack or any numpy commands do the concatenation in-place? Also, if I want to convert a numpy ndarray to tensor. If I do the following, are two copies existing in the memory at any given time? I am dealing with a massive dataset so big only one copy of it can exist in the memory at any given time.

# initially data is a huge ndarray
data = torch.Tensor(data)


title: How to warp the image with optical flow and grid_sample?

body:
I’ve some questions about image warping with optical flows.

I would like to test warp the images by grid_sample, and I’ve got samples(two images with flow) from PWC-Net as follows:
input images from --> https://github.com/NVlabs/PWC-Net/tree/master/PyTorch/data
flow file from --> https://github.com/NVlabs/PWC-Net/tree/master/PyTorch/tmp

result image

input image 1

input image 2

import torch
import torch.nn as nn
import torch.nn.functional as F
import cv2
import numpy as np
from imageio import imread

def warp(x, flo):
“”"
warp an image/tensor (im2) back to im1, according to the optical flow

x: [B, C, H, W] (im2)
flo: [B, 2, H, W] flow

"""
B, C, H, W = x.size()
# mesh grid
xx = torch.arange(0, W).view(1 ,-1).repeat(H ,1)
yy = torch.arange(0, H).view(-1 ,1).repeat(1 ,W)
xx = xx.view(1 ,1 ,H ,W).repeat(B ,1 ,1 ,1)
yy = yy.view(1 ,1 ,H ,W).repeat(B ,1 ,1 ,1)
grid = torch.cat((xx ,yy) ,1).float()

if x.is_cuda:
    grid = grid.cuda()
vgrid = Variable(grid) + flo

# scale grid to [-1,1]
vgrid[: ,0 ,: ,:] = 2.0 *vgrid[: ,0 ,: ,:].clone() / max( W -1 ,1 ) -1.0
vgrid[: ,1 ,: ,:] = 2.0 *vgrid[: ,1 ,: ,:].clone() / max( H -1 ,1 ) -1.0

vgrid = vgrid.permute(0 ,2 ,3 ,1)
flo = flo.permute(0 ,2 ,3 ,1)
output = F.grid_sample(x, vgrid)
mask = torch.autograd.Variable(torch.ones(x.size())).cuda()
mask = F.grid_sample(mask, vgrid)

mask[mask <0.9999] = 0
mask[mask >0] = 1

return output*mask
test = cv2.readOpticalFlow(‘reference_frame_0010.flo’)
test = torch.from_numpy(test)
test = test.view(1, 2, H, W)
test = Variable(test).cuda()

test_img2 = cv2.imread(‘frame_0011.png’)
test_img2 = torch.from_numpy(test_img2)
test_img2 = test_img2.view(1, 3, H, W)
test_img2 = Variable(test_img2).cuda()

out = warp(test_img2, test)
out = out.view(H, W, 3)
out = out.cpu()
out = np.float32(out)
cv2.imwrite(‘test3.png’, out*255)

I’ve run the code above and got a result image. As you can see the last image, it is not I expected.
It seems like several parts of the images with grid.
Is it right to get a warped target image with optical flow?
If not, how can I get whole image looks like the input color images?

Please, help me to solve this problem.


title: Using tensor.share_memory_() vs multiprocessing.Queue in PyTorch when training model across multiple processes

body:
I'm using the multiprocessing package in pytorch to split the training across multiple processes. My x and y, train and test data are CUDA tensors. I'm trying to understand the difference between using the tensor.share_memory_() and the multiprocessing.Queue method to share cuda tensors. Which is preferred and why?

Here's my current code using tensor.share_memory_(). What changes should I make?

def train(model, features, target, epochs=1000):

    X_train, x_test, Y_train, y_test = train_test_split(features,
                                                    target,
                                                    test_size=0.4,
                                                    random_state=0)

    Xtrain_ = torch.from_numpy(X_train.values).float().share_memory_()
    Xtest_ = torch.from_numpy(x_test.values).float().share_memory_()

    Ytrain_ = (torch.from_numpy(Y_train.values).view(1,-1)[0]).share_memory_()
    Ytest_ = (torch.from_numpy(y_test.values).view(1,-1)[0]).share_memory_()

    optimizer = optim.Adam(model.parameters(), lr = 0.01)
    loss_fn = nn.NLLLoss()

    for epoch in range(epochs):

        #training code here

target method ends here

mp.set_start_method('spawn')

model = Net()
model.share_memory()

processes = []

for rank in range(1):
    p = mp.Process(target=train, args=(model, features, target))
    p.start()
    processes.append(p)

Env details: Python-3 and Linux


title: Pytorch Autograd backwards() in-place memory error

body:
The following code ran without error on pytorch nightly (1.5.0.dev20200206), however once I installed the stable 1.5 build the RNN Forward method defined below started to throw an error:

def forward(self, sequence):
        print('Sequence shape:', sequence.shape)
        sequence = sequence.clone().view(len(sequence), 1, -1)
        print("flattened shape: ", sequence.shape)
        lstm_out, hidden = self.lstm(
            sequence, self.hidden
        )
        print(lstm_out.shape)
        out_space = self.hidden2out(lstm_out[:, -1])
        self.hidden = hidden
        print("hiddens")
        print(hidden[0].shape)
        print(hidden[1].shape)
        print(" out_space: ", out_space.shape)
        out_scores = torch.sigmoid(out_space)
        print("out_scores: ", out_scores.shape)
        out = out_scores.squeeze()
        print(out.shape)
        return out

I added the clone() function to prevent in place memory modifications from view() and made variable assignments obviously not in place. However I still get the following error:

Sequence shape: torch.Size([200, 19, 62])
flattened shape:  torch.Size([200, 1, 1178])
torch.Size([200, 1, 8])
hiddens
torch.Size([1, 1, 8])
torch.Size([1, 1, 8])
 out_space:  torch.Size([200, 1])
out_scores:  torch.Size([200, 1])
torch.Size([200])
Warning: Error detected in AddmmBackward. Traceback of forward call that caused the error:
  File "main.py", line 240, in <module>
    main_loop(args)
  File "main.py", line 115, in main_loop
    train.run(args)
  File "/data/learnedbloomfilter/python/classifier/train.py", line 519, in run
    args.log_every,
  File "/data/learnedbloomfilter/python/classifier/train.py", line 88, in train
    predictions = model(features)
  File "/data/miniconda3/envs/lbf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/learnedbloomfilter/python/classifier/embedding_lstm.py", line 65, in forward
    sequence, self.hidden
  File "/data/miniconda3/envs/lbf/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/miniconda3/envs/lbf/lib/python3.7/site-packages/torch/nn/modules/rnn.py", line 570, in forward
    self.dropout, self.training, self.bidirectional, self.batch_first)
 (print_stack at /opt/conda/conda-bld/pytorch_1587428190859/work/torch/csrc/autograd/python_anomaly_mode.cpp:60)
Traceback (most recent call last):
  File "main.py", line 240, in <module>
    main_loop(args)
  File "main.py", line 115, in main_loop
    train.run(args)
  File "/data/learnedbloomfilter/python/classifier/train.py", line 519, in run
    args.log_every,
  File "/data/learnedbloomfilter/python/classifier/train.py", line 97, in train
    loss.backward(retain_graph=True)
  File "/data/miniconda3/envs/lbf/lib/python3.7/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/data/miniconda3/envs/lbf/lib/python3.7/site-packages/torch/autograd/__init__.py", line 100, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [8, 32]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: the backtrace
further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

I have isolated the error to forward(), but can't find the intermediate tensor [torch.FloatTensor [8, 32]] that seems to be causing the problem (none of the tensor shapes in my forward method match, so it must be in the lstm forward() method). I only use CPU, not cuda.

For the rest of the rnn code see this gist: https://gist.github.com/yaatehr/aac21cae05b24101f2369c97cfecb47b

Thanks!


title: Is it normal in PyTorch for accuracy to increase and decrease repeatedly?

body:
I am new to PyTorch, currently working on a Transfer Learning simple code. When I am training my model, I am getting a big variance between increase and decrease of the accuracy and loss. I trained the network for 50 epochs, and below is the result:

Epoch [1/50], Loss: 0.5477, Train Accuracy: 63%
Epoch [2/50], Loss: 2.1935, Train Accuracy: 75%
Epoch [3/50], Loss: 1.8811, Train Accuracy: 79%
Epoch [4/50], Loss: 0.0671, Train Accuracy: 77%
Epoch [5/50], Loss: 0.2522, Train Accuracy: 80%
Epoch [6/50], Loss: 0.0962, Train Accuracy: 88%
Epoch [7/50], Loss: 1.8883, Train Accuracy: 74%
Epoch [8/50], Loss: 0.3565, Train Accuracy: 83%
Epoch [9/50], Loss: 0.0228, Train Accuracy: 81%
Epoch [10/50], Loss: 0.0124, Train Accuracy: 81%
Epoch [11/50], Loss: 0.0252, Train Accuracy: 84%
Epoch [12/50], Loss: 0.5184, Train Accuracy: 81%
Epoch [13/50], Loss: 0.1233, Train Accuracy: 86%
Epoch [14/50], Loss: 0.1704, Train Accuracy: 82%
Epoch [15/50], Loss: 2.3164, Train Accuracy: 79%
Epoch [16/50], Loss: 0.0294, Train Accuracy: 85%
Epoch [17/50], Loss: 0.2860, Train Accuracy: 85%
Epoch [18/50], Loss: 1.5114, Train Accuracy: 81%
Epoch [19/50], Loss: 0.1136, Train Accuracy: 86%
Epoch [20/50], Loss: 0.0062, Train Accuracy: 80%
Epoch [21/50], Loss: 0.0748, Train Accuracy: 84%
Epoch [22/50], Loss: 0.1848, Train Accuracy: 84%
Epoch [23/50], Loss: 0.1693, Train Accuracy: 81%
Epoch [24/50], Loss: 0.1297, Train Accuracy: 77%
Epoch [25/50], Loss: 0.1358, Train Accuracy: 78%
Epoch [26/50], Loss: 2.3172, Train Accuracy: 75%
Epoch [27/50], Loss: 0.1772, Train Accuracy: 79%
Epoch [28/50], Loss: 0.0201, Train Accuracy: 80%
Epoch [29/50], Loss: 0.3810, Train Accuracy: 84%
Epoch [30/50], Loss: 0.7281, Train Accuracy: 79%
Epoch [31/50], Loss: 0.1918, Train Accuracy: 81%
Epoch [32/50], Loss: 0.3289, Train Accuracy: 88%
Epoch [33/50], Loss: 1.2363, Train Accuracy: 81%
Epoch [34/50], Loss: 0.0362, Train Accuracy: 89%
Epoch [35/50], Loss: 0.0303, Train Accuracy: 90%
Epoch [36/50], Loss: 1.1700, Train Accuracy: 81%
Epoch [37/50], Loss: 0.0031, Train Accuracy: 81%
Epoch [38/50], Loss: 0.1496, Train Accuracy: 81%
Epoch [39/50], Loss: 0.5070, Train Accuracy: 76%
Epoch [40/50], Loss: 0.1984, Train Accuracy: 77%
Epoch [41/50], Loss: 0.1152, Train Accuracy: 79%
Epoch [42/50], Loss: 0.0603, Train Accuracy: 82%
Epoch [43/50], Loss: 0.2293, Train Accuracy: 84%
Epoch [44/50], Loss: 0.1304, Train Accuracy: 80%
Epoch [45/50], Loss: 0.0381, Train Accuracy: 82%
Epoch [46/50], Loss: 0.1833, Train Accuracy: 84%
Epoch [47/50], Loss: 0.0222, Train Accuracy: 84%
Epoch [48/50], Loss: 0.0010, Train Accuracy: 81%
Epoch [49/50], Loss: 1.0852, Train Accuracy: 79%
Epoch [50/50], Loss: 0.0167, Train Accuracy: 83%

There are some epochs that have a much better accuracy and loss than others. However, the model loses them in later epochs. As I know, the accuracy should improve every epoch. Did I write the training code wrongly? If not, then is that normal? Any way to solve it? Shall the previous accuracy be saved and only if the accuracy of the next epoch is greater than the previous one then train one more epoch? I have been working on Keras previously, and I haven't experienced that problem. I am fine tuning the resent by freezing previous weights and adding only 2 classes for the final layer. Below is my code:

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)

num_epochs = 50
for epoch in range (num_epochs):
    #Reset the correct to 0 after passing through all the dataset
    correct = 0
    for images,labels in dataloaders['train']:
        images = Variable(images)
        labels = Variable(labels)
        if torch.cuda.is_available():
            images = images.cuda()
            labels = labels.cuda()

        optimizer.zero_grad()
        outputs = model_conv(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()
        _, predicted = torch.max(outputs, 1)
        correct += (predicted == labels).sum()

    train_acc = 100 * correct / dataset_sizes['train']
    print ('Epoch [{}/{}], Loss: {:.4f}, Train Accuracy: {}%'
            .format(epoch+1, num_epochs, loss.item(), train_acc))


title: How to load pretrained googlenet model in pytorch

body:
I'm trying to finetune a GoogleNet network over a specific dataset but I'm having trouble loading it. What I try now is:

model = torchvision.models.googlenet(pretrained=True)

However I get an error:

AttributeError: module 'torchvision.models' has no attribute 'googlenet'

I have the latest version of torchvision but reinstalled just to be sure, the error is still there.


title: Pytorch: RGB value ranges 0-1 after rescaling, How do I normalize images?

body:
I wrote a class to rescale images, but the RGB value became ranging from 0 to 1 after preocessing. What happened to the RGB which intuitively should be ranging from 0-255 ? Following are the Rescale class and the RGB values after rescaling.

Question:

Do I still need a Min-Max Normalization, map the RGB value to 0-1?

How do I apply transforms.Normalization, where do I put the Normalization, before or after the Rescale, how do I calculate the mean and variance, use the RGB value ranging from 0-255 or 0-1？

Thanks for your time!

class Rescale(object):
    def __init__(self, output_size):
        assert isinstance(output_size, (int, tuple))
        self.output_size = output_size

    def __call__(self, sample):
        image, anno = sample['image'], sample['anno']

        # get orginal width and height of image
        h, w = image.shape[0:2]
        # if output_size is an integer

        if isinstance(self.output_size, int):
            if h > w:
                new_h, new_w = h * self.output_size / w, self.output_size
            else:
                new_h, new_w = self.output_size / h, w * self.output_size / h

        # if output size is a tuple (a, b)
        else:
            new_h, new_w = self.output_size
        new_h, new_w = int(new_h), int(new_w)

        image = transform.resize(image, (new_h, new_w))
        return {'image': image, 'anno': anno}

[[[0.67264216 0.50980392 0.34503034]
  [0.67243905 0.51208121 0.34528431]
  [0.66719145 0.51817184 0.3459951 ]
  ...
  [0.23645098 0.2654311  0.3759458 ]
  [0.24476471 0.28003857 0.38963938]
  [0.24885877 0.28807445 0.40935877]]

 [[0.67465196 0.50994608 0.3452402 ]
  [0.68067157 0.52031373 0.3531848 ]
  [0.67603922 0.52732436 0.35839216]
  ...
  [0.23458333 0.25195098 0.36822142]
  [0.2461343  0.26886127 0.38314558]
  [0.2454384  0.27233056 0.39977664]]

 [[0.67707843 0.51237255 0.34766667]
  [0.68235294 0.5219951  0.35553024]
  [0.67772059 0.52747687 0.35659176]
  ...
  [0.24485294 0.24514568 0.36592999]
  [0.25407436 0.26205475 0.38063318]
  [0.2597007  0.27202914 0.40214216]]

 ...

[[[172 130  88]
  [172 130  88]
  [172 130  88]
  ...
  [ 63  74 102]
  [ 65  76 106]
  [ 67  77 112]]

 [[173 131  89]
  [173 131  89]
  [173 131  89]
  ...
  [ 65  74 103]
  [ 64  75 105]
  [ 63  73 108]]

 [[173 131  89]
  [174 132  90]
  [174 132  90]
  ...
  [ 63  72 101]
  [ 62  71 102]
  [ 61  69 105]]
  ...


title: How do you implement variable-length recurrent neural networks?

body:
What is a full working example (not snippets) of variable-length sequence inputs into recurrent neural networks (RNNs)?

For example PyTorch supposedly can implement variable-length sequences as input into RNNs, but there do not seem to be examples of full working code.

Relevant:
https://github.com/pytorch/pytorch/releases/tag/v0.1.10
https://discuss.pytorch.org/t/about-the-variable-length-input-in-rnn-scenario/345


title: "RuntimeError: expected scalar type Double but found Float" in Pytorch CNN training

body:
I just begin to learn Pytorch and create my first CNN. The dataset contains 3360 RGB images and I converted them to a [3360, 3, 224, 224] tensor. The data and label are in the dataset(torch.utils.data.TensorDataset). Below is the training code.
    def train_net():
        dataset = ld.load()
        data_iter = Data.DataLoader(dataset, batch_size=168, shuffle=True)
        net = model.VGG_19()
        summary(net, (3, 224, 224), device="cpu")
        loss_func = nn.CrossEntropyLoss()
        optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9, dampening=0.1)
        scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=50, gamma=0.1)
        for epoch in range(5):
            print("epoch:", epoch + 1)
            train_loss = 0
            for i, data in enumerate(data_iter, 0):
                x, y = data
                print(x.dtype)
                optimizer.zero_grad()
                out = net(x)
                loss = loss_func(out, y)
                loss.backward()
                optimizer.step()
                train_loss += loss.item()
                if i % 100 == 99:
                    print("loss:", train_loss / 100)
                    train_loss = 0.0
        print("finish train")

Then I have this error:
    Traceback (most recent call last):
              File "D:/python/DeepLearning/VGG/train.py", line 52, in <module>
                train_net()
              File "D:/python/DeepLearning/VGG/train.py", line 29, in train_net
                out = net(x)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\DeepLearning\VGG\model.py", line 37, in forward
                out = self.conv3_64(x)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\lib\site-packages\torch\nn\modules\container.py", line 117, in forward
                input = module(input)
              File "D:\python\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
                result = self.forward(*input, **kwargs)
              File "D:\python\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
                return self._conv_forward(input, self.weight)
              File "D:\python\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
                return F.conv2d(input, weight, self.bias, self.stride,
            RuntimeError: expected scalar type Double but found Float

I think there is something wrong with x and I print its type by print(x.dtype):
torch.float64

which is double instead of float. Do you know what`s wrong? Thanks for your help!


title: Understanding input shape to PyTorch conv1D?

body:
This seems to be one of the common questions on here (1, 2, 3), but I am still struggling to define the right shape for input to PyTorch conv1D.
I have text sequences of length 512 (number of tokens per sequence) with each token being represented by a vector of length 768 (embedding). The batch size I am using is 6.
So my input tensor to conv1D is of shape [6, 512, 768].
input = torch.randn(6, 512, 768)

Now, I want to convolve over the length of my sequence (512) with a kernel size of 2 using the conv1D layer from PyTorch.
Understanding 1:
I assumed that "in_channels" are the embedding dimension of the conv1D layer. If so, then a conv1D layer will be defined in this way where
in_channels = embedding dimension (768)
out_channels = 100 (arbitrary number)
kernel = 2

convolution_layer = nn.conv1D(768, 100, 2)
feature_map = convolution_layer(input)

But with this assumption, I get the following error:
RuntimeError: Given groups=1, weight of size 100 768 2, expected input `[4, 512, 768]` to have 768 channels, but got 512 channels instead

Understanding 2:
Then I assumed that "in_channels" is the sequence length of the input sequence. If so, then a conv1D layer will be defined in this way where
in_channels = sequence length (512)
out_channels = 100 (arbitrary number)
kernel = 2

convolution_layer = nn.conv1D(512, 100, 2)
feature_map = convolution_layer(input)

This works fine and I get an output feature map of dimension [batch_size, 100, 767]. However, I am confused. Shouldn't the convolutional layer convolve over the sequence length of 512 and output a feature map of dimension [batch_size, 100, 511]?
I will be really grateful for your help.


title: How make customised dataset in Pytorch for images and their masks?

body:
I have two dataset folder of tif images, one is a folder called BMMCdata, and the other one is the mask of BMMCdata images called BMMCmasks(the name of images are corresponds). I am trying to make a customised dataset and also split the data randomly to train and test. at the moment I am getting an error

self.filenames.append(fn)
AttributeError: 'CustomDataset' object has no attribute 'filenames'

Any comment will be appreciated a lot.

import torch
from torch.utils.data.dataset import Dataset  # For custom data-sets
from torchvision import transforms
from PIL import Image
import os.path as osp
import glob

folder_data = "/Users/parto/PycharmProjects/U-net/BMMCdata/data"

class CustomDataset(Dataset):
def __init__(self, root):

    self.filename = folder_data
    self.root = root
    self.to_tensor = transforms.ToTensor()
    filenames = glob.glob(osp.join(folder_data, '*.tif'))
    for fn in filenames:
        self.filenames.append(fn)
    self.len = len(self.filenames)
    print(fn)

def __getitem__(self, index):
    image = Image.open(self.filenames[index])
    return self.transform(image)

def __len__(self):

    return self.len
custom_img = CustomDataset(folder_data)
# total images in set
print(custom_img.len)

train_len = int(0.6*custom_img.len)
test_len = custom_img.len - train_len
train_set, test_set = CustomDataset.random_split(custom_img, lengths=[train_len, test_len])
# check lens of subset
len(train_set), len(test_set)

train_set = CustomDataset(folder_data)
train_set = torch.utils.data.TensorDataset(train_set, train=True, batch_size=4)
train_loader = torch.utils.data.DataLoader(train_set, batch_size=4, shuffle=True, num_workers=1)
print(train_set)
print(train_loader)

test_set = torch.utils.data.DataLoader(Dataset, batch_size=4, sampler= train_sampler)
test_loader = torch.utils.data.DataLoader(Dataset, batch_size=4)


title: PyTorch installation fails Could not find a version that satisfies the requirement

body:
I'm trying to install PyTorch with PyCharm Community Edition 2020.2.3 x64 and Python 3.9.0 on Windows 10 pro 64-bit OS PC machine
I've tried:
pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

and:
python -m pip install torch==1.7.0 -f https://download.pytorch.org/whl/torch_stable.html

Should I downgrade Python version, let's say to Python 3.8.6 or PyTorch version, to make it work, or am I doing something else incorrectly  besides this, maybe missed something to install, for example I did not selected CUDA, but seems like it is different reason:

ERROR: Could not find a version that satisfies the requirement
torch==1.7.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.7.0+cpu

with pip3 install https://download.pytorch.org/whl/cpu//torch-1.7.0%2Bcpu-cp38-cp38-win_amd64.whl:

ERROR: torch-1.7.0+cpu-cp38-cp38-win_amd64.whl is not a supported
wheel on this platform.

and
pip install torch==1.4.0+cpu torchvision==0.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html -vvv

ERROR: Could not find a version that satisfies the requirement
torch==1.4.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.4.0+cpu

Any advice, guide or example would be helpful
Solution:
Installed successfully with Python 3.8.6


title: Splitting custom PyTorch dataset into train loader and validation loader: Length of both same, even though dataset was split?

body:
I'm trying to split one of the Pytorch custom datasets (MNIST) into a training set and a validation set as follows:
def get_train_valid_splits(data_dir,
                           batch_size,
                           random_seed=1,
                           valid_size=0.2,
                           shuffle=True,
                           num_workers=4,
                           pin_memory=False):

    normalize = transforms.Normalize((0.1307,), (0.3081,))  # MNIST

    # define transforms
    valid_transform = transforms.Compose([
            transforms.ToTensor(),
            normalize
        ])

        train_transform = transforms.Compose([
            transforms.ToTensor(),
            normalize
        ])

    # load the dataset
    train_dataset = datasets.MNIST(root=data_dir, train=True,
                download=True, transform=train_transform)

    valid_dataset = datasets.MNIST(root=data_dir, train=True,
                download=True, transform=valid_transform)

    dataset_size = len(train_dataset)
    indices = list(range(dataset_size))
    split = int(np.floor(valid_size * dataset_size))

    if shuffle == True:
        np.random.seed(random_seed)
        np.random.shuffle(indices)

    train_idx, valid_idx = indices[split:], indices[:split]

    train_sampler = sampler.SubsetRandomSampler(train_idx)
    valid_sampler = sampler.SubsetRandomSampler(valid_idx)

    print(len(train_sampler))
    print(len(valid_sampler))

    train_loader = torch.utils.data.DataLoader(train_dataset,
                    batch_size=batch_size, sampler=train_sampler,
                    num_workers=num_workers, pin_memory=pin_memory)

    valid_loader = torch.utils.data.DataLoader(valid_dataset,
                    batch_size=batch_size, sampler=valid_sampler,
                    num_workers=num_workers, pin_memory=pin_memory)

    print(len(train_loader.dataset))
    print(len(valid_loader.dataset))

    return (train_loader, valid_loader)

After calling the function I notice that the results of the indices to sample look right, 48000 and 12000:
print(len(train_sampler))
print(len(valid_sampler))

But when I look at the length of the data set associated with train_loader and valid_loader:
print(len(train_loader.dataset))
print(len(valid_loader.dataset))

I get the same length for both: 60000! Any idea what is going on here? Why is it giving the same length for both, even though I clearly split it by indices?


title: Reshaping Pytorch tensor

body:
I have a tensor of size (24, 2, 224, 224) in Pytorch.

24 = batch size
2 = matrixes representing foreground and
background
224 = image height dimension
224 = image width
dimension

This is the output of a CNN that performs binary segmentation. In each cell of the 2 matrixes is stored the probability for that pixel to be foreground or background: [n][0][h][w] + [n][1][h][w] = 1 for every coordinate

I want to reshape it into a tensor of size (24, 1, 224, 224). The values in the new layer should be 0 or 1 according to the matrix in which the probability was higher.

How can I do that? Which function should I use?


title: Column-dependent bounds in torch.clamp

body:
I would like to do something similar to np.clip on PyTorch tensors on a 2D array. More specifically, I would like to clip each column in a specific range of value (column-dependent). For example, in numpy, you could do:

x = np.array([-1,10,3])
low = np.array([0,0,1])
high = np.array([2,5,4])
clipped_x = np.clip(x, low, high)

clipped_x == np.array([0,5,3]) # True

I found torch.clamp, but unfortunately it does not support multidimensional bounds (only one scalar value for the entire tensor). Is there a "neat" way to extend that function to my case?

Thanks!


title: Save model with updated weights in pytorch

body:
I have a model that I want to train for 5 epochs. Then, I would like to see where the model was wrong and increase the training set accordingly. How can I save the following model with the learnt weight?

trainer_ = Trainer(network = network,
                       optimizer = optim.Adam(network.parameters(), lr=0.001),
                       loss_function = loss_function,
                       train_loader = train_loader,
                       valid_every = 100,
                       print_every = 50,
                       save_every = 15000,
                       save_path = ".",
                       cudaok = is_cuda_available)

trainer_.run(4,is_cuda_available)

I have tried this:

path = os.path.join(project_path, 'model.pth')
torch.save(network.cpu().state_dict(), path) # saving model

But I don't really think that the object network contains weights.

I am very confused here. Can anyone help? Thank you!


title: How to make a Trainer pad inputs in a batch with huggingface-transformers?

body:
I'm trying to train a model using a Trainer, according to the documentation (https://huggingface.co/transformers/master/main_classes/trainer.html#transformers.Trainer) I can specify a tokenizer:

tokenizer (PreTrainedTokenizerBase, optional) – The tokenizer used to
preprocess the data. If provided, will be used to automatically pad
the inputs the maximum length when batching inputs, and it will be
saved along the model to make it easier to rerun an interrupted
training or reuse the fine-tuned model.

So padding should be handled automatically, but when trying to run it I get this error:

ValueError: Unable to create tensor, you should probably activate
truncation and/or padding with 'padding=True' 'truncation=True' to
have batched tensors with the same length.

The tokenizer is created this way:
tokenizer = BertTokenizerFast.from_pretrained(pretrained_model)

And the Trainer like that:
trainer = Trainer(
    tokenizer=tokenizer,
    model=model,
    args=training_args,
    train_dataset=train,
    eval_dataset=dev,
    compute_metrics=compute_metrics
)

I've tried putting the padding and truncation parameters in the tokenizer, in the Trainer, and in the training_args. Nothing does. Any idea?


title: Derivative in both arguments of torch.nn.BCELoss()

body:
When using a torch.nn.BCELoss() on two arguments that are both results of some earlier computation, I get some curious error, which this question is about:

RuntimeError: the derivative for 'target' is not implemented

The MCVE is as follows:

import torch
import torch.nn.functional as F

net1 = torch.nn.Linear(1,1)
net2 = torch.nn.Linear(1,1)
loss_fcn = torch.nn.BCELoss()

x = torch.zeros((1,1))

y = F.sigmoid(net1(x)) #make sure y is in range (0,1)
z = F.sigmoid(net2(y)) #make sure z is in range (0,1)

loss = loss_fcn(z, y) #works if we replace y with y.detach()

loss.backward()

It turns out if we call .detach() on y the error disappears. But this results in a different computation, now in the .backward()-pass, the gradients with respect to the second argument of the BCELoss will not be computed.

Can anyone explain what I'm doing wrong in this case? As far as I know all pytorch modules in torch.nn should support computing gradients. And this error message seems to tell me that the derivative is not implemented for y, which is somehow strange, as you can compute the gradient of y, but not of y.detach() which seems to be contradictory.


title: Why tensor.view() is not working in pytorch?

body:
I have the following piece of code.

embedded = self.embedding(input).view(1, 1, -1)
embedded = self.drop(embedded)
print(embedded[0].size(), hidden[0].size())
concatenated_output = torch.cat((embedded[0], hidden[0]), 1)

The last line of the code is giving me the following error.

  RuntimeError: inconsistent tensor sizes at /data/users/soumith/miniconda2/conda-bld/pytorch-0.1.9_1487344852722/work/torch/lib/THC/generic/THCTensorMath.cu:141

Please note, when I am printing the tensor shapes at line no. 3, I am getting the following output.

torch.size([1, 300]) torch.size([1, 1, 300])

Why I am getting [1, 300] shape for embedded tensor even though I have used the view method as view(1, 1, -1)?

Any help would be appreciated!


title: Loading and Saving C++ torch model in Python and another OS

body:
I have a custom LSTM pytorch model in C++. After training I save the model using

torch::save(lstmNetwork, 'model.pt');

where lstmNetwork is std::shared_ptr<LSTMNetwork>;

class LSTMNetwork : public torch::nn::Module {
public:
  LSTMNetwork(const torch::nn::LSTMOptions &lstmOpts1,
            const torch::nn::LSTMOptions &lstmOpts2,
            const torch::nn::LinearOptions &linearOpts);
  ~LSTMNetwork();

  torch::Tensor forward(const torch::Tensor &x);

private:
  torch::nn::LSTM lstm1;
  torch::nn::LSTM lstm2;
  torch::nn::Linear linear;
};

LSTMNetwork::LSTMNetwork(const torch::nn::LSTMOptions &lstmOpts1,
                     const torch::nn::LSTMOptions &lstmOpts2,
                     const torch::nn::LinearOptions &linearOpts)
    : torch::nn::Module(),
      lstm1(register_module("lstm1", torch::nn::LSTM(lstmOpts1))),
      lstm2(register_module("lstm2", torch::nn::LSTM(lstmOpts2))),
      linear(register_module("linear", torch::nn::Linear(linearOpts))) {}

LSTMNetwork::~LSTMNetwork() {}

torch::Tensor LSTMNetwork::forward(const torch::Tensor &input) {
  torch::nn::RNNOutput lstm_out = this->lstm1->forward(input);
  lstm_out = this->lstm2->forward(lstm_out.output);
  torch::Tensor y_pred = this->linear(lstm_out.output[-1]);
  return y_pred.view({-1});
}

// .....
  torch::nn::LSTMOptions lstmOpts1(input, hidden);
  torch::nn::LSTMOptions lstmOpts2(hidden, hidden);
  lstmOpts1.layers(numLayers)
      .dropout(NetworkConstants::klsmt1DropOut)
      .with_bias(NetworkConstants::kIncludeBias);
  lstmOpts2.layers(numLayers)
      .dropout(NetworkConstants::klsmt2DropOut)
      .with_bias(NetworkConstants::kIncludeBias);

  torch::nn::LinearOptions linearOpts(hidden, output);
  linearOpts.with_bias(false);
  lstmNetwork = std::make_shared<LSTMNetwork>(lstmOpts1, lstmOpts2, linearOpts);

  torch::optim::AdamOptions opts(learningRate);
  optimizer = std::make_shared<torch::optim::Adam>(
      torch::optim::Adam(lstmNetwork->parameters(), opts));

  if (gpuAvailable) {
    lstmNetwork->to(torch::kCUDA);
  }
//....

I can reload the trained model in C++ without issue, using:

 torch::load(lstmNetwork, 'model.pt');

I have two questions related to the above snippets:

a) Is the trained model OS independent? Can I save the model on Windows 10 and reload it on Ubuntu 18.04 LTS?
(The application is complied on both the platforms)

b) How do I open the saved pytorch model ('model.pt') in python?

Following gives an error:

lstmModel = torch.load('model.pt')

Error : (showing the exact model name)

        lstmModel = torch.load('stockData\BOM500570.pt') Traceback (most recent call last):   File "C:\Python37\lib\tarfile.py", line 187, in
        nti
            n = int(s.strip() or "0", 8) ValueError: invalid literal for int() with base 8: 'ZZZZZZZZ'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):   File
  "C:\Python37\lib\tarfile.py", line 2289, in next
      tarinfo = self.tarinfo.fromtarfile(self)   File "C:\Python37\lib\tarfile.py", line 1095, in fromtarfile
      obj = cls.frombuf(buf, tarfile.encoding, tarfile.errors)   File "C:\Python37\lib\tarfile.py", line 1037, in frombuf
      chksum = nti(buf[148:156])   File "C:\Python37\lib\tarfile.py", line 189, in nti
      raise InvalidHeaderError("invalid header") tarfile.InvalidHeaderError: invalid header

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):   File
  "C:\Python37\lib\site-packages\torch\serialization.py", line 591, in
  _load
      return legacy_load(f)   File "C:\Python37\lib\site-packages\torch\serialization.py", line 502, in
  legacy_load
      with closing(tarfile.open(fileobj=f, mode='r:', format=tarfile.PAX_FORMAT)) as tar, \   File
  "C:\Python37\lib\tarfile.py", line 1591, in open
      return func(name, filemode, fileobj, **kwargs)   File "C:\Python37\lib\tarfile.py", line 1621, in taropen
      return cls(name, mode, fileobj, **kwargs)   File "C:\Python37\lib\tarfile.py", line 1484, in init
      self.firstmember = self.next()   File "C:\Python37\lib\tarfile.py", line 2301, in next
      raise ReadError(str(e)) tarfile.ReadError: invalid header

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):   File "", line 1, in
     File
  "C:\Python37\lib\site-packages\torch\serialization.py", line 422, in
  load
      return _load(f, map_location, pickle_module, **pickle_load_args)   File "C:\Python37\lib\site-packages\torch\serialization.py", line 595,
  in _load
      raise RuntimeError("{} is a zip archive (did you mean to use torch.jit.load()?)".format(f.name)) RuntimeError:
  stockData\BOM500570.pt is a zip archive (did you mean to use
  torch.jit.load()?)

However following executes correctly:

lstmModel = torch.jit.load('model.pt')

For the latter case, I cannot use python's forward()/ overloaded () function, and I'm not sure how to use the loaded model.

>>> lstmModel = torch.jit.load('stockData\BOM500570.pt')
>>> input = torch.arange(5)
>>> lstmModel.forward(input)

  Traceback (most recent call last):   File "", line 1, in
     File "C:\Python37\lib\site-packages\torch\jit__init__.py",
  line 1585, in getattr
      return super(ScriptModule, self).getattr(attr)   File "C:\Python37\lib\site-packages\torch\nn\modules\module.py", line 585,
  in getattr
      type(self).name, name)) AttributeError: 'ScriptModule' object has no attribute 'forward'

>>> lstmModel(input)

  Traceback (most recent call last):   File "", line 1, in
     File
  "C:\Python37\lib\site-packages\torch\nn\modules\module.py", line 541,
  in call
      result = self.forward(*input, **kwargs)   File "C:\Python37\lib\site-packages\torch\jit__init__.py", line 1585, in
  getattr
      return super(ScriptModule, self).getattr(attr)   File "C:\Python37\lib\site-packages\torch\nn\modules\module.py", line 585,
  in getattr
      type(self).name, name)) AttributeError: 'ScriptModule' object has no attribute 'forward'

>>> lstmModel

ScriptModule(
      original_name=Module
      (lstm1): ScriptModule(original_name=Module)
      (lstm2): ScriptModule(original_name=Module)
      (linear): ScriptModule(original_name=Module)
    )