title: Output size is too small for SpatialAveragePooling in Unet

body:
Tried this to use Resnext as encoder in Unet from here, but keep getting  RuntimeError: Given input size: (4320x4x4). Calculated output size: (4320x-6x-6). Output size is too small at /opt/conda/conda-bld/pytorch_1525796793591/work/torch/lib/THCUNN/generic/SpatialAveragePooling.cu:63

(input image 128*128 batch size 32)

class UNetResNext(nn.Module):
    def __init__(self, encoder_depth, num_classes, num_filters=32, dropout_2d=0.2,
                 pretrained=False, is_deconv=False):
        super().__init__()
        self.num_classes = num_classes
        self.dropout_2d = dropout_2d
    if encoder_depth == 34:
        self.encoder = resnext34()
        bottom_channel_nr = 512
    elif encoder_depth == 101:
        self.encoder = resnext101()
        bottom_channel_nr = 2048
    elif encoder_depth == 152:
        self.encoder = resnext152()
        bottom_channel_nr = 2048

    else:
        raise NotImplementedError('only 34, 101, 152 version of Resnext are implemented')

    self.pool = nn.MaxPool2d(2, 2)

    self.relu = nn.ReLU(inplace=True)

    self.conv1 = nn.Sequential(self.encoder.conv1,
                               self.encoder.bn1,
                               self.encoder.relu,
                               self.pool)

    self.conv2 = self.encoder.layer1
    self.conv3 = self.encoder.layer2
    self.conv4 = self.encoder.layer3
    self.conv5 = self.encoder.layer4
    self.center = DecoderCenter(bottom_channel_nr, num_filters * 8 *2, num_filters * 8, False)

    self.dec5 =  DecoderBlockV2(bottom_channel_nr + num_filters * 8, num_filters * 8 * 2, num_filters * 8,   is_deconv)
    self.dec4 = DecoderBlockV2(bottom_channel_nr // 2 + num_filters * 8, num_filters * 8 * 2, num_filters * 8, is_deconv)
    self.dec3 = DecoderBlockV2(bottom_channel_nr // 4 + num_filters * 8, num_filters * 4 * 2, num_filters * 2, is_deconv)
    self.dec2 = DecoderBlockV2(bottom_channel_nr // 8 + num_filters * 2, num_filters * 2 * 2, num_filters * 2 * 2,
                               is_deconv)
    self.dec1 = DecoderBlockV2(num_filters * 2 * 2, num_filters * 2 * 2, num_filters, is_deconv)
    self.dec0 = ConvRelu(num_filters, num_filters)
    self.final = nn.Conv2d(num_filters, num_classes, kernel_size=1)

def forward(self, x):
    conv1 = self.conv1(x)
    conv2 = self.conv2(conv1)
    conv3 = self.conv3(conv2)
    conv4 = self.conv4(conv3)
    conv5 = self.conv5(conv4)
    center = self.center(conv5)
    dec5 = self.dec5(torch.cat([center, conv5], 1))
    dec4 = self.dec4(torch.cat([dec5, conv4], 1))
    dec3 = self.dec3(torch.cat([dec4, conv3], 1))
    dec2 = self.dec2(torch.cat([dec3, conv2], 1))
    dec1 = self.dec1(dec2)
    dec0 = self.dec0(dec1)

    return self.final(F.dropout2d(dec0, p=self.dropout_2d))


title: PyTorch: nn.LSTM output different results for the same inputs in the same batch

body:
I try to implement a two-layer bidirectional LSTM with torch.nn.LSTM.

I made a toy example: a batch of 3 tensors, which are exactly the same (see my code below). And I expected the outputs of the BiLSTM  to be the the same along the batch dimension, i.e. out[:,0,:] == out[:,1,:] == out[:, 2, :].

But it seemed not to be the case. According to my experiments, 20%~40% of the time, the output were not the same. So I wonder where I got it wrong.

# Python 3.6.6, Pytorch 0.4.1
import torch

def test(hidden_size, in_size):
    seq_len, batch = 4, 3
    bilstm = torch.nn.LSTM(input_size=in_size, hidden_size=hidden_size,
                            num_layers=2, bidirectional=True)

    # create a batch with 3 exactly the same tensors
    a = torch.rand(seq_len, 1, in_size)  # (seq_len, 1, in_size)
    x = torch.cat((a, a, a), dim=1)

    out, _ = bilstm(x)  # (seq_len, batch, n_direction * hidden_size)

    # expect the output should be the same along the batch dimension
    assert torch.equal(out[:, 0, :], out[:, 1, :])
    assert torch.equal(out[:, 1, :], out[:, 2, :])

if __name__ == '__main__':
    count, total = 0, 0
    for h_size in range(1, 51):
        for in_size in range(1, 51):
            total += 1
            try:
                test(h_size, in_size)
            except AssertionError:
                count += 1
    print('percentage of assertion error:', count / total)


title: Element wise calculation breaks autograd

body:
I am using pytorch to calculate loss for a logistic regression (I know pytorch can do this automatically but I have to make it myself). My function is defined below but the cast to torch.tensor breaks autograd and gives me w.grad = None. Im new to pytorch so Im sorry.

logistic_loss = lambda X,y,w: torch.tensor([torch.log(1 + torch.exp(-y[i] * torch.matmul(w, X[i,:]))) for i in range(X.shape[0])], requires_grad=True)


title: How to take last hidden state of bidirectional LSTM in Pytorch?

body:
I'm not sure how to select the last hidden/cell states in a bidirectional LSTM in Pytorch.

output, (hn, cn) = bi_lstm(input, (h0, c0))

How can I use  output,  hn and cn in order to extract the last forward and backward hidden states?

In the case of backward LSTM, I want to extract the hidden state I get after processing the entire sequence backwards.


title: how can we use translate fucntion of fairseq without bpe

body:
i was using pretrained model of japanes translation , trained via fairseq http://www.kecl.ntt.co.jp/icl/lirg/jparacrawl/

this contains following files

model.pt dict.en.txt dict.ja.txt spm.en.nopretok.model spm.en.nopretok.vocab spm.ja.nopretok.model spm.ja.nopretok.vocab

i wanted to use

  from fairseq.models.transformer import TransformerModel
    zh2en = TransformerModel.from_pretrained(
      '/path/to/checkpoints',
      checkpoint_file='checkpoint_best.pt',
      data_name_or_path='data-bin/wmt17_zh_en_full',
      bpe='subword_nmt',
      bpe_codes='data-bin/wmt17_zh_en_full/zh.code'
    )

this function of fairseq but i do not have BPE codes file.

please suggest how can i use this pretrined model in python using fairseq


title: loss.backward() with minibatch in pytorch

body:
I came across this code online and I was wondering if I interpreted it correctly. Below is a part of a gradient descent process. full code available through the link https://jovian.ml/aakashns/03-logistic-regression. My question is as followed: During the training step, I guess the author is trying to minimize the loss for each batch by updating the parameters. However, how can we be sure the total loss of all training samples is minimized if loss.backward() is only applied to the batch loss?

def fit(epochs, lr, model, train_loader, val_loader, opt_func=torch.optim.SGD):
    history = []
    optimizer = opt_func(model.parameters(), lr)

for epoch in range(epochs):
    # Training Phase
    for batch in train_loader:
        loss = model.training_step(batch)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
    # Validation phase
    result = evaluate(model, val_loader)
    model.epoch_end(epoch, result)
    history.append(result)
return history


title: Why my model trained by fastai can not be used in pytorch's load function?

body:
and it show me that:
Error(s) in loading state_dict for ResNet:
    Missing key(s) in state_dict: "conv1.weight",...blabla

Why? They are resnet34 model both. Thanks for your help!


title: Exclude indices from Pytorch tensor

body:
I have an MxN table of distances between two distinct sets of points of size M <= N. I would like to find associate to each point of the first set M points in the second set in the following way.
Suppose that the shortest of all pairwise distances is between the i0 point of the first set and the j0 of the second. Then we attribute point i0 of the first set to j0 in the second. For the second pair, I have to find i1 != i0 and j1 != j0 such that the distance is minimal among remaining non-paired points.
I figure that I could do the first step by using torch.min function that will deliver me both minimal value as well as its 2d index in the matrix. But for the next steps I'll need to each time exclude a row a colunm, while keeping their original indices.
In other words, if I have a 3x4 matrix, and my first element is (1,2), I would like to be left with a 2x3 matrix with indices 0,2 and 0,1,3. So that, if my second desired element position in the original matrix is, say (2,3) I will be given (2,3) as a result of performing torch.max on the matrix with excluded row and column, rather than (1,2) again.
P.S. I could reach my goal by replacing the values in row and column I'd like to exclude by, say, positive infinities, but I think the question is still worth asking.


title: PyTorch GPU memory leak during inference

body:
I am trying to encode documents sentence-wise with a huggingface transformer module. I'm using the very small google/bert_uncased_L-2_H-128_A-2 pretrained model with the following code:
def pre_encode_wikipedia(model, tokenizer, device, save_path):

  document_data_list = []

  for iteration, document in enumerate(wikipedia_small['text']):
    torch.cuda.empty_cache()

    sentence_embeds_per_doc = [torch.randn(128)]
    attention_mask_per_doc = [1]
    special_tokens_per_doc = [1]

    doc_split = nltk.sent_tokenize(document)
    doc_tokenized = tokenizer.batch_encode_plus(doc_split, padding='longest', truncation=True, max_length=512, return_tensors='pt')

    for key, value in doc_tokenized.items():
      doc_tokenized[key] = doc_tokenized[key].to(device)

    with torch.no_grad():
      doc_encoded = model(**doc_tokenized)

    for sentence in doc_encoded['last_hidden_state']:
      sentence[0].to('cpu')
      sentence_embeds_per_doc.append(sentence[0])
      attention_mask_per_doc.append(1)
      special_tokens_per_doc.append(0)

    sentence_embeds = torch.stack(sentence_embeds_per_doc)
    attention_mask = torch.FloatTensor(attention_mask_per_doc)
    special_tokens_mask = torch.FloatTensor(special_tokens_per_doc)

    document_data = torch.utils.data.TensorDataset(*[sentence_embeds, attention_mask, special_tokens_mask])
    torch.save(document_data, f'{save_path}{time.strftime("%Y%m%d-%H%M%S")}{iteration}.pt')
    print(f"Document at {iteration} encoded and saved.")

After about 200-300 iterations on my local GTX 1060 3GB I get an error saying that my CUDA memory is full. Running this code on Colab with more GPU RAM gives me a few thousand iterations.
Things I've tried:

Adding torch.cuda.empty_cache() to the start of every iteration to clear out previously held tensors
Wrapping the model in torch.no_grad() to disable the computation graph
Setting model.eval() to disable any stochastic properties that might take up memory
Sending the output straight to CPU in hopes to free up memory

I'm baffled as to why my memory keeps overflowing. I've trained several models of bigger sizes, applying all the standard practices of a training loop (optimizer.zero_grad(), etc.) I've never had this problem. Why does it appear during this seemingly trivial task?
Edit #1
Changing sentence[0].to('cpu') to cpu_sentence = sentence[0].to('cpu') gave me a few thousand iterations before VRAM usage suddenly spiked, causing the run to crash:


title: How to vectorise a list of matrix vector multiplications using pytorch/numpy

body:
For example, I have a list of N B x H tensor(i.e. a N x B x H tensor) and a list of N vectors (i.e. N x B tensor). And I want multiply each B x H tensor in the list with corresponding B dimensional tensor, resulting a N x H tensor.

I know how to use a single for-loop with PyTorch to implement the computation, but is there any vectorised implantation? (i.e. no for-loop, just using PyTorch/numpy operations)


title: Periodic pattern in loss when training neural networks using pytorch

body:
I am using Pytorch train a model on MNIST, the loss curve has the periodic pattern shown in the figure. I've double checked the data loader and the dataset is shuffled in each epoch. Any suggestions for the possible reason? Thanks.

Loss curve during training, training loss in blue and test loss in red


title: Loss function used for multi-dimensional feature mapping

body:
I am working on a video animation project using PyTorch. My dataset contains 3904x60 mfcc audio features(input) and corresponding 3904x3 video features(output). The goal is to train a neural network model such that given an unknown audio feature, the model maps it into its corresponding video feature. In other words, the neural network performs a 60 to 3 feature mapping. I have already built the neural network following this tutorial:
class ConvNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Sequential(
            nn.Conv1d(1, 32, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv1d(32, 64, kernel_size=5, stride=1, padding=2),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=2))
        self.drop_out = nn.Dropout()
        self.fc1 = nn.Linear(15 * 64, 1000)
        self.fc2 = nn.Linear(1000, 3)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out

and my training code looks like:
model = ConvNet()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    for i, (a, v) in enumerate(train_loader):
        # Run the forward pass
        a = a.float()
        v = v.long()
        outputs = model(a.view(a.size(0),1,a.size(1)))
        loss = criterion(outputs, v)
        loss_list.append(loss.item())

        # Backprop and perform Adam optimisation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # Track the accuracy
        total = labels.size(0)
        _, predicted = torch.max(outputs.data, 1)
        correct = (predicted == labels).sum().item()
        acc_list.append(correct / total)

        if (i + 1) % 100 == 0:
            print('Epoch[{}/{}],Step[{}/{}],Loss{:.4f},Accuracy{:.2f}%'
              .format(epoch + 1, num_epochs, i + 1, total_step, loss.item(),
                      (correct / total) * 100))

but received an error in training:

---> 15         loss = criterion(outputs, v)
multi-target not supported at /Users/soumith/miniconda2/conda-bld/pytorch_1532623076075/work/aten/src/THNN/generic/ClassNLLCriterion.c:21

I defined the batch size to be 4 so each a and v in the iteration should be a 4 by 60 tensor and a 4 by 3 tensor, respectively. How do I solve this problem?


title: How to convert this TensorFlow code to PyTorch?

body:
I have this simple tensorflow code block, what is the equivalent in pytorch? I am stuck trying to code it. I have encountered multiple Runtime errors, due to the dimensions.
This is the tensorflow code:

x = tf.placeholder(tf.float32, [708, 256, 3])
y = tf.placeholder(tf.float32, [708, 4])

f1 = tf.Variable(tf.random_normal([5,3,8*3]))
f2 = tf.Variable(tf.random_normal([5,8*3,4*3]))
n1 = tf.Variable(tf.random_normal([61*12,12]))
n2 = tf.Variable(tf.random_normal([12,4]))

b1 = tf.Variable(tf.random_normal([8*3]))
b2 = tf.Variable(tf.random_normal([4*3]))
b3 = tf.Variable(tf.random_normal([12]))
b4 = tf.Variable(tf.random_normal([4]))

conv1 = tf.nn.conv1d(x,f1,stride=1,padding="VALID")
conv1 = tf.nn.bias_add(conv1, b1)
conv1 = tf.nn.sigmoid(conv1)

p1 = tf.layers.average_pooling1d(conv1, pool_size=2, strides=2, padding='VALID')

conv2 = tf.nn.conv1d(p1,f2,stride=1,padding="VALID")
conv2 = tf.nn.bias_add(conv2, b2)
conv2 = tf.nn.sigmoid(conv2)
p2 = tf.layers.average_pooling1d(conv2, pool_size=2, strides=2, padding='VALID')

nn = tf.layers.Flatten()(p2)

fc1 = tf.add(tf.matmul(nn, n1), b3)
fc1 = tf.nn.sigmoid(fc1)
out = tf.add(tf.matmul(fc1, n2), b4)
out = tf.nn.softmax(out)

How can I implement the same in pytorch? The code below is what I tried, but the dimensions are messed up, I think due to the channel inputs.

class TwoLayerNet(torch.nn.Module):
    def __init__(self):
        super(TwoLayerNet,self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(1,3, 3*8, kernel_size=5, stride=1),
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride=0))
        self.conv2 = nn.Sequential(
            nn.Conv1d(3*8, 12, kernel_size=5, stride=1),
            nn.Sigmoid(),
            nn.AvgPool1d(kernel_size=2, stride = 0))
        #self.drop_out = nn.Dropout()

        self.fc1 = nn.Linear(708, 732)
        self.fc2 = nn.Linear(732, 4)

    def forward(self, x):
        out = self.conv1(x)
        out = self.conv2(out)
        out = out.reshape(out.size(0), -1)
        out = self.drop_out(out)
        out = self.fc1(out)
        out = self.fc2(out)
        return out


title: How to determine PyTorch data input dimensions for model visualization?

body:
I am coming over from Keras to PyTorch and am having a hard time creating graph visualizations of models. In Keras, you can simply call plot_model(), and that function will use graphviz to produce a nice graph.

However, in PyTorch it is not so easy. I've seen there are a lot of tools available, such as TorchViz, but they all require that you pass in some input into the model. For example:

This StackOverflow question (How do I visualize a net in Pytorch?)
suggests using the torchviz package like so:

x = torch.zeros(1, 3, 224, 224, dtype=torch.float, requires_grad=False)
out = resnet(x)
make_dot(out)

The HiddenLayer package (example) also requires dummy input:

model = torchvision.models.vgg16()
g = hl.build_graph(model, torch.zeros([1, 3, 224, 224]))

So my question is: Suppose I download a pre-trained model that has little documentation. How can I determine what is the needed dummy input dimension in order to run these visualization functions?

For example, when I print() an AlexNet model, what part of the result below tells me that the input should be [1, 3, 224, 224]?

model = torchvision.models.alexnet(pretrained=True)
print(model)

# AlexNet(
#   (features): Sequential(
#     (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
#     (1): ReLU(inplace=True)
#     (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
#     (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
#     (4): ReLU(inplace=True)
#     (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
#     (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
#     (7): ReLU(inplace=True)
#     (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
#     (9): ReLU(inplace=True)
#     (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
#     (11): ReLU(inplace=True)
#     (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
#   )
#   (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
#   (classifier): Sequential(
#     (0): Dropout(p=0.5, inplace=False)
#     (1): Linear(in_features=9216, out_features=4096, bias=True)
#     (2): ReLU(inplace=True)
#     (3): Dropout(p=0.5, inplace=False)
#     (4): Linear(in_features=4096, out_features=4096, bias=True)
#     (5): ReLU(inplace=True)
#     (6): Linear(in_features=4096, out_features=1000, bias=True)
#   )
# )

The first layer says: Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2)). How would I know from that fact that I need to create [1, 3, 224, 224] input?


title: Why does this tensor.unfold method in this example add a dimension?

body:
I am familiarizing myself with the Pytorch unfold method from https://pytorch.org/docs/stable/tensors.html#torch.Tensor.unfold

I looked at their example which is

>>> x = torch.arange(1., 8)
>>> x
tensor([ 1.,  2.,  3.,  4.,  5.,  6.,  7.])
>>> x.unfold(0, 2, 1)
tensor([[ 1.,  2.],
        [ 2.,  3.],
        [ 3.,  4.],
        [ 4.,  5.],
        [ 5.,  6.],
        [ 6.,  7.]])

I understand above that when we unfold in dimension 0, we take chunks of size 2 at a time with stride 1 and therefore, the result is an arrangement of different chunks, which are [1., 2.], [2., 3.] and so on. As we have 6 chunks at the end, the chunks will be put together and the final shape is (6,2).

However, I have another example I ran as shown below.

In [115]: s = torch.arange(20).view(1,10,2)

In [116]: s
Out[116]:
tensor([[[ 0,  1],
         [ 2,  3],
         [ 4,  5],
         [ 6,  7],
         [ 8,  9],
         [10, 11],
         [12, 13],
         [14, 15],
         [16, 17],
         [18, 19]]])

In [117]: s.unfold(0,1,1)
Out[117]:
tensor([[[[ 0],
          [ 1]],

         [[ 2],
          [ 3]],

         [[ 4],
          [ 5]],

         [[ 6],
          [ 7]],

         [[ 8],
          [ 9]],

         [[10],
          [11]],

         [[12],
          [13]],

         [[14],
          [15]],

         [[16],
          [17]],

         [[18],
          [19]]]])

In [119]: s.unfold(0,1,1).shape
Out[119]: torch.Size([1, 10, 2, 1])

So you see my original tensor was of shape (1,10,2) and I asked for an unfolding operation with parameters s.unfold(0, 1, 1).

Going by original understanding from the previous example, I assumed this means in the dimension 0, we take 1 chunk at a time with stride 1. Thus, as we have go into dimension 0, we see that we have only one chunk of size (10,2). So the output should have just taken this chunk and may be it should have just added a dimension to wrap this chunk and given me an output of size (1, 10, 2).

However, it gives me an output of size (1, 10, 2, 1). Why does it have an extra dimension at the last? Can someone elaborate intuitively please?


title: PyTorch tensors have same value after being added to a list

body:
While learning gradients and optimizing the process by Pytorch, I wanted to figure out the change of loss function values vs weights values with the graph. While I tried to graph, I used both numpy and torch, because I wanted to compare. During to store list of grad. and loss function values.
It works at numpy:
def gradient(x,y, y_predicted):
  return np.dot(2*x, y_predicted-y).mean()

dw = gradient(x,y,y_pred)
dw_list.append(dw)

[-120.0, -112.8, -106.032, -99.67008, -93.68988, -88.06848, -82.78437, -77.81731, -73.14827, -68.75938]
It doesn't works at torch:
  for epoch in range(n_iters):
    # prediction = forward pass
    y_pred = forward(x)

    # loss
    l = loss(y, y_pred)
    loss_list.append(l)
    print('loss')
    print(l_list)
    # gradients = backward pass
    l.backward()  # calculate w.grad = dl/dw   and cumulative

    # update weights
    with torch.no_grad():
        w -= learning_rate * w.grad
    print(f'w.grad before zero setting = {w.grad}')
    dw_list.append(w.grad)
    print(dw_list)
    #print(f'w.grad before zero setting = {w.grad}')
    # zero gradients
    w.grad.zero_()

[tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485), tensor(-6.9485)]
Why dw_list.append(dw) works at numpy, but dw_list.append(w.grad) does not work at torch?
Why only the new value of grad had been filled up the whole array at each iter. at torch tensor?


title: How to binarize images in PyTorch?

body:
I am trying to make a ROC curve for my Image Dataset, and for that, I need to binarize the dataset. The dataset that I am using is Retinal OCT image dataset. But I am lost when I am trying to binarize the dataset for making the ROC plot.

I am making a classifier using PyTorch for 4 classes using resnet and printing the Train and test loss along with Accuracy. But after I get to know about Confusion matrix that accuracy may not give us the proper result if we have more than two classes. Now I am calculating the Sensitivity(fpr) and Specificity(tpr) of the dataset and trying to print the ROC plot using these parameters. I am following this tutorial from Scikit-Learn to print this -
https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html.
There I saw that first I have to binarize the test data, but when I try to do this, getting an error every time.

I am doing the entire program in Google Colab and this is how I am connecting my drive -

from google.colab import drive
drive.mount('/content/drive')
data = "/content/drive/My Drive/AMD_new"

Now I saw this code in Scikit learn's website to binarize the data -

# Import some data to play with
iris = datasets.load_iris()
X = iris.data
y = iris.target

# Binarize the output
y = label_binarize(y, classes=[0, 1, 2])
n_classes = y.shape[1]

This is my dataset and dataloader -

# choose the training and test datasets
train_data = datasets.ImageFolder(data+"/train", transform=transform_train)
test_data = datasets.ImageFolder(data+"/val", transform = transform_test)

test_data = label_binarize(test_data, classes=[0, 1, 2, 3])

dataloader_train = torch.utils.data.DataLoader(train_data, batch_size=32, shuffle=True, num_workers=2)
dataloader_test = torch.utils.data.DataLoader(test_data, batch_size=32, num_workers=2)

When I try to binarize test_data, it is giving me this error -
OSError: cannot identify image file <_io.BufferedReader name='/content/drive/My Drive/AMD_new/val/CNV/CNV-6294785-1.jpeg'>

in this line test_data = label_binarize(test_data, classes=[0, 1, 2, 3])

Can any of you guys please guide me how can I binarize the dataset and am I going in the right direction or not in printing the ROC plot? I a pretty new to this Pytorch domain, that's why I wonder if I am going in the right way to get the ROC or not. Thanks.


title: How get a Python pathlib Path from an Azure blob datastore?

body:
I am trying to do some custom manipulation of a torch.utils.data.DataLoader in AzureML but cannot get it to instantiate directly from my azureml.core.Datastore :

ws = Workspace( # ... etc ... )
ds = Datastore.get(ws, datastore_name='my_ds')
am = ds.as_mount()

# HOW DO I GET base_path, data_file from am?

dataloader = DataLoader(
        ListDataset(base_path, data_file), #... etc...
    )

The value of am.path() is "$AZUREML_DATAREFERENCE_my_ds" but I cannot figure out how to go from that to a pathlib.Path as is expected by the constructor to ListDataset. Things I've tried include Path(am.path()) and Path(os.environ[am.path()]) but they don't seem to work.

It's clear that there's some answer, since :

script_params = {
    '--base_path': ds.as_mount(),
    '--epochs': 30,
    '--batch_size' : 16,
    '--use_cuda': 'true'
}
torch = PyTorch(source_directory='./',
                        script_params=script_params,
                        compute_target=compute_target,
                        entry_script='train.py',
                        pip_packages=packages,
                        use_gpu=True)

seems to create a legit object.


title: How to compute the Surface Dice-Sørensen Coefficient in pytorch?

body:
I would like to compute the Surface Dice-Sørensen Coefficient from this  paper (page 19)in python3/pytorch.

I have to point out, that I do not try to implement the simple standard volumetric Dice-Sørensen Coefficient! This one would look as follows in my implementation:

import torch

def volumetric_DSC(M1, M2):
    M1 = M1.view(-1)
    M2 = M2.view(-1)
    dividend = 2 * (M1 * M2).sum()
    divisor = (M1 * M1).sum() + (M2 * M2).sum()
    return dividend / divisor

if __name__ == "__main__":
    m1 = torch.empty(5, 5, 5).uniform_(0, 1)
    m1 = torch.bernoulli(m1)
    m2 = torch.empty(5, 5, 5).uniform_(0, 1)
    m2 = torch.bernoulli(m2)
    loss = volumetric_DSC(m1, m2)
    print("loss = {0}".format(loss))

How can I extend this code to a Surface Dice-Sørensen Coefficient loss?


title: LibTorch, using deeplab model gives segfault on forwarding

body:
I am trying to run images through the DeepLab model in Libtorch to segment them. Using pytorch, i am converting the Deeplabv3 model like this:

import torch
import torchvision
from torchvision import models

deeplap_model = models.segmentation.deeplabv3_resnet101(pretrained=True)
deeplap_model.eval()

class wrapper(torch.nn.Module):
def __init__(self, model):
    super(wrapper, self).__init__()
    self.model = model

def forward(self, input):
    results = []
    output = self.model(input)
    for k, v in output.items():
        results.append(v)
        return tuple(results)

model = wrapper(deeplap_model)

example = torch.rand(1, 3, 224, 224)
# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

traced_script_module.save("model.pt")

Now, in c++ with LibTorch, i am trying to load the model and run data through it. This, however, fails:

std::shared_ptr<torch::jit::script::Module> module = torch::jit::load("model.pt");
module->to(torch::kCUDA);
assert(module != nullptr);
std::cout << "ok\n";

std::vector<torch::jit::IValue> inputs;
cv::Mat image;
image = cv::imread("pic.jpeg", 1);
cv::Mat image_resized;
cv::resize(image, image_resized, cv::Size(224, 224));
cv::cvtColor(image_resized, image_resized, cv::COLOR_BGR2RGB);
cv::Mat image_resized_float;
image_resized.convertTo(image_resized_float, CV_32F, 1.0 / 255);

auto img_tensor = torch::from_blob(image_resized_float.data, {1, 224, 224, 3}, torch::kFloat32);
cout << "img tensor loaded..\n";
img_tensor = img_tensor.permute({0, 3, 1, 2});
img_tensor[0][0] = img_tensor[0][0].sub(0.485).div(0.229);
img_tensor[0][1] = img_tensor[0][1].sub(0.456).div(0.224);
img_tensor[0][2] = img_tensor[0][2].sub(0.406).div(0.225);

// to GPU
img_tensor = img_tensor.to(at::kCUDA);

torch::Tensor out_tensor2 = module->forward({img_tensor}).toTensor();  // SEGFAULT

Where am i going wrong here?


title: Add sum of squares of model parameters to loss function

body:
Adding sum of squares of model weights is known to be a regularization method that prevents overfitting of the deep learning models.
Currently I have this loss function for my model which is implemented in pytorch:

lossfunc = nn.NLLLoss(ignore_index=0)

Then, I compute the loss during training as follows:

 ...
loss = lossfunc(out_perm, dec_pp[:,1:])

I want the regularized loss to look like:

loss = lossfunc(out_perm, dec_pp[:,1:])+sum_square_wights(enc)+sum_square_wights(dec)

But don't know how to implement sum_square_wights function. Any help will be appreciated.


title: Pytorch gradients has not calculated

body:
I create a NN. I'm having a problem with recounting gradients. The problem is that I scalarly multiply 2 tensors u @ v and normalize one of them. It is important that gradients cannot be calculated for h. Therefore, I use detach(). In addition, during the recalculation of gradients, normalization should not be taken into account (I do not know how to do this).

import torch
from torch import nn

class Nn(nn.Module):
    def __init__(self):
        super(Nn, self).__init__()
        self.ln = nn.Linear(5, 5)

    def forward(self, x):
        v = self.ln(x)

        u = v.clone()
        h = v.clone()

        u /= u.norm()
        h = h.detach()
        h /= h.norm()

        res = torch.stack([torch.stack([u @ h, u @ h])])

        return res

def patches_generator():
    while True:
        decoder = torch.rand((5, ))
        target = torch.randint(2, (1,))
        yield decoder, target

net = Nn()

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters())

net.train()
torch.autograd.set_detect_anomaly(True)
for decoder, targets in patches_generator():
    optimizer.zero_grad()
    outputs = net(decoder)
    loss = criterion(outputs, targets)
    loss.backward()
    optimizer.step()

As a result, I get the following error:

  RuntimeError: one of the variables needed for gradient computation has
  been modified by an inplace operation: [torch.FloatTensor [9, 512, 1,
  1]], which is output 0 of ReluBackward1, is at version 3; expected
  version 2 instead. Hint: the backtrace further above shows the
  operation that failed to compute its gradient. The variable in
  question was changed in there or anywhere later. Good luck!


title: Translating LSTM model from Pytorch to Tensorflow

body:
I have an existing model in Pytorch and I am trying to translate it to TensorFlow. It is a neural network with two hidden LSTM layers and a multiple output because my goal is to do multi-class classification. My input is a time serie where each value represents a given class and I want to predict the next one. My time series has been converted to a vector by using a window of size 10.
The thing that is bugging me is that in pytorch my model gives awesome predictions but in tensorflow my accuracy drops and my loss function gets stuck.

I am pretty sure that the data I input into model is in the shape it needs so i am only going to post the model itself for now.

Pytorch code

class Model(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_keys):
        super(Model, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, num_keys)

    def forward(self, input):
        h0 = torch.zeros(self.num_layers, input.size(0), self.hidden_size).to(device)
        c0 = torch.zeros(self.num_layers, input.size(0), self.hidden_size).to(device)
        out, _ = self.lstm(input, (h0, c0))
        out = self.fc(out[:, -1, :])
        return out

TF code

modeltf = keras.models.Sequential([
  keras.layers.LSTM(hidden_size, return_sequences=True,
                         batch_input_shape=(batch_size, 10, 1)),
  keras.layers.LSTM(hidden_size, return_sequences=False),
  keras.layers.Dense(num_classes),
])

modeltf.compile(optimizer=keras.optimizers.Adam(),
              loss='sparse_categorical_crossentropy',
              metrics=['sparse_categorical_accuracy'])

If anyone can see the difference or has any recommendation I'll be grateful :)


title: Confusion regarding batch size while using DataLoader in pytorch

body:
I am new to pytorch.
I am training an ANN for classification on the MNIST dataset.

train_loader = DataLoader(train_data,batch_size=200,shuffle=True)

I am confused. The dataset is of 60,000 images and I have set batch size of 6000 and my model has 30 epochs.
Will every epoch see only 6000 images or will every epoch see 10 batches of 6000 images?


title: Error trying to build docker for flask app

body:
I'm trying to build a docker to run a flask app. I've never done this before. I have the flask app working locally. Here is my approach:
My directory structure for the project looks like this:
model.pkl  README.md          images/            static/
Dockerfile         flaskapp.py        requirements.txt   templates/

I can launch the flask app by running python flaskapp.py and it runs in my browser (locally).
I want to create a Docker so other machines can run this project without dealing with all the dependency stuff. To do so, I've done the following:

I created a Dockerfile with this inside:

FROM python:3
COPY requirements.txt /tmp
COPY flaskapp.py /tmp
COPY model.pkl /tmp
COPY images /tmp
COPY static /tmp
COPY templates /tmp
WORKDIR /tmp
ADD flaskapp.py /
RUN pip install -r requirements.txt
CMD [ "python", "flaskapp.py" ]

Ran the command docker build -t python-barcode .

That worked, so. I ran docker run python-barcode. The terminal printed out * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit), but it didn't work and I got this error on the browser:

This site can’t be reached0.0.0.0 refused to connect.
Try:

Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED

So I did some digging and I updated my Dockerfile to this (adding the last line):
FROM python:3
COPY requirements.txt /tmp
COPY flaskapp.py /tmp
COPY model.pkl /tmp
COPY images /tmp
COPY static /tmp
COPY templates /tmp
WORKDIR /tmp
ADD flaskapp.py /
RUN pip install -r requirements.txt
CMD [ "python", "flaskapp.py" ]
CMD ["flask", "run", "--host", "0.0.0.0" ]

Then running docker run python-barcode again, I get this error:

Usage: flask run [OPTIONS]

Error: Could not locate Flask application. You did not provide the FLASK_APP environment variable.

For more information see http://flask.pocoo.org/docs/latest/quickstart/

How should I proceed?
If its relevant, my flaskapp.py looks like this:
model = load_learner('', 'model.pkl')

app = Flask(__name__)
app.config['SEND_FILE_MAX_AGE_DEFAULT'] = 0

def classify(document):
    X = document
    y = model.predict(X)

    return y

class ReviewForm(Form):
    pred = TextAreaField('',[validators.DataRequired(),validators.length(min=1)])

@app.route('/')
def index():
    form = ReviewForm(request.form)
    return render_template('reviewform.html', form=form)

@app.route('/results', methods=['POST'])
def results():
    form = ReviewForm(request.form)
    if request.method == 'POST' and form.validate():
        sequence = request.form['pred']
        y = classify(sequence)
        return render_template('results.html',
        y = y)
    return render_template('reviewform.html', form=form)

if __name__ == '__main__':
    app.run(host= '0.0.0.0')

EDIT 1
Now I am getting this error:
[2020-07-03 00:29:51,222] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "flaskapp.py", line 94, in index
    return render_template('reviewform.html', form=form)
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 133, in render_template
    return _render(ctx.app.jinja_env.get_or_select_template(template_name_or_list),
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 930, in get_or_select_template
    return self.get_template(template_name_or_list, parent, globals)
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 883, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 857, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/local/lib/python3.8/site-packages/jinja2/loaders.py", line 115, in load
    source, filename, uptodate = self.get_source(environment, name)
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 57, in get_source
    return self._get_source_fast(environment, template)
  File "/usr/local/lib/python3.8/site-packages/flask/templating.py", line 85, in _get_source_fast
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: reviewform.html


title: Is there a PyTorch with CUDA Unified GPU-CPU Memory fork?

body: So Training a DNN model can be a pain when a batch of one image takes 15GB. Speed is not so important for me, yet to fit bigger batches (and models is). So I wonder if there is a PyTorch with CUDA Unified Memory fork or something like that to fit giant models (having 16gb per GPU RAM, yet 250 on CPU side it seems quite resonable)?


title: Convert image to tensor with range [0,255] instead of [0,1]?

body:
I'm trying to use transforms.compose to convert my images into normalized images with a range of [0,255] instead of normalizing it as [0,1] for training my model. How do I make my code do this. Currently it normalizes the images from [0,1]. How would i just multiply this up to 255 to make it 0-255 or is it not that simple?
def build_model(self):
        """ DataLoader """
        train_transform = transforms.Compose([
            transforms.RandomHorizontalFlip(),
            transforms.Resize((self.img_size + 30, self.img_size+30)),
            transforms.RandomCrop(self.img_size),
            transforms.ToTensor(),
            transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
        ])
        test_transform = transforms.Compose([
            transforms.Resize((self.img_size, self.img_size)),
            transforms.ToTensor(),
            transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
        ])


title: Hessian of a black box function that uses Pytorch

body:
First of all I am very new to Python and machine learning, so please excuse my ignorance on what might be a very basic issue; I do appreciate any input on this question!

I have a very complicated scalar-valued multivariable function implemented in Python that uses Pytorch functionalities (it is actually a composition of a neural network and operations that depend on outputs from the network), and I wish to find the gradient vector and Hessian matrix of this function at certain points. Besides numdifftools which does finite difference and is not very fast and accurate when the dimension of input is high, are there alternatives? Something that seems promising is the the torch.autograd which I believe is used to compute the gradient of neural networks, however, can it compute the gradient and Hessian of any black-box function that runs Pytorch code? Any input is appreciated!


title: How do I use Pytorch DataLoader to output small 2D chunks of a larger 2D grid of x and y values?

body:
I'm trying to output "chunks" of x and y values that are smaller parts of a larger grid.

So for example, I want a first batch to have x values from 0-250 and y values from 0-250. Then the next batch would be x = 0-250 and y = 250-500. The another batch would be x = 250-500 and y = 250-500. And lastly, x = 250-500 and y = 0-250.

So every call to grab a batch with the DataLoader class would grab a small 2D chunk of a larger 2D grid.

This is what I have so far..

class inputDataset(Dataset):

  def __init__(self):
    self.z =
    torch.Tensor([0.234,0.123,0.831,0.456,0.910,0.356,0.276,0.081])
    self.xmax = 1000
    self.xmin = 0
    self.ymax = 1000
    self.ymin = 0

    self.x = torch.linspace(self.xmin,self.xmax-1,self.xmax)
    self.y = torch.linspace(self.ymin,self.ymax-1,self.ymax)
    self.r = torch.sqrt(torch.mul(self.x,self.x) + /
    torch.mul(self.y,self.y))

  def __len__(self):
    return self.xmax**2 *  (len(self.z) + 3)

  def __getitem__(self, idx):

    out = /
    torch.cat((torch.cat((self.z,self.x[idx].unsqueeze(dim=0)/
    )),torch.cat((self.y[idx].unsqueeze(dim=0)/
    ,self.r[idx].unsqueeze(dim=0)))))

    return out

Right now I can get x and y to output but they are the same numbers and not chunks.


title: Is there a method to vectorize applying a function to every element of a Tensor

body:
I want to implement an update rule in an optimization procedure. I wanted to vectorize it using pytorch functions to improve time complexity. Possibly run on a GPU.

The update rule is like

u[i][j] = multiplier*NORM(Q.column(i)-Q.column(j))

I have tried to write a simple nested loop for its execution but it seems to be absymally slow.

def u_vector(Q,parameters):
    u=torch.zeros(Q.shape[0],Q.shape[0],requires_grad=False)
    for i in range(Q.shape[0]):
        for j in range(i+1,Q.shape[1]):
            u[i][j]=0.5*(torch.norm(Q[:,i]-Q[:,j]))*parameters['gamma']
            u[j][i]=u[i][j]
    return u

Can someone help please?


title: Save a Hash Value as a Tensor in pytorch

body:
I have a dataset that contains identifiers that are saved as string.

I want to create a neural net that gets amongst other things these identifiers as labels and then checks if two identifier are exactly the same. If they are the same then I want to increase the loss if the network predicts wrong values.

As an example an identifier looks like this ec2c1cc2410a4e259aa9c12756e1d6e

It's always 32 values and uses hexadecimal characters (0-9a-f).

I want to work with this value in pytorch and save it as a tensor but I get the following problem

decimal_identifier = int(string_id, 16)
tensor_id = torch.ToTensor(decimal_identifier)

RuntimeError: Overflow when unpacking long

So I can't convert the value into a decimal because the values are too big.
Any idea how I could fix this?
I know that it's always 32 chars but I haven't found a char tensor in pytorch.

How can I feed this unique identifier in my neural net?


title: Convert Conv2d to Linear

body:
saying that I have a conv2d layer, I know that its a special case of a Linear layer, how can I convert a Conv2d Layer to Linear? (Pytorch)
For example the following :
Conv2d(96, 1000, kernel_size=torch.Size([10, 10]), stride=(1, 1))


title: How do I use a ByteTensor in a contrastive cosine loss function?

body:
I'm trying to implement the loss function in http://anthology.aclweb.org/W16-1617 in PyTorch. It is shown as follows:

I've implemented the loss as follows:

class CosineContrastiveLoss(nn.Module):
    """
    Cosine contrastive loss function.
    Based on: http://anthology.aclweb.org/W16-1617
    Maintain 0 for match, 1 for not match.
    If they match, loss is 1/4(1-cos_sim)^2.
    If they don't, it's cos_sim^2 if cos_sim < margin or 0 otherwise.
    Margin in the paper is ~0.4.
    """

    def __init__(self, margin=0.4):
        super(CosineContrastiveLoss, self).__init__()
        self.margin = margin

    def forward(self, output1, output2, label):
        cos_sim = F.cosine_similarity(output1, output2)
        loss_cos_con = torch.mean((1-label) * torch.div(torch.pow((1.0-cos_sim), 2), 4) +
                                    (label) * torch.pow(cos_sim * torch.lt(cos_sim, self.margin), 2))
        return loss_cos_con

However, I'm getting an error saying:

TypeError: mul received an invalid combination of arguments - got (torch.cuda.ByteTensor), but expected one of:
 * (float value)
      didn't match because some of the arguments have invalid types: (torch.cuda.ByteTensor)
 * (torch.cuda.FloatTensor other)
      didn't match because some of the arguments have invalid types: (torch.cuda.ByteTensor)

I know that torch.lt() returns a ByteTensor, but if I try to coerce it to a FloatTensor with torch.Tensor.float() I get AttributeError: module 'torch.autograd.variable' has no attribute 'FloatTensor'.

I'm really not sure where to go from here. It seems logical to me to do an element-wise multiplication between the cosine similarity tensor and a tensor with 0 or 1 based on a less-than rule.


title: pytorch how to compute grad after clone a tensor

body:
My simple code:

import torch

x = torch.randn(4, requires_grad=True).cuda()
y = torch.randn(4, requires_grad=True).cuda()
z = torch.zeros(4)
z = torch.clone(x)
z.retain_grad()
h = (z + y) * z
l = torch.randn(4).cuda()
loss = (l - h).pow(2).sum()
loss.backward()
print('x.grad=', x.grad)
print('z.grad=', z.grad)

output:

x.grad= None
z.grad= tensor([-15.3401,  -3.2623,  -2.1670,   0.1410], device='cuda:0')

Why x.grad is None but not same as z.grad ?
What should I do if I want they are same?


title: More efficient way of implement this equation in pytorch (or Numpy)

body:
I'm implementing the analytical form of this function

where k(x,y) is a RBF kernel k(x,y) = exp(-||x-y||^2 / (2h))

My function prototype is

def A(X, Y, grad_log_px，Kxy):
   pass

and X, Y are NxD matrix where N is batch size and D is a dimension. So X is a batch of x with size N in the above equation grad_log_px is some NxD matrix I've computed using autograd.

Kxy is NxN matrix where each entry (i,j) is the RBF kernel K(X[i],Y[j])

The challenge here is that in the above equation, y is just a vector with dimension D. I kind of want to pass into a batch of y. (So to pass matrix Y with NxD size)

The equation is fine using loop through the batch size but I'm having trouble to implement in a more neat way

here is my attempted loop solution:

def A(X, Y, grad_log_px，Kxy):
   res = []
   for i in range(Y.shape[0]):
       temp = 0
       for j in range(X.shape[0]):
           # first term of equation
           temp += grad_log_px[j].reshape(D,1)@(Kxy[j,i] * (X[i] - Y[j]) / h).reshape(1,D)
           temp += Kxy[j,i] * np.identity(D) - ((X[i] - Y[j]) / h).reshape(D,1)@(Kxy[j,i] * (X[i] - Y[j]) / h).reshape(1,D) # second term of equation
       temp /= X.shape[0]

        res.append(temp)
    return np.asarray(res) # return NxDxD array

In the equation: grad_{x} and grad_{y} both dimension D


I tried to run this simple model. Basically this model is for text classification. I have used random word embeddings.

import torch.nn as nn
import torch.nn.functional as F

class DSC_RE(nn.Module):
    def __init__(self, input_dim, embedding_dim, output_dim):

        super().__init__()

        self.embedding = nn.Embedding(input_dim, embedding_dim)

        self.fc = nn.Linear(embedding_dim, output_dim)

    def forward(self, text):

        #text = [batch size, sent len]

        text=torch.t(text)

        #text = [sent len, batch size]

        embed = self.embedding(text)

        #embed = [sent len, batch size, embedding dim]

        embed = embed.permute(1,2,0)

        #embed = [batch size, embedding dim, sent len]

        output = F.max_pool1d(embed, embed.size(2))

        #output = [batch size, embedding dim,1]

        output = output.squeeze(2)

        #output = [batch size, embedding dim]

        return self.fc(output)

However, I got this error when I ran the code

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-d82622e3ea00> in <module>()
      1 n_epochs=20
----> 2 run_model(model,train_dataloader, test_dataloader,optimizer,criterion,n_epochs)

6 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1368     if input.dim() == 2 and bias is not None:
   1369         # fused op is marginally faster
-> 1370         ret = torch.addmm(bias, input, weight.t())
   1371     else:
   1372         output = input.matmul(weight.t())

RuntimeError: cublas runtime error : resource allocation failed at /pytorch/aten/src/THC/THCGeneral.cpp:216

Any help is really appreciated..............

I ran this code in Google Colab using Tesla K80 GPU and Pytorch 1.x version


Maybe I found something strange on pytorch, which result in property setter not working. Below is a minimal example that demonstrates this:

import torch.nn as nn

class A(nn.Module):
    def __init__(self):
        super(A, self).__init__()
        self.aa = 1
        self.oobj = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    @property
    def obj(self):
        print('get attr [obj]: {0}'.format(self.oobj))
        return self.oobj
    @obj.setter
    def obj(self, val):
        print('set attr [obj] to {0}'.format(val))
        self.oobj = val

class B(nn.Module):
    def get_attr(self):
        print('no any attr.')

class C:
    def get_attr(self):
        print('no any attr.')

b = A()             # set obj, and prints my setter message

b.obj               # get obj using my getter

# respectively run the following 3 lines, only the last line not call the setter I defined explicitly.
b.obj = C()         # set obj, and prints my setter message

# b.obj = [1, 2, 3]   # set obj, and prints my setter message

# b.obj = B()         # set obj, but it doesn't print my setter message

The last line doesn't call property setter I defined on class A, but call setter on torch.nn.Module. Because A
    regard B as a nn.Module, call the setter on nn.Module to set attr [obj] as a Module, but it still strange, why
    not call the setter I explicitly defined on class A?

And my project needs to set a nn.Module attribute via setter I defined explicitly, which causes BUG( because it failed). Now I change my code solved the BUG, but still puzzle with the problem.


title: Why do 'loss.backward()' and 'weight.grad' return a tensor containing all zeros?

body:
When I run 'loss.backward()' and 'weight.grad' I get a tensor containing all zeros. Also, 'weight.grad_fn' retruns NONE.

However, it all seems to return the correct result for the second layer 'w2'.
If I play with simple operations such as x*2 or x**2 'backward()' and '.grad' return correct results

Here's my code:

import torch
from torch import nn
import torch.nn.functional as F
from torchvision import datasets, transforms

# Getting MNIST data
num_workers = 0
batch_size = 64
transform = transforms.ToTensor()
train_data = datasets.MNIST(root='data', train=True, download=True, transform=transform)
train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, num_workers=num_workers)
dataiter = iter(train_loader)
images, labels = dataiter.next()

#####################################
#####################################
#### NN Part

def activation(x):
    return 1/(1+torch.exp(-x))

inputs = torch.from_numpy(images.view())
# Flatten the inputs format from (64,1,28,28) into (64,784)
inputs = inputs.reshape(images.shape[0], int(images.shape[1]*images.shape[2]*images.shape[3]))

w1 = torch.randn(784, 256, requires_grad=True)# n_input, n_hidden
b1 = torch.randn(256)# n_hidden

w2 = torch.randn(256, 10, requires_grad=True)# n_hidden, n_output
b2 = torch.randn(10)# n_output

h = activation(torch.mm(inputs, w1) + b1)
y = torch.mm(h, w2) + b2

#print(h)
#print(y)

y.sum().backward()
print(w1.grad)
print(w1.grad_fn)
#print(w2.grad)
#print(w2.grad_fn)

By the way it gives me the same problem if I try to run it this way also:

images = images.reshape(images.shape[0], -1)

model = nn.Sequential(nn.Linear(784, 128),
                      nn.ReLU(),
                      nn.Linear(128, 64),
                      nn.ReLU(),
                      nn.Linear(64, 10),
                      nn.LogSoftmax(dim=1))

logits = model(images)
criterion = nn.NLLLoss()

loss = criterion(logits, labels)
print(loss)
print(loss.grad_fn)

print('Before backward pass: ', model[0].weight.grad)
loss.backward()
print('After: ', model[0].weight.grad)
#print('After: ', model[2].weight.grad)
#print('After: ', model[4].weight.grad)


title: Deployment of a PyTorch computer vision deep learning model into a windows desktop app

body:
I have trained a yolov3 model in PyTorch with my dataset and I also have written some utility codes for it that run alongside the model all in python language. Now I want to deploy this model and my utilities into a windows desktop app that takes a video and runs the model on its frames. How can I do this task with minimum change in my code or rewrite it in another language? What framework is the best option for designing the UI of the app?
Thanks.


title: torch gather 3D tensor with 2D tensor as "gather map"

body:
For the given tensors of dimensions data.shape [B,N,F] and indices.shape [N,K],
Where K are the indices of the "neighborhood" of each point N, is there a simple method of gathering the neighbors such that output.shape [B,N,K,F] ?
data = [[[1,2,3],[2,3,4],[3,4,5]],[[3,4,5],[6,7,8],[2,3,4]] # Shape 2,3,3
indices = [[1,2],[1,0],[0,0]] # Shape 3,2

output = [
         [[[2,3,4],[3,4,5]],[[2,3,4],[1,2,3]],[[1,2,3],[1,2,3]]],
         [[[6,7,8],[2,3,4]],[[6,7,8],[3,4,5]],[[3,4,5],[3,4,5]]]
         ]

For example, The first point in each batch "relates" to points [1,2], Thus output[0][0] = [[2,3,4],[3,4,5]] Which are points 1,2 in the first batch of the input.
My attempt:
batched_indices = indices.unsqueeze(0).unsqueeze(-1).repeat(batch_size,1,1,feature_size)
data_neighs = data.unsqueeze(2).repeat(1,1,num_neighs,1)
output = torch.gather(data_neighs,batched_indices,dim=1)


title: How to check if PyTorch is using my GPU with CUDA?

body:
I tried to run all the computations on gpu, but the increase in the speed wasn't as big as i expected, also nvidia-smi gives this: nvidia-smi output, while program's running, so does pytorch "see" my gpu or not. If not, how can i fix this?
I've install CUDA toolkit v11.1 and cuDNN. Merged what was in cuDNN into CUDA toolkit directory.
# In[1]:

import torch
import torch.nn.functional as F
import torchvision
import numpy as np
import matplotlib.pyplot as plt
from timeit import default_timer as timer
from typing import Tuple, List, Type, Dict, Any

# In[2]:

torch.manual_seed(0)
np.random.seed(0)

# In[3]:

class Perceptron(torch.nn.Module):

    def __init__(self,
                 input_resolution: Tuple[int, int] = (28, 28),
                 input_channels: int = 1,
                 hidden_layer_features: List[int] = [256, 256, 256],
                 activation: Type[torch.nn.Module] = torch.nn.Tanh,
                 num_classes: int = 10):

        super().__init__()

        self.input_resolution = input_resolution
        self.input_channels = input_channels
        self.hidden_layer_features = hidden_layer_features
        self.activation = activation()
        self.num_classes = num_classes

        # Layers:
        self.fc1 = torch.nn.Linear(self.input_resolution[0] * self.input_resolution[1], 128)
        self.fc2 = torch.nn.Linear(128, self.num_classes)

    def forward(self, X : Type[torch.Tensor]) -> Type[torch.Tensor]:

        X = self.fc1(X)
        X = self.activation(X)

        X = self.fc2(X)
        output = F.softmax(X, dim = 1)

        return output

# In[4]:

if torch.cuda.is_available():
    device = torch.device("cuda:0")
else:
    device = torch.device('cpu')

# In[5]:

model = Perceptron().to(device)
print(model)
print('Total number of trainable parameters',
      sum(p.numel() for p in model.parameters() if p.requires_grad))

# In[6]:

train_transforms = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])
valid_transforms = torchvision.transforms.Compose([torchvision.transforms.ToTensor()])

# In[7]:

train_dataset = torchvision.datasets.MNIST(root='./mnist',
                                           train=True,
                                           download=True,
                                           transform=train_transforms)

valid_dataset = torchvision.datasets.MNIST(root='./mnist',
                                         train=False,
                                         download=True,
                                         transform=valid_transforms)

# In[8]:

indices = np.random.randint(0, len(train_dataset), size=256)

fig, axes = plt.subplots(nrows=8, ncols=8, figsize=(10, 10))
for i, row in enumerate(axes):
    for j, ax in enumerate(row):
        sample_index = indices[i*16+j]
        sample, label = train_dataset[sample_index]
        ax.imshow(sample.cpu().numpy().transpose(1, 2, 0))
        ax.set_title(label)

# In[9]:

def train_single_epoch(model: torch.nn.Module,
                       optimizer: torch.optim.Optimizer,
                       loss_function: torch.nn.Module,
                       data_loader: torch.utils.data.DataLoader):

    for data in data_loader:

        X, y = data
        X, y = X.to(device), y.to(device)
        model.zero_grad()
        output = model(X.view(-1, 784))
        loss = loss_function(output, y)
        loss.backward()
        optimizer.step()

# In[10]:

def validate_single_epoch(model: torch.nn.Module,
                          loss_function: torch.nn.Module,
                          data_loader: torch.utils.data.DataLoader):

    loss_total = 0
    accuracy_total = 0

    for data in data_loader:

        X, y = data
        X, y = X.to(device), y.to(device)
        output = model(X.view(-1, 784))
        loss = loss_function(output, y)
        loss_total += loss

        y_pred = output.argmax(dim = 1, keepdim=True).to(device)
        accuracy_total += y_pred.eq(y.view_as(y_pred)).sum().item()

    loss_avg = loss_total / len(data_loader.dataset)
    accuracy_avg = 100.0 * accuracy_total / len(data_loader.dataset)

    return {'loss' : loss_avg, 'accuracy' : accuracy_avg}

# In[11]:

def train_model(model: torch.nn.Module,
                train_dataset: torch.utils.data.Dataset,
                valid_dataset: torch.utils.data.Dataset,
                loss_function: torch.nn.Module = torch.nn.CrossEntropyLoss(),
                optimizer_class: Type[torch.optim.Optimizer] = torch.optim,
                optimizer_params: Dict = {},
                initial_lr = 0.01,
                lr_scheduler_class: Any = torch.optim.lr_scheduler.ReduceLROnPlateau,
                lr_scheduler_params: Dict = {},
                batch_size = 64,
                max_epochs = 1000,
                early_stopping_patience = 20):

    optimizer = torch.optim.Adam(model.parameters(), lr=initial_lr, **optimizer_params)
    lr_scheduler = lr_scheduler_class(optimizer, **lr_scheduler_params)

    train_loader = torch.utils.data.DataLoader(train_dataset, shuffle=True, batch_size=batch_size)
    valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=batch_size)

    best_valid_loss = None
    best_epoch = None

    for epoch in range(max_epochs):

        print(f'Epoch {epoch}')
        start = timer()
        train_single_epoch(model, optimizer, loss_function, train_loader)
        valid_metrics = validate_single_epoch(model, loss_function, valid_loader)
        print('time:', timer() - start)
        print(f'Validation metrics: \n{valid_metrics}')

        lr_scheduler.step(valid_metrics['loss'])

        if best_valid_loss is None or best_valid_loss > valid_metrics['loss']:
            print(f'Best model yet, saving')
            best_valid_loss = valid_metrics['loss']
            best_epoch = epoch
            torch.save(model, './best_model.pth')

        if epoch - best_epoch > early_stopping_patience:
            print('Early stopping triggered')
            return

# In[ ]:

train_model(model,
            train_dataset=train_dataset,
            valid_dataset=valid_dataset,
            loss_function=torch.nn.CrossEntropyLoss(),
            initial_lr=0.01, max_epochs = 500)


title: how to load one type of image in cifar10 or stl10 with pytorch

body:
This is a very simple question, I'm just trying to select a specific class of images (eg "car") from a standard pytorch image dataset. At the moment the data loader looks like this:
def cycle(iterable):
    while True:
        for x in iterable:
            yield x

train_loader = torch.utils.data.DataLoader(
    torchvision.datasets.STL10('drive/My Drive/training/stl10', split='train+unlabeled', transform=torchvision.transforms.Compose([
        torchvision.transforms.ToTensor(),
    ])),
shuffle=True, batch_size=8)
train_iterator = iter(cycle(train_loader))
class_names = ['airplane', 'bird', 'car', 'cat', 'deer', 'dog', 'horse', 'monkey', 'ship', 'truck']

train_iterator = iter(cycle(train_loader))

The iterator returns a batch of shuffled images of all types, but I would like to be able to select what types of images are returned, eg. just images of deer, or ships


title: torchvision MNIST dataset download issue

body:
I am trying to get the MNIST dataset from torchvision package:
import torch
import torchvision
from torchvision import datasets, transforms
train = datasets.MNIST('', train=True, download=True,
                       transform=transforms.Compose([
                           transforms.ToTensor()
                       ]))

While running this code, I am getting the following error:
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to MNIST\raw\train-images-idx3-ubyte.gz

---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-3-f99c3ec55e27> in <module>
----> 1 train = datasets.MNIST('', train=True, download=True,
      2                        transform=transforms.Compose([
      3                            transforms.ToTensor()
      4                        ]))

C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets\mnist.py in __init__(self, root, train, transform, target_transform, download)
     77
     78         if download:
---> 79             self.download()
     80
     81         if not self._check_exists():

C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets\mnist.py in download(self)
    144         for url, md5 in self.resources:
    145             filename = url.rpartition('/')[2]
--> 146             download_and_extract_archive(url, download_root=self.raw_folder, filename=filename, md5=md5)
    147
    148         # process and save as torch files

C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_and_extract_archive(url, download_root, extract_root, filename, md5, remove_finished)
    312         filename = os.path.basename(url)
    313
--> 314     download_url(url, download_root, filename, md5)
    315
    316     archive = os.path.join(download_root, filename)

C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_url(url, root, filename, md5, max_redirect_hops)
    138             _urlretrieve(url, fpath)
    139         else:
--> 140             raise e
    141     # check integrity of downloaded file
    142     if not check_integrity(fpath, md5):

C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in download_url(url, root, filename, md5, max_redirect_hops)
    130     try:
    131         print('Downloading ' + url + ' to ' + fpath)
--> 132         _urlretrieve(url, fpath)
    133     except (urllib.error.URLError, IOError) as e:  # type: ignore[attr-defined]
    134         if url[:5] == 'https':

C:\ProgramData\Anaconda3\lib\site-packages\torchvision\datasets\utils.py in _urlretrieve(url, filename, chunk_size)
     27 def _urlretrieve(url: str, filename: str, chunk_size: int = 1024) -> None:
     28     with open(filename, "wb") as fh:
---> 29         with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": USER_AGENT})) as response:
     30             with tqdm(total=response.length) as pbar:
     31                 for chunk in iter(lambda: response.read(chunk_size), ""):

C:\ProgramData\Anaconda3\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    220     else:
    221         opener = _opener
--> 222     return opener.open(url, data, timeout)
    223
    224 def install_opener(opener):

C:\ProgramData\Anaconda3\lib\urllib\request.py in open(self, fullurl, data, timeout)
    529         for processor in self.process_response.get(protocol, []):
    530             meth = getattr(processor, meth_name)
--> 531             response = meth(req, response)
    532
    533         return response

C:\ProgramData\Anaconda3\lib\urllib\request.py in http_response(self, request, response)
    638         # request was successfully received, understood, and accepted.
    639         if not (200 <= code < 300):
--> 640             response = self.parent.error(
    641                 'http', request, response, code, msg, hdrs)
    642

C:\ProgramData\Anaconda3\lib\urllib\request.py in error(self, proto, *args)
    567         if http_err:
    568             args = (dict, 'default', 'http_error_default') + orig_args
--> 569             return self._call_chain(*args)
    570
    571 # XXX probably also want an abstract factory that knows when it makes

C:\ProgramData\Anaconda3\lib\urllib\request.py in _call_chain(self, chain, kind, meth_name, *args)
    500         for handler in handlers:
    501             func = getattr(handler, meth_name)
--> 502             result = func(*args)
    503             if result is not None:
    504                 return result

C:\ProgramData\Anaconda3\lib\urllib\request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650
    651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 503: Service Unavailable

I would like to know why this error is happening and what I need to do to solve this issue.
Really appreciate all the help.
Regards,
Arpan


title: Pytorch's DataParallel does not split data memory, instead seems to copy it

body:
I am running a batched LSTM model (batch size of 32) on PyTorch with 8xV100 GPUs, 16GB memory each GPU.

I have an LSTM model with the following structure:

# Class containing the LSTM model initialization and feed-forward logic
class LSTMClassifier(nn.Module):
    # LSTM initialization
    def __init__(self, embedding_dim, hidden_dim, vocab_size, label_size, static_size):
        super(LSTMClassifier, self).__init__()

        # Setting the hidden layer dimension of the LSTM
        self.hidden_dim = hidden_dim
        # Initializing the embedding layer
        self.embeddings = nn.Embedding(vocab_size, embedding_dim-2)
        # Initializing the LSTM layer with one hidden layer
        self.lstm = nn.LSTM(((embedding_dim*vocab_size)+static_size), hidden_dim, num_layers = 1, batch_first=True)
        # Initializing linear linear that takes the hidden layer output
        self.hidden2label = nn.Linear(hidden_dim, label_size)

    # Defining the hidden state of the LSTM
    def init_hidden(self, batch_size):
        # the first is the hidden h
        # the second is the cell  c
        return [autograd.Variable(torch.zeros(batch_size, 1, self.hidden_dim).cuda()),
                autograd.Variable(torch.zeros(batch_size, 1, self.hidden_dim).cuda())]

    # Defining the feed forward logic of the LSTM. It contains:
    # 1. The embedding layer
    # 2. The LSTM layer with one hidden layer
    # 3. The softmax layer
    def forward(self, seq, freq, time, static):
        print(seq.size())

        # Grab the mini-batch length and max sequence length (pre-ordered)
        # (need to do this in the forward logic because of data parallelism and how the GPU's will split up the batch)
        sequence_length = seq.size()[1]
        batch_length = seq.size()[0]

        # reset the LSTM hidden state.
        # Must be done before you run a new batch. Otherwise the LSTM will treat a new batch as a continuation of a sequence
        self.hidden = self.init_hidden(batch_length)

        # Permute the cell and hidden layers. This is because when using Batch_first = True on data parallel,
        # the hidden state will still expect an input of (nLayer, batch size, hidden dim), but we are feeding it (batch size, nLayer, hidden dim)
        # Thus, to fix it, we need to swap the first and 2nd inputs before feeding to hidden dim
        self.hidden[0] = self.hidden[0].permute(1, 0, 2).contiguous()
        self.hidden[1] = self.hidden[1].permute(1, 0, 2).contiguous()

        # This is the pass to the embedding layer.
        # The sequence is of dimension N and the output is N x Demb
        embeds = self.embeddings(seq)

        # Concatenate the embedding output with the time and frequency vectors
        embeds = torch.cat((embeds,freq), dim=3)
        embeds = torch.cat((embeds,time), dim=3)

        # Flatten the tensor
        x = embeds.view(batch_length, sequence_length, -1)

        # Concatenate the static information
        x = torch.cat((x, static), dim=2)

        # Grab the list of lengths of sequences, for the purpose of packing the padded sequenes
        seq_lengths = torch.LongTensor(list(map(len, seq)))

        # pack the padded sequence so that paddings are ignored
        packed_x = torch.nn.utils.rnn.pack_padded_sequence(x, seq_lengths, batch_first=True)

        # Feed to the LSTM layer
        self.lstm.flatten_parameters()
        lstm_out, self.hidden = self.lstm(packed_x, self.hidden)

        # Swap back the 1st and 2nd inputs to the hidden layer back to its original configuration
        self.hidden = list(self.hidden)
        self.hidden[0] = self.hidden[0].permute(1, 0, 2).contiguous()
        self.hidden[1] = self.hidden[1].permute(1, 0, 2).contiguous()

        # Unpack the packed padded sequence so that it is ready for prediction
        unpacked_lstm_out, input_sizes = torch.nn.utils.rnn.pad_packed_sequence(lstm_out, batch_first=True)

        # Feed the last layer of the LSTM into the linear layer
        y = self.hidden2label(unpacked_lstm_out[:,-1,:])

        # Produce the softmax probabilities
        log_probs = F.log_softmax(y)

        return log_probs

And I just run one iteration of a batched dataset (batch size, is 32):

torch.cuda.empty_cache()
EMBEDDING_DIM = 32
HIDDEN_DIM = 50
EPOCH = 10
BATCH_SIZE = 16
best_val_auc = 0.0

model = LSTMClassifier(embedding_dim=EMBEDDING_DIM, hidden_dim=HIDDEN_DIM, vocab_size=len(events_to_ix), label_size=len(targets_to_ix), static_size=(len(gender_to_ix)+1)).cuda()
model = torch.nn.DataParallel(model).cuda()

loss_function = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

no_up = 0

model.train()
avg_loss = 0.0
count = 0
truth_res = []
pred_res = []

g= train_data.groupby(np.arange(len(train_data)) // 32)

rows = g.get_group((list(g.groups)[0])) # Batch size of 32

torch.cuda.empty_cache()
# Grab the targets into a list and append it into the truth_res list in order to measure AUC performance
target = [targets_to_ix[target] for target in rows['event_target']]
truth_res.extend(target)

# Encode the data and output to tensors (based on the previous description)
seq, freq, time_data, static = encode_data(rows, events_to_ix)
print(len(seq))

# Pad the sequences
seq = rnn_utils.pad_sequence(seq, batch_first = True)
freq = rnn_utils.pad_sequence(freq, batch_first = True)
time_data = rnn_utils.pad_sequence(time_data, batch_first = True)
static = rnn_utils.pad_sequence(static, batch_first = True)

# Put the padded sequences into Variable and Cuda cores
seq = autograd.Variable(seq).cuda()
freq = autograd.Variable(freq).cuda()
time_data = autograd.Variable(time_data).cuda()
static = autograd.Variable(static).cuda()
target = autograd.Variable(torch.LongTensor(target)).cuda()

# Feed the tensor Variables into the model
pred = model(seq,freq,time_data,static)

This is my nvidia-smi output right before loading to the model:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:17.0 Off |                    0 |
| N/A   48C    P0    64W / 300W |   2438MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:18.0 Off |                    0 |
| N/A   42C    P0    44W / 300W |     11MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:19.0 Off |                    0 |
| N/A   41C    P0    46W / 300W |     11MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:00:1A.0 Off |                    0 |
| N/A   44C    P0    44W / 300W |     11MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 00000000:00:1B.0 Off |                    0 |
| N/A   44C    P0    42W / 300W |     11MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 00000000:00:1C.0 Off |                    0 |
| N/A   44C    P0    45W / 300W |     11MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 00000000:00:1D.0 Off |                    0 |
| N/A   42C    P0    44W / 300W |     11MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   45C    P0    47W / 300W |     11MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     93165      C   ...r/anaconda3/envs/pytorch_p36/bin/python  2427MiB |
+-----------------------------------------------------------------------------+

Right after loading the model, I call nvidia-smi in the start of the forward pass and the nvidia-smi output is:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104      Driver Version: 410.104      CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  On   | 00000000:00:17.0 Off |                    0 |
| N/A   50C    P0    59W / 300W |   3040MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  On   | 00000000:00:18.0 Off |                    0 |
| N/A   44C    P0    58W / 300W |   1342MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  On   | 00000000:00:19.0 Off |                    0 |
| N/A   44C    P0    62W / 300W |   1342MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  On   | 00000000:00:1A.0 Off |                    0 |
| N/A   47C    P0    57W / 300W |   1342MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  On   | 00000000:00:1B.0 Off |                    0 |
| N/A   47C    P0    57W / 300W |   1342MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  On   | 00000000:00:1C.0 Off |                    0 |
| N/A   47C    P0    59W / 300W |   1342MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  On   | 00000000:00:1D.0 Off |                    0 |
| N/A   44C    P0    59W / 300W |   1342MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  On   | 00000000:00:1E.0 Off |                    0 |
| N/A   48C    P0    62W / 300W |   1342MiB / 16130MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

I would expect the memory across 8 GPUs to be more uniform and much smaller, seeing how the batched data I passed in only took around ~2GB of space. Is there something I am doing wrong?


title: How to parallelize CNN evaluation of images with PyTorch

body:
I have a fairly long script that uses a CNN to classify people from drone footage as either human or non-human. The general flow is as such: (1) Create a video object and extract "captures" from it based on intervals of seconds specified. (2) Instantiate the Model class, loading a PyTorch CNN. (3) For each capture, break into much smaller (and overlapping) images which the CNN will classify as human or non-human. (4) Create a list of coordinates, which satisfy previous step. (5) Draw red squares surrounding coordinates, and save marked up image. (6) Repeat process for every capture.

There's a bottleneck at the image level. Crops are being produced and evaluated by the CNN sequentially. I would love to parallelize this process but it goes beyond my current knowledge/experience level.

Any recommendations? The script is below for reference.

import torch, torchvision
from torchvision import datasets, models, transforms
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
import time
from torchsummary import summary

import numpy as np
import matplotlib.pyplot as plt
import os

from PIL import Image
import shutil
from PIL import Image, ImageDraw
import random
import cv2

class Model():

    def __init__(self,model):

        self.idx_to_class = {1:'No human',0:'Human'}
        self.image_transforms = {
            'test': transforms.Compose([
                transforms.Resize(size=256),
                transforms.CenterCrop(size=224),
                transforms.ToTensor(),
                transforms.Normalize([0.485, 0.456, 0.406],
                                     [0.229, 0.224, 0.225])
            ])
        }
        self.model = torch.load(model)

    def predict(self, test_image_name):
        transform = self.image_transforms['test']
#         test_image = Image.open(test_image_name)
        test_image_tensor = transform(test_image_name)

        if torch.cuda.is_available():
            test_image_tensor = test_image_tensor.view(1, 3, 224, 224).cuda()
        else:
            test_image_tensor = test_image_tensor.view(1, 3, 224, 224)

        with torch.no_grad():
            self.model.eval()
            out = self.model(test_image_tensor)
            ps = torch.exp(out)
            topk, topclass = ps.topk(1, dim=1)
            return topclass.cpu().numpy()[0][0]

class Image_classifier():

    def __init__(self,image,folder,positive_location,model):
        self.name = image
        self.alias = image.split('.')[0]
        self.folder = folder
        self.src = Image.open(f"{self.folder}/{self.name}")
        self.width = self.src.size[0]
        self.square_size = int(self.width/25)
        self.max_down = int(self.src.height/self.square_size) * self.square_size - self.square_size
        self.max_right = int(self.src.width/self.square_size) * self.square_size - self.square_size
        self.offset = int(self.square_size/3)
        self.positive_location = positive_location
        self.model = model

    def window_coordinates(self):

        def right_pass(y):
            x_coords = [x for x in range(0,self.max_right,self.offset)]
            y_coords = [y for x in range(0,self.max_right,self.offset)]
            return [(x,y,x+self.square_size,y+self.square_size) for x,y in zip(x_coords,y_coords)]

        #v_pass = np.vectorize(right_pass)

        y_values = [y for y in range(0,self.max_down,self.offset)]
        coordinates = [right_pass(y) for y in y_values]
        self.coordinates = [item for sublist in coordinates for item in sublist]

    def predict_coord(self,coord):
        sample = self.src.crop(coord)
        return self.model.predict(sample)

    def parse_coordinates(self):

        new_coords = [coord if self.predict_coord(coord) == 0 else 0 for coord in self.coordinates]
        while 0 in new_coords:
            new_coords.remove(0)
        self.coordinates = new_coords

    def select_squares(self):
        self.window_coordinates()
        self.parse_coordinates()
        self.drawable = ImageDraw.Draw(self.src)
        for coord in self.coordinates:
            self.drawable.rectangle(list(coord), fill = None, outline = 'red')
        self.src.save(f"{self.positive_location}/{self.alias}.jpg")

class Video_classifier():
    def __init__(self,video,root,seconds):
        self.video = video
        self.alias = self.video.split('.')[0]
        self.root = root
        self.seconds = seconds
        self.model = Model(model='/home/team4/output/_model_142.pt')

        self.folder = f"{self.root}/{self.alias}"
        if os.path.exists(self.folder):
            shutil.rmtree(self.folder)
        os.mkdir(self.folder)

        self.positive_location = f"{self.root}/{self.alias}/positives"
        if os.path.exists(self.positive_location):
            shutil.rmtree(self.positive_location)
        os.mkdir(self.positive_location)

    def get_frames(self):
        import cv2
        cam = cv2.VideoCapture(f"{self.root}/{self.video}")

        (major_ver, minor_ver, subminor_ver) = (cv2.__version__).split('.')
        if int(major_ver)  < 3 :
            fps = round(video.get(cv2.cv.CV_CAP_PROP_FPS))
        else:
            fps = round(cam.get(cv2.CAP_PROP_FPS))

        current_frame = 0
        while(True):
            ret,frame = cam.read()
            if ret:
                if current_frame % (self.seconds*fps) == 0:
                    f_name = f"{self.alias}_{current_frame}.jpg"
                    cv2.imwrite(f_name, frame)
                    shutil.move(f_name, self.folder)

                current_frame += 1
            else:
                break

        cam.release()
        cv2.destroyAllWindows()

    def read_dir(self):
        self.files = [f for f in os.listdir(self.folder) if os.path.isfile(os.path.join(self.folder, f))]

    def classify_frames(self):
        self.get_frames()
        self.read_dir()
        for file in self.files:
            image = Image_classifier(image=file, folder=self.folder, positive_location=self.positive_location,
                                     model=self.model)
            image.select_squares()

test = Video_classifier(video='refugee_test.mp4',root='/home/team4/Untitled Folder 1', seconds=10)
test.classify_frames()

I apologize for the lack of comments, doc-strings, etc. It's the in the works.


title: How to perform pixelwise normalization on an image

body:
I am trying to perform pixel-wise normalization, but I dont understand the concept or how to implement it.

I have the MNIST Fashion data set, I have performed mean normalization on all the 1024 images.

# Assuming all images are the same size, get dimensions of the first image
w = 28
h = 28

N=len(data)
print(N)

print('data shape',data.shape)

# Create a numpy array of floats to store the average (assume RGB images)
arr=np.zeros((1,h,w),np.float)

print('arr size:', arr.shape)
# print(arr)

# Build up average pixel intensities, casting each image as an array of floats
for im in data:
    #print(type(im))
    imarr = np.array(np.mean(im.asnumpy(), axis=(0)), dtype=np.float)
    arr= arr + imarr / N

# Round values in array and cast as 8-bit integer
arr=np.array(np.round(arr),dtype=np.uint8)

I need to subtract a sample image from the dataset from the arr


title: Fastbert: BertDataBunch error for multilabel text classification

body:
I'm following the FastBert tutorial from huggingface https://medium.com/huggingface/introducing-fastbert-a-simple-deep-learning-library-for-bert-models-89ff763ad384

The problem is this the code is not exactly reproducible. The main issue I'm facing is the dataset preparation. In the tutorial, this dataset is used https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/data

But, if I set-up the folder structure according the tutorial, and place the dataset files in the folders I get errors with the databunch.

databunch = BertDataBunch(args['data_dir'], LABEL_PATH, args.model_name, train_file='train.csv', val_file='val.csv',
                          test_data='test.csv',
                          text_col="comment_text", label_col=label_cols,
                          batch_size_per_gpu=args['train_batch_size'], max_seq_length=args['max_seq_length'],
                          multi_gpu=args.multi_gpu, multi_label=True, model_type=args.model_type)

It complains about the file format being wrong. How should I format the dataset, labels for this dataset with fastbert?


title: How to convert darknet image classification weights file to pytorch pt?

body:
I created a model of darknet53.weights for image classification using my original data in darknet.
(This isn't a YOLO v3 model.)
Is there a way to convert a darknet53.weight to a pytorch pt model?
I tried quoting various codes on github etc., but all of them can convert only YOLOv3 weights file to pytorch's pt model.
I want to compare the accuracy of the darknet53 model created with darknet with other image classification models created with pytorch.
Initially, I tried to make a darknet53 model with pytorch, but that didn't work. Therefore, I created a darknet53 model with darknet.
If anyone knows a good way, please teach me.
Thanks.


title: Which PyTorch modules are affected by model.eval() and model.train()?

body:
The model.eval() method modifies certain modules (layers) which are required to behave differently during training and inference. Some examples are listed in the docs:

This has [an] effect only on certain modules. See documentations of particular modules for details of their behaviors in training/evaluation mode, if they are affected, e.g. Dropout, BatchNorm, etc.

Is there an exhaustive list of which modules are affected?


title: What is a good way to build/execute for-loop Tensorflow graph in parallel?

body:
y = []
for i in range(9999):
  x = ... # build/get sub graph
  y.append(x)
y = tf.stack(x)

I am trying to build a Tensorflow Graph that contains multiple separate sub-graph/inputs (independently each other) that I am using a python for-loop to achieve this. (assuming the for-loop can not achieve by tensor operation).
I figure out this part is really slow and I think it is because the python for-loop is not in parallel (process them one by one). I wonder is there solution I can make it parallel (because each sub-graph/input is independent each other and separate)? I wonder change to tf.while_loop can be helpful or any other suggestions?


title: cannot list parameters for pytorch network

body:
I am trying to build a nest unet with the following structure:

class EFUnet(nn.Module):

    backbone = efn.EfficientNetB3(
        weights=None,
        include_top=False,
        input_shape=(256,1600,3)
    )

    backbone.load_weights(('../input/efficientnet-keras-weights-b0b5/'
                           'efficientnet-b3_imagenet_1000_notop.h5'))

    def __init(self,in_channel = 3, out_channels = 1, features = 32):
        super(EFUnet, self).__init__()

        self.conv00 = backbone.input
        self.conv10 = backbone.get_layer('stem_activation').output
        self.conv20 = backbone.get_layer('block2c_add').output
        self.conv30 = backbone.get_layer('block3c_add').output
        self.conv40 = backbone.get_layer('block5e_add').output
        self.conv50 = backbone.get_layer('block7b_add').output

        self.conv01 = _H(features*(2^0+2^1), features*2^0)
        self.conv11 = _H(feathres*(2^1+2^2),feature*2^1)
        self.conv21 = _H(feathres*(2^2+2^3),feature*2^2)
        self.conv31 = _H(feathres*(2^3+2^4),feature*2^3)
        self.conv41 = _H(feathres*(2^4+2^5),feature*2^4)

        self.conv02 = _H(features*(2^0*2+2^1), features*2^0)
        self.conv12 = _H(features*(2^1*2+2^2), features*2^1)
        self.conv22 = _H(features*(2^2*2+2^3), features*2^2)
        self.conv32 = _H(features*(2^3*2+2^4), features*2^3)

        self.conv03 = _H(features*(2^0*3+2^1), features*2^0)
        self.conv13 = _H(features*(2^1*3+2^2), features*2^1)
        self.conv23 = _H(features*(2^2*3+2^3), features*2^2)

        self.conv04 = _H(features*(2^0*4+2^1), features*2^0)
        self.conv14 = _H(features*(2^1*4+2^2), features*2^1)

        self.conv05 = _H(features*(2^0*5+2^1), features*2^0)

        self.final1 = nn.Conv2d(features, 1, kernel_size=1)
        self.final2 = nn.Conv2d(features, 1, kernel_size=1)
        self.final3 = nn.Conv2d(features, 1, kernel_size=1)
        self.final4 = nn.Conv2d(features, 1, kernel_size=1)
        self.final5 = nn.Conv2d(features, 1, kernel_size=1)

        self.final = nn.Conv2d(feature*5, 4, (3,3), padding="same", activation="sigmoid")

    def forward(self,input):

        x00 = self.conv00(input)

        x10 = self.conv10(x00)
        x01 = self.conv01(torch.cat([x00,self._U(x10)],1))

        x20 = self.conv20(x10)
        x11 = self.conv11(torch.cat([x10,self._U(x20)],1))
        x02 = self.conv02(torch.cat([x00,x01,self._U(x11)],1))

        x30 = self.conv30(x20)
        x21 = self.conv21(torch.cat([x20,self._U(x30)],1))
        x12 = self.conv12(torch.cat([x10,x11,self._U(x21)],1))
        x03 = self.conv03(torch.cat([x00,x01,x02,self._U(x12)],1))

        x40 = self.conv40(x30)
        x31 = self.conv31(torch.cat([x30,self._U(x40)],1))
        x22 = self.conv22(torch.cat([x20,x21,self._U(x31)],1))
        x13 = self.conv13(torch.cat([x10,x11,x12,self._U(x22)],1))
        x04 = self.conv04(torch.cat([x00,x01,x02,x03,self._U(x13)],1))

        x50 = self.conv50(x40)
        x41 = self.conv41(torch.cat([x40,self._U(x50)],1))
        x32 = self.conv32(torch.cat([x30,x31,self._U(x41)],1))
        x23 = self.conv23(torch.cat([x20,x21,x22,self._U(x32)],1))
        x14 = self.conv14(torch.cat([x10,x11,x12,x13,self._U(x23)],1))
        x05 = self.conv05(torch.cat([x00,x01,x02,x03,x04,self._U(x14)],1))

        output1 = self.final1(x01)
        output2 = self.final2(x02)
        output3 = self.final3(x03)
        output4 = self.final4(x04)
        output5 = self.final4(x05)

        x_out = torch.cat([output1, output2, output3, output4, output5],1)
        x_out = self.final(x_out)

        return x_out

    def _H(in_channels, features, use_gn=True):
        if use_gn:
            norm = torch.nn.GroupNorm(num_channels = 3, num_groups=1)
        else:
            norm = BatchNormalization(number_features = features)

        return nn.Sequential(
            OrderedDict(
                [
                    (name + "conv", nn.Conv2D(in_channels, features, (2, 2), padding='same')),
                    (name + "norm", norm()),
                    (name + 'LReLU',LeakyReLU(alpha=0.1))
                ]
            )
        )

    def _U(in_channels, features, use_gn=True):
        if use_gn:
            norm = torch.nn.GroupNorm(num_channels = 3, num_groups=1)
        else:
            norm = BatchNormalization(number_features = features)

        return nn.Sequential(
            OrderedDict(
                [
                    (name + "upconv", nn.ConvTranspose2d(in_channels, features, (2, 2), padding='same')),
                    (name + "norm", norm()),
                    (name + 'LReLU',LeakyReLU(alpha=0.1))
                ]
            )
        )

When I put it in the adam optimizer. It complaint that
ValueError: optimizer got an empty parameter list

So i try to do some QC to check the parameters of the Unet. with the following code:

model = EFUnet()
model = model.cuda()
print(list(model.parameters))

However, python complaint that the output is a method which is not iterable.

TypeError: 'method' object is not iterable

Can any one help to see what is causing the issue that python not get the parameters?

Thank you!


title: Roberta detokenization without losing labels

body:
So I ran a custom Roberta Token Classification, and I have a tokenized output id that looks like this, each of the output ID's has a corresponding label.

  Ġlives
  Ġo
  Ġn
  ĠCali

Each of these tokens comes with its prediction label. Detokenizing the text is trivial, but it loses the match with the prediction labels.

detokenized = ''.join(tokens).replace('Ġ', ' ')

However, the challenge is to match the labels to the detokenized text. if anyone can point me to a script or method to use to solve this problem, would be really nice. Also below is my prediction script which was modified from a BERT script.

tags, t_words = [], []                                         #tokenize
for row in test_df:
  tokenized_sentence = tokenizer.encode(row)
  input_ids = torch.tensor([tokenized_sentence]).cuda()

  with torch.no_grad():                                                 #predict
    output = model(input_ids)
    label_indices = np.argmax(output[0].to('cpu').numpy(), axis=2)

     # join split tokens
    new_tokens, new_labels = [], []
    tokens = tokenizer.convert_ids_to_tokens(input_ids.to('cpu').numpy()[0] )
    for token, label_idx in zip(tokens, label_indices[0]):
        new_labels.append(tag_values[label_idx])
        new_tokens.append(token)
    tags.append(new_labels)
    t_words.append(new_tokens)

but is there any way to have the prediction labels match the detokenized inputs, else the entire attempt at prediction is pointless.


title: The loss doesn't decrease correctly while training BERT from scratch

body:
I am now using huggingface/transformers to train a BERT model on 1m wiki data from scratch, but the training loss looks so weird. Before showing the details of the training process, I would first share the scripts and configs I used:

python run_language_modeling.py --output_dir $OUTPUT_DIR \
                                --model_type bert \
                                --mlm \
                                --config_name $CONFIG_AND_DATA_DIR \
                                --tokenizer_name $CONFIG_AND_DATA_DIR \
                                --do_train \
                                --do_eval \
                                --num_train_epochs 20 \
                                --learning_rate 1e-4 \
                                --save_steps 250 \
                                --per_gpu_train_batch_size 64 \
                                --evaluate_during_training \
                                --seed 404 \
                                --block_size 256 \
                                --train_data_file $DATA_DIR/train.txt \
                                --eval_data_file $DATA_DIR/valid.txt \
                                --evaluate_during_training \
                                --logging_steps 250 > log.bert

where run_language_modeling.py is the python script provided by huggingface. I didn't change the config of BERT, except for the vocabulary size. The vocabulary, or the tokenizer, was trained using huggingface/tokenizers.

I put the log of training loss here. We can see that, after 20 epochs, the training loss decreased from 7.84 to 7.61. That's so weird, since the number of lines of the raw data is just 1 million, the training loss should decrease sharply with such an amount of data.

I have tried several ways to address this issue:

Set the batch size as big as the GPU can afford. Since BERT just predicts 15% tokens at each time step, big batch size may give the model more error signals while training. However, it doesn't help.
Maybe the learning rate is too small. I tried to adjust the learning_rate to 5e-4, unfortunately, the converged loss even became worse.

Thanks for your kindly help!!!