title: Max Pooling in VGG16 Before Global Average Pooling (GAP)?

body:
I am currently using VGG16 with Global Average Pooling (GAP) before final classification layer. The VGG16 model used is the one provided by torchvision.

However, I noticed that before the GAP layer, there is a Max Pooling layer. Is this okay or should the Max Pooling layer be removed before the GAP layer? The network architecture can be seen below.

VGG(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (1): ReLU(inplace=True)
    (2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (3): ReLU(inplace=True)
    (4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (5): Conv2d(64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (6): ReLU(inplace=True)
    (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (8): ReLU(inplace=True)
    (9): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (10): Conv2d(128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (13): ReLU(inplace=True)
    (14): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (15): ReLU(inplace=True)
    (16): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (17): Conv2d(256, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (18): ReLU(inplace=True)
    (19): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (20): ReLU(inplace=True)
    (21): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (22): ReLU(inplace=True)
    (23): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (24): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (25): ReLU(inplace=True)
    (26): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (27): ReLU(inplace=True)
    (28): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (29): ReLU(inplace=True)
    (30): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=1)   #GAP Layer
  (classifier): Sequential(
    (0): Linear(in_features=512, out_features=7, bias=True)
  )
)

Thanks in advance.


title: torch.cuda.is_available() False even though correct CUDA version and driver are installed

body:
I installed Pytorch 1.5.0 using conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
nvidia-smi outputs:
NVIDIA-SMI 440.64.00 Driver Version: 440.64.00 CUDA Version: 10.2

Regarding the information on this site everything should be fine.

However CUDA is still not working, as torch.cuda.is_available() returns False.
Does someone know how I can solve this?

Do I have to reboot the machine?


title: I get something wrong when use model.train() and model.eval() on pytorch

body:
I have prepare features and their labels as blow; I want to build a model which is constructed by transformers' encoder and then add a linear layer to predict a value. but I got some error when I use the model to predict after its training.
At first I run below code:
import torch
from torch import nn

features = torch.rand(bach_size, channels, lenght)
labels = torch.rand(batch_size)

class TransformerModel(nn.Module):

  def __init__(self):
    super(TransformerModel, self).__init__()
    encoder_layer = nn.TransformerEncoderLayer(d_model=8, nhead=8, dropout=0.5)
    self.transformer_encoder = nn.TransformerEncoder(encoder_layer, 6)
    self.decoder = nn.Linear(40, 1)

  def forward(self, src):
    encoded = self.transformer_encoder(src.transpose(1, 0)).transpose(1, 0)
    pred = self.decoder(encoded.reshape(encoded.shape[0], -1))
    return pred

model = TransformerModel()
criterion = nn.MSELoss()
lr = 0.3 # learning rate
optimizer = torch.optim.Adam(model.parameters(), lr=lr)

def train():

  model.train() # Turn on the train mode
  optimizer.zero_grad()

  output = model(features)

  loss = criterion(output.view(-1, 1), labels.view(-1, 1))
  loss.backward()
  torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
  optimizer.step()

  return loss.item()

for _ in range(100):
  train()

After that, I predict features by the below codes:
model.eval()
output = model(features)

I get all values of 'output' are the same, and if use 'model.train()', the 'output' seems Ok; so what is the problem? or the model was built wrong?


Installed PyTorch using pip install torchvision
Trying to import  _assert_no_grad and _Loss using

from torch.nn.modules.loss import _assert_no_grad, _Loss using

Getting this error even after reinstalling the Torch and PyTorch

Traceback (most recent call last):   File
"/home/ubuntu/Project/Segmentation/pytorch-discriminative-loss/src/loss.py",
line 7, in
from torch.nn.modules.loss import _assert_no_grad, _Loss ImportError: cannot import name '_assert_no_grad' from
'torch.nn.modules.loss'
(/home/ubuntu/anaconda3/lib/python3.8/site-packages/torch/nn/modules/loss.py)

N. B. - torch version = 1.7.0 and torchvision = 0.8.1


title: Understanding PyTorch CNN Channels

body:
I'm a bit confused at how CNNs and channels work. Specifically, how come these two implementations are not equal? Isn't the # of output channels just applying however many # of filters?

    self.conv1 = nn.Conv2d(1, 10, kernel_size=(3, self.embeds_size))
    self.conv2 = nn.ModuleList([nn.Conv2d(1, 1, kernel_size=(3, self.embeds_size)) for f in range(10)])
    ...

    conv1s = self.conv1(x)
    conv2s = [conv(x) for conv in self.conv2]
    conv2s = torch.stack(conv2s, 1).squeeze(2)
    print(torch.equal(conv1s, conv2s))


title: Problem when using py.test with a pytorch project via the python extension of visual studio code

body:
I'm working on a pytorch project using visual studio code and trying to use py.test.

However, when trying to discover (or run) tests using the vs-code extension, I get an error (see log) importing torch. When I run pytest from the terminal, everything works fine.

Question: What is VS-code doing differently compared to me running pytest myself?
How can I fix this?

Thanks

Config and further info

Win10, custom conda env, python=3.7, pytest=3.8, pytorch=0.4.1 [cuda92], vscode 1.27.2
When I remove some tests so that all remaining tests do not call any torch code, everything works fine (i.e. pytest config inside vs-code should be correct)
Python path pointing to correct env is set in vs-code Workspace Settings
VS Code seems to be using the correct conda env and pytest version according to Python Test Log (see below)
Running pytest directly from terminal works fine (see below)
I can successfully call modules using torch via python or import torch inside ipython
Edit 1: I'm aware of https://github.com/pytorch/pytorch/issues/4518 and have tried some of the more recent suggestions in that thread. (However, I don't think that's really my problem, because as said above I can import torch just fine.)
Edit 2: I'm getting the same error when running any modules that import torch inside vs-code's debugger

Output of Vs-Code Pytest Test Log after failed 'Discover Tests'

============================= test session starts =============================
platform win32 -- Python 3.7.0, pytest-3.8.0, py-1.6.0, pluggy-0.7.1
rootdir: c:\Users\stefan\dev\bnelearn, inifile: pytest.ini
collected 0 items / 1 errors

=================================== ERRORS ====================================
_____________ ERROR collecting bnelearn/tests/test_mechanisms.py ______________
ImportError while importing test module 'c:\Users\stefan\dev\bnelearn\bnelearn\tests\test_mechanisms.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
bnelearn\tests\test_mechanisms.py:3: in <module>
    import torch
..\..\Anaconda3\envs\bnelearn\lib\site-packages\torch\__init__.py:80: in <module>
    from torch._C import *
E   ImportError: DLL load failed: The specified module could not be found.
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.12 seconds ===========================

Output of running pytorch directly


title: RuntimeError: after reduction step 2: device-side assert triggered Pytorch on Jupyter Notebook

body:
I am training a binary classifier using Pytorch on Jupyter Notebook. The following is the architecture:

class AlexNet(nn.Module):
    def __init__(self, num_classes=1):
        super(AlexNet, self).__init__()
        self.conv_base = nn.Sequential(
            nn.Conv2d(3, 96, kernel_size=11, stride=4, padding=2, bias=False),
            nn.BatchNorm2d(96),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(96, 256, kernel_size=5, stride=1, padding=2, bias=False),
            nn.BatchNorm2d(256),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),

            nn.Conv2d(256, 384, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),

            nn.Conv2d(384, 384, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),

            nn.Conv2d(384, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2),
        )
        self.fc_base = nn.Sequential(
            nn.Dropout(),
            nn.Linear(256*6*6, 4096),
            nn.ReLU(inplace=True),

            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(inplace=True),

            nn.Linear(4096, num_classes),
        )

    def forward(self, x):
        x = self.conv_base(x)
        x = x.view(x.size(0), 256*6*6)
        x = self.fc_base(x)
        return x

These are my parameters for training:

criterion = nn.BCELoss()

# specify optimizer
optimizer = torch.optim.SGD(model.parameters(), lr=2e-5, momentum=0.9)

When I start the training, I get the following error:

  RuntimeError: after reduction step 2: device-side assert triggered

Some weird things are also happening:

1) It was training at one point in time, but the learning rate was small. So when I changed it and started the training process again, I got the error.

2) I'm also getting this error when I run this command:

model = AlexNet()
model.cuda()

the error:

  RuntimeError: CUDA error: device-side assert triggered

Some solutions that I found which worked temporarily are:

1) Add os.environ['CUDA_LAUNCH_BLOCKING'] = '1' to the notebook.
2) GPU memory being used. So I removed the process that was running the notebook.

However, nothing seems to be working permanently. Any suggestions?


title: Faster-RCNN Pytorch problem at prediction time with image dimensions

body:
I am finetuning Faster-RCNN using PyTorch according to this tutorial: https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

The results are pretty good but making predictions only work when feeding a single tensor to the model. For example:

# This works well
>>> img, _ = dataset_test[3]
>>> img.shape
torch.Size([3, 1200, 1600])
>>> model.eval()
>>> with torch.no_grad():
    .. preds = model([img.to(device)])

But when I feed multiple tensors at once I get that error:

>>> random_idx = torch.randint(high=50, size=(4,))
>>> images = torch.stack([dataset_test[idx][0] for idx in random_idx])
>>> images.shape
torch.Size([4, 3, 1200, 1600])
>>> with torch.no_grad():
    .. preds = model(images.to(device))
RuntimeError                              Traceback (most recent call last)
<ipython-input-101-52caf8fee7a4> in <module>()
      5 model.eval()
      6 with torch.no_grad():
----> 7   prediction =  model(images.to(device))

...

RuntimeError: The expanded size of the tensor (1600) must match the existing size (1066) at non-singleton dimension 2.  Target sizes: [3, 1200, 1600].  Tensor sizes: [3, 800, 1066]

Edit

Works when feeding a list of 3D tensors (IMO this behaviour is a bit weird, I cannot understand why it is not working with a 4D tensor):

>>> random_idx = torch.randint(high=50, size=(4,))
>>> images = [dataset_test[idx][0].to(device) for idx in random_idx]
>>> images.shape
torch.Size([4, 3, 1200, 1600])
>>> with torch.no_grad():
    .. preds = model(images)


The error Expected object of scalar type Float but got scalar type Long for argument #2 'mat1' in call to _th_addmm is being displayed after running the code below.

import numpy as np
import pandas as pd
import nltk
from nltk.stem import WordNetLemmatizer
nltk.download('wordnet')
import re
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import Dataset, DataLoader, TensorDataset

... (created a pandas dataframe containing a 'tweet', 'sentiment' and 47 one-hot bag of words cols)

# create train_target data set containing just the target
train_target = torch.tensor(df_train['sentiment'].values, dtype=torch.long)

# create train predictor features data set containing just the predictor features
train = torch.tensor(df_train.drop(['tweet','sentiment'], axis = 1).values, dtype=torch.long)

# convert to torch tensor and define data loader (train)
train_tensor = TensorDataset(train, train_target)
trainset = torch.utils.data.DataLoader(train_tensor, batch_size=2, shuffle=False)

# create test_target data set containing just the target
test_target = torch.tensor(df_test['sentiment'].values, dtype=torch.long)

# create test predictor features data set containing just the predictor features
test = torch.tensor(df_test.drop(['tweet','sentiment'], axis = 1).values, dtype=torch.long)

# convert to torch tensor and define data loader (test)
test_tensor = TensorDataset(test, test_target)
testset = torch.utils.data.DataLoader(test_tensor, batch_size=2, shuffle=False)

input_length = 47

class Net(nn.Module):   # create a new class called Net that inherits from nn's "Module" class

    # initialise our Net class
    def __init__(self):
        super().__init__()    # run the initialisation function of the nn.Module class (the parent class)
        self.fc1 = nn.Linear(input_length, 768)   # define the first fully connected layer.
        self.fc2 = nn.Linear(768, 768)    # define the 2nd layer
        self.fc3 = nn.Linear(768, 2)    # output layer. 2 classes so output is size 2

    # define how data will flow through feed-forward network
    def forward(self, x):
        x = F.relu(self.fc1(x))  # x becomes the output of running the 1st layer, after relu activation is applied
        x = F.relu(self.fc2(x))    # x then becomes the output of the 2nd layer...
        x = self.fc3(x)

        return F.log_softmax(x, dim=1)  # we want a probability distribution across the 2 classes so using softmax
                                        # dim=1 because we want the probabilities across the classes, not
                                        # the batches

# create a Net object called net
net = Net()
print(net)

# Train the model (net)
optimiser = optim.Adam(net.parameters(),  # net.parameters is everything thats adjustable in our model
                      lr=0.001)  # learning rate
epochs = 10
for epoch in range(epochs):
    for batch_data in trainset:
        X, y = batch_data  # set X as the input and y as the label
        net.zero_grad() # start with gradients of zero
        output = net(X.view(-1, input_length))  # run the model (could put batch size in instead of -1)
        loss = F.nll_loss(output, y)   # how wrong were we? Calculate loss.
        loss.backward()    # backpropagate the loss (how much did each weight contribute to the loss?)
        optimiser.step()    # adjust the weights based on the backpropagation
    print('Epoch: ', epoch, ' Loss: ', loss)

What do I need to do to fix this? I'm trying to create a neural network to do sentiment classification on some text data from tweets.


title: RuntimeError: CUDA out of memory. Problem with stanza lemmatazation using too much GPU memory

body:
Good day,
I have 11GB of GPU memory and I run into CUDA memory issue with pretrained lemmatazation.
I used this code:
snlp = stanza.Pipeline(lang="en", use_gpu=True) # tried different batch_size/ lemma_batch_size - did not help
nlp = StanzaLanguage(snlp)

def tokenize(text):
     tokens = nlp(text)
     doc_l = [token.lemma_ for token in doc]
     lower_tokens = [t.lower() for t in doc_l]
     alpha_only = [t for t in lower_tokens if t.isalpha()]
     no_stops = [t for t in alpha_only if t not in stopwords]
     #torch.cuda.empty_cache() # Tried this - did not work
     return no_stops

tfidf = TfidfVectorizer(tokenizer=tokenize, min_df=0.1, max_df=0.9)
# Construct the TF-IDF matrix
tfidf_matrix = tfidf.fit_transform(texts)

RuntimeError: CUDA out of memory. Tried to allocate 978.00 MiB (GPU 0;
11.00 GiB total capacity; 6.40 GiB already allocated; 439.75 MiB free; 6.53 GiB reserved in total by PyTorch).

I tried
 [(tokenize(t) for t in test]

It only lasted for 12 texts. They are 200 words on average each. Based on Error message - 'Tried to allocate 978.00 MiB' and this data - SNLP uses 1GiB of GPU memory per step??

This behavior seems strange to me (probably because I dont understand how library works) as model is already pretrained, so it should not get bigger when transforming new texts, right? Why it needs so much GPU memory?
Is there any way to clear memory after each run of lemma_ for each text? (#torch.cuda.empty_cache()-does not work) and batch_size does not work either.

It works on CPU, however allocates all of the available memory (32G of RAM), however. It is much slower on CPU. I need it to make it work on CUDA.


title: CUDA error: an illegal memory access was encountered

body:
I am unable to do device conversion for a PyTorch tensor on Google Colab. I attach the code snippet below:
Please help me.


title: How to decode an embedding in PyTorch efficiently?

body:
I am new to Pytorch and RNN. I am learning how to use RNN to predict numbers as a tutorial from the video: https://www.youtube.com/watch?v=MKA6v99uYKY

In his code, he use python 3 and do the decode like:

out_unembedded = out.view(-1, hidden_size) @ embedding.weight.transpose(0,1)

I am using Python 2 and try the code:

out_unembedded = out.view(-1, hidden_size).dot( embedding.weight.transpose(0,1))

But it seems not right, Then I try to decode like this:

import torch
import torch.nn as nn
from torch.autograd import Variable

word2id = {'hello': 0, 'world': 1, 'I': 2, 'am': 3,'writing': 4,'pytorch': 5}
embeds = nn.Embedding(6, 3)
word_embed = embeds(Variable(torch.LongTensor([word2id['am']])))

id2word = {v: k for k, v in word2id.iteritems()}
index = 0
for row in embeds.weight.split(1):
    if(torch.min( torch.eq(row.data,word_embed.data) ) == 1):
        print index
        print id2word[index]
    index+=1

Is there a more professional way to do this? Thanks!

------------ UPDATE ------------

I find the correct way to substitute @ in Python 2:

out_unembedded = torch.mm( embedded_output.view(-1, hidden_size),embedding.weight.transpose(0, 1))


title: Why do I see TypeError when Pytorch's mul() function is used in combination with numpy?

body:
I am getting the following error in the terminal:

Traceback (most recent call last):
  File "deep_Q_learner.py", line 289, in <module>
    agent.replay_experience()
  File "deep_Q_learner.py", line 170, in replay_experience
    self.learn_from_batch_experience(experience_batch)
  File "deep_Q_learner.py", line 151, in learn_from_batch_experience
    self.Q_target(next_obs_batch).max(1)[0].data
TypeError: mul(): argument 'other' (position 1) must be Tensor, not numpy.ndarray

The link to the code is:
https://github.com/PacktPublishing/Hands-On-Intelligent-Agents-with-OpenAI-Gym/blob/master/ch6/deep_Q_learner.py

The error is seen only when self.DQN=SLP (see line#76)

Is there a fix to this issue? Am I missing something here?


title: Transform images to white on black and predict

body:
I want to predict new customized images with LeNet model trained from here.
The customized images are black on white, so I need to convert them to white on black.

# Load & transform image
ori_img = Image.open('./test/2.png').convert('L')
img = np.invert(ori_img) #Transform images to white on black
t = transforms.Compose([
    transforms.Resize((32, 32)),
    transforms.ToTensor(),
    transforms.Normalize((0.1307,), (0.3081,))
])
img = torch.autograd.Variable(t(img).unsqueeze(0))
ori_img.close()

# Predict
model.eval()
output = model(img)
pred = output.data.max(1, keepdim=True)[1][0][0]
print('Prediction: {}'.format(pred))

The result I got:

TypeError                                 Traceback (most recent call last)
<ipython-input-182-abbffa2ce0d8> in <module>
      7     transforms.Normalize((0.1307,), (0.3081,))
      8 ])
----> 9 img = torch.autograd.Variable(t(img).unsqueeze(0))
     10 ori_img.close()

~/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py in __call__(self, img)
     59     def __call__(self, img):
     60         for t in self.transforms:
---> 61             img = t(img)
     62         return img
     63

~/.local/lib/python3.6/site-packages/torchvision/transforms/transforms.py in __call__(self, img)
    196             PIL Image: Rescaled image.
    197         """
--> 198         return F.resize(img, self.size, self.interpolation)
    199
    200     def __repr__(self):

~/.local/lib/python3.6/site-packages/torchvision/transforms/functional.py in resize(img, size, interpolation)
    236     """
    237     if not _is_pil_image(img):
--> 238         raise TypeError('img should be PIL Image. Got {}'.format(type(img)))
    239     if not (isinstance(size, int) or (isinstance(size, Iterable) and len(size) == 2)):
    240         raise TypeError('Got inappropriate size arg: {}'.format(size))

TypeError: img should be PIL Image. Got <class 'numpy.ndarray'>

When I comment img = np.invert(ori_img) I get no errors, but all prediction results are 2s.

Someone could help? Thanks  a lot.


title: How can I element-wise multiply tensors with different dimensions?

body:
I have a tensor expanded_mask, which has a size of torch.Size([1, 208]) and another one inputs which has a size of torch.Size([1, 208, 161]).

I want to elementwise multiply expanded_mask and input such that all 161 elements of the third dimension are multiplied with the 208 elements of expanded_mask.

As per jodag's answer, I tried:

masked_inputs = expanded_mask.unsqueeze(2) * inputs

inputs is:

tensor([1.8851e-02, 4.4921e-02, 7.5260e-02, 3.8994e-02, 3.5651e-02, 3.0457e-02,
        1.2933e-02, 2.5496e-02, 2.3260e-04, 2.4903e-03, 6.5678e-03, 1.0501e-02,
        1.2387e-02, 1.9434e-03, 1.0831e-03, 6.5691e-03, 5.3792e-03, 9.1925e-03,
        1.8146e-03, 4.9215e-03, 1.4623e-03, 9.4454e-03, 1.0504e-03, 3.3749e-03,
        2.1361e-03, 8.0782e-03, 1.7916e-03, 1.1577e-03, 1.1246e-04, 2.2520e-03,
        2.2255e-03, 2.1072e-03, 9.8782e-03, 2.2909e-03, 2.9957e-03, 5.8540e-03,
        1.1067e-02, 9.0582e-03, 5.6360e-03, 6.3841e-03, 5.9298e-03, 1.9501e-04,
        2.7967e-03, 3.5786e-03, 9.2363e-03, 8.3934e-03, 8.8185e-04, 5.4591e-03,
        2.2451e-04, 2.2307e-03, 2.4871e-03, 3.6736e-03, 1.3842e-04, 2.7455e-03,
        6.2199e-03, 1.1924e-02, 9.5953e-03, 1.6939e-03, 4.1919e-04, 9.3509e-05,
        1.8351e-03, 6.3350e-04, 1.1076e-03, 1.5472e-03, 1.2104e-03, 3.1803e-04,
        8.6507e-04, 3.0083e-03, 2.8435e-03, 1.6740e-03, 8.1023e-05, 7.5767e-04,
        9.1442e-04, 2.0204e-03, 1.3987e-03, 3.7729e-03, 5.2012e-04, 2.0367e-03,
        1.5177e-03, 1.6948e-03, 9.5833e-04, 1.2050e-03, 1.8356e-03, 9.4503e-04,
        4.8612e-04, 1.6844e-04, 1.2222e-04, 1.7526e-03, 2.6397e-04, 1.3026e-03,
        1.0704e-03, 3.6407e-04, 1.3135e-03, 2.6665e-03, 1.8639e-03, 3.0385e-05,
        1.0212e-03, 7.6236e-04, 1.7878e-03, 2.4298e-03, 7.2158e-05, 1.2488e-03,
        2.1347e-03, 3.9256e-03, 3.1436e-03, 3.1648e-03, 3.4657e-03, 1.3746e-03,
        1.6927e-03, 1.0794e-03, 8.8152e-04, 1.1757e-04, 3.2254e-04, 4.1866e-04,
        9.2787e-04, 2.0020e-03, 1.4813e-03, 1.1912e-03, 2.4577e-03, 2.2247e-03,
        1.7862e-03, 1.7460e-03, 1.4388e-03, 4.3175e-04, 6.7808e-04, 2.6875e-04,
        3.6475e-04, 8.7643e-04, 3.6790e-04, 2.1274e-04, 6.3725e-04, 2.0949e-03,
        2.4069e-03, 1.7348e-03, 1.0026e-03, 1.2451e-03, 4.7888e-04, 5.9790e-04,
        1.4343e-03, 4.0900e-03, 1.0176e-03, 5.5178e-04, 2.0624e-03, 1.2878e-03,
        6.9607e-04, 4.3259e-04, 1.8573e-03, 7.5521e-04, 5.2949e-04, 3.4758e-04,
        4.7898e-04, 7.5599e-04, 6.0631e-04, 1.7585e-03, 1.8156e-03, 3.2421e-04,
        8.9446e-04, 7.2131e-04, 6.2817e-04, 1.0827e-03, 2.0211e-03],
       device='cuda:0')

expanded_mask is:

tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
         1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]], device='cuda:0',
       grad_fn=<AsStridedBackward>)

then masked_inputs is:

tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
       device='cuda:0', grad_fn=<SelectBackward>)

Looks like the 1's isn't being multiplied through.


title: Python, class dataset, how to concatenate images with their respective labels in pytorch

body:
I am new to PyTorch, and in the last couple of days I have been struggling with the class Dataset that lets you build your custom dataset.

I am working with this dataset (https://www.kaggle.com/ianmoone0617/flower-goggle-tpu-classification/kernels) , the problem is that it has the images and their labels in separate folders, and I can’t figure out how to concatenate them.

This is the code I am using:

class MyDataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
        self.labels = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.labels)

    def __getitem__(self, index):
        if torch.is_tensor(index):
            index = index.tolist()

        image_name = os.path.join(self.root_dir, self.labels.iloc[index, 0])
        image = io.imread(image_name)

        if self.transform:
            image = self.transform(image)

        return (image, labels)

While the structure of the folders is the following one:

I really want to understand this so thank you in advance guys!!


title: Loading json file using torchtext

body:
I'm working on the dailydialog dataset, which I've converted into a
JSON file which looks something like this:
[{"response": "You know that is tempting but is really not good for our fitness.", "message": "Say, Jim, how about going for a few beers after dinner?"}, {"response": "Do you really think so? I don't. It will just make us fat and act silly. Remember last time?", "message": "What do you mean? It will help us to relax."}, {"response": "I suggest a walk over to the gym where we can play singsong and meet some of our friends.", "message": "I guess you are right. But what shall we do? I don't feel like sitting at home."}, {"response": "Sounds great to me! If they are willing, we could ask them to go dancing with us.That is excellent exercise and fun, too.", "message": "That's a good idea. I hear Mary and Sally often go there to play pingpong.Perhaps we can make a foursome with them."}, {"response": "All right.", "message": "Please lie down over there."}]
So, each item has two keys - response and message.
This is my first time using PyTorch, so I was following a few online available resources. These are the relevant snippets of my code:
def tokenize_en(text):
    return [tok.text for tok in spacy_en.tokenizer(text)]

src = Field(tokenize = tokenize_en,
            init_token = '<sos>',
            eos_token = '<eos>',
            lower = True)

fields = {'response': ('r', src)}

train_data, test_data, validation_data = TabularDataset.splits(
                                        path = 'FilePath',
                                        train = 'trainset.json',
                                        test = 'testset.json',
                                        validation = 'validationset.json',
                                        format = 'json',
                                        fields = fields
)

Although no errors are raised, despite having many items in my JSON file, the train, test and validation datasets strangely have only 1 example each, as seen in this image:
Image Showing the length of train_data, test_data and validation_data
I'd be really grateful if someone could point out the error to me.
Edit: I found out that the whole file is being treated as a single text string due to lack of indents in the file. But if I indent the JSON file, the TabularDataset function throws a JSONDecodeError to me, suggesting it can no more decode the file. How can I get rid of this problem?


title: How to get the inner module of Unet?

body:
I created a UNet with the the UnetGenerator. You can find the resulting structure here.

How do I get the module Conv2d(128, 256, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))?
How do I get the module (5): ConvTranspose2d(256, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1))?

I want to get the inner modules to change the attributes of certain layers.
I tried something like net.modules(i).modules(i), but it doesn't work. I refer the docs, I haven't get a good idea to do it.

My initial intention is to change the attributes of certain layers when training. I may add a custom layer myLayer, in which self.mode='normal'. When training, I hope I can change its attribute myLayer.mode = 'capture' to make it change its behaviour in training.


title: pytorch distributed training fails when use 'gloo' backend

body:
I run the distributed training code in the pytorch tutorial, when I use the 'gloo' backend, the code raises RuntimeError when init_process_group.

"""run.py:"""
#!/usr/bin/env python
import os
import torch
import torch.distributed as dist
from torch.multiprocessing import Process

def run(rank, size):
    """ Distributed function to be implemented later. """
    pass

def init_process(rank, size, fn, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_PORT'] = '29500'
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size)

if __name__ == "__main__":
    size = 2
    processes = []
    for rank in range(size):
        p = Process(target=init_process, args=(rank, size, run))
        p.start()
        processes.append(p)

    for p in processes:
        p.join()

While getting the error:

Process Process-2:
Traceback (most recent call last):
  File "/home/wuyiming/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/wuyiming/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-1-6cc8a94a6551>", line 16, in init_process
    dist.init_process_group(backend, rank=rank, world_size=size)
  File "/home/wuyiming/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 416, in init_process_group
    timeout=timeout)
  File "/home/wuyiming/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 484, in _new_process_group_helper
    timeout=timeout)
RuntimeError: [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/device.cc:198] ifa != nullptr. Unable to find interface for: [0.0.0.27]
Process Process-1:
Traceback (most recent call last):
  File "/home/wuyiming/anaconda3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/home/wuyiming/anaconda3/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-1-6cc8a94a6551>", line 16, in init_process
    dist.init_process_group(backend, rank=rank, world_size=size)
  File "/home/wuyiming/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 416, in init_process_group
    timeout=timeout)
  File "/home/wuyiming/anaconda3/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 484, in _new_process_group_helper
    timeout=timeout)
RuntimeError: [enforce fail at /pytorch/third_party/gloo/gloo/transport/tcp/device.cc:198] ifa != nullptr. Unable to find interface for: [0.0.0.27]

If I change the backbone from 'gloo' to 'NCCL', the code runs correctly.


title: Using pre-trained weights in PyTorch

body:
I am working on implementing a research paper based on computer vision in PyTorch. I have built the model architecture by referring to the paper. The author has uploaded saved weights on GitHub in ".pth.tar" format. I want to put the same weights in my model so that I can skip training and optimization part and directly get output from the neural net.
The paper is Learning to see in the dark.
Model architecture is as follow:
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv1d(32, 12, 1)
        .
        .
   def forward(self, x):
        x = F.relu(self.conv1(x))
        .
        .
        return x
net = Net()

And it is to be followed by importing trained weight from google drive/cloud storage and defining the function to put the trained weights in the net.
PS: Model architecture is exactly same for both


title: Code example in Hugging Face Pytorch-Transformers quickstart documentation

body:
I am following the quick start guide here. The problem is they have provided code for GPU machine and I am running the code on CPU based Ubuntu machine. I have commented the lines that put everything in CUDA. The code is now showing error and I don't know how to resolve it.
The question is "How can I make this work?"

I have checked this answer and this is not what I'm looking for.

The full code is here

1. Using BertModel to encode inputs in hidden-states:

#Load pre-trained model (weights)
model = BertModel.from_pretrained('bert-base-uncased')

#Set the model in evaluation mode to desactivate the DropOut modules
# This is IMPORTANT to have reproducible results during evaluation!
model.eval()

#***I have commented these 3 lines***

# If you have a GPU, put everything on cuda
#tokens_tensor = tokens_tensor.to('cuda')
#segments_tensors = segments_tensors.to('cuda')
#model.to('cuda')

#Rest all is untouched
# *** -----------------***---------------***

# Predict hidden states features for each layer
with torch.no_grad():
    # See the models docstrings for the detail of the inputs
    outputs = model(tokens_tensor, token_type_ids=segments_tensors)
 # PyTorch-Transformers models always output tuples.
 # See the models docstrings for the detail of all the outputs
 # In our case, the first element is the hidden state of the last layer of the Bert model
    encoded_layers = outputs[0]
# We have encoded our input sequence in a FloatTensor of shape (batch size, sequence length, model hidden dimension)
assert tuple(encoded_layers.shape) == (1, len(indexed_tokens), model.config.hidden_size)

Error for 1:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-40-a86e9643e7f3> in <module>
     11
     12 # We have encoded our input sequence in a FloatTensor of shape (batch size, sequence length, model hidden dimension)
---> 13 assert tuple(encoded_layers).shape == (1, len(indexed_tokens), model.config.hidden_size)

AttributeError: 'tuple' object has no attribute 'shape'

2. Using BertForMaskedLM to predict a masked token:

# Load pre-trained model (weights)
model = BertForMaskedLM.from_pretrained('bert-base-uncased')
model.eval()

#***---------------Commented--------------------------
# If you have a GPU, put everything on cuda
#tokens_tensor = tokens_tensor.to('cuda')
#segments_tensors = segments_tensors.to('cuda')
#model.to('cuda')

#***---------------------------------------------

# Predict all tokens
with torch.no_grad():
    outputs = model(tokens_tensor, token_type_ids=segments_tensors)
    predictions = outputs[0]

# confirm we were able to predict 'henson'
predicted_index = torch.argmax(predictions[0, masked_index]).item()
predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
assert predicted_token == 'henson'

Error for 2:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-42-9b965490d278> in <module>
     17 predicted_index = torch.argmax(predictions[0, masked_index]).item()
     18 predicted_token = tokenizer.convert_ids_to_tokens([predicted_index])[0]
---> 19 assert predicted_token == 'henson'

AssertionError:


title: CUDA Path Not Correctly Configured

body:
I am trying to change my environment path variables so Pytorch can access CUDA. According to nvidia-smi, the driver can support CUDA 10.0. CUDA 10.0 is already installed on the server. I modified my bash_profile according to these instructions.

export PATH=/usr/local/cuda-10.0/bin${PATH:+:${PATH}}
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64\{LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

nvcc shows the CUDA version. However, Pytorch still does not detect CUDA.

Is there some solution? I am a non-root user.


title: How to partition a neural network into sub-networks in Pytorch?

body:
I'd like to partition a neural network into two sub-networks using Pytorch. To make things concrete, consider this image:

In 1, I've a 3x4x1 neural network. What I want is, for example during epoch 1, I'd only like to update the weights in the sub-network 1, i.e., the weights that appear in the sub-network 2 must be frozen. Then again, in epoch 2, I'd like to train the weights that appear in sub-network 2 while the rest should be frozen.

How can I do that?


title: Is it safe to always use torch.tensor or torch.FloatTensor? Or do I need to treat Ints with care?

body:
https://pytorch.org/docs/stable/tensors.html

I am trying to understand the difference between tensor, FloatTensor, IntTensor - and am wondering if I can just always stick to tensor... or maybe FloatTensor.

I am going to be using a mix of different tensors that will be:

{integers:labels, floats:continuous, one-hot-encoded:categoricals}

Do I need to explicitly set each of these as kinds of variables as different types of tensors? Will they all work as floats? Will they work in combination with each other?

Would this get me into trouble downstream?

l_o_l = [[1,2,3],[1,2,3],[1,2,3]]

int_tnz = th.FloatTensor(l_o_l)
int_tnz

tensor([[1., 2., 3.],
        [1., 2., 3.],
        [1., 2., 3.]])

int_tnz.dtype

torch.float32

l_o_fl = [[1.1,2.2,3.3],[1.1,2.2,3.3],[1.1,2.2,3.3]]

int_tnz = th.tensor(l_o_fl)
int_tnz

tensor([[1.1000, 2.2000, 3.3000],
        [1.1000, 2.2000, 3.3000],
        [1.1000, 2.2000, 3.3000]])

int_tnz.dtype

torch.float32


title: WGAN-GP loss explodes

body:
I have been working on a project of Wasserstein GAN with gradient penalty. The project is with MRI/CT(3D data). When I was doing a 2D implementation(slice by slice) everything worked perfectly. However, when I tried extending it to 3D the Generators loss explodes to really high values(~10^7).

I am using a patch size of (144,176,20)
Batch size of 4, 4 GPUs.
The architectures of Critic is borrowed from https://github.com/igul222/improved_wgan_training
I am NOT using batch norm in Critic.
Doing 5 updates of Critic per one update of G.

I tried decreasing the number of filters in G, to make Critic and G more balanced. It still goes pretty high(10^5-10^6).


title: How to get pytorch to work in Jupyterhub?

body:
I'm using Jupyterhub, and am running someone else's code that uses pytorch. They include it via import torch, but I get the error "No module named 'torch'". I installed pytorch locally with conda, but still get the same error in Jupyterhub.

What I Want To Know:

For modules like pytorch that aren't part of Jupyterhub out of the box, how do you include them? Do you do it by installing them locally like I did? If so, what should I do differently?


title: How can I fix the weights of 'torch.nn.Linear'?

body:
I want to know why there are two tensors in parameter list of nn.linear??

I tried to set parameters, but it didn't work. How can I fix it?

XX=torch.from_numpy(X)
YY=torch.from_numpy(Y)
Ytt=torch.from_numpy(Yt)
XX=XX.view(100,1)
YY=YY.view(100,1)
Ytt=Ytt.view(100,1)
class model(torch.nn.Module):
  def __init__(self):
    super(model,self).__init__()
    self.linear1 = torch.nn.Linear(1,2).double()
    self.linear2 = torch.nn.Linear(2,2).double()
    self.linear3 = torch.nn.Linear(2,1).double()

  def forward(self,x):
    x=F.relu(self.linear1(x))
    x=F.relu(self.linear2(x))
    x=self.linear3(x)
    return x

M=model()
L=nn.MSELoss()
print(list(M.linear1.parameters()))
list(M.linear1.parameters())[0]=torch.Tensor([[-0.1],
        [ 0.2]])

print(list(M.linear1.parameters()))

Then
[Parameter containing:
tensor([[-0.2288],
        [ 0.2211]], dtype=torch.float64, requires_grad=True), Parameter containing:
tensor([-0.9185, -0.2458], dtype=torch.float64, requires_grad=True)]
[Parameter containing:
tensor([[-0.2288],
        [ 0.2211]], dtype=torch.float64, requires_grad=True), Parameter containing:
tensor([-0.9185, -0.2458], dtype=torch.float64, requires_grad=True)]


title: Process output data from YOLOv5 TFlite

body:
❔Question
Hi, I have successfully trained a custom model based on YOLOv5s and converted the model to TFlite. I feel silly asking, but how do you use the output data?
I get as output:

StatefulPartitionedCall: 0 = [1,25200,7]
from the converted YOLOv5 model
Netron YOLOv5s.tflite model

But I expect an output like:

StatefulPartitionedCall:3 = [1, 10, 4] # boxes
StatefulPartitionedCall:2 = [1, 10] # classes
StatefulPartitionedCall:1 = [1, 10] #scores
StatefulPartitionedCall:0 = [1] #count
(this one is from a tensorflow lite mobilenet model (trained to give 10 output data, default for tflite))
Netron mobilenet.tflite model

It may also be some other form of output, but I honestly have no idea how to get the boxes, classes, scores from a [1,25200,7] array.
(on 15-January-2021 I updated pytorch, tensorflow and yolov5 to the latest version)
The data contained in the [1, 25200, 7] array can be found in this file: outputdata.txt
0.011428807862102985, 0.006756599526852369, 0.04274776205420494, 0.034441519528627396, 0.00012877583503723145, 0.33658933639526367, 0.4722323715686798
0.023071227595210075, 0.006947836373001337, 0.046426184475421906, 0.023744791746139526, 0.0002465546131134033, 0.29862138628959656, 0.4498370885848999
0.03636947274208069, 0.006819264497607946, 0.04913407564163208, 0.025004519149661064, 0.00013208389282226562, 0.3155967593193054, 0.4081345796585083
0.04930267855525017, 0.007249316666275263, 0.04969717934727669, 0.023645592853426933, 0.0001222355494974181, 0.3123127520084381, 0.40113094449043274
...

Should I add a Non Max Suppression or something else, can someone help me please? (github YOLOv5 #1981)


title: How to Invert AvgPool2d?

body:
Is it possible to invert an avgPool2d operation in PyTorch, like maxunpool2d for a maxpool2d operation, and if so, how could that be done?

I've already checked the documentation, and there isn't an option to return the indices, like in the maxpool2d operation, so I assume the unpooling won't be possible in a similar way.

EDIT:
I found a document by Intel which describes how the unpooling works. After checking the math regarding the avgpool2d function the unpooling seems to be pretty straight forward, basically mirroring every input element onto multiple output elements, and apply padding in order to get a correct output size.


title: The similar function of "tf.Session().partial_run()" in Pytorch

body:
I am trying to use Pytorch to reimplement a project, but I have some troubles to find a similar function of "partial_run" of Tensorflow. Can you show me what it is?
Thanks.


title: What should be the loss function for classification problem in pytorch if sigmoid is used in the output layer

body: I am trying to implement a model for binary classification problem. Up to now, I was using softmax function (at the output layer) together with torch.NLLLoss function to calculate the loss. However, now I want to use the sigmoid function (instead of softmax) at the output layer. If I do that, should I also change the loss function (to BCELoss or binary_cross_entropy) or may I still use torch.NLLLoss function ?


title: Cuda 10.2 not recognised on Pip installed Pytorch 1.7.1

body:
Pip installing Pytorch 1.7.1 here on Gtx1660(latest drivers installed) doesn't recognise the installed Cuda toolkit 10.2 on my machine(Windows10).
As this is a personal project, I don't wish to use Anaconda.
How to resolve this?
#INSTALLATION (10.2 CUDA SUPPORT)
pip install torch===1.7.1 torchvision===0.8.2 torchaudio===0.7.2 -f https://download.pytorch.org/whl/torch_stable.html

#CODE
import torch
device = torch.device("cuda")
# device = torch.device("cuda:0") #ALSO NOT WORKING
cuda_available = torch.cuda.is_available()
cuda_init = torch.cuda.is_initialized()
print(f'device_current = {device},cuda_available = {cuda_available} and cuda_init = {cuda_init}\n')

torch.cuda.init()
print(f'device_current = {device} and cuda_init = {cuda_init}')

#TERMINAL
device_current = cuda:0,cuda_available = False and cuda_init = False

Traceback (most recent call last):
  File "f:\Script\Ai\Pytorch\mod_test\test1.py", line 8, in <module>
    torch.cuda.init()
  File "C:\Users\User0\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\cuda\__init__.py", line 137, in init
    _lazy_init()
  File "C:\Users\User0\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\cuda\__init__.py", line 166, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled


title: Running a portion of Python code in parallel on two different GPUs

body:
I have a PyTorch script similar to the following:

# Loading data
train_loader, test_loader = someDataLoaderFunction()

# Define the architecture
model = ResNet18()
model = model.cuda()

# Get method from program argument
method = args.method

# Training
train(method, model, train_loader, test_loader)

In order to run the script with two different methods (method1 and method2), it suffices to run the following commands in two different terminals:

CUDA_VISIBLE_DEVICES=0 python program.py --method method1
CUDA_VISIBLE_DEVICES=1 python program.py --method method2

The problem is, the above data loader function contains some randomness in it, which means that the two methods were applied to two different sets of training data. I would like them to train the exact same set of data, so I modified the script as follows:

# Loading data
train_loader, test_loader = someDataLoaderFunction()

# Define the architecture
model = ResNet18()
model = model.cuda()

## Run for the first method
method = 'method1'

 # Training
train(method, model, train_loader, test_loader)

## Run for the second method
method = 'method2'

# Must re-initialize the network first
model = ResNet18()
model = model.cuda()

 # Training
train(method, model, train_loader, test_loader)

Is it possible to make it run in parallel for each method?
Thank you so much in advance for your help!


title: Size mismatch error during VGG finetuning

body:
I have been following the ants and bees transfer learning tutorial from the official PyTorch Docs (http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html). I am trying to finetune a VGG19 model by changing the final layer to predict one of two classes. I am able to modify the last fc layer using the following code.

But I get an error when executing the train_model function. The error is “size mismatch at /opt/conda/conda-bld/pytorch_1513368888240/work/torch/lib/THC/generic/THCTensorMathBlas.cu:243”. Any idea what the issue is ?

model_conv = torchvision.models.vgg19(pretrained=True)
for param in model_conv.parameters():
    param.requires_grad = False

model_conv = nn.Sequential(*list(model_conv.classifier.children())[:-1] +
                     [nn.Linear(in_features=4096, out_features=2)])
if use_gpu:
    model_conv = model_conv.cuda()

criterion = nn.CrossEntropyLoss()

optimizer_conv = optim.SGD(model_conv._modules['6'].parameters(), lr=0.001, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)

model_conv = train_model(model_conv, criterion, optimizer_conv, exp_lr_scheduler, num_epochs=25)


title: Mistake in pytorch attention seq2seq tutorial?

body:
I am writing a sequence to sequence neural network in Pytorch. In the official Pytorch seq2seq tutorial, there is code for an Attention Decoder that I cannot understand/think might contain a mistake.

It computes the attention weights at each time step by concatenating the output and the hidden state at this time, and then multiplying by a matrix to get a vector of size equal to the output sequence length. Note, these attention weights don’t depend on the encoder sequence (named encoder_outputs in the code), which I think it should.

Also, the paper cited in the tutorial, lists three different score functions that can be used to compute attention weights (section 3.1 in the paper). None of these functions is just concatenating and multiplying by a matrix.

So it seems to me that the code in the tutorial is mistaken both in the function it applies and the arguments that are passed to this function. Am I missing something?


title: is loaded model does not need eval?

body:
tryint to use the test after I load the model

net = net.load_state_dict(torch.load(PATH))
net.eval()

but this spit the error

net.eval() AttributeError: '_IncompatibleKeys' object has no attribute 'eval'


title: How to convert string list to tensor in pytorch?

body:
I'm working on making and predicting several models using phytorch.
Because of memory problems, I made the tensor list into a data frame and saved it as Excel.
After that, I tried to predict the model by loading the data stored in Excel, but when I called Excel, the tensor list changed to a str list.
How can I change this str list back to the tensor list?
I'll refer to part of the code, the original tensor.

def BERT_reasoning(tokens_tensor, segments_tensors):
    model.eval()
    predictions=[]
    for i in range(len(tokens_tensor)):
        if torch.cuda.is_available():
            tokens_tensor[i] = tokens_tensor[i].to('cuda')
            segments_tensors[i] = segments_tensors[i].to('cuda')
            model.to('cuda')
            with torch.no_grad():
                outputs = model(tokens_tensor[i], token_type_ids=segments_tensors[i])
                predictions.append(outputs[0])
                torch.cuda.empty_cache()
    return(predictions)

predictions=[0 for i in range(len(target))]
for i in tqdm(range(len(target))):
    predictions[0]=BERT_reasoning(tokens_tensor[i],segments_tensors[i])
    globals()['df_pred_{}'.format(i)]=pd.DataFrame(predictions[0])
    del predictions[0]
    excel_name='prediction_{}.xlsx'.format(i)
    globals()['df_pred_{}'.format(i)].to_excel(excel_name)
    del globals()['df_pred_{}'.format(i)]
    torch.cuda.empty_cache()

Result :
orginal tensor -
tensor([[[ -7.2395,  -7.2337,  -7.2301,  ...,  -6.6463,  -6.5081,  -4.4686],
         [ -8.1057,  -8.1946,  -8.0791,  ...,  -8.4518,  -7.6345,  -5.3930],
         [-10.7883, -10.6919, -10.5438,  ...,  -9.9607, -10.0536,  -6.8828],
         ...,
         [ -9.0698,  -9.3295,  -8.9949,  ...,  -6.1696,  -7.4357,  -7.4828],
         [ -6.3161,  -6.4182,  -6.5455,  ...,  -5.5366,  -5.7362,  -2.2207],
         [-12.0209, -11.9511, -12.0039,  ..., -11.8723,  -9.6545,  -8.2306]]],
       device='cuda:0')

changed str
"tensor([[[ -7.2395,  -7.2337,  -7.2301,  ...,  -6.6463,  -6.5081,  -4.4686],\n         [ -8.1057,  -8.1946,  -8.0791,  ...,  -8.4518,  -7.6345,  -5.3930],\n         [-10.7883, -10.6919, -10.5438,  ...,  -9.9607, -10.0536,  -6.8828],\n         ...,\n         [ -9.0698,  -9.3295,  -8.9949,  ...,  -6.1696,  -7.4357,  -7.4828],\n         [ -6.3161,  -6.4182,  -6.5455,  ...,  -5.5366,  -5.7362,  -2.2207],\n         [-12.0209, -11.9511, -12.0039,  ..., -11.8723,  -9.6545,  -8.2306]]],\n       device='cuda:0')"


title: How to calculate cost for softmax regression with pytorch

body:
I would to calculate the cost for the softmax regression.
The cost function to calculate is given at the bottom of the page.

For numpy I can get the cost as follows:

"""
X.shape = 2,300 # floats
y.shape = 300,  # integers
W.shape = 2,3
b.shape = 3,1
"""
import numpy as np
np.random.seed(100)

# Data and labels
X = np.random.randn(300,2)
y = np.ones(300)
y[0:100] = 0
y[200:300] = 2
y = y.astype(np.int)

# weights and bias
W = np.random.randn(2,3)
b = np.random.randn(3)

N = X.shape[0]
scores = np.dot(X, W) + b
hyp = np.exp(scores-np.max(scores, axis=0, keepdims=True))
probs = hyp / np.sum(hyp, axis = 0)
logprobs = np.log(probs[range(N),y])
cost_data = -1/N * np.sum(logprobs)

print("hyp.shape = {}".format(hyp.shape)) # hyp.shape = (300, 3)
print(cost_data)

But, when I tried torch I could not get this.
So far I have got this:

"""
X.shape = 2,300 # floats
y.shape = 300,  # integers
W.shape = 2,3
b.shape = 3,1
"""
import numpy as np
import torch
from torch.autograd import Variable
np.random.seed(100)

# Data and labels
X = np.random.randn(300,2)
y = np.ones(300)
y[0:100] = 0
y[200:300] = 2
y = y.astype(np.int)

X = Variable(torch.from_numpy(X),requires_grad=True).type(torch.FloatTensor)
y = Variable(torch.from_numpy(y),requires_grad=True).type(torch.LongTensor)

# weights and bias
W = Variable(torch.randn(2,3),requires_grad=True)
b = Variable(torch.randn(3),requires_grad=True)

N = X.shape[0]
scores = torch.mm(X, W) + b
hyp = torch.exp(scores - torch.max(scores))
probs = hyp / torch.sum(hyp)
correct_probs = probs[range(N),y] # got problem HERE
# logprobs = np.log(correct_probs)
# cost_data = -1/N * torch.sum(logprobs)

# print(cost_data)

I got problem calculating the correct probabilities for the classes.

How can we solve this problem and get the correct cost value.

The cost function to calculate is given below:


title: GPU not available in Spyder

body:
A python program (PyTorch, via Spyder, running two GPUs) was running smoothly and using the two GPUs. Liunx-Ubunto-16.04 went suspended, due to locking the screen; then after log-ing, Spyder detected 0 GPUs. The GPUs, nonetheless, were available at the terminal ($ nvidia-smi). I had to restart the computer to get the GPUs back into Spyder.

Maybe there is a better way to initialize the GPUs, aside from managing the power settings, as the GPUs might be toggled off for other reasons.

import torch
torch.cuda.is_available()
Out[4]: False


title: Problems with relative import

body:
I run a file called test.py which has an import like this

from .nms.cpu_nms import cpu_nms, cpu_soft_nms
from .nms.gpu_nms import gpu_nms

When I run the file though, I get this error:

Traceback (most recent call last):
  File "test.py", line 9, in <module>
    from utils.nms_wrapper import nms
  File /media/ryan/shakira/InsightFace_Pytorch/FaceBoxes.PyTorch/utils/nms_wrapper.py",     line 7, in <module>
    from .nms.cpu_nms import cpu_nms, cpu_soft_nms
ModuleNotFoundError: No module named 'utils.nms.cpu_nms'

I have tried doing

sys.path.append('/path/to/the/main/directory/')

But that too does not work,

EDIT:

This is my directory structure:

FaceBoxes.PyTorch/
├── data
│   ├── AFW
│   │   └── img_list.txt
│   ├── config.py
│   ├── data_augment.py
│   ├── FDDB
│   │   └── img_list.txt
│   ├── __init__.py
│   ├── PASCAL
│   │   └── img_list.txt
│   ├── __pycache__
│   │   ├── config.cpython-36.pyc
│   │   ├── data_augment.cpython-36.pyc
│   │   ├── __init__.cpython-36.pyc
│   │   └── wider_voc.cpython-36.pyc
│   ├── WIDER_FACE
│   │   └── img_list.txt
│   └── wider_voc.py
├── layers
│   ├── functions
│   │   ├── prior_box.py
│   │   └── __pycache__
│   │       └── prior_box.cpython-36.pyc
│   ├── __init__.py
│   ├── modules
│   │   ├── __init__.py
│   │   ├── multibox_loss.py
│   │   └── __pycache__
│   │       ├── __init__.cpython-36.pyc
│   │       └── multibox_loss.cpython-36.pyc
│   └── __pycache__
│       └── __init__.cpython-36.pyc
├── LICENSE
├── make.sh
├── models
│   ├── faceboxes.py
│   └── __init__.py
├── README.md
├── test.py
├── train.py
└── utils
    ├── box_utils.py
    ├── build.py
    ├── __init__.py
    ├── nms
    │   ├── cpu_nms.c
    │   ├── cpu_nms.pyx
    │   ├── gpu_nms.cpp
    │   ├── gpu_nms.hpp
    │   ├── gpu_nms.pyx
    │   ├── __init__.py
    │   ├── nms_kernel.cu
    │   ├── __pycache__
    │   │   └── __init__.cpython-36.pyc
    │   └── py_cpu_nms.py
    ├── nms_wrapper.py
    ├── __pycache__
    │   ├── box_utils.cpython-36.pyc
    │   ├── __init__.cpython-36.pyc
    │   └── nms_wrapper.cpython-36.pyc
    └── timer.py

Any suggestions would be really helpful, Thanks in advance.


title: Trouble creating 3D rotation matrix in Pytorch - ValueError: only one element tensors can be converted to Python scalars

body:
I am trying to create 3D rotation matrices in pytorch as seen on the first page of this pdf, but I am encountering some problems. Here is my code so far:

zero = torch.from_numpy(np.zeros(len(cos)))
one = torch.from_numpy(np.ones(len(cos)))
R_transpose = torch.tensor([cos, -sin, zero, sin, cos, zero, zero, zero, one]).reshape(-1, 3, 3)

The cos and sin are matrices that look like this:

tensor([[[1.]],

    [[1.]],

    [[1.]],

    [[1.]],

    [[1.]]], dtype=torch.float64)

My goal is to create x number of rotation matrices(e.g. four matrices with the cos values shown above).

The code I currently have results in a "ValueError: only one element tensors can be converted to Python scalars"

How should I change my code to achieve my goal?


title: PyTorch: Relation between Dynamic Computational Graphs - Padding - DataLoader

body:
As far as I understand, the strength of PyTorch is supposed to be that it works with dynamic computational graphs. In the context of NLP, that means that sequences with variable lengths do not necessarily need to be padded to the same length. But, if I want to use PyTorch DataLoader, I need to pad my sequences anyway because the DataLoader only takes tensors - given that me as a total beginner does not want to build some customized collate_fn.

Now this makes me wonder - doesn’t this wash away the whole advantage of dynamic computational graphs in this context?
Also, if I pad my sequences to feed it into the DataLoader as a tensor with many zeros as padding tokens at the end (in the case of word ids), will it have any negative effect on my training since PyTorch may not be optimized for computations with padded sequences (since the whole premise is that it can work with variable sequence lengths in the dynamic graphs), or does it simply not make any difference?

I will also post this question in the PyTorch Forum...

Thanks!


title: RNN not training when batch size > 1 with variable length data

body:
I'm implementing a simple RNN network which predicts 1/0 for some variable length
 time-series data. The network would first feed the training data into an LSTM cell, and then use a linear layer for classification.

Usually, we would use mini-batches to train the network. But, the problem is that this simple RNN network is not training when I use batch_size > 1.

I manage to create a minimal code sample which can reproduce the problem. If you set batch_size=1 at line 95, the network trains successfully, but if you set batch_size=2, the network is not training at all, the losses just bouncing around. (requires python3, pytorch >= 0.4.0)

import numpy as np
import random
import torch
import torch.nn as nn
import torch.optim as optim
from torch.nn.utils.rnn import pad_sequence, pack_padded_sequence

class ToyDataLoader(object):

    def __init__(self, batch_size):
        self.batch_size = batch_size
        self.index = 0
        self.dataset_size = 10

        # generate 10 random variable length training samples,
        # each time step has 1 feature dimension
        self.X = [
            [[1], [1], [1], [1], [0], [0], [1], [1], [1]],
            [[1], [1], [1], [1]],
            [[0], [0], [1], [1]],
            [[1], [1], [1], [1], [1], [1], [1]],
            [[1], [1]],
            [[0]],
            [[0], [0], [0], [0], [0], [0], [0]],
            [[1]],
            [[0], [1]],
            [[1], [0]]
        ]

        # assign labels for the toy traning set
        self.y = torch.LongTensor([0, 0, 0, 0, 0, 1, 1, 1, 1, 1])

    def __len__(self):
        return self.dataset_size // self.batch_size

    def __iter__(self):
        return self

    def __next__(self):
        if self.index + self.batch_size > self.dataset_size:
            self.index = 0
            raise StopIteration()
        if self.index == 0:  # shufle the dataset
            tmp = list(zip(self.X, self.y))
            random.shuffle(tmp)
            self.X, self.y = zip(*tmp)
            self.y = torch.LongTensor(self.y)
        X = self.X[self.index: self.index + self.batch_size]
        y = self.y[self.index: self.index + self.batch_size]
        self.index += self.batch_size
        return X, y

class NaiveRNN(nn.Module):
    def __init__(self):
        super(NaiveRNN, self).__init__()
        self.lstm = nn.LSTM(1, 128)
        self.linear = nn.Linear(128, 2)

    def forward(self, X):
        '''
        Parameter:
            X: list containing variable length training data
        '''

        # get the length of each seq in the batch
        seq_lengths = [len(x) for x in X]

        # convert to torch.Tensor
        seq_tensor = [torch.Tensor(seq) for seq in X]

        # sort seq_lengths and seq_tensor based on seq_lengths, required by torch.nn.utils.rnn.pad_sequence
        pairs = sorted(zip(seq_lengths, seq_tensor),
                       key=lambda pair: pair[0], reverse=True)
        seq_lengths = torch.LongTensor([pair[0] for pair in pairs])
        seq_tensor = [pair[1] for pair in pairs]

        # padded_seq shape: (seq_len, batch_size, feature_size)
        padded_seq = pad_sequence(seq_tensor)

        # pack them up
        packed_seq = pack_padded_sequence(padded_seq, seq_lengths.numpy())

        # feed to rnn
        packed_output, (ht, ct) = self.lstm(packed_seq)

        # linear classification layer
        y_pred = self.linear(ht[-1])

        return y_pred

def main():
    trainloader = ToyDataLoader(batch_size=2)  # not training at all! !!
    # trainloader = ToyDataLoader(batch_size=1) # it converges !!!

    model = NaiveRNN()
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.Adadelta(model.parameters(), lr=1.0)

    for epoch in range(30):
        # switch to train mode
        model.train()

        for i, (X, labels) in enumerate(trainloader):

            # compute output
            outputs = model(X)
            loss = criterion(outputs, labels)

            # measure accuracy and record loss
            _, predicted = torch.max(outputs, 1)
            accu = (predicted == labels).sum().item() / labels.shape[0]

            # compute gradient and do SGD step
            optimizer.zero_grad()
            loss.backward()

            optimizer.step()

            print('Epoch: [{}][{}/{}]\tLoss {:.4f}\tAccu {:.3f}'.format(
                epoch, i, len(trainloader), loss, accu))

if __name__ == '__main__':
    main()

Sample output when batch_size=1:

...
Epoch: [28][7/10]       Loss 0.1582     Accu 1.000
Epoch: [28][8/10]       Loss 0.2718     Accu 1.000
Epoch: [28][9/10]       Loss 0.0000     Accu 1.000
Epoch: [29][0/10]       Loss 0.2808     Accu 1.000
Epoch: [29][1/10]       Loss 0.0000     Accu 1.000
Epoch: [29][2/10]       Loss 0.0001     Accu 1.000
Epoch: [29][3/10]       Loss 0.0149     Accu 1.000
Epoch: [29][4/10]       Loss 0.1445     Accu 1.000
Epoch: [29][5/10]       Loss 0.2866     Accu 1.000
Epoch: [29][6/10]       Loss 0.0170     Accu 1.000
Epoch: [29][7/10]       Loss 0.0869     Accu 1.000
Epoch: [29][8/10]       Loss 0.0000     Accu 1.000
Epoch: [29][9/10]       Loss 0.0498     Accu 1.000

Sample output when batch_size=2:

...
Epoch: [27][2/5]        Loss 0.8051     Accu 0.000
Epoch: [27][3/5]        Loss 1.2835     Accu 0.000
Epoch: [27][4/5]        Loss 1.0782     Accu 0.000
Epoch: [28][0/5]        Loss 0.5201     Accu 1.000
Epoch: [28][1/5]        Loss 0.6587     Accu 0.500
Epoch: [28][2/5]        Loss 0.3488     Accu 1.000
Epoch: [28][3/5]        Loss 0.5413     Accu 0.500
Epoch: [28][4/5]        Loss 0.6769     Accu 0.500
Epoch: [29][0/5]        Loss 1.0434     Accu 0.000
Epoch: [29][1/5]        Loss 0.4460     Accu 1.000
Epoch: [29][2/5]        Loss 0.9879     Accu 0.000
Epoch: [29][3/5]        Loss 1.0784     Accu 0.500
Epoch: [29][4/5]        Loss 0.6051     Accu 1.000

I've searched a lot of materials and still can't figure out why.


title: KeyError: 'predictions' When use SimpleSeq2SeqPredictor to predict string

body:
Please first search our GitHub repository for similar questions.  If you don't find a similar example you can use the following template:

System (please complete the following information):
 - OS: Ubunti 18.04
 - Python version: 3.6.7
 - AllenNLP version: v0.8.3
 - PyTorch version: 1.1.0

Question
When I Try to predict string using SimpleSeq2SeqPredictor, It always show that

Traceback (most recent call last):
  File "predict.py", line 96, in <module>
    p = predictor.predict(i)
  File "venv/lib/python3.6/site-packages/allennlp/predictors/seq2seq.py", line 17, in predict
    return self.predict_json({"source" : source})
  File "/venv/lib/python3.6/site-packages/allennlp/predictors/predictor.py", line 56, in predict_json
    return self.predict_instance(instance)
  File "/venv/lib/python3.6/site-packages/allennlp/predictors/predictor.py", line 93, in predict_instance
    outputs = self._model.forward_on_instance(instance)
  File "/venv/lib/python3.6/site-packages/allennlp/models/model.py", line 124, in forward_on_instance
    return self.forward_on_instances([instance])[0]
  File "/venv/lib/python3.6/site-packages/allennlp/models/model.py", line 153, in forward_on_instances
    outputs = self.decode(self(**model_input))
  File "/venv/lib/python3.6/site-packages/allennlp/models/encoder_decoders/simple_seq2seq.py", line 247, in decode
    predicted_indices = output_dict["predictions"]
KeyError: 'predictions'

I try to do a translate system, but I am newbie, most of code come from
https://github.com/mhagiwara/realworldnlp/blob/master/examples/mt/mt.py
http://www.realworldnlpbook.com/blog/building-seq2seq-machine-translation-models-using-allennlp.html

this is my training code

EN_EMBEDDING_DIM = 256
ZH_EMBEDDING_DIM = 256
HIDDEN_DIM = 256
CUDA_DEVICE = 0
prefix = 'small'

reader = Seq2SeqDatasetReader(
    source_tokenizer=WordTokenizer(),
    target_tokenizer=CharacterTokenizer(),
    source_token_indexers={'tokens': SingleIdTokenIndexer()},
    target_token_indexers={'tokens': SingleIdTokenIndexer(namespace='target_tokens')},
    lazy = True)
train_dataset = reader.read(f'./{prefix}-data/train.tsv')
validation_dataset = reader.read(f'./{prefix}-data/val.tsv')

vocab = Vocabulary.from_instances(train_dataset,
                                    min_count={'tokens': 3, 'target_tokens': 3})

en_embedding = Embedding(num_embeddings=vocab.get_vocab_size('tokens'),
                            embedding_dim=EN_EMBEDDING_DIM)
# encoder = PytorchSeq2SeqWrapper(
#     torch.nn.LSTM(EN_EMBEDDING_DIM, HIDDEN_DIM, batch_first=True))
encoder = StackedSelfAttentionEncoder(input_dim=EN_EMBEDDING_DIM, hidden_dim=HIDDEN_DIM, projection_dim=128, feedforward_hidden_dim=128, num_layers=1, num_attention_heads=8)

source_embedder = BasicTextFieldEmbedder({"tokens": en_embedding})

# attention = LinearAttention(HIDDEN_DIM, HIDDEN_DIM, activation=Activation.by_name('tanh')())
# attention = BilinearAttention(HIDDEN_DIM, HIDDEN_DIM)
attention = DotProductAttention()

max_decoding_steps = 20   # TODO: make this variable
model = SimpleSeq2Seq(vocab, source_embedder, encoder, max_decoding_steps,
                        target_embedding_dim=ZH_EMBEDDING_DIM,
                        target_namespace='target_tokens',
                        attention=attention,
                        beam_size=8,
                        use_bleu=True)
optimizer = optim.Adam(model.parameters())
iterator = BucketIterator(batch_size=32, sorting_keys=[("source_tokens", "num_tokens")])

iterator.index_with(vocab)
if torch.cuda.is_available():
    cuda_device = 0
    model = model.cuda(cuda_device)
else:
    cuda_device = -1
trainer = Trainer(model=model,
                    optimizer=optimizer,
                    iterator=iterator,
                    train_dataset=train_dataset,
                    validation_dataset=validation_dataset,
                    num_epochs=50,
                    serialization_dir=f'ck/{prefix}/',
                    cuda_device=cuda_device)

# for i in range(50):
    # print('Epoch: {}'.format(i))
trainer.train()

predictor = SimpleSeq2SeqPredictor(model, reader)

for instance in itertools.islice(validation_dataset, 10):
    print('SOURCE:', instance.fields['source_tokens'].tokens)
    print('GOLD:', instance.fields['target_tokens'].tokens)
    print('PRED:', predictor.predict_instance(instance)['predicted_tokens'])

# Here's how to save the model.
with open(f"ck/{prefix}/manually_save_model.th", 'wb') as f:
    torch.save(model.state_dict(), f)
vocab.save_to_files(f"ck/{prefix}/vocabulary")

and this is my predict code

EN_EMBEDDING_DIM = 256
ZH_EMBEDDING_DIM = 256
HIDDEN_DIM = 256
CUDA_DEVICE = 0
prefix = 'big'

reader = Seq2SeqDatasetReader(
    source_tokenizer=WordTokenizer(),
    target_tokenizer=CharacterTokenizer(),
    source_token_indexers={'tokens': SingleIdTokenIndexer()},
    target_token_indexers={'tokens': SingleIdTokenIndexer(namespace='target_tokens')},
    lazy = True)
# train_dataset = reader.read(f'./{prefix}-data/train.tsv')
# validation_dataset = reader.read(f'./{prefix}-data/val.tsv')

# vocab = Vocabulary.from_instances(train_dataset,
#                                     min_count={'tokens': 3, 'target_tokens': 3})
vocab = Vocabulary.from_files("ck/small/vocabulary")

en_embedding = Embedding(num_embeddings=vocab.get_vocab_size('tokens'),
                            embedding_dim=EN_EMBEDDING_DIM)
# encoder = PytorchSeq2SeqWrapper(
#     torch.nn.LSTM(EN_EMBEDDING_DIM, HIDDEN_DIM, batch_first=True))
encoder = StackedSelfAttentionEncoder(input_dim=EN_EMBEDDING_DIM, hidden_dim=HIDDEN_DIM, projection_dim=128, feedforward_hidden_dim=128, num_layers=1, num_attention_heads=8)

source_embedder = BasicTextFieldEmbedder({"tokens": en_embedding})

# attention = LinearAttention(HIDDEN_DIM, HIDDEN_DIM, activation=Activation.by_name('tanh')())
# attention = BilinearAttention(HIDDEN_DIM, HIDDEN_DIM)
attention = DotProductAttention()

max_decoding_steps = 20   # TODO: make this variable
model = SimpleSeq2Seq(vocab, source_embedder, encoder, max_decoding_steps,
                        target_embedding_dim=ZH_EMBEDDING_DIM,
                        target_namespace='target_tokens',
                        attention=attention,
                        beam_size=8,
                        use_bleu=True)

# And here's how to reload the model.
with open("./ck/small/best.th", 'rb') as f:
    model.load_state_dict(torch.load(f))

predictor = Seq2SeqPredictor(model, dataset_reader=reader)
# print(predictor.predict("The dog ate the apple"))

test = [
    'Surely ,he has no power over those who believe and put their trust in their Lord ;',
    'And assuredly We have destroyed the generations before you when they did wrong ,while their apostles came unto them with the evidences ,and they were not such as to believe . In this wise We requite the sinning people .',
    'And warn your tribe ( O Muhammad SAW ) of near kindred .',
    'And to the Noble Messengers whom We have mentioned to you before ,and to the Noble Messengers We have not mentioned to you ; and Allah really did speak to Moosa .',
    'It is He who gave you hearing ,sight ,and hearts ,but only few of you give thanks .',
    'spreading in them much corruption ?',
    'That will envelop the people . This will be a painful punishment .',
    'When you received it with your tongues and spoke with your mouths what you had no knowledge of ,and you deemed it an easy matter while with Allah it was grievous .',
    'of which you are disregardful .',
    'Whoever disbelieves ,then the calamity of his disbelief is only on him ; and those who do good deeds ,are preparing for themselves .'
]

for i in test:
    p = predictor.predict(i) # <------------------- ERROR !!!!!!!
    print(p)

Am I do something wrong ?


title: How to get the correct shape of the tensor in custom dataset

body:
I am using custom Dataset class, but the problem is that when I get the data from the Dataloader I am left with array that has different tensor shape than I want.

shape that I get:       torch.Size([1, 56, 128, 128])
shape that I want:    torch.Size([1, 56, 1, 128, 128])

my approach was to:

1) to apply numpy.expand_dims on the array and get   torch.Size([1, 1, 56, 128, 128])

2) then to np.transpose on the array to get the shape I want torch.Size([1, 56, 1, 128, 128])

after first step I am getting the error:

raise ValueError('pic should be 2/3 dimensional. Got {} dimensions.'.format(pic.ndim))
ValueError: pic should be 2/3 dimensional. Got 4 dimensions.

if I do the transpose first, none of the combination of np.transpose(array, axes=(1,2,0)) yields the shape torch.Size([56, 1, 128, 128])

if I convert array to Tensor first and then do torch.unsqueeze I get the error:

raise TypeError('pic should be PIL Image or ndarray. Got {}'.format(type(pic)))
TypeError: pic should be PIL Image or ndarray. Got <class 'torch.Tensor'>

here is my code:

class patientdataset(Dataset):
    def __init__(self, csv_file, root_dir, transform=None):
        self.annotations = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.annotations)

    def __getitem__(self, index):
        img_path = os.path.join(self.root_dir, self.annotations.iloc[index,0])
        # np_load_old = np.load
        # np.load = lambda *a, **k: np_load_old(*a, allow_pickle=True, **k)

        image= np.asarray(np.load(img_path))

        image= np.transpose(image, axes=(1,2,0))
        image = torch.Tensor (image)

        image = torch.unsqueeze(image, dim=1)

        y_label = torch.tensor(np.asarray(self.annotations.iloc[index,1]))

        if self.transform:
            image = self.transform(image)

        return (image, y_label)


title: Use only certain layers of pretrained torchvision network

body:
I'm trying to use only certain layers in a pretrained torchvision Faster-RCNN network initialized by:

model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()

This works. However, passing model.modules() or model.children() into an nn.Sequential yields an error. Even passing the whole model leads to errors, e.g.

model = torch.nn.Sequential(*model.modules())
model.eval()
# x is a [C, H, W] image
y = model(x)

leads to

AttributeError: 'dict' object has no attribute 'dim'

and

model = torch.nn.Sequential(*model.children())
model.eval()
# x is a [C, H, W] image
y = model(x)

leads to

TypeError: conv2d(): argument 'input' (position 1) must be Tensor, not tuple

This confuses me because I have modified other PyTorch pretrained models like that in the past. How can I use the FasterRCNN pretrained model to create a new (pretrained) model that uses only certain layers, e.g. all layers but the last one?


title: Pytorch differentiable conditional (index-based) sum

body:
I have an idx array like [0, 1, 0, 2, 3, 1] and another 2d array data like the following:

[[0,  1,  2],
 [3,  4,  5],
 [6,  7,  8],
 [9,  10, 11],
 [12, 13, 14],
 [15, 16, 17]]

I want my output to be 4x3 in which 4 is the max of idx and 3 is the feature size (data.shape[1]) and in the output each element is the sum of features with corresponding index in the idx. Then output in this example would be:

[[6,  8,  10],
 [18, 20, 22],
 [9,  10, 11],
 [12, 13, 14]]

I can do it with iterating over range(3) and creating a mask on data and summing up them but it's not differentiable (I suppose). Is there any function(s) in Pytorch for this purpose? something like scatter().

Update: It seems I am looking for something named scatter sum which is implemented in this repository.


title: Differentiable image compression operations in PyTorch

body:
During a CNN classification model training while calculating the loss I am applying the encoding jpeg compression on the image in PyTorch. While I call loss.backward() it must also backpropagate through encoding and compression operation performed on the images.

Are those compression algorithms (e.g. encoding and JPEG compression) are differentiable otherwise how to backpropagate the loss gradient through those operations?

If those operations are not differentiable is there any differentiable compression algorithm that exists in PyTorch which performs H.264 encoding and JPEG compression?

Any suggestions will be highly helpful.


title: Pytorch C++ (libtorch) outputs different results if I change shape

body:
So I'm learning Neural networks right now and I noticed something really really strange in my network.
I have an input layer created like this

convN1 = register_module("convN1", torch::nn::Conv2d(torch::nn::Conv2dOptions(4, 256, 3).padding(1)));

and an output layer that is a tanh function.

So it's expecting a torch::Tensor of shape
{/batchSize/, 4, /sideLength/, /sideLength/} which is and will output a tensor of just 1 float value.

As such for testing I have created a custom Tensor of shape {4, 15, 15}.

The really weird part is what's happening below

auto inputTensor = torch::zeros({ 1, 4, 15, 15});
inputTensor[0] = customTensor;
std::cout << network->forward(inputTensor); // Outputs something like 0.94142

inputTensor = torch::zeros({ 32, 4, 15, 15});
inputTensor[0] = customTensor;
std::cout << network->forward(inputTensor); // Outputs something like 0.1234 then 0.8543 31 times

So why is the customTensor getting 2 different values from my network just from the fact that the batchsize has changed? Am I not understanding some parts of how tensors work?

P.S. I did check and the above block of code was operating under eval mode.

Edit: Since it's been asked here's a more indepth look at my network

convN1 = register_module("convN1", torch::nn::Conv2d(torch::nn::Conv2dOptions(4, 256, 3).padding(1)));
batchNorm1 = register_module("batchNorm1", torch::nn::BatchNorm2d(torch::nn::BatchNormOptions(256)));

m_residualBatch1 = register_module(batch1Name, torch::nn::BatchNorm2d(torch::nn::BatchNormOptions(256)));
m_residualBatch2 = register_module(batch2Name, torch::nn::BatchNorm2d(torch::nn::BatchNormOptions(256)));
m_residualConv1 = register_module(conv1Name, torch::nn::Conv2d(torch::nn::Conv2dOptions(256, 256, 3).padding(1)));
m_residualConv2 = register_module(conv2Name, torch::nn::Conv2d(torch::nn::Conv2dOptions(256, 256, 3).padding(1)));

valueN1 = register_module("valueN1", torch::nn::Conv2d(256, 2, 1));
batchNorm3 = register_module("batchNorm3", torch::nn::BatchNorm2d(torch::nn::BatchNormOptions(2)));
valueN2 = register_module("valueN2", torch::nn::Linear(2 * BOARD_LENGTH, 64));
valueN3 = register_module("valueN3", torch::nn::Linear(64, 1));

And how it forwards is like so

torch::Tensor Net::forwadValue(torch::Tensor x)
{
    x = convN1->forward(x);
    x = batchNorm1->forward(x);
    x = torch::relu(x);

    torch::Tensor residualCopy = x.clone();
    x = m_residualConv1->forward(x);
    x = m_residualBatch1->forward(x);
    x = torch::relu(x);
    x = m_residualConv2->forward(x);
    x = m_residualBatch2->forward(x);
    x += residualCopy;
    x = torch::relu(x);

    x = valueN1->forward(x);
    x = batchNorm3->forward(x)
    x = torch::relu(x);
    x = valueN2->forward(x.reshape({ x.sizes()[0], 30 }))
    x = torch::relu(x);
    x = valueN3->forward(x)
    return torch::tanh(x);
}


title: How to extract images, labels from csv file and create a trainset using torch?

body:
I downloaded a dataset for facial key point detection the image and the labels were in a CSV file I extracted it using pandas but I don't know how to convert it into a tensor and load it into a data loader for training.

dataframe = pd.read_csv("training_facial_keypoints.csv")
dataframe['Image'] = dataframe['Image'].apply(lambda i: np.fromstring(i, sep=' '))
dataframe= dataframe.dropna()
images_array = np.vstack(dataframe['Image'].values)/255.0
images_array = images_array.astype(np.float32)
images_array = images_array.reshape(-1, 96, 96, 1)
print(images_array.shape)
labels_array = dataframe[dataframe.columns[:-1]].values
labels_array = (labels_array-48)/48
labels_array = labels_array.astype(np.float32)

I have the images and labels in two arrays. How do I create a training set from this and use transforms.
Then load it using a dataloader.


images, labels = next(iter(self.loader))
grid = torchvision.utils.make_grid(images)

  images, labels = next(iter(self.loader))

triggers the error.

I have a custom dataset class where I load each image (RGB) from an url :

  image = Image.open(urllib.request.urlopen(URL))

and I apply some albumentations transforms.

The code works when I read an image for which I have a path using cv2.
However, it doesn't work when I read an image from the url.
Note that I verified that the urls aren't broken.

Here's the traceback:

/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py in __next__(self)
    344     def __next__(self):
    345         index = self._next_index()  # may raise StopIteration
--> 346         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    347         if self._pin_memory:
    348             data = _utils.pin_memory.pin_memory(data)

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/content/transform_dataset.py in __getitem__(self, idx)
     49     labels = torch.from_numpy(item[2:].values.astype("float32"))
     50     #print("self.root,item,self.image_transform,self.transform,self.size", self.root,item,self.image_transform,self.transform,self.size)
---> 51     image = load_image(self.root,item.ID,item.URL,self.image_transform)
     52     return image, labels
     53

/content/transform_dataset.py in load_image(root, ID, URL, image_transform)
     81     print(image.shape)
     82     image = cv2.cvtColor(image,cv2.COLOR_BGR2RGB)
---> 83   image = image_transform(image=image)["image"]
     84   return image

/usr/local/lib/python3.6/dist-packages/albumentations/core/composition.py in __call__(self, **data)
    169                                               convert_keypoints_to_albumentations, data)
    170
--> 171             data = t(**data)
    172
    173             if dual_start_end is not None and idx == dual_start_end[1]:

/usr/local/lib/python3.6/dist-packages/albumentations/core/transforms_interface.py in __call__(self, **kwargs)
     26         if (random.random() < self.p) or self.always_apply:
     27             params = self.get_params()
---> 28             params = self.update_params(params, **kwargs)
     29             if self.targets_as_params:
     30                 targets_as_params = {k: kwargs[k] for k in self.targets_as_params}

/usr/local/lib/python3.6/dist-packages/albumentations/core/transforms_interface.py in update_params(self, params, **kwargs)
     66         if hasattr(self, 'interpolation'):
     67             params['interpolation'] = self.interpolation
---> 68         params.update({'cols': kwargs['image'].shape[1], 'rows': kwargs['image'].shape[0]})
     69         return params
     70

AttributeError: 'MpoImageFile' object has no attribute 'shape'


title: With a PyTorch LSTM, can I have a different hidden_size than input_size?

body:
I have:

    def __init__(self, feature_dim=15, hidden_size=5, num_layers=2):
        super(BaselineModel, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size

        self.lstm = nn.LSTM(input_size=feature_dim,
                            hidden_size=hidden_size, num_layers=num_layers)

and then I get an error:

RuntimeError: The size of tensor a (5) must match the size of tensor b (15) at non-singleton dimension 2

If I set the two sizes to be the same, then the error goes away. But I'm wondering if my input_size is some large number, say 15, and I want to reduce the number of hidden features to 5, why shouldn't that work?


title: Train Accuracy increases, Train loss is stable, Validation loss Increases, Validation Accuracy is low and increases

body:
My neural network trainign in pytorch is getting very wierd.

I am training a known dataset that came splitted into train and validation.
I'm shuffeling the data during training and do data augmentation on the fly.

I have those results:

Train accuracy start at 80% and increases

Train loss decreases and stays stable

Validation accuracy start at 30% but increases slowly

Validation loss increases

I have the following graphs to show:

How can you explain that the validation loss increases and the validation accuracy increases?
How can be such a big difference of accuracy between validation and training sets?  90% and 40%?

Update:

I balanced the data set.
It is binary classification. It now has now 1700 examples from class 1, 1200 examples from class 2. Total 600 for validation and 2300 for training.
I still see similar behavior:

**Can it be becuase I froze the weights in part of the network?

**Can it be becuase the hyperparametrs like lr?


title: Best way to define a scalar using nn.Parameter in Pytorch

body:
In my CNN at some stage I want to multiply a feature map with some scalar which should be learnt by the network. Which of the following is the best way to do it or all are same? The scalar has to be initialised to 5.
# Method 1
def __init__(self):
    super(..., self).__init__()
    ...
    ...
    alpha = nn.Parameter(5)
    ...

def forward(self, x):
    ...
    x = x * alpha
    return x

# Method 2
def __init__(self):
    super(..., self).__init__()
    ...
    ...
    alpha = nn.Parameter(torch.tensor(5))
    ...

def forward(self, x):
    ...
    x = x * alpha
    return x

# Method 3
def __init__(self):
    super(..., self).__init__()
    ...
    ...
    alpha = nn.Parameter(torch.ones(1)*5)
    ...

def forward(self, x):
    ...
    x = x * alpha
    return x

If all are same I would prefer Method 1 and let CNN learn the appropriate multiplier alpha for the feature map x. I hope in all cases alpha will be a float32 tensor initialised to 5. I am using PyTorch 1.3.1
Yours sincerely,
Mohit


title: Activation gradient penalty

body:
Here's a simple neural network, where I’m trying to penalize the norm of activation gradients:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=5)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=5)
        self.pool = nn.MaxPool2d(2, 2)
        self.relu = nn.ReLU()
        self.linear = nn.Linear(64 * 5 * 5, 10)

    def forward(self, input):
        conv1 = self.conv1(input)
        pool1 = self.pool(conv1)
        self.relu1 = self.relu(pool1)
        self.relu1.retain_grad()
        conv2 = self.conv2(relu1)
        pool2 = self.pool(conv2)
        relu2 = self.relu(pool2)
        self.relu2 = relu2.view(relu2.size(0), -1)
        self.relu2.retain_grad()
        return self.linear(relu2)

model = Net()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)

for i in range(1000):
    output = model(input)
    loss = nn.CrossEntropyLoss()(output, label)
    optimizer.zero_grad()
    loss.backward(retain_graph=True)

    grads = torch.autograd.grad(loss, [model.relu1, model.relu2], create_graph=True)

    grad_norm = 0
    for grad in grads:
        grad_norm += grad.pow(2).sum()

    grad_norm.backward()

    optimizer.step()

However, it does not produce the desired regularization effect. If I do the same thing for weights (instead of activations), it works well. Am I doing this right (in terms of pytorch machinery)? Specifically, what happens in grad_norm.backward() call? I just want to make sure the weight gradients are updated, and not activation gradients. Currently, when I print out gradients for weights and activations immediately before and after that line, both change - so I’m not sure what’s going on.


title: How to get sentence embeddings from encoder in Fastai learner language model

body:
I was able to fine tune a language model using fast ai.  I would like to extract sentence embeddings from the fine-tuned model for sentence similarity.  How do I get the encoder model embeddings? Also can embeddings be compared with dot product like other embeddings from other models such as USE?

data_lm = TextLMDataBunch.from_df(train_df = se1, valid_df = se2, path = "",text_cols='text')
learn = language_model_learner(data_lm,drop_mult=0.7,pretrained=True,arch=AWD_LSTM)
learn.fit_one_cycle(3, 1e-01)

My code is above how can I get encodings from learn?


title: LSTM after CNN how to feed in and what dimensions (input size)?

body:
I am trying to build a Convolutional Recurrent Network that takes a fixed input size of 32000 x 1 raw audio time series. It's just a numpy array of length 32000. So for example if we have a batch size of 1, my dimensions would be

torch.size([1,32000,1])

1 audio time series, 32000 in length, 1 channel

I had to reshape that to be

torch.size([1,1,32000])

so that it could be fed into the convolutional layer

As this is fed into Convolutional / Pooling layers my dimensions change as follows

torch.Size([1, 1, 32000])
torch.Size([1, 64, 7981])
torch.Size([1, 64, 1995])
torch.Size([1, 64, 1993])
torch.Size([1, 64, 498])
torch.Size([1, 128, 496])
torch.Size([1, 128, 494])
torch.Size([1, 128, 123])
torch.Size([1, 256, 121])
torch.Size([1, 256, 119])
torch.Size([1, 256, 29])
torch.Size([1, 512, 27])
torch.Size([1, 512, 25])
torch.Size([1, 512, 6])

So after the final max pooling layer, the size of the data is

torch.Size([1, 512, 6])

How do I go about this output and take it into the LSTM?

How would we feed this result into the LSTM? would it involve flattening our output from the last max pooling layer? Or is the input size for the LSTM going to be 512 or 6?

Please advise and thank you.

LSTM expects a 3D input shape of #Samples (audio time series), # Time steps (32000) , # features


title: PyTorch not detecting CUDA in Amazon Deep learning AMI

body:
I am using Amazon Deep Learning AMI (Ubuntu) Version 18.0 (with Ubuntu 16.04 LTS). After activating pytorch environment, PyTorch is not detecting CUDA. Also, nvidia-smi shows "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."
How to solve this problem?


title: How to compute pairwise distance between point set and lines in PyTorch?

body:
The point set A is a Nx3 matrix, and from two point sets B and C with the same size of Mx3 we could get the lines BC betwen them. Now I want to compute the distance from each point in A to each line in BC. B is Mx3 and C is Mx3, then the lines are from the points with correspoinding rows, so BC is a Mx3 matrix. The basic method is computed as follows:

D = torch.zeros((N, M), dtype=torch.float32)
for i in range(N):
    p = A[i]  # 1x3
    for j in range(M):
        p1 = B[j] # 1x3
        p2 = C[j] # 1x3
        D[i,j] = torch.norm(torch.cross(p1 - p2, p - p1)) / torch.norm(p1 - p2)

Are there any faster method to do this work? Thanks.


title: Facing error in pytorch v-1.1: "RuntimeError: all tensors must be on devices[0]"

body:
I am using nn.DataParallel() for my model but facing an error.

I am doing something like

    self.model = self.model.to(device)
    self.model = nn.DataParallel(self.model)

If the device is cuda:1 then I get RuntimeError: all tensors must be on devices[0].
But if I change the device to cuda:0 the parallel training on multiple GPUs works with no error. I am wondering what the problem is.


title: How to invert a PyTorch Embedding?

body:
I have an multi-task encoder/decoder model in PyTorch with a (trainable) torch.nn.Embedding embedding layer at the input.
In one particular task, I'd like to pre-train the model self-supervised (to re-construct masked input data) and use it for inference (to fill in gaps in data).
I guess for training time I can just measure loss as the distance between the input embedding and the output embedding... But for inference, how do I invert an Embedding to reconstruct the proper category/token the output corresponds to? I can't see e.g. a "nearest" function on the Embedding class...