title: binary image digit convolution neural network has a very high confidence when nothing is in the image

body:
I've been training my convolutional neural network on closely cropped printed digits, similar in style to the Mnist dataset. It works perfectly, with close to 100 percent on both the training and test data.

At first I trained it on  4 channel binary images where white was '255' and black was '0'. It has 10 outputs, one for each digit, which i then softmax in order to get a probability for each category.

However, I want to use a sliding window technique in order to analyse a page of digits. This is impossible because, for a fully white input, it returns almost %100 confidence that it is a 4, while pretty much nothing for everything else.

I thought it might be the fact the neural net was training on the white space in the image rather than the black space, as the black pixels had value 0, so I inverted the images and trained the network again.  again, it simply returns really  close to 100% confidence on a fully white image.

For both, it returns low percentages for each class when the image is completely black,as it should, although 4 is still the highest

I don't understand the intuition behind this, so any help would be great, even if you could just say its not usual behaviour. Is this to be expected? should i create another class for things that arent digits aswell and train it on that?

heres my neural network:
its fully convolutional so that i can implement fast sliding window with it, but the last convolutions are equivalent to fully connected layers

class fully_convolutional_1channel(nn.Module):
    def __init__(self):

        super(fully_convolutional_1channel, self).__init__()

        self.conv1 = nn.Conv2d(1, 6, 5)

        self.pool = nn.MaxPool2d(2, 2)

        self.conv2 = nn.Conv2d(6, 16, 5)

        self.fconv1 = nn.Conv2d(16, 120, (4,2))

        self.fconv2 = nn.Conv2d(120, 84,  1)
        self.fconv3 = nn.Conv2d(84, 10, 1)

    def forward(self, x):

        x = self.conv1(x)

        x = F.relu(x)
        #relu does not change size

        x = self.pool(x)

        x = self.pool(F.relu(self.conv2(x)))

        #x = x.view(-1, 16 * 4 * 2)

        x = F.relu(self.fconv1(x))

        x = F.relu(self.fconv2(x))

        x = self.fconv3(x)

        #print(list(x.size))

        return x


title: Faster alternative to compute convolution in the frequency domain in torch

body:
I'm implementing a custom convolution in torch, which converts an image to the frequency domain using FFT, computes the product between kernel and image and then computes the inverse FFT. Although it works, I've noticed that the product computation is rather slow. Is there a way to optimize it?

I've added a timer to everything to see how it goes and got the following results (testing on cpu):

squeezes - done in 4.17232513E-05s
real - done in 1.67846680E-04s
im - done in 7.53402710E-05s
stack - done in 8.36849213E-05s
sum - done in 3.96490097E-04s
bias - done in 1.64508820E-05s

Here is my implementation:

Note that I'm not computing the FFT and its inverse in here. The torchimplementation of these operations are rather fast.

def fconv2d(input, kernel, bias=None):
    # Computes the convolution in the frequency domain given
    # an input of shape (B, Cin, H, W) and kernel of shape (Cout, Cin, H, W).
    # Expects input and kernel already in frequency domain!

    with timer('squeezes'):
        kernel = kernel.unsqueeze(0)
        # Expand kernel to (B, Cout, Cin, H, W)
        # Expand input to (B, Cout, Cin, H, W)
        input = input.unsqueeze(1)
    # Compute the multiplication
    # (a+bj)*(c+dj) = (ac-bd)+(ad+bc)j
    with timer('real'):
        real = input[..., 0] * kernel[..., 0] - \
               input[..., 1] * kernel[..., 1]
    with timer('im'):
        im = input[..., 0] * kernel[..., 1] + \
             input[..., 1] * kernel[..., 0]
    # Stack both channels and sum-reduce the input channels dimension
    with timer('stack'):
        out = torch.stack([real, im], -1)

    with timer('sum'):
        out = out.sum(dim=-4)
    # Add bias
    with timer('bias'):
        if bias is not None:
            bias = bias.expand(1, 1, 1, bias.shape[0]).permute(0, 3, 1, 2)
            out += bias
    return out


title: Which ImageNet classes is PyTorch trained on?

body: PyTorch lets you download pre-trained models for classification here, but it's not clear how to go from the output tensor of probabilities back to the actual class labels. Any ideas?


title: Save a trained Autoencoder model and load it in another module in Pytorch

body:
I have an autoencoder that I have trained and now I want to save the best model. I would like to load this trained model in another Jupiter notebook and then use it in inference mode. This is my autoencoder class:
class DEC_AE(nn.Module):
"""
DEC auto encoder - this class defines the architecture of our Auto-encoder
"""

def __init__(self, num_classes, num_features): #what are number of classes (10) and number of features (10)?
    super(DEC_AE, self).__init__()
    self.dropout = nn.Dropout(p=0.1)#using dropout 0.1-is it according to the paper?
    self.fc1 = nn.Linear(28 * 28, 500)
    self.fc2 = nn.Linear(500, 500)
    self.fc3 = nn.Linear(500, 2000)
    self.fc4 = nn.Linear(2000, num_features)#the architecture used is exactly like the one in the paper
    self.relu = nn.ReLU()
    self.fc_d1 = nn.Linear(500, 28 * 28)
    self.fc_d2 = nn.Linear(500, 500)
    self.fc_d3 = nn.Linear(2000, 500)
    self.fc_d4 = nn.Linear(num_features, 2000)
    self.alpha = 1.0 #what is alpha?

    self.clusterCenter = nn.Parameter(torch.zeros(num_classes, num_features))#nn-Parameter
    self.pretrainMode = True
    #here we initialize all the weights
    for m in self.modules():
        if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
            torch.nn.init.xavier_uniform_(m.weight)

def setPretrain(self, mode):
    """To set training mode to pretrain or not,
    so that it can control to run only the Encoder or Encoder+Decoder"""
    self.pretrainMode = mode

#this one I need
def forward(self, x):#a batch of images
    x = x.view(-1, 1 * 28 * 28)
    x = self.dropout(x)
    x = self.fc1(x)
    x = self.relu(x)
    x = self.fc2(x)
    x = self.relu(x)
    x = self.fc3(x)
    x = self.relu(x)
    x = self.fc4(x)
    x_e = x #the output of encoder

    # if not in pre_train mode, we need encoder and t distribution output
    if self.pretrainMode is False:
        return x, F.softmax(x_e, dim=1)
        #why softmax of x?

    # encoder is done, followed by decoder
    x = self.fc_d4(x)
    x = self.relu(x)
    x = self.fc_d3(x)
    x = self.relu(x)
    x = self.fc_d2(x)
    x = self.relu(x)
    x = self.fc_d1(x)
    x_de = x.view(-1, 1, 28, 28)

    return x_e, x_de

and I save it like this
 torch.save(dec_ae, 'bestModel'.format(best_acc))

then in the other file I try loading the file like this:
def load_checkpoint(filepath):
   checkpoint = torch.load(filepath)
   model = checkpoint['model']
   model.load_state_dict(checkpoint['state_dict'])
   for parameter in model.parameters():
      parameter.requires_grad = False

  model.eval()
  return model

model = load_checkpoint('bestModel')
model.cuda()

But I get this error:
 AttributeError: Can't get attribute 'DEC_AE' on <module '__main__'>

I read in a post that I need to import the module like this:
from Autoencoder import DEC_AE

but it does not work. Any ideas on how to solve this? thanks!


title: How to get access intermediate layers parameters in BERT pretrained models?

body:
import torch
import transformers
tokenizer = transformers.AlbertTokenizer.from_pretrained('albert-base-v2', do_lower_case=True)
transformer = transformers.AlbertModel.from_pretrained("albert-base-v2")

I tried:
transformer.num_parameters

But, It gives all layers with parameters :

<bound method ModuleUtilsMixin.num_parameters of AlbertModel(
  (embeddings): AlbertEmbeddings(
    (word_embeddings): Embedding(30000, 128, padding_idx=0)
    (position_embeddings): Embedding(512, 128)
    (token_type_embeddings): Embedding(2, 128)
    (LayerNorm): LayerNorm((128,), eps=1e-12, elementwise_affine=True)
    (dropout): Dropout(p=0, inplace=False)
  )
  (encoder): AlbertTransformer(
    (embedding_hidden_mapping_in): Linear(in_features=128, out_features=768, bias=True)
    (albert_layer_groups): ModuleList(
      (0): AlbertLayerGroup(
        (albert_layers): ModuleList(
          (0): AlbertLayer(
            (full_layer_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            (attention): AlbertAttention(
              (query): Linear(in_features=768, out_features=768, bias=True)
              (key): Linear(in_features=768, out_features=768, bias=True)
              (value): Linear(in_features=768, out_features=768, bias=True)
              (attention_dropout): Dropout(p=0, inplace=False)
              (output_dropout): Dropout(p=0, inplace=False)
              (dense): Linear(in_features=768, out_features=768, bias=True)
              (LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)
            )
            (ffn): Linear(in_features=768, out_features=3072, bias=True)
            (ffn_output): Linear(in_features=3072, out_features=768, bias=True)
            (dropout): Dropout(p=0, inplace=False)
          )
        )
      )
    )
  )
  (pooler): Linear(in_features=768, out_features=768, bias=True)
  (pooler_activation): Tanh()
)>

I need to access to out_features = 768 the input of the last Linear function

((pooler): Linear(in_features=768, out_features=768, bias=True) )


title: How to input a numpy array to a neural network in pytorch?

body:
This is the neural network that I defined
class generator(nn.Module):
def __init__(self, n_dim, io_dim):
    super().__init__()
    self.gen = nn.Sequential(
        nn.Linear(n_dim,64),
        nn.LeakyReLU(.01),
        nn.Linear(64, io_dim),
    )

def forward(self, x):
    return self.gen(x)
#The input x is:
x = numpy.random.dirichlet([10,6,3],3)

Now I want the neural network to take dirichlet distributed samples (sampled using numpy.random.dirichlet([10,6,3],10) ) as an input. How to do that?


My program works properly with two GPUs but raise this error when reached to torch.bmm
energy =  torch.bmm(proj_query,proj_key) # transpose check

RuntimeError: cublas runtime error : the GPU program failed to execute at /pytorch/aten/src/THC/THCBlas.cu:450

how could I solve this issue ?


title: Pytorch simple text generator not working and loss keeps diverging

body:
I am new to pytorch and deep learning in general , and am trying to build a simple text generator. For reasons I don't understand, the loss keeps diverging and the model doesn't. Here's the code.

class RNN(nn.Module):
    def __init__(self, embed_size, hidden_size):
        super(RNN, self).__init__()
        self.embeds = nn.Embedding(num_chars, embed_size)
        self.l1 = nn.Linear(embed_size, hidden_size)
        self.l2 = nn.Linear(hidden_size, hidden_size)
        self.l3 = nn.Linear(hidden_size, num_chars)
        self.relu = nn.ReLU()
        self.softmax = nn.Softmax()

    def forward(self, inp):
        out = self.embeds(inp)
        out = self.l3(self.relu(self.l2(self.relu(self.l1(out)))))
        return self.softmax(out)

rnn = RNN(10, 50)
optimizer = torch.optim.Adam(rnn.parameters(), lr = 0.002)
criterion = nn.NLLLoss()

def charTensor(x):
    out = torch.zeros(1, num_chars)
    out[0][all_chars.index(x)] = 1
    return out.long()

for epoch in range(5):
    epoch_loss = 0
    rnn.zero_grad()
    for i in range(len(train_str[:400])-1):

        inp = charTensor(train_str[i])
        output = rnn(inp)
        loss = criterion(output, charTensor(train_str[i+1]))
        epoch_loss += loss

        loss.backward(retain_graph=True)
        optimizer.step()

    print("Epoch Loss:", epoch_loss) //epoch loss is always tensor(-399)

first_char = 'c'
inp_t = charTensor(first_char)
fin = first_char
for i in range(10):
    next_t = rnn(inp_t)
    next_char = all_chars[torch.argmax(next_t).numpy()]   //always ends up as 0, which is the char for space
    fin += next_char
    inp_t = charTensor(next_char)
print(fin)    //prints new line


title: how to get torch.Size([1, 3, 16, 112, 112]) instead of torch.Size([1, 3, 16, 64, 64]) after ConvTranspose3d

body:
I have a torch.Size([1, 64, 8, 32, 32]) which I want after my transpose 3d convolution to become torch.Size([1, 3, 16, 112, 112]).

Using this: nn.ConvTranspose3d(64, 3, kernel_size=4, stride=2, bias=False, padding=(1, 1, 1)) I get correct the output channels and the number of frames, but not the frame sizes:torch.Size([1, 3, 16, 64, 64])

What should I change in order to the right torch.Size?


title: PyTorch 1.0 loading VGGFace2 weights in Python3.7

body:
I am using Python3.7 and PyTorch 1.0 to develop a face recognition system. I want to use VGGFace2 Resnet50 pretrained model as described here as a feature extractor. I have downloaded the model and weights.
I run the following codes as project readme says:

MainModel = imp.load_source('MainModel', 'resnet50_128_pytorch.py')
model = torch.load('resnet50_128_pytorch.pth')

First line executed as expected but in the second line i got

  'ascii' codec can't decode byte 0xc3 in position 1124: ordinal not in
  range(128)

I searched in the Stackoverflow and Google and i saw that it may be about this model saved with Python2 and loading from Python3 creates a problem. Is there any way that i can solve this?

Thank you.


title: Size Mismatch using pytorch when trying to train data

body:
I am really new to pytorch and just trying to use my own dataset to do a simple Linear Regression Model. I am only using the numbers values as inputs, too.

I have imported the data from the CSV

dataset = pd.read_csv('mlb_games_overview.csv')

I have split the data into four parts X_train, X_test, y_train, y_test

X = dataset.drop(['date', 'team', 'runs', 'win'], 1)
y = dataset['win']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=True)

I have converted the data to pytorch tensors

X_train = torch.from_numpy(np.array(X_train))
X_test = torch.from_numpy(np.array(X_test))
y_train = torch.from_numpy(np.array(y_train))
y_test = torch.from_numpy(np.array(y_test))

I have created a LinearRegressionModel

class LinearRegressionModel(torch.nn.Module):
    def __init__(self):
        super(LinearRegressionModel, self).__init__()
        self.linear = torch.nn.Linear(1, 1)
    def forward(self, x):
        y_pred = self.linear(x)
        return y_pred

I have initialized the optimizer and the loss function

criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

Now when I start to train the data I get the runtime error mismatch

EPOCHS = 500
for epoch in range(EPOCHS):
    pred_y = model(X_train) # RUNTIME ERROR HERE
    loss = criterion(pred_y, y_train)
    optimizer.zero_grad() # zero out gradients to update parameters correctly
    loss.backward() # backpropagation
    optimizer.step() # update weights
    print('epoch {}, loss {}'. format(epoch, loss.data[0]))

Error Log:

RuntimeError                              Traceback (most recent call last)
<ipython-input-40-c0474231d515> in <module>
  1 EPOCHS = 500
  2 for epoch in range(EPOCHS):
----> 3     pred_y = model(X_train)
  4     loss = criterion(pred_y, y_train)
  5     optimizer.zero_grad() # zero out gradients to update parameters correctly
RuntimeError: size mismatch, m1: [3540 x 8], m2: [1 x 1] at
C:\w\1\s\windows\pytorch\aten\src\TH/generic/THTensorMath.cpp:752


title: In PyTorch calc Euclidean distance instead of matrix multiplication

body:
Let say we have 2 matrices:
mat = torch.randn([20, 7]) * 100
mat2 = torch.randn([7, 20]) * 100

n, m = mat.shape

The simplest usual matrix multiplication looks like this:
def mat_vec_dot_product(mat, vect):
    n, m = mat.shape

    res = torch.zeros([n])
    for i in range(n):
        for j in range(m):
            res[i] += mat[i][j] * vect[j]

    return res

res = torch.zeros([n, n])
for k in range(n):
    res[:, k] = mat_vec_dot_product(mat, mat2[:, k])

But what if I need to apply L2 norm instead of dot product? The code is next:
def mat_vec_l2_mult(mat, vect):
    n, m = mat.shape

    res = torch.zeros([n])
    for i in range(n):
        for j in range(m):
            res[i] += (mat[i][j] - vect[j]) ** 2

    res = res.sqrt()

    return res

for k in range(n):
    res[:, k] = mat_vec_l2_mult(mat, mat2[:, k])

Can we do this somehow in an optimal way using Torch or any other libraries? Cause naive O(n^3) Python code works really slow.


title: Docker build ERROR: Could not find a version that satisfies the requirement torch==1.5.1

body:
I have a Dockerfile that has the following format:
FROM python:latest
ADD requirements.txt .
RUN pip3 install -r requirements.txt
CMD ["python", "script.py"]

Where my requirements.txt looks like:
grpcio==1.31.0
grpcio-tools==1.31.0
torch==1.5.1

When I execute docker image build . I get the following error:
Sending build context to Docker daemon  61.44kB
Step 1/7 : FROM python:latest
 ---> dfc47c6cee13
Step 2/7 : ADD requirements.txt .
 ---> Using cache
 ---> 3be914d5b849
Step 3/7 : RUN pip3 install -r requirements.txt
 ---> Running in 372c329c76fa
Collecting grpcio==1.31.0
  Downloading grpcio-1.31.0.tar.gz (20.0 MB)
Collecting grpcio-tools==1.31.0
  Downloading grpcio-tools-1.31.0.tar.gz (2.1 MB)
ERROR: Could not find a version that satisfies the requirement torch==1.5.1 (from -r requirements.txt (line 3)) (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)
ERROR: No matching distribution found for torch==1.5.1 (from -r requirements.txt (line 3))
The command '/bin/sh -c pip3 install -r requirements.txt' returned a non-zero code: 1

This was a strange error to receive because if I recall correctly, torch was able to be installed fairly recently (as early as 2 weeks ago), so I'm not sure why this is happening.


title: "OSError: Model name './XX' was not found in tokenizers model name list" - cannot load custom tokenizer in Transformers

body:
I'm trying to create my own tokenizer with my own dataset/vocabulary using Sentencepiece and then use it with AlbertTokenizer transformers.
I followed really closely the tutorial on how to train a model from scratch by HuggingFace: https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb#scrollTo=hO5M3vrAhcuj
    # import relevant libraries
    from pathlib import Path
    from tokenizers import SentencePieceBPETokenizer
    from tokenizers.implementations import SentencePieceBPETokenizer
    from tokenizers.processors import BertProcessing
    from transformers import AlbertTokenizer

    paths = [str(x) for x in Path("./data").glob("**/*.txt")]

    # Initialize a tokenizer
    tokenizer = SentencePieceBPETokenizer(add_prefix_space=True)

    # Customize training
    tokenizer.train(files=paths,
                    vocab_size=32000,
                    min_frequency=2,
                    show_progress=True,
                    special_tokens=['<unk>'],)

    # Saving model
    tokenizer.save_model("Sent-AlBERT")

    tokenizer = SentencePieceBPETokenizer(
        "./Sent-AlBERT/vocab.json",
        "./Sent-AlBERT/merges.txt",)

    tokenizer.enable_truncation(max_length=512)

Everything is fine up until this point when I tried to re-create the tokenizer in transformers
    # Re-create our tokenizer in transformers
        tokenizer = AlbertTokenizer.from_pretrained("./Sent-AlBERT", do_lower_case=True)

This is the error message I kept receiving:
OSError: Model name './Sent-AlBERT' was not found in tokenizers model name list (albert-base-v1, albert-large-v1, albert-xlarge-v1, albert-xxlarge-v1, albert-base-v2, albert-large-v2, albert-xlarge-v2, albert-xxlarge-v2). We assumed './Sent-AlBERT' was a path, a model identifier, or url to a directory containing vocabulary files named ['spiece.model'] but couldn't find such vocabulary files at this path or url.

For some reason, it works with RobertaTokenizerFast but not with AlbertTokenzier.
If anyone could give me a suggestion or any sort of direction on how to use Sentencepiece with AlberTokenizer I would really appreciate.


title: fix number of model checkpoints to be kept pytorch

body:
Is there anything similar to max_to_keep in tf.Saver() in pytorch? How can I only save N number of model checkpoints? Is there a built in function in torch for this?

I'm using the function below to save checkpoints.

import os
import torch

def save_checkpoint(model, save_path):
    if not os.path.exists(os.path.dirname(save_path)):
        os.makedirs(os.path.dirname(save_path))

    torch.save(model.cpu().state_dict(), save_path)
    model.cuda()


title: How to efficiently compute gradients for each example in a large dataset using pytorch?

body:
Given a trained model (M), I'm interested in computing the utility of new (unseen) examples in a pool (for an active learning task).  For this, I need to compute the magnitude of the gradient when M is trained on each new example.  In code, it is something like:

losses, grads = [], []
for i in range(X_pool.shape[0]):
    pred = model(X_pool[i:i+1])
    loss = loss_func(pred, y_pool[i:i+1])

    model.zero_grad()
    loss.backward()

    losses.append(loss)
    grads.append(layer.weight.grad.norm())

However, this is quite slow when there is a large pool of examples, especially since this will be the inner loop in my scenario.  Is there a way to do this more efficiently in pytorch?


title: Installing torchvision from source libavcodec/avcodec.h not found

body:
I am trying to install torchvision from source, was able to get pytorch installed (needed it from source to use GPU) and now can't get torchvision to work.

I am getting the following error when I run the setup.py:

  C:\Users\hoski\vision\torchvision\csrc\cpu\decoder\defs.h(11): fatal error C1083: Cannot open include file: 'libavcodec/avcodec.h': No such file or directory
  error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe' failed with exit status 2

I downloaded the source for ffmpeg and added it to my path, and I checked the libavcodec folder and the file is there, it just isn't seeing it I guess?

Any advice would be greatly appreciated!


title: Why multiplication on GPU is slower than on CPU?

body:
Here is my code (simulate the feed-forward neural network):
import torch
import time

print(torch.cuda.is_available())    # True
device = torch.device('cuda:0' )

a = torch.tensor([1,2,3,4,5,6]).float().reshape(-1,1)
w1 = torch.rand(120,6)
w2 = torch.rand(1,120)
b1 = torch.rand(120,1)
b2 = torch.rand(1,1).reshape(1,1)

start = time.time()
for _ in range(100000):
    ans = torch.mm(w2, torch.mm(w1,a)+b1)+b2
end = time.time()
print(end-start)                    # 1.2725720405578613 seconds

a = a.to(device)
w1 = w1.to(device)
w2 = w2.to(device)
b1 = b1.to(device)
b2 = b2.to(device)

start = time.time()
for _ in range(100000):
    ans = torch.mm(w2, torch.mm(w1,a)+b1)+b2
end = time.time()
print(end-start)                    # 5.6569812297821045 seconds

I wonder if I did it the wrong way or what, and how can I change my code to show that GPU IS faster then CPU on matrix multiplication？


I have successfully finished the following steps:
Normal training using 1000 images from coco-2014 dataset with command:
python3 train.py --data data/coco2014.data --batch-size 16 -pt --weights yolov4.weights --cfg cfg/yolov4/yolov4.cfg
After this I obtained initial weights last.pt and using then I have successfully finished sparsity training with command:
python train.py --data data/coco2014.data -pt --weights weights/last.pt --batch-size 4 --cfg cfg/yolov4/yolov4.cfg --epochs 200 -sr --s 0.001 --prune 0
However, in the final pruning stage I am stuck while testing the final model step. The compactmodel2 is unable to go through eval_model function as when the tensors go through test.py they throw following error:
RuntimeError: Sizes of tensors must match except in dimension 2. Got 44 and 88 (The offending index is 0).
I will really appreciate for any solutions. Thanks in advance.
The command I use to run pruning is:
python layer_channel_regular_prune.py --data data/coco2014.data --weights weights/last.pt --cfg cfg/yolov4/yolov4.cfg --percent 0.5 --img-size 608


title: Implementing fast dense feature extraction in PyTorch

body:
I am trying to implement this paper in PyTorch Fast Dense Feature Extractor but I am having trouble converting the Torch implementation example they provide into PyTorch.

My attempt thus far has the issue that when adding an additional dimension to the feature map then the convolutional weights don't match the feature shape. How is this managed in Torch (from their implementation it seem that Torch doesn't care about this, but PyTorch does). My code: https://gist.github.com/system123/c4b8ef3824f2230f181f8cfba84f0cfd

Any other solutions to this problem would be great too. Basically, I have a feature extractor that converts a 128x128 patch into an embedding and I'd like to apply this in a dense manner across a larger image without using a for loop to evaluate the CNN on each location as that has a lot of duplicate computation.


title: How to get rid of Variable API in PyTorch.autograd?

body:
I am forwarding, and backpropping tensor data X through two simple nn.Module PyTorch models instances, model1 and model2.

I can't get this process to work without usage of the depreciated Variable API.

So this works just fine:

    y1 = model1(X)
    v = Variable(y1.data, requires_grad=training)         # Its all about this line!
    y2 = model2(v)
    criterion = nn.NLLLoss()
    loss = criterion(y2, y)
    loss.backward()
    y1.backward(v.grad)
    self.step()

But this will throw an error:

    y1 = model1(X)
    y2 = model2(y1)
    criterion = nn.NLLLoss()
    loss = criterion(y2, y)
    loss.backward()
    y1.backward(y1.grad) # it breaks here
    self.step()
>>> RuntimeError: grad can be implicitly created only for scalar outputs

I just can't seem to find a relevant difference between v in the first implementation, and y1 in the second. In both cases requires_grad is set to True. The only thing I could find was that y1.grad_fn=<ThnnConv2DBackward> and v.grad_fn=<ThnnConv2DBackward>

What am I missing here? What (tensor attributes?) do I not know about, and if Variable is depreciated, what other implementation would work?


title: TypeError: hook() takes 2 positional arguments but 3 were given

body:
I'm new to pytorch and I'm trying to use hook() and register_forward_pre_hook in my project

What I've tried is

def get_features_hook(module,input):
    print(input)

handle_feat = alexnet.features[0].register_forward_pre_hook(get_features_hook)

a = alexnet(input_data)

And I got belows error at a = alexnet(input_data)

  TypeError: get_features_hook() takes 2 positional arguments but 3 were given

I've lost few hours on this problem and I just can't able to figure it out.

Anyone likes to help me?

With Shai's help, I tried his codes, and I got this

Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
get_features_hook called with 2 args:
    arg of type Conv2d
    arg of type tuple

File "<input>", line 2, in get_features_hook
NameError: name 'args' is not defined


title: Importing torchvision to conda environment - Windows

body:
I have recently installed python (3.7.4) and conda (1.7.2) on my computer (OS: Windows 10).

I installed pytorch (1.3.1) and torchvision (0.4.2) using the conda prompt.

I created a new environment in the conda "Environment" tab.
When I try to search torchvision in my environment I can't see it in  the search result (even when I search "All" and not just "Installed").
However, when I switch to the "base(root)" environment, I can see torchvision.

How can I install torchvision to my environment?
Thank you!


title: Why is a simple Binary classification failing in a feedforward neural network?

body:
I am new to Pytorch. I was trying to model a binary classifier on the Kepler dataset. The following was my dataset class.

class KeplerDataset(Dataset):
    def __init__(self, test=False):
        self.dataframe_orig = pd.read_csv(koi_cumm_path)

        if (test == False):
            self.data = df_numeric[( df_numeric.koi_disposition == 1 ) | ( df_numeric.koi_disposition == 0 )].values
        else:
            self.data = df_numeric[~(( df_numeric.koi_disposition == 1 ) | ( df_numeric.koi_disposition == 0 ))].values

        self.X_data = torch.FloatTensor(self.data[:, 1:])
        self.y_data = torch.FloatTensor(self.data[:, 0])

    def __len__(self):
        return len(self.data)

    def __getitem__(self, index):
        return self.X_data[index], self.y_data[index]

Here, I created a custom classifier class with one hidden layer and a single output unit that produces sigmoidal probability of being in class 1 (planet).

class KOIClassifier(nn.Module):
    def __init__(self, input_dim, out_dim):
        super(KOIClassifier, self).__init__()
        self.linear1 = nn.Linear(input_dim, 32)

        self.linear2 = nn.Linear(32, 32)

        self.linear3 = nn.Linear(32, out_dim)

    def forward(self, xb):
        out = self.linear1(xb)
        out = F.relu(out)
        out = self.linear2(out)
        out = F.relu(out)
        out = self.linear3(out)
        out = torch.sigmoid(out)

        return out

I then created a train_model function to optimize the loss using SGD.

def train_model(X, y):
    criterion = nn.BCELoss()

    optim = torch.optim.SGD(model.parameters(), lr=0.001)

    n_epochs = 100
    losses = []

    for epoch in range(n_epochs):
        y_pred = model.forward(X)
        loss = criterion(y_pred, y)
        losses.append(loss.item())
        optim.zero_grad()
        loss.backward()
        optim.step()

losses = []
for X, y in train_loader:
    losses.append(train_model(X, y))

But after performing the optimization over the train_loader, When I try predicting on the trainn_loader itself, the prediction values are so much worse.

for features, y in train_loader:
    y_pred = model.predict(features)
    break

y_pred

> tensor([[4.5436e-02],
        [1.5024e-02],
        [2.2579e-01],
        [4.2279e-01],
        [6.0811e-02],
        .....

Why is my model not working properly? Is it the problem with the dataset or am I doing something wrong with implementing the Neural net? I will link my Kaggle notebook because more context might be helpful. Please help.


title: deep neural network model stops learning after one epoch

body:
I am training a unsupervised NN model and for some reason, after exactly one epoch (80 steps), model stops learning.
]
Do you have any idea why it might happen and what should I do to prevent it?
This is more info about my NN:
I have a deep NN that tries to solve an optimization problem. My loss function is customized and it is my objective function in the optimization problem.
So if my optimization problems is min f(x) ==> loss, now in my DNN loss = f(x). I have 64 input, 64 output, 3 layers in between :
self.l1 = nn.Linear(input_size, hidden_size)
self.relu1 = nn.LeakyReLU()
self.BN1 = nn.BatchNorm1d(hidden_size)

and last layer is:
self.l5 = nn.Linear(hidden_size, output_size)
self.tan5 = nn.Tanh()
self.BN5 = nn.BatchNorm1d(output_size)

to scale my network.
with more layers and nodes(doubles: 8 layers each 200 nodes), I can get a little more progress toward lower error, but again after 100 steps training error becomes flat!


title: how to change CNN model to Wide convolution in pytorch?

body:
I am learning CNN, I want to change CNN model with Wide Convolution by pytorch, who can help?

self.conv23 = nn.Conv2d(Ci, len(Ks) * Co, (3, Co), padding=1)

Traceback (most recent call last):
  File "E:/workspace/pycharmworkspace/cnn-text-classification-pytorch-update/main.py", line 137, in <module>
    train.train(train_iter, dev_iter, cnn, args)
  File "E:\workspace\pycharmworkspace\cnn-text-classification-pytorch-update\train.py", line 40, in train
    logit = model(feature)
  File "C:\Users\bamtercelboo\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "E:\workspace\pycharmworkspace\cnn-text-classification-pytorch-update\model.py", line 206, in forward
    x21 = self.conv(x11, self.conv23)  #(N,Co)
  File "E:\workspace\pycharmworkspace\cnn-text-classification-pytorch-update\model.py", line 91, in conv
    x = F.relu(conv(x)).squeeze(3)  # (N,Co,W)
  File "C:\Users\bamtercelboo\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 206, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\bamtercelboo\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 237, in forward
    self.padding, self.dilation, self.groups)
  File "C:\Users\bamtercelboo\Anaconda3\lib\site-packages\torch\nn\functional.py", line 43, in conv2d
    return f(input, weight, bias)
RuntimeError: kernel size should be greater than zero, but got kT: 3 kH: 200 kW: 0 at d:\downloads\pytorch-master-1\torch\lib\thnn\generic/VolumetricConvolutionMM.c:23


title: How to convert a model from torch to pytorch or caffe or any other model?

body:
I'm trying to convert this t7 model to pytorch or to caffe or to caffe2 or to any other model..
this is what I get when converting to pytorch with the code from:
https://github.com/clcarwin/convert_torch_to_pytorch

has anyone had this error or know what to do with it?

roy@roy-Lenovo:~/convert_torch_to_pytorch$ python convert_torch.py -m model_a.t7
    Traceback (most recent call last):
      File "convert_torch.py", line 314, in <module>
        torch_to_pytorch(args.model,args.output)
      File "convert_torch.py", line 262, in torch_to_pytorch
        slist = lua_recursive_source(lnn.Sequential().add(model))
      File "/home/roy/.local/lib/python2.7/site-packages/torch/legacy/nn/Sequential.py", line 15, in add
        self.output = module.output
      File "/home/roy/.local/lib/python2.7/site-packages/torch/utils/serialization/read_lua_file.py", line 99, in __getattr__
        return self._obj.get(k)
    AttributeError: 'list' object has no attribute 'get'


title: Pytorch sum tensors doing an operation within each set of numbers

body:
I have the following Pytorch tensor:

V1 = torch.tensor([[2, 4], [6, 4], [5, 3]])

I want to do the sum of the difference of each pair of numbers (applying absolute value), something like the code below;

result.sum(abs(2-4), abs(6-4), abs(5-3))

I can do this using a for statement:

total = 0
for i in range(0,vector.size(0)):
    total = total + torch.abs(vector.data[i][1] - vector.data[i][0])

But I want to do it without using a for.

Is there a way to do that?


title: Unable to reproduce results with PyTorch CPU

body:
I'm training my network on CPU. From PyTorch docs, I read that results may not be reproducible when using GPU because of cudnn. But on CPU, since cudnn is not used, results should be reproducible. I'm setting the seed as follows:

os.environ['PYTHONHASHSEED'] = str(seed)
random.seed(seed)
numpy.random.seed(seed)
torch.manual_seed(seed)

Even then I'm getting different results each time I try. Am I missing something?

Edit:
Sorry, my bad. There was a problem in code. The above was not called properly. Now, I'm able to get reproducible results.


title: Ordinal classes in a fast.ai ImageDataBunch

body:
I have ordinal categorical labels I want to use in an ImageDataBunch.

In my DataFrame I specify this as such:

from pandas.api.types import CategoricalDtype
cat_type = CategoricalDtype(categories=['L', 'M', 'H'], ordered=True)
label_df["adj_gr"] = label_df["adj_gr"].astype(cat_type)

#Series shows as: Categories (3, object): [L < M < H]

However when I made an ImageDataBunch from this column it reverts the categories to alphabetical order:

data = ImageDataBunch.from_df("", df=label_df, fn_col="path", label_col="adj_gr", valid_pct=0.2,
        ds_tfms=tfms, size=224, num_workers=4, padding_mode='border').normalize(imagenet_stats)
data.classes
OUT: [H, L, M]

Does the order not matter for neural net like it does for ML models?

Its frustrating as when I have several categories and I plot_confusion_matrix() the labels are out of order and its hard to get a visualization of how good the model is at this.

Setup: python==3.7, pytorch==1.4, fastai==1.0.60


title: Number of images per class in dataset, PyTorch

body: I am working on an image dataset where images are classified in 10 classes (CIFAR10 dataset). I am using PyTorch. Please, I would like to know how to determine the number of images per class by looping through the dataset. Thanks in advance for your response.


title: pretrained fasterrcnn_resnet50_fpn: "forward() takes 2 positional arguments but 5 were given" on single image test

body:
I am receving an error when executing the code below:
import torch
import torch.nn as nn
import torchvision.transforms as transforms
from torchvision.models.detection import fasterrcnn_resnet50_fpn
import PIL.Image as Image

device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

model = fasterrcnn_resnet50_fpn(pretrained=True)
model.roi_heads = nn.Sequential()
model.to(device)

img = Image.open('frame_00001.jpg')
transform = transforms.Compose([transforms.ToTensor(),
                                transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                                    std=[0.229, 0.224, 0.225])])
img = transform(img).unsqueeze(0).to(device)

model.eval()
output = model(img)

The last line causes a "TypeError: forward() takes 2 positional arguments but 5 were given."
The size of img is [1, 3, 960, 1280].
If I add square brackets around img before passing into the model (output = model([img])), I get a "ValueError: images is expected to be a list of 3d tensors of shape [C, H, W], got torch.Size([1, 3, 960, 1280])." Then, if I use "img.view(3, 960, 1280)," I get the original error again.
What is the solution to this problem? Thank you.


title: Can import pytorch but not torch

body:
I installed torch using conda install pytorch torchvision cudatoolkit=10.2 -c pytorch.
I can import pytorch but not import torch. I am getting the following error:
Traceback (most recent call last):

  File "<ipython-input-10-eb42ca6e4af3>", line 1, in <module>
    import torch

  File "C:\Users\anirb\Anaconda3\lib\site-packages\torch\__init__.py", line 81, in <module>
    ctypes.CDLL(dll)

  File "C:\Users\anirb\Anaconda3\lib\ctypes\__init__.py", line 356, in __init__
    self._handle = _dlopen(self._name, mode)


title: CNN model is overfitting to data after reaching 50% accuracy

body:
I am trying to identify 3 (classes) mental states based on EEG connectome data. The shape of the data is 99x1x34x34x50x130 (originally graph data, but now represented as a matrix), with respectably represent [subjects, channel, height, width, freq, time series]. For the sake of this study, can only input a 1x34x34 image of the connectome data. From previous studies, it was found that the alpha band (8-1 hz) had given the most information, thus the dataset was narrowed down to 99x1x34x34x4x130. The testing set accuracy on pervious machine learning techniques such as SVMs reached a testing accuracy of ~75%. Hence, by goal is to achieve a greater accuracy given the same data (1x34x34). Since my data is very limited 1-66 for training and 66-99 for testing (fixed ratios and have a 1/3 class distribution), I thought of splitting the data along the time series axis (6th axis) and then averaging the data to a shape of 1x34x34 (from ex. 1x34x34x4x10, 10 is the random sample of time series). This gave me ~1500 samples for training, and 33 for testing (testing is fixed, the class distributions are 1/3).
Model:
SimpleCNN(
  (conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (conv2): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  (pool1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  (drop1): Dropout(p=0.25, inplace=False)
  (fc1): Linear(in_features=9248, out_features=128, bias=True)
  (drop2): Dropout(p=0.5, inplace=False)
  (fc2): Linear(in_features=128, out_features=3, bias=True)
)
CrossEntropyLoss()
Adam (
Parameter Group 0
    amsgrad: False
    betas: (0.9, 0.999)
    eps: 1e-08
    lr: 5e-06
    weight_decay: 0.0001
)

Results:
The training set can achieve an accuracy of 100% with enough iteration, but at the cost of the testing set accuracy. After around 20-50 epochs of testing, the model starts to overfit to the training set and the test set accuracy starts to decrease (same with loss).

What I have tried:
I have tried tuning the hyperparameters: lr=.001-000001, weight decay=0.0001-0.00001. Training to 1000 epochs (useless bc overfitting in less than 100 epochs). I have also tried increasing/decreasing the model complexity with adding adding addition fc layers and varying amount of channels in CNN layers form 8-64. I have also tried adding more CNN layers and the model did a bit worse averaging around an accuracy of ~45% on the test set. I tried manually scheduling the learning rate every 10 epochs, the results were the same. Weight decay didn’t seem to effect the results much, changed it from 0.1-0.000001.
From previous testing, I have a model that achieves 60% acc on both the testing and the training set. However, when I try to retrain it, the acc instantly goes down to ~40 on both sets (training and testing), which makes no sense. I have tried altering the learning rate from 0.01 to 0.00000001, and also tried weight decay for this.
From training the model and the graphs, it seems like the model dosn’t know what it’s doing for the first 5-10 epochs and then starts to learn rapidly to around ~50%-60% acc on both sets. This is where the model starts to overfit, form there the model’s acc increases to 100% on the training set, and the acc for the testing set goes down to 33%, which is equivalent to guessing.
Any tips?
Edit:
The model’s outputs for the test set are very very close to each other.
0.33960407972335815, 0.311821848154068, 0.34857410192489624

The average standard deviation for the whole test set between predictions for each image are (softmax):
0.017695341517654846

However, the average std for the training set is .22 so...
F1 Scores:
Micro Average: 0.6060606060606061
Macro Average: 0.5810185185185186
Weighted Average: 0.5810185185185186
Scores for each class: 0.6875 0.5 0.55555556

Here is a confusion matrix:


title: The definition of "heads" in MultiheadAttention in Pytorch Transformer module

body:
I am a bit confused about the definition of Multihead.
Are [1] and [2] below the same?
[1]
My understanding about multiplhead is the multiple attention patterns as below.
"multiple sets of Query/Key/Value weight matrices (the Transformer uses eight attention heads, so we end up with eight sets for each encoder/decoder)."
http://jalammar.github.io/illustrated-transformer/
But
[2] in class MultiheadAttention(Module): in Pytorch Transformer module,
it seems like embed_dim is DIVIDED by the number of heads.. WHy?
Or... the embed_dim is meant to be the feature dimension times the number of heads in the first place?
self.head_dim = embed_dim // num_heads
assert self.head_dim * num_heads == self.embed_dim, "embed_dim must be divisible by num_heads"
https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/activation.py


title: Changing config and loading Hugging Face model fine-tuned on a downstream task

body:
I am using HuggingFace models for TokenClassification task. I have the following label2id mapping. I am using version 3.3.0 of the library
label2id = {
    "B-ADD": 4,
    "B-ARRESTED": 7,
    "B-CRIME": 2,
    "B-INCIDENT_DATE": 3,
    "B-SUSPECT": 9,
    "B-VICTIMS": 1,
    "B-WPN": 5,
    "I-ADD": 8,
    "I-ARRESTED": 13,
    "I-CRIME": 11,
    "I-INCIDENT_DATE": 10,
    "I-SUSPECT": 14,
    "I-VICTIMS": 12,
    "I-WPN": 6,
    "O": 0
  }

The following scenario works well and the model gets loaded correctly.
from transformers import AutoModelForTokenClassification, AutoTokenizer, AutoConfig

pretrained_model_name = "bert-base-cased"
config = AutoConfig.from_pretrained(pretrained_model_name)

id2label = {y:x for x,y in label2id.items()}
config.label2id = label2id
config.id2label = id2label
config._num_labels = len(label2id)

model = AutoModelForTokenClassification.from_pretrained(pretrained_model_name, config=config)

model

I get the following output. The last layer have been correctly initialized with 15 neurons (numer of token category to predict).
.....................
      (dropout): Dropout(p=0.1, inplace=False)
      (classifier): Linear(in_features=768, out_features=15, bias=True)
    )

but if I changed the pretrained_model_name to "dbmdz/bert-large-cased-finetuned-conll03-english", I am getting the following error
loading weights file https://cdn.huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english/pytorch_model.bin from cache at C:\Users\anu10961/.cache\torch\transformers\4b02c1fe04cf7f7e6972536150e9fb329c7b3d5720b82afdac509bd750c705d2.6dcb154688bb97608a563afbf68ba07ae6f7beafd9bd98b5a043cd269fcc02b4
All model checkpoint weights were used when initializing BertForTokenClassification.

All the weights of BertForTokenClassification were initialized from the model checkpoint at dbmdz/bert-large-cased-finetuned-conll03-english.
If your task is similar to the task the model of the checkpoint was trained on, you can already use BertForTokenClassification for predictions without further training.

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-15-2969a8092bf4> in <module>
----> 1 model = AutoModelForTokenClassification.from_pretrained(pretrained_model_name, config=config)

C:\ProgramData\Anaconda3\envs\arcgis183\lib\site-packages\transformers\modeling_auto.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1372         if type(config) in MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING.keys():
   1373             return MODEL_FOR_TOKEN_CLASSIFICATION_MAPPING[type(config)].from_pretrained(
-> 1374                 pretrained_model_name_or_path, *model_args, config=config, **kwargs
   1375             )
   1376

C:\ProgramData\Anaconda3\envs\arcgis183\lib\site-packages\transformers\modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
   1047                 raise RuntimeError(
   1048                     "Error(s) in loading state_dict for {}:\n\t{}".format(
-> 1049                         model.__class__.__name__, "\n\t".join(error_msgs)
   1050                     )
   1051                 )

RuntimeError: Error(s) in loading state_dict for BertForTokenClassification:
    size mismatch for classifier.weight: copying a param with shape torch.Size([9, 1024]) from checkpoint, the shape in current model is torch.Size([15, 1024]).
    size mismatch for classifier.bias: copying a param with shape torch.Size([9]) from checkpoint, the shape in current model is torch.Size([15]).

The only difference I could see is the model dbmdz/bert-large-cased-finetuned-conll03-english is already finetuned on TokenClassification task and it model config has these label2id mappings
label2id = {
    "B-LOC": 7,
    "B-MISC": 1,
    "B-ORG": 5,
    "B-PER": 3,
    "I-LOC": 8,
    "I-MISC": 2,
    "I-ORG": 6,
    "I-PER": 4,
    "O": 0
  }

But still I feel that we can change the last layer of this model and use it for my specific task (although I need to train the model first before using it for inferencing)


title: Assertion error in DataParallel when applying to RNN

body:
I'm trying to apply DataParallel to a RNN model.

this is part of my code:

if use_cuda:
    encoder = encoder.cuda()
    decoder = decoder.cuda()

    encoder = nn.DataParallel(encoder, dim=0)
    decoder = nn.DataParallel(decoder, dim=0)

class EncoderRNN(nn.Module):
    def __init__(self, vocal_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.embedding = nn.Embedding(vocal_size, hidden_size)
        self.gru = nn.GRU(hidden_size, hidden_size, batch_first=True)

    def forward(self, input_batch, input_batch_length, hidden):
        print(input_batch)
        print(input_batch_length)
        print(hidden)
        embedded = self.embedding(input_batch)
        packed_input = nn.utils.rnn.pack_padded_sequence(embedded, input_batch_length.cpu().numpy(), batch_first=True)
        output, hidden = self.gru(packed_input, hidden)
        return output, hidden

    def init_hidden(self, batch_size):
        result = torch.autograd.Variable(torch.zeros(1, batch_size, self.hidden_size))

        if use_cuda:
            return result.cuda()
        else:
            return result

I can guarantee that all inputs are cuda, but I received this error. It seems that my inputs are not all cuda.

Traceback (most recent call last):
  File "train.py", line 156, in <module>
    train_iteration(encoder, decoder, fileDataSet)
  File "train.py", line 122, in train_iteration
    target_indices, encoder, decoder, encoder_optimizer, decoder_optimizer, criterion)
  File "train.py", line 49, in train
    encoder_output, encoder_hidden = encoder(input_batch, input_batch_length, encoder_hidden)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 74, in forward
    return self.gather(outputs, self.output_device)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 86, in gather
    return gather(outputs, output_device, dim=self.dim)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 65, in gather
    return gather_map(outputs)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 60, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 60, in gather_map
    return type(out)(map(gather_map, zip(*outputs)))
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/utils/rnn.py", line 39, in __new__
    return super(PackedSequence, cls).__new__(cls, *args[0])
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/scatter_gather.py", line 57, in gather_map
    return Gather.apply(target_device, dim, *outputs)
  File "/home/cjunjie/anaconda3/lib/python3.6/site-packages/torch/nn/parallel/_functions.py", line 58, in forward
    assert all(map(lambda i: i.is_cuda, inputs))
AssertionError

This code works on the single GPU, but I didn't find any similar problem in Google.

Content of input_batch:

 2.0000e+00  6.2900e+02  5.4000e+01  ...   0.0000e+00  0.0000e+00  0.0000e+00
 2.0000e+00  1.6759e+04  6.0000e+00  ...   0.0000e+00  0.0000e+00  0.0000e+00
 2.0000e+00  7.2000e+01  3.3500e+02  ...   0.0000e+00  0.0000e+00  0.0000e+00
 2.0000e+00  5.4000e+01  1.2900e+02  ...   0.0000e+00  0.0000e+00  0.0000e+00
[torch.cuda.LongTensor of size (4,2687) (GPU 0)]

input_batch_length:

 1844
 1507
 1219
 1021
[torch.cuda.LongTensor of size (4,) (GPU 0)]

hidden:

( 0 ,.,.) =
   0   0   0  ...    0   0   0
   0   0   0  ...    0   0   0
   0   0   0  ...    0   0   0
   0   0   0  ...    0   0   0
[torch.cuda.FloatTensor of size (1,4,256) (GPU 0)]


title: Find global minimun with GD in PyTorch

body:
I have trained a NN with a vector input and scalar output (regression).
Now I want to find the global minimun of the NN using GD with pytorch.

I’m new to programming in general, python specifically, and pytorch even more specifically.

I believe what I’m trying to do must have been done a thousand times before, if not ten thousand times. I’ll be super happy and grateful if anyone could point me to some code somewhere (maybe in github) where there’s an example of what I’m trying to do that I could adjust to my needs.


title: Why my tensor defined in forward function can not be transformed in to cuda variable autonomously?

body:
In PyTorch, in the forward function of a model

class my_model(torch.nn.module):

    ......

    def forward():
        self.a=torch.zeros(1)
        #blabla

After model.cuda(), why self.a is still a cpu variable?


title: Why is the Pytorch Dropout layer affecting all values, not only the ones set to zero?

body:
The dropout layer from Pytorch changes the values that are not set to zero. Using Pytorch's documentation example: (source):

import torch
import torch.nn  as nn

m = nn.Dropout(p=0.5)
input = torch.ones(5, 5)

print(input)
tensor([[1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.],
        [1., 1., 1., 1., 1.]])

Then I pass it through a dropout layer:

output = m(input)
print(output)
tensor([[0., 0., 2., 2., 0.],
        [2., 0., 2., 0., 0.],
        [0., 0., 0., 0., 2.],
        [2., 2., 2., 2., 2.],
        [2., 0., 0., 0., 2.]])

The values that aren't set to zero are now 2. Why?


title: How to speed up slicing in python, not using the for loop

body:
I'm trying to speed up the following python code:

import torch
import numpy as np

A = torch.zeros(11, 16, 64)
B = torch.randn(11, 9, 64)

indices = np.random.randint(0,9,(11,16))

for i in range(len(A)):
    A[i,:,:] = B[i,indices[i],:]

Is there a nice way not to use the for loop? This way, it is really slow, especially when dealing with the big data . The indices is the pre-defined 2-dim matrix with size (11,16). What I need is to assign the elements of B to A according to the order of indices. After the speeding up, the result of A should be exactly the same with mine resulted A. Thanks!


title: Do I need to make multiple instances of a neural network in PyTorch to test multiple loss functions?

body:
I have written out a neural network in PyTorch and I would like to compare the results of two different loss functions on this one network

Should I go about making two different instances of the network and test one loss function per network like this

network_w_loss_1 = ANN().cuda()
network_w_loss_2 = ANN().cuda()

crit_loss_1 = loss_1()
crit_loss_2 = loss_2()

opt_loss_1 = optim.SGD('params')
opt_loss_2 = optim.SGD('params')

for epoch in range(num_epochs):
    for i, dat in enumerate(data_loader):
        #unpack data
        opt_loss_1.zero_grad()
        opt_loss_2.zero_grad()
        output1 = network_w_loss_1('params')
        output2 = network_w_loss_2('params')
        los_1 = crit_loss_1(output1)
        los_2 = crit_loss_2(output2)
        los_1.backward()
        los_2.backward()
        opt_loss_1.step()
        opt_loss_2.step()

or can I get away with doing this?

network = ANN().cuda()

crit_loss_1 = loss_1()
crit_loss_2 = loss_2()

opt = optim.SGD('params')

for epoch in range(num_epochs):
    for i, dat in enumerate(data_loader):
        #unpack data
        opt.zero_grad()
        output1 = network('params')
        output2 = network('params')
        los_1 = crit_loss_1(output1)
        los_2 = crit_loss_2(output2)
        los_1.backward()
        los_2.backward()
        opt.step()

I am using Python 3.6.5 and PyTorch 0.4.0


title: Pytorch detach() function failed to be excuated on different GPU severs

body:
Recently, our lab bought a new server with 9 GPUs and I want to run my programming on this machine. However, I do not change my right code and I got an unexpected error like the following.

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
  File "main.py", line 166, in <module>
    p_img.copy_(netG(p_z).detach())
  File "/usr/local/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/szhangcj/python/GBGAN/celebA_attention/sagan_models.py", line 100, in forward
    out,p1 = self.attn1(out)
  File "/usr/local/anaconda3/envs/tensorflow/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/szhangcj/python/GBGAN/celebA_attention/sagan_models.py", line 32, in forward
    energy =  torch.bmm(proj_query,proj_key) # transpose check
RuntimeError: cublas runtime error : the GPU program failed to execute at /opt/conda/conda-bld/pytorch_1535491974311/work/aten/src/THC/THCBlas.cu:411

However, I can run my programming successfully on the old machine with 4 GPUs. I am not sure what the problem is and it seems that the error is caused by the detach() function. My code is as the following.

z_b = torch.FloatTensor(opt.batch_size, opt.z_dim).to(device)
img_b = torch.FloatTensor(opt.batch_size, 3, 64, 64).to(device)
img_a = torch.FloatTensor(opt.batch_size, 3, 64, 64).to(device)
p_z = torch.FloatTensor(pool_size, opt.z_dim).to(device)
p_img = torch.FloatTensor(pool_size, 3, 64, 64).to(device)

## evaluation placeholder
show_z_b = torch.FloatTensor(100, opt.z_dim).to(device)
eval_z_b = torch.FloatTensor(250, opt.z_dim).to(device) # 250/batch * 120 --> 300000

optim_D = optim.Adam(netD.parameters(), lr=opt.lr_d) # other param?
optim_G = optim.Adam(netG.parameters(), lr=opt.lr_g) #?suitable
criterion_G = nn.MSELoss()

eta = 1
loss_GD = []
pre_loss = 0
cur_loss = 0
G_epoch = 1
for epoch in range(start_epoch, start_epoch + opt.num_epoch):
    print('Start epoch: %d' % epoch)
    ## input_pool: [pool_size, opt.z_dim] -> [pool_size, 32, 32]
    netD.train()
    netG.eval()
    p_z.normal_()
    # print(netG(p_z).detach().size())
    p_img.copy_(netG(p_z).detach())

    for t in range(opt.period):

        for _ in range(opt.dsteps):

            t = time.time()
            ### Update D
            netD.zero_grad()
            ## real
            real_img, _ = next(iter(dataloader)) # [batch_size, 1, 32, 32]
            img_b.copy_(real_img.squeeze().to(device))
            real_D_err = torch.log(1 + torch.exp(-netD(img_b))).mean()
            print("D real loss", netD(img_b).mean())
            # real_D_err.backward()

            ## fake
            z_b_idx = random.sample(range(pool_size), opt.batch_size)
            img_a.copy_(p_img[z_b_idx])
            fake_D_err = torch.log(1 + torch.exp(netD(img_a))).mean() # torch scalar[]
            loss_gp = calc_gradient_penalty(netD, img_b, img_a)
            total_loss = real_D_err + fake_D_err + loss_gp
            print("D fake loss", netD(img_a).mean())
            total_loss.backward()

            optim_D.step()

        ## update input pool
        p_img_t = p_img.clone().to(device)
        p_img_t.requires_grad_(True)
        if p_img_t.grad is not None:
            p_img_t.grad.zero_()
        fake_D_score = netD(p_img_t)

        fake_D_score.backward(torch.ones(len(p_img_t)).to(device))

        p_img = img_truncate(p_img + eta * p_img_t.grad)
        print("The mean of gradient", torch.mean(p_img_t.grad))


title: Size mismatch, m1: [1 x 5], m2: [7 x 100] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:752

body:
I am new to pytorch and trying to create a model, but I getting this error,

AttributeError                            Traceback (most recent call last)
<ipython-input-16-a0f31875b0ba> in <module>()
      1 for t in range(100):
      2     # Forward pass
----> 3     y_pred = model(X_train)
      4
      5     # Accuracy

AttributeError: 'numpy.ndarray' object has no attribute 'dim'

Here is my code:

x = np.array([2,4,6,18,20,30,50])
y = x * 2
print(x)
print(y)

shuffle_indices = torch.LongTensor(random.sample(range(0, len(x)),
len(x)))
x = x[shuffle_indices]
y = y[shuffle_indices]

x = torch.from_numpy(x).float()
y = torch.from_numpy(y.ravel()).long()

# Split datasets
test_start_idx = int(len(x) * 0.75)
X_train = x[:test_start_idx]
y_train = y[:test_start_idx]
X_test = x[test_start_idx:]
y_test = y[test_start_idx:]
print("We have %i train samples and %i test samples." % (len(X_train), len(X_test)))

  We have 5 train samples and 2 test samples.

class MLP(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(MLP, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, output_dim)

    def forward(self, x_in, apply_softmax=False):
        a_1 = F.relu(self.fc1(x_in))
        y_pred = self.fc2(a_1)

        if apply_softmax:
            y_pred = F.softmax(y_pred, dim=1)

        return y_pred

model = MLP(input_dim=len(X_train),
        hidden_dim=100,
        output_dim=len(set(y)))
print (model.named_modules)

  bound method Module.named_modules of MLP((fc1): Linear(in_features=7, out_features=100, bias=True)
    (fc2): Linear(in_features=100, out_features=7, bias=True)
  )

loss_fn = nn.CrossEntropyLoss()

optimizer = optim.Adam(model.parameters(), lr=1e-3)

def get_accuracy(y_pred, y_target):
    n_correct = torch.eq(y_pred, y_target).sum().item()
    accuracy = n_correct / len(y_pred) * 100
    return accuracy

for t in range(7):
    y_pred = model(X_train)
    _, predictions = y_pred.max(dim=1)
    accuracy = get_accuracy(y_pred=predictions.long(), y_target=y_train)
    loss = loss_fn(y_pred, y_train)
    if t%20==0:
        print ("epoch: {0} | loss: {1:.4f} | accuracy: {2:.1f}%".format(t, loss, accuracy))
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()


title: Simple policy gradients (REINFORCE) overfits one action when playing Atari Breakout

body:
Self-contained code: https://colab.research.google.com/drive/1HYEXMpicymPUySkhGOaCJdJ3pN4RzXYd

The problem: I am trying to use a CNN to play Atari breakout from pixels (Breakout-V0 OpenAI gym). I am trying to use the simple policy gradients algorithm, implemented in PyTorch, to do this. There are four possible actions in this game [<NO-OP>, <FIRE> (play), <LEFT>, <RIGHT>].

Expected results: I would expect the policy to learn to play <NO-OP>, <LEFT>, <RIGHT> with about equal probability, and only play <FIRE> on the first frame of the game.

Actual results: After ~4 weight updates, the network predicts ONE action with nearly 100% probability. This means the gradient vanishes and the policy never recovers.

What I've tried:

Only play random actions at the start of the game (take increasingly greedy actions that follow the policy). You can fiddle around with this in the notebook.
I have rewarded only one action, just to prove to myself that it learns to only play that one action. It does, so I think I can rule out any PyTorch specific implementation errors.
Introducing an entropy penalty to the loss to discourage high confidence in actions.

My understanding is that this should not be necessary to explicitly introduce random actions because action = categorical.sample() does this. And if one action becomes dominant, but does not lead to reward, it should be subsequently discouraged.

My thought was that my training batches were skewed so that taking the action <RIGHT> for example lead to more reward per episode than punishment, and hence it's likelihood kept increasing. I would have thought that taking totally random actions at the start of the game and only slowly beginning to listen to the policy would have fixed this, but in my experiments, it didn't.

I'm really very confused as to why this doesn't work. BIG thanks to anyone who can help. I tried to debug this with a PhD in RL for 5 hours yesterday and made no progress.

Extra Questions:

Is it common to play e.g. 1000 rollouts, then sample a batch randomly from this rollout buffer to learn from? My understanding of policy gradients was that the policy is updated after every episode.
Am I correct in thinking that in policy gradients, you only backprop through the neuron that you selected your action from, but the gradient gets distributed to ALL the network weights through the softmax?


title: Why does my PyTorch ModuleList object become a 'function' object after assigning it as a class variable?

body:
I really need your help with something annoying that I can't explain. I am fairly new to PyTorch and want to implementa a Module that requires the usage of ModuleList. However after instantiation of an empty ModuleList object, I can't use its append-method. Why?

self.modules = torch.nn.ModuleList([])
print(type(self.modules), type(torch.nn.ModuleList())) # -> <class 'method'> <class 'torch.nn.modules.container.ModuleList'>
for m in range(len(self.module_sizes)):

        prev_channels = 0 if m == 0 else self.module_out_channels[-m]
        vgg19_channels = self.vgg19_layer_channels[-(m+1)]
        module_out_channels = self.module_out_channels[-(m+1)]
        module_size = self.module_sizes[-(m+1)]

        self.modules.append(
            GeneratorModule(prev_channels=prev_channels, vgg19_channels=vgg19_channels, out_channels=module_out_channels, module_size=module_size, initial=(m==0))
            )

The error message says:

  File "main.py", line 90, in __init__
    self.modules.append(
AttributeError: 'function' object has no attribute 'append'

So far so good. I assume that as soon as I assign a ModuleList as a class variable to my custom nn.Module subclass, it becomes a 'function' object. So I changed the code in a way that the ModuleList is assigned to self.modules after the appending loop. But then it throws another error in a later piece of code, where it says:

for m, module in enumerate(self.modules):
     pass

TypeError: 'method' object is not iterable

I know that this should work, since I found other people do it this way on the internet.

I would be so happy if any more experienced PyTorch programmer can help me out. Thanks in advance!


title: An out of bounds index error when using Pytorch gather

body:
I have Two Tensors

I am trying to gather one from each row with the column being specified by these indices. So I am trying to get:

[0,0,0,0,0,0,0,0,0,0,0,0,0,0,1,1]

This is my code for this:

self.manDistMat.gather(1, state.unsqueeze(-1)))
self.manDistMat

being the 16x16 matrix and state.unsqueeze(-1) being the other matrix.

When I try this I get this error.

  RuntimeError: index 578437695752307201 is out of bounds for dimension 1 with size 16

What am I doing wrong?


title: Deep Learning NLP: "Efficient" BERT-like Implementations?

body:
I work in a legacy corporate setting where I only have 16 core 64GB VM to work with on an NLP project.  I have a multi-label NLP text classification problem where I would really like to utilize a deep representation learning model like BERT, RoBERTa, ALBERT, etc.

I have approximately 200,000 documents that need to be labeled and I have annotated set of about 2,000 to use as the ground truth for training/testing/fine tuning.  I also have a much larger volume of domain related documents to use for pre-training.  I will need to do the pre-training from scratch most likely, since this in a clinical domain.  I am also open to pre-trained models if they might have a chance working with just fine-tuning like Hugging Face, etc..

What models and their implementations that are PyTorch or Keras compatible would folks suggest as a starting point?  Or is this a computational non-starter with my existing compute resources?


title: PyTorch Extend Dataset: can't pickle local object

body:
In the context of creating my own Dataset to feed into a pytorch DataLoader,
I have designed a way to inherit from a class programmatically, so basically extending a class that's going to be used as a Dataset, in order to add 'custom' functionality to it. The dynamic extention works nicely. However, PyTorch doesn't like it, and when I start iterating the DataLoader based on it, it complains.
Here is a toy example for the extended class:
# Mock dataset. This has to be on a different file for some reason
from torch.utils.data import Dataset
class MyDataset(Dataset):
    def __init__(self):
        self.first = 1
        self.second = 2

    def __len__(self):
        return 1000

    def __getitem__(self, item):
        return self.first, self.second

import pickle
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
from utils import MyDataset

def extend_class(base_class):
    class B(base_class):
        def hello(self):
            print('Yo!')

    return B

if __name__ == '__main__':
    a = MyDataset()
    dataloader = DataLoader(a, batch_size=4, shuffle=True, num_workers=1)

    iterator = iter(dataloader)
    first, second = next(iterator)  # this works ok

    extended_class = extend_class(MyDataset)

    b = extended_class()
    b.hello() # this works!

    dataloader = DataLoader(b, batch_size=4, shuffle=True, num_workers=1)

    iterator = iter(dataloader)  # error here: AttributeError: Can't pickle local object 'extend_class.<locals>.B'

    first, second = next(iterator)

Any workaround to do this is appreciated!


title: clever image augmentation - random zoom out

body:
i'm building a CNN to identify facial keypoints. i want to make the net more robust, so i thought about applying some zoom-out transforms because most pictures have about the same location of keypoints, so the net doesn't learn much.
my approach:
i want augmented images to keep the original image size so apply MaxPool2d and then random (not equal) padding until the original size is reached.
first question
is it going to work with simple average padding or zero padding? i'm sure it would be even better if i made the padding appear more like a background but is there a simple way to do that?
second question
the keypoints are the target vector, they come as a row vector of 30. i'm getting confused with the logic needed to transform them to the smaller space.
generally if an original point was at (x=5,y=7) it transforms to (x=2,y=3)- i'm not sure about it but so far manually checked and it's correct. but what to do if to keypoints are in the same new pixel? i can't feed the network with less target values.
that's it. would be happy to hear your thoughts


title: Mask R-CNN optimizer and learning rate scheduler in Pytorch

body:
In the Mask R-CNN paper here the optimizer is described as follows training on MS COCO 2014/2015 dataset for instance segmentation (I believe this is the dataset, correct me if this is wrong)

We train on 8 GPUs (so effective minibatch
size is 16) for 160k iterations, with a learning rate of
0.02 which is decreased by 10 at the 120k iteration. We
use a weight decay of 0.0001 and momentum of 0.9. With
ResNeXt [45], we train with 1 image per GPU and the same
number of iterations, with a starting learning rate of 0.01.

I'm trying to write an optimizer and learning rate scheduler in Pytorch for a similar application, to match this description.
For the optimizer I have:
def get_Mask_RCNN_Optimizer(model, learning_rate=0.02):
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=0.0001)
    return optimizer

For the learning rate scheduler I have:
def get_MASK_RCNN_LR_Scheduler(optimizer, step_size):
    scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=step_size, gammma=0.1, verbose=True)
    return scheduler

When the authors say "decreased by 10" do they mean divide by 10? Or do they literally mean subtract by 10, in which case we have a negative learning rate, which seems odd/wrong. Any insights appreciated.


title: How do I train an LSTM in Pytorch?

body:
I am having a hard time understand the inner workings of LSTM in Pytorch.

Let me show you a toy example. Maybe the architecture does not make much sense, but I am trying to understand how LSTM works in this context.

The data can be obtained from here. Each row i (total = 1152) is a slice, starting from t = i until t = i + 91, of a longer time series. I will extract the last column of each row to use as labels.

import torch
import numpy as np
import pandas as pd
from torch import nn, optim
from sklearn.metrics import mean_absolute_error

data = pd.read_csv('data.csv', header = None).values
X = torch.tensor(data[:, :90], dtype = torch.float).view(1152, 1, 90)
y = torch.tensor(data[:, 90], dtype = torch.float).view(1152, 1, 1)

dataset = torch.utils.data.TensorDataset(X, y)
loader = torch.utils.data.DataLoader(dataset, batch_size = 50)

Then I am defining an LSTM regressor containing three LSTM layers with different structures.

class regressor_LSTM(nn.Module):
    def __init__(self):
        super().__init__()
        self.lstm1 = nn.LSTM(input_size = 49, hidden_size = 100)
        self.lstm2 = nn.LSTM(100, 50)
        self.lstm3 = nn.LSTM(50, 50, dropout = 0.3, num_layers = 2)
        self.dropout = nn.Dropout(p = 0.3)
        self.linear = nn.Linear(in_features = 50, out_features = 1)

    def forward(self, X):
        X, _ = self.lstm1(X)
        X = self.dropout(X)
        X, _ = self.lstm2(X)
        X = self.dropout(X)
        X, _ = self.lstm3(X)
        X = self.dropout(X)
        X = self.linear(X)

        return X

Initializing what needs to be initialized:

regressor = regressor_LSTM()
criterion = nn.MSELoss()
optimizer = optim.RMSprop(regressor.parameters())

Then training:

for epoch in range(25):
    acc_loss = 0.
    acc_mae = 0.
    for i, data in enumerate(loader):
        inputs, labels = data
        optimizer.zero_grad()
        outputs = regressor(inputs)
        loss = criterion(outputs, labels)
        loss.backward(retain_graph = True)
        optimizer.step()
        acc_loss += loss.item()
        mae = mean_absolute_error(labels.detach().cpu().numpy().flatten(), outputs.detach().cpu().numpy().flatten())
        acc_mae += mae
#       print('\rEPOCH {:3d} - Loop {:3d} of {:3d}: loss {:03.2f} - MAE {:03.2f}'.format(epoch+1, i+1, len(loader), loss, mae), end = '\r')
    print('\nEPOCH %3d FINISHED: loss %.5f - MAE %.5f' % (epoch+1, acc_loss/len(loader), acc_mae/len(loader)))

The thing is, after some initial decrease in both loss and MAE (expected behavior), both seem to get stuck (showing only first 10 epochs below):

EPOCH   1 FINISHED: loss 0.38506 - MAE 0.27322
EPOCH   2 FINISHED: loss 0.02825 - MAE 0.13601
EPOCH   3 FINISHED: loss 0.02593 - MAE 0.13117
EPOCH   4 FINISHED: loss 0.02568 - MAE 0.12705
EPOCH   5 FINISHED: loss 0.02546 - MAE 0.12920
EPOCH   6 FINISHED: loss 0.02502 - MAE 0.12763
EPOCH   7 FINISHED: loss 0.02445 - MAE 0.12659
EPOCH   8 FINISHED: loss 0.02310 - MAE 0.12328
EPOCH   9 FINISHED: loss 0.02277 - MAE 0.12237
EPOCH  10 FINISHED: loss 0.02352 - MAE 0.12476

When run with Keras, both metrics decrease consistently throughout the process. (I also noticed Keras takes much longer.)

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout, LSTM
import pandas as pd

data = pd.read_csv('data.csv', header = None).values
X = data[:, :90].reshape(1152, 90, 1)
y = data[:, 90]

regressor = Sequential()
regressor.add(LSTM(units = 100, return_sequences = True, input_shape = (90, 1)))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50, return_sequences = True))
regressor.add(Dropout(0.3))
regressor.add(LSTM(units = 50))
regressor.add(Dropout(0.3))
regressor.add(Dense(units = 1, activation = 'linear'))
regressor.compile(optimizer = 'rmsprop', loss = 'mean_squared_error', metrics = ['mean_absolute_error'])
regressor.fit(X, y, epochs = 25, batch_size = 32)

[OUTPUT]
Epoch 1/25
1152/1152 - 35s 30ms/sample - loss: 0.0307 - mean_absolute_error: 0.1225
Epoch 2/25
1152/1152 - 32s 28ms/sample - loss: 0.0156 - mean_absolute_error: 0.0978
Epoch 3/25
1152/1152 - 32s 28ms/sample - loss: 0.0126 - mean_absolute_error: 0.0871
Epoch 4/25
1152/1152 - 34s 30ms/sample - loss: 0.0111 - mean_absolute_error: 0.0806
Epoch 5/25
1152/1152 - 29s 25ms/sample - loss: 0.0103 - mean_absolute_error: 0.0785
Epoch 6/25
1152/1152 - 29s 25ms/sample - loss: 0.0088 - mean_absolute_error: 0.0718
Epoch 7/25
1152/1152 - 32s 27ms/sample - loss: 0.0085 - mean_absolute_error: 0.0699
Epoch 8/25
1152/1152 - 30s 26ms/sample - loss: 0.0069 - mean_absolute_error: 0.0640
Epoch 9/25
1152/1152 - 30s 26ms/sample - loss: 0.0077 - mean_absolute_error: 0.0660
Epoch 10/25
1152/1152 - 30s 26ms/sample - loss: 0.0070 - mean_absolute_error: 0.0644

I've been reading about hidden state initialization, I tried to set them to 0 in the beginning of the forward method (which, though, I understood to be the standard behavior), but nothing helped. I must confess that I do not understand what the parameters of an LSTM are, nor which should be reinitialized (if any) after each batch or epoch.

I appreciate any return!


title: Is there an easy way of implementing a radial basis function in pytorch?

body: I am a beginner in pytorch python library and want to implement  a Radial Baisis function(RBF) network. How does one look like in pytorch?


title: pytorch cnn model stop at loss.backward() without any prompt?

body:
My aim is to make a five-category text classification

I am running bert fine tuning with cnnbase model but my project stops at loss.backward() without any prompt in cmd.

My program runs successfully in rnn base such as lstm and rcnn.

But when I am running some cnnbase model a strange bug  appears.

My cnn model code:

import torch
import torch.nn as nn
import torch.nn.functional as F
# from ..Models.Conv import Conv1d
from transformers.modeling_bert import BertPreTrainedModel, BertModel
n_filters = 200
filter_sizes = [2,3,4]
class BertCNN(BertPreTrainedModel):
    def __init__(self, config):
        super(BertPreTrainedModel, self).__init__(config)
        self.num_filters = n_filters
        self.filter_sizes = filter_sizes
        self.bert = BertModel(config)
        for param in self.bert.parameters():
            param.requires_grad = True
        self.convs = nn.ModuleList(
            [nn.Conv2d(1, self.num_filters, (k, config.hidden_size))
                for k in self.filter_sizes])
        self.dropout = nn.Dropout(config.hidden_dropout_prob)
        self.fc_cnn = nn.Linear(self.num_filters *
                                len(self.filter_sizes), config.num_labels)

    def conv_and_pool(self, x, conv):
        x = F.relu(conv(x)).squeeze(3)
        x = F.max_pool1d(x, x.size(2)).squeeze(2)
        return x

    def forward(self, input_ids,
                attention_mask=None, token_type_ids=None, head_mask=None):
        outputs = self.bert(input_ids,
                            attention_mask=attention_mask,
                            token_type_ids=token_type_ids,
                            head_mask=head_mask)
        encoder_out, text_cls = outputs
        out = encoder_out.unsqueeze(1)
        out = torch.cat([self.conv_and_pool(out, conv)
                         for conv in self.convs], 1)
        out = self.dropout(out)
        out = self.fc_cnn(out)
        return out

My train code:

        for step, batch in enumerate(data):
            self.model.train()
            batch = tuple(t.to(self.device) for t in batch)
            input_ids, input_mask, segment_ids, label_ids = batch
            print("input_ids, input_mask, segment_ids, label_ids SIZE: \n")
            print(input_ids.size(), input_mask.size(),segment_ids.size(), label_ids.size())
            # torch.Size([2, 80]) torch.Size([2, 80]) torch.Size([2, 80]) torch.Size([2])
            logits = self.model(input_ids, segment_ids, input_mask)
            print("logits and label ids size: ",logits.size(), label_ids.size())
            # torch.Size([2, 5]) torch.Size([2])
            loss = self.criterion(output=logits, target=label_ids)
            if len(self.n_gpu) >= 2:
                loss = loss.mean()
            if self.gradient_accumulation_steps > 1:
                loss = loss / self.gradient_accumulation_steps
            if self.fp16:
                with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                    scaled_loss.backward()
                clip_grad_norm_(amp.master_params(self.optimizer), self.grad_clip)
            else:
                loss.backward() # I debug find that the program stop at this line without any error prompt

change the batchsize to 1
the bug still occured

the step1 logits ：

logits tensor([[ 0.8831, -0.0368, -0.2206, -2.3484, -1.3595]], device='cuda:1',
grad_fn=)

the step1 loss：

tensor(1.5489, device='cuda:1', grad_fn=NllLossBackward>)

but why can't loss.backward()?


title: What is the best file type to save a 4D tensor?

body:
I have some data I need to pre-process for a later step in a 3D Convolutional Network. The data comes in a file formatted like this:

   POSITION
x     y     z     (feature 1 x) (feature 1 y) (feature 1 z) (feature 2 x) (feature 2 y ...
1.2   0.54  2.3    0.04          0.2          -0.9           -0.2         0.65 ...
...(more rows of the same format)...

And after some other steps which involve operating on the positional data and the features, I get a pytorch tensor with dimensions [height][width][depth][features], or equivalently a numpy array, where the first three are positional data that I can use to plot the features using colours, and the [features] are vectors containing each of the feature values.

These are pretty large files and I'd like not have to perform the conversion from the first file format shown above to the tensor/array form later during processing. I'm thinking of using torch.save(tensor, 'file.pt').

My question is: what is the best file format to save this data so that it can be easily accessed later without the need for any pre-processing? Having to serialize it with PyTorch seems to be quite a convoluted way to save a type of data I would expect to have a more specific/designated file format.


title: What machine instance to use for running GPU workloads in Google Cloud Platform

body:
I am trying to run Elasticsearch BERT application and would like to understand the minimal configuration for fine-tuning the model using GPU. What machine configuration should I be using?
Reference github: Fast-Bert


title: torch.nn.CosineSimilarity --> IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

body:
I have the following code and I am trying to extract a 2048 dimension vector from the pretrained resnet50 network.
import torch
import torchvision
import torchvision.models as models
from PIL import Image
# Load the pretrained model
model = models.resnet50(pretrained=True)

# Use the model object to select the desired layer
layer = model._modules.get('avgpool')

# Set model to evaluation mode
model.eval()

transforms = torchvision.transforms.Compose([
    torchvision.transforms.Resize(256),
    torchvision.transforms.CenterCrop(224),
    torchvision.transforms.ToTensor(),
    torchvision.transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
])

def get_vector(image):
    # Create a PyTorch tensor with the transformed image
    t_img = transforms(image)

    # Create a vector of zeros that will hold our feature vector
    # The 'avgpool' layer has an output size of 512
    #my_embedding = torch.zeros(512)
    my_embedding = torch.zeros(2048)

    # Define a function that will copy the output of a layer
    def copy_data(m, i, o):
        my_embedding.copy_(o.flatten())                 # <-- flatten

    # Attach that function to our selected layer
    h = layer.register_forward_hook(copy_data)
    # Run the model on our transformed image
    with torch.no_grad():                               # <-- no_grad context
        model(t_img.unsqueeze(0))                       # <-- unsqueeze
    # Detach our copy function from the layer
    h.remove()
    # Return the feature vector
    return my_embedding
import torch.nn as nn
cos = nn.CosineSimilarity(dim=1, eps=1e-6)
img1 = Image.open('1.jpg')
img2 = Image.open('2.jpg')
img3 = Image.open('3.jpg')
output1 = cos(pic_vector1, pic_vector2)

I get the following error:
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-19-88106798854f> in <module>
----> 1 output1 = cos(pic_vector1, pic_vector2)

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/lib/python3.7/site-packages/torch/nn/modules/distance.py in forward(self, x1, x2)
     73
     74     def forward(self, x1: Tensor, x2: Tensor) -> Tensor:
---> 75         return F.cosine_similarity(x1, x2, self.dim, self.eps)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

How can I fix this?


I'm running into problems trying to use a PyTorch model exported as an ONNX model with Caffe2. Here is my export code

the_model = torchvision.models.densenet121(pretrained=True)
garbage, model_inputs = preprocessing("test.jpg")
torch_out = torch.onnx._export(the_model,
                           model_inputs,
                           "model_weights/chexnet-py.onnx",
                           export_params=True)

Now here is my testing code

model = onnx.load("model_weights/chexnet-py.onnx")
garbage, model_inputs = preprocessing("text.jpg")
prepared_backend = onnx_caffe2.backend.prepare(model)
W = {model.graph.input[0].name: model_inputs.numpy()}
c2_out = prepared_backend.run(W)[0]

This is returning the following error

ValueError: Don't know how to translate op Unsqueeze when running    converted PyTorch Model

Additional information
pytorch version 1.0.0a0+6f664d3
Caffe2 is latest version (attempted building from source, pip, and conda). All gave same result.


title: Pytorch errors: "received an invalid combination of arguments" in Jupyter Notebook

body:
I'm trying to learn Pytorch, but whenever I seem to try any online tutorial (https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html#sphx-glr-beginner-blitz-tensor-tutorial-py), I get errors when trying to run certain functions, but only in Jupyter Notebook.

When running

x = torch.empty(5, 3)

I get an error:

module 'torch' has no attribute 'empty'

Furthermore, when running

x = torch.zeros(5, 3, dtype=torch.long)

I get the error:

module 'torch' has no attribute 'long'

Some other functions work fine like:

x = torch.rand(5, 3)

But generally, most code I try to run seems to run into an error really quickly. I couldn't find any resolution online.

When I go into my docker container and simply run python in the shell, I can run these lines just fine with no errors.

I'm running pytorch in a Docker image that I extended from a fastai image, as it already included things like jupyter notebook and pytorch. I used anaconda to update everything, and committed it to a new image for myself.

I have absolutely no idea what the issue could be. I've tried updating packages through anaconda, pip, aptitude in my docker container, and making sure to commit my changes, but nothing seems to work. I also tried creating a new kernel with python 3.7 as I noticed that my Jupyter Notebook only runs in 3.6.4, and when I run python in the shell it is at 3.7.

I've also tried getting different docker images and extending them with what I need, but all images that I've tried have had errors with anaconda where it gets stuck on "Solving environment" step.


title: CRNN model for word data not learning anything after 10 epochs on 932 train images

body:
I am trying to predict words from images of Bangla words using CRNN approach. I am training it in a local environment with 932 images and 10 epochs. The loss is decreasing but accuracy is 0% . The model is kinda doing random guessing , and not learning anything . Following is my model :

import torch.nn as nn

class BidirectionalLSTM(nn.Module):

    def __init__(self, number_of_input, number_of_hidden, nunmer_of_out):
        super(BidirectionalLSTM, self).__init__()

        self.rnn = nn.LSTM(number_of_input, number_of_hidden, bidirectional=True)
        self.embedding = nn.Linear(number_of_hidden * 2, nunmer_of_out)

    def forward(self, input):
        recurrent, _ = self.rnn(input)
        T, b, h = recurrent.size()
        # print("size from recurrent forward :{} {} {}".format(T,b,h))
        t_rec = recurrent.view(T * b, h)
        # print("size after recurrent view : {}".format(t_rec.size()))
        output = self.embedding(t_rec)  # [T * b, nOut]

        output = output.view(T, b, -1)
        return output

class CRNN(nn.Module):

    def __init__(self, imgH, nc, nclass, nh, n_rnn=2, leakyRelu=False):
        super(CRNN, self).__init__()
        # assert imgH % 16 == 0, 'imgH has to be a multiple of 16'

        ks = [3, 3, 3, 3, 3, 3, 2]  # original [3, 3, 3, 3, 3, 3, 2]
        ps = [1, 1, 1, 1, 1, 1, 0]
        ss = [1, 1, 1, 1, 1, 1, 1]
        nm = [64, 128, 256, 256, 512, 512, 512]

        cnn = nn.Sequential()

        def conv_relu(i, batchNormalization=False):
            nIn = nc if i == 0 else nm[i - 1]
            nOut = nm[i]
            cnn.add_module('conv{0}'.format(i),
                           nn.Conv2d(nIn, nOut, ks[i], ss[i], ps[i]))
            if batchNormalization:
                cnn.add_module('batchnorm{0}'.format(i), nn.BatchNorm2d(nOut))
            if leakyRelu:
                cnn.add_module('relu{0}'.format(i),
                               nn.LeakyReLU(0.2, inplace=True))
            else:
                cnn.add_module('relu{0}'.format(i), nn.ReLU(True))

        conv_relu(0)
        cnn.add_module('pooling{0}'.format(0), nn.MaxPool2d(2, 2))  # 64x16x64 original
        conv_relu(1)
        cnn.add_module('pooling{0}'.format(1), nn.MaxPool2d(2, 2))  # 128x8x32 original
        conv_relu(2, True)
        conv_relu(3)
        cnn.add_module('pooling{0}'.format(2),
                       nn.MaxPool2d((2, 2), (2, 1), (0, 1)))  # 256x4x16 original
        conv_relu(4, True)
        conv_relu(5)
        cnn.add_module('pooling{0}'.format(3),
                       nn.MaxPool2d((2, 2), (2, 1), (0, 1)))  # 512x2x16 original
        conv_relu(6, True)  # 512x1x16
        # Extra max pull to bring down height to 1
        cnn.add_module('pooling{0}'.format(4),
                       nn.MaxPool2d((3, 3), (2, 4), (0, 1))) # final size 512x2x32
        self.cnn = cnn
        self.rnn = nn.Sequential(
            BidirectionalLSTM(512, nh, nh),
            BidirectionalLSTM(nh, nh, nclass))

    def forward(self, input):
        # conv features
        conv = self.cnn(input)
        b, c, h, w = conv.size()
        # assert h == 1, "the height of conv must be 1"
        # print('size before squeeze')
        # print(conv.size())
        conv = conv.squeeze(2)
        # print('size after squeeze')
        # print(conv.size())
        conv = conv.permute(2, 0, 1)  # [w, b, c]

        # rnn features
        output = self.rnn(conv)
        # print("from model {}".format(output))
        return output

And following is my main :

from data_preparation.dataloader import DataSetOCR
from utills.dataloader_services import *
from torch.utils.data import DataLoader
import parameters
from models import crnn
import torch.optim as optim
from torch.autograd import Variable
from torch.nn import CTCLoss
from utills.string_label_converter import averager, StrLabelConverter
import torch.nn.functional as F

train_dataset = DataSetOCR(
    csv_file_path= parameters.train_csv_path,
    text_file_path= parameters.text_file_path,
    root_directory= parameters.train_root)
assert train_dataset

test_dataset = DataSetOCR(
    csv_file_path= parameters.test_csv_path,
    text_file_path= parameters.text_file_path,
    root_directory= parameters.test_root)
assert test_dataset

dataloader_params = {
    'batch_size': 2,
    'shuffle': True,
    'collate_fn': my_collate

}

train_loader = DataLoader(train_dataset, **dataloader_params)
train_iter = iter(train_loader)

# custom weights initialization called on crnn
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

crnn = crnn.CRNN(parameters.max_image_height, 3, parameters.number_of_classes, 256)
crnn.apply(weights_init)
criterion = CTCLoss()
optimizer = optim.Adam(crnn.parameters(), lr=0.001)
loss_avg = averager()

image = torch.FloatTensor(2, 3, parameters.max_image_width, parameters.max_image_height)
text = torch.LongTensor(2 * 5)
length = torch.LongTensor(2)

string_converter = StrLabelConverter()
# string_converter.convert_integer_to_string()

def val(net, dataset, criterion, max_iter=100):
    print('Start val')

    for p in crnn.parameters():
        p.requires_grad = False

    net.eval()
    data_loader = torch.utils.data.DataLoader(
        dataset, shuffle=True, batch_size=2, collate_fn=my_collate)
    val_iter = iter(data_loader)

    i = 0
    n_correct = 0
    loss_avg = averager()

    max_iter = min(max_iter, len(data_loader))
    # print('length of dataloader')
    # print(len(data_loader))
    for i in range(max_iter):
        data = val_iter.next()
        i += 1
        images, texts = data
        batch_size = images.size(0)
        # print(images)
        # print(image)
        loadData(image, images)
        t, l = string_converter.convert_string_to_integer(texts, [])
        loadData(text, t)
        loadData(length, l)
        # print('actual label')
        # print(text)
        # print(text.size())

        # print('input from val')
        # print(i)
        print(image)
        preds = crnn(image)
        preds = F.log_softmax(preds, 2)
        print(preds)
        preds_size = Variable(torch.IntTensor([preds.size(0)] * batch_size))
        # print('predictions')
        # print(preds.size())
        cost = criterion(preds, text, preds_size, length) / batch_size
        print('validation {}'.format(cost))
        loss_avg.add(cost)

        # print('predictions before max')
        # print(preds.size())
        _, preds = preds.max(2)
        # print('after max function')
        # print(preds)
        # print(preds.size())
        preds = preds.squeeze(1) # originial 2
        # print('after squeeze')
        # print(preds.size())
        preds = preds.transpose(1, 0).contiguous().view(-1)
        # print('after transpose')
        # print(preds.size())
        sim_preds = string_converter.convert_integer_to_string(preds.data, preds_size.data)

        cpu_texts = string_converter.convert_integer_to_string(text, length)

        # for pred, target in zip(sim_preds, cpu_texts):
        #     # if pred == target:
        #     #     n_correct += 1
        #     print((pred, target))
    #
    # raw_preds = string_converter.convert_integer_to_string(preds.data, preds_size.data)[:2]
    # for raw_pred, pred, gt in zip(raw_preds, sim_preds, cpu_texts):
    #     print('%-20s => %-20s, gt: %-20s' % (raw_pred, pred, gt))
    #
    # accuracy = n_correct / float(max_iter * 100)
    # print('Test loss: %f, accuray: %f' % (loss_avg.val(), accuracy))

def trainBatch(train_iter, criterion, optimizer):
    try:
        data = train_iter.next()
    except StopIteration:
        train_iter = iter(train_loader)
        data = train_iter.next()

    images,texts = data
    loadData(image,images)
    t,l =string_converter.convert_string_to_integer(texts, [])
    loadData(text, t)
    loadData(length,l)
    batch_size = dataloader_params['batch_size']
    optimizer.zero_grad()
    # # test_util.loadData(image, data['image'])
    # text = Variable(data['integer_sequence_label'])
    # print("from train {}".format(image))
    preds = crnn(image)
    preds = F.log_softmax(preds, 2)
    preds_size = Variable(torch.IntTensor([preds.size(0)] * batch_size))

    cost = criterion(preds, text, preds_size, length) / batch_size
    # print('cost from train {}'.format(cost))
    crnn.zero_grad()
    cost.backward()
    optimizer.step()
    return cost

for epoch in range(25):
    i = 0
    while i < len(train_loader):
        for p in crnn.parameters():
            p.requires_grad = True
        crnn.train()

        cost = trainBatch(train_iter, criterion, optimizer)
        loss_avg.add(cost)
        i += 1

        if i % 50 == 0:
            print('[%d/%d][%d/%d] Loss: %f' % (epoch, 25, i, len(train_loader), loss_avg.val()))
            loss_avg.reset()

        if i % 50 == 0:
            val(crnn, test_dataset, criterion)

        # #do checkpointing
        # if i % 1 == 0:
        #     torch.save(crnn.state_dict(), '{0}/netCRNN_{1}_{2}.pth'.format('/home/bjit-531/PycharmProjects/python/bangla-ocr-version-2/bangla-ocr-version-2/weights/', epoch, i))
        #

I am generating the images in the data set with various fonts so they have different height and width. I am padding them during data loading , so the model receives each image with size 90 x 752 .
What am i doing wrong here? Why the model accuracy isn't improving ?


title: Torchscript incompatible with torch.cat for tensor lists

body:
Torch.cat throws error for tensor lists when used within torchscript

Here is a minimum reproducable example to reproduce the error

import torch
import torch.nn as nn

"""
Smallest working bug for torch.cat torchscript
"""

class Model(nn.Module):
    """dummy model for showing error"""

    def __init__(self):
        super(Model, self).__init__()
        pass

    def forward(self):
        a = torch.rand([6, 1, 12])
        b = torch.rand([6, 1, 12])
        out = torch.cat([a, b], axis=2)
        return out

if __name__ == '__main__':
    model = Model()
    print(model())  # works
    torch.jit.script(model)  # throws error

The expected result would be a torchscript output for torch.cat. Here is the error message provided:

File "/home/anil/.conda/envs/rnn/lib/python3.7/site-packages/torch/jit/__init__.py", line 1423, in _create_methods_from_stubs
    self._c._create_methods(self, defs, rcbs, defaults)
RuntimeError:
Arguments for call are not valid.
The following operator variants are available:

  aten::cat(Tensor[] tensors, int dim=0) -> (Tensor):
  Keyword argument axis unknown.

  aten::cat.out(Tensor[] tensors, int dim=0, *, Tensor(a!) out) -> (Tensor(a!)):
  Argument out not provided.

The original call is:
at smallest_working_bug_torch_cat_torchscript.py:19:14
    def forward(self):
        a = torch.rand([6, 1, 12])
        b = torch.rand([6, 1, 12])
        out = torch.cat([a, b], axis=2)
              ~~~~~~~~~ <--- HERE
        return out

Kindly let me know of a fix or a workaround for this problem.

Thanks!