title: CUDA out of memory

body:
I am getting error when trying to run BERT model for NER task.
"CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 3.82 GiB total capacity; 2.58 GiB already allocated; 25.38 MiB free; 6.33 MiB cached)I have also tried reducing batch size to 1c

enter code here
epochs = 10
max_grad_norm = 1.0

for _ in trange(epochs, desc="Epoch"):
    # TRAIN loop
    model.train()
    tr_loss = 0
    nb_tr_examples, nb_tr_steps = 0, 0
    for step, batch in enumerate(train_dataloader):
        # add batch to gpu
        batch = tuple(t.to(device) for t in batch)
        b_input_ids, b_input_mask, b_labels = batch
        # forward pass
        loss = model(b_input_ids, token_type_ids=None,
                     attention_mask=b_input_mask, labels=b_labels)
        # backward pass
        loss.backward()
        # track train loss
        tr_loss += loss.item()
        nb_tr_examples += b_input_ids.size(0)
        nb_tr_steps += 1
        # gradient clipping
        torch.nn.utils.clip_grad_norm_(parameters=model.parameters(), max_norm=max_grad_norm)
        # update parameters
        optimizer.step()
        model.zero_grad()
    # print train loss per epoch
    #print("Train loss: {}".format(tr_loss/nb_tr_steps))
    # VALIDATION on validation set
    model.eval()
    eval_loss, eval_accuracy = 0, 0
    nb_eval_steps, nb_eval_examples = 0, 0
    predictions , true_labels, true_inputs = [], [],[]
    for batch in valid_dataloader:
        batch = tuple(t.to(device) for t in batch)
        b_input_ids, b_input_mask, b_labels = batch

        with torch.no_grad():
            tmp_eval_loss = model(b_input_ids, token_type_ids=None,
                                  attention_mask=b_input_mask, labels=b_labels)
            logits = model(b_input_ids, token_type_ids=None,
                           attention_mask=b_input_mask)
        logits = logits.detach().cpu().numpy()
        label_ids = b_labels.to('cpu').numpy()
        inputs = b_input_ids.to('cpu').numpy()

        true_inputs.append(inputs)
        predictions.extend([list(p) for p in np.argmax(logits, axis=2)])
        true_labels.append(label_ids)

        tmp_eval_accuracy = flat_accuracy(logits, label_ids)

        eval_loss += tmp_eval_loss.mean().item()
        eval_accuracy += tmp_eval_accuracy

        nb_eval_examples += b_input_ids.size(0)
        nb_eval_steps += 1
    eval_loss = eval_loss/nb_eval_steps
    pred_tags = [tags_vals[p_i] for p in predictions for p_i in p]
    valid_tags = [tags_vals[l_ii] for l in true_labels for l_i in l for l_ii in l_i]
    valid_inputs = [[idx2word[l_ii] for l_ii in l_i] for l in  true_inputs  for l_i in l ]

    print("F1-Score: {}".format(f1_score(pred_tags, valid_tags)))
    print("Validation loss: {}".format(eval_loss))
    print("Validation Accuracy: {}".format(eval_accuracy/nb_eval_steps))

Attached is the output of nvidia-smi:-

nvidia-smi


title: Using Transforms in PyTorch

body: I have a tensor X of Cat/No cat images in PyTorch and wanted to apply Transformations on it. Is there any way to so without data loaders?


title: How can torchaudio.transform.Resample be called without __call__ function inside?

body:
if sample_rate != sr:
        waveform = torchaudio.transforms.Resample(sample_rate, sr)(waveform)
        sample_rate = sr

I was wondering how this Resamle works in there. So took a look at the docs of torchaudio. I thought there would be __call__ function. Because Resample is used as a function. I mean that Resample()(waveform). But inside, there are only __init__ and forward function. I think the forward function is the working function but I don't know why it is named 'forward' not __call__.  What am I missing?
class Resample(torch.nn.Module):
    r"""Resample a signal from one frequency to another. A resampling method can be given.

    Args:
        orig_freq (float, optional): The original frequency of the signal. (Default: ``16000``)
        new_freq (float, optional): The desired frequency. (Default: ``16000``)
        resampling_method (str, optional): The resampling method. (Default: ``'sinc_interpolation'``)
    """

    def __init__(self,
                 orig_freq: int = 16000,
                 new_freq: int = 16000,
                 resampling_method: str = 'sinc_interpolation') -> None:
        super(Resample, self).__init__()
        self.orig_freq = orig_freq
        self.new_freq = new_freq
        self.resampling_method = resampling_method

    def forward(self, waveform: Tensor) -> Tensor:
        r"""
        Args:
            waveform (Tensor): Tensor of audio of dimension (..., time).

        Returns:
            Tensor: Output signal of dimension (..., time).
        """
        if self.resampling_method == 'sinc_interpolation':

            # pack batch
            shape = waveform.size()
            waveform = waveform.view(-1, shape[-1])

            waveform = kaldi.resample_waveform(waveform, self.orig_freq, self.new_freq)

            # unpack batch
            waveform = waveform.view(shape[:-1] + waveform.shape[-1:])

            return waveform

        raise ValueError('Invalid resampling method: %s' % (self.resampling_method))

--edit--
I looked around torch.nn.module. There is no def __call__. But only
__call__ : Callable[..., Any] = _call_impl Would it be the way?


title: Custom data loader for images and labels in pytorch. Difficulty with inputs in loss function

body:
I was working on this https://www.kaggle.com/gti-upm/leapgestrecog data set lately. Its a hand gesture dataset and I was trying to make a classifier. Due to images available in different types of folder I made my on data loader. Here it is

class DatasetLoader(Dataset):

  def __init__(self,path):
    self.path_list = path
    self.labels = []
    self.to_tensor = transforms.ToTensor()
    self.resize = transforms.Resize((120,320))
    self.gray = transforms.Grayscale(num_output_channels = 1)
    self._init_dataset()

 def _init_dataset(self):
    labels = set()
    for diro in os.listdir("/kaggle/input/leapgestrecog/leapGestRecog"):
      for d in os.listdir(os.path.join("/kaggle/input/leapgestrecog/leapGestRecog",diro)):
        if len(d.split('_'))>2:
          labels.add("_".join(d.split("_")[-2:]))
        else:
          labels.add(d.split("_")[-1])
    labels = list(labels)
    ## help me on this line with some codes

  def __getitem__(self,idx):
    if torch.is_tensor(idx):
      idx = idx.tolist()
    img_name = self.path_list[idx]
    img = Image.open(img_name)
    img = self.resize(img)
    img = self.gray(img)
    img = self.to_tensor(img)
    if len(img_name.split('/')[-2].split('_')) > 2:
      label = "_".join(img_name.split('/')[-2].split('_')[-2:])
    else:
      label = img_name.split('/')[-2].split('_')[-1]
    label = ## Here also
    return img,label

  def __len__(self):
    return len(self.path_list)

I have problem with label which I am getting from this dataset loader. As I have created a model which takes n batches of data with 10 classes so during loss calculation I need my labels to of size(n,10). I dont know what to do. Here is my network design:

class Net(nn.Module):
    def __init__(self):

        super(Net,self).__init__()
        self.conv1 = nn.Conv2d(1,32,5)
        self.pool = nn.MaxPool2d(2,2)
        self.conv2 = nn.Conv2d(32,64,3)
        self.conv3 = nn.Conv2d(64,64,3)
        self.fc1 = nn.Linear(64*38*13,128)
        self.fc2 = nn.Linear(128,10)

    def forward(self,x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = x.view(64,64*38*13)
        x = F.relu(self.fc1(x))

        return F.log_softmax(self.fc2(x),dim = 1)

If y is label of an image. To train our network we feed loss function with y and output. But output we get is of size (64,10) , so I need help with the label in dataloader


title: Convert tensor list to tensor of tensors

body:
I have a variable a which is a bunch of tensors like this:

[tensor([0.0014, 0.0021, 0.0015, 0.0007, 0.0012, 0.0024, 0.0021, 0.0019, 0.0010,
        0.0010])]
[tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0])]

....

And when I want to do this as part of my code:

x = torch.tensor(a, dtype=torch.float)

I got this error:

ValueError: only one element tensors can be converted to Python scalars

Which I assumed that maybe I need to convert each tensor in a like this:

[tensor([[0.0014], [0.0021], [0.0015], [0.0007], [0.0012], [0.0024], [0.0021], [0.0019], [0.0010],
        [0.0010]])]
[tensor([[0], [0], [0], [0], [0], [0], [0], [0], [0], [0]])]

Is what I think right? or what I need exactly to avoid the error above?

Any help, please?


title: How to run a pre-trained pytorch model on the GPU?

body:
Here I am trying to use the mobilenetv2 mobile to train on a custom dataset. I can get it to work on the CPU, but I would prefer to run it on the GPU. Instead, I am getting errors like these:

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #2 'weight'

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #4 'mat1

So like my Post asks how can I get the pre-trained model to run on the GPU?

MobileNet = models.mobilenet_v2(pretrained = True)
if torch.cuda.is_available():
    MobileNet.cuda()

for param in MobileNet.parameters():
    param.requires_grad = False

    torch.manual_seed(50)

MobileNet.classifier = nn.Sequential(nn.Linear(1280, 1000), nn.ReLU(), nn.Dropout(0.5), nn.Linear(1000,3), nn.LogSoftmax(dim=1))

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(MobileNet.classifier.parameters(), lr=0.001)

train_transform = transforms.Compose([
        transforms.RandomRotation(10),      # rotate +/- 10 degrees
        transforms.RandomHorizontalFlip(),  # reverse 50% of images
        transforms.Resize(224),             # resize shortest side to 224 pixels
        transforms.CenterCrop(224),         # crop longest side to 224 pixels at center
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])
    ])

test_transform = transforms.Compose([
        transforms.Resize(224),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406],
                             [0.229, 0.224, 0.225])
    ])

train_data = datasets.ImageFolder('C:/Users/mixv/Pictures/Summer/datasets/train', transform=train_transform)
test_data = datasets.ImageFolder('C:/Users/mix/Pictures/Summer/datasets/test', transform=test_transform)

torch.manual_seed(42)
batch=64
train_loader = DataLoader(train_data, batch_size=batch, shuffle=True)
test_loader = DataLoader(test_data, batch_size=batch, shuffle=True)

if torch.cuda.is_available():
    train_loader = DataLoader(train_data, batch_size=batch, shuffle=True, pin_memory = True)
    test_loader = DataLoader(test_data, batch_size=batch, shuffle=True, pin_memory = True)

epochs = 10

train_losses = []
test_losses = []
train_correct = []
test_correct = []
start_time =time.time()
for i in range(epochs):
    trn_corr = 0
    tst_corr = 0

    # Run the training batches

    for b, (images, labels) in enumerate(train_loader):

        if torch.cuda.is_available():
            images = images.cuda()
            labels = labels.cuda()

        b+=1

        # Apply the model
        y_pred = MobileNet(images)
        loss = criterion(y_pred, labels)

        # Tally the number of correct predictions
        predicted = torch.max(y_pred.data, 1)[1]
        batch_corr = (predicted == labels).sum()
        trn_corr += batch_corr

        accuracy = trn_corr.item()*100/(b*batch)
        # Update parameters
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()


title: Pytorch reading tensors from file of tensors (stream training from disk)

body:
I have some really big input tensors and I was running into memory issues while building them, so I read them one by one into a .pt file. As I run the script that generates and saves the file, the file gets bigger and bigger, so I am assuming that the tensors are saving correctly. Here is that code:
with open(a_sync_save, "ab") as f:
     print("saved")
     torch.save(torch.unsqueeze(torch.cat(tensors, dim=0), dim=0), f)

I want to read a certain amount of these tensors from the file at a time, because I do not want to run into a memory issue again. When I try to read each tensor saved to the file I can only manage to get the first tensor.
with open(a_sync_save, "rb") as f:
    for tensor in torch.load(f):
        print(tensor.shape)

The output here is the shape of the first tensor, then quits peacfully.


title: removing sample from a Pytorch dataset object

body:
I'm loading images to pytorch using:
train_data = torchvision.datasets.ImageFolder(train_dir, transform=transform)

Which has some samples I want to omit.
Can i remove certain images from the loader? (by index? by name?)
Thx


title: why does pysptk error appear and how to correct it?

body:
I've got this error:

ModuleNotFoundError: No module named 'pysptk'

What does this error mean?

I tried installing pysptk using pip but did not install. I tried to install pysptk file from Github that didn't work too.


title: Training Mask RCNN but always get 0 positive bounding boxes from all proposals?

body:
I've just implemented a Mask RCNN in Pytorch from scratch and here is the code: https://github.com/arthurhero/Mask_R-CNN_Pytorch/blob/master/masker_ops.py

I have only one GTX1080 and one Quadro6000 on two separate machines, so I can only train with 2 imgs per iteration. My questions is, how long does it usually take for Mask RCNN to converge? And how can I tell whether it's on the right track or not?

Right now I'm only training the RPN head (so it's the first of the four "pragmatic steps"). After one night's training, this model still gives me a lot of "0 positive bboxes" notice, i.e., when it assigns labels to each bbox for a img, all of them are negative (does not have large enough iou with a ground-truth bbox). And it seems the number of "0 positive bboxes" notice did not decrease as the training goes at all.

I've thought of several reasons.

Right now, I will skip the whole batch if one of the img has "0 positive bboxes", just because it will make the tensor shape strange for later computations. Is this how people usually deal with this kind of situation? Is it bringing too much inefficiency to training?
The learning rate might be too big? I'm currently using 0.02 from the paper. But since my batch size is only 2, is this learning rate too huge?
I have a bit doubt in the way the offsets are applied to the anchors... Currently, the RPN head returns dy,dx,logh,logw relative to the anchor box. And the code for applying these offsets to anchor box is like this:

center_y = center_y + (dy*height)
 center_x = center_x + (dx*width)
 height = height * torch.exp(dh)
 width = width * torch.exp(dw)

The resulting bbox might be well out of the boundary of the original anchor (if dy and dx are large). But when this bbox is assigned "negative", the class label at the center of the original anchor will receive punishment if the label is positive. This does not look fair to me...... Am I having a misunderstanding here? Or does it just not matter? Should we put some hard constraint on those offsets so the resulting bbox cannot go too far away from the anchor?
Too many for loops. This is a problem for the ROIAlign layer and Proposal filter layer. Since each img might have different numbers of filtered proposals, and also just because it's hard to manipulate a 5D tensor... I usually just split the batch of bboxes or proposals along the batch axis, and use a for loop to deal with the set of proposals for one img at a time. But from my limited knowledge of OS, GPU's seem to be bad at doing serial work. Are those for loops dragging the speed of training? And is there a better way to implement those computations on 5d tensors without a for loop?
Too few GPU's. The fundamental question: is it even possible to train a Mask RCNN on only one GPU? If not... what are the recommendations for a object masker with cheaper cost? I heard YOLO is also very computationally expensive...

Many, many thanks!!

-Z


title: Create multiple images with different ratio PyTorch

body:
I'm trying to perform some digit recognition using PyTorch. I have implemented a convolutional version of the sliding window with size 32x32. Which makes me able to identify digits of this range of size in a picture.

But now let's imagine I have an image of size 300x300 with a digit that occupies the whole image. I will never be able to identify it...

I have seen people saying that the image needs to be rescaled and resized. Meaning that I need to create various scaled versions of my initial image and then to feed my network with those "new" images.

Does anyone have any idea how I can perform that?

Here is a part of my code, if it can help..

# loading dataset
size=200
height=200
width= 300

transformer_svhn_test = transforms.Compose([
    transforms.Grayscale(3),
    transforms.Resize((height, width)),
    transforms.CenterCrop((size, size)),
    transforms.ToTensor(),
    transforms.Normalize([.5,.5,.5], [.5,.5,.5])
])

SVHN_test = SVHN_(train=False, transform=transformer_svhn_test)
SVHN_test_loader = DataLoader(SVHN_test, batch_size=batch_size, shuffle=False, num_workers=3)

#loading network
model = Network()
model.to(device)
model.load_state_dict(torch.load("digit_classifier_gray_scale_weighted.pth"))

# loading one image and feeding the model with it
image = next(iter(SVHN_test_loader))[0]
image_tensor = image.unsqueeze(0) # creating a single-image batch
image_tensor = image_tensor.to(device)

model.eval()
output = model(image_tensor)


title: how to solve dimension mismatch error for CNN on images in pyTorch?

body:
I have input data (colored images) in the shape of (100, 64, 64, 3) and tried to train a CNN with 2 conv/pooling layers on it for binary classification. I keep encountering size mismatch error. Also attempted reshape the images into (-1, 3, 64, 64) size

class SimpleCNN(nn.Module):

    def __init__(self, input_dim, hidden_dim, output_dim, kernel_size):
        super(SimpleCNN, self).__init__()

        self.conv1 = nn.Conv2d(3, 10, kernel_size, padding=0)
        self.conv2 = nn.Conv2d(10, 20, kernel_size, padding=0)
        self.fc1 = nn.Linear(hidden_dim*16*16, hidden_dim)
        self.fc2 = nn.Linear(output_dim, output_dim)

    def forward(self, x):
        x = F.max_pool2d((self.conv1(x)), 2)
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return F.softmax(x)

net = SimpleCNN(in_channels, hidden_dim, out_channels, kernel_size)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

losses = []
idx = []
count=1
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        images = Variable(images.view(-1, 3, n_pixel, n_pixel))
        labels = Variable(labels)

        # Intialize the hidden weight to all zeros
        optimizer.zero_grad()
        # Forward pass: compute the output class given a image
        outputs = net(images)
        # Compute the loss: difference between the output class and the pre-given label
        loss = criterion(outputs, labels)
        # Backward pass: compute the weight
        loss.backward()
        # Optimizer: update the weights of hidden nodes
        optimizer.step()
        count+=1

        if (i+1) % 10 == 0:
            idx.append(count)
            losses.append(loss.data.numpy().tolist())

The full error msg is

  only batches of spatial targets supported (3D tensors) but got targets of dimension: 1 at /pytorch/aten/src/THNN/generic/SpatialClassNLLCriterion.c:61'


sal_maps_hf.shape
(11, 32, 32, 3)

I want to change the dtype of sal_maps_hf to nint8 so

sal_maps_hf.dtype
dtype('<f4')

sal_maps_hf = sal_maps_hf.astype(np.uint8)
print(sal_maps_hf.dtype)

  --------------------------------------------------------------------------- AttributeError                            Traceback (most recent call
  last)  in
  ----> 1 print(sal_maps_hf.dtype)

  AttributeError: 'AstypeContext' object has no attribute 'dtype'


title: Make PyTorch variables to float64

body:
How to make all the variables created in a PyTorch file to float64?
Is there a single line of code which can do that?


title: how to get alignment or attention information for translations produced by a torch hub model?

body:
The torch hub provides pretrained models, such as: https://pytorch.org/hub/pytorch_fairseq_translation/
These models can be used in python, or interactively with a CLI.
With the CLI it is possible to get alignments, with the --print-alignment flag. The following code works in a terminal, after installing fairseq (and pytorch)
curl https://dl.fbaipublicfiles.com/fairseq/models/wmt14.v2.en-fr.fconv-py.tar.bz2 | tar xvjf -
MODEL_DIR=wmt14.en-fr.fconv-py
fairseq-interactive \
    --path $MODEL_DIR/model.pt $MODEL_DIR \
    --beam 5 --source-lang en --target-lang fr \
    --tokenizer moses \
    --bpe subword_nmt --bpe-codes $MODEL_DIR/bpecodes \
    --print-alignment

In python it is possible to specify the keyword args verbose and print_alignment:
import torch

en2fr = torch.hub.load('pytorch/fairseq', 'transformer.wmt14.en-fr', tokenizer='moses', bpe='subword_nmt')

fr = en2fr.translate('Hello world!', beam=5, verbose=True, print_alignment=True)

However, this will only output the alignment as a logging message. And for fairseq 0.9 it seems to be broken and results in an error message (issue).
Is there a way to access alignment information (or possibly even the full attention matrix) from python code?


title: Torch Tensor & Input Conflicting: "Tensor Object Is Not Callable"

body:
Due to the code "torch.tensor," I am getting the error "Tensor object is not callable" when I add "input." Does anyone know how I can fix this?
import torch
from torch.nn import functional as F
from transformers import GPT2Tokenizer, GPT2LMHeadModel

tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
model = GPT2LMHeadModel.from_pretrained('gpt2')

text0 = "In order to"
text = tokenizer.encode("In order to")
input, past = torch.tensor([text]), None

logits, past = model(input, past = past)
logits = logits[0,-1]
probabilities = torch.nn.functional.softmax(logits)
best_logits, best_indices = logits.topk(5)
best_words = [tokenizer.decode([idx.item()]) for idx in best_indices]
text.append(best_indices[0].item())
best_probabilities = probabilities[best_indices].tolist()

for i in range(5):
    f = ('Generated {}: {}'.format(i, best_words[i]))
    print(f)

option = input("Pick a Option:")
z = text0.append(option)
print(z)

Error stacktrace:
TypeError                                 Traceback (most recent call last)

<ipython-input-2-82e8d88e81c1> in <module>()
     25
     26
---> 27 option = input("Pick a Option:")
     28 z = text0.append(option)
     29 print(z)

TypeError: 'Tensor' object is not callable


title: How to process TransformerEncoderLayer output in pytorch

body:
I am trying to use bio-bert sentence embeddings for text classification of longer pieces of text.
As it currently stands I standardize the number of sentences in each piece of text (some sentences are only comprised of ("[PAD]") and run each sentence through biobert to get sentence vectors as they do here:
https://pypi.org/project/biobert-embedding/
I then run those embeddings through a TrasnformerEncoder with 8 layers and 16 attention heads.
The TrasnformerEncoder outputs something of shape (batch_size, num_sentences, embedding_size).
I then try to decode this with a Linear layer and map it to my classes (of which there are 7) and softmax the output to get probabilities.
My loss function is simply nn.CrossEntropyLoss().
At first, I summed over dimensions 1 of the TransformerEncoder output to get something of size (batch_size, embedding_size). This invariable led to my network converging on always predicting one of the labels with absolute certainty. Usually the most common label in the dataset.
I then tried only taking the output for the last sentence of the TransformerEncoder output. i.e. TransformerEncoderOutput[:, -1, :].
This resulted in something similar.
I then tried running my Linear layer on each of the outputs of TransformerEncoder output to produce a tensor of size (batch_size, num_sentences, 7). I then sum over dim 1 (makes a tensor of size (batch_size, 7) and softmax as usual. The idea here is that every sentence gets to vote for the label after being informed about its place in the sequence.
This converged even more quickly to just predicting 1 for one of the labels and vanishingly small values for the others.
I feel like I am misunderstanding out to use the output of a pytorch Transformer somehow.
My learning rate is very low, 0.00001, and that helped delay the convergence but it converged eventually anyway.
What this suggests to me is that my network is incapable of figuring anything out about the text and is just learning to find the most common labels. I would guess that this is either a problem with my loss function or a problem with how I am using the Transformer.
Is there a glaring flaw in the architecture that I have laid out?


title: Classes in Coco dataset

body:
I have been checking out this detr repository and the total number of classes are 100, but 10 of these are empty string as shown here.
Is there any particular reason behind this?


title: How to use pretrained BERT word embedding vector to finetune (initialize) other networks?

body:
When I used to do classification work with textcnn, I had experience finetuning textcnn using pretrained word embedding with like Word2Vec and fasttext. And I use this process:

Create an embedding layer in textcnn
Load the embedding matrix of the words used this time by Word2Vec or
fasttext
Since the vector value of the embedding layer will change during training, the network is
being finetuning.

Recently I also want to try BERT to do this. I thought, 'As there should be few differences to use BERT pretrained embedding to initial other networks' embedding layer and finetuning, it should be easy!' But in fact yesterday I tried all day and still cannot do it.
The fact I found is that, as BERT's embedding is a contextual embedding, especially when extracting the word embeddings, the vector of each word from each sentence will vary, so it seems that there is no way to use that embedding to initialize the embedding layer of another network as usual...
Finally, I thought up one method to 'finetuning', as the following steps:

First, do not define an embedding layer in textcnn.
Instead of using embedding layer, in the network training part, I
firstly pass sequence tokens to the pretrained BERT model and get
the word embeddings for each sentence.
Put the BERT word embedding from 2. into textcnn and train the
textcnn network.

By using this method I was finally able to train, but thinking seriously, I don't think I'm doing a finetuning at all...
Because as you can see, every time when I start a new training loop, the word embedding generated from BERT is always the same vector, so just input these unchanged vectors to the textcnn wouldn't let the textcnn be finetuned at all, right?

UPDATE:
I thought up a new method to use the BERT embeddings and 'train' BERT and textcnn together.
Some part of my code is:
    BERTmodel = AutoModel.from_pretrained('bert-
                base-uncased',output_hidden_states=True).to(device)
    TextCNNmodel = TextCNN(EMBD_DIM, CLASS_NUM, KERNEL_NUM,
                   KERNEL_SIZES).to(device)
    optimizer = torch.optim.Adam(TextCNNmodel.parameters(), lr=LR)
    loss_func = nn.CrossEntropyLoss()

  for epoch in range(EPOCH):
    TextCNNmodel.train()
    BERTmodel.train()
    for step, (token_batch, seg_batch, y_batch) in enumerate(train_loader):
        token_batch = token_batch.to(device)
        y_batch = y_batch.to(device)

        BERToutputs = BERTmodel(token_batch)
        # I want to use the second-to-last hidden layer as the embedding, so
        x_batch = BERToutputs[2][-2]

        output = TextCNNmodel(x_batch)
        output = output.squeeze()
        loss = loss_func(output, y_batch)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

I think by enable BERTmodel.train() and delete torch.no_grad() when get the embedding, the loss gradient could be backward to BERTmodel, too. The training process of TextCNNmodel also went smoothly.
To use this model later, I saved the parameters of both TextCNNmodel and BERTmodel.
Then to experiment whether the BERTmodel was really being trained and changed, in another program I load the BERTModel, and input a sentence to test that whether the BERTModel was really being trained.
However, I found that the output (the embedding) of original 'bert-base-uncased' model and my 'BERTmodel' are the same, which is disappointing...
I really have no idea why the BERTmodel part did not change...


title: How to make R2 score in nn.LSTM pytorch

body:
I tried to make loss function with R2in nn.LSTM but i couldnt find any documentation about it . I already use RMSE and MAE loss from pytorch.
My data is a time series and im doing time series forecasting
This is the code where i use the loss function of RMSE in data training
model = LSTM_model(input_size=1, output_size=1, hidden_size=512, num_layers=2, dropout=0).to(device)
criterion = nn.MSELoss(reduction="sum")
optimizer = optim.Adam(model.parameters(), lr=0.001)
callback = Callback(model, early_stop_patience=10 ,outdir="model/lstm", plot_every=20,)

from tqdm.auto import tqdm

def loop_fn(mode, dataset, dataloader, model, criterion, optimizer,device):
    if mode =="train":
        model.train()
    elif mode =="test":
        model.eval()
    cost = 0
    for feature, target in tqdm(dataloader, desc=mode.title()):
        feature, target = feature.to(device), target.to(device)
        output , hidden = model(feature,None)
        loss = torch.sqrt(criterion(output,target))

        if mode =="train":
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

        cost += loss.item() * feature.shape[0]
    cost = cost / len(dataset)
    return cost

And this is the code to start data training
while True :
    train_cost = loop_fn("train", train_set, trainloader, model, criterion, optimizer,device)
    with torch.no_grad():
        test_cost  = loop_fn("test", test_set, testloader, model, criterion, optimizer,device)

    callback.log(train_cost, test_cost)

    callback.save_checkpoint()

    callback.cost_runtime_plotting()

    if callback.early_stopping(model, monitor="test_cost"):
        callback.plot_cost()
        break

Can anyone help me with the R2 loss function ? Thank you in advance


title: Trying to Serve a ML (pytorch) model on a Heroku, slug size too large

body:
I have a trained pytorch ML model that I would like to publish as a Heroku / Flask app.  The model is about 200 MB large, which puts me over the slug size limit on heroku.  I've tried to save the model in an S3 bucket (using heroku's Bucketeer) so that heroku can just download the model with boto3 once upon initialization, but then I get timeout errors. Is there any good way to serve a relatively large model with heroku?  If not, where should I migrate to?

I should say that I've also tried migrating everything to the Google App Engine Standard, but that didn't work since several needed libraries are not supported currently.  I am also unable to migrate to AWS directly since my AWS account was hacked a while ago and they basically banned me from using their services again.  (Their customer support was too incompetent to deal with the issue, though they are aware of the hack.).   I'm really at my wits end at this point and I don't want to waste another few days trying to get a new environment to work without some reasonable expectation that it will work....

Any help is appreciated, thanks!


title: AttributeError: 'numpy.ndarray' object has no attribute 'dim' with pytorch and scipy.optimize minimize

body:
I have the following small code snippet:
import torch
from scipy.optimize import minimize

def f(x):
    return torch.norm(x)

x = torch.tensor([1.0, 1.0])
y = minimize(f, x)
print(y)

However, this results in this error message:
> AttributeError                            Traceback (most recent call
> last) <ipython-input-17-cb070be7a142> in <module>
>       6
>       7 x = [1.0, 1.0]
> ----> 8 y = minimize(f, x)
>       9 print(y)
>
> ~\Anaconda3\lib\site-packages\scipy\optimize\_minimize.py in
> minimize(fun, x0, args, method, jac, hess, hessp, bounds, constraints,
> tol, callback, options)
>     610         return _minimize_cg(fun, x0, args, jac, callback, **options)
>     611     elif meth == 'bfgs':
> --> 612         return _minimize_bfgs(fun, x0, args, jac, callback, **options)
>     613     elif meth == 'newton-cg':
>     614         return _minimize_newtoncg(fun, x0, args, jac, hess, hessp, callback,
>
> ~\Anaconda3\lib\site-packages\scipy\optimize\optimize.py in
> _minimize_bfgs(fun, x0, args, jac, callback, gtol, norm, eps, maxiter, disp, return_all, finite_diff_rel_step, **unknown_options)    1100
> 1101     sf = _prepare_scalar_function(fun, x0, jac, args=args,
> epsilon=eps,
> -> 1102                                   finite_diff_rel_step=finite_diff_rel_step)    1103     1104     f =
> sf.fun
>
> ~\Anaconda3\lib\site-packages\scipy\optimize\optimize.py in
> _prepare_scalar_function(fun, x0, jac, args, bounds, epsilon, finite_diff_rel_step, hess)
>     260     # calculation reduces overall function evaluations.
>     261     sf = ScalarFunction(fun, x0, args, grad, hess,
> --> 262                         finite_diff_rel_step, bounds, epsilon=epsilon)
>     263
>     264     return sf
>
> ~\Anaconda3\lib\site-packages\scipy\optimize\_differentiable_functions.py
> in __init__(self, fun, x0, args, grad, hess, finite_diff_rel_step,
> finite_diff_bounds, epsilon)
>      74
>      75         self._update_fun_impl = update_fun
> ---> 76         self._update_fun()
>      77
>      78         # Gradient evaluation
>
> ~\Anaconda3\lib\site-packages\scipy\optimize\_differentiable_functions.py
> in _update_fun(self)
>     164     def _update_fun(self):
>     165         if not self.f_updated:
> --> 166             self._update_fun_impl()
>     167             self.f_updated = True
>     168
>
> ~\Anaconda3\lib\site-packages\scipy\optimize\_differentiable_functions.py
> in update_fun()
>      71
>      72         def update_fun():
> ---> 73             self.f = fun_wrapped(self.x)
>      74
>      75         self._update_fun_impl = update_fun
>
> ~\Anaconda3\lib\site-packages\scipy\optimize\_differentiable_functions.py
> in fun_wrapped(x)
>      68         def fun_wrapped(x):
>      69             self.nfev += 1
> ---> 70             return fun(x, *args)
>      71
>      72         def update_fun():
>
> <ipython-input-17-cb070be7a142> in f(x)
>       3
>       4 def f(x):
> ----> 5     return torch.norm(x)
>       6
>       7 x = [1.0, 1.0]
>
> ~\Anaconda3\lib\site-packages\torch\functional.py in norm(input, p,
> dim, keepdim, out, dtype)
>     738         (tensor(3.7417), tensor(11.2250))
>     739     """
> --> 740     ndim = input.dim()
>     741
>     742     # catch default case
>
> AttributeError: 'numpy.ndarray' object has no attribute 'dim'

I'm not sure why this is happening as I don't think I'm converting anything to a numpy.ndarray.
Any help would be appreciated.


title: How to replicate PyTorch's nn.functional.unfold function in Tensorflow?

body:
I want to use tensorflow to rewrite the pytorch's torch.nn.functional.unfold function:
#input x:[16, 1, 50, 36]
x = torch.nn.functional.unfold(x, kernel_size=(5, 36), stride=3)
#output x:[16, 180, 16]

I tried to use the function tf.extract_image_patches():
x = tf.extract_image_patches(x,ksizes=[1, 1,5, 98],strides=[1, 1, 3, 1], rates=[1, 1, 1, 1],padding='VALID')
The input x.shape:[16,1,64,98]
I get the output x.shape:[16,1,20,490]
Then I reshape the X to [16,490,20], that was I expect.
But I get the error when I feed the data:
UnimplementedError (see above for traceback): Only support ksizes across space.
[[Node:hcn/ExtractImagePatches = ExtractImagePatches[T=DT_FLOAT, ksizes=[1, 1, 5, 98], padding="VALID", rates=[1, 1, 1, 1], strides=[1, 1, 3, 1], _device="/job:localhost/replica:0/task:0/device:GPU:0"](hcn/Reshape)]]

How could I use tensorflow to rewrite pytorch torch.nn.functional.unfold function to change the X?


title: How to compute mean/max of HuggingFace Transformers BERT token embeddings with attention mask?

body:
I'm using the HuggingFace Transformers BERT model, and I want to compute a summary vector (a.k.a. embedding) over the tokens in a sentence, using either the mean or max function. The complication is that some tokens are [PAD], so I want to ignore the vectors for those tokens when computing the average or max.
Here's an example. I initially instantiate a BertTokenizer and a BertModel:
import torch
import transformers
from transformers import AutoTokenizer, AutoModel

transformer_name = 'bert-base-uncased'

tokenizer = AutoTokenizer.from_pretrained(transformer_name, use_fast=True)

model = AutoModel.from_pretrained(transformer_name)

I then input some sentences into the tokenizer and get out input_ids and attention_mask. Notably, an attention_mask value of 0 means that the token was a [PAD] that I can ignore.
sentences = ['Deep learning is difficult yet very rewarding.',
             'Deep learning is not easy.',
             'But is rewarding if done right.']
tokenizer_result = tokenizer(sentences, max_length=32, padding=True, return_attention_mask=True, return_tensors='pt')

input_ids = tokenizer_result.input_ids
attention_mask = tokenizer_result.attention_mask

print(input_ids.shape) # torch.Size([3, 11])

print(input_ids)
# tensor([[  101,  2784,  4083,  2003,  3697,  2664,  2200, 10377,  2075,  1012,  102],
#         [  101,  2784,  4083,  2003,  2025,  3733,  1012,   102,     0,     0,    0],
#         [  101,  2021,  2003, 10377,  2075,  2065,  2589,  2157,  1012,   102,   0]])

print(attention_mask.shape) # torch.Size([3, 11])

print(attention_mask)
# tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1],
#         [1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0],
#         [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0]])

Now, I call the BERT model to get the 768-D token embeddings (the top-layer hidden states).
model_result = model(input_ids, attention_mask=attention_mask, return_dict=True)

token_embeddings = model_result.last_hidden_state
print(token_embeddings.shape) # torch.Size([3, 11, 768])

So at this point, I have:

token embeddings in a [3, 11, 768] matrix: 3 sentences, 11 tokens, 768-D vector for each token.
attention mask in a [3, 11] matrix: 3 sentences, 11 tokens. A 1 value indicates non-[PAD].

How do I compute the mean / max over the vectors for the valid, non-[PAD] tokens?
I tried using the attention mask as a mask and then called torch.max(), but I don't get the right dimensions:
masked_token_embeddings = token_embeddings[attention_mask==1]
print(masked_token_embeddings.shape) # torch.Size([29, 768] <-- WRONG. SHOULD BE [3, 11, 768]

pooled = torch.max(masked_token_embeddings, 1)
print(pooled.values.shape) # torch.Size([29]) <-- WRONG. SHOULD BE [3, 768]

What I really want is a tensor of shape [3, 768]. That is, a 768-D vector for each of the 3 sentences.


title: Byte Embedding in mLSTM Conceptual Struggle

body:
I am trying to follow the OpenAI "Sentiment Neuron" experiment by reading through the PyTorch code posted on Github for training the model from scratch.
One thing I am not understanding is the byte-level embedding used in the code. I understood that the LSTM outputs a probability distribution for the value of the next byte and I assumed the "embedding" would just be a one-hot encoding of the byte value.
Looking at the code, I see that the model's input goes through a (trainable) dense embedding before going into the model. Confusingly, the output of the loss is computed between the model output and the upcoming byte value, which is not embedded.
My questions are:
1. How is the cross entropy loss computed? Does nn.CrossEntropyLoss take the softmax of its input and expand the target into a one-hot vector "under the hood"?
2. If we want to generate byte strings from this LSTM, how do we embed the output to feed back into the model for the next step? Do we embed the highest likelihood or take a softmax of the output and use some sort of weighted embedding?
I'm new to LSTM and I'm trying to learn but I just don't get it! I appreciate any help!


title: Pytorch - meaning of a command in a basic "forward" pass

body:
I am new with Pytorch, and will be glad if someone will be able to help me understand the following (and correct me if I am wrong), regarding the meaning of the command x.view in Pytorch first tutorial, and in general about the input of convolutional layers and the input of fully-connected layers:

def forward(self, x):
    x = self.pool(F.relu(self.conv1(x)))
    x = self.pool(F.relu(self.conv2(x)))
    x = x.view(-1, 16 * 5 * 5)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x

As far as I understand, an input 256X256 image to a convolutional layer is inserted in its 2D form (i.e. - a 256X256 matrix, or a 256X256X3 in the case of a color image). Nevertheless, when we insert an image to a fully-connected linear layer, we need to first reshape the 2D image into a 1D vector (am I right? Is this true also in general (or only in Pytorch)? ). Is this why we use the command “x = x.view(-1, 16 * 5 * 5)” before inserting x into the fully-connected layers?

If the input image x would be 3D (e.g. 256X256X256), would the syntax of the given above “forward” function remain the same?

Thanks a lot in advance


title: Pytorch incorrect value of member variable when using Multi-gpu

body:
Here is a simple class for running in multi-gpu environment. The member variable self.firstIter should be False after the first iteration.

Class TestNetwork(nn.Module):

    def __init__(self):
        super(TestNetwork, self).__init__()
        self.firstIter = True #indicates whether it's the first iteration

    def forward(self, input):
        print 'is firstIter: ', self.firstIter #always True!!
        if self.firstIter is True:
            self.firstIter = False
        # do otherthings

The code works as expected when using only one gpu.

However when using multi-gpu (i.e. nn.DataParallel), the value of self.firstIter is always printed as True.

Why does this happen? What is wrong with the code?

Using PyTorch version 0.3.1.


title: Maxpool2d error is showing while there is no Maxpool2d

body:
I run the following code to train a neural network that contains a CNN with max pooling and two fully-connected layers:

class Net(nn.Module):
    def __init__(self, vocab_size, embedding_size):
        torch.manual_seed(0)
        super(Net, self).__init__()
        self.word_embeddings = nn.Embedding(vocab_size, embedding_size)
        self.conv1 = nn.Conv1d(embedding_size, 64, 3)
        self.drop1 = nn.Dropout(0.5)
        self.max_pool1 = nn.MaxPool1d(2)
        self.flat1 = nn.Flatten()
        self.fc1 = nn.Linear(64*99, 100)
        self.fc2 = nn.Linear(100, 1)

    def forward(self, sentence):
        embedding = self.word_embeddings(sentence).permute(0, 2, 1)
        conv1 = F.relu(self.conv1(embedding))
        drop1 = self.drop1(conv1)
        max_pool1 = self.max_pool1(drop1)
        flat1 = self.flat1(max_pool1)
        fc1 = F.relu(self.fc1(flat1))
        fc2 = torch.sigmoid(self.fc2(fc1))
        return fc2

net = Net(vocab_size, EMBEDDING_SIZE)
EPOCHS = 10
net.cuda()
criterion = nn.BCELoss()
optimizer = optim.Adam(net.parameters(), lr=0.001)
loader = DataLoader(train, batch_size=32)

net.train()
for epoch in range(EPOCHS):
    progress = tqdm_notebook(loader, leave=False)
    for inputs, target in progress:
        net.zero_grad()
        output = net(inputs.to(device))
        loss = criterion(output, target.to(device))
        loss.backward()
        optimizer.step()
    print(loss)

and I get the following error (the error trace has been updated and it includes the complete trace):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py:498: UserWarning: Using a target size (torch.Size([32])) that is different to the input size (torch.Size([32, 1])) is deprecated. Please ensure they have the same size.
  return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-2-3c8a885417ba> in <module>()
     33     for inputs, target in progress:
     34         net.zero_grad()
---> 35         output = net(inputs.to(device))
     36         loss = criterion(output, target.to(device))
     37         loss.backward()

5 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in _max_pool1d(input, kernel_size, stride, padding, dilation, ceil_mode, return_indices)
    455         stride = torch.jit.annotate(List[int], [])
    456     return torch.max_pool1d(
--> 457         input, kernel_size, stride, padding, dilation, ceil_mode)
    458
    459 max_pool1d = boolean_dispatch(

RuntimeError: max_pool2d_with_indices_out_cuda_frame failed with error code 0

I do not have any Maxpool2ds in my code! Could anybody help me with this problem?


title: Deploying a hosted deep learning model on Heroku?

body:
I currently want to deploy a deep learning REST API using Flask on Heroku. The weights (Its a pre-trained BERT model) are stored here*as a .zip file. Is there a way I can directly deploy these?

From what I currently understand I have to have these uploaded on Github/S3. That's a bit of a hassle and seems pointless since they are already hosted. Do let me know!


title: Having problem with creating a dataloader class in PyTorch

body:
I am building a neural network for Bengali numerical digit classification using PyTorch. I am facing difficulties building the dataset class to load my dataset using a data loader. I have a folder with all the images (numerical digits from 0-9) and a CSV file with 2 columns, the first column contains the name of the image, and the second contains the label(0-9). This is my data loader class which probably isn't causing the error.

class BDRWDataset(Dataset):
"""BDRW dataset."""

def __init__(self, csv_file, imgs_dir, transform=None):
    """
    Args:
        csv_file (string): Path to the csv file with labels.
        imgs_dir (string): Directory with all the images.
        transform (callable, optional): Optional transform to be applied
            on a sample.
    """
    self.labels = pd.read_csv(csv_file).iloc[:, 1].to_numpy().reshape(-1,1)
    self.imgs_dir = imgs_dir
    self.transform = transform

def __len__(self):
    return len(self.labels)

def __getitem__(self, idx):
    if torch.is_tensor(idx):
        idx = idx.tolist()

    img_path = '/content/BDRW_train/digit_' + str(idx) + '.jpg'
    image = io.imread(img_name, plugin='matplotlib')
    image = Image.fromarray(np.uint8(image))
    label = self.labels[idx]
    label = float(label)

    if self.transform:
        image = self.transform(image)

    return (image, label)

I create an instance of this class.

transformed_dataset = BDRWDataset(csv_file='/content/labels.csv',imgs_dir='/content/BDRW_train',
                              transform=transforms.Compose([
                                           Rescale((28, 28)),
                                           transforms.Normalize((0.5,), (0.5,)),
                                           ToTensor()
                                           ]))

I have defined rescale and to tensor as follows

class Rescale(object):
"""Rescale the image in a sample to a given size.

Args:
    output_size (tuple or int): Desired output size. If tuple, output is
        matched to output_size. If int, smaller of image edges is matched
        to output_size keeping aspect ratio the same.
"""

def __init__(self, output_size):
    assert isinstance(output_size, (int, tuple))
    self.output_size = output_size

def __call__(self, sample):
    image, label = sample

    h, w = image.shape[:2]
    if isinstance(self.output_size, int):
        if h > w:
            new_h, new_w = self.output_size * h / w, self.output_size
        else:
            new_h, new_w = self.output_size, self.output_size * w / h
    else:
        new_h, new_w = self.output_size

    new_h, new_w = int(new_h), int(new_w)

    img = transform.resize(image, (new_h, new_w))

    return img, label

class ToTensor(object):
"""Convert ndarrays in sample to Tensors."""

def __call__(self, sample):
    image, label = sample['image'], sample['label']

    # swap color axis because
    # numpy image: H x W x C
    # torch image: C X H X W
    image = image.transpose((2, 0, 1))
    return (torch.from_numpy(image), torch.from_numpy(label))

Splitting the dataset into test and train and created train loader and validation loader using torch.utils.data.DataLoader

The neural network is

class CNN(nn.Module):
def __init__(self):
    super(CNN, self).__init__()
    self.layer1 = nn.Sequential(
        nn.Conv2d(1, 16, kernel_size=5, padding=2),
        nn.BatchNorm2d(16),
        nn.ReLU(),
        nn.MaxPool2d(2))
    self.layer2 = nn.Sequential(
        nn.Conv2d(16, 32, kernel_size=5, padding=2),
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.MaxPool2d(2))
    self.fc = nn.Linear(7*7*32, 10)

def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = out.view(out.size(0), -1)
    out = self.fc(out)
    return out

I create an instance of this class and start training

cnn = CNN()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(cnn.parameters(), lr=learning_rate)

losses = [];
for epoch in range(num_epochs):
    for i, (image, label) in enumerate(valloader):
        image = Variable(image.float())
        label = Variable(label)

        # Forward + Backward + Optimize
        optimizer.zero_grad()
        outputs = cnn(image)
        loss = criterion(output, label)
        loss.backward()
        optimizer.step()

        losses.append(loss.data[0]);

        if (i+1) % 100 == 0:
            print ('Epoch : %d/%d, Iter : %d/%d,  Loss: %.4f'
                   %(epoch+1, num_epochs, i+1, len(train_dataset)//batch_size, loss.data[0]))

This is where I get the error

AttributeError                            Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/PIL/Image.py in open(fp, mode)
   2812     try:
-> 2813         fp.seek(0)
   2814     except (AttributeError, io.UnsupportedOperation):

AttributeError: 'str' object has no attribute 'seek'

During handling of the above exception, another exception occurred:

AttributeError                            Traceback (most recent call last)
9 frames
/usr/local/lib/python3.6/dist-packages/PIL/Image.py in open(fp, mode)
   2813         fp.seek(0)
   2814     except (AttributeError, io.UnsupportedOperation):
-> 2815         fp = io.BytesIO(fp.read())
   2816         exclusive_fp = True
   2817

AttributeError: 'str' object has no attribute 'read'

It refers to PIL Image which I have used in the dataloader. So it seems to me that's where I am doing something wrong.

https://colab.research.google.com/drive/17XdP7gUoMNLxPCJ6PHEi3B09UQItzKyf?usp=sharing

This is the notebook I am working on. Please help me debug the errors in the code.

https://drive.google.com/open?id=1DznuHV9Fi5jVEbGdP-tg3ckmp5CNDOj1

This is the dataset I am working on.


I'm currently doing an DeepLearning course in Udemy.
I am currently designing an Restricted Boltzmann machine in which
my training runs perfectly, but I ended up with this error while testing

IndexError: The shape of the mask [1, 1682] at index 0 does not match the shape of the indexed tensor [100, 1682] at index 0

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.optim as optim
import torch.utils.data
from torch.autograd import Variable

# Importing dataset
movies = pd.read_csv('ml-1m/movies.dat',sep='::',header = None,
                     engine='python',encoding='latin-1')
users =  pd.read_csv('ml-1m/users.dat',sep='::',header = None,
                     engine='python',encoding='latin-1')
ratings = pd.read_csv('ml-1m/ratings.dat',sep='::',header = None,
                     engine='python',encoding='latin-1')

# preparing training and test set
training_set = pd.read_csv('ml-100k/u1.base',delimiter='\t')
training_set = np.array(training_set,dtype='int')
test_set = pd.read_csv('ml-100k/u1.test',delimiter='\t')
test_set = np.array(test_set,dtype='int')

#Getting the no of users and movies
nb_users = int(max(max(training_set[:,0]),max(test_set[:,0]))) #max out of both
nb_movies = int(max(max(training_set[:,1]),max(test_set[:,1])))

#Array with users in lines and movies in columns
def convert(data):
    new_data = []
    for id_users in range(1,nb_users+1):
        id_movies = data[:,1][data[:,0]==id_users]
        id_ratings = data[:,2][data[:,0]==id_users]
        ratings = np.zeros(nb_movies)
        ratings[id_movies-1]= id_ratings
        new_data.append(list(ratings))
    return new_data

training_set = convert(training_set)
test_set = convert(test_set)

# COnverting data to torcch tensors
training_set = torch.FloatTensor(training_set)
test_set = torch.FloatTensor(test_set)

# Converting the rating into binary ratings 1 (Liked) or 0 (Not liked)
training_set[training_set == 0] = -1 #taking all zero values in trainingset
training_set[training_set == 1] = 0
training_set[training_set == 2] = 0
training_set[training_set >= 3] = 1
test_set[test_set == 0] = -1 #taking all zero values in trainingset
test_set[test_set == 1] = 0
test_set[test_set == 2] = 0
test_set[test_set >= 3] = 1

# Creating the architecture of the Neural network
class RBM():
    def __init__(self, nv, nh):
        self.W = torch.randn(nh, nv)
        self.a = torch.randn(1, nh)
        self.b = torch.randn(1, nv)
    def sample_h(self, x):
        wx = torch.mm(x, self.W.t())
        activation = wx + self.a.expand_as(wx)
        p_h_given_v = torch.sigmoid(activation)
        return p_h_given_v, torch.bernoulli(p_h_given_v)
    def sample_v(self, y):
        wy = torch.mm(y, self.W)
        activation = wy + self.b.expand_as(wy)
        p_v_given_h = torch.sigmoid(activation)
        return p_v_given_h, torch.bernoulli(p_v_given_h)
    def train(self, v0, vk, ph0, phk):
        self.W += (torch.mm(v0.t(), ph0) - torch.mm(vk.t(), phk)).t()
        self.b += torch.sum((v0 - vk), 0)
        self.a += torch.sum((ph0 - phk), 0)
nv = len(training_set[0])
nh = 100
batch_size = 100
rbm = RBM(nv, nh)

#Training the RBM
nb_epoch = 10
for epoch in range(1, nb_epoch + 1):
    train_loss = 0
    s = 0.
    for id_user in range(0, nb_users - batch_size, batch_size):
        vk = training_set[id_user:id_user+batch_size]
        v0 = training_set[id_user:id_user+batch_size]
        ph0,_ = rbm.sample_h(v0)
        for k in range(10):
            _,hk = rbm.sample_h(vk)
            _,vk = rbm.sample_v(hk)
            vk[v0<0] = v0[v0<0]
        phk,_ = rbm.sample_h(vk)
        rbm.train(v0, vk, ph0, phk)
        train_loss += torch.mean(torch.abs(v0[v0>=0] - vk[v0>=0]))
        s += 1.
    print(f'epoch: {epoch} loss: {train_loss/s}')

# Testing the RBM
test_loss = 0
s = 0.
for id_user in range(nb_users):
    v = training_set[id_user:id_user+1]
    vt = training_set[id_user:id_user+1]
    if len(vt[vt>=0]) > 0:
        _,h = rbm.sample_h(v)
        _,v = rbm.sample_v(hk)
    test_loss += torch.mean(torch.abs(vt[vt>=0] - v[vt>=0]))
    s += 1.
print(f'test_loss: {test_loss/s}')


title: PyTorch: Calculating Top Eigenvector of Symmetric Matrix

body:
I am looking to find only the primary eigenvector of a real, positive-definite symmetric 20x2000x2000 PyTorch tensor:

K = torch.random.randn(20, 2000, 2000)
K_sym = K+K.transpose(1,2)/2
eig, ev = torch.symeig(K_sym, eigenvectors=True)
ev = ev[:,1999,:]

ev.shape

> torch.Size[20, 2000]

This implementation is very slow, even on the GPU,

Can anyone suggest a faster way of finding the first eigenvector?

For example, power iteration may be a solution. I have based this on the solution found in torch.nn.utils.spectral_norm :

def TopEigenvector(self, K, n_power_iterations=400, dim=1, eps=1e-12):
        v = torch.ones(K.shape[0],K.shape[1],1).to(self.device)
        with torch.no_grad():
            for _ in range(n_power_iterations):
                v = normalize(torch.bmm(K, v),dim=dim, eps=eps, out=v)
                if(n_power_iterations > 0):
                    v = v.clone(memory_format=torch.contiguous_format)
        return v

I have changed this to suit the problem as follows:

    def TopEigenvector(self, K, n_power_iterations=400, dim=1):
        v = torch.ones(K.shape[0],K.shape[1],1).to(self.device)
        for _ in range(n_power_iterations):
            m = torch.bmm(K, v)
            n = torch.norm(m, dim=1).unsqueeze(1)
            v = m/n
        return v

However, this relies on the dreaded for-loop, and is still slow, but faster than the original method.

Is there a better, faster method I could use?


title: How to calculate the median of a masked tensor along an axis?

body:
I have tensor X of floats of dimensions n x m and a tensor Y of booleans of dimensions n x m. I want to calculate values such as the mean, median and max of X, along one of the axes, but only considering the values in X which are true in Y. Something like X[Y].mean(dim=1). This is not possible because X[Y] is always a 1D tensor.
Edit:
For the mean, I was able to do it with:
masked_X = X * Y
masked_X_mean = masked_X.sum(dim=1) / Y.sum(dim=1)

For the max:
masked_X = X
masked_X[Y] = float('-inf')
masked_X_max = masked_X.max(dim=1)

But for the median, I was not able to be as creative. Any suggestions??
e.g.
X = torch.tensor([[1, 1, 1],
                  [2, 2, 4]]).type(torch.float32)
Y = torch.tensor([[0, 1, 0],
                  [1, 0, 1]]).type(torch.bool)

Expected Output
mean = [1., 3.]
median = [1., 2.]
var = [0., 1.]


title: comparing torch.nn.CrossEntropyLoss with label as int and prob tensor produces an error

body:
I have this criterion:
criterion = nn.CrossEntropyLoss()

And during training phase i have:
label = tensor([0.])
outputs = tensor([[0.0035, 0.0468]], grad_fn=<AddmmBackward>)

When i try to compare them:
 criterion(outputs, label)

I get this error:
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward


title: Why is this function parameter identical in every invocation, despite passing different value? (Creating closures in a loop)

body:
I'm using PyTorch and trying to register hooks on model parameters. The following code creates lambda functions to add to each model parameter, so I can see in the hook which tensor the gradient belongs to

import torch
import torchvision

# define model and random train batch
model = torchvision.models.alexnet()
input = torch.rand(10, 3, 224, 224)   # batch of 10 images
targets = torch.zeros(10).long()

def grad_hook_template(param, name, grad):
    print(f'Receive grad for {name} w whape {grad.shape}')

# add one lambda hook to each parameter
for name, param in model.named_parameters():
    print(f'Register hook for {name}')

    # use a lambda so we can pass additional information to the hook, which should only take one parameter
    param.register_hook(lambda grad: grad_hook_template(param, name, grad))

loss_fn = torch.nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=0.1)
optimizer.zero_grad()

prediction = model(input)
loss = loss_fn(prediction, targets)
loss.backward()
optimizer.step()

The result is that the name and param arguments to grad_hook_template always as the same value (and id), but the grad argument is always different (as expected). Why is it that when I register the hook, the lambdas seem to refer to the same local variables each time?

I read e.g. here that loops do not create new scopes and closures are lexical in Python, i.e. the name and param which I'm passing to the lambda are just pointers and whatever value they have at the end of the loop is seen by everyone with this pointer. But what can I do about it? copy.copy() the variables?


title: Pytorch - basic structure of cnn

body:
I just started pytorch today, and I am reading this at the moment. https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html.
The documentation has this code and also this illustration given. Are these related somehow? like, is this the structure or architecture of the network of the given code?

import torch
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

Link of the CNN Architecture: https://imgur.com/83UmBGc

Link of the image here


title: How to add pooling layer to BERT QA for large text

body:
I'm trying to implement a Question answering system that deal with large input text: so the idea is to split the large input text into subsequences of 510 tokens, after I will generate the representation of each sequence independently and using a pooling layer to generate the final representation of the input sequence.

I using the CamemBERT model for French language.

I have tried the following code:

class CamemBERTQA(nn.Module):

# the initialization of the model
   def __init__(self, do_lower_case: bool = True):
       super(CamemBERTQA, self).__init__()
       self.config_keys = ['do_lower_case']
       self.do_lower_case = do_lower_case
       self.camembert = CamembertForQuestionAnswering.from_pretrained('fmikaelian/camembert-base-fquad')
       self.tokenizer = CamembertTokenizer.from_pretrained('fmikaelian/camembert-base-fquad', do_lower_case=do_lower_case)
       self.cls_token_id = self.tokenizer.convert_tokens_to_ids([self.tokenizer.cls_token])[0]
       self.sep_token_id = self.tokenizer.convert_tokens_to_ids([self.tokenizer.sep_token])[0]
       self.pool = nn.MaxPool2d(2, 2)

# Split long input text into subsequences with overlapping
   def split_text(self, text, max_length, overlapp): #511 max
       f = []
       text = text.split()
       for i in range(0, int(len(text)-overlapp),(max_length-overlapp)):
           f.append(" ".join(text[i:i+max_length]))
#             print (f)
       return f

# Generate representation of a text,
   def text_representation(self, l): #  l here is a list
       result = []
       for i in l:
           input_ids = torch.tensor([self.tokenizer.encode(i, add_special_tokens=True)])
           with torch.no_grad():
               last_hidden_states = model(input_ids)[0]  # Models outputs are now tuples
               result.append(last_hidden_states)
#                     print(last_hidden_states[0])
       return result

   def forward(self, text, input_ids):
       # Split input text to subsequences of 511 with overlapping
       subsequences = self.split_text(text, 511, 10)

       # Generate IDs of each subsequence (Sequence representation)
       input_ids_list = self.text_representation(subsequences)
       print("input_ids_list")

       # Pooling layer
#         pool = self.pool(...)

###########      The problem is here: how can I add a pooling layer                  #################

#         input_ids = # the final output of the pooling layer, the result should contain 510 elements/tokens

       # generate the start and end logits of the answer
       start_scores, end_scores = self.camembert(torch.tensor([input_ids]))
       start_logits = torch.argmax(start_scores)
       end_logits = torch.argmax(end_scores)+1
       outputs = (start_logits, end_logits,)
#         print(outputs)

       return outputs

Since I'm a beginner with pyTorch, I'm not sure about if the code should be like that.

Please if you have any advice or if you need any further information contact me.


title: Can I create an Upper triangular tensor in pytorch?

body:
I want to create an upper triangular tensor in pytorch which I want that the lower half of the upper triangular tensor constant zeros. And the lower half of the upper triangular tensor have no grad.
When I use torch.triu() to get upper triangular tensor, the lower half of the upper triangular tensor have grad which means that such "zeros" are not constant.
So how to get an upper triangular tensor and let the lower half of the upper triangular tensor constant zeros?
import torch
a=torch.randn(5,5)
c=torch.randn(1,5)
b=torch.triu(a).requires_grad_()
loss=torch.matmul(c,b)
loss=loss.sum()
loss.backward()
print(b.grad)


title: Google Cloud - Pytorch Deep Learning VM - Error: 'module' object has no attribute '_cuda_setDevice'

body:
I have tried GCP following these tutorial https://cloud.google.com/deep-learning-vm/docs/pytorch_start_instance and I've tried to run my code using a GPU TeslaK80. It gives me the following AttributeError: 'module' object has no attribute '_cuda_setDevice'. Days before I had another VM and it worked fine. I had create another Deep Learning VM and now it gives me that error. I don't understand why it's failing now.

These is my stacktrace:

AttributeError                            Traceback (most recent call last)
<ipython-input-3-cc76841a4302> in <module>()
     45 # from config_training import config as config_training
     46 torch.manual_seed(0)
---> 47 torch.cuda.set_device(0)
     48
     49 model = import_module(args.model)

/usr/local/lib/python2.7/dist-packages/torch/cuda/__init__.pyc in set_device(device)
    262     device = _get_device_index(device)
    263     if device >= 0:
--> 264         torch._C._cuda_setDevice(device)
    265
    266

AttributeError: 'module' object has no attribute '_cuda_setDevice'


title: 1D correlation between 2 matrices

body:
I want to find 1D correlation between two matrices. These two matrices are the output of a convolution operation on two different images. Let's call the first matrix as matrix A and the other one as matrix B. Both these matrices have the shape 100 x 100 x 64 (say).

I've been following a research paper which basically computes 1D correlation between these two matrices (matrix A and matrix B) in one of the steps and the output of the correlation operation is also a matrix with the shape 100 x 100 x 64. The link to the paper can be found here. The network can be found on Page 4. The correlation part is in the bottom part of the network. A couple of lines have been mentioned about it in the 2nd paragraph of section 3.3 (on the same page, below the network).

I am not really sure what they mean by 1D correlation and more so how to implement it in Python. I am also confused as to how the shape of the output remains the same as the input after applying correlation. I am using the PyTorch library for implementing this network.

Any help will be appreciated. Thanks.


title: Apply preprocessing to the dataset

body:
I am implementing a paper on image segmentation in pytorch. I am required to do some preprocessing steps but as I am trying it the first time so I am unable to incorporate them in the traditional pipeline.
Following are the preprocessing steps-

1)   N(w, h) = I(w, h) − G(w, h), (1)
where N is the normalized image, I is the original image, and G is the Gaussian blurred image with kernel size 65*65 and 0 mean and standard deviation 10.

2)Normalizing the mean image and dividing each pixel by average standard deviation.

Following is my code snippet for the above steps-

def gaussian_blur(img):
    image = cv2.GaussianBlur(image,(65,65),10)
    new_image = img - image
return image

def normalise(img):
    img_normalised = np.empty(img.shape)
    img_std = np.std(img)
    img_mean = np.mean(img)
    img_normalized = (img-img_mean)/imgs_std

    for i in range(img.shape[1]):
        img_normalized[i] = (img_normalized -
         np.mean(img_normalized))/np.std(img_normalized)
return img_normalized

I am really not sure how to add above functions in the traditional pytorch data-loaders pipeline like first I should load the dataset using ImageFolder and then apply or first apply and then use ImageFolder method.


title: Pytorch Parameter-based Cropping custom sub-images from DataLoader image

body:
There are rectangular images in train & validation folders, and the images are accessed via Pytorch through DataLoader module. Without performing,

transforms.CenterCrop(size) or transforms.RandomCrop(size)

I need to preserve their original shape and size, so that while training/testing I can input square-shaped images, via some custom cropping technique. This CustomCrop function may take as inputs certain parameters, different for different images.

When I tried this custom-cropping approach only for testing, I had to exclude the cropping line from data_transforms for TEST.

data_transforms = {
    TRAIN: transforms.Compose([
        transforms.RandomCrop(size),
        transforms.ToTensor(),
    ]),
    TEST: transforms.Compose([
        #CROPPING LINE EXCLUDED FROM HERE
        transforms.ToTensor(),
    ])
}
image_datasets = {
    x: ImageFolder(
        os.path.join(path, mapping[x]),
        transform=data_transforms[x]
    )
    for x in [TRAIN, TEST]
}
dataloaders = {
       x: DataLoader(
       image_datasets[x], batch_size=batch_size
       )
    for x in [TRAIN, TEST]
}

And my intention is to do something like...

for imgs, lbls, paths in iter(dataloaders[TEST]):
    data = (imgs, lbls)
    inputs, labels = data
    cropped_input = myCustomCrop(inputs, param1, param2) #this crops it to square size, compatible for my model
    [...]  #some code
    optimizer.zero_grad()
    outputs = model(cropped_input)

...so that I can crop it during run-time as per my need.

But this gives me an error message (For image size (100,75) in folder):

--> 218     for imgs, lbls, paths in iter(dataloaders[TEST]):
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 100 and 75 in dimension 2 at ...

I am understanding that, DataLoader stacks the images and they need to be equal-sized at runtime. But, how can I then implement my custom-crop function, which may be different for different images (based on input parameter to CustomCrop function). Any suggestions would be highly appreciated. Thanks.


title: Pytorch Deep Learning - Class Model() and training function

body:
I am new to Pytorch and I am going through this tutorial to figure out how to do deep learning with this library. I have problem figuring out part of the code.

There is a class called Net and an object called model instantiated from it. Then there is the training function called  train(epoch). In the next line in the train function body I see this: model.train() which I can't make any sense of. Would you please help me in understanding this part of the code? how we are calling a method of a class when this method has not been defined inside the class? and how come the method has the exact same name as function it has been called inside?    This is the class definition:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 32, kernel_size=3)
        self.conv2 = nn.Conv2d(32, 64, kernel_size=3)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(2304, 256)
        self.fc2 = nn.Linear(256, 17)

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(x.size(0), -1) # Flatten layer
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.sigmoid(x)

model = Net() # On CPU
# model = Net().cuda() # On GPU

And this is the function defined after this class:

def train(epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        # data, target = data.cuda(async=True), target.cuda(async=True) # On GPU
        data, target = Variable(data), Variable(target)
        optimizer.zero_grad()
        output = model(data)
        loss = F.binary_cross_entropy(output, target)
        loss.backward()
        optimizer.step()
        if batch_idx % 10 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.data[0]))


title: How to deploy an existing pytorch model previously trained with Amazon Sagemaker and stored in S3 bucket

body:
I have trained a Pytorch model using SageMaker and the model is now stored in an S3 bucket.
I am trying to retrieve that model and deploying it.

This is the code I am using:

estimator = sagemaker.model.FrameworkModel(
    model_data= #link to  model location in s3
    image=  # image
    role=role,
    entry_point='train.py',
    source_dir='pytorch_source',
    sagemaker_session = sagemaker_session
)

predictor = estimator.deploy(initial_instance_count=1, instance_type="ml.p2.xlarge")

But after the deployment process (that seems to run smoothly), the predictor is just a NoneType.
I haven't found any weird message in the logs...

I have also made another attempt with the following code:

estimator = PyTorchModel(model_data= #link to model location in s3
                             role=role,
                             image= #image
                             entry_point='pytorch_source/train.py',
                            predictor_cls = 'pytorch_source/train.py',
                           framework_version = '1.1.0')

predictor = estimator.deploy(initial_instance_count=1, instance_type="ml.p2.xlarge")

But it doesn't even complete the deployment.

Can anyone help with this?


title: PyTorch Dataset Field for Sequence of Vectors (No Vocabulary)

body:
I have a "simple" machine translation task where I have a sequence of vectors to be mapped to a word or two. (the vector is 258 dimensions)
For example:

[[1, ..., 2], [3, ..., 4]] => "hello"
[[1, ..., 2], [3, ..., 4], [5, ..., 6]] => "hello world"

For the target field, I am using Field(eos_token="<eos>", is_target=True), which when batched does correctly give me a tensor with padding, in this case:
tensor([
  [1, 1], # 1 is "hello"
  [2, 3], # 2 is "world", 3 is <eos>
  [3, 0], # 0 is <pad>
])

However, the src field is not being padded the same way, as it is sequential, but does not have a vocabulary (Field(use_vocab=False)).
When I read the src from the BucketIterator, in a batch of size > 1, I get:

Traceback (most recent call last):
File "train.py", line 50, in train
for b, batch in enumerate(train_iter):

File "/torchtext/data/iterator.py", line 156, in iter
yield Batch(minibatch, self.dataset, self.device)

File "/torchtext/data/batch.py", line 34, in init
setattr(self, name, field.process(batch, device=device))

File "/torchtext/data/field.py", line 237, in process
tensor = self.numericalize(padded, device=device)

File "/torchtext/data/field.py", line 359, in numericalize
var = torch.tensor(arr, dtype=self.dtype, device=device)

ValueError: expected sequence of length 258 at dim 2 (got 5)

What I want to get is a tensor:
tensor([
  [[1, ..., 2], [1, ..., 2]],
  [[3, ..., 4], [3, ..., 4]],
  [[5, ..., 6], [0, ..., 0]],
  [[0, ..., 0], [0, ..., 0]],
])

What I think I might have but don't know how to confirm is:
tensor([
  [[1, ..., 2], [1, ..., 2]],
  [[3, ..., 4], [3, ..., 4]],
  [[5, ..., 6], 0],
  [0, 0],
])


title: Replacing nn.Upsample with alternative upsample operation

body:
I have a UNet++(view in private, code for model at the bottom of the article) which I'm trying to reconfigure. I'm getting some artifacts in some images so I'm following this article which suggests doing upsampling then a convolution operation.

I'm replacing the up-sample layers with sequential operation shown below but my model isn't learning. I suspect its to do with how I've configured the channels so I'd like another opinion.

Old up-sample operation:

self.up = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)

New operations:

class upConv(nn.Module):
"""
Up sampling/ deconv block by factor of 2
"""
def __init__(self, in_ch, out_ch):
    super().__init__()
    self.upc = nn.Sequential(
                             nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
                             nn.Conv2d(in_ch, out_ch*2, 3, stride=1, padding=1),
                             nn.BatchNorm2d(out_ch*2),
                             nn.ReLU(inplace=True)
                             )
def forward(self, x):
    out = self.upc(x)
    return out

My question is do these two operations have the same output/function within my model?


title: how to add layernorm in the model (pytorch)

body:
I tried to attach one layer norm to the conv3x3 to see the effect.
Following is my reproduce code
class conv3x3_layernorm(nn.Module):
    def __init__(self, in_planes=3, out_planes=3, stride=1, groups=1, dilation=1):
        super().__init__()
        self.conv = nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
                     padding=dilation, groups=groups, bias=False, dilation=dilation)
        self.layernorm = nn.LayerNorm([4, 3, 800, 800], elementwise_affine = False)

    def forward(self, x):
        x = self.conv(x)
        x = self.layernorm(x)
        return x

def show_conv3x3_layernorm(stride):
    model = conv3x3_layernorm(stride=stride)
    imgs1 = model(imgs)
    # show shape
    print(imgs1.shape)

    # show the effect
    fig, ax = plt.subplots(ROW, COL, figsize=(20, 20))
    for i in range(ROW):
        for j in range(COL):
            x = imgs1[i*ROW+j]
            ax[i, j].imshow(x.detach().permute(1, 2, 0)*0.5+0.5)

    m = imgs1.mean()
    s = imgs1.std()
    print('mean: ', m, 'std: ', s)

interact(show_conv3x3_layernorm, stride=widgets.IntSlider(min=1, max=10, step=1, value=1));

For LayerNorm, it requires the input size, but how to access that in the model?
In above case, I just hard code it, but that would have an error if I use the slider, namely changed the stride parameter, so how to solve this problem?
Methods I tried
I found this post https://discuss.pytorch.org/t/converting-from-nn-batchnorm2d-to-nn-layernorm-in-cnn/30638/2
is asking the same question, but the answer is just hard coding, so my question is not solved.
I also found this code to convert all BN layers to LayerNorm/InstanceNorm/GroupNorm, https://www.kaggle.com/c/aptos2019-blindness-detection/discussion/104686, which is useful. But how to change on this API?nn.LayerNorm
Thank you!!


title: Meta Learning with Transformers

body:
I want to train my transformer model with MAML gradient updates. I’ve found that higher is an efficient way to go about it. I’m open to other methods also.
So my MetaDataset wraps any GlueDataset to give a list containing all classes when meta_dataset[0] is called. So this will become, num_of_classes (N) way K shot example.
I’ve written this, which extends HF Trainer for MAML.
def train(self):

        self.create_optimizer_and_scheduler(
            int(
                len(self.train_dataloader)
                // self.args.gradient_accumulation_steps
                * self.args.num_train_epochs
            )
        )

        logger.info("***** Running training *****")

        self.global_step = 0
        self.epoch = 0

        eval_step = [2 ** i for i in range(1, 20)]
        inner_optimizer = torch.optim.SGD(
            self.model.parameters(), lr=self.args.step_size
        )
        self.model.train()

        tqdm_iterator = tqdm(self.train_dataloader, desc="Batch Index")

        #  n_inner_iter = 5
        self.optimizer.zero_grad()
        query_dataloader = iter(self.train_dataloader)

        for batch_idx, meta_batch in enumerate(tqdm_iterator):
            target_batch = next(query_dataloader)
            outer_loss = 0.0
            # Loop through all classes
            for inputs, target_inputs in zip(meta_batch, target_batch):

                for k, v in inputs.items():
                    inputs[k] = v.to(self.args.device)
                    target_inputs[k] = v.to(self.args.device)

                with higher.innerloop_ctx(
                    self.model, inner_optimizer, copy_initial_weights=False
                ) as (fmodel, diffopt):

                    inner_loss = fmodel(**inputs)[0]
                    diffopt.step(inner_loss)
                    outer_loss += fmodel(**target_inputs)[0]

            self.global_step += 1
            self.optimizer.step()

            outer_loss.backward()

            if (batch_idx + 1) % self.args.gradient_accumulation_steps == 0:
                torch.nn.utils.clip_grad_norm_(
                    self.model.parameters(), self.args.max_grad_norm
                )

            # Run evaluation on task list
            if self.global_step in eval_step:
                output = self.prediction_loop(self.eval_dataloader, description = "Evaluation")
                self.log(output.metrics)

                output_dir = os.path.join(
                    self.args.output_dir, f"{PREFIX_CHECKPOINT_DIR}-{self.global_step}",
                )
                self.save_model(output_dir)

The above code doesn’t seem to work properly as in the accuracy doesn’t improve. Any directions/tips would be appreciated.


title: BERT Embeddings in Pytorch Embedding Layer

body:
I'm working with word embeddings. I obtained word embeddings using 'BERT'.
I have a data like this
1992 regular unleaded 172 6 MANUAL all wheel drive 4 Luxury Midsize Sedan 21 16 3105 200

and as a label:
df['Make'] = df['Make'].replace(['Chrysler'],1)

I try to give embeddings as a LSTM inputs.
Using below code for BERT:
 tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
 bert_model = BertModel.from_pretrained('bert-base-uncased')

For tokinizing this code:
def tokenize_text(df, max_seq):
    return [
        tokenizer.encode(text, add_special_tokens=True)[:max_seq] for text in df
    ]

def pad_text(tokenized_text, max_seq):
    return np.array([el + [0] * (max_seq - len(el)) for el in tokenized_text])

def tokenize_and_pad_text(df, max_seq):
    tokenized_text = tokenize_text(df, max_seq)
    padded_text = pad_text(tokenized_text, max_seq)
    return padded_text

and :
train_indices = tokenize_and_pad_text(df_train, max_seq)

for BERT embedding matrix:
def get_bert_embed_matrix():
    bert = transformers.BertModel.from_pretrained('bert-base-uncased')
    bert_embeddings = list(bert.children())[0]
    bert_word_embeddings = list(bert_embeddings.children())[0]
    mat = bert_word_embeddings.weight.data.numpy()
    return mat

embedding_matrix = get_bert_embed_matrix()

and LSTM Model:
embedding_layer =Embedding(embedding_matrix.shape[0], embedding_matrix.shape[1], weights=[embedding_matrix], input_length=max_seq_len, trainable=True)
model = Sequential()
model.add(embedding_layer)
model.add(LSTM(128, dropout=0.3, recurrent_dropout=0.3))
model.add(Dense(1, activation='softmax'))

model.compile(
    optimizer=tf.keras.optimizers.Adam(1e-5),
    loss="categorical_crossentropy",
    metrics=["accuracy"],
)

model.summary()

and for model fit this code:
model.fit(train_indices, y_train, epochs=20, verbose=1)

I give a output like this:
Epoch 1/20
1/1 [==============================] - 3s 3s/step - loss: 0.0000e+00 - accuracy: 0.3704
Epoch 20/20
1/1 [==============================] - 0s 484ms/step - loss: 0.0000e+00 - accuracy: 0.3704

I don't know what I'm missing.
Firstly, what can we do about it?
Secondly, how can we implement Pytorch Model?
Thanks a lot.


title: Too many files open

body:
import PIL as Image
Image.fromarray(cv2.imread(link, cv2.IMREAD_GRAYSCALE))

I'm currently trying to complete a project but I'm constantly getting too many files open error on my linux GPU server which crashes the server.

I'm loading 3 images for CNN classification using the code as shown above. Anyone facing the same problem have a solution to this?

Thank you.


title: DQN stuck at suboptimal policy in Atari Pong task

body:
I am in the process of implementing the DQN model from scratch in PyTorch with the target environment of Atari Pong. After a while of tweaking hyper-parameters, I cannot seem to get the model to achieve the performance that is reported in most publications (~ +21 reward; meaning that the agent wins almost every volley).

My most recent results are shown in the following figure. Note that the x axis is episodes (full games to 21), but the total training iterations is ~6.7 million.

The specifics of my setup are as follows:

Model

class DQN(nn.Module):
    def __init__(self, in_channels, outputs):
        super(DQN, self).__init__()
        self.conv1 = nn.Conv2d(in_channels=in_channels, out_channels=32, kernel_size=8, stride=4)
        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=4, stride=2)
        self.conv3 = nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1)
        self.fc1 = nn.Linear(in_features=64*7*7 , out_features=512)
        self.fc2 = nn.Linear(in_features=512, out_features=outputs)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = x.view(-1, 64 * 7 * 7)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x    # return Q values of each action

Hyperparameters

batch size: 32
replay memory size: 100000
initial epsilon: 1.0
epsilon anneals linearly to 0.02 over 100000 steps
random warmstart episodes: ~50000
update target model every: 1000 steps
optimizer = optim.RMSprop(policy_net.parameters(),  lr=0.0025, alpha=0.9, eps=1e-02, momentum=0.0)

Additional info

OpenAI gym Pong-v0 environment
Feeding model stacks of 4 last observed frames, scaled and cropped to 84x84 such that only the "playing area" is visible.
Treat losing a volley (end-of-life) as a terminal state in the replay buffer.
Using smooth_l1_loss, which acts as Huber loss
Clipping gradients between -1 and 1 before optimizing
I offset the beginning of each episode with 4-30 no-op steps as the papers suggest

Has anyone had a similar experience of getting stuck around 6 - 9 average reward per episode like this?

Any suggestions for changes to hyperparameters or algorithmic nuances would be greatly appreciated!


title: TRN-Pytorch model - RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

body:
I'm using TRN-Pytorch model in colab and pytorch version is 0.4.1. While training the model I got a RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

This is code for training

!python3 main.py something RGB \
                     --arch BNInception --num_segments 3 \
                     --consensus_type TRN --batch-size 2

and I got this Error

storing name: TRN_something_RGB_BNInception_TRN_segment3

    Initializing TSN with base model: BNInception.
    TSN Configurations:
        input_modality:     RGB
        num_segments:       3
        new_length:         1
        consensus_module:   TRN
        dropout_ratio:      0.8
        img_feature_dim:    256

/content/drive/My Drive/TRN-pytorch/models.py:87: UserWarning: nn.init.normal is now deprecated in favor of nn.init.normal_.
  normal(self.new_fc.weight, 0, std)
/content/drive/My Drive/TRN-pytorch/models.py:88: UserWarning: nn.init.constant is now deprecated in favor of nn.init.constant_.
  constant(self.new_fc.bias, 0)
video number:4
/usr/local/lib/python3.6/dist-packages/torchvision/transforms/transforms.py:208: UserWarning: The use of the transforms.Scale transform is deprecated, please use transforms.Resize instead.
  "please use transforms.Resize instead.")
video number:1
group: first_conv_weight has 1 params, lr_mult: 1, decay_mult: 1
group: first_conv_bias has 1 params, lr_mult: 2, decay_mult: 0
group: normal_weight has 71 params, lr_mult: 1, decay_mult: 1
group: normal_bias has 71 params, lr_mult: 2, decay_mult: 0
group: BN scale/shift has 2 params, lr_mult: 1, decay_mult: 0
Freezing BatchNorm2D except the first one.
Traceback (most recent call last):
  File "main.py", line 324, in <module>
    main()
  File "main.py", line 128, in main
    train(train_loader, model, criterion, optimizer, epoch, log_training)
  File "main.py", line 175, in train
    prec1, prec5 = accuracy(output.data, target, topk=(1,5))
  File "main.py", line 301, in accuracy
    batch_size = target.size(1)
RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

main.py file is here

Help me to solve this issue


title: Convert image of dimension height,width,number of channels to n_masks, image_height, image_width

body:
I have an RGB image, it contains masks of different colors, each color represents a particular class.
I want to convert it into the format - n_masks, image_height, image_width where n_masks is number of masks present in the image. and each slice of the matrix along the 0th axis represents one binary mask.
So far, I have been able to convert it into the format of image_height, image_width where each array value represents which class it belongs to but I am kind of struck after it.
Below is my code to convert it into image_height,image_width format-

def mask_to_class(mask):
    target = torch.from_numpy(mask)
    h,w = target.shape[0],target.shape[1]
    masks = torch.empty(h, w, dtype=torch.long)
    colors = torch.unique(target.view(-1,target.size(2)),dim=0).numpy()
    target = target.permute(2, 0, 1).contiguous()
    mapping = {tuple(c): t for c, t in zip(colors.tolist(), range(len(colors)))}
    for k in mapping:
        idx = (target==torch.tensor(k, dtype=torch.uint8).unsqueeze(1).unsqueeze(2))
        validx = (idx.sum(0) == 3)
        masks[validx] = torch.tensor(mapping[k], dtype=torch.long)
    return masks

It converts an image of let's say format (512,512,3) to (512,512) where each pixel values represents the class it belongs to, but I don't have any idea how to proceed further.
P.S- I am coding it in pytorch but any approach involving numpy is also welcomed.


title: How to share the common parts of two models in pytorch？

body:
I had a problem implementing the model with pytorch.

I want to build two models, some of which are shared, and share the encoder part like this

Model1: input_1 -> encoder -> decoder_1 -> ouput_1
Model2: input_2 -> encoder -> decoder_2 -> ouput_2

What I want to do is make the two models use the encoder part together, but the decoder part is not the same. I looked up about parameter sharing, but it seems to be somewhat different from the requirements here.

My own idea is to build a model that includes encode, decoder_1, decoder_2 and then choose which decoder to use based on input.

I'm not sure about this method, if possible, can you give simple examples for using the common parts of two models?


title: RuntimeError: Unknown device when trying to run AlbertForMaskedLM on colab tpu

body:
I am running the following code on colab taken from the example here: https://huggingface.co/transformers/model_doc/albert.html#albertformaskedlm

import os
import torch
import torch_xla
import torch_xla.core.xla_model as xm

assert os.environ['COLAB_TPU_ADDR']

dev = xm.xla_device()

from transformers import AlbertTokenizer, AlbertForMaskedLM
import torch

tokenizer = AlbertTokenizer.from_pretrained('albert-base-v2')
model = AlbertForMaskedLM.from_pretrained('albert-base-v2').to(dev)
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1

data = input_ids.to(dev)

outputs = model(data, masked_lm_labels=data)
loss, prediction_scores = outputs[:2]

I haven't done anything to the example code except move input_ids and model onto the TPU device using .to(dev). It seems everything is moved to the TPU no problem as when I input data I get the following output: tensor([[    2, 10975,    15,    51,  1952,    25, 10901,     3]], device='xla:1')

However when I run this code I get the following error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-5-f756487db8f7> in <module>()
      1
----> 2 outputs = model(data, masked_lm_labels=data)
      3 loss, prediction_scores = outputs[:2]

9 frames
/usr/local/lib/python3.6/dist-packages/transformers/modeling_albert.py in forward(self, hidden_states, attention_mask, head_mask)
    277         attention_output = self.attention(hidden_states, attention_mask, head_mask)
    278         ffn_output = self.ffn(attention_output[0])
--> 279         ffn_output = self.activation(ffn_output)
    280         ffn_output = self.ffn_output(ffn_output)
    281         hidden_states = self.full_layer_layer_norm(ffn_output + attention_output[0])

RuntimeError: Unknown device

Anyone know what's going on?


title: An error while using tune for hyper-parameters a deep learning model

body:
I'm trying to use tune to tune my model in Pytorch (https://docs.ray.io/en/latest/tune.html), as a starting point; I just tried to run their small example:

import torch.optim as optim
from ray import tune
from ray.tune.examples.mnist_pytorch import get_data_loaders, ConvNet, train, test

def train_mnist(config):
    train_loader, test_loader = get_data_loaders()
    model = ConvNet()
    optimizer = optim.SGD(model.parameters(), lr=config["lr"])
    for i in range(10):
        train(model, optimizer, train_loader)
        acc = test(model, test_loader)
        tune.track.log(mean_accuracy=acc)

analysis = tune.run(
    train_mnist, config={"lr": tune.grid_search([0.001, 0.01, 0.1])})

print("Best config: ", analysis.get_best_config(metric="mean_accuracy"))

# Get a dataframe for analyzing trial results.
df = analysis.dataframe()

And I got this error:

I run it on a remote machine requesting 100 GB of memory. Any help with that please?