title: pytorch split gives wrong answer

body:
import torch
x = torch.rand(2, 3)
print(x)
splitted = x.split(split_size=2, dim=0)  # should get 2 tensors of 1 x 3  and 1 x 3 size I thought
print(splitted) #instead, get a tuple of len 1, with [0] =  tensor same as input
print(type(splitted), len(splitted))
print(splitted[0].shape)
print(torch.__version__)

gives the following output:

tensor([[0.0702, 0.1275, 0.3735],
        [0.0260, 0.9393, 0.9448]])
(tensor([[0.0702, 0.1275, 0.3735],
        [0.0260, 0.9393, 0.9448]]),)
<class 'tuple'> 1
torch.Size([2, 3])
1.3.1

Why do I not get two tensors in a tuple? I would have expected the input to be split in two. I am under Windows 10


title: simple CNN model training using CIFAR-10 dataset STUCK at a low accuracy

body:
Hi i have just learned to implement NN models in pytorch through the udacity course and thus created a simple model with a a few CNN and FC layers. after much struggle i got the model to work. but it seems that it is stuck at the same loss even after repeated executions. i dont know where i am going wrong. Must be some logical error which i cant see.
Here is the code.

model

class cifar_clasify(nn.Module):

    def __init__(self):

        super().__init__()

        self.conv1 = nn.Conv2d(3,16,3)
        self.BNorm1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(16,32,3)
        self.BNorm2 = nn.BatchNorm2d(32)
        self.fc1 = nn.Linear(32*6*6,256)
        self.fc2 = nn.Linear(256,512)
        self.fc3 = nn.Linear(512,10)

        self.drop = nn.Dropout(p =0.2)

    def forward(self,x):

        out = self.conv1(x)
        out = F.relu(out)
        #print(out.shape)
        out = F.max_pool2d(out,2)
        out = self.BNorm1(out)
        #print(out.shape)

        out = self.conv2(out)
        out = F.relu(out)
        #print(out.shape)
        out = F.max_pool2d(out,2)
        out = self.BNorm2(out)
        #print(out.shape)
        out = out.view(out.shape[0],-1)

        out = self.fc1(out)
        out = self.drop(F.relu(out))
        out = self.fc2(out)
        out = self.drop(F.relu(out))
        final = F.log_softmax(F.relu(self.fc3(out)) , dim = 1)

        return final

training code

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

model = cifar_clasify()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr =0.03)

epoch =2
step = 2
running_loss = 0
accuracy = 0
print_every = 5

model.to(device)

for e in range(epoch):

    for inputs,label_ in zip(train_X,train_labels):

        step +=1
        inputs = inputs.view((-1,3,32,32))
        inputs,label_ = inputs.to(device) , label_.to(device)

        #inputs.cuda()
        #label.cuda()

        optimizer.zero_grad()

        logps = model.forward(inputs)

        loss = criterion(logps , label_.reshape(1))

        loss.backward()

        optimizer.step()

        running_loss += loss.item()

        if step % print_every == 0:
            test_loss = 0
            accuracy = 0
            model.eval()
            with torch.no_grad():
                for testx, lab in zip(test_X , test_labels):

                    testx = testx.view((-1,3,32,32))

                    testx,lab = testx.to(device) , lab.to(device)
                    lab = lab.reshape(1)
                    logps = model.forward(testx)
                    batch_loss = criterion(logps , lab)
                    #print(batch_loss.item())
                    test_loss += batch_loss.item()

                    ps = torch.exp(logps)

                    top_p , topclass = ps.topk(1,dim = 1)

                    equals = topclass == lab.view(*topclass.shape)

                    accuracy += torch.mean(torch.mean(equals.type(torch.FloatTensor))).item()
            print(f"Epoch {e+1}/{epoch}.. "
                  f"Train loss: {running_loss/print_every:.3f}.. "
                  f"Test loss: {test_loss/len(test_X):.3f}.. "
                  f"Test accuracy: {accuracy/len(test_X):.3f}")
            running_loss = 0
            model.train()

Here is the result which i had to stop as it was not improving :

Epoch 1/2.. Train loss: 1.396.. Test loss: 5.288.. Test accuracy: 0.104
step =  5
Epoch 1/2.. Train loss: 3.038.. Test loss: 2.303.. Test accuracy: 0.104
step =  10
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  15
Epoch 1/2.. Train loss: 2.669.. Test loss: 2.318.. Test accuracy: 0.105
step =  20
Epoch 1/2.. Train loss: 3.652.. Test loss: 2.303.. Test accuracy: 0.104
step =  25
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  30
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  35
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  40
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  45
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  50
Epoch 1/2.. Train loss: 2.303.. Test loss: 2.303.. Test accuracy: 0.104
step =  55

Here is the code if you want any other information:

Simple CNN for CIFAR 10 classification in google colab


title: How can I matrix-multiply two PyTorch quantized Tensors?

body:
I am new to tensor quantization, and tried doing something as simple as
import torch
x = torch.rand(10, 3)
y = torch.rand(10, 3)

x@y.T

with PyTorch quantized tensors running on CPU. I thus tried
scale, zero_point = 1e-4, 2
dtype = torch.qint32
qx = torch.quantize_per_tensor(x, scale, zero_point, dtype)
qy = torch.quantize_per_tensor(y, scale, zero_point, dtype)

qx@qy.T # I tried...

..and got as error

RuntimeError: Could not run 'aten::mm' with arguments from the
'QuantizedCPUTensorId' backend. 'aten::mm' is only available for these
backends: [CUDATensorId, SparseCPUTensorId, VariableTensorId,
CPUTensorId, SparseCUDATensorId].

Is matrix multiplication just not supported, or am I doing something wrong?


title: CNN for image to feature vector

body:
I have a rather simple question, but I cannot wrap my head around this. You need to correct me in the case that I have understood something wrong!!! Typically, CNNs are used for image classification, where the FC layers are used to output the probability per given class.
I want actually something else:
Can I train a CNN to output a feature vector? One example would be: Input==Image; Output==Color histogram with 3 channels (RGB) and every channel value is the number of red, green, blue pixels.
(I actually want to learn other image feature descriptors for special images, but that should then work analogous).
Pointers, Links and Tutorials (esp. code that I can modify for my purposes) would be greatly appreciated.
Thanks so much in advance!!


title: "ImportError: No module named torch" in ROS package

body:
I've created a ROS package in which there are some python scripts. The python scripts are based on torch module (it is an inference code of pytorch models). when I try to run my scripts it gives me an error:
ImportError: No module named torch

To install ROS, I used  the instruction of ROS wiki. To validate my installation, I followed the sample code of ROS (a simple publisher and subscriber) and it works well. My sys-info is:

  python: 3.6.9
  torch: 1.1.0
  torchvision: 0.3.0
  OS kernel: Linux 4.15.0-74-generic
  OS distribution: Ubuntu 18.04.3

I want to import the below libs:

import torch
from cv_bridge import CvBridge
import cv2
import os
import numpy as np
from torch.autograd import Variable
from torchvision import transforms
import torch.nn.functional as F
import torch._utils
import time
from PIL import Image

My CMake file is as below:

cmake_minimum_required(VERSION 2.8.3)
project(inference_pytorch)

## Compile as C++11, supported in ROS Kinetic and newer
# add_compile_options(-std=c++11)

set(Torch_DIR ".local/lib/python3.6/site-packages/torch/share/cmake/Torch")

find_package(catkin REQUIRED COMPONENTS
  roscpp
  rospy
  std_msgs
  cv_bridge
)
find_package(Torch REQUIRED)
find_package(OpenCV 3 REQUIRED)
## System dependencies are found with CMake's conventions
# find_package(Boost REQUIRED COMPONENTS system)

###################################
## catkin specific configuration ##
###################################
catkin_package(
#  INCLUDE_DIRS include
#  LIBRARIES inference_pytorch
#  CATKIN_DEPENDS roscpp rospy std_msgs
#  DEPENDS system_lib
)

###########
## Build ##
###########

## Specify additional locations of header files
## Your package locations should be listed before other locations
include_directories(
# include
  ${catkin_INCLUDE_DIRS}
  ${OPENCV_INCLUDE_DIRS}
  ${Torch_INSTALL_INCLUDE}
  ${Torch_DIR}
)

#############
## Install ##
#############

# all install targets should use catkin DESTINATION variables
# See http://ros.org/doc/api/catkin/html/adv_user_guide/variables.html

install(PROGRAMS
   scripts/inference_test.py
   DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION}
 )

 catkin_install_python(PROGRAMS scripts/inference_test.py
                      DESTINATION ${CATKIN_PACKAGE_BIN_DESTINATION})

link_directories(
  ${Torch_INSTALL_LIB}
)

Now, how should I edit the CMAKE file to add the mentioned libs to my ROS package?


title: Loading a huge dataset batch-wise to train pytorch

body:
I am training a LSTM in-order to classify the time-series data into 2 classes(0 and 1).I have huge data-set on the drive where where the 0-class and the 1-class data are located in different folders.I am trying to train the LSTM batch-wise using by creating a Dataset class and wrapping the DataLoader around it. I have to do pre-processing such as reshaping.Here's my code which does that

`

class LoadingDataset(Dataset):
  def __init__(self,data_root1,data_root2,file_name):
    self.data_root1=data_root1#Has the path for class1 data
    self.data_root2=data_root2#Has the path for class0 data
    self.fileap1= pd.DataFrame()#Stores class 1 data
    self.fileap0 = pd.DataFrame()#Stores class 0 data
    self.file_name=file_name#List of all the files at data_root1 and data_root2
    self.labs1=None #Will store the class 1 labels
    self.labs0=None #Will store the class 0 labels

  def __len__(self):
    return len(self.fileap1)

  def __getitem__(self, index):
    self.fileap1 = pd.read_csv(self.data_root1+self.file_name[index],header=None)#read the csv file for class 1
    self.fileap1=self.fileap1.iloc[1:,1:].values.reshape(-1,WINDOW+1,1)#reshape the file for lstm
    self.fileap0 = pd.read_csv(self.data_root2+self.file_name[index],header=None)#read the csv file for class 0
    self.fileap0=self.fileap0.iloc[1:,1:].values.reshape(-1,WINDOW+1,1)#reshape the file for lstm
    self.labs1=np.array([1]*len(self.fileap1)).reshape(-1,1)#create the labels 1 for the csv file
    self.labs0=np.array([0]*len(self.fileap0)).reshape(-1,1)#create the labels 0 for the csv file
    # print(self.fileap1.shape,' ',self.fileap0.shape)
    # print(self.labs1.shape,' ',self.labs0.shape)
    self.fileap1=np.append(self.fileap1,self.fileap0,axis=0)#combine the class 0 and class one data
    self.fileap1 = torch.from_numpy(self.fileap1).float()
    self.labs1=np.append(self.labs1,self.labs0,axis=0)#combine the label0 and label 1 data
    self.labs1 = torch.from_numpy(self.labs1).int()
    # print(self.fileap1.shape,' ',self.fileap0.shape)
    # print(self.labs1.shape,' ',self.labs0.shape)

    return self.fileap1,self.labs1

data_root1 = '/content/gdrive/My Drive/Data/Processed_Data/Folder1/One_'#location of class 1 data
data_root2 = '/content/gdrive/My Drive/Data/Processed_Data/Folder0/Zero_'#location of class 0 data
training_set=LoadingDataset(data_root1,data_root2,train_ind)#train_ind is a list of file names that have to be read from data_root1 and data_root2
training_generator = DataLoader(training_set,batch_size =2,num_workers=4)

for epoch in range(num_epochs):
  model.train()#Setting the model to train mode after eval mode to train for next epoch once the testing for that epoch is finished
  for i, (inputs, targets) in enumerate(train_loader):
    .
    .
    .
    .

`
I get this error when the run this code

  RuntimeError: Traceback (most recent call last):
    File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
      samples = collate_fn([dataset[i] for i in batch_indices])
    File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 68, in default_collate
      return [default_collate(samples) for samples in transposed]
    File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 68, in
      return [default_collate(samples) for samples in transposed]
    File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/collate.py", line 43, in default_collate
      return torch.stack(batch, 0, out=out)
  RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 96596 and 25060 in dimension 1 at /pytorch/aten/src/TH/generic/THTensor.cpp:711

My Questions are
1.Have I Implemented this correctly, is this how you pre-process and then train a dataset batch-wise?

2.The batch_size of DataLoader and batch_size of the LSTM are different since the batch_size of DataLoader refers to the no. of files, whereas batch_size of the LSTM model refers to the no. of instances, so will I get another error here?

3.I have no idea how to scale this data-set since the MinMaxScaler has to be applied to the dataset in its entirety.

Responses are appreciated.Please let me know if I have to create separate posts for each question.

Thank You.


title: Everytime I use cuda() to remove Variable from CPU to GPU in pytorch,it takes about 5 to 10 minitues

body:
I just do this:

t = Variable(torch.randn(5))
t =t.cuda()
print(t)

but it takes 5 to 10 minitues,everytime.
I used cuda samples to test bandwidth, it's fine.
Then I used pdb to find which takes the most time.
I find in /anaconda3/lib/python3.6/site-packages/torch/cuda/__init__:

def _lazy_new(cls, *args, **kwargs):
    _lazy_init()
    # We need this method only for lazy init, so we can remove it
    del _CudaBase.__new__
    return super(_CudaBase, cls).__new__(cls, *args, **kwargs)

it takes about 5 minitues in the return
I don't know how to solve my problem by these imformation.
My environment is: Ubuntu 16.04 + CUDA 9.1


title: Can't change the Anchors in Faster RCNN

body:
I'm a newbie in pytorch and I was trying to put some custom anchors on my Faster RCNN network in pytorch. Basically, I'm using a resnet50 backbone and when I try to put the anchors, I got a mismatch error.

This is the code that I have:

backbone = torchvision.models.detection.backbone_utils.resnet_fpn_backbone('resnet50', True)
backbone.out_channels = 256

anchor_generator = AnchorGenerator(sizes=((4, 8, 16, 32, 64, 128),),
                                    aspect_ratios=((0.5, 1.0, 2.0),))

roi_pooler = torchvision.ops.MultiScaleRoIAlign(featmap_names=[0],
                                                    output_size=7,
                                                    sampling_ratio=2)

model = FasterRCNN(backbone,
                   num_classes=10,
                   rpn_anchor_generator=anchor_generator,
                   box_roi_pool=roi_pooler)

The error that I got is the following: shape '[1440000, -1]' is invalid for input of size 7674336.


title: how to identify wrong classification with batches in pytorch

body:
I have a script like this where batches of images were used

correct = 0
total = 0
incorrect_classification=[]
for (i, [images, labels]) in enumerate(test_loader):
  images = Variable(images.view(-1, n_pixel*n_pixel))
  outputs = net(images)
  _, predicted = torch.min(outputs.data, 1)
  total += labels.size(0)
  correct += (predicted == labels).sum()
print('Accuracy: %d %%' %
      (100 * correct / total))

With a batch size of 10, each enumerate returns 10 x image size tensor. How can I save all the wrong classifications into the array incorrect_classification or wrong img and their probability into an dictionary so I use can plt.imshow to inspect them later?

If batch size is 1 I could use this:

if (predicted==labels).item()==0:
    incorrect_examples.append(images.numpy())

But with a batch size specified (like 100 images per batch) how should I save the wrong classifications?

Thanks in advance for any answers.


title: how to adjust dataloader and make a new dataloader?

body:
let say I have a data loader of cifar10
if I want to remove some value from the dataloader and make a new dataloader
how should I do it?

def load_data_cifar10(batch_size=128,test=False):
    if not test:
        train_dset = torchvision.datasets.CIFAR10(root='/mnt/3CE35B99003D727B/input/pytorch/data', train=True,
                                                download=True, transform=transform)
    else:
        train_dset = torchvision.datasets.CIFAR10(root='/mnt/3CE35B99003D727B/input/pytorch/data', train=False,
                                               download=True, transform=transform)
    train_loader = torch.utils.data.DataLoader(train_dset, batch_size=batch_size, shuffle=True)
    print("LOAD DATA, %d" % (len(train_loader)))
    return train_loader


title: FastAI lrfind() method doesn’t work properly

body:
Update 1

I updated my lr according to " you want to be 10x back from that point, regardless of slope." and set it to
max_lr=-slice(1e-3, 1e-2)

And here is what I got

And the plots

What does this mean?

As you can see in the 2nd graph that

the loss was very good starting from 1e-08, but I never set my lr to 1e-08, why do I see this??
the loss went up and down between 1e-07 and 1e-04 and eventually it soared to almost 0.05 when the lr came back around 4e-05. What does this mean? Overfitting? How come initially when the Learning Rate was around the same value(4e-05) the loss looked okay?
from the Batches processed/Loss, I can see that train_loss and valid_loss went together and looked really well. This means the model was trained very well? If it was well trained, why the shot-up at the end of graph 2?
I have followed the rule about picking up the correct lr, why does not it work? May I conclude that the lr_find() does not work properly?

Here is my lr_find() plot

then according to its graph, I picked up the steepest slope section: 1e-2 to 1e-1 as my lr.

Here is the code:

learn.fit_one_cycle(20, max_lr=slice(1e-2,1e-1))

But here is what I got during training

And here are the plots for learn.recoder

learn.recorder.plot_lr()
learn.recorder.plot()
learn.recorder.plot_losses()

As you can see the valid_loss is getting worse cyclically. So my conclusion is lr_find() method doesn’t work properly.

How can I verify it?

If you want to see the entire code, here it is; the only difference is I use to_fp16():

learn = cnn_learner(data, models.resnet50, metrics=error_rate).to_fp16()

https://forums.fast.ai/t/train-loss-and-valid-loss-look-very-good-but-predicting-really-bad/60925


title: Why can't I use Cross Entropy Loss for multilabel?

body:
I'm in the process of finetuning a BERT model to the long answer task in the Natural Questions dataset. I'm training the model just like a SQuAD model (predicting start and end tokens).
I use Huggingface and PyTorch.
So the targets and labels have a shape/size of [batch, 2]. My problem is that I can't input "multi-targets" which I think is refering to the fact that the last shape is 2.

RuntimeError: multi-target not supported at /pytorch/aten/src/THCUNN/generic/ClassNLLCriterion.cu:18

Should I choose another loss function or is there another way to bypass this problem?
This code I'm using:
def loss_fn(preds, targets):
    return nn.CrossEntropyLoss()(preds,labels)

class DecoderModel(nn.Module):

    def __init__(self, model_args, encoder_config, loss_fn):
        super(DecoderModel, self).__init__()
        # ...

    def forward(self, pooled_output, labels):
        pooled_output = self.dropout(pooled_output)
        logits = self.linear(pooled_output)

        start_logits, end_logits = logits.split(1, dim = -1)
        start_logit = torch.squeeze(start_logits, axis=-1)
        end_logit = torch.squeeze(end_logits, axis=-1)

        # Concatenate into a "label"
        preds = torch.cat((start_logits, end_logits), -1)

        # Calculate loss
        loss = self.loss_fn(
            preds = preds,
            labels = labels)

        return loss, preds

The targets properties are:
torch.int64 & [3,2]
The predictions properties are:
torch.float32 & [3,2]
SOLVED - this is my solution
def loss_fn(preds:list, labels):
    start_token_labels, end_token_labels = labels.split(1, dim = -1)
    start_token_labels = start_token_labels.squeeze(-1)
    end_token_labels = end_token_labels.squeeze(-1)

    print('*'*50)
    print(preds[0].shape) # preds [0] and [1] has the same shape and dtype
    print(preds[0].dtype) # preds [0] and [1] has the same shape and dtype
    print(start_token_labels.shape) # labels [0] and [1] has the same shape and dtype
    print(start_token_labels.dtype) # labels [0] and [1] has the same shape and dtype

    start_loss = nn.CrossEntropyLoss()(preds[0], start_token_labels)
    end_loss = nn.CrossEntropyLoss()(preds[1], end_token_labels)

    avg_loss = (start_loss + end_loss) / 2
    return avg_loss

Basically I'm splitting the logits (just not concatinating them) and the labels. I then do Cross Entropy loss on both of them and at last taking the average loss between the two. Hope this gives you an idea to solve your own problem!


title: Huggingface Transformer - GPT2 resume training from saved checkpoint

body:
Resuming the GPT2 finetuning, implemented from run_clm.py
Does GPT2 huggingface  has a parameter to resume the training from the saved checkpoint, instead training again from the beginning? Suppose the python notebook crashes while training, the checkpoints will be saved, but when I train the model again still it starts the training from the beginning.
Source: here
finetuning code:
!python3 run_clm.py \
    --train_file source.txt \
    --do_train \
    --output_dir gpt-finetuned \
    --overwrite_output_dir \
    --per_device_train_batch_size 2 \
    --model_name_or_path=gpt2 \
    --save_steps 100 \
    --num_train_epochs=1 \
    --block_size=200 \
    --tokenizer_name=gpt2

From the above code, run_clm.py is a script provided by huggingface to finetune gpt2 to train with the customized dataset


title: Why do I have error with trained model from ubuntu to windows?

body:
I have trained the model on a super computer(ubuntu). After the training I used the model with Windows 10 and got this error:

SourceChangeWarning: source code of class 'torch.nn.modules.linear.Linear' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
  warnings.warn(msg, SourceChangeWarning)

I can't load the model I trained.

pytorch version in ubuntu: 1.1.0a0+9a7bcac

pytorch version in windows: 0.4.1

What is going wrong and how can I fix it?


title: How to use a PyTorch model without calculating gradients?

body:
I found a line of code in this file:

out_a, out_p, out_n = model(data_a), model(data_p), model(data_n)

As you can see the forward() function is invoked multiple times before backward(),In my test,the GPU consumption will increase accordingly,so GPU memory leaking happens. The questions here are:

How to solve this problem?
Is it possible use model as a pure feature extractor for these three consecutive invoking (with some modification,of course),then in fourth invoking,use it as forward(),if "Yes",how to implement it?


title: Getting started: Huggingface Model Cards

body:
I just recently started looking into the huggingface transformer library.
When I tried to get started using the model card code at e.g. community model

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

However, I got the following error:

Traceback (most recent call last):
  File "test.py", line 2, in <module>
    tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
  File "/Users/Lukas/miniconda3/envs/nlp/lib/python3.7/site-packages/transformers/tokenization_auto.py", line 124, in from_pretrained
    "'xlm', 'roberta', 'ctrl'".format(pretrained_model_name_or_path))
ValueError: Unrecognized model identifier in emilyalsentzer/Bio_ClinicalBERT. Should contains one of 'bert', 'openai-gpt', 'gpt2', 'transfo-xl', 'xlnet', 'xlm', 'roberta', 'ctrl'

If I try a different tokenizer such as "baykenney/bert-base-gpt2detector-topp92" I get the following error:

OSError: Model name 'baykenney/bert-base-gpt2detector-topp92' was not found in tokenizers model name list (bert-base-uncased, bert-large-uncased, bert-base-cased, bert-large-cased, bert-base-multilingual-uncased, bert-base-multilingual-cased, bert-base-chinese, bert-base-german-cased, bert-large-uncased-whole-word-masking, bert-large-cased-whole-word-masking, bert-large-uncased-whole-word-masking-finetuned-squad, bert-large-cased-whole-word-masking-finetuned-squad, bert-base-cased-finetuned-mrpc, bert-base-german-dbmdz-cased, bert-base-german-dbmdz-uncased). We assumed 'baykenney/bert-base-gpt2detector-topp92' was a path or url to a directory containing vocabulary files named ['vocab.txt'] but couldn't find such vocabulary files at this path or url.

Did I miss anything to get started? I feel like the model cards indicate that these three lines of code should should be enough to get started.

I am using Python 3.7 and the transformer library version 2.1.1 and pytorch 1.5.


title: Scope of with torch.no_grad() in pytorch

body:
with torch.no_grad():
    input = Variable(input).cuda()
    target = Variable(target).cuda(non_blocking=True)
y=model(input)
# many things here

Is the no_grad continue to having effect out of the "with" scope?


title: What is the role of pytorch dataset.imagefolder's loader option?

body:
Loader option
What is the role of PyTorch dataset.imagefolder's loader option?
Actually, now I have a 10,000 images in a folder, but I want to use only 100 images. Now I load all the image and subset dataset by index.
from torch.utils.data import Subset
a = dset.ImageFolder(root=path_F,transform=transform)
index = list(range(0,100))
b = Subset(a,index)

Can loader option make my code simple?


title: In PyTorch is GPU memory freed when a GPU tensor is assigned a new value?

body:
When a Cuda variable in PyTorch is assigned a new value, it becomes a CPU variable again (As illustrated by the code below). In this case, is the memory held by the variable on the GPU previously is freed automatically?
import torch

t1 = torch.empty(4,5)

if torch.cuda.is_available():
  t1 = t1.cuda()

print(t1.is_cuda)

t1 = torch.empty(4,5)
print(t1.is_cuda)

The output of the above code is:
True
False


title: How to train a simple, vanilla transformers translation model from scratch with Fairseq

body:
I have been familiarizing myself with the fairseq library recently, and have tried a couple of pretrained models. I thought that a good way to teach myself would be to train a plain vanilla transformers model with the data I have, and then I can modify and maybe add bells and whistles like pre-training from there. The fairseq documentation has an example of this with fconv architecture, and I basically would like to do the same with transformers.

Below is the code I tried:

In data preparation, I cleaned the data with moses script, tokenized words, and then applied BPE using subword-nmt, where I set number of BPE tokens to 15000.

For preprocessing:

fairseq-preprocess --source-lang zh --target-lang en \
    --trainpref data/train --validpref data/valid --testpref data/test \
    --joined-dictionary \
    --destdir data-bin \
    --workers 20

For training:

CUDA_VISIBLE_DEVICES=0,1,2,3

fairseq-train data-bin \
    --clip-norm 0.1 --dropout 0.2 --max-tokens 2048 \
    --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 \
    --lr 5e-4 --lr-scheduler inverse_sqrt \
    --criterion label_smoothed_cross_entropy \
    --lazy-load \
    --update-freq 4 \
    --keep-interval-updates 100 --save-interval-updates 3000  --log-interval 50 \
    --arch transformer --save-dir checkpoints/transformer

I trained this on a data set of ~19M samples, on 4 NVIDIA P100 GPUs, for about 8 hours -- at that point I had completed 1 epoch and a bit more. I tested this against my checkpoints -- for the first checkpoint at update 3000, the prediction was all "the the the"s -- but that might be ok because it was just the first checkpoint. However, I then tested this against the last checkpoint, and the prediction was the same sentence for all test samples!! -- The prediction was "committee on the peaceful uses of outer space" for everything, and the BLEU score was 0. My test set is not at all about outer space.

So after this extremely disappointing result, I realized that I should ask for some pointers on creating a basic transformers model:

First of all, is my result actually within expectation? The paper on which transformer.py is based, Jointly Learning to Align and Translate, stated that state of the art results are achieved on 64 Volta GPUs for 30k updates (!!!) -- my set up was much smaller, so maybe the result was expected? However, I have achieved better results in less time with less data, so I doubt that. Is it just that the learning rate was not set right so that it got stuck in some weird local minima? Or are there more things wrong with my setup above?
When would the above model stop? max_epoch and max_update are not required parameters and are set to math.inf when not given. From train.py, it looks like training goes on until learning rate gets to below args.min_lr, however I can't find where min_lr is set, and it is not a parameter in the documentation, so what is min_lr? Is it 0?
What is the best architecture to use for the "vanilla" transformer model that I'm looking for?

Thank you!


title: PyTorch has CUDA version 10.2 and torch_sparse has CUDA version 11.1 on Ubuntu 20.04

body:
I am working on Ubuntu 20.04 and it looks like CUDA10.2 does not support the latest version of Ubuntu.
Does anyone know how can I fix this problem?
The full error:

RuntimeError: Detected that PyTorch and torch_sparse were compiled
with different CUDA versions. PyTorch has CUDA version 10.2 and
torch_sparse has CUDA version 11.1. Please reinstall the torch_sparse
that matches your PyTorch install.

Thanks!


title: classification using 4-channel images in pytorch

body: I have some gray scale and color images with label. I want to combine this gray and color images (4-channel) and run transfer learning using 4-channel images. How to do that?


title: Huggingface transformers unusual memory use

body:
I have the following code attempting to use XL transformers to vectorize text:
  text = "Some string about 5000 characters long"

  tokenizer = TransfoXLTokenizerFast.from_pretrained('transfo-xl-wt103', cache_dir=my_local_dir, local_files_only=True)
  model = TransfoXLModel.from_pretrained("transfo-xl-wt103", cache_dir=my_local_dir, local_files_only=True)

  encoded_input = tokenizer(text, return_tensors='pt')
  output = model(**encoded_input)

This produces:
    output = model(**encoded_input)
  File "/home/user/w/default/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/w/default/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 863, in forward
    output_attentions=output_attentions,
  File "/home/user/w/default/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/w/default/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 385, in forward
    dec_inp, r, attn_mask=dec_attn_mask, mems=mems, head_mask=head_mask, output_attentions=output_attentions,
  File "/home/user/w/default/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/w//default/lib/python3.7/site-packages/transformers/modeling_transfo_xl.py", line 338, in forward
    attn_score = attn_score.float().masked_fill(attn_mask[:, :, :, None], -1e30).type_as(attn_score)
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 2007869696 bytes. Error code 12 (Cannot allocate memory)

I'm a little perplexed by this because this is asking for 2007869696, which is only 2GB and this machine has 64G of RAM. So I both don't understand why it is asking for this, and even more, why it is failing to get it.
Where can I change the setting that controls this and allow this process more RAM?  This is such a small invocation of the example code, and I just see very few places that would even accept this argument.


title: pytorch Crossentropy error in simple example of NN

body:
H1, I am try to make NN model that satisfy simple formula.
y = X1^2 + X2^2

But when i use CrossEntropyLoss for loss function, i get two different error message.
First, when i set code like this

x = torch.randn(batch_size, 2)
y_hat = model(x)
y = answer(x).long()

optimizer.zero_grad()
loss = loss_func(y_hat, y)
loss.backward()
optimizer.step()

i get this message

RuntimeError: Assertion `cur_target >= 0 && cur_target < n_classes' failed.  at
c:\programdata\miniconda3\conda-bld\pytorch_1533090623466\work\aten\src\thnn\generic/Cl

assNLLCriterion.c:93

Second, I change code like this

x = torch.randn(batch_size, 2)
y_hat = model(x)
y = answer(x).long().view(batch_size,1,1)

optimizer.zero_grad()
loss = loss_func(y_hat, y)
loss.backward()
optimizer.step()

then i get message like

RuntimeError: multi-target not supported at c:\programdata\miniconda3\conda-bld\pytorch_1533090623466\work\aten\src\thnn\generic/ClassNLLCriterion.c:21

How can i solve this problem? Thanks.(sorry for my English)
This is my code

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

def answer(x):

    y = x[:,0].pow(2) + x[:,1].pow(2)

    return y

class Model(nn.Module):

    def __init__(self, input_size, output_size):
        super(Model, self).__init__()

        self.linear1 = nn.Linear(input_size, 10)
        self.linear2 = nn.Linear(10, 1)

    def forward(self, x):

        y = F.relu(self.linear1(x))
        y = F.relu(self.linear2(y))

        return y

model = Model(2,1)
print(model, '\n')

loss_func = nn.CrossEntropyLoss()
optimizer = optim.SGD(model.parameters(), lr = 0.001)

batch_size = 3
epoch_n = 100
iter_n = 100

for epoch in range(epoch_n):
    loss_avg = 0

    for i in range(iter_n):

        x = torch.randn(batch_size, 2)
        y_hat = model(x)
        y = answer(x).long().view(batch_size,1,1)

        optimizer.zero_grad()
        loss = loss_func(y_hat, y)
        loss.backward()
        optimizer.step()

        loss_avg += loss

    loss_avg = loss_avg / iter_n

    if epoch % 10 == 0:
        print(loss_avg)

    if loss_avg < 0.001:
        break

Can i make those dataset using dataloader in pytorch? Thanks for your help.


title: pip install horovod fails on conda + OSX 10.14

body:
Running pip install horovod in a conda environment with pytorch installed resulted in

error: None of TensorFlow, PyTorch, or MXNet plugins were built. See errors above.

where the root problem near the top of stdout is

ld: library not found for -lstdc++
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  INFO: Unable to build PyTorch plugin, will skip it.


title: What is Tensorflow equivalent of pytorch's conv1d?

body:
Just wondering how I can perform 1D convolution in tensorflow. Specifically, looking to replace this code to tensorflow:

inputs = F.pad(inputs, (kernel_size-1,0), 'constant', 0)

output = F.conv1d(inputs, weight, padding=0, groups=num_heads)


title: Basic matrix operation in PyTorch/CuPy using GPU

body:
I have a numpy script where I do the following operation with big matrices (can go over 10000x10000 with float values):

F = (I - Q)^-1 * R

I first used pytorch tensors on CPU (i7-8750H) and it runs 2 times faster:

tensorQ = torch.from_numpy(Q)
tensorR = torch.from_numpy(R)
sub= torch.eye(a * d, dtype=float) - tensorQ
inv= torch.inverse(sub)
tensorF = torch.mm(inv, tensorR)
F = tensorF.numpy()

Now I'm trying to execute it on GPU (1050Ti Max-Q) to see if I can get another speedup but the code run slower than numpy version (I've already installed CUDA and cuDNN). Maybe pytorch it's not even the best library to do this kind of things, but I'm learning it now and I thought it could help me:

dev = torch.device('cuda')

tensorQ = torch.from_numpy(Q).to(dev)
tensorR = torch.from_numpy(R).to(dev)
sub= torch.eye(a * d, dtype=float).to(dev) - tensorQ
inv= torch.inverse(sub).to(dev)
tensorF = torch.mm(inv, tensorR).cpu()
F = tensorF.numpy()

Am I missing something?

Edit:
I also tried using CuPy but it's still slow:

Q = cp.array(matrixQ)
R = cp.array(matrixR)
sub = cp.identity(attacker * defender) - matrixQ
inv = cp.linalg.inv(sub)
F = cp.matmul(inv, matrixR)
F = cp.asnumpy(matrixF)

Probably the overhead of memory allocation is just too big compared to the computation of few operations.


title: Generating pytorch's theta from affine transform matrix

body:
I'm trying to perform a rigid + scale transformation on a 3D volume with pytorch, but I can't seem to understand how the theta required for torch.nn.functional.affine_grid works.

I have a transformation matrix of size (1,4,4) generated by multiplying the matrices Translation * Scale * Rotation. If I use this matrix in, for example, scipy.ndimage.affine_transform, it works with no issues. However, the same matrix (cropped to size (1,3,4)) fails completely with torch.nn.functional.affine_grid.

I have managed to understand how the translation works (range -1 to 1) and I have confirmed that the Translation matrix works by simply normalizing the values to that range. As for the other two, I am lost.

I tried using a basic Scaling matrix alone (below) as a most basic comparison but the results in pytorch are different than that of scipy

Scaling =
[[0.75, 0, 0, 0],
[[0, 0.75, 0, 0],
[[0, 0, 0.75, 0],
[[0, 0, 0, 1]]

How can I convert the (1,4,4) affine matrix to work the same with torch.nn.functional.affine_grid? Alternatively, is there a way to generate the correct matrix based on the transformation parameters (shift, euler angles, scaling)?


title: Running out of GPU memory with PyTorch

body:
I am running my own custom deep belief network code using PyTorch and using the LBFGS optimizer. After optimization starts, my GPU starts to run out of memory, fully running out after a couple of batches, but I'm not sure why. Should I be purging memory after each batch is run through the optimizer? My code is as follows (with the portion of code that causes the problem marked):
def fine_tuning(self, data, labels, num_epochs=10, max_iter=3):
        '''
        Parameters
        ----------
        data : TYPE torch.Tensor
            N x D tensor with N = num samples, D = num dimensions
        labels : TYPE torch.Tensor
            N x 1 vector of labels for each sample
        num_epochs : TYPE, optional
            DESCRIPTION. The default is 10.
        max_iter : TYPE, optional
            DESCRIPTION. The default is 3.

        Returns
        -------
        None.

        '''
        N = data.shape[0]
        #need to unroll the weights into a typical autoencoder structure
        #encode - code - decode
        for ii in range(len(self.rbm_layers)-1, -1, -1):
            self.rbm_layers.append(self.rbm_layers[ii])

        L = len(self.rbm_layers)
        optimizer = torch.optim.LBFGS(params=list(itertools.chain(*[list(self.rbm_layers[ii].parameters())
                                                                    for ii in range(L)]
                                                                  )),
                                      max_iter=max_iter,
                                      line_search_fn='strong_wolfe')

        dataset     = torch.utils.data.TensorDataset(data, labels)
        dataloader  = torch.utils.data.DataLoader(dataset, batch_size=self.batch_size*10, shuffle=True)
        #fine tune weights for num_epochs
        for epoch in range(1,num_epochs+1):
            with torch.no_grad():
                #get squared error before optimization
                v = self.pass_through_full(data)
                err = (1/N) * torch.sum(torch.pow(data-v.to("cpu"), 2))
            print("\nBefore epoch {}, train squared error: {:.4f}\n".format(epoch, err))

           #*******THIS IS THE PROBLEM SECTION*******#
            for ii,(batch,_) in tqdm(enumerate(dataloader), ascii=True, desc="DBN fine-tuning", file=sys.stdout):
                print("Fine-tuning epoch {}, batch {}".format(epoch, ii))
                with torch.no_grad():
                    batch = batch.view(len(batch) , self.rbm_layers[0].visible_units)
                    if self.use_gpu: #are we using a GPU?
                        batch = batch.to(self.device) #if so, send batch to GPU
                    B = batch.shape[0]
                    def closure():
                        optimizer.zero_grad()
                        output = self.pass_through_full(batch)
                        loss = nn.BCELoss(reduction='sum')(output, batch)/B
                        print("Batch {}, loss: {}\r".format(ii, loss))
                        loss.backward()
                        return loss
                    optimizer.step(closure)

The error I get is:
DBN fine-tuning: 0it [00:00, ?it/s]Fine-tuning epoch 1, batch 0
Batch 0, loss: 4021.35400390625
Batch 0, loss: 4017.994873046875
DBN fine-tuning: 0it [00:00, ?it/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/deep_autoencoder/deep_autoencoder.py", line 260, in fine_tuning
    optimizer.step(closure)
  File "/home/anaconda3/envs/torch_env/lib/python3.8/site-packages/torch/autograd
/grad_mode.py", line 15, in decorate_context
    return func(*args, **kwargs)
  File "/home/anaconda3/envs/torch_env/lib/python3.8/site-packages/torch/optim/lb
fgs.py", line 425, in step
    loss, flat_grad, t, ls_func_evals = _strong_wolfe(
  File "/home/anaconda3/envs/torch_env/lib/python3.8/site-packages/torch/optim/lb
fgs.py", line 96, in _strong_wolfe
    g_prev = g_new.clone(memory_format=torch.contiguous_format)
RuntimeError: CUDA out of memory. Tried to allocate 1.57 GiB (GPU 0; 24.00 GiB total capac
ity; 13.24 GiB already allocated; 1.41 GiB free; 20.07 GiB reserved in total by PyTorch)

This also racks up memory if I use CPU, so I'm not sure what the solution is here...


title: torch.nn.LSTM runtime error

body:
I'm trying to implement the structure from "Livelinet: A multimodal Deep Recurrent Neural Network to Predict Liveliness in Educational Videos".

For a brief explanation, I split 10-second audio clip into ten 1-second audio clip and get a spectrogram (a picture) from that 1-second audio clip. Then I use CNN to get a representation vector from the picture, finally getting 10 vectors of each 1-second video clips.

Next, I feed those 10 vectors to LSTM, and I got some errors there.
My code and the error traceback are as follows:

class AudioCNN(nn.Module):

def __init__(self):
    super(AudioCNN,self).__init__()
    self.features = alexnet.features
    self.features2 = nn.Sequential(*classifier)
    self.lstm = nn.LSTM(512, 256,2)
    self.classifier = nn.Linear(2*256,2)

def forward(self, x):
    x = self.features(x)
    print x.size()
    x = x.view(x.size(0),256*6*6)
    x = self.features2(x)
    x = x.view(10,1,512)
    h_0,c_0 = self.init_hidden()
    _, (_, _) = self.lstm(x,(h_0,c_0)) # x dim : 2 x 1 x 256
    assert False
    x = x.view(1,1,2*256)
    x = self.classifier(x)

    return x

def init_hidden(self):
    h_0 = torch.randn(2,1,256) #layer * batch * input_dim
    c_0 = torch.randn(2,1,256)
    return h_0, c_0

audiocnn = AudioCNN()
input = torch.randn(10,3,223,223)
input = Variable(input)
audiocnn(input)

Error:

RuntimeErrorTraceback (most recent call last)
<ipython-input-64-2913316dbb34> in <module>()
----> 1 audiocnn(input)

/home//local/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    222         for hook in self._forward_pre_hooks.values():
    223             hook(self, input)
--> 224         result = self.forward(*input, **kwargs)
    225         for hook in self._forward_hooks.values():
    226             hook_result = hook(self, input, result)

<ipython-input-60-31881982cca9> in forward(self, x)
     15         x = x.view(10,1,512)
     16         h_0,c_0 = self.init_hidden()
---> 17         _, (_, _) = self.lstm(x,(h_0,c_0)) # x dim : 2 x 1 x 256
     18         assert False
     19         x = x.view(1,1,2*256)

/home/local/lib/python2.7/site-packages/torch/nn/modules/module.pyc in __call__(self, *input, **kwargs)
    222         for hook in self._forward_pre_hooks.values():
    223             hook(self, input)
--> 224         result = self.forward(*input, **kwargs)
    225         for hook in self._forward_hooks.values():
    226             hook_result = hook(self, input, result)

/home//local/lib/python2.7/site-packages/torch/nn/modules/rnn.pyc in forward(self, input, hx)
    160             flat_weight=flat_weight
    161         )
--> 162         output, hidden = func(input, self.all_weights, hx)
    163         if is_packed:
    164             output = PackedSequence(output, batch_sizes)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, *fargs, **fkwargs)
    349         else:
    350             func = AutogradRNN(*args, **kwargs)
--> 351         return func(input, *fargs, **fkwargs)
    352
    353     return forward

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, weight, hidden)
    242             input = input.transpose(0, 1)
    243
--> 244         nexth, output = func(input, hidden, weight)
    245
    246         if batch_first and batch_sizes is None:

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, hidden, weight)
     82                 l = i * num_directions + j
     83
---> 84                 hy, output = inner(input, hidden[l], weight[l])
     85                 next_hidden.append(hy)
     86                 all_output.append(output)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in forward(input, hidden, weight)
    111         steps = range(input.size(0) - 1, -1, -1) if reverse else range(input.size(0))
    112         for i in steps:
--> 113             hidden = inner(input[i], hidden, *weight)
    114             # hack to handle LSTM
    115             output.append(hidden[0] if isinstance(hidden, tuple) else hidden)

/home//local/lib/python2.7/site-packages/torch/nn/_functions/rnn.pyc in LSTMCell(input, hidden, w_ih, w_hh, b_ih, b_hh)
     29
     30     hx, cx = hidden
---> 31     gates = F.linear(input, w_ih, b_ih) + F.linear(hx, w_hh, b_hh)
     32
     33     ingate, forgetgate, cellgate, outgate = gates.chunk(4, 1)

/home//local/lib/python2.7/site-packages/torch/nn/functional.pyc in linear(input, weight, bias)
    551     if input.dim() == 2 and bias is not None:
    552         # fused op is marginally faster
--> 553         return torch.addmm(bias, input, weight.t())
    554
    555     output = input.matmul(weight.t())

/home//local/lib/python2.7/site-packages/torch/autograd/variable.pyc in addmm(cls, *args)
    922         @classmethod
    923         def addmm(cls, *args):
--> 924             return cls._blas(Addmm, args, False)
    925
    926         @classmethod

/home//local/lib/python2.7/site-packages/torch/autograd/variable.pyc in _blas(cls, args, inplace)
    918             else:
    919                 tensors = args
--> 920             return cls.apply(*(tensors + (alpha, beta, inplace)))
    921
    922         @classmethod

RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition


title: Input labels for semantic segmentation with U-Net as single layer

body:
When doing semantic segmentation with the U-Net for example, it seems to be common-practice to provide the label data as one-hot-encoded tensors. In another SO question, a user pointed out that this is due to the labels usually representing categorical values. Feeding them to the network as class labels within only one layer (as greyscale intensity values) would introduce difficulties.

In another blog post however, the author explains that the labels

  "[...] sometimes [get] packaged as greyscale images, where the pixel intensity represents the class id [...]. This method can be the easiest to work with. It allows for a small file size for distribution and [...] One Hot Vector representations [use] up more memory than [the greyscale encoding format]."

My hardware is only very limited, and I am hoping that encoding the labels as 1-layered greyscale tensors, rather than n-layered (n being the number of classes to segment), will lead to lower memory usage. However, the author of the blog then also states:

  "Even if the deep learning framework you use accepts the labels data as class ids, as in [the greyscale format], it will convert that data to one-hot encoding behind the scenes."

Does this mean, there wouldn't be any savings memory-wise after all?

If it is worthwhile, how would I go on to implementing this in the dataset-reader? I also haven't encountered any implementation, where the greyscale labeling has in fact been practiced. I'd therefore also be thankful for any links to implementations that have been using greyscale labels for semantic segmentation!

I am working with PyTorch and my code is based on this implementation, with the difference that I have 3 classes to segment.

Any suggestions/links are greatly appreciated!


title: Custom initialization of weights for object of Conv1D class in PyTorch

body:
I am looking into torch.nn.Conv1D class that I would like to initialize with custom weights.

I would like these weights to be "differentiable" as they will be derived based on some other parameters.

Here is an example of what I would like to do:

# Initialize random convolution of size 3:
m = nn.Conv1d(1, 1, 3, stride=1,bias=False)

# Define custom weights
w1 = 1.
w2 =-2.
w3 = 1.
paramsNP = np.array([w1, w2, w3]).reshape((1,1,3))

# Assign weights to m.weight (THIS PART FAILS)
m.weight = torch.tensor(paramsNP, requires_grad=True)

This last assignment fails with

  TypeError: cannot assign 'torch.DoubleTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

This seams to be due to m.weight being of the type torch.nn.parameter.Parameter
What would be a good ways of performing the assignment/initialization of the Conv1D object m?

Instead of that last line I can do this assignment:

m.weight[0,0,0] = w1
m.weight[0,0,1] = w2
m.weight[0,0,2] = w3


title: Make GPU available again after numba.cuda.close()?

body:
So when I run cuda.select_device(0) and then cuda.close(). Pytorch cannot access the GPU again, I know that there is way so that PyTorch can utilize the GPU again without having to restart the kernel. But I forgot how. Does anyone else know?
from  numba import cuda as cu
import torch

# random tensor
a=torch.rand(100,100)

#tensor can be loaded onto the gpu()
a.cuda()

device = cu.get_current_device()
device.reset()

# thows error "RuntimeError: CUDA error: invalid argument"
a.cuda()

cu.close()
# thows error "RuntimeError: CUDA error: invalid argument"
a.cuda()

torch.cuda.isavailable()
>>> AttributeError: module 'torch.cuda' has no attribute 'isavailable'

And then trying to run cuda-based pytorch code yields:
RuntimeError: CUDA error: invalid argument


title: How to fix multiprocessing problems in python in windows10

body:
I try to use this tutorial to train my own car model recognition model: https://github.com/Helias/Car-Model-Recognition. And i want to use coda and my gpu perfomance to enhance training speed (preprocesssing step was completed without any errors).But when I try to train my model, I've got the following errors:

######### ERROR #######

        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

######### batch #######
Traceback (most recent call last):
  File "D:\Car-Model-Recognition\main.py", line 78, in train_model

######### ERROR #######
[Errno 32] Broken pipe
    for i, batch in enumerate(loaders[mode]):

######### batch #######  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__

    return _MultiProcessingDataLoaderIter(self)
Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
  File "main.py", line 78, in train_model
    w.start()
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    for i, batch in enumerate(loaders[mode]):
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Program Files\Python37\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    w.start()
    return Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
  File "C:\Program Files\Python37\lib\multiprocessing\process.py", line 112, in start
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    self._popen = self._Popen(self)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    _check_not_importing_main()
  File "C:\Program Files\Python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Program Files\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
    return Popen(process_obj)

I have used the exact code from given link, and if i start my code using wsl, everything is ok, but I can't use my gpu from wsl. Where should I insert this name == 'main' check to prevent such a mistake or how can i disable this multiprocessing


I currently have a neural network module:

import torch.nn as nn

class NN(nn.Module):
    def __init__(self,args,lambda_f,nn1, loss, opt):
        super().__init__()
        self.args = args
        self.lambda_f = lambda_f
        self.nn1 = nn1
        self.loss = loss
        self.opt = opt
        # more nn.Params stuff etc...

    def forward(self, x):
        #some code using fields
        return out

I am trying to checkpoint it but because pytorch saves using state_dicts it means I can't save the lambda functions I was actually using if I checkpoint with the pytorch torch.save etc. I literally want to save everything without issue and re-load to train on GPUs later. I currently am using this:

def save_ckpt(path_to_ckpt):
    from pathlib import Path
    import dill as pickle
    ## Make dir. Throw no exceptions if it already exists
    path_to_ckpt.mkdir(parents=True, exist_ok=True)
    ckpt_path_plus_path = path_to_ckpt / Path('db')

    ## Pickle args
    db['crazy_mdl'] = crazy_mdl
    with open(ckpt_path_plus_path , 'ab') as db_file:
        pickle.dump(db, db_file)

currently it throws no errors when I chekpoint it and it saved it.

I am worried that when I train it there might be a subtle bug even if no exceptions/errors are trained or something unexpected might happen (e.g. weird saving on disks in the clusters etc who knows).

Is this safe to do with pytorch classes/nn models? Especially if we want to resume training with GPUs?

Cross posted:

How does one pickle arbitrary pytorch models that use lambda functions?
https://discuss.pytorch.org/t/how-does-one-pickle-arbitrary-pytorch-models-that-use-lambda-functions/79026
https://www.reddit.com/r/pytorch/comments/gagpjg/how_does_one_pickle_arbitrary_pytorch_models_that/?
https://www.quora.com/unanswered/How-does-one-pickle-arbitrary-PyTorch-models-that-use-lambda-functions


title: How to create a Pytorch network with mixed categorical and continuous matrix input

body:
I'm creating a network network that will take a matrix of continuous values along with some categorical input represented as vectors of all the classes.

Now, I'm also looking extract features from the matrix with convolution. But this would not be possible if I reduce the matrix to dimension 1 and concatenate with the class vectors.

Is there a way to concat it together as a single input? Or do I have to create two separate input layer then somehow join them after the convolution? If its the latter, what function am I looking for?


title: Fast.ai learn.export() stopped working with no apparent code change

body:
I have been training and exporting several versions of a model on colab. My export code has always been this:
model.export('model.pkl')

I have been able to reload the model and make predictions to confirm everything works, like this:
x = load_learner('', 'model.pkl')
x.predict()

All of a sudden, the export code is behaving differently. I am still able to export and load_learner with not problem on colab, but when I download the model file from colab to my local machine and try to run locally, I get this error:
Traceback (most recent call last):
  File "process.py", line 127, in <module>
    predict_utterances_in_transcript(i, path_to_processed_and_predicted_transcript_dir, audio_id)
  File "process.py", line 94, in predict_utterances_in_transcript
    predict()
  File "/Users/src/predict.py", line 9, in predict_utterances
    model = load_learner('', 'models/model.pkl')
  File "/Users//anaconda3/envs/nlp/lib/python3.6/site-packages/fastai/basic_train.py", line 621, in load_learner
    state = torch.load(source, map_location='cpu') if defaults.device == torch.device('cpu') else torch.load(source)
  File "/Users//anaconda3/envs/nlp/lib/python3.6/site-packages/torch/serialization.py", line 586, in load
    with _open_zipfile_reader(f) as opened_zipfile:
  File "/Users//anaconda3/envs/nlp/lib/python3.6/site-packages/torch/serialization.py", line 246, in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))
AttributeError: 'PosixPath' object has no attribute 'tell'

Also, I have been able to open and read model pickle files in a text editor just fine until now. The broken model files cannot be read because an error says they are corrupted.
Nothing has changed in my code or my environment. Any ideas?


title: TypeError: 'torch.dtype' object is not callable. how to call this function?

body:
How to call this torch.dtype? because here the error shows it's not callable. before I used floatTensor and it shows the error like this can't convert np.ndarray of type numpy.object_ and now I using float64 it showing the error 'torch.dtype' object is not callable. please help with this issue.
import torch
    a = torch.cuda.is_available()
    b = torch.cuda.current_device()
    c = torch.cuda.get_device_name()
    d = torch.cuda.memory_reserved()
    e = torch.cuda.memory_allocated()
    var1 = torch.FloatTensor([1.0,2.0,3.0]).cuda()
    var1
    a1 = var1.device
    import pandas as pd
    df = pd.read_csv('diabetes.csv')
    df.head()
    b1 = df.isnull().sum()
    import seaborn as sns
    import numpy as np
    df['Outcome']=np.where(df['Outcome']==1,"Diabetic","No Diabetic")
    b2 = df.head()
    b3 = sns.pairplot(df,hue="Outcome")

    X=df.drop('Outcome',axis=1).values
    y=df['Outcome'].values

    from sklearn.model_selection import train_test_split

    X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)
    y_train

    import torch
    import torch.nn as nn
    import torch.nn.functional as F

    X_train=torch.FloatTensor(X_train).cuda()
    X_test=torch.FloatTensor(X_test).cuda()
    y_train=torch.float64(y_train).cuda()

this is the error:
 C:\Users\vinot\.conda\envs\python21\python.exe D:/python/python_work/pythonProject/diabetes.py
    Traceback (most recent call last):
      File "D:/python/python_work/pythonProject/diabetes.py", line 35, in <module>
        y_train=torch.float64(y_train).cuda()
    TypeError: 'torch.dtype' object is not callable

    Process finished with exit code 1


title: What happens when we call cpu().data.numpy() on a PyTorch tensor?

body:
I am working on a project and need to pass the data in loss tensor to a plotting library.

What happens when I perform this call -> loss.cpu().data.numpy()

Is there a risk of detaching the tensor from the computation graph?


title: Cannot install torchvision via poetry in windows

body:
I succeeded in pytorch installation thanks to answers here
Poetry and PyTorch.
But, I'm still failed to install torchvision via poetry.
> poetry add torchvision==0.8.2

Updating dependencies
Resolving dependencies...

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing torchvision (0.8.2)

  RuntimeError

  Unable to find installation candidates for torchvision (0.8.2)

  at ~\.poetry\lib\poetry\installation\chooser.py:72 in choose_for
       68│
       69│             links.append(link)
       70│
       71│         if not links:
    →  72│             raise RuntimeError(
       73│                 "Unable to find installation candidates for {}".format(package)
       74│             )
       75│
       76│         # Get the best link

Failed to add packages, reverting the pyproject.toml file to its original content.

I googled it and found some answers that say 'just pip install torchvision'.
But I'm suspicious that it works because according to PyPi(https://pypi.org/project/torchvision/#files), there is no wheel file for windows. And I tried it and it failed as I expected.
Is there any way to install latest torchvisin which is compatible with latest torch(1.7.1) in windows? + via poetry?


title: Python can't find torch module

body:
I'm simply trying to run a .py file, and not sure why it can't find the torch module. I am pretty sure all modules are up to date, and I can see the torch module in my lib/ folder...

Traceback (most recent call last):
  File "ptb-lm.py", line 86, in <module>
    import torch
  File "C:\Users\Public\Anaconda3\lib\site-packages\torch\__init__.py", line 80, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

To be honest I'm still confused as to when / why use conda or pip but I did check that both were up to date!


title: Pytorch - How to achieve higher accuracy with imdb review dataset using LSTM?

body:
I'm trying to practice with LSTM and Pytorch. I took IMDB movie review dataset to predict whether the review is positive or negative. I use 80% of the dataset for my training, remove punctuations, use GloVe (with 200 dims) as an embedding layer.

Before training, I also exclude too short (reviews with length smaller than 50 symbols) and too long (reviews with longer than 1000 symbols) reviews.

For the LSTM layer I use hidden dimension 256, num_layers 2 and one directional parameters with 0.5 dropout. Afterwards, I have fully connected layer.
For the training I used nn.BCELoss function with Adam optimizer (lr=0.001).

Currently I get 85% validation accuracy with 98% training accuracy after 7 epochs. I did following steps for preventing overfitting and getting higher accuracy:

used weight_decay for Adam optimizer,
tried SGD (lr=0.1, 0.001) instead of Adam,
tried to increase num_layers of LSTM,

In all of these cases model didn't learn at all, giving 50% of accuracy for both training and validation sets.

class CustomLSTM(nn.Module):
    def __init__(self, vocab_size, use_embed=False, embed=None, embedding_size=200, hidden_size=256,
                 num_lstm_layers=2, bidirectional=False, dropout=0.5, output_dims=2):
        super().__init__()

        self.vocab_size = vocab_size
        self.embedding_size = embedding_size
        self.hidden_size = hidden_size
        self.num_lstm_layers = num_lstm_layers
        self.bidirectional = bidirectional
        self.dropout = dropout

        self.embedding = nn.Embedding(vocab_size, embedding_size)
        if use_embed:
            self.embedding.weight.data.copy_(torch.from_numpy(embed))
#             self.embedding.requires_grad = False
        self.lstm = nn.LSTM(input_size=embedding_size,
                           hidden_size=hidden_size,
                           num_layers=num_lstm_layers,
                           batch_first=True,
                           dropout=dropout,
                           bidirectional=bidirectional)
#         print('output dims value ', output_dims)
        self.drop_fc = nn.Dropout(0.5)
        self.fc = nn.Linear(hidden_size, output_dims)
        self.sig = nn.Sigmoid()

I want to understand:

Why the model doesn't learn at all with that changes applied?
How to increase the accuracy?


title: Pytorch: TypeError 'torch.LongTensor' object is not reversible

body:
I'm trying to do a NLP task by pytorch and I used following code to pack my batch of sentences.

for iter in range(0, n_iters, batch_size):
    # batch size * max length Variable
    input_batch = input_data[iter:iter + batch_size]
    target_batch = target_data[iter:iter + batch_size]

    # batch size * 1 LongTensor
    input_batch_length = input_length[iter:iter + batch_size]
    target_batch_length = target_length[iter:iter + batch_size]

    # sort batch for pack_packed_sequence
    input_batch_length, input_indices = input_batch_length.sort(0, descending=True)
    input_batch = input_batch[input_indices]

    target_batch_length, target_indices = target_batch_length.sort(0, descending=True)
    target_batch = target_batch[target_indices]

    # TypeError 'torch.LongTensor' object is not reversible
    packed_input = nn.utils.rnn.pack_padded_sequence(input_batch, input_batch_length, batch_first=True)
    packed_target = nn.utils.rnn.pack_padded_sequence(target_batch, target_batch_length, batch_first=True)

    loss = train(packed_input, packed_target, input_batch_length, target_batch_length, encoder, decoder,
                 encoder_optimizer, decoder_optimizer, criterion)

However, I got an error that 'torch.LongTensor' object is not reversible in the line to pack the batch. Can anyone help me?


title: Pass variable sized input to Linear layer in Pytorch

body: I have a Linear() layer in Pytorch after a few Conv() layers. All the images in my dataset are black and white. However most of the images in my test set are of a different dimension than the images in my training set. Apart from resizing the images themselves, is there any way to define the Linear() layer in such a way that it takes a variable input dimension? For example something similar to view(-1)


title: PyTorch allocates more memory on the first available GPU (cuda:0)

body:
As a part of the reinforcement learning training system, I am training four policies in parallel using four GPUs. For each model, there are two processes - the actor and the learner, which only use their specific GPU (e.g. actor and learner corresponding to model #2 only use GPU #2 for all their tensors). Actor and learner share the model layers via torch's share_memory_().
Since the four training "subsystems" are completely symmetric, I would expect them to use the exact same amount of GPU memory on each of the four GPUs. In practice, however, I see a lot more GPU memory allocated on the first GPU (cuda:0).

It feels like all the memory sharing is somehow done via GPU #0. Is there a way to fix this?

So far, I tried setting CUDA_VISIBLE_DEVICES in child processes by explicitly altering os.environ in the process start function. This does not seem to have any effect, probably because child processes are forked from the main process, where PyTorch CUDA is already initialized, and envvars just seem to be ignored at this point.


title: Dropconnect implementation in pytorch

body:
I am trying to code dropconnect for Conv2D and transposeconv2D layer. Followed the tutorial in https://pytorchnlp.readthedocs.io/en/latest/_modules/torchnlp/nn/weight_drop.html to create it.

import torch
from torch.nn import Parameter

def _weight_drop(module, weights, dropout):
    for name_w in weights:
        w = getattr(module, name_w)
        del module._parameters[name_w]
        module.register_parameter(name_w + '_raw', Parameter(w))
    original_module_forward = module.forward

    def forward(*args, **kwargs):
        for name_w in weights:
            raw_w = getattr(module, name_w + '_raw')
            w = torch.nn.functional.dropout(raw_w, p=dropout, training=module.training)
            setattr(module, name_w, w)
        return original_module_forward(*args, **kwargs)
    setattr(module, 'forward', forward)

class WeightDropConv2d(torch.nn.Conv2d):
    def __init__(self, *args, weight_dropout=0.0, **kwargs):
        super().__init__(*args, **kwargs)
        weights = ['weight']
        _weight_drop(self, weights, weight_dropout)

class WeightDropConvTranspose2d(torch.nn.ConvTranspose2d):
    def __init__(self, *args, weight_dropout=0.0, **kwargs):
        super().__init__(*args, **kwargs)
        weights = ['weight']
        _weight_drop(self, weights, weight_dropout)

torch.version.cuda: 1.1.0
torch.version: 9.0.176

I get the following error in the 2nd epoch:

Traceback (most recent call last):
  File "dropconnect.py", line 110, in <module>
    out = model(image)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "dropconnect.py", line 73, in forward
    out = self.c1(x)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "dropconnect.py", line 34, in forward
    setattr(module, name_w, w)
  File "/home/sbhand2s/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 558, in __setattr__
    .format(torch.typename(value), name))
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

This error occurs in the second epoch when I switch from .eval() to .train(). This error does not occur if I don't call .eval()

Any suggestions on why this error is occurring or how to implement dropconnect in a better manner?

Code to replicate issue:

from collections import OrderedDict
import torch
from torch import nn

layers = []
layers.append(("conv_1", WeightDropConv2d(1,3,3,1,1,weight_dropout=0.5)))
layers.append(("conv_2", WeightDropConv2d(3,3,3,1,1,weight_dropout=0.5)))
layers.append(("conv_3", WeightDropConv2d(3,1,3,1,1,weight_dropout=0.5)))

model = nn.Sequential(OrderedDict(layers))

pred = model(torch.randn([1,1,3,3]))

model.eval()
pred = model(torch.randn([1,1,3,3]))

model.train()
pred = model(torch.randn([1,1,3,3]))


title: How to implement Pytorch 1D crosscorrelation for long signals in fourier domain?

body:
I have a series of signals length n = 36,000 which I need to perform crosscorrelation on. Currently, my cpu implementation in numpy is a little slow. I've heard Pytorch can greatly speed up tensor operations, and provides a way to perform computations in parallel on the GPU. I'd like to explore this option, but I'm not quite sure how to accomplish this using the framework.

Because of the length of these signals, I'd prefer to perform the crosscorrelation operation in the frequency domain.

Normally using numpy I'd perform the operation like so:

import numpy as np

signal_length=36000

# make the signals
signal_1 = np.random.uniform(-1,1, signal_length)
signal_2 = np.random.uniform(-1,1, signal_length)

# output target length of crosscorrelation
x_cor_sig_length = signal_length*2 - 1

# get optimized array length for fft computation
fast_length = np.fftpack.next_fast_len(x_cor_sig_length)

# move data into the frequency domain. axis=-1 to perform
# along last dimension
fft_1 = np.fft.rfft(src_data, fast_length, axis=-1)
fft_2 = np.fft.rfft(src_data, fast_length, axis=-1)

# take the complex conjugate of one of the spectrums. Which one you choose depends on domain specific conventions
fft_1 = np.conj(fft_1)

fft_multiplied = fft_1 * fft_2

# back to time domain.
prelim_correlation = np.fft.irfft(result, x_corr_sig_length, axis=-1)

# shift the signal to make it look like a proper crosscorrelation,
# and transform the output to be purely real

final_result = np.real(np.fft.fftshift(prelim_correlation),axes=-1)).astype(np.float64)

Looking at the Pytorch documentation, there doesn't seem to be an equivalent for numpy.conj(). I'm also not sure if/how I need to implement a fftshift after the irfft operation.

So how would you go about writing a 1D crosscorrelation in Pytorch using the fourier method?


title: NameError: name 'base' is not defined, while running open AI gym in GOOGLE COLAB

body:
I am going through the tutorial DQN reinforcement learning in Pytorch.org,https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html
But here when I am trying to render a screen and display using python display, I am getting name base not found. Can anyone help me here? If you want to clear any clarity about the question, I am here
Thanks in advance


title: Runtime Error : both arguments to matmul need to be at least 1d but they are 0d and 2d

body:
This is the code I have written, I have tried modifying here and there and have always gotten the same error. As I am a beginner to PyTorch, I am just trying things out to see if machine learning will work on a linear dataset. So, with random, I initialized a dataset. Then, made a single linear neural network. Then, train the neural network on the linear data. Nevertheless, I have an error stating that matmul expects a 1d data, but then my data is at least 1d but is read as 0d instead.
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import random

class LinearNetwork(nn.Module):
    def __init__(self):
        super(LinearNetwork, self).__init__()
        self.fc1 = nn.Linear(1, 1)

    def forward(self, input):
        x = self.fc1(input)
        return x;

#Generate a Dataset to train on. Just a linear function, and we wanna see if this thing is working
DataSet = []
grad = random.randint(0,100)
const = random.randint(0,100)
for i in range(0, 1000):
    DataSet.append([i, grad*i + const])

DataSet = torch.tensor(DataSet,dtype = torch.double, requires_grad = True)

#Declare a Linear Network
Net = LinearNetwork()
criterion = nn.MSELoss()
optimizer = optim.SGD(Net.parameters(), lr = 0.01)
print(DataSet)
#Train Model
for i, data in enumerate(DataSet, 0):
    Input, Target = data
    optimizer.zero_grad()
    Output = Net(Input)
    loss = criterion(Output, Target)
    loss.backward()
    optimizer.step()

print("Gradient of the Function is : " + str(grad))
print("Constant Value of the Function is : " + str(const))
print("Learned Gradient and Constant : " + list(Net.parameters()))


title: Cannot export PyTorch model to ONNX

body:
I am trying to convert a pre-trained torch model to ONNX, but recive the following error:
RuntimeError: step!=1 is currently not supported

I'm trying this on a pre-trained colorization model: https://github.com/richzhang/colorization
Here is the code I ran in Google Colab:
!git clone https://github.com/richzhang/colorization.git
cd colorization/
import colorizers
model = colorizer_siggraph17 = colorizers.siggraph17(pretrained=True).eval()
input_names = [ "input" ]
output_names = [ "output" ]
dummy_input = torch.randn(1, 1, 256, 256, device='cpu')
torch.onnx.export(model, dummy_input, "test_converted_model.onnx", verbose=True,
                  input_names=input_names, output_names=output_names)

I appreciate any help :)
UPDATE 1: @Proko suggestion solved the ONNX export issue. Now I have a new possibly related problem when I try to convert the ONNX to TensorRT. I get the following error:
[TensorRT] ERROR: Network must have at least one output

Here is the code I used:
import torch
import pycuda.driver as cuda
import pycuda.autoinit
import tensorrt as trt
import onnx

TRT_LOGGER = trt.Logger()

def build_engine(onnx_file_path):
    # initialize TensorRT engine and parse ONNX model
    builder = trt.Builder(TRT_LOGGER)
    builder.max_workspace_size = 1 << 25
    builder.max_batch_size = 1
    if builder.platform_has_fast_fp16:
        builder.fp16_mode = True

    network = builder.create_network()
    parser = trt.OnnxParser(network, TRT_LOGGER)

    # parse ONNX
    with open(onnx_file_path, 'rb') as model:
        print('Beginning ONNX file parsing')
        parser.parse(model.read())
    print('Completed parsing of ONNX file')

    # generate TensorRT engine optimized for the target platform
    print('Building an engine...')
    engine = builder.build_cuda_engine(network)
    context = engine.create_execution_context()
    print("Completed creating Engine")

    return engine, context

ONNX_FILE_PATH = 'siggraph17.onnx' # Exported using the code above
engine,_ = build_engine(ONNX_FILE_PATH)

I tried to force the build_engine function to use the output of the network by:
network.mark_output(network.get_layer(network.num_layers-1).get_output(0))

but it did not work.
I appropriate any help!


Why pytorch has two kinds of Non-linear activations?

Non-liner activations (weighted sum, nonlinearity):
https://pytorch.org/docs/stable/nn.html#non-linear-activations-weighted-sum-nonlinearity

Non-linear activations (other): https://pytorch.org/docs/stable/nn.html#non-linear-activations-other


title: How to use model generated with pytorch transfer learning module (.pth) in opencv's neural network module (caffe)?

body:
I have used transfer learning of pytorch to trained a model. It has an extension of .pth. I want to use it to recognize object in video. I have always been using opencv in video processing so I want to use it this time too. However, the dnn module from opencv does not accept model with .pth extension. Can I generate model with pytorch that can be accepted by opencv? Or can I use model with .pth in opencv?

Here's how I saved the trained model

torch.save(the_model.state_dict(), PATH)

I have read that post How should I save the model of PyTorch if I want it loadable by OpenCV dnn module but it is not helpful


title: PyTorch tensor indexing or conditional selection?

body:
for c in range(self.n_class):
        target[c][label == c] = 1

self.n_class is 32. and target is 32 x 1024 x 2048 tensor.

I know that target[c] select the each one of 1 x 1024 x 2048.
But I don't understand [label == c].

Because by thumb of rule, an integer should go in the square [].

Could someone explain what second square does and how it makes sense?


title: Importing pytorch in Spyder crashes kernel after installing matplotlib

body:
I created an environment in Anaconda3 and installed pytorch and spyder on a Linux machine. Here are the specifications:

spyder                    3.3.1
ipython                   7.0.1
python                    3.7.0
pytorch                   0.4.1
torchvision               0.2.1

When I open spyder and import torch, it works. Afterwards I installed matplotlib 3.0.1. Restarting spyder and importing again pytorch results in a message on the ipython window in spyder:

An error ocurred while starting the kernel
terminate called after throwing an instance of 'std::runtime_error'
what(): expected ) but found 'ident' here:
aten::_addmv(Tensor self, Tensor mat, Tensor vec, *, Scalar beta=1, Scalar alpha=1) ‑> Tensor
~~~~~~ <‑‑‑ HERE

On the bash terminal, I get the message:

js: Not allowed to load local resource:
file:///home/user/anaconda3/envs/myenv/lib/python3.7/site-packages/spyder/utils/help/static/css/default.css

I have been using all these packages in another environment for months (so they are in an older version), so it must be something with the new versions.
If I run ipython or python on the terminal, importing works, so I am concluding it has something to do with spyder.

The 'solution' is obvious: install older versions of the packages, but is there any other more sustainable solution?


title: Pytorch Convolutional Autoencoder output is blurry. How to improve it?

body:
I created Convolutional Autoencoder using Pytorch and I'm trying to improve it.

For the encoding layer I use first 4 layers of pre-trained ResNet 18 model from torchvision.models.resnet.

I have mid-layer with just one Convolutional layer with input and output channel sizes of 512. For the decoding layer I use Convolutional layers following with BatchNorm and ReLU activation function.

The decoding layer reduces the channel each layer: 512 -> 256 -> 128 -> 64 -> 32 -> 16 -> 3 and increases the resolution of the image with interpolation to match the dimension of the corresponding layer in the encoding part. For the last layer I use sigmoid instead of ReLu.

All Convolutional layers are:

self.up = nn.Sequential(
                nn.Conv2d(input_channels, output_channels,
                    kernel_size=5, stride=1,
                    padding=2, bias=False),
                nn.BatchNorm2d(output_channels),
                nn.ReLU()
            )

The input images are scaled to [0, 1] range and have shapes 224x224x3. Sample outputs are (First is from training set, the second from the test set):

First image

First image output

Second image

Second image output

Any ideas why output is blurry? The provided model has been trained around 160 epochs with ~16000 images using Adam optimizer with lr=0.00005.  I'm thinking about adding one more Convolutional layer in self.up given above. This will increase complexity of the model, but I'm not sure if it is the right way to improve the model.


title: PyTorch: Is retain_graph=True necessary in alternating optimization?

body:
I'm trying to optimize two models in an alternating fashion using PyTorch. The first is a neural network that is changing the representation of my data (ie a map f(x) on my input data x, parameterized by some weights W). The second is a Gaussian mixture model that is operating on the f(x) points, ie in the neural network space (rather than clustering points in the input space. I am optimizing the GMM using expectation maximization, so the parameter updates are analytically derived, rather than using gradient descent.

I have two loss functions here: the first is a function of the distances ||f(x) - f(y)||, and the second is the loss function of the Gaussian mixture model (ie how 'clustered' everything looks in the NN representation space). What I want to do is take a step in the NN optimization using both of the above loss functions (since it depends on both), and then do an expectation-maximization step for the GMM. The code looks like this (I have removed a lot since there is a ton of code):

data, labels = load_dataset()
net = NeuralNetwork()
net_optim = torch.optim.Adam(net.parameters(), lr=0.05, weight_decay=1)

# initialize weights, means, and covariances for the Gaussian clusters
concentrations, means, covariances, precisions = initialization(net.forward_one(data))

for i in range(1000):
    net_optim.zero_grad()
    pairs, pair_labels = pairGenerator(data, labels) # samples some pairs of datapoints
    outputs = net(pairs[:, 0, :], pairs[:, 1, :]) # computes pairwise distances
    net_loss = NeuralNetworkLoss(outputs, pair_labels) # loss function based on pairwise dist.

    embedding = net.forward_one(data) # embeds all data in the NN space

    log_prob, log_likelihoods = expectation_step(embedding, means, precisions, concentrations)
    concentrations, means, covariances, precisions = maximization_step(embedding, log_likelihoods)

    gmm_loss = GMMLoss(log_likelihoods, log_prob, precisions, concentrations)

    net_loss.backward(retain_graph=True)
    gmm_loss.backward(retain_graph=True)
    net_optim.step()

Essentially, this is what is happening:

Sample some pairs of points from the dataset
Push pairs of points through the NN and compute network loss based on those outputs
Embed all datapoints using the NN and perform a clustering EM step in that embedding space
Compute variational loss (ELBO) based on clustering parameters
Update neural network parameters using both the variational loss and the network loss

However, to perform (5), I am required to add the flag retain_graph=True, otherwise I get the error:

RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

It seems like having two loss functions means that I need to retain the computational graph?

I am not sure how to work around this, as with retain_graph=True, around iteration 400, each iteration is taking ~30 minutes to complete. Does anyone know how I might fix this? I apologize in advance – I am still very new to automatic differentiation.


title: FastAI PyTorch Train_loss and valid_loss look very good, but the model recognize nothing

body:
Update 1

I’m thinking that it might be the mistake in my detector code.
So, here is my code for using the trained learner/model to predict images.

import requests
import cv2

bytes = b''
stream = requests.get(url, stream=True)
bytes = bytes + stream.raw.read(1024) # I have my mobile video streaming to this url. the resolution for the video streaming is: 2048 x 1080
a = bytes.find(b'\xff\xd8')
b = bytes.find(b'\xff\xd9')
if a != -1 and b != -1:
      jpg = bytes[a:b+2]
      bytes = bytes[b+2:]
      img = cv2.imdecode(np.fromstring(jpg, dtype=np.uint8), cv2.IMREAD_COLOR)
      processedImg = Image(pil2tensor(img, np.float32).div_(255))
      predict = learn.predict(processedImg)
      self.objectClass = predict[0].obj

and I read the document of imdecode() method, it returns image in B G R order.

Could it because of different channel data used when in training and detecting?

Or

Could it because that I trained with image size 299 x 450, but when detecting the input image size from the video streaming is 2048 x 1080 without resizing it?

new to FastAi, ML and Python. I trained my “Birds Or Not-Birds” model. The train_loss, valid_loss and error_rate were improving. If I only trained 3 epochs, then the model worked(meaning it can recognize whether there are birds or no birds in images), then I increased to 30 epochs, all metrics look very good, but the model does not recognize things anymore, whatever images I input, the model always return Not-Birds.

here is the training output:

Here are the plots of learn.recorder

Here is my code:

from fastai.vision import *
from fastai.metrics import error_rate
from fastai.callbacks import EarlyStoppingCallback,SaveModelCallback
from datetime import datetime as dt
from functools import partial

path_img = '/minidata'
train_folder = 'train'
valid_folder = 'validation'

tunedTransform = partial(get_transforms, max_zoom=1.5)

data = ImageDataBunch.from_folder(path=path_img, train=train_folder, valid=valid_folder, ds_tfms=tunedTransform(),
                                  size=(299, 450), bs=40, classes=['birds', 'others'],
                                  resize_method=ResizeMethod.SQUISH)
data = data.normalize(imagenet_stats)

learn = cnn_learner(data, models.resnet50, metrics=error_rate)
learn.fit_one_cycle(30, max_lr=slice(5e-5,5e-4))

learn.recorder.plot_lr()
learn.recorder.plot()
learn.recorder.plot_losses()

Here is my dataset folder structure:

minidata

train

birds (7500 images)
others (around 7300 images)

validation

birds (1008 images)
others (around 872 images)


title: How to apply the histogram function in pytorch to a specific axis?

body:
I would like to use the torch.histc function to different samples in my training batch.
Here is an example:

>>> tt2 = torch.from_numpy(np.array([[-0.2, 1, 0.21], [-0.1, 0.32, 0.2]]))
>>> tt3 = torch.from_numpy(np.array([[-0.8, 0.6, 0.1], [-0.6, 0.5, 0.4]]))
>>> t = torch.cat((tt2, tt3), 1)
>>> t
tensor([[-0.2000,  1.0000,  0.2100, -0.8000,  0.6000,  0.1000],
        [-0.1000,  0.3200,  0.2000, -0.6000,  0.5000,  0.4000]],
       dtype=torch.float64)
>>> torch.histc(t, bins=1, min=0, max=5)
tensor([8.], dtype=torch.float64)

However, I don't want to apply the histogram function for all the values in t, I rather expect something like this:

>>> torch.histc(torch.tensor([[-0.2000,  1.0000,  0.2100, -0.8000,  0.6000,  0.1000]]), bins=1, min=0, max=5)
tensor([4.])
>>> torch.histc(torch.tensor([[-0.1000,  0.3200,  0.2000, -0.6000,  0.5000,  0.4000]]), bins=1, min=0, max=5)
tensor([4.])
>>>

And, finally, I want to aggregate all the histograms in the same tensor: tensor([[4.], [4.]]).
Thanks in advance


title: How to set hydra's parameter HYDRA_FULL_ERROR?

body:
When I use hydra in a python pytorch project, the operation result prompt “Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.”
But i don't konw how to set it.


title: Fit a Gaussian curve with a neural network using Pytorch

body:
Suppose the following model :

import torch.nn as nn

class PGN(nn.Module):
    def __init__(self, input_size):
        super(PGN, self).__init__()

        self.linear = nn.Sequential(
            nn.Linear(in_features=input_size, out_features=128),
            nn.ReLU(),
            nn.Linear(in_features=128, out_features=1)
        )

    def forward(self, x):
        return self.linear(x)

I figure I have to modify the model to fit a 2-dimensional curve.

Is there a way to fit a Gaussian curve with mu=0 and sigma=0 using Pytorch? If so, can you show me?


title: When will the computation graph be freed if I only do forward for some samples?

body:
I have a use case where I do forward for each sample in a batch and only accumulate loss for some of the samples based on some condition on the model output of the sample. Here is an illustrating code,

for batch_idx, (data, target) in enumerate(train_loader):
    optimizer.zero_grad()
    total_loss = 0

    loss_count_local = 0
    for i in range(len(target)):
        im = Variable(data[i].unsqueeze(0).cuda())
        y = Variable(torch.FloatTensor([target[i]]).cuda())

        out = model(im)

        # if out satisfy some condtion, we will calculate loss
        # for this sample, else proceed to next sample
        if some_condition(out):
            loss = criterion(out, y)
        else:
            continue

        total_loss += loss
        loss_count_local += 1

        if loss_count_local == 32 or i == (len(target)-1):
            total_loss /= loss_count_local
            total_loss.backward()
            total_loss = 0
            loss_count_local = 0

    optimizer.step()

My question is, as I do forward for all samples but only do backward for some of the samples. When will the graph for those samples which do not contribute to the loss be freed? Will these graphs be freed only after the for loop has ended or immediately after I do forward for the next sample? I am a little confused here.

Also for those samples that do contribute to the total_loss, their graph will be freed immediately after we do total_loss.backward(). Is that right?


title: Understanding Gradient in Pytorch

body:
I have some Pytorch code which demonstrates the gradient calculation within Pytorch, but I am thoroughly confused what got calculated and how it is used. This post here demonstrates the usage of it, but it does not make sense to me in terms of the back propagation algorithm. Looking at the gradient of in1 and in2 in the example below, I realized the gradient of in1 and in2 is the derivative of the loss function but my understanding is that the update needs to also account for the actual loss value as well? Where is the loss value getting used? Am I missing something here?

in1 = torch.randn(2,2,requires_grad=True)
in2 = torch.randn(2,2,requires_grad=True)
target = torch.randn(2,2)
l1 = torch.nn.L1Loss()
l2 = torch.nn.MSELoss()
out1 = l1(in1,target)
out2 = l2(in2,target)
out1.backward()
out2.backward()
in1.grad
in2.grad