title: Several questions when we install pytorch with pip or conda?

body:
When I install the tensorflow, they told me to inspect the Nvidia drive and So I did.
I install the Nvidia's drive on Nividia wibset. And when I input 'nvidia-smi' in CMD, it says " CUDA Version：10.1 " So I thought the cuda was installed.but after I install tensorflow-gpu, it turns out that I did't have the CUDA. So I wonder if the Nividia driver includs the "CUDA" and why the 'nvidia-smi' told me "CUDA Version：10.1" but I did't have one.And How do I confirm that CUDA is installed.
Then I try to install the pytorch on the Pytorch's website. The course told me the best way to install is using miniconda, because miniconda can install the cuda and cudnn automatically. but I'm used to using the pip,and I wonder if the pip can install the CUDA and cuDnn automatically ,just like miniconda？ Then I start to download with pip,but I find it is so slowly. it will take me 20h.I used to replace the source of the PIP to install the numpy and so on, but I can't change THE COMMAND，So I have no idea about how to accelerate it.
When I install the pytorch with the pip command above, it contains the CUDA unexpectly!  I don't understand why!! But when I use 'nvcc -V' in CMD, It doesn't work!
Finnaly , if I install the pytorch-gpu  with conda or pip , when I install tensorflow, should I install "CUDA" and "cuDnn" again?
I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this.


title: custom activation function in PyTorch - fix prediction

body:
I read this post about customa ctivation function, but still I can't implement my code. My  activation function can be expressed as a combination of existing PyTorch functions and it works fine function_pytorch(prediction, Q_sample). [Q_samples, is some variable I need it and it does't need gradient. ]
My activation function should receive the output of NN and , implement the function_pytorch and it's out put goes in the loss function. so:
class Activation_fun(nn.Module):

def __init__(self, prediction):
    super().__init__()

def forward(self, input, Q_samples):
    return function_pytorch(input, Q_samples)

in my NN I have
class NeuralNet(nn.Module):
  def __init__(self, input_size, hidden_size, output_size):
    super(NeuralNet, self).__init__()
    self.BN0 = nn.BatchNorm1d(input_size)
    self.l1 = nn.Linear(input_size, hidden_size)
    self.tan = nn.Tanh()
    self.BN = nn.BatchNorm1d(output_size)
    #custom activation
    self.l2 = Activation_fun()

def forward(self, x, q):
  out = self.BN0(x)
  out = self.l1(out)
  out = self.tan()
  out = self.BN9(out)
  out = self.l2(out, q)
  return out

 model = NeuralNet(input_size, hidden_size, output_size)

and in my training epochs:
outputs = model(inputs, q_samples)

The problem is: my prediction remains fix if I apply my customized activation function.
Is there any problem in my implementation?


title: Computing gradients of a depthwise convolution with respect to the layer's input in Pytorch?

body:
There's an utility function which does this in Tensorflow. However, I need it to be in Pytorch. I've successfully implemented a depthwise convolution with either "same" or "valid" padding in Pytorch with matches Tensorflow's implementation- e.g. depthwise_conv2d_torch(input, stride, kernel, padding) == tf.nn.depthwise_conv2d(input, stride, kernel, padding)
I'm not able to compute a matching gradient as follows:
def depthwise_conv2d_backprop_input_pt(depthwise_out, images):
    return T.autograd.grad(outputs = depthwise_out, inputs = images, grad_outputs = T.ones_like(depthwise_out))

I get wildly differing values for depthwise_conv2d_backprop_input_pt and tf.nn.depthwise_conv2d_backprop_input(input_sizes=images.shape, filter=filter, out_backprop=images, strides=strides, padding='SAME')
I've been stuck on this for a while and even tried using Tensorflow's reference cpp implementation to try to implement this but to no avail.


title: Pytorch: Node2Vec: TypeError: tuple indices must be integers or slices, not tuple

body:
I am trying to run Node2Vec from the torch_geometric.nn library. For reference, I am following this example.
While running the train() function I keep getting TypeError: tuple indices must be integers or slices, not tuple.
I am using torch version 1.6.0 with CUDA 10.1 and the latest versions of torch-scatter,torch-sparse,torch-cluster, torch-spline-conv and torch-geometric.
Here is the detailed error:
Part 1 of the Error
Part 2 of the Error
Thanks for any help.


title: How to completely modify the classifier of a pretrained network in PyTorch?

body:
I’m currently using a pretrained MobileNetV2 model for my audio classification.
I'm just starting to learn PyTorch and from reading other similar topics, I understand how to modify the first and the last layer:
from torchvision.models import mobilenet_v2
import torch
import torch.nn as nn
mobilenet_model = mobilenet_v2(pretrained=True)
mobilenet_model.features[0][0] = nn.Conv2d(1, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)  # Modify the first layer
mobilenet_model.classifier[1] = nn.Linear(1280, NUM_CLASSES, bias=True) # Modify the last layer

It works great, but now I want to experiment this model with different approach by applying different classifier (so changing the last layer), but I’m not sure how to change the last layer to use, let’s say, KNN instead of linear transformation.
Does anyone know how to do it? Any resource is also fine, but if you could explain it, I really appreciate!


title: Image Classification from Realtime Webcam video | PyTorch Model

body:
I am trying to implement a PyTorch image classification model in webcam on Jupyter Notebook. But getting the follow ing error:

error: OpenCV(4.5.1) /private/var/folders/nz/vv4_9tw56nv9k3tkvyszvwg80000gn/T/pip-req-build-39p1qqfs/opencv/modules/dnn/src/dnn.cpp:5449: error: (-2:Unspecified error) Cannot determine an origin framework of files: PyTorch, /Users/mdrahman/Collection/defect_classifier.pt in function 'readNet'

import cv2

#get the webcam video stream
webcam_video_stream = cv2.VideoCapture(0)

#create a while loop
while True:
    #get the current frame from video stream
    ret,current_frame = webcam_video_stream.read()
    #use the video current frame instead of image
    img_to_detect = current_frame
    img_height = img_to_detect.shape[0]
    img_width = img_to_detect.shape[1]
    # resize to match input size, convert to blob to pass into model
    resized_img_to_detect = cv2.resize(img_to_detect,(300,300))
    img_blob = cv2.dnn.blobFromImage(resized_img_to_detect,0.007843,(300,300),127.5)
    #recommended scale factor is 0.007843, width,height of blob is 300,300, mean of 255 is 127.5,

    # set of 21 class labels in alphabetical order (background + rest of 20 classes)
    class_labels = ["Perfect", "Defected"]

    # Loading pretrained model from prototext and caffemodel files
    # input preprocessed blob into model and pass through the model
    # obtain the detection predictions by the model using forward() method
    mobilenetssd = cv2.dnn.readNet('PyTorch','/Users/mdrahman/Collection/defect_classifier.pt')
    mobilenetssd.setInput(img_blob)
    obj_detections = mobilenetssd.forward()
    # returned obj_detections[0, 0, index, 1] , 1 => will have the prediction class index
    # 2 => will have confidence, 3 to 7 => will have the bounding box co-ordinates
    no_of_detections = obj_detections.shape[2]

    # loop over the detections
    for index in np.arange(0, no_of_detections):
        prediction_confidence = obj_detections[0, 0, index, 2]
        # take only predictions with confidence more than 20%
        if prediction_confidence > 0.20:

            #get the predicted label
            predicted_class_index = int(obj_detections[0, 0, index, 1])
            predicted_class_label = class_labels[predicted_class_index]

            #obtain the bounding box co-oridnates for actual image from resized image size
            bounding_box = obj_detections[0, 0, index, 3:7] * np.array([img_width, img_height, img_width, img_height])
            (start_x_pt, start_y_pt, end_x_pt, end_y_pt) = bounding_box.astype("int")

            # print the prediction in console
            predicted_class_label = "{}: {:.2f}%".format(class_labels[predicted_class_index], prediction_confidence * 100)
            print("predicted object {}: {}".format(index+1, predicted_class_label))

            # draw rectangle and text in the image
            cv2.rectangle(img_to_detect, (start_x_pt, start_y_pt), (end_x_pt, end_y_pt), (0,255,0), 2)
            cv2.putText(img_to_detect, predicted_class_label, (start_x_pt, start_y_pt-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0,255,0), 1)

    cv2.imshow("Detection Output", img_to_detect)

    #terminate while loop if 'q' key is pressed
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

#releasing the stream and the camera
#close all opencv windows
webcam_video_stream.release()
cv2.destroyAllWindows()


title: Running deep-daze on nvidia jetson - "Could not find a version that satisfies the requirement torchvision>=0.8.2"

body:
I have an nvidia Jetson Nano (the 4gb version).  I am attempting to run this project on it: https://github.com/lucidrains/deep-daze
I am attempting to run the command pip install deep-daze.  However, I do not have pip so I am running pip3 install deep-daze.  When I run that I get
chris@chris-desktop:~$ pip3 install deep-daze
Collecting deep-daze
  Using cached https://files.pythonhosted.org/packages/f1/ed/b3f3d9d92f5a48932b3807f683642b28da75722ae93da2f9bdc6af5f1768/deep_daze-0.7.2-py3-none-any.whl
Collecting tqdm (from deep-daze)
  Downloading https://files.pythonhosted.org/packages/f8/3e/2730d0effc282960dbff3cf91599ad0d8f3faedc8e75720fdf224b31ab24/tqdm-4.59.0-py2.py3-none-any.whl (74kB)
    100% |████████████████████████████████| 81kB 2.4MB/s
Collecting torchvision>=0.8.2 (from deep-daze)
  Could not find a version that satisfies the requirement torchvision>=0.8.2 (from deep-daze) (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3)
No matching distribution found for torchvision>=0.8.2 (from deep-daze)

I am pretty unfamiliar with several of the moving parts here and not sure how to fix this.  I thought these version numbers may be useful in answering this question:
chris@chris-desktop:~$ python3 --version
Python 3.6.9
chris@chris-desktop:~$ pip3 --version
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)
chris@chris-desktop:~$ python2 --version
Python 2.7.17
chris@chris-desktop:~$ pip2 --version
bash: pip2: command not found
chris@chris-desktop:~$ pip --version
bash: pip: command not found
chris@chris-desktop:~$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.5 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.5 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic


title: convnet better classification than regression performance

body:
I have a data set of 5 classes each with roughly 100 256x256 images. The code is mildly altered from the basic pytorch tutorials. Using vgg16 I get an accuracy of 77% for the classification. Then I switch the architecture to do regression by the following line of code:

model_ft.classifier[6] = nn.Linear(4096, 1)

Doing so I get a binned accuracy of 56% by rounding classification scores. Additionally I switch my loss to MSE resulting in a loss of .2698 . Are there any ideas why the regression performs worse than the classification model?

In depth Code Snippet:

from __future__ import print_function, division

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
from torch.autograd import Variable
import torchvision

import numpy as np
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy
import argparse
import transfer_learning_depths
import vgg_r
plt.ion()   # interactive mode

###############################################################################
def labelToConts(target):
    """
    'continous' classes- normalizes classes 0,1,2,3,4 to the domain [0,1]
    """
    target = target.numpy().astype(float)
    for i in range(0,target.size):
        #target[i] = math.floor((target[i] - 2)/2) for neg classes?
        target[i] = ((target[i])/4)
        #print(target[i])
    target = torch.from_numpy(target)
    return target
###############################################################################
"""my accuracy metric"""
def my_correct(outputs,labels):
    outputs_4 = outputs * 4
    outputs_4 = torch.round(outputs_4)
    outputs_4 = outputs_4 / 4
    return torch.sum(outputs_4 == labels.data.float())

###############################################################################

def train_model_classify(model,optimizer, scheduler, args):
    since = time.time()
    criterion = nn.CrossEntropyLoss()
    best_model_wts = copy.deepcopy(model.state_dict()) #keep track of best model?
    best_acc = 0.0
    for epoch in range(args.epochs):
        print('Epoch {}/{}'.format(epoch, args.epochs - 1))
        print('-' * 10)
        # Each epoch has a training and validation phase
        for phase in ['train', 'val']: #iterates through train and eval for each epoch
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode
            running_loss = 0.0
            running_corrects = 0
            # Iterate over data.
            for inputs, labels in dataloaders[phase]: #interesting
                inputs = inputs.to(device) #actually putting data to cpu or gpu #?grab subset of data needed, as not whol 1.2 million images is kept on system in cache
                labels = labels.to(device)
                # zero the parameter gradients
                optimizer.zero_grad()
                # forward
                # track history if only in train
                with torch.set_grad_enabled(phase == 'train'):
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)
                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()
                # statistics
                running_loss += loss.item() * inputs.size(0) #loss * batch size
                running_corrects += torch.sum(preds == labels.data)

            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]
            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))
            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())
    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))
    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

##############################################################################
def train_model_regress(model, optimizer, scheduler, num_epochs):
    since = time.time()
    best_model_wts = copy.deepcopy(model.state_dict()) #keep track of best model?
    best_acc = 0.0
    criterion = nn.MSELoss(size_average=args.mse)

    for epoch in range(num_epochs):
        print('Epoch {}/{}'.format(epoch, num_epochs - 1))
        print('-' * 10)

        # Each epoch has a training and validation phase
        for phase in ['train', 'val']: #iterates through train and eval for each epoch
            if phase == 'train':
                scheduler.step()
                model.train()  # Set model to training mode
            else:
                model.eval()   # Set model to evaluate mode
            running_loss = 0.0
            running_corrects = 0
            # Iterate over data.
            for inputs, labels in dataloaders[phase]: #interesting
                #manipulating those silly labels
                inputs = inputs.to(device) #actually putting data to cpu or gpu #?grab subset of data needed, as not whol 1.2 million images is kept on system in cache
                labels = labelToConts(labels)
                labels = labels.to(device)
                labels = labels.float()
                labels = labels.view(-1,1)
                #setting up for foward pass
                optimizer.zero_grad()# zero the parameter gradients
                with torch.set_grad_enabled(phase == 'train'): # track history if only in train
                    outputs = model(inputs)
                    _, preds = torch.max(outputs, 1)
                    loss = criterion(outputs, labels)
                    # backward + optimize only if in training phase
                    if phase == 'train':
                        loss.backward()
                        optimizer.step()
                    #print out final epochs outputs and labels
                    if ((phase == 'val') and (epoch == (num_epochs - 1))):
                        for i in range(outputs.shape[0]):
                            print(outputs[i].item(),labels[i].item())
                            with open("tab_" + args.save_dir , "a") as text_file:
                                text_file.write('{},{}\n'.format(outputs[i].item(),labels[i].item()))
                running_loss += loss.item() * inputs.size(0) #loss * batch size
                running_corrects += my_correct(outputs,labels) #torch.sum(preds == labels.data.long())
            #relaying results
            epoch_loss = running_loss / dataset_sizes[phase]
            epoch_acc = running_corrects.double() / dataset_sizes[phase]
            print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                phase, epoch_loss, epoch_acc))
            with open(args.save_dir, "a") as text_file:
                text_file.write('{} , {:.4f} , {:.4f}'.format(
                phase, epoch_loss, epoch_acc))
            # deep copy the model
            if phase == 'val' and epoch_acc > best_acc:
                best_acc = epoch_acc
                best_model_wts = copy.deepcopy(model.state_dict())
    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))
    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

##############################################################################
"""arg parse for command prompt"""
parser = argparse.ArgumentParser(description='PyTorch ImageNet Training')
parser.add_argument('-j', '--workers', default=8, type=int, metavar='N',
                    help='number of data loading workers (default: 0)')
parser.add_argument('--epochs', default=10, type=int, metavar='N',
                    help='number of total epochs to run')
parser.add_argument('-b', '--batch-size', default=8, type=int,
                    metavar='N', help='mini-batch size (default: 128)')
parser.add_argument('--lrc', default=1e-3, type=float,
                    metavar='LR', help='initial learning rate')
parser.add_argument('--lrr', default=5e-5, type=float,
                    metavar='LR', help='initial learning rate')
parser.add_argument('--momentum', default=0.9, type=float, metavar='M',
                    help='momentum')
parser.add_argument('--step-size', default=7, type=int, dest='step_size',
                    help='step size for learning rate decay')
parser.add_argument('--gamma', default=0.1, type=float,  metavar='M',
                    help='lr decay per step size')
parser.add_argument('--data-dir', dest='data_dir',
                    help='The directory used to load dataset',
                    default='sample/256_dataset', type=str)
parser.add_argument('--save-dir', dest='save_dir',
                    help='The directory used to save the trained models',
                    default='save_temp_r.txt', type=str)
parser.add_argument('--archs', metavar='N',
                    help='a case loop for transfer learning architecture (1 through 5)',
                    default=1, type=int)
parser.add_argument('--angle-rotation', dest='angle_rotation',
                    help='randomly rotate stuff',
                    default=15, type=int)
parser.add_argument('--mse', default=False, action='store_true', help='Bool type')

##############################################################################
"""setting up datasets"""
# Data augmentation and normalization for synthetic training set images
args = parser.parse_args()
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomAffine((args.angle_rotation * -1, args.angle_rotation)),
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = args.data_dir
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir,  x), #both training and testing data set?
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=args.batch_size, #dataset => loader
                                             shuffle=True, num_workers=args.workers)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']} #abstract of multiple classes: size of each set
class_names = image_datasets['train'].classes #labels

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") #gpu or cpu
##############################################################################
"""main method"""
#sanity check
with open(args.save_dir, "w") as text_file: #text file for logging (smaller than saving model params)
    text_file.write(str(args))
print(str(args))
"""classify"""
model_ft = models.vgg16_bn(pretrained=True)
model_ft.classifier[6] = nn.Linear(4096, 5)

optimizer_ft = optim.SGD(model_ft.parameters(), lr=args.lrc, momentum=args.momentum) # Observe that all parameters are being optimized
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=args.step_size, gamma=args.gamma) # Decay LR by a factor of 0.1 every 7 epochs
model_ft = model_ft.to(device) #move modelto CPU or GPU
model_ft = train_model_classify(model_ft, optimizer_ft, exp_lr_scheduler,args)
"""regression"""
for param in model_ft.features.parameters():
    param.requires_grad = False
model_ft.classifier[6] = nn.Linear(4096, 1)

optimizer_ft = optim.SGD(model_ft.classifier.parameters(), lr=args.lrr, momentum=args.momentum)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=args.step_size, gamma=args.gamma)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu") #gpu or cpu
model_ft = model_ft.to(device) #move to cpu or gpu
#go time
model_ft = train_model_regress(model_ft, optimizer_ft, exp_lr_scheduler,
                       num_epochs=args.epochs)

Update 1:
This is the experiment rerun so results are slightly different than above.
classification results

Namespace(angle_rotation=15, archs=1, batch_size=8, data_dir='sample/256_dataset', epochs=10, gamma=0.1, lrc=0.001, lrr=5e-05, momentum=0.9, mse=False, save_dir='save_temp_r.txt', step_size=7, workers=8)
Epoch 0/9
----------
train Loss: 1.2778 Acc: 0.4402
val Loss: 1.1267 Acc: 0.5569
Epoch 1/9
----------
train Loss: 1.0335 Acc: 0.5915
val Loss: 0.7418 Acc: 0.6228
Epoch 2/9
----------
train Loss: 0.8455 Acc: 0.6672
val Loss: 0.6596 Acc: 0.7186
Epoch 3/9
----------
train Loss: 0.8310 Acc: 0.6732
val Loss: 0.9665 Acc: 0.6527
Epoch 4/9
----------
train Loss: 0.7381 Acc: 0.7110
val Loss: 0.5091 Acc: 0.8144
Epoch 5/9
----------
train Loss: 0.6798 Acc: 0.7322
val Loss: 1.7675 Acc: 0.4850
Epoch 6/9
----------
train Loss: 0.6773 Acc: 0.7489
val Loss: 0.4319 Acc: 0.8323
Epoch 7/9
----------
train Loss: 0.6199 Acc: 0.7610
val Loss: 0.4467 Acc: 0.8383
Epoch 8/9
----------
train Loss: 0.5240 Acc: 0.8094
val Loss: 0.4593 Acc: 0.8323
Epoch 9/9
----------
train Loss: 0.5431 Acc: 0.7761
val Loss: 0.5307 Acc: 0.7784
Training complete in 10m 53s
Best val Acc: 0.838323

Regression Results

Epoch 0/9
----------
train Loss: 1.1029 Acc: 0.2753
val Loss: 0.7975 Acc: 0.2754
Epoch 1/9
----------
train Loss: 0.7637 Acc: 0.3222
val Loss: 0.4878 Acc: 0.4611
Epoch 2/9
----------
train Loss: 0.7492 Acc: 0.3147
val Loss: 0.7000 Acc: 0.3293
Epoch 3/9
----------
train Loss: 0.6129 Acc: 0.3782
val Loss: 0.9696 Acc: 0.2156
Epoch 4/9
----------
train Loss: 0.5920 Acc: 0.3676
val Loss: 0.4152 Acc: 0.3892
Epoch 5/9
----------
train Loss: 0.6302 Acc: 0.3464
val Loss: 0.8849 Acc: 0.1916
Epoch 6/9
----------
train Loss: 0.5896 Acc: 0.3707
val Loss: 0.4919 Acc: 0.2874
Epoch 7/9
----------
train Loss: 0.5016 Acc: 0.3722
val Loss: 0.4235 Acc: 0.3174
Epoch 8/9
----------
train Loss: 0.4701 Acc: 0.3949
val Loss: 0.4893 Acc: 0.3413
Epoch 9/9
----------
val Loss: 0.5068 Acc: 0.3114
Training complete in 4m 43s
Best val Acc: 0.461078


title: Maintaining packages with different versions in one python environment

body: I need an older version of torch (0.5.0) for a certain library (allennlp==0.7.2). But I already have an existing torch version (1.0.0) as required by flair and other libraries. How do I cater to different requirements of the same package of multiple libraries in the same python environment without having to manually specify the version every time I import that package?


title: Time series classification problem in python

body: My dataset looks like this. How to feed this to a RNN model. Should I be extracting values for each hour for each user as a feature and feed it to the RNN or is there is a way to feed the data directly like the way it is now. Thanks


title: How to modify the tensor data dimension in pytorch, thanks

body:
I want to transform the tensor data to numpy data and save it through Opencv, But the opencv require the data dimension must like such style [1, something, something, something], but my tensor data is a blend one, it'e size like [30, something, something, something],how can I modify the data dimension in pytorch.

PS, is there any function in pytorch can save data as a binary picture? I use "save_image" command to save my tensor data to a picture with all figure is 1 or 0, but the picture show is still a gray style. Maybe there is any other ways to save tensor data as a binary picture, please tell me.

               def save_image_tensor2cv2(input_tensor, filename):
                   assert (len(input_tensor.shape) == 4) and input_tensor.shape[0] == 1)
                   input_tensor = input_tensor.clone().detach()
                   input_tensor = input_tensor.to(torch.device('cpu'))
                   input_tensor = input_tensor.squeeze()
                   input_tensor = input_tensor.mul_(255).add_(0.5).clamp_(0, 255).permute(1, 2, 0).type(torch.uint8).numpy()
                   cv2.imwrite(filename, input_tensor)


title: PyTorch Model can only recognize birds when birds are close to camera

body:
update 1

The easiest way to stop cropping images is: to pass in a tuple to size parameter.

So it looks like this:

size=(299,299)

Unfortunately, it does not solve my problem. So the question is still opening.

I just trained my birds model. It works fine when I was testing it with close pictures.

But when I moved the pictures further away my camera, the model was not able to detect birds.

My guess is in this line of code

data = ImageDataBunch.from_folder(path=path_img, bs=48, valid_pct= 0.2, ds_tfms=get_transforms(), size=299, classes=['birds', 'others'])

The

  get_transforms()

and

  size=299

the function and size parameters crop my training images to centralize the images, so that birds in the images appear to be closer to the camera.

How to fix it?


title: How to circumvent AWS package and ephemeral limits for large packages + large models

body:
We have a production scenario with users invoking expensive NLP functions running for short periods of time (say 30s). Because of the high load and intermittent usage, we're looking into Lambda function deployment. However - our packages are big.

I'm trying to fit AllenNLP in a lambda function, which in turn depends on pytorch, scipy, spacy and numpy and a few other libs.

What I've tried

Following recommendations made here and the example here, tests and additional files are removed. I also use a non-cuda version of Pytorch which gets its' size down. I can package an AllenNLP deployment down to about 512mb. Currently, this is still too big for AWS Lambda.

Possible fixes?

I'm wondering if anyone of has experience with one of the following potential pathways:

Cutting PyTorch out of AllenNLP. Without Pytorch, we're in reach of getting it to 250mb. We only need to load archived models in production, but that does seem to use some of the PyTorch infrastructure. Maybe there are alternatives?
Invoking PyTorch in (a fork of) AllenNLP as a second lambda function.
Using S3 to deliver some of the dependencies: SIMlinking some of the larger .so files and serving them from an S3 bucket might help. This does create an additional problem: the Semnatic Role Labelling we're using from AllenNLP also requires some language models of around 500mb, for which the ephemeral storage could be used - but maybe these can be streamed directly into RAM from S3?

Maybe i'm missing an easy solution. Any direction or experiences would be much appreciated!


title: Gradient calculation in A2C

body:
In A2C, the actor and critic algorithm, the weights are updated via equations:

delta = TD Error and

theta = theta + alpha*delta*[Grad(log(PI(a|s,theta)))] and

w = w + beta*delta*[Grad(V(s,w))]

So my question is, when using neural networks to implement this,

how are the gradients calculated and
am I correct that the weights are updated via the optimization fmethods in TensorFlow or PyTorch?

Thanks, Jon


title: Helix Convolution in Pytorch (Machine Learning)

body:
I currently investigate the development of a convolutional neural network involving up to 5 or 6 dimensional arrays efficiently.

I was aware that many of the tools used for convolutional neural networks do not really deal with ND convolutions, so I decided to try and write an implementation of Helix Convolution, whereby the convolution can be treated as a large, 1D convolution (see Reference 1. http://sepwww.stanford.edu/public/docs/sep95/jon1/paper_html/node2.html , Reference 2 https://sites.ualberta.ca/~mostafan/Files/Papers/md_convolution_TLE2009.pdf for more details of the concept).

I did this under the (possibly incorrect) assumption that a large, single dimensional convolution was likely to be easier on a GPU than a multidimensional one, as well as that the method is trivially scalable to N dimensions.

Particularly, a quote from Reference 2. states:

We have not found important gains in computational efficiency between N-D standard convolution versus using the
algorithm described in the text. We have, however, found that
writing codes for seismic data regularization with the described
trick leads to algorithms that can easily handle regularization
problems with any number of spatial dimensions (Naghizadeh
and Sacchi, 2009).

I have written an implementation of the function below, which compares to signal.fftconvolve. It is slower on the CPU compared to this function, but I would nonetheless like to see how it performs on the GPU in PyTorch as a forward convolutional layer.

Can someone kindly help me port this code to PyTorch so I can verify how it behaves?

"""
HELIX CONVOLUTION FUNCTION

Shrink:
CROPS THE SIZE OF THE CONVOLVED SIGNAL DOWN TO THE ORIGINAL SIZE OF THE ORIGINAL.

Pad:
PADS THE DIFFERENCE BETWEEN THE ORIGINAL SHAPE AND THE DESIRED, CONVOLVED SHAPE FOR KERNEL AND SIGNAL.

GetLength:
EXTRACTS THE LENGTH OF THE UNWOUND STRIP OF THE SIGNAL AND KERNEL THAT IS TO BE CONVOLVED.

FFTConvolve:
USES THE NUMPY FFT PACKAGE TO PERFORM FAST FOURIER CONVOLUTION ON THE SIGNALS

Convolve:
USES HELIX CONVOLUTION ON AN INPUT ARRAY AND KERNEL.

"""

import numpy as np
from numpy import *
from scipy import signal
import operator
import time

class HelixCPU:
    @classmethod
    def Shrink(cls,array, bounding):
       start = tuple(map(lambda a, da: (a-da)//2, array.shape, bounding))
       end = tuple(map(operator.add, start, bounding))
       slices = tuple(map(slice, start, end))
       return array[slices]

    @classmethod
    def Pad(cls,array, target_shape):
       diff = target_shape-array.shape
       padder=[(0,val) for val in diff]
       padded = np.pad(array, padder, 'constant')
       return padded

    @classmethod
    def GetLength(cls,array_shape, padded_shape):
        temp=1
        steps=np.zeros_like(array_shape)

        for i, entry in enumerate(padded_shape[::-1]):
            if(i==len(padded_shape)-1):
               steps[i]=1
            else:
               temp=entry*temp
               steps[i]=temp

         steps=np.roll(steps, 1)
         steps=steps[::-1]
         ones=np.ones_like(array_shape)
         ones[-1]=0
         out=np.multiply(steps,array_shape - ones)
         length = np.sum(out)
         return length

    @classmethod
    def FFTConvolve(cls, in1, in2, len1, len2):
        s1 = len1
        s2 = len2
        shape = s1 + s2 - 1
        fsize = 2 ** np.ceil(cp.log2(shape)).astype(int)
        fslice = slice(0, shape)
        conv = np.fft.ifft(np.fft.fft(in1, int(fsize)) * np.fft.fft(in2, int(fsize)))[fslice].copy()
        return conv

    @classmethod
    def Convolve(cls,array, kernel):
        m = array.shape
        n = kernel.shape
        mn = np.add(m, n)
        mn = mn-np.ones_like(mn)
        k_pad=cls.Pad(kernel, mn)
        a_pad=cls.Pad(array, mn)
        length_k = cls.GetLength(kernel.shape, k_pad.shape);
        length_a = cls.GetLength(array.shape, a_pad.shape);
        k_flat = k_pad.flatten()[0:length_k]
        a_flat = a_pad.flatten()[0:length_a]
        conv = cls.FFTConvolve(a_flat, k_flat)
        conv = np.resize(conv,mn)
        conv = cls.Shrink(conv, m)
        return conv

def main():

    array=np.random.rand(25,25,41,51)
    kernel=np.random.rand(10, 10, 10, 10)

    start2 =time.process_time()
    test2 = HelixCPU.Convolve(array, kernel)
    end2=time.process_time()

    start1= time.process_time()
    test1 = signal.fftconvolve(array, kernel, "same")
    end1= time.process_time()

    print ("")
    print ("========================")
    print ("SOME LARGE CONVOLVED RANDOM ARRAYS. ")
    print ("========================")
    print("")
    print ("Random Calorimeter Image of Size {0} Created".format(array.shape))
    print ("Random Kernel of Size {0} Created".format(kernel.shape))
    print("")
    print ("Value\tOriginal\tHelix")
    print ("Time Taken [s]\t{0}\t{1}\t{2}".format( (end1-start1), (end2-start2), (end2-start2)/(end1-start1) ))
    print ("Maximum Value\t{:03.2f}\t{:13.2f}".format( np.max(test1), np.max(test2) ))
    print ("Matrix Norm \t{:03.2f}\t{:13.2f}".format( np.linalg.norm(test1), np.linalg.norm(test2) ))
    print ("All Close?\t{0}".format(np.allclose(test1, test2)))


title: PyTorch gradient calculation gives unexpected results

body:
The following code is intended to explain how the PyTorch gradient calculation works, and it should, IMO return the weights matrix, but it doesn't:

# the code calculates T x W + B ---> K1
# compute the mean of K1 --> km
# compute the gradient of km relative to T
#
import torch
torch.manual_seed(0)
t = torch.rand(2,3)
w = torch.rand(3,4)
b = torch.rand(1,4)
#
k1 = torch.mm(t, w) + b
#
#torch.set_grad_enabled(True)
print('k1_grad_fn ',k1.grad_fn)
#
print('t grad ',t.grad)
#
#
km = k1.mean()
km.requires_grad_(True)
print('k1 mean=',km)
km.backward()
print('t grad ',t.grad)
print('k1 grad ',k1.grad_fn)

The results are:

t grad  None
k1 mean= tensor(1.0396, requires_grad=True)
t grad  None
k1 grad  None```


title: How to calculate the histogram of an image in pytorch without for loops

body:
I'm trying to calculate a histogram of an image with PyTorch without using for loops (or the built in histc function), but just by leveraging the GPU. It should be simple code as this is for students.

Now, since there is a histc function, I assume I could just use that source code. However, I can't seem to find it at https://github.com/pytorch/pytorch


title: How to add cuda drivers to gcp Ubuntu vm?

body:
When I run a pytorch model on a google virtual machine using:

model.cuda()

I get this error:

AssertionError:
Found no NVIDIA driver on your system. Please check that you
have an NVIDIA GPU and installed a driver from
http://www.nvidia.com/Download/index.aspx


title: Reformer local and LSH attention in HuggingFace implementation

body:
The recent implementation of the Reformer in HuggingFace has both what they call LSH Self Attention and Local Self Attention, but the difference is not very clear to me after reading the documentation. Both use bucketing to avoid the quadratic memory requirement of vanilla transformers, but it is not clear how they differ.

Is it the case that local self attention only allows queries to attend to keys sequentially near them (i.e., inside a given window in the sentence), as opposed to the proper LSH hashing that LSH self attention does? Or is it something else?


title: Cannot iterate through a DataLoader object

body:
I have the ucf 101 dataset loaded as follows:
dataComplete = torchvision.datasets.UCF101('directory', frames_per_clip=16, step_between_clips=1, frame_rate=None, fold=1, train=True, transform=transformF, _precomputed_metadata=None, num_workers=1, _video_width=0, _video_height=0, _video_min_dimension=0, _audio_samples=0)
data = torch.utils.data.DataLoader(dataComplete, batch_size=3, shuffle=True)
#where transformF = transforms.ToTensor()

and I am trying to access the tensors stored in the dataLoader object, but everytime I try to execute:
for n_batch in data:
  #training...

I get the error
TypeError: pic should be PIL Image or ndarray. Got <class 'torch.Tensor'>

I tried using dataComplete instead and using enumerate(), range() ecc. but it shows the same error


title: Add tensor to a dataframe

body:
I have a tensor with the following characteristics:
<class 'torch.Tensor'>

torch.Size([600, 1])

I would like it to be a column in a data frame.
When I add the tensor to a df as is, I get the following for one of the rows.
tensor([-0.1186])

How do I get rid of the tensor([])? So that each row is just a number. For this row, I would just want -0.1186.


title: Pytorch + Docker: Multiple workers

body: I have a machine learning project using pytorch that is trained remotely. However the docker container is not started using the flags (--ipc=host nor --shm-size). The pytorch documentation (https://github.com/pytorch/pytorch#docker-image) says this is required to run multiple workers in a docker container. Setting the number of workers to 0 is not an adequate solution, because then training takes takes 10 times longer. Is there any way to get pytorch dataloaders working in a docker container? We have the ability to define the Dockerfile by our selfs.


title: Why is my loss function always returning nan?

body:
I'm trying to learn about ML and I tried to make a simple linear model, but when I run it, loss comes out as null:

So I tried to find whats the problem about. If I print first 10 y_pred, only about 17 of them have numbers, the rest of them are null. Maybe im doing something wrong, please help.
import torch
from torch.utils.data import TensorDataset, DataLoader
import torch.nn as nn
import numpy as np

#Input (temp, rainfall, humidity)
inputs = np.array([[73, 67, 43], [91, 88, 64], [87, 134, 58], [102, 43, 37], [69, 96, 70], [73, 67, 43], [91, 88, 64], [87, 134, 58], [102, 43, 37], [69, 96, 70], [73, 67, 43], [91, 88, 64], [87, 134, 58], [102, 43, 37], [69, 96, 70]], dtype='float32')

#Target (apples, oranges)
targets = np.array([[56, 70], [81, 101], [119, 133], [22, 37], [103, 119], [56, 70], [81, 101], [119, 133], [22, 37], [103, 119], [56, 70], [81, 101], [119, 133], [22, 37], [103, 119]], dtype='float32')

inputs = torch.from_numpy(inputs)
targets = torch.from_numpy(targets)

#Define datasets
train_ds = TensorDataset(inputs, targets)
train_ds[0:3]

#Hyperparameters
batch_size = 5
num_epochs = 100
learning_rate = 0.01

train_dl = DataLoader(dataset=train_ds, batch_size=batch_size, shuffle=True)

model = nn.Linear(3,2) #inputs(temp, rainfall, humidity) , targets(apples, oranges)

loss_f = nn.MSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

for epoch in range(num_epochs):
    for xb, yb in train_dl:
        y_pred = model(xb)
        loss = loss_f(y_pred, yb)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

    if(epoch+1) % 10 == 0:
        print(f'epoch = {epoch+1}/{num_epochs}, loss = {loss.item():.4f}')

print(f'Final loss = {loss.item():.4f}')

edit:
y_pred.shape = torch.Size([5, 2]), yb.shape = torch.Size([5, 2])


title: How To Use The First Layers Of Model In PyTorch

body:
I have uploaded a certain model
from efficientnet_pytorch import EfficientNet
model = EfficientNet.from_pretrained(model)

And I can see the model:
print(model.state_dict())

The model contains quite a few layers, and I want to take only the first 50. Please tell me how I can do this.


title: Getting error while executing program using torch.autograd "Legacy autograd function with non-static forward method is deprecated"

body:
I am using pytorch 1.18.5 (pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html) and running my code using google colab.
I am trying to run a code and it gives me error:
/usr/local/lib/python3.6/dist-packages/torch/autograd/function.py in __call__(self=<model_classes.GLinearApprox object>, *args=(tensor([[1.6634, 1.5887, 1.5473,  ..., 2.0960, 2...2]],
       device='cuda:0', dtype=torch.float64), tensor([[1.6634, 1.5887, 1.5473,  ..., 2.0960, 2...2]],
       device='cuda:0', dtype=torch.float64), tensor([[0.0194, 0.0243, 0.0261,  ..., 0.0812, 0...8]],
       device='cuda:0', dtype=torch.float64)), **kwargs={})
    147     def __call__(self, *args, **kwargs):
    148         raise RuntimeError(
--> 149             "Legacy autograd function with non-static forward method is deprecated. "
    150             "Please use new-style autograd function with static forward method. "
    151             "(Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)")'''

where code in model_classes.GLinearApprox is
class GLinearApprox(Function):
    """ Linear (gradient) approximation of G function at z"""
    def __init__(self, gamma_under, gamma_over):
        self.gamma_under = gamma_under
        self.gamma_over = gamma_over

    def forward(self, z, mu, sig):
        self.save_for_backward(z, mu, sig)
        p = st.norm(mu.cpu().numpy(),sig.cpu().numpy())
        return torch.DoubleTensor((self.gamma_under + self.gamma_over) * p.cdf(
            z.cpu().numpy()) - self.gamma_under).cuda()

    def backward(self, grad_output):
        z, mu, sig = self.saved_tensors
        p = st.norm(mu.cpu().numpy(),sig.cpu().numpy())
        pz = torch.DoubleTensor(p.pdf(z.cpu().numpy())).cuda()

        dz = (self.gamma_under + self.gamma_over) * pz
        dmu = -dz
        dsig = -(self.gamma_under + self.gamma_over)*(z-mu) / sig * pz
        return grad_output * dz, grad_output * dmu, grad_output * dsig '''


title: NaN value in WordEmbedding

body:
In my model I have a simple word embedding done like this:
self.word_embedding = nn.Embedding(
            num_embeddings=len(self.word_vocab),       # 381
            embedding_dim=self.word_emb_size,          # 100
            padding_idx=self.word_vocab.pad,           # 0
        )

after the first batch iteration in the first epoch, its values begin to be NaN.
In the forward() it is used like this:
we = self.word_embedding(w)

where w is a tensor of shape ([50,61]) and its values are never NaN.
How can I do to solve this problem that I propagate the NaN values around the whole model?
Thank you!
EDIT:
Trying to retrace all the steps of the training I noticed that after the loss.backward() and before the optimizer.step() the minimum and maximum values for the word embedding gradient are already Nan:
for epoch in range(args.epochs):
    for batch in chunker(train, args.batch_size):
        # clear gradients
        optimizer.zero_grad()
        # calculate loss
        loss = pat.train_conll(batch)
        # back propagate
        loss.backward()

        for name, params in pat.named_parameters():
            print("-->name:", name, "-->max_grad:", params.grad.max(), "-->min_grad:", params.grad.min())

        # optimize parameters
        optimizer.step()

output:
-->name: word_embedding.weight -->max_grad: tensor(nan, device='cuda:0') -->min_grad: tensor(nan, device='cuda:0')

EDIT2:
thanks to the helpful comment of @Nerveless_child.
The piece of code that was causing the problem is the following:
class Norm(nn.Module):
    def __init__(self, d_model, eps=1e-6):
        super().__init__()

        self.size = d_model
        # create two learnable parameters to calibrate normalisation
        self.alpha = nn.Parameter(torch.ones(self.size))
        self.bias = nn.Parameter(torch.zeros(self.size))
        self.eps = eps

    def forward(self, x):
        norm = self.alpha * (x - x.mean(dim=-1, keepdim=True)) \
               / (x.std(dim=-1, keepdim=True) + self.eps) + self.bias
        return norm

used after a self-attention BiLSTM.


title: pytorch loss accumulated when using mini-batch

body:
I am new to pytorch. May I ask what is the difference between adding 'loss.item()' or not? The following 2 parts of code:
for epoch in range(epochs):
    trainingloss =0
    for i in range(0,X.size()[1], batch_size):
        indices = permutation[i:i+batch_size]
        F = model.forward(X[n])
        optimizer.zero_grad()
        criterion = loss(X,n)
        criterion.backward()
        optimizer.step()
        trainingloss += criterion.item()

and this
for epoch in range(epochs):
    for i in range(0,X.size()[1], batch_size):
        indices = permutation[i:i+batch_size]
        F = model.forward(X[n])
        optimizer.zero_grad()
        criterion = loss(X,n)
        criterion.backward()
        optimizer.step()

If anyone has any idea please help. Thank you very much.


title: PyTorch tensors: new tensor based on old tensor and indices

body:
I'm new to tensors and having a headache over this problem:
I have an index tensor of size k with values between 0 and k-1:
tensor([0,1,2,0])

and the following matrix:
tensor([[[0, 9],
     [1, 8],
     [2, 3],
     [4, 9]]])

I want to create a new tensor which contains the rows specified in index, in that order. So I want:
tensor([[[0, 9],
     [1, 8],
     [2, 3],
     [0, 9]]])

Outside tensors I'd do this operation more or less like this:
new_matrix = [matrix[i] for i in index]

How do I do something similar in PyTorch on tensors?


title: PyTorch RuntimeError: Trying to backward through the graph a second time, but the saved intermediate results have already been freed

body:
I'm a college student just getting into Deep Learning, trying to create my first backward propagated model.
However, I keep getting the "Trying to backward through the graph a second time, but the saved intermediate results have already been freed." runtime error.
I have seen many others asking the same question here, but their sample code is often too advanced for me, and I don't understand the answers.
I have already tried adding convtraining.zero_grad() and loss.sum().backward(retain_graph = True). Neither seem to work.
My own code is the following:
# Import torch.
import torch
import torch.nn as nn

# Define a sample image.
image = torch.tensor([[1, 1, 0, 0, 0],
                     [0, 1, 1, 0, 0],
                     [0, 0, 1, 1, 0],
                     [0, 0, 0, 1, 1],
                     [1, 0, 0, 0, 1]], dtype = torch.float).reshape(1,1,5,5)

# Define a sample kernel.
kernel = torch.tensor([[0,-1, 0],
                       [-1, 1, -1],
                       [0,-1, 0]], dtype = torch.float).reshape(1,1,3,3)

# Define a convolution layer with the TRUE kernel weights assigned.
convolution = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(3, 3), bias=False)
convolution.weight = nn.Parameter(kernel)
# Define another convolution without kernel weights to PREDICT them.
convtraining = nn.Conv2d(in_channels=1, out_channels=1, kernel_size=(3, 3), bias=False)

# Run the first layer.
output_true = convolution(image)
# Reshape its output to make it suitable for comparison.
output_true = output_true.reshape((1, 1, 3, 3))

# Run the second model 5 times.
for i in range(5):
    output_prediction = convtraining(image)
    # Calculate the loss by squaring the error.
    loss = (output_prediction - output_true) ** 2
    convtraining.zero_grad()
    # Backward propagation.
    loss.sum().backward()
    # Adjust the kernel weights.
    convtraining.weight.data[:] -= 3e-2 * convtraining.weight.grad
    print(loss)

The strange thing is it worked before and I don't know what changed. Does anyone know what might be going wrong?


title: Pytorch: Combining Automatic and Manual Methods

body:
I am testing a two-step architecture that is composed of a conventional first section that can be implemented with any standard deep learning architecture and a second section that must be coded manually outside the declaration of the Pytorch graph (while still utilizing numpy-like torch functions).

My problem can be simplified to coding a feed-forward neural network with two hidden layers, where the first is implemented within the Pytorch graph and the second is implemented manually outside the Pytorch graph.

Architecture:

Input
-> Linear(28 * 28, 120) w/ Pytorch
-> ReLU w/ Pytorch
-> Linear(120, 84) w/ Pytorch
-> ReLU w/ Pytorch
-> Linear(84, 10) w/o Pytorch
-> Output

Problem: My implementation below achieves a very low ~74%, while a standard fully Pytorch implementation achieves ~95%. What is causing this disparity?

I believe my problem lies in manually passing back the deltas, although the math looks right, so I am stuck in finding a solution to this.

Implementation of architecture and training on MNIST:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28 * 28, 120)
        self.fc2 = nn.Linear(120, 84)

    def forward(self, x):
        x = x.view(-1, 28 * 28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x

net = Net()

criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.01)

# Initialize weights just as Pytorch does by default:
m = torch.distributions.uniform.Uniform(torch.tensor([-np.sqrt(1.0/84)]),
                                        torch.tensor([np.sqrt(1.0/84)]))
W = m.sample((84, 10)).reshape((84, 10))

# based on https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html
for epoch in range(2):  # loop over the dataset multiple times

    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # make one-hot encoding of labels
        targets = oneHot(labels)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        pytorch_outputs = net(inputs)
        pytorch_outputs = torch.autograd.Variable(pytorch_outputs,
                                                  requires_grad=True)

        manual_outputs = torch.mm(pytorch_outputs, W)

        delta_out = manual_outputs - targets.view(-1,10)  # = error_out
        dEdW3 = torch.mm(torch.t(pytorch_outputs), delta_out)
        W -= 0.01 * dEdW3  # gradient descent

        delta_h = torch.autograd.Variable(
                                  torch.t(torch.mm(W, torch.t(delta_out))))

        loss = criterion(pytorch_outputs, delta_h)
        loss.backward()
        optimizer.step()

Full code:

import torch
import torchvision
import torchvision.transforms as transforms
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.0, 0.0, 0.0), (1.0, 1.0, 1.0))])

trainset = torchvision.datasets.MNIST(root='./data', train=True,
                                     download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=1,
                                          shuffle=True, num_workers=1)

testset = torchvision.datasets.MNIST(root='./data', train=False,
                                     download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=1,
                                         shuffle=False, num_workers=1)

classes = ('0', '1', '2', '3',
           '4', '5', '6', '7', '8', '9')

def oneHot(a):
    b = torch.zeros(10)
    b[a] = 1
    return b

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(28 * 28, 120)
        self.fc2 = nn.Linear(120, 84)

    def forward(self, x):
        x = x.view(-1, 28 * 28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return x

net = Net()

criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.01)

# Initialize weights just as Pytorch does by default:
m = torch.distributions.uniform.Uniform(torch.tensor([-np.sqrt(1.0/84)]),
                                        torch.tensor([np.sqrt(1.0/84)]))
W = m.sample((84, 10)).reshape((84, 10))

# based on https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html
for epoch in range(2):  # loop over the dataset multiple times

    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # make one-hot encoding of labels
        targets = oneHot(labels)

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        pytorch_outputs = net(inputs)
        pytorch_outputs = torch.autograd.Variable(pytorch_outputs,
                                                  requires_grad=True)

        manual_outputs = torch.mm(pytorch_outputs, W)

        delta_out = manual_outputs - targets.view(-1,10)  # = error_out
        dEdW3 = torch.mm(torch.t(pytorch_outputs), delta_out)
        W -= 0.001*dEdW3  # gradient descent

        delta_h = torch.autograd.Variable(
                                  torch.t(torch.mm(W, torch.t(delta_out))))

        loss = criterion(pytorch_outputs, delta_h)
        loss.backward()
        optimizer.step()

correct = 0
total = 0
with torch.no_grad():
    for data in testloader:
        inputs, labels = data
        pytorch_outputs = torch.autograd.Variable(net(inputs),
                                                      requires_grad=True)
        manual_outputs = torch.mm(pytorch_outputs, W)
        _, predicted = torch.max(manual_outputs.data, 1)
        total += labels.size(0)
        correct += (predicted == labels).sum().item()

print('Accuracy of the network on the 10000 test images: %d %%' % (
    100 * correct / total))


title: Convolution Autoencoder Image Dimension Error

body:
I have the following Convolutional Autoencoder setup:

class autoencoder(nn.Module):
def  __init__(self):
    super(autoencoder, self).__init__()
    self.encoder = nn.Sequential(
        nn.Conv2d(1, 16, 3, stride=3, padding=1),  # b, 16, 10, 10
        nn.ReLU(True),
        nn.MaxPool2d(2, stride=2),  # b, 16, 5, 5
        nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
        nn.ReLU(True),
        nn.MaxPool2d(2, stride=1)  # b, 8, 2, 2
                                )

    self.decoder = nn.Sequential(
        nn.ConvTranspose2d(8, 16, 3, stride=2),  # b, 16, 5, 5
        nn.ReLU(True),
        nn.ConvTranspose2d(16, 8, 5, stride=3, padding=1),  # b, 8, 15, 15
        nn.ReLU(True),
        nn.ConvTranspose2d(8, 1, 2, stride=2, padding=1),  # b, 1, 28, 28
        nn.Tanh()
                                )

This is the main Loop:

for epoch in range(epochs):
running_loss = 0
for data in (train_loader):
    image,_=data
    inputs = image.view(image.size(0),-1)
    optimizer.zero_grad()
    #image = np.expand_dims(img, axis=0)

    outputs = net(inputs)
    loss = criterion(outputs,inputs)
    loss.backward()
    optimizer.step()
    running_loss += loss.data[0]
print('At Iteration : %d   ;  Mean-Squared Error : %f'%(epoch + 1,running_loss/(train_set.train_data.size(0)/batch_size)))

This is the Error:

RuntimeError: Expected 4-dimensional input for 4-dimensional weight [16, 1, 3, 3], but got input of size [1000, 784] instead

This has something to do with the flattening of the image but Im not exactly sure how to deflatten it.


title: PyCharm go to implementation of a method - what does this mean?

body:
I have the following python code:

import torch
t = torch.tensor([1, 2, 3])
print(t.dtype)

Within PyCharm, I select the dtype in the last line and go to its implementation:

It brings me to this _TensorBase.py file, with the beginning that looks like this:

and the line with dtype showing:

Now, this file actually doesn't exist in the pytorch package.  Could someone explain to me what exactly is going when I ask PyCharm to go to the implementation of dtype?  From looking at this, I'm guessing part of the source code is implemented in C and maybe I'm just looking at the interface generated by PyCharm, which explains why the file doesn't actually exist and none of the implementation exists.


title: how store train_loss and valid_loss separably from epoch_loss?

body:
I am trying to store the train_loss and valid_loss separably from epoch_loss as epoch_loss is return back the two loss values(first is train loss and second is valid loss). the epoch_loss is a float64 object. I tried to convert it to a numpy array and then access to the each slice but again return me two values.

this is snippet

criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)  # 1e-3

# Decay LR by a factor of 0.1 every 4 epochs.
#step size: Period of learning rate decay.
#gamma = Multiplicative factor of learning rate decay. Default: 0.1, should
float
scheduler = lr_scheduler.StepLR(optimizer, step_size=2, gamma=0.1)

def train_model(model, criterion, optimizer, scheduler, num_epochs=4):

 since = time.time()

 best_model_wts = copy.deepcopy(model.state_dict())
 best_acc = 0.0

 for epoch in range(num_epochs):  # loop over the dataset multiple times
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 30)

    # Each epoch has a training and validation phase
    for phase in ['train', 'valid']:
        if phase == 'train':
            scheduler.step()

            model.train()  # Set model to training mode
        else:
            model.eval()   # Set model to evaluate mode

        train_loss = 0.0
        total_train = 0
        correct_train = 0

        #iterate over data
        for t_image, mask, image_paths, target_paths in dataLoaders[phase]:

            # get the inputs
            t_image = t_image.to(device)
            mask = mask.to(device)

             # zeroes the gradient buffers of all parameters
            optimizer.zero_grad()

            # forward
            # track history if only in train
            with torch.set_grad_enabled(phase == 'train'):
                outputs = model(t_image)
                _, predicted = torch.max(outputs.data, 1)
                loss = criterion(outputs, mask) # calculate the loss

                # backward + optimize only if in training phase
                if phase == 'train':
                    loss.backward() # back propagation
                    optimizer.step() # update gradients

            # accuracy
            train_loss += loss.item()
            total_train += mask.nelement()  # number of pixel in the batch
            correct_train += predicted.eq(mask.data).sum().item() # sum all precited pixel values

        epoch_loss = train_loss / len(dataLoaders[phase].dataset)
        epoch_acc = (correct_train / total_train)

        print('{} Loss: {:.4f} Acc: {:.4f}'.format(phase, epoch_loss, epoch_acc))

         #deep copy the model
        if phase == 'valid' and epoch_acc > best_acc:
            best_acc = epoch_acc
            best_model_wts = copy.deepcopy(model.state_dict())

    print()
time_elapsed = time.time() - since

print('Training complete in {:.0f}m {:.0f}s'.format(
    time_elapsed // 60, time_elapsed % 60))
print('Best val Acc: {:4f}'.format(best_acc))

torch.save(model.state_dict(), 'train_valid_exp1.pth')
# load best model weights
model.load_state_dict(best_model_wts)


title: PyTorch: Is there a way to store model in CPU ram, but run all operations on the GPU for large models?

body: From what I see, most people seem to be initializing an entire model, and sending the whole thing to the GPU. But I have a neural net model that is too big to fit entirely on my GPU. Is it possible to keep the model saved in ram, but run all the operations on the GPU?


title: Why does a TypeError in commented out line stop my code from running?

body:
I keep getting a "TypeError: unsupported format string passed to numpy.ndarray.__format__" in a line that is commented out. Why is this stopping the script from running? What am I missing here? Thankful for any suggestions!

I am using jupyter notebook with python 3.6.3 and pytorch 1.0.0.

      62             loss = -torch.mean(self.conv_output)
 ---> 63             # print('Iteration:', str(i), 'Loss:', "{0:.2f}".format(str(loss.data.numpy())))

 TypeError: unsupported format string passed to numpy.ndarray.__format__


title: How to calculate a cumulative score with a LSTM using binary labels?

body:
So I might be missing something obvious here, but I'm new to LSTMs and I couldn't find anything on the internet that answers my question.

I need to use a few values (3 numerical values to be exact) to calculate a score which is used for binary classification. So my training dataset has binary labels. The score has to be cumulative, and not just based on the current values, and some values need to be remembered for longer than others. Therefore I decided to use a LSTM. My question is, what should be the basic architecture for such an LSTM? BTW the score should be a running score, i.e. it needs to be updated each time a new input comes in.

I'm using pyTorch, but I don't need exact code, just some guidance on how to design this network would be greatly appreciated.


title: Neural network from scratch - predict single example

body:
Here is a neural network I've modified from Coursera Deep Learning Specialization to train on a dataset containing a flattened array of training data :

%reset -s -f

import numpy as np
import math

def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def initialize_with_zeros(dim):

    w = np.zeros(shape=(dim, 1))
    b = 0

    return w, b

X = np.array([[1,1,1,1],[1,0,1,0] , [1,1,1,0], [0,0,0,0], [0,1,0,0], [0,1,0,1]])
Y = np.array([[1,0,1,1,1,1]])

X = X.reshape(X.shape[0], -1).T
Y = Y.reshape(Y.shape[0], -1).T

print('X shape' , X.shape)
print('Y shape' , Y.shape)

b = 1
w, b = initialize_with_zeros(4)

def propagate(w, b, X, Y) :

    m = X.shape[1]

    A = sigmoid(np.dot(w.T, X) + b)  # compute activation
    cost = (- 1 / m) * np.sum(Y * np.log(A) + (1 - Y) * (np.log(1 - A)))  # compute cost
    dw = (1./m)*np.dot(X,((A-Y).T))
    db = (1./m)*np.sum(A-Y, axis=1)
    cost = np.squeeze(cost)

    grads = {"dw": dw,
             "db": db}

    return grads, cost

propagate(w , b , X , Y)

learning_rate = .001
costs = []

def optimize(w , b, X , Y) :
    for i in range(2):

        grads, cost = propagate(w=w, b=b, X=X, Y=Y)

        dw = grads["dw"]
        db = grads["db"]

        w = w - learning_rate*dw
        b = b -  learning_rate*db

        if i % 100 == 0:
            costs.append(cost)

    return w , b

w , b = optimize(w , b , X , Y)

def predict(w, b, X):

    m = 6
    Y_prediction = np.zeros((1,m))
#     w = w.reshape(X.shape[0], 1)

    A = sigmoid(np.dot(w.T, X) + b)
    for i in range(A.shape[1]):

        if A[0, i] >= 0.5:
            Y_prediction[0, i] = 1

        else:
            Y_prediction[0, i] = 0

    return Y_prediction

predict(w , b, X)

This works as expected but I'm struggling to predict a single example.

If I use :

predict(w , b, X[0])

returns error :

ValueError: shapes (6,4) and (6,) not aligned: 4 (dim 1) != 6 (dim 0)

How to re-arrange matrix operation in order to predict a single instance ?


title: Different filters for 2d convolution

body: I’m having an input of shape (B(atch), F(features), N(odes), T(timestamps)). Right now if I apply a 2d convolution with a kernel of shape (1,2) I will have a total of (F_out, F_in, 1,2) weights to learn which is alright. I want to extend this so that for each Node in the input I have it’s own filter with shape (1,2). Does any of you have any idea where should I start from? So far I looped through all N and apply the filter to its respective input. Unfortunately this approach is very slow.


title: How to have two optimizers such that one optimizer trains the whole parameter and the other trains partial of the parameter?

body:
I have a model:

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.conv1 = nn.Conv2d(128, 128, (3,3))
        self.conv2 = nn.Conv2d(128, 256, (3,3))
        self.conv3 = nn.Conv2d(256, 256, (3,3))

    def forward(self,):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        return x

model = MyModel()

I want to train model in such a way that in every training step DATA_X1 should train
 ['conv1', 'conv2', 'conv3'] layers and DATA_X2 should train only ['conv3'] layers.

I tried making two optimizer:

# Full parameters train
all_params = model.parameters()
all_optimizer = optim.Adam(all_params, lr=0.01)

# Partial parameters train
partial_params = model.parameters()
for p, (name, param) in zip(list(partial_params), model.named_parameters()):
    if name in ['conv3']:
        p.requires_grad = True
    else:
        p.requires_grad = False
partial_optimizer = optim.Adam(partial_params, lr=0.01)

But this affects both the optimizer with required_grad = False

Is there any way I can do this?


title: Churn rate prediction based on sequence data

body:
I am trying to build a machine learning model that can predict if a certain user will churn based on its historical static and dynamic data. The data looks like below:

1) timestamp, user1, user info (static), event info(dynamic), 0
2) timestamp, user1, user info (static), event info(dynamic), 0
3) timestamp, user1, user info (static), event info(dynamic), 1

4) timestamp, user2, user info (static), event info(dynamic), 0
5) timestamp, user2, user info (static), event info(dynamic), 0
6) timestamp, user2, user info (static), event info(dynamic), 0
7) timestamp, user2, user info (static), event info(dynamic), 0
8) timestamp, user2, user info (static), event info(dynamic), 0

There are many user_id in the dataset and each could be a varied-length sequence. The features for each user are two parts. One is user info, which can be regarded as constant for each user. The other part is the event info, which includes features that change over time.

Please let me know what should be my approach to handle such a problem either in machine learning/deep learning, it is better to have a detailed step by step tutorial in pytorch.


title: PIP throwing logging error in google COLAB

body:
I tried

resetting of runtime does not help. Anyone know how to fix? Thanks


title: Predicting the next track using a vanilla RNN in PyTorch

body:
For some context, I have a set of 37 playlists of 12 tracks long. Each track has been hand-selected in a certain way. Early songs in the playlist are generally more chilled and as the playlist progresses tracks begin to increase in tempo. I decided to commit to a project and build a deep playlist generator.

I am implementing a many-to-many vanilla RNN in PyTorch and am seeking clarity on how to train the RNN one playlist at a time, where each track is then parsed and the model predicts the features of the next track.

Pictured is a Many-to-many RNN - for this case - each red box is the current track's features and the opposite blue box is the predicted next track's features:

The feature set (9), X, looks like so:

The target y simply mirrors the above feature set of the next track.

For my RNN Class it looks like so:

class RNNEstimator(nn.Module):
    def __init__(self, input_size=9, hidden_size=30, output_size=9):
        super(RNNEstimator, self).__init__()

        self.hidden_size = hidden_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)

    def forward(self, inp, hidden):
        print("inp", inp.shape)
        print("hid", hidden.shape)
        combined = torch.cat((inp, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        return output, hidden

    def initHidden(self):
        return torch.zeros(1, self.hidden_size)

This is taken from the PyTorch tutorials page. However, I have adapted the RNN Class to output 9 features rather than a binary classification.

The playlist dataset has been processed into a tensor of shape torch.Size([37, 12, 18]), and stride (12, 1, 444)) - meaning 37 playlist, 12 tracks longs with 9 X_features + 9 y_features (18).

The train_rnn function:

# Model Initiation
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = RNNEstimator(9, 30, 9)
optimizer = optim.Adam(model.parameters(), lr=0.001)
loss_fn = torch.nn.L1Loss()

# Training function for RNN
def train_rnn(model, train_loader, epochs, criterion, optimizer, device):

    model.train() # Make sure that the model is in training mode.

    # training loop is provided
    for epoch in range(1, epochs + 1):

        for batch in train_loader:

            total_loss = 0

            # get data
            batch_x = batch[:, :9, :].float().squeeze()
            batch_y = batch[:, 9:, :].float()

            batch_x = batch_x.to(device)
            batch_y = batch_y.to(device)

            optimizer.zero_grad()

            hidden = model.initHidden()

            # For each track in batch/playlist
            # TODO: THIS NEEDS WORK
            for x, y in zip(batch_x, batch_y):
                output, hidden = model(x, hidden)
                loss = criterion(output, y)
                loss.backward()
                optimizer.step()
                total_loss += loss.data.item()

        if epoch % 10 == 0:
            print('Epoch: {}/{}.............'.format(epoch, epochs), end=' ')
            print("Loss: {:.4f}".format(loss.item()))

What I am trying to understand is how to train this model one playlist per batch?

I am getting an error from the cat function like so:

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

The model should parse each track (t) - via the forward method - then output the next track (t+1). The hidden state will reset each playlist given they are independent of one another.


title: Pytorch - using pre-trained model, classifier training

body:
I'd like to use the pre-trained model and train the classifier part only, leaving the weights in the main part of the network unchanged.

I've imported the model, changed the output layer to match the number of classes I have in my data and instantiated the optimizer:

model = torch.hub.load('pytorch/vision:v0.6.0', 'mobilenet_v2', pretrained=True, force_reload=True)

model.classifier[1] = nn.Linear(1280, 50, True)

optimizer = torch.optim.Adam(model.classifier.parameters(), lr=0.05)

Then in the training part I just do (as always):

optimizer.zero_grad()
loss.backward()
optimizer.step()

In that manner, am I training only the classifier part of the network or it still somehow adjust the weights of the whole network and I'm missing something?


title: "Can't use matmul on the given tensors" error when converting pytorch to onnx JS

body:
I made a simple pytorch MLP(GAN generator) and converted it to onnx using the tutorial (https://www.youtube.com/watch?v=Vs730jsRgO8), my code is a bit different but I cant catch the error.

class Generator(nn.Module):
def __init__(self, g_input_dim, g_output_dim):
    super(Generator, self).__init__()
    # g_input = 100
    self.net = nn.Sequential(
      nn.Linear(g_input_dim, 256),
      nn.LeakyReLU(.2),
      nn.Linear(256, 512),
      nn.LeakyReLU(.2),
      nn.Linear(512, 1024),
      nn.LeakyReLU(.2),
      nn.Linear(1024, 784),
      nn.Tanh()
    )

# forward method
def forward(self, x):
    return self.net(x)

After training I export the model to onnx.

torch.save(G.state_dict(), "pytorch_model.pth")
import torch.onnx

model = Generator(z_dim,mnist_dim)
state_dict = torch.load("pytorch_model.pth")

model.load_state_dict(state_dict)
model.eval()

dummy_input = torch.zeros(100)

torch.onnx.export(model, dummy_input, "onnx_model.onnx", verbose=True)

Which gives the following onnx graph, which seems accurate.

graph(%input.1 : Float(100),
      %net.0.bias : Float(256),
      %net.2.bias : Float(512),
      %net.4.bias : Float(1024),
      %net.6.bias : Float(784),
      %25 : Float(100, 256),
      %26 : Float(256, 512),
      %27 : Float(512, 1024),
      %28 : Float(1024, 784)):
  %10 : Float(256) = onnx::MatMul(%input.1, %25) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1612:0
  %11 : Float(256) = onnx::Add(%10, %net.0.bias)
  %12 : Float(256) = onnx::LeakyRelu[alpha=0.20000000000000001](%11) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1239:0
  %14 : Float(512) = onnx::MatMul(%12, %26) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1612:0
  %15 : Float(512) = onnx::Add(%14, %net.2.bias)
  %16 : Float(512) = onnx::LeakyRelu[alpha=0.20000000000000001](%15) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1239:0
  %18 : Float(1024) = onnx::MatMul(%16, %27) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1612:0
  %19 : Float(1024) = onnx::Add(%18, %net.4.bias)
  %20 : Float(1024) = onnx::LeakyRelu[alpha=0.20000000000000001](%19) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1239:0
  %22 : Float(784) = onnx::MatMul(%20, %28) # /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py:1612:0
  %23 : Float(784) = onnx::Add(%22, %net.6.bias)
  %24 : Float(784)

Then I imported the code into javascript.

<html>
  <body>
    <script src="./onnx.min.js"></script>
    <script>
      async function test() {
        const sess = new onnx.InferenceSession()
        await sess.loadModel('./onnx_model.onnx')
        const input = new onnx.Tensor(new Float32Array(100), 'float32', [100])
        const outputMap = await sess.run([input])
        const outputTensor = outputMap.values().next().value
        console.log(`Output tensor: ${outputTensor.data}`)
      }
      test()
    </script>
  </body>
</html>

I know the input dimension is correct but onnx gives me the following error.

onnx.min.js:8 Uncaught (in promise) Error: Can't use matmul on the given tensors
    at e.createProgramInfo (onnx.min.js:8)
    at t.run (onnx.min.js:8)
    at e.run (onnx.min.js:8)
    at t.<anonymous> (onnx.min.js:14)
    at onnx.min.js:14
    at Object.next (onnx.min.js:14)
    at onnx.min.js:14
    at new Promise (<anonymous>)
    at r (onnx.min.js:14)
    at onnx.min.js:14

I also know that matmul is a supported operator with onnx, but I can't figure out how or if my input tensor is correct.


title: Convert custom Convolution from PyTorch to Tensorflow (2.2.0)

body:
I am currently attempting to convert a custom convolution from PyTorch to Tensorflow (V. 2.2.0).

The convolution is defined in PyTorch as:

    self.quantizer = q = nn.Conv1d(1, 2*nq, kernel_size=1, bias=True)
    a = (nq-1) / gap
    #1st half = lines passing to (min+x,1) and (min+x+1/a,0) with x = {nq-1..0}*gap/(nq-1)
    q.weight.data[:nq] = -a
    q.bias.data[:nq] = torch.from_numpy(a*min + np.arange(nq, 0, -1)) # b = 1 + a*(min+x)
    #2nd half = lines passing to (min+x,1) and (min+x-1/a,0) with x = {nq-1..0}*gap/(nq-1)
    q.weight.data[nq:] = a
    q.bias.data[nq:] = torch.from_numpy(np.arange(2-nq, 2, 1) - a*min) # b = 1 - a*(min+x)
    # first and last one are special: just horizontal straight line
    q.weight.data[0] = q.weight.data[-1] = 0
    q.bias.data[0] = q.bias.data[-1] = 1

where nq = 20, min = 0 and max = 1.

My reimplementation looks like this:

my_weight = my_init_weight((1,1,nq*2))
q = tf.nn.convolution(input_q, my_weight)
q = tf.nn.bias_add(q, my_init_bias((40,1), tf.float32))

with these these functions as weight and bias initialization:

def my_init_weight(shape, dtype=None):

    weights = np.zeros(shape, dtype=np.float32)
    weights[:, :, :nq] = -a
    weights[:, :, nq:] = a
    weights[:, :, 0] = weights[:, :, -1] = 0
    return tf.convert_to_tensor(weights, dtype=tf.float32)
def my_init_bias(shape, dtype=None):
    weights = np.zeros(shape[0], dtype=np.float32)
    weights[:nq] = a*min + np.arange(nq, 0, -1)
    weights[nq:] = np.arange(2-nq, 2, 1) - a*min
    weights[0] = weights[-1] = 1
    return weights

The input is a matrix with shape 1681, 1, 1600 for PyTorch (as it uses channels first) and 1681, 1600, 1 for Tensorflow (as it uses channels last) and the out put is 1681, 40, 1600 or 1681, 1600, 40. So it should be correct, however, the output of both convolutions is different.

Input, Output: Tensorflow on a random 100, 100 image:

my_weight = my_init_weight((1,1,nq*2))
my_weight = tf.nn.bias_add(my_weight, my_init_bias((40,1), tf.float32))
q = tf.nn.convolution(test_conv, my_weight)

q_left, q_right = tf.split(q, 2, axis=2)
q = tf.math.minimum(q_left, q_right)
nbs = tf.reduce_sum(q, axis=0)

Input, Output: PyTorch on a random 100, 100 image:

output = q(input_t_t)
output = torch.min(output[:,:nq], output[:,nq:]).clamp(min=0)
nbs = output.sum(dim=-1)


title: Passing variable to nn.Conv2d arguments within a class init definition python

body:
So I want to pass some new variables such as kernel_size when I initiate a new object. Let's say net=Net10(5,2,4,3,1,1). so that I get an object of this class with the parameters I want not something always constant, cos otherwise I will have to define lots of classes. Now, I want to pass kernel_size within the self.Conv2d, and I get a syntax error: positional argument follows keyword argument
Does anyone know how to fix this? Should I change it all to functions instead of classes?

    class Net10(nn.Module):
    def __init__ (self, kernel_size, stride, pooling, num_classes, neurons, ActFunn, *args):
        super(Net10, self).__init__()
        self.kernel = kernel_size
        self.stride = stride
        self.pooling = pooling
        self.num_classes = num_classes
        self.neurons = neurons
        self.Actfun = ActFunn
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=32, kernel, padding=2, stride=1)
        self.pool = nn.MaxPool2d(pooling, pooling)
        self.fcinput= round (28 / pooling)


title: pytoch RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1

body:
I am trying to train a Actor Critic Model with LSTM in both actor and critic.
I am new to all this and can not understand why "RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)" is comming.
I am forwardPropagating from actor and getting error
below is my code and error message.I am using pytorch version 0.4.1
Can someone please help to check what is wrong with this code.
import os
import time
import random
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.preprocessing import StandardScaler
import torch
import torch.nn as nn
import torch.nn.functional as F
from random import random as rndm
from torch.autograd import Variable
from collections import deque

torch.set_default_tensor_type('torch.DoubleTensor')

class Actor(nn.Module):

  def __init__(self, state_dim, action_dim, max_action):
    super(Actor, self).__init__()
    self.lstm = nn.LSTMCell(state_dim, 256)
    self.layer_1 = nn.Linear(256, 400)
    self.layer_2 = nn.Linear(400, 300)
    self.layer_3 = nn.Linear(300, action_dim)
    self.hx = torch.zeros(1,256)
    self.cx = torch.zeros(1,256)
    self.max_action = max_action

  def forward(self, x):
    self.hx, self.cx = self.lstm(x, (self.hx, self.cx))
    x = F.relu(self.layer_1(self.hx))
    x = F.relu(self.layer_2(x))
    x = self.max_action * torch.tanh(self.layer_3(x))
    return x

state_dim = 3
action_dim = 3
max_action = 1

policy = Actor(state_dim, action_dim, max_action)

s = torch.tensor([20,20,100])
next_action = policy(s)

and the error message is :
next_action = policy(s)
Traceback (most recent call last):

  File "<ipython-input-20-de717f0ad3d2>", line 1, in <module>
    next_action = policy(s)

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)

  File "<ipython-input-4-aed4daf511cb>", line 14, in forward
    self.hx, self.cx = self.lstm(x, (self.hx, self.cx))

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 704, in forward
    self.check_forward_input(input)

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 523, in check_forward_input
    if input.size(1) != self.input_size:

RuntimeError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

I am using pytorch version 0.4.1
Can someone please help to check what is wrong with this code.


title: Shouldn't `randperm` in the PyTorch C++ API return a tensor with default type int?

body:
When I try to generate a list of permuted integer indices with randperm using the C++ PyTorch API, the resulting tensor has the element type of CPUFloatType{10} instead of an integer type:
int N_SAMPLES = 10;
torch::Tensor shuffled_indices = torch::randperm(N_SAMPLES);
cout << shuffled_indices << endl;

returns
 9
 3
 8
 6
 2
 5
 4
 7
 1
 0
[ CPUFloatType{10} ]

Which cannot be used used for indexing of tensors because the element type is float and not an integer type. When tryig to use my_tensor.index(shuffled_indices) I get
terminate called after throwing an instance of 'c10::IndexError'
  what():  tensors used as indices must be long, byte or bool tensors

Environment:

python-pytorch, version 1.6.0-2 on Arch Linux
g++ (GCC) 10.1.0

Why does this happen?


title: How to normalize uint16 depth image for training?

body:
The depth image rendered from the ScanNet dataset is in uint16, after dividing the depth by the shift (1000), the depth values are in the range of [0, 1 < some positive float < 10], how to normalize the depth to [0, 1] for training? (per-dataset normalization)

Compute the mean and std over the training set but the mean and std I got are 1.6515 and 0.9373 which apparently cannot normalize the depth either in [0,1] or [-1,1].
Iterate over the training set to find the max_depth and min_depth=0 and use (depth - min_depth) / (max_depth - min_depth), max_depth in the training set is greater than the one in the test set.

Does the second one make sense or are there any ways?


title: Time-series data in PyTorch Geometric

body: I have time-series graph data. I try to predict graph representation for the next time period. Is there any Graph Convolution to handle time-series data or should I use PyTorch Geometric Temporal instead?


title: Optimize pytorch data loader for reading small patches in full HD images

body:
I'm training my neural network using PyTorch framework. The data is full HD images (1920x1080). But in each iteration, I just need to crop out a random 256x256 patch from these images. My network is relatively small (5 conv layers), and hence the bottleneck is being caused by loading the data. I've provided my current code below. Is there any way to optimize loading the data and speed up the training?
Code:
from pathlib import Path

import numpy
import skimage.io
import torch.utils.data as data

import Imath
import OpenEXR

class Ours(data.Dataset):
    """
    Loads patches of resolution 256x256. Patches are selected such that they contain atleast 1 unknown pixel
    """

    def __init__(self, data_dirpath, split_name, patch_size):
        super(Ours, self).__init__()
        self.dataroot = Path(data_dirpath) / split_name
        self.video_names = []
        for video_path in sorted(self.dataroot.iterdir()):
            for i in range(4):
                for j in range(11):
                    view_num = i * 12 + j
                    self.video_names.append((video_path.stem, view_num))
        self.patch_size = patch_size
        return

    def __getitem__(self, index):
        video_name, view_num = self.video_names[index]

        patch_start_pt = (numpy.random.randint(1080), numpy.random.randint(1920))

        frame1_path = self.dataroot / video_name / f'render/rgb/{view_num + 1:04}.png'
        frame2_path = self.dataroot / video_name / f'render/rgb/{view_num + 2:04}.png'
        depth_path = self.dataroot / video_name / f'render/depth/{view_num + 1:04}.exr'
        mask_path = self.dataroot / video_name / f'render/masks/{view_num + 1:04}.png'
        frame1 = self.get_image(frame1_path, patch_start_pt)
        frame2 = self.get_image(frame2_path, patch_start_pt)
        mask = self.get_mask(mask_path, patch_start_pt)
        depth = self.get_depth(depth_path, patch_start_pt, mask)

        data_dict = {
            'frame1': frame1,
            'frame2': frame2,
            'mask': mask,
            'depth': depth,
        }
        return data_dict

    def __len__(self):
        return len(self.video_names)

    @staticmethod
    def get_mask(path: Path, patch_start_point: tuple):
        h, w = patch_start_point
        mask = skimage.io.imread(path.as_posix())[h:h + self.patch_size, w:w + self.patch_size][None]
        return mask

    def get_image(self, path: Path, patch_start_point: tuple):
        h, w = patch_start_point
        image = skimage.io.imread(path.as_posix())
        image = image[h:h + self.patch_size, w:w + self.patch_size, :3]
        image = image.astype(numpy.float32) / 255 * 2 - 1
        image_cf = numpy.moveaxis(image, [0, 1, 2], [1, 2, 0])
        return image_cf

    def get_depth(self, path: Path, patch_start_point: tuple, mask: numpy.ndarray):
        h, w = patch_start_point

        exrfile = OpenEXR.InputFile(path.as_posix())
        raw_bytes = exrfile.channel('B', Imath.PixelType(Imath.PixelType.FLOAT))
        depth_vector = numpy.frombuffer(raw_bytes, dtype=numpy.float32)
        height = exrfile.header()['displayWindow'].max.y + 1 - exrfile.header()['displayWindow'].min.y
        width = exrfile.header()['displayWindow'].max.x + 1 - exrfile.header()['displayWindow'].min.x
        depth = numpy.reshape(depth_vector, (height, width))

        depth = depth[h:h + self.patch_size, w:w + self.patch_size]
        depth = depth[None]
        depth = depth.astype(numpy.float32)
        depth = depth * mask
        return depth

Finally, I'm creating a DataLoader as follows:
train_data_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True, pin_memory=True, num_workers=4)

What I've tried so far:

I've searched if it is possible to read a part of the image. Unfortunately, I didn't get any leads. Looks like python libraries read the full image.
I'm planning to read more patches from a single image so that I will need to read fewer images. But in PyTorch framework, the get_item() function has to return a single sample, not a batch. So, in each get_item() I can read only a patch.
I'm planning to circumvent this as follows: Read 4 patches in get_item() and return patches of shape (4,3,256,256) instead of (3,256,256). Later when I read a batch using dataloader, I'll get a batch of shape (BS,4,3,256,256) instead of (BS,3,256,256). I can then concatenate the data along dim=1 to convert (BS,4,3,256,256) to (BS*4,3,256,256). Thus I can reduce batch_size (BS) by 4 and hopefully this will speed up data loading by 4 times.

Are there any other options? I'm open to all kind of suggestions. Thanks!


title: Multi Head Attention: Correct implementation of Linear Transformations of Q, K, V

body:
I am implementing the Multi-Head Self-Attention in Pytorch now. I looked at a couple of implementations and they seem a bit wrong, or at least I am not sure why it is done the way it is. They would often apply the linear projection just once:
    self.query_projection = nn.Linear(input_dim, output_dim)
    self.key_projection = nn.Linear(input_dim, output_dim)
    self.value_projection = nn.Linear(input_dim, output_dim)

and then they would often reshape the projection as
    query_heads = query_projected.view(batch_size, query_lenght, head_count, head_dimension).transpose(1,2)
    key_heads = key_projected.view(batch_size, key_len, head_count, head_dimension).transpose(1, 2)  # (batch_size, heads_count, key_len, d_head)
    value_heads = value_projected.view(batch_size, value_len, head_count, head_dimension).transpose(1, 2)  # (batch_size, heads_count, value_len, d_head)

    attention_weights = scaled_dot_product(query_heads, key_heads)

According to this code, each head will work on a piece of a projected query. However, the initial paper says that we need to have a different Linear projection for each head in the encoder.
Is this displayed implementation correct?


title: Add a index selected tensor to another tensor with overlapping indices in pytorch

body:
This is a follow up question to this question. I want to do the exactly same thing in pytorch. Is it possible to do this? If yes, how?
import torch
image = torch.tensor([[246,  50, 101], [116,   1, 113], [187, 110,  64]])
iy = torch.tensor([[1, 0, 2], [1, 0, 2], [2, 2, 2]])
ix = torch.tensor([[0, 2, 1], [1, 2, 0], [0, 1, 2]])
warped_image = torch.zeros(size=image.shape)

I need something like torch.add.at(warped_image, (iy, ix), image) that gives the output as
[[  0.   0.  51.]
 [246. 116.   0.]
 [300. 211.  64.]]

Note that the indices at (0,1) and (1,1) point to the same location (0,2). So, I want warped_image[0,2] = image[0,1] + image[1,1] = 51.


title: Flair Framework with PyTorch - OverflowError: int too big to convert

body:
I'm trying to train a named entity recognition model with Flair Framework (https://github.com/flairNLP/flair), with this embedding: TransformerWordEmbeddings('emilyalsentzer/Bio_ClinicalBERT'). However, it always failed with OverflowError: int too big to convert. This is also happening in some other transformer word embedding such as XLNet. However, BERT and RoBERTa works fine.
Here is the full traceback of the error:
2021-04-15 09:34:48,106 ----------------------------------------------------------------------------------------------------
2021-04-15 09:34:48,106 Corpus: "Corpus: 778 train + 259 dev + 260 test sentences"
2021-04-15 09:34:48,106 ----------------------------------------------------------------------------------------------------
2021-04-15 09:34:48,106 Parameters:
2021-04-15 09:34:48,106  - learning_rate: "0.1"
2021-04-15 09:34:48,106  - mini_batch_size: "32"
2021-04-15 09:34:48,106  - patience: "3"
2021-04-15 09:34:48,106  - anneal_factor: "0.5"
2021-04-15 09:34:48,106  - max_epochs: "200"
2021-04-15 09:34:48,106  - shuffle: "True"
2021-04-15 09:34:48,106  - train_with_dev: "False"
2021-04-15 09:34:48,106  - batch_growth_annealing: "False"
2021-04-15 09:34:48,107 ----------------------------------------------------------------------------------------------------
2021-04-15 09:34:48,107 Model training base path: "/home/xxx/data/xxx-clinical-bert"
2021-04-15 09:34:48,107 ----------------------------------------------------------------------------------------------------
2021-04-15 09:34:48,107 Device: cuda:0
2021-04-15 09:34:48,107 ----------------------------------------------------------------------------------------------------
2021-04-15 09:34:48,107 Embeddings storage mode: gpu
2021-04-15 09:34:48,116 ----------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "train_medical_2.py", line 144, in <module>
    train_ner(d + '-base-ent',corpus_base)
  File "train_medical_2.py", line 136, in train_ner
    max_epochs=200)
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/flair/trainers/trainer.py", line 381, in train
    loss = self.model.forward_loss(batch_step)
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/flair/models/sequence_tagger_model.py", line 637, in forward_loss
    features = self.forward(data_points)
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/flair/models/sequence_tagger_model.py", line 642, in forward
    self.embeddings.embed(sentences)
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/flair/embeddings/token.py", line 81, in embed
    embedding.embed(sentences)
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/flair/embeddings/base.py", line 60, in embed
    self._add_embeddings_internal(sentences)
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/flair/embeddings/token.py", line 923, in _add_embeddings_internal
    self._add_embeddings_to_sentence(sentence)
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/flair/embeddings/token.py", line 999, in _add_embeddings_to_sentence
    truncation=True,
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/transformers/tokenization_utils_base.py", line 2438, in encode_plus
    **kwargs,
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py", line 472, in _encode_plus
    **kwargs,
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py", line 379, in _batch_encode_plus
    pad_to_multiple_of=pad_to_multiple_of,
  File "/home/d111199102201607101/flair/lib/python3.6/site-packages/transformers/tokenization_utils_fast.py", line 330, in set_truncation_and_padding
    self._tokenizer.enable_truncation(max_length, stride=stride, strategy=truncation_strategy.value)
OverflowError: int too big to convert

I have tried to change the embedding_storage_mode, hidden_size, and mini_batch_size. None of these gave me the fix to the issue.
Does anyone have the same issue?
Is there any way to resolve this?
Thanks