title: Why forward method of torch.autograd.Function has to be static

body:
I got warnings that

/tmp/pip-req-build-58y_cjjl/torch/csrc/autograd/python_function.cpp:638: UserWarning: Legacy autograd function with non-static forward method is deprecated and will be removed in 1.3.
 Please use new-style autograd function with static forward method. (Example: https://pytorch.org/docs/stable/autograd.html#torch.autograd.Function)
/tmp/pip-req-build-58y_cjjl/torch/csrc/autograd/python_function.cpp:664: UserWarning: Legacy autograd function object was called twice.  You will probably get incorrect gradients from
 this computation, as the saved tensors from the second invocation will clobber the saved tensors from the first invocation.  Please consider rewriting your autograd function in the m
odern style; for information on the new format, please see: https://pytorch.org/docs/stable/notes/extending.html#extending-torch-autograd

which means I must define static forward function in pytorch 1.3 or later version.
I just wonder why forward method of autograd.Function must be static?


title: What does local rank mean in distributed deep learning?

body:
https://github.com/huggingface/transformers/blob/master/examples/run_glue.py

I want to adapt this script to do text classification on my data. The computer for this task is one single machine with two graphic cards. So this involves kind of "distributed" training with the term local_rank in the script above, especially when local_rank equals 0 or -1 like in line 83.

After reading some materials from distributed computation I guess that local_rank is like an ID for a machine. And 0 may mean this machine is the "main" or "head" in the computation. But what is -1?


title: ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index) throws IndexError: Target 42 is out of bounds

body:
After loading CIFAR 100, I try to train my neural network. But I don't know why I get the out of bounds error shown below

Optimizing the network with batch size 25
Epoch: 0  of  30 Average loss: -
/mnt_home/klee/LBSBGenGapSharpnessResearch/vgg.py:43: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  return F.log_softmax(x)
Traceback (most recent call last):
  File "plot_parametric_pytorch_cifar100.py", line 130, in <module>
    loss_fn = F.nll_loss(ops, tgts)
  File "/home/klee/anaconda3/envs/sharpenv/lib/python3.7/site-packages/torch/nn/functional.py", line 2115, in nll_loss
    ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
IndexError: Target 42 is out of bounds.

Here is the script I'm running:

cudnn.benchmark = True
(X_train, y_train), (X_test, y_test) = cifar10.load_data()
X_train = X_train.astype('float32')
X_train = np.transpose(X_train, axes=(0, 3, 1, 2))
X_test = X_test.astype('float32')
X_test = np.transpose(X_test, axes=(0, 3, 1, 2))
X_train /= 255
X_test /= 255
device = torch.device('cuda:0')

# This is where you can load any model of your choice.
# I stole PyTorch Vision's VGG network and modified it to work on CIFAR-10.
# You can take this line out and add any other network and the code
# should run just fine.
model = vgg.vgg11_bn()
#model.to(device)

# Forward pass
opfun = lambda X: model.forward(Variable(torch.from_numpy(X)))

# Forward pass through the network given the input
predsfun = lambda op: np.argmax(op.data.numpy(), 1)

# Do the forward pass, then compute the accuracy
accfun = lambda op, y: np.mean(np.equal(predsfun(op), y.squeeze()))*100

# Initial point
x0 = deepcopy(model.state_dict())

# Number of epochs to train for
# Choose a large value since LB training needs higher values
# Changed from 150 to 30
nb_epochs = 30
batch_range = [25, 40, 50, 64, 80, 128, 256, 512, 625, 1024, 1250, 1750, 2048, 2500, 3125, 4096, 4500, 5000]

# parametric plot (i.e., don't train the network)
hotstart = False

if not hotstart:
    for batch_size in batch_range:
        optimizer = torch.optim.Adam(model.parameters())
        model.load_state_dict(x0)
        #model.to(device)
        average_loss_over_epoch = '-'
        print('Optimizing the network with batch size %d' % batch_size)
        np.random.seed(1337) #So that both networks see same sequence of batches
        for e in range(nb_epochs):
            model.eval()
            print('Epoch:', e, ' of ', nb_epochs, 'Average loss:', average_loss_over_epoch)
            average_loss_over_epoch = 0
            # Checkpoint the model every epoch
            torch.save(model.state_dict(), "./models/30EpochCIFAR100ExperimentBatchSize" + str(batch_size) + ".pth")
            array = np.random.permutation(range(X_train.shape[0]))
            slices = X_train.shape[0] // batch_size
            beginning = 0
            end = 1

            # Training loop!
            for _ in range(slices):
                start_index = batch_size * beginning
                end_index = batch_size * end
                smpl = array[start_index:end_index]
                model.train()
                optimizer.zero_grad()
                ops = opfun(X_train[smpl])
                tgts = Variable(torch.from_numpy(y_train[smpl]).long().squeeze())
                loss_fn = F.nll_loss(ops, tgts) <--- errorring linne
                average_loss_over_epoch += loss_fn.data.numpy() / (X_train.shape[0] // batch_size)
                loss_fn.backward()
                optimizer.step()
                beginning += 1
                end += 1

and here's the VGG model:

__all__ = [
    'VGG', 'vgg11', 'vgg11_bn', 'vgg13', 'vgg13_bn', 'vgg16', 'vgg16_bn',
    'vgg19_bn', 'vgg19',
]

model_urls = {
    'vgg11': 'https://s3.amazonaws.com/pytorch/models/vgg11-fb7e83b2.pth',
    'vgg13': 'https://s3.amazonaws.com/pytorch/models/vgg13-58758d87.pth',
    'vgg16': 'https://s3.amazonaws.com/pytorch/models/vgg16-82412952.pth',
    'vgg19': 'https://s3.amazonaws.com/pytorch/models/vgg19-341d7465.pth',
}

class VGG(nn.Module):
    def __init__(self, features):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Sequential(
            nn.Dropout(),
            nn.Linear(512, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Linear(4096, 10),
        )
        self._initialize_weights()

    def forward(self, x):
        x = self.features(x)
        x = x.view(x.size(0), -1)
        x = self.classifier(x)
        return F.log_softmax(x)

    def _initialize_weights(self):
        for m in self.modules():
            if isinstance(m, nn.Conv2d):
                n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
                m.weight.data.normal_(0, math.sqrt(2. / n))
                if m.bias is not None:
                    m.bias.data.zero_()
            elif isinstance(m, nn.BatchNorm2d):
                m.weight.data.fill_(1)
                m.bias.data.zero_()
            elif isinstance(m, nn.Linear):
                n = m.weight.size(1)
                m.weight.data.normal_(0, 0.01)
                m.bias.data.zero_()

def make_layers(cfg, batch_norm=False):
    layers = []
    in_channels = 3
    for v in cfg:
        if v == 'M':
            layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
        else:
            conv2d = nn.Conv2d(in_channels, v, kernel_size=3, padding=1)
            if batch_norm:
                layers += [conv2d, nn.BatchNorm2d(v), nn.ReLU(inplace=True)]
            else:
                layers += [conv2d, nn.ReLU(inplace=True)]
            in_channels = v
    return nn.Sequential(*layers)

cfg = {
    'A': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'B': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'D': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'E': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

def vgg11(pretrained=False, **kwargs):
    """VGG 11-layer model (configuration "A")

    Args:
        pretrained (bool): If True, returns a model pre-trained on ImageNet
    """
    model = VGG(make_layers(cfg['A']), **kwargs)
    if pretrained:
        model.load_state_dict(model_zoo.load_url(model_urls['vgg11']))
    return model

def vgg11_bn(**kwargs):
    """VGG 11-layer model (configuration "A") with batch normalization"""
    return VGG(make_layers(cfg['A'], batch_norm=True), **kwargs)

I'm not sure how to fix the index error. I think it has to do with the number of classes, but I'm  not sure where to fix that (in the above code): I read https://discuss.pytorch.org/t/indexerror-target-2-is-out-of-bounds/69614 but not sure where to go from here.


title: PyTorch Dataloader - List is not callable error when enumerating

body:
When iterating over a PyTorch dataloader, e.g.

# define dataset, dataloader
train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)
trainloader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=64)

# define model, optimizer, loss
# not included - irrelevant to the question

for ii, (inputs, labels) in enumerate(trainloader):

    # Move input and label tensors to the GPU
    inputs, labels = inputs.to(device), labels.to(device)

    start = time.time()

    outputs = model.forward(inputs)
    loss = criterion(outputs, labels)
    loss.backward()

I get a TypeError: 'list' object is not callable on this line

for ii, (inputs, labels) in enumerate(trainloader):

What dumb thing am I forgetting?


title: binarize input for pytorch

body:
may I ask how to make data loaded in pytorch become binarized once it is loaded?
Like Tensorflow can done this through:

train_data  = mnist.input_data.read_data_sets(data_directory, one_hot=True)

How can pytorch achieve the one_hot=True effect.

The data_loader I have now is:

torch.set_default_tensor_type('torch.FloatTensor')
train_loader = torch.utils.data.DataLoader(
        datasets.MNIST('data/', train=True, download=True,
                       transform=transforms.Compose([
                         # transforms.RandomHorizontalFlip(),
                           transforms.ToTensor()])),
        batch_size=batch_size, shuffle=False)

I want to make data in train_loader be binarized.
Now what I am doing is: After loading the data,

for data,_ in train_loader:
    torch.round(data)
    data = Variable(data)

Use the torch.round() function. Is this correct?


title: How to initialize weights in a pytorch model

body:
I've got a fairly straight forward problem here.
I've just finished re-configuring a network by replacing nn.Upsample with the upConv sequential container shown in the code below. I've verified that everything is lined up by running summary(UNetPP, (3, 128, 128)) which runs with no issue.

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

class blockUNetPP(nn.Module):
    def __init__(self, in_channels, middle_channels, out_channels):
        super().__init__()
        self.relu = nn.LeakyReLU(0.2, inplace=True)
        self.conv1 = nn.Conv2d(in_channels, middle_channels, 3, padding=1)
        self.bn1 = nn.BatchNorm2d(middle_channels)
        self.conv2 = nn.Conv2d(middle_channels, out_channels, 3, padding=1)
        self.bn2 = nn.BatchNorm2d(out_channels)

    def forward(self, x):
        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        return out

class upConv(nn.Module):
    def __init__(self, in_ch, out_ch):
        super().__init__()
        self.upc = nn.Sequential(
                                 nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True),
                                 nn.Conv2d(in_ch, out_ch*2, 3, stride=1, padding=1),
                                 nn.BatchNorm2d(out_ch*2),
                                 nn.ReLU(inplace=True)
                                 )
    def forward(self, x):
        out = self.upc(x)
        return out

My issue is that when I try to start training the model I get the following issue:

Traceback (most recent call last):
  File "runTrain.py", line 90, in <module>
    netG.apply(weights_init)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 289, in apply
    module.apply(fn)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 290, in apply
    fn(self)
  File "D:\Thesis Models\Deep_learning_models\UNet\train\NetC.py", line 8, in weights_init
    m.weight.data.normal_(0.0, 0.02)
  File "C:\Users\Anaconda3\envs\CFD\lib\site-packages\torch\nn\modules\module.py", line 594, in __getattr__
    type(self).__name__, name))
AttributeError: 'upConv' object has no attribute 'weight'

I've looked up  solutions which suggest looping over container modules, but I'm already doing this with weights_init(m). Could someone explain whats wrong with my current setup?


title: what is the first initialized weight in pytorch convolutional layer

body:
I do self-studying in Udacity PyTorch
Regarding to the last paragraph

Learning
In the code you've been working with, you've been setting the values of filter weights explicitly, but neural networks will actually learn the best filter weights as they train on a set of image data. You'll learn all about this type of neural network later in this section, but know that high-pass and low-pass filters are what define the behavior of a network like this, and you know how to code those from scratch!
In practice, you'll also find that many neural networks learn to detect the edges of images because the edges of object contain valuable information about the shape of an object.

I have studied all through the last 44th sections. But I couldn't be able to answer the following questions

What is the initialized weight when I do torch.nn.Conv2d? And how to define it myself?
How does PyTorch update weights in the convolutional layer?


title: pytorch variable index lost one dimension

body:
I am going to get every horizontal tensor in a variable but I got one dimension lost.
This is my code:
import torch
from torch.autograd import Variable
t = torch.rand((2,2,4))
x = Variable(t)
print(x)
shape = x.size()
for i in range(shape[0]):
    for j in range(shape[1]):
        print(x[i,j])

and the output is :
Variable containing:
(0 ,.,.) =
  0.6717  0.8216  0.5100  0.9106
  0.3280  0.8182  0.5781  0.3919

(1 ,.,.) =
  0.8823  0.4237  0.6620  0.0817
  0.5781  0.4187  0.3769  0.0498
[torch.FloatTensor of size 2x2x4]

Variable containing:
 0.6717
 0.8216
 0.5100
 0.9106
[torch.FloatTensor of size 4]

Variable containing:
 0.3280
 0.8182
 0.5781
 0.3919
[torch.FloatTensor of size 4]

Variable containing:
 0.8823
 0.4237
 0.6620
 0.0817
[torch.FloatTensor of size 4]

Variable containing:
 0.5781
 0.4187
 0.3769
 0.0498
[torch.FloatTensor of size 4]

and how can I get [torch.FloatTensor of size 1x4]?


title: Unable to import torch (ImportError: libcudart.so.10.0)

body:
I'm currently working on a Nvidia Jetson Nano and I'm not very familiar with Linux. I am trying to run a python file which imports a package called torch. I have installed it alongside with  torchvision while following the instructions from NVIDIA here.

When I run pip list on my terminal, I am able to see torch listed as one of the packages installed. However, I am unable to run the python file due to the error seen below. When I try to run it on python shell, the same error pops up.
FYI: Previously it had issues as the system was using python 2 by default but I have already fixed the path by switching to python 3 by editing the .bashrc file.

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jiayi/.local/lib/python3.6/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

I have tried uninstalling and installing via pip but to no avail. When I try to install the pytorch package (following the instructions from a github repo here), an error occurs as seen below and it is due to the same issue. It is able to detect that the torch package is installed but there seems to be an internal issue.

Requirement already satisfied: torch==1.4.0 from file:///home/jiayi/jetson-inference/build/torch-1.4.0-cp36-cp36m-linux_aarch64.whl in /home/jiayi/.local/lib/python3.6/site-packages (1.4.0)
[jetson-inference]  cloning torchvision...
[sudo] password for jiayi:
Cloning into 'torchvision-36'...
remote: Enumerating objects: 71, done.
remote: Counting objects: 100% (71/71), done.
remote: Compressing objects: 100% (56/56), done.
remote: Total 8219 (delta 37), reused 29 (delta 15), pack-reused 8148
Receiving objects: 100% (8219/8219), 10.22 MiB | 3.60 MiB/s, done.
Resolving deltas: 100% (5631/5631), done.
[jetson-inference]  building torchvision for Python 3.6...
Traceback (most recent call last):
  File "setup.py", line 14, in <module>
    import torch
  File "/home/jiayi/.local/lib/python3.6/site-packages/torch/__init__.py", line 81, in <module>
    from torch._C import *
ImportError: libcudart.so.10.0: cannot open shared object file: No such file or directory

[jetson-inference]  installation complete, exiting with status code 0
[jetson-inference]  to run this tool again, use the following commands:

    $ cd <jetson-inference>/build
    $ ./install-pytorch.sh


title: keras.preprocessing.text.Tokenizer equivalent in Pytorch?

body: Basically the title; is there any equivalent tokeras.preprocessing.text.Tokenizer in Pytorch? I have yet to find any that gives all the utilities without handcrafting things.


title: Load custom data from folder in dir Pytorch

body:
I am getting my hands dirty with Pytorch and I am trying to do what is apparently the hardest part in deep learning-> LOADING MY CUSTOM DATASET AND RUNNING THE PROGRAM<--    The problem is this " too many values to unpack (expected 2)" also I think I am loading the data wrong. Can someone please show me how to do this. How to use the Dataloader user one's own data.

import os
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.utils.data as data
import torchvision
from torchvision import transforms

# Hyper parameters
num_epochs = 20
batchsize = 100
lr = 0.001

EPOCHS = 2
BATCH_SIZE = 10
LEARNING_RATE = 0.003
TRAIN_DATA_PATH = "ImageFolder/images/train/"
TEST_DATA_PATH = "ImageFolder/images/test/"
TRANSFORM_IMG = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406],
                         std=[0.229, 0.224, 0.225] )
    ])

train_data = torchvision.datasets.ImageFolder(root=TRAIN_DATA_PATH, transform=TRANSFORM_IMG)
train_data_loader = data.DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,  num_workers=4)
test_data = torchvision.datasets.ImageFolder(root=TEST_DATA_PATH, transform=TRANSFORM_IMG)
test_data_loader  = data.DataLoader(test_data, batch_size=BATCH_SIZE, shuffle=True, num_workers=4)

# Data Loader (Input Pipeline)
train_loader = torch.utils.data.DataLoader(dataset=TRAIN_DATA_PATH,
                                           batch_size=batchsize,
                                           shuffle=True)
test_loader = torch.utils.data.DataLoader(dataset=TEST_DATA_PATH,
                                          batch_size=batchsize,
                                          shuffle=False)

# CNN model
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conlayer1 = nn.Sequential(
            nn.Conv2d(1,6,3),
            nn.Sigmoid(),
            nn.MaxPool2d(2))
        self.conlayer2 = nn.Sequential(
            nn.Conv2d(6,16,3),
            nn.Sigmoid(),
            nn.MaxPool2d(2))
        self.fc = nn.Sequential(
            nn.Linear(400,120),
            nn.Relu(),
            nn.Linear(120,84),
            nn.Relu(),
            nn.Linear(84,10))

    def forward(self, x):
        out = self.conlayer1(x)
        out = self.conlayer2(out)
        out = out.view(out.size(0),-1)
        out = self.fc(out)
        return out

cnn = CNN()

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(cnn.parameters(), lr=lr)

# Train the Model
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # Forward + Backward + Optimize
        optimizer.zero_grad()
        outputs = cnn(images)
        loss = criterion(outputs,labels)
        loss.backward()
        optimizer.step()

        if (i+1)%100 == 0:
            print('Epoch [%d/%d], Iter [%d/%d] Loss: %.4f' %
                  (epoch+1,num_epochs,i+1,len(train_dataset)//batchsize,loss.data[0]))

# Test the Model
cnn.eval()  # Change model to 'eval' mode (BN uses moving mean/var)
correct = 0
total = 0
for images, labels in test_loader:
    outputs = cnn(images)
    _, predicted = torch.max(outputs.data, 1)
    total += labels.size(0)
    correct += (predicted == labels).sum()
print('Test Accuracy of the model on test images: %.6f%%' % (100.0*correct/total))

#Save the Trained Model
torch.save(cnn.state_dict(),'cnn.pkl')


title: Pytorch installation problems with Anaconda

body:
After having upgraded my environment's python to Python 3.61, I attempted to install pytorch using this command:

conda install -c peterjc123 pytorch

However I got this error:

Fetching package metadata .............
Solving package specifications: .

UnsatisfiableError: The following specifications were found to be in conflict:

-pytorch
-pyqt

I also used the commands
conda install -c peterjc123 pytorch cuda90
conda install -c peterjc123 pytorch cuda80

But the result is still the same. Anyone got a clue how to solve this?


title: Auto updating custom layer parameters while backpropagating in pytorch

body:
I have a pytorch custom layer defined as:

class MyCustomLayer(nn.Module):
  def __init__(self):
    super(MyCustomLayer, self).__init__()

    self.my_parameter = torch.rand(1, requires_grad = True)

    # the following allows the previously defined parameter to be recognized as a network parameter when instantiating the model
    self.my_registered_parameter = nn.ParameterList([nn.Parameter(self.my_parameter)])

  def forward(self, x):
    return x*self.my_parameter

I then define my network where the custom layer is used:

class MyNet(nn.Module):
  def __init__(self):
    super(MyNet, self).__init__()
    self.layer1 = MyCustomLayer()

  def forward(self, x):
    x = self.layer1(x)
    return x

Now Let's instantiate MyNet and observe the issue:

# instantiate MyNet and run it over one input value
model = MyNet()
x = torch.tensor(torch.rand(1))
output = model(x)
criterion = nn.MSELoss()
loss = criterion(1, output)
loss.backward()

Iterating through model parameters shows None for custom layer parameter:

for p in model.parameters():
    print (p.grad)

None

while directly accessing that parameter shows the correct grad value:

print(model.layer1.my_parameter.grad)

tensor([-1.4370])

This, in turn, prevents the optim step from updating the inner parameters automatically and leaves me with the hassle of having to update those manually. Anyone knows how I can address this issue?


title: Why PyTorch nn.Module.cuda() not moving Module tensor but only parameters and buffers to GPU?

body:
nn.Module.cuda() moves all model parameters and buffers to the GPU.

But why not the model member tensor?

class ToyModule(torch.nn.Module):
    def __init__(self) -> None:
        super(ToyModule, self).__init__()
        self.layer = torch.nn.Linear(2, 2)
        self.expected_moved_cuda_tensor = torch.tensor([0, 2, 3])

    def forward(self, input: torch.Tensor) -> torch.Tensor:
        return self.layer(input)

toy_module = ToyModule()
toy_module.cuda()

next(toy_module.layer.parameters()).device
>>> device(type='cuda', index=0)

for the model member tensor, the device stays unchanged.

>>> toy_module.expected_moved_cuda_tensor.device
device(type='cpu')


title: How to access the predictions of pytorch classification model? (BERT)

body:
I'm running this file:
https://github.com/huggingface/pytorch-pretrained-BERT/blob/master/examples/run_classifier.py

This is the prediction code for one input batch:

  input_ids = input_ids.to(device)
  input_mask = input_mask.to(device)
  segment_ids = segment_ids.to(device)
  label_ids = label_ids.to(device)

  with torch.no_grad():
       logits = model(input_ids, segment_ids, input_mask, labels=None)

       loss_fct = CrossEntropyLoss()
       tmp_eval_loss = loss_fct(logits.view(-1, num_labels), label_ids.view(-1))

       eval_loss += tmp_eval_loss.mean().item()
       nb_eval_steps += 1
       if len(preds) == 0:
           preds.append(logits.detach().cpu().numpy())
       else:
           preds[0] = np.append(preds[0], logits.detach().cpu().numpy(), axis=0)

The task is a binary classification.
I want to access the binary output.

I've tried this:

  curr_pred = logits.detach().cpu()

  if len(preds) == 0:
      preds.append(curr_pred.numpy())
  else:
      preds[0] = np.append(preds[0], curr_pred.numpy(), axis=0)

  probablities = curr_pred.softmax(1).numpy()[:, 1]

But the results seem weird. So I'm not sure if it's the correct way.

My hypothesis - I'm receiving the output of the last layer, therefore after softmax, it's the true probabilities (vector of dim 2 - the probability to the 1st and probability to the 2nd class.)


title: How to ignore and initialize Missing key(s) in state_dict

body:
My saved state_dict does not contain all the layers that are in my model.
How can I ignore the Missing key(s) in state_dict error and initialize the remaining weights?


title: An invalid combination of arguments in pytorch

body:
I'm having some trouble to identify the error.
This is the error message that I'm getting:

return torch.exp(-((x-2)**2)) + 0.8*torch.exp(-(x+2)**2)
TypeError: torch.exp received an invalid combination of arguments - got (!float!), but expected (torch.FloatTensor source)

import torch
import time
dtype = torch.FloatTensor

def functionFit(x):
    return torch.exp(-((x-2)**2)) + 0.8*torch.exp(-(x+2)**2)

def createSample(mu,sig,N):
    return  mu + sig*torch.randn(N,1).type(dtype)

def updateMU(alpha,rho,x,N,I,mu):
    return alpha*torch.mean(x[I[int((1-rho)*N):N,0]]) + (1-alpha)*mu

def updateSIG(alpha,rho,x,N,I,sig):
    return alpha*torch.std(x[I[int((1-rho)*N):N,0]])  + (1-alpha)*sig

def CE(N,rho,alpha,epsilon,mu,sig): # initial std dev.
    start = time.time()
    k = 0
    while (sig > epsilon):
        x            = createSample(mu,sig,N)
        S            = functionFit(x)
        sorted_v , I = torch.sort(S,0)
        mu           = updateMU(alpha,rho,x,N,I,mu)
        sig          = updateSIG(alpha,rho,x,N,I,sig)
        k = k + 1

    end = time.time()
    xm  = torch.mean(x)
    ym  = functionFit(xm)
    print('x =',xm)
    print('y =',ym)
    print('time =',end - start,'s')
    print('iter =',k)

if __name__ == '__main__':
    N       = 50
    rho     = 0.5
    alpha   = 0.9
    epsilon = 0.001
    mu      = 20*torch.rand(1,1).type(dtype)-10 # init mu
    sig     = 5
    CE(N,rho,alpha,epsilon,mu,sig)


title: pyTorch: Creating a dataset

body:
I got a big pandas dataframe with time based measurement data (sensor values, the time information does not need to be an input to the network).
This dataframe is put into a tensor and then loaded with torch.Dataloader.

The dataloader is extremly slow, the GPU utilization is ~3%.

As you can see in the code below the dataset is put into a tensor an then put into a torch.utils.data.DataLoader. This loader is then used with enumerate(Dataloader).

During my research I found the torch.utils.data.TensorDataset, but when I try to put the tensor into this type I get the error:
TypeError: Variable data has to be a tensor, but got list

Any suggestions about that?

As well I found the samplers from torch. But when to use these, as I am getting out data of the dataloader, just very slow.

    dataset = big_dataframe_flt.values
    (looks like: array([[ 0.17114914, -0.67040386, -0.72875149, ..., -0.51023438,
     0.49735906, -0.74075046],
   [ 0.17114914, -0.67088608, -0.72631001, ..., -0.53046875,
     0.49741296, -0.74127526],)

    dataset = torch.tensor(dataset).float()
    dataset = torch.utils.data.TensorDataset(dataset)

    data_loader = torch.utils.data.DataLoader(dataset=dataset, batch_size=100, shuffle=True, num_workers=4, pin_memory=True)

    for epoch in range(num_epochs):
        model.train()
        for batch_idx, data in enumerate(data_loader):
            data = Variable(data).to(device)
            recon_batch, mu, var = model(data)

            # Backprop and optimize
            loss = loss_function(recon_batch, data, mu, var)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

data = Variable(data).to(device)

TypeError: Variable data has to be a tensor, but got list


title: What's the fastest way to copy values from one tensor to another in PyTorch?

body:
I am experimenting with dilation in convolution where I am trying to copy data from one 2D tensor to another 2D tensor using PyTorch. I'm copying values from tensor A to tensor B such that every element of A that is copied into B is surrounded by n zeros.

I have already tried using nested for loops, which is a very naive way. The performance, obviously is quite bad when I'm using large number of grayscale images as input.

for i in range(A.shape[0]):
   for j in range(A.shape[1]):
      B[n+i][n+j] = A[i][j]

Is there anything faster that doesn't need the usage of loops?


title: Error: shape '[-1, 270000]' is invalid for input of size 1440000

body:
I got an error:

  shape '[-1, 270000]' is invalid for the input of size 1440000

while running my  code for a CNN structure input tensor size is 64.

Class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 48, 2)
        self.conv2 = nn.Conv2d(48, 108, 2)
        self.conv3 = nn.Conv2d(108, 192, 2)
        self.conv4 = nn.Conv2d(192, 300, 2)
        self.pool = nn.MaxPool2d(2, 2)
        self.fc1 = nn.Linear(300* 30* 30, 864)
        self.fc2 = nn.Linear(864, 288)
        self.fc3 = nn.Linear(288, 2)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = F.relu(self.conv4(x))
        #x = self.pool(F.relu(self.conv4(x)))
        x = self.pool(x)
        x = x.view(-1, 300 * 30* 30)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return F.log_softmax(x)

Any idea why I am getting above error?


title: How to calculate perplexity for a language model using Pytorch

body:
I'm fine-tuning GPT-2 model for language generation task using huggingface Transformers library-pytorch, and I need to calculate an evaluation score(perplexity) for the fine-tuned model. But I'm not sure how that can be done using loss. I would like to know how perplexity can be calculated for the model with sum_loss or mean loss or any other suggestions are also welcome. Any help is appriciated.

Edit:

outputs = model(article_tens, labels=article_tens)

        loss, prediction_scores = outputs[:2]
        loss.backward()
        sum_loss = sum_loss + loss.detach().data

given above is how I calculate loss for each batch of data for the fine-tuning task.

sum loss 1529.43408203125
loss 4.632936000823975
prediction_scores tensor([[[-11.2291,  -9.2614, -11.8575,  ..., -18.1927, -17.7286, -11.9215],
         [-67.2786, -63.5928, -70.7110,  ..., -75.3516, -73.8672, -67.6637],
         [-81.1397, -80.0295, -82.9357,  ..., -83.7913, -85.7201, -78.9877],
         ...,
         [-85.3213, -82.5135, -86.5459,  ..., -90.9160, -90.4393, -82.3141],
         [-44.2260, -43.1702, -49.2296,  ..., -58.9839, -55.2058, -42.3312],
         [-63.2842, -59.7334, -61.8444,  ..., -75.0798, -75.7507, -54.0160]]],
       device='cuda:0', grad_fn=<UnsafeViewBackward>)

The above given is when loss is printed for a batch only


title: Bicubic interpolation in pytorch

body:
I'm trying to do bicubic interpolation on a torch.Tensor.

I know about torch.nn.functional.interpolate, but that method doesn't support bicubic interpolation yet.

I know that PIL images support bicubic interpolation, so I created this snippet (part of torch.nn.Module).

def build_transform(self, shape):
    h, w = shape[2] * self.scale_factor, shape[3] * self.scale_factor
    return Compose([
        ToPILImage(),
        Resize((h, w), interpolation=PIL.Image.BICUBIC),
        ToTensor()
      ])

def forward(self, x: Tensor):
    if self.transform is None:
        self.transform = self.build_transform(x.shape)

    shape = x.shape[0], x.shape[1], x.shape[2] * self.scale_factor, x.shape[3] * self.scale_factor
    new = Tensor(size=shape, device=x.device)

    for i in range(x.shape[0]):
        new[i] = self.transform(x[i])
    new.requires_grad_()

    return new

My assumption is that this method is slower than interpolating the Tensor in one pass.

My Question: Is there no other way to compute this without the loop? I assume that the difference would be noticeable (correct me if I'm wrong).


title: Pytorch Linear Layer now automatically reshape the input?

body:
I remember in the past, nn.Linear only accepts 2D tensors.

But today, I discover that nn.Linear now accepts 3D, or even tensors with arbitrary dimensions.

X = torch.randn((20,20,20,20,10))
linear_layer = nn.Linear(10,5)
output = linear_layer(X)
print(output.shape)
>>> torch.Size([20, 20, 20, 20, 5])

When I check the documentation for Pytorch, it does say that it now takes

  Input: :math:(N, *, H_{in}) where :math:* means any number of
            additional dimensions and :math:H_{in} = \text{in\_features}

So it seems to me that Pytorch nn.Linear now reshape the input by x.view(-1, input_dim) automatically.

But I cannot find any x.shape or x.view in the source code:

class Linear(Module):
    __constants__ = ['bias']

    def __init__(self, in_features, out_features, bias=True):
        super(Linear, self).__init__()
        self.in_features = in_features
        self.out_features = out_features
        self.weight = Parameter(torch.Tensor(out_features, in_features))
        if bias:
            self.bias = Parameter(torch.Tensor(out_features))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self):
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in)
            init.uniform_(self.bias, -bound, bound)

    @weak_script_method
    def forward(self, input):
        return F.linear(input, self.weight, self.bias)

    def extra_repr(self):
        return 'in_features={}, out_features={}, bias={}'.format(
            self.in_features, self.out_features, self.bias is not None
        )

Can anyone confirms this?


title: How to specify pytorch as a package requirement on windows?

body:
I have a python package which depends on pytorch and which I’d like windows users to be able to install via pip (the specific package is: https://github.com/mindsdb/lightwood, but I don’t think this is very relevant to my question).

What are the best practices for going about this ?

Are there some project I could use as examples ?

It seems like the pypi hosted version of torch & torchvision aren’t windows compatible and the “getting started” section suggests installing from the custom pytorch repository, but beyond that I’m not sure what the ideal solution would be to incorporate this as part of a setup script.


title: Pytorch super(class, self).__init__() vs super().__init__()

body:
So far out of all the paper and its corresponding code I see people use super(class, self).__init__() instead of super().__init__() within their def __init__ function regardless of which python version people are using. Why? I thought they are the same thing but different syntax for different version of python. Am I paranoid or is there an actual reason to use the old format in pytorch?

Edited: It's for backward compatibility and nothing else.


title: pytorch model.cuda() runtime error

body:
I'm building a text classifier using pytorch, and got into some trouble with .cuda() method. I know that .cuda() moves all parameters into gpu so that the training procedure can be faster. However, error occurred in .cuda() method like this:

start_time  = time.time()

for model_type in ('lstm',):

    hyperparam_combinations = score_util.all_combination(hyperparam_dict[model_type].values())
     # for selecting best scoring model

    for test_idx, setting in enumerate(hyperparam_combinations):
        args = custom_dataset.list_to_args(setting,model_type=model_type)
        print(args)
        tsv = "test %d\ttrain_loss\ttrain_acc\ttrain_auc\tval_loss\tval_acc\tval_auc\n"%(test_idx) # tsv record
        avg_score = [] # cv_mean score

        ### 4 fold cross validation
        for cv_num,(train_iter,val_iter) in enumerate(cv_splits):

            ### model initiation
            model = model_dict[model_type](args)

            if args.emb_type is not None: # word embedding init
                emb = emb_dict[args.emb_type]
                emb = score_util.embedding_init(emb,tr_text_field,args.emb_type)
                model.embed.weight.data.copy_(emb)

            model.cuda()

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-20-ff6cfce73c10> in <module>()
     23                 model.embed.weight.data.copy_(emb)
     24
---> 25             model.cuda()
     26
     27             optimizer= torch.optim.Adam(model.parameters(),lr=args.lr)

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in cuda(self, device_id)
    145                 copied to that device
    146         """
--> 147         return self._apply(lambda t: t.cuda(device_id))
    148
    149     def cpu(self, device_id=None):

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _apply(self, fn)
    116     def _apply(self, fn):
    117         for module in self.children():
--> 118             module._apply(fn)
    119
    120         for param in self._parameters.values():

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in _apply(self, fn)
    122                 # Variables stored in modules are graph leaves, and we don't
    123                 # want to create copy nodes, so we have to unpack the data.
--> 124                 param.data = fn(param.data)
    125                 if param._grad is not None:
    126                     param._grad.data = fn(param._grad.data)

RuntimeError: Variable data has to be a tensor, but got torch.cuda.FloatTensor

These are error traceback and I can't see why this happens.
This code worked very well before I set epoch parameter to 1 to run some tests. I set epoch to 1000 again, but the problem lingers on.
Aren't torch.cuda.FloatTensor object also Tensors? Any help would be much appreciated.

my model looks like this :

class TR_LSTM(nn.Module):
    def __init__(self,args,
                 use_hidden_average=False,
                 pretrained_emb = None):

        super(TR_LSTM,self).__init__()
        # arguments
        self.emb_dim = args.embed_dim
        self.emb_num = args.embed_num
        self.num_hidden_unit = args.hidden_state_dim
        self.num_lstm_layer = args.num_lstm_layer
        self.use_hidden_average = use_hidden_average
        self.batch_size = args.batch_size

        # layers
        self.embed = nn.Embedding(self.emb_num, self.emb_dim)
        if pretrained_emb is not None:
            self.embed.weight.data.copy_(pretrained_emb)

        self.lstm_layer = nn.LSTM(self.emb_dim, self.num_hidden_unit, self.num_lstm_layer, batch_first = True)
        self.fc_layer = nn.Sequential(nn.Linear(self.num_hidden_unit,self.num_hidden_unit),
                                      nn.Linear(self.num_hidden_unit,2))

    def forward(self,x):
        x = self.embed(x) # batch * max_seq_len * emb_dim
        h_0,c_0 = self.init_hidden(x.size(0))
        x, (_, _) = self.lstm_layer(x, (h_0,c_0)) # batch * seq_len * hidden_unit_num

        if not self.use_hidden_average:
            x = x[:,x.size(1)-1,:]
            x = x.squeeze(1)
        else:
            x = x.mean(1).squeeze(1)
        x = self.fc_layer(x)

        return x

    def init_hidden(self,batch_size):
        h_0, c_0 = torch.zeros(self.num_lstm_layer,batch_size , self.num_hidden_unit),\
                   torch.zeros(self.num_lstm_layer,batch_size , self.num_hidden_unit)
        h_0, c_0 = h_0.cuda(), c_0.cuda()
        h_0_param, c_0_param = torch.nn.Parameter(h_0), torch.nn.Parameter(c_0)
        return h_0_param, c_0_param


title: How does DataParallel figure out which gpu I want to use?

body:
I want to find a simple way to specify the gpus that my experiments run on. Currently, I know I can use prepend my python command with CUDA_VISIBLE_DEVICES=1,2,3,4 to set the gpu, and I am guessing DataParallel will then try to use all the gpu.

Is there a way to tell DataParallel directly the ids, like 4,7,9,12?


title: How to balance unbalanced data in PyTorch with WeightedRandomSampler?

body:
I have a 2-class problem and my data is unbalanced.
class 0 has 232550 samples and class 1 has 13498 samples.
PyTorch docs and the internet tells me to use the class WeightedRandomSampler for my DataLoader.

I have tried using the WeightedRandomSampler but I keep getting errors.

    trainratio = np.bincount(trainset.labels) #trainset.labels is a list of
    float [0,1,0,0,0,...]
    classcount = trainratio.tolist()
    train_weights = 1./torch.tensor(classcount, dtype=torch.float)
    train_sampleweights = train_weights[trainset.labels]
    train_sampler = WeightedRandomSampler(weights=train_sampleweights,
                                 num_samples=len(train_sampleweights))
    trainloader = DataLoader(trainset, sampler=train_sampler,
                                       shuffle=False)

Some dimensions I printed out:

train_weights = tensor([4.3002e-06, 4.3002e-06, 4.3002e-06,  ...,
4.3002e-06, 4.3002e-06, 4.3002e-06])

train_weights shape=  torch.Size([246048])

I can't see why I'm getting this error:

UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.weights = torch.tensor(weights, dtype=torch.double)

I have tried other similar workarounds but so far all attempts produce some error.
How should I implement this to balance my train, validation and test data?


title: Pytorch: how to make the trainloader use a specific amount of images?

body:
Assume I am using the following calls:

trainset = torchvision.datasets.ImageFolder(root="imgs/", transform=transform)
trainloader = torch.utils.data.DataLoader(trainset,batch_size=4,suffle=True,num_workers=1)

As far as I can tell, this defines the trainset as consisting of all the images in the folder "images", with labels as defined by the specific folder location.

My question is -
Is there any direct/easy way to define the trainset to be a sub-sample of the images in this folder? For example, define trainset to be a random sample of 10 images from every sub-folder?

Thanks in advance


title: What is the backward process of max operation in deep learning?

body:
I know that the backward process of deep learning follows the gradient descent algorithm. However, there is never a gradient concept for max operation.

How does deep learning frameworks like tensorflow, pytorch deal with the backward of 'max' operation like maxpooling?


title: Using pre-trained word embeddings - how to create vector for unknown / OOV Token?

body:
I wan't to add pre-trained embeddings to a model. But as it seems there is no out-of-vocabulary (OOV) token resp. no vector for unseen words existent.

So what can I do to handle OOV-tokens I come across? I have some ideas, but none of them seem to be very good:

I could just create a random vector for this token, but ideally I'd like the vector to within the logic of the existing model. If I just create it randomly I'm afraid the vector accidentally could be very similar to a very frequent word like 'the', 'for', 'that' etc. which is not my intention.
Or should I just initialize the vector with plain zeros instead?
Another idea would be averaging the token over other existing vectors. But averaging on what vectors then? On all? This doesn't seem to be very conclusive either.
I also thought about trying to train this vector. However this doesn't come very handy if I want to freeze the rest of the embedding during training.

(A general solution is appreciated, but I wanted to add that I'm using PyTorch - just in case PyTorch already comes with a handy solution to this problem.)

So what would be a good and easy strategy to create such a vector?


title: Pytorch and data augmentation: how to augmentate data with blur, rotations, etc

body:
I want to do some data augmentation with Pytorch, but i don't know the libraries very well:
I tried this:
def gaussian_blur(img):
    image = np.array(img)
    image_blur = cv2.GaussianBlur(image,(65,65),10)
    new_image = image_blur
    im = Image.fromarray(new_image)
    return im

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomRotation([-8,+8]),
        transforms.Lambda(gaussian_blur),
        transforms.ColorJitter(brightness=0, contrast=0.4, saturation=0, hue=0),
        transforms.Compose([transforms.Lambda(lambda x : x + torch.randn_like(x))]),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.RandomRotation([-8,+8]),
        transforms.Lambda(gaussian_blur),
        transforms.ColorJitter(brightness=0, contrast=0.4, saturation=0, hue=0),
        transforms.Compose([transforms.Lambda(lambda x : x + torch.randn_like(x))]),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

Because the effects i want to do are: gaussian blur/rotation/contrast/gamma+random noise
But i have errors considering several aspects, like the size of the images doesn't match.
Any suggestions?


title: HTTPError: HTTP Error 403: Forbidden on Google Colab

body:
I am trying to download MNIST data in PyTorch using the following code:

train_loader = torch.utils.data.DataLoader(
      datasets.MNIST('data',
                      train=True,
                      download=True,
                      transform=transforms.Compose([
                         transforms.ToTensor(),
                         transforms.Normalize((0.1307,), (0.3081,))
                     ])),
      batch_size=128, shuffle=True)

and it gives the following error.

Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to data/MNIST/raw/train-images-idx3-ubyte.gz
0it [00:00, ?it/s]
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-2-2fee284dabb8> in <module>()
      5                       transform=transforms.Compose([
      6                          transforms.ToTensor(),
----> 7                          transforms.Normalize((0.1307,), (0.3081,))
      8                      ])),
      9       batch_size=128, shuffle=True)

11 frames
/usr/lib/python3.6/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    648 class HTTPDefaultErrorHandler(BaseHandler):
    649     def http_error_default(self, req, fp, code, msg, hdrs):
--> 650         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    651
    652 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

How do I solve this? The notebook was working before, I'm trying to rerun it but I got this error.


title: Is a torch.FloatTensor not a Tensor?

body:
Although this example has no training, this is an adapted part of a larger program where training does occur. I simply want the generator network to spew out a random image in this case:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
from torchvision import transforms
from PIL import Image

class Generator(nn.Module):

    def __init__(self):
        """
        Generator component of GAN. requires an input slightly bigger
        than 300 x 300 (precisely 308 x 308)
        """
        super(Generator, self).__init__()

        # 5 x 5 square convolution.
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 4, 5)

    def forward(self, x):
        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        return x

def main():
    # Generate example image.
    generator = Generator()
    img = generator(Variable(torch.randn(1, 3, 308, 308))).data
    img_pil = transforms.ToPILImage()(img)
    img_pil.save("test.png")

if __name__ == "__main__":
    main()

Running this program gives the following:

(mgan-Csuh5VLx) ➜  mgan git:(broken) ✗ python test.py
Traceback (most recent call last):
  File "test.py", line 34, in <module>
    main()
  File "test.py", line 30, in main
    img_pil = transforms.ToPILImage()(img)
  File "/home/christopher/.local/share/virtualenvs/mgan-Csuh5VLx/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 92, in __call__
    return F.to_pil_image(pic, self.mode)
  File "/home/christopher/.local/share/virtualenvs/mgan-Csuh5VLx/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 96, in to_pil_image
    raise TypeError('pic should be Tensor or ndarray. Got {}.'.format(type(pic)))
TypeError: pic should be Tensor or ndarray. Got <class 'torch.FloatTensor'>.

I thought FloatTensors were essentially Tensors. Is there a way to get around this error?

(Note: I have four output channels for RGBA, but even switching to 3 output channels gives the same error.)


title: Updating a BERT model through Huggingface transformers

body:
I am attempting to update the pre-trained BERT model using an in house corpus. I have looked at the Huggingface transformer docs and I am a little stuck as you will see below.My goal is to compute simple similarities between sentences using the cosine distance but I need to update the pre-trained model for my specific use case.

If you look at the code below, which is precisely from the Huggingface docs. I am attempting to "retrain" or update the model and I assumed that special_token_1 and special_token_2 represent "new sentences" from my "in house" data or corpus. Is this correct? In summary, I like the already pre-trained BERT model but I would like to update it or retrain it using another in house dataset. Any leads will be appreciated.

import tensorflow as tf
import tensorflow_datasets
from transformers import *

model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

SPECIAL_TOKEN_1="dogs are very cute"
SPECIAL_TOKEN_2="dogs are cute but i like cats better and my
brother thinks they are more cute"

tokenizer.add_tokens([SPECIAL_TOKEN_1, SPECIAL_TOKEN_2])
model.resize_token_embeddings(len(tokenizer))
#Train our model
model.train()
model.eval()


title: What is a dimensional range of [-1,0] in Pytorch?

body:
So I'm struggling to understand some terminology about collections in Pytorch. I keep running into the same kinds of errors about the range of my tensors being incorrect, and when I try to Google for a solution often the explanations are further confusing.

Here is an example:

m = torch.nn.LogSoftmax(dim=1)
input = torch.tensor([0.3300, 0.3937, -0.3113, -0.2880])
output = m(input)

I don't see anything wrong with the above code, and I've defined my LogSoftmax to accept a 1 dimensional input. So according to my experience with other programming languages the collection [0.3300, 0.3937, -0.3113, -0.2880] is a single dimension.

The above triggers the following error for m(input):

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

What does that mean?

I passed in a one dimensional tensor, but then it tells me that it was expecting a range of [-1, 0], but got 1.

A range of what?
Why is the error comparing a dimension of 1 to [-1, 0]?
What do the two numbers [-1, 0] mean?

I searched for an explanation for this error, and I find things like this link which make no sense to me as a programmer:

https://github.com/pytorch/pytorch/issues/5554#issuecomment-370456868

So I was able to fix the above code by adding another dimension to my tensor data.

m = torch.nn.LogSoftmax(dim=1)
input = torch.tensor([[-0.3300, 0.3937, -0.3113, -0.2880]])
output = m(input)

So that works, but I don't understand how [-1,0] explains a nested collection.

Further experiments showed that the following also works:

m = torch.nn.LogSoftmax(dim=1)
input = torch.tensor([[0.0, 0.1], [1.0, 0.1], [2.0, 0.1]])
output = m(input)

So dim=1 means a collection of collections, but I don't understand how that means [-1, 0].

When I try using LogSoftmax(dim=2)

m = torch.nn.LogSoftmax(dim=2)
input = torch.tensor([[0.0, 0.1], [1.0, 0.1], [2.0, 0.1]])
output = m(input)

The above gives me the following error:

  IndexError: Dimension out of range (expected to be in range of [-2, 1], but got 2)

Confusion again that dim=2 equals [-2, 1], because where did the 1 value come from?

I can fix the error above by nesting collections another level, but at this point I don't understand what values LogSoftmax is expecting.

m = torch.nn.LogSoftmax(dim=2)
input = torch.tensor([[[0.0, 0.1]], [[1.0, 0.1]], [[2.0, 0.1]]])
output = m(input)

I am super confused by this terminology [-1, 0] and [-2, 1]?

If the first value is the nested depth, then why is it negative and what could the second number mean?

There is no error code associated with this error. So it's been difficult to find documentation on the subject. It appears to be an extremely common error people get confused by and nothing that I can find in the Pytorch documentation that talks specifically about it.


title: in pytorch, cuda.is_availbale(), but every operation fails with out of memory

body:
I've set up a new virtual machine (on GCP) with a K80 GPU on Ubuntu 16.
Followed installation instructions for the CUDA toolkit 9.1

nvidia-smi returns correctly:
NVIDIA-SMI 390.12                 Driver Version: 390.12
...

in pytorch, cuda.is_available() returns True,
but any operation fails:

torch.randn(10).cuda() ==>

RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-
bld/pytorch_1512378422383/work/torch/lib/THC/generic/THCStorage.cu:58

What am I missing in the installation?


title: Pytorch Test Loss increases while accuracy increases

body:
I am trying to implement
End to End Memory Network using Pytorch and BabI dataset.
The network architecture is :

MemN2N (
(embedding_A): Embedding(85, 120, padding_idx=0)
(embedding_B): Embedding(85, 120, padding_idx=0)
(embedding_C): Embedding(85, 120, padding_idx=0)
(match): Softmax ()
)

85 is the vocabulary size and 120 is embedding size.
Loss function is cross entropy and optimizer is RmsProp.
 The results is

Epoch    Train Loss  Test Loss  TrainAcc TestAcc
10       0.608         11.213       1.0     0.99
20       0.027         11.193       1.0     0.99
30       0.0017        11.740       1.0     0.99
40       0.0006        12.190       1.0     0.99
50       5.597e-05     12.319       1.0     0.99
60       3.366-05      12.379       1.0     0.99
70       2.72e-05      12.361       1.0     0.99
80       2.64e-05      12.333       1.0     0.99
90       2.63e-05      12.329       1.0     0.99
100      2.63e-05      12.329       1.0     0.99
110      2.63e-05      12.329       1.0     0.99
120      2.63e-05      12.329       1.0     0.99
Final TrainAcc TestAcc
    1.0     0.999

I know the accuracy is good, but I wonder the behaviour of the test loss.
Since training loss decreases, the test loss increases.
The calculation is the same for each loss value. Shouldn't it decrease too? I used Task 1 to display, but the behaviour is the same with other tasks.

Do you have any idea about this behaviour?


title: Accumulating Gradients

body:
I want to accumulate the gradients before I do a backward pass. So wondering what the right way of doing it is. According to this article
it's:

model.zero_grad()                                   # Reset gradients tensors
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss = loss_function(predictions, labels)       # Compute loss function
    loss = loss / accumulation_steps                # Normalize our loss (if averaged)
    loss.backward()                                 # Backward pass
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()

whereas I expected it to be:

model.zero_grad()                                   # Reset gradients tensors
loss = 0
for i, (inputs, labels) in enumerate(training_set):
    predictions = model(inputs)                     # Forward pass
    loss += loss_function(predictions, labels)       # Compute loss function
    if (i+1) % accumulation_steps == 0:             # Wait for several backward steps
        loss = loss / accumulation_steps            # Normalize our loss (if averaged)
        loss.backward()                             # Backward pass
        optimizer.step()                            # Now we can do an optimizer step
        model.zero_grad()
        loss = 0

where I accumulate the loss and then divide by the accumulation steps to average it.

Secondary question, if I am right, would you expect my method to be quicker considering I only do the backward pass every accumulation steps?


title: Pytorch based Resnet18 achieves low accuracy on CIFAR100

body:
I'm training a resnet18 on CIFAR100 dataset. After about 50 iterations the validation accuracy converged at about 34%. While the training accuracy reached almost 100%.
I doubt it's kinda overfitting, so i applied data augmentation like RandomHorizontalFlip and RandomRotation, which made the validation converge at about 40%.
I also tried decaying learning rate [0.1, 0.03, 0.01, 0.003, 0.001], decaying after each 50 iterations. Decaying learning rate seems not enhancing the performance.
Heard that Resnet on CIFAR100 may get 70%~80% accuracy. What trick else could I apply? Or is there anything wrong in my implementation? The same code on CIFAR10 can achieve about 80% accuracy.
My whole training and evaluation code is here below:
import torch
from torch import nn
from torch import optim
from torch.utils.data import DataLoader
from torchvision.models import resnet18
from torchvision.transforms import Compose, ToTensor, RandomHorizontalFlip, RandomRotation, Normalize
from torchvision.datasets import CIFAR10, CIFAR100
import os
from datetime import datetime
import matplotlib.pyplot as plt

def draw_loss_curve(histories, legends, save_dir):
    os.makedirs(save_dir, exist_ok=True)
    for key in histories[0][0].keys():
        if key != "epoch":
            plt.figure()
            plt.title(key)
            for history in histories:
                x = [h["epoch"] for h in history]
                y = [h[key] for h in history]
                # plt.ylim(ymin=0, ymax=3.0)
                plt.plot(x, y)
            plt.legend(legends)
            plt.savefig(os.path.join(save_dir, key + ".png"))

def cal_acc(out, label):
    batch_size = label.shape[0]
    pred = torch.argmax(out, dim=1)
    num_true = torch.nonzero(pred == label).shape[0]
    acc = num_true / batch_size
    return torch.tensor(acc)

class LrManager(optim.lr_scheduler.LambdaLR):
    def __init__(self, optimizer, lrs):
        def f(epoch):
            rate = 1
            for k in sorted(lrs.keys()):
                if epoch >= k:
                    rate = lrs[k]
                else:
                    break
            return rate
        super(LrManager, self).__init__(optimizer, f)

def main(cifar=100, epochs=250, batches_show=100):
    if torch.cuda.is_available():
        device = "cuda"
    else:
        device = "cpu"
        print("warning: CUDA is not available, using CPU instead")
    dataset_cls = CIFAR10 if cifar == 10 else CIFAR100
    dataset_train = dataset_cls(root=f"data/{dataset_cls.__name__}/", download=True, train=True,
                                transform=Compose([RandomHorizontalFlip(), RandomRotation(15), ToTensor(), Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))]))
    dataset_val = dataset_cls(root=f"data/{dataset_cls.__name__}/", download=True, train=False,
                              transform=Compose([ToTensor(), Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))]))
    loader_train = DataLoader(dataset_train, batch_size=128, shuffle=True)
    loader_val = DataLoader(dataset_val, batch_size=128, shuffle=True)
    model = resnet18(pretrained=False, num_classes=cifar).to(device)
    optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9, weight_decay=1e-5)
    lr_scheduler = LrManager(optimizer, {0: 1.0, 50: 0.3, 100: 0.1, 150: 0.03, 200: 0.01})
    criterion = nn.CrossEntropyLoss()

    history = []
    model.train()
    for epoch in range(epochs):
        print("-------------------  TRAINING  -------------------")
        loss_train = 0.0
        running_loss = 0.0
        acc_train = 0.0
        running_acc = 0.0
        for batch, data in enumerate(loader_train, 1):
            img, label = data[0].to(device), data[1].to(device)
            optimizer.zero_grad()
            pred = model(img)
            loss = criterion(pred, label)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()
            loss_train += loss.item()
            acc = cal_acc(pred, label)
            running_acc += acc.item()
            acc_train += acc.item()

            if batch % batches_show == 0:
                print(f"epoch: {epoch}, batch: {batch}, loss: {running_loss/batches_show:.4f}, acc: {running_acc/batches_show:.4f}")
                running_loss = 0.0
                running_acc = 0.0
        loss_train = loss_train / batch
        acc_train = acc_train / batch
        lr_scheduler.step()

        print("------------------- EVALUATING -------------------")
        with torch.no_grad():
            running_acc = 0.0
            for batch, data in enumerate(loader_val, 1):
                img, label = data[0].to(device), data[1].to(device)
                pred = model(img)
                acc = cal_acc(pred, label)
                running_acc += acc.item()
            acc_val = running_acc / batch
            print(f"epoch: {epoch}, acc_val: {acc_val:.4f}")

        history.append({"epoch": epoch, "loss_train": loss_train, "acc_train": acc_train, "acc_val": acc_val})
    draw_loss_curve([history], legends=[f"resnet18-CIFAR{cifar}"], save_dir=f"history/resnet18-CIFAR{cifar}[{datetime.now()}]")

if __name__ == '__main__':
    main()


title: How to permanently install package in GoogleColab using conda?

body:
I am trying yo use a PyTorch library SparseConvNet (https://github.com/facebookresearch/SparseConvNet) in Google Colaboratory. In order to install it properly, you need to first install Conda, and then using Conda install the SparseConvNet package. Here is the code I am using (following the instructions from scn readme file):

!wget -c https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
!chmod +x Anaconda3-5.1.0-Linux-x86_64.sh
!bash ./Anaconda3-5.1.0-Linux-x86_64.sh -b -f -p /usr/local

import sys
sys.path.append('/usr/local/lib/python3.6/site-packages/')

!conda install pytorch torchvision cudatoolkit=10.0 -c pytorch
!conda install google-sparsehash -c bioconda
!conda install -c anaconda pillow

!git clone https://github.com/facebookresearch/SparseConvNet.git
!cd SparseConvNet/
!bash develop.sh

When I run this it is working and I can successfully import sparseconvnet package, but I need to do it every time I enter the notebook or restart  runtime, and it's taking a lot of time. Is it possible to install these packages permanently?

There is one similar question, and the answer suggest that I should install it
on my drive, but I don't know how to do it using conda.
Thanks!


title: Is there a way to do Pytorch element wise equality treating each dimension as an element?

body:
I have two tensors and I want to check for equality treating an array in one dimension as the element

I have 2 tensors

lo = torch.Tensor(([1., 1., 0.],
                   [0., 1., 1.],
                   [0., 0., 0.],
                   [1., 1., 1.]))
lo = torch.Tensor(([1., 1., 0.],
                   [0., 0., 0.],
                   [0., 0., 0.],
                   [0., 0., 0.]))

I've tried using
torch.eq(lee, lo)
which returns a tensor like

tensor([[1, 1, 1],
        [1, 0, 0],
        [1, 1, 1],
        [0, 0, 0]], dtype=torch.uint8)

Is there a way to have the output become

tensor([1, 0, 1, 0])

as the only complete element that matches is the first?

edit:
I've come up with this solution

lee = lee.tolist()
lo = lo.tolist()
out = []
for i, j in enumerate(lee):
  if j == lo[i]:
    out.append(1)
  else:
    out.append(0)

and out will be [1, 0, 1, 0]
But is there an easier way?


title: Pytorch - Incorrect dimensions when using LSTM network

body:
I'm getting started with pytorch and used a few transformation to build the following model using one of the tutorials as a reference:

model = torch.nn.Sequential(
     torch.nn.Linear(D_in, H),
     torch.nn.ReLU(),
     torch.nn.Linear(H, D_out),
)

I want to use an LSTM network, so I tried to do the following:

model = torch.nn.Sequential(
      torch.nn.LSTM(D_in, H),
      torch.nn.Linear(H, D_out)
)

which gives me this error:

  RuntimeError: input must have 3 dimensions, got 2

Why am I seeing this error? I anticipate there's something fundamentally wrong in my understanding of how transformations (networks?) can be chained in pytorch...

EDIT

After following @esBee's suggestion, I found that following runs correctly. This is because an LSTM expects the input to be of the folowing dimension:

  input of shape (seq_len, batch, input_size): tensor containing the features of the input sequence. The input can also be a packed variable length sequence

local_x = local_x.unsqueeze(0)
y_pred, (hn, cn) = layerA(local_x)
y_pred = y_pred.squeeze(0)
y_pred = layerB(y_pred)

However, the fact that my original training/test dataset is only of sequence length 1 makes me feel like I'm doing something incorrectly. What is the purpose of this parameter in the context of neural networks?


title: Pytorch .to('cuda') or .cuda() doesn't work and just get stuck

body:
I am trying to do pytorch tutorial. When I try to set their device as cuda, it does not work and my code running get stuck.

For specific information, I am using conda environment of
python 3.7.3
pytorch 1.3.0
cuda 10.2 (NVIDIA RTX2080TI)

>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.device_count()
1
>>> torch.cuda.current_device()
0
>>> device = torch.device('cuda:0')
>>> device
device(type='cuda', index=0)
>>> aa = torch.randn(5)
>>> aa = tensor([-2.2084, -0.2700, 0.0921, -1.7678, 0.7642])
>>> aa.to(device)

nothing happens...

Can anybody please help me how to overcome this problem?


title: Setting up VGG-Face Descriptor in PyTorch

body:
I've been trying to use the VGG-Face descriptor model (http://www.robots.ox.ac.uk/~vgg/software/vgg_face/) for a project of mine. All I want to do is to simply obtain the outputs of the network from an input image.

I haven't used any of MatConvNet, Caffe or PyTorch before and so I picked PyTorch at random. It turns out that the model (of class torch.legacy.nn.Sequential.Sequential) was saved in an older version of PyTorch and the syntax was thus slightly different to the ones on PyTorch's documentation.

I was able to load the lua .t7 model like so:

vgg_net = load_lua('./vgg_face_torch/VGG_FACE.t7', unknown_classes=True)

And loading in the input image:

# load image
image = imread('./ak.png')
# convert to tensor
input = torch.from_numpy(image).float()

Gleefully, I loaded in the image into the model with much anticipation:

# load into vgg_net
output = vgg_net.forward(input)

However, my hopes of it cooperating at all was quickly dashed when the code fails to compile. Leaving behind a cryptic error message:

Traceback (most recent call last):
  File "~/Documents/python/vgg-face-test/vgg-pytorch.py", line 25, in <module>
    output = vgg_net.forward(input)
  File "~/.local/lib/python3.6/site-packages/torch/legacy/nn/Module.py", line 33, in forward
    return self.updateOutput(input)
  File "~/.local/lib/python3.6/site-packages/torch/utils/serialization/read_lua_file.py", line 235, in updateOutput_patch
    return obj.updateOutput(*args)
  File "~/.local/lib/python3.6/site-packages/torch/legacy/nn/Sequential.py", line 36, in updateOutput
    currentOutput = module.updateOutput(currentOutput)
TypeError: 'NoneType' object is not callable

Of which I am absolutely dumbfounded by.

This is why I sought help on Stackoverflow. I hope someone here could perhaps lend me a hand in setting up the model - not even necessarily in Torch, in fact any working model will do, where I can simply get the description for any particular image.


title: Are there any computational efficiency differences between nn.functional() Vs nn.sequential() in PyTorch

body:
The following is a Feed-forward network using the nn.functional() module in PyTorch

import torch.nn as nn
import torch.nn.functional as F

class newNetwork(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 128)
        self.fc2 = nn.Linear(128, 64)
        self.fc3 = nn.Linear(64,10)

    def forward(self,x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.softmax(self.fc3(x))
        return x

model = newNetwork()
model

The following is the same Feed-forward using nn.sequential() module to essentially build the same thing. What is the difference between the two and when would i use one instead of the other?

input_size = 784
hidden_sizes = [128, 64]
output_size = 10

Build a feed-forward network

 model = nn.Sequential(nn.Linear(input_size, hidden_sizes[0]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[0], hidden_sizes[1]),
                      nn.ReLU(),
                      nn.Linear(hidden_sizes[1], output_size),
                      nn.Softmax(dim=1))
    print(model)


title: Siamese Neural Network in Pytorch

body:
How can I implement a siamese neural network in PyTorch?

  What is a siamese neural network? A siamese neural network consists in two identical neural networks, each one taking one input. Identical means that the two neural networks have the exact same architecture and share the same weights.


title: pytorch model summary - forward func has more than one argument

body:
I am using torch summary

from torchsummary import summary

I want to pass more than one argument when printing the model summary, but the examples mentioned here: Model summary in pytorch taken only one argument. for e.g.:

model = Network().to(device)
summary(model,(1,28,28))

The reason is that the forward function takes two arguments as input, e.g.:

def forward(self, img1, img2):

How do I pass two arguments here?


title: Using a generator with pickled data in a Dataloader for PyTorch

body:
I have done some preprocessing and feature selection before, and I have a pickle training input data that consists of list of lists, e.g. (but pickled)

[[1,5,45,13], [23,256,4,2], [1,12,88,78], [-1]]
[[12,45,77,325], [23,257,5,28], [3,7,48,178], [12,77,89,99]]
[[13,22,78,89], [12,33,97], [-1], [-1]]

[-1] is a padding token, but I don't think that matters.

Because the file is numerous of gigabytes, I wish to spare memory and use a generator to read in the pickle line by line (list by list). I already found this answer that could be helpful. This would like follows:

def yield_from_pickle(pfin):
    with open(pfin, 'rb') as fhin:
        while True:
            try:
                yield pickle.load(fhin)
            except EOFError:
                break

The next thing is, that I wish to use this data in a PyTorch (1.0.1) Dataloader. From what I found in other answers, I must feed it a Dataset which you can subset, but which must contain __len__ and __getitem__. It could like this:

class TextDataset(Dataset):
    def __init__(self, pfin):
        self.pfin = pfin

    def __len__(self):
        # memory-lenient way but exhaust generator?
        return sum(1 for _ in self.yield_from_pickle())

    def __getitem__(self, index):
        # ???
        pass

    def yield_from_pickle(self):
        with open(self.pfin, 'rb') as fhin:
            while True:
                try:
                    yield pickle.load(fhin)
                except EOFError:
                    break

But I am not at all sure if this is even possible. How can I implement __len__ and __getitem__ in a sensible way? I don't think what I am doing with __len__ is a good idea because that'll exhaust the generator, and I have no idea at all how to safely implement __getitem__ while retaining the generator.

Is there a better way? To summarize, I want to build a Dataset that can be fed to PyTorch's Dataloader (because of its multiprocessing abilities) but in a memory-efficient way where I don't have to read the whole file into memory.


title: is crossentropy loss of pytorch different than "categorical_crossentropy" of keras?

body:
I am trying to mimic a pytorch neural network in keras.

I am confident that my keras version of the neural network is very close to the one in pytorch but during training, I see that the loss value of the pytorch network are much lower than the loss values of the keras network. I wonder if this is because I have not properly copied the pytorch network in keras or the loss computation is different in the two framework.

Pytorch loss definition:

loss_function = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=args.lr, momentum=0.9, weight_decay=5e-4)

Keras loss definition:

sgd = optimizers.SGD(lr=.1, momentum=0.9, nesterov=True)
resnet.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=['categorical_accuracy'])

Note that all the layers in the keras network have been implemented with L2 regularization kernel_regularizer=regularizers.l2(5e-4), also I used he_uniform initialization which I believe is default in pytorch, according to the source code.

The batch size for the two networks are the same: 128.

In the pytorch version, I get loss values around 4.1209 which decreases to around 0.5. In keras it starts around 30 and decreases to 2.5.


title: Pytorch: set indices in a particular Tensor dimension (similar to torch.index_select)

body:
I'm trying to get and set indices in a particular tensor dimension without reshaping if possible. I've been able to find the torch.index_select function which does what I want when getting values, but I haven't found an analogous setter function yet. Does one exist?

For context I have a tensor and a set of indexes

class_energy = torch.rand(3, 10, 32, 32)
class_logits = torch.empty_like(class_energy)
idxs = [2, 3, 5, 7]

I want to access items at those indices in a particular dimension so I can perform a log_softmax.

If I knew dim a-priori then I could simply use the fancy __getitem__ / __setitem__ syntax: For example, if dim=1, then class_energy[:, idxs]. Likewise, if dim=2 -> class_energy[:, :, idxs], dim=0 -> class_energy[idxs], etc...

In the case where dim=1, I essentially want this:

class_logits[:, idxs] = F.log_softmax(class_energy[:, idxs], dim=1)

Unfortunately, I don't know the value of dim ahead of time. Sure I could build up the fancy index ahead of time via:

fancy_index = tuple([slice(None)] * dim + [idxs])
class_logits[fancy_index] = F.log_softmax(class_energy[fancy_index], dim=dim)

However, I'm wondering if there is a better way to do this. For the case of __getitem__, I know for a fact that there is. The following code using torch.index_select is equivalent

fancy_index = tuple([slice(None)] * dim + [idxs])
index = torch.LongTensor(idxs).to(class_energy.device)
class_logits[fancy_index] = F.log_softmax(torch.index_select(class_energy, dim=dim, index=index, dim=dim))

Not only is it functionally the same, index_select is much faster (I've seen 2x improvement) than using the fancy getitem syntax.

My issue is that I can't seem to similar functionality for the __setitem__ portion of the code. It would be really nice if I could get rid of the fancy index all together. I've looked into Tensor.put_, torch.index_put, and torch.select, but none of these seem to have the functionality I want. Is there something I'm missing, or is the fancy index the only way to solve this problem currently?


title: Does Tensorflow or Pytorch work on RTX 20xx super series

body: I'm going to buy a new gpu to learn deep learning. The new Nvidia RTX 2060 Super seems to best fit my needs. But I wonder that is it compatible with CUDA and tensorflow or pytorch now?


title: How does the parameter 'dim' in torch.unique() work?

body:
I am trying to extract the unique values in each row of a matrix and returning them into the same matrix (with repeated values set to say, 0) For example, I would like to transform

torch.Tensor(([1, 2, 3, 4, 3, 3, 4],
              [1, 6, 3, 5, 3, 5, 4]])

to

torch.Tensor(([1, 2, 3, 4, 0, 0, 0],
              [1, 6, 3, 5, 0, 0, 4]])

or

torch.Tensor(([1, 2, 3, 4, 0, 0, 0],
              [1, 6, 3, 5, 4, 0, 0]])

I.e. the order does not matter in the rows. I have tried using pytorch.unique() and in the documentation it is mentioned that the dimension to take the unique values can be specified with the parameter dim. However, It doesn't seem to work for this case.

I've tried:

output= torch.unique(torch.Tensor([[4,2,52,2,2],[5,2,6,6,5]]), dim = 1)

output

Which gives

tensor([[ 2.,  2.,  2.,  4., 52.],
        [ 2.,  5.,  6.,  5.,  6.]])

Does anyone have a particular fix for this? If possible, I'm trying to avoid for loops.


title: torch NameError: name '_C' is not defined

body:
I'm getting the following error when trying to import torch in Jupyter:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-10-39178> in <module>
      1
----> 2 import torch

~/jupyter_py3/lib/python3.6/site-packages/torch/__init__.py in <module>
     84 from torch._C import *
     85
---> 86 __all__ += [name for name in dir(_C)
     87             if name[0] != '_' and
     88             not name.endswith('Base')]

NameError: name '_C' is not defined

     88             not name.endswith('Base')]

NameError: name '_C' is not defined

Cython is already installed, in case you may be wondering.

Any idea how to fix this?


The Grad sub object becomes "None" if expand the expression. Not sure why? Can somebody give some clue.
If expand the w.grand.zero_() throw error as "AttributeError: 'NoneType' object has no attribute 'zero_'"

Thanks,
Ganesh

import torch

x = torch.randint(size = (1,2), high = 10)
w = torch.Tensor([16,-14])
b = 36

y = w * x + b

epoch = 20
learning_rate = 0.01

w1 = torch.rand(size= (1,2), requires_grad= True)
b1 = torch.ones(size = [1], requires_grad= True)

for i in range(epoch):
    y1 = w1 * x + b1

    loss = torch.sum((y1-y)**2)

    loss.backward()

    with torch.no_grad():
        #w1 = w1 - learning_rate * w1.grad  //Not Working : w1.grad becomes "None" not sure how ;(
        #b1 = b1 - learning_rate * b1.grad

        w1 -= (learning_rate * w1.grad)  // Working code.
        b1 -= (learning_rate * b1.grad)

        w1.grad.zero_()
        b1.grad.zero_()

    print("B ", b1)
    print("W ", w1)


title: How to pass parameters to forward function of my torch nn.module from skorch.NeuralNetClassifier.fit()

body:
I have extended nn.Module to implement my network whose forward function is like this ...

def forward(self, X, **kwargs):

    batch_size, seq_len = X.size()

    length = kwargs['length']
    embedded = self.embedding(X) # [batch_size, seq_len, embedding_dim]
    if self.use_padding:
        if length is None:
            raise AttributeError("Length must be a tensor when using padding")
        embedded = nn.utils.rnn.pack_padded_sequence(embedded, length, batch_first=True)
        #print("Size of Embedded packed", embedded[0].size())

    hidden, cell = self.init_hidden(batch_size)
    if self.rnn_unit == 'rnn':
        out, _ = self.rnn(embedded, hidden)
    elif self.rnn_unit == 'lstm':
        out, (hidden, cell) = self.rnn(embedded, (hidden, cell))

    # unpack if padding was used
    if self.use_padding:
        out, _ = nn.utils.rnn.pad_packed_sequence(out, batch_first = True)

I initialized a skorch NeuralNetClassifier like this,

net = NeuralNetClassifier(
    model,
    criterion=nn.CrossEntropyLoss,
    optimizer=Adam,
    max_epochs=8,
    lr=0.01,
    batch_size=32
)

Now if I call net.fit(X, y, length=X_len) it throws an error

TypeError: __call__() got an unexpected keyword argument 'length'

According to the documentation fit function expects a fit_params dictionary,

**fit_params : dict
   Additional parameters passed to the ``forward`` method of
   the module and to the ``self.train_split`` call.

and the source code always send my parameters to train_split where obviously my keyword argument would not be recognized.

Is there any way around to pass the arguments to my forward function?


title: How to split data into train and test sets using torchvision.datasets.Imagefolder?

body: In my custom dataset, one kind of image is in one folder which torchvision.datasets.Imagefolder can handle, but how to split the dataset into train and test?


title: PyTorch - How to use k-fold cross validation when the data is loaded through ImageFolder?

body: My data, which is images, is stored on the filesystem, and it is fed into my convolutional neural network through the ImageFolder data loader of PyTorch. Therefore, the training, validation, and test data is manually splitted into different folders on the filesystem. So, how can I apply k-fold cross validation when using ImageFolder?


title: Loss function for ordinal multi classification in pytorch

body:
I am a beginner with DNN and pytorch.
I am dealing with a multi-classification problem where my label are encoded into a one-hotted vector, say of dimension D.
To this end, I am using the CrossEntropyLoss. However now I want to modify or change such criterion in order to penalize value distant from the actual one, say classify 4 instead of 5 is better than 2 instead of 5.

Is there a function already built-in in Pytorch that implement this behavior? Otherwise how can I modify the CrossEntropyLoss to achieve it?


title: Pytorch tensor.save() produces huge files for small tensors from MNIST

body:
I'm working with MNIST dataset from Kaggle challange and have troubles preprocessing with data. Furthermore, I don't know what are the best practices and was wondering if you could advise me on that.

Disclaimer: I can't just use torchvision.datasets.mnist because I need to use Kaggle's data for training and submission.

In this tutorial, it was advised to create a Dataset object loading .pt tensors from files, to fully utilize GPU. In order to achieve that, I needed to load the csv data provided by Kaggle and save it as .pt files:

import pandas as pd
import torch
import numpy as np

# import data
digits_train = pd.read_csv('data/train.csv')

train_tensor = torch.tensor(digits_train.drop(label, axis=1).to_numpy(), dtype=torch.int)
labels_tensor = torch.tensor(digits_train[label].to_numpy())

for i in range(train_tensor.shape[0]):
    torch.save(train_tensor[i], "data/train-" + str(i) + ".pt")

Each train_tensor[i].shape is torch.Size([1, 784])

However, each such .pt file has size of about 130MB.
A tensor of the same size, with randomly generated integers, has size of 6.6kB.
Why are these tensors so huge, and how can I reduce their size?

Dataset is 42 000 samples. Should I even bother with batching this data? Should I bother with saving tensors to separate files, rather than loading them all into RAM and then slicing into batches? What is the most optimal approach here?


title: Why am I getting different results after saving and loading model weights in pytorch?

body:
I have written a model, the architecture is follows:

CNNLSTM(
  (cnn): CNNText(
    (embed): Embedding(19410, 300, padding_idx=0)
    (convs1): ModuleList(
      (0): Conv2d(1, 32, kernel_size=(3, 300), stride=(1, 1))
      (1): Conv2d(1, 32, kernel_size=(5, 300), stride=(1, 1))
      (2): Conv2d(1, 32, kernel_size=(7, 300), stride=(1, 1))
    )
    (dropout): Dropout(p=0.6)
    (fc1): Linear(in_features=96, out_features=1, bias=True)
  )
  (lstm): RNN(
    (embedding): Embedding(19410, 300, padding_idx=0)
    (rnn): LSTM(300, 150, batch_first=True, bidirectional=True)
    (attention): Attention(
      (dense): Linear(in_features=300, out_features=1, bias=True)
      (tanh): Tanh()
      (softmax): Softmax()
    )
    (fc1): Linear(in_features=300, out_features=50, bias=True)
    (dropout): Dropout(p=0.5)
    (fc2): Linear(in_features=50, out_features=1, bias=True)
  )
  (fc1): Linear(in_features=146, out_features=1, bias=True)
)

I have used the RNN and the CNN differently on the same dataset and I have the weights saved. In the mixed model, I load the weights using the following function:

def load_pretrained_weights(self, model='cnn', path=None):
    if model not in ['cnn', 'rnn']:
        raise AttributeError("Model must be either rnn or cnn")
    if model == 'cnn':
        self.cnn.load_state_dict(torch.load(path))
    if model == 'rnn':
        self.lstm.load_state_dict(torch.load(path))

And freeze the sub modules using the function:

def freeze(self):
    for p in self.cnn.parameters():
        p.requires_grad = False
    for p in self.lstm.parameters():
        p.requires_grad = False

Then I train the model, and got better result compared to the each submodule trained and
evaluated alone.
I used an early-stopping technique in my epoch loop to save the best parameters.
After training I made a new instance of the same class and when I load the saved “best” parameters I am not getting similar result.
I tried the same thing with each submodule (RNN and CNNText here) alone, it worked. But in this case it is not giving the same performance.

Please help me understand it what is happening here. I am new to Deep Learning concepts.
Thank you.

Few Experiments I tried:

I loaded the saved weights of each submodule and loaded the best parameters, got somehow close to the best result.
Took the hidden layer from each submodule before applying the dropout, got better than the previous, but not the best!

EDIT

The init function of my class is as follows.
And the RNN and CNN are just usual implementations.

class CNNLSTM(nn.Module):

    def __init__(self, vocab_size, embedding_dim, embedding_weight, rnn_arch, isCuda=True, class_num=1, kernel_num=32, kernel_sizes=[3,4,5],train_wv=False, rnn_num_layers=1, rnn_bidirectional=True, rnn_use_attention=True):

        super(CNNLSTM, self).__init__()
        self.cnn = CNNText(vocab_size, embedding_dim, embedding_weight, class_num, kernel_num = kernel_num, kernel_sizes=kernel_sizes, static=train_wv,dropout=0.6)
        self.lstm = RNN(rnn_arch, vocab_size, embedding_dim, embedding_weight, num_layers=rnn_num_layers, rnn_unit='lstm', embedding_train=train_wv, isCuda=isCuda, bidirectional=rnn_bidirectional, use_padding=True, use_attention=rnn_use_attention, num_class=class_num)
        self.fc1 = nn.Linear(rnn_arch[-1] + len(kernel_sizes) * kernel_num , class_num)

After declaring the object

I Loaded individual pre-trained submodule as,

model.load_pretrained_weights('rnn', 'models/bilstm_2_atten.pth')
model.load_pretrained_weights('cnn', 'models/cnn2.pth')

model.freeze()

Then I train the last linear layer.
I saved the model parameter values as

torch.save(model.state_dict(),path)

So at 3rd/4th from last epoch I am getting the 'best' result.
And after training I loaded the parameters for best result with,

state_dict = torch.load(MODEL_PATH)
model.load_state_dict(state_dict)