title: Teacher forcing with pytorch RNN

body:
The pytorch tutorials do a great job of illustrating a bare-bones RNN by defining the input and hidden layers, and manually feeding the hidden layers back into the network to remember the state.  This flexibility then allows you to very easily perform teacher forcing.

Question 1: How do you perform teacher forcing when using the native nn.RNN() module (since the entire sequence is fed at once)?  Example simple RNN network would be:

class SimpleRNN(nn.Module):

    def __init__(self, vocab_size,
                 embedding_dim,
                 batch_sz,
                 hidden_size=128,
                 nlayers=1,
                 num_directions=1,
                 dropout=0.1):

        super(SimpleRNN, self).__init__()

        self.batch_sz = batch_sz
        self.hidden_size = hidden_size

        self.encoder = nn.Embedding(vocab_size, embedding_dim)
        self.rnn = nn.RNN(embedding_dim, hidden_size, nlayers, dropout=0.5)
        self.decoder = nn.Linear(hidden_size, vocab_size)

    def init_hidden(self):
        return autograd.Variable(torch.zeros(nlayers, batch_sz, hidden_size)).cuda()

    def forward(self, inputs, hidden):

        # -- encoder returns:
        # -- [batch_sz, seq_len, embed_dim]
        encoded = self.encoder(inputs)
        _, seq_len, _ = encoded.size()

        # -- rnn returns:
        # -- output.size() = [seq_len, batch_sz, hidden_sz]
        # -- hidden.size() = [nlayers, batch_sz, hidden_sz]
        output, hidden = self.rnn(encoded.view(seq_len, batch_sz, embedding_dim), hidden)

        # -- decoder returns:
        # -- output.size() = [batch_sz, seq_len, vocab_size]
        output = F.log_softmax(decoder(output.view(batch_sz, seq_len, self.hidden_size)))

        return output, hidden

Where I can call the network with:

model = SimpleRNN(vocab_size, embedding_dim, batch_sz).cuda()
x_data, y_data = get_sequence_data(train_batches[0])
output, hidden = model(x_data, model.init_hidden())

Just for completeness, here are my shapes of x_data,  output, and hidden:

print(x_data.size(), output.size(), hidden.size())
torch.Size([32, 80]) torch.Size([32, 80, 4773]) torch.Size([1, 32, 128])

Question 2: would it be possible to use this SimpleRNN network to then generate a sequence word-by-word, by first feeding it a <GO_TOKEN> and iterating until an <END_TOKEN> is reached?  I ask because when I run this:

x_data = autograd.Variable(torch.LongTensor([[word2idx['<GO>']]]), volatile=True).cuda()
output, hidden = model(x_data, model.init_hidden(1))

print(output, output.sum())

I get an output of all 0s, and the output.sum() = 0.  I get this even after training the network and backpropagating the loss.  Any ideas why?

Question 3: If not terribly inefficient, is it possible to train the SimpleRNN network above word-by-word, analogous to the pytorch tutorial shown (here)[http://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html] (albeit there they're training character-by-character).


title: Getting error: only one element tensors can be converted to Python scalars

body:
Could you please help how to solve this problem. Basically, I'm trying to get into Pytorch tensor function data that is vector not scalar. X1 and X2 are basically columns in CSV file that contains many strings.
How to kind of iterate through every single data from x1 and x2 and not just trying to parse the whole vector. I'm a newbie at Python and Pytorch as well.

import torch
import random
import pandas

data = pandas.read_csv('train/train.tsv', sep='\t')

learningrate = torch.tensor(0.01)
W = torch.rand([2, 2], dtype=torch.float, requires_grad=True)
b = torch.rand(2, dtype=torch.float, requires_grad=True)
U = torch.rand(2, dtype=torch.float, requires_grad=True)
c = torch.rand(1, dtype=torch.float, requires_grad=True)

def get_item():
 x1 = torch.tensor(data['Powierzchnia w m2'],
                  dtype=torch.float, requires_grad=True)
 x2 = torch.tensor(data['Liczba pokoi'],
                  dtype=torch.float, requires_grad=True)
 x = torch.tensor([x1, x2], dtype=torch.float)
 yexpected = torch.tensor(data['cena'].values, dtype=torch.float)
 return x, yexpected

for _ in range(100000):

 x, yexpected = get_item()
 h = torch.sigmoid(W @ x+b)

print(x)
print(yexpected)
print(h)
y = torch.sigmoid(U@h+c)
loss = (y-yexpected)**2
print(loss)
loss.backward()
with torch.no_grad():
    W -= learningrate * W.grad
    b -= learningrate * b.grad
    c -= learningrate * c.grad
    U -= learningrate * U.grad
    b.grad.zero_()
    W.grad.zero_()
    c.grad.zero_()
    U.grad.zero_()


title: Convert integer to pytorch tensor of binary bits

body:
Given an number and an encoding length, how can I convert the number to its binary representation as a tensor?

Eg, given the number 6 and width 8, how can I obtain the tensor:

(0, 0, 0, 0, 0, 1, 1, 0)


title: Custom convolution kernel and toroidal convolution in PyTorch

body:
I want to do two things with a PyTorch convolution which aren't mentioned in the documentation or code:

I want to create a convolution with a fixed kernel like this:

000010000
000010000
100010001
000010000
000010000

The horizontal aspect is like dilation, I guess, but the vertical part is different. I see that dilation is available as a parameter in the code, but it has to be a scalar or single-element tuple (not one element per dimension), so I don't think it can do what I want here.
I would like my convolutions to "wrap around" like a toroid, rather than use padding.

EDIT TO ADD: I see that there is an open issue for this , which also provides a suboptimal workaround. So, I guess that there's no "right" way to do it, yet.


title: torch.no_grad() affects on model accuracy

body:
I am getting an error "CUDA out of memory" then i add torch.no_grad() function into my code. Is it affect on my accuracy?
for iters in range(args.iterations):

with torch.no_grad():
    encoded, encoder_h_1, encoder_h_2, encoder_h_3 = encoder(
    res, encoder_h_1, encoder_h_2, encoder_h_3)

with torch.no_grad():
    code = binarizer(encoded)

with torch.no_grad():
    output, decoder_h_1, decoder_h_2, decoder_h_3, decoder_h_4 = decoder(
    code, decoder_h_1, decoder_h_2, decoder_h_3, decoder_h_4)

res = res - output.detach()
codes.append(code.data.cpu().numpy())
torch.cuda.empty_cache()
print('Iter: {:02d}; Loss: {:.06f}'.format(iters, res.data.abs().mean()))


title: Matrix multiplication in pyTorch

body:
I'm writing a simple neural network in pyTorch, where features and weights both are (1, 5) tensors. What are the differences between the two methods that I mention below?

y = activation(torch.sum(features*weights) + bias)

and

yy = activation(torch.mm(features, weights.view(5,1)) + bias)


title: PyTorch: Learning rate scheduler

body:
How do I use a learning rate scheduler with the following optimizer?

optimizer = torch.optim.Adam(optim_params,betas=(args.momentum, args.beta), weight_decay=args.weight_decay)

I have written the following scheduler:

scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=100, gamma=0.9)

I am not clear if I should step the scheduler or the optimizer. Which order should I take to perform the following?

optimizer.zero_grad()
scheduler.step()
optimizer.step()


title: PyTorch flatten doesn't maintain batch size

body: In Keras, using the Flatten() layer retains the batch size. For eg, if the input shape to Flatten is (32, 100, 100), in Keras output of Flatten is (32, 10000), but in PyTorch it is 320000. Why is it so?


title: LSTM autoencoder always returns the average of the input sequence

body:
I'm trying to build a very simple LSTM autoencoder with PyTorch. I always train it with the same data:

x = torch.Tensor([[0.0], [0.1], [0.2], [0.3], [0.4]])

I have built my model following this link:

inputs = Input(shape=(timesteps, input_dim))
encoded = LSTM(latent_dim)(inputs)

decoded = RepeatVector(timesteps)(encoded)
decoded = LSTM(input_dim, return_sequences=True)(decoded)

sequence_autoencoder = Model(inputs, decoded)
encoder = Model(inputs, encoded)

My code is running with no errors but y_pred converge to:

tensor([[[0.2]],
        [[0.2]],
        [[0.2]],
        [[0.2]],
        [[0.2]]], grad_fn=<StackBackward>)

Here is my code:

import torch
import torch.nn as nn
import torch.optim as optim

class LSTM(nn.Module):

    def __init__(self, input_dim, latent_dim, batch_size, num_layers):
        super(LSTM, self).__init__()
        self.input_dim = input_dim
        self.latent_dim = latent_dim
        self.batch_size = batch_size
        self.num_layers = num_layers

        self.encoder = nn.LSTM(self.input_dim, self.latent_dim, self.num_layers)

        self.decoder = nn.LSTM(self.latent_dim, self.input_dim, self.num_layers)

    def init_hidden_encoder(self):
        return (torch.zeros(self.num_layers, self.batch_size, self.latent_dim),
                torch.zeros(self.num_layers, self.batch_size, self.latent_dim))

    def init_hidden_decoder(self):
        return (torch.zeros(self.num_layers, self.batch_size, self.input_dim),
                torch.zeros(self.num_layers, self.batch_size, self.input_dim))

    def forward(self, input):
        # Reset hidden layer
        self.hidden_encoder = self.init_hidden_encoder()
        self.hidden_decoder = self.init_hidden_decoder()

        # Reshape input
        input = input.view(len(input), self.batch_size, -1)

        # Encode
        encoded, self.hidden = self.encoder(input, self.hidden_encoder)
        encoded = encoded[-1].repeat(5, 1, 1)

        # Decode
        y, self.hidden = self.decoder(encoded, self.hidden_decoder)
        return y

model = LSTM(input_dim=1, latent_dim=20, batch_size=1, num_layers=1)
loss_function = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.0001)

x = torch.Tensor([[0.0], [0.1], [0.2], [0.3], [0.4]])

while True:
    y_pred = model(x)
    optimizer.zero_grad()
    loss = loss_function(y_pred, x)
    loss.backward()
    optimizer.step()
    print(y_pred)


title: Pytorch network parameter calculation

body:
Can someone tell me please about how the network parameter (10) is calculated? Thanks in advance.

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        self.conv1 = nn.Conv2d(1, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(x.size()[0], -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
print(net)
print(len(list(net.parameters())))

Output:

Net(
  (conv1): Conv2d (1, 6, kernel_size=(5, 5), stride=(1, 1))
  (conv2): Conv2d (6, 16, kernel_size=(5, 5), stride=(1, 1))
  (fc1): Linear(in_features=400, out_features=120)
  (fc2): Linear(in_features=120, out_features=84)
  (fc3): Linear(in_features=84, out_features=10)
)
10

Best,
Zack


title: printing image paths from the dataloader in pytorch

body:
I am trying to learn One-shot learning with pytorch. I am experimenting with this Siamese Network in Pytorch example. Using that notebook as a guide, I simply would like to print out the image file paths for each pair of images, in addition to the dissimilarity scores.

From what I've been reading, it looks like I need to make some alterations to the dataloader in order to achieve this, as indicated here.

I do not have much experience yet in all of this. I would appreciate some guidance. I imported the altered dataloader (as in this gist) into my code.

The altered dataloader:

import torch
from torchvision import datasets

class ImageFolderWithPaths(datasets.ImageFolder):
    """Custom dataset that includes image file paths. Extends
    torchvision.datasets.ImageFolder
    """

    # override the __getitem__ method. this is the method that dataloader calls
    def __getitem__(self, index):
        # this is what ImageFolder normally returns
        original_tuple = super(ImageFolderWithPaths, self).__getitem__(index)
        # the image file path
        path = self.imgs[index][0]
        # make a new tuple that includes original and the path
        tuple_with_path = (original_tuple + (path,))
        return tuple_with_path

and the example usage:

data_dir = "your/data_dir/here"
dataset = ImageFolderWithPaths(data_dir) # our custom dataset
dataloader = torch.utils.DataLoader(dataset)

# iterate over data
for inputs, labels, paths in dataloader:
    # use the above variables freely
    print(inputs, labels, paths)

my code:

from pytorch_image_folder_with_file_paths import ImageFolderWithPaths

folder_dataset_test = dset.ImageFolder(root=Config.testing_dir)
siamese_dataset = SiameseNetworkDataset(imageFolderDataset=folder_dataset_test,
                                        transform=transforms.Compose([transforms.Resize((100,100)),
                                                                      transforms.ToTensor()
                                                                      ])
                                       ,should_invert=False)

test_dataloader = DataLoader(siamese_dataset,num_workers=6,batch_size=1,shuffle=True)

dataiter = iter(test_dataloader)
x0,_,_ = next(dataiter)

for i in range(10):
    _,x1,label2 = next(dataiter)

    concatenated = torch.cat((x0,x1),0)

    output1,output2 = net(Variable(x0).cuda(),Variable(x1).cuda())
    euclidean_distance = F.pairwise_distance(output1, output2)
    imshow(torchvision.utils.make_grid(concatenated),'Dissimilarity: {:.2f}'.format(euclidean_distance.item()))

for inputs, labels, paths in test_dataloader:
    print(inputs, labels, paths)

I get the paired images with the dissimilarity score, but I don't get the paths; I get

tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]]]]) tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,

...etc.

Thanks


title: How To Import The MNIST Dataset From Local Directory Using PyTorch

body:
I am writing a code of a well-known problem MNIST database of handwritten digits in PyTorch. I downloaded the train and testing dataset (from the main website) including the labeled dataset. The dataset format is t10k-images-idx3-ubyte.gz and after extract t10k-images-idx3-ubyte. My dataset folder looks like
MINST
 Data
  train-images-idx3-ubyte.gz
  train-labels-idx1-ubyte.gz
  t10k-images-idx3-ubyte.gz
  t10k-labels-idx1-ubyte.gz

Now, I wrote a code to load data like bellow
def load_dataset():
    data_path = "/home/MNIST/Data/"
    xy_trainPT = torchvision.datasets.ImageFolder(
        root=data_path, transform=torchvision.transforms.ToTensor()
    )
    train_loader = torch.utils.data.DataLoader(
        xy_trainPT, batch_size=64, num_workers=0, shuffle=True
    )
    return train_loader

My code is showing Supported extensions are: .jpg,.jpeg,.png,.ppm,.bmp,.pgm,.tif,.tiff,.webp
How can I solve this problem and I also want to check that my images are loaded (just a figure contains the first 5 images) from the dataset?


title: Can't import ToTensorV2 in Colab

body:
from albumentations.pytorch.transforms import ToTensorV2

I used the above code, and it doesn't work.


title: Loading model from checkpoint is not working

body:
I trained a vanilla vae which I modified from this repository. When I try and use the trained model I am unable to load the weights using load_from_checkpoint. It seems there is a mismatch between my checkpoint object and my lightningModule object.
I have setup an experiment (VAEXperiment) using pytorch-lightning LightningModule. I try to load the weights into the network with:
#building a new model
model = VanillaVAE(**config['model_params'])
model.build_layers()

#loading the weights
experiment = VAEXperiment(model, config['exp_params'])
experiment.load_from_checkpoint(path_to_checkpoint, config['exp_params'])

I also tried:
checkpoint = torch.load(path_to_checkpoint, map_location=lambda storage, loc: storage)
model.load_state_dict(checkpoint['state_dict'])

But I get an error
Unexpected key(s) in state_dict: "model.encoder.0.0.weight", "model.encoder.0.0.bias"...
I also followed the issue on
https://github.com/PyTorchLightning/pytorch-lightning/issues/924
https://github.com/PyTorchLightning/pytorch-lightning/issues/2798
Why I am I getting this error? Is it because of the encoder and decoder modules in my model? Based on the issues log on git it seems that the error resolved. What am I doing wrong?


title: How to setup pytorch in google-cloud-ml

body:
I try to throw job with Pytorch code in google-cloud-ml.
so I code the "setup.py" file. And add option "install_requires"

"setup.py"

from setuptools import find_packages
from setuptools import setup
REQUIRED_PACKAGES = ['http://download.pytorch.org/whl/cpu/torch-0.3.0.post4-cp27-cp27mu-linux_x86_64.whl','torchvision']

setup(
    name='trainer',
    version='0.1',
    install_requires=REQUIRED_PACKAGES,
    packages=find_packages(),
    include_package_data=True,
    description='My keras trainer application package.'
)

and throw the job to the google-cloud-ml, but it doesn't work

with error message

{
 insertId:  "3m78xtf9czd0u"
 jsonPayload: {
  created:  1516845879.49039
  levelname:  "ERROR"
  lineno:  829
  message:  "Command '['pip', 'install', '--user', '--upgrade', '--force-reinstall', '--no-deps', u'trainer-0.1.tar.gz']' returned non-zero exit status 1"
  pathname:  "/runcloudml.py"
 }
 labels: {
  compute.googleapis.com/resource_id:  "6637909247101536087"
  compute.googleapis.com/resource_name:  "cmle-training-master-5502b52646-0-ql9ds"
  compute.googleapis.com/zone:  "us-central1-c"
  ml.googleapis.com/job_id:  "run_ml_engine_pytorch_test_20180125_015752"
  ml.googleapis.com/job_id/log_area:  "root"
  ml.googleapis.com/task_name:  "master-replica-0"
  ml.googleapis.com/trial_id:  ""
 }
 logName:  "projects/exem-191100/logs/master-replica-0"
 receiveTimestamp:  "2018-01-25T02:04:55.421517460Z"
 resource: {
  labels: {…}
  type:  "ml_job"
 }
 severity:  "ERROR"
 timestamp:  "2018-01-25T02:04:39.490387916Z"
}

====================================================================

See detailed message here

so how can i use pytorch in google cloud ml engine?


title: L1 norm as regularizer in Pytorch

body:
I need to add an L1 norm as a regularizer to create a sparsity condition in my neural network. I would like to train my network for classification. I tried to construct an L1 norm by myself, like here, but it didn't work.
I need to add the regularizer after ConvTranspose2d, something like this Keras example:
model.add(Dense(64, input_dim=64,
            kernel_regularizer=regularizers.l2(0.01),
            activity_regularizer=regularizers.l1(0.01)))

But my network was created in PyTorch like so:
upconv = nn.ConvTranspose2d(inner_nc, outer_nc,
                            kernel_size=4, stride=2,
                            padding=1, bias=use_bias)
down = [downrelu, downconv]
up = [uprelu, upconv, upnorm]
model = down + up


RuntimeError: Given groups=1, weight of size [32, 3, 16, 16, 16], expected input[100, 16, 16, 16, 3] to have 3 channels, but got 16 channels instead

This is the portion of code I think where the problem is.

def __init__(self):
        super(Lightning_CNNModel, self).__init__()

        self.conv_layer1 = self._conv_layer_set(3, 32)
        self.conv_layer2 = self._conv_layer_set(32, 64)
        self.fc1 = nn.Linear(2**3*64, 128)
        self.fc2 = nn.Linear(128, 10)   # num_classes = 10
        self.relu = nn.LeakyReLU()
        self.batch=nn.BatchNorm1d(128)
        self.drop=nn.Dropout(p=0.15)

    def _conv_layer_set(self, in_c, out_c):
        conv_layer = nn.Sequential(
            nn.Conv3d(in_c, out_c, kernel_size=(3, 3, 3), padding=0),
            nn.LeakyReLU(),
            nn.MaxPool3d((2, 2, 2)),
        )

        return conv_layer

    def forward(self, x):
        out = self.conv_layer1(x)
        out = self.conv_layer2(out)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = self.relu(out)
        out = self.batch(out)
        out = self.drop(out)
        out = self.fc2(out)

        return out

This is the code I am working on


title: Correct way to create Pytorch dataset that returns sequence of data for RNN?

body:
I am attempting to train an RNN on time series data, and while there are plenty of tutorials out there on how to build a RNN model I am having some trouble with building the dataloader object for this task. The data is all going to be the same length, so no need for padding as well. The approach I have taken so far is to return a range of data in the getitem function on the dataset class and define the length as

len(data) - seq_len + 1

, however I feel that this is a bit "hacky" and that there should be a more proper way to do this. This method seems confusing and I feel that this would cause problems if collaborating with a group. More specifically, I think that somehow overriding the sampler function in the Pytorch Dataset constructor is the correct way, but I am having trouble understanding how to implement that. Below is the current dataset class I have built, can anyone point me in the right direction with how to fix it? Thank you in advance.

class CustomDataset(Dataset):
  def __init__(self, df, cats, y, seq_l):
    self.n, self.seq_l = len(df), seq_l
    self.cats  = np.array(np.stack([c.values for n,c in df[cats].items()], 1).astype(np.int64))
    self.conts = np.array(np.stack([c.values for n,c in df[[i for i in df.columns if i not in cats]].items()], 1).astype(np.float32))
    self.y = np.array(y)

  def __len__(self): return len(self.y) - self.seq_l + 1

  def __getitem__(self, idx):
    return [
      (torch.from_numpy(self.cats[idx:idx+self.seq_l]),
      torch.from_numpy(self.conts[idx:idx+self.seq_l])),
      self.y[idx+self.seq_l-1]
    ]


title: Invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:325

body:
I have a PyTorch tensor called out_probs which is produced like this:

out_probs=F.softmax(out_dec[:,0],dim=0)

Also, the shape of out_probs is [128,20004]

out_probs is the result of a softmax operation and it's not supposed to contain any negative value so naturally the result of out_probs[out_probs<0 is going to be an empty tensor(actually I checked and it was empty)
But when I'm running

torch.multinomial(out_probs, 1)

I'm getting :

RuntimeError: invalid argument 2: invalid multinomial distribution (encountering probability entry < 0) at /pytorch/aten/src/TH/generic/THTensorRandom.cpp:325

That implies my tensor has a negative entry and I don't know why this is happening?


title: How to fine tune BERT on its own tasks?

body: I wanted to pre-train BERT with the data from my own language since multilingual (which includes my language) model of BERT is not successful. Since whole pre-training costs a lot, I decided to fine tune it on its own 2 tasks: masked language model and next sentence prediction. There are previous implementation on different tasks (NER, sentiment analysis etc.), but I couldn't find any fine tuning on its own tasks. Is there an implementation that I couldn't see? If not, where should I start? I need some initial help.


title: padding='same' conversion to PyTorch padding=#

body:
I'm trying to convert the following Keras model code to pytorch, but am having problems dealing with padding='same'.

    model = Sequential()
    model.add(Conv2D(64, (3, 3), input_shape=img_size))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(Dropout(0.3))
    model.add(Conv2D(64, (3, 3), padding='same'))
    model.add(BatchNormalization(axis=1))
    model.add(Activation('relu'))
    model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2), padding='same'))

Which produces the following summary:

Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 30, 30, 64)        1792
_________________________________________________________________
batch_normalization_1 (Batch (None, 30, 30, 64)        120
_________________________________________________________________
activation_1 (Activation)    (None, 30, 30, 64)        0
_________________________________________________________________
dropout_1 (Dropout)          (None, 30, 30, 64)        0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 30, 30, 64)        36928
_________________________________________________________________
batch_normalization_2 (Batch (None, 30, 30, 64)        120
_________________________________________________________________
activation_2 (Activation)    (None, 30, 30, 64)        0
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 15, 15, 64)        0
=================================================================
Total params: 38,960
Trainable params: 38,840
Non-trainable params: 120

Right now, I would write:

self.features = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3,
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.Dropout(0.3),
            nn.Conv2d(64, 64, kernel_size=3, padding = ?
                      bias=False),
            nn.BatchNorm2d(64),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=3, stride=2, padding = ?),
        )

Where padding should have numerical value. I was wondering if there is an easier way to calculate this since we're using padding='same'.

Also, the next line of the Keras model looks like:

model.add(Conv2D(128, (3, 3), padding='same'))

So I really need to brush up on how to calculate padding, especially after stride too. From a rough eye only, is the padding 2?


title: How to change DataLoader in PyTorch to read one image for prediction?

body:
Currently, I have a pre-trained model that uses a DataLoader for reading a batch of images for training the model.
self.data_loader = torch.utils.data.DataLoader(dataset, batch_size=batch_size, shuffle=False,
   num_workers=1, pin_memory=True)

...

model.eval()
for step, inputs in enumerate(test_loader.data_loader):
   outputs = model(torch.cat([inputs], 1))

...

I want to process (make predictions) on images, as they arrive from a queue. It should be similar to a code that reads a single image and runs the model to make predictions on it. Something along the following lines:
from PIL import Image

new_input = Image.open(image_path)
model.eval()
outputs = model(torch.cat([new_input ], 1))

I was wondering if you could guide me how to do this and apply the same transformations in the DataLoader.


title: How to create a Pytorch Dataset from .pt files?

body:
I have transformed MNIST images saved as .pt files in a folder in Google drive. I'm writing my Pytorch code in Colab.

I would like to use these files, and create a Dataset that stores these images as Tensors. How can I do this?

Transforming images during training took too long. Hence, transformed them and saved them all as .pt files. I just want to load them back as a dataset and use them in my model.


title: How do I extract only subset of classes from torchvision.datasets.CIFAR10?

body:
How do I extract only 2 or 3 classes from torchvision.datasets.CIFAR10?

Standard way of loading all 10 classes

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
    trainset = torchvision.datasets.CIFAR10(root='./data', train=True, download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4, shuffle=True, num_workers=2)
    testset = torchvision.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)


title: Calling forward function without .forward()

body:
While looking at some pytorch code on pose estimation AlphaPose I noticed some unfamiliar syntax:

Basically, we define a Darknet class which inherits nn.Module properties like so: class Darknet(nn.Module)

This re-constructs the neural net from some config file and also defines functions to load pre-trained weights and a forward pass

Now, forward pass takes the following parameters:

def forward(self, x, CUDA)

I should note that in class definition forward is the only method that has a CUDA attribute (this will become important later on)

In the forward pass we get the predictions:

for i in range(number_of_modules):
     x = self.module[i](x)

where module[i] was constructed as:

module = nn.Sequential()
conv = nn.Conv2d(prev_fileters, filters, kernel_size, stride, pad, bias=bias)
module.add_module("conv_{0}".format(index), conv)

We then call invoke this model and (I presume) a forward method like so:

self.det_model = Darknet("yolo/cfg/yolov3-spp.cfg")
self.det_model.load_weights('models/yolo/yolov3-spp.weights')
self.det_model.cpu()
self.det_model.eval()

image = image.cpu()
prediction = self.det_model(img, CUDA = False)

I assume that the last line is the calling of the forward pass but why not use the .forward? Is this a pytorch specific syntax or am I missing some basic python principles?


title: pytorch conv2d value cannot be converted to type uint8_t without overflow

body:
I'm passing a torch.Tensor with a dtype of torch.uint8 to an nn.Conv2d module and it is giving the error

  RuntimeError: value cannot be converted to type uint8_t without
  overflow: -0.0344873

My conv2d is defined as self.conv1 = nn.Conv2d(3, 6, 5). The error comes in my forward method when I pass the tensor to the module like self.conv1(x).  The tensor has shape (4, 3, 480, 640). I'm not sure how to fix this.  Here is the stack trace

Traceback (most recent call last):

  File "cnn.py", line 54, in <module>

    outputs = net(inputs)

  File "/Users/my_repos/venv_projc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__

    result = self.forward(*input, **kwargs)

  File "cnn.py", line 24, in forward

    test = self.conv1(x)

  File "/Users/my_repos/venv_projc/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__

    result = self.forward(*input, **kwargs)

  File "/Users/my_repos/venv_projc/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward

    return self.conv2d_forward(input, self.weight)

  File "/Users/my_repos/venv_projc/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward

    self.padding, self.dilation, self.groups)

RuntimeError: value cannot be converted to type uint8_t without overflow: -0.0344873


title: Setting custom kernel for CNN in pytorch

body:
Is there a way to specify our own custom kernel values for a convolution neural network in pytorch? Something like kernel_initialiser in tensorflow? Eg. I want a 3x3 kernel in nn.Conv2d with initialization so that it acts as a identity kernel -

0 0 0
0 1 0
0 0 0

(this will effectively return the same output as my input in the very first iteration)

My non-exhaustive research on the subject -

I could use nn.init but it only has some pre-defined kernel initialisaition values.

I tried to follow the discussion on their official thread but it doesn't suit my needs.

I might have missed something in my research please feel free to point out.


title: Difficulty in understanding the tokenizer used in Roberta model

body:
from transformers import AutoModel, AutoTokenizer

tokenizer1 = AutoTokenizer.from_pretrained("roberta-base")
tokenizer2 = AutoTokenizer.from_pretrained("bert-base-cased")

sequence = "A Titan RTX has 24GB of VRAM"
print(tokenizer1.tokenize(sequence))
print(tokenizer2.tokenize(sequence))

Output:

['A', 'ĠTitan', 'ĠRTX', 'Ġhas', 'Ġ24', 'GB', 'Ġof', 'ĠVR', 'AM']

['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']

Bert model uses WordPiece tokenizer. Any word that does not occur in the WordPiece vocabulary is broken down into sub-words greedily. For example, 'RTX' is broken into 'R', '##T' and '##X' where ## indicates it is a subtoken.

Roberta uses BPE tokenizer but I'm unable to understand

a) how BPE tokenizer works?

b) what does G represents in each of tokens?


title: Mini batch training for inputs of variable sizes

body:
I have a list of LongTensors, and another list of labels. I'm new to PyTorch and RNN's so I'm quite confused as to how to implement minibatch training for the data I have. There is much more to this data, but I want to keep it simple, so I can understand only how to implement the minibatch training part. I'm doing multiclass classification based on the final hidden state of an LSTM/GRU trained on variable length inputs. I managed to get it working with batch size 1(basically SGD) but I'm struggling with implementing minibatches.

Do I have to pad the sequences to the maximum size and create a new tensor matrix of larger size which holds all the elements? I mean like this:

inputs = pad(sequences)
train = DataLoader(inputs, batch_size=batch_size, shuffle=True)
for i, data in train:
   #do stuff using LSTM and/or GRU models

Is this the accepted way of doing minibatch training on custom data? I couldn't find any tutorials on loading custom data using DataLoader(but I assume that's the way to create batches using pyTorch?)

Another doubt I have is with regards to padding. The reason I'm using LSTM/GRU is because of the variable length of the input. Doesn't padding defeat the purpose? Is padding necessary for minibatch training?


title: Display examples of augmented images in PyTorch

body:
I want to display some samples of augmented training images.

My transform includes the standard ImageNet transforms.Normalize like this:

train_transforms = transforms.Compose([transforms.RandomRotation(30),
                                       transforms.RandomResizedCrop(224),
                                       transforms.RandomHorizontalFlip(),
                                       transforms.ToTensor(),
                                       transforms.Normalize([0.485, 0.456, 0.406],
                                                            [0.229, 0.224, 0.225])])

However, because of the Normalise, the images display in weird colours.

This answer says I'd need access to the original image, which is difficult when the transforms are applied at load time:

image_datasets['train'] = datasets.ImageFolder(train_dir, transform=train_transforms)

How would I go about displaying a few sample augmented images in their usual colours while using the normalised ones for calculation?


title: How to fix '_pickle.UnpicklingError: invalid load key, '<' ' error in Pytorch

body:
The problem I encountered when I ran the official code of maskrcnn-benchmark for facebookresearch,which was wrong when loading the pre-training model.

The code runs on a remote server at the school and the graphics card is an NVIDIA P100.

checkpointer = DetectronCheckpointer(
        cfg, model, optimizer, scheduler, output_dir, save_to_disk)
extra_checkpoint_data = checkpointer.load(cfg.MODEL.WEIGHT)
arguments.update(extra_checkpoint_data)

I expect to run the code correctly and understand why this is the problem.


title: Tokens to Words mapping in the tokenizer decode step huggingface?

body:
Is there a way to know the mapping from the tokens back to the original words in the tokenizer.decode() function?
For example:

from transformers.tokenization_roberta import RobertaTokenizer

tokenizer = RobertaTokenizer.from_pretrained('roberta-large', do_lower_case=True)

str = "This is a tokenization example"
tokenized = tokenizer.tokenize(str)
## ['this', 'Ġis', 'Ġa', 'Ġtoken', 'ization', 'Ġexample']

encoded = tokenizer.encode_plus(str)
## encoded['input_ids']=[0, 42, 16, 10, 19233, 1938, 1246, 2]

decoded = tokenizer.decode(encoded['input_ids'])
## '<s> this is a tokenization example</s>'

And the objective is to have a function that maps each token in the decode process to the correct input word, for here it will be:
desired_output = [[1],[2],[3],[4,5],[6]] As this corresponds to id 42, while token and ization corresponds to ids [19244,1938] which are at indexes 4,5 of the input_ids array.


title: How to select index over two dimension in PyTorch?

body:
Given a = torch.randn(3, 2, 4, 5), how I can select sub tensor like  (2, :, 0, :), (1, :, 1, :), (2, :, 2, :), (0, :, 3, :) (a resulting tensor of size (2, 4, 5) or (4, 2, 5)?

While a[2, :, 0, :] gives

 0.5580 -0.0337  1.0048 -0.5044  0.6784
-1.6117  1.0084  1.1886  0.1278  0.3739
[torch.FloatTensor of size 2x5]

however, a[[2, 1, 2, 0], :, [0, 1, 2, 3], :] gives

  TypeError: Performing basic indexing on a tensor and encountered an error indexing dim 0 with an object of type list. The only supported types are integers, slices, numpy scalars, or if indexing with a torch.LongTensor or torch.ByteTensor only a single Tensor may be passed.

though numpy returns a (4, 2, 5) tensor successfully.


title: CTC loss goes down and stops

body:
I’m trying to train a captcha recognition model. Model details are resnet pretrained CNN layers + Bidirectional LSTM + Fully Connected. It reached 90% sequence accuracy on captcha generated by python library captcha. The problem is that these generated captcha seems to have similary location of each character. When I randomly add spaces between characters, the model does not work any more. So I wonder is LSTM learning segmentation during learning? Then I try to use CTC loss. At first, loss goes down pretty quick. But it stays at about 16 without significant drop later. I tried different layers of LSTM, different number of units. 2 Layers of LSTM reach lower loss, but still not converging. 3 layers are just like 2 layers. The loss curve:

#encoding:utf8
import os
import sys
import torch
import warpctc_pytorch
import traceback

import torchvision
from torch import nn, autograd, FloatTensor, optim
from torch.nn import functional as F
from torch.utils.data import DataLoader
from torch.optim.lr_scheduler import MultiStepLR
from tensorboard import SummaryWriter
from pprint import pprint

from net.utils import decoder

from logging import getLogger, StreamHandler
logger = getLogger(__name__)
handler = StreamHandler(sys.stdout)
logger.addHandler(handler)

from dataset_util.utils import id_to_character
from dataset_util.transform import rescale, normalizer
from config.config import MAX_CAPTCHA_LENGTH, TENSORBOARD_LOG_PATH, MODEL_PATH

class CNN_RNN(nn.Module):
    def __init__(self, lstm_bidirectional=True, use_ctc=True, *args, **kwargs):
        super(CNN_RNN, self).__init__(*args, **kwargs)
        model_conv = torchvision.models.resnet18(pretrained=True)
        for param in model_conv.parameters():
            param.requires_grad = False

        modules = list(model_conv.children())[:-1]  # delete the last fc layer.
        for param in modules[8].parameters():
            param.requires_grad = True

        self.resnet = nn.Sequential(*modules)            # CNN with fixed parameters from resnet as feature extractor
        self.lstm_input_size = 512 * 2 * 2
        self.lstm_hidden_state_size = 512
        self.lstm_num_layers = 2
        self.chracter_space_length = 64
        self._lstm_bidirectional = lstm_bidirectional
        self._use_ctc = use_ctc
        if use_ctc:
            self._max_captcha_length = int(MAX_CAPTCHA_LENGTH * 2)
        else:
            self._max_captcha_length = MAX_CAPTCHA_LENGTH

        if lstm_bidirectional:
            self.lstm_hidden_state_size = self.lstm_hidden_state_size * 2           # so that hidden size for one direction in bidirection lstm is the same as vanilla lstm
            self.lstm = self.lstm = nn.LSTM(self.lstm_input_size, self.lstm_hidden_state_size // 2, dropout=0.5, bidirectional=True, num_layers=self.lstm_num_layers)
        else:
            self.lstm = nn.LSTM(self.lstm_input_size, self.lstm_hidden_state_size, dropout=0.5, bidirectional=False, num_layers=self.lstm_num_layers)  # dropout doen't work for one layer lstm

        self.ouput_to_tag = nn.Linear(self.lstm_hidden_state_size, self.chracter_space_length)
        self.tensorboard_writer = SummaryWriter(TENSORBOARD_LOG_PATH)
        # self.dropout_lstm = nn.Dropout()

    def init_hidden_status(self, batch_size):
        if self._lstm_bidirectional:
            self.hidden = (autograd.Variable(torch.zeros((self.lstm_num_layers * 2, batch_size, self.lstm_hidden_state_size // 2))),
                           autograd.Variable(torch.zeros((self.lstm_num_layers * 2, batch_size, self.lstm_hidden_state_size // 2)))) # number of layers, batch size, hidden dimention
        else:
            self.hidden = (autograd.Variable(torch.zeros((self.lstm_num_layers, batch_size, self.lstm_hidden_state_size))),
                           autograd.Variable(torch.zeros((self.lstm_num_layers, batch_size, self.lstm_hidden_state_size)))) # number of layers, batch size, hidden dimention

    def forward(self, image):
        '''
        :param image:  # batch_size, CHANNEL, HEIGHT, WIDTH
        :return:
        '''
        features = self.resnet(image)                 # [batch_size, 512, 2, 2]
        batch_size = image.shape[0]
        features = [features.view(batch_size, -1) for i in range(self._max_captcha_length)]
        features = torch.stack(features)
        self.init_hidden_status(batch_size)
        output, hidden = self.lstm(features, self.hidden)
        # output = self.dropout_lstm(output)
        tag_space = self.ouput_to_tag(output.view(-1, output.size(2)))      # [MAX_CAPTCHA_LENGTH * BATCH_SIZE, CHARACTER_SPACE_LENGTH]
        tag_space = tag_space.view(self._max_captcha_length, batch_size, -1)

        if not self._use_ctc:
            tag_score = F.log_softmax(tag_space, dim=2)             # [MAX_CAPTCHA_LENGTH, BATCH_SIZE, CHARACTER_SPACE_LENGTH]
        else:
            tag_score = tag_space

        return tag_score

    def train_net(self, data_loader, eval_data_loader=None, learning_rate=0.008, epoch_num=400):
        try:
            if self._use_ctc:
                loss_function = warpctc_pytorch.warp_ctc.CTCLoss()
            else:
                loss_function = nn.NLLLoss()

            # optimizer = optim.SGD(filter(lambda p: p.requires_grad, self.parameters()), momentum=0.9, lr=learning_rate)
            # optimizer = MultiStepLR(optimizer, milestones=[10,15], gamma=0.5)

            # optimizer = optim.Adadelta(filter(lambda p: p.requires_grad, self.parameters()))
            optimizer = optim.Adam(filter(lambda p: p.requires_grad, self.parameters()))
            self.tensorboard_writer.add_scalar("learning_rate", learning_rate)

            tensorbard_global_step=0
            if os.path.exists(os.path.join(TENSORBOARD_LOG_PATH, "resume_step")):
                with open(os.path.join(TENSORBOARD_LOG_PATH, "resume_step"), "r") as file_handler:
                    tensorbard_global_step = int(file_handler.read()) + 1

            for epoch_index, epoch in enumerate(range(epoch_num)):
                for index, sample in enumerate(data_loader):
                    optimizer.zero_grad()
                    input_image = autograd.Variable(sample["image"])        # batch_size, 3, 255, 255
                    tag_score = self.forward(input_image)

                    if self._use_ctc:
                        tag_score, target, tag_score_sizes, target_sizes = self._loss_preprocess_ctc(tag_score, sample)
                        loss = loss_function(tag_score, target, tag_score_sizes, target_sizes)
                        loss = loss / tag_score.size(1)

                    else:
                        target = sample["padded_label_idx"]
                        tag_score, target = self._loss_preprocess(tag_score, target)
                        loss = loss_function(tag_score, target)

                    print("Training loss: {}".format(float(loss)))
                    self.tensorboard_writer.add_scalar("training_loss", float(loss), tensorbard_global_step)
                    loss.backward()
                    optimizer.step()

                    if index % 250 == 0:
                        print(u"Processing batch: {} of {}, epoch: {}".format(index, len(data_loader), epoch_index))
                        self.evaluate(eval_data_loader, loss_function, tensorbard_global_step)

                    tensorbard_global_step += 1

                self.save_model(MODEL_PATH + "_epoch_{}".format(epoch_index))

        except KeyboardInterrupt:
            print("Exit for KeyboardInterrupt, save model")
            self.save_model(MODEL_PATH)

            with open(os.path.join(TENSORBOARD_LOG_PATH, "resume_step"), "w") as file_handler:
                file_handler.write(str(tensorbard_global_step))

        except Exception as excp:
            logger.error(str(excp))
            logger.error(traceback.format_exc())

    def predict(self, image):
        # TODO ctc version
        '''
        :param image: [batch_size, channel, height, width]
        :return:
        '''
        tag_score = self.forward(image)
        # TODO ctc
        # if self._use_ctc:
        #     tag_score = F.softmax(tag_score, dim=-1)
        #     decoder.decode(tag_score)

        confidence_log_probability, indexes = tag_score.max(2)

        predicted_labels = []
        for batch_index in range(indexes.size(1)):
            label = ""
            for character_index in range(self._max_captcha_length):
                if int(indexes[character_index, batch_index]) != 1:
                    label += id_to_character[int(indexes[character_index, batch_index])]
            predicted_labels.append(label)

        return predicted_labels, tag_score

    def predict_pil_image(self, pil_image):
        try:
            self.eval()
            processed_image = normalizer(rescale({"image": pil_image}))["image"].view(1, 3, 255, 255)
            result, tag_score = self.predict(processed_image)
            self.train()

        except Exception as excp:
            logger.error(str(excp))
            logger.error(traceback.format_exc())
            return [""], None

        return result, tag_score

    def evaluate(self, eval_dataloader, loss_function, step=0):
        total = 0
        sequence_correct = 0
        character_correct = 0
        character_total = 0
        loss_total = 0
        batch_size = eval_data_loader.batch_size
        true_predicted = {}
        self.eval()
        for sample in eval_dataloader:
            total += batch_size
            input_images = sample["image"]
            predicted_labels, tag_score = self.predict(input_images)

            for predicted, true_label in zip(predicted_labels, sample["label"]):
                if predicted == true_label:                  # dataloader is making label a list, use batch_size=1
                    sequence_correct += 1

                for index, true_character in enumerate(true_label):
                    character_total += 1
                    if index < len(predicted) and predicted[index] == true_character:
                        character_correct += 1

                true_predicted[true_label] = predicted

            if self._use_ctc:
                tag_score, target, tag_score_sizes, target_sizes = self._loss_preprocess_ctc(tag_score, sample)
                loss_total += float(loss_function(tag_score, target, tag_score_sizes, target_sizes) / batch_size)

            else:
                tag_score, target = self._loss_preprocess(tag_score, sample["padded_label_idx"])
                loss_total += float(loss_function(tag_score, target))  # averaged over batch index

        print("True captcha to predicted captcha: ")
        pprint(true_predicted)
        self.tensorboard_writer.add_text("eval_ture_to_predicted", str(true_predicted), global_step=step)

        accuracy = float(sequence_correct) / total
        avg_loss = float(loss_total) / (total / batch_size)
        character_accuracy = float(character_correct) / character_total
        self.tensorboard_writer.add_scalar("eval_sequence_accuracy", accuracy, global_step=step)
        self.tensorboard_writer.add_scalar("eval_character_accuracy", character_accuracy, global_step=step)
        self.tensorboard_writer.add_scalar("eval_loss", avg_loss, global_step=step)
        self.zero_grad()
        self.train()

    def _loss_preprocess(self, tag_score, target):
        '''
        :param tag_score:  value return by self.forward
        :param target:     sample["padded_label_idx"]
        :return:           (processed_tag_score, processed_target)  ready for NLLoss function
        '''
        target = target.transpose(0, 1)
        target = target.contiguous()
        target = target.view(target.size(0) * target.size(1))
        tag_score = tag_score.view(-1, self.chracter_space_length)

        return tag_score, target

    def _loss_preprocess_ctc(self, tag_score, sample):
        target_2d = [
            [int(ele) for ele in sample["padded_label_idx"][row, :] if int(ele) != 0 and int(ele) != 1]
            for row in range(sample["padded_label_idx"].size(0))]
        target = []
        for ele in target_2d:
            target.extend(ele)
        target = autograd.Variable(torch.IntTensor(target))

        # tag_score = F.softmax(F.sigmoid(tag_score), dim=-1)
        tag_score_sizes = autograd.Variable(torch.IntTensor([self._max_captcha_length] * tag_score.size(1)))
        target_sizes = autograd.Variable(sample["captcha_length"].int())

        return tag_score, target, tag_score_sizes, target_sizes

    # def visualize_graph(self, dataset):
    #     '''Since pytorch use dynamic graph, an input is required to visualize graph in tensorboard'''
    #     # warning: Do not run this, the graph is too large to visualize...
    #     sample = dataset[0]
    #     input_image = autograd.Variable(sample["image"].view(1, 3, 255, 255))
    #     tag_score = self.forward(input_image)
    #     self.tensorboard_writer.add_graph(self, tag_score)

    def save_model(self, model_path):
        self.tensorboard_writer.close()
        self.tensorboard_writer = None          # can't be pickled
        torch.save(self, model_path)
        self.tensorboard_writer = SummaryWriter(TENSORBOARD_LOG_PATH)

    @classmethod
    def load_model(cls, model_path=MODEL_PATH, *args, **kwargs):
        net = cls(*args, **kwargs)
        if os.path.exists(model_path):
            model = torch.load(model_path)
            if model:
                model.tensorboard_writer = SummaryWriter(TENSORBOARD_LOG_PATH)
                net = model

        return net

    def __del__(self):
        if self.tensorboard_writer:
            self.tensorboard_writer.close()

if __name__ == "__main__":
    from dataset_util.dataset import dataset, eval_dataset
    data_loader = DataLoader(dataset, batch_size=2, shuffle=True)
    eval_data_loader = DataLoader(eval_dataset, batch_size=2, shuffle=True)

    net = CNN_RNN.load_model()

    net.train_net(data_loader, eval_data_loader=eval_data_loader)
    # net.predict(dataset[0]["image"].view(1, 3, 255, 255))

    # predict_pil_image test code
    # from config.config import IMAGE_PATHS
    # import glob
    # from PIL import Image
    #
    # image_paths = glob.glob(os.path.join(IMAGE_PATHS.get("EVAL"), "*.png"))
    # for image_path in image_paths:
    #     pil_image = Image.open(image_path)
    #     predicted, score = net.predict_pil_image(pil_image)
    #     print("True value: {}, predicted: {}".format(os.path.split(image_path)[1], predicted))

    print("Done")

The above codes are main part. If you need other components that makes it running, leave a comment. Got stuck here for quite long. Any advice for training crnn + ctc is appreciated.


When I try to load a pytorch checkpoint:

checkpoint = torch.load(pathname)

I see:

  RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at torch/csrc/cuda/Module.cpp:51

I created the checkpoint with a GPU available, but now only have CPU available.

How do I load the checkpoint?


title: Access parameter names in torch

body:
I need to convert a torch model to pytorch. Since the torch model has layers that pytorch doesn't support(such as inception and LRN), it is not possible to use the build-in APIs. In order to convert such models from torch to pytorch, it is necessary to implement such layers in pytorch and save all the parameters from torch model as hdf5 file, and reload them to python as a dictionary. I'm new to lua and I would like to ask how to access the 'nickname' of all the parameters in torch.

btw, this can be easily done in pytorch, for example:

import torch.nn as nn
model = nn.Sequential(
                nn.Conv2d(in_channels=3,out_channels=32,kernel_size=7,stride=1,bias=False),
                nn.ReLU(inplace=True),
                nn.BatchNorm2d(num_features=32,affine=True),
                nn.MaxPool2d(kernel_size=2,stride=2)
                )
for key in model.state_dict():
    value = model.state_dict().get(key)
    print(key, value.size())

if all the parameters are accessible in a dictionary format, reconstructing a model in pytorch can be done in the following code:

model = MyNewInceptionModel()
model.load_state_dict(param_dict)


title: What is the best way to use multiprocessing CPU inference for PyTorch models?

body:
I have to productionize a PyTorch BERT Question Answer model. The CPU inference is very slow for me as for every query the model needs to evaluate 30 samples. Out of the result of these 30 samples, I pick the answer with the maximum score. GPU would be too costly for me to use for inference.

Can I leverage multiprocessing / parallel CPU inference for this?
If Yes, what is the best practice to do so?
If No, is there a cloud option that bills me only for the GPU queries I make and not for continuously running the GPU instance?


I'm currently using fast.ai to train an image classifier model.

data = ImageDataBunch.single_from_classes(path, classes, ds_tfms=get_transforms(), size=224).normalize(imagenet_stats)
learner = cnn_learner(data, models.resnet34)

learner.model.load_state_dict(
    torch.load('stage-2.pth', map_location="cpu")
)

which results in :

  torch.load('stage-2.pth', map_location="cpu")   File
  "/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py",
  line 769, in load_state_dict
      self.class.name, "\n\t".join(error_msgs))) RuntimeError: Error(s) in loading state_dict for Sequential:

  ...

  Unexpected key(s) in state_dict: "model", "opt".

I have looked around in SO and tried to use the following solution:

# original saved file with DataParallel
state_dict = torch.load('stage-2.pth', map_location="cpu")
# create new OrderedDict that does not contain `module.`
from collections import OrderedDict
new_state_dict = OrderedDict()
for k, v in state_dict.items():
    name = k[7:] # remove `module.`
    new_state_dict[name] = v
# load params
learner.model.load_state_dict(new_state_dict)

which results in :

  RuntimeError: Error(s) in loading state_dict for Sequential:

  Unexpected key(s) in state_dict: "".

I'm using Google Colab to train my model and then port the trained model into docker and try to host in in a local server.

What could be the issue? Could it be the different version of pytorch which results in model mismatch?

In my docker config:

# Install pytorch and fastai
RUN pip install torch_nightly -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
RUN pip install fastai

While my Colab is using the following:

 !curl -s https://course.fast.ai/setup/colab | bash


title: How calculate the dice coefficient for multi-class segmentation task using Python?

body:
I am wondering how can I calculate the dice coefficient for multi-class segmentation.

Here is the script that would calculate the dice coefficient for the binary segmentation task. How can I loop over each class and calculate the dice for each class?

Thank you in advance

import numpy

def dice_coeff(im1, im2, empty_score=1.0):

    im1 = numpy.asarray(im1).astype(numpy.bool)
    im2 = numpy.asarray(im2).astype(numpy.bool)

    if im1.shape != im2.shape:
        raise ValueError("Shape mismatch: im1 and im2 must have the same shape.")

    im_sum = im1.sum() + im2.sum()
    if im_sum == 0:
        return empty_score

    # Compute Dice coefficient
    intersection = numpy.logical_and(im1, im2)

    return (2. * intersection.sum() / im_sum)


title: The result is different when I apply torch.manual_seed before loading cuda() after loading the model

body:
I tried to make sure my code to be reproducible (always get the same results)

So I applied below settings before my codes.

os.environ['PYTHONHASHSEED'] = str(args.seed)
random.seed(args.seed)
np.random.seed(args.seed)
torch.manual_seed(args.seed)
torch.cuda.manual_seed(args.seed)
torch.cuda.manual_seed_all(args.seed) # if you are using multi-GPU.
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True

With these settings, I always achieved the same results with the same environment and GPU.

Howerver, when I applied torch.manual_seed() after loading the model.

torch.manual_seed(args.seed)
model = Net()
Net.cuda()

torch.manual_seed(args.seed)
model = Net()
torch.manual_seed(args.seed)
Net.cuda()

The above two results were different.
How should I understand this situation?

Does seed reinitialize after loading the model?


title: How to extract fc7 features from AlexNet in pytorch as numpy array?

body:
I want to extract the 4096-dimensional feature vector from the fc7 layer of my finetuned AlexNet. My goal is to use this layer for clustering later on.
This is how I extract it:

alexnet = models.alexnet(pretrained=True);
fc7 = alexnet.classifier[6];

However, when I print it, fc7 is a Linear object:

Linear(in_features=4096, out_features=1000, bias=True)

What I am looking for is how to turn this Linear object into a numpy array so that I can do further manipulations on it.
What I am thinking of is to call its method'def forward(self, input)', but am not sure which input to provide? Should I provide the input image or should I provide the output the fc6 layer?

And I want the 4096-dim input array and get rid of the 1000 output array (presumably, since I don't think it will help me for clustering).


title: pytorch - net.cuda() seems don't work

body:
I wrote a cnn module to do digit recognition using pytorch, then try to train the network with gpu but got following error.

Traceback (most recent call last):
  File "main.py", line 51, in <module>
    outputs = cnn(inputs)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/daniel/Code/kaggle-competitions/digit-recognizer/Net.py", line 40, in forward
    x = self.pool(F.relu(self.conv[i](x)))
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 357, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/modules/conv.py", line 282, in forward
    self.padding, self.dilation, self.groups)
  File "/home/daniel/anaconda3/envs/pytorch/lib/python3.5/site-packages/torch/nn/functional.py", line 90, in conv2d
    return f(input, weight, bias)
RuntimeError: Input type (CUDAFloatTensor) and weight type (CPUFloatTensor) should be the same

here is my source code

It seems that cnn.cuda() didn't work properly because I got the same error after removing it. But I have no idea how to fix it.


title: pytorch - use device inside 'with statement'

body:
Is there a way of running pytorch inside the context of a specific (GPU) device (without having to specify the device for each new tensor, such as the .to option)?

Something like an equivalent of the tensorflow with tf.device('/device:GPU:0'):..

It seems that the default device is the cpu (unless I'm doing it wrong):

with torch.cuda.device('0'):
   a = torch.zeros(1)
   print(a.device)

>>> cpu


title: Errow Showing :'ResNet' object has no attribute 'classifier'

body:
I download Resnet18 model to train a model.

When I type

model

it shows

ResNet(
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (relu): ReLU(inplace=True)
  (maxpool): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)
  (layer1): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
    (1): BasicBlock(
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )

  (layer3): Sequential(
    (0): BasicBlock(
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (downsample): Sequential(
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    )
    (1): BasicBlock(
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      (relu): ReLU(inplace=True)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    )
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(1, 1))
  (fc): Linear(in_features=512, out_features=1000, bias=True)
  (classifer): Sequential(
    (fc1): Linear(in_features=512, out_features=256, bias=True)
    (relu): ReLU()
    (fc5): Linear(in_features=128, out_features=2, bias=True)
    (output): LogSoftmax()
  )
)

As you can see it clearly shows the classifier

but when I do

optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)

it shows an error

AttributeError: 'ResNet' object has no attribute 'classifier'

I don't know what mistake I am doing, if you can help that would be great. I can provide some extra details if you want.


title: pretrained object detection models in keras

body:
There are pretrained object recognition models in keras.applications library. But as far as I know, there is no pretrained object detection model available.

Does anyone know why it is the case? Object detection is a big part of problems when dealing with visual problems.


title: DataLoader not randomly sampling in PyTorch

body:
My DataLoader is returning me the same image with each epoch. My model is only looking at the same single image (indexed '0') each time (batch size is 1...although nothing changes with different batch sizes, anyways).

Here's my dataset, stripped down to the important bits:

class MyDataset(Dataset):

    def __init__(self, path, loader=pil_loader):
        self.path = path
        self.images = os.listdir(path)

    def __getitem__(self, index):
        image = self.images[index]

    . . .

And here's the DataSet:

train_ds = MyDataset('/data')

And here's my sampler:

train_sampler = RandomSampler(train_ds)

And here's my DataLoader:

train_dl = DataLoader(train_ds, batch_size=1, sampler=train_sampler)

I'm not sure why it is returning me the same image each time, during
 training.

Do I have RandomSampler incompletely set up? Or maybe I wrote the __getitem__ incorrectly? I can't figure it out.


title: Pytorch RuntimeError: Invalid index in gather

body:
I'm new to Pytorch and I encounter this error:

  x.gather(1, c)

  RuntimeError: Invalid index in gather at
  /pytorch/aten/src/TH/generic/THTensorEvenMoreMath.cpp:457

Here is some informations about the tensors:

print(x.size())
print(c.size())
print(type(x))
print(type(c))

torch.Size([128, 2])
torch.Size([128, 1])
<class 'torch.Tensor'>
<class 'torch.Tensor'>

x is filled with float values and c with integers, could it be the problem?


title: Index pytorch tensor with different dimension index array

body:
I have the following function, which does what I want using numpy.array, but breaks when feeding a torch.Tensor due to indexing errors.

import torch
import numpy as np

def combination_matrix(arr):
    idxs = np.arange(len(arr))
    idx = np.ix_(idxs, idxs)
    mesh = np.stack(np.meshgrid(idxs, idxs))

    def np_combination_matrix():
        output = np.zeros((len(arr), len(arr), 2, *arr.shape[1:]), dtype=arr.dtype)
        num_dims = len(output.shape)
        output[idx] = arr[mesh].transpose((2, 1, 0, *np.arange(3, num_dims)))
        return output

    def torch_combination_matrix():
        output = torch.zeros(len(arr), len(arr), 2, *arr.shape[1:], dtype=arr.dtype)
        num_dims = len(output.shape)
        print(arr[mesh].shape)  # <-- This is wrong/different to numpy!
        output[idx] = arr[mesh].permute(2, 1, 0, *np.arange(3, num_dims))
        return output

    if isinstance(arr, np.ndarray):
        return np_combination_matrix()
    elif isinstance(arr, torch.Tensor):
        return torch_combination_matrix()

The problem is that arr[mesh] results in different dimensions, depending on numpy and torch. Apparently, pytorch does not support indexing with index arrays of different dimensionality than the array being indexed. Ideally, the following should work:

features = np.arange(9).reshape(3, 3)
np_combs = combination_matrix(features)
features = torch.from_numpy(features)
torch_combs = combination_matrix(features)
assert np.array_equal(np_combs, torch_combs.numpy())

But the dimensions are different:

(2, 3, 3, 3)
torch.Size([3, 3])

Which results in an error (logically):

Traceback (most recent call last):
  File "/home/XXX/util.py", line 226, in <module>
    torch_combs = combination_matrix(features)
  File "/home/XXX/util.py", line 218, in combination_matrix
    return torch_combination_matrix()
  File "/home/XXX/util.py", line 212, in torch_combination_matrix
    output[idx] = arr[mesh].permute(2, 1, 0, *np.arange(3, num_dims))
RuntimeError: number of dims don't match in permute

How do I match the torch behavior to numpy?
I've read various questions on the torch forums (e.g. this one with only one dimension), but could find how to apply this here. Similarly, index_select only works for one dimension, but I need it to work for at least 2 dimensions.


title: Return predictions wav2vec fairseq

body:
I'm trying to use wav2vec to train my own Automatic Speech Recognition System:

https://github.com/pytorch/fairseq/tree/master/examples/wav2vec

import torch
from fairseq.models.wav2vec import Wav2VecModel

cp = torch.load('/path/to/wav2vec.pt')
model = Wav2VecModel.build_model(cp['args'], task=None)
model.load_state_dict(cp['model'])
model.eval()

First of all how can I use a loaded model to return predictions from a wav file?

Second, how can I pre-train using annotated data?
I don't see any text mention in the manifest scripts.


title: PyTorch tensor advanced indexing

body:
Say i have one matrix and one vector as follows:

x = torch.tensor([[1, 2, 3],
                  [4, 5, 6],
                  [7, 8, 9]])

y = torch.tensor([0, 2, 1])

is there a way to slice it x[y] so the result is:

res = [1, 6, 8]

so basically i take the first element of y and take the element in x that corresponds to the first row and the elements' column.

Cheers


I am designing an stacked autoencoder trying to train my neural network on movie rating if the user doesnt rate any movie it will not consider it

My training set runs perfectly but when i run test set it shows me this error

RuntimeError: The shape of the mask [1682] at index 0 does not match the shape of the indexed tensor [1, 1682] at index 0
I got error at the end test block i have commented there

CODE:-

# Auto Encoder

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
import torch.nn as nn
import torch.nn.parallel
import torch.optim as optim
import torch.utils.data
from torch.autograd import Variable

# Importing dataset
movies= pd.read_csv('ml-1m/movies.dat',sep ='::', header= None,engine ='python', encoding= 'latin-1')
users= pd.read_csv('ml-1m/users.dat',sep ='::', header= None,engine ='python', encoding= 'latin-1')
ratings = pd.read_csv('ml-1m/ratings.dat',sep ='::', header= None,engine ='python', encoding= 'latin-1')

# preparing the training set and the dataset

training_set =pd.read_csv('ml-100k/u1.base',delimiter ='\t')
training_set =np.array(training_set, dtype= 'int')

test_set =pd.read_csv('ml-100k/u1.test',delimiter ='\t')
test_set =np.array(test_set, dtype= 'int')

# Getting the number of users and  movies
# we are taking the maximum no of values from training set and test set

nb_users = int(max(max(training_set[:,0]), max(test_set[:,0])))
nb_movies = int(max(max(training_set[:,1]), max(test_set[:,1])))

# converting the data into an array within users in lines and movies in columns

def convert(data):
    new_data = []
    for id_users in range(1, nb_users +1):
        id_movies = data[:,1][data[:,0]==id_users]#movies id from data
        id_ratings = data[:,2][data[:,0]==id_users] #ratings
        ratings= np.zeros(nb_movies)
        ratings[id_movies-1] = id_ratings  # -1 for making it start from 1
        new_data.append(list(ratings))
    return new_data

training_set =convert(training_set)
test_set =convert(test_set)

# Converting the data into Torch tensor
training_set = torch.FloatTensor(training_set)
test_set = torch.FloatTensor(test_set)

# creating the architecture of the neural network
class SAE(nn.Module):

    def  __init__(self, ): # after comma it will consider parameters of module ie parent class
        super(SAE,self).__init__()#parent class inheritence
        self.fc1 = nn.Linear(nb_movies, 20)  #20 nodes in hidden layer
        self.fc2= nn.Linear(20,10)
        self.fc3 = nn.Linear(10,20)  #decoding
        self.fc4= nn.Linear(20, nb_movies) #decoding
        self.activation= nn.Sigmoid()
            #self.myparameters= nn.ParameterList(self.fc1,self.fc2,self.fc3,self.fc4,self.activation)

    def forward(self, x):
        x=self.activation(self.fc1(x))#encoding
        x=self.activation(self.fc2(x))#encoding
        x=self.activation(self.fc3(x)) #decoding
        x=self.fc4(x) #last layer machine understand automaically
        return x

sae= SAE()
criterion = nn.MSELoss()

optimizer= optim.RMSprop(sae.parameters(), lr= 0.01 , weight_decay =0.5)

# Training the SAE
nb_epoch = 200
for epoch in range(1, nb_epoch + 1):
    train_loss = 0
    s = 0.
    for id_user in range(nb_users):
        input = Variable(training_set[id_user]).unsqueeze(0)
        target = input.clone()
        if torch.sum(target.data > 0) > 0:
            output = sae(input)
            target.require_grad = False
            output[target == 0] = 0
            loss = criterion(output, target)
            mean_corrector = nb_movies/float(torch.sum(target.data > 0) + 1e-10)
            loss.backward()
            train_loss += np.sqrt(loss.data.item()*mean_corrector)
            s += 1.
            optimizer.step()
    print('epoch: '+str(epoch)+' loss: '+str(train_loss/s))

# Testing the SAE
test_loss = 0
s = 0.
for id_user in range(nb_users):
    input = Variable(training_set[id_user]).unsqueeze(0)
    target = Variable(test_set[id_user])
    if torch.sum(target.data > 0) > 0:
        output = sae(input)
        target.require_grad = False
        output[target == 0] = 0      # I get error at this line
        loss = criterion(output, target)
        mean_corrector = nb_movies/float(torch.sum(target.data > 0) + 1e-10)
        test_loss += np.sqrt(loss.data.item()*mean_corrector)
        s += 1.
print('test loss: '+str(test_loss/s))


title: Deep learning : How to build character level embedding?

body:
I am trying to use character level embedding in my model but I have few doubts regarding character level embedding.

So for word level embedding :

Sentence = 'this is a example sentence'

create the vocab :

vocab = {'this' : 0 , 'is' :1 , 'a': 2 'example' : 3, 'sentence' : 4 }

encode the sentence :

encoded_sentence = [ 0, 1 , 2 , 3 , 4 ]

now send it to any pre-trained embedding like word2vec or glove :

each id will be replaced with 300 or embedding dim :

embedding_sentence = [ [ 0.331,0.11 , ----300th dim ] , [ 0.331,0.11 , ----300th dim ] , [ 0.331,0.11 , ----300th dim ] , [ 0.331,0.11 , ----300th dim ] , [ 0.331,0.11 , ----300th dim ] ]

and if we are dealing with batches then we pad the sentences

So the shape goes like this :

[ batch_size , max_sentence_length , embedding_dim ]

Now for character level embedding I have few doubts :

so for char level embedding :

Sentence = 'this is a example sentence'

create the char_vocab :

char_vocab = [' ', 'a', 'c', 'e', 'h', 'i', 'l', 'm', 'n', 'p', 's', 't', 'x']

int_to_vocab = {n:m for m,n in enumerate(char_vocab)}

encoded the sentence by char level :

Now here is my confusion , so in word embedding we first tokenise the sentence and then encode each token with vocab id ( word_id)

but for char embedding if I am tokenzing the sentence and then encoding with character level then shape will be 4 dim and I can't feed this to LSTM.

But if i am not tokenising and directly encoding raw text then it's 3 dim and I can feed it to LSTM

for example :

with tokenization :

 token_sentence = ['this','is','a','example','sentence']

encoded_char_level = []

for words in token_sentence:
    char_lvel = [int_to_vocab[char] for char in words]
    encoded_char_level.append(char_lvel)

it's look like this:

[[0, 1, 2, 3],
 [2, 3],
 [5],
 [6, 7, 5, 8, 9, 10, 6],
 [3, 6, 11, 0, 6, 11, 12, 6]]

Now we have to pad this for two level , one is char_level padding and second is sentence level padding:

char_level_padding:

[[0, 1, 2, 3, 0, 0, 0,0],
 [2, 3, 0, 0, 0, 0, 0, 0],
 [5, 0, 0, 0, 0, 0, 0, 0],
 [6, 7, 5, 8, 9, 10, 6, 0],
 [3, 6, 11, 0, 6, 11, 12, 6]]

Now if we have 4 sentences then we have to pad each sentence with max sentence len so shape will be :

[batch_size , max_sentence_length , max_char_length ]

Now if we pass this to embedding layer then:

[ batch_size , max_sentence_length, max_char_length , embedding_dim ]

Which is 4 dim.

How to encode sentences with character level and use it with tensorflow LSTM layer?

Because lstm takes 3 dim input [ batch_size , max_sequence_length , embedding_dim ]

Can I use it something like :

[ Batch_size , ( max_sentence_length x max_char_length ) , dim ]

so for example :

[ 12 , [ 3 x 4 ] , 300 ]


title: Automatically set cuda variables in pytorch

body:
I'm trying to automatically set Variables to use cuda if a flag use_gpu is activated.

Usually I would do:

import torch
from torch import autograd
use_gpu = torch.cuda.is_available()

foo = torch.randn(5)

if use_gpu:
    fooV = autograd.Variable(foo.cuda())
else:
    fooV = autograd.Variable(foo)

However, to do things faster when I have to initialise variables in different parts of the code I wanted to do something like this:

import torch
from torch import autograd
use_gpu = torch.cuda.is_available()

class Variable(autograd.Variable):
    def __init__(self, data, *args, **kwargs):
        if use_gpu:
            data = data.cuda()
        super(Variable, self).__init__(data, *args, **kwargs)

foo = torch.randn(5)
fooV = Variable(foo)

Unfortunately, this doesn't seem to be working and fooV doesn't contain a cuda Tensor when the use_gpu is True.


title: Weighted Average of PyTorch Tensors

body: I have two Pytorch tensors of the form [y11, y12] and [y21, y22]. How do I get the weighted mean of the two tensors?


title: Torchvision 0.2.1 transforms.Normalize does not work as expected

body:
I am trying a new code using Pytorch. In this code, to load the dataset (CIFAR10), I am using torchvision's datasets. I define two transform functions ToTensor() and Normalize(). After normalize I expect the data in the dataset should be between 0 and 1. But the max value is still 255. I also inserted a print statement inside the '__call__' function of Normalize class in transforms.py (Lib\site-packages\torchvision\transforms\transforms.py). This print is not printed while running the code too. Not sure what is happening. Every page I visited in the internet, mentions the usage almost the same way as I do. For example some sites I visited
https://github.com/adventuresinML/adventures-in-ml-code/blob/master/pytorch_nn.py
https://github.com/pytorch/tutorials/blob/master/beginner_source/blitz/cifar10_tutorial.py

My code is given below. This reads the dataset with and without Normalize, then prints some stats. The min and max printed is an indicator of whether the data is normalized or not.

import torchvision as tv
import numpy as np

dataDir = 'D:\\general\\ML_DL\\datasets\\CIFAR'

trainTransform  = tv.transforms.Compose([tv.transforms.ToTensor()])
trainSet        = tv.datasets.CIFAR10(dataDir, train=True, download=False, transform=trainTransform)
print (trainSet.train_data.mean(axis=(0,1,2))/255)
print (trainSet.train_data.min())
print (trainSet.train_data.max())
print (trainSet.train_data.shape)

trainTransform  = tv.transforms.Compose([tv.transforms.ToTensor(), tv.transforms.Normalize((0.4914, 0.4822, 0.4466), (0.247, 0.243, 0.261))])
trainSet        = tv.datasets.CIFAR10(dataDir, train=True, download=False, transform=trainTransform)
print (trainSet.train_data.mean(axis=(0,1,2))/255)
print (trainSet.train_data.min())
print (trainSet.train_data.max())
print (trainSet.train_data.shape)

The output looks like,

[ 0.49139968  0.48215841  0.44653091]
0
255
(50000, 32, 32, 3)
[ 0.49139968  0.48215841  0.44653091]
0
255
(50000, 32, 32, 3)

Please help me understand this better. As most of the functions I tried, ends up with similar results - for example Grayscale, CenterCrop too.


title: PyTorch model saving error: "Can't pickle local object"

body:
When I try to save the PyTorch model with this piece of code:
checkpoint = {'model': Net(), 'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

I get the following error:
    E:\PROGRAM FILES\Anaconda\envs\staj_projesi\lib\site-packages\torch\serialization.py:251: UserWarning: Couldn't retrieve source code for container of type Net. It won't be checked for correctness upon loading.
...

      "type " + obj.__name__ + ". It won't be checked "
    Can't pickle local object 'trainModel.<locals>.Net'

When I try to save the PyTorch model with this piece of code:
checkpoint = {'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

I don't don't get any errors, but I want to save the ANN class. How can I solve this problem? Also, I could save the model with the first structure in the other projects before


I am new to the world of neural networks and I am trying to implement a CNN generator from this model  and these equations (N=32) in order to make motion generation. I wrote the following code, where H_txt is a dictionary containing, as keys, the name of my clips and as values, a vector representing the action shown in the clip, and z is a white gaussian noise of dimension (1, 256).

N=32

class CNNGenerator(nn.Module):

    def __init__(self, htxt = H_txt):
        super(CNNGenerator, self).__init__()

        self.htxt = htxt
        self.conv1 = nn.Conv1d(1, 1, 3)
        self.conv2 = nn.Conv1d(1, 1, 3)
        self.conv3 = nn.Conv1d(1, 1, 3)
        self.conv4 = nn.Conv1d(4, 4, 3)
        self.conv5 = nn.Conv1d(2, 2, 3)
        self.conv6 = nn.Conv1d(8, 8, 3)
        self.conv7 = nn.Conv1d(4, 4, 3)
        self.conv8 = nn.Conv1d(16, 16, 3)
        self.conv9 = nn.Conv1d(8, 8, 3)
        self.conv10 = nn.Conv1d(32, 32, 3)
        self.conv11 = nn.Conv1d(16, 16, 3)
        self.conv12 = nn.Conv1d(32, 32, 3)
        self.conv13 = nn.Conv1d(1, 1, 3)
        self.conv14 = nn.Conv1d(2, 2, 3)
        self.conv15 = nn.Conv1d(2, 2, 3)
        self.conv16 = nn.Conv1d(4, 4, 3)
        self.conv17 = nn.Conv1d(4, 4, 3)
        self.conv18 = nn.Conv1d(8, 8, 3)
        self.conv19 = nn.Conv1d(8, 8, 3)
        self.conv20 = nn.Conv1d(16, 16, 3)
        self.conv21 = nn.Conv1d(16, 16, 3)
        self.conv22 = nn.Conv1d(32, 32, 3)
        self.conv23 = nn.Conv1d(32, 32, 3)

    def forward(self, x):
        x[0] = self.conv1(F.relu(self.conv2(z) + self.htxt))

        x[1] = self.conv3(F.relu(self.conv4(z) + self.htxt))
        x[2] = self.conv5(F.relu(self.conv6(z) + self.htxt))
        x[3] = self.conv7(F.relu(self.conv8(z) + self.htxt))
        x[4] = self.conv9(F.relu(self.conv10(z) + self.htxt))
        x[5] = self.conv11(F.relu(self.conv12(z) + self.htxt))
        h = np.zeros(np.log2(N))

        h[0] = x[0]

        h[1] = nn.AdaptiveAvgPool1d(2*h[0]) + self.conv13(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv14(h[0])+x[1]))))
        h[2] = nn.AdaptiveAvgPool1d(2*h[1]) + self.conv15(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv16(h[1])+x[2]))))
        h[3] = nn.AdaptiveAvgPool1d(2*h[2]) + self.conv17(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv18(h[2])+x[3]))))
        h[4] = nn.AdaptiveAvgPool1d(2*h[3]) + self.conv19(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv20(h[3])+x[4]))))
        h[5] = nn.AdaptiveAvgPool1d(2*h[4]) + self.conv21(F.relu(nn.AdaptiveAvgPool1d(2*(self.conv22(h[4])+x[5]))))
        A = self.conv23(h[np.log2(N)])
        return A

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

    net = CNNGenerator()
    z= torch.randn(1, 1, 256)       #k=256
    out = net(z)
    print(out)

When I run my code, I get the following error message, coming from my  forward function :

TypeError: unsupported operand type(s) for +: 'Tensor' and 'dict'

My code doesn't like the fact that I try to sum a tensor with a dictionary. I haven't found any solution online, so I was wondering if it was possible to sum my tensor with a dictionary? Is there a function that can convert my dictionary into a tensor?
I tried to convert the dictionary into an array with np.asarray() but I got an error message saying I can't use a numpy function for this.

Thanks a lot for reading my message


title: Adding a linear layer to an existing model on Pytorch

body:
I'm trying to add a new layer to an existing network (as the first layer) and train it on the original input. When I add a convolutional layer everything works perfectly but when I change it to linear it doesn't seem to train. Any ideas why?
Here is the whole network:

class ActorCritic(torch.nn.Module): #original model
    def __init__(self, num_inputs, action_space):
        super(ActorCritic, self).__init__()
        self.conv1 = nn.Conv2d(num_inputs, 32, 3, stride=2, padding=1)
        self.conv2 = nn.Conv2d(32, 32, 3, stride=2, padding=1)
        self.conv3 = nn.Conv2d(32, 32, 3, stride=2, padding=1)
        self.conv4 = nn.Conv2d(32, 32, 3, stride=2, padding=1)

        self.lstm = nn.LSTMCell(32 * 3 * 3, 256)

        num_outputs = action_space.n
        self.critic_linear = nn.Linear(256, 1)
        self.actor_linear = nn.Linear(256, num_outputs)

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.conv1(inputs))
        x = F.elu(self.conv2(x))
        x = F.elu(self.conv3(x))
        x = F.elu(self.conv4(x))
        x = x.view(-1, 32 * 3 * 3)
        hx, cx = self.lstm(x, (hx, cx))
        x = hx
        return self.critic_linear(x), self.actor_linear(x), (hx, cx)

class TLModel(torch.nn.Module): #new model
    def __init__(self, pretrained_model, num_inputs):
        super(TLModel, self).__init__()
        self.new_layer = nn.Linear(1*1*42*42, 1*1*42*42)
        self.pretrained_model = pretrained_model

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.new_layer(inputs.view(-1, 1*1*42*42)))
        return self.pretrained_model((x.view(1,1,42,42), (hx, cx)))

I tried different activation functions (not just elu). it works with conv:

 class TLModel(torch.nn.Module):
    def __init__(self, pretrained_model, num_inputs):
        super(TLModel, self).__init__()
        self.new_layer = nn.Conv2d(num_inputs, num_inputs, 1)
        self.pretrained_model = pretrained_model

    def forward(self, inputs):
        inputs, (hx, cx) = inputs
        x = F.elu(self.new_layer(inputs))
        return self.pretrained_model((x, (hx, cx)))

The number of inputs is 1 and the size of an input is 1x1x42x42


title: How to find the max index for each row in a tensor object?

body:
So I'm creating a pytorch model and for the forward pass, I'm applying my forward pass method to get the scores tensor which contains the prediction scores for each class. The shape of this tensor is [100, 10]. Now, I want to get the accuracy by comparing it to y which contains the actual scores. This tensor has the shape [100]. To compare the two I'll be using torch.mean(scores == y) and I'll count how many are the same.

The problem is that I need to convert the scores tensor so that each row simply contains the index of the highest value in each row. For example if the tensor looked like this,

tensor(
    [[0.3232, -0.2321, 0.2332, -0.1231, 0.2435, 0.6728],

    [0.2323, -0.1231, -0.5321, -0.1452, 0.5435, 0.1722],

    [0.9823, -0.1321, -0.6433, 0.1231, 0.023, 0.0711]]
)

Then I'd want it to be converted so that it looks like this.

tensor([5, 4, 0])

How could I do that?


title: How can torch multiply two 10000*10000 matrices in almost zero time? Why does the speed change so much from 349 ms down to 999 µs?

body:
Here is an excerpt from Jupyter:
In [1]:
import torch, numpy as np, datetime
cuda = torch.device('cuda')

In [2]:
ac = torch.randn(10000, 10000).to(cuda)
bc = torch.randn(10000, 10000).to(cuda)
%time cc = torch.matmul(ac, bc)
print(cc[0, 0], torch.sum(ac[0, :] * bc[:, 0]))

Wall time: 349 ms
tensor(17.0374, device='cuda:0') tensor(17.0376, device='cuda:0')

The time is low but still reasonable (0.35 sec for 1e12 multiplications)
But if we repeat the same:
ac = torch.randn(10000, 10000).to(cuda)
bc = torch.randn(10000, 10000).to(cuda)
%time cc = torch.matmul(ac, bc)
print(cc[0, 0], torch.sum(ac[0, :] * bc[:, 0]))

Wall time: 999 µs
tensor(-78.7172, device='cuda:0') tensor(-78.7173, device='cuda:0')

1e12 multiplications in 1ms?!
Why did the time change from 349ms to 1ms?
Info:

Tested on GeForce RTX 2070;
Can be reproduced on Google Colab.


title: ValueError: sampler option is mutually exclusive with shuffle pytorch

body:
i'm working on face recognition project using pytorch and mtcnn and after trained my training dataset , now i want to make prediction on test data set

this my trained code

optimizer = optim.Adam(resnet.parameters(), lr=0.001)
scheduler = MultiStepLR(optimizer, [5, 10])

trans = transforms.Compose([
   np.float32,
   transforms.ToTensor(),
   fixed_image_standardization
])
dataset = datasets.ImageFolder(data_dir, transform=trans)
img_inds = np.arange(len(dataset))
np.random.shuffle(img_inds)
train_inds = img_inds[:int(0.8 * len(img_inds))]
val_inds = img_inds[int(0.8 * len(img_inds)):]

train_loader = DataLoader(
   dataset,
   num_workers=workers,
   batch_size=batch_size,
   sampler=SubsetRandomSampler(train_inds)
)
val_loader = DataLoader(
   dataset,
   shuffle=True,
   num_workers=workers,
   batch_size=batch_size,
   sampler=SubsetRandomSampler(val_inds)
)

and if remove sampler=SubsetRandomSampler(val_inds) and put val_inds instead it will rise this error

  val_inds
      ^
  SyntaxError: positional argument follows keyword argument

i want to make prediction (select randomly from test data set) in pytorch?thats why i should use shuffle=True
i followed this repo facenet-pytorch


title: Train-Valid-Test split for custom dataset using PyTorch and TorchVision

body:
I have some image data for a binary classification task and the images are organised into 2 folders as data/model_data/class-A and data/model_data/class-B.

There are a total of N images. I want to have a 70/20/10 split for train/val/test.
I am using PyTorch and Torchvision for the task. Here is the code I have so far.

from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils, datasets, models

data_transform = transforms.Compose([
    transforms.RandomResizedCrop(224),
    transforms.RandomHorizontalFlip(),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])])

model_dataset = datasets.ImageFolder(root, transform=data_transform)
train_count = int(0.7 * total_count)
valid_count = int(0.2 * total_count)
test_count = total_count - train_count - valid_count
train_dataset, valid_dataset, test_dataset = torch.utils.data.random_split(model_dataset, (train_count, valid_count, test_count))
train_dataset_loader = torch.utils.data.DataLoader(train_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER)
valid_dataset_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=BATCH_SIZE, shuffle=True, num_workers=NUM_WORKER)
test_dataset_loader  = torch.utils.data.DataLoader(test_dataset , batch_size=BATCH_SIZE, shuffle=False,num_workers=NUM_WORKER)
dataloaders = {'train': train_dataset_loader, 'val': valid_dataset_loader, 'test': test_dataset_loader}

I feel that this isn't the correct way to be doing this because of 2 reasons.

I am applying the same transform to all the splits. (This is not what I want to do, obviously! The solution for this is most probably the answer here.)
Usually people first separate the original data into test/train and then they
separate train  into train/val, whereas I am directly separating the
original data into train/val/test. (Is this correct?)

So, my question is, is what I am doing correct? (Probably not)
And if it is not correct, how do I go about writing the data loaders to achieve the required splits, so that I can apply separate transforms to each of train/test/val?


title: Different Pytorch random initialization with the same seed

body:
I am very new to Pytorch, so I apologise if the question is very straightforward.
My problem is that I have defined class net1 and initialised its parameters randomly with a fixed manual seed.

random.seed(opt.manualSeed)
torch.manual_seed(opt.manualSeed)
if torch.cuda.is_available():
    torch.cuda.manual_seed_all(opt.manualSeed)

class net1(nn.Module):
    def __init__(self):
        super(net1, self).__init__()
        self.main_body = nn.Sequential(
            # Define the layers...#
        )
    def forward(self, x):
        return self.main_body(x)

# custom weights initialization called on net1
def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

net1_ = net1()
net1_.apply(weights_init)

However, when I add another class net2 to the code:

class net2(nn.Module):
    def __init__(self):
        super(net2, self).__init__()
        self.main_body = nn.Sequential(
            # Define the layers
        )
    def forward(self, x):
        return self.main_body(x)

net2_ = net2()

and instantiate it, even though I do not use it anywhere else and it is not connected to my main graph (which is built on net1_), I get different outputs from my graph.
Is it a reasonable outcome?


title: Choosing the learning_rate using fastai's learn.lr_find()

body:
I am going over this Heroes Recognition ResNet34 notebook published on Kaggle.

The author uses fastai's learn.lr_find() method to find the optimal learning rate.

Plotting the loss function against the learning rate yields the following figure:

It seems that the loss reaches a minimum for 1e-1, yet in the next step the author passes 1e-2 as the max_lr in fit_one_cycle in order to train his model:

learn.fit_one_cycle(6,1e-2)

Why use 1e-2 over 1e-1 in this example? Wouldn't this only make the training slower?


title: How to Save pytorch tensor in append mode

body:
How to save several tensor appending using torch.save()?

For example

for i in range(20):
       ......
    loss = criterion(scores, labels)
    torch.save(loss,'loss.pt')

How to save these all 20 losses?


title: Pytorch and Torchvision are compiled different CUDA versions

body:
RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA versions. PyTorch has CUDA Version=10.2 and torchvision has CUDA Version=10.1. Please reinstall the torchvision that matches your PyTorch install.

I am trying to run YOLACT on my Google Colab and found this error. Can someone help solve this issue?


title: Why do we normalize the image to mean=0.5, std=0.5?

body:
I was lookking for GAN code in Github. The code I found uses pytorch. In this code, we first normalized the image to mean = 0.5, std = 0.5. Normally, normalize to min = 0 and max = 1. Or normal distribution with mean = 0 and std = 1. Why is this normalized to mean = 0.5 and std = 0.5?

transformtransfo  = transforms.Compose([
        transforms.ToTensor(),
        transforms.Normalize(mean=(0.5, 0.5, 0.5), std=(0.5, 0.5, 0.5))
])


title: Found no NVIDIA driver on your system error on WSL2 conda environment with Python 3.7

body:
I have a Nvidia 1080Ti GPU, and I want to run Pytorch on WSL2, but I got error "Found no NVIDIA driver on your system" but I did installed the NVIDIA driver. Here is the step I did.

I installed WSL2, and installed NVIDIA driver for Cuda on WSL from GeForce Driver:
https://developer.nvidia.com/cuda/wsl/download

I activate a clean conda environment with Python 3.7

Then I run the Pytorch installation:
conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

Then the error occurred saying Found no NVIDIA driver. I came across a post on Pytorch forum, and someone did get it to run in a similar settings: Ubuntu 18.04 + Conda + Pytorch
https://discuss.pytorch.org/t/found-no-nvidia-driver-on-your-system-but-its-there/35063/4
I don't have multiple GPUs, so I don't know how to get my driver recognized in WSL2. Thank you for any thoughts!


My data iterator currently runs on the CPU as device=0 argument is deprecated. But I need it to run on the GPU with the rest of the model etc.

Here is my code:

pad_idx = TGT.vocab.stoi["<blank>"]
model = make_model(len(SRC.vocab), len(TGT.vocab), N=6)
model = model.to(device)
criterion = LabelSmoothing(size=len(TGT.vocab), padding_idx=pad_idx, smoothing=0.1)
criterion = criterion.to(device)
BATCH_SIZE = 12000
train_iter = MyIterator(train, device, batch_size=BATCH_SIZE,
                        repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),
                        batch_size_fn=batch_size_fn, train=True)
valid_iter = MyIterator(val, device, batch_size=BATCH_SIZE,
                        repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),
                        batch_size_fn=batch_size_fn, train=False)
#model_par = nn.DataParallel(model, device_ids=devices)

The above code gives this error:

The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.
The `device` argument should be set by using `torch.device` or passing a string as an argument. This behavior will be deprecated soon and currently defaults to cpu.

I have tried passing in 'cuda' as an argument instead of device=0 but I receive this error:

<ipython-input-50-da3b1f7ed907> in <module>()
    10     train_iter = MyIterator(train, 'cuda', batch_size=BATCH_SIZE,
    11                             repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),
---> 12                             batch_size_fn=batch_size_fn, train=True)
    13     valid_iter = MyIterator(val, 'cuda', batch_size=BATCH_SIZE,
    14                             repeat=False, sort_key=lambda x: (len(x.src), len(x.trg)),

TypeError: __init__() got multiple values for argument 'batch_size'

I have also tried passing in device as an argument. Device being defined as device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

But receive the same error as just above.

Any suggestions would be much appreciated, thanks.


title: How to "denormalize" a pytorch tensor?

body:
I would like to normalize the labels for my neural network but also be able to reverse the normalization process, so that I can scale my prediction outputs back to the original size.

My labels are of the form:

[[ 2.25, 2345123.23],
 [ 1.13, 234565.11],
 ...
 [0.12, 990232.98]]

Now I would like to normalize each column such that the values range from 0 to 1.

I found that pytorch has the torch.nn.functional.normalize function which allows me to normalize along a specific dimension using whichever p-norm I desire.

Is there an existing inverse function that allows me to scale my normalized values?

Currently, I am using a custom function that performs feature scaling as follows which scales both columns together, however, I wanted to make sure I am not creating a function that already exists in the torch library before rewriting it.

 A = lbl_min
 B = lbl_max
 a = lower_bound
 b = upper_bound
 self.labels = a*np.ones(shape) + (self.labels - A*np.ones(shape))*(b-a)/(B-A)

Additional information:
I am using a regression model to output continuous real world values. I need to "denormalize" my predictions in order to validate them against the true values. Also, if I were to use my model in a practical application, I would need my output back in its original units.

My neural network predicts multiple values at once that are in very different ranges.
The two values have different distributions but can be predicted from the same input.
My input is an image into two CNN layers that then are fed into two fully connected layers