title: Loss dimensionality issue in PyTorch (sequence to label learning)

body:
I am doing a sequence to label learning model in PyTorch. I have two sentences and I am classifying whether they are entailed or not (SNLI dataset). I concatenate two 50 word sentences together (sometimes padded) into a vector of length 100. I then send in minibatches into word embeddings -> LSTM -> Linear layer. I am doing cross entropy loss but I need a vector of [mini_batch, C] to go into the CrossEntropyLoss function. Instead I still have the 100 words in my vector as [mini_batch, 100, C]

Here is my model:

class myLSTM(nn.Module):
    def __init__(self, h_size=128, v_size=10, embed_d=300, mlp_d=256):
        super(myLSTM, self).__init__()
        self.embedding = nn.Embedding(v_size, embed_d)
        self.lstm = nn.LSTM(embed_d, h_size, num_layers=1, bidirectional=True, batch_first=True)
        self.mlp = nn.Linear(mlp_d, 1024)

        # Set static embedding vectors
        self.embedding.weight.requires_grad = False

        #self.sm = nn.CrossEntropyLoss()

    def display(self):
        for param in self.parameters():
            print(param.data.size())

    def filter_params(self):
        # Might not be compatible with python 3
        #self.parameters = filter(lambda p: p.requires_grad, self.parameters())
        pass

    def init_hidden(self):
        # Need to init hidden weights in LSTM
        pass

    def forward(self, sentence):
        print(sentence.size())
        embeds = self.embedding(sentence)
        print(embeds.size())
        out, _ = self.lstm(embeds)
        print(out.size())
        out = self.mlp(out)
        return out

My training sequences with output:

batch_size = 3
SGD_optimizer = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.01, weight_decay=1e-4)
ADM_optimizer = optim.Adam(filter(lambda p: p.requires_grad, model.parameters()), lr=0.01)

criterion = nn.CrossEntropyLoss()
num_epochs = 50

from torch.autograd import Variable
from torch import optim

for epoch in range(num_epochs):
    print("Epoch {0}/{1}: {2}%".format(epoch, num_epochs, float(epoch)/num_epochs))
    for start, end in tqdm(batch_index_gen(batch_size, len(n_data))):

        # Convert minibatch to numpy
        s1, s2, y = convert_to_numpy(n_data[start:end])

        # Convert numpy to Tensor
        res = np.concatenate((s1,s2), axis=1) # Attach two sentences into 1 input vector
        input_tensor = torch.from_numpy(res).type(torch.LongTensor)
        target_tensor = torch.from_numpy(y).type(torch.FloatTensor)
        data, target = Variable(input_tensor), Variable(target_tensor)

        # Zero gradients
        SGD_optimizer.zero_grad()

        # Forward Pass
        output = model.forward(data)
        print("Output size: ")
        print(output.size())
        print("Target size: ")
        print(target.size())
        # Calculate loss with respect to training labels
        loss = criterion(output, target)

        # Backprogogate and update optimizer
        loss.backward()
        SGD_optimizer.step()
        #ADAM_optimizer.step()

output:

Epoch 0/50: 0.0%
torch.Size([3, 100])
torch.Size([3, 100, 300])
torch.Size([3, 100, 256])
Output size:
torch.Size([3, 100, 1024])
Target size:
torch.Size([3])

error:

ValueError: Expected 2 or 4 dimensions (got 3)

EDITED -------------------------------------------------------------------

I have now got my model training but I am getting low accuracy. Is there an issue with my LSTM outputs being concatenated and then condensed to a smaller tensor to go through my linear layer?

New Model:

class myLSTM(nn.Module):
    def __init__(self, h_size=128, v_size=10, embed_d=300, mlp_d=256, num_classes=3, lstm_layers=1):
        super(myLSTM, self).__init__()
        self.num_layers = lstm_layers
        self.hidden_size = h_size
        self.embedding = nn.Embedding(v_size, embed_d)
        self.lstm = nn.LSTM(embed_d, h_size, num_layers=lstm_layers, bidirectional=True, batch_first=True)
        self.mlp = nn.Linear(2 * h_size * 2, num_classes)

        # Set static embedding vectors
        self.embedding.weight.requires_grad = False

    def forward(self, s1, s2):
        # Set initial states
        #h0 = Variable(torch.zeros(self.num_layers*2, s1.size(0), self.hidden_size)).cuda() # 2 for bidirection
        #c0 = Variable(torch.zeros(self.num_layers*2, s1.size(0), self.hidden_size)).cuda()

        batch_size = s1.size()[0]
        embeds_1 = self.embedding(s1)
        embeds_2 = self.embedding(s2)
        _, (h_1_last, _) = self.lstm(embeds_1)#, (h0, c0)) #note the change here. Last hidden state is taken
        _, (h_2_last, _) = self.lstm(embeds_2)#, (h0, c0))
        concat = torch.cat( (h_1_last, h_2_last), dim=2) #double check the dimension
        concat = concat.view(batch_size, -1)
        scores = self.mlp(concat)
        return scores

New Training

batch_size = 64
SGD_optimizer = optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=0.001, weight_decay=1e-4)

criterion = nn.CrossEntropyLoss()
num_epochs = 10
model.train()

if cuda:
    model = model.cuda()
    criterion = criterion.cuda()

from torch.autograd import Variable
from torch import optim

epoch_losses = []

for epoch in range(num_epochs):
    print("Epoch {0}/{1}: {2}%".format(epoch, num_epochs, 100*float(epoch)/num_epochs))

    # Batch loss aggregator
    losses = []

    for start, end in tqdm(batch_index_gen(batch_size, len(n_data))):
        # Convert minibatch to numpy
        s1, s2, y = convert_to_numpy(n_data[start:end])

        # Convert numpy to Tensor
        s1_tensor = torch.from_numpy(s1).type(torch.LongTensor)
        s2_tensor = torch.from_numpy(s2).type(torch.LongTensor)
        target_tensor = torch.from_numpy(y).type(torch.LongTensor)

        s1 = Variable(s1_tensor)
        s2 = Variable(s2_tensor)
        target = Variable(target_tensor)

        if cuda:
            s1 = s1.cuda()
            s2 = s2.cuda()
            target = target.cuda()

        # Zero gradients
        SGD_optimizer.zero_grad()

        # Forward Pass
        output = model.forward(s1,s2)

        # Calculate loss with respect to training labels
        loss = criterion(output, target)
        losses.append(loss.data[0])

        # Backprogogate and update optimizer
        loss.backward()
        SGD_optimizer.step()

    # concat losses to epoch losses
    epoch_losses += losses

training with tensor sizes printed:

Epoch 0/10: 0.0%
Batch size: 64
Sentences
torch.Size([64, 50])
torch.Size([64, 50])
torch.Size([64, 50, 300])
torch.Size([64, 50, 300])
Hidden states
torch.Size([2, 64, 128])
torch.Size([2, 64, 128])
Concatenated hidden states
torch.Size([2, 64, 256])
Reshaped tensors for linear layer
torch.Size([64, 512])
Linear propogation
torch.Size([64, 3])

Evaluation

def eval_model(model, mode='dev'):
    file_name = 'snli_1.0/snli_1.0_dev.jsonl' if mode == 'dev' else 'snli_1.0/snli_1.0_test.jsonl'

    dev_data, _ = obtain_data(file_name)
    dev_n_data = vocab.process_data(dev_data)

    print("Length of data: {}".format(len(dev_n_data)))

    eval_batch_size = 1024
    model.eval()

    total = len(dev_n_data)
    hit = 0
    correct = 0

    # Batch dev eval
    for start, end in batch_index_gen(eval_batch_size, len(dev_n_data)):

        s1, s2, y = convert_to_numpy(dev_n_data[start:end])

        s1_tensor = torch.from_numpy(s1).type(torch.LongTensor)
        s2_tensor = torch.from_numpy(s2).type(torch.LongTensor)
        target_tensor = torch.from_numpy(y).type(torch.LongTensor)

        s1 = Variable(s1_tensor, volatile=True)
        s2 = Variable(s2_tensor, volatile=True)
        target = Variable(target_tensor, volatile=True)

        if cuda:
            s1 = s1.cuda()
            s2 = s2.cuda()
            target = target.cuda()

        output = model.forward(s1,s2)
        loss = criterion(output, target)

        #print("output size: {}".format(output.size()))
        #print("target size: {}".format(target.size()))
        pred = output.data.max(1)[1] # get the index of the max log-probability
        #print(pred[:5])
        #print(output[:])
        correct += pred.eq(target.data).cpu().sum()

    return correct / float(total)

eval_model(model)


title: Pytorch on google-colaboratory GPU - Illegal memory access

body:
I am using pytorch(0.4.0) on google-colaboratory ( NVIDIA-SMI 396.44                 Driver Version: 396.44)

When running my code outside any function, I am able to send pytorch tensors and model to the GPU :

...
model.cuda()
data_tensor = data_tensor.cuda()
...

And my CNN model is trained successfully with 98% accurancy.

But when I put the same code in a function,

def main(...):
    ....
    model.cuda()
    data_tensor= data_tensor.cuda()
    ...

if __name__ == "__main__":
    main('...)

I have the following error:

cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/generic/THCTensorCopy.c:20

UPDATE(18/11/21):

It turned out that being part or not of a function is irrelevant. Usually, I have first a CUDNN_STATUS_EXECUTION_FAILED error then the second time a  cuda runtime error (77) as shown below. But it sometimes works a few times before failing.

CUDNN_STATUS_EXECUTION_FAILED (first try) :

RuntimeError                              Traceback (most recent call last)
<ipython-input-27-53476e08e017> in <module>()
      1 main('mnist', 'to', 'ndd', Xd=16, epo=5, bs=100, tXn=-1, vXn=300,
----> 2      lr=0.05, suf="s1", n_class=10, cuda=True)

<ipython-input-23-918584456207> in main(ds, framework, format, Xd, epo, bs, tXn, vXn, lr, suf, n_class, cuda)
     12     opt = torch.optim.SGD(net.parameters(), lr)
     13
---> 14     train(net, opt, Xd, epo, bs, cuda, tXn, tX, tT, vX, vT,lr)
     15

<ipython-input-26-6b574a9e8af6> in train(model, optimizer, Xd, epo, bs, cuda, Xn, tX, tT, vX, vT, lr)
     26             #t = t.cuda()
     27             optimizer.zero_grad()
---> 28             z = model(x)
     29             bat_loss = criterion(z, t)
     30             bat_loss.backward()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

<ipython-input-22-b4bc2e0b39b8> in forward(self, X)
     10         H0 = torch.zeros(self.n_H, X.size(0), self.Wh)
     11         C0 = torch.zeros(self.n_H, X.size(0), self.Wh)
---> 12         O, (Hn, Cn), = self.lstm1(X, (H0, C0))
     13         O = self.linear1(O[:, -1, :])
     14         return O

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in forward(self, input, hx)
    190             flat_weight=flat_weight
    191         )
--> 192         output, hidden = func(input, self.all_weights, hx, batch_sizes)
    193         if is_packed:
    194             output = PackedSequence(output, batch_sizes)

/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py in forward(input, *fargs, **fkwargs)
    321             func = decorator(func)
    322
--> 323         return func(input, *fargs, **fkwargs)
    324
    325     return forward

/usr/local/lib/python3.6/dist-packages/torch/nn/_functions/rnn.py in forward(input, weight, hx, batch_sizes)
    285             batch_first, dropout, train, bool(bidirectional),
    286             list(batch_sizes.data) if variable_length else (),
--> 287             dropout_ts)
    288
    289         if cx is not None:

RuntimeError: CUDNN_STATUS_EXECUTION_FAILED

cuda runtime error (77)  (other tries):

RuntimeError                              Traceback (most recent call last)
<ipython-input-28-53476e08e017> in <module>()
      1 main('mnist', 'to', 'ndd', Xd=16, epo=5, bs=100, tXn=-1, vXn=300,
----> 2      lr=0.05, suf="s1", n_class=10, cuda=True)

<ipython-input-23-918584456207> in main(ds, framework, format, Xd, epo, bs, tXn, vXn, lr, suf, n_class, cuda)
     12     opt = torch.optim.SGD(net.parameters(), lr)
     13
---> 14     train(net, opt, Xd, epo, bs, cuda, tXn, tX, tT, vX, vT,lr)
     15

<ipython-input-26-6b574a9e8af6> in train(model, optimizer, Xd, epo, bs, cuda, Xn, tX, tT, vX, vT, lr)
      4     if cuda and torch.cuda.is_available():
      5         print("tX type (before):", tX.type())
----> 6         model.cuda()
      7         tX = tX.cuda()
      8         tT = tT.cuda()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in cuda(self, device)
    247             Module: self
    248         """
--> 249         return self._apply(lambda t: t.cuda(device))
    250
    251     def cpu(self):

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    174     def _apply(self, fn):
    175         for module in self.children():
--> 176             module._apply(fn)
    177
    178         for param in self._parameters.values():

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/rnn.py in _apply(self, fn)
    109
    110     def _apply(self, fn):
--> 111         ret = super(RNNBase, self)._apply(fn)
    112         self.flatten_parameters()
    113         return ret

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    180                 # Tensors stored in modules are graph leaves, and we don't
    181                 # want to create copy nodes, so we have to unpack the data.
--> 182                 param.data = fn(param.data)
    183                 if param._grad is not None:
    184                     param._grad.data = fn(param._grad.data)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in <lambda>(t)
    247             Module: self
    248         """
--> 249         return self._apply(lambda t: t.cuda(device))
    250
    251     def cpu(self):

RuntimeError: cuda runtime error (77) : an illegal memory access was encountered at /pytorch/aten/src/THC/generic/THCTensorCopy.c:20


title: PyTorch mapping operators to functions

body:
What are all the PyTorch operators, and what are their function equivalents?

Eg, is a @ b equivalent to a.mm(b) or a.matmul(b)?

I'm after a canonical listing of operator -> function mappings.

I'd be happy to be given a PyTorch documentation link as an answer - my googlefu couldn't track it down.


title: Create a nnModule that's just the identity

body:
I'm trying to debug a pretty complex interaction between different nnModules. It would be very helpful for me to be able to replace one of them with just an identity network for debugging purposes. For example:

net_a = NetworkA()
net_b = NetworkB()
net_c = NetworkC()

input = Autograd.Variable(torch.rand(10,2))

out = net_a(input)
out = net_b(out)
out = net_c(out)

I would like to be able to just change the second line to net_b = IdentityNet(), instead of having to go through and reconnect all my As to Cs. But, when I make a completely empty nnModule, the optimizer throws ValueError: optimizer got an empty parameter list.

Is there any workaround this?

A minimum non-working example:

import torch.optim as optim

class IdentityModule(nnModule):
    def forward(self, inputs):
        return inputs

identity = IdentityModule()
opt = optim.Adam(identity, lr=0.001)
out = identity(any_tensor)
error = torch.mean(out)
error.backward()
opt.step()


title: Add extra dimension to an axes

body: I have a batch of segmentation masks of shape [5,1,100,100] (batch_size x dims x ht x wd) which I have to display in tensorboardX with an RGB image batch [5,3,100,100]. I want to add two dummy dimensions to the second axes of the segmentation mask to make it [5,3,100,100] so there will not be any dimension mismatch error when I pass it to torch.utils.make_grid. I have tried unsqueeze, expand and view but I am not able to do it. Any suggestions?


While working on Udacity's AI course I encountered the error in the title.

At first it looks like the image isn't formatted for the model, however the instruction for the course has us format the images using resize, so I get an image dimension of 3x224x224, that's 3 color channels and 224 pixels by 224 pixels. The image array is in the required dimensions, [3,224,224].

Return/Output:
(3, 224, 224)
(3, 224, 224)
torch.Size([1, 3, 224, 224])

Next, I figured it was the model, but when I go back, before the training loop and change the original model to fix this error I get a similar error in revers "up stream".

I want this predict function to use my model to predict the category of an image.

Posted are:

Original classifier
Save checkpoint block
Process image function
Predict image function

classifier = nn.Sequential(OrderedDict([
                      ('fc1', nn.Linear(25088, 500)),
                      ('relu', nn.ReLU()),
                      ('drop', nn.Dropout(0.25)),
                      ('fc2', nn.Linear(500, 102)),
                      ('output', nn.LogSoftmax(dim=1))
                      ]))

model.load_state_dict(state_dict)

checkpoint= {'input_size': 672,
            'output_size': 102,
            'hidden_layers': 224,
            'state_dict': model.state_dict()}

torch.save(checkpoint, 'checkpoint.pth')

def predict(image_path, model, topk=5):
''' Predict the class (or classes) of an image using a trained deep learning model.
'''

model= model.cuda()
model.eval()

# TODO: Implement the code to predict the class from an image file

pil_img= Image.open(image_path)
#print(pil_img)
processed_image= process_image(pil_img)
print(processed_image.shape)
torch_image= torch.from_numpy(processed_image)
torch_image= torch_image.unsqueeze_(0)
print(torch_image.shape)
torch_image= torch_image.float().to('cuda')
output= model.forward(torch_image)
#top_k= predict.topk(topk)
#print(top_k)

return torch_img

image_path = 'flowers/test/8/image_03299.jpg'
predict(image_path, model, 5)

> RuntimeError                              Traceback (most recent call last)
<ipython-input-21-d84c5999ad68> in <module>()
     23
     24 image_path = 'flowers/test/8/image_03299.jpg'
---> 25 predict(image_path, model, 5)

<ipython-input-21-d84c5999ad68> in predict(image_path, model, topk)
     16     print(torch_image.shape)
     17     torch_image= torch_image.float().to('cuda')
---> 18     output= model.forward(torch_image)
     19     #top_k= predict.topk(topk)
     20     #print(top_k)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     89     def forward(self, input):
     90         for module in self._modules.values():
---> 91             input = module(input)
     92         return input
     93

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     53
     54     def forward(self, input):
---> 55         return F.linear(input, self.weight, self.bias)
     56
     57     def extra_repr(self):

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
    992         return torch.addmm(bias, input, weight.t())
    993
--> 994     output = input.matmul(weight.t())
    995     if bias is not None:
    996         output += bias

    RuntimeError                              Traceback (most recent call last)
<ipython-input-21-d84c5999ad68> in <module>()
     23
     24 image_path = 'flowers/test/8/image_03299.jpg'
---> 25 predict(image_path, model, 5)

<ipython-input-21-d84c5999ad68> in predict(image_path, model, topk)
     16     print(torch_image.shape)
     17     torch_image= torch_image.float().to('cuda')
---> 18     output= model.forward(torch_image)
     19     #top_k= predict.topk(topk)
     20     #print(top_k)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/container.py in forward(self, input)
     89     def forward(self, input):
     90         for module in self._modules.values():
---> 91             input = module(input)
     92         return input
     93

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    489             result = self._slow_forward(*input, **kwargs)
    490         else:
--> 491             result = self.forward(*input, **kwargs)
    492         for hook in self._forward_hooks.values():
    493             hook_result = hook(self, input, result)

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     53
     54     def forward(self, input):
---> 55         return F.linear(input, self.weight, self.bias)
     56
     57     def extra_repr(self):

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
    992         return torch.addmm(bias, input, weight.t())
    993
--> 994     output = input.matmul(weight.t())
    995     if bias is not None:
    996         output += bias

RuntimeError: size mismatch, m1: [672 x 224], m2: [672 x 224] at /opt/conda/conda-bld/pytorch_1524584710464/work/aten/src/THC/generic/THCTensorMathBlas.cu:249


title: How to add a L2 regularization term in my loss function

body:
I’m going to compare the difference between with and without regularization, so I want to custom two loss functions.

My loss function with L2 norm:

###NET
class CNN(nn.Module):
def __init__(self):
    super(CNN,self).__init__()
    self.layer1 = nn.Sequential(
        nn.Conv2d(3, 16, kernel_size = 5, padding=2),
        nn.ReLU(),
        nn.MaxPool2d(2))
    self.layer2 = nn.Sequential(
        nn.Conv2d(16, 32, kernel_size = 5, padding=2),
        nn.ReLU(),
        nn.MaxPool2d(2))
    self.layer3 = nn.Sequential(
        nn.Conv2d(32, 32, kernel_size = 5, padding=2),
        nn.ReLU(),
        nn.MaxPool2d(4))
    self.fc = nn.Linear(32*32*32,11)
def forward(self, x):
    out = self.layer1(x)
    out = self.layer2(out)
    out = self.layer3(out)
    out = out.view(out.size(0), -1)
    out = self.fc(out)
    return out

net = CNN()

###OPTIMIZER
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr = LR, momentum = MOMENTUM)

  1.How can I add a L2 norm in my loss function?

  2.If I want to write the loss function by myself (without using optim.SGD) and do the grad-decent by autograd, how can I do?

Thanks for your help!


title: example of doing simple inference with pytorch-lightning

body:
I have an existing model where I load some pre-trained weights and then do inference (one image at a time) in pytorch. I am trying to basically convert it to a pytorch lightning module and am confused about a few things.

So currently, my __init__ method for the model looks like this:

self._load_config_file(cfg_file)
# just creates the pytorch network
self.create_network()

self.load_weights(weights_file)

self.cuda(device=0)  # assumes GPU and uses one. This is probably suboptimal
self.eval()  # inference mode

What I can gather from the lightning docs, I can pretty much do the same, except not to do the cuda() call. So something like:

self.create_network()

self.load_weights(weights_file)
self.freeze()  # inference mode

So, my first question is whether this is the correct way to use lightning? How would lightning know if it needs to use the GPU? I am guessing this needs to be specified somewhere.

Now, for the inference, I have the following setup:

def infer(frame):
    img = transform(frame)  # apply some transformation to the input
    img = torch.from_numpy(img).float().unsqueeze(0).cuda(device=0)
    with torch.no_grad():
        output = self.__call__(Variable(img)).data.cpu().numpy()
    return output

This is the bit that has me confused. Which functions do I need to override to make a lightning compatible inference?

Also, at the moment, the input comes as a numpy array. Is that something that would be possible from the lightning module or do things always have to use some sort of a dataloader?

At some point, I want to extend this model implementation to do training as well, so want to make sure I do it right but while most examples focus on training models, a simple example of just doing inference at production time on a single image/data point might be useful.

I am using 0.7.5 with pytorch 1.4.0 on GPU with cuda 10.1


title: Generating new images with PyTorch

body:
I am studying GANs I've completed the one course which gave me an example of a program that generates images based on examples inputed.

The example can be found here:

https://github.com/davidsonmizael/gan

So I decided to use that to generate new images based on a dataset of frontal photos of faces, but I am not having any success. Differently from the example above, the code only generates noise, while the input has actual images.

Actually I don't have any clue about what should I change to make the code point to the right direction and learn from images. I haven't change a single value on the code provided in the example, yet it does not work.

If anyone can help me understand this and point me to the right direction would be very helpful. Thanks in advance.

My Discriminator:

class D(nn.Module):

    def __init__(self):
        super(D, self).__init__()
        self.main = nn.Sequential(
                nn.Conv2d(3, 64, 4, 2, 1, bias = False),
                nn.LeakyReLU(0.2, inplace = True),
                nn.Conv2d(64, 128, 4, 2, 1, bias = False),
                nn.BatchNorm2d(128),
                nn.LeakyReLU(0.2, inplace = True),
                nn.Conv2d(128, 256, 4, 2, 1, bias = False),
                nn.BatchNorm2d(256),
                nn.LeakyReLU(0.2, inplace = True),
                nn.Conv2d(256, 512, 4, 2, 1, bias = False),
                nn.BatchNorm2d(512),
                nn.LeakyReLU(0.2, inplace = True),
                nn.Conv2d(512, 1, 4, 1, 0, bias = False),
                nn.Sigmoid()
                )

    def forward(self, input):
        return self.main(input).view(-1)

My Generator:

class G(nn.Module):

    def __init__(self):
        super(G, self).__init__()
        self.main = nn.Sequential(
                nn.ConvTranspose2d(100, 512, 4, 1, 0, bias = False),
                nn.BatchNorm2d(512),
                nn.ReLU(True),
                nn.ConvTranspose2d(512, 256, 4, 2, 1, bias = False),
                nn.BatchNorm2d(256),
                nn.ReLU(True),
                nn.ConvTranspose2d(256, 128, 4, 2, 1, bias = False),
                nn.BatchNorm2d(128),
                nn.ReLU(True),
                nn.ConvTranspose2d(128, 64, 4, 2, 1, bias = False),
                nn.BatchNorm2d(64),
                nn.ReLU(True),
                nn.ConvTranspose2d(64, 3, 4, 2, 1, bias = False),
                nn.Tanh()
                )

    def forward(self, input):
        return self.main(input)

My function to start the weights:

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

Full code can be seen here:

https://github.com/davidsonmizael/criminal-gan

Noise generated on epoch number 25:

Input with real images:


title: How to import Pytorch model into MATLAB

body:
I've created a model within Pytorch and I wish to transfer it to MATLAB, a minimal example is shown

import torch.nn as nn
import torch
class cnn(nn.Module):
    def __init__(self):
        super(cnn, self).__init__()
        self.fc1 = nn.Sequential(
            nn.Linear(10, 1),
            nn.ReLU(True)
        )

    def forward(self, x):
        out = self.fc1(x)
        return out
the_net = cnn()
torch.save(the_net,'desperation.h5')

Within MATLAB I then call

net = importKerasLayers('desperation.h5')

This gives the error message

Error using importKerasLayers (line 104)
Unable to read HDF5 file 'desperation.h5'. The error message was: 'The filename specified was either
not found on the MATLAB path or it contains unsupported characters.''

The file is on the path and I am able to load the model back into Python. What I really want is any solution that allows me to transfer the model from Pytorch into MATLAB without manually copying all of the weights.

I am running MATLAB 2018b, Python 3.6 and Pytorch 0.4.0


title: Tying weights in neural machine translation

body:
I want to tie weights of the embedding layer and the next_word prediction layer of the decoder. The embedding dimension is set to 300 and the hidden size of the decoder is set to 600. Vocabulary size of the target language in NMT is 50000, so embedding weight dimension is 50000 x 300 and weight of the linear layer which predicts the next word is 50000 x 600.

So, how can I tie them? What will be the best approach to achieve weight tying in this scenario?


title: How to declare a scalar as a parameter in pytorch?

body:
I am new to pytorch. I want to know how to declare a scalar as a parameter. I am wondering what's the difference between the following two ways?

x = torch.randn(1,1, requires_grad=True)

and

tensor = torch.randn(1,1)
x = Variable(tensor, requires_grad=True)


title: PyTorch specify model parameters

body:
I am trying to create a convolutional model in PyTorch where

one layer is fixed (initialized to prescribed values)
another layer is learned (but initial guess taken from prescribed values).

Here is a sample code for model definition:

import torch.nn as nn

class Net(nn.Module):
    def __init__(self, weights_fixed, weights_guess):
        super(Net, self).__init__()
        self.convL1 = nn.Conv1d(1, 3, 3, bias=False)
        self.convL1.weight = weights_fixed # I want to keep these weights fixed

        self.convL2 = nn.Conv1d(3, 1, 1, bias=False)
        self.convL1.weight = weights_guess # I want to learn these weights

    def forward(self, inp_batch):
        out1 = self.convL1(inp_batch)
        out2 = self.convL2(out1)

        return out2

and the sample use:

weights_fixed = ...
weights_guess = ...

model = Net(weights_fixed, weights_guess)

loss_fn = nn.CrossEntropyLoss()
optim = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)

train_dataset = ... #define training set here

for (X, y) in train_dataset:
    optim.zero_grad()
    out = model(X)
    loss = loss_fn(out, y)
    loss.backward()
    optim.step()

How can I make the weights
weights_fixed - fixed and
weights_guess - learnable?

My guess would be
    weights_fixed = nn.Parameter(W1,requires_grad=False)
    weights_guess = nn.Parameter(W2,requires_grad=True)
where for the sake of completeness
    import numpy as np
    import torch

krnl = np.zeros((5,order+1))
krnl[:,0] = [ 0. , 1., 0. ]
krnl[:,1] = [-0.5, 0., 0.5]
krnl[:,2] = [ 1. ,-2., 1. ]
W1 = torch.tensor(krnl)

a = np.array((1.,2.,3.))
W2 = torch.tensor(a)

But I am utterly confused. Any suggestions or references would be greatly appreciated. Of course I went over PyTorch docs, but it did not add clarity to my understanding.


title: PyTorch - RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress

body:
I have the problem that when I run my model on Google Colab I very frequently get this error:

RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

I does not happen all the time. Sometimes it works fine but then again it doesn't.
(If it works or not seems to be related to the batch size, see edit at the bottom.)

When running only on CPU there is no problem, so it seems related to GPU/CUDA.

The error traceback shows that the error happens at some point during backward().

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-1-c8bbb05fb58a> in <module>()
     65 out   = model(chars, pos)
     66 loss  = F.binary_cross_entropy_with_logits(out, labels)
---> 67 loss.backward()

1 frames
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    164                 products. Defaults to ``False``.
    165         """"""
--> 166         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    167
    168     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     97     Variable._execution_engine.run_backward(
     98         tensors, grad_tensors, retain_graph, create_graph,
---> 99         allow_unreachable=True)  # allow_unreachable flag
    100
    101

RuntimeError: transform: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered

Once this error has occurred I cannot do anything related with GPU on that runtime. Whenever I try then to create some tensors or so on GPU then I get a slightly different error message:

RuntimeError: CUDA error: an illegal memory access was encountered

I need to restart the runtime/notebook in order to do something on the GPU (with Pytorch haven't tried other frameworks).

Here is a code snippet that should be able to reproduce the issue in Google Colab:

import os
#os.environ['CUDA_LAUNCH_BLOCKING'] = '1'

import torch
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print(torch.__version__, torch.cuda.get_device_name(0))
from torch import nn
from torch.nn import functional as F
class MyModel(nn.Module):
    def __init__(self, char_vocab, num_pos, dim, hidden, dropout):
        super().__init__()
        self.emb_char  = nn.Embedding(char_vocab, dim)
        self.cnn1 = nn.Conv1d(dim, dim // 2, kernel_size=3)
        self.bn1  = nn.BatchNorm1d(dim // 2)
        self.cnn2 = nn.Conv1d(dim // 2, dim // 4, kernel_size=2)
        self.bn2  = nn.BatchNorm1d(dim // 4)
        self.pooling = nn.MaxPool1d(2)
        self.lin1 = nn.Linear(11, hidden)
        self.bn3  = nn.BatchNorm1d(hidden)
        self.lin2 = nn.Linear(hidden, dim // 4)
        self.bn4  = nn.BatchNorm1d(dim // 4)

        self.out  = nn.Linear(dim // 4, 1)
        self.drop = nn.Dropout(dropout)

    def forward(self, chars, pos):
        x = self.emb_char(chars).transpose(1, 2)
        x = self.drop(x)

        x = self.cnn1(x)
        x = self.bn1(x)
        x = self.pooling(x)
        x = F.relu(x)

        x = self.cnn2(x)
        x = self.bn2(x)
        x = self.pooling(x).squeeze(-1)
        x = F.relu(x)
        x = self.drop(x)

        y = F.relu(self.lin1(pos))
        y = self.bn3(y)
        y = self.drop(y)
        y = F.relu(self.lin2(y))
        y = self.bn4(y)
        y = self.drop(y)

        return self.out(x+y)

model = MyModel(char_vocab=80, num_pos=11, dim=32, hidden=8, dropout=0.5).to(device)

batch_size = 160000

# inputs
chars = torch.randint(0, 79, (batch_size, 8), device=device)
pos   = torch.rand(batch_size, 11, device=device)

# labels
labels = torch.ones(batch_size, 1, dtype=torch.float, device=device)

# forward
out   = model(chars, pos)
loss  = F.binary_cross_entropy_with_logits(out, labels)

# backward
loss.backward()

Pytorch version is 1.3.1 the GPU it was running on was P100-PCIE-16GB.

Any ideas how to get rid of this error?

Edit:

I added a piece of code with generated input that can reproduce the issue.
It seems to be related to the high batch size I'm using. The batch size of 160000 I'm using in the code seems to reliably cause this error, however I noticed that it does not solely depend on the batch size. I tried to reduce it down to 80000 and still got this error. But I know that in my previous runs I have had higher batch sizes up to 180000 without problems. Also starting with a batch size of 60000 worked, then using the same runtime I could run it again with an increased batch size of 140000 without error (that failed before). Note that the memory usage is very low / only ~1-2 GB
Setting CUDA_LAUNCH_BLOCKING=1 leads to a different error message:

RuntimeError: fractional_max_pool2d_backward_out_cuda failed with error code 0

But at the same point in code and with same traceback.


title: How can I specify the flatten layer input size after many conv layers in PyTorch?

body:
Here is my problem, I do a small test on CIFAR10 dataset, how can I specify the flatten layer input size in PyTorch? like the following, the input size is 16*5*5, however I don't know how to calculate this and I want to get the input size through some function.Can someone just write a simple function in this Net class and solve this?

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3,6,5)
        self.conv2 = nn.Conv2d(6,16,5)

        # HERE , the input size is 16*5*5, but I don't know how to get it.
        self.fc1 = nn.Linear(16*5*5, 120)
        self.fc2 = nn.Linear(120,84)
        self.fc3 = nn.Linear(84,10)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        x = F.max_pool2d(F.relu(self.conv2(x)),2)
        x = x.view(x.size()[0],-1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x


title: How exactly should the input file be formatted for the language model finetuning (BERT through Huggingface Transformers)?

body:
I wanted to employ the examples/run_lm_finetuning.py from the Huggingface Transformers repository on a pretrained Bert model. However, from following the documentation it is not evident how a corpus file should be structured (apart from referencing the Wiki-2 dataset). I've tried

One document per line (multiple sentences)
One sentence per line. Documents are separated by a blank line (this I found in some older pytorch-transformers documentation)

By looking at the code of examples/run_lm_finetuning.py it is not directly evident how sequence pairs for the Next Sentence Prediction objective are formed. Would the --line-by-line option help here? I'd be grateful, if someone could give me some hints how a text corpus file should look like.

Many thanks and cheers,

nminds


title: How to share weights between modules in Pytorch?

body:
What is the correct way of sharing weights between two layers(modules) in Pytorch?
Based on my findings in the Pytorch discussion forum, there are several ways for doing this.
As an example, based on this discussion, I thought simply assigning the transposed weights would do it. That is doing :
 self.decoder[0].weight = self.encoder[0].weight.t()

This however, proved to be wrong and causes an error.
I then tried wrapping the above line in a nn.Parameter():
self.decoder[0].weight = nn.Parameter(self.encoder[0].weight.t())

This eliminates the error, but then again, there is no sharing happening here. by this I just initialized a new tensor with the same values as the encoder[0].weight.t().
I then found this link which provides different ways for sharing weights. however, I'm skeptical if all methods given there are actually correct.
For example, one way is demonstrated like this :
# tied autoencoder using off the shelf nn modules
class TiedAutoEncoderOffTheShelf(nn.Module):
    def __init__(self, inp, out, weight):
        super().__init__()
        self.encoder = nn.Linear(inp, out, bias=False)
        self.decoder = nn.Linear(out, inp, bias=False)

        # tie the weights
        self.encoder.weight.data = weight.clone()
        self.decoder.weight.data = self.encoder.weight.data.transpose(0,1)

    def forward(self, input):
        encoded_feats = self.encoder(input)
        reconstructed_output = self.decoder(encoded_feats)
        return encoded_feats, reconstructed_output

Basically it creates a new weight tensor using nn.Parameter() and assigns it to each layer/module like this :
weights = nn.Parameter(torch.randn_like(self.encoder[0].weight))
self.encoder[0].weight.data = weights.clone()
self.decoder[0].weight.data = self.encoder[0].weight.data.transpose(0, 1)

This really confuses me, how is this sharing the same variable between these two layers?
Is it not just cloning the 'raw' data?
When I used this approach, and visualized the weights, I noticed the visualizations were different and that make me even more certain something is not right.
I'm not sure if the different visualizations were solely due to one being the transpose of the other one, or as I just already suspected, they are optimized independently (i.e. the weights are not shared between layers)
example weight initialization :


title: Anaconda reading wrong CUDA version

body:
I have a conda environment with PyTorch and Tensorflow, which both require CUDA 9.0 (~cudatoolkit 9.0 from conda). After installing pytorch with torchvision and the cudatoolkit (like they provided on their website) I wanted to install Tensorflow, the problem here is that I get this error:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: /
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - tensorflow==1.12.0 -> python[version='2.7.*|3.6.*']
  - tensorflow==1.12.0 -> python[version='>=2.7,<2.8.0a0|>=3.6,<3.7.0a0']

Your python: python=3.5

If python is on the left-most side of the chain, that's the version you've asked for.
When python appears to the right, that indicates that the thing on the left is somehow
not available for the python version you are constrained to. Note that conda will not
change your python version to a different minor version unless you explicitly specify
that.

The following specifications were found to be incompatible with your system:

  - feature:/linux-64::__cuda==10.2=0
  - feature:|@/linux-64::__cuda==10.2=0

Your installed version is: 10.2

If I run nvcc or nvidia-smi on my host or the activated conda environment, I get that I have installed CUDA 10.2, even though conda list shows me that cudatoolkit 9.0 is installed. Any solution to this?
EDIT:
When running this code sample:
# setting device on GPU if available, else CPU
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
print('Using device:', device)
print()

#Additional Info when using cuda
if device.type == 'cuda':
    print(torch.cuda.get_device_name(0))
    print('Memory Usage:')
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:   ', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')

print(torch.version.cuda)

I get this output:
GeForce GTX 1050
Memory Usage:
Allocated: 0.0 GB
Cached:    0.0 GB
9.0.176

So PyTorch does get the correct CUDA version, I just cant get tensorflow-gpu installed.


Following my previous question , I have written this code to train an autoencoder and then extract the features.
(There might be some changes in the variable names)
# Autoencoder class
#https://medium.com/pytorch/implementing-an-autoencoder-in-pytorch-19baa22647d1
class AE_class(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.encoder_hidden_layer = nn.Linear(
            in_features=kwargs["input_shape"], out_features=128
        )
        self.encoder_output_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_hidden_layer = nn.Linear(
            in_features=128, out_features=128
        )
        self.decoder_output_layer = nn.Linear(
            in_features=128, out_features=kwargs["input_shape"]
        )

    def forward(self, features):
        #print("in forward")
        #print(type(features))
        activation = self.encoder_hidden_layer(features)
        activation = torch.relu(activation)
        code = self.encoder_output_layer(activation)
        code = torch.relu(code)
        activation = self.decoder_hidden_layer(code)
        activation = torch.relu(activation)
        activation = self.decoder_output_layer(activation)
        reconstructed = torch.relu(activation)
        return reconstructed

    def encode(self, features_h):
        activation_h = self.encoder_hidden_layer(features_h)
        activation_h = torch.relu(activation_h)
        code_h = self.encoder_output_layer(activation_h)
        code_h = torch.relu(code_h)
        return code_h

And then, for training:
def retrieve_AE_features(X_before, n_voxel_region):

        #  use gpu if available
    #https://discuss.pytorch.org/t/runtimeerror-tensor-for-out-is-on-cpu-tensor-for-argument-1-self-is-on-cpu-but-expected-them-to-be-on-gpu-while-checking-arguments-for-addmm/105453
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    # create a model from `AE` autoencoder class
    # load it to the specified device, either gpu or cpu
    model_AE = AE_class(input_shape=n_voxel_region).to(device)

    # create an optimizer object
    # Adam optimizer with learning rate 1e-3
    optimizer = optim.Adam(model_AE.parameters(), lr=1e-3)

    # mean-squared error loss
    criterion = nn.MSELoss()

    X_tensor = torch.tensor(X_before, dtype=torch.float32)

    print(type(X_tensor))

    train_loader = torch.utils.data.DataLoader(
        X_tensor, batch_size=64, shuffle=True, num_workers=2, pin_memory=True
    )

    test_loader = torch.utils.data.DataLoader(
        X_tensor, batch_size=32, shuffle=False, num_workers=2
    )

    print(type(train_loader))

    for epoch in range(epochs_AE):
        loss = 0

        for batch_features in train_loader:
            # reshape mini-batch data to [N, 784] matrix
            # load it to the active device
            #batch_features = batch_features.view(-1, 784).to(device)

            #print(batch_features.shape)

            # reset the gradients back to zero
            # PyTorch accumulates gradients on subsequent backward passes
            optimizer.zero_grad()

            # compute reconstructions
            outputs = model_AE(batch_features)

            # compute training reconstruction loss
            train_loss = criterion(outputs, batch_features)

            # compute accumulated gradients
            train_loss.backward()

            # perform parameter update based on current gradients
            optimizer.step()

            # add the mini-batch training loss to epoch loss
            loss += train_loss.item()

        # compute the epoch training loss
        loss = loss / len(train_loader)

        # display the epoch training loss

        print("AE, epoch : {}/{}, loss = {:.6f}".format(epoch + 1, epochs_AE, loss))

    #After training
    hidden_features = model_AE.encode(X_before)
    return hidden_features

However, I received the following error:

Tensor for argument #2 'mat1' is on CPU, but expected it to be on GPU
(while checking arguments for addmm)

It seems some of my variables should be defined in another way to be able to be executed on GPU.
My questions:

How can I understand which variables will be executed on GPU and which ones on CPU?
How to fix it? In other words, how to define a variable executable on GPU?

Thanks in advance


title: How do I rotate a PyTorch image tensor around it's center in a way that supports autograd?

body:
I'd like to randomly rotate an image tensor (B, C, H, W) around it's center (2d rotation I think?). I would like to avoid using NumPy and Kornia, so that I basically only need to import from the torch module. I'm also not using torchvision.transforms, because I need it to be autograd compatible. Essentially I'm trying to create an autograd compatible version of torchvision.transforms.RandomRotation() for visualization techniques like DeepDream (so I need to avoid artifacts as much as possible).
import torch
import math
import random
import torchvision.transforms as transforms
from PIL import Image

# Load image
def preprocess_simple(image_name, image_size):
    Loader = transforms.Compose([transforms.Resize(image_size), transforms.ToTensor()])
    image = Image.open(image_name).convert('RGB')
    return Loader(image).unsqueeze(0)

# Save image
def deprocess_simple(output_tensor, output_name):
    output_tensor.clamp_(0, 1)
    Image2PIL = transforms.ToPILImage()
    image = Image2PIL(output_tensor.squeeze(0))
    image.save(output_name)

# Somehow rotate tensor around it's center
def rotate_tensor(tensor, radians):
    ...
    return rotated_tensor

# Get a random angle within a specified range
r_degrees = 5
angle_range = list(range(-r_degrees, r_degrees))
n = random.randint(angle_range[0], angle_range[len(angle_range)-1])

# Convert angle from degrees to radians
ang_rad = angle * math.pi / 180

# test_tensor = preprocess_simple('path/to/file', (512,512))
test_tensor = torch.randn(1,3,512,512)

# Rotate input tensor somehow
output_tensor = rotate_tensor(test_tensor, ang_rad)

# Optionally use this to check rotated image
# deprocess_simple(output_tensor, 'rotated_image.jpg')

Some example outputs of what I'm trying to accomplish:


title: Pytorch RuntimeError: CUDA error: out of memory at loss.backward() , No error when using CPU

body:
I'm training a Fully convolutional network (FCN32) for semantic segmentation on Tesla K80 with more than 11G memory.

The input image is pretty large: 352x1216. Network structure is shown below. I used batch_size=1, but still encounter the
out_of_memory error.

Criterion is nn.BCEWithLogitsLoss()

The network works fine when I run on CPU.

    Layer (type)               Output Shape       #  Param
        Conv2d-1        [-1, 64, 352, 1216]           1,792
        Conv2d-2        [-1, 64, 352, 1216]          36,928
     MaxPool2d-3         [-1, 64, 176, 608]               0
        Conv2d-4        [-1, 128, 176, 608]          73,856
        Conv2d-5        [-1, 128, 176, 608]         147,584
     MaxPool2d-6         [-1, 128, 88, 304]               0
        Conv2d-7         [-1, 256, 88, 304]         295,168
        Conv2d-8         [-1, 256, 88, 304]         590,080
        Conv2d-9         [-1, 256, 88, 304]         590,080
    MaxPool2d-10         [-1, 256, 44, 152]               0
       Conv2d-11         [-1, 512, 44, 152]       1,180,160
       Conv2d-12         [-1, 512, 44, 152]       2,359,808
       Conv2d-13         [-1, 512, 44, 152]       2,359,808
    MaxPool2d-14          [-1, 512, 22, 76]               0
       Conv2d-15          [-1, 512, 22, 76]       2,359,808
       Conv2d-16          [-1, 512, 22, 76]       2,359,808
       Conv2d-17          [-1, 512, 22, 76]       2,359,808
    MaxPool2d-18          [-1, 512, 11, 38]               0
       Conv2d-19         [-1, 4096, 11, 38]     102,764,544
       Conv2d-20         [-1, 4096, 11, 38]      16,781,312
       Conv2d-21          [-1, 1, 11, 38]           4,097  ConvTranspose2d-22         [-1, 1, 352, 1216]             4,096

Error message:

  --------------------------------------------------------------------------- RuntimeError                              Traceback (most recent call
  last)  in ()
       36         print (loss)
       37         #torch.cuda.empty_cache()
  ---> 38         loss.backward()
       39         optimizer.step()
       40

  /anaconda/envs/py35/lib/python3.5/site-packages/torch/tensor.py in
  backward(self, gradient, retain_graph, create_graph)
       91                 products. Defaults to False.
       92         """
  ---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
       94
       95     def register_hook(self, hook):

  /anaconda/envs/py35/lib/python3.5/site-packages/torch/autograd/init.py
  in backward(tensors, grad_tensors, retain_graph, create_graph,
  grad_variables)
       88     Variable._execution_engine.run_backward(
       89         tensors, grad_tensors, retain_graph, create_graph,
  ---> 90         allow_unreachable=True)  # allow_unreachable flag
       91
       92

  RuntimeError: CUDA error: out of memory


title: Merge two tensor in pytorch

body:
Tensor a:

tensor([[1, 2, 3],
        [1, 2, 3],
        [1, 2, 3],
        [1, 2, 3]])

Tensor b:

tensor([4,4,4,4])

Question 1:

How to merge two tensors and get result c:

tensor([[1, 2, 3, 4],
        [1, 2, 3, 4],
        [1, 2, 3, 4],
        [1, 2, 3, 4]])

Question 2: How to divide tensor c and get original a and b.


title: Why VGG-16 takes input size 512 * 7 * 7?

body:
According to https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py

I don`t understand why VGG models take 512 * 7 * 7 input_size of fully-connected layer.
Last convolution layer is

nn.Conv2d(512, 512, kernel_size=3, padding=1),
nn.ReLU(True),
nn.MaxPool2d(kernel_size=2, stride=2, dilation=1)

Codes in above link.

class VGG(nn.Module):

    def __init__(self, features, num_classes=1000, init_weights=True):
        super(VGG, self).__init__()
        self.features = features
        self.classifier = nn.Sequential(
            nn.Linear(512 * 7 * 7, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, 4096),
            nn.ReLU(True),
            nn.Dropout(),
            nn.Linear(4096, num_classes),
        )


As I outlined here I am stuck using old versions of pytorch and torchvision due to hardware e.g. using ppc64le IBM architectures.
For this reason, I am having issues when sending and receiving checkpoints between different computers, clusters and my personal mac. I wonder if there is any way to load models in a way to avoid this issue? e.g. perhaps saving models in with a old and new format when using 1.6.x. Of course for the 1.3.1 to 1.6.x is impossible but at leat I was hoping something would work.
Any advice? Of course my ideal solution is that I don't have to worry about it and I can always load and save my checkpoints and everything I usually pickle uniformly across all my hardware.

The first error I got was a zip jit error:
RuntimeError: /home/miranda9/data/f.pt is a zip archive (did you mean to use torch.jit.load()?)

so I used that (and other pickle libraries):
# %%
import torch
from pathlib import Path

def load(path):
    import torch
    import pickle
    import dill

    path = str(path)
    try:
        db = torch.load(path)
        f = db['f']
    except Exception as e:
        db = torch.jit.load(path)
        f = db['f']
        #with open():
        # db = pickle.load(open(path, "r+"))
        # db = dill.load(open(path, "r+"))
        #raise ValueError(f'FAILED: {e}')
    return db, f

p = "~/data/f.pt"
path = Path(p).expanduser()

db, f = load(path)

Din, nb_examples = 1, 5
x = torch.distributions.Normal(loc=0.0, scale=1.0).sample(sample_shape=(nb_examples, Din))

y = f(x)

print(y)
print('Success!\a')

but I get complains of different pytorch versions which I am forced to use:
Traceback (most recent call last):
  File "hal_pg.py", line 27, in <module>
    db, f = load(path)
  File "hal_pg.py", line 16, in load
    db = torch.jit.load(path)
  File "/home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/jit/__init__.py", line 239, in load
    cpp_module = torch._C.import_ir_module(cu, f, map_location, _extra_files)
RuntimeError: version_number <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 1. Your PyTorch installation may be too old. (init at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xbc (0x7fff7b527b9c in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1d98 (0x7fff1d293c78 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x88 (0x7fff1d2950d8 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::jit::import_ir_module(std::shared_ptr<torch::jit::script::CompilationUnit>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) + 0x64 (0x7fff1e624664 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: <unknown function> + 0x70e210 (0x7fff7c0ae210 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x28efc4 (0x7fff7bc2efc4 in /home/miranda9/.conda/envs/wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #26: <unknown function> + 0x25280 (0x7fff84b35280 in /lib64/libc.so.6)
frame #27: __libc_start_main + 0xc4 (0x7fff84b35474 in /lib64/libc.so.6)

any ideas how to make everything consistent across the clusters? I can't even open the pickle files.

maybe this is just impossible with the current pytorch version I am forced to use :(
RuntimeError: version_number <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131, please report a bug to PyTorch. Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 1. Your PyTorch installation may be too old. (init at /opt/anaconda/conda-bld/pytorch-base_1581395437985/work/caffe2/serialize/inline_container.cc:131)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xbc (0x7fff83ba7b9c in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: caffe2::serialize::PyTorchStreamReader::init() + 0x1d98 (0x7fff25993c78 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #2: caffe2::serialize::PyTorchStreamReader::PyTorchStreamReader(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x88 (0x7fff259950d8 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #3: torch::jit::import_ir_module(std::shared_ptr<torch::jit::script::CompilationUnit>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, c10::optional<c10::Device>, std::unordered_map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&) + 0x64 (0x7fff26d24664 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch.so)
frame #4: <unknown function> + 0x70e210 (0x7fff8472e210 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x28efc4 (0x7fff842aefc4 in /home/miranda9/.conda/envs/automl-meta-learning_wmlce-v1.7.0-py3.7/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #23: <unknown function> + 0x25280 (0x7fff8d335280 in /lib64/libc.so.6)
frame #24: __libc_start_main + 0xc4 (0x7fff8d335474 in /lib64/libc.so.6)

using code:
from pathlib import Path

import torch

path = '/home/miranda9/data/dataset/'
path = Path(path).expanduser() / 'fi_db.pt'
path = str(path)

# db = torch.load(path)
# torch.jit.load(path)
db = torch.jit.load(str(path))

print(db)

related links:

How to load checkpoints across different versions of pytorch (1.3.1 and 1.6.x) using ppc64le and x86?
https://discuss.pytorch.org/t/how-to-load-checkpoints-across-different-versions-of-pytorch-1-3-1-and-1-6-x-using-ppc64le-and-x86/97829
related gitissue: https://github.com/pytorch/pytorch/issues/43766
reddit: https://www.reddit.com/r/pytorch/comments/jvza7v/how_to_load_checkpoints_across_different_versions/


title: How to install torchtext 0.4.0 on conda

body: The torchtext 0.4.0 library exists (can be downloaded thru pip), but conda install torchtext=0.4.0 will not work. How can I download torchtext to a anaconda environment?


title: RuntimeError: Expected hidden[0] size (2, 20, 256), got (2, 50, 256)

body:
I get this error while trying to build a multiclass text classification network using LSTM (RNN). The code seems to run fine for the training part of the code whereas it throws the error for the validation part. Below is the network architecture and training code. Appreciate any help here.

I tried taking an existing code that predicts sentiment using RNN and replaced sigmoid with softmax function in the end and loss function from BCE Loss to NLLLoss()

  def forward(self, x, hidden):
    """
    Perform a forward pass of our model on some input and hidden state.
    """
    batch_size = x.size(0)
    embeds = self.embedding(x)
    lstm_out,hidden= self.lstm(embeds,hidden)

     # stack up lstm outputs
    lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)

    # dropout and fully-connected layer
    out = self.dropout(lstm_out)
    out = self.fc(out)

    # softmax function
    soft_out = self.sof(out)

    # reshape to be batch_size first
    soft_out = soft_out.view(batch_size, -1)
#         soft_out = soft_out[:, -1] # get last batch of labels

    # return last sigmoid output and hidden state
    return soft_out, hidden

def init_hidden(self, batch_size):
    ''' Initializes hidden state '''
    # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
    # initialized to zero, for hidden state and cell state of LSTM
    weight = next(self.parameters()).data

    if (train_on_gpu):
        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),
              weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())
    else:
        hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                  weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())

    return hidden

# Instantiate the model w/ hyperparams
vocab_size = len(vocab_to_int)+1
output_size = 44
embedding_dim = 100
hidden_dim = 256
n_layers = 2

net = ClassificationRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)

print(net)

# loss and optimization functions
lr=0.001

criterion = nn.NLLLoss()

optimizer = torch.optim.Adam(net.parameters(), lr=lr)

# training params

epochs = 4 # 3-4 is approx where I noticed the validation loss stop decreasing

counter = 0
print_every = 100
clip=5 # gradient clipping

# move model to GPU, if available
if(train_on_gpu):
    net.cuda()

net.train()
# train for some number of epochs
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    # batch loop
    for inputs, labels in train_loader:
        counter += 1

        if(train_on_gpu):
            inputs, labels = inputs.cuda(), labels.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)

#         print('output:',output.squeeze())
#         print('labels:',labels.float())

        # calculate the loss and perform backprop
        loss = criterion(output, labels)
        loss.backward()
        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        optimizer.step()

        # loss stats
        if counter % print_every == 0:
            # Get validation loss
            val_h = net.init_hidden(batch_size)
            val_losses = []
            net.eval()
            for inputs, labels in valid_loader:

                # Creating new variables for the hidden state, otherwise
                # we'd backprop through the entire training history
                val_h = tuple([each.data for each in val_h])

                if(train_on_gpu):
                    inputs, labels = inputs.cuda(), labels.cuda()

                output, val_h = net(inputs, val_h)

                val_loss = criterion(output, labels)

                val_losses.append(val_loss.item())

            net.train()
            print("Epoch: {}/{}...".format(e+1, epochs),
                  "Step: {}...".format(counter),
                  "Loss: {:.6f}...".format(loss.item()),
                  "Val Loss: {:.6f}".format(np.mean(val_losses)))

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-41-805ed880b453> in <module>()
     58                     inputs, labels = inputs.cuda(), labels.cuda()
     59
---> 60                 output, val_h = net(inputs, val_h)
     61
     62                 val_loss = criterion(output, labels)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

<ipython-input-38-dbfb8d384231> in forward(self, x, hidden)
     34         batch_size = x.size(0)
     35         embeds = self.embedding(x)
---> 36         lstm_out,hidden= self.lstm(embeds,hidden)
     37
     38          # stack up lstm outputs


title: How to display graphs of loss and accuracy on pytorch using matplotlib

body:
I am new to pytorch, and i would like to know how to display graphs of loss and accuraccy And how exactly should i store these values,knowing that i'm applying a cnn model for image classification using CIFAR10.
here is my current implementation :
    def train(num_epochs,optimizer,criterion,model):
        for epoch in range(num_epochs):
            for i, (images, labels) in enumerate(trainloader):
                # origin shape: [4, 3, 32, 32] = 4, 3, 1024
                # input_layer: 3 input channels, 6 output channels, 5 kernel size
                images = images.to(device)
                labels = labels.to(device)

                # Forward pass
                outputs = model(images)
                loss = criterion(outputs, labels)

                # Backward and optimize
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

                if (i+1) % 2000 == 0:
                    print (f'Epoch [{epoch+1}/{num_epochs}], Step [{i+1}/{n_total_steps}], Loss: {loss.item():.4f}')

        PATH = './cnn.pth'
        torch.save(model.state_dict(), PATH)

    def test ():
        with torch.no_grad():
            n_correct = 0
            n_samples = 0
            n_class_correct = [0 for i in range(10)]
            n_class_samples = [0 for i in range(10)]

            for images, labels in testloader:
                images = images.to(device)
                labels = labels.to(device)
                outputs = model(images)
                # max returns (value ,index)
                _, predicted = torch.max(outputs, 1)
                n_samples += labels.size(0)
                n_correct += (predicted == labels).sum().item()

                for i in range(batch_size):
                    label = labels[i]
                    pred = predicted[i]

                    if (label == pred):
                        n_class_correct[label] += 1
                    n_class_samples[label] += 1

            acc = 100.0 * n_correct / n_samples
            print(f'Accuracy of the network: {acc} %')

            for i in range(10):
                acc = 100.0 * n_class_correct[i] / n_class_samples[i]
                print(f'Accuracy of {classes[i]}: {acc} %')

            test_score = np.mean([100 * n_class_correct[i] / n_class_samples[i] for i in range(10)])
            print("the score test is : {0:.3f}%".format(test_score))
            return acc


I have installed cpuonly pytorch and torchvision in anaconda. But when i try to import torchvision i get the following error.
ImportError: cannot import name 'Optional' from 'torch.jit.annotations'(C:\Users\MSI\Anaconda3\lib\site-packages\torch\jit\annotations.py)
How can i fix this?


title: Is it possible to implement a multilayered LSTM with LSTMCells modules in PyTorch?

body:
In PyTorch there is a LSTM module which in addition to input sequence, hidden states, and  cell states accepts a num_layers argument which specifies how many layers will our LSTM have.

There is however another module LSTMCell which has just input size and number of hidden states as parameters, there is no num_layers since this is a single cell in a multi-layered LSTM.

My question is what is the proper way to connect together the LSTMCell modules to achieve a same effect as a multi layered LSTM with num_layers > 1


title: PyTorch data loading from multiple different-sized datasets

body:
I have multiple datasets, each with a different number of images (and different image dimensions) in it. In the training loop I want to load a batch of images randomly from among all the datasets but so that each batch only contains images from a single dataset. For example, I have datasets A, B, C, D and each has images 01.jpg, 02.jpg, … n.jpg (where n depends on the dataset), and let’s say the batch size is 3. In the first loaded batch, for example, I may get images [B/02.jpg, B/06.jpg, B/12.jpg], in the next batch [D/01.jpg, D/05.jpg, D/12.jpg], etc.

So far I have considered the following:

Use a different DataLoader for each dataset, e.g. dataloaderA, dataloaderB, etc., and then in each training loop randomly select one of the dataloaders and get a batch from it. However, this will require a for loop and for large number of datasets it would be very slow since it can’t be split among workers to do it in parallel.
Use a single DataLoader with all of the images from all datasets together but with a custom collate_fn which will create a batch using only images from the same dataset. (I’m not sure how exactly to go about this.)
I have looked at the ConcatDataset class but from its source code it looks like if I use it and try getting a new batch the images in it will be mixed up from among different datasets which I don’t want.

What would be the best way to do this? Thanks!


title: How can I load a model in PyTorch without redefining the model?

body:
I am looking for a way to save a pytorch model, and load it without the model definition. By this I mean that I want to save my model including model definition.

For example, I would like to have two scripts. The first would define, train, and save the model. The second would load and predict the model without including the model definition.

The method using torch.save(), torch.load() requires me to include the model definition in the prediction script, but I want to find a way to load a model without redefining it in the script.


title: Pytorch, `backward` RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed

body:
I'm implementing DDPG with PyTorch (0.4) and got stuck backproping the loss.
So, first my Code performing the update:
def update_nets(self, transitions):
    """
    Performs one update step
    :param transitions: list of sampled transitions
    """
    # get batches
    batch = transition(*zip(*transitions))
    states = torch.stack(batch.state)
    actions = torch.stack(batch.action)
    next_states = torch.stack(batch.next_state)
    rewards = torch.stack(batch.reward)

    # zero gradients
    self._critic.zero_grad()

    # compute critic's loss
    y = rewards.view(-1, 1) + self._gamma * \
        self.critic_target(next_states, self.actor_target(next_states))

    loss_critic = F.mse_loss(y, self._critic(states, actions),
                             size_average=True)

    # backpropagte it
    loss_critic.backward()
    self._optim_critic.step()

    # zero gradients
    self._actor.zero_grad()

    # compute actor's loss
    loss_actor = ((-1.) * self._critic(states, self._actor(states))).mean()

    # backpropagate it
    loss_actor.backward()
    self._optim_actor.step()

    # do soft updates
    self.perform_soft_update(self.actor_target, self._actor)
    self.perform_soft_update(self.critic_target, self._critic)

Where self._actor, self._crtic, self.actor_target and self.critic_target are Nets.
If I run this, I get the following error in the second iteration:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

at

line 221, in update_nets
    loss_critic.backward()
line 93, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
line 89, in backward
    allow_unreachable=True)  # allow_unreachable flag

and I don't know what is causing it.
What I know till now is, that the loss_critic.backward() call causes the error.
I already debugged loss_critic - it got a valid value.
If I replace the loss computation with a simple
loss_critic = torch.tensor(1., device=self._device, dtype=torch.float, requires_grad=True)

Tensor containing the value 1 it works fine.
Also, I already checked that I'm not saving some results which could cause the error.
Additionally updating the actor with the loss_actor doesn't cause any problems.
Does anyone know what is going wrong here?
Thanks!
Update
I replaced
    # zero gradients
    self._critic.zero_grad()

and
    # zero gradients
    self._actor.zero_grad()

with
    # zero gradients
    self._critic.zero_grad()
    self._actor.zero_grad()
    self.critic_target.zero_grad()
    self.actor_target.zero_grad()

(both calls) but it is still failing with the same error.
Additionally, the code performing the update at the end of one iteration
def perform_soft_update(self, target, trained):
    """
    Preforms the soft update
    :param target: Net to be updated
    :param trained: Trained net - used for update
    """
    for param_target, param_trained in \
            zip(target.parameters(), trained.parameters()):
        param_target.data.copy_(
            param_target.data * (
                    1.0 - self._tau) + param_trained * self._tau
        )


title: Change input shape dimensions for ResNet model (pytorch)

body:
I want to feed my 3,320,320 pictures in an existing ResNet model. The model actually expects input of size 3,32,32. As I am afraid of loosing information I don't simply want to resize my pictures.
What is the best way to preprocess my images, so that they are able to run on the ResNet34?
Should I add additional layers in the forward method of ResNet? If yes, what would be a suitable combination in my case?

import torch
import torch.nn as nn
import torch.nn.functional as F
from pytorch_fitmodule import FitModule
from torch.autograd import Variable
import numpy as np

def conv3x3(in_planes, out_planes, stride=1):
    return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride, padding=1, bias=False)

class BasicBlock(FitModule):
    expansion = 1

    def __init__(self, in_planes, planes, stride=1):
        super(BasicBlock, self).__init__()
        self.conv1 = conv3x3(in_planes, planes, stride)
        self.bn1 = nn.BatchNorm2d(planes)
        self.conv2 = conv3x3(planes, planes)
        self.bn2 = nn.BatchNorm2d(planes)

        self.shortcut = nn.Sequential()
        if stride != 1 or in_planes != self.expansion * planes:
            self.shortcut = nn.Sequential(
                nn.Conv2d(in_planes, self.expansion * planes,
                          kernel_size=1, stride=stride, bias=False),
                nn.BatchNorm2d(self.expansion * planes)
            )

    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        out += self.shortcut(x)
        out = F.relu(out)
        return out

class ResNet(FitModule):
    def __init__(self, block, num_blocks, num_classes=10):
        super(ResNet, self).__init__()
        self.in_planes = 64

        self.conv1 = conv3x3(3, 64)
        self.bn1 = nn.BatchNorm2d(64)
        self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
        self.linear = nn.Linear(512 * block.expansion, num_classes)

    def _make_layer(self, block, planes, num_blocks, stride):
        strides = [stride] + [1] * (num_blocks - 1)
        layers = []
        for stride in strides:
            layers.append(block(self.in_planes, planes, stride))
            self.in_planes = planes * block.expansion
        return nn.Sequential(*layers)

    def forward(self, x):    # add additional layers here?
        x = x.float()
        out = F.relu(self.bn1(self.conv1(x).float()).float())
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = F.avg_pool2d(out, 4)
        out = out.view(out.size(0), -1)
        out = self.linear(out)
        return out

def ResNet34():
    return ResNet(BasicBlock, [3, 4, 6, 3])

Thanks plenty!

Regards,
Fabian


title: Transfer learning in Pytorch using fasterrcnn_resnet50_fpn

body:
I am looking for Object Detection for custom dataset in PyTorch.

Tutorial here provides a snippet to use pre-trained model for custom object classification

model_ft = models.resnet18(pretrained=True)
num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, 2)

model_ft = model_ft.to(device)

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                   num_epochs=25)

I tried to use similar method for Object Detection using faster rcnn model.

# load a model pre-trained pre-trained on COCO
model = torchvision.models.detection.fasterrcnn_resnet50_fpn(pretrained=True)
model.eval()
for param in model.parameters():
    param.requires_grad = False
# replace the classifier with a new one, that has
# num_classes which is user-defined
num_classes = 1  # 1 class (person) + background
print(model)
model = model.to(device)
criterion = nn.CrossEntropyLoss()
# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)
model = train_model(model, criterion, optimizer_ft, exp_lr_scheduler,num_epochs=25)

PyTorch throws these errors . Is this approach correct in the first place ?

Epoch 0/24
----------
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-69-527ca4db8e5d> in <module>()
----> 1 model = train_model(model, criterion, optimizer_ft, exp_lr_scheduler,num_epochs=25)

2 frames
/usr/local/lib/python3.6/dist-packages/torchvision/models/detection/generalized_rcnn.py in forward(self, images, targets)
     43         """
     44         if self.training and targets is None:
---> 45             raise ValueError("In training mode, targets should be passed")
     46         original_image_sizes = [img.shape[-2:] for img in images]
     47         images, targets = self.transform(images, targets)

ValueError: In training mode, targets should be passed

Is there a way modify this example for custom object detection ?
https://www.learnopencv.com/faster-r-cnn-object-detection-with-pytorch/


title: how to apply gradients manually in pytorch

body:
Starting to learn pytorch and was trying to do something very simple, trying to move a randomly initialized vector of size 5 to a target vector of value [1,2,3,4,5].

But my distance is not decreasing!! And my vector x just goes crazy. No idea what I am missing.

import torch
import numpy as np
from torch.autograd import Variable

# regress a vector to the goal vector [1,2,3,4,5]

dtype = torch.cuda.FloatTensor # Uncomment this to run on GPU

x = Variable(torch.rand(5).type(dtype), requires_grad=True)
target = Variable(torch.FloatTensor([1,2,3,4,5]).type(dtype),
requires_grad=False)
distance = torch.mean(torch.pow((x - target), 2))

for i in range(100):
  distance.backward(retain_graph=True)
  x_grad = x.grad
  x.data.sub_(x_grad.data * 0.01)


title: Can you train a BERT model from scratch with task specific architecture?

body:
BERT pre-training of the base-model is done by a language modeling approach, where we mask certain percent of tokens in a sentence, and we make the model learn those missing mask. Then, I think in order to do downstream tasks, we add a newly initialized layer and we fine-tune the model.

However, suppose we have a gigantic dataset for sentence classification. Theoretically, can we initialize the BERT base architecture from scratch, train both the additional downstream task specific layer + the base model weights form scratch with this sentence classification dataset only, and still achieve a good result?

Thanks.


title: Load data into GPU directly using PyTorch

body:
In training loop, I load a batch of data into CPU and then transfer it to GPU:

import torch.utils as utils

train_loader = utils.data.DataLoader(train_dataset, batch_size=128, shuffle=True, num_workers=4, pin_memory=True)

for inputs, labels in train_loader:
    inputs, labels = inputs.to(device), labels.to(device)

This way of loading data is very time-consuming. Any way to directly load data into GPU without transfer step ?


title: Sharing GPU memory between process on a same GPU with Pytorch

body:
I'm trying to implement an efficient way of doing concurrent inference in Pytorch.

Right now, I start 2 processes on my GPU (I have only 1 GPU, both process are on the same device). Each process load my Pytorch model and do the inference step.

My problem is that my model takes quite some space on the memory. I have 12Gb of memory on the GPU, and the model takes ~3Gb of memory alone (without the data). Which means together, my 2 processes takes 6Gb of memory just for the model.

Now I was wondering if it's possible to load the model only once, and use this model for inference on 2 different processes. What I want is only 3Gb of memory is consumed by the model, but still have 2 processes.

I came accross this answer mentioning IPC, but as far as I understood it means the process #2 will copy the model from process #1, so I will still end up with 6Gb allocated for the model.

I also checked on the Pytorch documentation, about DataParallel and DistributedDataParallel, but it seems not possible.

This seems to be what I want, but I couldn't find any code example on how to use with Pytorch in inference mode.

I understand this might be difficult to do such a thing for training, but please note I'm only talking about the inference step (the model is in read-only mode, no need to update gradients). With this assumption, I'm not sure if it's possible or not.


title: Why are Embeddings in PyTorch implemented as Sparse Layers?

body:
Embedding Layers in PyTorch are listed under "Sparse Layers" with the limitation:

  Keep in mind that only a limited number of optimizers support sparse gradients: currently it’s optim.SGD (cuda and cpu), and optim.Adagrad (cpu)

What is the reason for this? For example in Keras I can train an architecture with an Embedding Layer using any optimizer.


title: How to take the average of the weights of two networks?

body: Suppose in PyTorch I have model1 and model2 which have the same architecture. They were further trained on same data or one model is an earlier version of the othter, but it is not technically relevant for the question. Now I want to set the weights of model to be the average of the weights of model1 and model2. How would I do that in PyTorch?


title: What "exactly" happens inside embedding layer in pytorch?

body:
From multiple searches and pytorch documentation itself I could figure out that inside embedding layer there is a lookup table where the embedding vectors are stored. What I am not able to understand:

what exactly happens during training in this layer?
What are the weights and how the gradients of those weights are computed?
My intuition is that at least there should be a function with some parameters that produces the keys for the lookup table. If so, then what is that function?

Any help in this will be appreciated. Thanks.


title: PyTorch - Incorrect labeling using torchvision.datasets.ImageFolder

body:
I have structured my dataset in the following way:

dataset/train/0/456.jpg
dataset/train/1/456456.jpg
dataset/train/2/456.jpg
dataset/train/...

dataset/val/0/878.jpg
dataset/val/1/234.jpg
dataset/val/2/34554.jpg
dataset/val/...

So I used torchvision.datasets.ImageFolder to import my dataset to PyTorch. However, it seems like it is not giving the right label to the right image. I've added my code below:

data_transforms = {
    'train': transforms.Compose(
        [transforms.Resize((176,176)),
         transforms.RandomRotation((0,360)),
         transforms.RandomHorizontalFlip(),
         transforms.RandomVerticalFlip(),
         transforms.CenterCrop(128),
         transforms.Grayscale(),
         transforms.ToTensor(),
         transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))
    ]),
    'val': transforms.Compose(
        [transforms.Resize((128,128)),
         transforms.Grayscale(),
         transforms.ToTensor(),
         transforms.Normalize((0.5,0.5,0.5), (0.5,0.5,0.5))
    ]),
}

data_dir = 'dataset'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

I found out that the labels are wrong using the following function:

def imshow(img):
    img = img / 2 + 0.5
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

dataiter = iter(dataloaders['val'])
images, labels = dataiter.next()

imshow(torchvision.utils.make_grid(images))
print(labels)

Using the shown images and the labels, I manually checked whether they are correct. Unfortunately, the labels do not correspond to the images. Can someone tell me what I'm doing wrong?


title: PyTorch DataLoader returns the batch as a list with the batch as the only entry. How is the best way to get a tensor from my DataLoader

body:
I currently have the following situation where I want to use DataLoader to batch a numpy array:

import numpy as np
import torch
import torch.utils.data as data_utils

# Create toy data
x = np.linspace(start=1, stop=10, num=10)
x = np.array([np.random.normal(size=len(x)) for i in range(100)])
print(x.shape)
# >> (100,10)

# Create DataLoader
input_as_tensor = torch.from_numpy(x).float()
dataset = data_utils.TensorDataset(input_as_tensor)
dataloader = data_utils.DataLoader(dataset,
                                   batch_size=100,
                                  )
batch = next(iter(dataloader))

print(type(batch))
# >> <class 'list'>

print(len(batch))
# >> 1

print(type(batch[0]))
# >> class 'torch.Tensor'>

I expect the batchto be already a torch.Tensor. As of now I index the batch like so, batch[0] to get a Tensor but I feel this is not really pretty and makes the code harder to read.

I found that the DataLoader takes a batch processing function called collate_fn.  However, setting data_utils.DataLoader(..., collage_fn=lambda batch: batch[0]) only changes the list to a tuple (tensor([ 0.8454, ..., -0.5863]),) where the only entry is the batch as a Tensor.

You would help me a lot by helping me finding out how to elegantly transform the batch to a tensor (even if this would include telling me that indexing the single entry in batch is okay).


title: PyTorch model to C++

body:
I have trained the detection algorithm and saved my best model. Now I want to convert my model (pretrained) to C++ and use it in my app. I wanted to know what are the possible ways to convert a pyTorch model to c++?

Thanks!


title: How do I install Pytorch offline?

body:
I need to install Pytorch on a computer with no internet connection.

I've tried finding information about this online but can't find a single piece of documentation.

Do you know how I can do this? Is it even possible?


title: What are C classes for a NLLLoss loss function in Pytorch?

body:
I'm asking about C classes for a NLLLoss loss function.

The documentation states:

  The negative log likelihood loss. It is useful to train a classification problem with C classes.

Basically everything after that point depends upon you knowing what a C class is, and I thought I knew what a C class was but the documentation doesn't make much sense to me. Especially when it describes the expected inputs of (N, C) where C = number of classes. That's where I'm confused, because I thought a C class refers to the output only. My understanding was that the C class was a one hot vector of classifications. I've often found in tutorials that the NLLLoss was often paired with a LogSoftmax to solve classification problems.

I was expecting to use NLLLoss in the following example:

# Some random training data
input = torch.randn(5, requires_grad=True)
print(input)  # tensor([-1.3533, -1.3074, -1.7906,  0.3113,  0.7982], requires_grad=True)
# Build my NN (here it's just a LogSoftmax)
m = nn.LogSoftmax(dim=0)
# Train my NN with the data
output = m(input)
print(output)  # tensor([-2.8079, -2.7619, -3.2451, -1.1432, -0.6564], grad_fn=<LogSoftmaxBackward>)
loss = nn.NLLLoss()
print(loss(output, torch.tensor([1, 0, 0])))

The above raises the following error on the last line:

  ValueError: Expected 2 or more dimensions (got 1)

We can ignore the error, because clearly I don't understand what I'm doing. Here I'll explain my intentions of the above source code.

input = torch.randn(5, requires_grad=True)

Random 1D array to pair with one hot vector of [1, 0, 0] for training. I'm trying to do a binary bits to one hot vector of decimal numbers.

m = nn.LogSoftmax(dim=0)

The documentation for LogSoftmax says that the output will be the same shape as the input, but I've only seen examples of LogSoftmax(dim=1) and therefore I've been stuck trying to make this work because I can't find a relative example.

print(loss(output, torch.tensor([1, 0, 0])))

So now I have the output of the NN, and I want to know the loss from my classification [1, 0, 0]. It doesn't really matter in this example what any of the data is. I just want a loss for a one hot vector that represents classification.

At this point I get stuck trying to resolve errors from the loss function relating to expected output and input structures. I've tried using view(...) on the output and input to fix the shape, but that just gets me other errors.

So this goes back to my original question and I'll show the example from the documentation to explain my confusion:

m = nn.LogSoftmax(dim=1)
loss = nn.NLLLoss()
input = torch.randn(3, 5, requires_grad=True)
train = torch.tensor([1, 0, 4])
print('input', input)  # input tensor([[...],[...],[...]], requires_grad=True)
output = m(input)
print('train', output, train)  # tensor([[...],[...],[...]],grad_fn=<LogSoftmaxBackward>) tensor([1, 0, 4])
x = loss(output, train)

Again, we have dim=1 on LogSoftmax which confuses me now, because look at the input data. It's a 3x5 tensor and I'm lost.

Here's the documentation on the first input for the NLLLoss function:

  Input: (N, C)(N,C) where C = number of classes

The inputs are grouped by the number of classes?

So each row of the tensor input is associated with each element of the training tensor?

If I change the second dimension of the input tensor, then nothing breaks and I don't understand what is going on.

input = torch.randn(3, 100, requires_grad=True)
# 3 x 100 still works?

So I don't understand what a C class is here, and I thought a C class was a classification (like a label) and meaningful only on the outputs of the NN.

I hope you understand my confusion, because shouldn't the shape of the inputs for the NN be independent from the shape of the one hot vector used for classification?

Both the code examples and documentations say that the shape of the inputs is defined by the number of classifications, and I don't really understand why.

I have tried to study the documentations and tutorials to understand what I'm missing, but after several days of not being able to get past this point I've decided to ask this question. It's been humbling because I thought this was going to be one of the easier things to learn.


title: Tried installing FastAi but I got "ERROR: No matching distribution found for torchvision>=0.7"

body:
I tried installing Fast Ai using the command
pip install fastai

But then I got this error

ERROR: Could not find a version that satisfies the requirement torchvision>=0.7 (from fastai) (from versions: 0.1.6, 0.1.7, 0.1.8, 0.1.9, 0.2.0, 0.2.1, 0.2.2, 0.2.2.post2, 0.2.2.post3, 0.3.0, 0.4.1, 0.5.0)
ERROR: No matching distribution found for torchvision>=0.7 (from fastai)

What happened and how do I solve this? I tried installing some specific torch versions but none works.


title: How to handle the BatchNorm layer when training fully convolutional networks by finetuning?

body:
Training fully convolutional nerworks (FCNs) for pixelwise semantic segmentation is very memory intensive. So we often use batchsize=1 for traing FCNs. However, when we finetune the pretrained networks with BatchNorm (BN) layers, batchsize=1 doesn't make sense for the BN layers. So, how to handle the BN layers?

Some options:

delete the BN layers (merge the BN layers with the preceding layers for the pretrained model)
Freeze the parameters and statistics of the BN layers
....

which is better and any demo for implementation in pytorch/tf/caffe?


title: Can't open jupyter notebook in docker

body:
I am trying to open the jupyter notebook in a container, but I just came cross this situation:
[I 10:01:25.051 NotebookApp] The Jupyter Notebook is running at:
[I 10:01:25.051 NotebookApp] http://8c1eb91f0492:8888/?token=7671a7abe557349c8d8ad1cbf207702451925efd2c27c84e
[I 10:01:25.051 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 10:01:25.051 NotebookApp]
Copy/paste this URL into your browser when you connect for the first time,
to login with a token:
    http://8c1eb91f0492:8888/?token=7671a7abe557349c8d8ad1cbf207702451925efd2c27c84e&token=7671a7abe557349c8d8ad1cbf207702451925efd2c27c84e

As you can see the url is the container ID, I tried many solutions, like the ip setting. All don't help at all. Could someone give the hints? Thanks in advance!