title: How to run lua ML model from Google Colab

body:
I am trying to run this neural style transfer model from Github but on Google Colab because my computer doesn't have enough memory/CPU/anything.

I have mounted my google drive to my notebook, cloned the repo onto my drive by following this tutorial downloaded the models to my drive folder, and just to test that it works, I'm using the most basic example of running the Brad Pitt example in the readme using:

th neural_style.lua -style_image examples/inputs/picasso_selfport1907.jpg -content_image examples/inputs/brad_pitt.jpg

But for some reason it won't work.

Using subprocess.run() just returned No such file or directory
Using !th neural_style.lua... returns th: command not found

I've tried like four other things and they all give me a variant of the above two error messages. Any thoughts?

Here is the full notebook code to reproduce my setup on Colab from start to end/error:

# Mount the drive
from google.colab import drive
drive.mount('/content/drive')

# Clone the repo onto the drive
!git clone https://github.com/jcjohnson/neural-style

# Install Pytorch
!pip3 install http://download.pytorch.org/whl/cu80/torch-0.3.0.post4-cp36-cp36m-linux_x86_64.whl
!pip3 install torchvision

# Download the models per the github repo instructions
!bash models/download_models.sh

!th neural_style.lua -style_image examples/inputs/picasso_selfport1907.jpg -content_image examples/inputs/brad_pitt.jpg


title: How to balance the generator and the discriminator performances in a GAN?

body:
It's the first time I'm working with GANs and I am facing an issue regarding the Discriminator repeatedly outperforming the Generator. I am trying to reproduce the PA model from this article and I'm looking at this slightly different implementation to help me out.

I have read quite a lot of papers on how GANs work and also followed some tutorials to understand them better. Moreover, I've read articles on how to overcome the major instabilities, but I can't find a way to overcome this behavior.

In my environment, I'm using PyTorch and BCELoss(). Following the DCGAN PyTorch tutorial, I'm using the following training loop:

criterion = nn.BCELoss()
train_d = False
# Discriminator true
optim_d.zero_grad()
disc_train_real = target.to(device)
batch_size = disc_train_real.size(0)
label = torch.full((batch_size,), 1, device=device).cuda()
output_d = discriminator(disc_train_real).view(-1)
loss_d_real = criterion(output_d, label).cuda()
if lossT:
    loss_d_real *= 2
if loss_d_real.item() > 0.3:
    loss_d_real.backward()
    train_d = True
D_x = output_d.mean().item()
# Discriminator false
output_g = generator(image)
output_d = discriminator(output_g.detach()).view(-1)
label.fill_(0)
loss_d_fake = criterion(output_d, label).cuda()
D_G_z1 = output_d.mean().item()
if lossT:
    loss_d_fake *= 2
loss_d = loss_d_real + loss_d_fake
if loss_d_fake.item() > 0.3:
    loss_d_fake.backward()
    train_d = True
if train_d:
    optim_d.step()

# Generator
label.fill_(1)
output_d = discriminator(output_g).view(-1)
loss_g = criterion(output_d, label).cuda()
D_G_z2 = output_d.mean().item()
if lossT:
    loss_g *= 2

loss_g.backward()
optim_g.step()

and, after a period of settlement, everything seems to work fine:

Epoch 1/5 - Step: 1900/9338  Loss G: 3.057388  Loss D: 0.214545  D(x): 0.940985  D(G(z)): 0.114064 / 0.114064
Time for the last step: 51.55 s    Epoch ETA: 01:04:13
Epoch 1/5 - Step: 2000/9338  Loss G: 2.984724  Loss D: 0.222931  D(x): 0.879338  D(G(z)): 0.159163 / 0.159163
Time for the last step: 52.68 s    Epoch ETA: 01:03:24
Epoch 1/5 - Step: 2100/9338  Loss G: 2.824713  Loss D: 0.241953  D(x): 0.905837  D(G(z)): 0.110231 / 0.110231
Time for the last step: 50.91 s    Epoch ETA: 01:02:29
Epoch 1/5 - Step: 2200/9338  Loss G: 2.807455  Loss D: 0.252808  D(x): 0.908131  D(G(z)): 0.218515 / 0.218515
Time for the last step: 51.72 s    Epoch ETA: 01:01:37
Epoch 1/5 - Step: 2300/9338  Loss G: 2.470529  Loss D: 0.569696  D(x): 0.620966  D(G(z)): 0.512615 / 0.350175
Time for the last step: 51.96 s    Epoch ETA: 01:00:46
Epoch 1/5 - Step: 2400/9338  Loss G: 2.148863  Loss D: 1.071563  D(x): 0.809529  D(G(z)): 0.114487 / 0.114487
Time for the last step: 51.59 s    Epoch ETA: 00:59:53
Epoch 1/5 - Step: 2500/9338  Loss G: 2.016863  Loss D: 0.904711  D(x): 0.621433  D(G(z)): 0.440721 / 0.435932
Time for the last step: 52.03 s    Epoch ETA: 00:59:02
Epoch 1/5 - Step: 2600/9338  Loss G: 2.495639  Loss D: 0.949308  D(x): 0.671085  D(G(z)): 0.557924 / 0.420826
Time for the last step: 52.66 s    Epoch ETA: 00:58:12
Epoch 1/5 - Step: 2700/9338  Loss G: 2.519842  Loss D: 0.798667  D(x): 0.775738  D(G(z)): 0.246357 / 0.265839
Time for the last step: 51.20 s    Epoch ETA: 00:57:19
Epoch 1/5 - Step: 2800/9338  Loss G: 2.545630  Loss D: 0.756449  D(x): 0.895455  D(G(z)): 0.403628 / 0.301851
Time for the last step: 51.88 s    Epoch ETA: 00:56:27
Epoch 1/5 - Step: 2900/9338  Loss G: 2.458109  Loss D: 0.653513  D(x): 0.820105  D(G(z)): 0.379199 / 0.103250
Time for the last step: 53.50 s    Epoch ETA: 00:55:39
Epoch 1/5 - Step: 3000/9338  Loss G: 2.030103  Loss D: 0.948208  D(x): 0.445385  D(G(z)): 0.303225 / 0.263652
Time for the last step: 51.57 s    Epoch ETA: 00:54:47
Epoch 1/5 - Step: 3100/9338  Loss G: 1.721604  Loss D: 0.949721  D(x): 0.365646  D(G(z)): 0.090072 / 0.232912
Time for the last step: 52.19 s    Epoch ETA: 00:53:55
Epoch 1/5 - Step: 3200/9338  Loss G: 1.438854  Loss D: 1.142182  D(x): 0.768163  D(G(z)): 0.321164 / 0.237878
Time for the last step: 50.79 s    Epoch ETA: 00:53:01
Epoch 1/5 - Step: 3300/9338  Loss G: 1.924418  Loss D: 0.923860  D(x): 0.729981  D(G(z)): 0.354812 / 0.318090
Time for the last step: 52.59 s    Epoch ETA: 00:52:11

that is, the gradients on the Generator are higher and start to decrease after a while, and in the meanwhile the gradients on the Discriminator rise up. As for the losses, the Generator goes down while the Discriminator goes up. If compared to the tutorial, I guess this can be acceptable.

Here's my first question: I've noticed that on the tutorial (usually) as D_G_z1 rises, D_G_z2 decreases (and viceversa), while in my example this happens a lot less. Is it just a coincidence or am I doing something wrong?

Given that, I've let the training procedure go on, but now I'm noticing this:

Epoch 3/5 - Step: 1100/9338  Loss G: 4.071329  Loss D: 0.031608  D(x): 0.999969  D(G(z)): 0.024329 / 0.024329
Time for the last step: 51.41 s    Epoch ETA: 01:11:24
Epoch 3/5 - Step: 1200/9338  Loss G: 3.883331  Loss D: 0.036354  D(x): 0.999993  D(G(z)): 0.043874 / 0.043874
Time for the last step: 51.63 s    Epoch ETA: 01:10:29
Epoch 3/5 - Step: 1300/9338  Loss G: 3.468963  Loss D: 0.054542  D(x): 0.999972  D(G(z)): 0.050145 / 0.050145
Time for the last step: 52.47 s    Epoch ETA: 01:09:40
Epoch 3/5 - Step: 1400/9338  Loss G: 3.504971  Loss D: 0.053683  D(x): 0.999972  D(G(z)): 0.052180 / 0.052180
Time for the last step: 50.75 s    Epoch ETA: 01:08:41
Epoch 3/5 - Step: 1500/9338  Loss G: 3.437765  Loss D: 0.056286  D(x): 0.999941  D(G(z)): 0.058839 / 0.058839
Time for the last step: 52.20 s    Epoch ETA: 01:07:50
Epoch 3/5 - Step: 1600/9338  Loss G: 3.369209  Loss D: 0.062133  D(x): 0.955688  D(G(z)): 0.058773 / 0.058773
Time for the last step: 51.05 s    Epoch ETA: 01:06:54
Epoch 3/5 - Step: 1700/9338  Loss G: 3.290109  Loss D: 0.065704  D(x): 0.999975  D(G(z)): 0.056583 / 0.056583
Time for the last step: 51.27 s    Epoch ETA: 01:06:00
Epoch 3/5 - Step: 1800/9338  Loss G: 3.286248  Loss D: 0.067969  D(x): 0.993238  D(G(z)): 0.063815 / 0.063815
Time for the last step: 52.28 s    Epoch ETA: 01:05:09
Epoch 3/5 - Step: 1900/9338  Loss G: 3.263996  Loss D: 0.065335  D(x): 0.980270  D(G(z)): 0.037717 / 0.037717
Time for the last step: 51.59 s    Epoch ETA: 01:04:16
Epoch 3/5 - Step: 2000/9338  Loss G: 3.293503  Loss D: 0.065291  D(x): 0.999873  D(G(z)): 0.070188 / 0.070188
Time for the last step: 51.85 s    Epoch ETA: 01:03:25
Epoch 3/5 - Step: 2100/9338  Loss G: 3.184164  Loss D: 0.070931  D(x): 0.999971  D(G(z)): 0.059657 / 0.059657
Time for the last step: 52.14 s    Epoch ETA: 01:02:34
Epoch 3/5 - Step: 2200/9338  Loss G: 3.116310  Loss D: 0.080597  D(x): 0.999850  D(G(z)): 0.074931 / 0.074931
Time for the last step: 51.85 s    Epoch ETA: 01:01:42
Epoch 3/5 - Step: 2300/9338  Loss G: 3.142180  Loss D: 0.073999  D(x): 0.995546  D(G(z)): 0.054752 / 0.054752
Time for the last step: 51.76 s    Epoch ETA: 01:00:50
Epoch 3/5 - Step: 2400/9338  Loss G: 3.185711  Loss D: 0.072601  D(x): 0.999992  D(G(z)): 0.076053 / 0.076053
Time for the last step: 50.53 s    Epoch ETA: 00:59:54
Epoch 3/5 - Step: 2500/9338  Loss G: 3.027437  Loss D: 0.083906  D(x): 0.997390  D(G(z)): 0.082501 / 0.082501
Time for the last step: 52.06 s    Epoch ETA: 00:59:03
Epoch 3/5 - Step: 2600/9338  Loss G: 3.052374  Loss D: 0.085030  D(x): 0.999924  D(G(z)): 0.073295 / 0.073295
Time for the last step: 52.37 s    Epoch ETA: 00:58:12

not only D(x) has increased again and it's stuck to almost one, but also both D_G_z1 and D_G_z2 always show the same value. Moreover, looking at the losses it seems pretty clear that the Discriminator has outperformed the Generator. This behavior has gone on and on for the rest of the epoch and for all the next one, until the end of the training.

Hence my second question: is this normal? If not, what am I doing wrong within the procedure? How can I achieve a more stable training?

EDIT: I've tried to train the network using the MSELoss() as suggested and this is the output:

Epoch 1/1 - Step: 100/9338  Loss G: 0.800785  Loss D: 0.404525  D(x): 0.844653  D(G(z)): 0.030439 / 0.016316
Time for the last step: 55.22 s    Epoch ETA: 01:25:01
Epoch 1/1 - Step: 200/9338  Loss G: 1.196659  Loss D: 0.014051  D(x): 0.999970  D(G(z)): 0.006543 / 0.006500
Time for the last step: 51.41 s    Epoch ETA: 01:21:11
Epoch 1/1 - Step: 300/9338  Loss G: 1.197319  Loss D: 0.000806  D(x): 0.999431  D(G(z)): 0.004821 / 0.004724
Time for the last step: 51.79 s    Epoch ETA: 01:19:32
Epoch 1/1 - Step: 400/9338  Loss G: 1.198960  Loss D: 0.000720  D(x): 0.999612  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.47 s    Epoch ETA: 01:18:09
Epoch 1/1 - Step: 500/9338  Loss G: 1.212810  Loss D: 0.000021  D(x): 0.999938  D(G(z)): 0.000000 / 0.000000
Time for the last step: 52.18 s    Epoch ETA: 01:17:11
Epoch 1/1 - Step: 600/9338  Loss G: 1.216168  Loss D: 0.000000  D(x): 0.999945  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.24 s    Epoch ETA: 01:16:02
Epoch 1/1 - Step: 700/9338  Loss G: 1.212301  Loss D: 0.000000  D(x): 0.999970  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.61 s    Epoch ETA: 01:15:02
Epoch 1/1 - Step: 800/9338  Loss G: 1.214397  Loss D: 0.000005  D(x): 0.999973  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.58 s    Epoch ETA: 01:14:04
Epoch 1/1 - Step: 900/9338  Loss G: 1.212016  Loss D: 0.000003  D(x): 0.999932  D(G(z)): 0.000000 / 0.000000
Time for the last step: 52.20 s    Epoch ETA: 01:13:13
Epoch 1/1 - Step: 1000/9338  Loss G: 1.215162  Loss D: 0.000000  D(x): 0.999988  D(G(z)): 0.000000 / 0.000000
Time for the last step: 52.28 s    Epoch ETA: 01:12:23
Epoch 1/1 - Step: 1100/9338  Loss G: 1.216291  Loss D: 0.000000  D(x): 0.999983  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.78 s    Epoch ETA: 01:11:28
Epoch 1/1 - Step: 1200/9338  Loss G: 1.215526  Loss D: 0.000000  D(x): 0.999978  D(G(z)): 0.000000 / 0.000000
Time for the last step: 51.88 s    Epoch ETA: 01:10:35

As can be seen, the situation gets even worse. Moreover, reading the EnhanceNet paper all over again, Section 4.2.4 (Adversarial Training) states that the adversarial loss function used is a BCELoss(), as I would expect to solve the vanishing gradients problem that I get with MSELoss().


title: Save and load checkpoint pytorch

body:
i make a model and save the configuration as:

def checkpoint(state, ep, filename='./Risultati/checkpoint.pth'):
    if ep == (n_epoch-1):
        print('Saving state...')
        torch.save(state,filename)
checkpoint({'state_dict':rnn.state_dict()},ep)

and then i want load this configuration :

state_dict= torch.load('./Risultati/checkpoint.pth')
    rnn.state_dict(state_dict)

when i try, this is the error:

Traceback (most recent call last):
File "train.py", line 288, in <module>
rnn.state_dict(state_dict)
File "/home/marco/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 593, in state_dict
destination._metadata[prefix[:-1]] = dict(version=self._version)
AttributeError: 'dict' object has no attribute '_metadata'

where i do wrong?

thx in advance


Does pytorch have function to calculate correlation coefficient matrix like numpy.corrcoef ()


title: pytorch out of GPU memory

body:
I am trying to implement Yolo-v2 in pytorch. However, I seem to be running out of memory just passing data through the network. The model is large and is shown below. However, I feel like I'm doing something stupid here with my network (like not freeing memory somewhere). The network works as expected on cpu.

The test code (where memory runs out) is:

x = torch.rand(32,3,416, 416).cuda()
model = Yolov2().cuda()
y = model(x.float())

Question

Anything that is obviously wrong with my model?
How do I make this more efficient with memory?
Other comments?

The model:

import torch
from torch import nn
import torch.nn.functional as F

class Yolov2(nn.Module):

    def __init__(self):
        super(Yolov2, self).__init__()

        self.conv1 = nn.Conv2d(in_channels=3, out_channels=32, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm1 = nn.BatchNorm2d(32)

        self.conv2 = nn.Conv2d(in_channels=32, out_channels=64, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm2 = nn.BatchNorm2d(64)

        self.conv3 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm3 = nn.BatchNorm2d(128)
        self.conv4 = nn.Conv2d(in_channels=128, out_channels=64, kernel_size=1, stride=1, padding=0, bias=False)
        self.batchnorm4 = nn.BatchNorm2d(64)
        self.conv5 = nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm5 = nn.BatchNorm2d(128)

        self.conv6 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm6 = nn.BatchNorm2d(256)
        self.conv7 = nn.Conv2d(in_channels=256, out_channels=128, kernel_size=1, stride=1, padding=0, bias=False)
        self.batchnorm7 = nn.BatchNorm2d(128)
        self.conv8 = nn.Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm8 = nn.BatchNorm2d(256)

        self.conv9 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm9 = nn.BatchNorm2d(512)
        self.conv10 = nn.Conv2d(in_channels=512, out_channels=256, kernel_size=1, stride=1, padding=0, bias=False)
        self.batchnorm10 = nn.BatchNorm2d(256)
        self.conv11 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm11 = nn.BatchNorm2d(512)
        self.conv12 = nn.Conv2d(in_channels=512, out_channels=256, kernel_size=1, stride=1, padding=0, bias=False)
        self.batchnorm12 = nn.BatchNorm2d(256)
        self.conv13 = nn.Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm13 = nn.BatchNorm2d(512)

        self.conv14 = nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm14 = nn.BatchNorm2d(1024)
        self.conv15 = nn.Conv2d(in_channels=1024, out_channels=512, kernel_size=1, stride=1, padding=0, bias=False)
        self.batchnorm15 = nn.BatchNorm2d(512)
        self.conv16 = nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm16 = nn.BatchNorm2d(1024)
        self.conv17 = nn.Conv2d(in_channels=1024, out_channels=512, kernel_size=1, stride=1, padding=0, bias=False)
        self.batchnorm17 = nn.BatchNorm2d(512)
        self.conv18 = nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm18 = nn.BatchNorm2d(1024)

        self.conv19 = nn.Conv2d(in_channels=1024, out_channels=1024, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm19 = nn.BatchNorm2d(1024)
        self.conv20 = nn.Conv2d(in_channels=1024, out_channels=1024, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm20 = nn.BatchNorm2d(1024)

        self.conv21 = nn.Conv2d(in_channels=3072, out_channels=1024, kernel_size=3, stride=1, padding=1, bias=False)
        self.batchnorm21 = nn.BatchNorm2d(1024)

        self.conv22 = nn.Conv2d(in_channels=1024, out_channels=125, kernel_size=1, stride=1, padding=0)

    def reorg_layer(self, x):
        stride = 2
        batch_size, channels, height, width = x.size()
        new_ht = int(height/stride)
        new_wd = int(width/stride)
        new_channels = channels * stride * stride

#         from IPython.core.debugger import Tracer; Tracer()()
        passthrough = x.permute(0, 2, 3, 1)
        passthrough = passthrough.contiguous().view(-1, new_ht, stride, new_wd, stride, channels)
        passthrough = passthrough.permute(0, 1, 3, 2, 4, 5)
        passthrough = passthrough.contiguous().view(-1, new_ht, new_wd, new_channels)
        passthrough = passthrough.permute(0, 3, 1, 2)
        return passthrough

    def forward(self, x):
        out = F.max_pool2d(F.leaky_relu(self.batchnorm1(self.conv1(x)), negative_slope=0.1), 2, stride=2)
        out = F.max_pool2d(F.leaky_relu(self.batchnorm2(self.conv2(out)), negative_slope=0.1), 2, stride=2)

        out = F.leaky_relu(self.batchnorm3(self.conv3(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm4(self.conv4(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm5(self.conv5(out)), negative_slope=0.1)
        out = F.max_pool2d(out, 2, stride=2)

        out = F.leaky_relu(self.batchnorm6(self.conv6(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm7(self.conv7(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm8(self.conv8(out)), negative_slope=0.1)
        out = F.max_pool2d(out, 2, stride=2)

        out = F.leaky_relu(self.batchnorm9(self.conv9(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm10(self.conv10(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm11(self.conv11(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm12(self.conv12(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm13(self.conv13(out)), negative_slope=0.1)
#         from IPython.core.debugger import Tracer; Tracer()()
        passthrough = self.reorg_layer(out)
        out = F.max_pool2d(out, 2, stride=2)

        out = F.leaky_relu(self.batchnorm14(self.conv14(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm15(self.conv15(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm16(self.conv16(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm17(self.conv17(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm18(self.conv18(out)), negative_slope=0.1)

        out = F.leaky_relu(self.batchnorm19(self.conv19(out)), negative_slope=0.1)
        out = F.leaky_relu(self.batchnorm20(self.conv20(out)), negative_slope=0.1)

        out = torch.cat([passthrough, out], 1)
        out = F.leaky_relu(self.batchnorm21(self.conv21(out)), negative_slope=0.1)
        out = self.conv22(out)

        return out

Additional Info:

Torch version is '0.4.1.post2'
Running on aws p2.xlarge (limit 12gb GPU memory).
The number of parameters for this model is 67137565. Which would occupy <500MB.
this thread from pytorch might be related.


title: optim.lr_scheduler.ReduceLROnPlateau gives error value cannot be converted to type float without overflow: inf

body:
I am using pytorch with this install command: pip3 install http://download.pytorch.org/whl/cu80/torch-0.3.1-cp35-cp35m-linux_x86_64.whl.

I have a model that trains without issues, but when I add a learning rate scheduler, I get an error

My scheduler:

# In init
self.optimizer = optim.Adam(self.model.parameters(), lr=0.01)
self.scheduler = optim.lr_scheduler.ReduceLROnPlateau(
    self.optimizer, 'min', factor=0.1, patience=5, verbose=True
)

# after each epoch
self.scheduler.step(loss)

The error:

...
my_project/.env/lib/python3.5/site-packages/torch/optim/lr_scheduler.py in <lambda>(a, best)
    330         if mode == 'min' and threshold_mode == 'rel':
    331             rel_epsilon = 1. - threshold
--> 332             self.is_better = lambda a, best: a < best * rel_epsilon
    333             self.mode_worse = float('Inf')
    334         elif mode == 'min' and threshold_mode == 'abs':

RuntimeError: value cannot be converted to type float without overflow: inf

Doc: http://pytorch.org/docs/master/optim.html#torch.optim.lr_scheduler.ReduceLROnPlateau
Related thread: https://discuss.pytorch.org/t/value-cannot-be-converted-to-type-double-without-overflow-inf/11752/7


title: Understanding when to call zero_grad() in pytorch, when training with multiple losses

body:
I am going through an open-source implementation of a domain-adversarial model (GAN-like). The implementation uses pytorch and I am not sure they use zero_grad() correctly. They call zero_grad() for the encoder optimizer (aka the generator) before updating the discriminator loss. However zero_grad() is hardly documented, and I couldn't find information about it.

Here is a psuedo code comparing a standard GAN training (option 1), with their implementation (option 2). I think the second option is wrong, because it may accumulate the D_loss gradients with the E_opt. Can someone tell if these two pieces of code are equivalent?

Option 1 (a standard GAN implementation):

X, y = get_D_batch()
D_opt.zero_grad()
pred = model(X)
D_loss = loss(pred, y)
D_opt.step()

X, y = get_E_batch()
E_opt.zero_grad()
pred = model(X)
E_loss = loss(pred, y)
E_opt.step()

Option 2 (calling zero_grad() for both optimizers at the beginning):

E_opt.zero_grad()
D_opt.zero_grad()

X, y = get_D_batch()
pred = model(X)
D_loss = loss(pred, y)
D_opt.step()

X, y = get_E_batch()
pred = model(X)
E_loss = loss(pred, y)
E_opt.step()


title: Data Augmentation with torchvision.transforms in pytorch

body:
I found out data augmentation can be done in PyTorch by using torchvision.transforms. I also read that transformations are apllied at each epoch. So I'm wondering whether or not the effect of copying each sample multiple times and then applying random transformation to them is same as using torchvision.transforms on original data set(unique images) and just training it for a longer time(more epochs).
Thanks in advance.


title: Pytorch reconstruction loss

body:
If i have two tensors

truth = [N, 1, 224, 224]

net_output = [N, 1, 224, 224]

and I want to minimize the net_output to match the truth, is this the correct way to apply the loss?

mse = nn.L1Loss().cuda()
FPL = mse(truth, net_output.detach())

N is the batch size. Will my l1 loss be summed over the N images?


title: How to use backpropagation with a self-defined loss in pytorch?

body:
I am trying to implement a Siamese network with a ranking loss between two images. If I define my own loss would I be able to do the backpropagation step as follows? When I run it sometimes it seems to me that it is giving the same results that the single network gives.

with torch.set_grad_enabled(phase == 'train'):
    outputs1 = model(inputs1)
    outputs2 = model(inputs2)
    preds1 = outputs1;
    preds2 = outputs2;

    alpha = 0.02;
    w_r = torch.tensor(1).cuda(async=True);

    y_i, y_j, predy_i, predy_j = labels1,labels2,outputs1,outputs2;
    batchRankLoss =  torch.tensor([max(0,alpha - delta(y_i[i], y_j[i])*predy_i[i] - predy_j[i])) for i in range(batchSize)],dtype = torch.float)
    rankLossPrev = torch.mean(batchRankLoss)
    rankLoss = Variable(rankLossPrev,requires_grad=True)

    loss1 = criterion(outputs1, labels1)
    loss2 = criterion(outputs2, labels2)

    #total loss = loss1 + loss2 + w_r*rankLoss
    totalLoss = torch.add(loss1,loss2)
    w_r = w_r.type(torch.LongTensor)
    rankLossPrev = rankLossPrev.type(torch.LongTensor)
    mult = torch.mul(w_r.type(torch.LongTensor),rankLossPrev).type(torch.FloatTensor)
    totalLoss = torch.add(totalLoss,mult.cuda(async = True));

     # backward + optimize only if in training phase
         if phase == 'train':
            totalLoss.backward()
            optimizer.step()

            running_loss += totalLoss.item() * inputs1.size(0)


title: How to change a Pytorch CNN to take color images instead of black and white?

body:
This code I found has a neural net that is setup to take black and white images. (It's a siamese network but that part's not relevant). When I change it to take my images and NOT convert them to black and white I get an error shown below.
I tried changing the first Conv2d, the sixth line down from a 1 to a 3

class SiameseNetwork(nn.Module):
    def __init__(self):
        super(SiameseNetwork, self).__init__()
        self.cnn1 = nn.Sequential(
            nn.ReflectionPad2d(1),
            # was nn.Conv2d(1, 4, kernel_size=3),
            nn.Conv2d(3, 4, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(4),

            nn.ReflectionPad2d(1),
            nn.Conv2d(4, 8, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(8),

            nn.ReflectionPad2d(1),
            nn.Conv2d(8, 8, kernel_size=3),
            nn.ReLU(inplace=True),
            nn.BatchNorm2d(8))

        self.fc1 = nn.Sequential(
            nn.Linear(8*300*300, 500),
            nn.ReLU(inplace=True),

            nn.Linear(500, 500),
            nn.ReLU(inplace=True),

            nn.Linear(500, 5))

    def forward_once(self, x):
        output = self.cnn1(x)
        output = output.view(output.size()[0], -1)
        output = self.fc1(output)
        return output

    def forward(self, input1, input2):
        output1 = self.forward_once(input1)
        output2 = self.forward_once(input2)
        return output1, output2

My error when the images are NOT converted to black and white and remain in color.

RuntimeError: invalid argument 0: Sizes of tensors must match
except in dimension 0. Got 3 and 1 in dimension 1 at
/opt/conda/conda-bld/pytorch-nightly_1542963753679/work/aten/src/TH/generic/THTensorMoreMath.cpp:1319

I checked the shapes of the images as arrays (right before they go into the model) as black and white vs in color...

B&W

torch.Size([1, 1, 300, 300])

In color

torch.Size([1, 3, 300, 300])

Here is a link to a Jupyter Notebook of the entire original code I am working with... https://github.com/harveyslash/Facial-Similarity-with-Siamese-Networks-in-Pytorch/blob/master/Siamese-networks-medium.ipynb

EDIT: UPDATE: I seemed to have solved it by converting the images to RBG in the SiameseNetworkDataset part of the code

img0 = img0.convert("L")

changed to

img0 = img0.convert("RGB")

I just had the line commented out before and thought this left it in RGB but it was something else the model didn't understand.
Also, the change in the OP was needed.

nn.Conv2d(1, 4, kernel_size=3),

changed to

nn.Conv2d(3, 4, kernel_size=3),

If you'd like to answer with an explaination of what the model is doing that makes it clear I'll give you the green check. Don't really understand nn.Conv2d


title: How to handle class imbalance in multi-label classification using pytorch

body:
We are attempting to implement multi-label classification using CNN in pytorch. We have 8 labels and around 260 images using a 90/10 split for train/validation sets.

The classes are highly imbalanced with the most frequent class occurring in over 140 images. On the other hand, the least frequent class occurs in less than 5 images.

We attempted BCEWithLogitsLoss function initially that led to the model predicting the same label for all images.

We then implemented a focal loss approach to handle class imbalance as follows:

    import torch.nn as nn
    import torch

    class FocalLoss(nn.Module):
        def __init__(self, alpha=1, gamma=2):
            super(FocalLoss, self).__init__()
            self.alpha = alpha
            self.gamma = gamma

        def forward(self, outputs, targets):
            bce_criterion = nn.BCEWithLogitsLoss()
            bce_loss = bce_criterion(outputs, targets)
            pt = torch.exp(-bce_loss)
            focal_loss = self.alpha * (1 - pt) ** self.gamma * bce_loss
            return focal_loss

This resulted in the model predicting empty sets (no labels) for every image since it could not get a greater than 0.5 confidence for any classes.

Is there a approach in pytorch to help address this situation?


title: Examples or explanations of pytorch dataloaders?

body:
I am fairly new to Pytorch (and have never done advanced coding). I am trying to learn the basics of deep learning using the d2l.ai textbook but am having trouble with understanding the logic behind the code for dataloaders. I read the torch.utils.data docs and am not sure what the DataLoader class is meant for, and when for example I am supposed to use the  torch.utils.data.TensorDataset class in combination with it. For example, d2l defines a function:
def load_array(data_arrays, batch_size, is_train=True):
    """Construct a PyTorch data iterator."""
    dataset = data.TensorDataset(*data_arrays)
    return data.DataLoader(dataset, batch_size, shuffle=is_train)

I assume this is supposed to return an iterable that iterates over different batches. However, I don't understand what the data.TensorDataset part does (seems like there are a lot of options listed on the docs page). Also, the documents say that there are two types of datasets: iterable and map style. When describing the former type, it says
"This type of datasets is particularly suitable for cases where random reads are expensive or even improbable, and where the batch size depends on the fetched data."
What does it mean for "a random read to be expensive or improbable" and for the batch_size to depend on the fetched data? Can anyone give an example of this?
If there is any source where a CompSci noob like me can learn these basics, I'd really appreciate tips!
Thanks very much!


title: how to use custom python object in torchscript

body:
I am ready to convert a pytorch module to ScriptModule and then load it in c++，but I am blocked by this error This attribute exists on the Python module, but we failed to convert Python type: 'Vocab' to a TorchScript type, the Vocab is a python object I define.
the demo code is here:
import torch
class Vocab(object):
    def __init__(self, name):
        self.name = name

    def show(self):
        print("dict:" + self.name)

class Model(torch.nn.Module):
    def __init__(self, ):
        super(Model, self).__init__()
        self.layers = torch.nn.Linear(2, 3)
        self.encoder = 4
        self.vocab = Vocab("vocab")

    def forward(self, x):
        name = self.vocab.name
        print("forward show encoder:" + str(self.encoder))
        print("vocab:" + name)
        enc_hidden = []
        step = len(x) // 2
        for i in range(step):
            enc_hidden.append((x[2*i] + x[2*i + 1])/2)
        enc_hidden = torch.stack(enc_hidden, 0)
        enc_hidden = self.__show(enc_hidden)
        return self.layers(enc_hidden)

    @torch.jit.export
    def __show(self, x):
        return x + 1

model = Model()
data = torch.randn(10, 2)
script_model = torch.jit.script(model)
print(script_model)
r1 = model(data)
print(r1)

the error msg:
Traceback (most recent call last):
  File "/mnt/d/python_projects/pytorch_deploy/model4.py", line 47, in <module>
    script_model = torch.jit.script(model)
  File "/mnt/d/anaconda3/lib/python3.6/site-packages/torch/jit/__init__.py", line 1261, in script
    return torch.jit._recursive.create_script_module(obj, torch.jit._recursive.infer_methods_to_compile)
  File "/mnt/d/anaconda3/lib/python3.6/site-packages/torch/jit/_recursive.py", line 305, in create_script_module
    return create_script_module_impl(nn_module, concrete_type, stubs_fn)
  File "/mnt/d/anaconda3/lib/python3.6/site-packages/torch/jit/_recursive.py", line 361, in create_script_module_impl
    create_methods_from_stubs(concrete_type, stubs)
  File "/mnt/d/anaconda3/lib/python3.6/site-packages/torch/jit/_recursive.py", line 279, in create_methods_from_stubs
    concrete_type._create_methods(defs, rcbs, defaults)
RuntimeError:
  Module 'Model' has no attribute 'vocab' (This attribute exists on the Python module, but we failed to convert Python type: 'Vocab' to a TorchScript type.):
  File "/mnt/d/python_projects/pytorch_deploy/model4.py", line 26
  def forward(self, x):
    name = self.vocab.name
           ~~~~~~~~~~ <--- HERE
    print("forward show encoder:" + str(self.encoder))
    print("vocab:" + name)

so how can I use my own python object in torchscript?


title: Can we deploy Pytorch models to Google Cloud ML Engine? If so, how?

body:
I searched if anybody has. Somebody has provided a work around but couldn't confirm its working. Here are the links -

How to setup pytorch in google-cloud-ml

https://discuss.pytorch.org/t/pytorch-in-google-cloud-ml-engine/12818

Currently, I am using Floydhub, but it is good to start with. I work mostly with Google Cloud products, so want to keep my deep learning projects as close to it as possible.


title: PyTorch - shape of nn.Linear weights

body:
Yesterday I came across this question and for the first time noticed that the weights of the linear layer nn.Linear need to be transposed before applying matmul.

Code for applying the weights:

output = input.matmul(weight.t())

What is the reason for this?

Why are the weights not in the transposed shape just from the beginning, so they don't need to be transposed every time before applying the layer?


title: model.cuda() in pytorch

body: If I call model.cuda() in pytorch where model is a subclass of nn.Module, and say if I have four GPUs, how it will utilize the four GPUs and how do I know which GPUs that are using?


title: pytorch - gradients not calculated for parameters

body:
a = torch.nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float, device=device))
b = torch.nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float, device=device))
c = a + 1
d = torch.nn.Parameter(c, requires_grad=True,)
for epoch in range(n_epochs):
    yhat = d + b * x_train_tensor
    error = y_train_tensor - yhat
    loss = (error ** 2).mean()
    loss.backward()
    print(a.grad)
    print(b.grad)
    print(c.grad)
    print(d.grad)

Prints out

None
tensor([-0.8707])
None
tensor([-1.1125])

How do I learn the gradient for a and c? variable d needs to stay a parameter


title: How to know input/output layer names and sizes for Pytorch model?

body:
I have Pytorch model.pth using Detectron2's COCO Object Detection Baselines pretrained model R50-FPN.
I am trying to convert the .pth model to onnx.
My code is as follows.
import io
import numpy as np

from torch import nn
import torch.utils.model_zoo as model_zoo
import torch.onnx
from torchvision import models

model = torch.load('output_object_detection/model_final.pth')
x = torch.randn(1, 3, 1080, 1920, requires_grad=True)#0, in_cha, in_h, in_w
torch_out = torch_model(x)
print(model)
torch.onnx.export(torch_model,               # model being run
                  x,                         # model input (or a tuple for multiple inputs)
                  "super_resolution.onnx",   # where to save the model (can be a file or file-like object)
                  export_params=True,        # store the trained parameter weights inside the model file
                  opset_version=10,          # the ONNX version to export the model to
                  do_constant_folding=True,  # whether to execute constant folding for optimization
                  input_names = ['input'],   # the model's input names
                  output_names = ['cls_score','bbox_pred'], # the model's output names
                  dynamic_axes={'input' : {0 : 'batch_size'},    # variable lenght axes
                                'output' : {0 : 'batch_size'}})

Is it correct way to convert ONNX model?
If it is the right way, how to know input_names and output_names?
Used netron to see input and output, but the graph doesn't show input/output layers.


title: Disable GPU on certain applications

body:
I was trying to train some neural network on an Nvidia GPU, but it seems the desktop environment (KDE) is occupying the GPU:

$ nvidia-smi
Sat Apr 22 09:04:16 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.39                 Driver Version: 375.39                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960M    Off  | 0000:01:00.0     Off |                  N/A |
| N/A   52C    P0    N/A /  N/A |   1295MiB /  2002MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1139    G   /usr/lib/xorg/Xorg                             681MiB |
|    0      1591    G   kwin_x11                                        50MiB |
|    0      1594    G   /usr/bin/krunner                                13MiB |
|    0      1596    G   /usr/bin/plasmashell                           126MiB |
|    0      2267    G   ...el-token=FF7F1AB0E04D51461A7E5E08B2463625   136MiB |
+-----------------------------------------------------------------------------+

Here is the python code I was running:

import torch
import torchvision
import torchvision.transforms as transforms

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')

import matplotlib.pyplot as plt
import numpy as np

def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))

# get some random training images
dataiter = iter(trainloader)
images, labels = dataiter.next()
imshow(torchvision.utils.make_grid(images))
print(' '.join('%5s' % classes[labels[j]] for j in range(4)))

from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

net = Net()
net.cuda()

import torch.optim as optim
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

for epoch in range(2):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data

        # wrap them in Variable
        inputs, labels = Variable(inputs.cuda()), Variable(labels.cuda())

        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = net(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.data[0]
        if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0

print('Finished Training')

Error:

Traceback (most recent call last):
  File "<input>", line 64, in <module>
  File "/home/kaiyin/virtualenvs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 147, in cuda
    return self._apply(lambda t: t.cuda(device_id))
  File "/home/kaiyin/virtualenvs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 118, in _apply
    module._apply(fn)
  File "/home/kaiyin/virtualenvs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 124, in _apply
    param.data = fn(param.data)
  File "/home/kaiyin/virtualenvs/pytorch/lib/python3.5/site-packages/torch/nn/modules/module.py", line 147, in <lambda>
    return self._apply(lambda t: t.cuda(device_id))
  File "/home/kaiyin/virtualenvs/pytorch/lib/python3.5/site-packages/torch/_utils.py", line 65, in _cuda
    return new_type(self.size()).copy_(self, async)
RuntimeError: cuda runtime error (46) : all CUDA-capable devices are busy or unavailable at /b/wheel/pytorch-src/torch/lib/THC/generic/THCStorage.cu:66
  car   cat  bird   dog

How can disable those kde related processes from using the GPU, and let them use Intel graphics instead?


title: RuntimeError: mat1 dim 1 must match mat2 dim 0 Pytorch

body:
I am trying to evaluate my model on random weight and bias before training by calling the  evaluate(model, valid_dl) but I get the below error.
I have also tried debugging my code by the printing the size of tensor after taking a step by step walk through on my error traceback.
I have also check similar problem on SO but the solutions are not working for me.
I think the error is coming from self.classifier. it appear there is something I am not getting right there. Please I need your help .
Error traceback
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-31-1c4791c5f87e> in <module>
----> 1 history = [evaluate(model, valid_dl)]
      2 history

/opt/conda/lib/python3.7/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     24         def decorate_context(*args, **kwargs):
     25             with self.__class__():
---> 26                 return func(*args, **kwargs)
     27         return cast(F, decorate_context)
     28

<ipython-input-30-b49ebc28d6a3> in evaluate(model, val_loader)
      2 def evaluate(model, val_loader):
      3     model.eval()
----> 4     outputs = [model.validation_step(batch) for batch in val_loader]
      5     return model.validation_epoch_end(outputs)
      6

<ipython-input-30-b49ebc28d6a3> in <listcomp>(.0)
      2 def evaluate(model, val_loader):
      3     model.eval()
----> 4     outputs = [model.validation_step(batch) for batch in val_loader]
      5     return model.validation_epoch_end(outputs)
      6

<ipython-input-23-e9609521578c> in validation_step(self, batch)
     12     def validation_step(self, batch):
     13         images, labels = batch
---> 14         out = self(images)                   # Generate predictions
     15         loss = F.cross_entropy(out, labels)   # Calculate loss
     16         acc = accuracy(out, labels)           # Calculate accuracy

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-28-3b2b2b49d22e> in forward(self, xb)
     34         print(out.shape)
     35
---> 36         out = self.classifier(out)
     37         print(out.shape)
     38         return out

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/container.py in forward(self, input)
    115     def forward(self, input):
    116         for module in self:
--> 117             input = module(input)
    118         return input
    119

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

/opt/conda/lib/python3.7/site-packages/torch/nn/modules/linear.py in forward(self, input)
     91
     92     def forward(self, input: Tensor) -> Tensor:
---> 93         return F.linear(input, self.weight, self.bias)
     94
     95     def extra_repr(self) -> str:

/opt/conda/lib/python3.7/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1688     if input.dim() == 2 and bias is not None:
   1689         # fused op is marginally faster
-> 1690         ret = torch.addmm(bias, input, weight.t())
   1691     else:
   1692         output = input.matmul(weight.t())

RuntimeError: mat1 dim 1 must match mat2 dim 0

Model
def conv_block(in_channels, out_channels, pool=False):
    layers = [nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
              nn.BatchNorm2d(out_channels),
              nn.ReLU(inplace=True)]
    if pool: layers.append(nn.MaxPool2d(2))
    return nn.Sequential(*layers)

class ResNet9(ImageClassificationBase):
    def __init__(self, in_channels, num_classes):
        super().__init__()

        self.conv1 = conv_block(in_channels, 64)
        self.conv2 = conv_block(64, 128, pool=True)
        self.res1 = nn.Sequential(conv_block(128, 128), conv_block(128, 128))

        self.conv3 = conv_block(128, 256, pool=True)
        self.conv4 = conv_block(256, 512, pool=True)
        self.res2 = nn.Sequential(conv_block(512, 512), conv_block(512, 512))

        self.classifier = nn.Sequential(nn.MaxPool2d(4),
                                        nn.Flatten(),
                                        nn.Dropout(0.2),
                                        nn.Linear(512, num_classes))

    def forward(self, xb):
        out = self.conv1(xb)
        print(out.shape)
        out = self.conv2(out)
        print(out.shape)
        out = self.res1(out) + out
        print(out.shape)
        out = self.conv3(out)
        print(out.shape)
        out = self.conv4(out)
        print(out.shape)
        out = self.res2(out) + out
        print(out.shape)
        out = self.classifier(out)
        #Error
        print(out.shape)

        return out
enter code here

Output shapes from the forward pass
torch.Size([32, 64, 400, 400])
torch.Size([32, 128, 200, 200])
torch.Size([32, 128, 200, 200])
torch.Size([32, 256, 100, 100])
torch.Size([32, 512, 50, 50])
torch.Size([32, 512, 50, 50])

but when I check the shape after self.classifier I get am error


title: Why isn't there inplace flag in F.sigmoid in pytorch?

body:
Both relu, leakyrelu have inplace flag, so why not sigmoid?

Signature: F.sigmoid(input)

F.relu(input, inplace=False)


title: How can I calculate the network gradients w.r.t weights for all inputs in PyTorch?

body:
I'm trying to figure out how I can calculate the gradient of the network for each input. And I'm a bit lost. Essentially, what I would want, is to calculate d self.output/d weight1 and d self.output/d weight2 for all values of input x. So, I would have a matrix of size (1000, 5) for example. Where the 1000 is for the size of the input x, and 5 is the number of weights in the layer.

The example I've included below returns weights as size (1,5). What exactly is being calculated here? Is this d self.output/ d weight1 for 1 input of x, or an average of all inputs?

Secondly, would a matmul of features.grad and weight1.grad be the same as what I'm asking? A matrix of all the gradients of weight1 for all values of x.

class Network(torch.nn.Module):

    def __init__(self, iNode, hNode, oNode):
        super(Network, self).__init__()

        print("Building Model...")

        iNode = int(iNode) ; self.iNode = iNode
        hNode = int(hNode) ; self.hNode = hNode
        oNode = int(oNode) ; self.oNode = oNode

        self.fc1 = nn.Linear(iNode, hNode, bias=False)
        self.fc2 = nn.Linear(hNode, oNode, bias=False)

    def forward(self, x):
        self.hidden_probs = self.fc1(x)
        self.hidden = self.actFunc1(self.hidden_probs)
        self.output_probs = self.fc2(self.hidden)
        self.output = self.actFunc2(self.output_probs)
        return self.output

    def actFunc1(self, x):
        return 1.0/(1.0+torch.exp(-x))

    def actFunc2(self, x):
        return x

    def trainData(self, features, labels, epochs, alpha, optimisation, verbose=False):

        for epoch in range(0,epochs):
            net_pred = self.forward(features)
            net_pred.backward(gradient=torch.ones(features.size())) #calc. dout/dw for all w
print(features.grad.size()) #returns (1000,1)

            with torch.no_grad():
                for name, param in self.named_parameters():
                    if(param.requires_grad):
                        param -= alpha*param.grad

                for name, param in self.named_parameters():
                    if(param.requires_grad):
                        param.grad.zero_()

            sys.stdout.write("Epoch: %06i\r" % (epoch))
            sys.stdout.flush()
        sys.stdout.write("\n")


title: How to use map_location='cpu' due to "RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False"

body:
I was trying to download the following model at https://pytorch.org/hub/nvidia_deeplearningexamples_tacotron2/

import torch
tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2')

I received:

>>> import torch
>>> tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2')
Using cache found in .cache\torch\hub\nvidia_DeepLearningExamples_torchhub
...
  File "Anaconda3\envs\env3_pytorch\lib\site-packages\torch\serialization.py", line 79, in validate_cuda_device
    raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location='cpu' to map your storages to the CPU.

So I used the following with map_location='cpu', but still get the same error.

>>> tacotron2 = torch.hub.load('nvidia/DeepLearningExamples:torchhub', 'nvidia_tacotron2', map_location='cpu')

torch.version.cuda shows a version but torch.cuda.is_available() is false.

>>> import torch
>>> torch.version.cuda
'9.0'
>>> torch.cuda.is_available()
False

How to get around this error related to map_location as I used exactly the command error message asked to use?
How can I use cuda in loading the model?


title: How to save and load models of custom dataset in Detectron2?

body:
I have tried to save and load the model using:
All keys are mapped but there is no prediction in  output
#1
from detectron2.modeling import build_model
model = build_model(cfg)
torch.save(model.state_dict(), 'checkpoint.pth')
model.load_state_dict(torch.load(checkpoint_path,map_location='cpu'))

I also tried doing it using the official doc but can't understand the input format part
from detectron2.checkpoint import DetectionCheckpointer
DetectionCheckpointer(model).load(file_path_or_url)  # load a file, usually from cfg.MODEL.WEIGHTS
checkpointer = DetectionCheckpointer(model, save_dir="output")
checkpointer.save("model_999")  # save to output/model_999.pth


title: How to solve size mismatch of Multi Head Attention in pytorch?

body:
Learning how to coding  Multi Head Attention in pytorch now,

I can't solve the problem of size_mismatch in case dimension of input tensor  have 4 dims.

I refer to the  def and class codes in
http://nlp.seas.harvard.edu/2018/04/03/attention.html

Sorry for the inconvenience,Can you give me advice?

#attention def and class

def clones(module, N):
    "Produce N identical layers."
    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])

def attention(query, key, value, mask=None, dropout=None):
    "Compute 'Scaled Dot Product Attention'"
    d_k = query.size(-1)
    scores = torch.matmul(query, key.transpose(-2, -1)) \
             / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    p_attn = F.softmax(scores, dim = -1)
    if dropout is not None:
        p_attn = dropout(p_attn)

    return torch.matmul(p_attn, value), p_attn

# MultiHead Attention class

class MultiHeadedAttention(nn.Module):
    def __init__(self, h, d_model, dropout=0.1):
        "Take in model size and number of heads."
        super(MultiHeadedAttention, self).__init__()
        assert d_model % h == 0
        # We assume d_v always equals d_k
        self.d_k = d_model // h
        self.h = h
        self.linears = clones(nn.Linear(d_model, d_model), 4)
        self.attn = None
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, query, key, value, mask=None):
        "Implements Figure 2"
        if mask is not None:
            # Same mask applied to all h heads.
            mask = mask.unsqueeze(1)
        nbatches = query.size(0)

        # 1) Do all the linear projections in batch from d_model => h x d_k
        query, key, value = \
            [l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)
             for l, x in zip(self.linears, (query, key, value))]

        # 2) Apply attention on all the projected vectors in batch.
        x, self.attn = attention(query, key, value, mask=mask,
                                 dropout=self.dropout)

        # 3) "Concat" using a view and apply a final linear.
        x = x.transpose(1, 2).contiguous() \
             .view(nbatches, -1, self.h * self.d_k)
        return self.linears[-1](x)

# create test_4_dim tensor
X=torch.randn(10,5,64,64)
X=X.view(X.shape[0],X.shape[1],X.shape[2]*X.shape[3])
#X:torch.Size([10, 5, 4096])
query_=X.transpose(2,1)
key_=X
value_=X

print("query:",query_.size())
print("key:",key_.size())
print("value:",value_.size())
#query: torch.Size([10, 4096, 5])
#key: torch.Size([10, 5, 4096])
#value: torch.Size([10, 5, 4096])

multihead_testmodel= MultiHeadedAttention(h=4,d_model=4096,dropout=0.1)
#print(multihead_model)

output=multihead_testmodel(query=query_,key=key_,value=value_)
print("model output:",output.size())

#size mismatch, m1: [40960 x 5], m2: [4096 x 4096] at #../aten/src/TH/generic/THTensorMath.cpp:197

in case tensor seze: torch.randn(5,64,64),this code has no error.

X=torch.randn(5,64,64)
#X=X.view(X.shape[0],X.shape[1],X.shape[2]*X.shape[3])

query_=X.transpose(2,1)
key_=X
value_=X

print("query:",query_.size())
print("key:",key_.size())
print("value:",value_.size())

#query: torch.Size([5, 64, 64])
#key: torch.Size([5, 64, 64])
#value: torch.Size([5, 64, 64])

multihead_model= MultiHeadedAttention(h=4,d_model=64,dropout=0.1)
temp_output=multihead_model(query=query_,key=key_,value=value_)
print(temp_output.size())
#torch.Size([5, 64, 64])


title: Pytorch - compute accuracy UNet multi-class segmentation

body:
I'm trying to run on pytorch a UNet model for a multi-class image segmentation. I found an architecture of the model online that is apparently working ... I have 100 classes, my input is corresponding to a tensor size [8, 3, 32, 32], my label is [8, 32, 32] and as expected my output is [8, 100, 32, 32].

I want to compute the accuracy for every iteration so I followed this code for the computation of the accuracy :

def multi_acc(pred, label):
    probs = torch.log_softmax(pred, dim = 1)
    _, tags = torch.max(probs, dim = 1)
    corrects = (tags == label).float()
    acc = corrects.sum()/len(corrects)
    acc = torch.round(acc)*100
    return acc

But then when i'm running the training i get an accuracy that is always the same as :

 Epoch : [2] [1/38311] Loss : 0.3168763518333435
 Acc: 102400.0
 Epoch : [2] [2/38311] Loss : 0.31527179479599
 Acc: 102400.0
 Epoch : [2] [3/38311] Loss : 0.2920961081981659
 Acc: 102400.0

And it keep going like this...
If anyone has an idea to better understand that would be super great !

Thanks for the answers.


title: Pytorch Android mobile: PytorchStreamReader failed reading zip archive error

body:
I want use my pytorch model in my Android project.
I save my model.pt as below:
model.load_state_dict(torch.load(Mymodel.pt)
model.eval()
example_inputs=torch.rand(1,1,266,300)
Mymodel_traced=torch.jit.trace(model,example_inputs=example_inputs)
Mymodel_traced.save("model_traced.pt")

But when I copy this model in my Android Studio project, I have error:
Process: org.pytorch.helloworld, PID: 15383
    java.lang.RuntimeException: Unable to start activity ComponentInfo{org.pytorch.helloworld/org.pytorch.helloworld.MainActivity}: com.facebook.jni.CppException: [enforce fail at inline_container.cc:143] . PytorchStreamReader failed reading zip archive: failed finding central directory
    (no backtrace available)

In my gradle I have:
dependencies {
    implementation 'androidx.appcompat:appcompat:1.1.0'
   implementation 'org.pytorch:pytorch_android:1.4.0'
    implementation 'org.pytorch:pytorch_android_torchvision:1.4.0'

}

Why I have this problem? I read that it can be caused by as the model was salved, but I followed the guidelines of Pytorch website (https://pytorch.org/mobile/android/)


title: tf.losses.log_loss and tf.nn.softmax in Tensorflow and Pytorch

body:
I am trying to implement a network which has the following loss function definition in Pytorch

logits = F.log_softmax(layer_output)
loss = F.nll_loss(logits, labels)

This link https://discuss.pytorch.org/t/pytorch-equivalence-to-sparse-softmax-cross-entropy-with-logits-in-tensorflow/18727 mentions that log_softmax should be used instead of softmax as it is more stable before calculating nll loss

In tensorflow i have the following code

logits = tf.nn.log_softmax(layer_output)
loss = .tf.losses.log_loss(logits, labels)

This leads to a NAN value in loss from first iteration. If i use tf.nn.softmax I don't have a NAN value. But the link mentions log_loss is more stable. Is there any specific reason for this? I could get rid of the NANs using tf.clip_by_value but that leads to vanishing gradients.


title: How to combine datasets in PyTorch to return image and numpy file simultaneously

body:
I am trying to build a dataloader that will take images and poses. The images are saved in the form of .jpg files, and the poses in the form of .npy files. The images and poses are in different folders but have the same sub-folder structure and name. The sub-folders are in the form of classes, i.e., each class has a corresponding folder. I want to apply image transformations and then return the images (for which I am using torchvision datasets.ImageFolder). For the poses, I am using torchvision datasets.DatasetFolder. How do I combine these two datasets so that I get both pose and image of the same name simultaneously?

class ReIDFolder_images(datasets.ImageFolder):

    def __init__(self, root, transform):
        super().__init__(root, transform)
        targets = np.asarray([s[1] for s in self.samples])
        self.targets = targets
        self.img_num = len(self.samples)
        print(self.img_num)

    def _get_cam_id(self, path):
        camera_id = []
        filename = os.path.basename(path)
        camera_id = filename.split('c')[1][0]
        return int(camera_id)-1

    def _get_pos_sample(self, target, index, path):
        pos_index = np.argwhere(self.targets == target)
        pos_index = pos_index.flatten()
        pos_index = np.setdiff1d(pos_index, index)
        if len(pos_index)==0:  # in the query set, only one sample
            return path
        else:
            rand = random.randint(0,len(pos_index)-1)
        return self.samples[pos_index[rand]][0]

    def _get_neg_sample(self, target):
        neg_index = np.argwhere(self.targets != target)
        neg_index = neg_index.flatten()
        rand = random.randint(0,len(neg_index)-1)
        return self.samples[neg_index[rand]]

    def __getitem__(self, index):
        path, target = self.samples[index]
        sample = self.loader(path)

        pos_path = self._get_pos_sample(target, index, path)
        pos = self.loader(pos_path)

        if self.transform is not None:
            sample = self.transform(sample)
            pos = self.transform(pos)

        if self.target_transform is not None:
            target = self.target_transform(target)

        return sample, target, pos

class ReIDFolder_poses(datasets.DatasetFolder):

    def __init__(self, root):
        super().__init__(root, loader=self.npy_loader, extensions='.npy')

        targets = np.asarray([s[1] for s in self.samples])
        self.targets = targets
        self.img_num = len(self.samples)
        print(self.img_num)

    def npy_loader(self, path):
        sample = torch.Tensor(np.load(path))
        return sample

    def _get_cam_id(self, path):
        camera_id = []
        filename = os.path.basename(path)
        camera_id = filename.split('c')[1][0]
        return int(camera_id)-1

    def _get_pos_sample(self, target, index, path):
        pos_index = np.argwhere(self.targets == target)
        pos_index = pos_index.flatten()
        pos_index = np.setdiff1d(pos_index, index)
        if len(pos_index)==0:  # in the query set, only one sample
            return path
        else:
            rand = random.randint(0,len(pos_index)-1)
        return self.samples[pos_index[rand]][0]

    def _get_neg_sample(self, target):
        neg_index = np.argwhere(self.targets != target)
        neg_index = neg_index.flatten()
        rand = random.randint(0,len(neg_index)-1)
        return self.samples[neg_index[rand]]

    def __getitem__(self, index):
        path, target = self.samples[index]
        sample = self.loader(path)

        pos_path = self._get_pos_sample(target, index, path)
        pos = self.loader(pos_path)

        return sample, target, pos


title: Trying to understand Pytorch's implementation of LSTM

body:
I have a dataset containing 1000 examples where each example has 5 features (a,b,c,d,e). I want to feed 7 examples to an LSTM so it predicts the feature (a) of the 8th day.

Reading Pytorchs documentation of nn.LSTM() I came up with the following:

input_size = 5
hidden_size = 10
num_layers = 1
output_size = 1

lstm = nn.LSTM(input_size, hidden_size, num_layers)
fc = nn.Linear(hidden_size, output_size)

out, hidden = lstm(X)  # Where X's shape is ([7,1,5])
output = fc(out[-1])

output  # output's shape is ([7,1])

According to the docs:

The input of the nn.LSTM is "input of shape (seq_len, batch, input_size)" with "input_size – The number of expected features in the input x",

And the output is: "output of shape (seq_len, batch, num_directions * hidden_size): tensor containing the output features (h_t) from the last layer of the LSTM, for each t."

In this case, I thought seq_len would be the sequence of 7 examples, batchis 1 and input_size is 5. So the lstm would consume each example containing 5 features refeeding the hidden layer every iteration.

What am I missing?


title: How do I use BertForMaskedLM or BertModel to calculate perplexity of a sentence?

body:
I want to use BertForMaskedLM or BertModel to calculate perplexity of a sentence, so I write code like this:
import numpy as np
import torch
import torch.nn as nn
from transformers import BertTokenizer, BertForMaskedLM
# Load pre-trained model (weights)
with torch.no_grad():
    model = BertForMaskedLM.from_pretrained('hfl/chinese-bert-wwm-ext')
    model.eval()
    # Load pre-trained model tokenizer (vocabulary)
    tokenizer = BertTokenizer.from_pretrained('hfl/chinese-bert-wwm-ext')
    sentence = "我不会忘记和你一起奋斗的时光。"
    tokenize_input = tokenizer.tokenize(sentence)
    tensor_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])
    sen_len = len(tokenize_input)
    sentence_loss = 0.

    for i, word in enumerate(tokenize_input):
        # add mask to i-th character of the sentence
        tokenize_input[i] = '[MASK]'
        mask_input = torch.tensor([tokenizer.convert_tokens_to_ids(tokenize_input)])

        output = model(mask_input)

        prediction_scores = output[0]
        softmax = nn.Softmax(dim=0)
        ps = softmax(prediction_scores[0, i]).log()
        word_loss = ps[tensor_input[0, i]]
        sentence_loss += word_loss.item()

        tokenize_input[i] = word
    ppl = np.exp(-sentence_loss/sen_len)
    print(ppl)

I think this code is right, but I also notice BertForMaskedLM's paramaters masked_lm_labels, so could I use this paramaters to calculate PPL of a sentence easiler?
I know the input_ids argument is the masked input, the masked_lm_labels argument is the desired output. But I couldn't understand the actual meaning of its output loss, its code like this:
if masked_lm_labels is not None:
    loss_fct = CrossEntropyLoss()  # -100 index = padding token
    masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size),
    masked_lm_labels.view(-1))
    outputs = (masked_lm_loss,) + outputs


title: Saving Pytorch model.state_dict() to s3

body:
I am trying to save a trained Pytorch model to S3. However, the torch.save(model.state_dict(), file_name) seems to support only local files. How can the state dict be saved to an S3 file?

I'm using Torch 0.4.0


title: Which PyTorch version is CUDA 3.0 compatible?

body:
I have a Nvidia GeForce GTX 770, which is CUDA 3.0 compatible, but upon running PyTorch training on the GPU, I get the warning

Found GPU0 GeForce GTX 770 which is of cuda capability 3.0.
PyTorch no longer supports this GPU because it is too old.
The minimum cuda capability that we support is 3.5.

and subsequently the error RuntimeError: CUDA error: no kernel image is available for execution on the device.

Was there an old PyTorch version, that supported graphics cards like mine with CUDA capability 3.0? If yes, which version, and where to find this information? Is there a table somewhere, where I can find the supported CUDA versions and compatibility versions?

If it is relevant, I have CUDA 10.1 installed.


title: fastapi could not find model defintion when run with uvicorn

body:
I want to host a pytorch model in a fastapi backend. When I run the code with python it is working fine. the depickled model can use the defined class. When the same file is started with uvicorn it cannot find the class definition.
Sourcecode looks like this:
import uvicorn
import json
from typing import List
from fastapi import Body, FastAPI
from fastapi.encoders import jsonable_encoder
import requests
from pydantic import BaseModel

#from model_ii import Model_II_b

import dill as pickle
import torch as T
import sys

app = FastAPI()
current_model = 'model_v2b_c2_small_ep15.pkl'
verbose_model = False  # for model v2

class Model_II_b(T.nn.Module):
[...]
@app.post('/function')
def API_call(req_json: dict = Body(...)):
    try:
        # load model...
        model = pickle.load(open('models/' + current_model, 'rb'))
        result = model.dosomething_with(req_json)

        return result

    except Exception as e:
        raise e
        return {"error" : str(e)}

if __name__ == "__main__":
    uvicorn.run(app, host="0.0.0.0", port=8000)

When I run this with python main.py it is working fine and I am gettings results. When I run it with  uvicorn main:app and send a request I get the following error:
AttributeError: Can't get attribute 'Model_II_b' on <module '__mp_main__' from '/opt/webapp/env/bin/uvicorn'>

both should be using the same python env as I use the uvicorn from within the env.
I hope someone has an idea what is wrong with my setup or code.
Update Stacktrace:
(model_2) root@machinelearning-01:/opt/apps# uvicorn main:app --env-file /opt/apps/env/pyvenv.cfg --reload
INFO:     Loading environment from '/opt/apps/env/pyvenv.cfg'
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
INFO:     Started reloader process [164777] using statreload
INFO:     Started server process [164779]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     127.0.0.1:33872 - "POST /ml/v2/predict HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/opt/apps/env/lib/python3.6/site-packages/uvicorn/protocols/http/httptools_impl.py", line 385, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/opt/apps/env/lib/python3.6/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/opt/apps/env/lib/python3.6/site-packages/fastapi/applications.py", line 183, in __call__
    await super().__call__(scope, receive, send)  # pragma: no cover
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/applications.py", line 102, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/routing.py", line 550, in __call__
    await route.handle(scope, receive, send)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/opt/apps/env/lib/python3.6/site-packages/fastapi/routing.py", line 197, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/opt/apps/env/lib/python3.6/site-packages/fastapi/routing.py", line 149, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/opt/apps/env/lib/python3.6/site-packages/starlette/concurrency.py", line 34, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "./main.py", line 155, in API_call
    raise e
  File "./main.py", line 129, in API_call
    model = pickle.load(open('models/' + current_model, 'rb'))
  File "/opt/apps/env/lib/python3.6/site-packages/dill/_dill.py", line 270, in load
    return Unpickler(file, ignore=ignore, **kwds).load()
  File "/opt/apps/env/lib/python3.6/site-packages/dill/_dill.py", line 473, in load
    obj = StockUnpickler.load(self)
  File "/opt/apps/env/lib/python3.6/site-packages/dill/_dill.py", line 463, in find_class
    return StockUnpickler.find_class(self, module, name)
AttributeError: Can't get attribute 'Model_II_b' on <module '__mp_main__' from '/opt/apps/env/bin/uvicorn'>
enter code here


title: NumPy is faster than PyTorch for larger cross or outer products

body:
I'm computing huge outer products between vectors of size (50500,) and found out that NumPy is (much?) faster than PyTorch while doing so.

Here are the tests:

# NumPy

In [64]: a = np.arange(50500)
In [65]: b = a.copy()

In [67]: %timeit np.outer(a, b)
5.81 s ± 56.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

-------------

# PyTorch

In [73]: t1 = torch.arange(50500)
In [76]: t2 = t1.clone()

In [79]: %timeit torch.ger(t1, t2)
7.73 s ± 143 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

I'd ideally like to have the computation done in PyTorch. So, how can I speed things up for computing outer product in PyTorch for such huge vectors?

Note: I tried to move the tensors to GPU but I was treated with MemoryError because it needs around 19 GiB of space. So, I eventually have to do it on the CPU.


I don't know how to solve this error but I hope some of you guys know how to solve this issue.
Error:TypeError: __init__() got an unexpected keyword argument 'filepath'
Full error message:
  File "train.py", line 167, in <module>
    main(args)
  File "train.py", line 113, in main
    checkpoint_callback=checkpoint_callback(),
  File "train.py", line 86, in checkpoint_callback
    return ModelCheckpoint(
TypeError: __init__() got an unexpected keyword argument 'filepath'

from pytorch_lightning.callbacks import ModelCheckpoint

save_model_path = path/to/your/dir
def checkpoint_callback():
    return ModelCheckpoint(
        filepath= save_model_path,
        save_top_k=True,
        verbose=True,
        monitor='val_loss',
        mode='min',
        prefix=''
    )


title: PyTorch installation on Ubuntu 16.04 (CPU)

body:
I want to install pytorch-cpu and torch-vision-cpu on a Ubuntu 16.04 machine using conda.

I followed the instructions specified in pytorch.org.

conda install pytorch-cpu torchvision-cpu -c pytorch

I'm getting the following error report.

ninja-1.8.2          |  1.3 MB | ####################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: failed

ERROR conda.core.link:_execute(502): An error occurred while uninstalling package 'defaults::conda-4.5.4-py36_0'.
PermissionError(13, 'Permission denied')
Attempting to roll back.

Rolling back transaction: done

PermissionError(13, 'Permission denied')

Environment info

  environment variables:
                 CIO_TEST=<not set>
          COMPIZ_BIN_PATH=/usr/bin/
               CONDA_ROOT=/home/thechain/anaconda3
            DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
           MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
                     PATH=/home/thechain/anaconda3/bin:/home/thechain/anaconda3/bin:/home/thecha
                          in/gems/bin:/home/thechain/google-cloud-sdk/bin:/home/thechain/gems/bi
                          n:/home/thechain/bin:/home/thechain/.local/bin:/usr/local/sbin:/usr/lo
                          cal/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/sna
                          p/bin
       REQUESTS_CA_BUNDLE=<not set>
            SSL_CERT_FILE=<not set>
            XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
         XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0

     active environment : None
       user config file : /home/thechain/.condarc
 populated config files :
          conda version : 4.5.11
    conda-build version : 3.10.5
         python version : 3.6.5.final.0
       base environment : /home/thechain/anaconda3  (read only)
           channel URLs : https://conda.anaconda.org/pytorch/linux-64
                          https://conda.anaconda.org/pytorch/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/free/linux-64
                          https://repo.anaconda.com/pkgs/free/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
                          https://repo.anaconda.com/pkgs/pro/linux-64
                          https://repo.anaconda.com/pkgs/pro/noarch
          package cache : /home/thechain/anaconda3/pkgs
                          /home/thechain/.conda/pkgs
       envs directories : /home/thechain/.conda/envs
                          /home/thechain/anaconda3/envs
               platform : linux-64
             user-agent : conda/4.5.11 requests/2.18.4 CPython/3.6.5 Linux/4.13.0-36-generic ubuntu/16.04 glibc/2.23
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False


title: torch.cuda.is_available() returns false

body:
I have the following configuration:

OS: Windows 10
GCC version: (GCC) 3.4.5 (mingw-vista special r3)

Python version: 3.7.5
CUDA runtime version: 9.0.176
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip] numpy==1.17.4
[pip] torch==1.1.0
[pip] torch-nightly==1.2.0.dev20190723
[pip] torchfile==0.1.0
[pip] torchvision==0.3.0
[conda] blas                      1.0                         mkl    defaults
[conda] cuda92                    1.0                           0    pytorch
[conda] mkl                       2019.0                   pypi_0    pypi
[conda] mkl-service               2.3.0            py37hb782905_0    defaults
[conda] mkl_fft                   1.0.15           py37h14836fe_0    defaults
[conda] mkl_random                1.1.0            py37h675688f_0    defaults
[conda] pytorch                   1.1.0           py3.7_cuda90_cudnn7_1    pytorch
[conda] torch-nightly             1.2.0.dev20190723          pypi_0    pypi
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.3.0               py37_cu90_1    pytorch

And finnaly torch.cuda.is_available() returns false.
I've tried a lot of solutions but could not solve this problem.
My graphics card is Nvidia GeForce GT 650M


title: I can not install pytorch with Anaconda just because I ran out of disk space the first time

body:
I cannot install pytorch 1.5.0. Firstly, I opened Anaconda Prompt and activated my venv as follows:

conda activate gym

After that, I wanted to install pytorch in gym, so I wrote:

conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

This is from pytorch official website.

But I ran out of space in my hard disk so the installation ended with an error. I've cleaned some space and then ran that command again, but it said:

WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(140): Could not remove or rename D:\Python\Anaconda\pkgs\pytorch-1.5.0-py3.8_cuda102_cudnn7_0\Lib\site-packages\torch\lib\torch_cuda.dll.  Please remove this file manually (you may need to reboot to free file handles)

I have deleted the whole archive and rebooted my PC. The error persisted. I did these steps several times.
Then I thought to install it in another way. I deleted the archive again and I was sure that there was no package left called pytorch (the archive was under pkgs folder under Anaconda installation). So I went to Anaconda official website where I found this command:

conda install -c pytorch pytorch

I tried it and now it says:

InvalidArchiveError('Error with archive D:\\Python\\Anaconda\\pkgs\\pytorch-1.5.0-py3.8_cuda102_cudnn7_0.tar.bz2.  You probably need to delete and re-download or re-create this file.  Message from libarchive was:\n\nFile size could not be restored')

The archive was deleted for sure from my disk.

I even updated all packages in Anaconda with conda update -n gym --all and tried these steps several times.

All these problems just because I ran out of disk space? ...

I do not know what to do. Can you please help me? Thanks a lot for any answer! I really appreciate it!


title: libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: No such file or directory

body: I tried to implement the code from PyTorch Tutorial, and it seems there is an issue with libc++abi.dylib:


title: Do I need to define the backward() in custom loss function?

body:
I have already to define my own loss function. It does work. The feedforward may not have problem. But I am not sure whether it is correct because I don't define the backward().

class _Loss(nn.Module):
    def __init__(self, size_average=True):
        super(_Loss, self).__init__()
        self.size_average = size_average
class MyLoss(_Loss):
    def forward(self, input, target):
        loss = 0
        weight = np.zeros((BATCH_SIZE,BATCH_SIZE))
        for a in range(BATCH_SIZE):
            for b in range(BATCH_SIZE):
                weight[a][b] = get_weight(target.data[a][0])
        for i in range(BATCH_SIZE):
            for j in range(BATCH_SIZE):
                a_ij= (input[i]-input[j]-target[i]+target[j])*weight[i,j]
                loss += F.relu(a_ij)
        return loss

The question I want to ask is that

1) Do I need to define the backward() to loss function?

2) How to define the backward()?

3) Is there are any way to do the index of the data while doing SGD in torch?


title: Pytorch softmax along different masks without for loop

body:
Say I have a vector a , with an index vector b of the same length. The indexs are in range 0~N-1, corresponding to N groups. How can I do softmax for every group without for loop?

I'm doing some sort of attention operation here. The numbers for every group are not the same, so I can't reshape a to a matrix and use the dim in standard Softmax() API.

Toy example:

a = torch.rand(10)
a: tensor([0.3376, 0.0557, 0.3016, 0.5550, 0.5814, 0.1306, 0.2697, 0.9989, 0.4917,
        0.6306])
b = torch.randint(0,3,(1,10), dtype=torch.int64)
b: tensor([[1, 2, 0, 2, 2, 0, 1, 1, 1, 1]])

I want to do softmax like

for index in range(3):
    softmax(a[b == index])

but without the for loop to save time.


title: How do I calculate cross-entropy from probabilities in PyTorch?

body:
By default, PyTorch's cross_entropy takes logits (the raw outputs from the model) as the input. I know that CrossEntropyLoss combines LogSoftmax (log(softmax(x))) and NLLLoss (negative log likelihood loss) in one single class. So, I think I can use NLLLoss to get cross-entropy loss from probabilities as follows:

  true labels: [1, 0, 1]
  probabilites: [0.1, 0.9], [0.9, 0.1], [0.2, 0.8]

where, y_i,j denotes the true value i.e. 1 if sample i belongs to class j and 0 otherwise.
and p_i,j denotes the probability predicted by your model of sample i belonging to class j.

If I calculate by hand, it turns out to be:

>>> -(math.log(0.9) + math.log(0.9) + math.log(0.8))
0.4338

Using PyTorch:

>>> labels = torch.tensor([1, 0, 1], dtype=torch.long)
>>> probs = torch.tensor([[0.1, 0.9], [0.9, 0.1], [0.2, 0.8]], dtype=torch.float)
>>> F.nll_loss(torch.log(probs), labels)
tensor(0.1446)

What am I doing wrong? Why is the answer different?


title: Slicing a 4D tensor with a 3D tensor-index in PyTorch

body:
I have a 4D tensor (which happens to be a stack of three batches of 56x56 images where each batch has 16 images) with the size of [16, 3, 56, 56]. My goal is to select the correct one of those three batches (with my index map that has the size of [16, 56, 56]) for each pixel and get the images that I want.

Now, I want to select the particular batches of images inside those three batches, with a  which has values such as

       [[[ 0,  0,  2,  ...,  0,  0,  0],
         [ 0,  0,  2,  ...,  0,  0,  0],
         [ 0,  0,  0,  ...,  0,  0,  0],
         ...,
         [ 0,  0,  0,  ...,  0,  0,  0],
         [ 0,  2,  0,  ...,  0,  0,  0],
         [ 0,  2,  2,  ...,  0,  0,  0]],

        [[ 0,  2,  0,  ...,  1,  1,  0],
         [ 0,  2,  0,  ...,  0,  0,  0],
         [ 0,  0,  0,  ...,  0,  2,  0],
         ...,
         [ 0,  0,  0,  ...,  0,  2,  0],
         [ 0,  0,  2,  ...,  0,  2,  0],
         [ 0,  0,  2,  ...,  0,  0,  0]]]

So for the 0s, the value will be selected from the first batch, where 1 and 2 will mean I want to select the values from the second and the third batch.

Here are some of the visualizations of the indices, each color denoting another batch.

I have tried to transpose the 4D tensor to match the dimensions of my indices, but it did not work. All it does is to give me a copy of the dimensions I have tried to select. Means

tposed = torch.transpose(fourD, 0,1) print(indices.size(),
outs.size(), tposed[:, indices].size())

outputs

torch.Size([16, 56, 56]) torch.Size([16, 3, 56, 56]) torch.Size([3, 16, 56, 56, 56, 56])

while the shape I need is

torch.Size([16, 56, 56]) or torch.Size([16, 1, 56, 56])

and as an example, if I try to select the right values for only the first image on the batch with

fourD[0,indices].size()

I get a shape like

torch.Size([16, 56, 56, 56, 56])

Not to mention that I get an out of memory error when I try this on the whole tensor.

I appreciate any help for using these indices to select either one of these three batches for each pixel in my images.

Note :

I have tried the option

outs[indices[:,None,:,:]].size()

and that returns

torch.Size([16, 1, 56, 56, 3, 56, 56])

Edit : torch.take does not help much since it treats the input tensor as a single dimensional array.


title: Getting an error(cannot import name 'BertPreTrainedModel') while importing classification model from simpletransformers

body:
Getting the following error while  trying to import the classificationmodel from simpletransformers.

ImportError                               Traceback (most recent call last)
<ipython-input-1-29f08e6c2d87> in <module>()
----> 1 from simpletransformers.classification import ClassificationModel, ClassificationArgs

3 frames
/usr/local/lib/python3.6/dist-packages/simpletransformers/classification/transformer_models/roberta_model.py in <module>()
      2 import torch.nn as nn
      3 from torch.nn import CrossEntropyLoss, MSELoss
----> 4 from transformers.modeling_roberta import (
      5     ROBERTA_PRETRAINED_MODEL_ARCHIVE_LIST,
      6     BertPreTrainedModel,

ImportError: cannot import name 'BertPreTrainedModel'

---------------------------------------------------------------------------


title: Pytorch: AttributeError: cannot assign module before Module.__init__() call even if initialized

body:
I'm getting the following error:

  AttributeError: cannot assign module before Module.init() call

I'm trying to create an instance of my class :

class ResNetGenerator(nn.Module):
    def __init__(self, input_nc=3, output_nc=3, n_residual_blocks=9, use_dropout=False):
        # super(ResNetGenerator, self).__init__()
        super().__init__()

I'm calling super().__init__() but in vain.

What I am doing wrong here?

Complete Traceback:

File "train.py", line 40, in <module>
    model = ColorizationCycleGAN(args)
File "/path/cycle_gan.py", line 27, in __init__
    self.G_A2B = ResNetGenerator(input_nc=self.input_nc, output_nc=self.output_nc, n_residual_blocks=9, use_dropout=False)
File "/path/.local/lib/python3.6/site packages/torch/nn/modules/module.py", line 544, in __setattr__
    "cannot assign module before Module.__init__() call")
AttributeError: cannot assign module before Module.__init__() call


title: Cifar10 dataset: read certain number of images from a class

body:
I am currently learning deep learning with Pytorch and doing some experiment with Cifar 10 dataset. Which is having 10 classes each class is having 5000 test images. I want to use only 60% of dog and deer classes data and 100% data of other classes.

As per my understanding I need to use custom dataset. But I am not actually able to figure it out. Any Idea, sample code or link if you can share will be helpful for me.


title: TypeError: stat: path should be string, bytes, os.PathLike or integer, not cv2.VideoCapture ,detect using neural network using mtcnn?

body:
i've tried to detect faces from my webcam , but it was only detect the first face which appeared in the first frame , then it will crush , i have used mtcnn to detection operation

import cv2
i = 0
capture = cv2.VideoCapture(0)
while(True):
    ret, frame = capture.read()

    frames_tracked = []
    print('\rTracking frame: {}'.format(i + 1), end='')
    frame_pil = cv2.cvtColor(frame,cv2.COLOR_BGR2RGB)
    frame_pil = Image.fromarray(frame_pil)
    boxes,_ = mtcnn.detect(frame_pil)
    frame_draw = frame_pil.copy()
    draw = ImageDraw.Draw(frame_draw)
    for box in boxes:
        draw.rectangle(box.tolist(), outline=(255, 0, 0), width=6)

        frames_tracked.append(frame_draw.resize((640, 360), Image.BILINEAR))
    d = display.display(frames_tracked[0], display_id=True)
    i = 1
    try:
        while True:
            d.update(frames_tracked[i % len(frames_tracked)])
            i += 1
    except KeyboardInterrupt:
        pass

if cv2.waitKey('q') == 27:
   break

capture.release()
cv2.destroyAllWindows()

while there is no face in the first frame it will crush , and rise the error

  TypeError: 'NoneType' object is not iterable

i want to even there is no faces it will remain capturing and looking for faces .
after detecting the first face in the first frame raised this error :

  TypeError: stat: path should be string, bytes, os.PathLike or integer, not cv2.VideoCapture

  http://dpaste.com/04WXP33

thanks for helping


title: pytorch nllloss function target shape mismatch

body:
I'm training a LSTM model using pytorch with batch size of 256 and NLLLoss() as loss function.
The loss function is having problem with the data shape.

The softmax output from the forward passing has shape of torch.Size([256, 4, 1181]) where 256 is batch size, 4 is sequence length, and 1181 is vocab size.

The target is in the shape of torch.Size([256, 4]) where 256 is batch size and 4 is the output sequence length.

When I was testing earlier with batch size of 1, the model works fine but when I add batch size, it is breaking. I read that NLLLoss() can take class target as input instead of one hot encoded target.

Am I misunderstanding it? Or did I not format the shape of the target correctly?

class LSTM(nn.Module):

    def __init__(self, embed_size=100, hidden_size=100, vocab_size=1181, embedding_matrix=...):
        super(LSTM, self).__init__()
        self.hidden_size = hidden_size
        self.word_embeddings = nn.Embedding(vocab_size, embed_size)
        self.word_embeddings.load_state_dict({'weight': torch.Tensor(embedding_matrix)})
        self.word_embeddings.weight.requires_grad = False
        self.lstm = nn.LSTM(embed_size, hidden_size)
        self.hidden2out = nn.Linear(hidden_size, vocab_size)

    def forward(self, tokens):
        batch_size, num_steps = tokens.shape
        embeds = self.word_embeddings(tokens)
        lstm_out, _ = self.lstm(embeds.view(batch_size, num_steps, -1))
        out_space = self.hidden2out(lstm_out.view(batch_size, num_steps, -1))
        out_scores = F.log_softmax(out_space, dim=1)
        return out_scores

model = LSTM(self.config.embed_size, self.config.hidden_size, self.config.vocab_size, self.embedding_matrix)
loss_function = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=self.config.lr)

Error:

~/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1846         if target.size()[1:] != input.size()[2:]:
   1847             raise ValueError('Expected target size {}, got {}'.format(
-> 1848                 out_size, target.size()))
   1849         input = input.contiguous().view(n, c, 1, -1)
   1850         target = target.contiguous().view(n, 1, -1)

ValueError: Expected target size (256, 554), got torch.Size([256, 4])


title: pytorch torch.jit.trace returns function instead of torch.jit.ScriptModule

body:
I need to run in c++ a pre-trained pytorch nn model (trained in python) to make predictions.

To do so, I'm following the instructions on how to load a pytorch model in c++ given here: https://pytorch.org/tutorials/advanced/cpp_export.html

But when I try to get the torch.jit.ScriptModule via tracing as stated in the first step of the tutorial:

    traced_script_module =
        torch.jit.trace(model, (input_tensor_1, input_tensor_2))

Instead of returning a torch.jit.ScriptModule, it returns a function:

    print(type(traced_script_module))
    <type 'function'>

Which, when I run:

    traced_script_module.save("model.pt")

then leads into the following error:

Traceback (most recent call last):
  File "serialize_model.py", line 60, in <module>
    traced_script_module.save("model.pt")
AttributeError: 'function' object has no attribute 'save'

Any ideas on what I'm doing wrong?


title: why do we need to call `detach_` in `zero_grad`?

body:
I am reading the source code of optimizer.zero_grad()

http://pytorch.org/docs/master/_modules/torch/optim/optimizer.html#Optimizer.zero_grad

def zero_grad(self):
    """Clears the gradients of all optimized :class:`Variable` s."""
    for group in self.param_groups:
        for p in group['params']:
            if p.grad is not None:
                p.grad.detach_()
                p.grad.zero_()

I wonder why detach_() is necessary? What does it even mean to detach a gradient variable instead of a normal variable?

Why isn't zero_() enough?


I'm trying to build a model for document classification. I'm using BERT with PyTorch.
I got the bert model with below code.
bert = AutoModel.from_pretrained('bert-base-uncased')

This is the code for training.
for epoch in range(epochs):

    print('\n Epoch {:} / {:}'.format(epoch + 1, epochs))

    #train model
    train_loss, _ = modhelper.train(proc.train_dataloader)

    #evaluate model
    valid_loss, _ = modhelper.evaluate()

    #save the best model
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(modhelper.model.state_dict(), 'saved_weights.pt')

    # append training and validation loss
    train_losses.append(train_loss)
    valid_losses.append(valid_loss)

    print(f'\nTraining Loss: {train_loss:.3f}')
    print(f'Validation Loss: {valid_loss:.3f}')

this is my train method, accessible with the object modhelper.
def train(self, train_dataloader):
    self.model.train()
    total_loss, total_accuracy = 0, 0

    # empty list to save model predictions
    total_preds=[]

        # iterate over batches
    for step, batch in enumerate(train_dataloader):

        # progress update after every 50 batches.
        if step % 50 == 0 and not step == 0:
            print('  Batch {:>5,}  of  {:>5,}.'.format(step, len(train_dataloader)))

        # push the batch to gpu
        #batch = [r.to(device) for r in batch]

        sent_id, mask, labels = batch

        # clear previously calculated gradients
        self.model.zero_grad()

        print(sent_id.size(), mask.size())
        # get model predictions for the current batch
        preds = self.model(sent_id, mask) #This line throws the error

        # compute the loss between actual and predicted values
        self.loss = self.cross_entropy(preds, labels)

        # add on to the total loss
        total_loss = total_loss + self.loss.item()

        # backward pass to calculate the gradients
        self.loss.backward()

        # clip the the gradients to 1.0. It helps in preventing the exploding gradient problem
        torch.nn.utils.clip_grad_norm_(self.model.parameters(), 1.0)

        # update parameters
        self.optimizer.step()

        # model predictions are stored on GPU. So, push it to CPU
        #preds=preds.detach().cpu().numpy()

        # append the model predictions
        total_preds.append(preds)

    # compute the training loss of the epoch
    avg_loss = total_loss / len(train_dataloader)

    # predictions are in the form of (no. of batches, size of batch, no. of classes).
    # reshape the predictions in form of (number of samples, no. of classes)
    total_preds  = np.concatenate(total_preds, axis=0)

    #returns the loss and predictions
    return avg_loss, total_preds

preds = self.model(sent_id, mask) this line throws the following error(including full traceback).
 Epoch 1 / 1
torch.Size([32, 4000]) torch.Size([32, 4000])
Traceback (most recent call last):

File "<ipython-input-39-17211d5a107c>", line 8, in <module>
train_loss, _ = modhelper.train(proc.train_dataloader)

File "E:\BertTorch\model.py", line 71, in train
preds = self.model(sent_id, mask)

File "E:\BertTorch\venv\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)

File "E:\BertTorch\model.py", line 181, in forward
#pass the inputs to the model

File "E:\BertTorch\venv\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)

File "E:\BertTorch\venv\lib\site-packages\transformers\modeling_bert.py", line 837, in forward
embedding_output = self.embeddings(

File "E:\BertTorch\venv\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)

File "E:\BertTorch\venv\lib\site-packages\transformers\modeling_bert.py", line 201, in forward
embeddings = inputs_embeds + position_embeddings + token_type_embeddings

RuntimeError: The size of tensor a (4000) must match the size of tensor b (512) at non-singleton dimension 1

If you observe I've printed the torch size in the code.
print(sent_id.size(), mask.size())
The output of that line of code is torch.Size([32, 4000]) torch.Size([32, 4000]).
as we can see that size is the same but it throws the error. Please put your thoughts. Really appreciate it.
please comment if you need further information. I'll be quick to add whatever is required.