title: Loss with custom backward function in PyTorch - exploding loss in simple MSE example

body:
Before working on something more complex, where I knew I would have to implement my own backward pass, I wanted to try something nice and simple. So, I tried to do linear regression with mean squared error loss using PyTorch. This went wrong (see third implementation option below) when I defined my own backward method and I suspect it's because I'm not thinking very clearly about what I need to send PyTorch as gradients. So, I suspect what I need is some explanation/clarification/advice on what PyTorch expects me to provide in what form here.
I am using PyTorch 1.7.0, so a bunch of old examples no longer work (different way of working with user-defined autograd functions as described in the documentation).
First approach (standard PyTorch MSE loss function)
Let's first do it the standard way without a custom loss function:
import torch
import torch.nn as nn
import torch.nn.functional as F

# Let's generate some fake data
torch.manual_seed(42)
resid = torch.rand(100)
inputs = torch.tensor([ [ xx ] for xx in range(100)] , dtype=torch.float32)
labels = torch.tensor([ (2 + 0.5*yy + resid[yy]) for yy in range(100)], dtype=torch.float32)

# Now we define a linear regression model
class linearRegression(torch.nn.Module):
    def __init__(self, inputSize, outputSize):
        super(linearRegression, self).__init__()
        self.bn = torch.nn.BatchNorm1d(num_features=1)
        self.linear = torch.nn.Linear(inputSize, outputSize)

    def forward(self, inx):
        x = self.bn(inx) # Adding BN to standardize input helps us use a higher learning rate
        x = self.linear(x)
        return x

model = linearRegression(1, 1)

# Using the standard mse_loss of PyTorch
epochs = 25
mseloss = F.mse_loss
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=1e-3)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.1)

for epoch in range(epochs):
    model.train()
    optimizer.zero_grad()
    outputs = model(inputs)
    loss = mseloss(outputs.view(-1), labels)
    loss.backward()
    optimizer.step()
    scheduler.step()
    print(f'epoch {epoch}, loss {loss}')

This train just fine and I get to a loss of about 0.0824 and a plot of the fit looks fine.
Second approach (custom loss function, but relying on PyTorch's automatic gradient calculation)
So, now I replace the loss function with my own implementation of the MSE loss, but I still rely on PyTorch autograd. The only things I change here are defining the custom loss function, correspondingly defining the loss based on that, and a minor detail for how I hand over the predictions and true labels to the loss function.
#######################################################3
class MyMSELoss(nn.Module):

    def __init__(self):
        super(MyMSELoss, self).__init__()

    def forward(self, inputs, targets):
        tmp = (inputs-targets)**2
        loss =  torch.mean(tmp)
        return loss
#######################################################3

model = linearRegression(1, 1)

mseloss = MyMSELoss()
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=1e-3)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.1)

for epoch in range(epochs):
    model.train()
    outputs = model(inputs)
    loss = mseloss(outputs.view(-1), labels)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    scheduler.step()

    print(f'epoch {epoch}, loss {loss}')

This gives completely identical results as using the standard MSE loss function. Loss over epochs looks like this:
epoch 0, loss 884.2006225585938
epoch 1, loss 821.930908203125
epoch 2, loss 718.7732543945312
epoch 3, loss 538.1835327148438
epoch 4, loss 274.50909423828125
epoch 5, loss 55.115299224853516
epoch 6, loss 2.405021905899048
epoch 7, loss 0.47621214389801025
epoch 8, loss 0.1584305614233017
epoch 9, loss 0.09725229442119598
epoch 10, loss 0.0853077694773674
epoch 11, loss 0.08297089487314224
epoch 12, loss 0.08251354098320007
epoch 13, loss 0.08242412656545639
epoch 14, loss 0.08240655809640884
epoch 15, loss 0.08240310847759247
epoch 16, loss 0.08240246027708054
epoch 17, loss 0.08240233361721039
epoch 18, loss 0.08240240067243576
epoch 19, loss 0.08240223675966263
epoch 20, loss 0.08240225911140442
epoch 21, loss 0.08240220695734024
epoch 22, loss 0.08240220695734024
epoch 23, loss 0.08240220695734024
epoch 24, loss 0.08240220695734024

Third approach (custom loss function with my own backward method)
Now, the final version, where I implement my own gradients for the MSE. For that I define my own backward method in the loss function class and apparently need to do mseloss = MyMSELoss.apply.
from torch.autograd import Function

#######################################################
class MyMSELoss(Function):

    @staticmethod
    def forward(ctx, y_pred, y):
        ctx.save_for_backward(y_pred, y)
        return ( (y - y_pred)**2 ).mean()

    @staticmethod
    def backward(ctx, grad_output):
        y_pred, y = ctx.saved_tensors
        grad_input = torch.mean( -2.0 * (y - y_pred)).repeat(y_pred.shape[0])
        # This fails, as does grad_input = -2.0 * (y-y_pred)
        # I've also messed around with the sign and that's not the sole problem, either.
        return grad_input, None

#######################################################

model = linearRegression(1, 1)
mseloss = MyMSELoss.apply
optimizer = torch.optim.SGD(model.parameters(), lr=1e-2, momentum=1e-3)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, step_size=20, gamma=0.1)

for epoch in range(epochs):
    model.train()
    outputs = model(inputs)
    loss = mseloss(outputs.view(-1), labels)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    scheduler.step()
    print(f'epoch {epoch}, loss {loss}')

This is where things go wrong and instead of the training loss decreasing, I get increasing training loss. Now it looks like this:
epoch 0, loss 884.2006225585938
epoch 1, loss 3471.384033203125
epoch 2, loss 47768555520.0
epoch 3, loss 1.7422577779621402e+33
epoch 4, loss inf
epoch 5, loss nan
epoch 6, loss nan
epoch 7, loss nan
epoch 8, loss nan
epoch 9, loss nan
epoch 10, loss nan
epoch 11, loss nan
epoch 12, loss nan
epoch 13, loss nan
epoch 14, loss nan
epoch 15, loss nan
epoch 16, loss nan
epoch 17, loss nan
epoch 18, loss nan
epoch 19, loss nan
epoch 20, loss nan
epoch 21, loss nan
epoch 22, loss nan
epoch 23, loss nan
epoch 24, loss nan


title: How to implement an upper bound JSD loss in Pytorch?

body:
I’m trying to “replicate” TextGAN using pytorch and I’m new to pytorch. My current concern is to replicate the L_G (eq. 7 page 3), and here’s my current code:

class JSDLoss(nn.Module):

    def __init__(self):
        super(JSDLoss,self).__init__()

    def forward(self, batch_size, f_real, f_synt):
        assert f_real.size()[1] == f_synt.size()[1]

        f_num_features = f_real.size()[1]
        identity = autograd.Variable(torch.eye(f_num_features)*0.1, requires_grad=False)

        if use_cuda:
            identity = identity.cuda(gpu)

        f_real_mean = torch.mean(f_real, 0, keepdim=True)
        f_synt_mean = torch.mean(f_synt, 0, keepdim=True)

        dev_f_real = f_real - f_real_mean.expand(batch_size,f_num_features)
        dev_f_synt = f_synt - f_synt_mean.expand(batch_size,f_num_features)

        f_real_xx = torch.mm(torch.t(dev_f_real), dev_f_real)
        f_synt_xx = torch.mm(torch.t(dev_f_synt), dev_f_synt)

        cov_mat_f_real = (f_real_xx / batch_size) - torch.mm(f_real_mean, torch.t(f_real_mean)) + identity
        cov_mat_f_synt = (f_synt_xx / batch_size) - torch.mm(f_synt_mean, torch.t(f_synt_mean)) + identity

        cov_mat_f_real_inv = torch.inverse(cov_mat_f_real)
        cov_mat_f_synt_inv = torch.inverse(cov_mat_f_synt)

        temp1 = torch.trace(torch.add(torch.mm(cov_mat_f_synt_inv, cov_mat_f_real), torch.mm(cov_mat_f_real_inv, cov_mat_f_synt)))
        temp1 = temp1.view(1,1)
        temp2 = torch.mm(torch.mm((f_synt_mean - f_real_mean), (cov_mat_f_synt_inv + cov_mat_f_real_inv)), torch.t(f_synt_mean - f_real_mean))
        loss_g = torch.add(temp1, temp2).mean()

        return loss_g

It works. However, I have a suspicion that it's not the way to create a custom loss. Any kind of help is well appreciated! Thanks in advance :)


title: Caffe 2 Install fails on Windows 10

body:
It's been a fairly treacherous journey with dozens of error messages. I am installing Caffe 2 from source onto a Windows 10 machine with an Nvidia GPU. I have installed VS2017 Community Edition (CE), CUDA 9.2 and cuDNN. I am able to build the CUDA sample .sln files into .exe's and run the .exe's successfully.

Presently, I am trying to install Caffe 2. Naturally, there's an error on this step as well. Like my previous issues, I have tried Googling this error. Unfortunately, I am unable to find others with the same issue and consequently cannot find hints for a solution.

According to the Caffe 2 installation instrucitons at: https://caffe2.ai/docs/getting-started.html?platform=windows&configuration=compile:

1. git clone --recursive https://github.com/pytorch/pytorch.git
2. build_windows.bat from the Developer Command Prompt for VS 2017

on step 2., I get the following error:

       "C:\Users\andcy\pytorch\build\ALL_BUILD.vcxproj" (default target) (1) ->
       "C:\Users\andcy\pytorch\build\caffe2\caffe2.vcxproj" (default target) (13) ->
       (ClCompile target) ->
         c:\users\andcy\pytorch\third_party\eigen\eigen\src\core\products\generalblockpanelkernel.h(1902): fatal error C1001: An internal error has occurred in the compiler. [C:\Users\andcy\pytorch\build\caffe2\caffe2.vcxproj]
         cl : Command line error D8040: error creating or communicating with child process [C:\Users\andcy\pytorch\build\caffe2\caffe2.vcxproj]

    3282 Warning(s)
    2 Error(s)

Time Elapsed 00:05:11.58
"Caffe2 building failed"

C:\Users\andcy\pytorch\scripts>

Does anyone have any advice on how to resolve this issue? I've been trying to install Caffe 2 on my local machine for about a week.


title: Why my estimated negative log likelihood loss is constant for all the input images

body:
I am trying to learn the manifold of a particular class (say digit 1 for the MNIST dataset) using the latent space density estimation. I am using a Gaussian model (Mixture Density Network, MDN) to estimate the densities of the latent space (obtained through an autoencoder). I am using a neural network to approximate mean and variance for the MDN.

I facing a problem that, all of my estimated densities and consequently the negative log-likelihood loss is coming constant. This can be true for learning the manifold of digit 1 (although there should be some variance) but should be constant while testing it with the other digits.

Below you can see the loss plot for the learned class (normal class) and the new class (anomaly class)

Below is the code for the MDN and MDn loss estimation -

import sys
import torch
import warnings
import numpy as np
import torch.nn as nn
import seaborn as sns
import torch.optim as optim
import matplotlib.pyplot as plt
import torch.nn.functional as F
from torch.autograd import Variable
from sklearn.model_selection import train_test_split
from scipy import ndimage
#from hessian import hessian
warnings.filterwarnings('ignore')

COEFS = 10
IN_DIM =32
OUT_DIM = IN_DIM

class MDN(nn.Module):

    def __init__(self, input_dim=IN_DIM, out_dim=OUT_DIM, layer_size=10, coefs=COEFS, test = False, sd =0.2):
        super(MDN, self).__init__()
        self.fc = nn.Linear(input_dim, layer_size)
        self.fc2 = nn.Linear(layer_size, layer_size)
        self.pi = nn.Linear(layer_size, coefs)
        self.mu = nn.Linear(layer_size, out_dim*coefs) # mean
#        self.mu = nn.Linear(layer_size, coefs) # mean
        self.sigma_sq = nn.Linear(layer_size, coefs*out_dim) # isotropic independent variance
        self.out_dim = out_dim
        self.coefs = coefs
        self.test = test
        self.sd = sd

    def forward(self, x):

        if self.test:
            y = x
        else:
            ## Adding Noise ##
            y = add_noise(x, noise_type="gaussian", sd =self.sd)
#            y =x

        ## Learning ditribution parameters ##
        for i in range(x.size(0)):
            x = F.leaky_relu(self.fc(y[i]))
            x = F.leaky_relu(self.fc2(x))
            pi = F.softmax(self.pi(x), dim=-1)
            sigma_sq = F.softplus(self.sigma_sq(x).view(self.coefs,-1)) #logvar
            mu = self.mu(x).view(self.coefs,-1) #mean
        return pi, mu, sigma_sq

'''
    functions to compute the log-likelihood of data according to a gaussian mixture model.
    All the computations are done with log because exp will lead to numerical underflow errors.
'''

def log_gaussian(x, mean, logvar):
    '''
    Computes the Gaussian log-likelihoods

    Parameters:
        x: [samples,features]  data samples
        mean: [features]  Gaussian mean (in a features-dimensional space)
        logvar: [features]  the logarithm of variances [no linear dependance hypotesis: we assume one variance per dimension]

    Returns:
         [samples]   log-likelihood of each sample
    '''
    a = (x - mean) ** 2  # works on multiple samples thanks to tensor broadcasting
    log_p = (logvar + a / logvar.exp()).sum(-1)
    log_p = -0.5 * ( np.log(2 * np.pi) + log_p )
#    log_p = -0.5 * log_p  #forgetting constants
    return log_p

def log_gmm(x, means, logvars, weights, total=True):
    '''
    Computes the Gaussian Mixture Model log-likelihoods

    Parameters:
        x: [samples,features]  data samples
        means:  [K,features]   means for K Gaussians
        logvars: [K,features]  logarithm of variances for K Gaussians  [no linear dependance hypotesis: we assume one variance per dimension]
        weights: [K]  the weights of each Gaussian
        total:   wether to sum the probabilities of each Gaussian or not (see returning value)

    Returns:
        [samples]  if total=True. Log-likelihood of each sample
        [K,samples] if total=False. Log-likelihood of each sample for each model

    '''
    res = log_gaussian(
            x[None,:,:],  # (1,examples,features)
            means[:,None,:],  # (K,1,features)
            logvars[:,None,:] # (K,1,features)
    )
    # res is (K,examples)
    res = weights[:,None] * res

    if total:
        return res.sum(0)
    else:
        return res

# for Mishra
def mdn_loss_function(x, means, logvars, weights, test = False):
    if test:
        res = -log_gmm(x, means, logvars, weights)
    else:
        res = torch.mean(-log_gmm(x, means, logvars, weights))
    return res

def loss_fn(y, pi, mu, sigma_sq, model, sd =0.2, test = False):
    loss1 = mdn_loss_function(y, mu,sigma_sq, pi, test = test)
    if test:
        return loss1
    else:

    #    hes = hessian(loss1, model.parameters(), out=h)
        hes = torch.tensor(ndimage.filters.laplace(y.detach().cpu().numpy())).cuda()
    #    print(f"hessian : {hes}")

        loss = loss1 + 0.5*(torch.tensor(sd)**2)*torch.sum(hes)
        return loss

##### Adding Noise ############

def add_noise(latent, noise_type="gaussian", sd =0.2):
    """Here we add noise to the latent features concatenated from the 4 autoencoders.
    Arguments:
    'gaussian' (string): Gaussian-distributed additive noise.
    'speckle' (string) : Multiplicative noise using out = image + n*image, where n is uniform noise with specified mean & variance.
    'sd' (integer) : standard deviation used for generating noise

    Input :
        latent : numpy array or cuda tensor.

    Output:
        Array: Noise added input, can be np array or cuda tnesor.
    """
    assert sd >= 0.0
    if noise_type=="gaussian":
        mean=0.

        n = torch.distributions.Normal(torch.tensor([mean]), torch.tensor([sd]))
        noise = n.sample(latent.size()).squeeze(2).cuda()
        latent = latent + noise
        return latent

    if noise_type=="speckle":
        noise = torch.randn(latent.size()).cuda()
        latent = latent + latent*noise
        return latent


title: How can I remove elements across a dimension that are all zero with PyTorch?

body:
I have a tensor:

tensor([[0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1.,
         1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
        [0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
         0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.]],
       device='cuda:0', grad_fn=<AsStridedBackward>)
torch.Size([3, 176])

You see that the first and the third have all zeroes. I'd like to remove it and change this to have size of [1, 176].

If I do print((mask != 0).nonzero()), then I see:

tensor([[  0, 136],
        [  0, 137],
        [  0, 138],
        [  0, 139],
        [  0, 140],
        [  0, 141],
        [  0, 142],
        [  0, 143],
        [  0, 144],
        [  0, 145],
        [  0, 146],
        [  0, 147],
        [  0, 148],
        [  0, 149],
        [  0, 150],
        [  0, 151],
        [  0, 152],
        [  0, 153],
        [  0, 154],
        [  0, 155],
        [  0, 156],
        [  0, 157],
        [  0, 158],
        [  0, 159]], device='cuda:0')

So I feel like that's a part of it, but how do I further reduce the tensor?


Update #1 (original question and details below):

As per the suggestion of @MatthijsHollemans below I've tried to run this by removing dynamic_axes from the initial create_onnx step below. This removed both:

Description of image feature 'input_image' has missing or non-positive width 0.

and

Input 'input_image' of layer '63' not found in any of the outputs of the preceeding layers.

Unfortunately this opens up two sub-questions:

I still want to have a functional ONNX model. Is there a more appropriate way to make H and W dynamic? Or should I be saving two versions of the ONNX model, one without dynamic_axes for the CoreML conversion, and one with for use as a valid ONNX model?
Although this solves the compilation error in xcode (specified below) it introduces the following runtime issues:

Finalizing CVPixelBuffer 0x282f4c5a0 while lock count is 1.
[espresso] [Espresso::handle_ex_plan] exception=Invalid X-dimension 1/480 status=-7
[coreml] Error binding image input buffer input_image: -7
[coreml] Failure in bindInputsAndOutputs.

I am calling this the same way I was calling the fixed size model, which does still work fine. The image dimensions are 640 x 480.

As specified below the model should accept any image between 64x64 and higher.

For flexible shape models, do I need to provide an input differently in xcode?

Original Question (parts still relevant)

I have been slowly working on converting a style transfer model from pytorch > onnx > coreml. One of the issues that has been a struggle is flexible/dynamic input + output shape.

This method (besides i/o renaming) has worked well on iOS 12 & 13 when using a static input shape.

I am using the following code to do the onnx > coreml conversion:

def create_coreml(name):
    mlmodel = convert(
            model="onnx/" + name + ".onnx",
            preprocessing_args={'is_bgr': True},
            deprocessing_args={'is_bgr': True},
            image_input_names=['input_image'],
            image_output_names=['stylized_image'],
            minimum_ios_deployment_target='13'
            )

    spec = mlmodel.get_spec()

    img_size_ranges = flexible_shape_utils.NeuralNetworkImageSizeRange()

    img_size_ranges.add_height_range((64, -1))
    img_size_ranges.add_width_range((64, -1))

    flexible_shape_utils.update_image_size_range(
        spec,
        feature_name='input_image',
        size_range=img_size_ranges)

    flexible_shape_utils.update_image_size_range(
        spec,
        feature_name='stylized_image',
        size_range=img_size_ranges)

    mlmodel = coremltools.models.MLModel(spec)

    mlmodel.save("mlmodel/" + name + ".mlmodel")

Although the conversion 'succeeds' there are a couple of warnings (spaces added for readability):

Translation to CoreML spec completed. Now compiling the CoreML model.
/usr/local/lib/python3.7/site-packages/coremltools/models/model.py:111:
RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was:
Error compiling model:
"Error reading protobuf spec. validator error: Description of image feature 'input_image' has missing or non-positive width 0.".
  RuntimeWarning)

Model Compilation done.
/usr/local/lib/python3.7/site-packages/coremltools/models/model.py:111:
RuntimeWarning: You will not be able to run predict() on this Core ML model. Underlying exception message was:
Error compiling model:
"compiler error:  Input 'input_image' of layer '63' not found in any of the outputs of the preceeding layers.
".
  RuntimeWarning)

If I ignore these warnings and try to compile the model for latest targets (13.0) I get the following error in xcode:

coremlc: Error: compiler error:  Input 'input_image' of layer '63' not found in any of the outputs of the preceeding layers.

Here is what the problematic area appears to look like in netron:

My main question is how can I get these two warnings out of the way?

Happy to provide any other details.

Thanks for any advice!

Below is my pytorch > onnx conversion:

def create_onnx(name):
    prior = torch.load("pth/" + name + ".pth")
    model = transformer.TransformerNetwork()
    model.load_state_dict(prior)

    dummy_input = torch.zeros(1, 3, 64, 64) # I wasn't sure what I would set the H W to here?

    torch.onnx.export(model, dummy_input, "onnx/" + name + ".onnx",
                      verbose=True,
                      opset_version=10,
                      input_names=["input_image"], # These are being renamed from garbled originals.
                      output_names=["stylized_image"], # ^
                      dynamic_axes={'input_image':
                                    {2: 'height', 3: 'width'},
                                    'stylized_image':
                                    {2: 'height', 3: 'width'}}
                      )

    onnx.save_model(original_model, "onnx/" + name + ".onnx")


title: How to add a L1 or L2 regularization to weights in pytorch

body:
In tensorflow, we can add a L1 or L2 regularizations in the sequential model. I couldn't find equivalent approach in pytorch. How can we add regularizations to weights in pytorch in the definition of the net:
class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)   # hidden layer
        """ How to add a L1 regularization after a certain hidden layer?? """
        """ OR How to add a L1 regularization after a certain hidden layer?? """
        self.predict = torch.nn.Linear(n_hidden, n_output)   # output layer

    def forward(self, x):
        x = F.relu(self.hidden(x))      # activation function for hidden layer
        x = self.predict(x)             # linear output
        return x

net = Net(n_feature=1, n_hidden=10, n_output=1)     # define the network
# print(net)  # net architecture
optimizer = torch.optim.SGD(net.parameters(), lr=0.2)
loss_func = torch.nn.MSELoss()  # this is for regression mean squared loss


title: plt.show() gives some response in IPython but not in Visual Studio Code

body:
I'm new to ML & AI in general, but I'm attempting to make sure I installed everything properly first before I do any real experimentation. I've been following a tutorial of sorts and I'm currently experiencing something strange.

It appears that when you install Anaconda to make it easier to get Pytorch, it also installs IPython. I'm not entirely sure what it is but I assume its some sort of IDE as it can run code. I noticed by following the tutorial that plt.plot(a) doesn't spit out any response when I run the code in Visual Studio Code. However, the next line plt.show() works and spits out the expected graph image. I'm really confused about what is happening here. Is it because the line with the red icon in IPython symbolizes something internal that visual studio code doesn't print out? Also I noticed that in VSC that I have to have Anaconda 5.2.0 on the lower left instead of Python 3.7(64 bit). Is it because pytorch is only accessible though Anaconda when I installed pytorch with Anaconda? Should I just leave it at Anaconda?


title: PyTorch dataloader performance slump on HDD

body:
I've just built a new PC for DL and I'm testing it on the official Imagenet example from PyTorch. I'm seeing reasonable performance when the dataset resides on my SSD (GoodRam IRDM Pro 240GB SATA3 (IRP-SSDPR-S25B-240)), but it becomes ridiculously slow on my HDD (Toshiba P300 (HDWD120UZSVA)). It seems to be all about the DataLoader. Of course, SDD is expected to perform better than HDD, but I don't think this workload should even be bottlenecked by disc read (on other machines I used, it was either GPU bound, or CPU bound due to preprocessing), let alone to this extent. To investigate, I wrote a quick iterator wrapper to time the dataloader calls

def time_iter(iter):
    while True:
        start = time.time()
            item = next(iter)
            print('Yielding in', time.time() - start)
            yield item
        except StopIteration:
            break

And checked the results by evaluating a pretrained net on Imagenet. The results on HDD:

jatentaki@Dzik:~/Programs/pytorch-examples/imagenet$ python main.py --pretrained -e -b 768 -j 10 ~/2tb/Datasets/ILSVRC2012/
=> using pre-trained model 'resnet18'
Yielding in 94.4384298324585
Test: [0/66]    Time 97.014 (97.014)    Loss 0.6302 (0.6302)    Prec@1     82.943 (82.943)  Prec@5 95.573 (95.573)
Yielding in 0.00038623809814453125
Yielding in 0.00019431114196777344
Yielding in 0.0001766681671142578
Yielding in 0.0002028942108154297
Yielding in 0.00017595291137695312
Yielding in 0.00017023086547851562
Yielding in 0.000179290771484375
Yielding in 0.00019288063049316406
Yielding in 0.00017714500427246094
Yielding in 85.04550909996033
Test: [10/66]   Time 85.408 (16.858)    Loss 1.1352 (0.8930)    Prec@1 69.661 (77.190)  Prec@5 91.927 (92.779)
Yielding in 0.15804052352905273
Yielding in 0.00020647048950195312
Yielding in 2.0329136848449707
Yielding in 0.00020360946655273438

The ~90 second spikes consistently appear every 9 iterations. On SDD:

jatentaki@Dzik:~/Programs/pytorch-examples/imagenet$ python main.py --pretrained -e -b 768 -j 10 ~/FastDatasets/
=> using pre-trained model 'resnet18'
Yielding in 11.228104829788208
Test: [0/66]    Time 14.272 (14.272)    Loss 0.6302 (0.6302)    Prec@1 82.943 (82.943)  Prec@5 95.573 (95.573)
Yielding in 0.00038361549377441406
Yielding in 0.00030112266540527344
Yielding in 0.0002224445343017578
Yielding in 0.0002486705780029297
Yielding in 0.00018787384033203125
Yielding in 0.0002593994140625
Yielding in 0.00020194053649902344
Yielding in 0.0003197193145751953
Yielding in 0.00019288063049316406
Yielding in 3.4066810607910156
Test: [10/66]   Time 4.013 (1.946)  Loss 1.1352 (0.8930)    Prec@1 69.661     (77.190)  Prec@5 91.927 (92.779)
Yielding in 1.5148968696594238
Yielding in 0.0003371238708496094
Yielding in 0.0002467632293701172

Other diagnostics: iotop total disk read peaks at 12-14M/s for HDD and 100-120M/s for SDD. hdparm results:

jatentaki@Dzik:~$ sudo hdparm -Tt /dev/sda1
/dev/sda1: # SSD
Timing cached reads:   20976 MB in  2.00 seconds = 10503.39 MB/sec
Timing buffered disk reads: 512 MB in  1.02 seconds = 501.77 MB/sec

jatentaki@Dzik:~$ sudo hdparm -Tt /dev/sdb1
/dev/sdb1: # HDD
Timing cached reads:   19484 MB in  2.00 seconds = 9755.75 MB/sec
Timing buffered disk reads: 586 MB in  3.01 seconds = 194.69 MB/sec

What is a possible cause of the problems? I am considering improper installation/synergy between the hardware pieces (I'm a total hardware noob) or some issues with how the DataLoader works. Another possibility is improperly configured OS (Xubuntu 18.04 LTS). Apparently there's a lot of caching going on (all the 0.000...s yields), could it go wrong?

System: Xubuntu 18.04 LTS
PyTorch: 0.4.1, from Anaconda, with CUDA 9.2 and 396 nvidia driver.
GPU: GTX 1080 Ti


title: Import LSTM from Tensorflow to PyTorch by hand

body:
I am trying to import a pretrained Model from tensorflow to PyTorch. It takes a single input and maps it onto a single output.
Confusion comes up, when I try to import the LSTM weights

I read the weights and their variables from the file with the following function:

def load_tf_model_weights():

    modelpath = 'models/model1.ckpt.meta'

    with tf.Session() as sess:
        tf.train.import_meta_graph(modelpath)
        init = tf.global_variables_initializer()
        sess.run(init)
        vars = tf.trainable_variables()
        W = sess.run(vars)

    return W,vars

W,V = load_tf_model_weights()

Then I am inspecting the shapes of the weights

In [33]:  [w.shape for w in W]
Out[33]: [(51, 200), (200,), (100, 200), (200,), (50, 1), (1,)]

furthermore the variables are defined as

In [34]:    V
Out[34]:
[<tf.Variable 'rnn/multi_rnn_cell/cell_0/lstm_cell/kernel:0' shape=(51, 200) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_0/lstm_cell/bias:0' shape=(200,) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/lstm_cell/kernel:0' shape=(100, 200) dtype=float32_ref>,
<tf.Variable 'rnn/multi_rnn_cell/cell_1/lstm_cell/bias:0' shape=(200,) dtype=float32_ref>,
<tf.Variable 'weight:0' shape=(50, 1) dtype=float32_ref>,
<tf.Variable 'FCLayer/Variable:0' shape=(1,) dtype=float32_ref>]

So I can say that the first element of  W defines the  Kernel of an LSTM and the second element define its bias. According to this post, the shape for the Kernel is defined as
[input_depth + h_depth, 4 * self._num_units]
and the bias as [4 * self._num_units]. We already know that input_depth is 1. So we get, that h_depth and _num_units both have the value 50.

In pytorch my LSTMCell, to which I want to assign the weights, looks like this:

In [38]: cell = nn.LSTMCell(1,50)
In [39]: [p.shape for p in cell.parameters()]
Out[39]:
[torch.Size([200, 1]),
torch.Size([200, 50]),
torch.Size([200]),
torch.Size([200])]

The first two entries can be covered by the first value of W which has the shape (51,200). But the LSTMCell from Tensorflow yields only one bias of shape (200) while pytorch wants two of them

And by leaving the bias out I have weights left over:

cell2 = nn.LSTMCell(1,50,bias=False)
[p.shape for p in cell2.parameters()]
Out[43]: [torch.Size([200, 1]), torch.Size([200, 50])]

Thanks!


title: Running Detectron2 inference in Caffe2

body:
I have a Detectron2 .pth model that I converted successfully to Caffe2 .pb via the Detectron2 tools functionality located here: https://github.com/facebookresearch/detectron2/blob/master/tools/caffe2_converter.py

As recommended, used the --run-eval flag to confirm results while converting and the results are very similar to original detectron2 results.

To run inference on a new image using the resulting model.pb and model_init.pb files, used functionality located here:
https://github.com/facebookresearch/detectron2/blob/master/detectron2/export/api.py (mostly)
https://github.com/facebookresearch/detectron2/blob/master/detectron2/export/caffe2_inference.py

However, inference results are not even close. Can anybody suggest reasons why this might happen? Detectron2 repo says all preprocessing is done in the caffe2 scripts, but am I missing something?

I can provide my inference code:

caffe2_model = Caffe2Model.load_protobuf(input_directory)
img = cv2.imread(input_image)
image = torch.as_tensor(img.astype("float32").transpose(2, 0, 1))
data = {'image': image, 'height': image.shape[1], 'width': image.shape[2]}
output = caffe2_model([data])


title: Fairseq Transform model not working (Float can't be cast to long)

body:
I've installed python 3.8, pytorch 1.7, and fairseq 0.10.1, on a new machine, then copied in a script and model from a machine with python 3.6, pytorch 1.4 and fairseq 0.9.0, where it is working.
The model is loaded and prepared with:
model = TransformerModel.from_pretrained(...)
model.eval()
model.cuda()

Then used with:
inputs = [model.binarize(encode(src, str)) for str in texts]
batched_hypos = model.generate(inputs, beam)

inputs looks like e.g. [tensor([ 116, 1864,  181,    6,    2]), tensor([    5,   432,     7,   2])]
It asserts, with the last bit of the call stack being:
  ...
    batched_hypos = model.generate(inputs, beam)
  File "/path/to/fairseq/hub_utils.py", line 125, in generate
    sample = self._build_sample(tokens)
  File "/path/to/fairseq/hub_utils.py", line 196, in _build_sample
    assert torch.is_tensor(src_tokens)

If instead I use fairseq-interactive from the commandline it fails with RuntimeError: result type Float can't be cast to the desired output type Long. (Full stack trace below.)
As using the cli also fails, my hunch is that my model built with fairseq 0.9.x cannot be used with fairseq 0.10.x. If so, is there a way to update the model (i.e. without having to retrain it). And if not, what could the problem be, and how do I fix it?
BTW, exactly the same error if I add --cpu to the commandline args, so the GPU or cuda version can be eliminated as a possible cause.
$ fairseq-interactive path/to/dicts --path models/big.pt --source-lang ja --target-lang en  --remove-bpe sentencepiece

  File "/path/to/bin/fairseq-interactive", line 11, in <module>
    sys.exit(cli_main())
  File "/path/to/lib/python3.8/site-packages/fairseq_cli/interactive.py", line 190, in cli_main
    main(args)
  File "/path/to/lib/python3.8/site-packages/fairseq_cli/interactive.py", line 149, in main
    translations = task.inference_step(generator, models, sample)
  File "/path/to/lib/python3.8/site-packages/fairseq/tasks/fairseq_task.py", line 265, in inference_step
    return generator.generate(models, sample, prefix_tokens=prefix_tokens)
  File "/path/to/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/path/to/lib/python3.8/site-packages/fairseq/sequence_generator.py", line 113, in generate
    return self._generate(model, sample, **kwargs)
  File "/path/to/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/path/to/lib/python3.8/site-packages/fairseq/sequence_generator.py", line 376, in _generate
    cand_scores, cand_indices, cand_beams = self.search.step(
  File "/path/to/lib/python3.8/site-packages/fairseq/search.py", line 81, in step
    torch.div(self.indices_buf, vocab_size, out=self.beams_buf)
RuntimeError: result type Float can't be cast to the desired output type Long


title: Cuda error out of memory while running a python code

body:
I am trying to run this following code.
I am getting an error. The code is from Deep Image Prior downloaded from Github. Can anyone tell me why I am getting this error and where I am wrong?
I read something about this and it said to reduce the batch size. How to reduce it?
Is it because the .tif file is around 100 MB?
The error is RuntimeError: CUDA error: out of memory and I am using 12 GB GPU and CUDA 9.2

from __future__ import print_function
import matplotlib.pyplot as plt
%matplotlib inline
import argparse
import os
#os.environ['CUDA_VISIBLE_DEVICES'] = '0'
import numpy as np
from models import *
import torch
import torch.optim
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import warnings
warnings.filterwarnings("ignore")
from skimage.measure import compare_psnr
from models.downsampler import Downsampler
from utils.sr_utils import *
torch.backends.cudnn.enabled = True
torch.backends.cudnn.benchmark =True
dtype = torch.cuda.FloatTensor
device = torch.device("cuda:0" if torch.cuda.is_available() else"cpu")
imsize = (819,819)
factor = 2 # 8
enforse_div32 = 'CROP' # we usually need the dimensions to be divisible by     a power of two (32 in this case)
PLOT = True
# To produce images from the paper we took *_GT.png images from LapSRN     viewer for corresponding factor,
# e.g. x4/zebra_GT.png for factor=4, and x8/zebra_GT.png for factor=8
with torch.no_grad():
path_to_image = '/home/smitha/deep-image-prior/resize.tif'
imgs = load_LR_HR_imgs_sr(path_to_image , imsize, factor, enforse_div32)
imgs['bicubic_np'], imgs['sharp_np'], imgs['nearest_np'] =  get_baselines(imgs['LR_pil'], imgs['HR_pil'])
 if PLOT:
 plot_image_grid([imgs['HR_np'], imgs['bicubic_np'], imgs['sharp_np'],   imgs['nearest_np']], 4,12);
print ('PSNR bicubic: %.4f   PSNR nearest: %.4f' %  (
                                    compare_psnr(imgs['HR_np'], imgs['bicubic_np']),
                                    compare_psnr(imgs['HR_np'], imgs['nearest_np'])))
input_depth = 8
INPUT =     'noise'
pad   =     'reflection'
OPT_OVER =  'net'
KERNEL_TYPE='lanczos2'
LR = 1
tv_weight = 0.0
OPTIMIZER = 'adam'
if factor == 2:
num_iter = 10
reg_noise_std = 0.01
elif factor == 8:
num_iter = 40
reg_noise_std = 0.05
else:
assert False, 'We did not experiment with other factors'
net_input = get_noise(input_depth, INPUT, (imgs['HR_pil'].size[1],    imgs['HR_pil'].size[0])).type(dtype).detach()
NET_TYPE = 'skip' # UNet, ResNet
net = get_net(input_depth, 'skip', pad,
          skip_n33d=512,
          skip_n33u=512,
          skip_n11=4,
          num_scales=5,
          upsample_mode='bilinear').type(dtype)
      # Losses
      mse = torch.nn.MSELoss().type(dtype)
      img_LR_var = np_to_torch(imgs['LR_np']).type(dtype)
      downsampler = Downsampler(n_planes=3, factor=factor,   kernel_type=KERNEL_TYPE, phase=0.5, preserve_size=True).type(dtype)
     def closure():
     global i, net_input
     if reg_noise_std > 0:
     net_input = net_input_saved + (noise.normal_() * reg_noise_std)
     out_HR = net(net_input)
     out_LR = downsampler(out_HR)
     total_loss = mse(out_LR, img_LR_var)
     if tv_weight > 0:
     total_loss += tv_weight * tv_loss(out_HR)
     total_loss.backward()
     # Log
     psnr_LR = compare_psnr(imgs['LR_np'], torch_to_np(out_LR))
     psnr_HR = compare_psnr(imgs['HR_np'], torch_to_np(out_HR))
     print ('Iteration %05d    PSNR_LR %.3f   PSNR_HR %.3f' % (i, psnr_LR,  psnr_HR), '\r', end='')
      # History
      psnr_history.append([psnr_LR, psnr_HR])
      if PLOT and i % 100 == 0:
      out_HR_np = torch_to_np(out_HR)
      plot_image_grid([imgs['HR_np'], imgs['bicubic_np'],  np.clip(out_HR_np, 0, 1)], factor=13, nrow=3)
      i += 1
      return total_loss
       psnr_history = []
       net_input_saved = net_input.detach().clone()
       noise = net_input.clone()
       i = 0
       p = get_params(OPT_OVER, net, net_input)
       optimize(OPTIMIZER, p, closure, LR, num_iter)
       out_HR_np = np.clip(torch_to_np(net(net_input)), 0, 1)
       result_deep_prior = put_in_center(out_HR_np, imgs['orig_np'].shape[1:])
       # For the paper we acually took `_bicubic.png` files from LapSRN viewer   and used `result_deep_prior` as our result
       plot_image_grid([imgs['HR_np'],
             imgs['bicubic_np'],
             out_HR_np], factor=4, nrow=1);


title: How to set the environment default python in anaconda?

body:
It makes me crazy, In anaconda I create the environment with the defualt iterpreter python3.4   Next I install pytorch 0.4.1

conda install pytorch=0.4.1 cuda80 -c pytorch

After this I found that the pytorch was installed in python3.6!

And the environment defualt interpreter is chaged from python3.4 to python3.6.

I am very confused what happend ? How shoud I fix it back? change defualt python back to python3.4? Hope some one could help me.

The commands I typed in are as follows:

conda create -n pointgen python=3.4 ipykernel
source activate pointgen
conda install pytorch=0.4.1 cuda80 -c pytorch

Thats all. What Novak said is right, there is remaining question is how could I manually change the python version from 3.6 back to 3.4, is there any config file I can deal with?


title: How to change default optimization in spotlight from pytorch e.g. torch.optim.SGD?

body:
I'm currently using spotlight https://github.com/maciejkula/spotlight/tree/master/spotlight
to implement Matrix Factorization in recommender system. spotlight is based on pytorch, it's a integrated platform implementing RS. In spotlight/factorization/explicit, it uses torch.optim.Adam as optimizer, I want to change it to torch.optim.SGD.
I tried

emodel = ExplicitFactorizationModel(n_iter=15,
                                embedding_dim=32,
                                use_cuda=False,
                                loss='regression',
                                l2=0.00005,
                                optimizer_func=optim.SGD(lr=0.001, momentum=0.9))

but it gives:TypeError: init() missing 1 required positional argument: 'params'
Any suggestions?


title: Saving and reloading an LSTM model in Pytorch

body:
Has anyone ever tried to train a Pytorch LSTM model, save it, reload it somewhere else and then continue training? I've been trying to do something like this for the past 2 weeks with no good results (I kept track using the training loss). It seems everytime I reload the model and train it, it behaves arbitrarily (I know this because if I train it continuously it shows very different training losses).

I have tried using the pytorch.save() fn, saving the state_dict and loading it, native pickle mechanism as well as joblib for the same but all of them have the same issue. I even saved the optimizer states and reloaded it without much luck.

Could it somehow be related to the hidden and cell states of the LSTM layers? Should I save and reload them as well everytime I want to train? Or could it be something else entirely?

I have searched extensively for this issue but to no avail. Any help would be much appreciated.

Some more info: I'm trying to detect anomalies in data using an autoencoder and so each time I reload the model, it is trained only on a batch of data and then saved again to be reused for next batch.


title: what's the differences of the from_pretrained and weight.data.copy in Pytorch.Embedding

body:
recently , I use glove to initialize the weight of Embedding, I find out the differences of from_pretrained and weight.data.copy_ is that : from_pretrained = {weight.data.copy_() ; embedding.weight.requires_grad = False}
I want to know if what I said is correct?
and I got 10% less accurate with the glove than I did without the glove, but when I count the words that are not covered by glove, I find that only 10% of the words are not in glove.

import numpy as np
import pandas as pd

import unicodedata, re, string
import nltk

import torch
import torch.nn as nn
from torch.utils.data import TensorDataset, DataLoader

import seaborn as sns

sns.set(color_codes=True)

df_train = pd.read_csv(r'E:\Mice\train.tsv', delimiter='\t')  # r代表反转义
df_test = pd.read_csv(r'E:\Mice\test.tsv', delimiter='\t')

def remove_non_ascii(words):
    """Remove non-ASCII characters from list of tokenized words"""
    new_words = []
    for word in words:
        new_word = unicodedata.normalize('NFKD', word).encode('ascii', 'ignore').decode('utf-8', 'ignore')
        new_words.append(new_word)
    return new_words

def to_lowercase(words):
    """Convert all characters to lowercase from list of tokenized words"""
    new_words = []
    for word in words:
        new_word = word.lower()
        new_words.append(new_word)
    return new_words

def remove_punctuation(words):
    """Remove punctuation from list of tokenized words"""
    new_words = []
    for word in words:
        new_word = re.sub(r'[^\w\s]', '', word)
        if new_word != '':
            new_words.append(new_word)
    return new_words

def remove_numbers(words):
    """Remove all interger occurrences in list of tokenized words with textual representation"""
    new_words = []
    for word in words:
        new_word = re.sub(r"\d+", "", word)
        if new_word != '':
            new_words.append(new_word)
    return new_words

def normalize(words):
    words = remove_non_ascii(words)
    words = to_lowercase(words)
    words = remove_punctuation(words)
    words = remove_numbers(words)
    #    words = remove_stopwords(words)
    return words

df_train['Words'] = df_train['Phrase'].apply(nltk.word_tokenize)
df_train['Words'] = df_train['Words'].apply(normalize)
word_set = set()
for l in df_train['Words']:
    for e in l:
        word_set.add(e)

# 从这里开始构建词向量，简单的索引。
word_to_int = {word: ii for ii, word in enumerate(word_set, 1)}
df_train['Tokens'] = df_train['Words'].apply(lambda l: [word_to_int[word] for word in l])
max_len = df_train['Tokens'].str.len().max()
all_tokens = np.array([t for t in df_train['Tokens']])
encoded_labels = np.array([l for l in df_train['Sentiment']])
features = np.zeros((len(all_tokens), max_len), dtype=int)
# for each phrase, add zeros at the end
for i, row in enumerate(all_tokens):
    features[i, :len(row)] = row
# 到这里词向量构建结束

# 下面是另一种, 用glove预训练。
embeddings_index = {}
f = open('E:/Mice/glove.6B.100d.txt', encoding='utf-8')
for line in f:
    values = line.split()
    word = values[0]
    coefs = np.asarray(values[1:], dtype='float32')
    embeddings_index[word] = coefs
f.close()
# 同样属于glove, 构建词矩阵。
num = 0  # 计算下有多少数没有算进去。
tem_word = 0

embedding_matrix = np.zeros((len(word_to_int) + 1, 100))  # 因为预训练的glove就是100
for word, i in word_to_int.items():
    embedding_vector = embeddings_index.get(word)
    if embedding_vector is not None:
        # words not found in embedding index will be all-zeros.
        embedding_matrix[i] = embedding_vector
    else:
        num = num + 1
        if num == 1:
            temp_word = i
# 要放在weight 中的。
print(embedding_matrix[tem_word])
print("未统计的数字：", num)
print("glove length", len(embeddings_index))
print("word length", len(word_to_int))

# split_frac = 0.8
# split_idx = int(len(features) * 0.8)
# train_x, remaining_x = features[:split_idx], features[split_idx:]
# train_y, remaining_y = encoded_labels[:split_idx], encoded_labels[split_idx:]
# test_idx = int(len(remaining_x) * 0.5)
# val_x, test_x = remaining_x[:test_idx], remaining_x[test_idx:]
# val_y, test_y = remaining_y[:test_idx], remaining_y[test_idx:]

split_frac = 0.8
split_idx = int(len(features) * 0.8)
train_x, remaining_x = features[:split_idx], features[split_idx:]
train_y, remaining_y = encoded_labels[:split_idx], encoded_labels[split_idx:]
test_idx = int(len(remaining_x) * 0.5)
val_x, test_x = remaining_x[:test_idx], remaining_x[test_idx:]
val_y, test_y = remaining_y[:test_idx], remaining_y[test_idx:]

train_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))
valid_data = TensorDataset(torch.from_numpy(val_x), torch.from_numpy(val_y))
test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))
batch_size = 54
# make sure the SHUFFLE your training data
train_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)
valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size)
test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size)

train_on_gpu = torch.cuda.is_available()
if train_on_gpu:
    print('Training on GPU.')
else:
    print('No GPU available, training on CPU.')

class SentimentRNN(nn.Module):
    """
    The RNN model that will be used to perform Sentiment analysis.
    """

    def __init__(self, vocab_size, output_size, embedding_dim, hidden_dim, n_layers, drop_prob=0.5):
        """
        Initialize the model by setting up the layers.
        """
        super(SentimentRNN, self).__init__()

        self.output_size = output_size
        self.n_layers = n_layers
        self.hidden_dim = hidden_dim

        # embedding and LSTM layers
        self.embedding = nn.Embedding(vocab_size, embedding_dim)
        self.embedding.weight.requires_grad = False
        self.lstm = nn.LSTM(embedding_dim, hidden_dim, n_layers,
                            dropout=drop_prob, batch_first=True)

        # dropout layer
        self.dropout = nn.Dropout(0.3)

        # linear
        self.fc = nn.Linear(hidden_dim, output_size)

    def forward(self, x, hidden):
        """
        Perform a forward pass of our model on some input and hidden state.
        """
        batch_size = x.size(0)  # input.shape = (54, 48)

        # embeddings and lstm_out
        x = x.long()  # 走的坑, 这里新增加了 .long()
        embeds = self.embedding(x)  # input 应该是： LongTensor of arbitrary shape containing the indices to extract
        # embeds.shape = (54, 48, 400)
        lstm_out, hidden = self.lstm(embeds, hidden)  # lstm_out.shape = (54, 48, 256)

        # transform lstm output to input size of linear layers
        lstm_out = lstm_out.transpose(0, 1)  # 48, 54, 256
        lstm_out = lstm_out[-1]  # lstm_out.shape = 54, 256

        out = self.dropout(lstm_out)
        out = self.fc(out)

        return out, hidden

    def init_hidden(self, batch_size):
        """ Initializes hidden state """
        # Create two new tensors with sizes n_layers x batch_size x hidden_dim,
        # initialized to zero, for hidden state and cell state of LSTM
        weight = next(self.parameters()).data  # 这里的shape是[16210, 400]  也是embedding的两个参数

        if train_on_gpu:
            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda(),
                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_().cuda())
            # new 和 view、reshape的区别于何处， zero_ 还是初始化成了 0
        else:
            hidden = (weight.new(self.n_layers, batch_size, self.hidden_dim).zero_(),
                      weight.new(self.n_layers, batch_size, self.hidden_dim).zero_())

        return hidden

# Instantiate the model w/ hyperparams
# vocab_size = len(word_to_int) + 1  # +1 for the 0 padding
vocab_size = len(word_set) + 1
output_size = 5
embedding_dim = 100
hidden_dim = 256
n_layers = 2

net = SentimentRNN(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)
net.embedding.weight.data.copy_(torch.from_numpy(embedding_matrix))

lr = 0.003

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(net.parameters(), lr=lr)

# training params
epochs = 3  # 3-4 is approx where I noticed the validation loss stop decreasing

counter = 0
print_every = 100
clip = 5  # gradient clipping

# move model to GPU, if available
if train_on_gpu:
    net.cuda()

net.train()
# train for some number of epochs
for e in range(epochs):
    # initialize hidden state
    h = net.init_hidden(batch_size)

    # batch loop
    for inputs, labels in train_loader:
        # inputs.shape = (54, 48)    # batch_size 就是54 . 48是 token的长度  # labels.shpe = 54, 每一项都是分类.
        counter += 1

        if train_on_gpu:
            inputs, labels = inputs.cuda(), labels.cuda()

        # Creating new variables for the hidden state, otherwise
        # we'd backprop through the entire training history
        h = tuple([each.data for each in h])

        # zero accumulated gradients
        net.zero_grad()

        # get the output from the model
        output, h = net(inputs, h)  # output.shape = [54, 5]
        # calculate the loss and perform backprop
        loss = criterion(output.squeeze(), labels.long())  # 走的坑, 这里新增加了 .squeeze() ; .long()
        loss.backward()
        # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.
        nn.utils.clip_grad_norm_(net.parameters(), clip)
        optimizer.step()

        # loss stats
        if counter % print_every == 0:
            # Get validation loss
            val_h = net.init_hidden(batch_size)
            val_losses = []
            net.eval()
            for inputs, labels in valid_loader:

                # Creating new variables for the hidden state, otherwise
                # we'd backprop through the entire training history
                val_h = tuple([each.data for each in val_h])

                if train_on_gpu:
                    inputs, labels = inputs.cuda(), labels.cuda()

                output, val_h = net(inputs, val_h)
                val_loss = criterion(output.squeeze(), labels.long())

                val_losses.append(val_loss.item())

            net.train()
            print("Epoch: {}/{}...".format(e + 1, epochs),
                  "Step: {}...".format(counter),
                  "Loss: {:.6f}...".format(loss.item()),
                  "Val Loss: {:.6f}".format(np.mean(val_losses)))

test_losses = []  # track loss
num_correct = 0
# init hidden state
h = net.init_hidden(batch_size)
net.eval()
# iterate over test data
for inputs, labels in test_loader:
    # Creating new variables for the hidden state, otherwise
    # we'd backprop through the entire training history
    h = tuple([each.data for each in h])

    if train_on_gpu:
        inputs, labels = inputs.cuda(), labels.cuda()

    # get predicted outputs
    output, h = net(inputs, h)
    # calculate loss
    test_loss = criterion(output.squeeze(), labels.long())
    test_losses.append(test_loss.item())

    # convert output probabilities to predicted class
    _, pred = torch.max(output, 1)

    # compare predictions to true label
    correct_tensor = pred.eq(labels.view_as(pred))
    correct = np.squeeze(correct_tensor.numpy()) if not train_on_gpu else np.squeeze(correct_tensor.cpu().numpy())
    num_correct += np.sum(correct)
    # -- stats! -- ##
    # avg test loss
    print("Test loss: {:.3f}".format(np.mean(test_losses)))
    # accuracy over all test data
test_acc = num_correct / len(test_loader.dataset)
print("Test accuracy: {:.3f}".format(test_acc))
print(net.embedding.weight[tem_word])


title: How to get quick documentation working with PyCharm and Pytorch

body:
I'm running PyCharm on Windows 10, and installed PyTorch following the getting started guide. Where I used Chocolatey and Anaconda to set up everything.

I can run the PyTorch tutorials from inside the PyCharm IDE without any problems. So I feel like I have a proper set up, but there aren't any intellisense documentations for any of the PyTorch APIs.

For example;

import torch

x = torch.randn(128, 20)

If I mouse over randn and press CTRL+Q then PyCharm shows me a popup of the function definition without any documentation.

I'm expecting to see Python comments from the API documentation for that function:

https://pytorch.org/docs/stable/torch.html?highlight=randn#torch.randn

I'm a new beginner with Pytorch and Python, but this is something that I often have access to from inside the IDE with many other languages and libraries. So I feel like this should be possible to get working, but I can't seem to find any instructions on how to fix this.


title: PyTorch Gradient not Working after adding to("cuda:0")

body:
I'm going to give a context-free set of code.  The code works before adding "to(device)".
def get_input_layer(word_idx) :
x = torch.zeros(vocabulary_size).float().to(device)
x[word_idx] = 1.0
return x

embedding_dims = 5
device = torch.device("cuda:0")
W1 = Variable(torch.randn(embedding_dims, vocabulary_size).float(), requires_grad=True).to(device)
W2 = Variable(torch.randn(vocabulary_size, embedding_dims).float(), requires_grad=True).to(device)
num_epochs = 100
learning_rate = 0.001

x = Variable(get_input_layer(data)).float().to(device)
y_true = Variable(torch.from_numpy(np.array([target])).long()).to(device)
z1 = torch.matmul(W1, x).to(device)
z2 = torch.matmul(W2, z1).to(device)

log_softmax = F.log_softmax(z2, dim=0).to(device)

loss = F.nll_loss(log_softmax.view(1,-1), y_true).to(device)
loss_val += loss.data
loss.backward().to(device)

## Optimize values. This is done by hand rather than using the optimizer function
W1.data -= learning_rate * W1.grad.data
W2.data -= learning_rate * W2.grad.data

I get
Traceback (most recent call last):
File "<input>", line 1, in <module>
AttributeError: 'NoneType' object has no attribute 'data'

Which triggers specifically on the line
W1.data -= learning_rate * W1.grad.data

Checking, this is confirmed because W1.grad is None for some reason.
And this loops after clearing the gradients. This works just fine if I remove all of the .to(device).  What is it that I'm doing wrong in trying to run this on my GPU?
Thank you for your time.


title: How to prune certain weights (rather than freeze a layer) with Keras?

body:
I am trying to prune some connections in a deep model with Keras. These connections are in different locations and different fully-connection layers, so setting layer.trainable=False doesn't work for me.

So how to prune these connections (freeze their weights to 0 during training) in Keras? Do I need to writing my own layers and how to?
Does the other tools like PyTorch support this prune (or freeze) operation easily (no need to write my own layers)?


title: How to use gensim with pytorch to create an intent classifier (With LSTM NN)?

body:
The problem to solve: Given a sentence, return the intent behind it (Think chatbot)

Reduced example dataset (Intent on the left of dict):

data_raw    = {"mk_reservation" : ["i want to make a reservation",
                                   "book a table for me"],
               "show_menu"      : ["what's the daily menu",
                                   "do you serve pizza"],
               "payment_method" : ["how can i pay",
                                   "can i use cash"],
               "schedule_info"  : ["when do you open",
                                   "at what time do you close"]}

I have stripped down the sentences with spaCy, and tokenized each word by using the word2vec  algorithm provided by the gensim library.

This is what resulted from the use of word2vec model GoogleNews-vectors-negative300.bin:

[[[ 5.99331968e-02  6.50703311e-02  5.03010787e-02 ... -8.00536275e-02
    1.94782894e-02 -1.83010306e-02]
  [-2.14406010e-02 -1.00447744e-01  6.13847338e-02 ... -6.72588721e-02
    3.03986594e-02 -4.14126664e-02]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  ...
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]]

 [[ 4.48647663e-02 -1.03907576e-02 -1.78682189e-02 ...  3.84555124e-02
   -2.29179319e-02 -2.05144612e-03]
  [-5.39291985e-02 -9.88398306e-03  4.39085700e-02 ... -3.55276838e-02
   -3.66208404e-02 -4.57760505e-03]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  ...
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]
  [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 ...  0.00000000e+00
    0.00000000e+00  0.00000000e+00]]]

This is a List of sentences, and each sentence is a list of words ([sentences[sentence[word]]])
Each sentence (list) must be of size 10 words (I am padding the remaining with zeroes)
Each word (list) has 300 elements (word2vec dimensions)

By following some tutorials i transformed this to a TensorDataset.

At this moment, i am very confused on how to use the word2vec and probably i have just been wasting time, as of now i believe the embeddings layer from an LSTM configuration should be composed by importing the word2vec model weights using:

import gensim
model = gensim.models.KeyedVectors.load_word2vec_format('path/to/file')
weights = torch.FloatTensor(model.vectors)
word_embeddings = nn.Embedding.from_pretrained(weights)

This is not enough as pytorch is saying it does not accept embeddings where indices are not INT type  .

EDIT: I found out that importing the weight matrix from gensim word2vec is not straightforward, one has to import the word_index table as well.

As soon as i fix this issue i'll post it here.


title: Difficulties calculating mean square error between 2 tensors

body:
I'm trying to build a loss function which will calculate the mean squared error of 2 tenors of the same size.
In other words I need a function that calcualtes the difference of every 2 cells (with the same row and column) cell on matrix A and matrix B, squares it and calculate the mean of the differences.
As far as I understand nn.MSELoss should do exactly that.
When I pass the 2 tensors to nn.MSELoss I'm getting the following error message:
RuntimeError: Boolean value of Tensor with more than one value is ambiguous

Here's my code
nn.MSELoss(stack_3[0,:],stack_7[0,:])

The tensors are floats of same shape.
stack_3.shape, stack_7.shape
(torch.Size([6131, 784]), torch.Size([6131, 784]))


title: What is the difference between a .ckpt and a .pth file in Pytorch?

body:
I am following a code from GitHub that uses Pytorch.
The model is saved using :
model.save(ARGS.working_dir + '/model_%d.ckpt' % (epoch+1)).

What is the difference between using .pth and .ckpt in Pytorch?


title: How can I invert a MelSpectrogram with torchaudio and get an audio waveform?

body:
I have a MelSpectrogram generated from:
eval_seq_specgram = torchaudio.transforms.MelSpectrogram(sample_rate=sample_rate, n_fft=256)(eval_audio_data).transpose(1, 2)

So eval_seq_specgram now has a size of torch.Size([1, 128, 499]), where 499 is the number of timesteps and 128 is the n_mels.
I'm trying to invert it, so I'm trying to use GriffinLim, but before doing that, I think I need to invert the melscale, so I have:
inverse_mel_pred = torchaudio.transforms.InverseMelScale(sample_rate=sample_rate, n_stft=256)(eval_seq_specgram)

inverse_mel_pred has a size of torch.Size([1, 256, 499])
Then I'm trying to use GriffinLim:
pred_audio = torchaudio.transforms.GriffinLim(n_fft=256)(inverse_mel_pred)

but I get an error:
Traceback (most recent call last):
  File "evaluate_spect.py", line 63, in <module>
    main()
  File "evaluate_spect.py", line 51, in main
    pred_audio = torchaudio.transforms.GriffinLim(n_fft=256)(inverse_mel_pred)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torchaudio/transforms.py", line 169, in forward
    return F.griffinlim(specgram, self.window, self.n_fft, self.hop_length, self.win_length, self.power,
  File "/home/shamoon/.local/share/virtualenvs/speech-reconstruction-7HMT9fTW/lib/python3.8/site-packages/torchaudio/functional.py", line 179, in griffinlim
    inverse = torch.istft(specgram * angles,
RuntimeError: The size of tensor a (256) must match the size of tensor b (129) at non-singleton dimension 1

Not sure what I'm doing wrong or how to resolve this.


title: Dataset Labeled as not found or Corrupt, but the dataset is not corrupt

body:
I have been trying to use this Github (https://github.com/AntixK/PyTorch-VAE) and call the CelebA dataset using the config file listed.  Specifically under the vae.yaml I have placed the path of the unzipped file where I have downloaded the celeba dataset (https://www.kaggle.com/jessicali9530/celeba-dataset) on my computer. And every time I run the program, I keep getting these errors:
File "/usr/local/lib/python3.6/dist-packages/torchvision/datasets/celeba.py", line 67, in init
' You can use download=True to download it')
RuntimeError: Dataset not found or corrupted. You can use download=True to download it
AttributeError: 'VAEXperiment' object has no attribute '_lazy_train_dataloader'
I have tried to download the dataset, but nothing changes. So I have no idea why the program is not running.
The run.py calls the experiment.py which uses this dataloader to retrieve the information:
def train_dataloader(self):
        transform = self.data_transforms()

        if self.params['dataset'] == 'celeba':
            dataset = CelebA(root = self.params['data_path'],
                             split = "train",
                             transform=transform,
                             download=False)
        else:
            raise ValueError('Undefined dataset type')

        self.num_train_imgs = len(dataset)
        return DataLoader(dataset,
                          batch_size= self.params['batch_size'],
                          shuffle = True,
                          drop_last=True)

The config file grabs the information passed on the root. So what I did was upload a few files to google colab (some .jpg files) and when I run the command stated in the GItHub, python run.py -c config/vae.yaml, it states that the dataset is not found or is corrupt. I have tried this on my linux machine and the same error occurs, even when I used the downloaded and unzip link. I have gone further to attempt to change the self.params['data_path'] to the actual path and that still does not work. Any ideas what I can do?


title: Adding matrix rows to columns in numpy

body:
Say I have two 3D matrices/tensors with dimensions:

[10, 3, 1000]
[10, 4, 1000]

How do I add each combination of the third dimensions of each vector together such that to get a dimension of:

[10, 3, 4, 1000]

So each row if you will, in the second x third dimension for each of the vectors adds to the other one in every combination. Sorry if this is not clear I'm having a hard time articulating this...

Is there some kind of clever way to do this with numpy or pytorch (perfectly happy with a numpy solution, though I'm trying to use this in a pytorch context so a torch tensor manipulation would be even better) that doesn't involve me writing a bunch of nested for loops?

Nested loops example:

x = np.random.randint(50, size=(32, 16, 512))
y = np.random.randint(50, size=(32, 21, 512))
scores = np.zeros(shape=(x.shape[0], x.shape[1], y.shape[1], 512))

for b in range(x.shape[0]):
    for i in range(x.shape[1]):
        for j in range(y.shape[1]):

            scores[b, i, j, :] = y[b, j, :] + x[b, i, :]


title: Why my Pytorch Doesn't raise overflow/underflow warning during the running

body:
Consider

  torch.Tensor([torch.exp(torch.Tensor([10000]))])

it gives

  tensor([inf.])

without raising any warning or error. During the training, many time I end up with nan value; and have to debug all the way to figure when did the numerical issue occurs.

Is there anyway to fix it

I'm using pytorch 0.4.0

(Both jupyter notebook and pycharm doesn't give any numerical warning)


title: Take the mean of multi torch.Tensor list

body:
The project needs to iterate over the data multiple times and then average it

gradslist = []
for data_epoch in interval_steps:
    output = cnn.forward(b_x)[0]
    loss = nn.CrossEntropyLoss()(output, b_y)
    cnn.zero_grad()
    loss.backward()
    w_c1 = cnn.conv1[0].weight.grad
    b_c1 = cnn.conv1[0].bias.grad
    w_c2 = cnn.conv2[0].weight.grad
    b_c2 = cnn.conv2[0].bias.grad
    w_o = cnn.out.weight.grad
    b_o = cnn.out.bias.grad
    grads = [w_c1,b_c1,w_c2,b_c2,w_o,b_o]
    gradslist.append(grads)

w_c1 type= torch.Tensor  shape=  torch.Size([64, 1, 5, 5])
b_c1 type= torch.Tensor  shape=  torch.Size([64])
w_c2 type= torch.Tensor  shape=  torch.Size([144, 64, 5, 5])
b_c2 type= torch.Tensor  shape=  torch.Size([144])
w_o type= torch.Tensor  shape=  torch.Size([10, 2304])
b_o type= torch.Tensor  shape=  torch.Size([10])

I want the average

avggrads = [avg_w_c1, avg_b_c1, avg_w_c2, avg_b_c2, avg_w_o, avg_w_0]

The result of avggrads is not a list of values, but a torch.tensor list, such as avg_w_c1.shape= torch.size ([64, 1, 5, 5]), not a value. it is the average of the corresponding position (Matrix averaging)

Please advise how to average multiple grads in gradslist.


title: Pytorch: Index with tensor along multiple axes OR scatter to more than one index at once

body:
I am trying to update very specific indices of a multidimensional tensor in Pytorch, and I am not sure how to access the correct indices. I can do this in a very straightforward way in Numpy:

import numpy as np
#set up the array containing the data
data = 100*np.ones((10,10,2))
data[5:,:,:] = 0
#select the data points that I want to update
idxs = np.nonzero(data.sum(2))
#generate the updates that I am going to do
updates = np.random.randint(5,size=(idxs[0].shape[0],2))
#update the data
data[idxs[0],idxs[1],:] = updates

I need to implement this in Pytorch but I am not sure how to do this. It seems like I need the scatter function but that only works along a single dimension instead of the multiple dimensions that I need. How can I do this?


title: "ERROR: Invalid requirement: pip3 install torch==1.5.0+cpu torchvision==0.6.0+cpu -f", while deployment

body:
I'm deploying a application on Heroku and that application needs Torch module to install on the server and I'm unable to push the app using git because of this error....
I tried many methods but unable to push it...
Tell me what should I write in the requirements.txt in order to install torch


title: Convolution layer padding difference between pytorch and tensorflow

body:
torch.nn.Conv2d(7, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3))

In the above line of code for convolution in pytorch what is happening with the padding = (3,3) parameter. How can we implement the same padding in tensorflow.
tf.keras.layers.Conv2D(7, 64, stride = (2,2), padding="same padding as pytorch")

How can we implement this in tensorflow ?


title: Using multiple loss functions in pytorch

body:
I was working on an image restoration task and I considered multiple loss functions . My plan was to consider 3 routes:
1: Use multiple losses for monitoring but use only a few for training itself
2: Out of those loss functions that are used for training, I needed to give each a weight - currently I am specifying the weight. I would like to make that parameter adaptive.
3: If in between training - if I observe a saturation I would like to change the loss function . or its components. Currently I considered re-training a network (if in the first training the model saturated) such that it trained with a particular loss function for the the first say M epochs after which I change the loss.

Except the last case I developed a code which computes these losses but I am not sure whether it will work. - ie whether it will backpropagate? (code given below)

is it possible to give the weights adaptively when using combination of loss functions - ie can we train the network so that these weights are also learned ?

can this implementation be used for the above mentioned case 3 of changing loss functions

Sorry if anything given here is not clear or wrong. Please let me know if I have to improve the question. (I am kinda new to PyTorch)
criterion = _criterion
#--training
prediction = model(input)
loss = criterion(prediction, target)
loss.backward()

class _criterion(nn.Module):

    def __init__(self, model_type="CNN"):

        super(_criterion).__init__()

        self.model_type = model_type

    def forward(self, pred, ref):

        loss_1 = lambda x,y : nn.MSELoss(size_average=False)(x,y)

        loss_2 = lambda x,y : nn.L1Loss(size_average=False)(x,y)

        loss_3 = lambda x,y : nn.SmoothL1Loss(size_average=False)(x,y)

        loss_4 = lambda x,y : L1_Charbonnier_loss_()(x,y)     #user-defined

        if opt.loss_function_order == 1:

            loss_function_1 = get_loss_function(opt.loss_function_1)

            loss = lambda x,y: 1*loss_function_1(x,y)

        elif opt.loss_function_order == 2:

            loss_function_1 = get_loss_function(opt.loss_function_1)

            loss_function_2 = get_loss_function(opt.loss_function_2)

            weight_1 = opt.loss_function_1_weight

            weight_2 = opt.loss_function_2_weight

            loss = lambda x,y: weight_1*loss_function_1(x,y) + weight_2*loss_function_2(x,y)

        elif opt.loss_function_order == 3:

            loss_function_1 = get_loss_function(opt.loss_function_1)

            loss_function_2 = get_loss_function(opt.loss_function_2)

            loss_function_3 = get_loss_function(opt.loss_function_3)

            weight_1 = opt.loss_function_1_weight

            weight_2 = opt.loss_function_2_weight

            weight_3 = opt.loss_function_3_weight

            loss = lambda x,y: weight_1*loss_function_1(x,y) + weight_2*loss_function_2(x,y) +  weight_3*loss_function_3(x,y)

        elif opt.loss_function_order == 4:

            loss_function_1 = get_loss_function(opt.loss_function_1)

            loss_function_2 = get_loss_function(opt.loss_function_2)

            loss_function_3 = get_loss_function(opt.loss_function_3)

            loss_function_4 = get_loss_function(opt.loss_function_4)

            weight_1 = opt.loss_function_1_weight

            weight_2 = opt.loss_function_2_weight

            weight_3 = opt.loss_function_3_weight

            weight_4 = opt.loss_function_4_weight

            loss = lambda x,y: weight_1*loss_function_1(x,y) + weight_2*loss_function_2(x,y) +  weight_3*loss_function_3(x,y)  +  weight_4*loss_function_4(x,y)

        else:

            raise Exception("_criterion : unable to interpret loss_function_order")

        return loss(ref,pred), loss_1(ref,pred), loss_2(ref,pred), loss_3(ref,pred), loss_4(ref,pred)

def get_loss_function(loss):

    if loss == "MSE":

        criterion = nn.MSELoss(size_average=False)

    elif loss == "MAE":

        criterion = nn.L1Loss(size_average=False)

    elif loss == "Smooth-L1":

        criterion = nn.SmoothL1Loss(size_average=False)

    elif loss == "Charbonnier":

        criterion = L1_Charbonnier_loss_()
    else:

        raise Exception("not implemented")
    return criterion

class L1_Charbonnier_loss_(nn.Module):

    def __init__(self):

        super(L1_Charbonnier_loss_, self).__init__()

        self.eps = 1e-6

    def forward(self, X, Y):

        diff = torch.add(X, -Y)

        error = self.eps*((torch.sqrt(1+((diff * diff)/self.eps)))-1)

        loss = torch.sum(error)

        return loss


title: why pytorch linear model isn't using sigmoid function

body:
I'm expecting that the linear model in pytorch yields sigmoid(WX+b). But I see it is only returning Wx+b. Why is this the case?
In Udacity "Intro to deep learning with pytorch" -> Lesson 2: Introduction to Neural Networks, They say that the output is sigmoid:
̂ =(11+22+)

From the below code I was expecting y cap to be 0.38391371665752183, but it is just the value of WX+b, which I confirmed the output. Why is this discrepancy?
import torch
from torch import nn
import numpy as np

torch.manual_seed(0)
model = nn.Linear(2,1)
w1 = model.weight.detach().numpy()
b1 = model.bias.detach().numpy()
print (f'model.weight = {w1}, model.bias={b1}')
x = torch.tensor([[0.2877, 0.2914]])
print(f'model predicted {model(x)}')
z = x.numpy()[0][0] * w1[0][0] + x.numpy()[0][1] * w1 [0][1] + b1[0]
print(f'manual multiplication yielded {z}')
ycap = 1/(1+ np.exp(-z))
print(f'y cap is {ycap}')

Output:
model.weight = [[-0.00529398  0.3793229 ]], model.bias=[-0.58198076]
model predicted tensor([[-0.4730]], grad_fn=<AddmmBackward>)
manual multiplication yielded -0.4729691743850708
y cap is 0.38391371665752183


title: How to combine multiple models together?

body:
I am trying to "parallelize" Neural Network models to speed up training. One idea I had was to run two models on two computers and combine the results somehow.

Is this possible? If not, what are the options to parallelize model training on two computers?

I am open to use any neural network framework.


title: PyTorch - How to set Activation Rules of neurons to increase efficiency of Neural Network?

body:
I'm trying to make a Back Propagation Neural Network with PyTorch. I can successfully execute and test its accuracy, but it doesn't work very efficiently. Now, I'm supposed to increase its efficiency by setting different activation rules for neurons, so that those neurons that don't contribute to the final output get excluded (pruned) from the computations, thereby increasing the time and accuracy.

My code looks like this (extracted snippets) -

# Hyper Parameters
input_size = 20
hidden_size = 50
num_classes =130
num_epochs = 500
batch_size = 5
learning_rate = 0.1

# normalise input data
for column in data:
# the last column is target
if column != data.shape[1] - 1:
    data[column] = data.loc[:, [column]].apply(lambda x: (x - x.mean()) / x.std())

# randomly split data into training set (80%) and testing set (20%)
msk = np.random.rand(len(data)) < 0.8
train_data = data[msk]
test_data = data[~msk]

# define train dataset and a data loader
train_dataset = DataFrameDataset(df=train_data)
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)

# Neural Network
class Net(nn.Module):
    def __init__(self, input_size, hidden_size, num_classes):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.sigmoid = nn.Sigmoid()
        self.fc2 = nn.Linear(hidden_size, num_classes)

    def forward(self, x):
        out = self.fc1(x)
        out = self.sigmoid(out)
        out = self.fc2(out)
        return out
net = Net(input_size, hidden_size, num_classes)

# train the model by batch
for epoch in range(num_epochs):
    for step, (batch_x, batch_y) in enumerate(train_loader):
        # convert torch tensor to Variable
        X = Variable(batch_x)
        Y = Variable(batch_y.long())

        # Forward + Backward + Optimize
        optimizer.zero_grad()  # zero the gradient buffer
        outputs = net(X)
        loss = criterion(outputs, Y)
        all_losses.append(loss.data[0])
        loss.backward()
        optimizer.step()

        if epoch % 50 == 0:
            _, predicted = torch.max(outputs, 1)
            # calculate and print accuracy
            total = predicted.size(0)
            correct = predicted.data.numpy() == Y.data.numpy()

            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f, Accuracy: %.2f %%' % (epoch + 1, num_epochs, step + 1, len(train_data) // batch_size + 1, loss.data[0], 100 * sum(correct)/total))

Can someone tell me how to do that in PyTorch as I'm very new to PyTorch.


title: (Single Layer) Perceptron in PyTorch, bad convergence

body:
I'm trying to develop a simple single layer perceptron with PyTorch (v0.4.0) to classify AND boolean operation.
I want to develop it by using autograd to calculate gradient of weights and bias and then update them in a SGD manner.

The code is very simple and is the following:

# AND points and labels
data = torch.tensor([
    [0, 0],
    [0, 1],
    [1, 0],
    [1, 1]
    ], dtype=torch.float32)
labels = torch.tensor([0,0,0,1], dtype=torch.float32)

weights = torch.zeros(2, dtype=torch.float32, requires_grad=True)
bias = torch.zeros(1, requires_grad=True)
losses = []
epochs = 100
eta = 0.01
for epoch in range(epochs):
    total_loss = 0
    for idx in range(4):
        # take current input
        X = data[idx,:]
        y = labels[idx]

        # compute output and loss
        out = torch.add(torch.dot(weights, X), bias)
        loss = (out-y).pow(2)
        total_loss += loss.item()
        # backpropagation
        loss.backward()

        # compute accuracy and update parameters
        with torch.no_grad():
            weights -= eta * weights.grad
            bias -= eta * bias.grad
            # reset gradient to zero
            weights.grad.zero_()
            bias.grad.zero_()
    losses.append(total_loss)

The model converges, as you can see from the learning curve

but the resulting plane is:

with 50% of accuracy.

I tried with different inital parameters and also by using the SGD optimizer from PyTorch but nothing changed.
I know that MSE is a regression loss but I don't think the problem is there.

Any ideas?

Update
The plane is computed with these 2 lines of code

xr = np.linspace(0, 1, 10)
yr = (-1 / weights[1].item()) * (weights[0].item() * xr  + bias.item())
plt.plot(xr,yr,'-')


title: Implementing Curriculum Dropout in Pytorch

body:
Is there anyone who can help me implement Curriculum Dropout by Pytorch. Thanks in advance, and any kind of help will be appreciable.

I want to do some experiments of Curriculum Dropout in Pytorch. Curriculum Dropout tries to use a time scheduling for adjusting the dropout rate in the neural networks. The related paper can be downloaded from here.
The source code in python can be found here


title: Tensorflow no_grad concept

body:
I know with pytorch you can turn off training by calling eval() on your model.
Also you can set requires_grad=False.

How can you ensure that a TensorFlow element is not modified during training?


title: windows spyder invalid syntax error while running py file

body:
I am trying to run the last example from the page. I have cloned the repository in the directory C:/Users/nn/Desktop/BERT/transformers-master. I am on windows machine and using spyder IDE. Why i do get below error and how could i resolve it? How do i input the initial part of the poem?

import os

os.chdir('C:/Users/nn/Desktop/BERT/transformers-master/examples')
os.listdir()# It shows run_generation.py file

python run_generation.py \
    --model_type=gpt2 \
    --length=100 \
    --model_name_or_path=gpt2 \

python run_generation.py \
    --model_type=gpt2 \
    --length=100 \
    --model_name_or_path=gpt2 \
  File "<ipython-input-10-501d266b0e64>", line 1
    python run_generation.py \
                        ^
SyntaxError: invalid syntax

I went to command prompt and tried below

cd C:/Users/nn/Desktop/BERT/transformers-master/examples
python3 run_generation.py \--model_type=gpt2 \--length=100 \--model_name_or_path=gpt2 \--promt="Hello world"

nothing happens :(

when i try the same with python command i get an error as below :(

python run_generation.py \--model_type=gpt2 \--length=100 \--model_name_or_path=gpt2 \--promt="Hello world"
2019-12-04 11:23:36.345648: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2019-12-04 11:23:36.352875: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
usage: run_generation.py [-h] --model_type MODEL_TYPE --model_name_or_path
                         MODEL_NAME_OR_PATH [--prompt PROMPT]
                         [--padding_text PADDING_TEXT] [--xlm_lang XLM_LANG]
                         [--length LENGTH] [--num_samples NUM_SAMPLES]
                         [--temperature TEMPERATURE]
                         [--repetition_penalty REPETITION_PENALTY]
                         [--top_k TOP_K] [--top_p TOP_P] [--no_cuda]
                         [--seed SEED] [--stop_token STOP_TOKEN]
run_generation.py: error: the following arguments are required: --model_type, --model_name_or_path

####update 2 ------------------

I followed suggestions in the comments and it worked. It seems that the code downloads 3 files.

Can i copy those files manually so that I dont have to rely on downloading them every time in a temp folder?
Where should i store those files? which folder location? would it be C:\Users\nnn\Desktop\BERT\transformers-master\examples - same as run_generation.py file?

abc

C:\Users\nnn\Desktop\BERT\transformers-master\examples>python run_generation.py --model_type=gpt2 --length=100 --model_name_or_path=gpt2 --prompt="My job is"

2019-12-12 11:11:57.740810: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_100.dll'; dlerror: cudart64_100.dll not found
2019-12-12 11:11:57.748330: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
12/12/2019 11:12:01 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json not found in cache or force_download set to True, downloading to C:\Users\nnn\AppData\Local\Temp\tmpt_29gyqi
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1042301/1042301 [00:00<00:00, 2275416.04B/s]
12/12/2019 11:12:02 - INFO - transformers.file_utils -   copying C:\Users\nnn\AppData\Local\Temp\tmpt_29gyqi to cache at C:\Users\nnn\.cache\torch\transformers\f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
12/12/2019 11:12:02 - INFO - transformers.file_utils -   creating metadata file for C:\Users\nnn\.cache\torch\transformers\f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
12/12/2019 11:12:02 - INFO - transformers.file_utils -   removing temp file C:\Users\nnn\AppData\Local\Temp\tmpt_29gyqi
12/12/2019 11:12:03 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt not found in cache or force_download set to True, downloading to C:\Users\nnn\AppData\Local\Temp\tmpj1_y4sn8
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 456318/456318 [00:00<00:00, 1456594.78B/s]
12/12/2019 11:12:03 - INFO - transformers.file_utils -   copying C:\Users\nnn\AppData\Local\Temp\tmpj1_y4sn8 to cache at C:\Users\nnn\.cache\torch\transformers\d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
12/12/2019 11:12:03 - INFO - transformers.file_utils -   creating metadata file for C:\Users\nnn\.cache\torch\transformers\d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
12/12/2019 11:12:03 - INFO - transformers.file_utils -   removing temp file C:\Users\nnn\AppData\Local\Temp\tmpj1_y4sn8
12/12/2019 11:12:03 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json from cache at C:\Users\nnn\.cache\torch\transformers\f2808208f9bec2320371a9f5f891c184ae0b674ef866b79c58177067d15732dd.1512018be4ba4e8726e41b9145129dc30651ea4fec86aa61f4b9f40bf94eac71
12/12/2019 11:12:03 - INFO - transformers.tokenization_utils -   loading file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt from cache at C:\Users\nnn\.cache\torch\transformers\d629f792e430b3c76a1291bb2766b0a047e36fae0588f9dbc1ae51decdff691b.70bec105b4158ed9a1747fea67a43f5dee97855c64d62b6ec3742f4cfdb5feda
12/12/2019 11:12:04 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json not found in cache or force_download set to True, downloading to C:\Users\nnn\AppData\Local\Temp\tmpyxywrts1
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 176/176 [00:00<00:00, 17738.31B/s]
12/12/2019 11:12:04 - INFO - transformers.file_utils -   copying C:\Users\nnn\AppData\Local\Temp\tmpyxywrts1 to cache at C:\Users\nnn\.cache\torch\transformers\4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
12/12/2019 11:12:04 - INFO - transformers.file_utils -   creating metadata file for C:\Users\nnn\.cache\torch\transformers\4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
12/12/2019 11:12:04 - INFO - transformers.file_utils -   removing temp file C:\Users\nnn\AppData\Local\Temp\tmpyxywrts1
12/12/2019 11:12:04 - INFO - transformers.configuration_utils -   loading configuration file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json from cache at C:\Users\nnn\.cache\torch\transformers\4be02c5697d91738003fb1685c9872f284166aa32e061576bbe6aaeb95649fcf.085d5f6a8e7812ea05ff0e6ed0645ab2e75d80387ad55c1ad9806ee70d272f80
12/12/2019 11:12:04 - INFO - transformers.configuration_utils -   Model config {
  "attn_pdrop": 0.1,
  "embd_pdrop": 0.1,
  "finetuning_task": null,
  "initializer_range": 0.02,
  "layer_norm_epsilon": 1e-05,
  "n_ctx": 1024,
  "n_embd": 768,
  "n_head": 12,
  "n_layer": 12,
  "n_positions": 1024,
  "num_labels": 1,
  "output_attentions": false,
  "output_hidden_states": false,
  "output_past": true,
  "pruned_heads": {},
  "resid_pdrop": 0.1,
  "summary_activation": null,
  "summary_first_dropout": 0.1,
  "summary_proj_to_labels": true,
  "summary_type": "cls_index",
  "summary_use_proj": true,
  "torchscript": false,
  "use_bfloat16": false,
  "vocab_size": 50257
}

12/12/2019 11:12:04 - INFO - transformers.file_utils -   https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin not found in cache or force_download set to True, downloading to C:\Users\nnn\AppData\Local\Temp\tmpn8i9o_tm
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 548118077/548118077 [01:12<00:00, 7544610.26B/s]
12/12/2019 11:13:18 - INFO - transformers.file_utils -   copying C:\Users\nnn\AppData\Local\Temp\tmpn8i9o_tm to cache at C:\Users\nnn\.cache\torch\transformers\4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
12/12/2019 11:13:24 - INFO - transformers.file_utils -   creating metadata file for C:\Users\nnn\.cache\torch\transformers\4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
12/12/2019 11:13:24 - INFO - transformers.file_utils -   removing temp file C:\Users\nnn\AppData\Local\Temp\tmpn8i9o_tm
12/12/2019 11:13:24 - INFO - transformers.modeling_utils -   loading weights file https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-pytorch_model.bin from cache at C:\Users\nnn\.cache\torch\transformers\4295d67f022061768f4adc386234dbdb781c814c39662dd1662221c309962c55.778cf36f5c4e5d94c8cd9cefcf2a580c8643570eb327f0d4a1f007fab2acbdf1
12/12/2019 11:13:32 - INFO - __main__ -   Namespace(device=device(type='cpu'), length=100, model_name_or_path='gpt2', model_type='gpt2', n_gpu=0, no_cuda=False, num_samples=1, padding_text='', prompt='My job is', repetition_penalty=1.0, seed=42, stop_token=None, temperature=1.0, top_k=0, top_p=0.9, xlm_lang='')
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 100/100 [00:23<00:00,  2.49it/s]
 to know when it will change, it's up to you."

National Communications Director Alex Brynner said the Trump administration needs to help then-Secretary of State Rex Tillerson learn from him.

"The Cabinet, like any other government job, has to be attentive to the needs of an individual that might challenge his or her position," Brynner said. "This is especially true in times of renewed volatility."

Brynner said Tillerson has not "failed at vetting


title: Broken Pipe in pytorch DataLoader

body:
I was trying to understand how DataLoader works.

This is how I applied it:

# DATASET
class Word2VecDataset(torch_data.Dataset):

def __init__(self, vocabulary):
    super(Word2VecDataset, self).__init__()
    self.data_list = []
    self.vocab = vocabulary
    self.generate_batch_list()

def __getitem__(self, index):
    return self.data_list[index]

def __len__(self):
    return len(self.data_list)

def generate_batch_list(self):
    training_data = self.vocab.get_training_phrases()
    for query in training_data.Query:
        query = utils.skip_gram_tokenize(vocab=self.vocab, sentence=query)
        for entry in query:
            self.data_list.append(entry)
    for response in training_data.Response:
        response = utils.skip_gram_tokenize(vocab=self.vocab, sentence=response)
        for entry in response:
            self.data_list.append(entry)

And this is the actual dataloader part:

 dataset = Word2VecDataset(self.vocab)
 data_loader = torch_data.DataLoader(dataset, self.batch_size, True, num_workers=4)

 print('Model Initialized')
 for epo in range(self.num_epochs):
     loss_val = None
     for i_batch, sample_batched in enumerate(data_loader): # This seems to be causing issues. For some reason this is the part that 'reboots' the whole model, making it print twice (more info under the code)
         loss_val = 0
         for data, target in sample_batched:
         ....

Now, weirdly enough both the initialization phase (which you don't see here) that says 'This is the gpu detected: xxx', and the print('Model Initialized') get printed two times.

Finally here (pastebin) is the full console log (with the error).


title: How to set lr_mult for convolutional layer in pytorch?

body:
In caffe, it has the option to set the learning multiple for convolution as follows

layer {
  name: "conv1a"
  type: "Convolution"
  bottom: "data"
  top: "conv1a"
  param {
    lr_mult: 1
  }
  param {
    lr_mult: 2
  }
  convolution_param {
    num_output: 64
    kernel_size: 3
    pad: 1
    stride: 1
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}

In pytorch, how to set the lr_mult for convolutional layer only with SGD? Thanks


title: Pytorch BCELoss not accepting lists

body:
My convLSTM model returns a list of hidden states (17 total, size (1,3,128,128)) and my target is a list of 17 images( all tensors size: (3,128,128)
When the loss function is called, I get the following error:

  File "/Users/xyz/opt/anaconda3/envs/matrix/lib/python3.7/site->packages/torch/nn/modules/loss.py", line 498, in forward
    return F.binary_cross_entropy(input, target, weight=self.weight, >reduction=self.reduction)
  File "/Users/xyz/opt/anaconda3/envs/matrix/lib/python3.7/site->packages/torch/nn/functional.py", line 2052, in binary_cross_entropy
     if target.size() != input.size():
  AttributeError: 'list' object has no attribute 'size'

Part of the training loop:

    hc = model.init_hidden(batch_size=1)
    for batch_idx, (data, target) in enumerate(train_loader):
        optimizer.zero_grad()
        # Set target, images 2 to 18
        target = data[1:]
        if gpu:
            data = data.cuda()
            target = target.cuda()
            hc.cuda()
        # Get outputs of LSTM
        output = model(data, hc)
        # Calculate loss
        loss = criterion(output, target)
        loss.backward()
        optimizer.step()

I was expecting a size mismatch error but got this instead. How can I fix this?


title: Multiply feature map by a scalar in pytorch

body:
I have a binary classification problem, there are image and variable in the data set, I have an idea for compare image and variable together.

Every time when I pass conv-layer, I want to multiply a weight scalar to all feature map, where weight scalar is computed from a fc-layer.

For example, suppose batch size is 8, there are two tensor x1 and x2, where x1’s size is (8,3,224,224) and x2’s size is (8,16).

import torch
from torch.nn import Module, Sequential
from torch.nn import Conv2d, BatchNorm2d, ReLU, MaxPool2d, Softmax, Linear
import numpy
batch_size = 8
x1 = torch.rand(batch_size*3*224*224).view(batch_size,3,224,224)
x2 = torch.rand(batch_size*16).view(batch_size,16)

I define conv-layer and fc-layer and compute the output from image and variable.

conv_01 = Conv2d(in_channels=3, out_channels= 9, kernel_size=3, stride=1, padding=1)
linear_02 = Linear(16, 1)
c1 = conv_01(x1) ## torch.Size([8, 9, 224, 224])
c2 = linear_02(x2) ## torch.Size([8, 1])

The problem is to write the suitable code as below.

## I want to do like blow
## c1[0,:,:,:] = c1[0,:,:,:] * c2[0,0] # 1st data in the mini-batch
## c1[1,:,:,:] = c1[1,:,:,:] * c2[0,1] # 2nd data in the mini-batch
## c1[2,:,:,:] = c1[2,:,:,:] * c2[0,2]
## c1[3,:,:,:] = c1[3,:,:,:] * c2[0,3]
## c1[4,:,:,:] = c1[4,:,:,:] * c2[0,4]
## c1[5,:,:,:] = c1[5,:,:,:] * c2[0,5]
## c1[6,:,:,:] = c1[6,:,:,:] * c2[0,6]
## c1[7,:,:,:] = c1[7,:,:,:] * c2[0,7]
## output is a (8, 9, 224, 224)
## and do more layer like this operation

I already took a look in the Multiply feature map by a learnable scalar.
But this only support when batch size is 1, but in my case, batch size is more than 1.
How to write a suitable code to the forward function in my case?
Thanks a lot.


title: How to use fairseq interactive.py non-interactively?

body:
I am trying to translate from English to Arabic using Fairseq. But the interactive.py script translate pieces of text fragment on-the-fly. But I need to use it as reading an input text file and writing output text file write. I referred this GitHub issue - https://github.com/pytorch/fairseq/issues/858 But it doesn't clearly explain on how to do it in general.
Any suggestions ?