title: Tensorflow (or Pytorch?) with a cost function defined in terms of Monte Carlo sampling (cost function that changes every epoch)

body:
Edit: I've essentially rewritten this to be (hopefully) much clearer and more concise.  Here's the new version, followed by the original version.

I'm currently using TensorFlow for a physics optimization problem (the physics details/background should be mostly irrelevant, though see below if you're interested and understand the jargon).  This actually has nothing to do with machine learning; I'm just using TF for the automatic differentiation.

So, say one is optimizing x1, x2, ..., xN with a loss function L(x1, x2, ..., xN).  The problem I have is that L(x1, x2, ..., xN) is impossible to calculate exactly and efficiently; however, it can be approximated by sampling.  That is, L(x1, x2, ..., xN) is nearly equal to some:

l(x1, x2, ..., xN) = Σi λi(x1, x2, ..., xN)

where each λi can be efficiently calculated. The catch, though, is the form of each λi depends on the current values of x1, x2, ..., xN.  That is, the sampling isn't "random" but is done in a controlled way based on the current value of the parameters.

For the optimization problem, the idea is to use l instead of L as the cost function.  The complication is that l changes every iteration (or "epoch" in ML terminology).  So, the algorithm is:

At each epoch:

start with l=0; form of l as a function of the xk is unknown at this point
determine the form of λ1.  Then evaluate λ1 (and automatically ∂λ1/∂xk)
Do the sum over i, in the process defining and evaluating l (and automatically ∂l/∂xk).
update x1, ..., xN based on the derivatives ∂li/∂xk.

I'm not sure how to deal with this in TensorFlow, since "if" statements don't work.  I know there are TensorFlow conditionals (tf.cond) but I've had various problems with that, and I'm not even sure if this is the right place to use tf.cond.  Would PyTorch be any better?

[On the physics background: I'm finding the ground state of a quantum spin system; the loss function is the energy, which I'm calculating with the variational Monte Carlo method.  What I called λi is the local energy of a certain spin configuration.  Then a random spin is flipped, and this new spin configuration is accepted with probability = ratio of the wavefunction amplitudes.  λi+1 is the local energy of the new spin configuration, and so on.]

Original version:
Basically, the issue is that my loss function is not rigidly defined, but changes every iteration.

How this arises:
Let's say my variables being optimized are x1, x2, ..., xN and my loss function is L(x1, x2, ..., xN).  In the problem I'm working with, to exactly calculate L(x1, x2, ..., xN) would involve doing an intractably large sum, so the only way is to calculate L(x1, x2, ..., xN) by a Monte Carlo sampling method.  Specifically, the method is a Markov chain Monte Carlo (i.e. sequential sampling) method.

In practice, what my situation basically is:

With the sampling method, I have something like L(x1, ..., xN) = L1(xi11,...,xi1n) + L2(xi21,...,xi2n) + ... + LM(xiM1,...,xiMn).  The important point is that the functions L1, L2, ... are not set in stone (both in terms of their functional form, and in terms of which subset of variables they depend on).  To calculate L, one fist starts with an L1 (essentially, the general form of the Lk is set, but some free parameters remain.  Typically we start with a "random" L1).  Then we generate an L2 "candidate" by randomly generating the free parameters within it.  This is then accepted or rejected depending on the values of some quantities depending on the x1...N variables.  Once an L2 is found, it's evaluated and added to the total L, and we move on to L3, and so on.  To calculate L again, the whole process is repeated.

For optimization, this whole process is repeated each iteration (or "epoch" in ML terminology).   Even though the precise functional form of  L varies from epoch to epoch, for any given values of x1, ..., xN, the value of L(x1, ..., xN) is well defined (to within the accuracy of the approximation), so the optimization should still converge.

The question is: can one do this in TensorFlow?  It's easy to write regular Python code, using if statements, to evaluate L(x1, ..., xN), but that doesn't translate well to TensorFlow.  Would PyTorch be any better?


title: Reduction parameter of the negative log likelihood

body:
What is the intuitive explanation of the reduction parameter in negative log likelihood loss function in PyTorch? The parameter can take values such as 'mean' or 'sum'. Is it summing over the elements of the batch?

torch.nn.functional.nll_loss(outputs.mean(0), target, reduction="sum")


title: how to vectorize the scatter-matmul operation

body:
I have many matrices w1, w2, w3...wn with shapes (k*n1, k*n2, k*n3...k*nn) and x1, x2, x3...xn with shapes (n1*m, n2*m, n3*m...nn*m).
I want to get w1@x1, w2@x2, w3@x3 ... respectively.

The resulting matrix is multiple k*m matrices and can be concatenated into a large matrix with shape (k*n)*m.

Multiply them one by one will be slow. How to vectorize this operation?

Note: The input can be a k*(n1+n2+n3+...+nn) matrix and a (n1+n2+n3+...+nn)*m matrix, and we may use a batch index to indicate those submatrices.

This operation is related to the scatter operations implemented in pytorch_scatter, so I refer it as "scatter_matmul".


title: Importance weighted autoencoder doing worse than VAE

body:
I've been implementing VAE and IWAE models on the caltech silhouettes dataset and am having an issue where the VAE outperforms IWAE by a modest margin (test LL ~120 for VAE, ~133 for IWAE!). I don't believe this should be the case, according to both theory and experiments produced here.

I'm hoping someone can find some issue in how I'm implementing that's causing this to be the case.

The network I'm using to approximate q and p is the same as that detailed in the appendix of the paper above. The calculation part of the model is below:

data_k_vec = data.repeat_interleave(K,0) # Generate K samples (in my case K=50 is producing this behavior)

mu, log_std = model.encode(data_k_vec)
z = model.reparameterize(mu, log_std) # z = mu + torch.exp(log_std)*epsilon (epsilon ~ N(0,1))
decoded = model.decode(z) # this is the sigmoid output of the model

log_prior_z = torch.sum(-0.5 * z ** 2, 1)-.5*z.shape[1]*T.log(torch.tensor(2*np.pi))
log_q_z = compute_log_probability_gaussian(z, mu, log_std) # Definitions below
log_p_x = compute_log_probability_bernoulli(decoded,data_k_vec)

if model_type == 'iwae':
    log_w_matrix = (log_prior_z + log_p_x  - log_q_z).view(-1, K)
elif model_type =='vae':
    log_w_matrix = (log_prior_z + log_p_x  - log_q_z).view(-1, 1)*1/K

log_w_minus_max = log_w_matrix - torch.max(log_w_matrix, 1, keepdim=True)[0]
ws_matrix = torch.exp(log_w_minus_max)
ws_norm = ws_matrix / torch.sum(ws_matrix, 1, keepdim=True)

ws_sum_per_datapoint = torch.sum(log_w_matrix * ws_norm, 1)

loss = -torch.sum(ws_sum_per_datapoint) # value of loss that gets returned to training function. loss.backward() will get called on this value

Here are the likelihood functions. I had to fuss with the bernoulli LL in order to not get nan during training

def compute_log_probability_gaussian(obs, mu, logstd, axis=1):
    return torch.sum(-0.5 * ((obs-mu) / torch.exp(logstd)) ** 2 - logstd, axis)-.5*obs.shape[1]*T.log(torch.tensor(2*np.pi))

def compute_log_probability_bernoulli(theta, obs, axis=1): # Add 1e-18 to avoid nan appearances in training
    return torch.sum(obs*torch.log(theta+1e-18) + (1-obs)*torch.log(1-theta+1e-18), axis)

In this code there's a "shortcut" being used in that the row-wise importance weights are being calculated in the model_type=='iwae' case for the K=50 samples in each row, while in the model_type=='vae' case the importance weights are being calculated for the single value left in each row, so that it just ends up calculating a weight of 1. Maybe this is the issue?

Any and all help is huge - I thought that addressing the nan issue would permanently get me out of the weeds but now I have this new problem.

EDIT:
Should add that the training scheme is the same as that in the paper linked above. That is, for each of i=0....7 rounds train for 2**i epochs with a learning rate of 1e-4 * 10**(-i/7)


title: Tensorflow loss higher than pytorch

body:
I have been trying to perform a comparison between pytorch and tensorflow on a variational autoencoder model. I have built exactly the same model on both. However, the pytorch model outperforms the tensorflow one always showing around 4 times less loss, and seems to be able to learn better loss keeps reducing on most of the epochs.
I do not really see why this difference should be that big. And I have checked that all hyperparameters are the same on both: Adam optimizer, BatchNormalization, ReLu functions. I have even changed the initializer on tensorflow to be the same as pytorch for the linear/Dense function.
Tensorflow:
class Encoder(tf.keras.layers.Layer):
    def __init__(self, input_dim,intermediate_dim,latent_dim):
        super(Encoder, self).__init__()
        self.hidden_layer = tf.keras.Sequential([
            tf.keras.layers.InputLayer(input_shape=(input_dim,)),
            tf.keras.layers.Dense(units=intermediate_dim,use_bias=True),
            tf.keras.layers.BatchNormalization(momentum=0.01, epsilon=0.001),
            tf.keras.layers.ReLU()
            ])

        self.q_mu = tf.keras.layers.Dense(units=latent_dim,activation=None,use_bias=True)
        self.q_var = tf.keras.layers.Dense(units=latent_dim,activation=None,use_bias=True)

    @tf.function
    def reparameterize(self, mean, logvar):
        x = tfp.distributions.Normal(mean,tf.math.sqrt(logvar)).sample()
        return x

    @tf.function
    def call(self, input_features):
        hidden = self.hidden_layer(input_features)

        mu = self.q_mu(hidden)
        var = tf.math.exp(self.q_var(hidden)) + 1e-4
        z = self.reparameterize(mu,var)
        return mu, var,z

class Decoder(tf.keras.layers.Layer):
    def __init__(self, output_dim,intermediate_dim,latent_dim):
        super(Decoder, self).__init__()

        self.px_decoder = tf.keras.Sequential([
            tf.keras.layers.InputLayer(input_shape=(latent_dim,)),
            tf.keras.layers.Dense(units=intermediate_dim,use_bias=True),
            tf.keras.layers.BatchNormalization(momentum=0.01, epsilon=0.001),
            tf.keras.layers.ReLU()
            ])

        self.scale_decoder = tf.keras.Sequential([
            tf.keras.layers.Dense(units=output_dim,activation=None,use_bias=True), tf.keras.layers.Activation('softmax')])
        self.px_r_decoder = tf.keras.layers.Dense(units=output_dim,use_bias=True,activation=None)
        self.px_dropout_decoder = tf.keras.layers.Dense(units=output_dim,use_bias=True,activation=None)

    @tf.function
    def call(self, z):
        px = self.px_decoder(z)

        px_scale = self.scale_decoder(px)
        px_dropout = self.px_dropout_decoder(px)

        px_rate = px_scale#*tf.math.exp(library)

        px_r = self.px_r_decoder(px)
        return px_rate, px_r,px_dropout

class VAE(tf.keras.Model):
    def __init__(self, latent_dim,intermediate_dim, output_dim):
        super(VAE,self).__init__()
        self.q_encoder = Encoder(output_dim,intermediate_dim,latent_dim)
        self.decoder = Decoder(output_dim,intermediate_dim,latent_dim)

    @tf.function
    def log_nb_positive(self,x, mu, theta, eps=1e-8):
        log_theta_mu_eps = tf.math.log(theta + mu + eps)

        negative_likelihood = (theta * (tf.math.log(theta + eps) - log_theta_mu_eps)
            + x * (tf.math.log(mu + eps) - log_theta_mu_eps)
            + tf.math.lgamma(x + theta)
            - tf.math.lgamma(theta)
            - tf.math.lgamma(x + 1)
        )

        return negative_likelihood

    @tf.function
    def loss_f(self,outputs,batch_features):
        qz_m = outputs['qz_m']
        qz_v = outputs['qz_v']
        px_rate = outputs['px_rate']
        px_r = outputs['px_r']
        ql_m = outputs['ql_m']
        ql_v =  outputs['ql_v']

        RCL = tf.math.reduce_sum(-self.log_nb_positive(batch_features, px_rate, px_r),axis=1)

        Muz = tf.ones_like(qz_m)*0
        Varz = tf.ones_like(qz_v)*1

        local_l_mean,local_l_var = tf.math.reduce_mean(batch_features,axis=1), tf.math.reduce_variance(batch_features,axis=1)

        kld = tfp.distributions.kl_divergence(tfp.distributions.Normal(qz_m, tf.math.sqrt(qz_v)), tfp.distributions.Normal(Muz, Varz))

        loss = RCL,tf.math.reduce_sum(kld,axis=1)
        return loss

    @tf.function
    def inference(self, input_features):
        qz_m, qz_v, z,ql_m,ql_v,library = self.q_encoder(input_features)
        px_rate,px_r,_ = self.decoder(z)
        px_r = tf.math.exp(px_r)
        return {'px_rate':px_rate,'px_r':px_r,'qz_m':qz_m,'qz_v':qz_v,'ql_m':ql_m,'ql_v':ql_v}
    @tf.function
    def call(self, input_features):
        outputs = self.inference(input_features)
        loss = self.loss_f(outputs,input_features)
        return loss

#### Loss is obtained using gradienttape something like this:
class Trainer:
    def __init__(self, model,batch_features,lr=1e-3,eps=0.01):
        import datetime
        self.model = model
        self.batch_features = batch_features
        self.opt = tf.optimizers.Adam(learning_rate=lr,epsilon=eps,amsgrad=True,decay=0)
        self.logdir = os.path.join("logs", datetime.datetime.now().strftime("%Y%m%d-%H%M%S"))
        self.writer = tf.summary.create_file_writer(self.logdir)
        self.train_loss = tf.keras.metrics.Mean(name='train_loss')

    @tf.function
    def single_pass(self,curr_epoch,n_epochs,bf):
        with tf.GradientTape(persistent=True) as tape:
            batch_size = bf.shape[0]
            rcl,kld = self.model(bf)
            kl_weight = 1
            lossSingle_Pass = tf.reduce_mean(rcl + kld*kl_weight)

        gradients = tape.gradient(lossSingle_Pass, self.model.trainable_variables)
        gradient_variables = zip(gradients, self.model.trainable_variables)
        self.opt.apply_gradients(gradient_variables)

    def train(self,n_epochs=100):

        self.n_epochs = n_epochs

        with self.writer.as_default():
            with tf.summary.record_if(True):
                for e in n_epochs:
                        for step,bf in enumerate(training_dataset):
                            batch_size = bf.shape[0]
                            self.single_pass(e,self.n_epochs,bf)
                            kl_weight = min(1, e / n_epochs)
                            rcl, kld = self.model(bf)
                            lossTrain = tf.reduce_mean(rcl + kld*kl_weight)

                            running += lossTrain/batch_size
                            likelihood += tf.reduce_mean(rcl)/batch_size

                        tf.summary.scalar('Running Loss', lossTrain, step=e)

Pytorch
def reparameterize_gaussian(mu, var):
    return Normal(mu, var.sqrt()).rsample()
torch.manual_seed(1)

################################################################ Encoder Network ##########################################################################################

class Encoder(nn.Module):
    def __init__(self, input_dim, hidden_dim, latent_dim):

        super().__init__()

        '''  Gene  '''
        self.q_linear = nn.Sequential(
            nn.Linear(input_dim,hidden_dim,bias=True),
            nn.BatchNorm1d(hidden_dim,momentum=0.01, eps=0.001),
            nn.ReLU())
        self.q_mu = nn.Sequential(
            nn.Linear(hidden_dim,latent_dim))

        self.q_var = nn.Sequential(
            nn.Linear(hidden_dim,latent_dim))

    def forward(self, x):
        hidden = self.q_linear(x)
        mean = self.q_mu(hidden)
        log_var = torch.exp(self.q_var(hidden))+ 1e-4
        z= reparameterize_gaussian(mean,log_var)
        return mean, log_var,z

################################################################ Decoder Network ##########################################################################################

class Decoder(nn.Module):

    def __init__(self, latent_dim, hidden_dim, output_dim):
        super().__init__()
        self.px_decoder = nn.Sequential(
            nn.Linear(latent_dim, hidden_dim,bias=True),
            nn.BatchNorm1d(hidden_dim,momentum=0.01, eps=0.001),
            nn.ReLU()
            )

        self.scale_decoder = nn.Sequential(
            nn.Linear(hidden_dim, output_dim),nn.Softmax(dim=-1))
        self.px_r_decoder = nn.Sequential(nn.Linear(hidden_dim, output_dim))
        self.px_dropout_decoder = nn.Sequential(nn.Linear(hidden_dim,output_dim))

    def forward(self, x):
        px = self.px_decoder(x)
        px_scale = self.scale_decoder(px)
        px_dropout = self.px_dropout_decoder(px)
        px_rate = px_scale
        px_r = self.px_r_decoder(px)
        return px_rate, px_r,px_dropout

################################################################ Variational Autoencoder scVI Negative Binomial Model ##########################################################################################

class VAE(nn.Module):
    def __init__(self, input_dim, hidden_dim, latent_dim):
        super().__init__()

        self.q_encoder = Encoder(input_dim, hidden_dim, latent_dim)
        self.decoder = Decoder(latent_dim, hidden_dim, input_dim)

    def sample_from_posterior(self, x, give_mean=False):

        qz_m, qz_v, z,_,_,_ = self.q_encoder(x)  # y only used in VAEC
        if give_mean:
            z = qz_m
        return z

    def forward(self, x):
        qz_m, qz_v, z,ql_m,ql_v,library = self.q_encoder(x)

        px_rate,px_r,_ = self.decoder(z,library)
        px_r = torch.exp(px_r)
        return px_rate, px_r, qz_m, qz_v,ql_m,ql_v,library

def log_nb_positive(x, mu, theta, eps=1e-8):
    log_theta_mu_eps = torch.log(theta + mu + eps)

    res = (theta * (torch.log(theta + eps) - log_theta_mu_eps)
        + x * (torch.log(mu + eps) - log_theta_mu_eps)
        + torch.lgamma(x + theta)
        - torch.lgamma(theta)
        - torch.lgamma(x + 1))
    return res

def get_reconstruction_loss(x, px_rate, px_r, **kwargs):
    reconst_loss = -log_nb_positive(x, px_rate, px_r).sum(dim=-1)
    return reconst_loss

def calculate_lossNEW(x, p_mean, p_log_var,qz_mu,qz_var):
    RCL = get_reconstruction_loss(x, p_mean, p_log_var)

    mean = torch.zeros_like(qz_mu)
    scale = torch.ones_like(qz_var)
    kld = kl(Normal(qz_mu, torch.sqrt(qz_var)), Normal(mean, scale)).sum(dim=1)
    return RCL,kld
optimizer = optim.Adam(model.parameters(), lr=1e-3,eps=0.01)

num_epochs = 100

losses = []
rlosses1 = []

for e in num_epochs:
    running_loss1 = 0
    for z,x in enumerate(mini_batches):

        #========forward pass=====================================]
        px_rate,px_r,qz_mu, qz_var = model(x)

        loss = calculate_lossNEW(x, px_rate,px_r,qz_mu,qz_var)

        if e is not None:
            kl_weight = min(1, e / 100)
        else:
            kl_weight = 1.0

        like = loss[0].mean()
        klditem = loss[1].mean()
        loss = (loss[0] + loss[1]*kl_weight).mean()

        #=======backward pass=====================================
        optimizer.zero_grad() # zero the gradients on each pass before the update
        loss.backward() # backpropagate the loss through the model
        optimizer.step() # update the gradients w.r.t the loss

        losses.append(loss.item())
        running_loss1 += loss.item()/batch_size

    rlosses1.append(running_loss1)

Sorry for the length of the code, but the loss functions are exactly the same. And the KL divergence should also be in theory similar between the pytorch and tensorflow calculations.


title: Why pytorch 1.5 is slower than Tensorflow 2.x？

body:
I tried two version code for a simple classification task.
Tensorflow version
import  tensorflow as tf

from    tensorflow.keras

import datasets, layers, optimizers, Sequential
import time
(xs, ys),(xs_, ys_) = datasets.mnist.load_data()

xs = tf.convert_to_tensor(xs, dtype=tf.float32)/255.

db = tf.data.Dataset.from_tensor_slices((xs,ys))

db = db.batch(30000)

network = Sequential([layers.Dense(8, activation='relu'),
                     layers.Dense(8, activation='relu'),
                     layers.Dense(10)])
network.build(input_shape=(None, 28*28))
optimizer = optimizers.SGD(lr=0.1)
tic=time.time()
for epoch in range(5):
    for step, (x,y) in enumerate(db):

        with tf.GradientTape() as tape:
            # [b, 28, 28] => [b, 784]
            x = tf.reshape(x, (-1, 28*28))
            # [b, 784] => [b, 10]
            # print(tf.reduce_max(x),tf.reduce_min(x))
            out = network(x)
            # [b] => [b, 10]
            y_onehot = tf.one_hot(y, depth=10)
            # [b, 10]
            # print(out,y_onehot)

            loss=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
                logits=out, labels=y_onehot))
        grads = tape.gradient(loss, network.trainable_variables)
        optimizer.apply_gradients(zip(grads, network.trainable_variables))
    print('epoch=',epoch,'loss=',loss.numpy())
toc=time.time()
print('elapsed time:',toc-tic)
# test

xs_=tf.convert_to_tensor(xs_,dtype=tf.float32)/255.
xs_=tf.reshape(xs_,[-1,28*28])

out = network(xs_)

pred=tf.argmax(out,axis=1)

correct=tf.reduce_sum((pred==ys_).numpy().astype('int'))
print(correct.numpy())

import  tensorflow as tf
from    tensorflow.keras import datasets, layers, optimizers, Sequential
import time
(xs, ys),(xs_, ys_) = datasets.mnist.load_data()

xs = tf.convert_to_tensor(xs, dtype=tf.float32)/255.

db = tf.data.Dataset.from_tensor_slices((xs,ys))

db = db.batch(30000)

network = Sequential([layers.Dense(8, activation='relu'),
                     layers.Dense(8, activation='relu'),
                     layers.Dense(10)])
network.build(input_shape=(None, 28*28))
optimizer = optimizers.SGD(lr=0.1)
tic=time.time()
for epoch in range(5):
    for step, (x,y) in enumerate(db):

        with tf.GradientTape() as tape:
            # [b, 28, 28] => [b, 784]
            x = tf.reshape(x, (-1, 28*28))
            # [b, 784] => [b, 10]
            # print(tf.reduce_max(x),tf.reduce_min(x))
            out = network(x)
            # [b] => [b, 10]
            y_onehot = tf.one_hot(y, depth=10)
            # [b, 10]
            # print(out,y_onehot)

            loss=tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(
                logits=out, labels=y_onehot))
        grads = tape.gradient(loss, network.trainable_variables)
        optimizer.apply_gradients(zip(grads, network.trainable_variables))
    print('epoch=',epoch,'loss=',loss.numpy())
toc=time.time()
print('elapsed time:',toc-tic)
# test

xs_=tf.convert_to_tensor(xs_,dtype=tf.float32)/255.
xs_=tf.reshape(xs_,[-1,28*28])

out = network(xs_)

pred=tf.argmax(out,axis=1)

correct=tf.reduce_sum((pred==ys_).numpy().astype('int'))
print(correct.numpy())

Pytorch version
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
import matplotlib.pyplot as plt
import numpy
import time
print(torch.cuda.is_available())

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
print(device)
input_size  = 28*28   # images are 28x28 pixels
output_size = 10      # there are 10 classes

train_loader = torch.utils.data.DataLoader(
    datasets.MNIST('~/data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       # transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=30000, shuffle=True,num_workers=2,pin_memory=True)

test_loader = torch.utils.data.DataLoader(
    datasets.MNIST('~/data', train=False, transform=transforms.Compose([
                       transforms.ToTensor(),
                       # transforms.Normalize((0.1307,), (0.3081,))
                   ])),
    batch_size=1000, shuffle=True)

class FC2Layer(nn.Module):
    def __init__(self, input_size, n_hidden, output_size):
        super(FC2Layer, self).__init__()
        self.input_size = input_size
        self.network = nn.Sequential(
            nn.Linear(input_size, n_hidden),
            nn.ReLU(),
            nn.Linear(n_hidden, n_hidden),
            nn.ReLU(),
            nn.Linear(n_hidden, output_size),
            nn.LogSoftmax(dim=1)
        )

    def forward(self, x):
        #x = x.view(-1, self.input_size)
        return self.network(x)

accuracy_list = []

def train(epoch, model, perm=torch.arange(0, 784).long()):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        # send to device
        data, target = data.to(device), target.to(device)
        # print(torch.min(data))
        # permute pixels
        data = data.view(-1, 28 * 28)
        #data = data[:, perm]
        #data = data.view(-1, 1, 28, 28)

        optimizer.zero_grad()
        output = model(data)
        loss = F.nll_loss(output, target)

        loss.backward()
        optimizer.step()

    print('epoch=',epoch,'loss=',loss.cpu().item())

def test(model, perm=torch.arange(0, 784).long()):
    model.eval()
    test_loss = 0
    correct = 0
    for data, target in test_loader:
        # send to device
        data, target = data.to(device), target.to(device)

        # permute pixels
        data = data.view(-1, 28 * 28)
        #data = data[:, perm]
        #data = data.view(-1, 1, 28, 28)
        output = model(data)
        test_loss += F.nll_loss(output, target,
                                reduction='sum').item()  # sum up batch loss
        pred = output.data.max(1, keepdim=True)[
            1]  # get the index of the max log-probability
        correct += pred.eq(target.data.view_as(pred)).cpu().sum().item()

    test_loss /= len(test_loader.dataset)
    accuracy = 100. * correct / len(test_loader.dataset)
    accuracy_list.append(accuracy)
    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        accuracy))

n_hidden = 8 # number of hidden units

model_fnn = FC2Layer(input_size, n_hidden, output_size)
model_fnn.to(device)
optimizer = optim.SGD(model_fnn.parameters(), lr=0.01)

tic=time.time()
for epoch in range(0, 5):
    train(epoch, model_fnn)
toc=time.time()
print('elapsed time:',toc-tic)
test(model_fnn)

Two code above use the same neural nets architecture, run 5 epoch.
Only the training time is counted.
Results are as follows.

Why is the elapsed time different so large？
Thank you!


title: How do I map Joblib's Parallel function to PyTorch's DistributedDataParallel

body:
I have the following code below which uses Joblib's Parallel and I want to implement this in PyTorch and run it with GPUs. I am reading through PyTorch's DistributedDataParallel documentation, but I can't seem to figure this out.
import numpy as np
import torch
from joblib import Parallel, delayed
from torch.nn.parallel import DistributedDataParallel as DDP

X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]])
X = torch.DoubleTensor(X).cuda()

def X_power_func(j):
    X_power = X**j
    return X_power

results = Parallel(n_jobs = 4)(delayed(X_power_func)(j) for j in range(8))   # how do I map this to
                                                                             # PyTorch's
                                                                             # DistributedDataParallel

Any help would really be appreciated. Many thanks in advance!


title: How to perform Col2Im operation on 4D tensor in Pytorch?

body:
I am currently developing a custom locally connected autoencoder for a computer vision project.
I found a Pytorch implementation of a feedforward locally connected layer on the this feature request (https://github.com/pytorch/pytorch/issues/499) that match my needs for my project.
Here is the code :
import numpy as np
import torch
import torch.nn as nn
from torch.nn import init

# Locally connected layer
class LocallyConnected2D(nn.Module):
    def calculate_spatial_output_shape(self, inputShape, kernelSize, dilation, padding, stride):
        return [np.floor(((inputShape[index]+2*padding[index]-dilation[index]*(kernelSize[index]-1)-1)/stride[index])+1).astype(int) for index in range(len(inputShape))]
    def __init__(self, inputShape, inChannels, outChannels, kernelSize, dilation, padding, stride):
        super().__init__()
        self.inputShape = inputShape
        self.inChannels = inChannels
        self.outChannels = outChannels
        self.kernelSize = kernelSize
        self.dilation = dilation
        self.padding = padding
        self.stride = stride
        # Calculate desired output shape
        self.outputHeight, self.outputWidth = self.calculate_spatial_output_shape(self.inputShape, self.kernelSize, self.dilation, self.padding, self.stride)
        print(self.outputHeight, self.outputWidth)
        self.weightTensorDepth = self.inChannels * self.kernelSize[0] * self.kernelSize[1]
        self.spatialBlocksSize = self.outputHeight * self.outputWidth
        # init weight and bias
        self.weights = nn.Parameter(torch.empty((1, self.weightTensorDepth, self.spatialBlocksSize, self.outChannels),requires_grad=True, dtype=torch.float))
        torch.nn.init.xavier_uniform_(self.weights)
        self.bias = nn.Parameter(torch.empty((1, outChannels, self.outputHeight, self.outputWidth),requires_grad=True, dtype=torch.float))
        torch.nn.init.xavier_uniform_(self.bias)

    def forward(self, input):
        # Perform Vol2Col/Im2Col operation on the input feature given kernel, stride, padding and dilation size
        inputUnfold = torch.nn.functional.unfold(input, self.kernelSize, dilation=self.dilation, padding=self.padding, stride=self.stride)
        # Apply the weight to the unfolded image
        localOpUnfold = (inputUnfold.view((*inputUnfold.shape, 1)) * self.weights)
        return localOpUnfold.sum(dim=1).transpose(2, 1).reshape((-1, self.outChannels, self.outputHeight, self.outputWidth)) + self.bias

# Exemple test :
# distributed locally connected layer where there is no overlaping over the receptive field
inLayer = LocallyConnected2D(inputShape=[128,128], inChannels=3, outChannels=49, kernelSize=[5,5], dilation=[1,1], padding=[0,0], stride=[5,5])
errorLayer = LocallyConnected2D(inputShape=[128,128], inChannels=3, outChannels=49, kernelSize=[5,5], dilation=[1,1], padding=[0,0], stride=[5,5])
# locally connected lateral/recurrent connection -> each neuron see the previous state of their neighbor (RF = 5X5)
latRLayer = LocallyConnected2D(inputShape=[25,25], inChannels=49, outChannels=49, kernelSize=[5,5], dilation=[1,1], padding=[2,2], stride=[1,1])

Based on this implementation, I tried to write a transposed version of this layer (decoding part).
Unfortunately, the torch.nn.functional.fold() operator (similar to Col2Im or Col2Vol) doesnt't support 4D tensor with shape T = [batch, channel, Width, Height] which is essential for the spatial padding of the feature space.
My question is : based on the current implementation; how to code a substitute of the col2im/fold operation for a 4D tensor in the context of a transposed locally connected network ?
I hope my question is clear ^^
And I look forward for your answer !
PS: I raise a feature request on the Pytorch git -> https://github.com/pytorch/pytorch/issues/38394


title: Try to run torch but encounter INTERNAL ASSET FAILED

body:
Tried to uninstall and reinstall several different versions from the attempted solution in other threads.
But still encounter this issue of not getting the "supported version for reading".
Wonder if it is due to the compatibility of windows?
in __init__
    super(_open_zipfile_reader, self).__init__(torch._C.PyTorchFileReader(name_or_buffer))

RuntimeError: version_ <= kMaxSupportedFileFormatVersion INTERNAL ASSERT FAILED
at ..\caffe2\serialize\inline_container.cc:132, please report a bug to PyTorch.
Attempted to read a PyTorch file with version 3, but the maximum supported version for reading is 2.
Your PyTorch installation may be too old. (init at ..\caffe2\serialize\inline_container.cc:132)
(no backtrace available)

windows 10 1909 (18363.1016)
CPU: AMD 4800HS
GPU: 1650Ti
Pytorch version:1.4.0+cu92
torchvision version: 0.5.0
Python version: 3.7
Compiler: Spyder(anaconda 3)
import syft


title: Insecureplatform warning

body:
I am trying to install pytorch on ubuntu 14.04, and i am using python 2.7.6/ cuda 8.0. But error comes out.

error 1:

torch-1.0.0-cp27-cp27m-linux_x86_64.whl is not a supported wheel on this platform.

error 2:

  InsecurePlatformWarning
  Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '_ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure'),)': /whl/cu80/torch-1.0.0-cp27-cp27mu-linux_x86_64.whl
/usr/local/lib/python2.7/dist-packages/pip/_vendor/urllib3/util/ssl_.py:150: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='download.pytorch.org', port=443): Max retries exceeded with url: /whl/cu80/torch-1.0.0-cp27-cp27mu-linux_x86_64.whl (Caused by SSLError(SSLError(1, '_ssl.c:510: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure'),))

I found that Insecureplatformwarning always appeared whenever i tried to install new modules. Is it possible to solve this problem without changing version of python or cuda? Since i am using tensorflow1.3.0 for python 2.7 and cuda 8.0, I am reluctant to change the version of python or cuda.


title: Is there a possible way to break a dataloader loop?

body:
I use a PyTorch DataLoader in my code, and there is loop using data from this dataloader.
However, when I try to end this loop, it failed.
There is no error info. The program is still.
GPU is not out of memory, but the volatile GPU-util keeps to be 0%

def function():
    seq_dt = SequenceLoader(image_list)
    seq_loader = DataLoader(seq_dt, batch_size=1, shuffle=False, sampler=None,
    num_workers=1, pin_memory=True)
    for image_idx, img in enumerate(seq_loader):
        ...
        ...
        my_variable = ...
        ...
        if my_variable<my_threshold:
            return my_flag

Could you please help me with this problem?
Thanks


title: replacement of nditer for numpy array for pytorch tensor

body:
I am looking for similar functionality for pytorch tensor as of nditer of numpy array, see this link with a small example.
https://discuss.pytorch.org/t/replacement-of-np-nditer-for-torch/64024?u=songqsh


title: Creating SequenceTaggingDataset from list, not file

body:
I would like to create a SequenceTaggingDataset from two lists that I have created dynamically inside my code - train_sentences and train_tags. I would want to write something like this:

train_data = SequenceTaggingDataset(examples=(zip(train_sentences, train_tags)))

However, the constructor must receive a path. And not only that - it looks from the code as though, even if I were to provide the examples, it will override those, and initialize examples to be an empty list.

For various reasons, I do not want to save the lists I created in a file from which the SequenceTaggingDataset could read. Is there any way around this, save defining my own custom class?


title: How to break PyTorch autograd with in-place ops

body:
I'm trying to understand better the role of in-place operations in PyTorch autograd.
My understanding is that they are likely to cause problems since they may overwrite values needed during the backward step.

I'm trying to build an example where an in-place operation breaks the auto differentiation, my idea is to overwrite some value needed during the backpropagation after it has been used to compute some other tensor.

I'm using the assignment as the in-place operation (I tried += with the same result), I double-checked it is an in-place op in this way:

x = torch.arange(5, dtype=torch.float, requires_grad=True)
y = x
y[3] = -1
print(x)

prints:

tensor([ 0.,  1.,  2., -1.,  4.], grad_fn=<CopySlices>)

This is my attempt to break autograd:

Without the in-place op:

x = torch.arange(5, dtype=torch.float, requires_grad=True)
out1 = x ** 2
out2 = out1 / 10
# out1[3] += 100
out2.sum().backward()
print(x.grad)

This prints

tensor([0.0000, 0.2000, 0.4000, 0.6000, 0.8000])

With the in-place op:

x = torch.arange(5, dtype=torch.float, requires_grad=True)
out1 = x ** 2
out2 = out1 / 10
out1[3] = 0
out2.sum().backward()
print(x.grad)

This prints:

tensor([0.0000, 0.2000, 0.4000, 0.6000, 0.8000])

I was expecting to obtain differents grads.

What is the item assignment doing? I don't get the grad_fn=<CopySlices>.
Why does it return the same grads?
Is there a working example of in-place operations that break autograd?
Is there a list of non-backwards compatible PyTorch ops?


title: Why do we average the losses of all elements of the batch if we typically should average the gradients (rather than the losses)?

body:
My loss output is

tensor([0.0430, 0.0443, 0.0430, 0.0430, 0.0443, 0.0466, 0.0466, 0.0466],
       grad_fn=<AddBackward0>)

When I execute loss.backward(), I obtained *** RuntimeError: grad can be implicitly created only for scalar outputs

In some places, they rather suggested loss.mean().backward() or loss.sum().backward().

Why is .mean().backward() used, i.e. why do we average the losses of all elements of the batch if we typically should average the gradients (rather than the losses)?

Here is my code


title: Cuda out of memory in loop on second forward pass?

body:
I have the following problem:
model = MyModel()
model.load_state_dict(checkpoint[weights])
model.train()
data, label = get_data() # just take one trainings example
data.cuda()
for i in range(10): # lets predict data 10 times
    output = model(data)
    print(i)

I can do one forward step (i is printed 0) but then I get a Cuda OOM error at output = model(data).
It works if I use with torch.no_grad() but I'd like to train in this forward loop and so I need the gradients later.
Do I somehow have to clear the earlier output or data so it doesn't take up my Vram?
output = model(data.clone()) # this doesn't fix the problem

for i in range(10):
  model.zero_grad() # doesn't work either
  output = model(data)


title: Inverse transform

body:
I am not able to find an efficient way to give batch input to this function and return the batch output. I want to do this during the training of my neural network.
Inverse_Norm = transforms.Normalize(
   mean = [-m/s for m, s in zip(mean, std)],
   std = [1/s for s in std]
)
inverse_norm_input = Inverse_Norm(input)


title: RNN implementation in PyTorch vs Tensorflow

body:
I’m getting started in PyTorch and have a few years experience with Tensorflow v1. I’m a bit confused about how RNNs work in PyTorch.
It seems to me that the provided RNNs in ‘nn’ are all C implementations and I can’t seem to find an equivalent to Tensorflow’s ‘scan’ or ‘dynamic_rnn’ function. Furthermore, all custom implementations of RNNs in PyTorch seem to work using Python for-loops. Wouldn’t this result in multiple calls to the GPU, which slows everything down?
Second: I am used to dealing with minibatches containing variable length sequences in Tensorflow by providing the length of each sequence to the RNN function. This has the advantage (over using an explicit pad token) that you do not need to create a useless entry in the vocabulary just for a pad token. The padded data can be any existing token which will be ignored because it lies outside of the declared length. But it seems that PyTorch assumes a pad token the same way that Keras does. Am I understanding this right? Do I have to reserve an entry in my embedding matrix for pad tokens?
Thanks!


title: nn.Functinal vs nn - Pytorch

body:
while adding loss in pytorch I have same function in nn.Functional as well as nn. what is the difference ?
 torch.nn.CrossEntropyLoss() and torch.nn.functional.cross_entropy


title: pytorch LSTM model not learning

body:
I created a simple LSTM model to predict Uniqlo closing price. The problem is, my model doesn't seem to learn anything. This is the link to my notebook
This is my model creation class (I tried relu activation function before, get the same outcome):
class lstm(torch.nn.Module):
  def __init__(self,hidden_layers):
    super(lstm,self).__init__()
    self.hidden_layers = hidden_layers
    self.lstm = torch.nn.LSTM(input_size = 2,hidden_size = 100,num_layers = self.hidden_layers,batch_first=True)
    self.hidden1 = torch.nn.Linear(100,80)
    self.dropout1 = torch.nn.Dropout(0.1)
    self.hidden2 = torch.nn.Linear(80,60)
    self.dropout2 = torch.nn.Dropout(0.1)
    self.output = torch.nn.Linear(60,1)

  def forward(self,x):
    batch = len(x)
    h = torch.randn(self.hidden_layers,batch,100).requires_grad_().cuda()
    c = torch.randn(self.hidden_layers,batch,100).requires_grad_().cuda()

    x,(ho,co)= self.lstm(x.view(batch,10,2),(h.detach(),c.detach()))
    x = torch.reshape(x[:,-1,:],(batch,-1))
    x = self.hidden1(x)
    x = torch.nn.functional.tanh(x)
    x = self.dropout1(x)
    x = self.hidden2(x)
    x = torch.nn.functional.tanh(x)
    x = self.dropout2(x)
    x = self.output(x)
    return x

model = lstm(10)

This is my training loss plot:
training loss
This is my validation loss plot:
validation loss
This is my ground truth (blue) vs prediction (orange):
ground truth vs prediction
Can anyone please point out what did I do wrongly?


title: Is there a way to convert the quint8 pytorch format to np.uint8 format?

body:
I'm using the code below to get the quantized unsiged int 8 format in pytorch. However, I'm not able to convert the quant variable to the to np.uint8. Is there possible to do that?
import torch

quant = torch.quantize_per_tensor(torch.tensor([-1.0, 0.352, 1.321, 2.0]), 0.1, 10, torch.quint8)


title: PyTorch, when divided by zero, set the result value with 0

body:
In Pytorch, when values are divided by zero, replace the result value with 0, as it will output NaN. Here is an example,
a = th.from_numpy(np.array([ [1, 0], [0, 1], [1, 1]]))
b = th.zeros_like(a)
b[0, :] = 2
a = a / b

How can I do that?


title: Pytorch: why does torch.where method does not work like numpy.where?

body:
In order to replace positive values with a certain number and negative ones with another number in a random vector using Numpy one can do the following:
npy_p = np.random.randn(4,6)
quant = np.where(npy_p>0, c_plus , np.where(npy_p<0, c_minus , npy_p))

However where method in Pytorch throws out the following error:

expected scalar type double but found float

Can you help me with that?


title: problem with nn.ModuleDict ('method' object is not subscriptable)

body:
I am tring to use nn.ModuleDict following this documentation:  https://pytorch.org/docs/stable/generated/torch.nn.ModuleDict.html
I have this pytorch network:
class Net(nn.Module):
    def __init__(self, kernel_size):
        super(Net, self).__init__()
        modules = {}
        modules["layer1"] = nn.Conv2d(3, 16, kernel_size=kernel_size, stride=1, padding=2)
        self.modules = nn.ModuleDict(modules)

    def forward(self, x):
        x = self.modules["layer1"](x)

when i use the forward method, I get the following error:
'method' object is not subscriptable

when I change the forward method to:
  def forward(self, x):
        x = self.modules()["layer1"](x)

I get the following error:
TypeError: 'generator' object is not subscriptable


title: What do the charts in the System Panels signify in Wandb (PyTorch)

body:
I recently started using the wandb module with my PyTorch script, to ensure that the GPU's are operating efficiently. However, I am unsure as to what exactly the charts indicate.
I have been following the tutorial in this link, https://lambdalabs.com/blog/weights-and-bias-gpu-cpu-utilization/ , and was confused by this plot:

I am uncertain about the GPU % and the GPU Memory Access % charts. The descriptions in the blog are as following:

GPU %: This graph is probably the most important one. It tracks the percent of the time over the past sample period during which one or more kernels was executing on the GPU. Basically, you want this to be close to 100%, which means GPU is busy all the time doing data crunching. The above diagram has two curves. This is because there are two GPUs and only of them (blue) is used for the experiment. The Blue GPU is about 90% busy, which means it is not too bad but still has some room for improvement. The reason for this suboptimal utilization is due to the small batch size (4) we used in this experiment. The GPU fetches a small amount of data from its memory very often, and can not saturate the memory bus nor the CUDA cores. Later we will see it is possible to bump up this number by merely increasing the batch size.

GPU Memory Access %: This is an interesting one. It measures the percent of the time over the past sample period during which GPU memory was being read or written. We should keep this percent low because you want GPU to spend most of the time on computing instead of fetching data from its memory. In the above figure, the busy GPU has around 85% uptime accessing memory. This is very high and caused some performance problem. One way to lower the percent here is to increase the batch size, so data fetching becomes more efficient.

I had the following questions:

The aforementioned values do not sum to 100%. It seems as though our GPU can either be spending time on computation or spending time on reading/writing memory. How can the sum of these two values be greater than 100%?
Why does increasing batch size decrease the time spent accessing GPU Memory?


title: Layer normalization in pytorch

body:
I'm trying to test layer normalization function of PyTorch.
But I don't know why b[0] and result have different values here
Did I do something wrong ?
import numpy as np
import torch
import torch.nn as nn

a = torch.randn(1, 5)

m = nn.LayerNorm(a.size()[1:], elementwise_affine= False)

b = m(a)

Result:
input: a[0] = tensor([-1.3549, 0.3857,  0.1110, -0.8456, 0.1486])

output: b[0] = tensor([-1.5561, 1.0386,  0.6291, -0.7967, 0.6851])

mean = torch.mean(a[0])
var = torch.var(a[0])
result = (a[0]-mean)/(torch.sqrt(var+1e-5))

Result:
result = tensor([-1.3918, 0.9289, 0.5627, -0.7126, 0.6128])

And, for n*2 normalization , the result of pytorch layer norm is always [1.0 , -1.0] (or [-1.0, 1.0]) . I can't understand why. Please let me know if you have any hints
a = torch.randn(1, 2)

m = nn.LayerNorm(a.size()[1:], elementwise_affine= False)

b = m(a)

Result:
b = tensor([-1.0000, 1.0000])


title: Shape mismatch in two equivalent networks?

body:
So, the answer is probably "the two networks are not equivalent" and there's something obvious I'm missing, but in my understanding they should do the same thing but the dimensions of intermediate outputs are not the same. My main issue is with pytorch ConvTranspose3d and Conv3d. In particular, I have an input that has shape (1,44,68,120), where 44 is the depth dimension while 68 and 120 are width and height. I use strided conv/convtranspose to go down/up in dimensions. Shouldn't this:

self.conv3d1_1 = nn.Conv3d(in_channels=1, out_channels=32, groups=1, stride=(2,2,2), kernel_size=3, padding=1)

be equivalent to these two layers in terms of output dimensions?

self.conv3d1_1 = nn.Conv3d(in_channels=1, out_channels=32, groups=1, stride=(2,1,1), kernel_size=3, padding=1)
self.conv3d1_2 = nn.Conv3d(in_channels=32, out_channels=32, groups=1, stride=(1,2,2), kernel_size=3, padding=1)

where the first layer halves all the dimensions simultaneously, while the second block decreases the temporal dimension first, and the spatial dimensions after?

So, first network:

    self.conv3d1_1 = nn.Conv3d(in_channels=1, out_channels=32, groups=1, stride=(2,2,2), kernel_size=3, padding=1)  # depthwise convolution (1 ch, 44 depth dimension, h, w)
    self.conv3d1_2 = nn.Conv3d(in_channels=32, out_channels=64, groups=1, kernel_size=3, padding=1)

    self.conv3d2_1 = nn.Conv3d(in_channels=64, out_channels=128, groups=1, stride=(2, 2, 2), kernel_size=3, padding=1)
    self.conv3d2_2 = nn.Conv3d(in_channels=128, out_channels=128, groups=1, kernel_size=3, padding=1)

    self.conv3d3_1 = nn.ConvTranspose3d(in_channels=128, out_channels=128, stride=(2, 2, 2), groups=1, kernel_size=4, padding=(1,1,1))
    self.conv3d3_2 = nn.Conv3d(in_channels=128, out_channels=128, groups=1, kernel_size=3, padding=1)

    self.conv3d4_1 = nn.ConvTranspose3d(in_channels=128, out_channels=128, stride=(2, 2, 2), groups=1, kernel_size=4, padding=1)
    self.conv3d4_2 = nn.Conv3d(in_channels=128, out_channels=64, groups=1, kernel_size=3, padding=1)

Produces these intermediate dimensions:

torch.Size([2, 1, 44, 68, 120])
torch.Size([2, 32, 22, 34, 60])
torch.Size([2, 64, 22, 34, 60])
torch.Size([2, 128, 11, 17, 30])
torch.Size([2, 128, 11, 17, 30])
torch.Size([2, 128, 22, 34, 60])
torch.Size([2, 128, 22, 34, 60])
torch.Size([2, 128, 44, 68, 120])
torch.Size([2, 64, 44, 68, 120])

Everything looks fine.

Second network (only 1 decrease in dimensions to keep it short, but same and worst happens when going down 2 times (so 4x))

    elf.conv3d1_1 = nn.Conv3d(in_channels=1, out_channels=32, groups=1, stride=(2,1,1), kernel_size=3, padding=1)  # depthwise convolution (1 ch, 44 depth dimension, h, w)
    self.conv3d1_2 = nn.Conv3d(in_channels=32, out_channels=64, groups=1, kernel_size=3, padding=1)

    self.conv3d2_1 = nn.Conv3d(in_channels=64, out_channels=128, groups=1, stride=(1, 2, 2), kernel_size=3, padding=1)
    self.conv3d2_2 = nn.Conv3d(in_channels=128, out_channels=128, groups=1, kernel_size=3, padding=1)

    self.conv3d3_1 = nn.ConvTranspose3d(in_channels=128, out_channels=128, stride=(1, 2, 2), groups=1, kernel_size=4, padding=(1,1,1))
    self.conv3d3_2 = nn.Conv3d(in_channels=128, out_channels=128, groups=1, kernel_size=3, padding=1)

    self.conv3d4_1 = nn.ConvTranspose3d(in_channels=128, out_channels=128, stride=(2, 1, 1), groups=1, kernel_size=4, padding=1)
    self.conv3d4_2 = nn.Conv3d(in_channels=128, out_channels=64, groups=1, kernel_size=3, padding=1)

And these are the intermediate dimensions:

torch.Size([2, 1, 44, 68, 120])
torch.Size([2, 32, 22, 68, 120])
torch.Size([2, 64, 22, 68, 120])
torch.Size([2, 128, 22, 34, 60])
torch.Size([2, 128, 22, 34, 60])
torch.Size([2, 128, 23, 68, 120])
torch.Size([2, 128, 23, 68, 120])
torch.Size([2, 128, 46, 69, 121])
torch.Size([2, 64, 46, 69, 121])

For some reason, the first ConvTranspose3d increases the temporal dimension, while it should be only working on the spatial dimensions? I initially thought it was a padding issue but changing the padding doesn't fix the issue. Same happens with the temporal ConvTranspose3d, which increases the spatial dimensions by 1.

Any clues? Thanks in advance.


title: Pytorch not building with cmake in Developer Console

body:
I've mostly used languages with simple IDEs until now, so I don't have the best knowledge of compiling and running git and cmake and everything else through command line. I need to use Pytorch for a project though, so it's necessary to use those skills. I'm installing it according to the tutorial for windows found here:

https://caffe2.ai/docs/getting-started.html?platform=windows&configuration=compile

I've gotten to the point where I'm running build_windows.bat, but I'm getting this output with an error from the Developer Command Prompt.

The system cannot find the drive specified.
Requirement already satisfied: pyyaml in g:\programs\python27\lib\site-packages (3.13)
CAFFE2_ROOT=G:\Programs\Caffe2\pytorch\scripts\..
CMAKE_GENERATOR="Visual Studio 14 2015 Win64"
CMAKE_BUILD_TYPE=Release
-- Selecting Windows SDK version  to target Windows 10.0.17134.
CMake Error at CMakeLists.txt:6 (project):
  Failed to run MSBuild command:

    MSBuild.exe

  to get the value of VCTargetsPath:

    Microsoft (R) Build Engine version 15.8.169+g1ccb72aefa for .NET Framework
    Copyright (C) Microsoft Corporation. All rights reserved.

    Build started 9/25/2018 4:20:32 PM.
    Project "G:\Programs\Caffe2\pytorch\build\CMakeFiles\3.12.2\VCTargetsPath.vcxproj" on node 1 (default targets).
    G:\Programs\Caffe2\pytorch\build\CMakeFiles\3.12.2\VCTargetsPath.vcxproj(14,2): error MSB4019: The imported project "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.Cpp.Default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
    Done Building Project "G:\Programs\Caffe2\pytorch\build\CMakeFiles\3.12.2\VCTargetsPath.vcxproj" (default targets) -- FAILED.

    Build FAILED.

    "G:\Programs\Caffe2\pytorch\build\CMakeFiles\3.12.2\VCTargetsPath.vcxproj" (default target) (1) ->
      G:\Programs\Caffe2\pytorch\build\CMakeFiles\3.12.2\VCTargetsPath.vcxproj(14,2): error MSB4019: The imported project "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.Cpp.Default.props" was not found. Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.

        0 Warning(s)
        1 Error(s)

    Time Elapsed 00:00:00.15

  Exit code: 1

-- Configuring incomplete, errors occurred!
See also "G:/Programs/Caffe2/pytorch/build/CMakeFiles/CMakeOutput.log".
"Caffe2 building failed"

The CMakeOutput.log file only contains this:

The system is: Windows - 10.0.17134 - AMD64

Which isn't very useful. I'm not sure what I'm doing wrong here.


title: How to properly optimize shared network between actor and critic?

body:
I'm building an actor-critic reinforcment learning algorithm to solve environments.  I want to use a single encoder to find representation of my environment.

When I share the encoder with the actor and the critic, my network isn't learning anything:

class Encoder(nn.Module):
  def __init__(self, state_dim):
    super(Encoder, self).__init__()

    self.l1 = nn.Linear(state_dim, 512)

  def forward(self, state):
    a = F.relu(self.l1(state))
    return a

class Actor(nn.Module):
  def __init__(self, state_dim, action_dim, max_action):
    super(Actor, self).__init__()

    self.l1 = nn.Linear(state_dim, 128)
    self.l3 = nn.Linear(128, action_dim)

    self.max_action = max_action

  def forward(self, state):
    a = F.relu(self.l1(state))
    # a = F.relu(self.l2(a))
    a = torch.tanh(self.l3(a)) * self.max_action
    return a

class Critic(nn.Module):
  def __init__(self, state_dim, action_dim):
    super(Critic, self).__init__()

    self.l1 = nn.Linear(state_dim + action_dim, 128)
    self.l3 = nn.Linear(128, 1)

  def forward(self, state, action):
    state_action = torch.cat([state, action], 1)

    q = F.relu(self.l1(state_action))
    # q = F.relu(self.l2(q))
    q = self.l3(q)
    return q

However, when I use different encoder for the actor and different for the critic, it learn properly.

class Actor(nn.Module):
def __init__(self, state_dim, action_dim, max_action):
    super(Actor, self).__init__()

    self.l1 = nn.Linear(state_dim, 400)
    self.l2 = nn.Linear(400, 300)
    self.l3 = nn.Linear(300, action_dim)

    self.max_action = max_action

def forward(self, state):
    a = F.relu(self.l1(state))
    a = F.relu(self.l2(a))
    a = torch.tanh(self.l3(a)) * self.max_action
    return a

class Critic(nn.Module):
  def __init__(self, state_dim, action_dim):
    super(Critic, self).__init__()

    self.l1 = nn.Linear(state_dim + action_dim, 400)
    self.l2 = nn.Linear(400, 300)
    self.l3 = nn.Linear(300, 1)

  def forward(self, state, action):
    state_action = torch.cat([state, action], 1)

    q = F.relu(self.l1(state_action))
    q = F.relu(self.l2(q))
    q = self.l3(q)
    return q

Im pretty sure its becuase of the optimizer.
In the shared encoder code, I define it as foolow:

self.actor_optimizer = optim.Adam(list(self.actor.parameters())+
                                      list(self.encoder.parameters()))
self.critic_optimizer = optim.Adam(list(self.critic.parameters()))
                                         +list(self.encoder.parameters()))

In the seperate encoder, its just:

self.actor_optimizer = optim.Adam((self.actor.parameters()))
self.critic_optimizer = optim.Adam((self.critic.parameters()))

two optimizers must be becuase of the actor critic algorithm.

How can I combine two optimizers to optimize correctly the encoder?


title: how to find the inplace operation: one of the variables needed for gradient computation has been modified by an inplace operation

body:
I encounter the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation When I run the folloing codes. I tried searching for some solutions to this problem, but failed to solve my problem.

this codes run on Pytorch 0.4

import torch
import torch.nn as nn
from torch.nn import functional as F
import pdb

class Summ(nn.Module):
    def __init__(self, in_dim=1024, hid_dim=256, num_layers=1, cell='lstm', action='wo', semantic=False):
        super(DSN, self).__init__()
        assert cell in ['lstm', 'gru'], "cell must be either 'lstm' or 'gru'"
        if cell == 'lstm':
            self.rnn = nn.LSTM(in_dim, hid_dim, num_layers=num_layers, bidirectional=True, batch_first=True)
        else:
            self.rnn = nn.GRU(in_dim, hid_dim, num_layers=num_layers, bidirectional=True, batch_first=True)
        if semantic:
            self.lstm_video = nn.LSTM(in_dim, hid_dim, num_layers=num_layers, bidirectional=False, batch_first=True)
            self.lstm_summary = nn.LSTM(in_dim, hid_dim, num_layers=num_layers, bidirectional=False, batch_first=True)

        self.fc = nn.Linear(hid_dim*2, 1)
        self.fc1 = nn.Linear(in_dim, 1)
        self.fc2 = nn.Linear(in_dim, 1)
        self.fc3 = nn.Linear(in_dim, 1)
        self.action = action
        self.semantic = semantic

    def forward(self, x):
        # x:1* T*1024
        if self.semantic:
            ori = x.clone()
            h_video, _ = self.lstm_video(x) # embedding of video

        T = x.size()[1]
        D = x.size()[2]
        d1 = torch.zeros(T,D,requires_grad=True)
        d2 = torch.zeros(T,D,requires_grad=True)
        d3 = torch.zeros(T,D,requires_grad=True)
        for i in range(T-1):
            d1[i,:] = abs(x[0,i+1,:] - x[0,i,:])

        for i in range(T-2):
            d2[i,:] = abs(x[0,i+2,:] - x[0,i,:])

        for i in range(T-4):
            d3[i,:] = abs(x[0,i+4,:] - x[0,i,:])

        f1 = self.fc1(d1.cuda())
        f2 = self.fc2(d2.cuda())
        f3 = self.fc3(d3.cuda())

        if self.action in ['softmax', 'softmax_skip']:
            f = F.softmax(f1 + f2 + f3, dim=0)
        else:
            f = f1 + f2 + f3

        for i in range(T):
            with torch.no_grad():
                if self.action == 'softmax_skip':
                    x[0,i,:] = x[0,i,:].clone() * (1 + f[i,0])
                else:
                    x[0,i,:] = x[0,i,:].clone() * f[i,0]
        h, _ = self.rnn(x)
        p = torch.sigmoid(self.fc(h))

        if self.semantic:
            for i in range(T):
                with torch.no_grad():
                    ori[0,i,:] = ori[0,i,:].clone() * p[0,i,0]
            h_summary, _ = self.lstm_summary(ori) # embedding of summary

            return p, h_video, h_summary

        return p

def semantic_loss(hv, hs):
    h_vd = torch.squeeze(hv)
    h_sm = torch.squeeze(hs)

    gap_h = h_vd - h_sm
    loss = torch.norm(gap_h)
    return loss


title: pytorch LSTM Regression; the Loss function peaks up in every epoch

body:
I'm self learning the pytorch and trying out LSTM to apply in Remaining Useful Lifetime regression (RUL) problem.

My input matrix consists of feature variables in vector stacking down along the rows [2D matrix of time-step by feature vector] I'm feeding the input with preprocessed DataLoader with the size of [batchsize, sequence length, number of featurs] for each enumerate(train_loader)

I was trying loss functions like MSELoss, L1Loss with RMSprop, SGD optimizer and the target is the simple declining RUL values along the timesteps.

So the problem is, at every epochs, the loss calucation starts from the scratch as the new epoch proceeds. It could possibly due to the high integer values of the target, causing the high loss values. However, as the epoch repeats, the starting value of the loss were expected to be decreasing since the learning is somehow been done. but the result is not what i expected..

Hopefully somebody could help me from this problem.
Thank you in advance..!!!

following the the model source:

class GRU(nn.Module):
    def __init__(self, input_dim, hidden_dim, batch_size, window_size, output_dim=1, num_layers=2):
        super(GRU, self).__init__()
        self.input_dim = input_dim
        self.hidden_dim = hidden_dim
        self.batch_size = batch_size
        self.window_size = window_size

        # Define the GRU layer
        self.gru1 = nn.GRU(
                input_size=self.input_dim,
                hidden_size=hidden_dim,
                bias=True)

        # Define the output layer
        self.linear1 = nn.Linear(hidden_dim, 500)
        self.linear2 = nn.Linear(500, 1)

    def forward(self, input, hidden):
        input=input.transpose(1,0)
        gru_out, self.hidden1 = self.gru1(input.view(self.window_size, self.batch_size, -1), hidden)

        y_pred = self.linear1(gru_out[-1].view(self.batch_size, -1))
        y_pred = self.linear2(y_pred)

        return y_pred.view(-1), self.hidden1.detach()

following is my train source:

#input shape: [sequence_length, batch_size, input_size]
#target shape: [batch_size, 1]

for j in range(epoch):
    for i, (dat, target) in enumerate(train_loader):

        out, hidden = model(dat, hidden)
        loss = criterion(out, target.view(-1))
        loss.backward(retain_graph=True)

        optimizer.step()
        optimizer.zero_grad()

        loss_history.append(loss.item())

        if i % 100 == 0:
            print("{} epoch: {}/{}timestep, time: {}, loss {}:".format(j, i,len(train_loader), round(t2-t1,4), loss))
        RUL=torch.cat((RUL,out.data.view(-1)))

    plt.plot(RUL[-28000:].detach().numpy(), linewidth=0.5)
    plt.show()


title: Use pytorch optimizer to fit a user defined function

body:
I have read many tutorials on how to use PyTorch to make a regression over a data set, using, for instance, a model composed of several linear layers and the MSE loss.

Well, imagine that I know the function F depends on a variable x and some unknown parameters (p_j: j=0,..., P-1) with P relatively small, but the function is a composition of special function. So, my problem is the classical minimization knowing the data {x_i,y_i}_i<=N

Min_{ {p_j} } Sum_i (F(x_i;{p_j}) - y_i)^2

I would like to know if I can use the PyTorch optimizers and if yes how I can do it?

Thanks.


title: How are neural networks, loss and optimizer connected in PyTorch?

body:
I've seen answers to this question, but I still don't understand it at all. As far as I know, this is the most basic setup:

net = CustomClassInheritingFromModuleWithDefinedInitAndForward()
criterion = nn.SomeLossClass()
optimizer = optim.SomeOptimizer(net.parameters(), ...)
for _, data in enumerate(trainloader, 0):
    inputs, labels = data
    optimizer.zero_grad()
    outputs = net(inputs)
    loss = criterion(outputs, labels)
    loss.backward()
    optimizer.step()

What I don't understand is:

Optimizer is initialized with net.parameters(), which I thought are internal weights of the net.

Loss does not access these parameters nor the net itself. It only has access to net's outputs and input labels.

Optimizer does not access loss either.

So if loss only works on outputs and optimizer only on net.parameters, how can they be connected?


title: Pytorch: How do I deal with different input sizes within one batch?

body:
I am implementing something closely related to the DeepSets architecture on point clouds:

https://arxiv.org/abs/1703.06114

That means I am working with a set of inputs (coordinates), have fully connected layers process each of those seperately and then perform average pooling over them (to then do further processing).

The input for each sample i is a tensor of shape [L_i, 3] where L_i is the number of points and the last dimension is 3 because each points has x,y,z coordinates. Crucially, L_i depends on the sample. So I have a different number of points per instance. When I put everything into a batch, I currently have the input in the shape [B, L, 3] where L is larger than L_i for all i. The individual samples are padded with 0's. The issue is that 0's are not ignored by the network, they are processed and fed into the average pooling. Instead I would like the average pooling to only consider actual points (not padded 0's). I do have another array which stores the lengths [L_1, L_2, L_3, L_4...], but I am not sure how to use it.

My Question is: How do you handle different input sizes wihtin one batch in the most graceful manner?

This is how the model is define:

encoder = nn.Sequential(nn.Linear(3, 128), nn.ReLU(), nn.Linear(128, 128))
x = self.encoder(x)
x = x.max(dim=1)[0]
decoder = ...


title: Torch: backpropagate gradients without updating variables

body:
In pytorch, is there an efficient way to back propagate the gradients but not update their corresponding variables? It seems making a copy of the weights every time during update is too expensive. But no_grad & set_grad_enabled does not allow back propagation.

Ex. The following seems to take too much time, as it needs to do a copy of the model every time weights are updated:

    def __init__():
        …
        self.model = MyModel()
        self.func1 = FuncModel1()
        self.func2 = FuncModel2()
        …
    def trainstep(input):
        f1 = self.func1(input)
        f2 = self.func2(input)
        …
        # want to update weights in model & f1 with respect to loss1
        loss1 = my_loss(model(f1), y1)

        # don’t want to update weights in self.model with respect to loss2
        # but want to update weights in f2 for loss2
        copy_model = MyModel()
        copy_model.load_state_dict(self.model.state_dict())
        loss2 = my_loss(copy_model(f2), y2)

        total_loss = loss1 + loss2
        …

        total_loss.backward()
        optimizer.step()


title: How can I get the mean of 5 tensors along an axis?

body:
I have 5 tensors of shape torch.Size([7, 20, 180])

I want to get the mean of each along dim=1 so that my final shape will be torch.Size([7, 20, 180]).

Basically, I want the first element from the first tensor, then the first element from the second tensor, and so on to be averaged.


title: How can torch parameters be frozen if an optimizer with momentum is used?

body:
I am training a torch model, where I want to freeze (and later unfreeze) certain parameters.
I was under the impression, that simply setting
param.requires_grad = False
would accomplish this. This does not seem to be the case for optimizers with momentum. I know that I can either instantiate a new optimizer or change the parameters of the existing one, but neither would allow me to unfreeze parameters (easily) and without keeping an extra reference to all the parameters that the optimizer was changing initially.

I think the desired result could be achieved by setting the momentum_buffer in the state of the optimizer to zero, but I am not sure how to do this, at it can not be easily accessed.

The code bellow can be used to reproduce the effects, with both known "solutions" commented out.

import torch
import torch.nn as nn
import torchvision
from tqdm import tqdm

class Flatten(nn.Module):

    def __init__(self):
        super(Flatten, self).__init__()

    def forward(self, x):
        return x.view((x.size()[0], -1))

def main():
    data = torchvision.datasets.MNIST("./data",download=True,
                                       transform=torchvision.transforms.Compose([
                                       torchvision.transforms.ToTensor(),
                                       torchvision.transforms.Normalize((0.1307,), (0.3081,))
                                        ]))
    data_loader = torch.utils.data.DataLoader(data,
                                              batch_size=1000,
                                              shuffle=True)

    net=nn.Sequential(*[Flatten(),
                    nn.Linear(28*28,100),
                    nn.ReLU(),
                    nn.Linear(100,10)])
    opt=torch.optim.SGD(net.parameters(), lr=0.01, momentum=0.9)

    for e in range(2):
        old_params = [p.clone() for p in net.parameters()]
        if e == 1:
            for j,p in enumerate(net.parameters()):
                if j<2:
                    p.requires_grad = False
            # opt.param_groups[0]['params'] = opt.param_groups[0]['params'][2:]

        # opt = torch.optim.SGD(net.parameters(), lr=0.01, momentum=0.9)

        for data, label in tqdm(data_loader):
            loss=torch.nn.functional.cross_entropy(net(data),label)
            opt.zero_grad()
            loss.backward()
            opt.step()
        print(loss)

        new_params=[p.clone() for p in net.parameters()]
        change = [(~(p1 == p2).all()).item() for p1, p2 in zip(old_params, new_params)]
        print("Epoch: %d \t params changed: %s" % (e, change))
        print([p.requires_grad for p in net.parameters()])

if __name__ == '__main__':
    main()


title: Pytorch: How to know if GPU memory being utilised is actually needed or is there a memory leak

body:
I have 3 Tesla V100(16 GB). I am doing transfer learning using efficeint net (63 Million parameters) on images of (512,512) with a batch size of 20.
My GPU memory utilisation is below -

As you can see, it has almost filled up all the 3 GPUs(almost 80%).
My question is is there any theoretical way of calculating that the utilisation of GPU memory being shown is what is required by the model at a certain image and batch size or is there a memory leak in my GPU?


title: Trouble installing PyTorch for OpenNMT

body:
I'm trying to train an openNMT model using the instructions here - https://pypi.org/project/OpenNMT-py/

However I run into a problem while installing (using python setup.py install)

It runs into an error -

Processing dependencies for OpenNMT-py==1.1.1
Searching for torch>=1.4.0
Reading https://pypi.org/simple/torch/
No local packages or working download links found for torch>=1.4.0
error: Could not find suitable distribution for Requirement.parse('torch>=1.4.0')

I tried manually installing manually - python -m pip install torch==1.4.0 (because the link earlier says 1.4.0 is the only one supported, but it doesn't work. (Says No Matching Distribution found for torch==1.4.0)


title: Where can I find a pre-trained variational autoencoder that's trained on natural images?

body:
Does anyone know a pre-trained variational autoencoder (VAE) or a VAE-GAN that's trained on natural images?

I have been searching for a variational autoencoder that is trained on natural images (e.g. ImageNet) where I can extract latent features from. It could be implemented in any framework, preferably MXNet, Chainer or PyTorch.


title: Finetuning (German) Bert. Is it underfitting?

body:
With pytorch-lightning and transformers, I finetuned a Bert model on german service tickets. The dataset has the following size:
FULL Dataset: (1220, 2)
TRAIN Dataset: (854, 2)
VAL Dataset: (366, 2)

Every ticket can be in exactly 1 out of 10 categories. This is why my model is initilized in def init like:
#changing the configuration to X lables instead of 2
self.bert = transformers.BertModel.from_pretrained(MODEL_NAME)
self.drop = th.nn.Dropout(p=0.1)
self.out = th.nn.Linear(self.bert.config.hidden_size, NUM_LABELS)
self.softmax = th.nn.Softmax(dim=1)
self.loss = th.nn.CrossEntropyLoss(reduction="none")

This produceds a probability distributuion over the 10 classes for each sample. As the forward function is:
 def forward(self, input_ids, mask):
    _, pooled_output = self.bert(
                                input_ids=input_ids,
                                attention_mask=mask
                        )
    output= self.drop(pooled_output)
    output = self.out(output)
    return self.softmax(output)

As a loss function torch.nn.CrossEntropyLoss is defined and within the training_step called. With a Batch_Size of 16, logits.shape = [16,10] and batch['targets'].shape = [16] and batch['targets'] = [1,5,2,4,8,6,9,0,0,1,2,7,7,7,5,3]. Is CrossEntropyLoss the right loss function? And is the optimizer even working?
 def training_step(self, batch, batch_idx):
    logits = self.forward(batch['input_ids'], batch['mask']).squeeze()
    loss = self.loss(logits, batch['targets']).mean()
    return {'loss': loss, 'log': {'train_loss': loss}}

Same goes for validation_step:
  def validation_step(self,batch, batch_idx):
      logits = self.forward(batch['input_ids'], batch['mask']).squeeze()
      acc = (logits.argmax(-1) == batch['targets']).float()
      loss = self.loss(logits, batch['targets'])
      return {'loss': loss, 'acc': acc}

In the end, the model produces the same probabilities no matter what input sequence it gets. Is this underfitting?
A example is:
model.eval()
text = "Warum kann mein Kollege in SAP keine Transaktionen mehr     ausführen?"
input = tokenizer(
   text,
   None,
   add_special_tokens=True,
   max_length=200,
   pad_to_max_length=True,
   return_token_type_ids=True,
   truncation=True,
   padding='max_length',
   return_tensors="pt"
   )

input = input.to(device)

out = model(input_ids=input['input_ids'], mask=input['attention_mask'])
text, out

    --> ('Warum kann mein Kollege in SAP keine Transaktionen mehr ausführen?',
 tensor([[2.9374e-03, 3.1926e-03, 8.7949e-03, 3.0573e-01, 2.6428e-04, 5.2946e-02,
      2.4758e-01, 6.2161e-03, 3.6930e-01, 3.0384e-03]], device='cuda:0',
    grad_fn=<SoftmaxBackward>))

And another example is:
    ('Auf meinem Telefon erscheinen keine Nummern mehr.',
 tensor([[2.9374e-03, 3.1926e-03, 8.7949e-03, 3.0573e-01, 2.6428e-04, 5.2946e-02,
          2.4758e-01, 6.2161e-03, 3.6930e-01, 3.0384e-03]], device='cuda:0',
        grad_fn=<SoftmaxBackward>))

It is german, but as you can see, the predictions match exactly. Which is a shame :D And my problem.


title: Aspect based sentiment analysis

body:
I've read a paper called "A multi-task learning model for chinese-oriented aspect polarity classification and aspect term extraction" proposed by Yang et. al (2019).
The code is available here:
https://github.com/yangheng95/LCF-ATEPC
The code works with the SemEval 2014 Dataset but when I try to use my dataset I have a problem to convert it to the structure, which works with code.
this my dataset (json file) with the following structure https://github.com/afi1289/LCF-ATEPC/blob/master/atepc_datasets/restaurant/restaurant_sentences_for_ABSA.json:
"98497_5": {"true token": ["Mussels", "-", "So", "much", "flavor", "!"], "label": ["BA", "O", "BS", "IS", "IS", "O"], "reference": ["", "", "1", "", "", ""], "polarity": ["", "", "2", "", "", ""], "aspect category": ["f", "", "", "", "", ""], "modifier": [[], [], ["STR"], [], [], []], "sentence": "Mussels - So much flavor!"}
Only the following structure works with LCF-ATEPC Model:
But O -1
the O -1
staff B-ASP 0
was O -1
so O -1
horrible O -1
to O -1
us O -1
. O -1
we've tried to convert it, then we get:
Mussels B-ASP 2

O -1
So O -1
much O -1
flavor O -1
! O -1

Nevertheless, the LCF-ATEPC model could not starts to train?
How I could convert my json Dataset to be able to work with LCF-ATEPC model?
Thank you in advance!


title: Element wise batch matrix multiplication of a row with every other row in matrix, in PyTorch

body:
I have two matrices of sizes (30, 24, 512) respectively where 30 is the batch size. Let us call them A and B.
Now what I need to do is this:
For every batch in A, I want to compute element-wise batch matrix multiplication of each row in a single batch of A with each row in a single batch of B and sum them.
In other words, for every batch, I have a (24, 512) matrix on left-hand side (A) and on right-hand side (B). Now for each row (1 * 512) in A, I want to compute element-wise matrix multiplication of that row with each of the (24 * 512) rows in B, one by one, and sum them.
This operation would thus result in a (1 * 512) sized vector.
Done for each row in A, it would result in a (24 * 512) sized matrix and then, done for each batch it would result in (30, 24, 512) sized matrix, which is the preferred dimension of my output.
How do I go about doing this?
I do not want to use for loops, since that would be inefficient. I also do not want to use repeat method for the same reason.


title: TypeError: 'NoneType' object is not callable while use the model

body:
I'm using PyTorch for training GCN, I have a simple model that saved using this command:
torch.save(MyNet().state_dict(), PATH)

Then I load it by doing the following:
model = MyNet()
model.load_state_dict(torch.load(PATH))
model.eval()

But then when I tried to input the data as follows:
output = model(dataset)

I got this error for the previous line:
TypeError: 'NoneType' object is not callable

I don't know what's the reason for that, could you please help me?


title: PyTorch: How to multiply elements in a list containing tuples of integer and tensor

body:
I have the following list ABC_lst containing tuples of an integer and a tensor.
Code:
import numpy as np
import torch

A_int = 40
A_tsr = torch.tensor(np.array([[1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5], [1,2,3,4,5]]))
A_tpl = (A_int, A_tsr)

B_int = 42
B_tsr = torch.tensor(np.array([[4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8], [4,5,6,7,8]]))
B_tpl = (B_int, B_tsr)

C_int = 38
C_tsr = torch.tensor(np.array([[7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11], [7,8,9,10,11]]))
C_tpl = (C_int, C_tsr)

ABC_lst = [A_tpl, B_tpl, C_tpl]
ABC_lst

Output:
[(40, tensor([[1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5]])),
 (42, tensor([[4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8]])),
 (38, tensor([[ 7,  8,  9, 10, 11],
          [ 7,  8,  9, 10, 11],
          [ 7,  8,  9, 10, 11],
          [ 7,  8,  9, 10, 11],
          [ 7,  8,  9, 10, 11]]))]

How do I multiply the integer with the corresponding tensor, for eg. multiply 40 with
tensor([[1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5],
          [1, 2, 3, 4, 5]])

multiply 42 with
tensor([[4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8],
          [4, 5, 6, 7, 8]])

and so on.
The returned object should be a tensor, which looks like this:
tensor([[[ 40.,  80., 120., 160., 200.],
         [ 40.,  80., 120., 160., 200.],
         [ 40.,  80., 120., 160., 200.],
         [ 40.,  80., 120., 160., 200.],
         [ 40.,  80., 120., 160., 200.]],

        [[168., 210., 252., 294., 336.],
         [168., 210., 252., 294., 336.],
         [168., 210., 252., 294., 336.],
         [168., 210., 252., 294., 336.],
         [168., 210., 252., 294., 336.]],

        [[266., 304., 342., 380., 418.],
         [266., 304., 342., 380., 418.],
         [266., 304., 342., 380., 418.],
         [266., 304., 342., 380., 418.],
         [266., 304., 342., 380., 418.]]])

In the above eg., I have 3 "sets" of integer and tensor. How do I generalize a code for the multiplication above for any arbitrary "sets" of integer and tensor?
Would really appreciate it if anyone could help.
EDIT: I need to do all the above in GPU, so need to work with tensors.


title: calculate cosine similarity in Pytorch

body:
I have a candidate document embedding tensor, namely cdd_doc_embeddings of size [batch_size, cdd_size, signal_length, embedding_dim], a history clicked document embedding tensor, namely his_doc_embeddings of size [batch_size, his_size, signal_length, embedding_dim].
Now I want to compute the cosine similarity between them, yielding a tensor fusion_matrix of size [batch_size, cdd_size, his_size, signal_length, signal_length] where entry [ b,i,j,u,v ] denotes the cosine similarity between the u th word in i th candidate document in b th batch and the v th word in j th history clicked document in b th batch.
How can I do it efficiently with PyTorch?


title: Does NVLink accelerate training with DistributedDataParallel?

body:
Nvidia's NVLink accelerates data transfer between several GPUs on the same machine.
I train large models on such a machine using PyTorch.
I see why NVLink would make model-parallel training faster, since one pass through a model will involve several GPUs.
But would it accelerate a data-parallel training process using DistributedDataParallel?


title: Legacy torchtext 0.9.0

body: In the latest release of torchtext they moved a lot of features to torchtext.legacy, I want to do the same things like torchtext.legacy.data.Field and other features without using legacy, is that can be done? and how?


title: Non linear mapping to vector of higher dimension

body:
I am learning Keras and need help on the following. I currently have a sequence of floats in lists X and Y. What I need to do is to have a non-linear mapping to map each element to a vector of higher dimension following the below equation.

pos(i) = tanh(W.[concat(X[i],Y[i]])
#where W is a learnable weight matrix, concat performs the concatenation and pos(i) is a vector of 16x1. (I'm trying to create 16 channel inputs for a CNN).

I found that Pytorch implementation for the above is

m = nn.linear(2,16)
input = torch.cat(X[i],Y[i])
torch.nn.functional.tanh(m(input))

Currently I've tried the concat and tanh in numpy and it seems that is not what I want here.

Can you help me implement the above using Keras.


title: How to perform finetuning on a Pytorch net

body:
I'm using this implementation of SegNet in Pytorch, and I want to finetune it.
I've read online and I've found this method (basically freezing all layers except the last one in your net). My problem is that SegNet has more than 100 layers and I'm looking for a simpler way to do it, rather than writing 100 lines of code.

Do you think this could work? Or is this utter nonsense?

import torch.optim as optim

model = SegNet()
for name, param in model.named_modules():
    if name != 'conv11d':    # the last layer should remain active
        param.requires_grad = False

optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.5)

def train():
    ...

How can I check if this is working as intended?


title: Code conversion from python 2 to python 3

body:
I'm setting up a new algorithm which combines an object detector(bounding box detector) which is in python 3 and a mask generator which is in python 2. The problem here is I have several python 2 files which is required for the mask generation algorithm. So I tried 2to3 to convert all my python 2 files to python 3. The script seemed like working but as it was a deep learning algorithm(for mask generation when bounding box coordinates are given as input) which needs some pytorch weights to be loaded, while testing the model in python 3 the program was throwing out error like

  "RuntimeError: Expected object of type torch.FloatTensor but found
  type torch.cuda.FloatTensor for argument #2 ‘weight’"

I have searched in PyTorch forums but none of the posts were useful to me. Is it because my mask generation code is trained in python 2 ?

Does that means while loading the weights and testing the model I should use python 2 not python 3 ? It would be great if someone can shed some light on this. As a work around I can still use the object detector code downgraded to python 2. But still I want to know why it was throwing the error.


title: Keyword arguments in torch.nn.Sequential (pytroch)

body:
a question regarding keywords in torch.nn.Sequential, it is possible in some way to forward keywords to specific models in a sequence?

    model = torch.nn.Sequential(model_0, MaxPoolingChannel(1))
    res = model(input_ids_2, keyword_test=mask)

here, keyword_test should be forwarded only to the first model.

Thank a lot and best regards!

my duplicate from - https://discuss.pytorch.org/t/keyword-arguments-in-torch-nn-sequential/53282


title: How can I replace the entries of a tensor with the appropriate ranking of each entry?

body:
suppose that I have the following tensor:

>> i = 3
>> j = 5
>> k = 2
>> sor = torch.randn(i,j,k)

>> sor
Out[20]:
tensor([[[ 0.5604, -0.9675],
         [-1.0953, -0.5615],
         [ 0.4250, -0.9176],
         [-1.6188, -1.0217],
         [-0.0778,  1.9407]],

        [[-0.1034, -0.7925],
         [-0.2955,  0.8058],
         [-0.5349,  1.1040],
         [ 1.1240,  0.8249],
         [ 0.0827, -1.2471]],

        [[ 0.5924,  0.4777],
         [-2.4640, -1.9527],
         [-0.4519,  0.4788],
         [-0.2308, -0.2368],
         [-1.6786,  0.1360]]])

suppose that for every fixed i and j, I want to compute the numeric rank of elements across k, and replace the elements of the tensor sor with those ranks. For instance, from the example above, I want to change the entry [ 0.5604, -0.9675], which is sor[0,0,:] , into [1, 2], since 0.5604 > -0.9675

Thank you,


title: How to compare one picture to all data test in siamese neural network?

body:
I've been build siamese neural network using pytorch. But I've just test it by inserting 2 pictures and calculate the similarity score, where 0 says that picture is different and 1 says picture is same.

import numpy as np
import os, sys
from PIL import Image
dir_name = "/Users/tania/Desktop/Aksara/Compare" #this should contain 26 images only
X = []
for i in os.listdir(dir_name):
    if ".PNG" in i:
        X.append(torch.from_numpy(np.array(Image.open("./Compare/" + i))))

x1 = np.array(Image.open("/Users/tania/Desktop/Aksara/TEST/Ba/B/B.PNG"))
x1 = transforms(x1)
x1 = torch.from_numpy(x1)

#x1 = torch.stack([x1])

closest = 0.0  #highest similarity
closest_letter_idx = 0  #index of closest letter 0=A, 1=B, ...
cnt = 0

for i in X:
    output = model(x1,i) #assuming x1 is your input image
    output = torch.sigmoid(output)
    if output > closest:
        closest_letter_idx = cnt
        closest = output
    cnt += 1

Both pictures are different, so the output

  File "test.py", line 83, in <module>
    X.append(torch.from_numpy(Image.open("./Compare/" + i)))
TypeError: expected np.ndarray (got PngImageFile)

this is the directory


title: training and evaluating an stacked auto-encoder model in pytorch

body:
I am trying to train a model in pytorch.

input: 686-array
first layer: 64-array
second layer: 2-array
output: predition either 1 or 0

this is what I have so far:

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder_softmax = nn.Sequential(
            nn.Linear(686, 256),
            nn.ReLU(True),
            nn.Linear(256, 2),
            nn.Softmax()
        )

    def forward(self, x):
        x = self.encoder_softmax(x)
        return x

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(device)

net = net.to(device)

iterations = 10
learning_rate = 0.98
criterion = nn.CrossEntropyLoss()

optimizer = torch.optim.Adam(
    net.parameters(), lr=learning_rate, weight_decay=1e-5)

for epoch in range(iterations):
    loss = 0.0
    print("train_dl len: ", len(train_dl))

    # net.train()
    for i, data in enumerate(train_dl, 0):
        inputs, labels, vectorize = data

        labels = labels.long().to(device)
        inputs = inputs.float().to(device)
        optimizer.zero_grad()
        outputs = net(inputs)

        train_loss = criterion(outputs, labels)

        train_loss.backward()
        optimizer.step()

        loss += train_loss.item()

    loss = loss / len(train_dl)

but when I train the model, the loss is not going down. What am I doing wrong?


title: PyTorch: Differentiable operations to go from coordinate tensor to grid tensor

body:
I have a tensor that looks like
coords = torch.Tensor([[0, 0, 1, 2],
                       [0, 2, 2, 2]])

The first row is the x-coordinates of objects on a grid and the second row is the corresponding y-coordinates.
I need a differentiable way (i.e. gradients can flow) to go from this tensor to the corresponding "grid" tensor, where a 1 represents the presence of an object in that location (row index, column index) and 0 represents no object:
grid = torch.Tensor([[1, 0, 1],
                     [0, 0, 1],
                     [0, 0, 1]])

In general, coords can be large (the grid size is 300x300). If coords was a sparse tensor I could simply call to_dense on it, but for various reasons specific to my application I cannot store coords as sparse. Additionally, I cannot create a new sparse tensor from coords and call to_dense on it because creating a new tensor is not differentiable.
Any help is appreciated!


title: Pytorch image view invalid for input of size

body:
Data:
horses_and_humans

train

horses (500 image)
humans (500 image)

test

horses (128 image)
humans (128 image)

Problem:
I am trying to implement the ResNet method with Pytorch. I successfully implemented this with another dataset in the ResNet training video. But in this dataset, I started to get the following error in the last part.
batch_size = 2000

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(trainloader):
        print(images.size())
        images = images.view(batch_size, 64, 64)
        images = images.float()

torch.Size([655, 4096])
---> 11         images = images.view(batch_size, 64, 64)
RuntimeError: shape '[2000, 64, 64]' is invalid for input of size 2682880


title: Pytorch: Differentiable counting

body: I need to count the number of elements in a tensor that satisfy a condition, such as counting the number of people with age == 60, or people with age >= 50. Is there a differentiable approximation to the counting function?


title: Fluctuating loss during training for text binary classification

body:
I'm doing a finetuning of a Longformer on a document text binary classification task using Huggingface Trainer class and I'm monitoring the measures of some checkpoints with Tensorboard.
Even if the F1 score and accuracy is quite high, I have perplexities about the fluctuations of training loss.
I read online a reason for that can be:

the too high learning rate, but I tried with 3 values (1e-4, 1e-5 and 1e-6) and all of them made the same effect
a small batch size. I'm using a Sagemaker notebook p2.8xlarge which has 8xK80 GPUs. The batch size per GPU I can use to avoid the CUDA out of memory error is 1. So the total batch size is 8. My intuition is that a bs of 8 is too small for a dataset containing 57K examples (7K steps per epoch). Unfortunately it's the highest value I can use.

Here I have reported the trend of F1, accuracy, loss and smoothed loss. The grey line is with 1e-6 of learning rate while the pink one is 1e-5.

I reasume all the info of my training:

batch size: 1 x 8GPU = 8
learning rate: 1e-4, 1e-5, 1e-6 (all of them tested without improvement on loss)
model: Longformer
dataset:

training set: 57K examples
dev set: 12K examples
test set: 12K examples

Which could be the reason? Can this be considered a problem despite the quite good F1 and accuracy results?