title: Getting nans after applying softmax

body:
I am trying to develop a deep Markov Model using following tutorial:

https://pyro.ai/examples/dmm.html

This model parameterises transitions and emissions with using a neural network and for variational inference part they use RNN to map observable 'x' to latent space. And in order to ensure that their model is learning something they try to maximise ELBO or minimise negative ELBO. They refer to negative ELBO as NLL.

I am basically converting pyro code to pure pytorch. And I've put together pretty much every thing. However, now I want to get a reconstruct error on test sequences. My input is one hot encoding of my sequences and it is of shape (500,1900,4) where 4 refers to number of features and 1900 is sequence length, and 500 refers to total number of examples.

The way I am generating data is:

generation model p(x_{1:T} | z_{1:T}) p(z_{1:T})

batch_size, _, x_dim = x.size() # number of time steps we need to process in the mini-batch
T_max = x_lens.max()
z_prev = self.z_0.expand(batch_size, self.z_0.size(0)) # set z_prev=z_0 to setup the recursive conditioning in p(z_t|z_{t-1})
for t in range(1, T_max + 1):
    # sample z_t ~ p(z_t | z_{t-1}) one time step at a time
    z_t, z_mu, z_logvar = self.trans(z_prev) # p(z_t | z_{t-1})
    p_x_t = F.softmax(self.emitter(z_t),dim=-1)  # compute the probabilities that parameterize the bernoulli likelihood
    safe_tensor = torch.where(torch.isnan(p_x_t), torch.zeros_like(p_x_t), p_x_t)
    print('generate p_x_t : ',safe_tensor)
    x_t = torch.bernoulli(safe_tensor) #sample observe x_t according to the bernoulli distribution p(x_t|z_t)
    print('generate x_t : ',x_t)
    z_prev = z_t

So my emissions are being defined by Bernoulli distribution. And I use softmax in order to compute probabilities that parameterise the Bernoulli likelihood. And then I sample my observables x_t according the Bernoulli distribution.

At first when I ran my model, I was getting nans at times and so I introduced the line (show below) in order to convert nans to zero:

safe_tensor = torch.where(torch.isnan(p_x_t), torch.zeros_like(p_x_t), p_x_t)

However, after 1 epoch or so 'x_t' that I sample, this tensor is just zero. Basically, what I want is that after applying softmax, I want my function to pick the highest probability and give me the corresponding label for it which is either of the 4 features. But I get nans and then when I convert all nans to zero, I end up getting all zeros in all tensors after 1 epoch or so.

Also, when I look at p_x_t tensor I get probabilities that add up to one. But when I look at x_t tensor it gives me 0's all the way. So for example:

p_x_t : tensor([[0.2168, 0.2309, 0.2555, 0.2967]
.....], device='cuda:0',
       grad_fn=<SWhereBackward>)

generate x_t : tensor([[0., 0., 0., 0.],...
], device='cuda:0', grad_fn=<BernoulliBackward0>)

Here fourth label/feature is giving me highest probability. Shouldn't the x_t tensor give me 1 at least in that position so be like:

generate x_t : tensor([[0., 0., 0., 1.],...
], device='cuda:0', grad_fn=<BernoulliBackward0>)

How can I get rid of this problems?

EDIT

My transition (which is called as self.trans in generate function mentioned above):

class GatedTransition(nn.Module):
    """
    Parameterizes the gaussian latent transition probability `p(z_t | z_{t-1})`
    See section 5 in the reference for comparison.
    """
    def __init__(self, z_dim, trans_dim):
        super(GatedTransition, self).__init__()
        self.gate = nn.Sequential(
            nn.Linear(z_dim, trans_dim),
            nn.ReLU(),
            nn.Linear(trans_dim, z_dim),
            nn.Softmax(dim=-1)
        )
        self.proposed_mean = nn.Sequential(
            nn.Linear(z_dim, trans_dim),
            nn.ReLU(),
            nn.Linear(trans_dim, z_dim)
        )
        self.z_to_mu = nn.Linear(z_dim, z_dim)
        # modify the default initialization of z_to_mu so that it starts out as the identity function
        self.z_to_mu.weight.data = torch.eye(z_dim)
        self.z_to_mu.bias.data = torch.zeros(z_dim)
        self.z_to_logvar = nn.Linear(z_dim, z_dim)
        self.relu = nn.ReLU()

    def forward(self, z_t_1):
        """
        Given the latent `z_{t-1}` corresponding to the time step t-1
        we return the mean and scale vectors that parameterize the (diagonal) gaussian distribution `p(z_t | z_{t-1})`
        """
        gate = self.gate(z_t_1) # compute the gating function
        proposed_mean = self.proposed_mean(z_t_1) # compute the 'proposed mean'
        mu = (1 - gate) * self.z_to_mu(z_t_1) + gate * proposed_mean # compute the scale used to sample z_t, using the proposed mean from
        logvar = self.z_to_logvar(self.relu(proposed_mean))
        epsilon = torch.randn(z_t_1.size(), device=z_t_1.device) # sampling z by re-parameterization
        z_t = mu + epsilon * torch.exp(0.5 * logvar)    # [batch_sz x z_sz]

        if torch.isinf(z_t).any().item():
            print('something is infinity')
            print('z_t : ',z_t)
            print('logvar : ',logvar)
            print('epsilon : ',epsilon)
            print('mu : ',mu)

        return z_t, mu, logvar

I do not get any inf in z_t tensor when doing training and validation. Only during testing. This is how I am training, validating and testing my model:

    for epoch in range(config['epochs']):

        train_loader=torch.utils.data.DataLoader(dataset=train_set, batch_size=config['batch_size'], shuffle=True, num_workers=1)
        train_data_iter=iter(train_loader)
        n_iters=train_data_iter.__len__()

        epoch_nll = 0.0 # accumulator for our estimate of the negative log likelihood (or rather -elbo) for this epoch
        i_batch=1
        n_slices=0
        loss_records={}
        while True:
            try: x, x_rev, x_lens = train_data_iter.next()
            except StopIteration: break # end of epoch
            x, x_rev, x_lens = gVar(x), gVar(x_rev), gVar(x_lens)

            if config['anneal_epochs'] > 0 and epoch < config['anneal_epochs']: # compute the KL annealing factor
                min_af = config['min_anneal']
                kl_anneal = min_af+(1.0-min_af)*(float(i_batch+epoch*n_iters+1)/float(config['anneal_epochs']*n_iters))
            else:
                kl_anneal = 1.0 # by default the KL annealing factor is unity

            loss_AE = model.train_AE(x, x_rev, x_lens, kl_anneal)

            epoch_nll += loss_AE['train_loss_AE']
            i_batch=i_batch+1
            n_slices=n_slices+x_lens.sum().item()

        loss_records.update(loss_AE)
        loss_records.update({'epo_nll':epoch_nll/n_slices})
        times.append(time.time())
        epoch_time = times[-1] - times[-2]
        log("[Epoch %04d]\t\t(dt = %.3f sec)"%(epoch, epoch_time))
        log(loss_records)
        if args.visual:
            for k, v in loss_records.items():
                tb_writer.add_scalar(k, v, epoch)

        # do evaluation on test and validation data and report results
        if (epoch+1) % args.test_freq == 0:
            save_model(model, epoch)
            #test_loader=torch.utils.data.DataLoader(dataset=test_set, batch_size=config['batch_size'], shuffle=False, num_workers=1)
            valid_loader=torch.utils.data.DataLoader(dataset=valid_set, batch_size=config['batch_size'], shuffle=False, num_workers=1)

            for x, x_rev, x_lens in valid_loader:
                x, x_rev, x_lens = gVar(x), gVar(x_rev), gVar(x_lens)
                loss,kl_loss = model.valid(x, x_rev, x_lens)
                #print('x_lens sum : ',x_lens.sum().item())
                #print('loss : ',loss)
                valid_nll = loss/x_lens.sum().item()
            log("[Epoch %04d]\t\t(valid_loss = %.8f)\t\t(kl_loss = %.8f)\t\t(valid_nll = %.8f)"%(epoch, loss, kl_loss, valid_nll ))

    test_loader=torch.utils.data.DataLoader(dataset=test_set, batch_size=config['batch_size'], shuffle=False, num_workers=1)
    for x, x_rev, x_lens in test_loader:
        x, x_rev, x_lens = gVar(x), gVar(x_rev), gVar(x_lens)
        loss,kl_loss = model.valid(x, x_rev, x_lens)
        test_nll = loss/x_lens.sum().item()
        model.generate(x, x_rev, x_lens)
        log("[test_nll epoch %08d]  %.8f" % (epoch, test_nll))

The test is outside the for loop for epoch. Because I want to test my model on test sequences using generate function. Can you now help me understand if there's something wrong I am doing?


title: how is the batch size determined?

body:
I'm looking at this pytorch starter tutorial:

  https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html#sphx-glr-beginner-blitz-neural-networks-tutorial-py

the zero_grad() function is being used to zero the gradients which means that it's running with mini-batches, is this a correct assumption? If so, where is the batch size defined??

I found the following for nn.conv2d:

For example, nn.Conv2d will take in a 4D Tensor of nSamples x nChannels x Height x Width.

in that case nSamples is the batch size?

but how do you specify the batch size for a nn.Linear layer? do you decide what your mini-batches are when you load the data or what?

I am making a few assumptions here that may be totally incorrect, pls correct me if i'm wrong.
thank you!


title: Speech Recognition - how to split a sentence into words?

body:
I'm new to Speech Recognition, and I'm looking for an approach to split a sentence (or multiple sentences) in the form of audio/wav files, into individual words? This sounds like a standard problem, so I'm wondering how people in the industry approach it.

ps: yes this question was asked three years ago, but I'm looking for an up-to-date answer using newer libraries (i.e. pytorch and tensorflow 2.0). Thanks!


title: Pytorch - Should backward() function be in the loop of epoch or batch?

body:
When training nn models using Pytorch, is there a difference regarding where we place the backward method? For example, which one of below is correct?

Calculate gradient across the batch:

for e in range(epochs):
    for i in batches_list:
        out = nn_model(i)
        loss = loss_function(out, actual)
        loss_sum += loss.item()
        lstm.zero_grad()
        loss.backward()
        optimizer.step()
loss_list.append(loss_sum / num_train_obs)

Calculate gradient across the epoch:

for e in range(epochs):
    for i in batches_list:
        out = nn_model(i)
        loss = loss_function(out, actual)
        loss_sum += loss.item()
    lstm.zero_grad()
    loss_sum.backward()
    optimizer.step()
loss_list.append(loss_sum / num_train_obs)


title: Getting model class labels from torchvision pretrained models

body:
I am using a pre-trained Alexnet model (without fine-tuning) from torchvision. The issue is that even though I am able to run the model on some data and get the output probability distribution, I am unable to find class labels to map it to.
Following this official documentation
import torch
model = torch.hub.load('pytorch/vision:v0.6.0', 'alexnet', pretrained=True)
model.eval()

AlexNet(
  (features): Sequential(
    (0): Conv2d(3, 64, kernel_size=(11, 11), stride=(4, 4), padding=(2, 2))
    (1): ReLU(inplace=True)
    (2): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (3): Conv2d(64, 192, kernel_size=(5, 5), stride=(1, 1), padding=(2, 2))
    (4): ReLU(inplace=True)
    (5): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
    (6): Conv2d(192, 384, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (7): ReLU(inplace=True)
    (8): Conv2d(384, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (9): ReLU(inplace=True)
    (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (11): ReLU(inplace=True)
    (12): MaxPool2d(kernel_size=3, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (avgpool): AdaptiveAvgPool2d(output_size=(6, 6))
  (classifier): Sequential(
    (0): Dropout(p=0.5, inplace=False)
    (1): Linear(in_features=9216, out_features=4096, bias=True)
    (2): ReLU(inplace=True)
    (3): Dropout(p=0.5, inplace=False)
    (4): Linear(in_features=4096, out_features=4096, bias=True)
    (5): ReLU(inplace=True)
    (6): Linear(in_features=4096, out_features=1000, bias=True)
  )
)

Following some steps on processing the image, I am able to use it to get an output for a single image as a (1,1000) dim vector which I will use a softmax on to get a probability distribution -
#Output -

tensor([-1.6531e+00, -4.3505e+00, -1.8172e+00, -4.2143e+00, -3.1914e+00,
         3.4163e-01,  1.0877e+00,  5.9350e+00,  8.0425e+00, -7.0242e-01,
        -9.4130e-01, -6.0822e-01, -2.4097e-01, -1.9946e+00, -1.5288e+00,
        -3.2656e+00, -5.5800e-01,  1.0524e+00,  1.9211e-01, -4.7202e+00,
        -3.3880e+00,  4.3048e+00, -1.0997e+00,  4.6132e+00, -5.7404e-03,
        -5.3437e+00, -4.7378e+00, -3.3974e+00, -4.1287e+00,  2.9064e-01,
        -3.2955e+00, -6.7051e+00, -4.7232e+00, -4.1778e+00, -2.1859e+00,
        -2.9469e+00,  3.0465e+00, -3.5882e+00, -6.3890e+00, -4.4203e+00,
        -3.3685e+00, -5.0983e+00, -4.9006e+00, -5.5235e+00, -3.7233e+00,
        -4.0204e+00,  2.6998e-01, -4.4702e+00, -5.6617e+00, -5.4880e+00,
        -2.6801e+00, -3.2129e+00, -1.6294e+00, -5.2289e+00, -2.7495e+00,
        -2.6286e+00, -1.8206e+00, -2.3196e+00, -5.2806e+00, -3.7652e+00,
        -3.0987e+00, -4.1421e+00, -5.2531e+00, -4.6505e+00, -3.5815e+00,
        -4.0189e+00, -4.0008e+00, -4.5512e+00, -3.2248e+00, -7.7903e+00,
        -1.4484e+00, -3.8347e+00, -4.5611e+00, -4.3681e+00,  2.7234e-01,
        -4.0162e+00, -4.2136e+00, -5.4524e+00,  1.1744e+00, -4.7785e+00,
        -1.8335e+00,  4.1288e-01,  2.2239e+00, -9.9919e-02,  4.8216e+00,
        -8.4304e-01,  5.6911e-01, -4.0484e+00, -3.3013e+00,  2.8698e+00,
        -1.1419e+00, -9.1690e-01, -2.9284e+00, -2.6097e+00, -1.8213e-01,
        -2.5429e+00, -2.1095e+00,  2.2419e+00, -1.6280e+00,  7.4458e+00,
         2.3184e+00, -5.7408e+00, -7.4332e-01, -5.4066e+00,  1.5177e+01,
        -4.4737e-02,  1.8237e+00, -3.7741e+00,  9.2271e-01, -4.3687e-01,
        -1.4003e+00, -4.3026e+00,  6.3782e-01, -1.0808e+00, -1.4173e+00,
         2.6194e+00, -3.8418e+00,  1.1598e+00, -2.6876e+00, -3.6103e+00,
        -4.9281e+00, -4.1411e+00, -3.3603e+00, -3.4296e+00, -1.4997e+00,
        -2.8381e+00, -1.2843e+00,  1.5745e+00, -1.7449e+00,  4.2903e-01,
         3.1234e-01, -2.8206e+00,  3.6688e-01, -2.1033e+00,  1.6481e+00,
         1.4222e+00, -2.7303e+00, -3.6292e+00,  1.2864e+00, -2.5541e+00,
        -2.9663e+00, -4.1575e+00, -3.1954e+00, -4.6487e-01,  1.8916e+00,
        -7.4721e-01,  4.5986e+00, -2.5443e+00, -6.2003e+00, -1.3215e+00,
        -2.6225e+00,  9.9639e+00,  9.7772e+00,  9.6715e+00,  9.0857e+00,...

Where do I get the class labels from? I couldn't find any method that let me get that from the model object.


title: Check the number of parameters from state_dict in pytorch

body:
SO has an answer about how to check the total # of params from the model:
pytorch_total_params = sum(p.numel() for p in model.parameters())

However, how does one check the total # of params from
the state_dict?

state_dict = torch.load(model_path, map_location='cpu')?


title: Command line python and jupyter notebooks use two different versions of torch

body:
On my conda environment importing torch from command line Python and from a jupyter notebook yields two different results.

Command line Python:

$ source activate GNN
(GNN) $ python
>>> import torch
>>> print(torch.__file__)
/home/riccardo/.local/lib/python3.7/site-packages/torch/__init__.py
>>> print(torch.__version__)
0.4.1

Jupyter:

(GNN) $ jupyter notebook --no-browser --port=8890

import torch

print(torch.__file__)
/home/riccardo/.local/lib/python3.6/site-packages/torch/__init__.py

print(torch.__version__)
1.2.0+cu92

I tried the steps suggested in Conda environments not showing up in Jupyter Notebook

$ conda install ipykernel
$ source activate GNN
(GNN) $ python -m ipykernel install --user --name GNN --display-name "Python (GNN)"
Installed kernelspec GNN in /home/riccardo/.local/share/jupyter/kernels/gnn

but that did not solve the problem.


title: Confusion in Pre-processing text for Roberta Model

body:
I want to apply Roberta model for  text similarity. Given a pair of sentences,the input should be in the format <s> A </s></s> B </s>. I figure out two possible ways to generate the input ids namely

a)

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('roberta-base')

list1 = tokenizer.encode('Very severe pain in hands')

list2 = tokenizer.encode('Numbness of upper limb')

sequence = list1+[2]+list2[1:]

In this case, sequence is [0, 12178, 3814, 2400, 11, 1420, 2, 2, 234, 4179, 1825, 9, 2853, 29654, 2]

b)

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained('roberta-base')

list1 = tokenizer.encode('Very severe pain in hands', add_special_tokens=False)

list2 = tokenizer.encode('Numbness of upper limb', add_special_tokens=False)

sequence = [0]+list1+[2,2]+list2+[2]

In this case, sequence is [0, 25101, 3814, 2400, 11, 1420, 2, 2, 487, 4179, 1825, 9, 2853, 29654, 2]

Here 0 represents <s> token and 2 represents </s> token. I'm not sure which is the correct way to encode the given two sentences for calculating sentence similarity using Roberta model.


title: gpu not fully used, model takes same time to run in cpu as in gpu

body:
So Ive been playing with pytorch lately and I end up with a model that seems poorly optimized. The gpu usage is around 30% all the time during training and depending on batch_size the time required to run a epoch can be drastically reduced (if batch_size is high) or long (if batch_size is small).

The full code sits in the following repository https://github.com/jiwidi/las-pytorch.

The training loop is located here and the "juicy" part of the code where training on gpu happens is located here

I also attach here the output of calling my script with torch.utils.bottleneck for 51 minibatches training for both train and validation dataset (a full epoch takes almost a full day, the dataset is 1.000 of Speech2Text)

`bottleneck` is a tool that can be used as an initial step for debugging
bottlenecks in your program.

It summarizes runs of your script with the Python profiler and PyTorch's
autograd profiler. Because your script will be profiled, please ensure that it
exits in a finite amount of time.

For more complicated uses of the profilers, please see
https://docs.python.org/3/library/profile.html and
https://pytorch.org/docs/master/autograd.html#profiler for more information.
Running environment analysis...
Running your script with cProfile
Running your script with the autograd profiler...

--------------------------------------------------------------------------------
  Environment Summary
--------------------------------------------------------------------------------
PyTorch 1.5.0 compiled w/ CUDA 10.2
Running with Python 3.6 and

`pip list` truncated output:
numpy==1.18.1
torch==1.5.0
torchaudio==0.5.0a0+3305d5c
torchvision==0.6.0a0+82fd1c8
--------------------------------------------------------------------------------
  cProfile output
--------------------------------------------------------------------------------
         3139301 function calls (3045612 primitive calls) in 48.917 seconds

   Ordered by: internal time
   List reduced from 2862 to 15 due to restriction <15>

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
       51   31.576    0.619   31.576    0.619 {method 'run_backward' of 'torch._C._EngineBase' objects}
      183    2.985    0.016    2.985    0.016 {method 'cuda' of 'torch._C._TensorBase' objects}
    11606    2.903    0.000    2.903    0.000 {built-in method lstm}
    11504    1.034    0.000    1.034    0.000 {method 'repeat' of 'torch._C._TensorBase' objects}
    23110    0.907    0.000    0.907    0.000 {built-in method cat}
    11504    0.803    0.000    4.975    0.000 /mnt/04DC6FC8DC6FB31A/github/las-pytorch/model/las_model.py:275(forward)
    23008    0.750    0.000    0.750    0.000 {built-in method addmm}
        3    0.478    0.159    0.478    0.159 {built-in method _cudnn_rnn_flatten_weight}
       51    0.451    0.009    0.451    0.009 {built-in method torch._C._cuda_emptyCache}
      362    0.442    0.001    0.442    0.001 {method 'acquire' of '_thread.lock' objects}
        2    0.402    0.201    0.821    0.411 /mnt/04DC6FC8DC6FB31A/github/las-pytorch/utils/data.py:35(__init__)
    23008    0.384    0.000    0.384    0.000 {built-in method relu}
    11504    0.351    0.000    0.351    0.000 {method 'matmul' of 'torch._C._TensorBase' objects}
    34512    0.342    0.000    1.786    0.000 /home/jiwidi/anaconda3/envs/torch-gpu/lib/python3.6/site-packages/torch/nn/functional.py:1591(linear)
       51    0.304    0.006    9.883    0.194 /mnt/04DC6FC8DC6FB31A/github/las-pytorch/model/las_model.py:186(forward)

--------------------------------------------------------------------------------
  autograd profiler output (CPU mode)
--------------------------------------------------------------------------------
        top 15 events sorted by cpu_time_total

-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------
Name                     Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  Input Shapes
-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------
lstm                     7.66%            26.315ms         7.66%            26.315ms         26.315ms         NaN              0.000us          0.000us          1                []
_cudnn_rnn               7.65%            26.277ms         7.65%            26.277ms         26.277ms         NaN              0.000us          0.000us          1                []
CudnnRnnBackward         7.19%            24.689ms         7.19%            24.689ms         24.689ms         NaN              0.000us          0.000us          1                []
_cudnn_rnn_backward      7.19%            24.672ms         7.19%            24.672ms         24.672ms         NaN              0.000us          0.000us          1                []
lstm                     7.08%            24.296ms         7.08%            24.296ms         24.296ms         NaN              0.000us          0.000us          1                []
_cudnn_rnn               7.07%            24.258ms         7.07%            24.258ms         24.258ms         NaN              0.000us          0.000us          1                []
lstm                     6.69%            22.980ms         6.69%            22.980ms         22.980ms         NaN              0.000us          0.000us          1                []
_cudnn_rnn               6.68%            22.943ms         6.68%            22.943ms         22.943ms         NaN              0.000us          0.000us          1                []
lstm                     6.38%            21.893ms         6.38%            21.893ms         21.893ms         NaN              0.000us          0.000us          1                []
_cudnn_rnn               6.37%            21.854ms         6.37%            21.854ms         21.854ms         NaN              0.000us          0.000us          1                []
lstm                     6.19%            21.240ms         6.19%            21.240ms         21.240ms         NaN              0.000us          0.000us          1                []
_cudnn_rnn               6.18%            21.202ms         6.18%            21.202ms         21.202ms         NaN              0.000us          0.000us          1                []
CudnnRnnBackward         5.92%            20.315ms         5.92%            20.315ms         20.315ms         NaN              0.000us          0.000us          1                []
_cudnn_rnn_backward      5.91%            20.291ms         5.91%            20.291ms         20.291ms         NaN              0.000us          0.000us          1                []
lstm                     5.85%            20.087ms         5.85%            20.087ms         20.087ms         NaN              0.000us          0.000us          1                []
-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------
Self CPU time total: 343.311ms
CUDA time total: 0.000us

--------------------------------------------------------------------------------
  autograd profiler output (CUDA mode)
--------------------------------------------------------------------------------
        top 15 events sorted by cpu_time_total

    Because the autograd profiler uses the CUDA event API,
    the CUDA time column reports approximately max(cuda_time, cpu_time).
    Please ignore this output if your code does not use CUDA.

-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------
Name                     Self CPU total %  Self CPU total   CPU total %      CPU total        CPU time avg     CUDA total %     CUDA total       CUDA time avg    Number of Calls  Input Shapes
-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------
CudnnRnnBackward         7.13%            25.220ms         7.13%            25.220ms         25.220ms         7.13%            25.248ms         25.248ms         1                []
_cudnn_rnn_backward      7.12%            25.187ms         7.12%            25.187ms         25.187ms         7.13%            25.228ms         25.228ms         1                []
lstm                     7.05%            24.953ms         7.05%            24.953ms         24.953ms         7.07%            25.008ms         25.008ms         1                []
CudnnRnnBackward         7.01%            24.826ms         7.01%            24.826ms         24.826ms         6.97%            24.672ms         24.672ms         1                []
_cudnn_rnn               7.01%            24.814ms         7.01%            24.814ms         24.814ms         7.03%            24.880ms         24.880ms         1                []
_cudnn_rnn_backward      7.00%            24.792ms         7.00%            24.792ms         24.792ms         6.96%            24.648ms         24.648ms         1                []
lstm                     7.00%            24.772ms         7.00%            24.772ms         24.772ms         7.00%            24.770ms         24.770ms         1                []
_cudnn_rnn               6.98%            24.708ms         6.98%            24.708ms         24.708ms         6.98%            24.709ms         24.709ms         1                []
lstm                     6.47%            22.903ms         6.47%            22.903ms         22.903ms         6.47%            22.901ms         22.901ms         1                []
lstm                     6.46%            22.866ms         6.46%            22.866ms         22.866ms         6.46%            22.864ms         22.864ms         1                []
_cudnn_rnn               6.45%            22.814ms         6.45%            22.814ms         22.814ms         6.45%            22.816ms         22.816ms         1                []
_cudnn_rnn               6.44%            22.798ms         6.44%            22.798ms         22.798ms         6.44%            22.798ms         22.798ms         1                []
lstm                     5.97%            21.120ms         5.97%            21.120ms         21.120ms         5.98%            21.180ms         21.180ms         1                []
lstm                     5.97%            21.114ms         5.97%            21.114ms         21.114ms         5.97%            21.112ms         21.112ms         1                []
_cudnn_rnn               5.95%            21.062ms         5.95%            21.062ms         21.062ms         5.95%            21.064ms         21.064ms         1                []
-----------------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  ---------------  -----------------------------------
Self CPU time total: 353.950ms
CUDA time total: 353.898ms

Self CPU time total: 353.950ms
CUDA time total: 353.898ms

By looking at the results seems like it takes the same time to train on cpu mode than with cuda (with a rtx 2060). There must be something wrong..

Im failry new to pytorch so I apreciate any help with help to debug the cpu bottlenecks my code could have.

Thanks


title: How to create a copy of nn.Sequential in torch?

body:
I am trying to create a copy of a nn.Sequential network. For example, the following is the easiest way to do the same-
net = nn.Sequential(
        nn.Conv2d(16, 32, 3, stride=2),
        nn.ReLU(),
        nn.Conv2d(32, 64, 3, stride=2),
        nn.ReLU(),
    )

net_copy = nn.Sequential(
        nn.Conv2d(16, 32, 3, stride=2),
        nn.ReLU(),
        nn.Conv2d(32, 64, 3, stride=2),
        nn.ReLU(),
    )

However, it is not so great to define the network again. I tried the following ways but it didn't work-

net_copy = nn.Sequential(net): In this approach, it seems that net_copy is just a shared pointer of net
net_copy = nn.Sequential(*net.modules()): In this approach, net_copy contains many more layers.

Finally, I tired deepcopy in the following way which worked fine-
net_copy = deepcopy(net)

However, I am wondering if it is the proper way. I assume it is fine because it works.


title: Best way to bound outputs from neural networks on reinforcement learning

body:
I am training a neural network (feedforward, Tanh hidden layers) that receives states as inputs and gives actions as outputs. I am following the REINFORCE algorithm for policy-gradient reinforcement learning.

However, I need my control actions to be bounded (let us say from 0-5). Currently the way I am doing this is by using a sigmoid output function and multiplying the output by 5. Although my algorithm has a moderate performance, I find the following drawback from using this “bounding scheme” for the output:

I know for regression (hence I guess for reinforcement learning) a linear output is best, and although the sigmoid has a linear part I am afraid the network has not been able to capture this linear output behaviour correctly, or it captures it way too slowly (as its best performance is for classification, therefore polarizing the output).

I am wondering what other alternatives there are, and maybe some heuristics on the matter.


title: PyTorch Optimization: Is using map function on a torch.tensor without damaging computational graphs possible?

body:
I have multiple output tensors of my neural network and I want to gather them in a single list  at the end of the Forward-Propagation function. For that reason I was using for loop like that:

output = []
append = output.append
for i in range(self.number):
            append(self.out[i](input))

where self.out is a torch.nn.ModuleList of nn.Sequential layers as output of the model, input is input of the model and self.number is number of outputs.

I want to optimize this with built-in map method as follows:

append = output.append
output = map(lambda x: self.out[x](input)), range(0,self.number))

With both functions, outputs are same and training works. But map is somehow not much faster than for loop for self.number = 6. Sometimes even slower.

1-Might there be any wrong steps made by me, which harms the process(like damaging computational graphs) and avoids the optimization?

2-Or can't we expect perfect results from map for small loops ?

3-Would there be any better method to optimize a for loop without damaging the torch data?

Thanks in advance!


title: pytorch: How to do layer wise multiplication?

body:
I have a tensor containing five 2x2 matrices - shape (1,5,2,2), and a tensor containing 5 elements -  shape ([5]). I want to multiply each 2x2 matrix(in the former tensor) with the corresponding value (in the latter tensor). The resultant tensor should be of shape (1,5,2,2). How to do that?

Getting the following error when I run this code

a = torch.rand(1,5,2,2)
print(a.shape)
b = torch.rand(5)
print(b.shape)
mul = a*b

RuntimeError: The size of tensor a (2) must match the size of tensor b (5) at non-singleton dimension 3


title: Conda ModuleNotFound after installing PyTorch

body:
I've had Conda installed and working fine, but after trying to install Pytorch in my base environment, I have been getting ModuleNotFoundError: No module named 'conda'.

This is the installation trace:

$ conda install pytorch torchvision -c pytorch
Collecting package metadata: done
Solving environment: done

## Package Plan ##

  environment location: /Users/yitianzou/anaconda3

  added / updated specs:
    - pytorch
    - torchvision

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    cffi-1.12.2                |   py36hb5b8e2f_1         213 KB
    ------------------------------------------------------------
                                           Total:         213 KB

The following packages will be UPDATED:

  cffi                                1.11.5-py37h6174b99_1 --> 1.12.2-py36hb5b8e2f_1
  olefile                                     0.45.1-py36_0 --> 0.46-py36_0
  pytorch                     0.4.0-py36_cuda0.0_cudnn0.0_1 --> 1.0.1-py3.6_2
  torchvision        pytorch/osx-64::torchvision-0.2.1-py3~ --> pytorch/noarch::torchvision-0.2.2-py_3

The following packages will be DOWNGRADED:

  mkl_fft                              1.0.6-py37hb8a8100_0 --> 1.0.6-py36hb8a8100_0
  mkl_random                           1.0.2-py37h27c97d8_0 --> 1.0.1-py36h5d10147_1
  numpy                               1.15.4-py37h6a91979_0 --> 1.15.4-py36h6a91979_0
  numpy-base                          1.15.4-py37h8a80b8c_0 --> 1.15.4-py36h8a80b8c_0
  pip                                         19.0.3-py37_0 --> 19.0.3-py36_0
  pycparser                                     2.19-py37_0 --> 2.19-py36_0
  python                                   3.7.2-haf84260_0 --> 3.6.8-haf84260_0
  setuptools                                  40.8.0-py37_0 --> 40.8.0-py36_0
  six                                         1.12.0-py37_0 --> 1.12.0-py36_0
  wheel                                       0.33.1-py37_0 --> 0.33.1-py36_0

Proceed ([y]/n)? y

Downloading and Extracting Packages
cffi-1.12.2          | 213 KB    | ################################### | 100%
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
Traceback (most recent call last):
  File "/Users/yitianzou/anaconda3/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

It looks like my conda path is still fine:

$ which conda
/Users/yitianzou/anaconda3/condabin/conda

And my python also looks fine:

$ python --version
Python 3.6.8 :: Anaconda custom (x86_64)

However, whenever I try to use conda I get this:

$ conda env list
Traceback (most recent call last):
  File "/Users/yitianzou/anaconda3/bin/conda", line 12, in <module>
    from conda.cli import main
ModuleNotFoundError: No module named 'conda'

Any ideas on how to approach this issue?


title: PyTorch v1.3 with GPU on Google Colab?

body:
Google colab notebook comes packaged with PyTorch and GPU support -- see this related SO answer, and then you should go change notebook runtime to GPU from CPU (from the top Colab menubar).

Yet trying to train a model from GPyTorch always fails on

/usr/local/lib/python3.6/dist-packages/gpytorch/utils/cholesky.py in psd_safe_cholesky(A, upper, out, jitter)
     19     """
     20     try:
---> 21         L = torch.cholesky(A, upper=upper, out=out)
     22         # TODO: Remove once fixed in pytorch (#16780)
     23         if A.dim() > 2 and A.is_cuda:

RuntimeError: CUDA error: invalid device function

Why?

Relevant info:

import torch
torch.cuda.is_available()

True

print(torch.__version__)
print(gpytorch.__version__)
print(torch.backends.cudnn.version())

1.3.0+cu100
0.3.6
7603


title: SyntaxError: only named arguments may follow *expression on nn.Sequential()

body:
def forward(self, x):
    x = x.view(-1)
    fwd = nn.Sequential(self.linear_1, *self.middle, self.output)
    return fwd(x)

It works python3 but not python2.7. The error is:

  fwd = nn.Sequential(self.linear_1, *self.middle, self.output)
  SyntaxError: only named arguments may follow *expression
  )

What can I convert this code to work on python2.7?


title: what is the pytorch equivalent of a tensorflow linear regression?

body:
I am learning pytorch, that to do a basic linear regression on this data created this way here:
from sklearn.datasets import make_regression

x, y = make_regression(n_samples=100, n_features=1, noise=15, random_state=42)
y = y.reshape(-1, 1)
print(x.shape, y.shape)

plt.scatter(x, y)

I know that using tensorflow this code can solve:
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Dense(units=1, activation='linear', input_shape=(x.shape[1], )))

model.compile(optimizer=tf.keras.optimizers.SGD(lr=0.05), loss='mse')

hist = model.fit(x, y, epochs=15, verbose=0)

but I need to know what the pytorch equivalent would be like, what I tried to do was this:
# Model Class
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.linear = nn.Linear(1,1)

    def forward(self, x):
        x = self.linear(x)
        return x

    def predict(self, x):
        return self.forward(x)

model = Net()

loss_fn = F.mse_loss
opt = torch.optim.SGD(modelo.parameters(), lr=0.05)

# Funcao para treinar
def fit(num_epochs, model, loss_fn, opt, train_dl):

    # Repeat for given number of epochs
    for epoch in range(num_epochs):

        # Train with batches of data
        for xb, yb in train_dl:

            # 1. Generate predictions
            pred = model(xb)

            # 2. Calculate Loss
            loss = loss_fn(pred, yb)

            # 3. Campute gradients
            loss.backward()

            # 4. Update parameters using gradients
            opt.step()

            # 5. Reset the gradients to zero
            opt.zero_grad()

        # Print the progress
        if (epoch+1) % 10 == 0:
            print('Epoch [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, loss.item()))

# Training
fit(200, model, loss_fn, opt, data_loader)

But the model doesn't learn anything, I don't know what I can do anymore.
The input/output dimensions is (1/1)


title: PyTorch TypeError: scatter() missing 1 required positional argument: 'scatter_list'

body:
all. I have a problem with starting a distributed program while using “mpi” as the backend. The program is as follows:

def run(rank, size, hostname):
    print("I am {} of {} in {}".format(rank, size, hostname))
    tensor = torch.zeros(1)
    group=dist.new_group([0,1,2])
    if rank == 0:
        scatter_list=[torch.zeros(1) for _ in range(3)]
        dist.scatter(tensor= tensor, src=0, scatter_list=scatter_list, group=group)
        print("Master has completed Scatter")
    else:
        tensor += 1
        dist.scatter(tensor= tensor, src=0, group=group)
        print("worker has completed scatter")
    print('Rank', rank, 'has data', tensor[0])

def init_process(rank, size, hostname, fn, backend='tcp'):
    dist.init_process_group(backend, rank=rank, world_size=size)
    fn(rank, size,hostname)

if __name__ == "__main__":
    world_size = int(os.environ['OMPI_COMM_WORLD_SIZE'])
    world_rank = int(os.environ['OMPI_COMM_WORLD_RANK'])
    hostname = socket.gethostname()
    p = Process(target = init_process,
                     args=(world_rank, world_size, hostname, run, 'mpi'))
    p.start()
    p.join()

However, when the program starts, it always throws an error like:

  File "mpi_test.py", line 17, in run
dist.scatter(tensor= tensor, src=0, group=group)

TypeError: scatter() missing 1 required positional argument: 'scatter_list'

The error is emitted by ranks 1 and 2 which don’t need argument: ‘scatter_list’.
I tried many ways, but failed. Does anybody know why?
Thanks for your reading.


title: Loading a model with pytorch

body:
I'm having a problem loading my model in the Image Classifier project. First, I saved it:

model.class_to_idx = train_data.class_to_idx

checkpoint = {'arch': 'vgg19',
              'learn_rate': learn_rate,
              'epochs': epochs,
              'state_dict': model.state_dict(),
              'class_to_idx': model.class_to_idx,
              'optimizer': optimizer.state_dict(),
              'input_size': 25088,
              'output_size': 102,
              'momentum': momentum,
              'batch_size':64,
              'classifier' : classifier
             }

torch.save(checkpoint, 'checkpoint.pth')

Then I tried to load the project I had saved:

def load_checkpoint(filepath):
    checkpoint = torch.load(filepath)

    learn_rate = checkpoint['learn_rate']

    optimizer.load_state_dict(checkpoint['optimizer'])

    model = models.vgg16(pretrained=True)
    model.epochs = checkpoint['epochs']
    model.load_state_dict(checkpoint['state_dict'])
    model.class_to_idx = checkpoint['class_to_idx']
    model.classifier = checkpoint['classifier']

    return learn_rate, optimizer, model

learn_rate, optimizer, model = load_checkpoint('checkpoint.pth')

And I get an error when I try to load:

<ipython-input-75-5bd1aa042c7f> in load_checkpoint(filepath)
      9     model = models.vgg16(pretrained=True)
     10     model.epochs = checkpoint['epochs']
---> 11     model.load_state_dict(checkpoint['state_dict'])
     12     model.class_to_idx = checkpoint['class_to_idx']
     13     model.classifier = checkpoint['classifier']

RuntimeError: Error(s) in loading state_dict for VGG:
    Missing key(s) in state_dict: "classifier.0.weight", "classifier.0.bias", "classifier.3.weight", "classifier.3.bias", "classifier.6.weight", "classifier.6.bias".
    Unexpected key(s) in state_dict: "classifier.fc1.weight", "classifier.fc1.bias", "classifier.fc2.weight", "classifier.fc2.bias".

This seems to be classifier issue. Does anyone know what's going on?


title: Pytorch: Memory Efficient weighted sum with weights shared along channels

body:
Inputs:

1) I = Tensor of dim (N, C, X)      (Input)

2) W = Tensor of dim (N, X, Y)    (Weight)

Output:

1) O = Tensor of dim (N, C, Y)     (Output)

I want to compute:

I = I.view(N, C, X, 1)
W = W.view(N, 1, X, Y)
PROD = I*W
O = PROD.sum(dim=2)
return O

without incurring N * C * X * Y memory overhead.

Basically I want to calculate the weighted sum of a feature map wherein the weights are the same along the channel dimension, without incurring memory overhead per channel.

Maybe I could use

from itertools import product

O = torch.zeros(N, C, Y)
for n, x, y in product(range(N), range(X), range(Y)):
    O[n, :, y] += I[n, :, x]*W[n, x, y]
return O

but that would be slower (no broadcasting) and I'm not sure how much memory overhead would be incurred by saving variables for the backward pass.


title: Pytorch doesn't work with CUDA in PyCharm/IntelliJ

body:
I have just downloaded PyTorch with CUDA via Anaconda and when I type into the Anaconda terminal:
import torch
if torch.cuda.is_available():
    print('it works')

then he outputs that;  that means that it worked and it works with PyTorch.
But when I go to my IDE (PyCharm and IntelliJ) and write the same code, it doesn't output anything.
Could someone please explain to me how I can somehow get this to work in the IDE?


title: How to run Python script on a Discrete Graphics AMD GPU?

body:
WHAT I WANT TO DO:

I have a script that I use for factorizing prime numbers given a certain range:
# Python program to display all the prime numbers within an interval

lower = 900
upper = 1000

print("Prime numbers between", lower, "and", upper, "are:")

for num in range(lower, upper + 1):
   # all prime numbers are greater than 1
   if num > 1:
       for i in range(2, num):
           if (num % i) == 0:
               break
       else:
           print(num)

I would like to use the GPU instead of the CPU to run such script so it would be faster

THE PROBLEM:

I don't have a NVIDIA GPU on my Intel NUC NUC8i7HVK but a "Discrete GPU"

If I run this code to check what are my GPUs:
import pyopencl as cl
import numpy as np

a = np.arange(32).astype(np.float32)
res = np.empty_like(a)

ctx = cl.create_some_context()
queue = cl.CommandQueue(ctx)

mf = cl.mem_flags
a_buf = cl.Buffer(ctx, mf.READ_ONLY | mf.COPY_HOST_PTR, hostbuf=a)
dest_buf = cl.Buffer(ctx, mf.WRITE_ONLY, res.nbytes)

prg = cl.Program(ctx, """
    __kernel void sq(__global const float *a,
    __global float *c)
    {
      int gid = get_global_id(0);
      c[gid] = a[gid] * a[gid];
    }
    """).build()

prg.sq(queue, a.shape, None, a_buf, dest_buf)

cl.enqueue_copy(queue, res, dest_buf)

print (a, res)

I receive:

[0] <pyopencl.Platform 'AMD Accelerated Parallel Processing' at 0x7ffb3d492fd0>
[1] <pyopencl.Platform 'Intel(R) OpenCL HD Graphics' at 0x187b648ed80>

THE POSSIBLE APPROACH TO THE PROBLEM:

I found a guide that takes you by the hand and explains step by step how to run it on your GPU. But all Pyhton libraries that pipes Python through the GPU like PyOpenGL, PyOpenCL, Tensorflow (Force python script on GPU), PyTorch, etc... are tailored for NVIDIA.
In case you have an AMD all libraries ask for ROCm but such software still doesn't support integrated GPU or Discrete GPU as far as I know (see my own reply below).
I only found a guide that talks about such approach but I cannot make it work.
Is there hope or I'm just tying to do something impossible?

EDIT: Reply to @chapelo

If I choose 0 the reply is:
Set the environment variable PYOPENCL_CTX='0' to avoid being asked again.
[ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15. 16. 17.
 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31.] [  0.   1.   4.   9.  16.  25.  36.  49.  64.  81. 100. 121. 144. 169.
 196. 225. 256. 289. 324. 361. 400. 441. 484. 529. 576. 625. 676. 729.
 784. 841. 900. 961.]

If I choose 1 the reply is:
Set the environment variable PYOPENCL_CTX='1' to avoid being asked again.
[ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10. 11. 12. 13. 14. 15. 16. 17.
 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31.] [  0.   1.   4.   9.  16.  25.  36.  49.  64.  81. 100. 121. 144. 169.
 196. 225. 256. 289. 324. 361. 400. 441. 484. 529. 576. 625. 676. 729.
 784. 841. 900. 961.]


title: Speed up the PSO algorithm using Pytorch?

body:
I've implemented a PSO algorithm using PyTorch, torch.cuda.FloatTensor.
I manage to speed it up 2 times but I was expecting more than that.
It think the steps of checking limits (line 41-55) and the parameters update (line 58-72) are the cause. There's a "GPU any" of doing that?

There's is my code:

import torch
import numpy as np
import matplotlib.pyplot as plt
dtype = torch.cuda.FloatTensor

def fitness(x,y):
    return -torch.abs(torch.sin(x)*torch.cos(y)*torch.exp(torch.abs(1-(torch.sqrt(x**2+y**2))/(3.1415))))

def velocity(v, gxbest, pxbest, pybest, x, pop, w, c1, c2):
    return w*torch.rand(pop).type(dtype)*v + \
       c1*torch.rand(pop).type(dtype)*(pxbest - x) + \
       c2*torch.rand(pop).type(dtype)*(gxbest.expand(x.size(0)) - x)

def PSO(pop, xmax, xmin, niter, wmax, wmin, c1, c2):

    vx                 = torch.rand(pop).type(dtype)
    vy                 = torch.rand(pop).type(dtype)
    best               = np.zeros(niter)
    x                 = (xmax -  xmin)*torch.rand(pop).type(dtype) + xmin
    y                 = (xmax -  xmin)*torch.rand(pop).type(dtype) + xmin
    z                 = fitness(x,y)
    [minz, indexminz] = z.min(0)
    gxbest            = x[indexminz]
    gybest            = y[indexminz]
    pxbest            = x
    pybest            = y
    pzbest            = z

    for K in range(niter):
        w      = wmax - ((wmax - wmin) / niter) * (K)
        vnextx = velocity(vx, gxbest, pxbest, pybest, x, pop, w, c1, c2)
        xnext  = x + vnextx
        vnexty = velocity(vy, gxbest, pxbest, pybest, y, pop, w, c1, c2)
        ynext  = y + vnexty

        **(41)** xnext = xnext.cpu()
        ynext = ynext.cpu()
        idxmax        = (xnext > xmax) # elements that are bigger that upper limit
        idxmim        = (xnext < xmin)  # elements that are smaller that upper limit
        xnext[idxmax] = xmax
        xnext[idxmim] = xmin
        idymax        = (ynext > xmax) # elements that are bigger that upper limit
        idymim        = (ynext < xmin)  # elements that are smaller that upper limit
        ynext[idymax] = xmax
        ynext[idymim] = xmin

        xnext = xnext.cuda()
        **(55)** ynext = ynext.cuda()

        znext = fitness(xnext,ynext)

        **(58)**[minznext, indexminznext]  = znext.min(0)

        if (minznext[0] < minz[0]):
            minz   = minznext
            gxbest = xnext[indexminznext]
            gybest = ynext[indexminznext]

        indexpbest         = (znext < pzbest)
        pxbest[indexpbest] = xnext[indexpbest]
        pybest[indexpbest] = ynext[indexpbest]
        pzbest[indexpbest] = znext[indexpbest]
        x                  = xnext
        y                  = ynext
        vx                 = vnextx
        **(72)** vy                 = vnexty

        best[K] = minz.cpu().numpy()
    return gxbest, gybest , minz, best

def main():

    pop, xmax, xmin, niter = 10000, 10, -10, 10
    wmax = 0.9
    wmin = 0.4
    c1   = 2.05
    c2   = 2.05
    xbest, ybest, fitbest, best = PSO(pop, xmax, xmin, niter, wmax, wmin, c1, c2)
    print(xbest)
    print(ybest)
    print(fitbest)
    t = np.linspace(0,niter,niter)
    plt.plot(t, best, 'k.-')
    plt.show()

main()

Thank so much for your help.


title: Python Kernel Died when using triple for loop containing requires_grad=True parameters (Pytorch)

body:
I just want to store para_k in every position of COV matrix, where:

para_k=torch.tensor([2.0],requires_grad=True).

And because I want to update parameters by optimising a loss function later(omitted here), I have to add another for loop. Therefore, the whole code is as follows:

import torch
para_k=torch.tensor([2.0],requires_grad=True)
for t in range(5):
    num_row=300
    num_col=300
    COV = torch.zeros((num_row,num_col))
    for i in range(num_row):
        for j in range(num_col):
            COV[i,j]=para_k
            print('i= %d ,j= %d' % (i,j) )

This simple computation cannot be finished by Spyder!!
Things can go well if I change nrow and ncol to be smaller than 100.
If I use number larger than 100, it will send error message (sometimes even one iteration cannot be finished) that Kernel died and restarting: Spyder Kernel Died

I even uninstall and install again my Anaconda and reinstall pytorch. I also tried to run Spyder as administrator, still got the error.
I check the event of my system, the  error is: ntdll.dll regarding to pythonw.exe. I listed pythonw.exe as exception of firewall, but still cannot solve the problem.


title: Modification to Caffe VGG 16 to handle 1 channel images on PyTorch

body:
I am converting a VGG16 network to be a Fully Convolutional network and also modifying the the input to accept a single channel image. The complete code for reproducibilty is given below.

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable

import numpy as np
import torchvision.datasets as datasets
import copy

from torch.utils import model_zoo
from torchvision import models
from collections import OrderedDict

def convolutionalize(modules, input_size):
    """
    Recast `modules` into fully convolutional form
    """
    fully_conv_modules = []
    x = Variable(torch.zeros((1, ) + input_size))
    for m in modules:
         if isinstance(m, nn.Linear):
              n = nn.Conv2d(x.size(1), m.weight.size(0), kernel_size=(x.size(2), x.size(3)))
              n.weight.data.view(-1).copy_(m.weight.data.view(-1))
              n.bias.data.view(-1).copy_(m.bias.data.view(-1))
              m = n
         fully_conv_modules.append(m)
         x = m(x)
    return fully_conv_modules

def vgg16(is_caffe=True):
     """
     Load the VGG-16 net for use as a fully convolutional backbone.
     """
     vgg16 = models.vgg16(pretrained=True)
     # cast into fully convolutional form (as list of layers)
     vgg16 = convolutionalize(list(vgg16.features) + list(vgg16.classifier),
                         (3, 224, 224))
     # name layers like the original paper
     names = ['conv1_1', 'relu1_1', 'conv1_2', 'relu1_2', 'pool1',
    'conv2_1', 'relu2_1', 'conv2_2', 'relu2_2', 'pool2',
    'conv3_1', 'relu3_1', 'conv3_2', 'relu3_2', 'conv3_3', 'relu3_3', 'pool3',
    'conv4_1', 'relu4_1', 'conv4_2', 'relu4_2', 'conv4_3', 'relu4_3', 'pool4',
    'conv5_1', 'relu5_1', 'conv5_2', 'relu5_2', 'conv5_3', 'relu5_3', 'pool5',
    'fc6', 'relu6', 'drop6', 'fc7', 'relu7', 'drop7', 'fc8']

    vgg16 = nn.Sequential(OrderedDict(zip(names, vgg16)))

    if is_caffe:
        # substitute original Caffe weights for improved fine-tuning accuracy
        # see https://github.com/jcjohnson/pytorch-vgg
        caffe_params = model_zoo.load_url('https://s3-us-west-2.amazonaws.com/'
                                      'jcjohns-models/vgg16-00b39a1b.pth')
        for new_p, old_p in zip(vgg16.parameters(), caffe_params.values()):
            new_p.data.copy_(old_p.view_as(new_p))
        # surgery: decapitate final classifier
   del vgg16._modules['fc8']  # note: risky use of private interface
   # surgery: keep fuller spatial dims by including incomplete pooling regions
   for m in vgg16.modules():
       if isinstance(m, nn.MaxPool2d):
                m.ceil_mode = True
   return vgg16

class Learner(nn.Module):

     def __init__(self, num_classes, singleChannel=False):
          super().__init__()

          backbone = vgg16(is_caffe=True)
          for k in list(backbone._modules)[-6:]:
                del backbone._modules[k]

          supp_backbone = copy.deepcopy(backbone)

          # Modify conv1_1 of conditioning branch to have 1 input channels
          # Init the weights in the new channels to the channel-wise mean
          # of the pre-trained conv1_1 weights
          if singleChannel==True:
               old_conv1 = backbone._modules['conv1_1'].weight.data
               mean_conv1 = torch.mean(old_conv1, dim=1, keepdim=True)
               new_conv1 = nn.Conv2d(1, old_conv1.size(0), kernel_size=old_conv1.size(2), stride=1, padding=1)
               new_conv1.weight.data = mean_conv1
               new_conv1.bias.data = backbone._modules['conv1_1'].bias.data
               backbone._modules['conv1_1'] = new_conv1

          self.encoder = copy.deepcopy(backbone)
          self.num_classes=num_classes

     def forward(self,im):

          # encode image
          supp_feats = self.encoder(im)

          return supp_feats

 model=Learner(num_classes=2,singleChannel=True).cpu()
 mnist_trainset = datasets.MNIST(root='./data', train=True, download=True, transform=None)
 im2arr = np.array(mnist_trainset[1][0])
 im2arr = im2arr[np.newaxis,:, :,] # shape(1,28,28)

 model.train()
 x=model(torch.from_numpy(im2arr))

I expected x to be a torch tensor output but get an error message
'ValueError: Expected 4D tensor as input, got 3D tensor instead.' on the last line


title: How to change the axis on which 1 dimensional convolution is performed on embedding layer in PyTorch?

body:
I've been playing around with text classification in PyTorch and I've encountered a problem with 1 dimensional convolutions.

I've set an embedding layer of dimesions (x, y, z) where:
x - denotes the batch size
y - denotes the length of a sentence (fixed with padding, so 40 words)
z - the dimensionality of pre-trained word embedding (for now 100)

For simplicity sake, let's assume I put in a matrix of (1,40, 100)

However, when to my knowledge once I perform torch.nn.conv1d(*args),
The resulting matrix becomes (batch size = 1, word size = 40, feature map size = 98) with kernel size of 3.

Basically, as I understand it convolves around y axis instead of x axis and it turn does not capture the spacial relationship between word embeddings.

Is there any way to change the convolutional layer so it calculates feature maps around different axis?

TL, DR:

Torch conv1d layer behaves this way on embedding layer:
enter image description here

But I want it to behave like this

enter image description here

Any help would be much appreciated.


title: index out of range at ..\aten\src\TH/generic/THTensorEvenMoreMath.cpp:193

body:
I am trying to build junction tree vae.I got struck over this error.My embedding size is same as the vocabulary size and and the other parameter is hidden size length.Where am I going wrong?Please help someone.

I have read many blogs and tried solving this error but I can't.I know there is some problem with embedding but I don't know how to solve?

def __init__(self, vocab, hidden_size, latent_size, depthT, depthG):
        super(JTNNVAE, self).__init__()
        self.vocab = vocab
        #print(int(vocab.size()))
        self.hidden_size = int(hidden_size)
        self.latent_size = latent_size = latent_size / 2 #Tree and Mol has two vectors
        self.latent_size=int(self.latent_size)
        self.jtnn = JTNNEncoder(hidden_size, depthT, nn.Embedding(int(vocab.size()), int(hidden_size)))
        self.decoder = JTNNDecoder(vocab, hidden_size, latent_size, nn.Embedding(int(vocab.size()), int(hidden_size)))

        self.jtmpn = JTMPN(hidden_size, depthG)
        self.mpn = MPN(hidden_size, depthG)

        self.A_assm = nn.Linear(int(latent_size), int(hidden_size), bias=False)
        self.assm_loss = nn.CrossEntropyLoss(size_average=False)

        self.T_mean = nn.Linear(int(hidden_size), int(latent_size))
        self.T_var = nn.Linear(int(hidden_size), int(latent_size))
        self.G_mean = nn.Linear(int(hidden_size), int(latent_size))
        self.G_var = nn.Linear(int(hidden_size), int(latent_size))

Expected to solve this error.

Thanks.


title: Why is stft(istft(x)) ≠ x?

body:
Why is stft(istft(x)) ≠ x?

Using PyTorch, I have computed the short-time Fourier transform of the inverse short-time Fourier transform of a tensor.
I have done so as shown below given a tensor x.
For x, the real and imaginary part are equal, or the imaginary part is set to zero -- both produces the same problem.

torch.stft(torchaudio.functional.istft(x, n_fft), n_fft)

As shown in the image, only a single one of the stripes in the tensor remains after applying stft(istft(x)) -- all other stripes vanish.
If stft(istft(x)) (bottom) was equal to x (top), both images would look similar.
Why are they so different?
It seems like stft(istft(x)) can pick up only certain frequencies of x.

x (top) and stft of istft of x (bottom)

I have also tried the same with scipy.signal.istft and scipy.signal.stft which causes the same issue.
Moreover, I have tried it with a wide range of tensors x, e.g., different randomized distributions, images, and other stripes.
Also, I have tried a variety of hyper-parameters for stft/istft.
Only for x generated by a short-time Fourier transform from a sound wave, it works.


title: pytorch model not updating

body:
I put my training code below. I am using  torch.optim.SGD as optimizer. I thought optimizer.step() would be doing the update but the model accuracy seems to stay the same. My friend said he didn't use the optimizer.step() and his works fine.
I tried taking it out, still the same result. What can I be doing wrong?
I don't think there's a problem with the accuracy calculation.

class FNet(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()
        self.fc1 = nn.Linear(128*256, 1024)
        self.fc2 = nn.Linear(1024, 256)
        self.fc3 = nn.Linear(256, 2)

    def forward(self, X):
        X = F.relu(self.fc1(X))
        X = F.relu(self.fc2(X))
        X = self.fc3(X)
        return F.softmax(X, dim=1)

def main():

    learning_rate = 0.01
    model = FNet()
    optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=0.9, weight_decay=5e-04) # you can play with momentum and weight_decay parameters as well
    accs = [0,]

    for epoch in range(max_epoch):

        train(epoch, model, optimizer, train_batch)

        acc = test(model, val_batch)
        accs.append(acc)

def train(epoch, model, optimizer, trainloader):
    model.train()
    optimizer.zero_grad()
    for batch_idx, (data, labels) in enumerate(trainloader):

        outputs = model(data)
        loss = F.nll_loss(outputs, labels)
        loss.backward()

        optimizer.step()

def test(model, testloader):
    correct = 0
    total = 0
    model.eval()

    for batch_idx, (data, labels) in enumerate(testloader):

            outputs = model(data.view(-1,128*256))

            for sample_idx,output in enumerate(outputs):

                if torch.argmax(output) == labels[sample_idx]:
                    correct = correct + 1
                total = total + 1

    accuracy = correct/total
    return accuracy


title: PyTorch - Save just the model structure without weights and then load and train it

body:
I want to separate model structure authoring and training. The model author designs the model structure, saves the untrained model to a file and then sends it training service which loads the model structure and trains the model.
Keras has the ability to save the model config and then load it.
How can the same be accomplished with PyTorch?


title: Custom loss function error: tensor does not have a grad_fn

body:
Trying to utilize a custom loss function and getting error ‘RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn’. Error occurs during loss.backward()
I’m aware that all computations must be done in tensors with ‘require_grad = True’. I’m having trouble implementing that as my code requires a nested for loop. I believe it could be the for loop. Is there a way to create an empty tensor and append it? Below is my code.
def Gaussian_Kernal(x, mu, sigma):
  p = (1./(math.sqrt(2. * math.pi * (sigma**2)))) * torch.exp((-1.) * (((Variable(x)**2) - mu)/(2. * (sigma**2))))
  return p

class MEE(torch.nn.Module):
  def __init__(self):
    super(MEE,self).__init__()

  def forward(self,output, target, mu, variance):

    error = torch.subtract(Variable(output),Variable(target))

    error_diff = []
    for i in range(0, error.size(0)):
      for j in range(0, error.size(0)):
        error_diff.append(error[i] - error[j])

    error_diff = torch.cat(error_diff)
    torch.tensor(error_diff,requires_grad=True)

    loss = (1./(target.size(0)**2)) * torch.sum(Gaussian_Kernal(Variable(error_diff), mu, variance*(2**0.5)))

    loss = Variable(loss)

    return loss


title: What does unsqueeze do in pytorch?

body:
lower_bounds = torch.max(set_1[:, :2].unsqueeze(1),
                         set_2[:, :2].unsqueeze(0))   #(n1, n2, 2)

This code snippet uses unsqueeze(1) for one tensor, but unsqeeze(0) for another. What is the difference between them?


title: Google Colab does not run all of one cell

body:
I have a training method defined in one cell of a google colab notebook, shown below:
def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
    since = time.time()

    best_model_wts = copy.deepcopy(model.state_dict())
    best_acc = 0.0
    print("Training started")
    for epoch in range(num_epochs):
      print("epoch: ", epoch)
      # Training first
      model.train()
      running_loss = 0.0
      running_corrects = 0

      for step, (inputs, labels) in enumerate(train_iter):
        # zero the parameter gradients
        optimizer.zero_grad()
        # Track history if only in training
        torch.set_grad_enabled(True)
        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)
        loss = criterion(outputs, labels)
        #print(loss)
        # Backwards and optimization:
        loss.backward()
        optimizer.step()
        print("step: ",step)
      # statistics

      running_loss += loss.item() * inputs.size(0)
      running_corrects += torch.sum(preds == labels.data)

      epoch_loss = running_loss / train_size
      epoch_acc = running_corrects.double() / train_size

      # Use validation set after each epoch
      # Save the best model
      model.eval()
      torch.no_grad()
      num_correct_val = 0
      print("Validation started")
      for step, (inputs, labels) in enumerate(val_iter):
        outputs = model(inputs)
        _, preds = torch.max(outputs, 1)
        loss = criterion(outputs, labels)
        num_correct_val += torch.sum(preds == labels.data)

      val_acc = num_correct_val.double() / val_size

      if val_acc > best_acc:
        best_acc = val_acc
        best_model_wts = copy.deepcopy(model.state_dict())

    time_elapsed = time.time() - since
    print('Training complete in {:.0f}m {:.0f}s'.format(
        time_elapsed // 60, time_elapsed % 60))
    print('Best val Acc: {:4f}'.format(best_acc))

    # load best model weights
    model.load_state_dict(best_model_wts)
    return model

I define my ResNet18 model as below:
resnet18 = torchvision.models.resnet18(pretrained=True)
for param in resnet18.parameters():
  param.requires_grad = False
num_ftrs = resnet18.fc.in_features
num_classes = 2
resnet18.fc = nn.Linear(num_ftrs, num_classes)

criterion = nn.CrossEntropyLoss()

optimizer = optim.SGD(resnet18.fc.parameters(), lr = 0.001, momentum = 0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer, step_size=7, gamma=0.1)

And I finally call the the train method on my model:
try:
  resnet18 = train_model(resnet18, criterion, optimizer, exp_lr_scheduler)
except:
  pass

For some reason, my output is just ( I skipped some steps to make it cleaner and shorter):
Training started
epoch:  0
step:  0
step:  1
step:  2
step:  3
step:  4
step:  5
step:  6
.
.
.
step:  155
step:  156

Why doesn't google colab execute the rest of the code in the training method? Is it because the number of epochs is large? Or am I doing something wrong in my code?
Note: Every block of code is a separate cell in my notebook, and I run them sequentially.


title: Jupyter doesn't recognize torchaudio

body:
I am trying to install torchaudio to use in a Jupyter notebook but when i import it i get the error:
ModuleNotFoundError: No module named 'torchaudio'

I tried to import it in a .py file that the notebook uses but to with no success. I thought maybe it wasnt installed properly but when i try to install it using pip install torchaudio i get "requirement already satisfied".
Im lost, how can i import it successfully?


title: TypeError: forward() missing 1 required positional argument: 'x' while implementing Grad CAM

body:
I made model that takes grayscale image as input, and returns single value as output.
Backbone of this model is basically resnet18, but as it takes different input/output, I modified it.
model = models.resnet18(pretrained=True)
model.conv1 = nn.Conv2d(1, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
model = nn.Sequential(model,nn.Linear(1000,1))

This model worked pretty well on test dataset, so I decided to make activation heatmap using Grad CAM.
class model(nn.Module):
def __init__(self):
    super(model,self).__init__()

    self.model = torch.load('model.pth')

    self.features=nn.Sequential( "STACKED ARCHITECTURES")

    self.avgpool = nn.AdaptiveAvgPool2d(output_size=(1, 1))
    self.classifier = nn.Sequential(nn.Linear(in_features=512, out_features=1000, bias=True),
                                    nn.Linear(in_features=1000, out_features=1, bias=True))
    self.gradient = None

# hook for the gradients
def activations_hook(self, grad):
    self.gradient = grad

def get_gradient(self):
    return self.gradient

def get_activations(self, x):
    return self.features(x)

def forward(self, x):
    identity = x

    x = self.features(x)        # extract the features

    h = x.register_hook(self.activations_hook)

    # complete the forward pass
    x = self.avgpool(x)
    x = x.view((1, -1))
    x = self.classifier(x)

    return x

and then, after I tried to initiate it with code below, I always face with TypeError
model = model()
_ = model.eval()
image, _ = next(iter(grad_loader))
pred = model(image)

TypeError: forward() missing 1 required positional argument: 'x'
Can anyone give any advice to solve this?
p.s Dataloader(grad_loader) worked fine while I was training my model, so I think dataloader is probably not the problem


so ive been trying to work on some example of bert that i found on github as its the first time im trying to use bert and see how it works. The respiratory im working with is the following: https://github.com/prateekjoshi565/Fine-Tuning-BERT/blob/master/Fine_Tuning_BERT_for_Spam_Classification.ipynb
im using a different dataset however im getting the issue TypeError: linear(): argument 'input' (position 1) must be Tensor, not str" and honestly i dont know what im doing wrong. is there anyone that could help me?
the code ive been using is the following:
# convert class weights to tensor
weights= torch.tensor(class_wts,dtype=torch.float)
weights = weights.to(device)

# loss function
cross_entropy  = nn.NLLLoss(weight=weights)

# number of training epochs
epochs = 10

def train():

  model.train()

  total_loss, total_accuracy = 0, 0

  # empty list to save model predictions
  total_preds=[]

  # iterate over batches
  for step,batch in enumerate(train_dataloader):

    # progress update after every 50 batches.
    if step % 50 == 0 and not step == 0:
      print('  Batch {:>5,}  of  {:>5,}.'.format(step, len(train_dataloader)))

    # push the batch to gpu
    batch = [r.to(device) for r in batch]

    sent_id, mask, labels = batch

    # clear previously calculated gradients
    model.zero_grad()

    # get model predictions for the current batch
    preds = model(sent_id, mask)

    # compute the loss between actual and predicted values
    loss = cross_entropy(preds, labels)

    # add on to the total loss
    total_loss = total_loss + loss.item()

    # backward pass to calculate the gradients
    loss.backward()

    # clip the the gradients to 1.0. It helps in preventing the exploding gradient problem
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)

    # update parameters
    optimizer.step()

    # model predictions are stored on GPU. So, push it to CPU
    preds=preds.detach().cpu().numpy()

    # append the model predictions
    total_preds.append(preds)

  # compute the training loss of the epoch
  avg_loss = total_loss / len(train_dataloader)

  # predictions are in the form of (no. of batches, size of batch, no. of classes).
  # reshape the predictions in form of (number of samples, no. of classes)
  total_preds  = np.concatenate(total_preds, axis=0)

  #returns the loss and predictions
  return avg_loss, total_preds

def evaluate():

  print("\nEvaluating...")

  # deactivate dropout layers
  model.eval()

  total_loss, total_accuracy = 0, 0

  # empty list to save the model predictions
  total_preds = []

  # iterate over batches
  for step,batch in enumerate(val_dataloader):

    # Progress update every 50 batches.
    if step % 50 == 0 and not step == 0:

      # Calculate elapsed time in minutes.
      elapsed = format_time(time.time() - t0)

      # Report progress.
      print('  Batch {:>5,}  of  {:>5,}.'.format(step, len(val_dataloader)))

    # push the batch to gpu
    batch = [t.to(device) for t in batch]

    sent_id, mask, labels = batch

    # deactivate autograd
    with torch.no_grad():

      # model predictions
      preds = model(sent_id, mask)

      # compute the validation loss between actual and predicted values
      loss = cross_entropy(preds,labels)

      total_loss = total_loss + loss.item()

      preds = preds.detach().cpu().numpy()

      total_preds.append(preds)

  # compute the validation loss of the epoch
  avg_loss = total_loss / len(val_dataloader)

  # reshape the predictions in form of (number of samples, no. of classes)
  total_preds  = np.concatenate(total_preds, axis=0)

  return avg_loss, total_preds

# set initial loss to infinite
best_valid_loss = float('inf')

# empty lists to store training and validation loss of each epoch
train_losses=[]
valid_losses=[]

#for each epoch
for epoch in range(epochs):

    print('\n Epoch {:} / {:}'.format(epoch + 1, epochs))

    #train model
    train_loss, _ = train()

    #evaluate model
    valid_loss, _ = evaluate()

    #save the best model
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'saved_weights.pt')

    # append training and validation loss
    train_losses.append(train_loss)
    valid_losses.append(valid_loss)

    print(f'\nTraining Loss: {train_loss:.3f}')
    print(f'Validation Loss: {valid_loss:.3f}')

the traceback i receive is:
Epoch 1 / 10
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-105-c5138ddf6b25> in <module>()
     12
     13     #train model
---> 14     train_loss, _ = train()
     15
     16     #evaluate model

5 frames
<ipython-input-103-3236a6e339dd> in train()
     24
     25     # get model predictions for the current batch
---> 26     preds = model(sent_id, mask)
     27
     28     # compute the loss between actual and predicted values

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

<ipython-input-99-9ebdcf410f97> in forward(self, sent_id, mask)
     28       _, cls_hs = self.bert(sent_id, attention_mask=mask)
     29
---> 30       x = self.fc1(cls_hs)
     31
     32       x = self.relu(x)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/linear.py in forward(self, input)
     92
     93     def forward(self, input: Tensor) -> Tensor:
---> 94         return F.linear(input, self.weight, self.bias)
     95
     96     def extra_repr(self) -> str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in linear(input, weight, bias)
   1751     if has_torch_function_variadic(input, weight):
   1752         return handle_torch_function(linear, (input, weight), input, weight, bias=bias)
-> 1753     return torch._C._nn.linear(input, weight, bias)
   1754
   1755

TypeError: linear(): argument 'input' (position 1) must be Tensor, not str


title: Pytorch hidden cell LSTM with mini-batches

body:
Problem

I don't understand how to handle the LSTM hidden cells when training in mini-batches, since the training data is sent to the network in batches of n sequences, while only 1 sequence is processed each time during test.

Code

Specifically, my network is:

class Pytorch_LSTM(nn.Module):
    def __init__(self, params):
        super(Pytorch_LSTM, self).__init__()
        self.params = params
        self.hidden_layer_size = params['hidden_layer_size']
        # Define layers
        self.lstm = nn.LSTM(input_size = params['in_features'], hidden_size = params['hidden_layer_size'])
        self.linear1 = nn.Linear(params['hidden_layer_size'], params['hidden_layer_size'])
        self.linear2 = nn.Linear(params['hidden_layer_size'], params['out_features'])
        self.hidden_cell = (torch.zeros(1,self.params['batch_size'],self.hidden_layer_size),
                           torch.zeros(1,self.params['batch_size'],self.hidden_layer_size))

    def forward(self, input_seq):
        lstm_out, self.hidden_cell = self.lstm(input_seq.view(self.params['time_window'],-1,self.params['in_features']), self.hidden_cell)
        linear1_out = self.linear1(lstm_out)
        predictions = self.linear2(linear1_out)
        return predictions[-1]

In my train() method:

def train(self, input_sequence, params, test_idx, final, verbose=True):

        ....
        ....

        # Model
        self.model = Pytorch_LSTM(params)
        # Let's train the model
        for epoch in range(epochs):
            for count_1,seq in enumerate(train_data_batch):
                optimizer.zero_grad()
                self.model.hidden_cell = (torch.zeros(1, params['batch_size'], self.model.hidden_layer_size),
                                          torch.zeros(1, params['batch_size'], self.model.hidden_layer_size))
                y_pred = self.model(seq)     # seq.shape: (n_batches, 25, 4)
                single_loss = mse_loss(y_pred, y_label)    # y_pred.shape, y_label.shape : (batch_size, 4)

This trains the model in mini-batches, I believe correctly.
When I test it, I only have one single sequence per time, instead of multiple batches. In my test():

for count,seq in enumerate(val_data[j]):
    y_pred = self.model(seq)           # seq.shape: (25,4)
    single_loss = mse_loss(y_pred, y_label)

This returns the error:

RuntimeError: Expected hidden[0] size (1, 1, 100), got (1, 704, 100)

where n_batches= 704.

How should I handle the hidden_cell?


title: Is cudatoolkit necessary for pytorch?

body: When installing pytorch in conda, cudatoolkit is also installed. My understanding is that the pytorch code is pre-compiled into machine code. If this is true, is the cudatoolkit used when writing/running regular pytorch programs or is it just there in the case you want to write custom CUDA code?


title: Unable to install Pytorch in Ubuntu

body:
I'm using the following command to install pytorch in my conda environment.
conda install pytorch=0.4.1 cuda90 -c pytorch

However, I'm getting the following error

Solving environment: failed
PackagesNotFoundError: The following packages are not available from
current channels:

pytorch=0.4.1
cuda90

Current channels:

https://conda.anaconda.org/pytorch/linux-32
https://conda.anaconda.org/pytorch/noarch
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/linux-32
https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/noarch
https://repo.anaconda.com/pkgs/main/linux-32
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/free/linux-32

To search for alternate channels that may provide the conda package
you're looking for, navigate to
https://anaconda.org

How can I sort this out?
I have ofcourse installed cuda 9 and nvcc works.


title: The difference between the GRUCell function in pytorch and tensorflow

body:
I want to migrate my code from pytorch to tensorflow2.0, but I don't know how GRU and GRUCell in pytorch are implemented in tensorflow2.0.

self.dec_gru_a = nn.GRUCell(self.embed_size, self.hidden_size)
hidden = self.dec_gru_a(h, hidden)


title: convert float image array to int in PIL via Image.fromarray

body:
I have a PyTorch tensor of shape 3,256,256 where 3 is the number of channels and the image dimensions are 256, with all float values.

I am trying to feed this into the network and use PIL to do some transforms. To this end, I do:

img = Image.fromarray((255*imgs[i]).numpy().astype(np.uint8))

but I get:

TypeError: Cannot handle this data type: (1, 1, 256), |u1

When I check the output of (255*imgs[i]).numpy().astype(np.uint8), I do however see, for example:

[[[ 62  57  59 ...  63  46  36]
  [ 72  71  67 ...  80  76  82]
  [ 58  63  63 ... 145 152 169]
  ...
  [238 240 243 ...   7   7   7]
  [241 239 240 ...   5   5   6]
  [241 243 242 ...   4   3   5]]

 [[ 83  78  80 ...  86  70  61]
  [ 91  90  85 ...  95  93  98]
  [ 80  83  80 ... 141 150 168]
  ...
  [176 178 181 ...  14  14  14]
  [177 176 178 ...  15  15  17]
  [179 180 180 ...  13  13  15]]

 [[147 141 143 ... 150 136 128]
  [147 149 148 ... 154 149 154]
  [141 149 148 ... 178 182 196]
  ...
  [129 131 134 ...  43  43  43]
  [130 130 131 ...  45  45  47]
  [133 134 133 ...  44  44  46]]]

I am not an image expert by a long shot, and I am struggling to troubleshoot this issue now.


title: running a python file from the ipython notebook using command line in loop

body:
I have built a model that trains using training.py. I want to tune the hyperparameters and run the following script from the notebook in loop by varying the arguments passed.

python training.py --cuda --emsize 1500 --nhid 1500 --dropout 0.65 --epochs 10

For eg: If the hyperparameter is dropout, I want to be able to run the script in loop by varying dropout values and plot the graph.


title: Pytorch mask missing values when calculating rmse

body:
I'm trying to calculate the rmse error of two torch tensors. I would like to ignore/mask the rows where the labels are 0 (missing values). How could I modify this line to take that restriction into account?

torch.sqrt(((preds.detach() - labels) ** 2).mean()).item()

Thank you in advance.


title: How can I deploy my Pytorch model into IOS?

body: I have a deep learning neural network I built on Pytorch I am seeking to deploy onto IOS.


title: Tensor output from final layer is of the wrong shape in PyTorch

body:
I am building a sequence-to-label classifier, where the input data are text sequences and output labels are binary. The model is very simple, with GRU hidden layers and a Word Embeddings input layer. I want a [n, 60] input to output a [n, 1] label, but the Torch model returns a [n, 60] output.

My model, with minimal layers:

class Model(nn.Module):
    def __init__(self, weights_matrix, hidden_size, num_layers):
        super(Model, self).__init__()
        self.embedding, num_embeddings, embedding_dim = create_emb_layer(weights_matrix, True)
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.gru = nn.GRU(embedding_dim, hidden_size, num_layers, batch_first=True)
        self.out = nn.Linear(hidden_size, 1)
    def forward(self, inp, hidden):
        emb = self.embedding(inp);
        out, hidden = self.gru(emb, hidden)
        out = self.out(out);
        return out, hidden;

    def init_hidden(self, batch_size):
        return torch.zeros(self.num_layers, batch_size, self.hidden_size).to(device);

Model Layers:

Model(
  (embedding): Embedding(184901, 100)
  (gru): GRU(100, 60, num_layers=3, batch_first=True)
  (out): Linear(in_features=60, out_features=1, bias=True)
)

Input shapes of my data are: X : torch.Size([64, 60]), and Y : torch.Size([64, 1]), for a single batch of size 64.

When I run the X tensor through the model, it should output a single label, however, the output from the classifier is torch.Size([64, 60, 1]). To run the model, I do the following:

for epoch in range(1):
    running_loss = 0.0;

    batch_size = 64;
    hidden = model.init_hidden(batch_size)
    for ite, data in enumerate(train_loader, 0):
        x, y = data[:,:-1], data[:,-1].reshape(-1,1)

        optimizer.zero_grad();

        outputs, hidden = model(x, hidden);

        hidden = Variable(hidden.data).to(device);
        loss = criterion(outputs, y);

        loss.backward();
        optimizer.step();

        running_loss = running_loss + loss.item();
        if ite % 2000 == 1999:
            print('[%d %5d] loss: %.3f'%(epoch+1, ite+1, running_loss / 2000))
        running_loss = 0.0;

When I print the shape of outputs, it is 64x60x1 rather than 64x1. What I also don't get is how the criterion function is able to calculate the loss when the shapes of outputs and labels are inconsistent. With Tensorflow, this would always throw an error, but it doesn't with Torch.


title: Unable to install fastai on Jupyter Notebook

body:
I'm currently trying to get fastai installed on a conda environment using the command conda install -c fastai fastai as shown in the installation guide. This is what appears when that command is ran:

(fastai) C:\>conda install -v -c fastai fastai
Collecting package metadata (current_repodata.json): ...working... Unable to retrieve repodata (response: 404) for https://conda.anaconda.org/fastai/win-64/current_repodata.json

done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working... failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): ...working... done
Solving environment: ...working... failed with initial frozen solve. Retrying with flexible solve.
Solving environment: ...working...
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
failed
Traceback (most recent call last):
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\exceptions.py", line 1079, in __call__
    return func(*args, **kwargs)
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\cli\main.py", line 84, in _main
    exit_code = do_call(args, p)
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\cli\conda_argparse.py", line 82, in do_call
    return getattr(module, func_name)(args, parser)
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\cli\main_install.py", line 20, in execute
    install(args, parser, 'install')
  File "C:\Users\username\AppData\Local\Continuum\anaconda2\lib\site-packages\conda\cli\install.py", line 308, in install
    raise e
UnsatisfiableError: The following specifications were found to be incompatible with each other:

Output in format: Requested package -> Available versionsThe following specifications were found to be incompatible with your CUDA driver:

  - feature:/win-64::__cuda==9.2=0

Your installed CUDA driver is: 9.2

I can't say I completely understand what the issue is here. I previously thought it was an issue with PyTorch but after installing PyTorch successfully, I am still greeted with this message. Any ideas on how I can this package installed on my environment? Any help is appreciated. Thank you!


title: TypeError: can't convert np.ndarray of type numpy.object_. It shows the supported files are: float64, float32 etc

body:
when I run this code it has a TypeError. I don't know how to solve this. please help me with this issue. and I have a question that it is running in GPU or not?
import torch
torch.cuda.is_available()
torch.cuda.current_device()
torch.cuda.get_device_name(0)
torch.cuda.memory_reserved()
torch.cuda.memory_allocated()
torch.cuda.memory_allocated()
var1=torch.FloatTensor([1.0,2.0,3.0]).cuda(0)
var1
var1.device
import pandas as pd
df=pd.read_csv('diabetes.csv')
df.head()
df.isnull().sum()

import seaborn as sns
import numpy as np
df['Outcome']=np.where(df['Outcome']==1,"Diabetic","No Diabetic")
df.head()
sns.pairplot(df,hue="Outcome")
X=df.drop('Outcome',axis=1).values### independent features
y=df['Outcome'].values###dependent features
from sklearn.model_selection import train_test_split

X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.2,random_state=0)
y_train
import torch
import torch.nn as nn
import torch.nn.functional as F

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

X_train=torch.FloatTensor(X_train).cuda(0)
X_test=torch.FloatTensor(X_test).cuda()
y_train=torch.LongTensor(y_train).cuda()
y_test=torch.LongTensor(y_test).cuda()

The error is:

C:\Users\vinot\.conda\envs\python21\python.exe D:/python/python_work/pythonProject/diabetes.py
Traceback (most recent call last):
  File "D:/python/python_work/pythonProject/diabetes.py", line 35, in <module>
    y_train=torch.LongTensor(y_train).cuda()
TypeError: can't convert np.ndarray of type numpy.object_. The only supported types are: float64, float32, float16, complex64, complex128, int64, int32, int16, int8, uint8, and bool.

Process finished with exit code 1


title: Who changed my weight initialization?

body:
I know it is the reset_parameters() in conv.py that is responsible for the default weight initialization,

I changed the function to

 def reset_parameters(self):
        n = self.in_channels
        for k in self.kernel_size:
            n *= k
        stdv = 1. / math.sqrt(n)
        print('reset w, stdv=',stdv)
        self.weight.data.uniform_(-stdv, stdv)
        if self.bias is not None:
            print('reset b, stdv=',stdv)
            self.bias.data.uniform_(-stdv, stdv)
        print('w:',self.weight.data.norm(), 'b:',self.bias.data.norm())

and after my model was created, I apply the follow manual weights_init() to change the weight initialization method of my conv layer

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        print(m)
        print(m.weight.data.norm())
        print(m.bias.data.norm())

        std_w = m.weight.size(1) * m.weight.size(2) * m.weight.size(3)
        std_b = m.weight.size(0)
        std_w = 1. / math.sqrt(std_w)
        # std_b = 1. / math.sqrt(std_b)
        std_b = std_w
        m.weight.data.uniform_(-std_w, std_w)
        m.bias.data.uniform_(-std_b, std_b)
        print(m.weight.data.norm())
        print(m.bias.data.norm())
        print('\n\n')

I think they are same , but I found it print below info in reset_params function.

  reset w, stdv= 0.19245008972987526 reset b, stdv= 0.19245008972987526
  w: 4.651364750286455 b: 0.9658572124243059 reset w, stdv=
  0.041666666666666664 reset b, stdv= 0.041666666666666664 w: 4.60668514079571 b: 0.19021859795685142 reset w, stdv= 0.041666666666666664 reset b, stdv= 0.041666666666666664 w: 6.529658534196003 b: 0.24801288097906313 reset w, stdv= 0.029462782549439483 reset b, stdv= 0.029462782549439483 w: 6.544403663970284 b: 0.20246035190569983 reset w, stdv= 0.029462782549439483 reset b, stdv= 0.029462782549439483 w: 9.237618805061214 b: 0.2699324704165474 reset w, stdv= 0.020833333333333332 reset b, stdv= 0.020833333333333332 w: 9.240560902888104 b: 0.18776950085546212 reset w, stdv= 0.020833333333333332 reset b, stdv= 0.020833333333333332 w: 9.23323252375467 b: 0.19598698034213305 reset w, stdv= 0.020833333333333332 reset b, stdv= 0.020833333333333332 w: 9.247914516750834 b: 0.19991090497324737 reset w, stdv= 0.020833333333333332 reset b, stdv= 0.020833333333333332 w: 13.062441360447233 b: 0.2709608856088436 reset w, stdv= 0.014731391274719742 reset b, stdv= 0.014731391274719742 w: 13.058955297303523 b: 0.19297756771652977 reset w, stdv= 0.014731391274719742 reset b, stdv= 0.014731391274719742 w: 13.064573213326009 b: 0.19342352625500445 reset w, stdv= 0.014731391274719742 reset b, stdv= 0.014731391274719742 w: 13.060771314609305 b: 0.1931597201764238 reset w, stdv= 0.014731391274719742 reset b, stdv= 0.014731391274719742 w: 13.068217941106957 b: 0.1944194648771781 reset w, stdv= 0.014731391274719742 reset b, stdv= 0.014731391274719742 w: 13.064472494773318 b: 0.1871614021517605 reset w, stdv= 0.014731391274719742 reset b, stdv= 0.014731391274719742 w: 13.065174600640301 b: 0.19473828458164352 reset w, stdv= 0.014731391274719742 reset b, stdv= 0.014731391274719742 w: 13.064107007871193 b: 0.18669129860732317

but during my weights_init() ,it print below info:

  Conv2d (3, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  2.5413928031921387
  0.0
  4.599651336669922
  0.8222851753234863

  Conv2d (64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  11.280376434326172
  0.0
  4.624712944030762
  0.18499651551246643

  Conv2d (64, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  11.323299407958984
  0.0
  6.527068614959717
  0.2626609206199646

  Conv2d (128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  16.010761260986328
  0.0
  6.516138553619385
  0.18262024223804474

  Conv2d (128, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
  16.00145149230957
  0.0
  9.22119426727295
  0.2743944823741913 etc …

Obviously, it has been changed somewhere, for example, the bias was changed to 0 at least.
I don’t know if there another point in the source code which modify the conv weight initialization, who can explain this question to me will be much appreciated! My English is poor, hoping you can understand it :smile:


title: "No route to host" error in torch.distributed.init_process_group

body:
I'm trying to train a simple pytorch model on two GPU servers in parallel. I have compiled pytorch from source. The program gives the "RuntimeError: No route to host" when the process runs on the second server. How do I fix it?

I have tried the anaconda installation and source installation of pytorch, cuda, NCCL.

I have copied the following code from https://yangkky.github.io/2019/07/08/distributed-pytorch-tutorial.html

import os
from datetime import datetime
import argparse
import torch.multiprocessing as mp
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torch.distributed as dist
#from apex.parallel import DistributedDataParallel as DDP
#from apex import amp

class ConvNet(nn.Module):
    def __init__(self, num_classes=10):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(1, 16, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(16),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )
        self.layer2 = nn.Sequential(
            nn.Conv2d(16, 32, kernel_size=5, stride=1, padding=2),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2)
        )
        self.fc = nn.Linear(7*7*32, num_classes)

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc(out)
        return out

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('-n', '--nodes', default=1, type=int, metavar='N')
    parser.add_argument('-g', '--gpus', default=1, type=int, help='number of gpus per node')
    parser.add_argument('-nr', '--nr', default=0, type=int, help='ranking within the nodes')
    parser.add_argument('--epochs', default=2, type=int, metavar='N', help='number of total epochs to run')
    args = parser.parse_args()

    args.world_size = args.gpus * args.nodes
    os.environ['MASTER_ADDR'] = '192.168.0.238'
    os.environ['MASTER_PORT'] = '8888'
    mp.spawn(train, nprocs=args.gpus, args=(args,))

    #train(0, args)

def train(gpu, args):
    rank = args.nr * args.gpus + gpu
    dist.init_process_group(
            backend='nccl',
            init_method='env://',
            world_size=args.world_size,
            rank=rank
            )
    model = ConvNet()
    print('gpu:', gpu)
    torch.cuda.set_device(gpu)
    model.cuda(gpu)
    batch_size= 100

    criterion = nn.CrossEntropyLoss().cuda(gpu)
    optimizer = torch.optim.SGD(model.parameters(), 1e-4)

    model = nn.parallel.DistributedDataParallel(model, device_ids=[gpu])

    train_dataset = torchvision.datasets.MNIST(root='./data',
                                              train=True,
                                              transform=transforms.ToTensor(),
                                              download=True)

    train_sampler = torch.utils.data.distributed.DistributedSample(
            train_dataset,
            num_replicas=args.world_size,
            rank=rank
            )

    train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                              batch_size=batch_size,
                                              shuffle=False,
                                              num_workers=0,
                                              pin_memory=True,
                                              sampler=train_sampler)

    start = datetime.now()
    total_step = len(train_loader)
    for epoch in range(args.epochs):
        for i, (images, labels) in enumerate(train_loader):
            images = images.cuda(non_blocking=True)
            labels = labels.cuda(non_blocking=True)

            outputs = model(images)
            loss = criterion(outputs, labels)

            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if (i+1) % 100 == 0 and gpu == 0:
                print('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(
                    epoch + 1,
                    args.epochs,
                    i + 1,
                    total_step,
                    loss.item()
                ))

    if gpu == 0:
        print("Training complete in:" + str(datetime.now() - start))

if __name__ == '__main__':
    main()