title: Pytorch detection of CUDA

body:
Which is the command to see the "correct" CUDA Version that pytorch in conda env is seeing?  This, is a similar question, but doesn't get me far.

nvidia-smi says I have cuda version 10.1

conda list tells me cudatoolkit version is 10.2.89

torch.cuda.is_available() shows FALSE, so it sees No CUDA?

print(torch.cuda.current_device()), I get 10.0.10 (10010??) (it
looks like):

AssertionError: The NVIDIA driver on your system is too old
(found version 10010)

print(torch._C._cuda_getCompiledVersion(), 'cuda compiled version') tells me my version is 10.0.20 (10020??)?

10020 cuda compiled version

Why are there so many different versions? What am I missing?
P.S
I have Nvidia driver 430 on Ubuntu 16.04 with Geforce 1050. It comes
with libcuda1-430 when I installed the driver from additional drivers tab in ubuntu (Software and Updates). I installed pytorch
with conda which also installed the cudatoolkit using conda install -c fastai -c pytorch -c anaconda fastai


title: Pycharm/Pytorch - 'tensor' is not callable

body:
When creating a pytorch (1.0) tensor :

import torch
W = torch.tensor(([1.0]))

Pycharm (2018.3.1) gives me the following warning :

'tensor' is not callable less... (Ctrl+F1)
Inspection info: This inspection highlights attempts to call objects which are not callable, like, for example, tuples

My code works fine (tensor() is callable) but I'd like to understand and get rid of this warning.


I want to run a git project used pytorch and torchtext but when I run it, it raise error:
  File "main.py", line 60, in <module>
    main()
  File "main.py", line 50, in main
    train_iters, dev_iters, test_iters, vocab = load_dataset(config)
  File "/home/esmailza/style transfer/style-transformer/data.py", line 23, in load_dataset
    TEXT = data.Field(batch_first=True, eos_token='<eos>')
AttributeError: module 'torchtext.data' has no attribute 'Field'

torch version  = 1.8.0
torchtext version = 0.9

def load_dataset(config, train_pos='train.pos', train_neg='train.neg',
                 dev_pos='dev.pos', dev_neg='dev.neg',
                 test_pos='test.pos', test_neg='test.neg'):

    root = config.data_path
    TEXT = data.Field(batch_first=True, eos_token='<eos>')

    dataset_fn = lambda name: data.TabularDataset(
        path=root + name,
        format='tsv',
        fields=[('text', TEXT)]
    )


title: What is the difference between .flatten() and .view(-1) in PyTorch?

body:
Both .flatten() and .view(-1) flatten a tensor in PyTorch. What's the difference?

Does .flatten() copy the data of the tensor?
Is .view(-1) faster?
Is there any situation that .flatten() doesn't work?


title: PyTorch Gradient Descent

body:
I am trying to manually implement gradient descent in PyTorch as a learning exercise. I have the following to create my synthetic dataset:

import torch
torch.manual_seed(0)
N = 100
x = torch.rand(N,1)*5
# Let the following command be the true function
y = 2.3 + 5.1*x
# Get some noisy observations
y_obs = y + 2*torch.randn(N,1)

Then I create my predictive function (y_pred) as shown below.

w = torch.randn(1, requires_grad=True)
b = torch.randn(1, requires_grad=True)
y_pred = w*x+b
mse = torch.mean((y_pred-y_obs)**2)

which uses MSE to infer the weights w,b. I use the block below to update the values according to the gradient.

gamma = 1e-2
for i in range(100):
  w = w - gamma *w.grad
  b = b - gamma *b.grad
  mse.backward()

However, the loop only works in the first iteration. The second iteration onwards w.grad is set to None. I am fairly sure the reason this happens is because I am setting w as a function of it self (I might be wrong).

The question is how do I update the weights properly with the gradient information?


title: Higher order gradients in pytorch

body:
I have implemented the following Jacobian function in pytorch.  Unless I have made a mistake, it computes the Jacobian of any tensor w.r.t. any dimensional inputs:

import torch
import torch.autograd as ag

def nd_range(stop, dims = None):
    if dims == None:
        dims = len(stop)
    if not dims:
        yield ()
        return
    for outer in nd_range(stop, dims - 1):
        for inner in range(stop[dims - 1]):
            yield outer + (inner,)

def full_jacobian(f, wrt):
    f_shape = list(f.size())
    wrt_shape = list(wrt.size())
    fs = []

    f_range = nd_range(f_shape)
    wrt_range = nd_range(wrt_shape)

    for f_ind in f_range:
        grad = ag.grad(f[tuple(f_ind)], wrt, retain_graph=True, create_graph=True)[0]
        for i in range(len(f_shape)):
            grad = grad.unsqueeze(0)
        fs.append(grad)

    fj = torch.cat(fs, dim=0)
    fj = fj.view(f_shape + wrt_shape)
    return fj

On top of this, I have tried to implement a recursive function to calculate nth order derivatives:

def nth_derivative(f, wrt, n):
    if n == 1:
        return full_jacobian(f, wrt)
    else:
        deriv = nth_derivative(f, wrt, n-1)
        return full_jacobian(deriv, wrt)

I ran a simple test:

op = torch.ger(s, s)
deep_deriv = nth_derivative(op, s, 5)

Unfortunately, this succeeds in getting me the Hessian...but no higher order derivatives.  I'm aware many higher order derivatives should be 0, but I'd prefer if pytorch can analytically compute that.

One fix has been to change the gradient calculation to:

try:
            grad = ag.grad(f[tuple(f_ind)], wrt, retain_graph=True, create_graph=True)[0]
        except:
            grad = torch.zeros_like(wrt)

Is this the accepted correct way to handle this?  Or is there a better option?  Or do I have the reason for my issue completely wrong to begin with?


title: RuntimeError: Given groups=1, weight of size [64, 3, 3, 3], expected input[4, 5000, 5000, 3] to have 3 channels, but got 5000 channels instead

body:
So, I have a U-Net model and I'm feeding images of 5000x5000x3 into the model and I and getting the error above.

So here is my model.

import torch
import torch.nn as nn

def double_conv(in_channels, out_channels):
    return nn.Sequential(
        nn.Conv2d(in_channels, out_channels, 3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(out_channels, out_channels, 3, padding=1),
        nn.ReLU(inplace=True)
    )

class UNeT(nn.Module):
    def __init__(self, n_class):
        super().__init__()
        self.dconv_down1 = double_conv(3, 64)
        self.dconv_down2 = double_conv(64, 128)
        self.dconv_down3 = double_conv(128, 256)
        self.dconv_down4 = double_conv(256, 512)
        self.maxpool = nn.MaxPool2d(2)
        self.upsample = nn.Upsample(scale_factor=2, mode='bilinear',
                                    align_corners=True)
        self.dconv_up3 = double_conv(256 + 512, 256)
        self.dconv_up2 = double_conv(128 + 256, 128)
        self.dconv_up1 = double_conv(128 + 64, 64)
        self.conv_last = nn.Conv2d(64, n_class, 1)

    def forward(self, x):
        conv1 = self.dconv_down1(x)
        x = self.maxpool(conv1)
        conv2 = self.dconv_down2(x)
        x = self.maxpool(conv2)
        conv3 = self.dconv_down3(x)
        x = self.maxpool(conv3)
        x = self.dconv_down4(x)
        x = self.upsample(x)
        x = torch.cat([x, conv3], dim=1)
        x = self.dconv_up3(x)
        x = self.upsample(x)
        x = torch.cat([x, conv2], dim=1)
        x = self.dconv_up2(x)
        x = self.upsample(x)
        x = torch.cat([x, conv1], dim=1)
        x = self.dconv_up1(x)
        out = self.conv_last(x)
        return out

I tried to do model(inputs.unsqueeze_(0)) but I got a different error.


title: How to apply a custom function to specific columns in a matrix in PyTorch

body:
I have a tensor of size [150, 182, 91], the first part is just the batch size while the matrix I am interested in is the 182x91 one.

I need to run a function on the 182x91 matrix for each of the 50 dimensions separately.

I need to get a diagonal matrix stripe of the 182x91 matrix, and the function I am using is the following one (based on my previous question: Getting diagonal matrix stripe automatically in numpy or pytorch):

 def stripe(a):

    i, j = a.size()
    assert (i >= j)

    out = torch.zeros((i - j + 1, j))
    for diag in range(0, i - j + 1):
        out[diag] = torch.diag(a, -diag)
    return out

The stripe function expects a matrix of size IxJ and can't deal with the 3rd dimension.

So when I run this:

some_matrix = x # <class 'torch.autograd.variable.Variable'> torch.Size([150, 182, 91])
get_diag = stripe(some_matrix)

I get this Error: ValueError: too many values to unpack (expected 2)

If I just try to skip the first dimension by doing x, i, j = a.size(),
I get this: RuntimeError: invalid argument 1: expected a matrix or a vector at

I am still on PyTorch 0.3.1. Any help is appreciated!


title: PyTorch: new_ones vs ones

body:
In PyTorch what is the difference between new_ones() vs ones(). For example,

x2.new_ones(3,2, dtype=torch.double)

vs

torch.ones(3,2, dtype=torch.double)


title: ModuleNotFoundError: No module named 'past' when installing tensorboard with pytorch 1.2

body:
I'm trying out tensorboard with pytorch by following this: https://pytorch.org/docs/stable/tensorboard.html

I've installed tensorboard with pip install tb-nightly

The command tensorboard --logdir=runs starts ok.

But the line self.writer = SummaryWriter()

Gives the following error:

ModuleNotFoundError: No module named 'past'

How do I fix this?


title: Replace diagonal elements with vector in PyTorch

body:
I have been searching everywhere for something equivalent of the following to PyTorch, but I cannot find anything.

L_1 = np.tril(np.random.normal(scale=1., size=(D, D)), k=0)
L_1[np.diag_indices_from(L_1)] = np.exp(np.diagonal(L_1))

I guess there is no way to replace the diagonal elements in such an elegant way using Pytorch.


title: Pytorch - Stack dimension must be exactly the same?

body:
In pytorch, given the tensors a of shape (1X11) and  b of shape (1X11), torch.stack((a,b),0) would give me a tensor of shape (2X11)

However, when a is of shape (2X11) and b is of shape (1X11), torch.stack((a,b),0) will raise an error cf. "the two tensor size must exactly be the same".

Because the two tensor are the output of a model (gradient included), I can't convert them to numpy to use np.stack() or np.vstack().

Is there any possible solution for least GPU memory usage?


title: Why does Pytorch expect a DoubleTensor instead of a FloatTensor?

body:
From everything I see online, FloatTensors are Pytorch's default for everything, and when I create a tensor to pass to my generator module it is a FloatTensor, but when I try to run it through a linear layer it complains that it wants a DoubleTensor.

class Generator(nn.Module):
  def __init__(self):
    super(Generator, self).__init__()
    self.fully_connected = nn.Linear(100, 1024*4*4, bias=False)

  def forward(self, zvec):
    print(zvec.size())
    fc = self.fully_connected(zvec)
    return(fc.size())

gen = Generator();

gen(torch.from_numpy(np.random.normal(size=100)))

Which produces

RuntimeError: Expected object of type torch.DoubleTensor but found type torch.FloatTensor for argument #2 'mat2'


title: I change the expected object of scalar type float but still got Long in Pytorch

body:
To do the binary class classification. I use binary cross entropy to be the loss function(nn.BCEloss()), and the units of last layer is one.

Before I put (input, target) into loss function, I cast target from Long to float. Only the final step of the DataLoader comes the error messages, and the error message is as bellows.

"RuntimeError: Expected object of scalar type Float but got scalar type Long for argument #2 'target'"

The DataLoader(I drop the last batch if the batch size is not match) is defined in the code, I'm not sure if there is a correlation with the error.

I have tried to print the type of the target and input(output of the Neural Network), and the type of both variable is float. I put the "type result" and the code bellow.

trainloader = torch.utils.data.DataLoader(trainset, batch_size=BATCH_SIZE,
                                          shuffle=True, drop_last=True)
loss_func = nn.BCELoss()

# training
for epoch in range(EPOCH):
    test_loss = 0
    train_loss = 0

    for step, (b_x, b_y) in enumerate(trainloader):        # gives batch data
        b_x = b_x.view(-1, TIME_STEP, 1)              # reshape x to (batch, time_step, input_size)
        print("step: ", step)
        b_x = b_x.to(device)
        print("BEFORE|b_y type: ",b_y.type())
        b_y = b_y.to(device, dtype=torch.float)
        print("AFTER|b_y type: ",b_y.type())
        output = rnn(b_x)                               # rnn output
        print("output type:", output.type())
        loss = loss_func(output, b_y)  # !!!error occurs when trainloader enumerate the final step!!!

        train_loss = train_loss + loss

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

#### type result and the error message####
...
step:  6
BEFORE|b_y type:  torch.LongTensor
AFTER|b_y type:  torch.cuda.FloatTensor
output type: torch.cuda.FloatTensor
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-18-e028fcb6b840> in <module>
     30         b_y = b_y.to(device)
     31         output = rnn(b_x)
---> 32         loss = loss_func(output, b_y)
     33         test_loss = test_loss + loss
     34         rnn.train()

~/venvs/tf1.12/lib/python3.5/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~/venvs/tf1.12/lib/python3.5/site-packages/torch/nn/modules/loss.py in forward(self, input, target)
    502     @weak_script_method
    503     def forward(self, input, target):
--> 504         return F.binary_cross_entropy(input, target, weight=self.weight, reduction=self.reduction)
    505
    506

~/venvs/tf1.12/lib/python3.5/site-packages/torch/nn/functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2025
   2026     return torch._C._nn.binary_cross_entropy(
-> 2027         input, target, weight, reduction_enum)
   2028
   2029

RuntimeError: Expected object of scalar type Float but got scalar type Long for argument #2 'target'


title: How to solve UserWarning: Using a target size (torch.Size([])) that is different to the input size (torch.Size([1]))?

body:
I am trying to run code from a book I purchased about reinforcement learning in Pytorch.
The code should work according to the book, but for me the model doesn't converge and the reward remains negative. It also get the following user warning:

/home/user/.local/lib/python3.6/site-packages/ipykernel_launcher.py:30: UserWarning: Using a target size (torch.Size([])) that is different to the input size (torch.Size([1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

I am a complete beginner in Pytorch, but I assume a size([]) is not a valid tensor size? I think something is going wrong in the code, but after trying to work trough it for a while, I have yet to find anything. I also messaged the book publisher some time ago, but I unfortunately did not hear back from them.

That's why I would like to ask here if anyone has ever seen this error and perhaps knows how to fix it?

The code is for implementing A2C reinforcement learning on a mountain car gym environment. I can also be found here: https://github.com/PacktPublishing/PyTorch-1.x-Reinforcement-Learning-Cookbook/blob/master/Chapter08/chapter8/actor_critic_mountaincar.py

'''
Source codes for PyTorch 1.0 Reinforcement Learning (Packt Publishing)
Chapter 8: Implementing Policy Gradients and Policy Optimization
Author: Yuxi (Hayden) Liu
'''

import torch
import gym
import torch.nn as nn
import torch.nn.functional as F

env = gym.make('MountainCarContinuous-v0')

class ActorCriticModel(nn.Module):
    def __init__(self, n_input, n_output, n_hidden):
        super(ActorCriticModel, self).__init__()
        self.fc = nn.Linear(n_input, n_hidden)
        self.mu = nn.Linear(n_hidden, n_output)
        self.sigma = nn.Linear(n_hidden, n_output)
        self.value = nn.Linear(n_hidden, 1)
        self.distribution = torch.distributions.Normal

    def forward(self, x):
        x = F.relu(self.fc(x))
        mu = 2 * torch.tanh(self.mu(x))
        sigma = F.softplus(self.sigma(x)) + 1e-5
        dist = self.distribution(mu.view(1, ).data, sigma.view(1, ).data)
        value = self.value(x)
        return dist, value

class PolicyNetwork():
    def __init__(self, n_state, n_action, n_hidden, lr=0.001):
        self.model = ActorCriticModel(n_state, n_action, n_hidden)
        self.optimizer = torch.optim.Adam(self.model.parameters(), lr)

    def update(self, returns, log_probs, state_values):
        """
        Update the weights of the Actor Critic network given the training samples
        @param returns: return (cumulative rewards) for each step in an episode
        @param log_probs: log probability for each step
        @param state_values: state-value for each step
        """
        loss = 0
        for log_prob, value, Gt in zip(log_probs, state_values, returns):
            advantage = Gt - value.item()
            policy_loss = - log_prob * advantage

            value_loss = F.smooth_l1_loss(value, Gt)

            loss += policy_loss + value_loss

        self.optimizer.zero_grad()
        loss.backward()
        self.optimizer.step()

    def predict(self, s):
        """
        Compute the output using the continuous Actor Critic model
        @param s: input state
        @return: Gaussian distribution, state_value
        """
        self.model.training = False
        return self.model(torch.Tensor(s))

    def get_action(self, s):
        """
        Estimate the policy and sample an action, compute its log probability
        @param s: input state
        @return: the selected action, log probability, predicted state-value
        """
        dist, state_value = self.predict(s)
        action = dist.sample().numpy()
        log_prob = dist.log_prob(action[0])
        return action, log_prob, state_value

def actor_critic(env, estimator, n_episode, gamma=1.0):
    """
    continuous Actor Critic algorithm
    @param env: Gym environment
    @param estimator: policy network
    @param n_episode: number of episodes
    @param gamma: the discount factor
    """
    for episode in range(n_episode):
        log_probs = []
        rewards = []
        state_values = []
        state = env.reset()

        while True:
            state = scale_state(state)
            action, log_prob, state_value = estimator.get_action(state)
            action = action.clip(env.action_space.low[0],
                                 env.action_space.high[0])
            next_state, reward, is_done, _ = env.step(action)

            total_reward_episode[episode] += reward
            log_probs.append(log_prob)
            state_values.append(state_value)
            rewards.append(reward)

            if is_done:
                returns = []

                Gt = 0
                pw = 0

                for reward in rewards[::-1]:

                    Gt += gamma ** pw * reward
                    pw += 1
                    returns.append(Gt)

                returns = returns[::-1]
                returns = torch.tensor(returns)
                returns = (returns - returns.mean()) / (returns.std() + 1e-9)

                estimator.update(returns, log_probs, state_values)
                print('Episode: {}, total reward: {}'.format(episode, total_reward_episode[episode]))

                break

            state = next_state

import sklearn.preprocessing
import numpy as np

state_space_samples = np.array(
    [env.observation_space.sample() for x in range(10000)])
scaler = sklearn.preprocessing.StandardScaler()
scaler.fit(state_space_samples)

def scale_state(state):
    scaled = scaler.transform([state])
    return scaled[0]

n_state = env.observation_space.shape[0]
n_action = 1
n_hidden = 128
lr = 0.0003
policy_net = PolicyNetwork(n_state, n_action, n_hidden, lr)

n_episode = 200
gamma = 0.9
total_reward_episode = [0] * n_episode

actor_critic(env, policy_net, n_episode, gamma)


title: ImportError: Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training

body:
cannot install apex for distributed and fp16 training of bert model
i have tried to install by cloning the apex from github and tried to install packages using pip

i have tried to install apex by cloning from git hub using following command:

git clone https://github.com/NVIDIA/apex.git

and cd apex to goto apex directory and tried to install package using following pip command:

pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext"

full code is:

def main(server_ip,server_port,local_rank,no_cuda,fp16,train_batch_size,gradient_accumulation_steps,seed,do_train,do_eval,output_dir,task_name,data_dir,do_lower_case,bert_model,num_train_epochs,cache_dir,learning_rate,warmup_proportion,loss_scale,max_seq_length):
        if server_ip and server_port:
            # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
            import ptvsd
            print("Waiting for debugger attach")
            ptvsd.enable_attach(address=(server_ip, server_port), redirect_output=True)
            ptvsd.wait_for_attach()

        processors = {"ner":NerProcessor}
        print(processors)

        if local_rank == -1 or no_cuda:
            device = torch.device("cuda" if torch.cuda.is_available() and not no_cuda else "cpu")
            n_gpu = torch.cuda.device_count()
        else:
            torch.cuda.set_device(local_rank)
            device = torch.device("cuda", local_rank)
            n_gpu = 1
            # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
            torch.distributed.init_process_group(backend='nccl')
        logger.info("device: {} n_gpu: {}, distributed training: {}, 16-bits training: {}".format(
            device, n_gpu, bool(local_rank != -1), fp16))

        if gradient_accumulation_steps < 1:
            raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
                                args.gradient_accumulation_steps))

        train_batch_size = train_batch_size // gradient_accumulation_steps

        random.seed(seed)
        np.random.seed(seed)
        torch.manual_seed(seed)

        if not do_train and not do_eval:
            raise ValueError("At least one of `do_train` or `do_eval` must be True.")

        if os.path.exists(output_dir) and os.listdir(output_dir) and do_train:
            raise ValueError("Output directory ({}) already exists and is not empty.".format(output_dir))
        if not os.path.exists(output_dir):
            os.makedirs(output_dir)

        task_name = task_name.lower()

        if task_name not in processors:
            raise ValueError("Task not found: %s" % (task_name))

        processor = processors[task_name]()
        label_list = processor.get_labels()
        num_labels = len(label_list) + 1

        tokenizer = BertTokenizer.from_pretrained(bert_model, do_lower_case=do_lower_case)

        train_examples = None
        num_train_optimization_steps = None
        if do_train:
            train_examples = processor.get_train_examples(data_dir)
            num_train_optimization_steps = int(
                len(train_examples) / train_batch_size / gradient_accumulation_steps) * num_train_epochs
            if local_rank != -1:
                num_train_optimization_steps = num_train_optimization_steps // torch.distributed.get_world_size()

    #     # Prepare model
        cache_dir = cache_dir if cache_dir else os.path.join(str(PYTORCH_PRETRAINED_BERT_CACHE), 'distributed_{}'.format(local_rank))
        model = Ner.from_pretrained(bert_model,
                  cache_dir=cache_dir,
                  num_labels = num_labels)
        if fp16:
            model.half()
        # model.cuda()
        model.to(device)
        if local_rank != -1:
            try:
                from apex.parallel import DistributedDataParallel as DDP
            except ImportError:
                raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")

            model = DDP(model)
        elif n_gpu > 1:
            model = torch.nn.DataParallel(model)

        param_optimizer = list(model.named_parameters())
        no_decay = ['bias', 'LayerNorm.bias', 'LayerNorm.weight']
        optimizer_grouped_parameters = [
            {'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
            {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
             ]
        if fp16:
            try:
                from apex.optimizers import FP16_Optimizer
                from apex.optimizers import FusedAdam
            except ImportError:
                raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use distributed and fp16 training.")

            optimizer = FusedAdam(optimizer_grouped_parameters,
                                  lr=learning_rate,
                                  bias_correction=False,
                                  max_grad_norm=1.0)
            if loss_scale == 0:
                optimizer = FP16_Optimizer(optimizer, dynamic_loss_scale=True)
            else:
                optimizer = FP16_Optimizer(optimizer, static_loss_scale=loss_scale)

        else:
            optimizer = BertAdam(optimizer_grouped_parameters,
                                 lr=learning_rate,
                                 warmup=warmup_proportion,
                                 t_total=num_train_optimization_steps)

        global_step = 0
        nb_tr_steps = 0
        tr_loss = 0
        label_map = {i : label for i, label in enumerate(label_list,1)}
        if do_train:
            train_features = convert_examples_to_features(
                train_examples, label_list, max_seq_length, tokenizer)
            logger.info("***** Running training *****")
            logger.info("  Num examples = %d", len(train_examples))
            logger.info("  Batch size = %d", train_batch_size)
            logger.info("  Num steps = %d", num_train_optimization_steps)
            all_input_ids = torch.tensor([f.input_ids for f in train_features], dtype=torch.long)
            all_input_mask = torch.tensor([f.input_mask for f in train_features], dtype=torch.long)
            all_segment_ids = torch.tensor([f.segment_ids for f in train_features], dtype=torch.long)
            all_label_ids = torch.tensor([f.label_id for f in train_features], dtype=torch.long)
            all_valid_ids = torch.tensor([f.valid_ids for f in train_features], dtype=torch.long)
            all_lmask_ids = torch.tensor([f.label_mask for f in train_features], dtype=torch.long)
            train_data = TensorDataset(all_input_ids, all_input_mask, all_segment_ids, all_label_ids,all_valid_ids,all_lmask_ids)
            if local_rank == -1:
                train_sampler = RandomSampler(train_data)
            else:
                train_sampler = DistributedSampler(train_data)
            train_dataloader = DataLoader(train_data, sampler=train_sampler, batch_size=train_batch_size)

            model.train()
            for _ in trange(int(num_train_epochs), desc="Epoch"):
                tr_loss = 0
                nb_tr_examples, nb_tr_steps = 0, 0
                for step, batch in enumerate(tqdm(train_dataloader, desc="Iteration")):
                    batch = tuple(t.to(device) for t in batch)
                    input_ids, input_mask, segment_ids, label_ids, valid_ids,l_mask = batch
                    loss = model(input_ids, segment_ids, input_mask, label_ids,valid_ids,l_mask)
                    del loss
                    if n_gpu > 1:
                        loss = loss.mean() # mean() to average on multi-gpu.
                    if gradient_accumulation_steps > 1:
                        loss = loss / gradient_accumulation_steps

                    if fp16:
                        optimizer.backward(loss)
                    else:
                        loss.backward()

                    tr_loss += loss.item()
                    nb_tr_examples += input_ids.size(0)
                    nb_tr_steps += 1
                    if (step + 1) % gradient_accumulation_steps == 0:
                        if fp16:
                            # modify learning rate with special warm up BERT uses
                            # if args.fp16 is False, BertAdam is used that handles this automatically
                            lr_this_step = learning_rate * warmup_linear(global_step/num_train_optimization_steps, warmup_proportion)
                            for param_group in optimizer.param_groups:
                                param_group['lr'] = lr_this_step
                        optimizer.step()
                        optimizer.zero_grad()
                        global_step += 1

main('','',-1,True,True,8,1,42,True,True,'jpt','ner','data/',True,'bert-base-cased',5,'cache_dir',5e-5,0.4,0,128)


title: Saving PyTorch model with no access to model class code

body:
How can I save a PyTorch model without a need for the model class to be defined somewhere?

Disclaimer:
In Best way to save a trained model in PyTorch?, there are no solutions (or a working solution) for saving the model without access to the model class code.


title: Pytorch RuntimeError: [enforce fail at CPUAllocator.cpp:56] posix_memalign(&data, gAlignment, nbytes) == 0. 12 vs 0

body:
I'm building a simple content based recommendations system. In order to compute the cosine similarity in a GPU accelerated way, i'm using Pytorch.

At the time of creating the tfidf vocabulary tensor from a csr_matrix, it promts the following RuntimeErrorr

RuntimeError: [enforce fail at CPUAllocator.cpp:56] posix_memalign(&data, gAlignment, nbytes) == 0. 12 vs 0

I'm doing it in this way:

coo = tfidf_matrix.tocoo()
values = coo.data
indices = np.vstack( (coo.row, coo.col ))
i = torch.LongTensor(indices)
v = torch.FloatTensor(values)
tfidf_matrix_tensor = torch.sparse.FloatTensor(i, v, torch.Size(coo1.shape)).to_dense()
# Prompts the error

I tried with a small test (tfidf matrix size = 10,296) dataset and it works.
The tfidf matrix size from the real dataset is (27639, 226957)


title: Converting python list to pytorch tensor

body:
I have a problem converting a python list of numbers to pytorch Tensor :
this is my code :

caption_feat = [int(x)  if x < 11660  else 3 for x in caption_feat]

printing caption_feat gives : [1, 9903, 7876, 9971, 2770, 2435, 10441, 9370, 2]
I do the converting like this : tmp2 = torch.Tensor(caption_feat)
now printing tmp2 gives : tensor([1.0000e+00, 9.9030e+03, 7.8760e+03, 9.9710e+03, 2.7700e+03, 2.4350e+03,
        1.0441e+04, 9.3700e+03, 2.0000e+00])
However I expected to get : tensor([1. , 9903, , 9971. ......])
Any Idea?


title: How to use PyTorch Tensor.index_select()?

body:
Currently trying to implement REINFORCE algorithm using PyTorch. I wish to be able to gather responsible outputs after having discounted the reward. Hence, given the actions memory, I create a Tensor of indexes, and try to use Tensor.index_select, but no success. Can anyone help ?

    rH = np.array(rH) # discounted reward
    aH = np.array(aH) # action_holder
    sH = np.vstack(np.array(sH)) # states holder

    statesTensor = Variable(torch.from_numpy(sH).type(torch.FloatTensor))
    out = model.forward(statesTensor)

    indexes = GuiltyOnes(out, aH)
    flat = out.view(1,-1)

    respos = torch.index_select(flat, 1, torch.from_numpy(indexes).type(torch.LongTensor))

I get the following error:

    return IndexSelect.apply(self, dim, index)
    RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition


title: Does pytorch apply softmax automatically in nn.Linear

body:
In pytorch a classification network model is defined as this,

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)   # hidden layer
        self.out = torch.nn.Linear(n_hidden, n_output)   # output layer

    def forward(self, x):
        x = F.relu(self.hidden(x))      # activation function for hidden layer
        x = self.out(x)
        return x

Is softmax applied here? In my understanding, things should be like,

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)   # hidden layer
        self.relu =  torch.nn.ReLu(inplace=True)
        self.out = torch.nn.Linear(n_hidden, n_output)   # output layer
        self.softmax = torch.nn.Softmax(dim=n_output)
    def forward(self, x):
        x = self.hidden(x)      # activation function for hidden layer
        x = self.relu(x)
        x = self.out(x)
        x = self.softmax(x)
        return x

I understand that F.relu(self.relu(x)) is also applying relu, but the first block of code doesn't apply softmax, right?


title: Is there a method in Pytorch to count the number of unique values in a way that can be back propagated?

body:
Given the following tensor (which is the result of a network [note the grad_fn]):

tensor([121., 241., 125.,   1., 108., 238., 125., 121.,  13., 117., 121., 229.,
        161.,  13.,   0., 202., 161., 121., 121.,   0., 121., 121., 242., 125.],
       grad_fn=<MvBackward>)

Which we will define as:

xx = torch.tensor([121., 241., 125.,   1., 108., 238., 125., 121.,  13., 117., 121., 229.,
        161.,  13.,   0., 202., 161., 121., 121.,   0., 121., 121., 242., 125.]).requires_grad_(True)

I would like to define an operation which counts the number of occurrences of each value in such a way that the operation will output the following tensor:

tensor([2, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
        0, 7, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0,
        0, 1, 1])

i.e. there are 2 zeros, 1 one, 2 thirteens, etc... the total number of possible values is set upstream, but in this example is 243

So far I have tried the following approaches, which successfully produce the desired tensor, but do not do so in a way that allows computing gradients back through the network:

Attempt 1

tt = []
for i in range(243):
    tt.append((xx == i).unsqueeze(0))
torch.cat(tt,dim=0).sum(dim=1)

Attempt 2

tvtensor = torch.tensor([i for i in range(243)]).unsqueeze(1).repeat(1,xx.shape[0]).float().requires_grad_(True)
(xx==tvtensor).sum(dim=1)

EDIT: Added Attempt

Attempt 3

-- Didn't really expect this to back prop, but figured I would give it a try anyway

ll = torch.zeros((1,243))
for x in xx:
    ll[0,x.long()] += 1

Any help is appreciated

EDIT: As requested the end goal of this is the following:

I am using a technique for calculating structural similarity between two time sequences. One is real and the other is generated. The technique is outlined in this paper (https://link.springer.com/chapter/10.1007/978-3-642-02279-1_33) where a time series is converted to a sequence of code words and the distribution of code words (similar to the way that Bag of Words is used in NLP) is used to represent the time series. Two series are considered similar when the two signal distributions are similar. This is what the counting statistics tensor is for.

What is desired is to be able to construct a loss function which consumes this tensor and measures the distance between the real and generated signal (euclidian norm on the time domain data directly does not work well and this approach claimed better results), so that it can update the generator appropriately.


title: Pytorch - Pick best probability after softmax layer

body:
I have a logistic regression model using Pytorch 0.4.0, where my input is high-dimensional and my output must be a scalar - 0, 1 or 2.

I'm using a linear layer combined with a softmax layer to return a n x 3 tensor, where each column represents the probability of the input falling in one of the three classes (0, 1 or 2).

However, I must return a n x 1 tensor, so I need to somehow pick the highest probability for each input and create a tensor indicating which class had the highest probability. How can I achieve this using Pytorch?

To illustrate, my Softmax outputs this:

[[0.2, 0.1, 0.7],
 [0.6, 0.2, 0.2],
 [0.1, 0.8, 0.1]]

And I must return this:

[[2],
 [0],
 [1]]


title: How to learn the embeddings in Pytorch and retrieve it later

body: I am building a recommendation system where I predict the best item for each user given their purchase history of items. I have userIDs and itemIDs and how much itemID was purchased by userID. I have Millions of users and thousands of products. Not all products are purchased(there are some products that no one has bought them yet). Since the users and items are big I don't want to use one-hot vectors. I am using pytorch and I want to create and train the embeddings so that I can make the predictions for each user-item pair. I followed this tutorial https://pytorch.org/tutorials/beginner/nlp/word_embeddings_tutorial.html. If it's an accurate assumption that the embedding layer is being trained, then do I retrieve the learned weights through model.parameters() method or should I use the embedding.data.weight option?


title: Simple LSTM in PyTorch with Sequential module

body:
In PyTorch, we can define architectures in multiple ways. Here, I'd like to create a simple LSTM network using the Sequential module.

In Lua's torch I would usually go with:

model = nn.Sequential()
model:add(nn.SplitTable(1,2))
model:add(nn.Sequencer(nn.LSTM(inputSize, hiddenSize)))
model:add(nn.SelectTable(-1)) -- last step of output sequence
model:add(nn.Linear(hiddenSize, classes_n))

However, in PyTorch, I don't find the equivalent of SelectTable to get the last output.

nn.Sequential(
  nn.LSTM(inputSize, hiddenSize, 1, batch_first=True),
  # what to put here to retrieve last output of LSTM ?,
  nn.Linear(hiddenSize, classe_n))


title: Pytorch DataLoader multiple data source

body:
I am trying to use Pytorch dataloader to define my own dataset, but I am not sure how to load multiple data source:

My current code:

class MultipleSourceDataSet(Dataset):
    def __init__ (self, json_file, root_dir, transform = None):
        with open(root_dir + 'block0.json') as f:
            self.result = torch.Tensor(json.load(f))

    self.root_dir = root_dir
    self.transform = transform

    def __len__(self):
        return len(self.result[0])

    def __getitem__ (self):
        None

The data source is 50 blocks under root_dir = ~/Documents/blocks/

I split them and avoid to combine them directly before since this is a very big dataset.

How can I load them into a single dataloader?


I´m using Mac OS el capitán and I am trying to follow the quick start tutorial for OpenNMT pytorch version.  In the training step I get the following warning message:

OpenNMT-py/onmt/modules/GlobalAttention.py:177: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.

align_vectors = self.sm(align.view(batch*targetL, sourceL))
/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/torch/nn/modules/container.py:67: UserWarning: Implicit dimension choice for log_softmax has been deprecated. Change the call to include dim=X as an argument.
  input = module(input)

Step 1: Preprocess data (works as expected)

python preprocess.py -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/demo

Step 2: Train model (produces the warning message)

python train.py -data data/demo -save_model demo-model

Has anyone come across this warning or have any pointers to solve it?


title: Using pytorch Cuda on MacBook Pro

body:
I am using MacBook Pro (16-inch, 2019, macOS 10.15.5 (19F96))
GPU

AMD Radeon Pro 5300M
Intel UHD Graphics 630

I am trying to use Pytorch with Cuda on my mac.
All of the guides I saw assume that i have Nvidia graphic card.
I found this: https://github.com/pytorch/pytorch/issues/10657 issue, but it looks like I need to install ROCm, and according to their Supported Operating Systems, it only supports Linux.
Is it possible to run Pytorch on GPU using mac and AMD Graphic card?


title: How can I use KNN, Random Forest models in Pytorch?

body: This may seem like a X Y problem, but initially I had huge data and I was not able to train in given resources (RAM problem). So I thought I could use batch feature of Pytorch. But I want to use Methods like KNN, Random Forest, Clustering except Deep Learning. So is it possible or can I use scikit libraries in Pytorch?


title: Error in Google Colaboratory - AttributeError: module 'PIL.Image' has no attribute 'register_decoder'

body:
I am running this code on Google Colaboratory and I am getting error of register decoder

image_data = dset.ImageFolder(root="drive/SemanticDataset/train/", transform = transforms.Compose([
                                        transforms.Scale(size=img_size),
                                        transforms.CenterCrop(size=(img_size,img_size*2)),
                                        transforms.ToTensor(),
                                        ]))
enter code herelabel_data = dset.ImageFolder(root="drive/SemanticDataset/label/", transform = transforms.Compose([
                                        transforms.Scale(size=img_size),
                                        transforms.CenterCrop(size=(img_size,img_size*2)),
                                        transforms.ToTensor(),
                                        ]))

image_batch = data.DataLoader(image_data, batch_size=batch_size, shuffle=False, num_workers=2)
label_batch = data.DataLoader(label_data, batch_size=batch_size, shuffle=False, num_workers=2)
for i in range(epoch):
   for _, (image, label) in enumerate(zip(image_batch, label_batch)):
       optimizer.zero_grad()

       x = Variable(image, requires_grad=True).cuda()
       y = Variable(label).cuda()

       out = model.forward(x)
       loss = loss_func(out, y)

       loss.backward()
       optimizer.step()

       if _ % 100 == 0:
           print("Epoch: "+i+"| Loss: " , loss)

here is the screenshot of error


title: How to load tfrecord in pytorch?

body:
How to use tfrecord with pytorch?

I have downloaded "Youtube8M" datasets with video-level features, but it is stored in tfrecord.
I tried to read some sample from these file to convert it to numpy and then load in pytorch. But it failed.

    reader = YT8MAggregatedFeatureReader()
    files = tf.gfile.Glob("/Data/youtube8m/train*.tfrecord")
    filename_queue = tf.train.string_input_producer(
        files, num_epochs=5, shuffle=True)
    training_data = [
        reader.prepare_reader(filename_queue) for _ in range(1)
    ]

    unused_video_id, model_input_raw, labels_batch, num_frames = tf.train.shuffle_batch_join(
        training_data,
        batch_size=1024,
        capacity=1024 * 5,
        min_after_dequeue=1024,
        allow_smaller_final_batch=True  ,
        enqueue_many=True)

    with tf.Session() as sess:
        label_numpy = labels_batch.eval()
        print(type(label_numpy))

But this step have no result, just stuck for a long while without any response.


title: How to mask weights in PyTorch weight parameters?

body:
I am attempting to mask (force to zero) specific weight values in PyTorch. The weights I am trying to mask are defined as so in the def __init__

class LSTM_MASK(nn.Module):
        def __init__(self, options, inp_dim):
            super(LSTM_MASK, self).__init__()
            ....
            self.wfx = nn.Linear(input_dim, curernt_output, bias=add_bias)

The mask is also defined in def __init__ as

self.mask_use = torch.Tensor(curernt_output, input_dim)

The mask is a constant and the .requires_grad_() is False for the mask parameter. Now in the def forward part of the class I attempt to do an element-wise multiplication of the weight parameter and the mask before the linear operation is completed

def forward(self, x):
    ....
    self.wfx.weight = self.wfx.weight * self.mask_use
    wfx_out = self.wfx(x)

I get ann error message:

self.wfx.weight = self.wfx.weight * self.mask_use
  File "/home/xyz/anaconda2/lib/python2.7/site-packages/torch/nn/modules/module.py", line 537, in __setattr__
    .format(torch.typename(value), name))
TypeError: cannot assign 'torch.cuda.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)

But when I check on the both parameters with .type() both of them come up as torch.cuda.FloatTensor. I am not sure why there is an error here.


title: PyTorch DataLoader shuffle

body:
I did an experiment and I did not get the result I was expecting.

For the first part, I am using

trainloader = torch.utils.data.DataLoader(trainset, batch_size=128,
                                          shuffle=False, num_workers=0)

I save trainloader.dataset.targets to the variable a, and trainloader.dataset.data to the variable b before training my model. Then, I train the model using trainloader. After the training is finished, I save trainloader.dataset.targets to the variable c, and trainloader.dataset.data to the variable d. Finally, I check a == c and b == d and they both give True, which was expected because the shuffle parameter of the DataLoader is False.

For the second part, I am using

trainloader = torch.utils.data.DataLoader(trainset, batch_size=128,
                                          shuffle=True, num_workers=0)

I save trainloader.dataset.targets to the variable e, and trainloader.dataset.data to the variable f before training my model. Then, I train the model using trainloader. After the training is finished, I save trainloader.dataset.targets to the variable g, and trainloader.dataset.data to the variable h. I expect e == g and f == h to be both False since shuffle=True, but they give True again. What am I missing from the definition of DataLoader class?


title: error installing pytorch using pip on windows 10

body:
I am trying to install pytorch with pip using

pip install torch

or

pip3 install torch===1.3.1 torchvision===0.4.2 -f https://download.pytorch.org/whl/torch_stable.html

with python 3.7.4

and with python 3.8 (latest stable release)

both on 32 and 64 bit.

and getting

Collecting torch   Using cached
https://files.pythonhosted.org/packages/f8/02/880b468bd382dc79896eaecbeb8ce95e9c4b99a24902874a2cef0b562cea/torch-0.1.2.post2.tar.gz
Collecting pyyaml (from torch)   Downloading
https://files.pythonhosted.org/packages/bc/3f/4f733cd0b1b675f34beb290d465a65e0f06b492c00b111d1b75125062de1/PyYAML-5.1.2-cp37-cp37m-win_amd64.whl
(215kB)
    100% |████████████████████████████████| 225kB 1.2MB/s Installing collected packages: pyyaml, torch   Running setup.py install for torch
... error
    Complete output from command C:\Noam\Code\threadart\stav-rl\venv\Scripts\python.exe -u -c "import
setuptools,
tokenize;__file__='C:\\Users\\noams\\AppData\\Local\\Temp\\pip-install-djc6s2t8\\torch\\setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read( ).replace('\r\n',
'\n');f.close();exec(compile(code, __file__, 'exec'))" install
--record C:\Users\noams\AppData\Local\Temp\pip-record-zohv2zo7\install-record.txt
--single-version-externally-managed --compile --install-headers C:\Noam\Code\threadart\stav-rl\venv\inclu de\site\python3.7\torch:
    running install
    running build_deps
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\noams\AppData\Local\Temp\pip-install-djc6s2t8\torch\setup.py",
line 265, in <module>
        description="Tensors and Dynamic neural networks in Python with strong GPU acceleration",
      File "C:\Noam\Code\threadart\stav-rl\venv\lib\site-packages\setuptools-40.8.0-py3.7.egg\setuptools\__init__.py",
line 145, in setup
      File "C:\Python37_x64\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Python37_x64\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "C:\Python37_x64\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\noams\AppData\Local\Temp\pip-install-djc6s2t8\torch\setup.py",
line 99, in run
        self.run_command('build_deps')
      File "C:\Python37_x64\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Python37_x64\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\noams\AppData\Local\Temp\pip-install-djc6s2t8\torch\setup.py",
line 51, in run
        from tools.nnwrap import generate_wrappers as generate_nn_wrappers
    ModuleNotFoundError: No module named 'tools.nnwrap'

    ---------------------------------------- Command "C:\Noam\Code\threadart\stav-rl\venv\Scripts\python.exe -u -c "import
setuptools,
tokenize;__file__='C:\\Users\\noams\\AppData\\Local\\Temp\\pip-install-djc6s2t8\\torch\\setup.py';f=getattr(tokenize,
'open', open)(__file__);code=f.read().replace('\r\n', '\n');
f.close();exec(compile(code, __file__, 'exec'))" install --record
C:\Users\noams\AppData\Local\Temp\pip-record-zohv2zo7\install-record.txt
--single-version-externally-managed --compile --install-headers C:\Noam\Code\threadart\stav-rl\venv\include\site\python3.7\torch"
failed with error code 1 in
C:\Users\noams\AppData\Local\Temp\pip-install-djc6s2t8\torch\

clearly, I am doing something wrong.

Please help!


title: Deleting Rows in Torch Tensor

body:
I have a torch tensor as follows -

a = tensor(
[[0.2215, 0.5859, 0.4782, 0.7411],
[0.3078, 0.3854, 0.3981, 0.5200],
[0.1363, 0.4060, 0.2030, 0.4940],
[0.1640, 0.6025, 0.2267, 0.7036],
[0.2445, 0.3032, 0.3300, 0.4253]],  dtype=torch.float64)

If the first value of each row is less than 0.2 then the whole row needs to be deleted. Thus I need the output like -

tensor(
[[0.2215, 0.5859, 0.4782, 0.7411],
[0.3078, 0.3854, 0.3981, 0.5200],
[0.2445, 0.3032, 0.3300, 0.4253]],  dtype=torch.float64)

I have tried to loop through the tensor and append the valid value to a new empty tensor but was not successful. Is there any way to get the results efficiently?


title: Torchtext AttributeError: 'Example' object has no attribute 'text_content'

body:
I'm working with RNN and using Pytorch & Torchtext. I've got a problem with building vocab in my RNN. My code is as follows:

TEXT = Field(tokenize=tokenizer, lower=True)
LABEL = LabelField(dtype=torch.float)

trainds = TabularDataset(
    path='drive/{}'.format(TRAIN_PATH), format='tsv',
    fields=[
        ('label_start', LABEL),
        ('label_end', None),
        ('title', None),
        ('symbol', None),
        ('text_content', TEXT),
    ])

testds = TabularDataset(
    path='drive/{}'.format(TEST_PATH), format='tsv',
    fields=[
        ('text_content', TEXT),
    ])

TEXT.build_vocab(trainds, testds)

When I want to build vocab, I'm getting this annoying error:

AttributeError: 'Example' object has no attribute 'text_content'

I'm sure, that there is no missing text_content attr. I made try-catch in order to display this specific case:

try:
    print(len(trainds[i]))
except:
    print(trainds[i].text_content)

Surprisingly, I don't get any error and this specific print command shows:

['znana', 'okresie', 'masarni', 'walc', 'y', 'myśl', 'programie', 'sprawy', ...]

So it indicates, that there is text_content attr. When I perform this on a smaller dataset, it works like a charm. This problem occurs when I want to work with proper data. I ran out of ideas. Maybe someone had a similar case and can explain it.

My full traceback:

AttributeError                            Traceback (most recent call last)
<ipython-input-16-cf31866a07e7> in <module>()
    155
    156 if __name__ == "__main__":
--> 157     main()
    158

<ipython-input-16-cf31866a07e7> in main()
    117             break
    118
--> 119     TEXT.build_vocab(trainds, testds)
    120     print('zbudowano dla text')
    121     LABEL.build_vocab(trainds)

/usr/local/lib/python3.6/dist-packages/torchtext/data/field.py in build_vocab(self, *args, **kwargs)
    260                 sources.append(arg)
    261         for data in sources:
--> 262             for x in data:
    263                 if not self.sequential:
    264                     x = [x]

/usr/local/lib/python3.6/dist-packages/torchtext/data/dataset.py in __getattr__(self, attr)
    152         if attr in self.fields:
    153             for x in self.examples:
--> 154                 yield getattr(x, attr)
    155
    156     @classmethod

AttributeError: 'Example' object has no attribute 'text_content'


title: How to count the amount of layers in a CNN?

body:
The Pytorch implementation of ResNet-18.
has the following structure, which appears to be 54 layers, not 18.
So why is it called "18"? How many layers does it actually have?

ResNet (
  (conv1): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
  (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
  (relu): ReLU (inplace)
  (maxpool): MaxPool2d (size=(3, 3), stride=(2, 2), padding=(1, 1), dilation=(1, 1))
  (layer1): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
    )
    (1): BasicBlock (
      (conv1): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(64, 64, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer2): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(64, 128, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(64, 128, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer3): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(128, 256, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (layer4): Sequential (
    (0): BasicBlock (
      (conv1): Conv2d(256, 512, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (downsample): Sequential (
        (0): Conv2d(256, 512, kernel_size=(1, 1), stride=(2, 2), bias=False)
        (1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      )
    )
    (1): BasicBlock (
      (conv1): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
      (relu): ReLU (inplace)
      (conv2): Conv2d(512, 512, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)
      (bn2): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True)
    )
  )
  (avgpool): AvgPool2d (
  )
  (fc): Linear (512 -> 1000)
)


title: PyTorch next(iter(training_loader)) extremely slow, simple data, can't num_workers?

body:
Here x_dat and y_dat are just really long 1-dimensional tensors.

class FunctionDataset(Dataset):
    def __init__(self):
        x_dat, y_dat = data_product()

        self.length = len(x_dat)
        self.y_dat = y_dat
        self.x_dat = x_dat

    def __getitem__(self, index):
        sample = self.x_dat[index]
        label = self.y_dat[index]
        return sample, label

    def __len__(self):
        return self.length

...

data_set = FunctionDataset()

...

training_sampler = SubsetRandomSampler(train_indices)
validation_sampler = SubsetRandomSampler(validation_indices)

training_loader = DataLoader(data_set, sampler=training_sampler, batch_size=params['batch_size'], shuffle=False)
validation_loader = DataLoader(data_set, sampler=validation_sampler, batch_size=valid_size, shuffle=False)

I have also  tried pinning the memory for the two loaders. Setting num_workers to > 0 gives me run-time errors between the processes (like EOF error and interruption errors). I get my batch with:

x_val, target = next(iter(training_loader))

The entire data-set would fit into memory/gpu but I would like to emulate batches for this experiment. Profiling my process gives me the following:

16276989 function calls (16254744 primitive calls) in 38.779 seconds

   Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   1745/1    0.028    0.000   38.780   38.780 {built-in method builtins.exec}
        1    0.052    0.052   38.780   38.780 simple aprox.py:3(<module>)
        1    0.000    0.000   36.900   36.900 simple aprox.py:519(exploreHeatmap)
        1    0.000    0.000   36.900   36.900 simple aprox.py:497(optFromSample)
        1    0.033    0.033   36.900   36.900 simple aprox.py:274(train)
  705/483    0.001    0.000   34.495    0.071 {built-in method builtins.next}
      222    1.525    0.007   34.493    0.155 dataloader.py:311(__next__)
      222    0.851    0.004   12.752    0.057 dataloader.py:314(<listcomp>)
  3016001   11.901    0.000   11.901    0.000 simple aprox.py:176(__getitem__)
       21    0.010    0.000   10.891    0.519 simple aprox.py:413(validationError)
      443    1.380    0.003    9.664    0.022 sampler.py:136(__iter__)
  663/221    2.209    0.003    8.652    0.039 dataloader.py:151(default_collate)
      221    0.070    0.000    6.441    0.029 dataloader.py:187(<listcomp>)
      442    6.369    0.014    6.369    0.014 {built-in method stack}
  3060221    2.799    0.000    5.890    0.000 sampler.py:68(<genexpr>)
  3060000    3.091    0.000    3.091    0.000 tensor.py:382(<lambda>)
      222    0.001    0.000    1.985    0.009 sampler.py:67(__iter__)
      222    1.982    0.009    1.982    0.009 {built-in method randperm}
  663/221    0.002    0.000    1.901    0.009 dataloader.py:192(pin_memory_batch)
      221    0.000    0.000    1.899    0.009 dataloader.py:200(<listcomp>)
....

Suggesting the data loader is immensely slow compared to the remaining activity of my experiment (training the model and lots of other computations etc.). What's going wrong and what would be the best way to speed this up?


title: Masking tensor of same shape in PyTorch

body:
Given an array and mask of same shapes, I want the masked output of the same shape and containing 0 where mask is False.

For example,

# input array
img = torch.randn(2, 2)
print(img)
# tensor([[0.4684, 0.8316],
#        [0.8635, 0.4228]])
print(img.shape)
# torch.Size([2, 2])

# mask
mask = torch.BoolTensor(2, 2)
print(mask)
# tensor([[False,  True],
#        [ True,  True]])
print(mask.shape)
# torch.Size([2, 2])

# expected masked output of shape 2x2
# tensor([[0, 0.8316],
#        [0.8635, 0.4228]])

Issue: The masking changes the shape of the output as follows:

#1: shape changed
img[mask]
# tensor([0.8316, 0.8635, 0.4228])


title: pytorch how to remove cuda() from tensor

body:
I got TypeError: expected torch.LongTensor (got torch.cuda.FloatTensor).

How do I convert torch.cuda.FloatTensor to torch.LongTensor?

  Traceback (most recent call last):
  File "train_v2.py", line 110, in <module>
    main()
  File "train_v2.py", line 81, in main
    model.update(batch)
  File "/home/Desktop/squad_vteam/src/model.py", line 131, in update
    loss_adv = self.adversarial_loss(batch, loss, self.network.lexicon_encoder.embedding.weight, y)
  File "/home/Desktop/squad_vteam/src/model.py", line 94, in adversarial_loss
    adv_embedding = torch.LongTensor(adv_embedding)
TypeError: expected torch.LongTensor (got torch.cuda.FloatTensor)


title: Batch-Matrix multiplication in Pytorch - Confused with the handling of the output's dimension

body:
I got two arrays :

A
B

Array A contains a batch of RGB images, with shape:

[batch, Width, Height, 3]

whereas Array B contains coefficients needed for a "transformation-like" operation on images, with shape:

[batch, 4, 4, 3]

To put it simply, the operation for a single image is a multiplication that outputs an environment map (normalMap * Coefficients).

The output I want should hold shape:

[batch, Width, Height, 3]

I tried using torch.bmm but failed. Is this possible somehow?


title: Implement dropout to fully connected layer in PyTorch

body:
How to apply dropout to the following fully connected network in Pytorch:

class NetworkRelu(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784,128)
        self.fc2 = nn.Linear(128,64)
        self.fc3 = nn.Linear(64,10)

    def forward(self,x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.softmax(self.fc3(x),dim=1)
        return x


title: PyTorch Linear layer input dimension mismatch

body:
Im getting this error when passing the input data to the Linear (Fully Connected Layer) in PyTorch:

matrices expected, got 4D, 2D tensors

I fully understand the problem since the input data has a shape (N,C,H,W) (from a Convolutional+MaxPool layer) where:

N: Data Samples
C: Channels of the data
H,W: Height and Width

Nevertheless I was expecting PyTorch to do the "reshaping" of the data form:

[ N , D1,...Dn] --> [ N, D] where D = D1*D2*....Dn

I try to reshape the Variable.data, but I've read that this approach is not recommended since the gradients will conserve the previous shape, and that in general you should not mutate a Variable.data shape.

I am pretty sure there is a simple solution that goes along with the framework, but i haven't find it.

Is there a good solution for this?

PD: The Fully connected layer has as input size the value C * H * W


title: Understanding accumulated gradients in PyTorch

body:
I am trying to comprehend inner workings of the gradient accumulation in PyTorch. My question is somewhat related to these two:

Why do we need to call zero_grad() in PyTorch?

Why do we need to explicitly call zero_grad()?

Comments to the accepted answer to the second question suggest that accumulated gradients can be used if a minibatch is too large to perform a gradient update in a single forward pass, and thus has to be split into multiple sub-batches.

Consider the following toy example:

import numpy as np
import torch

class ExampleLinear(torch.nn.Module):

    def __init__(self):
        super().__init__()
        # Initialize the weight at 1
        self.weight = torch.nn.Parameter(torch.Tensor([1]).float(),
                                         requires_grad=True)

    def forward(self, x):
        return self.weight * x

if __name__ == "__main__":
    # Example 1
    model = ExampleLinear()

    # Generate some data
    x = torch.from_numpy(np.array([4, 2])).float()
    y = 2 * x

    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)

    y_hat = model(x)          # forward pass

    loss = (y - y_hat) ** 2
    loss = loss.mean()        # MSE loss

    loss.backward()           # backward pass

    optimizer.step()          # weight update

    print(model.weight.grad)  # tensor([-20.])
    print(model.weight)       # tensor([1.2000]

Which is exactly the result one would expect. Now assume that we want to process the dataset sample-by-sample utilizing gradient accumulation:

    # Example 2: MSE sample-by-sample
    model2 = ExampleLinear()
    optimizer = torch.optim.SGD(model2.parameters(), lr=0.01)

    # Compute loss sample-by-sample, then average it over all samples
    loss = []
    for k in range(len(y)):
        y_hat = model2(x[k])
        loss.append((y[k] - y_hat) ** 2)
    loss = sum(loss) / len(y)

    loss.backward()     # backward pass
    optimizer.step()    # weight update

    print(model2.weight.grad)  # tensor([-20.])
    print(model2.weight)       # tensor([1.2000]

Again as expected, the gradient is calculated when the .backward() method is called.

Finally to my question: what exactly happens 'under the hood'?

My understanding is that the computational graph is dynamically updated going from <PowBackward> to <AddBackward> <DivBackward> operations for the loss variable, and that no information about the data used for each forward pass is retained anywhere except for the loss tensor which can be updated until the backward pass.

Are there any caveats to the reasoning in the above paragraph? Lastly, are there any best practices to follow when using gradient accumulation (i.e. can the approach I use in Example 2 backfire somehow)?


title: Despite installing the torch vision pytorch library, I am getting an error saying that there is no module named torch vision

body:
The error that I am getting when I use import torchvision is this:

Error Message

"*Traceback (most recent call last):
  File "/Users/gokulsrin/Desktop/torch_basics/data.py", line 4, in <module>
    import torchvision
ModuleNotFoundError: No module named 'torchvision'*"

I don't know what to do. I have tried changing the version of python from the native one to the one downloaded through anaconda. I am using anaconda as a package manager and have installed torch vision through anaconda as well as through pip commands.


title: How can I select single indices over a dimension in pytorch?

body:
Assume I have a tensor sequences of shape [8, 12, 2]. Now I would like to make a selection of that tensor for each first dimension which results in a tensor of shape [8, 2]. The selection over dimension 1 is specified by indices stored in a long tensor indices of shape [8].

I tried this, however it selects each index in indices for each first dimension in sequences instead of only one.

sequences[:, indices]

How can I make this query without a slow and ugly for loop?


title: TypeError: Cannot handle the data type in PIL Image

body:
I have a Pytorch tensor of size (4,3,224,224). When I am trying to convert the first tensor into an Image object, it says:

TypeError: Cannot handle this data type

I ran the following command:

img = Image.fromarray(data[0][i].numpy().astype(np.uint8))

where data is the Pytorch tensor

I tried other solutions but couldn't find any solution.

Please suggest !!


title: How to solve "RuntimeError: CUDA error: invalid device ordinal"?

body:
I'm trying to run this code. I don't know what is wrong with it, but this code is not running. and I don't know how to solve this problem.
import cv2
from facial_emotion_recognition import EmotionRecognition

emotion_detector = EmotionRecognition(device='gpu', gpu_id=1)
camera = cv2.VideoCapture(0)

while True:
    image = camera.read()[1]
    image = emotion_detector.recognise_emotion(image, return_type='BGR')
    cv2.imshow('Camera', image)

    key = cv2.waitKey(1)
    if key == 27:
        break

camera.release()
cv2.destroyAllWindows()

but I'm getting this error:
Traceback (most recent call last):
  File "/home/fahim/Documents/Python_projects/Python tutorials/pantech AI Master/Computer_Vision/Day 8 Face emotion recognition/emotion.py", line 4, in <module>
    emotion_detector = EmotionRecognition(device='gpu', gpu_id=1)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/facial_emotion_recognition/facial_emotion_recognition.py", line 25, in __init__
    self.network = NetworkV2(in_c=1, nl=32, out_f=7).to(self.device)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 607, in to
    return self._apply(convert)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 354, in _apply
    module._apply(fn)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 376, in _apply
    param_applied = fn(param)
  File "/home/fahim/anaconda3/envs/Computer_Vision/lib/python3.7/site-packages/torch/nn/modules/module.py", line 605, in convert
    return t.to(device, dtype if t.is_floating_point() else None, non_blocking)
RuntimeError: CUDA error: invalid device ordinal

Process finished with exit code 1

This is my the configuration of my computer:
GPU: NVIDIA GeForce MX130
CPU: Intel i5-10210U (8) @ 4.200GHz
Help me to solve this please.


title: Stratified cross validation with Pytorch

body:
My goal is to make binary classification, using neural network.
The problem is that dataset is unbalanced, I have 90% of class 1 and 10 of class 0.
To deal with it I want to use Stratified cross-validation.

The problem that is I am working with Pytorch, I can't find any example and documentation doesn't provide it, and I'm student, quite new for neural networks.

Can anybody help?
Thank you!


title: How does pytorch backprop through argmax?

body:
I'm building Kmeans in pytorch using gradient descent on centroid locations, instead of expectation-maximisation. Loss is the sum of square distances of each point to its nearest centroid.  To identify which centroid is nearest to each point, I use argmin, which is not differentiable everywhere.  However, pytorch is still able to backprop and update weights (centroid locations), giving similar performance to sklearn kmeans on the data.

Any ideas how this is working, or how I can figure this out within pytorch? Discussion on pytorch github suggests argmax is not differentiable: https://github.com/pytorch/pytorch/issues/1339.

Example code below (on random pts):

import numpy as np
import torch

num_pts, batch_size, n_dims, num_clusters, lr = 1000, 100, 200, 20, 1e-5

# generate random points
vector = torch.from_numpy(np.random.rand(num_pts, n_dims)).float()

# randomly pick starting centroids
idx = np.random.choice(num_pts, size=num_clusters)
kmean_centroids = vector[idx][:,None,:] # [num_clusters,1,n_dims]
kmean_centroids = torch.tensor(kmean_centroids, requires_grad=True)

for t in range(4001):
    # get batch
    idx = np.random.choice(num_pts, size=batch_size)
    vector_batch = vector[idx]

    distances = vector_batch - kmean_centroids # [num_clusters, #pts, #dims]
    distances = torch.sum(distances**2, dim=2) # [num_clusters, #pts]

    # argmin
    membership = torch.min(distances, 0)[1] # [#pts]

    # cluster distances
    cluster_loss = 0
    for i in range(num_clusters):
        subset = torch.transpose(distances,0,1)[membership==i]
        if len(subset)!=0: # to prevent NaN
            cluster_loss += torch.sum(subset[:,i])

    cluster_loss.backward()
    print(cluster_loss.item())

    with torch.no_grad():
        kmean_centroids -= lr * kmean_centroids.grad
        kmean_centroids.grad.zero_()


title: How to train a neural network model with bert embeddings instead of static embeddings like glove/fasttext?

body:
I am looking for some heads up to train a conventional neural network model with bert embeddings that are generated dynamically (BERT contextualized embeddings which generates different embeddings for the same word which when comes under different context).

In normal neural network model, we would initialize the model with glove or fasttext embeddings like,

import torch.nn as nn

embed = nn.Embedding(vocab_size, vector_size)

embed.weight.data.copy_(some_variable_containing_vectors)

Instead of copying static vectors like this and use it for training, I want to pass every input to a BERT model and generate embedding for the words on the fly, and feed them to the model for training.

So should I work on changing the forward function in the model for incorporating those embeddings?

Any help would be appreciated!


title: why is my Neural Network stuck at high loss value after the first epochs

body:
I'm doing regression using Neural Networks. It should be a simple task for NN to do, I have 10 features and 1 output that I want to predict.I’m using pytorch for my project but my Model is not learning well. the loss start with a very high value (40000), then after the first 5-10 epochs the loss decrease rapidly to 6000-7000 and then it stuck there, no matter what I make. I tried even to change to skorch instead of pytorch so that I can use cross validation functionality but that also didn’t help. I tried different optimizers and added layers and neurons to the network but that didn’t help, it stuck at 6000 which is a very high loss value. I’m doing regression here, I have 10 features and I’m trying to predict one continuous value. that should be easy to do that’s why it is confusing me more.

here is my network:
I tried here all the possibilities from making more complex architectures like adding layers and units to batch normalization, changing activations etc.. nothing have worked

class BearingNetwork(nn.Module):
    def __init__(self, n_features=X.shape[1], n_out=1):
        super().__init__()
        self.model = nn.Sequential(

            nn.Linear(n_features, 512),
            nn.BatchNorm1d(512),
            nn.LeakyReLU(),
            nn.Linear(512, 64),
            nn.BatchNorm1d(64),
            nn.LeakyReLU(),
            nn.Linear(64, n_out),
#             nn.LeakyReLU(),
#             nn.Linear(256, 128),
#             nn.LeakyReLU(),
#             nn.Linear(128, 64),
#             nn.LeakyReLU(),
#             nn.Linear(64, n_out)
        )

    def forward(self, x):
        out = self.model(x)
        return out

and here are my settings:
using skorch is easier than pytorch. here I'm monitoring also the R2 metric and I made RMSE as a custom metric to also monitor the performance of my model. I also tried the amsgrad for Adam but that didn't help.

R2 = EpochScoring(r2_score, lower_is_better=False, name='R2')
explained_var_score = EpochScoring(EVS, lower_is_better=False, name='EVS Metric')
custom_score = make_scorer(RMSE)
rmse = EpochScoring(custom_score, lower_is_better=True, name='rmse')

bearing_nn = NeuralNetRegressor(

    BearingNetwork,
    criterion=nn.MSELoss,
    optimizer=optim.Adam,
    optimizer__amsgrad=True,
    max_epochs=5000,
    batch_size=128,
    lr=0.001,
    train_split=skorch.dataset.CVSplit(10),
    callbacks=[R2, explained_var_score, rmse, Checkpoint(), EarlyStopping(patience=100)],
    device=device

)

I also standardize the Input values.

my Input have the shape:

torch.Size([39006, 10])

and shape of output is:

torch.Size([39006, 1])

I’m using 128 as my Batch_size but I also tried other values like 32, 64, 512 and even 1024. Although normalizing output is not necessary but I also tried that and It didn’t work when I predict values, the loss is high. Please someone help me on this, I would appreciate every helpful advice. I ll also add a screenshot of my training and val losses and metrics over epochs to visualize how the loss is decreasing in the first 5 epochs and then it stays like forever at the value 6000 which is a very high value for a loss.


title: Quickly find indices that have values larger than a threshold in Numpy/PyTorch

body:
Task

Given a numpy or pytorch matrix, find the indices of cells that have values that are larger than a given threshold.

My implementation

#abs_cosine is the matrix
#sim_vec is the wanted

sim_vec = []
for m in range(abs_cosine.shape[0]):
    for n in range(abs_cosine.shape[1]):
        # exclude diagonal cells
        if m != n and abs_cosine[m][n] >= threshold:
            sim_vec.append((m, n))

Concerns

Speed. All other computations are built on Pytorch, using numpy is already a compromise, because it has moved computations from GPU to CPU. Pure python for loops will make the whole process even worse (for small data set already 5 times slower). I was wondering if we can move the whole computation to Numpy (or pytorch) without invoking any for loops?

An improvement I can think of (but got stuck...)

  bool_cosine = abs_cosine > threshold

which returns a boolean matrix of True and False. But I cannot find a way to quick retrieve the indices of the True cells.


When I run the program I get this error:

  RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

However I had set gen_y = torch.tensor(gen_y,requires_grad=True), but this has not helped, gen_y.grad_fn is None. And I also try x = torch.tensor(x,requires_grad=True), it is not working either. I guess it could be a problem related to version of pytorch. How can I solve this problem?

    def training(self, net, datasets):
        """
          input:
            net: (object) model & optimizer
            datasets : (list) [train, val] dataset object
        """
        args = self.args
        net.model.train()
        steps = len(datasets[0]) // args.batch_size
        if args.trigger == 'epoch':
            args.epochs = args.terminal
            args.iters = steps * args.terminal
            args.iter_interval = steps * args.interval
        else:
            args.epochs = args.terminal // steps + 1
            args.iters = args.terminal
            args.iter_interval = args.interval

        train_loss, train_acc = 0, 0
        start = time.time()
        for epoch in range(1, args.epochs + 1):
            self.epoch = epoch
            # setup data loader
            data_loader = DataLoader(datasets[0], args.batch_size, num_workers=4,
                                     shuffle=True)
            batch_iterator = iter(data_loader)
            for step in range(steps):
                self.iter += 1
                if self.iter > args.iters:
                    self.iter -= 1
                    break
                # convert numpy.ndarray into pytorch tensor
                x, y = next(batch_iterator)
                x = Variable(x)
                #None
                y = Variable(y)
                if args.cuda:
                    x = x.cuda()
                    y = y.cuda()
                # training
                x = torch.tensor(x,requires_grad=True)
                gen_y = net.model(x)
                gen_y = torch.tensor(gen_y,requires_grad=True)
                print(gen_y.requires_grad)
                print(gen_y.grad_fn)
                if self.is_multi:
                    gen_y = gen_y[0]
                    y = y[0]
                loss = F.binary_cross_entropy(gen_y, y)
                # Update generator parameters

                net.optimizer.zero_grad()
                loss.backward()


title: How to get the input and output channels in a CNN?

body:
I am specifically looking at the AlexNet architecture found here:
https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py

I am confused as to how they are getting the input and output channels. Based on my readings of the AlexNet, I can't figure out where they are getting outputchannels = 64 from (as the second argument to the Conv2d function). Even if the 256 is split across 2 GPUs, that should give 128 rather than 64. The input channel of 3 initially represents the color channels as per my assumption. However, the other input and output channels don't make sense to me either.

Could anyone clarify what the input and output channels are?

class AlexNet(nn.Module):

def __init__(self, num_classes=1000):
    super(AlexNet, self).__init__()
    self.features = nn.Sequential(
        nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2), #why 64?
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),
        nn.Conv2d(64, 192, kernel_size=5, padding=2),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),
        nn.Conv2d(192, 384, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(384, 256, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.Conv2d(256, 256, kernel_size=3, padding=1),
        nn.ReLU(inplace=True),
        nn.MaxPool2d(kernel_size=3, stride=2),

)


title: PyTorch and CUDA driver

body:
I have CUDA 9.2 installed. For example:

(base) c:\>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Wed_Apr_11_23:16:30_Central_Daylight_Time_2018
Cuda compilation tools, release 9.2, V9.2.88

I installed PyTorch on Windows 10 using:

conda install pytorch cuda92 -c pytorch
pip3 install torchvision

I ran the test script:

(base) c:\>python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from __future__ import print_function
>>> import torch
>>> x = torch.rand(5, 3)
>>> print(x)
tensor([[0.7041, 0.5685, 0.4036],
        [0.3089, 0.5286, 0.3245],
        [0.3504, 0.8638, 0.1118],
        [0.6517, 0.9209, 0.6801],
        [0.0315, 0.1923, 0.8720]])
>>> quit()

So for, so good.  Then I ran:

(base) c:\>python
Python 3.6.5 |Anaconda custom (64-bit)| (default, Mar 29 2018, 13:32:41) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.cuda.is_available()
False
>>>

Why did PyTorch say CUDA was not available?

The GPU is a compute capability 3.0 Quadro K3000M:

(base) C:\Program Files\NVIDIA Corporation\NVSMI>nvidia-smi.exe

Mon Oct 01 16:36:47 2018
NVIDIA-SMI 385.54 Driver Version: 385.54
-------------------------------+----------------------+----------------------
GPU Name TCC/WDDM              | Bus-Id Disp.A        | Volatile Uncorr.
ECC Fan Temp Perf Pwr:Usage/Cap| Memory-Usage         | GPU-Util Compute M.
0 Quadro K3000M WDDM           | 00000000:01:00.0 Off |
N/A N/A 35C  P0 N/A / N/A      | 29MiB / 2048MiB      | 0% Default


title: Finding mean and standard deviation across image channels PyTorch

body:
Say I have a batch of images in the form of tensors with dimensions (B x C x W x H) where B is the batch size, C is the number of channels in the image, and W and H are the width and height of the image respectively. I'm looking to use the transforms.Normalize() function to normalize my images with respect to the mean and standard deviation of the dataset across the C image channels, meaning that I want a resulting tensor in the form 1 x C. Is there a straightforward way to do this?

I tried torch.view(C, -1).mean(1) and torch.view(C, -1).std(1) but I get the error:

view size is not compatible with input tensor's size and stride (at least one dimension spans across two contiguous subspaces). Use .reshape(...) instead.

Edit

After looking into how view() works in PyTorch, I know realize why my approach doesn't work; however, I still can't figure out how to get the per-channel mean and standard deviation.


title: PyTorch CUDA vs Numpy for arithmetic operations? Fastest?

body:
I performed element-wise multiplication using Torch with GPU support and Numpy using the functions below and found that Numpy loops faster than Torch which shouldn't be the case, I doubt.

I want to know how to perform general arithmetic operations with Torch using GPU.

Note: I ran these code snippets in Google Colab notebook

Define the default tensor type to enable global GPU flag

torch.set_default_tensor_type(torch.cuda.FloatTensor if
                              torch.cuda.is_available() else
                              torch.FloatTensor)

Initialize Torch variables

x = torch.Tensor(200, 100)  # Is FloatTensor
y = torch.Tensor(200,100)

Function in question

def mul(d,f):
    g = torch.mul(d,f).cuda()  # I explicitly called cuda() which is not necessary
    return g

When call the function above as
%timeit mul(x,y)

Returns:

  The slowest run took 10.22 times longer than the fastest. This could
  mean hat   an intermediate result is being cached.  10000 loops, best
  of 3: 50.1 µs per loop

Now trial with numpy,

Used the same values from torch variables

x_ = x.data.cpu().numpy()
y_ = y.data.cpu().numpy()

def mul_(d,f):
    g = d*f
    return g

%timeit mul_(x_,y_)

Returns

  The slowest run took 12.10 times longer than the fastest. This could
  mean that an intermediate result is being cached.  100000 loops, best
  of 3: 7.73 µs per loop

Needs some help to understand GPU enabled Torch operations.


title: CUDA HOME in pytorch installation

body:
I installed pytorch via conda with cuda 7.5

conda install pytorch=0.3.0 cuda75 -c pytorch

>>> import torch
>>> torch.cuda.is_available()
True

I didn't do any other installations for cuda other than this, since it looks like pytorch comes with cuda

Now, I am trying to setup yolo2 https://github.com/longcw/yolo2-pytorch

However, I am getting error in ./make.sh command

this is the error

  OSError: The nvcc binary could not be located in your $PATH. Either add it to your path, or set $CUDAHOME

I'm assuming I need to set CUDAHOME in my path, but I am not able to locate any cuda directory having nvcc binary. Any pointers on it?


title: Implementing Luong Attention in PyTorch

body:
I am trying to implement the attention described in Luong et al. 2015 in PyTorch myself, but I couldn't get it work. Below is my code, I am only interested in the "general" attention case for now. I wonder if I am missing any obvious error. It runs, but doesn't seem to learn.

class AttnDecoderRNN(nn.Module):
    def __init__(self, hidden_size, output_size, dropout_p=0.1):
        super(AttnDecoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.output_size = output_size
        self.dropout_p = dropout_p

        self.embedding = nn.Embedding(
            num_embeddings=self.output_size,
            embedding_dim=self.hidden_size
        )
        self.dropout = nn.Dropout(self.dropout_p)
        self.gru = nn.GRU(self.hidden_size, self.hidden_size)
        self.attn = nn.Linear(self.hidden_size, self.hidden_size)
        # hc: [hidden, context]
        self.Whc = nn.Linear(self.hidden_size * 2, self.hidden_size)
        # s: softmax
        self.Ws = nn.Linear(self.hidden_size, self.output_size)

    def forward(self, input, hidden, encoder_outputs):
        embedded = self.embedding(input).view(1, 1, -1)
        embedded = self.dropout(embedded)

        gru_out, hidden = self.gru(embedded, hidden)

        # [0] remove the dimension of directions x layers for now
        attn_prod = torch.mm(self.attn(hidden)[0], encoder_outputs.t())
        attn_weights = F.softmax(attn_prod, dim=1) # eq. 7/8
        context = torch.mm(attn_weights, encoder_outputs)

        # hc: [hidden: context]
        out_hc = F.tanh(self.Whc(torch.cat([hidden[0], context], dim=1)) # eq.5
        output = F.log_softmax(self.Ws(out_hc), dim=1) eq. 6

        return output, hidden, attn_weights

I have studied the attention implemented in

https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

and

https://github.com/spro/practical-pytorch/blob/master/seq2seq-translation/seq2seq-translation.ipynb

The first one isn't the exact attention mechanism I am looking for. A major disadvantage is that its attention depends on the sequence length (self.attn = nn.Linear(self.hidden_size * 2, self.max_length)), which could be expensive for long sequences.
The second one is more similar to what's described in the paper, but still not the same as there is not tanh. Besides, it is really slow after updating it to latest version of pytorch (ref). Also I don't know why it takes the last context (ref).


title: torch.cuda.is_available() returns false in colab

body:
I am trying to use GPU in google colab. Below are the details of the versions of pytorch and cuda installed in my colab.

Torch 1.3.1 CUDA 10.1.243

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

I am pretty new to using a GPU for transfer learning on pytorch models. My torch.cuda.is_available() returns false and I am unabel to use a GPU. torch.backends.cudnn.enabled returns true. What might be going wrong here?


title: layer Normalization in pytorch?

body:
shouldn't the layer normalization of x = torch.tensor([[1.5,0,0,0,0]]) be [[1.5,-0.5,-0.5,-0.5]] ? according to this paper paper and the equation from the pytorch doc. But the torch.nn.LayerNorm gives [[ 1.7320, -0.5773, -0.5773, -0.5773]]

Here is the example code:

x = torch.tensor([[1.5,.0,.0,.0]])
layerNorm = torch.nn.LayerNorm(4, elementwise_affine = False)

y1 = layerNorm(x)

mean = x.mean(-1, keepdim = True)
var = x.var(-1, keepdim = True)
y2 = (x-mean)/torch.sqrt(var+layerNorm.eps)

where:

y1 == tensor([[ 1.7320, -0.5773, -0.5773, -0.5773]])
y2 == tensor([[ 1.5000, -0.5000, -0.5000, -0.5000]])


title: pytorch: Convert a tuple of FloatTensors into a numpy array

body:
I just got a torchvision.datasets object with MNIST data

train_dataset= dsets.MNIST(root='./data',train=True,transform=transforms.ToTensor(),download=True)

I want to convert this tuple into a set of numpy arrays of shape 60000x28x28 and labels of 60000

I know that the form that the data is provided, can be directly applied into a pytorch neuron in order to be used as training data, but I would like to convert this data into numpy arrays.

the first thing I did was to divide the tuples of (data,labels) with zip(*train_dataset)

data,labels = zip(*train_dataset)

labels is easy to convert into a numpy array, however I have not been able to convert "data" into a numpy array the way I would like. When I try to convert all of the data into numpy.array like

data[:].numpy()

I receive an error telling me 'tuple' object has no attribute 'numpy'. and if I convert this data[:] into numpy, just the first dimension (the 60000 data) is converted into numpy.array but the rest remains as Tensors.

I can convert all of the data using a loop (actually I did it) and it works, however it is really slow. Has anyone done any conversion like this before?

Thanks.


title: Why pytorch DataLoader behaves differently on numpy array and list?

body:
The only difference is one of the parameter passed to DataLoader is in type "numpy.array" and the other is in type "list", but the DataLoader gives totally different results.

You can use the following code to reproduce it:

from torch.utils.data import DataLoader,Dataset
import numpy as np

class my_dataset(Dataset):
    def __init__(self,data,label):
        self.data=data
        self.label=label
    def __getitem__(self, index):
        return self.data[index],self.label[index]
    def __len__(self):
        return len(self.data)

train_data=[[1,2,3],[5,6,7],[11,12,13],[15,16,17]]
train_label=[-1,-2,-11,-12]

########################### Look at here:

test=DataLoader(dataset=my_dataset(np.array(train_data),train_label),batch_size=2)
for i in test:
    print ("numpy data:")
    print (i)
    break

test=DataLoader(dataset=my_dataset(train_data,train_label),batch_size=2)
for i in test:
    print ("list data:")
    print (i)
    break

The result is:

numpy data:
[tensor([[1, 2, 3],
        [5, 6, 7]]), tensor([-1, -2])]
list data:
[[tensor([1, 5]), tensor([2, 6]), tensor([3, 7])], tensor([-1, -2])]


title: Why would Pytorch (CUDA) be running slow on GPU

body:
I have been playing around with Pytorch on Linux for some time now and recently decided to try get more scripts to run with my GPU on my Windows desktop. Since trying this I have noticed a massive performance difference between my GPU execution time and my CPU execution time, on the same scripts, such that my GPU is significantly slow than CPU. To illustrate this I just a tutorial program found here (https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-tensors)

import torch
import datetime
print(torch.__version__)

dtype = torch.double
#device = torch.device("cpu")
device = torch.device("cuda:0")

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1000, 100, 10

# Create random input and output data
x = torch.randn(N, D_in, device=device, dtype=dtype)
y = torch.randn(N, D_out, device=device, dtype=dtype)

# Randomly initialize weights
w1 = torch.randn(D_in, H, device=device, dtype=dtype)
w2 = torch.randn(H, D_out, device=device, dtype=dtype)

start = datetime.datetime.now()
learning_rate = 1e-6
for t in range(5000):
    # Forward pass: compute predicted y
    h = x.mm(w1)
    h_relu = h.clamp(min=0)
    y_pred = h_relu.mm(w2)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    #print(t, loss)

    # Backprop to compute gradients of w1 and w2 with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_w2 = h_relu.t().mm(grad_y_pred)
    grad_h_relu = grad_y_pred.mm(w2.t())
    grad_h = grad_h_relu.clone()
    grad_h[h < 0] = 0
    grad_w1 = x.t().mm(grad_h)

    # Update weights using gradient descent
    w1 -= learning_rate * grad_w1
    w2 -= learning_rate * grad_w2

end = datetime.datetime.now()

print(end-start)

I increased the number of Epoch's from 500 to 5000 as I have read that the first CUDA call is very slow due to initialisation. However the performance issue still exists.

With device = torch.device("cpu") the final time printed out is normal around 3-4 seconds, well device = torch.device("cuda:0") executes in around 13-15 seconds

I have reinstalled Pytorch a number of different ways (uninstalling the previous installation of course) and the problem still persists. I am hoping that someone can help me, if I have perhaps missed a set (didn't install some other API/program) or am doing something wrong in the code.

Python: v3.6

Pytorch:v0.4.1

GPU: NVIDIA GeForce GTX 1060 6GB

Any help would be appreciated :slight_smile:


title: Why is it in Pytorch when I make a COPY of a network's weight it would be automatically updated after back-propagation?

body:
I wrote the following code as a test because in my original network I use a ModuleDict and depends on what index I feed it would slice and train only parts of that network.

I wanted to make sure that only the sliced layers would update their weight so I wrote some test code to double check. Well I am getting some weird results. Say if my model has 2 layers, layer1 is an FC and layer 2 is Conv2d, if I slice the network and ONLY use layer2 I would expect layer1's weight to be unchanged because they are unused and layer2's weight will get updated after 1 epoch.

So my plan was to used a for loop to grab all the weights from the network Before training then I would do it AFTER 1 optimizer.step(). Both times I would store those weights completely separate in 2 Python lists so I can compare their results later. Well for some reason the two lists are completely the same if I compare them with torch.equal() I thought its because maybe there is still some sort of hidden link in memory? So I tried to use .detach() on the weights when I grab them from the loop and the result is still the same. Layer2's weight should be different in this case because it should contain weights from the network before training.

Noted in the code below I am actually using layer1 and ignoring layer2.

Full code:

class mymodel(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer1 = nn.Linear(10, 5)
        self.layer2 = nn.Conv2d(1, 5, 4, 2, 1)
        self.act = nn.Sigmoid()
    def forward(self, x):
        x = self.layer1(x) #only layer1 and act are used layer 2 is ignored so only layer1 and act's weight should be updated
        x = self.act(x)
        return x
model = mymodel()

weights = []

for param in model.parameters(): # loop the weights in the model before updating and store them
    print(param.size())
    weights.append(param)

critertion = nn.BCELoss() #criterion and optimizer setup
optimizer = optim.Adam(model.parameters(), lr = 0.001)

foo = torch.randn(3, 10) #fake input
target = torch.randn(3, 5) #fake target

result = model(foo) #predictions and comparison and backprop
loss = criterion(result, target)
optimizer.zero_grad()
loss.backward()
optimizer.step()

weights_after_backprop = [] # weights after backprop
for param in model.parameters():
    weights_after_backprop.append(param) # only layer1's weight should update, layer2 is not used

for i in zip(weights, weights_after_backprop):
    print(torch.equal(i[0], i[1]))

# **prints all Trues when "layer1" and "act" should be different, I have also tried to call param.detach in the loop but I got the same result.