title: Why does the CrossEntropy loss not go down during training of my network?

body:
I am fairly new to Pytorch and I am currently trying to implement the network in this paper: https://arxiv.org/pdf/1811.06621.pdf?fbclid=IwAR3Ya9ZfBNN40UO0wct7dGupjlBFEpU47IRHK-wXmejI4U2UQGf03sXHMlw.

I have provided the class for this network and some training code that uses dummy data. The code compiles and runs but the loss that's printed each iteration is always the same (8.371). This leads me to believe that there is something wrong with the way I implemented my network. Is there anything glaringly wrong with my implementation?

import torch
import torch.nn as nn
import torchvision.datasets as dsets
import torchvision.transforms as transforms
from torch.autograd import Variable

torch.manual_seed(1)

# Hyper Parameters
sequence_length = 1
input_size = 320
hidden_size = 2048
recurrent_size = 640
num_layers = 8
num_classes = 10
batch_size = 10
num_epochs = 2
learning_rate = 0.01

# RNNT Model

class RNNTModel(nn.Module):
    def __init__(self, input_size, hidden_size, recurrent_size, bias=True):
        super(RNNTModel, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.recurrent_size = recurrent_size
        self.bias = bias
        self.downsample_fc = nn.Linear(self.recurrent_size * 2, self.recurrent_size)
        self.joint_fc = nn.Linear(self.recurrent_size * 2, self.recurrent_size)
        self.out_fc = nn.Linear(640, 4096)
        self.softmax = nn.LogSoftmax(dim=1)

        self.encoder_1 = nn.ModuleDict({
            'lstm1': nn.LSTM(self.input_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj1': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm2': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj2': nn.Linear(self.hidden_size, self.recurrent_size)
        })

        self.encoder_2 = nn.ModuleDict({
            'lstm3': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj3': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm4': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj4': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm5': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj5': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm6': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj6': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm7': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj7': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm8': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj8': nn.Linear(self.hidden_size, self.recurrent_size)
        })

        self.prediction_net = nn.ModuleDict({
            'fc1': nn.Linear(4096, 76),
            'lstm1': nn.LSTM(76, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj1': nn.Linear(self.hidden_size, self.recurrent_size),
            'lstm2': nn.LSTM(self.recurrent_size, self.hidden_size, num_layers=1, bias=bias, batch_first=True),
            'proj2': nn.Linear(self.hidden_size, self.recurrent_size)
        })

    def forward(self, x, ):
        y = [torch.zeros(1, x.size(1), 4096)]
        for i in range(x.size(0) // 2):

            # Unrolled loop of encoder 1
            enc_out, (h1, c1) = self.encoder_1['lstm1'](torch.stack([x[2 * i], x[2 * i + 1]]))
            enc_out = self.encoder_1['proj1'](enc_out)
            enc_out, _ = self.encoder_1['lstm2'](enc_out)
            enc_out = self.encoder_1['proj2'](enc_out)

            # Downsample by halving framrate
            enc_out = enc_out.view(1, -1, 2 * self.recurrent_size)
            enc_out = self.downsample_fc(enc_out)

            # Unrolled loop of encoder 2
            enc_out, _ = self.encoder_2['lstm3'](enc_out)
            enc_out = self.encoder_2['proj3'](enc_out)
            enc_out, _ = self.encoder_2['lstm4'](enc_out)
            enc_out = self.encoder_2['proj4'](enc_out)
            enc_out, _ = self.encoder_2['lstm5'](enc_out)
            enc_out = self.encoder_2['proj5'](enc_out)
            enc_out, _ = self.encoder_2['lstm6'](enc_out)
            enc_out = self.encoder_2['proj6'](enc_out)
            enc_out, _ = self.encoder_2['lstm7'](enc_out)
            enc_out = self.encoder_2['proj3'](enc_out)
            enc_out, _ = self.encoder_2['lstm7'](enc_out)
            enc_out = self.encoder_2['proj3'](enc_out)
            enc_out, _ = self.encoder_2['lstm8'](enc_out)
            enc_out = self.encoder_2['proj8'](enc_out)

            # Unrolled loop of prediction net
            pred_out = self.prediction_net['fc1'](y[i])
            pred_out, _ = self.prediction_net['lstm1'](pred_out)
            pred_out = self.prediction_net['proj1'](pred_out)
            pred_out, _ = self.prediction_net['lstm2'](pred_out)
            pred_out = self.prediction_net['proj2'](pred_out)

            # Unrolled loop of joint layers
            joint_out = torch.cat([enc_out, pred_out], dim=-1)
            joint_out = self.joint_fc(joint_out)
            joint_out = self.out_fc(joint_out)
            joint_out = self.softmax(joint_out)

            y.append(joint_out)

        return(torch.stack(y[1:]))

rnnt = RNNTModel(input_size, hidden_size, recurrent_size, bias=True)
# y = rnnt(torch.rand(batch_size, sequence_length, input_size))

training_data = [(torch.rand(batch_size, sequence_length, input_size), torch.ones(batch_size//2, 1, 4096).long()) for _ in range(100)]

# Loss and Optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(rnnt.parameters(), lr=learning_rate)

# Train the model
for epoch in range(num_epochs):
    for i, (x, y) in enumerate(training_data):
        x = Variable(x)
        y = Variable(y)
        # Forward + Backward + Optimize
        optimizer.zero_grad()
        outputs = rnnt(x).view(-1, 4096)
        loss = criterion(outputs, torch.max(y, 2)[1].squeeze())
        loss.backward()
        optimizer.step

        if (i+1) % 1 == 0:
            print('Epoch [%d/%d], Step [%d/%d], Loss: %.4f'
                   %(epoch+1, num_epochs, i+1, len(training_data)//batch_size, loss.item()))


title: Pytorch model do not update weights

body:
I'm trying to solve the CartPole problem with pytorch, but after several iterations the parameters don't update.

The code that I'm trying to reproduce is this one [cartpole https://github.com/gsurma/cartpole/blob/master/cartpole.py] made in keras.

import random
from collections import deque

import gym
import numpy as np
import torch.nn as nn
import torch

GAMMA = 0.95

MEMORY_SIZE = 1000000
BATCH_SIZE = 50

EXPLORATION_MAX = 1.0
EXPLORATION_MIN = 0.01
EXPLORATION_DECAY = 0.995

# Principal Neural Netword module
class DQNSolver(nn.Module):
    def __init__(self, observation_space, action_space):
        super(DQNSolver, self).__init__()
        self.action_space = action_space
        self.observation_space = observation_space

        self.hiddenSpace = 24

        self.fc1 = nn.Linear(self.observation_space, self.hiddenSpace)
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(self.hiddenSpace, self.action_space)

    def forward(self, x):
        out = self.fc1(x)
        out = self.relu(out)
        out = self.fc2(out)
        return out

class cartpole():

    def __init__(self):

        # Device configuration
        device = torch.device('cpu' if torch.cuda.is_available() else 'cpu')

        self.env = gym.make("CartPole-v1")
        self.observation_space = self.env.observation_space.shape[0]
        self.action_space = self.env.action_space.n

        self.dqn_solver = DQNSolver(self.observation_space, self.action_space).to(device)

        # Create the memory
        self.memory = deque(maxlen=MEMORY_SIZE)
        self.exploration_rate = EXPLORATION_MAX

        # Create the optimizer and the loss
        self.optimizer = torch.optim.Adam(self.dqn_solver.parameters(), lr=0.1)
        self.loss_func = torch.nn.MSELoss()

    def remember(self, state, action, reward, next_state, done):
        self.memory.append((state, action, reward, next_state, done))

    # This custom function will receive a observation of the state of the
    # environment, and then return the action
    def predict_action(self, observation):
        # if np.random.rand() < self.exploration_rate:
        #     randomQ = np.random.rand(1, self.action_space)[0]
        #     return np.double(randomQ)
        predicted = self.dqn_solver(observation)
        predicted = predicted.cpu().data.numpy()
        return np.double(predicted)

    def optimize_model(self, state, q_values):
        output = self.predict_action(state)
        output = torch.tensor(output, requires_grad=False)
        qValues = torch.tensor(q_values, requires_grad=True)

        self.optimizer.zero_grad()
        loss = self.loss_func(output,  qValues)
        loss.backward()
        self.optimizer.step()

        print("Loss: {}".format(loss))

    def experience_replay(self):
        if len(self.memory) < BATCH_SIZE:
            return
        batch = random.sample(self.memory, BATCH_SIZE)
        for state, action, reward, state_next, terminal in batch:
            q_update = reward
            if not terminal:
                next_action = self.predict_action(state_next)
                q_update = (reward + GAMMA * np.amax(next_action))
            q_values = self.predict_action(state)
            q_values[action] = q_update

            self.optimize_model(state, q_values)
        print("Finished replay")
        print('weights after backpropagation = ',   list(self.dqn_solver.parameters()))
        self.exploration_rate *= EXPLORATION_DECAY
        self.exploration_rate = max(EXPLORATION_MIN, self.exploration_rate)

    def run(self):
        while True:
            state = self.env.reset()
            # state = np.reshape(state, [1, self.observation_space])
            state = torch.Tensor(state)
            while True:
                self.env.render()
                action = self.predict_action(state)
                action = np.argmax(action)
                state_next, reward, terminal, info = self.env.step(action)
                reward = reward if not terminal else -reward
                state_next = torch.Tensor(state_next)
                self.remember(state, action, reward, state_next, terminal)
                state = state_next
                if terminal:
                    break
                self.experience_replay()

if __name__ == "__main__":
    cartpole().run()

after several epochs the parameters printed in this line are the same:

print('weights after backpropagation = ',   list(self.dqn_solver.parameters()))

also Loss values are closer to a random value, but don't increase or decrease.
What could be wrong?


title: Autograd function in Pytorch documentation

body:
In the Pytorch documentation https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#sphx-glr-beginner-blitz-autograd-tutorial-py
In the image, I am unable to understand what y.backward(v) means and why do we need to define another tensor v to do the backward operation and also how we got the results of x.grad
Thanks in advance


title: pytorch xla: The element types of the operands to Pad do not match

body:
(edited to provide and explain a minimal reproducible example)

I am seeing the below error when a backward hook is used with pytorch xla.

the error is not seen when the pytorch-xla is replaced with normal pytorch (aka pytorch cuda).
and the error is not seen with pytorch-xla when the line copying the gradient is commented out in the backward hook.

Traceback (most recent call last):
  File "test1.py", line 30, in <module>
    l.backward()
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/tensor.py", line 198, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch/autograd/__init__.py", line 99, in backward
    allow_unreachable=True)  # allow_unreachable flag
RuntimeError: Error while lowering: f32[1,2,16,16]{3,2,1,0} aten::constant_pad_nd, pad=[0, -1, 0, -1, 0, 0, 0, 0], value=0
XLA builder error: Invalid argument: The element types of the operands to Pad do not match.:
Python Frames:

The minimal code that creates this error:

import torch
import torch_xla
import torch_xla.core.xla_model as xm

def loss(output, target):
    l = torch.sum(output - target)
    return l

model = torch.nn.Sequential(
          # minimal model to reproduce the error
          torch.nn.ConstantPad2d((0, 1, 0, 1), 0),
          torch.nn.Conv2d(1, 2, kernel_size=(3, 3), stride=(2, 2)),
          torch.nn.ConstantPad2d((0, 1, 0, 1), 0),
          torch.nn.Conv2d(2, 2, kernel_size=(3, 3), stride=(2, 2))
        )
model = model.to(xm.xla_device())

def dummyHook(module, gradIn, gradOut):
    # error is not seen if i comment out the below line
    g = gradOut[0].cpu()
    print(str(module))

x = torch.randn((1, 1, 32, 32), device=xm.xla_device(), dtype=torch.float)
target = torch.ones((1, 2, 8, 8), device=xm.xla_device(), dtype=torch.float)

for module in model.modules():
    module.register_backward_hook(dummyHook)

y = model(x)
l = loss(y, target)
l.backward()

what could be going wrong ?


title: Best practice to pass PyTorch device name to model

body:
Currently, I separated train.py with model.py for my deep learning project.

So for the datasets, they are sent to cuda device inside the epoch for loop like below.

train.py

...
device = torch.device('cuda:2' if torch.cuda.is_available() else 'cpu')
model = MyNet(~).to(device)
...
for batch_data in train_loader:
    s0 = batch_data[0].to(device)
    s1 = batch_data[1].to(device)
    pred = model(s0, s1)

However, inside my model (in model.py), it also needs to access the device variable for skip connection like method. To make a new copy of hidden unit (for residual connection)

model.py

class MyNet(nn.Module):
    def __init__(self, in_feats, hid_feats, out_feats):
        super(MyNet, self).__init__()
        self.conv1 = GCNConv(in_feats, hid_feats)
        ...

    def forward(self, data):
        x, edge_index = data.x, data.edge_index
        x1 = copy.copy(x.float())
        x = self.conv1(x, edge_index)
        skip_conn = torch.zeros(len(data.batch), x1.size(1)).to(device)  # <--
        (some opps for x1 -> skip_conn)
        x = torch.cat((x, skip_conn), 1)

In this case, I am currently passing device as a parameter, however, I believe this is not a best practice.

Where should be the best practice to send the dataset to CUDA?
In the case of multiple scripts need to access device, how sould I handle this? (parameter, global variable?)


title: Deep Q Learning : How to visualize convergence?

body:
I have trained an RL agent in an environment similar to the Puckworld. Theres no puck though! The agent is in continuous space and wants to reach a fixed target. Each episode the agent is born at a random location and there is an added noise to each action to make learning less trivial.
The reward is given every step as a scaled version of the distance to the target.

I want to plot the convergence of the neural network. The same problem in discrete space and using Q learning, I would plot the sum of all elements in Q matrix vs episode number. This gave me a good understanding of the performance of the network. How can i do the same for a neural network?

Plotting the reward collected in an episode vs episode number is not optimal here.
I use PyTorch. Any help is appreciated


title: ValueError: Expected input batch_size (1) to match target batch_size (26)

body:
I’m getting the following error when using BERT with BiLSTM (my batch_size on BERT is 26). I want to concat last 4 hidden layer of BERT then feed it to BiLSTM. Here is my model:

from transformers import BertPreTrainedModel, BertModel
import torch.nn as nn
import torch
import torch.nn.functional as F

class BERT(BertPreTrainedModel):
    def __init__(self, config):
        super(BERT, self).__init__(config)
        self.device = config.device
        self.num_labels = config.num_labels
        self.bert = BertModel(config)
        self.dropout = nn.Dropout(0.1)

        self.lstm = nn.LSTM(input_size=config.hidden_size * 4, hidden_size=500, num_layers=3, dropout=0.5, bidirectional=True)
        self.qa_outputs = nn.Linear(500*2, config.num_labels)

        self.weight_class = config.weight_class
        self.init_weights()

    def forward(self, input_ids, attention_mask=None, token_type_ids=None):
        with torch.no_grad():
            outputs = self.bert(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
            cls_output = torch.cat((outputs[2][-4][:,0, ...],outputs[2][-3][:,0, ...], outputs[2][-2][:,0, ...], outputs[2][-1][:,0, ...]),-1)
            cls_output = self.lstm(cls_output.unsqueeze(0))[0]
            logits = self.qa_outputs(cls_output)
            return logits

    def loss(self, input_ids, attention_mask, token_type_ids, label):
        outputs = self.bert(input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids)
        cls_output = torch.cat((outputs[2][-4][:,0, ...],outputs[2][-3][:,0, ...], outputs[2][-2][:,0, ...], outputs[2][-1][:,0, ...]),-1)
        cls_output = self.lstm(cls_output.unsqueeze(0))[0]
        logits = self.qa_outputs(cls_output)

        target = label
        class_weights = torch.FloatTensor(self.weight_class).to(self.device)
        loss = F.cross_entropy(logits, target, weight=class_weights)

        predict_value = torch.max(logits, 1)[1]
        list_predict = predict_value.cpu().numpy().tolist()
        list_target = target.cpu().numpy().tolist()

        return loss, list_predict, list_target

Really don't know how to debug this. Any solution for this error.  Thanks in advance.


title: How do I preprocess and train a translation model with OpenNMT-py without parallel data?

body:
I have done tokenization and embedding for two languages.
Thereafter, I use facebookresearch/MUSE to align these two embeddings.
How do I use OpenNMT-py to perform the translation training?
train.py in OpenNMT-py requires preprocessed data, I am guessing using this.
onmt_preprocess -train_src data/src-train.txt -train_tgt data/tgt-train.txt -valid_src data/src-val.txt -valid_tgt data/tgt-val.txt -save_data data/demo

Do I just put the same data for both src and tgt after concatenating the two aligned embeddings?


title: multiplying a vector (1 x N) by a tensor (N x M x M)

body:
I am looking for a matrix operation in numpy or preferably in pytorch that allows one to multiply a vector (1 x N) by a tensor (N x M x M) and get (1 x M x M).  This is easily accomplished using a for loop, but the for loop does not allow back propagation during training. I tried using matmul in numpy and pytorch (and several others such as dot and bmm), but could not get any to work. Here is an example (where M=2, but is 256 in my use case) of what I am trying to do:
a = np.array([1,2,3])
b = np.array([[[1,2],[3,4]],[[5,6],[7,8]],[[9,10],[11,12]]])

I would like to perform the operation: 1*[[1,2],[3,4]] + 2*[[5,6],[7,8]] + 3*[[9,10],[11,12]], which can be achieved with a for loop like this:
for i in range(3):
  matrix_sum += a[i]*b[i]

Any advice or solution would be greatly appreciated.


title: Mixed precision training implementation

body:
So with the release of 1.16 I tried to optimize some of my code for mixed precision and see if I would see a measurable speed up.
I tried two training loops (both with 99% gpu usage on rtx 2060) but I don't measure any speed up, so I would like to show them here to check if my implementation is correct first. My implementation is based on the code sample at https://pytorch.org/blog/accelerating-training-on-nvidia-gpus-with-pytorch-automatic-mixed-precision/
The train function WITHOUT mixed precision:
def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, writer):
    model.train()
    data_len = len(train_loader.dataset)
    train_start_time = time.time()
    for batch_idx, _data in enumerate(train_loader):
        spectrograms, labels, input_lengths, label_lengths = _data
        spectrograms, labels = spectrograms.to(device), labels.to(device)

        optimizer.zero_grad()

        output = model(spectrograms)  # (batch, time, n_class)
        output = F.log_softmax(output, dim=2)
        output = output.transpose(0, 1)  # (time, batch, n_class)

        loss = criterion(output, labels, input_lengths, label_lengths)
        loss.backward()

        writer.add_scalar("Loss/train", loss.item(), iter_meter.get())
        writer.add_scalar("learning_rate", scheduler.get_last_lr()[0], iter_meter.get())
        optimizer.step()
        scheduler.step()
        iter_meter.step()

    return loss.item()

And the train function WITH mixed precision
def train(model, device, train_loader, criterion, optimizer, scheduler, epoch, iter_meter, scaler, writer):
    model.train()
    data_len = len(train_loader.dataset)
    train_start_time = time.time()
    for batch_idx, _data in enumerate(train_loader):

        spectrograms, labels, input_lengths, label_lengths = _data
        spectrograms, labels = spectrograms.to(device), labels.to(device)

        optimizer.zero_grad()
        with torch.cuda.amp.autocast():
            output = model(spectrograms)  # (batch, time, n_class)
            output = F.log_softmax(output, dim=2)
            output = output.transpose(0, 1)  # (time, batch, n_class)

            loss = criterion(output, labels, input_lengths, label_lengths)

        # Mixed precision
        scaler.scale(loss).backward()  # loss.backward()

        scaler.step(optimizer)  # optimizer.step()
        scheduler.step() #Should I also put this steps in the scaler?
        iter_meter.step()

        # Updates the scale for next iteration
        scaler.update()

        writer.add_scalar("Loss/train", loss.item(), iter_meter.get())
        writer.add_scalar("learning_rate", scheduler.get_last_lr()[0], iter_meter.get())
    return loss.item()

Thhe main difference is that I pass the inputs to the model and calculate the loss inside with torch.cuda.amp.autocast(): and later backward with the scaler.
Times for training with both functions take the same time, my thoughts on possible reasons are:

Automatic mixed precision training from pytorch (making both code functions equal)
I missed something in my implementation
My model can't benefit from a speed up from mixed precision training

In case someone wants to check the original model that is being trained I have it on Github
Any experience eyes that can give it a look?


title: How to Perform 2d indexing, gather In Pytorch?

body:
Use case:
I am writing beam search submodule in an NLP program which requires 2d indexing for my imlementation.
The setup is:

k_next_words is shape (batch_size, d1, d2) = (4,5,6)
ix is shape (batch_size, k_num_to_select, ix_dim=2) = (4,7,2),
where the 4 dim is batch size, the 7 dim is number of samples to look up, and the 2 dim is a 2d index corresponding to d1 and d2 in k_next_words (such that the first entry in the index is between 0 and 4 and the second entry is between 0 and 5).

Problem: We want to, for each of the 4 batches gather the 7 entries corresponding to the elements of k_next_words at the 2d index.
This is probably a use case for gather, but I haven’t gotten it to work so far!
Example:
Here is a toy example to illustrate my use case. Let's ignore the batch size (=4) for now.
k_next_words is a torch.Tensor of size (5,6) which looks like:
[[1,  2,  3,  4,  5,  6]
[11, 12, 13, 14, 15, 16]
[21, 22, 23, 24, 25, 26]
[31, 32, 33, 34, 35, 36]
[41, 42, 43, 44, 45, 46]]

ix is a (7,2) tensor of coordinates we want to look up in k_next_words. As you can see, the first entry in the last dimension is always between 0-4 and the second entry in the last dimension is always between 0-5
[[4,5],
 [1,0],
 [0,2],
 [2,2],
 [2,1],
 [3,2],
 [1,1]]

Given these two inputs, I want to look up the 7 entries of k_next_words determined by the indices in ix (note: indexing start at 0), that is, in the toy example:
[[46],
 [11],
 [3],
 [23],
 [22],
 [33],
 [12]]

However, I want to do the above, but also for a given batch size (=4) in the example.


title: Modify existing Pytorch code to run on multiple GPUs

body:
I'm trying to run Pytoch UNet from the following link on 2 or more GPUs
Pytorch-UNet github
the changes the I did till now is:
1.
from:
net = UNet(n_channels=3, n_classes=1, bilinear=True)
logging.info(f'Network:\n'
             f'\t{net.module.n_channels} input channels\n'
             f'\t{net.module.n_classes} output channels (classes)\n'
             f'\t{"Bilinear" if net.module.bilinear else "Transposed conv"} upscaling')

to:
net = UNet(n_channels=3, n_classes=1, bilinear=True)
net = nn.DataParallel(net)
logging.info(f'Network:\n'
             f'\t{net.module.n_channels} input channels\n'
             f'\t{net.module.n_classes} output channels (classes)\n'
             f'\t{"Bilinear" if net.module.bilinear else "Transposed conv"} upscaling')

in each place where was:
net.<something>

replaced to:
net.module.<something>

I know that pytorch see more that 1 GPU because torch.cuda.device_count() return

2

.
But as long a I try to run train that need more momery than what the first GPU have I'm getting:

RuntimeError: CUDA out of memory. Tried to allocate 512.00 MiB (GPU 0;
11.91 GiB total capacity; 10.51 GiB already allocated; 82.56 MiB free; 818.92 MiB cached)

I change required memory to train by changing the batch size.
Any help welcome
EDIT
I see that training run twice faster with 2 GPUs, but max batch size of run with single GPU is the same as for two GPU. Is there any way to use the memory of 2 GPUs together during a single training run?


title: Embedding documents on multi-GPU single-node Docker using pre-trained models of huggingface transformers and pytorch DistributedDataParallel

body:
I am trying to embed some documents including a couple of sentences using huggingface transformers models. I have multi-gpu single-node and I want to do embedding parallel and distributed in all 8 gpus. I tried to use pytorch DistributedDataParallel, but I think all sentences are sending to all GPUs and for all sentences, it is returning one tensor. this is a sample code:
import torch
import torch.nn as nn
import torch.nn.functional as F
import time

import argparse
import os

from transformers import AlbertTokenizer, AlbertModel
import numpy
from tqdm import tqdm
from torch.utils.data import DataLoader,TensorDataset

def parse_args():
    parse = argparse.ArgumentParser()
    parse.add_argument(
            '--local_rank',
            dest = 'local_rank',
            type = int,
            default = 0,
            )
    parse.add_argument("--gpu", type=str, default='None',
                        help="choose gpu device.")
    return parse.parse_args()

def train():
    args = parse_args()

    if not args.gpu == 'None':
        device = torch.device("cuda")
        os.environ["CUDA_VISIBLE_DEVICES"]=args.gpu
    else:
        device = torch.device("cpu")

    torch.cuda.set_device(args.local_rank)

    torch.distributed.init_process_group(
                backend='nccl',
                init_method='env://',
                )

    tokenizer = AlbertTokenizer.from_pretrained('albert-xxlarge-v2')

    sentences=['I love tea',
               'He hates tea',
               'We love tea',
               'python coder',
               'geeksforgeeks',
               'coder in geeksforgeeks']

    sentence_tokens = []
    for sent in (sentences):

        token_id = tokenizer.encode(sent, max_length=128, add_special_tokens=True, pad_to_max_length=True)

        sentence_tokens.append(token_id)
    original_sentences = torch.tensor(sentence_tokens)
    train_dataset = TensorDataset(original_sentences)

    #setup training sampler
    train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset,num_replicas=len(sentences))

    #setup training data loader with the train sampler setup
    train_dataloader = DataLoader(train_dataset, batch_size=16,sampler=train_sampler, shuffle=False)

    model = AlbertModel.from_pretrained('albert-xxlarge-v2', return_dict=True)

    model = model.to(device)
    model = nn.parallel.DistributedDataParallel(model,
            device_ids = [args.local_rank, ],
            output_device = args.local_rank,\
            find_unused_parameters=True
            )
    for batch in (train_dataloader):

        batch_input_tensors = batch[0].to('cuda')
        outputs = model(batch_input_tensors)
        last_hidden_states = outputs.last_hidden_state

        average= torch.mean(last_hidden_states,dim=1)

if __name__ == "__main__":
    train()

all of sentences are sending to all 8 GPUs and output as last_hidden_states is only one tensor. I got the average of tensor elements because I thought at the end they should be same but they aren't.
how can do it distributed and sentences distribute to GPUs and embed over there? and finally for each sentence or for my final case each Doc I have one tensor as feature vector?
thanks


Is RNN for image classification available only for gray image?
The following program works for gray image classification.
If RGB images are used, I have this error:

Expected input batch_size (18) to match target batch_size (6)

at this line loss = criterion(outputs, labels).
My data loading for train, valid and test are as follows.
input_size  = 300
inputH = 300
inputW = 300

#Data transform (normalization & data augmentation)
stats = ((0.4914, 0.4822, 0.4465), (0.2023, 0.1994, 0.2010))
train_resize_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])

train_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.RandomHorizontalFlip(),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
valid_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                         tt.ToTensor(),
                         tt.Normalize(*stats)])
test_tfms = tt.Compose([tt.Resize((inputH, inputW), interpolation=2),
                        tt.ToTensor(),
                        tt.Normalize(*stats)])

#Create dataset
train_ds = ImageFolder('./data/train', train_tfms)
valid_ds = ImageFolder('./data/valid', valid_tfms)
test_ds = ImageFolder('./data/test', test_tfms)

from torch.utils.data.dataloader import DataLoader
batch_size = 6

#Training data loader
train_dl = DataLoader(train_ds, batch_size, shuffle = True, num_workers = 8, pin_memory=True)
#Validation data loader
valid_dl = DataLoader(valid_ds, batch_size, shuffle = True, num_workers = 8, pin_memory=True)
#Test data loader
test_dl = DataLoader(test_ds, 1, shuffle = False, num_workers = 1, pin_memory=True)

My model is as follows.
num_steps = 300
hidden_size = 256 #size of hidden layers
num_classes = 5
num_epochs = 20
learning_rate = 0.001
# Fully connected neural network with one hidden layer
num_layers = 2 # 2 RNN layers are stacked
class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_classes):
        super(RNN, self).__init__()
        self.num_layers = num_layers
        self.hidden_size = hidden_size
        self.rnn = nn.RNN(input_size, hidden_size, num_layers, batch_first=True, dropout=0.2)#batch must have first dimension
        #our inpyt needs to have shape
        #x -> (batch_size, seq, input_size)
        self.fc = nn.Linear(hidden_size, num_classes)#this fc is after RNN. So needs the last hidden size of RNN

    def forward(self, x):
        #according to ducumentation of RNN in pytorch
        #rnn needs input, h_0 for inputs at RNN (h_0 is initial hidden state)

        #the following one is initial hidden layer
        h0 = torch.zeros(self.num_layers, x.size(0), self.hidden_size).to(device)#first one is number of layers and second one is batch size
        #output has two outputs. The first tensor contains the output features of the hidden last layer for all time steps
        #the second one is hidden state f
        out, _ = self.rnn(x, h0)
        #output has batch_size, num_steps, hidden size
        #we need to decode hidden state only the last time step
        #out (N, 30, 128)
        #Since we need only the last time step
        #Out (N, 128)
        out = out[:, -1, :] #-1 for last time step, take all for N and 128
        out = self.fc(out)
        return out

stacked_rnn_model = RNN(input_size, hidden_size, num_layers, num_classes).to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()#cross entropy has softmax at output
#optimizer = torch.optim.Adam(stacked_rnn_model.parameters(), lr=learning_rate) #optimizer used gradient optimization using Adam
optimizer = torch.optim.SGD(stacked_rnn_model.parameters(), lr=learning_rate)
# Train the model
n_total_steps = len(train_dl)
    for epoch in range(num_epochs):
        t_losses=[]
        for i, (images, labels) in enumerate(train_dl):
            # origin shape: [6, 3, 300, 300]
            # resized: [6, 300, 300]
            images = images.reshape(-1, num_steps, input_size).to(device)
            print('images shape')
            print(images.shape)
            labels = labels.to(device)

            # Forward pass
            outputs = stacked_rnn_model(images)
            print('outputs shape')
            print(outputs.shape)
            loss = criterion(outputs, labels)
            t_losses.append(loss)
            # Backward and optimize
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

Printing images and outputs shapes are
images shape
torch.Size([18, 300, 300])
outputs shape
torch.Size([18, 5])

Where is the mistake?


title: Pytorch CNN not learning

body:
I am currently attempting to train a PyTorch CNN to classify demented and non-demented individuals based on MRI scans. However, during training, the loss of the model remains constant and the accuracy, while attempting to differentiate 3 classes, remains at .333. I have tried many of the suggestions offered by respondents to similar questions, yet none of them have worked for my specific task. These pieces of advice included changing the amount of convolutional units in the model, trying different loss functions, training the model on the raw dataset and then scaling up to a larger, augmented set of images, and altering parameters such as learning rate and batch size. I have attached my code and examples of input imagery below.
Image Examples
Healthy Brain
Mild Cognitive Impairment Brain
Alzheimer's Brain
Preprocessing Code
torch.cuda.set_device(0)
g = True
if g == True:
    for f in final_MRI_data:
        path = os.path.join(final_MRI_dir, f)
        matrix = nib.load(path)
        matrix.get_fdata()
        matrix = matrix.get_fdata()
        matrix.shape
        slice_ = matrix[90, :, :]
        img = Image.fromarray(slice_)
        img = img.crop((left, top, right, bottom))
        img = ImageOps.grayscale(img)
        data_matrices.append(img)

postda_data = []
for image in data_matrices:
    for i in range(30):
        transformed_img = transforms(image)
        transformed_img = np.asarray(transformed_img)
        postda_data.append(transformed_img)

final_MRI_labels = list(itertools.chain.from_iterable(itertools.repeat(x, 30) for x in
final_MRI_labels))

X = torch.Tensor(np.asarray([i for i in postda_data])).view(-1, 145, 200)
print(X.size())

y = torch.Tensor([i for i in final_MRI_labels]) #Target labels for cross entropy loss function

z = []
for val in final_MRI_labels:
    z.append(np.eye(3)[val])
z = torch.Tensor(np.asarray(z)) #Target one-hot encoded matrices for model testing function

Network Class
class Hl_Model(nn.Module):

    torch.cuda.set_device(0)

    def __init__(self):
        super().__init__()

        self.conv1 = nn.Conv2d(1, 32, 3, stride=2)
        self.conv2 = nn.Conv2d(32, 64, 3, stride=2)
        self.conv3 = nn.Conv2d(64, 128, 3, stride=2)
        self.conv4 = nn.Conv2d(128, 256, 3, stride=2)

        x = torch.randn(145,200).view(-1,1,145,200)
        self._to_linear = None
        self.convs(x)

        self.fc1 = nn.Linear(self._to_linear, 128, bias=True)
        self.fc2 = nn.Linear(128, 3)

    def convs(self, x):

        x = F.relu(self.conv1(x))
        x = F.relu(self.conv2(x))
        x = F.relu(self.conv3(x))
        x = F.max_pool2d(F.relu(self.conv4(x)), (2, 2), stride=2)

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.softmax(x, dim=1)

Training Function
def train(net, train_fold_x, train_fold_y):

    optimizer = optim.Adam(net.parameters(), lr=0.05)
    BATCH_SIZE = 5
    EPOCHS = 50
    for epoch in range(EPOCHS):
        for i in tqdm(range(0, len(train_fold_x), BATCH_SIZE)):

            batch_x = train_fold_x[i:i+BATCH_SIZE].view(-1, 1, 145, 200)
            batch_y = train_fold_y[i:i+BATCH_SIZE]

            batch_x, batch_y = batch_x.to(device), batch_y.to(device)

            optimizer.zero_grad()
            outputs = net(batch_x)

            batch_y = batch_y.long()
            loss = loss_func(outputs, batch_y)

            loss.backward()
            optimizer.step()

        print(f"Epoch: {epoch} Loss: {loss}")

Testing Function
def test(net, test_fold_x, test_fold_y):

    test_fold_x.to(device)
    test_fold_y.to(device)

    correct = 0
    total = 0
    with torch.no_grad():
        for i in tqdm(range(len(test_fold_x))):
            real_class = torch.argmax(test_fold_y[i]).to(device)
            net_out = net(test_fold_x[i].view(-1, 1, 145, 200).to(device))
            pred_class = torch.argmax(net_out)

            if pred_class == real_class:
                correct += 1
            total +=1

Cross-Validation Loop
for i in range(6):
    result = next(skf.split(X, y))
    X_train = X[result[0]]
    X_test = X[result[1]]
    y_train = y[result[0]]
    y_test = z[result[1]]
    train(hl_model, X_train, y_train)
    test(hl_model, X_test, y_test)

Output During Training:
  0%|          | 0/188 [00:00<?, ?it/s]
  1%|          | 1/188 [00:01<05:35,  1.79s/it]
  5%|4         | 9/188 [00:01<03:45,  1.26s/it]
  9%|8         | 16/188 [00:02<02:32,  1.13it/s]
 12%|#2        | 23/188 [00:02<01:43,  1.60it/s]
 16%|#6        | 31/188 [00:02<01:09,  2.27it/s]
 21%|##        | 39/188 [00:02<00:46,  3.20it/s]
 25%|##5       | 47/188 [00:02<00:31,  4.49it/s]
 30%|##9       | 56/188 [00:02<00:21,  6.26it/s]
 35%|###4      | 65/188 [00:02<00:14,  8.67it/s]
 39%|###9      | 74/188 [00:02<00:09, 11.85it/s]
 44%|####4     | 83/188 [00:02<00:06, 15.92it/s]
 49%|####8     | 92/188 [00:02<00:04, 21.01it/s]
 54%|#####3    | 101/188 [00:03<00:03, 27.03it/s]
 59%|#####8    | 110/188 [00:03<00:02, 33.91it/s]
 63%|######3   | 119/188 [00:03<00:01, 41.21it/s]
 68%|######8   | 128/188 [00:03<00:01, 48.36it/s]
 73%|#######2  | 137/188 [00:03<00:00, 55.36it/s]
 78%|#######7  | 146/188 [00:03<00:00, 61.09it/s]
 82%|########2 | 155/188 [00:03<00:00, 65.87it/s]
 87%|########7 | 164/188 [00:03<00:00, 69.85it/s]
 92%|#########2| 173/188 [00:03<00:00, 72.93it/s]
 97%|#########6| 182/188 [00:04<00:00, 74.88it/s]
100%|##########| 188/188 [00:04<00:00, 45.32it/s]
Epoch: 0 Loss: 1.5514447689056396

  0%|          | 0/188 [00:00<?, ?it/s]
  5%|4         | 9/188 [00:00<00:02, 85.13it/s]
 10%|9         | 18/188 [00:00<00:02, 84.42it/s]
 14%|#4        | 27/188 [00:00<00:01, 83.22it/s]
 19%|#9        | 36/188 [00:00<00:01, 82.64it/s]
 24%|##3       | 45/188 [00:00<00:01, 82.23it/s]
 29%|##8       | 54/188 [00:00<00:01, 82.17it/s]
 34%|###3      | 63/188 [00:00<00:01, 82.13it/s]
 38%|###8      | 72/188 [00:00<00:01, 81.66it/s]
 43%|####2     | 80/188 [00:00<00:01, 79.76it/s]
 47%|####6     | 88/188 [00:01<00:01, 79.66it/s]
 52%|#####1    | 97/188 [00:01<00:01, 80.58it/s]
 56%|#####6    | 106/188 [00:01<00:01, 80.36it/s]
 61%|######1   | 115/188 [00:01<00:00, 80.64it/s]
 66%|######5   | 124/188 [00:01<00:00, 80.84it/s]
 71%|#######   | 133/188 [00:01<00:00, 80.54it/s]
 76%|#######5  | 142/188 [00:01<00:00, 80.98it/s]
 80%|########  | 151/188 [00:01<00:00, 80.86it/s]
 85%|########5 | 160/188 [00:01<00:00, 80.77it/s]
 90%|########9 | 169/188 [00:02<00:00, 78.81it/s]
 94%|#########4| 177/188 [00:02<00:00, 78.53it/s]
 98%|#########8| 185/188 [00:02<00:00, 77.88it/s]
100%|##########| 188/188 [00:02<00:00, 80.35it/s]
Epoch: 1 Loss: 1.5514447689056396

  0%|          | 0/188 [00:00<?, ?it/s]
  5%|4         | 9/188 [00:00<00:02, 83.56it/s]
 10%|9         | 18/188 [00:00<00:02, 82.41it/s]
 14%|#3        | 26/188 [00:00<00:01, 81.49it/s]
 19%|#8        | 35/188 [00:00<00:01, 81.65it/s]
 23%|##3       | 44/188 [00:00<00:01, 81.55it/s]
 28%|##7       | 52/188 [00:00<00:01, 80.41it/s]
 32%|###1      | 60/188 [00:00<00:01, 79.40it/s]
 37%|###6      | 69/188 [00:00<00:01, 80.17it/s]
 41%|####1     | 78/188 [00:00<00:01, 80.29it/s]
 46%|####6     | 87/188 [00:01<00:01, 80.81it/s]
 51%|#####1    | 96/188 [00:01<00:01, 80.95it/s]
 55%|#####5    | 104/188 [00:01<00:01, 80.24it/s]
 60%|######    | 113/188 [00:01<00:00, 80.56it/s]
 65%|######4   | 122/188 [00:01<00:00, 80.56it/s]
 70%|######9   | 131/188 [00:01<00:00, 80.78it/s]
 74%|#######4  | 140/188 [00:01<00:00, 79.65it/s]
 79%|#######9  | 149/188 [00:01<00:00, 80.14it/s]
 84%|########4 | 158/188 [00:01<00:00, 80.70it/s]
 89%|########8 | 167/188 [00:02<00:00, 80.88it/s]
 94%|#########3| 176/188 [00:02<00:00, 81.22it/s]
 98%|#########8| 185/188 [00:02<00:00, 81.03it/s]
100%|##########| 188/188 [00:02<00:00, 80.66it/s]
Epoch: 2 Loss: 1.5514447689056396

This output repeats all the way to "Epoch: 49 Loss: 1.5514447689056396"
Thanks in advance for any advice.


title: PULSE on github (link provided) RuntimeError: CUDA out of memory.... preventing the program "run.py" from executing

body:
(As a student I am kind of new to this but did quite a bit of research and I got pretty far, I'm super into learning something new through this!)
This issue is for the project pulse -> https://github.com/adamian98/pulse
the readme if you scroll down a bit on the page, gives a much better explanation than I could. It will also give a direct "correct" path to judge my actions against and make solving the problem a lot easier.
Objective: run program using the run.py file
Issue: I got a "RuntimeError: CUDA out of memory" despite having a compatible gpu and enough vram
Knowledge: when it comes to coding i just started a few days ago and have a dozen hours with anaconda now, comfterable creating environments.
What I did was... (the list below is a summary and the specific details are after it)

install anaconda

use this .yml file -> https://github.com/leihuayi/pulse/blob/feature/docker/pulse.yml (it changes dependencies to work for windows which is why I needed to grab a different one than the one supplied on the master github page) to create a new environment and install the required packages. It worked fantastically! I only got an error trying to install dlib, it didn't seem compatible with A LOT of the packages and my python version.

I installed the cuda toolkit 10.2 , cmake 3.17.2, and tried to install dlib into the environment directly. the errors spat out in a blaze of glory. The dlib package seems to be only needed for a different .py file and not run.py though so I think it may be unrelated to this error

logs are below and I explain my process in more detail
START DETAILS AND LOGS: from here until the "DETAILS 2" section should be enough information to solve, the rest past there is in case
error log for runing out of memory--> (after executing the "run.py" file)
Loading Synthesis Network
Loading Mapping Network
Running Mapping Network
Traceback (most recent call last):
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\run.py", line 58, in
model = PULSE(cache_dir=kwargs["cache_dir"])
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\PULSE.py", line 44, in init
latent_out = torch.nn.LeakyReLU(5)(mapping(latent))
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\stylegan.py", line 233, in forward
x = super().forward(x)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\container.py", line 100, in forward
input = module(input)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\modules\module.py", line 550, in call
result = self.forward(*input, **kwargs)
File "C:\Users\micha\anaconda3\envs\Pulse1\pulse-master\stylegan.py", line 38, in forward
return F.linear(x, self.weight * self.w_mul, bias)
File "C:\Users\micha\anaconda3\envs\pulse3\lib\site-packages\torch\nn\functional.py", line 1610, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 6.00 GiB total capacity; 3.92 GiB already allocated; 744.91 MiB free; 3.93 GiB reserved in total by PyTorch)

End of error log.
NVIDIA-SMI LOG WHILE RUNNING (checking free memory)
C:\Users\micha>nvidia-smi --query-gpu=memory.free --format=csv --loop=1
memory.free [MiB]

5991 MiB
5991 MiB
5991 MiB
5991 MiB
5897 MiB
5781 MiB
5685 MiB
1643 MiB
5991 MiB
5991 MiB

the program stopped at the 1643MiB
DETAILS PART 1:
I have an nvidia gpu with 6gb of memory which, (according to other logs posted by the creator of the project-> should have enough memory for this to work "We ran our tests with 8GB of memory but I believe that you should be able to run the code with 4GB as well" -adamian98 ). I'm trying to fix that error and get the run.py to work as intended.
Here is the system info using the numba -s command in anaconda (includes hardware info: gpu, my windows version, memory, python version etc.)
START CUDA INFO

Hardware Information
Machine : AMD64
CPU Name : znver1
CPU Count : 16
Number of accessible CPUs : 16
List of accessible CPUs cores : 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
CFS Restrictions (CPUs worth of runtime) : None

CPU Features : 64bit adx aes avx avx2 bmi bmi2
clflushopt clwb clzero cmov cx16
cx8 f16c fma fsgsbase fxsr lzcnt
mmx movbe mwaitx pclmul popcnt
prfchw rdpid rdrnd rdseed sahf sha
sse sse2 sse3 sse4.1 sse4.2 sse4a
ssse3 wbnoinvd xsave xsavec
xsaveopt xsaves

Memory Total (MB) : 15789
Memory Available (MB) : 8421

OS Information
Platform Name : Windows-10-10.0.18362-SP0
Platform Release : 10
OS Name : Windows
OS Version : 10.0.18362
OS Specific Version : 10 10.0.18362 SP0 Multiprocessor Free
Libc Version : ?

Python Information
Python Compiler : MSC v.1916 64 bit (AMD64)
Python Implementation : CPython
Python Version : 3.8.5
Python Locale : en_CA.cp1252

LLVM Information
LLVM Version : 10.0.1

CUDA Information
CUDA Device Initialized : True
CUDA Driver Version : 10020
CUDA Detect Output:
Found 1 CUDA devices
id 0 b'GeForce GTX 1660 Ti with Max-Q Design' [SUPPORTED]
compute capability: 7.5
pci device id: 0
pci bus id: 1
Summary:
1/1 devices are supported

CUDA Librairies Test Output:
Finding cublas from
named cublas.dll
trying to open library... ERROR: failed to open cublas:
Could not find module 'cublas.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cusparse from
named cusparse.dll
trying to open library... ERROR: failed to open cusparse:
Could not find module 'cusparse.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cufft from
named cufft.dll
trying to open library... ERROR: failed to open cufft:
Could not find module 'cufft.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding curand from
named curand.dll
trying to open library... ERROR: failed to open curand:
Could not find module 'curand.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding nvvm from
named nvvm.dll
trying to open library... ERROR: failed to open nvvm:
Could not find module 'nvvm.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding cudart from
named cudart.dll
trying to open library... ERROR: failed to open cudart:
Could not find module 'cudart.dll' (or one of its dependencies). Try using the full path with constructor syntax.
Finding libdevice from
searching for compute_20... ERROR: can't open libdevice for compute_20
searching for compute_30... ERROR: can't open libdevice for compute_30
searching for compute_35... ERROR: can't open libdevice for compute_35
searching for compute_50... ERROR: can't open libdevice for compute_50

ROC information
ROC Available : False
ROC Toolchains : None
HSA Agents Count : 0
HSA Agents:
None
HSA Discrete GPUs Count : 0
HSA Discrete GPUs : None

SVML Information
SVML State, config.USING_SVML : True
SVML Library Loaded : True
llvmlite Using SVML Patched LLVM : True
SVML Operational : True

Threading Layer Information
TBB Threading Layer Available : False
+--> Disabled due to Unknown import problem.
OpenMP Threading Layer Available : True
+-->Vendor: MS
Workqueue Threading Layer Available : True
+-->Workqueue imported successfully.

Numba Environment Variable Information
None found.

Conda Information
Conda Build : 3.20.5
Conda Env : 4.9.2
Conda Platform : win-64
Conda Python Version : 3.8.5.final.0
Conda Root Writable : True

Installed Packages
blas 1.0 mkl
ca-certificates 2020.12.8 haa95532_0
certifi 2020.12.5 py38haa95532_0
cffi 1.14.0 py38h7a1dbc1_0
chardet 3.0.4 py38haa95532_1003
cryptography 2.9.2 py38h7a1dbc1_0
cudatoolkit 10.2.89 h74a9793_1 anaconda
cycler 0.10.0 py38_0
freetype 2.9.1 ha9979f8_1
icc_rt 2019.0.0 h0cc432a_1
icu 58.2 ha925a31_3
idna 2.9 py_1
intel-openmp 2019.4 245
jpeg 9b hb83a4c4_2
kiwisolver 1.2.0 py38h74a9793_0
libcxx 7.0.0 h1ad3211_1002 conda-forge
libpng 1.6.37 h2a8f88b_0
libtiff 4.1.0 h56a325e_0
llvm-meta 7.0.0 0 conda-forge
m2-bash 4.3.042 5
m2-gcc-libs 5.3.0 4
m2-libedit 3.1 20150326
m2-libffi 3.2.1 2
m2-libreadline 6.3.008 8
m2-msys2-runtime 2.5.0.17080.65c939c 3
m2-ncurses 6.0.20160220 2
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs-core 5.3.0 7
m2w64-gmp 6.1.0 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
matplotlib 3.1.3 py38_0
matplotlib-base 3.1.3 py38h64f37c6_0
mkl 2019.4 245
mkl-service 2.3.0 py38h196d8e1_0
mkl_fft 1.0.15 py38h14836fe_0
mkl_random 1.1.0 py38hf9181ef_0
msys2-conda-epoch 20160418 1
ninja 1.9.0 py38h74a9793_0
numpy 1.18.1 py38h93ca92e_0
numpy-base 1.18.1 py38hc3f5095_1
olefile 0.46 py_0
openssl 1.1.1i h2bbff1b_0
pandas 1.0.3 py38h47e9c7a_0
pillow 7.1.2 py38hcc1f983_0
pip 20.0.2 py38_3
powershell_shortcut 0.0.1 3
pycparser 2.20 py_2
pyopenssl 19.1.0 pyhd3eb1b0_1
pyparsing 2.4.7 py_0
pyqt 5.9.2 py38ha925a31_4
pysocks 1.7.1 py38haa95532_0
python 3.8.2 he1778fa_13
python-dateutil 2.8.1 py_0
pytorch 1.5.0 py3.8_cuda102_cudnn7_0 pytorch
pytz 2020.1 py_0
qt 5.9.7 vc14h73c81de_0
requests 2.23.0 py38_0
scipy 1.4.1 py38h9439919_0
setuptools 46.2.0 py38_0
sip 4.19.13 py38ha925a31_0
six 1.14.0 py38haa95532_0
sqlite 3.31.1 h2a8f88b_1
tk 8.6.8 hfa6e2cd_0
torchvision 0.6.0 py38_cu102 pytorch
tornado 6.0.4 py38he774522_1
urllib3 1.25.8 py38_0
vc 14.2 h21ff451_1
vs2015_runtime 14.27.29016 h5e58377_2
wheel 0.34.2 py38_0
win_inet_pton 1.1.0 py38haa95532_0
wincertstore 0.2 py38_0
xz 5.2.5 h62dcd97_0
zlib 1.2.11 h62dcd97_4
zstd 1.3.7 h508b16e_0

No errors reported.

Warning log
Warning (roc): Error initialising ROC: No ROC toolchains found.
Warning (roc): No HSA Agents found, encountered exception when searching: Error at driver init:

HSA is not currently supported on this platform (win32).`

END CUDA INFO
The section below is most likely not needed for this question, here in case:
DETAILS PART 2 (It sounded like dlib (and cmake) were only required to align the faces before therefore theoretically it isn't needed in run.py, but I'm not 100 percent sure. I omitted the section including the dlib specific errors):
I got the .yml installed successfully into the anaconda environment I named "pulse3" with no errors as well as the CUDA toolkit 10.2 and cmake 3.17.2. Dlib is the only thing that gave me some trouble spiting out a bunch of incompatibility errors with the other packages on windows with my python version 3.8.2.


title: How to convert PyTorch tensor to C++ torch::Tensor vice versa?

body:
I want to receive a dictionary includes PyTorch Tensor in C++ module using pybind11, and return the result dictionary with some modification that includes C++ torch::Tensor back. As far as I was looking for, there seems no clear way to convert PyTorch Tensor to C++ Tensor, and C++ Tensor to PyTorch Tensor. For a last trial, I tried to convert PyObject to torch::Tensor but seems not working as well. (https://discuss.pytorch.org/t/is-it-possible-to-get-pyobject-from-a-torch-tensor/85980/2) I want to know if it is correct and there are there any workarounds. I share my code snippet on the below.
py::dict quantize(py::dict target) {
    ...
    for (auto item: target) {
        py::str key(item.first);
        torch::Tensor test = item.second.ptr(); // it fails to compile
    }
    ...
    return py::dict("name"_a="test", "tensor"_a=torch::rand({3, 3, 3})); // it fails on runtime
}


title: Calculate padding for 3D CNN in Pytorch

body:
I'm currently trying to apply a 3D CNN to a set of images with the dimensions of 193 x 229 x 193 and would like to retain the same image dimensions through each convolutional layer (similar to tensorflow's padding=SAME). I know that the padding can be calculated as follow:
S=Stride
P=Padding
W=Width
K=Kernal size

P = ((S-1)*W-S+K)/2

Which yields a padding of 1 for the first layer:
P = ((1-1)*193-1+3)/2
P= 1.0

Although I also get a result of 1.0 for each of the subsequent layers. Anyone have any suggestions? Sorry, beginner here!
Reproducible example:
import torch
import torch.nn as nn

x = torch.randn(1, 1, 193, 229, 193)

padding = ((1-1)*96-1+3)/2
print(padding)

x = nn.Conv3d(in_channels=1, out_channels=8, kernel_size=3, padding=1)(x)
print("shape after conv1: " + str(x.shape))
x = nn.Conv3d(in_channels=8, out_channels=8, kernel_size=3,padding=1)(x)
x = nn.BatchNorm3d(8)(x)
print("shape after conv2 + batch norm: " + str(x.shape))
x = nn.ReLU()(x)
print("shape after reLU:" + str(x.shape))
x = nn.MaxPool3d(kernel_size=2, stride=2)(x)
print("shape after max pool" + str(x.shape))
x = nn.Conv3d(in_channels=8, out_channels=16, kernel_size=3,padding=1)(x)
print("shape after conv3: " + str(x.shape))
x = nn.Conv3d(in_channels=16, out_channels=16, kernel_size=3,padding=1)(x)
print("shape after conv4: " + str(x.shape))

Current output:
shape after conv1: torch.Size([1, 8, 193, 229, 193])
shape after conv2 + batch norm: torch.Size([1, 8, 193, 229, 193])
shape after reLU:torch.Size([1, 8, 193, 229, 193])
shape after max pooltorch.Size([1, 8, 96, 114, 96])
shape after conv3: torch.Size([1, 16, 96, 114, 96])
shape after conv4: torch.Size([1, 16, 96, 114, 96])

Desired output:
shape after conv1: torch.Size([1, 8, 193, 229, 193])
shape after conv2 + batch norm: torch.Size([1, 8, 193, 229, 193])
...
shape after conv3: torch.Size([1, 16, 193, 229, 193])
shape after conv4: torch.Size([1, 16, 193, 229, 193])


title: Xid problem when trying to enable CUDA in TensorFlow and Pytorch

body:
I used to have a working CUDA before my last Linux update(Sabayon Linux, which is effectively Gentoo). Now, I get a very strange error when I try to enable it.
For example:
$python3 -c 'import torch; torch.rand(3).cuda()'
will return:
RuntimeError: CUDA unknown error - this may be due to an incorrectly set up environment, e.g. changing env variable CUDA_VISIBLE_DEVICES after program start. Setting the available devices to be zero.

I have tested my CUDA with nvcc - it works. My python has optirun enabled. optirun works as expected. So there is no problem with the video card or with the general software. The only hint of a problem is this error in dmesg:
NVRM: GPU at PCI:0000:01:00: GPU-02d2502c-1e0f-3318-1ca8-3c980ce85560
NVRM: Xid (PCI:0000:01:00): 31, pid=9410, Ch 00000002, intr 10000000. MMU Fault: ENGINE HOST6 HUBCLIENT_HOST faulted @ 0x1_01010000. Fault is of type FAULT_PDE ACCESS_TYPE_READ

I read that Xid errors might be due to nvidia-persistenced so I run:
sudo nvidia-smi -pm 1
Persistence mode is already Enabled for GPU 00000000:01:00.0.
All done.

I also did try adding to my notebook:
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1"
os.environ['TF_XLA_FLAGS'] = '--tf_xla_enable_xla_devices'

which leads to both
torch._C._cuda_getDeviceCount()  and len(tf.config.list_physical_devices('GPU')) reporting 0.
I really have no idea what more to do. Has anyone encountered similar problem?
Edit: Ok, it seems all the things I did worked, the problem was that I didn't restart after installing cuda-toolkit and adding the path to nvcc.
Now pytorch finds the GPU but TensorFlow doesn't. If I use:
os.environ['TF_XLA_FLAGS'] = '--tf_xla_enable_xla_devices'

device_lib.list_local_devices() finds my GPU. But I don't know how to use it. If I don't use this command, tensorflow finds 0 GPUs which is clearly incorrect. PyTorch works with my GPU fine.


title: Append tensor to each element of another tensor

body:
I have a pytorch tensor: x = torch.zeros(2, 2), and another tensor of variable values: item = torch.tensor([[1, 2], [3, 4]]), I am just giving this tensor for example.
I would like add the item tensor as each element of the x tensor, such that
x = [[item, item],
     [item, item]]

So x is a tensor with tensors inside it.
I have tried assigning item directly to x, but got an error: RuntimeError: The expanded size of the tensor must match the existing size at non-singleton dimension


title: Installing ROCm 4.0.1

body:
With the latest update, Pytorch offers support for ROCm 4.0.1.
I've tried to install the latest version of ROCm using the guide on the official AMD site, but doing that I've only managed to install version 4.0.0.
As I understand at this step
wget -q -O - https://repo.radeon.com/rocm/rocm.gpg.key | sudo apt-key add -

echo 'deb [arch=amd64] https://repo.radeon.com/rocm/apt/debian/ xenial main' | sudo tee /etc/apt/sources.list.d/rocm.list

I should change the repository, but I can't figure out how.
Could someone help me with the exact lines that I should run in order to install 4.0.1?


I was training the lstm network using pytorch and encountered this error in jupyter notebook.

RuntimeError                              Traceback (most recent call last)
<ipython-input-16-b6b1e0b8cad1> in <module>()
      4
      5 # train the model
----> 6 train(net, encoded, epochs=n_epochs, batch_size=batch_size, seq_length=seq_length, lr=0.001, print_every=10)

<ipython-input-14-43dc0cc515e7> in train(net, data, epochs, batch_size, seq_length, lr, clip, val_frac, print_every)
     55
     56             # calculate the loss and perform backprop
---> 57             loss = criterion(output, targets.view(batch_size*seq_length))
     58             loss.backward()
     59             # `clip_grad_norm` helps prevent the exploding gradient problem in RNNs / LSTMs.

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torch\nn\modules\loss.py in forward(self, input, target)
    902     def forward(self, input, target):
    903         return F.cross_entropy(input, target, weight=self.weight,
--> 904                                ignore_index=self.ignore_index, reduction=self.reduction)
    905
    906

~\Anaconda3\lib\site-packages\torch\nn\functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   1968     if size_average is not None or reduce is not None:
   1969         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 1970     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   1971
   1972

~\Anaconda3\lib\site-packages\torch\nn\functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   1788                          .format(input.size(0), target.size(0)))
   1789     if dim == 2:
-> 1790         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   1791     elif dim == 4:
   1792         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: Expected object of scalar type Long but got scalar type Int for argument #2 'target'


title: How do I use Pytorch's "tanslation with a seq2seq" using my own inputs?

body:
I am following the guide here

Currently this is the model:

SOS_token = 0
EOS_token = 1

class Lang:
    def __init__(self, name):
        self.name = name
        self.word2index = {}
        self.word2count = {}
        self.index2word = {0: "SOS", 1: "EOS"}
        self.n_words = 2  # Count SOS and EOS

    def addSentence(self, sentence):
        for word in sentence.split(' '):
            self.addWord(word)

    def addWord(self, word):
        if word not in self.word2index:
            self.word2index[word] = self.n_words
            self.word2count[word] = 1
            self.index2word[self.n_words] = word
            self.n_words += 1
        else:
            self.word2count[word] += 1
def unicodeToAscii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
    )

# Lowercase, trim, and remove non-letter characters

def normalizeString(s):
    s = unicodeToAscii(s.lower().strip())
    s = re.sub(r"([.!?])", r" \1", s)
    s = re.sub(r"[^a-zA-Z.!?]+", r" ", s)
    return s
def readLangs(lang1, lang2, reverse=False):
    print("Reading lines...")

    # Read the file and split into lines
    lines = open('Scribe/%s-%s.txt' % (lang1, lang2), encoding='utf-8').\
        read().strip().split('\n')

    # Split every line into pairs and normalize
    pairs = [[normalizeString(s) for s in l.split('\t')] for l in lines]

    # Reverse pairs, make Lang instances
    if reverse:
        pairs = [list(reversed(p)) for p in pairs]
        input_lang = Lang(lang2)
        output_lang = Lang(lang1)
    else:
        input_lang = Lang(lang1)
        output_lang = Lang(lang2)

    return input_lang, output_lang, pair
MAX_LENGTH = 5000

eng_prefixes = (
    "i am ", "i m ",
    "he is", "he s ",
    "she is", "she s ",
    "you are", "you re ",
    "we are", "we re ",
    "they are", "they re "
)

def filterPair(p):
    return len(p[0].split(' ')) < MAX_LENGTH and \
        len(p[1].split(' ')) < MAX_LENGTH and \
        p[1].startswith(eng_prefixes)

def filterPairs(pairs):
    return [pair for pair in pairs if filterPair(pair)]
def prepareData(lang1, lang2, reverse=False):
    input_lang, output_lang, pairs = readLangs(lang1, lang2, reverse)
    print("Read %s sentence pairs" % len(pairs))
    pairs = filterPairs(pairs)
    print("Trimmed to %s sentence pairs" % len(pairs))
    print("Counting words...")
    for pair in pairs:
        input_lang.addSentence(pair[0])
        output_lang.addSentence(pair[1])
    print("Counted words:")
    print(input_lang.name, input_lang.n_words)
    print(output_lang.name, output_lang.n_words)
    return input_lang, output_lang, pairs

The difference between what I'm trying to do and the guide is that I'm trying  to insert my input languages as list of strings instead of reading them from a file:

pairs=['string one goes like this', 'string two goes like this']
input_lang = Lang(pairs[0][0])
output_lang = Lang(pairs[1][1])

But I it seems like when I try to count the number of words input_lang.n_words in my string I always get 2.

Is there something I'm missing in calling the class Lang?

Update:
I ran

language = Lang('english')

for sentence in pairs: language.addSentence(sentence)

print (language.n_words)

and that gave me the number of words in pairs
Though, that doesn't give me input_lang and output_lang like the guide did:

for pair in pairs:
    input_lang.addSentence(pair[0])
    output_lang.addSentence(pair[1])


title: Unexpected data types when trying to train a pytorch model

body:
I'm putting together a basic neural network to learn pytorch. Attempting to train it always fails with the message "Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'". I suspect I'm doing something wrong with putting the data together, but I don't know what.

The data in question is a couple of one-dimensional lists of numbers that I've generated, which should be linearly separable.

I've pasted my code below.

class MyDataset(Dataset):
    def __init__(self, xs, ys):
        assert len(xs) == len(ys), "Input and output tensors must be the same length"
        self.xs = np.array(xs, dtype=np.double)
        self.ys = np.array(ys, dtype=np.double)

    def __getitem__(self, idx):
        return (self.xs[idx], self.ys[idx])

    def __len__(self):
        return len(self.xs)

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.layer1 = nn.Linear(1, 1)

    def forward(self, x):
        x = F.relu(self.layer1(x))
        return x

def train(data, validation, net, epochs=100):
    learning_rate = 0.01
    optimizer = optim.SGD(net.parameters(), lr=learning_rate)
    criterion = nn.MSELoss()

    for epoch in range(0, epochs):
        print('Beginning epoch ', epoch+1)
        training_losses = []
        validation_losses = []
        for x_batch, y_batch in data:
            optimizer.zero_grad()
            yhat = net(x_batch)
            loss = criterion(y_batch, yhat)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            training_losses.append(loss)

        with torch.no_grad():
            for x_batch, y_batch in validation:
                net.eval()
                yhat = net(x_batch)
                loss = criterion(y_batch, yhat)
                validation_losses.append(loss)

        print('Ending epoch ', epoch+1, 'Training loss: ', np.mean(training_losses), 'Validation loss: ', np.mean(validation_losses))

And this is how I'm generating the data and attempting to train it:

num_samples = 10000
foos = [100 + np.random.normal(scale=20) for x in range(0, num_samples)]
bars = [200 + np.random.normal(scale=20) for x in range(0, num_samples)]

xs = foos + bars
xs = torch.tensor([[x] for x in xs])
ys = np.concatenate([np.zeros(num_samples), np.ones(num_samples)])
ys = torch.tensor([[y] for y in ys])

dataset = MyDataset(xs, ys)

train_dataset, val_dataset = random_split(dataset, [16000, 4000])

train_loader = DataLoader(dataset=train_dataset, batch_size=16)
val_loader = DataLoader(dataset=val_dataset, batch_size=20)

net = Net()
train(train_loader, val_loader, net)

Finally, here's the stack trace:

<ipython-input-114-ab674ae015a5> in train(data, validation, net, epochs)
     13             print('x_batch: ', type(x_batch[0].item()))
     14             print('y_batch: ', type(y_batch[0].item()))
---> 15             yhat = net(x_batch)
     16             loss = criterion(y_batch, yhat)
     17             loss.backward()

/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

<ipython-input-58-ec2e6d981760> in forward(self, x)
      5
      6     def forward(self, x):
----> 7         x = F.relu(self.layer1(x))
      8         return x

/usr/local/lib/python3.6/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    487             result = self._slow_forward(*input, **kwargs)
    488         else:
--> 489             result = self.forward(*input, **kwargs)
    490         for hook in self._forward_hooks.values():
    491             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/site-packages/torch/nn/modules/linear.py in forward(self, input)
     65     @weak_script_method
     66     def forward(self, input):
---> 67         return F.linear(input, self.weight, self.bias)
     68
     69     def extra_repr(self):

/usr/local/lib/python3.6/site-packages/torch/nn/functional.py in linear(input, weight, bias)
   1350     if input.dim() == 2 and bias is not None:
   1351         # fused op is marginally faster
-> 1352         ret = torch.addmm(torch.jit._unwrap_optional(bias), input, weight.t())
   1353     else:
   1354         output = input.matmul(weight.t())

RuntimeError: Expected object of scalar type Float but got scalar type Double for argument #4 'mat1'

I've attempted to debug by logging the types of x_batch and y_batch from within the train method, but they're both showing as float, so I'm stumped as to where the Double is coming from.

Any suggestions?


title: SageMaker Neo custom model_fn and predict_fn

body:
I'm looking at the PyTorch SageMaker Neo compilation examples. They seem to place the required neo_preprocess and neo_postprocess functions inside the train.py file, rather than a serve.py that also encorporates model_fn and predict_fn.

I was wondering, is it possible to instead have a custom serve.py with my own model_fn and predict_fn's, while still using Neo compilation?


title: Where to start on creating a method that saves desired changes in a Tensor with PyTorch?

body:
I have two tensors that I am calculating the Spearmans Rank Correlation from, and I would like to be able to have PyTorch automatically adjust the values in these Tensors in a way that increases my Spearmans Rank Correlation number as high as possible.

I have explored autograd but nothing I've found has explained it simply enough.

Initialized tensors:
a=Var(torch.randn(20,1),requires_grad=True)
psfm_s=Var(torch.randn(12,20),requires_grad=True)

How can I have a loop of constant adjustments of the values in these two tensors to get the highest spearmans rank correlation from 2 lists I make from these 2 tensors while having PyTorch do the work? I just need a guide of where to go. Thank you!


title: Understanding deep Markov model code on pyro

body:
I am going through an a deep Markov model code given on pyro's website:

https://pyro.ai/examples/dmm.html

I am really confused about combiner module that they've implemented. You an find combiner module on line 104 on this GitHub page:

https://github.com/pyro-ppl/pyro/blob/dev/examples/dmm/dmm.py

The article they are following is:

https://arxiv.org/abs/1609.09869

They explain combiner module in section 4 (Structured inference networks). I am really confused about why they are doing three linear transformations in the code starting from line 104 on GitHub. Aren't they just suppose to use RNN to produce the distribution or am I missing something? Insights would be appreciated.


title: Creating Pytorch variables directly on GPU seems to still create variable on CPU first, judging by CPU RAM usage

body:
I need to create variables directly on the GPU because I am very limited in my CPU ram.

I found the method to do this here

https://discuss.pytorch.org/t/how-to-create-a-tensor-on-gpu-as-default/2128/3

Which mentions using

torch.set_default_tensor_type(‘torch.cuda.FloatTensor’)

However, when I tried

torch.set_default_tensor_type('torch.cuda.FloatTensor')
pytorchGPUDirectCreate = torch.FloatTensor(20000000, 128).uniform_(-1, 1).cuda()

It still seemed to take up mostly CPU RAM, before being transferred to GPU ram.

I am using Google Colab. To view RAM usage during the variable creation process, after running the cell, go to Runtime -> Manage Sessions

With and without using torch.set_default_tensor_type('torch.cuda.FloatTensor') , the CPU RAM bumps up to 11.34 GB while GPU ram stays low, and then GPU RAM goes to 9.85 and CPU ram goes back down.

It seems that torch.set_default_tensor_type(‘torch.cuda.FloatTensor’) didn't make a difference

For convenience here's a direct link to a notebook anyone can directly run

https://colab.research.google.com/drive/1LxPMHl8yFAATH0i0PBYRURo5tqj0_An7


title: Concept of mini batch in deep generative model using pyro

body:
I am new to probabilistic programming and ML. I am following a code on deep Markov model given on pyro's website. The link to the github page to that code is:

https://github.com/pyro-ppl/pyro/blob/dev/examples/dmm/dmm.py

I understand most part of the code. The part I don't understand is mini batch idea they are using from line 175.

Question 1:
Could someone explain what are they doing there when they are using mini-batch?

In pyro documentation they say

  mini_batch is a three dimensional tensor, with the first dimension being the batch dimension, the second dimension being the temporal dimension, and the final dimension being the features (88-dimensional in our case)'

Question 2:
What does temporal dimension means here?

Because I want to use this code on my dataset which is a sequential data. I have done one hot encoding of my data such that it's dimension is (10000,500,20) where 10000 is the number of examples/Sequences, 500 is the length of each of these sequences and 20 is the number of features.

Question 3:
How can I use my one hot encoded data as mini batch here?

I'm sorry if it is a really basic question but, insights will be appreciated.

Link to that documentation is:

https://pyro.ai/examples/dmm.html


title: Custom Dataset, Dataloader, Sampler, or something else?

body:
I'm working on a project that requires training a PyTorch framework NN on a very large dataset of images. Some of these images are completely irrelevant to the problem, and but these irrelevant images are not labelled as such. However, there are some metrics I can use to calculate if they are irrelevant (e.g. summing all the pixel values would give me a good sense of which are the relevant images and which are not). What I would ideally like to do is have a Dataloader that can take in a Dataset class, and create batches only with the relevant images. The Dataset class would just know the list of images and their labels, and the Dataloader would interpret whether or not the image it is making a batch with is relevant or not, and would then only make batches with relevant images.

To apply this to an example, lets say I have a dataset of black and white images. The white images are irrelevant, but they are not labelled as such. I want to be able to load batches from a file location, and have these batches only contain the black images. I could filter at some point by summing all the pixels and finding it equals to 0.

What I am wondering is if a custom Dataset, Dataloader, or Sampler would be able to solve this task for me? I already have written a custom Dataset that stores the directory of all the saved images, and a list of all the images in that directory, and can return an image with its label in the getitem function. Is there something more I should add there to filter out certain images? Or should that filter be applied in a custom Dataloader, or Sampler?

Thank you!


title: Pytorch: multiply two high dimensions tensor, (2, 5, 3) * (2, 5) into (2, 5, 3)

body:
I want to multiply two high dimensions tensor, (2, 5, 3) * (2, 5) into (2, 5, 3), which multiply each row vector by a scalar.
E.g.
emb = nn.Embedding(6, 3)

input = torch.tensor([[1, 2, 3, 4, 5,],
                      [2, 3, 1, 4, 5,]])
input_emb = emb(input)

print(input.shape)
> torch.Size([2, 5])

print(input_emb.shape)
> torch.Size([2, 5, 3])

print(input_emb)
> tensor([[[-1.9114, -0.1580,  1.2186],
         [ 0.4627,  0.9119, -1.1691],
         [ 0.6452, -0.6944,  1.9659],
         [-0.5048,  0.6411, -1.3568],
         [-0.2328, -0.9498,  0.7216]],

        [[ 0.4627,  0.9119, -1.1691],
         [ 0.6452, -0.6944,  1.9659],
         [-1.9114, -0.1580,  1.2186],
         [-0.5048,  0.6411, -1.3568],
         [-0.2328, -0.9498,  0.7216]]], grad_fn=<EmbeddingBackward>)

I want to multiply may as follows:
// It is written in this way for convenience, not mathematical true.

// multiply each row vector by a scalar
[[
         [-1.9114, -0.1580,  1.2186] * 1
         [ 0.4627,  0.9119, -1.1691] * 2
         [ 0.6452, -0.6944,  1.9659] * 3
         [-0.5048,  0.6411, -1.3568] * 4
         [-0.2328, -0.9498,  0.7216] * 5
]
[
         [ 0.4627,  0.9119, -1.1691] * 2
         [ 0.6452, -0.6944,  1.9659] * 3
         [-1.9114, -0.1580,  1.2186] * 1
         [-0.5048,  0.6411, -1.3568] * 4
         [-0.2328, -0.9498,  0.7216] * 5
]]

Except for the multi-for-loops ways, how to implement it in a concise way by PyTorch APIs?
Thanks in advances.


plt.imshow(torchvision.utils.make_grid(images[3], nrow=5).permute(1, 2, 0 )

Why does the following error popup while running the above code?

  ---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-78-ce90d0e2159a> in <module>
----> 1 plt.imshow(torchvision.utils.make_grid(images[3], nrow=5).permute(1, 2, 0))

~\Anaconda3\lib\site-packages\torchvision\utils.py in make_grid(tensor, nrow, padding, normalize, range, scale_each, pad_value)
     74     xmaps = min(nrow, nmaps)
     75     ymaps = int(math.ceil(float(nmaps) / xmaps))
---> 76     height, width = int(tensor.size(2) + padding), int(tensor.size(3) + padding)
     77     num_channels = tensor.size(1)
     78     grid = tensor.new_full((num_channels, height * ymaps + padding, width * xmaps + padding), pad_value)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 2)

This happens after running the following code

model= nn.Sequential(nn.Linear (150528, 1000),
                     nn.ReLU(),
                     nn.Linear(1000, 250) ,
                     nn.ReLU(),
                     nn.Linear(250, 32),
                     nn.LogSoftmax(dim=1),
                     )
criterion = nn.CrossEntropyLoss()
images = images.view(images.shape[0], -1)
logits = model (images)
loss = criterion(logits, labels)
print(loss)

And re-running the above code that is,

plt.imshow(torchvision.utils.make_grid(images[3], nrow=5).permute(1, 2, 0 )


title: How to convert string to tensor having torchtext.data.Field created?

body: So I have a torchtext.data.Field with build_vocab done. I have trained my model on my dataset. How to use it to convert raw string into tensor for model to be able to do prediction?


title: How to calculate the "Normalized distance" in a Pose Estimation model?

body:
I am currently trying to reproduce the Convolutional Pose Machines model presented in this paper (https://arxiv.org/pdf/1602.00134.pdf). I have trained the model on the LSP Dataset and want to evaluate it. In the paper the authors are talking about a normalized distance between the predicted keypoint and the groundtruth. Can anybody explain to me what this exactly means?
Currently I calculate the PCKh@0.5 metric as followed

euclidean distance between prediction and groundtruth = d
euclidean distance between head_top and neck keypoints = head_neck
correct prediction if: d smaller than head_neck * 0.5

I calculate the euclidean distances after the resizing back to the original images size (model input size is 368x368 and output is a 46x46 heatmap for each part).
When I evaluate my model I only reach roughly 45 percent correct predictions on my test set (500 imgs) which is much worse than the results reported in the paper so maybe I do something wrong.
Link to my github repo https://github.com/johanngerberding/convolutional_pose_machines
I hope somebody can explain what I'm doing wrong. Thanks!


title: Predictions using Logistic Regression in Pytorch return infinity

body:
I started watching a tutorial on PyTorch and I am learning the concept of logistic regression.
I tried it using some stock data that I had. I have inputs, which contains two parameters trade_quantity and trade_value, and targets which has the corresponding stock price.
inputs = torch.tensor([[182723838.00, 2375432.00],
                       [185968153.00, 2415558.00],
                       [181970093.00, 2369140.00],
                       [221676832.00, 2811589.00],
                       [339785916.00, 4291782.00],
                       [225855390.00, 2821301.00],
                       [151430199.00, 1889032.00],
                       [122645372.00, 1552998.00],
                       [129015052.00, 1617158.00],
                       [121207837.00, 1532166.00],
                       [139554705.00, 1789392.00]])

targets = torch.tensor([[76.90],
                        [76.90],
                        [76.90],
                        [80.70],
                        [78.95],
                        [79.60],
                        [80.05],
                        [78.90],
                        [79.40],
                        [78.95],
                        [77.80]])

I defined the model function, the loss as the mean square error, and tried to run it a few times to get some predictions. Here's the code:
def model(x):
    return x @ w.t() + b

def mse(t1, t2):
    diff = t1 - t2
    return torch.sum(diff * diff) / diff.numel()

preds = model(inputs)
loss = mse(preds, targets)
loss.backward()
with torch.no_grad():
    w -= w.grad * 1e-5
    b -= b.grad * 1e-5
    w.grad.zero_()
    b.grad.zero_()

I am using Jupyter for this and ran the last part of the code a few times, after which the predictions come as:
tensor([[inf],
        [inf],
        [inf],
        [inf],
        [inf],
        [inf],
        [inf],
        [inf],
        [inf],
        [inf],
        [inf]], grad_fn=<AddBackward0>)

If I run it for a few more times the predictions become nan. Can you please tell me why is this happening?


title: understanding conv net layers

body: I've been reading about Conv nets and I've programmed a few models myself. When I see visual diagrams of other models it shows each layer being smaller and deeper than the last ones. Layers have 3 dimensions like 256x256x32. What is this third number? I assume the first two numbers are the number of nodes but I don't know what the depth is.


title: How to use VGG19 transfer learning pretraining

body:
I'm working on a VQA model, and I need some help as I'm new to this.
I want to use transfer learning from the VGG19 network before running the train, so when I start the train, I will have the image features ahead (trying to solve performance issue).
Does it possible to do so? If so, can someone please share an example with pytorch?
below is the relevant code:
class img_CNN(nn.Module):
  def __init__(self, img_size):

        super(img_CNN, self).__init__()
        self.model = models.vgg19(pretrained=True)
        self.in_features = self.model.classifier[-1].in_features
        self.model.classifier = nn.Sequential(*list(self.model.classifier.children())[:-1]) # remove vgg19 last layer
        self.fc = nn.Linear(in_features, img_size)

  def forward(self, image):
    #with torch.no_grad():
    img_feature = self.model(image) # (batch, channel, height, width)
    img_feature = self.fc(img_feature)
    return img_feature

class vqamodel(nn.Module):
  def __init__(self, output_dim,input_dim, emb_dim, hid_dim, n_layers, dropout, answer_len, que_size, img_size,model_vgg,in_features):
    super(vqamodel,self).__init__()
    self.image=img_CNN(img_size)
    self.question=question_lstm(input_dim, emb_dim, hid_dim, n_layers, dropout,output_dim,que_size)
    self.tanh=nn.Tanh()
    self.relu=nn.ReLU()
    self.dropout=nn.Dropout(dropout)
    self.fc1=nn.Linear(que_size,answer_len) #the input to the linear network is equal to the combain vector
    self.softmax=nn.Softmax(dim=1)

  def forward(self, image, question):
    image_emb=self.image(image)
    question_emb=self.question(question)
    combine =question_emb*image_emb
    out_feature=self.fc1(combine)
    out_feature=self.relu(out_feature)

    return (out_feature)

How can I take out the models.vgg19(pretrained=True),run it before the train on the image dataloader and save the image representation in NumPy array?
thank you!


title: java BindException: Address already in use: bind even though port is free

body:
I am testing out torchserve with its default configuration... its inference api listen to port 8080. and I got this error which means the port is in use. my bad, I still have apache process listening to that port. I killed the apache process. But i still get the same error. Tried again by changing port configuration of torchserve to use port that I know for sure is not being used by any process. still got the same error. with TCPView app, I could see that for a second, the configured port has been opened in LISTENING state for the torcheserve process. But soon after it got removed again.
so anyone has anymore idea why is this happening? I am pretty sure this is no longer an issue of port conflict
the stack trace
java.io.IOException: Failed to bind
        at io.grpc.netty.shaded.io.grpc.netty.NettyServer.start(NettyServer.java:264)
        at io.grpc.internal.ServerImpl.start(ServerImpl.java:183)
        at io.grpc.internal.ServerImpl.start(ServerImpl.java:90)
        at org.pytorch.serve.ModelServer.startGRPCServer(ModelServer.java:396)
        at org.pytorch.serve.ModelServer.startGRPCServers(ModelServer.java:377)
        at org.pytorch.serve.ModelServer.startAndWait(ModelServer.java:116)
        at org.pytorch.serve.ModelServer.main(ModelServer.java:95)
Caused by: java.net.BindException: Address already in use: bind
        at java.base/sun.nio.ch.Net.bind0(Native Method)
        at java.base/sun.nio.ch.Net.bind(Net.java:461)
        at java.base/sun.nio.ch.Net.bind(Net.java:453)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
        at io.grpc.netty.shaded.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:134)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:550)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1334)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:506)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:491)
        at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:973)
        at io.grpc.netty.shaded.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:248)
        at io.grpc.netty.shaded.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:356)
        at io.grpc.netty.shaded.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
        at io.grpc.netty.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)
        at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
        at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:834)
Exception in thread "Thread-0" java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926)
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353)
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828)
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818)
        at io.netty.channel.AbstractChannelHandlerContext.safeExecute(AbstractChannelHandlerContext.java:989)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:608)
        at io.netty.channel.AbstractChannelHandlerContext.close(AbstractChannelHandlerContext.java:472)
        at io.netty.channel.DefaultChannelPipeline.close(DefaultChannelPipeline.java:957)
        at io.netty.channel.AbstractChannel.close(AbstractChannel.java:232)
        at org.pytorch.serve.ModelServer.stop(ModelServer.java:473)
        at org.pytorch.serve.ModelServer$1.run(ModelServer.java:91)


title: Load custom knowledge-graph data to pytorch geometric

body:
I am new to PyTorch geometric and want to know how we can load our own knowledge-graph dataset into PyTorch geometric DataLoader. I have my data in the CSV file which looks like for example:
Dataset consists of 1000's of such triples
I went through PyTorch documentation but couldn't understand how this kind of data can be used with Pytorch geometric.
I was using this data earlier with ampligraph to do link prediction and thought of giving it try with GNN (PyTorch geometric).
Any help on this!!


title: Using BERT and Keras's neural network for text classification

body:
I'm trying to run a binary supervised text classification task using BERT, but I'm not sure how to do that.
I have tried to run BERT with the Hugging Face () library, but I have no idea what to do with the output of the process.
After a lot of internet searches I ended up with the following class (according to https://towardsdatascience.com/build-a-bert-sci-kit-transformer-59d60ddd54a5):
class BertTransformer(BaseEstimator, TransformerMixin):
    def __init__(self):
        self.tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
        self.model = BertModel.from_pretrained("bert-base-uncased")
        self.model.eval()
        self.embedding_func = lambda x: x[0][:, 0, :].squeeze()

    def _tokenize(self, text: str):
        # Tokenize the text with the provided tokenizer
        tokenized_text = self.tokenizer.encode_plus(text,
                                                    add_special_tokens=True,
                                                    truncation=True
                                                    )["input_ids"]

        # Create an attention mask telling BERT to use all words
        attention_mask = [1] * len(tokenized_text)

        # bert takes in a batch so we need to unsqueeze the rows
        return (
            torch.tensor(tokenized_text).unsqueeze(0),
            torch.tensor(attention_mask).unsqueeze(0),
        )

    def _tokenize_and_predict(self, text: str) -> torch.tensor:
        tokenized, attention_mask = self._tokenize(text)

        embeddings = self.model(tokenized, attention_mask)
        return self.embedding_func(embeddings)

    def transform(self, text: List[str]):
        if isinstance(text, pd.Series):
            text = text.tolist()

        with torch.no_grad():
            return torch.stack([self._tokenize_and_predict(string) for string in text])

    def fit(self, X, y=None):
        return self

This class suitable for use in Sikict-Learn which is good for me, but I want also to run it with deep learning models using Keras. How can I make this work with Keras's neural networks (such as RNN and CNN)?

From what I understand, the above code takes only the CLS token and not all of the tokens. I don't know if that's alright. Maybe I should use all of them? If so, how can I do that?

Any help would be appreciated.


title: AllenNLP DatasetReader.read returns generator instead of AllennlpDataset

body:
While studying AllenNLP framework (version 2.0.1), I tried to implement the example code from https://guide.allennlp.org/training-and-prediction#1.
While reading the data from a Parquet file I got:
TypeError: unsupported operand type(s) for +: 'generator' and 'generator'

for the next line:
vocab = build_vocab(train_data + dev_data)

I suspect the return value should be AllennlpDataset but maybe I got it mixed up.
What did I do wrong?
Full code:
train_path = <some_path>
test_path = <some_other_path>

class ClassificationJobReader(DatasetReader):
    def __init__(self,
                 lazy: bool = False,
                 tokenizer: Tokenizer = None,
                 token_indexers: Dict[str, TokenIndexer] = None,
                 max_tokens: int = None):
        super().__init__(lazy)
        self.tokenizer = tokenizer or WhitespaceTokenizer()
        self.token_indexers = token_indexers or {'tokens': SingleIdTokenIndexer()}
        self.max_tokens = max_tokens

    def _read(self, file_path: str) -> Iterable[Instance]:
      df = pd.read_parquet(data_path)
      for idx in df.index:
        text = row['title'][idx] + ' ' + row['description'][idx]
        print(f'text : {text}')
        label = row['class_id'][idx]
        print(f'label : {label}')
        tokens = self.tokenizer.tokenize(text)
        if self.max_tokens:
            tokens = tokens[:self.max_tokens]
        text_field = TextField(tokens, self.token_indexers)
        label_field = LabelField(label)
        fields = {'text': text_field, 'label': label_field}
        yield Instance(fields)

def build_dataset_reader() -> DatasetReader:
    return ClassificationJobReader()

def read_data(reader: DatasetReader) -> Tuple[Iterable[Instance], Iterable[Instance]]:
    print("Reading data")
    training_data = reader.read(train_path)
    validation_data = reader.read(test_path)
    return training_data, validation_data

def build_vocab(instances: Iterable[Instance]) -> Vocabulary:
    print("Building the vocabulary")
    return Vocabulary.from_instances(instances)

dataset_reader = build_dataset_reader()
train_data, dev_data = read_data(dataset_reader)
vocab = build_vocab(train_data + dev_data)

Thanks for your help


I want to create a dataset from a tsv file with pytorch.
I was thinking of using
torchtext.data.TabularDataset.splits

but I'm getting an error message.
AttributeError:module 'torchtext.data' has no attribute 'TabularDataset'


title: Is a .pth file a security risk, and how can we sanitise it?

body:
It's well-established that pickled files are  [unsafe][1] to simply load directly. However, the advice on that SE post concludes that basically one should not use a pickled file if they are not sure of its provenance.
What about PyTorch machine-learning models that are stored as .pth files on, say, public repos on Github, which we want to use for inference? For example, if I have a model.pth which I plan to load with torch.load(model.pth), is it necessary to check it's safe to do so? Assuming I have no choice but to use the model, how should one go about checking it?
Given these models are ultimately just weights, could we do something like make a minimal Docker container with PyTorch, load the model inside there, and then resave the weights? Is this necessary, and what sort of checking should we do (i.e. assuming it is safe to load within the container, what sort of code treatment should be applied to sanitise the model for shipping?).
EDIT: response to asking for clarification: say I have a model.pth - (1) do I need to be careful with it, like I would with a .pkl, given a .pth is meant to contain only model weights? Or can I just go ahead and throw it into torch.load(model.pth)? (2) If I can't, what can I do before torch.load() to provide some peace-of-mind?
EDIT 2: An answer should show some focus on ML pretrained models in particular. An example: see the model https://download.pytorch.org/models/resnet34-333f7ec4.pth (warning: 80 MB download from TorchHub - feel free to use another Resnet .pth model, but any cleaning does need to be reasonably fast). Currently I would download this and then load it in PyTorch using load_state_dict (explained in detail [here][2] for example). If I didn't know this was a safe model, how could I try to sanitise it first before loading it in load_state_dict?

  [1]: https://stackoverflow.com/questions/25353753/python-can-i-safely-unpickle-untrusted-data
  [2]: https://www.programmersought.com/article/95324315915/


title: Efficiently way of assembling PyTorch `Tensor`s that require gradients

body:
I need to construct a 4-dimensional PyTorch Tensor where one of the dimensions comes from multiplying a constant sparse matrix with a dense vector. The dense vector, and the resulting 4D Tensor, require gradients to be tracked. Since PyTorch only supports sparse matrices, I can't express the whole thing as a Tensor-Tensor multipllication, and I think I have to do the matrix multiplication part of the construction in a loop. In that case, I'd at least like to preallocate the result 4D Tensor and let the sparse mm fill in one dimension in a loop.

How do I in that case keep track of the resulting 4D Tensor's gradient requirements? Can I manually attach it into the gradient graph once it's been created?

My current approach is extremely inefficient, essentially building up one dimension at a time in a list, can cating.