title: Pytorch CNN loss is not changing,

body:
I making a CNN for a binary classification problem between bees and ants images.
Images are of 500x500 dimension with 3 channels.

Here is my code.

Dataloader:

def load_data(path):
    data = []
    ant = 0
    bee = 0
    for folder in os.listdir(path):
        print(folder)
        curfolder = os.path.join(path, folder)
        for file in os.listdir(curfolder):
            image = plt.imread(curfolder+'/'+file)
            image = cv2.resize(image, (500,500))
            if folder == 'ants':
                ant += 1
                data.append([np.array(image) , np.eye(2)[0]])
            elif folder == 'bees':
                bee += 1
                data.append([np.array(image) , np.eye(2)[1]])

    np.random.shuffle(data)
    np.save('train.npy',data)
    print('ants : ',ant)
    print('bees : ',bee)

training_data = np.load("train.npy",allow_pickle=True)
print(len(training_data))

CNN class

class Net(nn.Module):
    def __init__(self):
        super().__init__() # just run the init of parent class (nn.Module)
        self.conv1 = nn.Conv2d(3, 32, 5) # input is 1 image, 32 output channels, 5x5 kernel / window
        self.conv2 = nn.Conv2d(32, 64, 5) # input is 32, bc the first layer output 32. Then we say the output will be 64 channels, 5x5 kernel / window
        self.conv3 = nn.Conv2d(64, 128, 5)

        x = torch.randn(3,500,500).view(-1,3,500,500)
        self._to_linear = None
        self.convs(x)
        print(self._to_linear)
        self.fc1 = nn.Linear(self._to_linear, 512) #flattening.
        self.fc2 = nn.Linear(512, 2) # 512 in, 2 out bc we're doing 2 classes (dog vs cat).

    def convs(self, x):
        # max pooling over 2x2
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        if self._to_linear is None:
            self._to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.convs(x)
        x = x.view(-1, self._to_linear)  # .view is reshape ... this flattens X before
        x = F.relu(self.fc1(x))
        x = self.fc2(x) # bc this is our output layer. No activation here.
        return F.softmax(x, dim=1)

net = Net()
print(net)

loss and optimizer

import torch.optim as optim

optimizer = optim.Adam(net.parameters(), lr=0.001)
loss_function = nn.MSELoss()

Data Flat

train_X = torch.Tensor([i[0] for i in training_data]).view(-1,3,500,500)
train_X = train_X/255.0
train_y = torch.Tensor([i[1] for i in training_data])

training the model

device = torch.device("cuda:0")
net = Net().to(device)
print(len(train_X))
epochs = 10
BATCH_SIZE = 1
for epoch in range(epochs):
        for i in range(0, len(train_X), BATCH_SIZE): # from 0, to the len of x, stepping BATCH_SIZE at a time. [:50] ..for now just to dev
            #print(f"{i}:{i+BATCH_SIZE}")
            batch_X = train_X[i:i+BATCH_SIZE]
            batch_y = train_y[i:i+BATCH_SIZE]

            batch_X, batch_y = batch_X.to(device), batch_y.to(device)
            net.zero_grad()
            outputs = net(batch_X)
            loss = loss_function(outputs, batch_y)
            loss.backward()
            optimizer.step()     # Does the update

        print(f"Epoch : {epoch}. Loss: {loss}")

The loss does not update for every epoch. I have tried changing the learning rate but the problem still remains.

  Epoch : 0. Loss: 0.23345321416854858

  Epoch : 1. Loss: 0.23345321416854858

  Epoch : 2. Loss: 0.23345321416854858

  Epoch : 3. Loss: 0.23345321416854858

  Epoch : 4. Loss: 0.23345321416854858

  Epoch : 5. Loss: 0.23345321416854858

  Epoch : 6. Loss: 0.23345321416854858

  Epoch : 7. Loss: 0.23345321416854858

  Epoch : 8. Loss: 0.23345321416854858

  Epoch : 9. Loss: 0.23345321416854858

Thank you in advance.


title: KeyError: 'answers' error when using BioASQ dataset using Huggingface Transformers

body:
I am using run_squad.py https://github.com/huggingface/transformers/blob/master/examples/run_squad.py from Huggingface Transformers for fine-tuning on BioASQ Question Answering dataset.

I have converted the tensorflow weights provided by the authors of BioBERT https://github.com/dmis-lab/bioasq-biobert to Pytorch as discussed here https://github.com/huggingface/transformers/issues/312.

Further, I am using the preprocessed data of BioASQ https://github.com/dmis-lab/bioasq-biobert which is converted to the SQuAD form.
However, when I am running the run_squad.py script with the below parameters

 --model_type bert \
  --model_name_or_path /scratch/oe7/uk1594/BioBERT/BioBERT-PyTorch/BioBERTv1.1-SQuADv1.1-Factoid-PyTorch/ \
  --do_train \
  --do_eval \
  --save_steps 1000 \
  --train_file $data/BioASQ-train-factoid-6b.json \
  --predict_file $data/BioASQ-test-factoid-6b-1.json \
  --per_gpu_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2.0 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /scratch/oe7/uk1594/BioBERT/BioBERT-PyTorch/QA_output_squad/BioASQ-factoid-6b/BioASQ-factoid-6b-1-issue-23mar/

I get the below error:

03/23/2020 12:53:12 - INFO - transformers.modeling_utils -   loading weights file /scratch/oe7/uk1594/BioBERT/BioBERT-PyTorch/QA_output_squad/BioASQ-factoid-6b/BioASQ-factoid-6b-1-issue-23mar/pytorch_model.bin
03/23/2020 12:53:15 - INFO - __main__ -   Creating features from dataset file at .

  0%|          | 0/1 [00:00<?, ?it/s]
  0%|          | 0/1 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "run_squad.py", line 856, in <module>
    main()
  File "run_squad.py", line 845, in main
    result = evaluate(args, model, tokenizer, prefix=global_step)
  File "run_squad.py", line 299, in evaluate
    dataset, examples, features = load_and_cache_examples(args, tokenizer, evaluate=True, output_examples=True)
  File "run_squad.py", line 475, in load_and_cache_examples
    examples = processor.get_dev_examples(args.data_dir, filename=args.predict_file)
  File "/scratch/oe7/uk1594/lib/python3.7/site-packages/transformers/data/processors/squad.py", line 522, in get_dev_examples
    return self._create_examples(input_data, "dev")
  File "/scratch/oe7/uk1594/lib/python3.7/site-packages/transformers/data/processors/squad.py", line 549, in _create_examples
    answers = qa["answers"]
KeyError: 'answers'

Really appreciate your help.

Thanks a lot for your guidance.

The evaluaton dataset is looks like this:

{
  "version": "BioASQ6b",
  "data": [
    {
      "title": "BioASQ6b",
      "paragraphs": [
        {
          "context": "emMAW: computing minimal absent words in external memory. Motivation: The biological significance of minimal absent words has been investigated in genomes of organisms from all domains of life. For instance, three minimal absent words of the human genome were found in Ebola virus genomes",
          "qas": [
            {
              "question": "Which algorithm is available for computing minimal absent words using external memory?",
              "id": "5a6a3335b750ff4455000025_000"
            }
          ]
        }
    ]
}
]
}


title: Using pretrained VGG-16 to get a feature vector from an image

body:
I want to get a feature vector out of an image by passing the image through a pre-trained VGG-16. I used the pretrained Resnet50 to get a feature vector and that worked perfectly. But when I use the same method to get a feature vector from the VGG-16 network, I don’t get the 4096-d vector which I assume I should get. I got the code from a variety of sources and it is as follows:

vgg16_model=models.vgg16(pretrained=True)
modules=list(vgg16_model.children())[:-1]
vgg16_model=nn.Sequential(*modules)

data=moveaxis(data,2,0)
img_var=Variable(torch.from_numpy(data).unsqueeze(0)).float()
features_var=vgg16_model(img_var)
features=features_var.data
features=features.data.numpy()
print(features.shape)

The variable “data” is an image numpy array of dimensions (300, 400, 3)
Hence I use the move axis to jumble the axis so that I have 3 channels and not 300.
The output(features.shape) which I get is : (1, 512, 7, 7)
I want a 4096-d vector as the VGG-16 gives before the softmax layer.
I even tried the list(vgg16_model.classifier.children())[:-1] approach but that did not go too well either. There are a lot of discussions about this but none of them worked for me. Let me know where I might be going wrong. Thank you!


title: How do I specify and install the latest version of PyTorch via Conda in AWS Sagemaker?

body:
I'm attempting to use a recent version of PyTorch (1.7.0) in a Conda environment on Sagemaker by specifying the package version in an environment.yml file. However, I'm getting a ResolvePackageNotFound error. Note that I'm just working in a Jupyter notebook with a kernel corresponding to this Conda environment. I'm not using a deep learning image.
Steps to reproduce:
Save the code below as a .yml file, navigate to where this .yml file is stored, and run conda env create environment.yml.
name: test_env
dependencies:
  - numpy
  - pandas
  - pytorch>=1.7.0
  - torchvision
  - scipy
  - ipykernel
  - torchvision

I've tried this on instances of various types (ml.p2.xlarge, ml.p3.2xlarge, and ml.p3.8xlarge) and have gotten the same error each time. I tried it with conda version 4.8.3 and 4.9.2 as well. If I specify pytorch>=1.5.0, I'm able to create the environment successfully.
Does anyone have any idea why I can't create an environment successfully with more recent versions of PyTorch? Based on this documentation, I wonder if Sagemaker has preinstalled certain versions of PyTorch, and something is going awry when I attempt to use a more recent version.


I was trying to implement the Discriminator and Generator model from DCGAN paper on Google colab. After running model (from where the Discriminator, Generator, initialize_weights functions were called) and utils (from where the gradient_penalty, save_checkpoint, load_checkpoint functions were called) when I try to run train I encounter an error.
import torch
import torch.nn as nn
import torch.optim as optim
import torchvision
import torchvision.datasets as datasets
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
#from utils import gradient_penalty, save_checkpoint, load_checkpoint
#from model import Discriminator, Generator, initialize_weights

# Hyperparameters etc.
device = "cuda" if torch.cuda.is_available() else "cpu"
LEARNING_RATE = 1e-4
BATCH_SIZE = 64
IMAGE_SIZE = 256
CHANNELS_IMG = 1
NUM_CLASSES = 10
GEN_EMBEDDING = 100
Z_DIM = 100
NUM_EPOCHS = 100
FEATURES_CRITIC = 16
FEATURES_GEN = 16
CRITIC_ITERATIONS = 5
LAMBDA_GP = 10

transforms = transforms.Compose(
    [
        transforms.Resize(IMAGE_SIZE),
        transforms.ToTensor(),
        transforms.Normalize(
            [0.5 for _ in range(CHANNELS_IMG)], [0.5 for _ in range(CHANNELS_IMG)]),
           #[0.5 for _ in range(CHANNELS_IMG)], [0.99 for _ in range(CHANNELS_IMG)]),
    ]

)
#dataset = datasets.MNIST(root="/content/datasets", transform=transforms, download=True)
# comment mnist above and uncomment below for training on CelebA
dataset = datasets.ImageFolder(root="/content/datasets", transform=transforms)
loader = DataLoader(
    dataset,
    batch_size=BATCH_SIZE,
    shuffle=True,
)

# initialize gen and disc, note: discriminator should be called critic,
# according to WGAN paper (since it no longer outputs between [0, 1])
gen = Generator(Z_DIM, CHANNELS_IMG, FEATURES_GEN, NUM_CLASSES, IMAGE_SIZE, GEN_EMBEDDING).to(device)
critic = Discriminator(CHANNELS_IMG, FEATURES_CRITIC, NUM_CLASSES, IMAGE_SIZE).to(device)
initialize_weights(gen)
initialize_weights(critic)

# initializate optimizer
opt_gen = optim.Adam(gen.parameters(), lr=LEARNING_RATE, betas=(0.0, 0.9))
opt_critic = optim.Adam(critic.parameters(), lr=LEARNING_RATE, betas=(0.0, 0.9))

# for tensorboard plotting
fixed_noise = torch.randn(32, Z_DIM, 1, 1).to(device)
writer_real = SummaryWriter(f"logs/GAN_MNIST/real")
writer_fake = SummaryWriter(f"logs/GAN_MNIST/fake")
step = 0

gen.train()
critic.train()

for epoch in range(NUM_EPOCHS):

    for batch_idx, (real,labels) in enumerate(loader):
        real = real.to(device)
        cur_batch_size = real.shape[0]
        labels = labels.to(device)

        # Train Critic: max E[critic(real)] - E[critic(fake)]
        # equivalent to minimizing the negative of that
        for _ in range(CRITIC_ITERATIONS):
            noise = torch.randn(cur_batch_size, Z_DIM, 1, 1).to(device)
            fake = gen(noise, labels)
            critic_real = critic(real, labels).reshape(-1)
            critic_fake = critic(fake, labels).reshape(-1)
            gp = gradient_penalty(critic, labels, real, fake, device=device)
            loss_critic = (
                -(torch.mean(critic_real) - torch.mean(critic_fake)) + LAMBDA_GP * gp
            )
            critic.zero_grad()
            loss_critic.backward(retain_graph=True)
            opt_critic.step()

        # Train Generator: max E[critic(gen_fake)] <-> min -E[critic(gen_fake)]
        gen_fake = critic(fake, labels).reshape(-1)
        loss_gen = -torch.mean(gen_fake)
        gen.zero_grad()
        loss_gen.backward()
        opt_gen.step()

        # Print losses occasionally and print to tensorboard
        if batch_idx % 100 == 0 and batch_idx > 0:
            print(
                f"Epoch [{epoch}/{NUM_EPOCHS}] Batch {batch_idx}/{len(loader)} \
                  Loss D: {loss_critic:.4f}, loss G: {loss_gen:.4f}"
            )

            with torch.no_grad():
                fake = gen(fixed_noise, labels)
                # take out (up to) 32 examples
                img_grid_real = torchvision.utils.make_grid(real[:32], normalize=True)
                img_grid_fake = torchvision.utils.make_grid(fake[:32], normalize=True)

                writer_real.add_image("Real", img_grid_real, global_step=step)
                writer_fake.add_image("Fake", img_grid_fake, global_step=step)

            step += 1

This is the error I encounter
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-17-abb0d634be43> in <module>()
     70 for epoch in range(NUM_EPOCHS):
     71
---> 72     for batch_idx, (real,labels) in enumerate(loader):
     73         real = real.to(device)
     74         cur_batch_size = real.shape[0]

5 frames
/usr/local/lib/python3.7/dist-packages/torch/utils/data/_utils/collate.py in default_collate(batch)
     53             storage = elem.storage()._new_shared(numel)
     54             out = elem.new(storage)
---> 55         return torch.stack(batch, 0, out=out)
     56     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
     57             and elem_type.__name__ != 'string_':

RuntimeError: stack expects each tensor to be equal size, but got [3, 256, 341] at entry 0 and [3, 256, 386] at entry 1

How can I resolve this? I tried adding transforms.Resize() inside transforms.Compose() at the end but with no results.


title: Pytorch tensor multiplication and addition too long on GPU

body:
I am having strange profiling results on a FISTA algorithm implemented with pytorch, and running on GPU. I am using the line-profiler tool in a notebook, and before I launch the notebook I export CUDA_LAUNCH_BLOCKING=1 to prevent asynchronous launch of cuda (in order to have a faithful profiling).

Here is the minimal code I am using :

x_old = torch.randn(20,64,49,49).cuda()
image = torch.randn(20,1,64,64).cuda()
dico = torch.randn(64,1,16,16).cuda()

def MinimalCode(x_old, image, dico):
    i=1
    z_new=x_old
    while i <= 1000:
        residual = f.conv_transpose2d(z_new, dico) - image
        dx = f.conv2d(residual, dico)
        x_new = f.relu(torch.add(z_new - dx/100, -0.2/100))
        z_new = x_new + torch.mul(x_new - x_old, ((1.25 - 1) / 1.24))
        x_old=x_new
        i+=1

I'm only profiling the following code :

test = MinimalCode(x_old, image, dico)

My profiling results are :

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
 1                                           def MinimalCode(x_old, image, dico):
 2         1          2.0      2.0      0.0      L=100
 3         1          1.0      1.0      0.0      i=1
 4         1          1.0      1.0      0.0      z_new=x_old
 5      1001        777.0      0.8      0.0      while i <= 1000:
 6      1000    4830461.0   4830.5     29.0          residual = f.conv_transpose2d(z_new, dico) - image
 7      1000    2846208.0   2846.2     17.1          dx = f.conv2d(residual, dico)
 8      1000    4998924.0   4998.9     30.0          x_new = f.relu(torch.add(z_new - dx/L, -0.2/100))
 9      1000    3992261.0   3992.3     23.9          z_new = x_new + torch.mul(x_new - x_old, ((1.25 - 1) / 1.24))
10      1000       1610.0      1.6      0.0          x_old=x_new
11      1000        938.0      0.9      0.0          i+=1

In the profiling results, I am very surprised of the following lines :

line #8 : a simple relu applied to simple tensor & operation takes more time than a complex deconvolution (line #6). Mathematically speaking, the complexity of line #8 is way under the one of line #6
line #9 : same here

Why simple tensor addition and scalar multiplication are taking approximating the same time than a complex convolution ? Is there a way to make this code faster ? I'am missing something in the profiling process ?

Thanks in advance for your help !!


title: How to use checkpoint model file in pytorch to test the CIFAR-10 dataset?

body:
model = SqueezeNext()
model = model.to(device)

def load_checkpoint(model, optimizer, losslogger, filename='SqNxt_23_1x_Cifar.ckpt'):
# Note: Input model & optimizer should be pre-defined.  This routine only updates their states.
start_epoch = 0
if os.path.isfile(filename):
    print("=> loading checkpoint '{}'".format(filename))
    checkpoint = torch.load(filename)
    start_epoch = checkpoint['epoch']
    model.load_state_dict(checkpoint['state_dict'])
    optimizer.load_state_dict(checkpoint['optimizer'])
    losslogger = checkpoint['losslogger']
    print("=> loaded checkpoint '{}' (epoch {})"
              .format(filename, checkpoint['epoch']))
else:
    print("=> no checkpoint found at '{}'".format(filename))

return model, optimizer, start_epoch, losslogger

model, optimizer, start_epoch, losslogger = load_checkpoint(model, optimizer, losslogger)

  TypeError: Traceback (most recent call last)
   in ()
       41 test_loader   = torch.utils.data.DataLoader(test_dataset, batch_size=80, num_workers=8, shuffle=False)
       42
  ---> 43 model = SqueezeNext()
       44 model = model.to(device)
       45 def load_checkpoint(model, optimizer, losslogger, filename='SqNxt_23_1x_Cifar.ckpt'): TypeError: init() missing 3
  required positional arguments: 'width_x', 'blocks', and 'num_classes'

I think I am not implementing this in right manner!!


title: nn.Linear should be mismatch, but it works successfully

body:
I'm confused about the in-feature of nn.linear. For out-feature of the model VGG-19's last nn.MaxPool2d, the tensor size is (512, 7, 7). The model below uses pooling function and resizes the tensor to (512, 49), then uses nn.linear(512, 7) directly. Why can't it work successfully without mismatch problem?

source

'''VGG11/13/16/19 in Pytorch.'''
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable

cfg = {
    'VGG11': [64, 'M', 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'VGG13': [64, 64, 'M', 128, 128, 'M', 256, 256, 'M', 512, 512, 'M', 512, 512, 'M'],
    'VGG16': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 'M', 512, 512, 512, 'M', 512, 512, 512, 'M'],
    'VGG19': [64, 64, 'M', 128, 128, 'M', 256, 256, 256, 256, 'M', 512, 512, 512, 512, 'M', 512, 512, 512, 512, 'M'],
}

class VGG(nn.Module):
    def __init__(self, vgg_name):
        super(VGG, self).__init__()
        self.features = self._make_layers(cfg[vgg_name])
        self.classifier = nn.Linear(512, 7)

    def forward(self, x):
        out = self.features(x)
        out = out.view(out.size(0), -1)
        out = F.dropout(out, p=0.5, training=self.training)
        out = self.classifier(out)
        return out

    def _make_layers(self, cfg):
        layers = []
        in_channels = 3
        for x in cfg:
            if x == 'M':
                layers += [nn.MaxPool2d(kernel_size=2, stride=2)]
            else:
                layers += [nn.Conv2d(in_channels, x, kernel_size=3, padding=1),
                           nn.BatchNorm2d(x),
                           nn.ReLU(inplace=True)]
                in_channels = x
        layers += [nn.AvgPool2d(kernel_size=1, stride=1)]
        return nn.Sequential(*layers)


title: How to keep loaded data in RAM when using torch.utils.data.DataLoder?

body:
I'm using torch.utils.data.Dataloader to load my data. I stored my data in a global python dict variable when dataloader loaded it through a __getitem__ function in a torch.utils.data.Dataset object. But it released all the memory after every training epoch was done. Then it loaded all data from the hard disk again.

# in data.py file
from torch.utils.data import Dataset, DataLoader

PRELOAD_DATA = dict()

class Data(Dataset):
    def __init__(self, some_path):
        self.file_list = some_path  # a list contains all the file path

    def __len__(self):
        return len(self.file_list)

    def __getitem__(self, index):
        if index in PRELOAD_DATA:
            this_blob = PRELOAD_DATA[index]
        else:
            this_blob = load_data_from_disk(self.file_list[index])
            PRELOAD_DATA[index] = this_blob  # I try to store data in this line
        return this_blob

def multithread_dataloader(some_path):
    this_data = Data(some_path)
    this_dataloader = DataLoader(this_data, batch_size=1, shuffle=True, num_workers=8, drop_last=False)
    return this_dataloader

# in main.py file
from data import multithread_dataloader

file_path_list = ...  # a list contains all the file path
train_data = multithread_dataloader(file_path_list)
for epoch in range(100):
    for blob in train_data:
        # training


title: SSH into Docker? or docker on SSH? and I need command

body:
I'm new to DL, and docker and even not familiar with Linux and internet things (SSH and port.. DNS things.. part of them are only existing in my mind). Thus, I'd be so much happy with "specific explanation + command" (or reference sites).

My basic questions are:

what is superior concept between Docker and SSH? (running SSH on the Docker ? or running Docker on SSH? or both are possible?)
Which specific command should I use if I want to use SSH+Docker+Pytorch+Jupyternotebook+visdom ?

2-1) I run SSH first (which is my lab's server, thus I am not usually root user, so if I want to run python file here, I face with permission denied often), let say SSH address is 123.456.789.999

2-2) use docker after running ssh (however, what I found from many posts are about running docker FIRST and then access to SSH.. how's it different?)

2-2-1) so for this, I now I have to pull a image which includes pytorch, jupyternotebook. I've done it

2-2-2) I need to RUN DOCKER using image with proper COMMAND LINE. What makes me confused is here.

$docker run -it --[name] -p 8888:8888 [docker_image_with_pytorch]

this is what I found. I assume to use jupyter notebook (let's say if I want to use 4444 instead of 8888, and visdom as 5555 instead of 8097) then I need to map port from host to docker for two times and is it right?

$docker run -it --[name] -p 4444:8888 -p 5555:8097 [docker_image_with_pytorch]

Finally I need to link SSH(let's say SSH port num:22 as general, ip: 123.456.789.999, id=heyjude) for SSH, I also found command below.

$ ssh -L <host port>:localhost:<remote port> user@remote

But is it general to use command after running docker instead of adding option when I first run docker? also if I assume to use that command for setting SSH, i'm confused which things I need to input  ;(
(host port=22 ? for SSH? localhost is just literal expression? remote port is arbitrary thing?) Below is my assumption..

$ ssh -L <22>:localhost:<12345> heyjude@123.456.789.999

I know it's so messy and you may catch how my thinking is twisted..
it would be very helpful for me to explain from the scratch..

Thank you.


title: How do I augment data after spliting traininng datset into train and validation set for CIFAR10 using PyTorch?

body:
When classifying the CIFAR10 in PyTorch, there are normally 50,000 training samples and 10,000 testing samples. However, if I need to create a validation set, I can do it by splitting the training set into 40000 train samples and 10000 validation samples. I used the following codes

train_transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])
test_transform = transforms.Compose([transforms.ToTensor(),transforms.Normalize((0.5,0.5,0.5),(0.5,0.5,0.5))])

cifar_train_L = CIFAR10('./data',download=True, train= True, transform = train_transform)
cifar_test = CIFAR10('./data',download=True, train = False, transform= test_transform)

train_size = int(0.8*len(cifar_training))
val_size = len(cifar_training) - train_size
cifar_train, cifar_val = torch.utils.data.random_split(cifar_train_L,[train_size,val_size])

train_dataloader = torch.utils.data.DataLoader(cifar_train, batch_size= BATCH_SIZE, shuffle= True, num_workers=2)
test_dataloader = torch.utils.data.DataLoader(cifar_test,batch_size= BATCH_SIZE, shuffle= True, num_workers= 2)
val_dataloader = torch.utils.data.DataLoader(cifar_val,batch_size= BATCH_SIZE, shuffle= True, num_workers= 2)

Normally, when augmenting data in PyTorch, different augmenting processes are used under the
transforms.Compose function (i.e., transforms.RandomHorizontalFlip()). However, if I use these augmentation processes before splitting the training set and validation set, the augmented data will also be included in the validation set. Is there any way, I can fix this problem?

  In short, I want to manually split the
  training dataset into train and validation set as well as I want to
  use the data augmentation technique into the new training set.


title: Torch/numpy: groupby pandas alternative

body:
Is there some efficient way to rewrite the following code to avoid installing & importing pandas and using torch/numpy instead? I am used to work with pandas, so I wrote it like this, but I am trying to learn numpy and torch, so I am looking for alternative solutions that do not use pandas.

bins = torch.LongTensor(3072).random_(0, 35)
weights = torch.rand((3072))
df = pd.DataFrame({'weights': weights.numpy(), 'bins': bins.numpy()})
bins_sum = df.groupby('bins').weights.sum().values

So, basically: how, without using pandas, get a sum of weights grouped by bins?


title: Function tensor.cpu() takes a lot of time with pytorch xla multicore

body:
I am using pytorch xla with TPU on google colab. As expected computations are indeed faster, however when I go back to cpu to save my tensors, the "action" of going back to the cpu takes a lot of time for big tensors. Here is the function I run in parallel:

def _mp_fn(index, batch_size, model, typee, directory_join):
    """
    batch_size: Size of batch
    model: BERT model from hugging face
    typee: Type of drug to get embeddings from
    directory_join: save directory
    """
    device = xm.xla_device()
    xm.master_print(f"Here")
    if not xm.is_master_ordinal():
        xm.rendezvous("download_only_once")

    loader = sherBERTEmbedding(f_df, typee)

    if xm.is_master_ordinal():
        xm.rendezvous("download_only_once")
    xm.master_print(f"There")

    sampler = DistributedSampler(loader, num_replicas=xm.xrt_world_size(), rank=xm.get_ordinal(), shuffle=False)
    dataloader = DataLoader(
        loader,
        batch_size=batch_size,
        sampler=sampler,
        # drop_last=True,
    )

    model = model.to(device)
    model.eval()

    length = len(dataloader)

    para_loader = pl.ParallelLoader(dataloader, [device])
    num_b = len(dataloader)
    tensor_embeddings = torch.zeros((batch_size*num_b, 153, 768), dtype=torch.float).to(device)
    tensor_ids = torch.zeros((batch_size*num_b, 153), dtype=torch.long).to(device)
    tensor_index = torch.zeros(batch_size*num_b, dtype=torch.long).to(device)

    toremove = 0
    prev_ind = 0

    with torch.no_grad():
        for t, (x, x_pad_mask, index) in tqdm(
            enumerate(para_loader.per_device_loader(device)), total=length
        ):
            x = x.to(device=device, dtype=torch.long)
            x_pad_mask = x_pad_mask.to(device=device, dtype=torch.long)
            result = get_embeddings_best_token(x, x_pad_mask, model, index_best_tokens)
            ids = get_ids_best_tokens(x, index_best_tokens)

            tensor_embeddings[prev_ind: prev_ind + result.size(0), :, :] = result
            tensor_ids[prev_ind: prev_ind + ids.size(0), :] = ids
            tensor_index[prev_ind: prev_ind +index.size(0)] = index.squeeze(1)
            prev_ind += result.size(0)

            if result.size(0) != batch_size:
                toremove = (batch_size - result.size(0))

        if toremove == 0:
            toremove = tensor_embeddings.size(0)+1

        #saving
        directory_embeddings = os.path.join(os.path.join(directory_join, typee), "embeddings")
        directory_ids = os.path.join(os.path.join(directory_join, typee), "ids")
        directory_index = os.path.join(os.path.join(directory_join, typee), "index")

        tosave = tensor_index[:-toremove]
        ltosave = tosave.cpu()
        torch.save(ltosave, os.path.join(directory_index, str(xm.get_ordinal())+".p"))
        print("Saving 1")

        tosave = tensor_ids[:-toremove, :].squeeze()
        ltosave = tosave.cpu()
        torch.save(ltosave, os.path.join(directory_ids, str(xm.get_ordinal())+".p"))
        print("Saving 2")

        tosave = tensor_embeddings[:-toremove, :, : ].view(-1,768).detach()
        ltosave = tosave.cpu()
        print("Saving 3")
        torch.save(ltosave, os.path.join(directory_embeddings, str(xm.get_ordinal())+".p"))

def get_embeddings_best_token(root, mask, model, index_best_tokens):
    res = model((root, mask))
    return res[:,index_best_tokens[0]:index_best_tokens[-1]+1, :]

def get_ids_best_tokens(x, index_best_tokens):
    return x[:,index_best_tokens[0]:index_best_tokens[-1]+1]

And I run in parallel with this:

get_embeddings(8, model_embed, "myType", JOIN_DIR)

I know that going back to cpu between computations on TPU is a bottleneck, however here I am just trying to save the tensors after the "computations". I know that a special function exists to save tensors: torch_xla.core.xla_model.save(data, file_or_path, master_only=True, global_master=False). However, they explain that basically, it converts tensors from xla tensors to cpu tensors and then saves them. When I use this function, it takes around the same amount of time.

The saving takes around 440 seconds while the previous computation (meaning computing model's result) takes 60 seconds in comparison.


title: CUDA Illegal Memory Access on PyTorch 1.3

body:
@staticmethod
    def backward(ctx, grad_output):
         grad_label = grad_output.clone()
        num_ft = grad_output.shape[0]
        # grad_label.data.resize_(num_ft, 32, 41)
        lin_indices_3d, lin_indices_2d = ctx.saved_variables
        num_ind = lin_indices_3d.data[0]
        grad_label.data.view(num_ft, -1).index_copy_(1, lin_indices_2d.data[1:1 + num_ind],
                                                     torch.index_select(grad_output.data.contiguous().view(num_ft, -1),
                                                                        1, lin_indices_3d.data[1:1 + num_ind]))
        # raw_input('sdflkj')
        return grad_label, None, None, None

This is the code snippet I am trying to run on PyTorch. However, I strangely keep getting the error of Illegal Memory Access. When I tried to use a Debugger and try and find the culprit, I would see

As such I am not certain what is wrong here. The same code was running on PyTorch 0.4 and now I am trying to run it on PyTorch 1.3 and it does not work. The same error remains on versions 1.4 and 1.5 which are the latest versions for the framework. Any help shall be highly appreciated.


title: Does cv2.rectangle() have an argument named “rec”?

body:
I was trying to draw some rectangles within an image but the system gave me this error:
Here is my code:
cv2.rectangle(img=sample,
                      pt1=(box[0], box[1]),
                      pt2=(box[2], box[3]),
                      color=(220, 0, 0), thickness=2)

Any advice will be appreciated!
PS: I tried this piece of codes on kaggle notebok and it ran successfully but it crashed when I want to apply streamlit to deploy a web app (using my local machine). Not sure if it makes a difference, just FYI.


title: Split and extract values from a PyTorch tensor according to given dimensions

body: I have a tensor Aof sizetorch.Size([32, 32, 3, 3]) and I want to split it and extract a tensor B of size torch.Size([16, 16, 3, 3]) from it. The tensor can be 1d or 4d and split has to be according to the given new tensor dimensions. I have been able to generate the target dimensions but I'm unable to split and extract the values from the source tensor. I ave tried torch.narrow but it takes only 3 arguments and I need 4 in many cases. torch.split takes dim as an int due to which tensor is split along one dimension only. But I want to split it along multiple dimensions.


title: pytorch Gradient with respect to 3D input

body:
I want to construct sobolev network for 3D input regression
In TensorFlow, the gradients of neural network model can be computed using tf.gradient like:
dfdx,dfdy,dfdz = tf.gradients(pred,[x,y,z])

Let M be a torch neural network with 5 layers.
If X is a set of (x,y,z)  (3dim data)
and M.forward(X)  is a 1 dim output
How can I compute like gradient of M.forward(X) with respect to X?  Something like:
tf.gradient(M.forward(X),X)


I'm trying to train my model (custom model) in pytorch and I'm getting this error:
RuntimeError: Given groups=1, weight of size [32, 32, 3, 3], expected input[13, 4, 64, 768] to have 32 channels, but got 4 channels instead
this is the code for my custom model:
class BERT_Arch(nn.Module):

    def __init__(self, bert):
        super(BERT_Arch, self).__init__()
        self.bert = BertModel.from_pretrained('bert-base-uncased')
        self.conv = nn.Conv2d(32, 32, kernel_size = 3)
        self.relu = nn.ReLU()
        self.pool = nn.MaxPool2d(2, 2)
        # this number is what i got when i printed the shape of x after max pooling #
        # 3 is the number of classes (labels) #
        self.fc = nn.Linear(13*31*383, 3)
        self.softmax = nn.LogSoftmax()

    def forward(self, sent_id, mask):
        bert_output = self.bert(sent_id, attention_mask=mask, output_hidden_states=True)
        x = bert_output[2]
        x = torch.stack([x[i] for i in range(len(x))])
        x = self.conv(x)
        x = self.relu(x)
        x = self.pool(x)
        x = x.view(-1, 13*31*383)
        x = self.fc(x)
        return self.softmax(x)

now to be honest, I don't know exactly how to use the conv2d because I don't understand the parameters of the function, and I tried reading everywhere and nothing ..
I don't know how to adjust my date shapes and sizes to make it as it should be for conv2d.
my batch size is 32
and my is 13 matrixes of the size (12, 768), that's why I used torch.stack([x[i] for I in range(len(x))]).
you can ignore the numbers(parameters) that I already set in my code, I was just trying to make it work.
Can someone help me to set the parameters ? I'm really confused about this.


title: How would one use Pytorch's Autograd to backprop multiple layers (matracies)?

body:
Just in order to learn I'm trying to modify this three layer neural network from scratch to still use everything from scratch EXCEPT I want to use the Pytorch Autograd function to calculate the deltas and backprop the weights. Specifically I just want to take the mean squared error of the prediction minus y stored as 'loss' and do a 'loss.backwards()' on it.
I made a successful linear regression model with Autograd but it's not working here

I've tried putting the layers and the weights inside and outside the for loop. I've done tinkering with trying to get everything to 'require_grad'.
As is, this is my current error with Python3.6

Traceback (most recent call last):
  File "nn_scratch_trash.py", line 96, in <module>
    layer_2.data -= layer_2.grad.data
AttributeError: 'NoneType' object has no attribute 'data'

The code...

import numpy as np
import random
import torch

np.random.seed(111)

X = np.random.randn(6,4)
y = np.random.randn(6,1)

def zeround(x):
    x = 1 / ( 1 + (np.exp(-x)))
    x = np.around(x)
    return x

# Create Random X and y arrays of random 1s and 0s with the zeround function I made.
y = zeround(y)
X = zeround(X)

X, y = V(X), V(y)

# randomly initialize our weights
weight_0 = V(np.random.randn(4,6), requires_grad=True) - 1
weight_1 = V(np.random.randn(6,7), requires_grad=True) - 1
weight_2 = V(np.random.randn(7,1), requires_grad=True) - 1

# Feed forward through layers 0, 1, and 2
layer_1 = V(((X @ weight_0)), requires_grad=True)
layer_2 = V(((layer_1 @ weight_1)), requires_grad=True)
layer_3 = V(((layer_2 @ weight_2)), requires_grad=True)

print((layer_1))
print((layer_2))
print((layer_3))
print(y)

for j in range(10000):
    # torch backprop
    loss = ((layer_3 - y))
    loss.backward()

    print(weight_2.requires_grad)

    # UPDATE WEIGHTS
    weight_0.data -= weight_0.grad.data
    weight_1.data -= weight_1.grad.data
    weight_2.data -= weight_2.grad.data

    weight_0.grad.data.zero_()
    weight_1.grad.data.zero_()
    weight_2.zero_()

    if j == 9999:
        print (loss.item())


title: Pytorch:DataParallel can't reduce model inference time with same total batch size iamges

body:
When I use torch.nn.DataParallel() to realize data parallel computation, I find the parallel model inference time for the same total amount batch size don't significantly decrease compared with serial model.

As shown below.

model = SNModel()
criterion = nn.CrossEntropyLoss()
if parallel_enable:
    model = nn.DataParallel(model, device_ids=gpu_ids) # gpu_ids = [0,1,2,3] total 4 gpus
model.to(args.device)
criterion.to(args.device)

calculate 100 iteration model inference time:

torch.cuda.synchronize()
st = time.time()
outputs, loss = model(images, path, labels, criterion, gpu_nums)
torch.cuda.synchronize()
total_tm += time.time() - st

With single gpu:

100 iter model time  23s (around)

With 4 gpus:

100 iter model time  103s (around)

There is only 103 - 23*4  = 11s reduced.

Is there any wrong? Hope you can help me! I have been troubled by this for a long time. I know that there is something wrong that I didn't find out. This phenomenon has appeared in other models, so that I have always abandoned the use of parallelism.


I'm getting the following error when calling .backward():

Encounter the RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Here's the code:

for i, j, k in zip(X, Y, Z):
    A[:, i, j] = A[:, i, j] + k

I've tried .clone(), torch.add(), and so on.

Please help!


title: Cannot convert Pandas Dataframe columns to float

body:
I am using Pandas to read a CSV file containing several columns that must be converted to floats:

    df = pd.read_csv(r'dataset.csv',  low_memory=False,  sep = ',')
    df.head(2)

    Coal Flow 01    Air Flow 01 Outlet Temp 01  Inlet Temp 01   Bowl DP 01  Current 01  Vibration 01
    0   51.454407   101.432340  64.917089   234.2488932 2.470623    96.727352   1.874374
    1   51.625368   100.953089  64.726890   233.2340394 2.495698    96.309512   1.996391

Next I specify the columns that need to be converted to floats in a variable called features:

    features = ['Coal Flow 01', 'Air Flow 01', 'Outlet Temp 01', 'Inlet Temp 01',
           'Bowl DP 01', 'Current 01', 'Vibration 01']

Then I needed to convert the the value of the columns to float, but I got an error.

features = np.stack([df[col].values for col in features], 1)
features = torch.tensor(features, dtype=torch.float)
features[:5]

and the error that Pandas is showing me is:

  KeyError: "None of [Index([  51.45440668,   101.4323397,   64.91708906, '234.2488932',\n         2.470623484,   96.72735193,    1.87437372],\n      dtype='object')] are in the [columns]"


title: kth-value per row in pytorch?

body:
Given

import torch
A = torch.rand(9).view((3,3)) # tensor([[0.7455, 0.7736, 0.1772],\n[0.6646, 0.4191, 0.6602],\n[0.0818, 0.8079, 0.6424]])
k = torch.tensor([0,1,0])
A.kthvalue_vectoriezed(k) -> [0.1772,0.6602,0.0818]

Meaning I would like to operate on each column with a different k.

Not kthvalue nor topk offers such API.
Is there a vectorized way around that?
Remark - kth value is not the value in the kth index, but the kth smallest element. Pytorch docs

  torch.kthvalue(input, k, dim=None, keepdim=False, out=None) -> (Tensor, LongTensor)
  Returns a namedtuple (values, indices) where values is the k th smallest element of each row of the input tensor in the given dimension dim. And indices is the index location of each element found.


title: How to make pip run a specific command from requirements file?

body:
I would like to add lines for torch and torchvision on my requirements.txt file, to allow for easy clone, and I will be moving from computer to computer and to cloud in the near future.

I want an easy pip install -r requirements.txt and be done with it for my project.

> pip freeze > requirements.txt

gives something like

...
torch==1.5.0
torchvision==0.6.0
...

However, pip install -r requirements.txt (which is in fact pip install torch) doesn't work, and instead, as the official torch site, clearly says the command should be:

pip install torch===1.5.0 torchvision===0.6.0 -f https://download.pytorch.org/whl/torch_stable.html

How do I make the requirements file reflect this?

My desktop is Windows 10.

Bonus question:

My cloud is Linux based.
How do I make the requirements file fit both desktop and cloud?


title: How to train two pytorch networks with different inputs together?

body:
I'm totally new to pytorch, so it might be a very basic question. I have two networks that should be trained together.

First one takes data as input and returns its embedding as output.
Second one takes pairs of embedded datapoints and returns their 'similarity' as output.
Partial loss is then computed for every datapoint, and then all the losses are combined.
This final loss should be backpropagated through both networks.

How should the code for that look like? I'm thinking something like this:

def train_models(inputs, targets):
    network1.train()
    network2.train()
    embeddings = network1(inputs)
    paired_embeddings = pair_embeddings(embeddings)
    similarities = network2(similarities)

    """
        I don't know how the loss should be calculated here.
        I have a loss formula for every embedded datapoint,
        but not for every similarity.
        But if I only calculate loss for every embedding (using similarites),
        won't backpropagate() only modify network1,
        since embeddings are network1's outputs
        and have not been modified in network2?
    """

    optimizer1.step()
    optimizer2.step()
    scheduler1.step()
    scheduler2.step()
    network1.eval()
    network2.eval()

I hope this specific enough. I'll gladly share more details if necessary. I'm just so inexperienced with pytorch and deep learning in general, that I'm not even sure how to ask this question.


title: Is it possible to use non-pytoch augmentation in transform.compose

body:
I am working on a data classification problem that takes images as an input in Pytorch. I would like the use the imgaug library, but unfortunatly I keep on getting errors. Here is my code.
#import necessary libraries
from torch import nn
from torchvision import models
import imgaug as ia
import imgaug.augmenters as iaa
from torchvision import datasets
from torch.utils.data.dataloader import DataLoader
from torchvision import transforms
from torch import optim
import numpy as np
from PIL import Image
import glob
from matplotlib import image

#preprocess images
#create data transformers
seq = iaa.Sequential([iaa.Sometimes(0.5,iaa.GaussianBlur(sigma=(0,3.0))),
                      iaa.Sometimes(0.5,iaa.LinearContrast((0.75,1.5))),
                      iaa.AdditiveGaussianNoise(loc=0,scale=(0.0,0.05*255),per_channel=0.5),
                      iaa.Sometimes(0.5,iaa.Affine(
        scale={"x": (0.8, 1.2), "y": (0.8, 1.2)},
        translate_percent={"x": (-0.2, 0.2), "y": (-0.2, 0.2)},
        rotate=(-25, 25),
        shear=(-8, 8)))],random_order=True)

train_transformation = transforms.Compose([transforms.RandomResizedCrop(300),
                                           seq,
                                           transforms.ToTensor()])

train_data = datasets.ImageFolder(root = 'train')
train_loader = DataLoader(train_data,shuffle = True,batch_size = 32,num_workers = 0)
train_iter = iter(train_loader)
train_iter.next()

Jupyter Server: local
Python 3.8.4 64-bit: Idle
CNN Cancer Detector
Melanoma
Intro
Skin cancer is the most common form of cancer, with 1 in 5 Americans developping it by the time they reach 70 years old. Over 2 people die of skin cancer in the US every hour.[1] Early detection is key in saving peoples lives with skin cancer, with the early detection 5 year survival rate being 99%[1]. Dermatologist have to look at patients one by one, and must assess by eye whether or not a blemish is malignant or benign. Dermatologist's have around a 66% accuracy rate in assessing 752 different skin diseases, while CNN's, such as the one detailed in *Dermatologist-level classification of skin cancer with deep neural networks* published in Nature have achieved greater accuracy levels then dermatologist's, around 72.1%[2].
By converting cancer detection to easily deployable software, you could allow people to get accurate cancer testing at home, saving resources and time. By making cancer detection more accesible, people would be more likely to get tested, saving lives in the process. Below I will detail my process and results from a melanoma (the most deadly form of skin cancer) detector model using CNN's.

[2]

from PIL import Image
import glob
from matplotlib import image

[3]

#preprocess images
#create data transformers
seq = iaa.Sequential([iaa.Sometimes(0.5,iaa.GaussianBlur(sigma=(0,3.0))),
                      iaa.Sometimes(0.5,iaa.LinearContrast((0.75,1.5))),
                      iaa.AdditiveGaussianNoise(loc=0,scale=(0.0,0.05*255),per_channel=0.5),
                      iaa.Sometimes(0.5,iaa.Affine(
        scale={"x": (0.8, 1.2), "y": (0.8, 1.2)},
        translate_percent={"x": (-0.2, 0.2), "y": (-0.2, 0.2)},
        rotate=(-25, 25),
        shear=(-8, 8)))],random_order=True)
…train_iter = iter(train_loader)
train_iter.next()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in
     20 train_loader = DataLoader(train_data,shuffle = True,batch_size = 32,num_workers = 0)
     21 train_iter = iter(train_loader)
---> 22 train_iter.next()

D:\Python\lib\site-packages\torch\utils\data\dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

D:\Python\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

D:\Python\lib\site-packages\torch\utils\data\_utils\fetch.py in fetch(self, possibly_batched_index)
     45         else:
     46             data = self.dataset[possibly_batched_index]
---> 47         return self.collate_fn(data)

D:\Python\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     77     elif isinstance(elem, container_abcs.Sequence):
     78         transposed = zip(*batch)
---> 79         return [default_collate(samples) for samples in transposed]
     80
     81     raise TypeError(default_collate_err_msg_format.format(elem_type))

D:\Python\lib\site-packages\torch\utils\data\_utils\collate.py in (.0)
     77     elif isinstance(elem, container_abcs.Sequence):
     78         transposed = zip(*batch)
---> 79         return [default_collate(samples) for samples in transposed]
     80
     81     raise TypeError(default_collate_err_msg_format.format(elem_type))

D:\Python\lib\site-packages\torch\utils\data\_utils\collate.py in default_collate(batch)
     79         return [default_collate(samples) for samples in transposed]
     80
---> 81     raise TypeError(default_collate_err_msg_format.format(elem_type))

TypeError: default_collate: batch must contain tensors, numpy arrays, numbers, dicts or lists; found

I am aware that the input to the imgaug transformer must be a numpy array, but I am not sure how to incorporate that into my transform.compose (if I can at all that is.). When the imgaug seq is not in the transform.compose it works properly.
Thank you for the help!


title: Pytorch counting tensor

body:
I am trying to create a Python function that, given inputs dim and m, generates the tensor of size [m ** dim, dim] of the form
[[1,...,1,1],
 [1,...,1,2],
 ...
 [1,...,1,m],
 [1,...,2,1],
 [1,...,2,2],
 ...
 [m,...,m,m]]

What is the best way of doing this in Pytorch?
Thanks


title: How can I get argmaxed torch tensor excluding certain index?

body:
I wonder if I can get torch.argmax of my input excluding certain index.
For example,
target = torch.tensor([1,2])
input = torch.tensor([[0.1,0.5,0.2,0.2], [0.1,0.5,0.1,0.3]])

I want to get the maximum value in input excluding the index on the target, so that the result would be
output = torch.tensor([[0.2],[0.5]])


title: Segmentation Fault when exporting to onnx a quantized Pytorch model

body:
I am trying to export a model to the onnx format. The architecture is complicated so I won't share it here, but basically, I have the network weights in a .pth file. I'm able to load them, create the network and perform inference with it.
It's  important to note that I have adapted the code to be able to quantize the network. I have added quantize and dequantize operators as well as some torch.nn.quantized.FloatFunctional() operators.
However, whenever I try to export it with
torch.onnx.export(torch_model,               # model being run
                  input_example,             # model input
                  model_name,                # where to save the model
                  export_params=True,        # store the trained parameter
                  opset_version=11,          # the ONNX version to export
                  # the model to
                  do_constant_folding=True,  # whether to execute constant
                  # folding for optimization
                 )

I get Segmentation fault (core dumped)
I am working on Ubuntu 20, with the following packages installed :
torch==1.6.0
torchvision==0.7.0
onnx==1.7.0
onnxruntime==1.4.0

Note that the according to some prints I have left in the code, the inference part of the exporting completes. The segmentation fault happens afterward.
Does anyone see any reason why this may happen ?
[Edit] : I can export my network when it is not adapted for quantized operations. Therefore, the problem is not a broken installation but more a problem of some quantized operators for onnx saving.


title: Using a subset of classes in ImageNet

body:
I'm aware that subsets of ImageNet exist, however they don't fulfill my requirement. I want 50 classes at their native ImageNet resolutions.
To this end, I used torch.utils.data.dataset.Subset to select specific classes from ImageNet. However, it turns out, class labels/indices must be greater than 0 and less than num_classes.
Since ImageNet contains 1000 classes, the idx of my selected classes quickly goes over 50. How can I reassign the class indices and do so in a way that allows for evaluation later down the road as well?
Is there a way more elegant way to select a subset?


title: Can't interpolate frames with pytorch

body:
So I was trying to test an algorithm I found in a github page that interpolates frames. I was testing the algorithm on the datasets middlebury provided, the link: http://vision.middlebury.edu/flow/data/ .(I use the coloured ones with only two frames. The error also occured in the Yosemite image file) The algorithm worked fine on every image except the last one. It raises the error:

Traceback (most recent call last):
File "run.py", line 257, in
tensorInputFirst = torch.FloatTensor(numpy.rollaxis(numpy.asarray(PIL.Image.open(arguments_strFirst))[:,:,::-1], 2, 0).astype(numpy.float32) / 255.0)
IndexError: too many indices for array

I know what the error means, an index thats not in the list, but I don't know its connection with frame interpolation and why does it happen. Can somebody help.

The code I found on github:

#!/usr/bin/env python2.7

import sys
import getopt
import math
import numpy
import torch
import torch.utils.serialization
import PIL
import PIL.Image

from moviepy.video.io.ffmpeg_reader import FFMPEG_VideoReader
from moviepy.video.io.ffmpeg_writer import FFMPEG_VideoWriter

from SeparableConvolution import SeparableConvolution # the custom SeparableConvolution layer

torch.cuda.device(1)

torch.backends.cudnn.enabled = True

##########################################################

arguments_strModel = 'lf'
arguments_strFirst = './images/first.png'
arguments_strSecond = './images/second.png'
arguments_strOut = './result.png'
arguments_strVideo = ''
arguments_strVideoOut = ''

for strOption, strArgument in getopt.getopt(sys.argv[1:], '', [ strParameter[2:] + '=' for strParameter in sys.argv[1::2] ])[0]:
    if strOption == '--model':
        arguments_strModel = strArgument # which model to use, l1 or lf, please see our paper for more details

    elif strOption == '--first':
        arguments_strFirst = strArgument # path to the first frame

    elif strOption == '--second':
        arguments_strSecond = strArgument # path to the second frame

    elif strOption == '--out':
        arguments_strOut = strArgument # path to where the output should be stored

    elif strOption == '--video':
        arguments_strVideo = strArgument # path to the video

    elif strOption == '--video-out':
        arguments_strVideoOut = strArgument # path to the video

##########################################################

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()

        def Basic(intInput, intOutput):
            return torch.nn.Sequential(
                torch.nn.Conv2d(in_channels=intInput, out_channels=intOutput, kernel_size=3, stride=1, padding=1),
                torch.nn.ReLU(inplace=False),
                torch.nn.Conv2d(in_channels=intOutput, out_channels=intOutput, kernel_size=3, stride=1, padding=1),
                torch.nn.ReLU(inplace=False),
                torch.nn.Conv2d(in_channels=intOutput, out_channels=intOutput, kernel_size=3, stride=1, padding=1),
                torch.nn.ReLU(inplace=False)
            )

        def Subnet():
            return torch.nn.Sequential(
                torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
                torch.nn.ReLU(inplace=False),
                torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
                torch.nn.ReLU(inplace=False),
                torch.nn.Conv2d(in_channels=64, out_channels=51, kernel_size=3, stride=1, padding=1),
                torch.nn.ReLU(inplace=False),
                torch.nn.Upsample(scale_factor=2, mode='bilinear'),
                torch.nn.Conv2d(in_channels=51, out_channels=51, kernel_size=3, stride=1, padding=1)
            )

        self.moduleConv1 = Basic(6, 32)
        self.modulePool1 = torch.nn.AvgPool2d(kernel_size=2, stride=2)

        self.moduleConv2 = Basic(32, 64)
        self.modulePool2 = torch.nn.AvgPool2d(kernel_size=2, stride=2)

        self.moduleConv3 = Basic(64, 128)
        self.modulePool3 = torch.nn.AvgPool2d(kernel_size=2, stride=2)

        self.moduleConv4 = Basic(128, 256)
        self.modulePool4 = torch.nn.AvgPool2d(kernel_size=2, stride=2)

        self.moduleConv5 = Basic(256, 512)
        self.modulePool5 = torch.nn.AvgPool2d(kernel_size=2, stride=2)

        self.moduleDeconv5 = Basic(512, 512)
        self.moduleUpsample5 = torch.nn.Sequential(
            torch.nn.Upsample(scale_factor=2, mode='bilinear'),
            torch.nn.Conv2d(in_channels=512, out_channels=512, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(inplace=False)
        )

        self.moduleDeconv4 = Basic(512, 256)
        self.moduleUpsample4 = torch.nn.Sequential(
            torch.nn.Upsample(scale_factor=2, mode='bilinear'),
            torch.nn.Conv2d(in_channels=256, out_channels=256, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(inplace=False)
        )

        self.moduleDeconv3 = Basic(256, 128)
        self.moduleUpsample3 = torch.nn.Sequential(
            torch.nn.Upsample(scale_factor=2, mode='bilinear'),
            torch.nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(inplace=False)
        )

        self.moduleDeconv2 = Basic(128, 64)
        self.moduleUpsample2 = torch.nn.Sequential(
            torch.nn.Upsample(scale_factor=2, mode='bilinear'),
            torch.nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
            torch.nn.ReLU(inplace=False)
        )

        self.moduleVertical1 = Subnet()
        self.moduleVertical2 = Subnet()
        self.moduleHorizontal1 = Subnet()
        self.moduleHorizontal2 = Subnet()

        self.modulePad = torch.nn.ReplicationPad2d([ int(math.floor(51 / 2.0)), int(math.floor(51 / 2.0)), int(math.floor(51 / 2.0)), int(math.floor(51 / 2.0)) ])

        self.load_state_dict(torch.load('./network-' + arguments_strModel + '.pytorch'))

    def forward(self, variableInput1, variableInput2):
        variableJoin = torch.cat([variableInput1, variableInput2], 1)

        variableConv1 = self.moduleConv1(variableJoin)
        variablePool1 = self.modulePool1(variableConv1)

        variableConv2 = self.moduleConv2(variablePool1)
        variablePool2 = self.modulePool2(variableConv2)

        variableConv3 = self.moduleConv3(variablePool2)
        variablePool3 = self.modulePool3(variableConv3)

        variableConv4 = self.moduleConv4(variablePool3)
        variablePool4 = self.modulePool4(variableConv4)

        variableConv5 = self.moduleConv5(variablePool4)
        variablePool5 = self.modulePool5(variableConv5)

        variableDeconv5 = self.moduleDeconv5(variablePool5)
        variableUpsample5 = self.moduleUpsample5(variableDeconv5)

        variableDeconv4 = self.moduleDeconv4(variableUpsample5 + variableConv5)
        variableUpsample4 = self.moduleUpsample4(variableDeconv4)

        variableDeconv3 = self.moduleDeconv3(variableUpsample4 + variableConv4)
        variableUpsample3 = self.moduleUpsample3(variableDeconv3)

        variableDeconv2 = self.moduleDeconv2(variableUpsample3 + variableConv3)
        variableUpsample2 = self.moduleUpsample2(variableDeconv2)

        variableCombine = variableUpsample2 + variableConv2

        variableDot1 = SeparableConvolution()(self.modulePad(variableInput1), self.moduleVertical1(variableCombine), self.moduleHorizontal1(variableCombine))
        variableDot2 = SeparableConvolution()(self.modulePad(variableInput2), self.moduleVertical2(variableCombine), self.moduleHorizontal2(variableCombine))

        return variableDot1 + variableDot2

moduleNetwork = Network().cuda()

##########################################################

def process(tensorInputFirst, tensorInputSecond, tensorOutput):
    assert(tensorInputFirst.size(1) == tensorInputSecond.size(1))
    assert(tensorInputFirst.size(2) == tensorInputSecond.size(2))

    intWidth = tensorInputFirst.size(2)
    intHeight = tensorInputFirst.size(1)

    assert(intWidth <= 1280) # while our approach works with larger images, we do not recommend it unless you are aware of the implications
    assert(intHeight <= 720) # while our approach works with larger images, we do not recommend it unless you are aware of the implications

    intPaddingLeft = int(math.floor(51 / 2.0))
    intPaddingTop = int(math.floor(51 / 2.0))
    intPaddingRight = int(math.floor(51 / 2.0))
    intPaddingBottom = int(math.floor(51 / 2.0))
    modulePaddingInput = torch.nn.Module()
    modulePaddingOutput = torch.nn.Module()

    if True:
        intPaddingWidth = intPaddingLeft + intWidth + intPaddingRight
        intPaddingHeight = intPaddingTop + intHeight + intPaddingBottom

        if intPaddingWidth != ((intPaddingWidth >> 7) << 7):
            intPaddingWidth = (((intPaddingWidth >> 7) + 1) << 7) # more than necessary

        if intPaddingHeight != ((intPaddingHeight >> 7) << 7):
            intPaddingHeight = (((intPaddingHeight >> 7) + 1) << 7) # more than necessary

        intPaddingWidth = intPaddingWidth - (intPaddingLeft + intWidth + intPaddingRight)
        intPaddingHeight = intPaddingHeight - (intPaddingTop + intHeight + intPaddingBottom)

        modulePaddingInput = torch.nn.ReplicationPad2d([intPaddingLeft, intPaddingRight + intPaddingWidth, intPaddingTop, intPaddingBottom + intPaddingHeight])
        modulePaddingOutput = torch.nn.ReplicationPad2d([0 - intPaddingLeft, 0 - intPaddingRight - intPaddingWidth, 0 - intPaddingTop, 0 - intPaddingBottom - intPaddingHeight])

    if True:
        tensorInputFirst = tensorInputFirst.cuda()
        tensorInputSecond = tensorInputSecond.cuda()

        modulePaddingInput = modulePaddingInput.cuda()
        modulePaddingOutput = modulePaddingOutput.cuda()

    if True:
        variablePaddingFirst = modulePaddingInput(torch.autograd.Variable(data=tensorInputFirst.view(1, 3, intHeight, intWidth), volatile=True))
        variablePaddingSecond = modulePaddingInput(torch.autograd.Variable(data=tensorInputSecond.view(1, 3, intHeight, intWidth), volatile=True))
        variablePaddingOutput = modulePaddingOutput(moduleNetwork(variablePaddingFirst, variablePaddingSecond))

        tensorOutput.resize_(3, intHeight, intWidth).copy_(variablePaddingOutput.data[0])

    if True:
        tensorInputFirst.cpu()
        tensorInputSecond.cpu()
        tensorOutput.cpu()

tensorOutput = torch.FloatTensor()

if arguments_strVideo and arguments_strVideoOut:
    reader = FFMPEG_VideoReader(arguments_strVideo, False)
    writer = FFMPEG_VideoWriter(arguments_strVideoOut, reader.size, reader.fps*2)
    reader.initialize()
    nextFrame = reader.read_frame()
    for x in range(0, reader.nframes):
        firstFrame = nextFrame
        nextFrame = reader.read_frame()
        tensorInputFirst = torch.FloatTensor(numpy.rollaxis(firstFrame[:,:,::-1], 2, 0) / 255.0)
        tensorInputSecond = torch.FloatTensor(numpy.rollaxis(nextFrame[:,:,::-1], 2, 0) / 255.0)
        process(tensorInputFirst, tensorInputSecond, tensorOutput)
        writer.write_frame(firstFrame)
        writer.write_frame((numpy.rollaxis(tensorOutput.clamp(0.0, 1.0).numpy(), 0, 3)[:,:,::-1] * 255.0).astype(numpy.uint8))
    writer.write_frame(nextFrame)
    writer.close()
else:
    tensorInputFirst = torch.FloatTensor(numpy.rollaxis(numpy.asarray(PIL.Image.open(arguments_strFirst))[:,:,::-1], 2, 0).astype(numpy.float32) / 255.0)
    tensorInputSecond = torch.FloatTensor(numpy.rollaxis(numpy.asarray(PIL.Image.open(arguments_strSecond))[:,:,::-1], 2, 0).astype(numpy.float32) / 255.0)
    process(tensorInputFirst, tensorInputSecond, tensorOutput)
    PIL.Image.fromarray((numpy.rollaxis(tensorOutput.clamp(0.0, 1.0).numpy(), 0, 3)[:,:,::-1] * 255.0).astype(numpy.uint8)).save(arguments_strOut)

UPDATE

I found the reason of the error. It is because the image is grayscale while the other images I tested was coloured as coloured ones are 24 bit depth while grayscales are 8 bit depth. How can I modify the code for it to work on grayscale images.


title: Problem loading Pytourch 3.0 model unexpected key "module.features.0.weight" in state_dict

body:
I am trying to load a model I have trained using Pytorch,
but I keep getting the following error:

  File "convert.py", line 12, in
      model.load_state_dict(torch.load('model/model_vgg2d_2.pth'))   File
  "/usr/local/lib/python3.5/dist-packages/torch/nn/modules/module.py",
  line 490, in load_state_dict
      .format(name)) KeyError: 'unexpected key "module.features.0.weight" in state_dict'

Below is my code:

import torch.onnx
import torch.nn as nn

class TempModel(nn.Module):
    def __init__(self):
        super(TempModel, self).__init__()
        self.conv1 = nn.Conv2d(3, 5, (3, 3))
    def forward(self, inp):
        return self.conv1(inp)

model = nn.DataParallel(TempModel())
model.load_state_dict(torch.load('model/model_vgg2d_2.pth'))
dummy_input = Variable(torch.randn(1, 3, 224, 224))
torch.onnx.export(model, dummy_input, "model_onnx/model_vgg2d_0.onnx")

I am working on the same machine that I have used to train the model(which has multiple GPUs).
Any ideas what am I doing wrong?


title: Nsight Compute can't profile Waveglow (PyTorch application)

body:
I tried to profile  https://github.com/NVIDIA/waveglow  by this command:

nv-nsight-cu-cli --export ./nsight_output ~/.virtualenvs/waveglow/bin/python3 inference.py -f <(ls mel_spectrograms/*.pt) -w waveglow_256channels.pt -o . --is_fp16 -s 0.6

Python command is from instruction of https://github.com/NVIDIA/waveglow#generate-audio-with-our-pre-existing-model ,
and it works with Nsight System, not Nsight Compute.

Profiling doesn't end printing this log; so I pressed Ctrl+C.
Also, It profiles only one kernel but I have more kernels. (checked by Nsight Systems)

...
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 286: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 287: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 288: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 289: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 290: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 291: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 292: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 293: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 294: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 295: 0%....50%....100% - 48 passes
==PROF== Profiling "weight_norm_fwd_first_dim_ker..." - 296: 0%....50%...^C
==PROF== Received signal, trying to shutdown target application
 - 43 passes
==ERROR== Failed to profile kernel "weight_norm_fwd_first_dim_ker..." in process
==ERROR== An error occurred while trying to profile.
==ERROR== An error occurred while trying to profile
==PROF== Report: nsight_compute_result.nsight-cuprof-report

OS:  CentOS Linux 7, Nsight Compute (2019.3.1, Build 26317742),
GPU: Tesla V100-PCIE-32GB

How can I fix this?


title: how to use RGB values in feedforward neural network?

body:
I have data set of colored images in the form of ndarray (100, 20, 20, 3) and 100 corresponding labels. When passing them as input to a fully connected neural network (not CNN), what should I do with the 3 values of RGB? Average them perhaps lose some information, but if not manipulating them, my main issue is batch size, as demo-ed below in pytorch.

for epoch in range(n_epochs):
    for i, (images, labels) in enumerate(train_loader):
        # because of rgb values, now images is 3 times the length of labels
        images = Variable(images.view(-1, 400))
        labels = Variable(labels)
        optimizer.zero_grad()
        outputs = net(images)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

This returns 'ValueError: Expected input batch_size (300) to match target batch_size (100).' Should I have reshaped images into (1, 1200) dimension tensors? Thanks in advance for answers.


title: Pytorch. Optimizing the input to a model: Trying to backward through the graph a second time, but the buffers have already been freed

body:
Currently, I'm trying to optimize the values of an input tensor, x, to a model.
I want to restrict the input to only contain values in the range [0.0;1.0].

There is not too much information about how to do this, when not working with a layer as such.

I've created a minimum working example below, which gives the error message in the title of this post.

The magic happens in the optimize_x() function

If I comment out the line: model.x = model.x.clamp(min=0.0, max=1.0) the issue is fixed, but the tensor is obviously not clamped.

I'm aware that I could just set retain_graph=True - but it's not clear whether this is the right way to go, or if there is a better way of achieving this functionality?

import torch
from torch.distributions import Uniform

class OptimizeInputModel(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = torch.nn.Sequential(
                torch.nn.Linear(123, 1000),
                torch.nn.Dropout(0.4),
                torch.nn.ReLU(),
                torch.nn.Linear(1000, 100),
                torch.nn.Dropout(0.4),
                torch.nn.ReLU(),
                torch.nn.Linear(100, 1),
                torch.nn.Sigmoid(),
        )

        in_shape = (1, 123)
        self.x = torch.ones(in_shape) * 0.1
        self.x.requires_grad = True

    def forward(self) -> torch.Tensor:
        return self.model(self.x)

class MyLossFunc(torch.nn.Module):

    def forward(self, y: torch.Tensor) -> torch.Tensor:
        loss = torch.sum(-y)
        return loss

def optimize_x():
    model = OptimizeInputModel()
    optimizer = torch.optim.Adam([model.x], lr=1e-4)
    loss_fn = MyLossFunc()
    for epoch in range(50000):
        # Constrain X to have no values < 0
        model.x = model.x.clamp(min=0.0, max=1.0)
        y = model()
        loss = loss_fn(y)

        if epoch % 9 == 0:
            print(f'Epoch: {epoch}\t Loss: {loss}')

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

optimize_x()

Full error message:
RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.


title: 'MINST' downloading in Colab from pytorch's torchvision.datasets

body:
The following code run in Colab and I got the following error:

NameError: name 'MINST' is not defined

What do I need to do?
import torch
import torchvision
from torchvision.datasets import MNIST
dataset = MINST(root='data/', download=True)
len(dataset)
test_dataset = MINST(root='data/', train=False)
len(test_dataset)
dataset[0]


title: Is there a way to retrieve the specific parameters used in a random torchvision transform?

body:
I can augment my data during training by applying a random transform (rotation/translation/rescaling) but I don't know the value that was selected.
I need to know what values were applied. I can manually set these values, but then I lose a lot of the benefits that torch vision transforms provide.
Is there an easy way to get these values are implement them in a sensible way to apply during training?
Here is an example. I would love to be able print out the rotation angle, translation/rescaling being applied at each image:
import numpy as np
import matplotlib.pyplot as plt
from torchvision import transforms

RandAffine = transforms.RandomAffine(degrees=0, translate=(0.1, 0.1), scale=(0.8, 1.2))

rotate = transforms.RandomRotation(degrees=45)
shift = RandAffine
composed = transforms.Compose([rotate,
                               shift])

# Apply each of the above transforms on sample.
fig = plt.figure()
sample = np.zeros((28,28))
sample[5:15,7:20] = 255
sample = transforms.ToPILImage()(sample.astype(np.uint8))
title = ['None', 'Rot','Aff','Comp']
for i, tsfrm in enumerate([None,rotate, shift, composed]):
    if tsfrm:
        t_sample = tsfrm(sample)
    else:
        t_sample = sample
    ax = plt.subplot(1, 5, i + 2)
    plt.tight_layout()
    ax.set_title(title[i])
    ax.imshow(np.reshape(np.array(list(t_sample.getdata())), (-1,28)), cmap='gray')

plt.show()


title: torch.nn.embedding has run time error

body:
I want to use torch.nn.Embedding. I have followed the codes in the documentation of embedding command.
here is the code:

# an Embedding module containing 10 tensors of size 3
embedding = nn.Embedding(10, 3)
# a batch of 2 samples of 4 indices each
input = torch.LongTensor([[1,2,4,5],[4,3,2,9]])
embedding(input)

The documentation says that you will receive this output:

tensor([[[-0.0251, -1.6902,  0.7172],
         [-0.6431,  0.0748,  0.6969],
         [ 1.4970,  1.3448, -0.9685],
         [-0.3677, -2.7265, -0.1685]],

        [[ 1.4970,  1.3448, -0.9685],
         [ 0.4362, -0.4004,  0.9400],
         [-0.6431,  0.0748,  0.6969],
         [ 0.9124, -2.3616,  1.1151]]])

but I don't receive this output. instead I receive this error:

Traceback (most recent call last):
  File "/home/mahsa/PycharmProjects/PyTorch_env_project/PyTorchZeroToAll-master/temporary.py", line 12, in <module>
    embedding(input)
  File "/home/mahsa/anaconda3/envs/pytorch_env/lib/python3.5/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mahsa/anaconda3/envs/pytorch_env/lib/python3.5/site-packages/torch/nn/modules/sparse.py", line 94, in forward
    self.scale_grad_by_freq, self.sparse
RuntimeError: save_for_backward can only save input or output tensors, but argument 0 doesn't satisfy this condition

Can anybody guide me about this error? and about the work of torch.nn.Embedding?


title: Several questions regarding my implementation of PPO on Pytorch

body:
I’ve been learning RL this summer and this week I’ve tried to make a PPO implementation on Pytorch with the help of some repositories from github with similiar algorithms.

The code runs OpenAI’s Lunar Lander but I have several errors that I have not been able to fix, the biggest one being that the algorithm quickly converges to doing the same action regardless of the state. The other major problem I’ve found is that even though I’m using backwards() only once, I get an error asking me to set retain_graph to True.

Because of that, I see no improvement of the rewards obtained over 1000 steps, I don’t know if the algorithm needs more steps to be able to see an improvement.

I’m really sorry if this kind of problems have no place in this forums, I just didn’t know where to post this.
Also I’m sorry for the messy code, it’s my first time doing this kind of algorithms, and I’m fairly new with pytorch and machine learning in general.

import torch
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import matplotlib.pyplot as plt
from torch.distributions import Categorical
import gym

class actorCritic(nn.Module):
    def __init__(self):
        super(actorCritic, self).__init__()
        self.fc = nn.Sequential(
        nn.Linear(8, 16),
        nn.Linear(16, 32),
        nn.Linear(32, 64),
        nn.ReLU(inplace=True)
        )

        self.pi = nn.Linear(64, 4)
        self.value = nn.Linear(64, 1)
    def forward(self, x):
        x = self.fc(x)
        pi_1 = self.pi(x)
        pi_out = F.softmax(pi_1, dim=-1)
        value_out = self.value(x)
        return pi_out, value_out

def GAE(rewards, values, masks):
    gamma = 0.99
    lamb = 0.95
    advan_t = 0
    sizes = rewards.size()

    advantages = torch.zeros(1, sizes[1])
    for t in reversed(range(sizes[1])):
        delta = rewards[0][t] + gamma*values[0][t+1]*masks[0][t] - values[0][t]
        advan_t = delta + gamma*lamb*advan_t*mask[0][t]
        advantages[0][t] = advan_t

    real_values = values[:,:sizes[1]] + advantages

    return advantages, real_values
def plot_rewards(rewards):
    plt.figure(2)
    plt.clf()
    plt.plot(rewards)
    plt.pause(0.001)
    plt.savefig('TruePPO 500 steps.png')

def interact(times, states):

    rewards = torch.zeros(1, times)

    actions = torch.zeros(1, times)
    mask = torch.ones(1, times)
    for steps in range(times):
        action_probs, _ = network(states[steps])
        m = Categorical(action_probs)
        action = int(m.sample())

        obs, reward, done, _ = env.step(action)

        if done:
            obs = env.reset()
            mask[0][steps] = 0

        states[steps+1] = torch.from_numpy(obs).float()
        rewards[0][steps] = reward
        actions[0][steps] = action

    return states, rewards, actions, mask

#Parameters

total_steps = 1000
batch_size = 10
env = gym.make('LunarLander-v2')
network = actorCritic()
old_network = actorCritic()
optimizer = torch.optim.Adam(network.parameters(), lr = 0.001)
states = torch.zeros(batch_size+1, 8)
steps = 0
obs_ = env.reset()
obs = torch.from_numpy(obs_).float()
states[0] = obs
reward_means = []
nn_paramD = network.state_dict()
old_network.load_state_dict(nn_paramD)

while steps < total_steps:
    print (steps)
    states, rewards, actions, mask = interact(batch_size, states)
    #calculate values, GAE, normalize advantages, randomize, calculate loss, backprop,
    _, values = network(states)
    values = values.view(-1, batch_size+1)

    advantages, v_targ = GAE(rewards, values, mask)

    advantages = (advantages - advantages.mean()) / (advantages.std() + 1e-5)

    optimizer.zero_grad()
    for n in range(rewards.size()[1]):

        probabilities, _ = network(states[n])
        print (probabilities)
        m = Categorical(probabilities)
        action_prob = m.probs[int(actions[0][n])]

        entropia = m.entropy()

        old_probabilities, _ = old_network(states[n])
        m_old = Categorical(old_probabilities)
        old_action_prob = m.probs[actions[0][n].int()]
        old_action_prob.detach()

        ratio = action_prob / old_action_prob

        surr1 = ratio*advantages[0][n]
        surr2 = torch.clamp(ratio, min = (1.-0.2), max = (1.+0.2))

        policy_loss = -torch.min(surr1, surr2)

        value_loss = 0.5*(values[0][n]-v_targ[0][n])**2

        entropy_loss = -entropia

        total_loss = policy_loss + value_loss + 0.01*entropy_loss

        total_loss.backward(retain_graph = True)

        optimizer.step()

    reward_means.append(rewards.numpy().mean())
    old_network.load_state_dict(nn_paramD)
    nn_paramD = network.state_dict()
    steps += 1
    plot_rewards(reward_means)


title: Frequently getting CUDA error out of memory?

body:
I have written a code for training a deep learning model where I am deleting cuda tensors after every batch and doing torch.cuda.empty_cache() after that. I am pretty sure that batch size is not large enough to cause this error. What could be the possible reason for this?

    for epoch in range(1+last_epoch, self.num_epochs+1):
        for phase in ['train', 'val']:
            loss_arr = []
            if phase == 'train':
                model.train()
                scheduler.step()
                was_training = True
            else:
                model.eval()
                was_training = False
            for i_batch, sample_batched in enumerate(dataloaders[phase]):
                X = sample_batched[0]
                y = sample_batched[1].type(torch.LongTensor)
                w = sample_batched[2]

                if model.is_cuda:
                    X, y, w = X.cuda(non_blocking=True), y.cuda(non_blocking=True),  w.cuda(non_blocking=True)

                output = model(X)
                loss = self.loss_func(output, y, w)
                if phase == 'train':
                    curr_iteration+=1
                    optim.zero_grad()
                    loss.backward()
                    optim.step()
                    if (curr_iteration % log_nth == 0):
                        self.logWriter.loss_per_iter(loss.item(), curr_iteration)

                loss_arr.append(loss.item())

                with torch.no_grad():
                    self.logWriter.update_cm_per_iter(output, y, self.labels, phase)

                del X, y, w, output, loss
                torch.cuda.empty_cache()

            self.logWriter.loss_per_epoch(loss_arr, phase, epoch)

            epoch_output, epoch_labels = model.predict(dataloaders[phase].dataset.X), dataloaders[phase].dataset.y
            self.logWriter.dice_score_per_epoch(epoch_output, epoch_labels, phase, epoch)
            index = np.random.choice(len(dataloaders[phase].dataset), 3, replace=False)
            self.logWriter.image_per_epoch(epoch_output[index], epoch_labels[index], phase, epoch)
            self.logWriter.cm_per_epoch(self.labels, phase, epoch, i_batch)
            del epoch_output, epoch_labels

        print("==== Epoch ["+str(epoch)+" / "+str(self.num_epochs)+"] done ====")
        model.save('models/' + self.exp_dir_name + '/quicknat_epoch' + str(epoch) + '.model')

And the perdict function inside the model is as follows

def predict(self, X, enable_dropout = False):
    """
    Predicts the outout after the model is trained.
    Inputs:
    - X: Volume to be predicted
    """
    self.eval()

    if type(X) is np.ndarray:
        X = torch.tensor(X, requires_grad = False).cuda(non_blocking=True)
    elif type(X) is torch.Tensor and not X.is_cuda:
        X = X.cuda(non_blocking=True)

    if enable_dropout:
        self.enable_test_dropout()

    with torch.no_grad():
        out = self.forward(X)

    max_val, idx = torch.max(out,1)
    idx = idx.data.cpu().numpy()
    prediction = np.squeeze(idx)
    del X, out, idx, max_val
    return prediction


title: Installing PyTorch on PyCharmin dont work anymore

body:
I cant install PyTorch in PyCharm anymore on one of my computers. PyCharm shows me some weird error messages and i was not able to solve this problem.

Im running Python 3.7 with pip 19.0.3.

Everytime i try to use commands like:

pip3 install torch==1.3.0+cpu torchvision==0.4.1+cpu -f https://download.pytorch.org/whl/torch_stable.html

it ends up with following error message:

  Could not find a version that satisfies the requirement
  torch==1.3.0+cpu (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2) No
  matching distribution found for torch==1.3.0+cpu

If i try with this command:

pip3 install torch

this error shows up:

  ModuleNotFoundError: No module named 'tools.nnwrap'

I searched for a solution on Google all the time but did not find a solution.
I have reinstalled PyCharm to check out if it was only a bug. The problem is still there.
I reinstalled pip but nothing.

Can you help me?


title: Forward pass gets 10000x slower after iterating for a while

body:
I implemented a simple Deconv network just like pytorch's official DCGAN tutorial.
I repeatedly pass a zeros vector to it. The time taken slows down significantly after a while. I am wondering what the reason is and how I can resolve it.

Code:

import torch
import torch.nn as nn
import time

# JUST TO MEASURE TIME
class Timer:
    def __init__(self, msg):
        self.msg = msg

    def __enter__(self):
        self.start = time.process_time()
        return self

    def __exit__(self, *args):
        self.end = time.process_time()
        self.interval = self.end - self.start

        print('{}: {:.5f}'.format(self.msg, self.interval))

device = torch.device("cuda")

ngf, nc, nz, batchSize = 64, 1, 6, 1<<16
class Generator(nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        self.main = nn.Sequential(
            # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 4, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 4 x 4
            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 8 x 8
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 16 x 16
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 32 x 32
        )

    def forward(self, input):
        return self.main(input)

# Create the generator
netG = Generator().to(device)

def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        nn.init.normal_(m.weight.data, 0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        nn.init.normal_(m.weight.data, 1.0, 0.02)
        nn.init.constant_(m.bias.data, 0)

netG.apply(weights_init)

# torch.backends.cudnn.benchmark=True

while True:
    with Timer('Time elapsed'):
        with torch.no_grad():
            netG(torch.zeros([batchSize, nz, 1, 1], device=device))

Result:

  Time elapsed: 0.02309 Time elapsed: 0.00072 Time elapsed: 0.00208 Time
  elapsed: 0.00128 Time elapsed: 0.00119 Time elapsed: 0.00153 Time
  elapsed: 0.00176 Time elapsed: 0.00170 Time elapsed: 0.00185 Time
  elapsed: 0.00188 Time elapsed: 0.00191 Time elapsed: 0.00190 Time
  elapsed: 0.00171 Time elapsed: 0.00176 Time elapsed: 0.00167 Time
  elapsed: 0.00120 Time elapsed: 0.00168 Time elapsed: 0.00169 Time
  elapsed: 0.00166 Time elapsed: 0.00167 Time elapsed: 0.00171 Time
  elapsed: 0.00168 Time elapsed: 0.00168 Time elapsed: 0.00168 Time
  elapsed: 0.00169 Time elapsed: 0.00177 Time elapsed: 0.00173 Time
  elapsed: 0.00176 Time elapsed: 0.00173 Time elapsed: 0.00171 Time
  elapsed: 0.00168 Time elapsed: 0.00173 Time elapsed: 0.00168 Time
  elapsed: 0.00178 Time elapsed: 0.00169 Time elapsed: 0.00171 Time
  elapsed: 0.00168 Time elapsed: 0.00169 Time elapsed: 0.00169 Time
  elapsed: 0.00173 Time elapsed: 0.00154 Time elapsed: 0.00170 Time
  elapsed: 0.00167 Time elapsed: 0.00224 Time elapsed: 0.00117 Time
  elapsed: 0.00175 Time elapsed: 0.00168 Time elapsed: 0.00173 Time
  elapsed: 0.00169 Time elapsed: 12.62760 Time elapsed: 12.71425 Time
  elapsed: 12.71379 Time elapsed: 12.71846 Time elapsed: 12.71909 Time
  elapsed: 12.71898 Time elapsed: 12.72288 Time elapsed: 12.72157 Time
  elapsed: 12.72226 Time elapsed: 12.72456 Time elapsed: 12.72350 Time
  elapsed: 12.72480 Time elapsed: 12.72644 Time elapsed: 12.72337 Time
  elapsed: 12.72424 Time elapsed: 12.72538 Time elapsed: 12.72533 Time
  elapsed: 12.72510 Time elapsed: 12.72507 Time elapsed: 12.72806 Time
  elapsed: 12.72865 Time elapsed: 12.72764 Time elapsed: 12.72431

My GPU: Titan RTX
PyTorch version: 1.4
Python version: 3.7


title: Applying a 1D Convolution on a Tensor in Pytorch

body:
I have a Tensor that represents a set of 1D signals, that are concatenated along the column axis. So say I have 300 1D signals that are of size 64. So [64x300]

I want to apply a smooth convolution / moving average kernel on it [0.2 0.2 0.2 0.2 0.2] on the GPU, but I am not sure exactly what is the API to do it. Can I be provided an example?


title: PyTorch - get list of sums of 2D tensors in 3D tensor

body:
I have a 3D tensor consisting of 2D tensors, e. g.:
t = torch.tensor([[[0, 0, 1],
                   [0, 1, 0],
                   [1, 0, 0]],

                  [[0, 0, 1],
                   [0, 1, 0],
                   [1, 0, 0]],

                  [[0, 0, 1],
                   [0, 1, 0],
                   [1, 0, 0]]
                  ])

I need a list or tensor of sums of those 2D tensors, e. g.: sums = [3, 3, 3]. So far I have:
sizes = [torch.sum(t[i]) for i in range(t.shape[0])]

I think this can be done with PyTorch only, but I've tried using torch.sum() with all possible dimensions and I always get sums over the individual fields of those 2D tensors, e. g.:
[[0, 0, 3],
[0, 3, 0],
[3, 0, 0]]

How can this be done in PyTorch?


title: Tensorflow not detecting GPU where as Pytorch does

body:
ubuntu 18.04,
nvidia mx150,
cuda 10.1,
tensorflow installed using both $ pip install tensorflow and $ pip install tensorflow-gpu.
the problem is that it doesn't detect the GPU, but when tried with pytorch it does. cant seem to find the problem.
Thanks in advance.
Jupyterlab output when tried to test GPU
edit 1 - I have set the CUDA_VISIBLE_DEVICES to 0, i'm able to detect the GPU manually but tensorflow is not able to, i did not try the source build option yet.


title: U-net training Error: The size of tensor a (16) must match the size of tensor b (6) at non-singleton dimension 1

body:
I’m trying to train a Unit model on LandCoverNet dataset, which is a satellite imagery dataset that contains input images and corresponding land cover type masks.
I have created a custom dataset to get my images and masks:
# Create custom dataset that accepts 4 channels images
from torch.utils.data import Dataset, DataLoader, sampler
from pathlib import Path
from PIL import Image
import matplotlib.pyplot as plt
import os
import numpy as np
import rasterio as rio
from torchvision import transforms, datasets, models
# We have two dir: inputs(folder for each image) and tatgets

class LandCoverNetDataset(BaseDataset):

  CLASSES = ['otherland', 'cropland', 'pastureland', 'bare soil', 'openwater', 'forestland']

  def __init__(self, inputs_dir, targets_dir,
               classes = None,
               augmentation=None ,
               preprocessing = False,
               pytorch=True):

    super().__init__()
    self.samples = []
    self.pytorch = pytorch
    self.augmentation = augmentation
    self.preprocessing = preprocessing

    # Convert str names to class values on masks
    self.class_value = [self.CLASSES.index(cls.lower()) for cls in classes]

    # Create dictionary for images and targets
    for sub_dir in os.listdir(inputs_dir):
      files = {}
      files = {
         'img_bands' : os.path.join(inputs_dir, sub_dir),
          'target' : os.path.join(targets_dir, sub_dir[:13] + "_LC_10m.png")
        }
      self.samples.append(files)

  def __len__(self):
    return len(self.samples)

  def normalize(self, band):

    '''Notmalize a numpy array to have values between 0 and 1'''
    band_min, band_max = band.min(), band.max()
    np.seterr(divide='ignore', invalid='ignore')
    normalized_band = ((band - band_min)/(band_max - band_min))
    #Remove any nan value and subtitute by zero
    where_are_NaNs = isnan(normalized_band)
    normalized_band[where_are_NaNs] = 0
    return normalized_band

  def open_as_array(self, idx, include_ndvi = False):
    '''
      Merge the 4 bands into one image and normalize the bands
    '''
    # List indivisual bands in each image folder
    # Stack them togather
    list_bands = []
    for img_file in os.listdir(self.samples[idx]['img_bands']):
      # Get the ndvi band
      if 'NDVI' in img_file:
        ndvi_band = os.path.join(self.samples[idx]['img_bands'], img_file)
      else:
        # Get the rgb bands
        band = rio.open(os.path.join(self.samples[idx]['img_bands'], img_file)).read(1)

        if self.preprocessing:
          # preprocess the bands before stacking them (only rgb)
          band = self.normalize(band)
        list_bands.append(band)

    # Stack the bands
    raw_rgb = np.stack(list_bands, axis=2).astype('float32')

    if include_ndvi:
      # Include the NDVI band in the input images
      ndvi = np.expand_dims(rio.open(ndvi_band).read(1).astype('float32'), 2)
      raw_rgb = np.concatenate([raw_rgb, ndvi], axis=2)

    if self.augmentation:
      transformed = self.augmentation(image = raw_rgb)
      raw_rgb  = transformed["image"]

    if self.preprocessing:
      # transpose to tensor shape
      raw_rgb = raw_rgb.transpose((2,0,1)).astype('float32')

    return raw_rgb

  def open_mask(self, idx):
    # Extract certain classes from mask

    mask = cv2.imread(self.samples[idx]['target'], 0)
    masks = [(mask == v) for v in self.class_value]
    mask = np.stack(masks, axis=-1).astype('long')

    if self.augmentation:
      transformed = self.augmentation(image = mask)
      mask  = transformed["image"]

    if self.preprocessing:
      # preprocess the mask
      mask = self.normalize(mask)
      # transpose to tensor shape
      mask = mask.transpose((2, 0, 1)).astype('long')
      mask = mask[0, :, :]
    return mask

  def __getitem__(self, idx):
    x = torch.tensor(self.open_as_array(idx, include_ndvi=True), dtype=torch.float)
    y = torch.tensor(self.open_mask(idx), dtype=torch.long)

    return x, y

  def open_as_pil(self, idx):
    arr = 256*self.open_as_array(idx)
    return Image.fromarray(arr.astype(np.uint8), 'RGB')

  def __repr__(self):
    s = 'Dataset class with {} files'.format(self.__len__())
    return s

The input here is 4 bands.
This is the shape of the first batch for both input/target
torch.Size([16, 4, 224, 224])
torch.Size([16, 224, 224])
I’m using a model from segmentation-models-pytorch library, and here is how I customized it for my case:
ENCODER = 'se_resnext50_32x4d'
ENCODER_WEIGHTS = 'imagenet'
ACTIVATION = 'softmax2d'
DEVICE = 'cuda'

model = smp.FPN(ENCODER, classes=len(CLASSES), activation=ACTIVATION)

# Replace the model.conv1 to accept 4 channels
# first: copy the layer's weights
weight = model.encoder.layer0.conv1.weight.clone()
model.encoder.layer0.conv1 = nn.Conv2d(4, 64,kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)
with torch.no_grad():
  model.encoder.layer0.conv1.weight[:, :3] = weight
  model.encoder.layer0.conv1.weight[:, 3] = model.encoder.layer0.conv1.weight[:, 0]

loss = smp.utils.losses.NLLLoss()
metrics = [
    smp.utils.metrics.IoU(threshold=0.5),
]
optimizer = torch.optim.SGD([
    dict(params=model.parameters(), lr=0.001, weight_decay=1e-8, momentum=0.9),
])

# create epoch runners
# it is a simple loop of iterating over dataloader`s samples
train_epoch = smp.utils.train.TrainEpoch(
    model,
    loss=loss,
    metrics=metrics,
    optimizer=optimizer,
    device=DEVICE,
    verbose=True,
)

valid_epoch = smp.utils.train.ValidEpoch(
    model,
    loss=loss,
    metrics=metrics,
    device=DEVICE,
    verbose=True,
)

And here is my training loop

# train model for 40 epochs

max_score = 0

for i in range(0, 40):

    print('\nEpoch: {}'.format(i))
    train_logs = train_epoch.run(train_loader)
    valid_logs = valid_epoch.run(valid_loader)

    # do something (save model, change lr, etc.)
    if max_score < valid_logs['iou_score']:
        max_score = valid_logs['iou_score']
        torch.save(model, './best_model.pth')
        print('Model saved!')

    if i == 25:
        optimizer.param_groups[0]['lr'] = 1e-5
        print('Decrease decoder learning rate to 1e-5!')

At first, the target shape was [16, 6, 224, 224] but I had an error and found this thread that it should be [batch_size, height, width]
That’s why I added this line in the Dataset class : mask = mask[0, :, :]
to get ride of the number of classes dim, and here where things get confusing for me, because the output of me model is torch.Size([10, 6, 224, 224]).
This is the entire error message:
Epoch: 0
train:   0%|          | 0/157 [00:00<?, ?it/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-215-2ae39e205dee> in <module>()
      7
      8     print('\nEpoch: {}'.format(i))
----> 9     train_logs = train_epoch.run(train_loader)
     10     valid_logs = valid_epoch.run(valid_loader)
     11

3 frames
/usr/local/lib/python3.6/dist-packages/segmentation_models_pytorch/utils/functional.py in iou(pr, gt, eps, threshold, ignore_channels)
     32     pr, gt = _take_channels(pr, gt, ignore_channels=ignore_channels)
     33
---> 34     intersection = torch.sum(gt * pr)
     35     union = torch.sum(gt) + torch.sum(pr) - intersection + eps
     36     return (intersection + eps) / union

RuntimeError: The size of tensor a (16) must match the size of tensor b (6) at non-singleton dimension 1

Thanks!


title: Neural Chess: Sample neural network gets stuck at value

body:
I am attempting to write a neural network to play chess, but I am running into problems with the output. I am using python-chess library, and built rewards in. The network has 4 outputs and three Fully Connected Layers. The 4 outputs should map between 0 and 7, and the first two and last two outputs map each map to a square's rank and file. To squash the output, I sigmoid each output and multiply by seven. The problem is that after a few learning epochs the learning hits a wall. The network outputs the same output regardless of initial seed, something like 3443 or 4333, and the outputs before sigmoid are all super close to zero. I figured it was the fact that the negative reward from failed moves and the sigmoid derivative was moving the output to 0, and thus outputted 3s and 4s. I need this network to learn through reinforcement learning, but this is stalling its learning badly.

Code:

def forward(self, x):
    x = F.relu(self.action_head2(F.relu(self.action_head(F.relu(self.affine1(x))))))
    action_scores = self.action_head3(x)
    state_values = self.value_head(x)
    print(action_scores)
    return F.sigmoid(action_scores)*7, state_values


title: python - Machine Learning 2D Regression with confidence bands

body:
I have measures of a variable versus time.
I want to obtain a regression with confidence bands so that the plot sounds like this:

Given an arbitrary x, I "predicted" the y and its confidence by evaluating mean and std of N closest measured points. But I feel like this is quite naive.

I was wondering if it is possible to obtain a similar result with Machine Learning, Neural Network, ...

I'm quite familiar with python, sklearn and pytorch so if you please could suggest a solution that implements those tools I would be very thankfull.


title: Visualize DGLGraph based model

body:
I'm working on a graph convolutional model taking DGLGraph as input. Is there any method to visualize the model structure?

I found libraries such as torchsummary and visualdl requires the input to be pytorch tensors or python data types.


title: How to select a GPU based on PCI bus id?

body: I would like to know if it is possible to select a GPU in Tensorflow and Pytorch based on pci bus id or uuid?