title: Pytorch not using GPU . Worked on Fastai

body:
My GPU Nvidia gtx1050 Ti

I am trying to train it on GPU but I only see CPU utilization 60-90 percent and GPU around 5 percent during training may be due to the copying of tensors to GPU I don't know. But it just goes up 5 percent and comes down.

I tried increasing batch size to 64 or 128 based on some solutions online but it just gives me Cuda out of memory error. I have used like 2.3gb out of 4gb and says need 120 MB and has only 110 mb.---- I dont even understand  how it works at this point

Then tried decreasing batch size to 16 then also got Cuda out of memory error saying need like 40 MB and has 16mb same 2.3 GB used out of 4gb

In the end it worked for 8 batch size but it only uses CPU no GPU

I used fastai with batch size 128 it works fine and uses gpu . I dont know where I did wrong. Any help appreciated. Below is my code wrote based on pytorch image classifier tutorial

Model Resnet pretrained True
has 205 labels with 117000 images approx as data for training.
Just using pretrained weights such that weights will modify accordingly without freezing weights and not training from scratch I assume is what the code does. Feel free to correct me if I did something wrong or better solution. In the end I am just noob in pytorch .... My first code in pytorch

trainset....... PIL Image format with rgb converted to prevent error of 4 channel and removed if not opened by PIL --invalid images: corrupted image

device = torch.device("cuda:0") #0 device is my nvidia gtx 1050 ti when printed

model.fc=nn.Linear(2048, 205)

from torchvision import transforms
t = transforms.Compose([
 transforms.Resize(256),                    #[2]
 transforms.CenterCrop(224),                #[3]
 transforms.ToTensor(),                     #[4]
 transforms.Normalize(                      #[5]
 mean=[0.485, 0.456, 0.406],                #[6]
 std=[0.229, 0.224, 0.225] )                 #[7]
 ])
trainloader = torch.utils.data.DataLoader(trainset, batch_size=8,
                                          shuffle=True)

if torch.cuda.is_available():
    print('yes gpu')
    torch.set_default_tensor_type('torch.cuda.FloatTensor')
    model = model.cuda()

import torch.optim as optim
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

for epoch in range(6):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs; data is a list of [inputs, labels]
        inputs, labels,img_name = data
        inputs = inputs.to(device)
        labels = labels.to(device)
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        total=0
        correct=0
        if i % 2000 == 1999:
            # print every 2000 mini-batches

            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            outputs = model(inputs)
            _, predicted = torch.max(outputs.data, 1)
            total += labels.size(0)
            correct += (predicted == labels).sum().item()

            print('Accuracy of the network on the batch size  images: %d %%' % (
                100 * correct / total)) #training accuraccy

            running_loss = 0.0

print('Finished Training')


title: Why is PyTorch slower than PyOpenCL, which is slower than Numba on GPU?

body:
I was working on a FDTD program that used the discrete Laplacian, which I can be implemented as a convolution operation. From what I have read, the main component of PyTorch is a tensor library optimized to perform operations commonly used for machine learning (such as convolutions). I was interested to compare it to other frameworks I have used, so I wrote a test program to apply the discrete Laplacian to a 1d array multiple times and compare execution times:

import torch as tr
import time
from numba import jit, cuda
import numpy as np
import pyopencl as cl
from pyopencl import array

#parameters
number_of_timesteps = 1000
number_of_elements = 10000000

#set up the inital conditions
torch_data = tr.rand((1,1,number_of_elements),dtype=tr.double) #torch convolution needs shape (minibatch,in_channels,iW)
numba_data = np.array([0] + list(torch_data[0][0].numpy()) + [0]) #add padding [0] for convolution. handled automatically in torch.
opencl_data = np.array([0] + list(torch_data[0][0].numpy()) + [0])

#test Torch
device = "cuda"
torch_data_a = torch_data.to(device)
torch_data_b = torch_data.to(device)
kernel = tr.tensor([[[1,-2,1]]],dtype=tr.double,device=device)
with tr.no_grad():
    start_time = time.time()
    for t in range(round(number_of_timesteps/2)): # /2 because each loop is two convolutions
        torch_data_b = torch_data_a + 0.1* tr.nn.functional.conv1d(torch_data_a,kernel,padding=1)
        torch_data_a = torch_data_b + 0.1* tr.nn.functional.conv1d(torch_data_b,kernel,padding=1)
    print("Torch GPU time:",time.time()-start_time)
    torch_data_numpy = np.array([0] + list(torch_data_a[0][0].cpu().numpy()) + [0])

#Numba GPU kernel
@cuda.jit
def numba_conv_cuda(x,x_new):
    gid = cuda.grid(1)
    if 0 < gid < x.size - 1 :  # Check array boundaries
        x_new[gid] = x[gid] + 0.1*(x[gid+1]+x[gid-1]-2*x[gid])

threadsperblock = 100
blockspergrid = (numba_data.size + (threadsperblock - 1)) // threadsperblock
x_a = cuda.to_device(numba_data)
x_b = cuda.to_device(numba_data)
start_time = time.time()
#actually run the kernel
for t in range(round(number_of_timesteps/2)): #again /2 because each loop is two convolutions
    numba_conv_cuda[blockspergrid, threadsperblock](x_a,x_b)
    numba_conv_cuda[blockspergrid, threadsperblock](x_b,x_a)
print("Numba GPU time:",time.time()-start_time)
numba_data = x_a.copy_to_host()

#test OpenCL
context = cl.create_some_context(interactive=False,answers=[0])
queue = cl.CommandQueue(context)
mem_flags = cl.mem_flags
program = cl.Program(context, """
    #pragma OPENCL EXTENSION cl_khr_fp64 : enable //enable double precision calculations
    __kernel void update_psi(__global const double *x, __global double *x_new)
    {
        int gid = get_global_id(0);
        if(0 < gid && gid < x.size - 1){
            x_new[gid] = x[gid] + 0.1*(x[gid+1]+x[gid-1]-2*x[gid]);
        }
    }
    """.replace("x.size",str(opencl_data.size))).build()
x_a_buf = cl.Buffer(context, mem_flags.READ_WRITE | mem_flags.COPY_HOST_PTR, hostbuf=opencl_data)
x_b_buf = cl.Buffer(context, mem_flags.READ_WRITE | mem_flags.COPY_HOST_PTR, hostbuf=opencl_data)

#actually run the OpenCL
start_time = time.time()
for t in range(round(number_of_timesteps/2)): #again /2 because each loop is two convolutions
    event = program.update_psi(queue, [threadsperblock*blockspergrid], [threadsperblock], x_a_buf, x_b_buf)
    event.wait()
    event = program.update_psi(queue, [threadsperblock*blockspergrid], [threadsperblock], x_b_buf, x_a_buf)
    event.wait()
print("OpenCL GPU time:",time.time()-start_time)
event = cl.enqueue_copy(queue, opencl_data, x_a_buf)
event.wait()

print("Results are same?",np.allclose(torch_data_numpy,numba_data) and np.allclose(numba_data,opencl_data))

And these are the results testing on an Nvidia GPU:

Torch GPU time: 13.544365406036377
Numba GPU time: 0.2404193878173828
OpenCL GPU time: 0.9025869369506836
Results are same? True

I am surprised that the results show that a library designed for applying operations such as convolutions to be so much slower than Numba or PyOpenCL (which is not even optimized because I did not use any local memory on the GPU). Is this really the case, or did I do something wrong?

Additionally, why is the kernel written in c more than 3x slower than the kernel written in Python?


title: fp16 inference on cpu Pytorch

body:
I have a pretrained pytorch model I want to inference on fp16 instead of fp32, I have already tried this while using the gpu but when I try it on cpu I get:
"sum_cpu" not implemented for 'Half' torch.
any fixes?


title: TensorRT (C++ API) undefined reference to `createNvOnnxParser_INTERNAL'

body:
I am trying to create a tensorrt engine from ONNX model using the TensorRT C++ API. I have written code to read, serialize and write a tensorrt engine to disk as per the documentation. I have installed tensorrt7 on colab using debian installation instructions.
This is my c++ code that I am compiling using g++ rnxt.cpp -o rnxt
#include <cuda_runtime_api.h>
#include <NvOnnxParser.h>
#include <NvInfer.h>

#include <cstdlib>
#include <fstream>
#include <iostream>
#include <sstream>
#include <iterator>
#include <algorithm>

class Logger : public nvinfer1::ILogger
 {
     void log(Severity severity, const char* msg) override
     {
         // suppress info-level messages
         if (severity != Severity::kINFO)
             std::cout << msg << std::endl;
     }
 } gLogger;

int main(){

    int maxBatchSize = 32;

    nvinfer1::IBuilder* builder = nvinfer1::createInferBuilder(gLogger);
    const auto explicitBatch = 1U << static_cast<uint32_t>(nvinfer1::NetworkDefinitionCreationFlag::kEXPLICIT_BATCH);
    nvinfer1::INetworkDefinition* network = builder->createNetworkV2(explicitBatch);

    nvonnxparser::IParser* parser = nvonnxparser::createParser(*network, gLogger);

    parser->parseFromFile("saved_resnext.onnx", 1);
    for (int i = 0; i < parser->getNbErrors(); ++i)
    {
        std::cout << parser->getError(i)->desc() << std::endl;
    }

    builder->setMaxBatchSize(maxBatchSize);
    nvinfer1::IBuilderConfig* config = builder->createBuilderConfig();
    config->setMaxWorkspaceSize(1 << 20);
    nvinfer1::ICudaEngine* engine = builder->buildEngineWithConfig(*network, *config);

    parser->destroy();
    network->destroy();
    config->destroy();
    builder->destroy();

    nvinfer1::IHostMemory *serializedModel = engine->serialize();

    std::ofstream engine_file("saved_resnext.engine");

    engine_file.write((const char*)serializedModel->data(),serializedModel->size());

    serializedModel->destroy();
    return 0;
 }

While compiling, I get the following error:
/tmp/ccJaGxCX.o: In function `nvinfer1::(anonymous namespace)::createInferBuilder(nvinfer1::ILogger&)':
rnxt.cpp:(.text+0x19): undefined reference to `createInferBuilder_INTERNAL'
/tmp/ccJaGxCX.o: In function `nvonnxparser::(anonymous namespace)::createParser(nvinfer1::INetworkDefinition&, nvinfer1::ILogger&)':
rnxt.cpp:(.text+0x43): undefined reference to `createNvOnnxParser_INTERNAL'
collect2: error: ld returned 1 exit status

I also get error related to <cuda_runtime_api.h> so I have added (pasted) those files from cuda's include directory (/usr/local/cuda-11.0/targets/x86_64-linux/include) to the /usr/include directoryafter which I am getting the said error. I don't have much experience with C++ and any help would be appreciated.
Edit: I have also installed libnvinfer using
!apt-get install -y libnvinfer7=7.1.3-1+cuda11.0
!apt-get install -y libnvinfer-dev=7.1.3-1+cuda11.0


title: Anaconda / Pytorch - Getting error when trying to get Pytorch working

body:
I've tried the installing Pytorch with Anaconda on a Windows 10 system but have gotten a variety of errors back with each attempt. I downloaded a windows compatible tar file of the package from https://anaconda.org/peterjc123/pytorch/files and was apparently able to get the the package installed using conda install filename.tar.bz2. By apparently, I mean that the Pytorch shows up in the list of packages available in the environment I set up for using Pytorch. But when I go to load the package I get this error:

>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\conner\Anaconda3\envs\pytorch\lib\site-packages\torch\__init__.py", line 76, in <module>
    from torch._C import *
ImportError: DLL load failed: The specified module could not be found.

I also tried updating Anaconda with conda update --all which raised a relevant warning:

(pytorch) C:\Users\User 1\Downloads>conda update --all
Fetching package metadata .............
Solving package specifications:
Warning: ['Dependency missing in current win-64 channels: \n  - pytorch -> mkl >=2018'], skipping
Fetching package metadata .............
Solving package specifications:
Warning: ['Dependency missing in current win-64 channels: \n  - pytorch -> mkl >=2018'], skipping

NoPackagesFoundError: Dependency missing in current win-64 channels:
  - pytorch -> mkl >=2018

The mkl package also shows up in the list of packages connected to the pytorch environment. I interpret the error to mean that there isn't a recent enough version of mkl. Would that be correct? Any other insights or advice? I really need to get pytorch installed.


title: forcing pytorch to use gpu

body:
i've recently followed a tutorial here https://www.provideocoalition.com/automatic-rotoscopingfor-free/

And ended with a functional bit of code that generate masks outlining interestings objects.

But now, i want ot run it on my gpu, since cpu is way too slow.

I have CUDA installed and all, but pytorch refuses to use it. I've used most tricks like setting torch.device and all, but to no avail; pytorch keep using 0 gpu.

here's the code :

from PIL import Image
import torch
import torchvision.transforms as T
from torchvision import models
import numpy as np

fcn = None

device = torch.device('cuda')
torch.cuda.set_device(0)
print('Using device:', device)
print()

if device.type == 'cuda':
    print(torch.cuda.get_device_name(0))
    print('Memory Usage:')
    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')
    print('Cached:', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')

def getRotoModel():
    global fcn
    #fcn = models.segmentation.fcn_resnet101(pretrained=True).eval()
    fcn = models.segmentation.deeplabv3_resnet101(pretrained=True).eval()

# Define the helper function
def decode_segmap(image, nc=21):

    label_colors = np.array([(0, 0, 0),  # 0=background
                           # 1=aeroplane, 2=bicycle, 3=bird, 4=boat, 5=bottle
               (128, 0, 0), (0, 128, 0), (128, 128, 0), (0, 0, 128), (128, 0, 128),
               # 6=bus, 7=car, 8=cat, 9=chair, 10=cow
               (0, 128, 128), (128, 128, 128), (64, 0, 0), (192, 0, 0), (64, 128, 0),
               # 11=dining table, 12=dog, 13=horse, 14=motorbike, 15=person
               (192, 128, 0), (64, 0, 128), (192, 0, 128), (64, 128, 128), (192, 128, 128),
               # 16=potted plant, 17=sheep, 18=sofa, 19=train, 20=tv/monitor
               (0, 64, 0), (128, 64, 0), (0, 192, 0), (128, 192, 0), (0, 64, 128)])

    r = np.zeros_like(image).astype(np.uint8)
    g = np.zeros_like(image).astype(np.uint8)
    b = np.zeros_like(image).astype(np.uint8)

    for l in range(0, nc):
        idx = image == l
        r[idx] = label_colors[l, 0]
        g[idx] = label_colors[l, 1]
        b[idx] = label_colors[l, 2]

    rgb = np.stack([r, g, b], axis=2)
    return rgb

def createMatte(filename, matteName, size):
    img = Image.open(filename)
    trf = T.Compose([T.Resize(size),
                     T.ToTensor(),
                     T.Normalize(mean = [0.485, 0.456, 0.406],
                                 std = [0.229, 0.224, 0.225])])
    inp = trf(img).unsqueeze(0)
    if (fcn == None): getRotoModel()
    out = fcn(inp)['out']
    om = torch.argmax(out.squeeze(), dim=0).detach().cpu().numpy()
    rgb = decode_segmap(om)
    im = Image.fromarray(rgb)
    im.save(matteName)

What could i do ? thanks.


title: Basic multi GPU parallelization of matrix multiplication

body:
I want to parallelize the simple following expression on 2 GPUs: C =  A^n + B^n by calculating A^n on GPU 0 and B^n on GPU 1 before summing the results.

In TensorFlow I would go like:

with tf.device('/gpu:0'):
    An = matpow(A, n)
with tf.device('/gpu:1'):
    Bn = matpow(B, n)
with tf.Session() as sess:
    C = sess.run(An + Bn)

However, since PyTorch is dynamic, I'm having trouble doing the same thing. I tried the following but it only takes more time.

with torch.cuda.device(0):
    A = A.cuda()
with torch.cuda.device(1):
    B = B.cuda()
C = matpow(A, n) + matpow(B, n).cuda(0)

I know there is a module to parallelize models on the batch dimension using torch.nn.DataParallel but here I try to do something more basic.


title: BatchNorm1d needs 2d input?

body:
I want to fix problem in PyTorch.
I wrote the following code that is learning sine functions as tutorial.

import torch
from torch import nn
from torch import optim
from torch.autograd import Variable as V
from torch.utils.data import TensorDataset, DataLoader
import numpy as np

# y=sin(x1)
numTrain = 512
numTest = 128
noiseScale = 0.01
PI2 = 3.1415 * 2
X_train = np.random.rand(numTrain,1) * PI2
y_train = np.sin(X_train) + np.random.randn(numTrain,1) * noiseScale + 1.5
X_test  = np.random.rand(numTest,1) * PI2
y_test  = np.sin(X_test) + np.random.randn(numTest,1) * noiseScale

# Construct DataSet
X_trainT = torch.Tensor(X_train)
y_trainT = torch.Tensor(y_train)
X_testT = torch.Tensor(X_test)
y_testT = torch.Tensor(y_test)
ds_train = TensorDataset(X_trainT, y_trainT)
ds_test = TensorDataset(X_testT, y_testT)

# Construct DataLoader
loader_train = DataLoader(ds_train, batch_size=64, shuffle=True)
loader_test = DataLoader(ds_test, batch_size=64, shuffle=False)

# Construct network
net = nn.Sequential(
    nn.Linear(1,10),
    nn.ReLU(),
    nn.BatchNorm1d(10),
    nn.Linear(10,5),
    nn.ReLU(),
    nn.BatchNorm1d(5),
    nn.Linear(5,1),
)
optimizer = optim.Adam(net.parameters())
loss_fn = nn.SmoothL1Loss()

# Training
losses = []
net.train()
for epoc in range(100):
    for data, target in loader_train:
        y_pred = net(data)
        loss = loss_fn(target,y_pred)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        losses.append(loss.data)

# evaluation
%matplotlib inline
from matplotlib import pyplot as plt

#plt.plot(losses)
plt.scatter(X_train, y_train)

net.eval()
sinsX = []
sinsY = []
for t in range(128):
    x = t/128 * PI2
    output = net(V(torch.Tensor([x])))
    sinsX.append(x)
    sinsY.append(output.detach().numpy())
plt.scatter(sinsX,sinsY)

Training is done without error, But the next line caused an error, "expected 2D or 3D input (got 1D input)"

output = net(V(torch.Tensor([x])))

This error doesn't occur if it is without BatchNorm1d().
I feel strange because the input is 1D.

How to fix it?

Thanks.

Update: How did I fix

arr = np.array([x])
output = net(V(torch.Tensor(arr[None,...])))


title: Cosine Similarity on large matrix

body:
I am trying to compute cosine distance between all pairs of a large matrix (3m x 2048) and extract the top30 similar vectors using pytorch.
the following is my code which works fine but it take about 30 sec for each iteration which is too long for 3 million word vectors.
Any idea to speed it up ?

import torch.nn.functional as F
import torch
from tqdm import tqdm
import gc

sym_dict={}
tmp_list=[]

tot_dict=torch.load('xbx.pt')

all_tensors = torch.cat([v.unsqueeze(0) for k,v in tot_dict.items()], dim=0)
token_list= [i for i in tot_dict.keys()]

del tot_dict
gc.collect()

for counter ,value in tqdm(enumerate(token_list)):

  uniq_vec=torch.unsqueeze(all_tensors[counter],dim=0)

  dist = 1 - F.cosine_similarity(uniq_vec,all_tensors)
  index_sorted = torch.argsort(dist)

  roll_me=index_sorted[:30].cpu().numpy().tolist()

  for ind in roll_me:
    tmp_list.append(token_list[ind])
  sym_dict.update({value:tmp_list})
  tmp_list=[]

#save .pt file
torch.save(sym_dict,'sym_dict.pt')


title: How to implement LSTM layer with multiple cells in Pytorch?

body:
I intend to implement an LSTM with 2 layers and 256 cells in each layer. I am trying to understand the PyTorch LSTM framework for the same. The variables in torch.nn.LSTM that I can edit are input_size, hidden_size, num_layers, bias, batch_first, dropout and bidirectional.

However, how do I have multiple cells in a single layer?


title: pytorch: RuntimeError: bool value of Tensor with more than one value is ambiguous

body:
it works with

x[x >= 0.2] = 1
x[x < 0.2] = 0

x is a tensor here.

but when i am trying to use

x[x > 0 and x < 1] = 1

it reports: RuntimeError: bool value of Tensor with more than one value is ambiguous ?

dose anyone know why?


title: Error in training opennmt - caffe2_detectron_ops.dll not found

body:
I have torch 1.6 and python 3.8. When training OpenNMT, it throws the following error -
OSError: [WinError 126] The specified module could not be found. Error loading "C:\Users\Girish\AppData\Local\Programs\Python\Python38\lib\sitepackages\torch\lib\caffe2_detectron_ops.dll" or one of its dependencies.
I checked the folder, the file is present there. I have tried uninstalling torch and reinstalling it, but no help.
Any help will be appreciated. thanks


title: Saving Model State and Load in Google Colab

body: I have 500 epochs in total to train . But it is taking 8 minutes per epoch to be completed in google colab. Can any one help me how can I save my Model state after a Particular number of epoch  completion and start the training again from where I left in google Colab ??


title: Variation between custom convolution vs pytorch conv2d results?

body:
I am trying to build a custom convolution using the method shown in pytorch unfold function

The custom convolution function is given below:

import torch
from torch import nn
import torch.nn.functional as F
from torch.nn.parameter import Parameter
import math
from torch.nn.modules.utils import _pair

class customConv(nn.Module):
    def __init__(self, n_channels, out_channels, kernel_size, dilation=1, padding=0, stride=1, bias=True):
        super(customConv, self).__init__()

        self.kernel_size = _pair(kernel_size)
        self.out_channels = out_channels
        self.dilation = _pair(dilation)
        self.padding = _pair(padding)
        self.stride = _pair(stride)
        self.n_channels = n_channels
        self.weight = Parameter(torch.Tensor(self.out_channels, self.n_channels, self.kernel_size[0], self.kernel_size[1]))
        if bias:
            self.bias = Parameter(torch.Tensor(out_channels))
        else:
            self.register_parameter('bias', None)
        self.reset_parameters()

    def reset_parameters(self):
        n = self.n_channels
        for k in self.kernel_size:
            n *= k
        stdv = 1. / math.sqrt(n)
        self.weight.data.uniform_(-stdv, stdv)
        if self.bias is not None:
            self.bias.data.uniform_(-stdv, stdv)

    def forward(self, input_):

        hout = ((input_.shape[2] + 2 * self.padding[0] - self.dilation[0] * (self.kernel_size[0]-1)-1)//self.stride[0])+1
        wout = ((input_.shape[3] + 2 * self.padding[1] - self.dilation[1] * (self.kernel_size[1]-1)-1)//self.stride[1])+1

        inputUnfolded = F.unfold(input_, kernel_size=self.kernel_size, padding=self.padding, dilation=self.dilation, stride=self.stride)
        if self.bias:
            convolvedOutput = (inputUnfolded.transpose(1, 2).matmul(
            self.weight.view(self.weight.size(0), -1).t()).transpose(1, 2)) + self.bias.view(-1, 1)
        else:
            convolvedOutput = (inputUnfolded.transpose(1, 2).matmul(self.weight.view(self.weight.size(0), -1).t()).transpose(1, 2))
        convolutionReconstruction = convolvedOutput.view(input_.shape[0], self.out_channels, hout, wout)
        return convolutionReconstruction

But when I try comparing it with the pytorch implementation, I do not get the exact value. The code to check for difference is provided below

import torch
from torch import nn
from customConvolve import customConv
torch.manual_seed(1)

input = torch.randn (10,3,64,64)

conv1 = nn.Conv2d(input.shape[1],5, kernel_size=3, dilation=1, padding=1, stride=1 ,bias = False)
conv1_output = conv1(input)

conv2 = customConv(n_channels=input.shape[1], out_channels=5, kernel_size=3,  dilation=1, stride =1, padding = 1, bias = False)
conv2_output = conv2(input)

print(torch.equal(conv1.weight.data, conv2.weight.data))

print(torch.equal(conv1_output, conv2_output))

I would like to know why the variation exists and how to solve this?
Thank you.


title: PyTorch: What is the difference between tensor.cuda() and tensor.to(torch.device("cuda:0"))?

body:
In PyTorch, what is the difference between the following two methods in sending a tensor (or model) to GPU:
Setup:
X = np.array([[1, 3, 2, 3], [2, 3, 5, 6], [1, 2, 3, 4]]) # X = model()
X = torch.DoubleTensor(X)

Method 1
Method 2

X.cuda()
device = torch.device("cuda:0")X = X.to(device)

(I don't really need a detailed explanation of what is happening in the backend, just want to know if they are both essentially doing the same thing)


title: PyTorch - loading images without sub folders

body:
First of all I wanted to say that I am new to PyTorch, so i apologize in advice if the level of my questions is not that high.
I was wondering if you can help me with something (I actually have 2 questions). The story behind them: I am working on image classification. My test data is divided into sub folders based on their labels and I am loading them via DataLoader.

First question:
1) Is it true that if you have trained your model with specific batch sized, testing it with other sizes will affect the accuracy?
2) Is there a way to load and use the model with test data located in a single folder (without sub folders). DataLoader needs sub folders as far as i know.

Thank you in advance!


title: How to initialize the weights of different layers of nn.Sequential block in different styles in pytorch?

body:
Let's suppose I have a nn.Sequential block, it has 2 linear layers. I want to initialize the weights of first layer by uniform distribution but want to initialize the weights of second layer as constant 2.0.

net = nn.Sequential()
net.add_module('Linear_1', nn.Linear(2, 5, bias = False))
net.add_module('Linear_2', nn.Linear(5, 5, bias = False)


title: Custom backward/optimization steps in pytorch-lightning

body:
I would like to implement the training loop below in pytorch-lightning (to be read as pseudo-code). The peculiarity is that the backward and optimization steps are not performed for every batch.

(Background: I am trying to implement a few-shots learning algorithm; although I need to make predictions at every step -- forward method
-- I need to perform the gradient updates at random -- if- block.

for batch in batches:
    x, y = batch
    loss = forward(x,y)

    optimizer.zero_grad()

    if np.random.rand() > 0.5:
        loss.backward()
        optimizer.step()

My proposed solution entails implementing the backward and the optimizer_step methods as follows:

def backward(self, use_amp, loss, optimizer):
        self.compute_grads = False
        if np.random.rand() > 0.5:
            loss.backward()
            nn.utils.clip_grad_value_(self.enc.parameters(), 1)
            nn.utils.clip_grad_value_(self.dec.parameters(), 1)
            self.compute_grads = True
        return

    def optimizer_step(self, current_epoch, batch_nb, optimizer, optimizer_i, second_order_closure=None):
        if self.compute_grads:
            optimizer.step()
            optimizer.zero_grad()
        return

Note: In this way I need to store a compute_grads attribute at the class level.

What is the "best-practice" way to implement it in pytorch-lightning? Is there a better way to use the hooks?


title: How to sort the pytorch tensors by specific key value?

body:
I'm new to Pytorch. Given a tensor set, I need to sort these tensors by the key value.
For example,

A =
[[0.9133, 0.5071, 0.6222, 3.],
 [0.5951, 0.9315, 0.6548, 1.],
 [0.7704, 0.0720, 0.0330, 2.]]

My expected result after sorting is:

A' =
[[0.5951, 0.9315, 0.6548, 1.],
 [0.7704, 0.0720, 0.0330, 2.],
 [0.9133, 0.5071, 0.6222, 3.]]

I tried to use sorted function in python, but it was time-consuming in my training process.
How to achieve it more efficiently?
Thanks!


title: IndexError: Target 60972032 is out of bounds

body:
I am trying to call cross entropy loss but it says the index is out of range
loss = nn.CrossEntropyLoss()
target = torch.empty(128, dtype=torch.long)
result = loss(output, target)

Note the output has the shape torch.Size([128, 10])


title: RuntimeError: Given groups=3, weight of size 12 64 3 768, expected input[32, 12, 30, 768] to have 192 channels, but got 12 channels instead

body:
I started working with Pytorch recently so my understanding of it isn't quite strong. I previously had a 1 layer CNN but wanted to extend it to 2 layers, but the input and output channels have been throwing errors I can seem to decipher. Why does it expect 192 channels? Can someone give me a pointer to help me understand this better? I have seen several related problems on here, but I don't understand those solutions either.

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from transformers import BertConfig, BertModel, BertTokenizer
import math
from transformers import AdamW, get_linear_schedule_with_warmup

def pad_sents(sents, pad_token):  # Pad list of sentences according to the longest sentence in the batch.
    sents_padded = []
    max_len = max(len(s) for s in sents)
    for s in sents:
        padded = [pad_token] * max_len
        padded[:len(s)] = s
        sents_padded.append(padded)
    return sents_padded

def sents_to_tensor(tokenizer, sents, device):
    tokens_list = [tokenizer.tokenize(str(sent)) for sent in sents]
    sents_lengths = [len(tokens) for tokens in tokens_list]
    tokens_list_padded = pad_sents(tokens_list, '[PAD]')
    sents_lengths = torch.tensor(sents_lengths, device=device)
    masks = []
    for tokens in tokens_list_padded:
        mask = [0 if token == '[PAD]' else 1 for token in tokens]
        masks.append(mask)
    masks_tensor = torch.tensor(masks, dtype=torch.long, device=device)
    tokens_id_list = [tokenizer.convert_tokens_to_ids(tokens) for tokens in tokens_list_padded]
    sents_tensor = torch.tensor(tokens_id_list, dtype=torch.long, device=device)

    return sents_tensor, masks_tensor, sents_lengths

class ConvModel(nn.Module):

    def __init__(self, device, dropout_rate, n_class, out_channel=16):
        super(ConvModel, self).__init__()
        self.bert_config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True)
        self.dropout_rate = dropout_rate
        self.n_class = n_class
        self.out_channel = out_channel
        self.bert = BertModel.from_pretrained('bert-base-uncased', config=self.bert_config)
        self.out_channels = self.bert.config.num_hidden_layers * self.out_channel
        self.tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', config=self.bert_config)
        self.conv = nn.Conv2d(in_channels=self.bert.config.num_hidden_layers,
                              out_channels=self.out_channels,
                              kernel_size=(3, self.bert.config.hidden_size),
                              groups=self.bert.config.num_hidden_layers)
        self.conv1 = nn.Conv2d(in_channels=self.out_channels,
                               out_channels=48,
                               kernel_size=(3, self.bert.config.hidden_size),
                               groups=self.bert.config.num_hidden_layers)
        self.hidden_to_softmax = nn.Linear(self.out_channels, self.n_class, bias=True)
        self.dropout = nn.Dropout(p=self.dropout_rate)
        self.device = device

    def forward(self, sents):
        sents_tensor, masks_tensor, sents_lengths = sents_to_tensor(self.tokenizer, sents, self.device)
        encoded_layers = self.bert(input_ids=sents_tensor, attention_mask=masks_tensor)
        hidden_encoded_layer = encoded_layers[2]
        hidden_encoded_layer = hidden_encoded_layer[0]
        hidden_encoded_layer = torch.unsqueeze(hidden_encoded_layer, dim=1)
        hidden_encoded_layer = hidden_encoded_layer.repeat(1, 12, 1, 1)
        conv_out = self.conv(hidden_encoded_layer)  # (batch_size, channel_out, some_length, 1)
        conv_out = self.conv1(conv_out)
        conv_out = torch.squeeze(conv_out, dim=3)  # (batch_size, channel_out, some_length)
        conv_out, _ = torch.max(conv_out, dim=2)  # (batch_size, channel_out)
        pre_softmax = self.hidden_to_softmax(conv_out)

        return pre_softmax

def batch_iter(data, batch_size, shuffle=False, bert=None):
    batch_num = math.ceil(data.shape[0] / batch_size)
    index_array = list(range(data.shape[0]))
    if shuffle:
        data = data.sample(frac=1)
    for i in range(batch_num):
        indices = index_array[i * batch_size: (i + 1) * batch_size]
        examples = data.iloc[indices]
        sents = list(examples.train_BERT_tweet)
        targets = list(examples.train_label.values)
        yield sents, targets  # list[list[str]] if not bert else list[str], list[int]

def train():
    label_name = ['Yes', 'Maybe', 'No']
    device = torch.device("cpu")

    df_train = pd.read_csv('trainn.csv')  # , index_col=0)
    train_label = dict(df_train.train_label.value_counts())
    label_max = float(max(train_label.values()))
    train_label_weight = torch.tensor([label_max / train_label[i] for i in range(len(train_label))], device=device)
    model = ConvModel(device=device, dropout_rate=0.2, n_class=len(label_name))
    optimizer = AdamW(model.parameters(), lr=1e-3, correct_bias=False)
    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps=100, num_training_steps=1000)  # changed the last 2 arguments to old ones
    model = model.to(device)
    model.train()
    cn_loss = torch.nn.CrossEntropyLoss(weight=train_label_weight, reduction='mean')
    train_batch_size = 16

    for epoch in range(1):

        for sents, targets in batch_iter(df_train, batch_size=train_batch_size, shuffle=True):  # for each epoch
            optimizer.zero_grad()
            pre_softmax = model(sents)
            loss = cn_loss(pre_softmax, torch.tensor(targets, dtype=torch.long, device=device))
            loss.backward()
            optimizer.step()
            scheduler.step()
TrainingModel = train()

Here's a snippet of data https://github.com/Kosisochi/DataSnippet


title: How to resize a batch of images for use with Pytorch Linear Regression?

body:
I am trying to create a simple linear regression neural net for use with batches of images. The input dimensions are [BatchSize, 3, Width, Height]  with the second dimension representing the RGB channels of the input image.
Here is my (broken) attempt at that regression model:
class LinearNet(torch.nn.Module):
    def __init__(self, Chn, W,H, nHidden):
        """
        Input: A [BatchSize x Channels x Width x Height] set of images
        Output: A fitted regression model with weights dimension : [Width x Height]
        """
        super(LinearNet, self).__init__()
        self.Chn = Chn
        self.W = W
        self.H = H
        self.hidden = torch.nn.Linear(Chn*W*H,nHidden)   # hidden layer
        self.predict = torch.nn.Linear(nHidden, Chn*W*H)   # output layer

    def forward(self, x):
        torch.reshape(x, (-1,self.Chn*self.W*self.H)) # FAILS here
        # x = x.resize(-1,self.Chn*self.W*self.H)
        x = F.relu(self.hidden(x))      # activation function for hidden layer
        x = self.predict(x)             # linear output
        x = x.resize(-1,self.Chn, self.W,self.H)
        return x

When sending in a batch of images with dimensions [128 x 3 x 96 x 128]  this fails on the indicated line:
RuntimeError: mat1 and mat2 shapes cannot be multiplied (36864x128 and 36864x256)

How should the matrix dimensions be properly manipulated to use these pytorch functions?
Update Based on a (since deleted) comment I have updated the code to use torch.reshape.


title: type mismatch in pytorch forward method

body:
I am trying to augment the MNIST found here by loading my own data.
To do so I have numpy arrays training_x, training_y which I transform as follows:

training_X = torch.from_numpy(training_X)
training_y = torch.from_numpy(training_y)

[Note that the original training_y is the output of scikit-learn's LabelEncoder]

and add to the DataLoader by doing:

torch.utils.data.TensorDataset(training_X, training_y)

when training I receive the following error:

TypeError: addmm_ received an invalid combination of arguments - got (int, int, torch.cuda.FloatTensor, torch.FloatTensor), but expected one of:
* (torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
* (torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
* (float beta, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
* (float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
* (float beta, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
* (float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
* (float beta, float alpha, torch.cuda.FloatTensor mat1, torch.cuda.FloatTensor mat2)
  didn't match because some of the arguments have invalid types: (int, int, torch.cuda.FloatTensor, !torch.FloatTensor!)
v * (float beta, float alpha, torch.cuda.sparse.FloatTensor mat1, torch.cuda.FloatTensor mat2)
  didn't match because some of the arguments have invalid types: (int, int, !torch.cuda.FloatTensor!, !torch.FloatTensor!)

I have tried changing the input tensors to float, double and long but it seems I am still missing something crucial.

How can I get the model to accept my input?


title: Plotting predicted values causes error: 'Tensor' object has no attribute 'ndim'

body:
How do I properly write the code to plot the predicted values in this linear regression model?

I am using this tutorial to learn Linear regression:https://www.deeplearningwizard.com/deep_learning/practical_pytorch/pytorch_linear_regression/

I was able to successfully implement GPU. My issue is plotting the predicted values. I tried searching for solutions to learn how to keep the values as a tensor, but it seems that I don't have the syntactical knowledge to do this.

Here is where I'll start

epochs = 100
for epoch in range(epochs):
    epoch += 1

    # Convert numpy array to torch Variable
    if torch.cuda.is_available():
        inputs = (torch.from_numpy(x_train).cuda())
        labels = (torch.from_numpy(y_train).cuda())
    else:
        inputs = (torch.from_numpy(x_train))
        labels = (torch.from_numpy(y_train))

    # Clear gradients w.r.t. parameters
    optimizer.zero_grad()

    # Forward to get output
    outputs = model(inputs)

    # Calculate Loss
    loss = criterion(outputs, labels)

    # Getting gradients w.r.t. parameters
    loss.backward()

    # Updating parameters
    optimizer.step()

    # Logging
    print('epoch {}, loss {}'.format(epoch, loss.item()))

Prediction is made here and I choose to use cuda

predicted = model(Variable(torch.from_numpy(x_train).requires_grad_().cuda()))

print("Predicted")
print(predicted)
print("Output")
print(y_train)

plt.clf()

# Get predictions

#predicted = model(Variable(torch.from_numpy(x_train).requires_grad_().cuda()))

# Plot true data
plt.plot(x_train, y_train, 'go', label='True data', alpha=0.5)

The error is called here after it is unable to plot

# Plot predictions
plt.plot(x_train, predicted, '--', label='Predictions', alpha=0.5)

# Legend and plot
plt.legend(loc='best')
plt.show()

Error Given:

Traceback (most recent call last):
  File "D:/Test with GPU/Linear regression.py", line 101, in <module>
    plt.plot(x_train, predicted, '--', label='Predictions', alpha=0.5)
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\pyplot.py", line 2795, in plot
    is not None else {}), **kwargs)
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\axes\_axes.py", line 1666, in plot
    lines = [*self._get_lines(*args, data=data, **kwargs)]
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\axes\_base.py", line 225, in __call__
    yield from self._plot_args(this, kwargs)
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\axes\_base.py", line 391, in _plot_args
    x, y = self._xy_from_xy(x, y)
  File "D:\Anaconda3\envs\gputest\lib\site-packages\matplotlib\axes\_base.py", line 271, in _xy_from_xy
    if x.ndim > 2 or y.ndim > 2:
AttributeError: 'Tensor' object has no attribute 'ndim'


title: Usage of data.to(device) with cuda GPUs

body:
I'm trying to implement a neural network to run across 8 GPUs and i just want clarification on what exactly these commands do

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)
data.to(device)

Will this automatically spread the training across the 8 GPUs ?
Thanks!


title: Why do TensorFlow and PyTorch gradients of the eigenvalue decomposition differ from each other and the analytic solution?

body:
The following code computes the eigenvalue decomposition of a real symmetric matrix. Then, the gradient of the first eigenvalue with respect to the matrix is computed. This is done three times: 1) using the analytic formula, 2) using TensorFlow, 3) using PyTorch. This yields three different results. Can someone explain this behavior to me?

import numpy as np
import torch
import tensorflow as tf

np.set_printoptions(precision=3)
np.random.seed(123)

# random matrix
matrix_np = np.random.randn(4, 4)
# make symmetric
matrix_np = matrix_np + matrix_np.T
matrix_torch = torch.autograd.Variable(torch.from_numpy(matrix_np), requires_grad=True)
matrix_tf = tf.constant(matrix_np, dtype=tf.float64)

#
# compute eigenvalue decompositions
#
# NumPy
eigvals_np, eigvecs_np = np.linalg.eigh(matrix_np)
# PyTorch
eigvals_torch, eigvecs_torch = torch.symeig(matrix_torch, eigenvectors=True, upper=True)
# TensorFlow
eigvals_tf, eigvecs_tf = tf.linalg.eigh(matrix_tf)

# make sure all three versions computed the same eigenvalues
if not np.allclose(eigvals_np, eigvals_torch.data.numpy()):
    print('NumPy and PyTorch have different eigenvalues')
if not np.allclose(eigvals_np, tf.keras.backend.eval(eigvals_tf)):
    print('NumPy and TensorFlow have different eigenvalues')

#
# compute derivative of first eigenvalue with respect to the matrix
#
# analytic gradient, see "On differentiating eigenvalues and eigenvectors" by Jan R. Magnus
grad_analytic = np.outer(eigvecs_np[:, 0], eigvecs_np[:, 0])
# PyTorch gradient
eigvals_torch[0].backward()
grad_torch = matrix_torch.grad.numpy()
# TensorFlow gradient
grad_tf = tf.gradients(eigvals_tf[0], matrix_tf)[0]
grad_tf = tf.keras.backend.eval(grad_tf)

#
# print all derivatives
#
print('-'*6, 'analytic gradient', '-'*6)
print(grad_analytic)
print('-'*6, 'Pytorch gradient', '-'*6)
print(grad_torch)
print('-'*6, 'TensorFlow gradient', '-'*6)
print(grad_tf)

Prints

------ analytic gradient ------
[[ 0.312 -0.204 -0.398 -0.12 ]
 [-0.204  0.133  0.26   0.079]
 [-0.398  0.26   0.509  0.154]
 [-0.12   0.079  0.154  0.046]]
------ Pytorch gradient ------
[[ 0.312 -0.407 -0.797 -0.241]
 [ 0.     0.133  0.52   0.157]
 [ 0.     0.     0.509  0.308]
 [ 0.     0.     0.     0.046]]
------ TensorFlow gradient ------
[[ 0.312  0.     0.     0.   ]
 [-0.407  0.133  0.     0.   ]
 [-0.797  0.52   0.509  0.   ]
 [-0.241  0.157  0.308  0.046]]

The main diagonals of the three results are identical. The off-diagonal elements of TensorFlow and PyTorch are twice as large as the analytic elements or equal to zero.

Is this intended behavior? Why is it not documented? Are the gradients wrong?

Version infos: TensorFlow 1.14.0, PyTorch 1.0.1


title: Pytorch not using cuda device

body:
I have the following code:

from __future__ import print_function
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import scipy.io

folder = 'small/'
mat = scipy.io.loadmat(folder+'INISTATE.mat');
ini_state = np.float32(mat['ini_state']);
ini_state = torch.from_numpy(ini_state);
ini_state = ini_state.cuda();

mat = scipy.io.loadmat(folder+'TARGET.mat');
target = np.float32(mat['target']);
target = torch.from_numpy(target);
target = target.cuda();

class MLPNet(nn.Module):
    def __init__(self):
        super(MLPNet, self).__init__()
        self.fc1 = nn.Linear(3, 64)
        self.fc2 = nn.Linear(64, 128)
        self.fc3 = nn.Linear(128, 128)
        self.fc4 = nn.Linear(128, 41)
    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = self.fc4(x)
        return x

    def name(self):
        return "MLP"

model = MLPNet();
model = model.cuda();

criterion = nn.MSELoss();
criterion = criterion.cuda();
learning_rate = 0.001;
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

batch_size = 20
iter_size = int(target.size(0)/batch_size)
print(iter_size)

for epoch in range(50):
    for i in range(iter_size):
        start = i*batch_size;
        end = (i+1)*batch_size-1;
        samples = ini_state[start:end,:];
        labels = target[start:end,:];

        optimizer.zero_grad()  # zero the gradient buffer
        outputs = model(samples)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        if (i+1) % 500 == 0:
            print("Epoch %s, batch %s, loss %s" % (epoch, i, loss))
    if (epoch+1) % 7 == 0:
        for g in optimizer.param_groups:
            g['lr'] = g['lr']*0.1;

But when I train the simple MLP, the CPU usage is around 100% while the gpu is only around 10%. What is the problem that prevents using the GPU?


title: AttributeError: module 'tensorflow' has no attribute 'value'

body:
I am training pytorch-yolov3 in custom dataset. I prepared all the required txt, data and names files .

while runninng following command:

python3 train.py --model_def config/yolov3.cfg --data_config config/custom.data

I got following error:

Warning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead. (expandTensors at /pytorch/aten/src/ATen/native/IndexingUtils.h:20)
Warning: indexing with dtype torch.uint8 is now deprecated, please use a dtype torch.bool instead. (expandTensors at /pytorch/aten/src/ATen/native/IndexingUtils.h:20)
Traceback (most recent call last):
  File "train.py", line 136, in <module>
    logger.list_of_scalars_summary(tensorboard_log, batches_done)
  File "/home/sudip/torch/PyTorch-YOLOv3/utils/logger.py", line 16, in list_of_scalars_summary
    summary = tf.summary(value=[tf.summary.Value(tag=tag, simple_value=value) for tag, value in tag_value_pairs])
  File "/home/sudip/torch/PyTorch-YOLOv3/utils/logger.py", line 16, in <listcomp>
    summary = tf.summary(value=[tf.summary.Value(tag=tag, simple_value=value) for tag, value in tag_value_pairs])
AttributeError: module 'tensorboard.summary._tf.summary' has no attribute 'Value'

This is logger.py  file:

import tensorflow as tf
class Logger(object):
    def __init__(self, log_dir):
        self.writer = tf.summary.create_file_writer(log_dir)

    def scalar_summary(self, tag, value, step):
        """Log a scalar variable."""
        summary = tf.summary(value=[tf.summary.Value(tag=tag, simple_value=value)])
        self.writer.add_summary(summary, step)

    def list_of_scalars_summary(self, tag_value_pairs, step):
        """Log scalar variables."""
        summary = tf.summary(value=[tf.summary.Value(tag=tag, simple_value=value) for tag, value in tag_value_pairs])
        self.writer.add_summary(summary, step)

Any idea or suggestions to solve this problem?

Any help would be appreciated.

Thank you


title: Python Dataset Class + PyTorch Dataloader: Stuck at __getitem__, how to get Index, Label and so on during Testing?

body:
I have a, maybe small problem but I am stuck for quite a while now. Hope someone can help me with that. I am currently on a Kddcup99 dataset which I like to train via DeepLearning (CNN Network)

I have a "Dataset" Class which includes the Panda Dataframe. Thus i split up into normal and validate dataset. So far, no problem.
I load it into a Numpy vector, torch it to Tensor and then direct it to the DataLoader.

The Dataset Class has these two important classes for iterating through:

def __len__(self):
        return len(self.val_df)

def __getitem__(self, index):
        img, target = self.val_df[index][:-1], self.val_df[index][-1]
        return img, target, index

Not in the class is the DataLoader string:

test_dataloader = DataLoader(datat.val_df, batch_size=10, shuffle=True)

In my Trainer Class i have a for loop which should iterate through the Dataloader:

with torch.no_grad():
            for data in dataloader:
                inputs, labels, idx = data
                inputs = inputs.to(self.device)

But it won't. I can't access the labels, index and such.

My question is now: Why?
How can I access Labels, Index from the given Dataset via the Dataloader?

Thank you all for your help!
Much appreciate it.


title: Understanding the reason behind 'RuntimeError: leaf variable has been moved into the graph interior'

body:
I am trying to understand pytorch and how autograd works in it. I tried creating a tensor by filling it with values from other tensors and then checking the gradients. However, I am running into RuntimeError: leaf variable has been moved into the graph interior if I don't set requires_grad equal to False.
code:
x = torch.ones(3,5,requires_grad=True)

y = x+2

z = y*y*3

out1 = z.mean()
out2 = 2*z.mean()

outi = torch.empty(2,requires_grad=True)

outi[0] = out1
outi[1] = out2

outi.backward(torch.tensor([0.,1.]))

output:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-22-1000fc52a64c> in <module>
     13 outi[1] = out2
     14
---> 15 outi.backward(torch.tensor([0.,1.]))

~/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    183                 products. Defaults to ``False``.
    184         """
--> 185         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    186
    187     def register_hook(self, hook):

~/anaconda3/envs/pytorch/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    123         retain_graph = create_graph
    124
--> 125     Variable._execution_engine.run_backward(
    126         tensors, grad_tensors, retain_graph, create_graph,
    127         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: leaf variable has been moved into the graph interior

however, I can change the requires_grad to False and it will all work just fine
x = torch.ones(3,5,requires_grad=True)

y = x+2

z = y*y*3

out1 = z.mean()
out2 = 2*z.mean()

outi = torch.empty(2,requires_grad=False)

outi[0] = out1
outi[1] = out2

outi.backward(torch.tensor([0.,1.]))

output:
empty. it worked

Can somebody please help me understand what happened under the hood and, what changed with setting require_grad to True that resulted in this behavior? Thank you for reading


title: Pytorch tutorial loss is not decreasing as expected

body:
I am reimplementing the pytorch tutorial of the Pytorch cifar10 tutorial

But I want to use a different model.
I don't want to use fully connected (in pytorch linear) layers and I want to add Batch Normalization.

My model looks like this:

import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
  def __init__(self):
      super(Net, self).__init__()
      self.pool = nn.MaxPool2d(2,2)
      self.conv1 = nn.Conv2d(in_channels=3,out_channels=16,kernel_size=3, padding=1, padding_mode='zeros')
      self.conv1_bn = nn.BatchNorm2d(16)
      self.conv2 = nn.Conv2d(in_channels=16,out_channels=32,kernel_size=3, padding=1, padding_mode='zeros')
      self.conv2_bn = nn.BatchNorm2d(32)
      self.conv3 = nn.Conv2d(in_channels=32,out_channels=64,kernel_size=3, padding=1, padding_mode='zeros')
      self.conv3_bn = nn.BatchNorm2d(64)
      self.conv4 = nn.Conv2d(64,64,3, padding=1, padding_mode='zeros')
      self.conv4_bn = nn.BatchNorm2d(64)
      self.conv5 = nn.Conv2d(64,10,2,padding=0)
  def forward(self, x): # x has shape (4,32,32,3)
      x = self.pool(F.relu(self.conv1_bn(self.conv1(x)))) # feature map resolution is now 16*16
      x = self.pool(F.relu(self.conv2_bn(self.conv2(x)))) # resolution now 8*8
      x = self.pool(F.relu(self.conv3_bn(self.conv3(x)))) #resolution now 4*4
      x = self.pool(F.relu(self.conv4_bn(self.conv4(x)))) # now 2*2
      x = F.relu(self.conv5(x)) # The output shape is (batchsize, 1,1,10)

      return x

Batchsize is 4 and image resolution is 32*32 so inputsize is 4,32,32,3
The convolution layers don't reduce the resolution size of the feature maps because of the padding. The resolution is halved with the maxpool layers. Conv5 gets an input with shape 4,2,2,64.
Now I use filtersize 2 and no padding to get a resolution of 1*1.
I have 10 classes so I use 10 filters. Each of the last filters should predict it's corresponding class.
The shape of the output is now (4,1,1,10).
But when I try to train this model the loss doesn't decrease. The amount of parameters of the tutorial model and my net are about the same at ~62k.

Here is the rest of the code. This is identical to the code in the tutorial but I have to reshape the output so it fits. (output in the tutorial was (4,10) and mine is 4,1,1,10)

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])
trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

net = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(net.parameters(), lr=0.01, momentum=0.9)
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
net.to(device)
for epoch in range(2):  # loop over the dataset multiple times

  running_loss = 0.0
  for i, data in enumerate(trainloader, 0):
      # get the inputs; data is a list of [inputs, labels]
      inputs, labels = data[0].to(device), data[1].to(device)

      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = net(inputs) # I get the values as 4,1,1,10
      outputs_reshaped = outputs.reshape(4,10)
      loss = criterion(outputs_reshaped, labels)
      loss.backward()
      optimizer.step()
      running_loss +=loss.item()
      if i % 2000 == 1999:    # print every 2000 mini-batches
          print('[%d, %5d] loss: %.3f' %
                (epoch + 1, i + 1, running_loss / 2000))
          running_loss = 0.0

My loss looks like this.

[1,  2000] loss: 2.348
[1,  2000] loss: 2.477
[1,  4000] loss: 2.482
[1,  6000] loss: 2.468
[1,  8000] loss: 2.471
[1, 10000] loss: 2.482
[1, 12000] loss: 2.485
[2,  2000] loss: 2.486
[2,  4000] loss: 2.470
[2,  6000] loss: 2.479
[2,  8000] loss: 2.481
[2, 10000] loss: 2.474
[2, 12000] loss: 2.470

My model doesn't seem to learn anything. Anyone an idea why this might happen?


I try to get the outputs from several models, stack it together and feed into another classifier, but I get the following error:
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior.
Below is the forward method of my model:
def forward(self, x):
        mem_outpus = []
        for i in range(5):
            output = self.models[i](x)
            mem_outpus.append(output)

        ensemle_inputs = torch.stack(mem_outpus, dim=1).unsqueeze(dim=1)
        ensemle_inputs = (ensemle_inputs - self.mu)/self.std
        ensemle_outputs = self.classifier(ensemle_inputs)
        return ensemle_inputs


title: How to increase number of channels of a pytorch tensor?

body: I have a pytorch tensor of format [N, 2, H, W], where 2 is the number of channels. However, the model I am using (pretrained resnet18) requires me to have dimensions [N, 3, H, W]. How do I increase the number of channels from 2 to 3?


title: What is mean by (AttributeError: 'NoneType' object has no attribute '__array_interface__') error?

body:
I am trying to build a ML model to detect landmarks on a cartoon image face. When I split the image dataset in to training and validation sets I got the following error. Here I am using pytorch to build the model. So what is mean by this error?
This is how I split the dataset.
# split the dataset into validation and test sets
len_valid_set = int(0.2*len(dataset))
len_train_set = len(dataset) - len_valid_set

print("The length of Train set is {}".format(len_train_set))
print("The length of Valid set is {}".format(len_valid_set))

train_dataset , valid_dataset,  = torch.utils.data.random_split(dataset , [len_train_set, len_valid_set])

# shuffle and batch the datasets
train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=32, shuffle=True, num_workers=4)
valid_loader = torch.utils.data.DataLoader(valid_dataset, batch_size=8, shuffle=True, num_workers=4)

images, landmarks = next(iter(train_loader))

This is the error I got.
The length of Train set is 105
The length of Valid set is 26

AttributeError                            Traceback (most recent call last)
<ipython-input-61-ffb86a628e37> in <module>()
----> 1 images, landmarks = next(iter(train_loader))
      2
      3 print(images.shape)
      4 print(landmarks.shape)

3 frames
/usr/local/lib/python3.6/dist-packages/torch/_utils.py in reraise(self)
    426             # have message field
    427             raise self.exc_type(message=msg)
--> 428         raise self.exc_type(msg)
    429
    430

AttributeError: Caught AttributeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 198, in _worker_loop
    data = fetcher.fetch(index)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py", line 272, in __getitem__
    return self.dataset[self.indices[idx]]
  File "<ipython-input-12-5595ac89d75d>", line 38, in __getitem__
    image, landmarks = self.transform(image, landmarks, self.crops[index])
  File "<ipython-input-9-e38df55ee0d4>", line 46, in __call__
    image = Image.fromarray(image)
  File "/usr/local/lib/python3.6/dist-packages/PIL/Image.py", line 2670, in fromarray
    arr = obj.__array_interface__
AttributeError: 'NoneType' object has no attribute '__array_interface__'


title: How does one implement adversarial examples in pytorch?

body:
I wanted to reproduce:

from the paper https://arxiv.org/pdf/1312.6199.pdf. I was wondering, how does one actually implement this in pytorch? My main confusion is that for loss_f I am using a torch.nn.CrossEntropy() criterion for example. Do i just need to change the code that I already have from:

loss = criterion(outputs+r, labels)
loss.backward()

to:

loss = criterion(outputs+r, labels)
loss = loss + c * r.norm(2)
loss.backward()

or something along those lines (of course include r in the optimizer!). I know its not quite right cuz I did not explicitly show how I implemented x+r or the hypercube constraint but those are parts that I still need to figure out.

I think for the moment I want to focus first without the hypercube constraint. If we assume I am ok with going out of that is the above correct? I just want to know if:

loss = loss + c * r.norm(2)

works as it should.

Now if we do include the hypercube constraint how does my solution change? Is that were the "penalty function method" come into place?

https://discuss.pytorch.org/t/how-does-one-implement-adversarial-examples-in-pytorch/15668


title: What is the difference between parameters and children?

body:
It looks like parameters and children show the same info, so what is the difference between them?

import torch

print('torch.__version__', torch.__version__)

m = torch.load('imagenet_resnet18.pth')

print(m.parameters)
print(m.children)


title: Split dataset based on file names in pytorch Dataset

body:
Is there a way to divide the dataset into training and testing based on the filenames. I have a folder containing two folders: input and output. Input folder has the images and output are the labels for that image. The file names in the input folder are something like input01_train.png and input01_test.png like shown below.

                          Dataset
                          /     \
                     Input       Output
                      |             |
           input01_train.png   output01_train.png
                    .                 .
                    .                 .
           input01_test.png    output01_test.png

The code I have only divides the dataset into inputs and labels not test and train.

class CancerDataset(Dataset):
  def __init__(self, dataset_folder):#,label_folder):
    self.dataset_folder = torchvision.datasets.ImageFolder(dataset_folder ,transform = transforms.Compose([transforms.Resize(512),transforms.ToTensor()]))
    self.label_folder = torchvision.datasets.ImageFolder(dataset_folder ,transform = transforms.Compose([transforms.Resize(512),transforms.ToTensor()]))

  def __getitem__(self,index):
    img = self.dataset_folder[index]
    label = self.label_folder[index]
    return img,label

  def __len__(self):
    return len(self.dataset_folder)

trainset = CancerDataset(dataset_folder = '/content/drive/My Drive/cancer_data/')
trainsetloader = DataLoader(trainset,batch_size = 1, shuffle = True,num_workers = 0,pin_memory = True)

I would like to be able to divide the train and test set by their names if that is possible .


title: torch.cuda.is_availabe() continuously returns False

body:
I'm currently in a lab at the school I'm at and we use GPU servers for our tasks. I've tried running the torch.cuda.is_available() function but it continuously returns False despite me having tried this and that.

By "this and that" I specifically mean that I've tried installing and reinstalling PyTorch, I've ensured that CUDA is indeed available by running the nvidia-smi command, I've also checked the CUDA_VISIBLE_DEVICES environmental variable and made sure that it's correctly assigned.

The GPU devices are either 0 or 1, and my environmental variable is currently set to 1.

Currently the specs that I have are:

OS: Ubuntu 16.04

CUDA: 9.0176

Python: 3.7.0

cuDNN: 9.0-v7.0.5

Nvidia driver: 384.81

Has anyone experienced this issue? If so, how was it solved?

EDIT

I've currently re-installed PyTorch from the Github source as suggested PyTorch - Get Started - Building From Source, and it's fixed my issue. However, it would still be greatly helpful to me and the community in general if anyone could provide some insight onto why such problems occur in the first place. Thank you.

EDIT 2

As @kHarshit suggested in the comments, I ran collect_env.py located in pytorch/torch/utils/ and got the following output:

Collecting Environment information...
PyTorch version: 1.3.0a0+6a4ca9a
is debug build: No
CUDA used to build PyTorch: 9.0.176

OS: Ubuntu 16.04.3 LTS
GCC version: (Ubuntu 5.4.0-6ubuntu1~16.04.10) 5.4.0 20160609
CMake version: version 3.14.0

Python version: 3.7
Is CUDA available: No
CUDA runtime version: 9.0.176
GPU models and configuration:
GPU 0: TITAN Xp
GPU 1: TITAN Xp

Nvidia driver version: 384.81
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.17.2
[pip3] numpydoc=0.8.0
[pip3] torch=1.3.0a0+6a4ca9a
[pip3] torch-cluster=1.4.4
[pip3] torch-geometric=1.3.1
[pip3] torch-scatter==1.3.1
[pip3] torch-sparse==0.4.0
[pip3] torch-spline-conv==1.1.0
[pip3] torchvision==0.4.0
[conda] blas                    1.0                    mkl
[conda] mkl                     2019.4                 243
[conda] mkl-include             2019.4                 243
[conda] mkl-service             2.3.0       py37he904b0f_0
[conda] mkl_fft                 1.0.14      py37ha843d7b_0
[conda] mkl_random              1.1.0       py37hd6b4f25_0
[conda] torch                   1.2.0               pypi_0     pypi
[conda] torch-cluster           1.4.4               pypi_0     pypi
[conda] torch-geometric         1.3.1               pypi_0     pypi
[conda] torch-scatter           1.3.1               pypi_0     pypi
[conda] torch-sparse            0.4.0               pypi_0     pypi
[conda] torch-spline-conv       1.1.0               pypi_0     pypi
[conda] torchvision             0.4.0               pypi_0     pypi

I'm not too familiar with this and am not sure if there is any information indicative of a potential problem. If there is, would anyone be kind enough to point it out? Thanks.


title: Missing XLA configuration when running pytorch/xla

body:
I am trying to run GCP TPU with Pytorch/XLA, I am using a VM with debian-9-torch-xla-v20200818 image, I initiate the TPU and check it is running using ctpu status which shows that both the CPU and TPU are running, I then activate the torch-xla-nightly environment, but when I try to invoke this simple code:
import torch
import torch_xla
import torch_xla.core.xla_model as xm

dev = xm.xla_device()
t1 = torch.ones(3, 3, device = dev)
print(t1)

this error comes up:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 231, in xla_device
devkind=devkind if devkind is not None else None)
File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 136, in get_xla_supported_devices
 xla_devices = _DEVICES.value
File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/utils/utils.py", line 32, in value
self._value = self._gen_fn()
File "/anaconda3/envs/torch-xla-nightly/lib/python3.6/site-packages/torch_xla/core/xla_model.py", line 18, in <lambda>
_DEVICES = xu.LazyProperty(lambda: torch_xla._XLAC._xla_get_devices())
RuntimeError: tensorflow/compiler/xla/xla_client/computation_client.cc:274 : Missing XLA configuration

I tried everything but nothing seem to work.


title: Get the Cross Entropy Loss in pytorch as in Keras

body:
I am struggling to port a classification model form keras to pytorch. Especially the cross entropy loss seems to return totally different numbers.
import numpy as np
import torch as t
import torch.nn as nn
import tensorflow.keras.backend as K

y_true = np.array([[0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0]])
y_pred = np.array([[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 1, 0.41, 0.31, 0.21, 0.11]])

print("Keras", K.categorical_crossentropy(K.constant(y_true), K.constant(y_pred)))
print("PyTorch", nn.CrossEntropyLoss()(t.tensor(y_pred).argsort(dim=-1).float(), t.tensor(y_true).argmax(dim=-1)))```

prints:

Keras tf.Tensor([2.3369865], shape=(1,), dtype=float32)
PyTorch tensor(1.4587)

Since I have a custom loss function where cross entropy is a part of it, I would need to get similar if not the same numbers.


title: Filter out NaN values from a PyTorch N-Dimensional tensor

body:
This question is very similar to filtering np.nan values from pytorch in a -Dimensional tensor. The difference is that I want to apply the same concept to tensors of 2 or higher dimensions.
I have a tensor that looks like this:
import torch

tensor = torch.Tensor(
[[1, 1, 1, 1, 1],
 [float('nan'), float('nan'), float('nan'), float('nan'), float('nan')],
 [2, 2, 2, 2, 2]]
)

>>> tensor.shape
>>> [3, 5]

I would like to find the most pythonic / PyTorch way of to filter out (remove) the rows of the tensor which are nan. By filtering this tensor along the first (0th axis) I want to obtain a filtered_tensor which looks like this:
>>> print(filtered_tensor)
>>> torch.Tensor(
[[1, 1, 1, 1, 1],
 [2, 2, 2, 2, 2]]
)
>>> filtered_tensor.shape
>>> [2, 5]


title: PyTorch Design: Why does torch.distributions.multivariate_normal have methods outside of its class?

body:
I'm trying to understand the design of pytorch a little bit better. I was trying to draw samples from a multivariate normal, and found torch.distributions.multivariate_normal, which to my surprise is a module with many protected functions defined outside of its MultivariateNormal() class.

I was confused as to why this was the case. Why not just define all of these functions as class methods inside the MultivariateNormal() class? That way, we could instantiate an object of this class by

torch.distributions.multivariate_normal(mu,sigma)

rather than

torch.distributions.multivariate_normal.MultivariateNormal(mu,sigma).

Any thoughts?

Thanks.


title: Pytorch: Visualize model while training

body:
I am training a neural network by regression but it is predicting a constant value during testing. Which is why I want to visualize the weights of the neural network change during training and see the weights change dynamically in the jupyter notebook.
Currently, my model looks like this:

import torch
from torch import nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.inp = nn.Linear(2, 40)
        self.act1 = nn.Tanh()
        self.h1 = nn.Linear(40, 40)
        self.act2 = nn.Tanh()
        self.h2 = nn.Linear(40, 2)
        self.act3 = nn.Tanh()
        #self.h3 = nn.Linear(20, 20)
        #self.act4=nn.Tanh()
        self.h4 = nn.Linear(2, 1)

    def forward_one_pt(self, x):
        out = self.inp(x)
        out = self.act1(out)
        out = self.h1(out)
        out = self.act2(out)
        out = self.h2(out)
        out = self.act3(out)
        #out = self.h3(out)
        #out = self.act4(out)
        out = self.h4(out)
        return out

    def forward(self, config):
        E = torch.zeros([config.shape[0], 1])
        for i in range(config.shape[0]):
            E[i] = self.forward_one_pt(config[i])
            # print("config[",i,"] = ",config[i],"E[",i,"] = ",E[i])
        return torch.sum(E, 0)

and my main function looks like this:

def main()  :
    learning_rate = 0.5
    n_pts = 1000
    t_pts = 100
    epochs = 15

    coords,E = load_data(n_pts,t_pts)

    #generating my data to NN
    G = get_symm(coords,save,load_symmetry,symmtery_pickle_file,eeta1,eeta2,Rs,ex,lambdaa,zeta,boxl,Rc,pi,E,scale)
    net = Net()
    if(cuda_flag):
        net.cuda()
    criterion = nn.MSELoss()
    optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

    net_trained = train(save,text_output,epochs,n_pts,G,E,net,t_pts,optimizer,criterion,out,cuda_flag)
    test(save,n_pts,t_pts,G,E,net_trained,out,criterion,cuda_flag)

    torch.save(net,save_model)

any tutorials or answers would be helpful


title: How to revert back to PyTorch 1.4?

body:
I have had PyTorch 1.4 installed on my machine for a little while now, and just tried to install 1.5 using the directions found directly on PyTorch.org.

Unfortunately, it failed, as do most things I try to do exactly as told. No shocker there.

Now, I have some half-installation of PyTorch 1.5 that will not import, but insists that it's on my machine. It fails with a FileNotFoundError, referring specifically to caffe2_nvrtc.dll.

I found an issue on GitHub with this problem that was closed last month, but of course it still is an issue, and many other people have reported it since it was closed. There has been no real resolution.

I just want to go back. What is the best way of getting rid of 1.5 without screwing up my machine and its environments even worse? And then, where do I find v1.4? It is not listed on the "Previous PyTorch Versions" page.


title: Choosing loss function for lstm trained on word2vec vectors when target is a vector of same dimensions

body:
I have an lstm I'm using as a sequence-generator trained on word2vec vectors. The previous implementation produced a probability distribution for all the different labels. There was one label for every word in the vocabulary. This implementation used Pytorch's CrossEntropyLoss. I now want to change this so the lstm outputs a vector that has the same dimensions as the vectors used for training. This way I could use the euclydian distance measure to match wit to nearby vectors in the vocabulary. The problem is that in order to do this, I have to use a different loss function, because CrossEntropyLoss is appropriate for classifiers and not for regression problems.

I tried changing the format of the target vector, but torch's CrossEntropyLoss function requires integer input, and I have a word vector. Having looked at a few options it seems Cosine Embedding Loss might be a good idea but I don't understand how it works and what kind of input it takes.

I have already changed my fully connected layer to output vectors of the same dimensions as the Word Embeddings used for training:

nn.Linear(in_features=self.cfg.lstm.lstm_num_hidden,out_features=self.cfg.lstm.embedding_dim,bias=True)

Any advice and examples would be much appreciated.


title: How to cache Pytorch models for use when not connected to the internet?

body:
I'm using vgg19 in a classification problem. I have access to the campus research computer to train on, but the nodes where the computation is done don't have access to the internet. So running a line of code like self.net = models.vgg19(pretrained=True) fails with the error urllib.error.URLError: <urlopen error [Errno 101] Network is unreachable>

Is there a way I could cache the model on the head node (where I have internet access), and load the model from the cache instead of the internet on the compute node?


title: output prediction of pytorch lightning model

body:
This is potentially a very easy question. I just started with PyTorch lightning and can't figure out how to receive the output of my model after training.
I am interested in both predictions of y_train and y_test as an array of some sort (PyTorch tensor or NumPy array in a later step) to plot next to the labels using different scripts.
dataset = Dataset(train_tensor)
val_dataset = Dataset(val_tensor)
training_generator = torch.utils.data.DataLoader(dataset, **train_params)
val_generator = torch.utils.data.DataLoader(val_dataset, **val_params)
mynet = Net(feature_len)
trainer = pl.Trainer(gpus=0,max_epochs=max_epochs, logger=logger, progress_bar_refresh_rate=20, callbacks=[early_stop_callback], num_sanity_val_steps=0)
trainer.fit(mynet)

In my lightning module I have the functions:
def __init__(self, random_inputs):

def forward(self, x):

def train_dataloader(self):

def val_dataloader(self):

def training_step(self, batch, batch_nb):

def training_epoch_end(self, outputs):

def validation_step(self, batch, batch_nb):

def validation_epoch_end(self, outputs):

def configure_optimizers(self):

Do I need a specific predict function or is there any already implemented way I don't see?


title: Iterating over subsets from torch.utils.data.random_split

body:
I am currently loading a folder with AI training data in it. The subfolders represent the label names with the corresponding images inside. This works well by using pyTorch's ImageFolder loader.

def load_dataset():
    data_path = 'C:/example_folder/'

    train_dataset_manual = torchvision.datasets.ImageFolder(
        root=data_path,
        transform=torchvision.transforms.ToTensor()
    )

    train_loader_manual = torch.utils.data.DataLoader(
        train_dataset_manual,
        batch_size=1,
        num_workers=0,
        shuffle=True
    )

    return train_loader_manual

full_dataset = load_dataset()

Now I want to have this dataset split into a training and a test data set. I am using the random_split function for this:

training_data_size = 0.8

train_size = int(training_data_size * len(full_dataset))
test_size = len(full_dataset) - train_size
train_dataset, test_dataset = torch.utils.data.random_split(full_dataset, [train_size, test_size])

The full_dataset is an object of type torch.utils.data.dataloader.DataLoader. I can iterate through it with a loop like this:

for batch_idx, (data, target) in enumerate(full_dataset):
    print(batch_idx)

The train_dataset is an object of type torch.utils.data.dataset.Subset. If I try to loop through it, I get:

  TypeError 'DataLoader' object is not subscriptable:

for batch_idx, (data, target) in enumerate(train_dataset):
    print(batch_idx)

How can I loop through it? I am relatively new to Python.

Thanks!


title: Where in the code of pytorch or huggingface/transformer label gets "renamed" into labels?

body:
My question concerns the example, available in the great huggingface/transformers library.
I am using a notebook, provided by library creators as a starting point for my pipeline. It presents a pipeline of finetuning a BERT for Sentence Classification on Glue dataset.
When getting into the code, I noticed a very weird thing, which I cannot explain.
In the example, input data is introduced to the model as the instances of the InputFeatures class from here:
This class has 4 attributes, including the label attribute:
class InputFeatures:
    ...
    input_ids: List[int]
    attention_mask: Optional[List[int]] = None
    token_type_ids: Optional[List[int]] = None
    label: Optional[Union[int, float]] = None

which are later passed as a dictionary of inputs to the forward() method of the model. This is done by the Trainer class, for example in the lines 573-576 here:
    def _training_step(
        self, model: nn.Module, inputs: Dict[str, torch.Tensor], optimizer: torch.optim.Optimizer
    ) -> float:
        model.train()
        for k, v in inputs.items():
            inputs[k] = v.to(self.args.device)

        outputs = model(**inputs)

However, the forward() method expects labels (note the plural form) input parameter (taken from here):
    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        output_attentions=None,
    ):

So my question is where does the label become labels in this pipeline?
To give some extra info on the issue, I created my own pipeline, which uses nothing, related, with Glue data and pipe, basically it relies only on the Trainer class of transformers. I even use another model (Flaubert). I replicated the InputFeature class and my code works for both cases below:
class InputFeature:
    def __init__(self, text, label):
        self.input_ids = text
        self.label = label

class InputFeaturePlural:
    def __init__(self, text, label):
        self.input_ids = text
        self.labels = label

But it does not work if I name the second attribute as self.labe or by any other names. Why is it possible to use both attribute names?
It's not like it is extremely important in my case, but I feel uncomfortable passing around the data in the variable, which "changes name" somewhere along the way.


title: Deep learning model stuck in local minima or overfit?

body: I trained an image classification model of 10 classes by finetuning EfficientNet-B4 for 100 epochs. I split my training data to 70/30. I used stochastic gradient descent with Nesterov momentum of 0.9 and the starting learning rate is 0.001. The batch size is 10. The test loss seemed to stuck at 84% for the next 50 epochs (51st - 100th). I do not know whether the model was stuck in local minima or the model was overfitted. Below is an image of the test and train loss from 51st epoch to 100th. I need your help a lot. Thanks. Train test loss image from 51st to 100th epoch.


title: Building recurrent neural network with feed forward network in pytorch

body:
I was going through this tutorial. I have a question about the following class code:

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()

        self.input_size = input_size
        self.hidden_size = hidden_size
        self.output_size = output_size

        self.i2h = nn.Linear(input_size + hidden_size, hidden_size)
        self.i2o = nn.Linear(input_size + hidden_size, output_size)
        self.softmax = nn.LogSoftmax()

    def forward(self, input, hidden):
        combined = torch.cat((input, hidden), 1)
        hidden = self.i2h(combined)
        output = self.i2o(combined)
        output = self.softmax(output)
        return output, hidden

    def init_hidden(self):
        return Variable(torch.zeros(1, self.hidden_size))

This code was taken from Here. There it was mentioned that

  Since the state of the network is held in the graph and not in the layers, you can simply create an nn.Linear and reuse it over and over again for the recurrence.

What I don't understand is, how can one just increase input feature size in nn.Linear and say it is a RNN. What am I missing here?


title: Data loading in torch.utils.data.TensorDataset

body:
I would like to run RAM model on my own data set. My training set is train.t7 . I have encountered a problem while loading my data set because it is my first time that I have used python. I have tried very hard to understand both the software and package but I need your help in solving this problem.

I run the following code:

o=torchfile.load('train.t7')
x=tf.convert_to_tensor(o[b'data'],dtype=tf.uint8)
y=tf.convert_to_tensor(o[b'labels'],dtype=tf.uint8)
x=tf.reshape(x,[50000,784])
dataset = torch.utils.data.TensorDataset(x, y)

But I got the following error:

File "/home/wing92518/.local/lib/python3.6/site-packages/torch/utils/data/dataset.py", line 36, in <genexpr>
    assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)
AttributeError: 'Tensor' object has no attribute 'size'


title: Pytorch dataset and shared memory?

body: I would want to cache data in a torch.utils.data.Dataset. The simple solution is to just persist certain tensors in a member of the dataset. However, since the torch.utils.data.DataLoader class spawns multiple processes, the cache would only be local to each instance and would cause me to possibly cache multiple copies of the same tensors. Is there a way to use Python's multiprocessing library to share data between the different loader processes?


title: where could I find training.pt / test.pt

body:
In the Pytorch docs for MNIST I read:

root (string): Root directory of dataset where MNIST/processed/training.pt
and  MNIST/processed/test.pt exist.

Where could I find these two files traing.pt, test.pt? And what are their format?


title: How to get per channel convolution in tensorflow like the conv2d parameter "groups" in Pytorch?

body:
I am working on a project for which I am referring a network which has been built in Pytorch. One of its layer has conv2d layer with group =  some value. While I am building the project in Tensorflow . How do I do this particular layer considering I didn't find any group parameter in the conv2d layer of tensorflow. I searched online but couldn't find anything concrete like what group says is it is a per-channel convolution. The close thing I found was tf.nn.separable_conv2d but still not sure how to set the arguments in it.

The Pytorch layer is conv2d(input ,weight , groups = x )
where input and weight are 4d tensors. Could anyone help out how I create this layer in Tensorflow.


title: Using bias in PyTorch for basic function approximation

body:
Using R, it is very easy to approximate basic functions through a neural network:

library(nnet)
x <- sort(10*runif(50))
y <- sin(x)
nn <- nnet(x, y, size=4, maxit=10000, linout=TRUE, abstol=1.0e-8, reltol = 1.0e-9, Wts = seq(0, 1, by=1/12) )
plot(x, y)
x1 <- seq(0, 10, by=0.1)
lines(x1, predict(nn, data.frame(x=x1)), col="green")
predict( nn , data.frame(x=pi/2) )

A simple neural network with one hidden layer of a mere 4 neurons is sufficient to approximate a sine. (As per stackoverflow question Approximating function with Neural Network.)

But I cannot obtain the same in PyTorch.

In fact, the neural network created by R contains not only an input, four hidden and an output, but also two "bias" neurons - the first connected towards the hidden layer, the second towards the output.

The plot above is obtained through the following:

library(devtools)
library(scales)
library(reshape)
source_url('https://gist.github.com/fawda123/7471137/raw/cd6e6a0b0bdb4e065c597e52165e5ac887f5fe95/nnet_plot_update.r')
plot.nnet(nn$wts,struct=nn$n, pos.col='#007700',neg.col='#FF7777')   ### this plots the graph
plot.nnet(nn$wts,struct=nn$n, pos.col='#007700',neg.col='#FF7777', wts.only=1)   ### this prints the weights

Attempting the same with PyTorch produces a different network: the bias neurons are missing.

Following is an attempt to do in PyTorch what was done previously in R. The results will not be satisfactory: the function is not approximated. The most evident difference is that absence of the bias neurons.

import torch
from torch.autograd import Variable

import random
import math

N, D_in, H, D_out = 1000, 1, 4, 1

l_x = []
l_y = []

for a in range(1000):
    r = random.random()*10
    l_x.append( [r] )
    l_y.append( [math.sin(r)] )

tx = torch.cuda.FloatTensor(l_x)
ty = torch.cuda.FloatTensor(l_y)

x = Variable(tx, requires_grad=False)
y = Variable(ty, requires_grad=False)

w1 = Variable(torch.randn(D_in, H ).type(torch.cuda.FloatTensor), requires_grad=True)
w2 = Variable(torch.randn(H, D_out).type(torch.cuda.FloatTensor), requires_grad=True)

learning_rate = 1e-5
for t in range(1000):
    y_pred = x.mm(w1).clamp(min=0).mm(w2)

    loss = (y_pred - y).pow(2).sum()
    if t<10 or t%100==1: print(t, loss.data[0])

    loss.backward()

    w1.data -= learning_rate * w1.grad.data
    w2.data -= learning_rate * w2.grad.data

    w1.grad.data.zero_()
    w2.grad.data.zero_()

t = [ [math.pi] ]
print( str(t) +" -> "+ str( (Variable(torch.cuda.FloatTensor( t ))).mm(w1).clamp(min=0).mm(w2).data ) )
t = [ [math.pi/2] ]
print( str(t) +" -> "+ str( (Variable(torch.cuda.FloatTensor( t ))).mm(w1).clamp(min=0).mm(w2).data ) )

How to make the network approximate to the given function (sine in this case), through either inserting the "bias" neurons or other missing detail?

Moreover: I have difficulties in understanding why R inserts the "bias". I found information that the bias could be akin to the "Intercept in a Regression Model" - I still find it not clear. Any information would be appreciated.
EDIT: an excellent explanation turned out to be at stackoverflow question Role of Bias in Neural Networks

EDIT:

An example to obtain the result, though using the "fuller" framework ("not reinventing the wheel") is as follows:

import torch
from torch.autograd import Variable
import torch.nn.functional as F

import math

N, D_in, H, D_out = 1000, 1, 4, 1

l_x = []
l_y = []

for a in range(1000):
    t = (a/1000.0)*10
    l_x.append( [t] )
    l_y.append( [math.sin(t)] )

x = Variable( torch.FloatTensor(l_x) )
y = Variable( torch.FloatTensor(l_y) )

class Net(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output):
        super(Net, self).__init__()
        self.to_hidden = torch.nn.Linear(n_feature, n_hidden)
        self.to_output = torch.nn.Linear(n_hidden,  n_output)

    def forward(self, x):
        x = self.to_hidden(x)
        x = F.tanh(x)           # activation function
        x = self.to_output(x)
        return x

net = Net(n_feature = D_in, n_hidden = H, n_output = D_out)

learning_rate =  0.01
optimizer = torch.optim.Adam( net.parameters() , lr=learning_rate )

for t in range(1000):
    y_pred = net(x)

    loss = (y_pred - y).pow(2).sum()
    if t<10 or t%100==1: print(t, loss.data[0])

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

t = [ [math.pi] ]
print( str(t) +" -> "+ str( net( Variable(torch.FloatTensor( t )) ) ) )
t = [ [math.pi/2] ]
print( str(t) +" -> "+ str( net( Variable(torch.FloatTensor( t )) ) ) )

Unfortunately, while this code works properly, it does not solve the matter of making the original, more "low level" code work as expected (e.g. introducing the bias).


When does PyTorch automatically cast Tensor dtype? Why does it sometimes do it automatically and other times throws and error?

For example this automatically casts c to be a float:

a = torch.tensor(5)
b = torch.tensor(5.)
c = a*b

a.dtype
>>> torch.int64

b.dtype
>>> torch.float32

c.dtype
>>> torch.float32

But this throws an error:

a = torch.ones(2, dtype=torch.float)
b = torch.ones(2, dtype=torch.long)
c = torch.matmul(a,b)

Traceback (most recent call last):

  File "<ipython-input-128-fbff7a713ff0>", line 1, in <module>
    torch.matmul(a,b)

RuntimeError: Expected object of scalar type Float but got scalar type Long for argument #2 'tensor'

I'm confused since Numpy seems to automatically cast all arrays as necessary e.g.

a = np.ones(2, dtype=np.long)
b = np.ones(2, dtype=np.float)

np.matmul(a,b)
>>> 2.0

a*b
>>> array([1., 1.])


title: Given list of image filenames for eachset,Split large dataset to train/valid/test directories?

body:
I am trying to split a large dataset into train/valid/test sets from Food101 dataset  for image classification

and the structure of a dataset is like this and has all images in one folder

'',
'Structure:',
'----------',
'pec/',
'    images/',
'        <class_name>/',
'            <image_id>.jpg',
'    meta/',
'        classes.txt',
'        labels.txt',
'        test.json',
'        test.txt',
'        train.json',
'        train.txt',
'',
'All images can be found in the "images" folder and are organized per class. All',
'image ids are unique and correspond to the foodspotting.com review ids.
'',
'The test/train splitting used in the experiment of our paper can be found in',
'the "meta" directory.', (edited) ```

I want to divide images dataset to train/valid/test  with the list of filenames given in train.txt and test.txt, which author used

the Shape of train,valid, test lists:  (101, 600) , (101, 150) , 25250

In colab ,i run following code

for x in range(train.shape[0]):
    for y in range(train.shape[1]):

     temp = train[x,y] + ".jpg"

     foldername = temp.split('/')[0]

     !mv /content/food-101/images/$temp /content/food101/train/$foldername/

Individually moving images by running nested loop by taking filenames in lists, is  taking forever time to create folders as there are 100100 images total so,

I have a list of filenames for train/valid and test set but how to make them into folders so that we can feed it to image classifier in pytorch image folder format(i mean train / valid / test set are three different folders and each folder has subfolders of each class)

Please Do tell if anyone knows how to do this please, and please I really need your help here,  THanks:smile:


title: Why pytorch needs much more memory than it should?

body:
I'm just playing around with pytorch and I'm wondering why it consumes so much memory of my GPU?

I'm using Cuda 10.0 with pythorch 1.2.0 and torchvision 0.4.0.

import torch
gpu = torch.device("cuda")
x = torch.ones(int(4e8), device=gpu)
y = torch.ones(int(1e5), device=gpu)

Running the above code I get the error:
RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 2.00 GiB total capacity; 1.49 GiB already allocated; 0 bytes free; 0 bytes cached)

So, does pytorch needs ~500MB of the gpu memory as overhead? Or what is the problem here?


title: torch assign not in place by tensor slicing in pytorch

body:
I am trying to convert my current code, which assigns tensors in place, to an outer operation.
Meaning currently the code is

self.X[:, nc:] = D

Where D is in the same shape as self.X[:, nc:]
But I would like to convert it to

sliced_index = ~ somehow create an indexed tensor from self.X[:, nc:]
self.X = self.X.scatter(1,sliced_index,mm(S_, Z[:, :n - nc]))

And don't know how to create that index mask tensor that represents only the entries in the sliced tensor

Minimal example:

a = [[0,1,2],[3,4,5]]
D = [[6],[7]]
Not_in_place = [[0,1,6],[3,4,7]]


title: PyTorch LSTM dropout vs Keras LSTM dropout

body:
I'm trying to port my sequential Keras network to PyTorch. But I'm having trouble with the LSTM units:

LSTM(512,
     stateful = False,
     return_sequences = True,
     dropout = 0.5),
LSTM(512,
     stateful = False,
     return_sequences = True,
     dropout = 0.5),

How should I formulate this in PyTorch? Especially dropout appears to work very differently in PyTorch than it does in Keras.


title: Getting diagonal "stripe" from matrix in NumPy or PyTorch

body:
I need to get the diagonal "stripe" of a matrix. Say I have a matrix of size KxN (K>N):
[[ 0  1  2]
 [ 3  4  5]
 [ 6  7  8]
 [ 9 10 11]]

From it I need to extract a diagonal stripe, in this case, a matrix MxV size that is created by truncating the original one:
[[ 0  x  x]
 [ 3  4  x]
 [ x  7  8]
 [ x  x  11]]

So the result matrix is:
[[ 0  4  8]
 [ 3  7  11]]

I could define a bolean mask like so:
import numpy as np

X=np.arange(12).reshape(4,3)
mask=np.asarray([
  [ True,  False,  False],
  [ True,  True,  False],
  [ False, True,  True],
  [ False, False,  True]
])

>>> X
array([[ 0,  1,  2],
       [ 3,  4,  5],
       [ 6,  7,  8],
       [ 9, 10, 11]])

>>> X.T[mask.T].reshape(3,2).T
array([[ 0,  4,  8],
       [ 3,  7, 11]])

But I don't see how such a mask could be automatically generated to an arbitrary KxN matrix (e.g. 39x9, 360x96)
Is there a function that does this automatically either in numpy, scipy or pytorch?

Additional question: is it possible to get a "reverse stripe" instead? i.e.
[[ x   x   2]
 [ x   4   5]
 [ 6   7   x]
 [ 9   x   x]]


title: Pytorch: how to get all the tensors in a graph

body:
I would like to access all the tensors instances of a graph. For example, I can check if a tensor is detached or I can check the size. It can be done in tensorflow.

I don't want visualization of the graph.


title: Truncate SVD decomposition of Pytorch tensor without transfering to cpu

body:
I'm training a model in Pytorch and I want to use truncated SVD decomposition of input. For calculating SVD I transfer input witch is a Pytorch Cuda Tensor to CPU and using TruncatedSVD from scikit-learn perform truncate, after that, I transfer the result back to GPU. The following is code for my model:

 class ImgEmb(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(ImgEmb, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.drop = nn.Dropout(0.2)
        self.mlp = nn.Linear(input_size/2, hidden_size)
        self.relu = nn.Tanh()
        self.svd = TruncatedSVD(n_components=input_size/2)

    def forward(self, input):
        svd=self.svd.fit_transform(input.cpu())
        svd_tensor=torch.from_numpy(svd)
        svd_tensor=svd_tensor.cuda()
        mlp=self.mlp(svd_tensor)
        res = self.relu(mlp)
        return res

I wonder is a way to implement truncated SVD without transferring back and forth to GPU? (Because it's very time consuming and is not efficient at all)


title: PyTorch: How to implement attention for graph attention layer

body:
I have implemented the attention (Eq. 1) of https://arxiv.org/pdf/1710.10903.pdf but it's clearly not memory efficient and can run only a single model on my GPU (it takes 7-10GB).

Currently, I have

class MyModule(nn.Module):

def __init__(self, in_features, out_features):
    super(MyModule, self).__init__()
    self.in_features = in_features
    self.out_features = out_features

    self.W = nn.Parameter(nn.init.xavier_uniform(torch.Tensor(in_features, out_features).type(torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor), gain=np.sqrt(2.0)), requires_grad=True)
    self.a = nn.Parameter(nn.init.xavier_uniform(torch.Tensor(2*out_features, 1).type(torch.cuda.FloatTensor if torch.cuda.is_available() else torch.FloatTensor), gain=np.sqrt(2.0)), requires_grad=True)

def forward(self, input):
    h = torch.mm(input, self.W)
    N = h.size()[0]

    a_input = torch.cat([h.repeat(1, N).view(N * N, -1), h.repeat(N, 1)], dim=1).view(N, -1, 2 * self.out_features)
    e = F.elu(torch.matmul(a_input, self.a).squeeze(2))
    return e

Where my insight to compute all the e_ij terms is

In [8]: import torch

In [9]: import numpy as np

In [10]: h = torch.LongTensor(np.array([[1,1], [2,2], [3,3]]))

In [11]: N=3

In [12]: h.repeat(1, N).view(N * N, -1)
Out[12]:

1     1
1     1
1     1
2     2
2     2
2     2
3     3
3     3
3     3

[torch.LongTensor of size 9x2]

In [13]: h.repeat(N, 1)
Out[13]:

1     1
2     2
3     3
1     1
2     2
3     3
1     1
2     2
3     3

[torch.LongTensor of size 9x2]

And finally concatenate both hs and feed matrix a.

Is there a way to do it in a more memory-friendly way ?