title: PyTorch ValueError: too many dimensions 'str'

body:
I am trying to use torchvision.transforms to apply transformtation the training data, but getting the following traceback error, Traceback (most recent call last):

File "train4.py", line 113, in
targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
ValueError: too many dimensions 'str'

import argparse
import datetime
import glob
import os
import random
import shutil
import time
from os.path import join

import numpy as np
import pandas as pd
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
from torchvision.transforms import ToTensor, RandomChoice, Compose, functional
from tqdm import tqdm

from convnet0 import Convnet
from dataset2 import CellsDataset
import pdb

parser = argparse.ArgumentParser('Predicting hits from pixels')
parser.add_argument('name',type=str,help='Name of experiment')
parser.add_argument('data_dir',type=str,help='Path to data directory containing images and gt.csv')
parser.add_argument('--num_steps',type=int,default=20000,help='Number of training iterations')
parser.add_argument('--batchsize',type=int,default=16,help='Size of batch')
parser.add_argument('--weight_decay',type=float,default=0.0,help='Weight decay coefficient (something like 10^-5)')
parser.add_argument('--lr',type=float,default=0.0001,help='Learning rate')
parser.add_argument('--resume',action='store_true',help='Resume experiments from checkpoint directory')
parser.add_argument('--seed',type=int,default=1337,help='RNG seed')
args = parser.parse_args()

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')

torch.manual_seed(args.seed)
torch.cuda.manual_seed(args.seed)
np.random.seed(args.seed)
random.seed(args.seed)

# create output directory tree:
if os.path.isdir(args.name) and not args.resume:
    shutil.rmtree(args.name)
logs_path = join(args.name,'logs')
checkpoints_path = join(args.name,'checkpoints')
checkpoint_path = join(checkpoints_path,'checkpoint.pth')
if not os.path.isdir(args.name):
    os.mkdir(args.name)
    os.mkdir(logs_path)
    os.mkdir(checkpoints_path)

# record arguments for future reference:
with open(join(args.name,'arguments.txt'),'w') as fout:
    fout.write(str(args))

# create datasets:
toTensor = ToTensor()
dataset_train = CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)

# duplicate the dataset and transform the duplicates
transforms = Compose([RandomChoice([functional.hflip,functional.vflip])])
dataset_train.duplicate_and_transform(transforms)
print(dataset_train.Size())

# load metadata:
metadata = pd.read_csv(join(args.data_dir,'gt.csv'))
metadata.set_index('filename', inplace=True)

# assign to train or test
random.shuffle(dataset_train.files)
split_point = int(len(dataset_train) * 0.9) # 90/10 train/val split
dataset_test = CellsDataset(args.data_dir,transform=ToTensor(),return_filenames=True)

dataset_test.files = dataset_train.files[split_point:]
dataset_train.files = dataset_train.files[:split_point]

loader_train = DataLoader(dataset_train,batch_size=args.batchsize,shuffle=True,num_workers=4,pin_memory=True)
loader_test =  DataLoader(dataset_test, batch_size=args.batchsize,shuffle=True,num_workers=4,pin_memory=True)

# create model:
model = Convnet()
model.to(device)

# create optimizer:
optimizer = torch.optim.Adam(model.parameters(),lr=args.lr,weight_decay=args.weight_decay)

if args.resume:
    try:
        checkpoint = torch.load(checkpoint_path)
        print('Resuming from checkpoint...')
        model.load_state_dict(checkpoint['state_dict'])
        globalStep = checkpoint['globalStep']
        optimizer.load_state_dict(checkpoint['optimizer_state_dict'])
        dataset_train.files = checkpoint['train_paths']
        dataset_test.files = checkpoint['test_paths']
    except FileNotFoundError:
        globalStep = 0

# create logger:
timestamp = datetime.datetime.now().strftime("%Y-%m-%d_%H:%M")
writer = SummaryWriter()

# main training loop
global_step = 0
best_test_error = 10000
for epoch in range(15):
    print("Epoch %d" % epoch)
    model.train()
    for images, paths in tqdm(loader_train):
        images = images.to(device)
        pdb.set_trace()
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

        # backward pass:
        loss = torch.mean((preds - targets)**2)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        # logging:
        count_error = torch.abs(preds - targets).mean()
        writer.add_scalar('train_loss', loss.item(), global_step=global_step)
        writer.add_scalar('train_count_error', count_error.item(), global_step=global_step)

        print("Step %d, loss=%f, count error=%f" % (global_step,loss.item(),count_error.item()))

        global_step += 1

    mean_test_error = 0
    model.eval()
    for images, paths in tqdm(loader_test):
        images = images.to(device)
        targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
        targets = targets.float().to(device)

        # forward pass:
        output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
        preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

        # logging:
        loss = torch.mean((preds - targets)**2)
        count_error = torch.abs(preds - targets).mean()
        mean_test_error += count_error
        writer.add_scalar('test_loss', loss.item(), global_step=global_step)
        writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)

        global_step += 1

    mean_test_error = mean_test_error / len(loader_test)
    print("Test count error: %f" % mean_test_error)
    if mean_test_error < best_test_error:
        best_test_error = mean_test_error
        torch.save({'state_dict':model.state_dict(),
                    'optimizer_state_dict':optimizer.state_dict(),
                    'globalStep':global_step,
                    'train_paths':dataset_train.files,
                    'test_paths':dataset_test.files},checkpoint_path)


title: Tensorboard is showing a blank page (Refused to execute script from 'http://localhost:6006/index.js' because its MIME type)

body:
When Trying to open tesnorflow I just get a plank page:

This is how it looks like in firefox:

I get the error message in the chrome console:

Refused to execute script from 'http://localhost:6006/index.js' because its MIME type ('text/plain') is not executable, and strict MIME type checking is enabled.

In the firefox console I get the error message:

The resource from “http://localhost:6006/index.js” was blocked due to MIME type (“text/plain”) mismatch (X-Content-Type-Options: nosniff)

and

Loading failed for the <script> with source “http://localhost:6006/index.js”.

I tried:
Unable to open Tensorboard in browser
Tensorboard get blank page

I typed in the console:

tensorboard --logdir=runs --bind_all
tensorboard --logdir=./runs --bind_all
tensorboard --logdir=./runs/ --bind_all
tensorboard --logdir=./runs --host localhost --port 6006
tensorboard --logdir=./runs --host localhost
tensorboard --logdir=./runs --port 6006 --bind_all

I have tensorboard version: 2.1.0
I generated my data like that:

 train_set = torchvision.datasets.FashionMNIST(
        root="./data/FashionMNIST",
        train=True,
        download=True,
        transform=transforms.Compose([
            transforms.ToTensor()
        ])
    )
train_loader = torch.utils.data.DataLoader(train_set, batch_size=1000)
tb = SummaryWriter()

network = Network()
images, labels = next(iter(train_loader))
grid = torchvision.utils.make_grid(images)

tb.add_image("image", grid)
tb.add_graph(network, images)
tb.close()

I followed this tutorial: TensorBoard with PyTorch - Visualize Deep Learning Metrics


title: unknown resampling filter error when trying to create my own dataset with pytorch

body:
I am trying to create a CNN implemented with data augmentation in pytorch to classify dogs and cats. The issue that I am having is that when I try to input my dataset and enumerate through it I keep getting this error:

  Traceback (most recent call last):

  File "<ipython-input-55-6337e0536bae>", line 75, in <module>
    for i, (inputs, labels) in enumerate(trainloader):

  File "/usr/local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 188, in __next__
    batch = self.collate_fn([self.dataset[i] for i in indices])

  File "/usr/local/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 188, in <listcomp>
    batch = self.collate_fn([self.dataset[i] for i in indices])

  File "/usr/local/lib/python3.6/site-packages/torchvision/datasets/folder.py", line 124, in __getitem__
    img = self.transform(img)

  File "/usr/local/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 42, in __call__
    img = t(img)

  File "/usr/local/lib/python3.6/site-packages/torchvision/transforms/transforms.py", line 147, in __call__
    return F.resize(img, self.size, self.interpolation)

  File "/usr/local/lib/python3.6/site-packages/torchvision/transforms/functional.py", line 197, in resize
    return img.resize((ow, oh), interpolation)

  File "/usr/local/lib/python3.6/site-packages/PIL/Image.py", line 1724, in resize
    raise ValueError("unknown resampling filter")

ValueError: unknown resampling filter

and I really dont know whats wrong with my code. I have provided the code below:

# Creating the CNN

# Importing the libraries
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.autograd import Variable
import torchvision
from torchvision import transforms

#Creating the CNN Model
class CNN(nn.Module):
    def __init__(self, nb_outputs):
        super(CNN, self).__init__() #activates the inheritance and allows the use of all the tools in the nn.Module
        #making the 3 convolutional layers that will be used in the convolutional neural network
        self.convolution1 = nn.Conv2d(in_channels = 1, out_channels = 32, kernel_size = 5) #kernal_size -> the deminson of the feature detector e.g kernel_size = 5 => feature detector of size 5x5
        self.convolution2 = nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 2)
        #making 2 full connections one to connect the inputs of the ANN to the hidden layer and another to connect the hidden layer to the outputs of the ANN
        self.fc1 = nn.Linear(in_features = self.count_neurons((1, 64,64)), out_features = 40)
        self.fc2 = nn.Linear(in_features = 40, out_features = nb_outputs)

    def count_neurons(self, image_dim):
        x = Variable(torch.rand(1, *image_dim)) #this variable repersents a fake image to allow us to compute the number of neruons
        #in order to pass the elements of the tuple image_dim into our function as a list of arguments we need to add a * before image_dim
        #since x will be going into our neural network we need to convert it into a torch variable using the Variable() function
        x = F.relu(F.max_pool2d(self.convolution1(x), 3, 2)) #first we apply the convolution to x then apply max_pooling to the convolutional fake images and then activate all the neurons in the pooling layer
        x = F.relu(F.max_pool2d(self.convolution2(x), 3, 2)) #the signals are now propragated up to the thrid convoulational layer
        #Now to flatten x to obtain the number of neurons in the flattening layer
        return x.data.view(1, -1).size(1) #this will flatten x into a huge vector and returns the size of the vector, that size repersents the number of neurons that will be inputted into the ANN
        #even though x is not a real image from the game since the size of the flattened vector only depends on the dimention of the inputted image we can just set x to have the same dimentions as the image

    def forward(self, x):
        x = F.relu(F.max_pool2d(self.convolution1(x), 3, 2)) #first we apply the convolution to x then apply max_pooling to the convolutional fake images and then activate all the neurons in the pooling layer
        x = F.relu(F.max_pool2d(self.convolution2(x), 3, 2))
        #flattening layer of the CNN
        x = x.view(x.size(0), -1)
        #x is now the inputs to the ANN
        x = F.relu(self.fc1(x)) #we propagte the signals from the flatten layer to the full connected layer and activate the neruons by breaking the linearilty with the relu function
        x = F.sigmoid(self.fc2(x))
        #x is now the output neurons of the ANN
        return x

train_tf = transforms.Compose([transforms.RandomHorizontalFlip(),
                               transforms.Resize(64,64),
                               transforms.RandomRotation(20),
                               transforms.RandomGrayscale(.2),
                               transforms.ToTensor()])

test_tf = transforms.Compose([transforms.Resize(64,64),
                              transforms.ToTensor()])

training_set = torchvision.datasets.ImageFolder(root = './dataset/training_set',
                                                transform = train_tf)
test_set = torchvision.datasets.ImageFolder(root = './dataset/test_set',
                                                transform = transforms.Compose([transforms.Resize(64,64),
                              transforms.ToTensor()]) )

trainloader = torch.utils.data.DataLoader(training_set, batch_size=32,
                                              shuffle=True, num_workers=0)
testloader = torch.utils.data.DataLoader(test_set, batch_size= 32,
                                             shuffle=False, num_workers=0)

#training the model
cnn = CNN(1)
cnn.train()
loss = nn.BCELoss()
optimizer = optim.Adam(cnn.parameters(), lr = 0.001) #the optimizer => Adam optimizer
nb_epochs = 25

for epoch in range(nb_epochs):
    train_loss = 0.0
    train_acc = 0.0
    total = 0.0
    for i, (inputs, labels) in enumerate(trainloader):
        inputs, labels = Variable(inputs), Variable(labels)
        cnn.zero_grad()
        outputs = cnn(inputs)
        loss_error = loss(outputs, labels)
        optimizer.step()

        _, pred = torch.max(outputs.data, 1)
        total += labels.size(0)
        train_loss += loss_error.data[0]
        train_acc += (pred == labels).sum()

    train_loss = train_loss/len(training_loader)
    train_acc = train_acc/total
    print('Epoch: %d, loss: %.4f, accuracy: %.4f' %(epoch+1, train_loss, train_acc))

The folder arrangement for the code is /dataset/training_set and inside the training_set folder are two more folders one for all the cat images and the other for all the dog images. Each image is name either dog.xxxx.jpg or cat.xxxx.jpg, where the xxxx represents the number so for the first cat image it would be cat.1.jpg up to cat.4000.jpg. This is the same format for the test_set folder. The number of training images is 8000 and the number of test images is 2000. If anyone can point out my error I would greatly appreciate it.

Thank you


title: Updating Pytorch variables

body:
First, I should mention I’m on torch version 0.3.1, but am happy to upgrade if necessary.

I am still learning, so please let me know if I misunderstood anything here.

I am building several graphs imperatively, using Variables. The mathematical expressions might be quite complicated as time goes on, and certain parts of the graphs may be generated using autograd’s gradient functions, which may then be operated on.

I can create whatever expression I want just fine. The only catch is building up this graph can be quite “slow” - for relatively simple operations (4x4 matrices being multiplied together), it can take a few ms.

Once these graphs are generated, though, I expect that I should be able to update certain Variables’ values, and evaluate the output of the graph (or any node, really), much faster. I believe that that is probably the case since evaluation should be able to happen completely in the C back-end, which should be pretty otimized for speed? In other words, building the graph in python might be slow, since it involves python for loops and the like, but re-evaluating a graph with static topology should be fast.

Is this intuition correct? If so, how can I efficiently re-assign a value and re-evaluate the graph?


title: Pytorch: nn.Parameter to tensorflow variable

body:
I am trying to convert pytorch code to tensorflow code. I am not able to find how to convert parameter in tensorflow, I was thinking to use tf.get_variable but not sure about it.

Pytorch code looks like this :

def gen_adj(A):
    D = torch.pow(A.sum(1).float(), -0.5)
    D = torch.diag(D)
    adj = torch.matmul(torch.matmul(A, D).t(), D)
    return adj

self.A = Parameter(torch.from_numpy(_adj).float())
inp = inp[0]
adj = gen_adj(self.A).detach()
x = self.new_fucn(inp, adj)

So for this line

self.A = Parameter(torch.from_numpy(_adj).float())

What I tried is :

self.A = tf.get_variable(name="self.A",
                                 shape=[_adj.shape[0],_adj.shape[1]],
                                 initializer=tf.constant_initializer(np.array(_adj)),
                                 trainable=False
                                )

and get_adj function like this:

def tensorflow_adj(A):

    D = tf.pow(tf.reduce_sum(A, 1),-0.5)
    D = tf.diag_part(D)
    adj = tf.matmul( tf.transpose(tf.matmul(A,D),[1,0]),D)

    return adj

I would greatly appreciate it if anyone kindly give me some advice if my converted code is correct.

Thank you !


title: Parallelize simple for loop using Pytorch on single GPU

body:
I have a for-loop which operates on independent columns of a large matrix. I have parallelized the for-loop on CPU using the prange function in Numba. Now I want to perform this operation using PyTorch tensors on a GPU. I'm a PyTorch novice and don't know how to do it.

Any help will be really appreciated.

My Python code is as follows:

def select_next(X, gains, current_values, mask):
    for idx in prange(X.shape[0]):
        if mask[idx] == 1:
            continue

        a = numpy.maximum(X[idx], current_values)
        gains[idx] = (a - current_values).sum()
    return numpy.argmax(gains)

My PyTorch code is as follows:

def select_next(X, gains, current_values, mask):
    for idx in range(X.shape[0]):
        if mask[idx].item() == 1:
            continue

        a = torch.max(X[idx], current_values)
        gains[idx] = torch.sum(torch.sub(a, current_values))
    return torch.argmax(gains)

How can I parallelize the for loops?


title: How can I create a PyCUDA GPUArray from a gpu memory address?

body: I'm working with PyTorch and want to do some arithmetic on Tensor data with the help of PyCUDA. I can get a memory address of a cuda tensor t via t.data_ptr(). Can I somehow use this address and my knowledge of the size and data type to initialize a GPUArray? I am hoping to avoid copying the data, but that would also be an alternative.


title: Using trained BERT Model and Data Preprocessing

body:
When using a pre-trained BERT embeddings from pytorch (which are then fine-tuned), should the text data fed into the model be pre-processed like in any standard NLP task?
For instance, should  stemming, removing low frequency words, de-captilisation, be performed or should the raw text simply be passed to `transformers.BertTokenizer'?


title: PyTorch error in trying to backward through the graph a second time

body:
I'm trying to run this code: https://github.com/aitorzip/PyTorch-CycleGAN
I modified only the dataloader and transforms to be compatible with my data.
When trying to run it I get this error:

Traceback (most recent call last):
File "models/CycleGANs/train",
line 150, in
loss_D_A.backward()
File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in
backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File
"/opt/conda/lib/python3.8/site-packages/torch/autograd/init.py",
line 130, in backward
Variable._execution_engine.run_backward(
RuntimeError: Trying to backward through the graph a second time, but the saved intermediate
results have already been freed. Specify retain_graph=True when
calling backward the first time.

This is the train loop up to the point of error:
for epoch in range(opt.epoch, opt.n_epochs):
for i, batch in enumerate(dataloader):
    # Set model input
    real_A = Variable(input_A.copy_(batch['A']))
    real_B = Variable(input_B.copy_(batch['B']))

    ##### Generators A2B and B2A #####
    optimizer_G.zero_grad()

    # Identity loss
    # G_A2B(B) should equal B if real B is fed
    same_B = netG_A2B(real_B)
    loss_identity_B = criterion_identity(same_B, real_B)*5.0
    # G_B2A(A) should equal A if real A is fed
    same_A = netG_B2A(real_A)
    loss_identity_A = criterion_identity(same_A, real_A)*5.0

    # GAN loss
    fake_B = netG_A2B(real_A)
    pred_fake = netD_B(fake_B)
    loss_GAN_A2B = criterion_GAN(pred_fake, target_real)

    fake_A = netG_B2A(real_B)
    pred_fake = netD_A(fake_A)
    loss_GAN_B2A = criterion_GAN(pred_fake, target_real)

    # Cycle loss
    # TODO: cycle loss doesn't allow for multimodality. I leave it for now but needs to be thrown out later
    recovered_A = netG_B2A(fake_B)
    loss_cycle_ABA = criterion_cycle(recovered_A, real_A)*10.0

    recovered_B = netG_A2B(fake_A)
    loss_cycle_BAB = criterion_cycle(recovered_B, real_B)*10.0

    # Total loss
    loss_G = loss_identity_A + loss_identity_B + loss_GAN_A2B + loss_GAN_B2A + loss_cycle_ABA + loss_cycle_BAB
    loss_G.backward()

    optimizer_G.step()

    ##### Discriminator A #####
    optimizer_D_A.zero_grad()

    # Real loss
    pred_real = netD_A(real_A)
    loss_D_real = criterion_GAN(pred_real, target_real)

    # Fake loss
    fake_A = fake_A_buffer.push_and_pop(fake_A)
    pred_fale = netD_A(fake_A.detach())
    loss_D_fake = criterion_GAN(pred_fake, target_fake)

    # Total loss
    loss_D_A = (loss_D_real + loss_D_fake)*0.5
    loss_D_A.backward()

I am not familiar at all what it means. My guess is it's something to do with fake_A_buffer. It's just a fake_A_buffer = ReplayBuffer()
class ReplayBuffer():
def __init__(self, max_size=50):
    assert (max_size > 0), 'Empty buffer or trying to create a black hole. Be careful.'
    self.max_size = max_size
    self.data = []

def push_and_pop(self, data):
    to_return = []
    for element in data.data:
        element = torch.unsqueeze(element, 0)
        if len(self.data) < self.max_size:
            self.data.append(element)
            to_return.append(element)
        else:
            if random.uniform(0,1) > 0.5:
                i = random.randint(0, self.max_size-1)
                to_return.append(self.data[i].clone())
                self.data[i] = element
            else:
                to_return.append(element)
    return Variable(torch.cat(to_return))

Error after setting `loss_G.backward(retain_graph=True)

Traceback (most recent call last):   File "models/CycleGANs/train",
line 150, in
loss_D_A.backward()   File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in
backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)   File
"/opt/conda/lib/python3.8/site-packages/torch/autograd/init.py",
line 130, in backward
Variable._execution_engine.run_backward( RuntimeError: one of the variables needed for gradient computation has been modified by an
inplace operation: [torch.FloatTensor [3, 64, 7, 7]] is at version 2;
expected version 1 instead. Hint: enable anomaly detection to find the
operation that failed to compute its gradient, with
torch.autograd.set_detect_anomaly(True).

And after setting torch.autograd.set_detect_anomaly(True)

/opt/conda/lib/python3.8/site-packages/torch/autograd/init.py:130:
UserWarning: Error detected in MkldnnConvolutionBackward. Traceback of
forward call that caused the error:
File "models/CycleGANs/train",
line 115, in
fake_B = netG_A2B(real_A)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/Histology-Style-Transfer-Research/models/CycleGANs/models.py",
line 67, in forward
return self.model(x)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py",
line 117, in forward
input = module(input)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/Histology-Style-Transfer-Research/models/CycleGANs/models.py",
line 19, in forward
return x + self.conv_block(x)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/container.py",
line 117, in forward
input = module(input)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py",
line 727, in _call_impl
result = self.forward(*input, **kwargs)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py",
line 423, in forward
return self._conv_forward(input, self.weight)
File "/opt/conda/lib/python3.8/site-packages/torch/nn/modules/conv.py",
line 419, in _conv_forward
return F.conv2d(input, weight, self.bias, self.stride,  (Triggered internally at
/opt/conda/conda-bld/pytorch_1603729096996/work/torch/csrc/autograd/python_anomaly_mode.cpp:104.)
Variable._execution_engine.run_backward(
Traceback (most recent call
last):   File "models/CycleGANs/train", line 133, in
loss_G.backward(retain_graph=True)
File "/opt/conda/lib/python3.8/site-packages/torch/tensor.py", line 221, in
backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File
"/opt/conda/lib/python3.8/site-packages/torch/autograd/init.py",
line 130, in backward
Variable._execution_engine.run_backward( RuntimeError: Function 'MkldnnConvolutionBackward' returned nan values in its 2th output.


title: Pytorch: backpropagating from sum of matrix elements to leaf variable

body:
I'm trying to understand backpropagation in pytorch a bit better. I have a code snippet that successfully does backpropagation from the output d to the leaf variable a, but then if I add in a reshape step, the backpropagation no longer gives the input a gradient.

I know reshape is out-of-place, but I'm still not sure how to contextualize this.

Any thoughts?

Thanks.

#Works
a = torch.tensor([1.])
a.requires_grad = True
b = torch.tensor([1.])
c = torch.cat([a,b])
d = torch.sum(c)
d.backward()

print('a gradient is')
print(a.grad) #=> Tensor([1.])

#Doesn't work
a = torch.tensor([1.])
a.requires_grad = True
a = a.reshape(a.shape)
b = torch.tensor([1.])
c = torch.cat([a,b])
d = torch.sum(c)
d.backward()

print('a gradient is')
print(a.grad) #=> None


title: Pytorch equivalent of `tf.reverse_sequence`?

body:
I would like to do backward-direction LSTM on a padded sequence, which requires reversing the input sequence without the padding.

For a batch like this (where _ stands for padding):

a b c _ _ _
d e f g _ _
h i j k l m

if would like to get:

c b a _ _ _
g f e d _ _
m l k j i h

TensorFlow has a function tf.reverse_sequence that takes the input tensor and lengths of the sequences in the batch and returns the reversed batch. Is there an easy way of doing it in Pytorch?


title: Convert list of two dimensional DataFrame to Torch Tensor

body:
Goal: I am working with RNNs in PyTorch, and my data is given by a list of DataFrames, where each DataFrame means one observation like:

import numpy as np
data = [pd.DataFrame(np.zeros((5,50))) for x in range(100)]

which means 100 observation, with 50 parameters and 5 timesteps each. For my Model i need a tensor of shape (100,5,50).

Issue: I tried a lot of things but nothing seems to work, does anyone know how this is done?
This approaches doesn't work:

import torch
torch.tensor(np.array(data))

I thing the problem is to convert the DataFrames into Arrays and the List into a Tensor at the same time.


title: Convert a torch t7 model to keras h5

body:
How can we convert a t7 model to keras' h5 ?
I am trying to do so for c3d-sports1m-kinetics.t7 that you can find here https://github.com/kenshohara/3D-ResNets/releases

The least I can ask for is a way to load the t7 model to python (pytorch) and then extract its weights, but I couldn't do it with the load_lua() function...

I get an error while trying to do it using this function https://github.com/pytorch/pytorch/blob/c6529f4851bb8ac95f05d3f17dea178a0367aaee/torch/utils/serialization/read_lua_file.py

The error that I get is the following :

Traceback (most recent call last):
File "convert_t7_to_hdf5.py", line 574, in <module>
    a = load_lua("model.t7")
  File "convert_t7_to_hdf5.py", line 571, in load_lua
    return reader.read()
  File "convert_t7_to_hdf5.py", line 542, in read
    typeidx = self.read_int()
  File "convert_t7_to_hdf5.py", line 440, in read_int
    return self._read('i')
  File "convert_t7_to_hdf5.py", line 431, in _read
    result = struct.unpack(fmt, self.f.read(sz))
ValueError: read of closed file


title: 'Sequential' object has no attribute 'features' while extracting vgg19 pytorch features

body:
I'm trying to extract the features of images using VGG19 network (the output should be of dim : [1 , 7 , 7 , 512] per frame
here is the code I have used :

    deep_net = models.vgg19(pretrained=True).cuda()
    deep_net = nn.Sequential(*list(deep_net.children())[:-2])
    deep_net.eval()
    save_file_sample_path = '/media/data1/out.npy'
    input_image = torch.zeros(1, 3, 224, 224)
    output_feat = np.zeros(shape=[1, 49, 512])
    with torch.no_grad():
      im = default_loader('/media/data1/images/frame612.jpg')
      im = transform(im)
      input_image[0, :, :] = im
      input_image = input_image.cuda()
      output_feat = deep_net(input_image)
      output_feat = output_feat.features[:-2].view(1, 512, 49).transpose(1, 2)

But I get the following error :

  AttributeError: 'Sequential' object has no attribute 'features'

At the line :

          output_feat = output_feat.features[:-2].view(1, 512, 49).transpose(1, 2)

Any idea why this does not work anymore? and how to fix?
 Thanks!


title: PyTorch 1.5.0 CUDA 10.2 installation via pip always installs CUDA 9.2

body:
I am currently installing an environment and need pytorch 1.5.0 with CUDA 10.2 . CUDA drivers are set up and all is fine, but the pytorch download via pip is broken.
The official doc for the previous versions says the installation should go as folllows:

CUDA 10.2
pip install torch==1.5.0 torchvision==0.6.0 -f https://download.pytorch.org/whl/torch_stable.html
CUDA 10.1
pip install torch==1.5.0+cu101 torchvision==0.6.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
CUDA 9.2
pip install torch==1.5.0+cu92 torchvision==0.6.0+cu92 -f https://download.pytorch.org/whl/torch_stable.html
CPU only
pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

(source: https://pytorch.org/get-started/previous-versions/)
but when i try to install the first one for CUDA 10.2 it installs the one for CUDA 9.2:
$ pip install torch==1.5.0 torchvision==0.6.0 -f https://download.pytorch.org/whl/torch_stable.html
 Looking in links: https://download.pytorch.org/whl/torch_stable.html
 Collecting torch==1.5.0
 Downloading https://download.pytorch.org/whl/cu92/torch-1.5.0%2Bcu92-cp36-cp36m-linux_x86_64.whl (603.7 MB)
      |████████████████████████████████| 603.7 MB 985 bytes/s
 Collecting torchvision==0.6.0
 Downloading  https://download.pytorch.org/whl/cu92/torchvision-0.6.0%2Bcu92-cp36-cp36m-linux_x86_64.whl (6.5 MB)
      |████████████████████████████████| 6.5 MB 600 kB/s
Requirement already satisfied: future in /home/---/site-packages (from torch==1.5.0) (0.18.2)
Requirement already satisfied: numpy in /home/---/python3.6/site-packages (from torch==1.5.0) (1.19.1)
Requirement already satisfied: pillow>=4.1.1 in /home/---/site-packages (from torchvision==0.6.0) (7.2.0) Installing collected packages: torch, torchvision
Successfully installed torch-1.5.0+cu92
 torchvision-0.6.0+cu92

So, it downloads and installs the wrong version. Explicitely adding +cu102 to the version like with the other version also doesnt work, since then it gives the error:
$ pip install torch==1.5.0+cu102 torchvision==0.6.0+cu102 -f https://download.pytorch.org/whl/torch_stable.html
Looking in links: https://download.pytorch.org/whl/torch_stable.html
ERROR: Could not find a version that satisfies the requirement torch==1.5.0+cu102 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2, 0.3.0.post4, 0.3.1, 0.4.0, 0.4.1, 1.0.0, 1.0.1, 1.0.1.post2, 1.1.0, 1.2.0, 1.2.0+cpu, 1.2.0+cu92, 1.3.0, 1.3.0+cpu, 1.3.0+cu100, 1.3.0+cu92, 1.3.1, 1.3.1+cpu, 1.3.1+cu100, 1.3.1+cu92, 1.4.0, 1.4.0+cpu, 1.4.0+cu100, 1.4.0+cu92, 1.5.0, 1.5.0+cpu, 1.5.0+cu101, 1.5.0+cu92, 1.5.1, 1.5.1+cpu, 1.5.1+cu101, 1.5.1+cu92, 1.6.0, 1.6.0+cpu, 1.6.0+cu101, 1.6.0+cu92)
ERROR: No matching distribution found for torch==1.5.0+cu102

Manually downloading the wheel and changing https://download.pytorch.org/whl/cu92/torch-1.5.0%2Bcu92-cp36-cp36m-linux_x86_64.whl by deleting the cu92 part or replacing it with cu102 doesnt work, either and results in an Error 403 from the pytorch server.
Sadly, I do rely on pip in this case and can't use the conda install command. Does anybody have a solution to this problem?


title: How can I select specific row in a torch Tensor

body:
I want to select specific row in a 3-dimension torch Tensor, just as below:
It could get random number of rows from every sub 2-dimension tensor, and I want to merge them together.
What I want is a 3-dimension tensor:

tensor([[[0.7185, 0.2953, 0.6841, 0.1045]],

        [[0.6817, 0.4053, 0.2318, 0.1309]],

        [[0.8265, 0.8029, 0.3165, 0.2020],
        [0.6041, 0.0118, 0.8386, 0.5076]],

        [[0.6985, 0.7313, 0.4613, 0.1862],
        [0.5678, 0.4485, 0.4514, 0.7747]]])

But here is the result 2-dimension:

tensor([[0.7185, 0.2953, 0.6841, 0.1045],
        [0.6817, 0.4053, 0.2318, 0.1309],
        [0.8265, 0.8029, 0.3165, 0.2020],
        [0.6041, 0.0118, 0.8386, 0.5076],
        [0.6985, 0.7313, 0.4613, 0.1862],
        [0.5678, 0.4485, 0.4514, 0.7747]])

The demo code:

import torch
a= torch.rand(4,4,4)
a
tensor([[[0.1227, 0.8073, 0.0308, 0.6210],
         [0.7185, 0.2953, 0.6841, 0.1045],
         [0.2089, 0.3731, 0.7066, 0.9211],
         [0.0326, 0.4471, 0.8805, 0.3516]],     1
--------------------------------------------------
        [[0.3817, 0.2368, 0.9351, 0.0448],
         [0.6817, 0.4053, 0.2318, 0.1309],
         [0.1490, 0.4178, 0.2769, 0.7073],
         [0.0593, 0.6327, 0.0792, 0.3341]],     2
--------------------------------------------------
        [[0.3492, 0.0924, 0.8318, 0.4404],
         [0.8265, 0.8029, 0.3165, 0.2020],
         [0.6041, 0.0118, 0.8386, 0.5076],
         [0.3121, 0.7751, 0.5351, 0.9866]],     3
--------------------------------------------------
        [[0.6985, 0.7313, 0.4613, 0.1862],
         [0.5678, 0.4485, 0.4514, 0.7747],
         [0.2221, 0.6104, 0.2327, 0.9274],
         [0.2359, 0.2159, 0.3979, 0.2519]]])    4
---------------------------------------------------
i = a[:,:,0] > 0.5
i
tensor([[0, 1, 0, 0],
        [0, 1, 0, 0],
        [0, 1, 1, 0],
        [1, 1, 0, 0]])

b = a[i]
b
tensor([[0.7185, 0.2953, 0.6841, 0.1045],
        [0.6817, 0.4053, 0.2318, 0.1309],
        [0.8265, 0.8029, 0.3165, 0.2020],
        [0.6041, 0.0118, 0.8386, 0.5076],
        [0.6985, 0.7313, 0.4613, 0.1862],
        [0.5678, 0.4485, 0.4514, 0.7747]])

=====================================================
What I want is as below:

tensor([[[0.7185, 0.2953, 0.6841, 0.1045]],        1
------------------------------------------------------
        [[0.6817, 0.4053, 0.2318, 0.1309]],        2
-------------------------------------------------------
        [[0.8265, 0.8029, 0.3165, 0.2020],
        [0.6041, 0.0118, 0.8386, 0.5076]],         3
-----------------------------------------------------
        [[0.6985, 0.7313, 0.4613, 0.1862],
        [0.5678, 0.4485, 0.4514, 0.7747]]])        4
----------------------------------------------------


title: Given 4 points, how to crop a quadrilateral from an image in pytorch/torchvision?

body:
Problem is pretty straightforward -

I want to crop a quadrilateral from an image in pytorch/torchvision. Given, I have four coordinates of the corners of this quadrilateral.

Please note that these four points confine a quadrilateral within themselves which may or may not be a rectangle. So please refrain from suggesting answers involving slicing of the image.

Please comment if I have missed any relevant detail.


title: Equivalent of Keras's binary_crossentropy in PyTorch?

body:
I want to port some code from keras to pytorch, but I cann't find equivalent of Keras's binary_crossentropy in PyTorch. PyTorch's binary_cross_entropy has different behavior with keras's.

import torch
import torch.nn.functional as F
input = torch.tensor([[ 0.6845,  0.2454],
                      [ 0.7186,  0.3710],
                      [ 0.3480,  0.3374]])
target = torch.tensor([[ 0.,  1.],
                       [ 1.,  1.],
                       [ 1.,  1.]])
F.binary_cross_entropy(input, target, reduce=False)
#tensor([[ 1.1536,  1.4049],
#    [ 0.3305,  0.9916],
#    [ 1.0556,  1.0865]])
import keras.backend as K
K.eval(K.binary_crossentropy(K.variable(input.detach().numpy()), K.variable(target.detach().numpy())))

 #[[11.032836 12.030124]
 #[ 4.486187 10.02776 ]
 #[10.394435 10.563424]]

Is there anyone know why these two results are different? thanks!


title: Adapting pytorch softmax function

body:
I am currently looking into the softmax function and I would like to adapt the orignally implemented for ome small tests.

I have been to the docs but there wasn't that much of usefull information about the function. This is the pytorch python implementation:

def __init__(self, dim=None):
    super(Softmax, self).__init__()
    self.dim = dim

def __setstate__(self, state):
    self.__dict__.update(state)
    if not hasattr(self, 'dim'):
        self.dim = None

def forward(self, input):
    return F.softmax(input, self.dim, _stacklevel=5)

Where can I find the F.softmax impementation?

One off the things I want to try for instance is the soft-margin softmax described here: Soft-Margin Softmax for Deep Classification

Where would be the best place to start?
Thanks in advance!


title: pytorch lstm tutorial initializing Variable

body:
I am going through the pytorch tutorial for lstm and here's the code they use:

lstm = nn.LSTM(3, 3)  # Input dim is 3, output dim is 3
inputs = [autograd.Variable(torch.randn((1, 3)))
          for _ in range(5)]  # make a sequence of length 5

# initialize the hidden state.
hidden = (autograd.Variable(torch.randn(1, 1, 3)),
          autograd.Variable(torch.randn((1, 1, 3))))
for i in inputs:
    # Step through the sequence one element at a time.
    # after each step, hidden contains the hidden state.
    out, hidden = lstm(i.view(1, 1, -1), hidden)

For variable hidden, it initializes into a tuple and the result is:

(Variable containing:
(0 ,.,.) =
  0.4251 -1.2328 -0.6195
[torch.FloatTensor of size 1x1x3]
, Variable containing:
(0 ,.,.) =
  1.5133  1.9954 -0.6585
[torch.FloatTensor of size 1x1x3]
)

What I don't understand is

Is (0, ., .) an index? And shouldn't it initialize all three numbers since we said (torch.randn(1,1,3))?
What is the difference between torch.randn(1, 1, 3) and torch.randn((1,1,3))?


title: PyTorch: How to do inference in batches (inference in parallel)

body:
How to do inference in batches in PyTorch? How to do inference in parallel to speed up that part of the code.
I've started with the standard way of doing inference:
with torch.no_grad():
    for inputs, labels in dataloader['predict']:
        inputs = inputs.to(device)
        output = model(inputs)
        output = output.to(device)

And I've researched and the only mention of doing inference in parallel (in the same machine) seems to be with the library Dask: https://examples.dask.org/machine-learning/torch-prediction.html
Currently attempting to understand that library and create a working example. In the meanwhile do you know of a better way?


title: Why separable convolution implemented by tensorflow is slower than the normal convolution?

body:
I have tested the speed of separable_conv2d and normal conv2d implemented in TF, it seems the only depthwise_conv2d is faster than the normal conv2d but the performance of the dw_conv2d is poor obviously.

The separable_conv2d mentioned in MobileNet, its FLOPs is 1/9 of the normal conv when the kernel_size=3, but considering the Memory Access Cost the separable one cannot be 9 times faster than the normal one but in my experiment, the separable one is too much slower.

I modeling the experiment like this separable_conv2d is too slow. In this experiment, the separable_conv2d seems faster than the normal one when the depth_multiply=1, but when I use tf.nn to implement it as follows:

IMAGE_SIZE= 512
REPEAT = 100
KERNEL_SIZE = 3
data_format = 'NCHW'
#CHANNELS_BATCH_SIZE =  2048 # channe# ls * batch_size
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"

def normal_layers(inputs, nfilter, name=''):
    with tf.variable_scope(name, reuse=tf.AUTO_REUSE):
         shape = inputs.shape.as_list()
         in_channels = shape[1]
         filter = tf.get_variable(initializer=tf.initializers.random_normal,
                                  shape=[KERNEL_SIZE, KERNEL_SIZE,
                                   in_channels, nfilter], name='weight')
         conv = tf.nn.conv2d(input= inputs, filter=filter, strides=
                           [1,1,1,1],padding='SAME',data_format=data_format,
                           name='conv')
    return conv
def sep_layers(inputs, nfilter, name=''):
    with tf.variable_scope(name, reuse=tf.AUTO_REUSE):
         shape= inputs.shape.as_list()
         in_channels = shape[1]
         dw_filter=
         tf.get_variable(initializer=tf.initializers.random_normal,
                         shape=[KERNEL_SIZE, KERNEL_SIZE,
                         in_channels, 1], name='dw_weight')

         pw_filter =
         tf.get_variable(initializer=tf.initializers.random_normal,
                                shape=[1,1,in_channels, nfilter],
                         name='pw_weight')
         conv = tf.nn.depthwise_conv2d_native(input=inputs,
                                              filter=dw_filter,
                                              strides=[1,1,1,1],
                                              padding='SAME',
                                              data_format=data_format)
         conv = tf.nn.conv2d(input=conv,
                             filter=pw_filter,
                             strides=[1,1,1,1],
                             padding='SAME',
                             data_format=data_format)
    return conv

Each layer run in 100 times,
different from the link is I set the batch_size as a constant 10,
and the channels is in [32, 64, 128],
inputs is[batch_size, channels, img_size, img_size]
and the duration of them as follows:

Channels: 32
Normal Conv 0.7769527435302734s, Sep Conv 1.4197885990142822s
Channels: 64
Normal Conv 0.8963277339935303s, Sep Conv 1.5703468322753906s
Channels: 128
Normal Conv 0.9741833209991455s, Sep Conv 1.665834665298462s

It seems that when batch_size is a constant only change channel the time cost of the normal one and the separable one is growing gradually.

And when setting batch_size * channels as a constant
Inputs shape as [CHANNELS_BATCH_SIZE // channels, channels, imgsize, imgsize]

Channels: 32
Normal Conv 0.871959924697876s, Sep Conv 1.569300651550293s
Channels: 64
Normal Conv 0.909860372543335s, Sep Conv 1.604109525680542s
Channels: 128
Normal Conv 0.9196009635925293s, Sep Conv 1.6144189834594727s

What confuses me is the result is different from the result of the link above: the time cost of sep_conv2d does not have obvious change.

My Questions are:

What makes it different between the link above and the experiment by myself?
I am a newbie, so is there something wrong in my code to implement the separable_conv2d?
How to implement the separable_conv2d can be faster than the normal one in TF or in Pytorch?

Any help would be appreciated. Thank in advance.


title: How to load BertforSequenceClassification models weights into BertforTokenClassification model?

body:
Initially, I have a fine-tuned BERT base cased model using a text classification dataset and I have used BertforSequenceClassification class for this.

from transformers import BertForSequenceClassification, AdamW, BertConfig

# Load BertForSequenceClassification, the pretrained BERT model with a single
# linear classification layer on top.
model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased", # Use the 12-layer BERT model, with an uncased vocab.
    num_labels = 2, # The number of output labels--2 for binary classification.
                    # You can increase this for multi-class tasks.
    output_attentions = False, # Whether the model returns attentions weights.
    output_hidden_states = False, # Whether the model returns all hidden-states.
)

Now I want to use this fine-tuned BERT model weights for Named Entity Recognition and I have to use BertforTokenClassification class for this. I'm unable to figure out how to load the fine-tuned BERT model weights into the new model created using BertforTokenClassification.

Thanks in advance.......................


title: Normalization (Feature scaling) of Point Cloud Dataset

body:
I have point cloud data set where single data is represented by N * 3 where N is number of points. Similarly I have "M" number of points clouds in Dataset. The range of these point clouds varies largely. Some have very large values (e.g., in term of 10^6 for all N points) while some has very small values (e.g., in term of 10^1 for all N points). I want to normalize each point cloud. How should i do that?

Q1. Should I normalize(min-max) each point cloud (single point cloud N*3) individually along x, y, z dimension by choosing min and max from this point cloud only. In this scenario, for all "M" point clouds we have different min-max. The same is done for output  point cloud. Please view the image for more understanding https://i.stack.imgur.com/tKauw.jpg

Q2. Or Should I normalize (min-max) all point clouds along x, y, z dimension by choosing min and max (from M * N *3 in x, y, z columns) from this entire data set only. In this scenario, for all "m" point clouds we have same min-max. Please view the image for more understanding  https://i.stack.imgur.com/0HAhn.jpg


title: Anchor boxes and offsets in SSD object detection

body:
How do you calculate anchor box offsets for object detection in SSD? As far as I understood anchor boxes are the boxes in 8x8 feature map, 4x4 feature map, or any other feature map in the output layer.

So what are the offsets?

Is it the  distance between the centre of the bounding box and the centre of a particular box in say a 4x4 feature map?

If I am using a 4x4 feature map as my output, then my output should be of the dimension:

(4x4, n_classes + 4)

where 4 is for my anchor box co-ordinates.
This 4 co-ordinates can be something like:

(xmin, xmax, ymin, ymax)

This will correspond to the top-left and bottom-right corners of the bounding box.
So why do we need offsets and if so how do we calculate them?

Any help would be really appreciated!


title: pytorch runs much slower than tensorflow even for a shallow CNN

body:
The input data is an EEG data which was converted into PSD. And its shape is [103600, 59, 51], where 103600 is the number of samples, i.e. the total samples for an epoch. The data was loaded into memory.

And the model I used is very simple, however, the time required for training an epoch was 50s with pytorch 0.4 and a GTX 1080. However, the time for val set interface was 0.3s.

I also tested this model using tensorflow 1.2.0 the time was 8s. I have no idea if I got it wrong.

So I made some test. This img shows the time for runing.

where
time_to_tensor is the time for inputs, targets = torch.tensor(inputs, dtype=torch.float), torch.tensor(targets, dtype=torch.int64)

time_to_cuda for inputs, targets = inputs.to(self.device), targets.to(self.device)

time_to_model for out = self.model(inputs)

time_criterion for loss = self.criterion(out, targets)

time_backward for self.optimizer.step()

It seems the operation for tensor.to(self.device) (or tensor.cuda()) wasted much time. So I tried to move the data to GPU at once, using

self.X_train = torch.tensor(self.X_train, dtype=torch.float).to(self.device)
self.y_train = torch.tensor(self.y_train, dtype=torch.int64).to(self.device)

and commented

inputs, targets = torch.tensor(inputs, dtype=torch.float), torch.tensor(targets, dtype=torch.int64)
inputs, targets = inputs.to(self.device), targets.to(self.device)

However I got the running time in the following img.

The time required for an epoch still was about 50s, and time for each operation changed. I am confused.

Could someone help me to find the problem? Thanks.

Code was here.

It's the model I used

import torch
import torch.nn as nn
import torch.nn.functional as F

class TxtCNN4(nn.Module):

    def __init__(self, n_classes=6, time_steps=59, psd_lenght=51, filter_numbers=128):
        super(TxtCNN4, self).__init__()
        self.n_classes = n_classes
        self.time_steps = time_steps
        self.psd_lenght = psd_lenght
        self.filters = filter_numbers

        self.conv1 = nn.Conv2d(in_channels=1, out_channels=self.filters//2,
                                kernel_size=(3, psd_lenght), stride=1, padding=0, bias=False)
        self.bn1 = nn.BatchNorm2d(self.filters//2, momentum=0.05)
        self.pool1 = nn.MaxPool2d(kernel_size=(self.time_steps - 3 + 1, 1))

        self.conv2 = nn.Conv2d(in_channels=self.filters//2, out_channels=self.filters,
                                    kernel_size=(3, 1), stride=1, padding=0, bias=False)
        self.bn2 = nn.BatchNorm2d(self.filters, momentum=0.05)
        self.pool2 = nn.MaxPool2d(kernel_size=(self.time_steps - 5 + 1, 1))

        self.conv3 = nn.Conv2d(in_channels=self.filters, out_channels=self.filters,
                                    kernel_size=(3, 1), stride=1, padding=0, bias=False)
        self.bn3 = nn.BatchNorm2d(self.filters, momentum=0.05)
        self.pool3 = nn.MaxPool2d(kernel_size=(self.time_steps - 7 + 1, 1))

        self.fc = nn.Linear(in_features=2*self.filters+self.filters//2, out_features=128)
        self.cls = nn.Linear(in_features=128, out_features=self.n_classes)
        self.criterion = nn.CrossEntropyLoss(weight=None)

    def forward(self, inputs):
        inputs = inputs.view(-1, self.time_steps, self.psd_lenght, 1)
        inputs = inputs.permute(0, 3, 1, 2)
        conv1 = F.relu(self.bn1(self.conv1(inputs)), inplace=True)
        conv2 = F.relu(self.bn2(self.conv2(conv1)), inplace=True)
        conv3 = F.relu(self.bn3(self.conv3(conv2)), inplace=True)

        x = torch.cat((self.pool1(conv1), self.pool2(conv2), self.pool3(conv3)), dim=1)
        x = x.view(-1, 2*self.filters+self.filters//2)
        x = F.dropout(x, p=0.5, training=self.training)
        x = F.relu(self.fc(x), inplace=True)
        x = self.cls(x)

        return x

It's the main python file for traning the model

file_path = '/mnt/disk2/wy/SLEEP_EDF/'

batch_size = 32*4*2
dropout_rate = 0.5
nb_classes = 5
max_epochs = 100
early_stop_epoch = 10

learning_rate = 1e-3

model_type='TxtCNN4'
filter_numbers = 128

fs = 100    # Sample frequence
n_Channels = 2
n_Samples = 30*100

load_flag = False
model_path = ''

def iterate_minibatches(inputs, targets, batchsize, shuffle=False):
    input_len = inputs.shape[0]
    assert input_len == len(targets)

    if shuffle:
        indices = np.arange(input_len)
        np.random.shuffle(indices)
    for start_idx in range(0, input_len, batchsize):
        if shuffle:
            excerpt = indices[start_idx:start_idx + batchsize]
        else:
            excerpt = slice(start_idx, start_idx + batchsize)

class Trainer():

    def __init__(self, X_inputs, labels, fold, subj_id, log_path, model_type=model_type, args=None):
        self.subj_id = subj_id
        self.model_type = model_type
        self.writer_root_path = os.path.abspath(
                                os.path.join(
                                    os.path.curdir,
                                    log_path+'_'+model_type,
                                    model_type + '_' + str(subj_id)))
        self.train_writer = SummaryWriter(os.path.join(self.writer_root_path, 'train'))
        self.val_writer = SummaryWriter(os.path.join(self.writer_root_path, 'val'))
        self.test_writer = SummaryWriter(os.path.join(self.writer_root_path, 'test'))

        self.curr_epcoh = 0
        self.curr_iter = 0

        self.train_end_flag = False

        (self.X_train, self.y_train), (self.X_val, self.y_val), (self.X_test, self.y_test) = reformatInput(X_inputs, labels, fold)
        print('Test set label and BiLi:\t', np.unique(self.y_test, return_counts=True))
        # normalization between all data
        X_mean = self.X_train.mean()
        X_std = self.X_train.std()
        self.X_train = (self.X_train - X_mean)/X_std
        self.X_val = (self.X_val - X_mean)/X_std
        self.X_test = (self.X_test - X_mean)/X_std

        if model_type == 'TxtCNN4':
            model = TxtCNN4()
        else:
            model = DeepSleepNet()
        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
        self.model = model.to(self.device)
        self.criterion = nn.CrossEntropyLoss(weight=None)

        # self.X_train = torch.tensor(self.X_train, dtype=torch.float).to(self.device)
        # self.y_train = torch.tensor(self.y_train, dtype=torch.int64).to(self.device)

        self.optimizer = optim.Adam(self.model.parameters(), lr=learning_rate)

    def train(self, epoch):
        train_loss = AvgMeter()
        train_acc = AvgMeter()
        start_time = time.time()

        time_to_tensor = 0
        time_to_cuda = 0
        time_model = 0
        time_criterion = 0
        time_backward = 0
        time_step = 0
        time_writer = 0
        self.curr_epcoh = epoch
        self.model.train()
        for (inputs, targets) in iterate_minibatches(self.X_train, self.y_train, batch_size, shuffle=False):
            self.curr_iter += 1
            time_1 = time.time()
            inputs, targets = torch.tensor(inputs, dtype=torch.float), torch.tensor(targets, dtype=torch.int64)  # dtype=torch.int64
            time_to_tensor += (time.time() - time_1)
            time_1 = time.time()
            inputs, targets = inputs.to(self.device), targets.to(self.device)
            time_to_cuda += (time.time() - time_1)

            # zero the parameter gradients
            self.optimizer.zero_grad()

            # forward + backward + optimize
            time_1 = time.time()
            out = self.model(inputs)
            time_model += (time.time() - time_1)

            time_1 = time.time()
            loss = self.criterion(out, targets)
            time_criterion += (time.time() - time_1)

            time_1 = time.time()
            loss.backward()
            time_backward += (time.time() - time_1)

            time_1 = time.time()
            self.optimizer.step()
            time_step += (time.time() - time_1)

            if self.curr_iter % 20 == 0:
                time_1 = time.time()
                pred = out.max(dim=1)[1].cpu().numpy()
                gt = targets.cpu().numpy()
                acc = np.mean(pred==gt)
                _loss = loss.detach().cpu().numpy()
                train_acc.update(acc)
                train_loss.update(_loss)

                self.train_writer.add_scalar('learning_rate', curr_lr, self.curr_iter)
                self.train_writer.add_scalar('acc', acc, self.curr_iter)
                self.train_writer.add_scalar('loss', _loss, self.curr_iter)
                time_writer += (time.time() - time_1)

        fmt_str = "Train\tEpoch [{:d}/{:d}]  train_Loss: {:.4f}\ttrain_Acc: {:.2f}\tTime per Epoch: {:.4f}"
        print_str = fmt_str.format(self.curr_epcoh,
                                    max_epochs,
                                    train_loss.avg,
                                    train_acc.avg*100,
                                    time.time()-start_time)
        print(print_str)
        print('time_to_tensor: ',time_to_tensor)
        print('time_to_cuda: ',time_to_cuda)
        print('time_model: ',time_model)
        print('time_criterion: ',time_criterion)
        print('time_backward: ',time_backward)
        print('time_step: ',time_step)
        print('time_writer: ',time_writer)
        print(len(self.y_train))
        train_acc.reset()
        train_loss.reset()

    def validation(self):
        self.model.eval()
        with torch.no_grad():
            # val set
            loss_gather = AvgMeter()
            pred_gather = PreGather()
            tran_time = 0
            for (inputs, targets) in iterate_minibatches(self.X_val, self.y_val, batch_size, shuffle=False):
                time_1 = time.time()
                inputs, targets = torch.tensor(inputs, dtype=torch.float), torch.tensor(targets, dtype=torch.int64)  # dtype=torch.int64
                inputs, targets = inputs.to(self.device), targets.to(self.device)
                tran_time += (time.time() - time_1)

                out = self.model(inputs)
                _loss = self.criterion(out, targets).detach().cpu().numpy()
                pred = out.max(dim=1)[1].cpu().numpy()
                loss_gather.update(_loss)
                pred_gather.update(pred)

            val_loss = loss_gather.avg
            pred = pred_gather.pred
            val_acc = np.mean(pred==self.y_val)
            val_kappa = cohen_kappa_score(self.y_val, pred)
            val_BCA = recall_score(self.y_val, pred, average='macro')
            loss_gather.reset()
            pred_gather.reset()

            # Then we print the results for this epoch:
            fmt_str = "VAL \tEpoch [{:d}/{:d}]  val_Loss: {:.4f}\tval_Acc: {:.2f}\tval_kappa: {:.2f}\tval_BCA: {:.2f}"
            print_str = fmt_str.format(self.curr_epcoh,
                                        max_epochs,
                                        val_loss,
                                        val_acc*100,
                                        val_kappa*100,
                                        val_BCA*100)
            print(print_str)
            print(tran_time)
            print(len(self.y_val))
            self.val_writer.add_scalar('acc', val_acc, self.curr_iter)
            self.val_writer.add_scalar('loss', val_loss, self.curr_iter)
            self.val_writer.add_scalar('kappa', val_kappa, self.curr_iter)
            self.val_writer.add_scalar('bca', val_BCA, self.curr_iter)

def train_all_subject(num_epochs=max_epochs, log_path=None):
    # Leave-Subject-Out cross validation
    subj_nums, fold_pairs, EEGs, labels = load_data(file_path, subj_nums=38, channel=1)
    for subj_id in range(subj_nums):
        print('The subj_id', subj_id, '\t\t Training the ' + model_type + ' Model...')
        trainer = Trainer(EEGs, labels, fold_pairs[subj_id], subj_id, log_path)
        for epoch in range(trainer.curr_epcoh, max_epochs):
            if trainer.train_end_flag is False:
                trainer.train(epoch)
                trainer.validation()
                print('-'*50)
            else:
                break


title: Maxpool of an image in pytorch

body:
I'm trying to just apply maxpool2d (from torch.nn) on a single image (not as a maxpool layer). Here is my code right now:

name = 'astronaut'
imshow(images[name], name)
img = images[name]
# pool of square window of size=3, stride=1
m = nn.MaxPool2d(3,stride = 1)
img_transform = torch.Tensor(images[name])
plt.imshow(m(img_transform).view((512,510)))

The issue is, this code gives me a very green image as a result. I am sure the problem is with the dimensions of view, but I was unable to find how to apply maxpool to just one image so I couldn't fix it. The dimension of the image I'm considering is 512x512. The arguments for view make no sense for me right now, it's just the only number that gives a result...

If for example, I gave 512,512 as the argument for view, I get the following error:

RuntimeError: shape '[512, 512]' is invalid for input of size 261120

If anyone can tell me how to apply maxpool, avgpool, or minpool to an image and display the result I would be super grateful!

Thanks (:


title: Extract features from pretrained resnet50 in pytorch

body:
Hy guys, i want to extract the in_features of Fully connected layer of my pretrained resnet50.

I create before a method that give me the vector of features:

def get_vector(image):

#layer = model._modules.get('fc')

layer = model.fc
my_embedding = torch.zeros(2048) #2048 is the in_features of FC , output of avgpool

def copy_data(m, i, o):

    my_embedding.copy_(o.data)

h = layer.register_forward_hook(copy_data)
tmp = model(image)

h.remove()

# return the vector
return my_embedding

after I call this method here:

column = ["FlickrID", "Features"]

path = "./train_dataset/train_imgs/"

pathCSV = "./train_dataset/features/img_info_TRAIN.csv"

f_id=[]
features_extr=[]

df = pd.DataFrame(columns=column)

tr=transforms.Compose([transforms.Resize(256),
                       transforms.CenterCrop(224),
                       transforms.ToTensor(),
                       transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])])

test = Dataset(path, pathCSV, transform=tr)

test_loader = DataLoader(test, batch_size=1, num_workers=2, shuffle = False)

#Leggiamo le immagini
for batch in test_loader:
    nome = batch['FlickrID']
    f_id.append(nome)
    image = batch['image']

    #print(image)
    with torch.no_grad():
        pred = get_vector(image)

    features_extr.append(pred)

df["FlickrID"] = f_id
df["Features"] = features_extr

df.to_hdf("Places.h5", key='df', mode='w')

I have an error like this:
output with shape [2048] doesn't match the broadcast shape [1, 2048, 1, 2048]

How can I take the in_feature of Fully Connected of this resnet50?
The Dataset is a customized Dataset class.

Sorry for my bad english


title: Error Utilizing Pytorch Transforms and Custom Dataset

body:
This question mainly concerns the return value of __getitem__ in a pytorch Dataset which I've seen as both a tuple and a dict in the source code.

I have been following this tutorial for creating a dataset class within my code, which is following this tutorial on transfer learning. It has the following definition of a dataset.

class FaceLandmarksDataset(Dataset):
"""Face Landmarks dataset."""

    def __init__(self, csv_file, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.landmarks_frame = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.landmarks_frame)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir,
                                self.landmarks_frame.iloc[idx, 0])
        image = io.imread(img_name)
        landmarks = self.landmarks_frame.iloc[idx, 1:].as_matrix()
        landmarks = landmarks.astype('float').reshape(-1, 2)
        sample = {'image': image, 'landmarks': landmarks}

        if self.transform:
            sample = self.transform(sample)

        return sample

As you can see, __getitem__ returns a dictionary with two entries.
In the transfer learning tutorial, the following calls are made to transform a dataset:

    data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = 'hymenoptera_data'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

use_gpu = torch.cuda.is_available()

inputs, classes = next(iter(dataloaders['train']))

That last line of code causes an error in my code by attempting to run transform on a sample in my custom dataset.

  'dict' object has no attribute 'size'

But if the tutorial dataset is implemented correctly, shouldn't it function correctly with a transform? My own hybrid implementation is below:

import torch
import torch.nn as nn
import torch.optim as optim
from torch.optim import lr_scheduler
from torch.autograd import Variable
import numpy as np
import torchvision
from torchvision import datasets, models, transforms
import matplotlib.pyplot as plt
import time
import os
import copy
from torch.utils.data import *
from skimage import io, transform
plt.ion()

class NumsDataset(Dataset):
    """Face Landmarks dataset."""

    def __init__(self, root_dir, transform=None):
        """
        Args:
            csv_file (string): Path to the csv file with annotations.
            root_dir (string): Directory with all the images.
            transform (callable, optional): Optional transform to be applied
                on a sample.
        """
        self.docs = []
        for file in os.listdir(root_dir):
            #print(file)
            if file.endswith(".txt"):
                path = os.path.join(root_dir, file)
                with open(path, 'r') as f:
                    self.docs.append( (  file , list(f.read()) ) ) #tup containing file, image values pairs
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self): #returns number of images
        i = 0
        for j in self.docs:
            i += len(j[1])
        return i

    def len2(self): #returns number of batches
        return len(self.docs)

    def __getitem__(self, idx):
        idx1 = idx // self.len2()
        idx2 = idx % self.len2()
        imglabel = self.docs[idx1][0] #label with filename for batch error calculation later
        imgdir = os.path.join(self.root_dir, self.docs[idx1][0].strip(".txt"))
        img = None
        l = idx2

        for file in os.listdir(imgdir):
            file = os.path.join(imgdir, file)
            if(l == 0):
                img = io.imread(file)
            l -= 1
        sample = (img , imglabel)
        sample ={'image': img, 'label': imglabel}
        if self.transform:
            sample = self.transform(sample)

        return sample

data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}
data_dir = "images"
image_datasets = {x: NumsDataset(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=5)
              for x in ['train', 'val']}

dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = ["one", "two", "four"]

use_gpu = torch.cuda.is_available()
# Get a batch of training data
inputs, classes = next(iter(dataloaders['train']))

directory structure:

images
     /train
        /file1
            *.jpg
        /file2...
            *.jpg
        file1.txt
        file2.txt...
     /val
        /file1
            *.jpg
        /file2...
            *.jpg
        file1.txt
        file2.txt...

Is the sample I'm returning formatted incorrectly?


title: Prepare Decoder of a Sequence to Sequence Network in PyTorch

body:
I was working with Sequence to Sequence models in Pytorch. Sequence to Sequence Models comprises of an Encoder and a Decoder.

The Encoder convert a (batch_size X input_features X num_of_one_hot_encoded_classes) -> (batch_size X input_features X hidden_size)

The Decoder will take this input sequence and convert it into (batch_size X output_features X num_of_one_hot_encoded_classes)

An example would be like-

So on the above example, I would need to convert the 22 input features to 10 output features. In Keras it could be done with a RepeatVector(10).

An Example -

model.add(LSTM(256, input_shape=(22, 98)))
model.add(RepeatVector(10))
model.add(Dropout(0.3))
model.add(LSTM(256, return_sequences=True))

Although, I'm not sure if it's the proper way to convert the input sequences into the output ones.

So, my question is -

What's the standard way to convert the input sequences to
output ones. eg. converting from (batch_size, 22, 98) -> (batch_size,
10, 98)? Or how should I prepare the Decoder?

Encoder Code snippet (Written in Pytorch) -

class EncoderRNN(nn.Module):
    def __init__(self, input_size, hidden_size):
        super(EncoderRNN, self).__init__()
        self.hidden_size = hidden_size
        self.lstm = nn.LSTM(input_size=input_size, hidden_size=hidden_size,
          num_layers=1, batch_first=True)

    def forward(self, input):
        output, hidden = self.lstm(input)
        return output, hidden


title: Pytorch Batchnorm layer different from Keras Batchnorm

body:
I'm trying to copy pre-trained BN weights from a pytorch model to its equivalent Keras model but I keep getting different outputs.

I read Keras and Pytorch BN documentation and I think that the difference lies in the way they calculate the "mean" and "var".

Pytorch:

  The mean and standard-deviation are calculated per-dimension over the
  mini-batches

source: Pytorch BatchNorm

Thus, they average over samples.

Keras:

  axis: Integer, the axis that should be normalized (typically the
  features axis). For instance, after a Conv2D layer with
  data_format="channels_first", set axis=1 in BatchNormalization.

source: Keras BatchNorm

and here they average over the features (channels)

What's the right way? How to transfer BN weights between the models?


title: How to do cubic spline interpolation and integration in Pytorch

body: In Pytorch, is there cubic spline interpolation similar to Scipy's? Given 1D input tensors x and y, I want to interpolate through those points and evaluate them at xs to obtain ys. Also, I want an integrator function that finds Ys, the integral of the spline interpolation from x[0] to xs.


title: How to normalize PIL image between -1 and 1 in pytorch for transforms.Compose?

body:
So I have been trying to find a way to normalize some PIL image pixel values between -1 and 1. I searched through documentation and didn't find solution. Only normalization in documentation is transforms.Normalize which normalizes with mean and std. So I am stuck on how to do it. This is my code:

train_transform = transforms.Compose([
        transforms.RandomHorizontalFlip(p=0.5),
        transforms.Resize(40),
        transforms.RandomCrop(32),
        # Normalize(-1, 1)  # Something like that
])

How can I normalize tensor between 2 numbers? For example:

[[1,2,3]
[3,2,1]]

Between -1 and 1

[[-1, 0, 1],
[1, 0, -1]]


title: How to properly implement 1D CNN for numerical data in PyTorch?

body:
I have a 500x2000 matrix, where each row represents an individual and each column is a measurement of some particular quality about that individual. I'm using a batch size of 64, so the input for each cycle of the network is actually a 64x2000 matrix. I'm trying to build a CNN in PyTorch to classify individuals given a set of these measurements. However, I've stumbled on the parameters for the convolutional layer.

Below is my current definition for a simple convolutional neural network.

class CNNnet(nn.Module)
    def __init__(self):
        self.conv1 = nn.Conv1d(2000, 200, (1,2), stride=10)
        self.pool = nn.MaxPool1d(kernel_size = (1, 2), stride = 2)

        self.fc1 = nn.Linear(64, 30)
        self.fc2 = nn.Linear(30, 7)

    def forward(self, x):
        x = x.view(64, 2000, 1)
        x = F.relu(self.conv1(x))
        x = self.pool(x)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

Attempting to train this model produces the following error:

  "RuntimeError: Expected 4-dimensional input for 4-dimensional weight
  200 2000 1 2, but got 3-dimensional input of size [64, 2000, 1]
  instead".

I'm confused on why it's expecting a 4D 200x2000x1x2 matrix (shouldn't the number of output channels be irrelevant to the input? And why is there a 2 at the end?).

My question is what would be the proper syntax or approach for writing a CNN (specifically the convolutional layer) when dealing with 1D data. Any help is greatly appreciated.


title: Error implementing torch.optim.lr_scheduler.LambdaLR in Pytorch

body:
I'm working on an Image classifier and trying to implement Cyclical Learning Rates to have a better results. I'm using lr_scheduler.LambdaLR to adjust the learning rate during training, but I'm having an error that I'm not sure what is the cause of it.
this is my code:

lr_find_epochs = 2
start_lr = 1e-7
end_lr = 0.1
# Set up the model, optimizer and loss function for the experiment

optimizer = torch.optim.SGD(model.parameters(), start_lr)
criterion = nn.NLLLoss()
# LR function lambda

lr_lambda = lambda x: math.exp(x * math.log(end_lr / start_lr) / (lr_find_epochs * len( train_loader)))
scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lr_lambda)

and this is the error I'm having:

The error in lr_scheduler.py.


title: Pytorch batch indexing

body:
So the output of my network looks like this:

output = tensor([[[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.0584, -0.2549],
     [ 0.0482, -0.2386],
     [ 0.0410, -0.2234],
     [ 0.0362, -0.2111],
     [ 0.0333, -0.2018],
     [ 0.0318, -0.1951],
     [ 0.0311, -0.1904],
     [ 0.0310, -0.1873],
     [ 0.0312, -0.1851],
     [ 0.0315, -0.1837],
     [ 0.0318, -0.1828],
     [ 0.0322, -0.1822],
     [ 0.0324, -0.1819],
     [ 0.0327, -0.1817],
     [ 0.0328, -0.1815],
     [ 0.0330, -0.1815],
     [ 0.0331, -0.1814],
     [ 0.0332, -0.1814],
     [ 0.0333, -0.1814],
     [ 0.0333, -0.1814],
     [ 0.0334, -0.1814],
     [ 0.0334, -0.1814],
     [ 0.0334, -0.1814]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.0584, -0.2549],
     [ 0.0482, -0.2386],
     [ 0.0410, -0.2234],
     [ 0.0362, -0.2111],
     [ 0.0333, -0.2018],
     [ 0.0318, -0.1951],
     [ 0.0311, -0.1904],
     [ 0.0310, -0.1873],
     [ 0.0312, -0.1851],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.0584, -0.2549],
     [ 0.0482, -0.2386],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.0584, -0.2549],
     [ 0.0482, -0.2386],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.0716, -0.2668],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]],

    [[ 0.0868, -0.2623],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164],
     [ 0.1003, -0.2164]]])

Which is a shape of [8, 24, 2]
Now 8 is my batch size. And i would like to get a data point from every batch, at the following locations:

index = tensor([24, 10,  3,  3,  1,  1,  1,  0])

So the 24th value from the first batch, the 10th value from the second batch, and so on.

Now i have problems figuring out the syntax.
I've tried

torch.gather(output, 0, index)

But it keeps telling me, that my dimensions don't match.
And trying

output[ : ,index]

Just gets me the values at all the indexes for each batch.
What would be the correct syntax here, to get these values?


title: PyTorch Softmax Output Doesn't Sum to 1

body:
Cross posting my question from the PyTorch forum:

I started receiving negative KL divergences between a target Dirichlet distribution and my model’s output Dirichlet distribution. Someone online suggested that this might be indicative that the parameters of the Dirichlet distribution don’t sum to 1. I thought this was ridiculous since the output of the model is passed through

output = F.softmax(self.weights(x), dim=1)

But after looking into it more closely, I found that torch.all(torch.sum(output, dim=1) == 1.) returns False! Looking at the problematic row, I see that it is tensor([0.0085, 0.9052, 0.0863], grad_fn=<SelectBackward>). But torch.sum(output[5]) == 1. produces tensor(False).

What am I misusing about softmax such that output probabilities do not sum to 1?

This is PyTorch version 1.2.0+cpu. Full model is copied below:

import torch
import torch.nn as nn
import torch.nn.functional as F

def assert_no_nan_no_inf(x):
    assert not torch.isnan(x).any()
    assert not torch.isinf(x).any()

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.weights = nn.Linear(
            in_features=2,
            out_features=3)

    def forward(self, x):
        output = F.softmax(self.weights(x), dim=1)
        assert torch.all(torch.sum(output, dim=1) == 1.)
        assert_no_nan_no_inf(x)
        return output


title: How to solve the famous `unhandled cuda error, NCCL version 2.7.8` error?

body:
I've seen multiple issue about the:
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1614378083779/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled cuda error, NCCL version 2.7.8
ncclUnhandledCudaError: Call to CUDA function failed.

but none seem to fix it for me:

https://github.com/pytorch/pytorch/issues/54550
https://github.com/pytorch/pytorch/issues/47885
https://github.com/pytorch/pytorch/issues/50921
https://github.com/pytorch/pytorch/issues/54823

I've tried to do torch.cuda.set_device(device) manually at the beginning of every script. That didn't seem to work for me. I've tried different GPUS. I've tried downgrading pytorch version and cuda version. Different combinations of 1.6.0, 1.7.1, 1.8.0 and cuda 10.2, 11.0, 11.1. I am unsure what else to do. What did people do to solve this issue?

very related perhaps?

Pytorch "NCCL error": unhandled system error, NCCL version 2.4.8"

More complete error message:
('jobid', 4852)
('slurm_jobid', -1)
('slurm_array_task_id', -1)
('condor_jobid', 4852)
('current_time', 'Mar25_16-27-35')
('tb_dir', PosixPath('/home/miranda9/data/logs/logs_Mar25_16-27-35_jobid_4852/tb'))
('gpu_name', 'GeForce GTX TITAN X')
('PID', '30688')
torch.cuda.device_count()=2

opts.world_size=2

ABOUT TO SPAWN WORKERS
done setting sharing strategy...next mp.spawn
INFO:root:Added key: store_based_barrier_key:1 to store for rank: 1
INFO:root:Added key: store_based_barrier_key:1 to store for rank: 0
rank=0
mp.current_process()=<SpawnProcess name='SpawnProcess-1' parent=30688 started>
os.getpid()=30704
setting up rank=0 (with world_size=2)
MASTER_ADDR='127.0.0.1'
59264
backend='nccl'
--> done setting up rank=0
setup process done for rank=0
Traceback (most recent call last):
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 279, in <module>
    main_distributed()
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 188, in main_distributed
    spawn_return = mp.spawn(fn=train, args=(opts,), nprocs=opts.world_size)
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 230, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 188, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 150, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/miranda9/ML4Coq/ml4coq-proj/embeddings_zoo/tree_nns/main_brando.py", line 212, in train
    tactic_predictor = move_to_ddp(rank, opts, tactic_predictor)
  File "/home/miranda9/ultimate-utils/ultimate-utils-project/uutils/torch/distributed.py", line 162, in move_to_ddp
    model = DistributedDataParallel(model, find_unused_parameters=True, device_ids=[opts.gpu])
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 446, in __init__
    self._sync_params_and_buffers(authoritative_rank=0)
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 457, in _sync_params_and_buffers
    self._distributed_broadcast_coalesced(
  File "/home/miranda9/miniconda3/envs/metalearning11.1/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1155, in _distributed_broadcast_coalesced
    dist._broadcast_coalesced(
RuntimeError: NCCL error in: /opt/conda/conda-bld/pytorch_1616554793803/work/torch/lib/c10d/ProcessGroupNCCL.cpp:825, unhandled cuda error, NCCL version 2.7.8
ncclUnhandledCudaError: Call to CUDA function failed.


title: Is it possible to create a FIFO queue with pyTorch?

body:
I need to create a fixed length Tensor in pyTorch that acts like a FIFO queue.

I have this fuction to do it:

def push_to_tensor(tensor, x):
    tensor[:-1] = tensor[1:]
    tensor[-1] = x
    return tensor

For example, I have:

tensor = Tensor([1,2,3,4])

>> tensor([ 1.,  2.,  3.,  4.])

then using the function will give:

push_to_tensor(tensor, 5)

>> tensor([ 2.,  3.,  4.,  5.])

However, I was wondering:

Does pyTorch have a native method for doing this?
If not, is there a more clever way of doing it?


title: How to use AlexNet with one channel

body:
I am new to pytorch and had a problem with channels in AlexNet.
I am using it for a ‘gta san andreas self driving car’ project, I collected the dataset from a black and white image that has one channel and trying to train AlexNet using the script:

from AlexNetPytorch import*
import torchvision
import torchvision.transforms as transforms
import torch.optim as optim
import torch.utils.data
import numpy as np
import torch
from IPython.core.debugger import set_trace

AlexNet = AlexNet()

criterion = nn.CrossEntropyLoss()
optimizer = optim.SGD(AlexNet.parameters(), lr=0.001, momentum=0.9)

all_data = np.load('training_data.npy')
inputs= all_data[:,0]
labels= all_data[:,1]
inputs_tensors = torch.stack([torch.Tensor(i) for i in inputs])
labels_tensors = torch.stack([torch.Tensor(i) for i in labels])

data_set = torch.utils.data.TensorDataset(inputs_tensors,labels_tensors)
data_loader = torch.utils.data.DataLoader(data_set, batch_size=3,shuffle=True, num_workers=2)

if __name__ == '__main__':
 for epoch in range(8):
  runing_loss = 0.0
  for i,data in enumerate(data_loader , 0):
     inputs= data[0]
     inputs = torch.FloatTensor(inputs)
     labels= data[1]
     labels = torch.FloatTensor(labels)
     optimizer.zero_grad()
     # set_trace()
     inputs = torch.unsqueeze(inputs, 1)
     outputs = AlexNet(inputs)
     loss = criterion(outputs , labels)
     loss.backward()
     optimizer.step()

     runing_loss +=loss.item()
     if i % 2000 == 1999:    # print every 2000 mini-batches
            print('[%d, %5d] loss: %.3f' %
                  (epoch + 1, i + 1, running_loss / 2000))
            running_loss = 0.0
 print('finished')

I am using AlexNet from the link:
https://github.com/pytorch/vision/blob/master/torchvision/models/alexnet.py

But changed line 18 from :

nn.Conv2d(3, 64, kernel_size=11, stride=4, padding=2)

To :

nn.Conv2d(1, 64, kernel_size=11, stride=4, padding=2)

Because I am using only one channel in training images, but I get this error:

 File "training_script.py", line 44, in <module>
    outputs = AlexNet(inputs)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Mukhtar\Documents\AI_projects\gta\AlexNetPytorch.py", line 34, in forward
    x = self.features(x)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\container.py", line 91, in forward
    input = module(input)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\modules\pooling.py", line 142, in forward
    self.return_indices)
  File "C:\Users\Mukhtar\Anaconda3\lib\site-packages\torch\nn\functional.py", line 396, in max_pool2d
    ret = torch._C._nn.max_pool2d_with_indices(input, kernel_size, stride, padding, dilation, ceil_mode)
RuntimeError: Given input size: (256x1x1). Calculated output size: (256x0x0). Output size is too small at c:\programdata\miniconda3\conda-bld\pytorch-cpu_1532499824793\work\aten\src\thnn\generic/SpatialDilatedMaxPooling.c:67

I don't know what is wrong, is it wrong to change the channel size like this, and if it is wrong can you please lead me to a neural network that work with one channel , as I said I am a newbie in pytorch and I don't want to write the nn myself.


title: pytorch inception model outputs the wrong label for every input image

body:
For the pytorch models I found this tutorial explaining how to classify an image. I tried to apply the same prcedure for an inception model. However the model fails for every image I load in

Code:

# some people need these three lines to make it work
#from torchvision.models.inception import model_urls
#name = 'inception_v3_google'
#model_urls[name] = model_urls[name].replace('https://', 'http://')

from torch.autograd import Variable
import torchvision
import requests
from torchvision import models, transforms
from PIL import Image
import io
from PIL import Image

LABELS_URL = 'https://s3.amazonaws.com/outcome-blog/imagenet/labels.json'

# cat
IMG_URL1 = 'http://farm2.static.flickr.com/1029/762542019_4f197a0de5.jpg'

# dog
IMG_URL2 = 'http://farm3.static.flickr.com/2314/2518519714_98b01968ee.jpg'

# lion
IMG_URL3 = 'http://farm1.static.flickr.com/62/218998565_62930f10fc.jpg'

labels = {int(key):value for (key, value)
          in requests.get(LABELS_URL).json().items()}

model = torchvision.models.inception_v3(pretrained=True)
model.training = False
model.transform_input = False

def predict_url_img(url):
    response = requests.get(url)
    img_pil = Image.open(io.BytesIO(response.content))

    normalize = transforms.Normalize(
       mean=[0.485, 0.456, 0.406],
       std=[0.229, 0.224, 0.225]
    )
    preprocess = transforms.Compose([
       transforms.Scale(256),
       transforms.CenterCrop(299),
       transforms.ToTensor(),
       normalize
    ])

    img_tensor = preprocess(img_pil)
    img_tensor.unsqueeze_(0)
    img_variable = Variable(img_tensor)
    fc_out = model(img_variable)
    print("prediction:", labels[fc_out.data.numpy().argmax()])

predict_url_img(IMG_URL1)
predict_url_img(IMG_URL2)
predict_url_img(IMG_URL3)

As output I get this:

  ('prediction:', u"plunger, plumber's helper")

  ('prediction:', u'plastic bag')

  ('prediction:', u"plunger, plumber's helper")


title: How to integrate LIME with PyTorch?

body:
Using this mnist image classification model :

%reset -f

import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch
import torch.nn as nn
import torchvision
import torchvision.transforms as transforms
import torch.utils.data as data_utils
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_moons
from matplotlib import pyplot
from pandas import DataFrame
import torchvision.datasets as dset
import os
import torch.nn.functional as F
import time
import random
import pickle
from sklearn.metrics import confusion_matrix
import pandas as pd
import sklearn

trans = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

root = './data'
if not os.path.exists(root):
    os.mkdir(root)
train_set = dset.MNIST(root=root, train=True, transform=trans, download=True)
test_set = dset.MNIST(root=root, train=False, transform=trans, download=True)

batch_size = 64

train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True)
test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
shuffle=True)

class NeuralNet(nn.Module):
    def __init__(self):
        super(NeuralNet, self).__init__()
        self.fc1 = nn.Linear(28*28, 500)
        self.fc2 = nn.Linear(500, 256)
        self.fc3 = nn.Linear(256, 2)
    def forward(self, x):
        x = x.view(-1, 28*28)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

num_epochs = 2
random_sample_size = 200

values_0_or_1 = [t for t in train_set if (int(t[1]) == 0 or int(t[1]) == 1)]
values_0_or_1_testset = [t for t in test_set if (int(t[1]) == 0 or int(t[1]) == 1)]

print(len(values_0_or_1))
print(len(values_0_or_1_testset))

train_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1,
                 batch_size=batch_size,
                 shuffle=True)

test_loader_subset = torch.utils.data.DataLoader(
                 dataset=values_0_or_1_testset,
                 batch_size=batch_size,
                 shuffle=False)

train_loader = train_loader_subset

# Hyper-parameters
input_size = 100
hidden_size = 100
num_classes = 2
# learning_rate = 0.00001
learning_rate = .0001
# Device configuration
device = 'cpu'
print_progress_every_n_epochs = 1

model = NeuralNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

N = len(train_loader)
# Train the model
total_step = len(train_loader)

most_recent_prediction = []
test_actual_predicted_dict = {}

rm = random.sample(list(values_0_or_1), random_sample_size)
train_loader_subset = data_utils.DataLoader(rm, batch_size=4)

for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader_subset):
        # Move tensors to the configured device
        images = images.reshape(-1, 2).to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    if (epoch) % print_progress_every_n_epochs == 0:
        print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'.format(epoch+1, num_epochs, i+1, total_step, loss.item()))

predicted_test = []
model.eval()  # eval mode (batchnorm uses moving mean/variance instead of mini-batch mean/variance)
probs_l = []

predicted_values = []
actual_values = []
labels_l = []

with torch.no_grad():
    for images, labels in test_loader_subset:
        images = images.to(device)
        labels = labels.to(device)
        outputs = model(images)
        _, predicted = torch.max(outputs.data, 1)
        predicted_test.append(predicted.cpu().numpy())

        sm = torch.nn.Softmax()
        probabilities = sm(outputs)
        probs_l.append(probabilities)
        labels_l.append(labels.cpu().numpy())

    predicted_values.append(np.concatenate(predicted_test).ravel())
    actual_values.append(np.concatenate(labels_l).ravel())

if (epoch) % 1 == 0:
    print('test accuracy : ', 100 * len((np.where(np.array(predicted_values[0])==(np.array(actual_values[0])))[0])) / len(actual_values[0]))

I'm to attempting to integrate 'Local Interpretable Model-Agnostic Explanations for machine learning classifiers' : https://marcotcr.github.io/lime/

It appears PyTorch support is not enabled as it is not mentioned in doc and following tutorial :

https://marcotcr.github.io/lime/tutorials/Tutorial%20-%20images.html

With my updated code for PyTorch :

from lime import lime_image
import time

explainer = lime_image.LimeImageExplainer()
explanation = explainer.explain_instance(images[0].reshape(28,28), model(images[0]), top_labels=5, hide_color=0, num_samples=1000)

Causes error :

/opt/conda/lib/python3.6/site-packages/skimage/color/colorconv.py in gray2rgb(image, alpha)
    830     is_rgb = False
    831     is_alpha = False
--> 832     dims = np.squeeze(image).ndim
    833
    834     if dims == 3:

AttributeError: 'Tensor' object has no attribute 'ndim'

So appears tensorflow object is expected here ?

How to integrate LIME with PyTorch image classification ?


title: PyTorch torch_sparse installation without CUDA

body:
I am new in PyTorch and I have faced one issue, namely I cannot get my torch_sparse module properly installed.
In general, I wanted to use module torch_geometric - this I have installed. However, when during the execution of the program I keep receiving the error ModuleNotFoundError: No module named ‘torch_sparse’ .
I try to intall it, but when I use the command pip install torch-sparse in anaconda, I get an error:
UserWarning: CUDA initialization:Found no NVIDIA driver on your system.
My system does not have a CUDA. So how could I install torch_sparse module without it?
Thank you in advance!
Kind regards
Rostyslav


title: how to install pytorch version 0.1.12 in anaconda 3.6 windows 10?

body:
Tried to install with this command but still didn't work-> conda install -c peterjc123 pytorch=0.1.12

Also tried installing using this command:
conda install pytorch=0.1.12 -c pytorch
How can it be installed with python anaconda 3.6?