title: The output of numerical calculation of BCEWithLogitsLoss for PyTorch example

body:
When I looked the example code of BCEWithLogitsLoss in PyTorch Docs. I am confusing about the output result of loss function and formula.

PyTorch Example:

>>> loss = nn.BCEWithLogitsLoss()
>>> input = torch.randn(3, requires_grad=True)
>>> target = torch.empty(3).random_(2)
>>> output = loss(input, target)
>>> output.backward()

input : tensor([0.4764, -2.4063,  0.1563], requires_grad=True)
target: tensor([0., 1., 1.])
output: tensor(1.3567, grad_fn=<BinaryCrossEntropyWithLogitsBackward>)

But according to the formula by showing:

The output of loss function should have shape (3,) instead of single value ,since the shape of input and output both are (3,) . I was thinking that the output may was the sum of Ln or else, but still no idea. Could someone help me to explain that?

As @Dishin H Goyani reminded that the default reduction is 'mean'. I did a simple test.

>>> target_n = target.numpy()
>>> input_n  = input.detach().numpy()
>>> def sigmoid(array):return 1/(1+np.exp(-array))
>>> output_n = -1*(target_n*np.log(sigmoid(input_n))+(1-target_n)*np.log(1-sigmoid(input_n)))
output_n : array([0.95947516, 2.4926252 , 0.61806685], dtype=float32)
>>> np.mean(output_n)
1.3567224

The result is matched.

As you seen, the default Wn is 1.


title: PyTorch embedding layer raises "expected...cuda...but got...cpu" error

body:
I'm working on translating a PyTorch model from CPU (where it works) to GPU (where it so far doesn't). The error message (clipped to the important bits) is as follows:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-a7bb230c924c> in <module>
      1 model = FeedforwardTabularModel()
      2 model.cuda()
----> 3 model.fit(X_train_sample.values, y_train_sample.values)

<ipython-input-11-40b1edae7417> in fit(self, X, y)
    100         for epoch in range(self.n_epochs):
    101             for i, (X_batch, y_batch) in enumerate(batches):
--> 102                 y_pred = model(X_batch).squeeze()
    103                 # scheduler.batch_step()  # Disabled due to a bug, see above.
    104                 loss = self.loss_fn(y_pred, y_batch)

[...]

/opt/conda/lib/python3.6/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

Here is the full model definition:

import torch
from torch import nn
import torch.utils.data
# ^ https://discuss.pytorch.org/t/attributeerror-module-torch-utils-has-no-attribute-data/1666

class FeedforwardTabularModel(nn.Module):
    def __init__(self):
        super().__init__()

        self.batch_size = 512
        self.base_lr, self.max_lr = 0.001, 0.003
        self.n_epochs = 5
        self.cat_vars_embedding_vector_lengths = [
            (1115, 80), (7, 4), (3, 3), (12, 6), (31, 10), (2, 2), (25, 10), (26, 10), (4, 3),
            (3, 3), (4, 3), (23, 9), (8, 4), (12, 6), (52, 15), (22, 9), (6, 4), (6, 4), (3, 3),
            (3, 3), (8, 4), (8, 4)
        ]
        self.loss_fn = torch.nn.MSELoss()
        self.score_fn = torch.nn.MSELoss()

        # Layer 1: embeddings.
        self.embeddings = []
        for (in_size, out_size) in self.cat_vars_embedding_vector_lengths:
            emb = nn.Embedding(in_size, out_size)
            self.embeddings.append(emb)

        # Layer 1: dropout.
        self.embedding_dropout = nn.Dropout(0.04)

        # Layer 1: batch normalization (of the continuous variables).
        self.cont_batch_norm = nn.BatchNorm1d(16, eps=1e-05, momentum=0.1)

        # Layers 2 through 9: sequential feedforward model.
        self.seq_model = nn.Sequential(*[
            nn.Linear(in_features=215, out_features=1000, bias=True),
            nn.ReLU(),
            nn.BatchNorm1d(1000, eps=1e-05, momentum=0.1),
            nn.Dropout(p=0.001),
            nn.Linear(in_features=1000, out_features=500, bias=True),
            nn.ReLU(),
            nn.BatchNorm1d(500, eps=1e-05, momentum=0.1),
            nn.Dropout(p=0.01),
            nn.Linear(in_features=500, out_features=1, bias=True)
        ])

    def forward(self, x):
        # Layer 1: embeddings.
        inp_offset = 0
        embedding_subvectors = []
        for emb in self.embeddings:
            index = torch.tensor(inp_offset, dtype=torch.int64).cuda()
            inp = torch.index_select(x, dim=1, index=index).long().cuda()
            out = emb(inp)
            out = out.view(out.shape[2], out.shape[0], 1).squeeze()
            embedding_subvectors.append(out)
            inp_offset += 1
        out_cat = torch.cat(embedding_subvectors)
        out_cat = out_cat.view(out_cat.shape[::-1])

        # Layer 1: dropout.
        out_cat = self.embedding_dropout(out_cat)

        # Layer 1: batch normalization (of the continuous variables).
        out_cont = self.cont_batch_norm(x[:, inp_offset:])

        out = torch.cat((out_cat, out_cont), dim=1)

        # Layers 2 through 9: sequential feedforward model.
        out = self.seq_model(out)

        return out

    def fit(self, X, y):
        self.train()

        # TODO: set a random seed to invoke determinism.
        # cf. https://github.com/pytorch/pytorch/issues/11278

        X = torch.tensor(X, dtype=torch.float32).cuda()
        y = torch.tensor(y, dtype=torch.float32).cuda()

        # The build of PyTorch on Kaggle has a blog that prevents us from using
        # CyclicLR with ADAM. Cf. GH#19003.
        # optimizer = torch.optim.Adam(model.parameters(), lr=max_lr)
        # scheduler = torch.optim.lr_scheduler.CyclicLR(
        #     optimizer, base_lr=base_lr, max_lr=max_lr,
        #     step_size_up=300, step_size_down=300,
        #     mode='exp_range', gamma=0.99994
        # )
        optimizer = torch.optim.Adam(model.parameters(), lr=(self.base_lr + self.max_lr) / 2)
        batches = torch.utils.data.DataLoader(
            torch.utils.data.TensorDataset(X, y),
            batch_size=self.batch_size, shuffle=True
        )

        for epoch in range(self.n_epochs):
            for i, (X_batch, y_batch) in enumerate(batches):
                y_pred = model(X_batch).squeeze()
                # scheduler.batch_step()  # Disabled due to a bug, see above.
                loss = self.loss_fn(y_pred, y_batch)
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()
            print(
                f"Epoch {epoch + 1}/{self.n_epochs}, Loss {loss.detach().numpy()}"
            )

    def predict(self, X):
        self.eval()
        with torch.no_grad():
            y_pred = model(torch.tensor(X, dtype=torch.float32).cuda())
        return y_pred.squeeze()

    def score(self, X, y):
        y_pred = self.predict(X)
        y = torch.tensor(y, dtype=torch.float32).cuda()
        return self.score_fn(y, y_pred)

model = FeedforwardTabularModel()
model.cuda()
model.fit(X_train_sample.values, y_train_sample.values)

This type of error typically occurs when there is a tensor in the model that should be on GPU but is on CPU instead. But as far as I can tell, I've already placed .cuda() calls at all of the necessary places: every time a torch.tensor is declared, and running model.cuda() before model.fit.

What is causing this error?


title: What are the difference between .bin and .pt pytorch saved model types?

body:
Sometimes I see .bin files for pretrained pytorch, like the one here

https://github.com/allenai/scibert#pytorch-models

However, the files are usually saved as .pt files.

What's the difference between these two parameter weights file formats? Why are there two?


title: Pytorch RuntimeError: size mismatch, m1: [1 x 7744], m2: [400 x 120]

body:
In a simple CNN that classifies 5 objects, I get a size mis-match error:

"RuntimeError: size mismatch, m1: [1 x 7744], m2: [400 x 120]" in the convolutional layer .

my model.py file:

import torch.nn as nn
import torch.nn.functional as F

class FNet(nn.Module):

    def __init__(self,device):
        # make your convolutional neural network here
        # use regularization
        # batch normalization
        super(FNet, self).__init__()
        num_classes = 5
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.conv2 = nn.Conv2d(6, 16, 5)
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 5)

    def forward(self, x):

        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))

        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

if __name__ == "__main__":
    net = FNet()

Complete Error:

Traceback (most recent call last):
  File "main.py", line 98, in <module>
    train_model('../Data/fruits/', save=True, destination_path='/home/mitesh/E yantra/task1#hc/Task 1/Task 1B/Data/fruits')
  File "main.py", line 66, in train_model
    outputs = model(images)
  File "/home/mitesh/anaconda3/envs/HC#850_stage1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mitesh/E yantra/task1#hc/Task 1/Task 1B/Code/model.py", line 28, in forward
    x = F.relu(self.fc1(x))
  File "/home/mitesh/anaconda3/envs/HC#850_stage1/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mitesh/anaconda3/envs/HC#850_stage1/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "/home/mitesh/anaconda3/envs/HC#850_stage1/lib/python3.6/site-packages/torch/nn/functional.py", line 1024, in linear
    return torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [1 x 7744], m2: [400 x 120] at /opt/conda/conda-bld/pytorch-cpu_1532576596369/work/aten/src/TH/generic/THTensorMath.cpp:2070


title: Creating a torch tensor from a generator

body:
I attempt to construct a tensor from a generator as follows:

>>> torch.tensor(i**2 for i in range(10))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: Could not infer dtype of generator

Currently I just do:

>>> torch.tensor([i**2 for i in range(10)])
tensor([ 0,  1,  4,  9, 16, 25, 36, 49, 64, 81])

Is there a way to avoid needing this intermediate list?


title: Weighted summation of embeddings in pytorch

body:
I have a sequence of 12 words which I represent using a 12x256 matrix (using word embeddings). Let us refer to these as . I wish to take this as input and output a 1x256 vector. However I don't want to use a (12x256) x 256 dense layer. Instead I want to create the output embedding using a weighted summation of the 12 embeddings

where the wi s are scalars (thus there is weight sharing).

How can I create trainable wi s in pytorch? I am new and only familiar with the standard modules like nn.Linear.


title: Run inference on CPU using pytorch and multiprocessing

body:
I have trained a CNN model on GPU using FastAI (PyTorch backend). I am now trying to use that model for inference on the same machine, but using CPU instead of GPU. Along with that, I am also trying to make use of multiple CPU cores using the multiprocessing module. Now here is the issue,

  Running the code on single CPU (without multiprocessing) takes only 40 seconds to process nearly 50
  images

  Running the code on multiple CPUs using torch multiprocessing takes more than 6 minutes to process the same 50 images

from torch.multiprocessing import Pool, set_start_method
os.environ['CUDA_VISIBLE_DEVICES']=""
from fastai.vision import *
from fastai.text import *
defaults.device = torch.device('cpu')

def process_image_batch(batch):

    learn_cnn  = load_learner(scripts_folder, 'cnn_model.pkl')
    learn_cnn.model.training = False
    learn_cnn.model = learn_cnn.model.eval()
    # for image in batch:
    #     prediction = ... # predicting the image here
    #     return prediction

if __name__ == '__main__':
    #
    # image_batches = ..... # retrieving the image batches (It is a list of 5 lists)
    # n_processes = 5
    set_start_method('spawn', force=True)
    try:
        pool = Pool(n_processes)
        pool.map(process_image_batch, image_batches)
    except Exception as e:
        print('Main Pool Error: ', e)
    except KeyboardInterrupt:
        exit()
    finally:
        pool.terminate()
        pool.join()

I am not sure what's causing this slowdown in multiprocessing mode. I've read a lot of posts discussing similar issue but couldn't find a proper solution anywhere.


title: Convolutional auto-encoder error - 'RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.FloatTensor) should be the same'

body:
For below model I received error 'Expected stride to be a single value integer or list'. I used suggested answer from https://discuss.pytorch.org/t/expected-stride-to-be-a-single-integer-value-or-a-list/17612/2
 and added

img.unsqueeze_(0)

I now receive error :

RuntimeError: Input type (torch.cuda.ByteTensor) and weight type (torch.FloatTensor) should be the same

For below code I three sample images and attempt to learn a representation of them using an auto-encoder :

%reset -f

import torch.utils.data as data_utils
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from matplotlib import pyplot as plt
from sklearn import metrics
import datetime
from sklearn.preprocessing import MultiLabelBinarizer
import seaborn as sns
sns.set_style("darkgrid")
from ast import literal_eval
import numpy as np
from sklearn.preprocessing import scale
import seaborn as sns
sns.set_style("darkgrid")
import torch
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable
from os import listdir
import cv2
import torch.nn.functional as F
import numpy as np
from numpy.polynomial.polynomial import polyfit
import matplotlib.pyplot as plt

number_channels = 3

%matplotlib inline

x = np.arange(10)
m = 1
b = 2
y = x * x
plt.plot(x, y)
plt.axis('off')
plt.savefig('1-increasing.jpg')

x = np.arange(10)
m = 0.01
b = 2
y = x * x * x
plt.plot(x, y)
plt.axis('off')
plt.savefig('2-increasing.jpg')

x = np.arange(10)
m = 0
b = 2
y = (m*x)+b
plt.plot(x, y)
plt.axis('off')
plt.savefig('constant.jpg')

batch_size_value = 2

train_image = []

train_image.append(cv2.imread('1-increasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('2-increasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('decreasing.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))
train_image.append(cv2.imread('constant.jpg', cv2.IMREAD_UNCHANGED).reshape(3, 288, 432))

data_loader = data_utils.DataLoader(train_image, batch_size=batch_size_value, shuffle=False,drop_last=True)

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.utils import save_image
from torchvision.datasets import MNIST
import os

if not os.path.exists('./dc_img'):
    os.mkdir('./dc_img')

def to_img(x):
    x = 0.5 * (x + 1)
    x = x.clamp(0, 1)
    x = x.view(x.size(0), 1, 28, 28)
    return x

num_epochs = 100
# batch_size = 128
batch_size = 2

learning_rate = 1e-3
dataloader = data_loader

class autoencoder(nn.Module):
    def __init__(self):
        super(autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(3, 16, 3, stride=3, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2),  # b, 16, 5, 5
            nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
            nn.ReLU(True),
            nn.MaxPool3d(3, stride=1)  # b, 8, 2, 2
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose3d(8, 16, 3, stride=2),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose3d(16, 8, 5, stride=3, padding=1),  # b, 8, 15, 15
            nn.ReLU(True),
            nn.ConvTranspose3d(8, 1, 2, stride=2, padding=1),  # b, 1, 28, 28
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

model = autoencoder()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,
                             weight_decay=1e-5)

for epoch in range(num_epochs):
    for data in dataloader:
        img, _ = data
        img.unsqueeze_(0)
#         img.unsqueeze_(0)
#         print(img)
#         img.unsqueeze_(0)
        img = Variable(img).cuda()
        # ===================forward=====================
        output = model(img)
        loss = criterion(output, img)
        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    # ===================log=================to_img=======
    print('epoch [{}/{}], loss:{:.4f}'
          .format(epoch+1, num_epochs, loss.data[0]))
    if epoch % 10 == 0:
        pic = to_img(output.cpu().data)
        save_image(pic, './dc_img/image_{}.png'.format(epoch))

torch.save(model.state_dict(), './conv_autoencoder.pth')

But as stated earlier this results in error :

299     def forward(self, input):
300         return F.conv2d(input, self.weight, self.bias, self.stride,

  --> 301                         self.padding, self.dilation, self.groups)
      302
      303

  RuntimeError: Input type (torch.cuda.ByteTensor) and weight type
  (torch.FloatTensor) should be the same

The issue appears to be related to img.unsqueeze_(0) ?

How to train the auto-encoder on these images ?


title: No module named 'torch.nn.functional'

body:
I have python file with lines:

import argparse
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torch.autograd import Variable

It generates errors:

  File "C:\gdrive\python\a.py", line 5, in <module>
import torch.nn.functional as F
ModuleNotFoundError: No module named 'torch.nn.functional'

How to fix that error?

I have installed pytorch by using command:

conda install pytorch-cpu torchvision-cpu -c pytorch


title: Assigning values to torch tensors

body:
I'm trying to assign some values to a torch tensor. In the sample code below, I initialized a tensor U and try to assign a tensor b to its last 2 dimensions. In reality, this is a loop over i and j that solves some relation for a number of training data (here 10) and assigns it to its corresponding location.

import torch

U = torch.zeros([10, 1, 4, 4])
b = torch.rand([10, 1, 1, 1])

i = 2
j = 2
U[:, :, i, j] = b

I was expecting vector b to be assigned for dimensions i and j of corresponding training data (shape of training data being (10,1)) but it gives me an error. The error that I get is the following

RuntimeError: expand(torch.FloatTensor{[10, 1, 1, 1]}, size=[10, 1]): the number of sizes provided (2) must be greater or equal to the number of dimensions in the tensor (4)

Any suggestions on how to fix it would be appreciated.

As an example, you can think of this as if '[10, 1]' is the shape of my data. Imagine it is 10 images, each of which has one channel. Then imagine each image is of shape '[4, 4]'. In each iteration of the loop, pixel '[i, j]' for all images and channels is being calculated.


title: Unable to update PyTorch 1.4.0 to 1.5.0 using Conda

body:
When I tried to udpdate PyTorch from 1.4.0 to 1.5.0, Anaconda says that all the packages are already installed.

$ conda install -c pytorch pytorch torchvision

Collecting package metadata (current_repodata.json): done
Solving environment: done

# All requested packages already installed.

$ conda list | grep -i torch

_pytorch_select           0.2                       gpu_0
pytorch                   1.4.0           py3.7_cuda10.0.130_cudnn7.6.3_0    pytorch
torchvision               0.5.0                py37_cu100    pytorch

I believe 1.5.0 is available in the pytorch channel

$ conda search -c pytorch pytorch=1.5.0

Loading channels: done
# Name                       Version           Build  Channel
pytorch                        1.5.0     py3.5_cpu_0  pytorch
pytorch                        1.5.0 py3.5_cuda10.1.243_cudnn7.6.3_0  pytorch
pytorch                        1.5.0 py3.5_cuda10.2.89_cudnn7.6.5_0  pytorch
pytorch                        1.5.0 py3.5_cuda9.2.148_cudnn7.6.3_0  pytorch
pytorch                        1.5.0     py3.6_cpu_0  pytorch
pytorch                        1.5.0 py3.6_cuda10.1.243_cudnn7.6.3_0  pytorch
pytorch                        1.5.0 py3.6_cuda10.2.89_cudnn7.6.5_0  pytorch
pytorch                        1.5.0 py3.6_cuda9.2.148_cudnn7.6.3_0  pytorch
pytorch                        1.5.0     py3.7_cpu_0  pytorch
pytorch                        1.5.0 py3.7_cuda10.1.243_cudnn7.6.3_0  pytorch
pytorch                        1.5.0 py3.7_cuda10.2.89_cudnn7.6.5_0  pytorch
pytorch                        1.5.0 py3.7_cuda9.2.148_cudnn7.6.3_0  pytorch
pytorch                        1.5.0     py3.8_cpu_0  pytorch
pytorch                        1.5.0 py3.8_cuda10.1.243_cudnn7.6.3_0  pytorch
pytorch                        1.5.0 py3.8_cuda10.2.89_cudnn7.6.5_0  pytorch
pytorch                        1.5.0 py3.8_cuda9.2.148_cudnn7.6.3_0  pytorch

Why is conda not updating PyTorch to 1.5.0?

Using Python 3.7.3 & conda 4.8.3 on Ubuntu 18.04

Thanks!


title: transfer learning [resnet18] using PyTorch. Dataset: Dog-Breed-Identification

body:
I am trying to implement a transfer learning approach in PyTorch. This is the dataset that I am using: Dog-Breed

Here's the step that I am following.

1. Load the data and read csv using pandas.
2. Resize (60, 60) the train images and store them as numpy array.
3. Apply stratification and split the train data into 7:1:2 (train:validation:test)
4. use the resnet18 model and train.

Location of dataset

LABELS_LOCATION = './dataset/labels.csv'
TRAIN_LOCATION = './dataset/train/'
TEST_LOCATION = './dataset/test/'
ROOT_PATH = './dataset/'

Reading CSV (labels.csv)

def read_csv(csvf):
    # print(pandas.read_csv(csvf).values)
    data=pandas.read_csv(csvf).values
    labels_dict = dict(data)
    idz=list(labels_dict.keys())
    clazz=list(labels_dict.values())
    return labels_dict,idz,clazz

I did this because of a constraint which I will mention next when I am loading the data using DataLoader.

def class_hashmap(class_arr):
    uniq_clazz = Counter(class_arr)
    class_dict = {}
    for i, j in enumerate(uniq_clazz):
        class_dict[j] = i
    return class_dict

labels, ids, class_names = read_csv(LABELS_LOCATION)
train_images = os.listdir(TRAIN_LOCATION)
class_numbers = class_hashmap(class_names)

Next, I resize the image to 60,60 using opencv, and store the result as numpy array.

resize = []
indexed_labels = []
for t_i in train_images:
    # resize.append(transform.resize(io.imread(TRAIN_LOCATION+t_i), (60, 60, 3)))  # (60,60) is the height and widht; 3 is the number of channels
    resize.append(cv2.resize(cv2.imread(TRAIN_LOCATION+t_i), (60, 60)).reshape(3, 60, 60))
    indexed_labels.append(class_numbers[labels[t_i.split('.')[0]]])

resize = np.asarray(resize)
print(resize.shape)

Here in indexed_labels, I give each label a number.

Next, I split the data into 7:1:2 part

X = resize  # numpy array of images [training data]
y = np.array(indexed_labels)  # indexed labels for images [training labels]

sss = StratifiedShuffleSplit(n_splits=3, test_size=0.2, random_state=0)
sss.get_n_splits(X, y)

for train_index, test_index in sss.split(X, y):
    X_temp, X_test = X[train_index], X[test_index]  # split train into train and test [data]
    y_temp, y_test = y[train_index], y[test_index]  # labels

sss = StratifiedShuffleSplit(n_splits=3, test_size=0.123, random_state=0)
sss.get_n_splits(X_temp, y_temp)

for train_index, test_index in sss.split(X_temp, y_temp):
    print("TRAIN:", train_index, "VAL:", test_index)
    X_train, X_val = X[train_index], X[test_index]  # training and validation data
    y_train, y_val = y[train_index], y[test_index]  # training and validation labels

Next, I loaded the data from the previous step into torch DataLoaders

batch_size = 500
learning_rate = 0.001

train = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train))
train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, shuffle=False)

val = torch.utils.data.TensorDataset(torch.from_numpy(X_val), torch.from_numpy(y_val))
val_loader = torch.utils.data.DataLoader(val, batch_size=batch_size, shuffle=False)

test = torch.utils.data.TensorDataset(torch.from_numpy(X_test), torch.from_numpy(y_test))
test_loader = torch.utils.data.DataLoader(test, batch_size=batch_size, shuffle=False)

# print(train_loader.size)

dataloaders = {
    'train': train_loader,
    'val': val_loader
}

Next, I load the pretrained rensnet model.

model_ft = models.resnet18(pretrained=True)

# freeze all model parameters
# for param in model_ft.parameters():
#     param.requires_grad = False

num_ftrs = model_ft.fc.in_features
model_ft.fc = nn.Linear(num_ftrs, len(class_numbers))

if use_gpu:
    model_ft = model_ft.cuda()
    model_ft.fc = model_ft.fc.cuda()

criterion = nn.CrossEntropyLoss()

# Observe that all parameters are being optimized
optimizer_ft = optim.SGD(model_ft.fc.parameters(), lr=0.001, momentum=0.9)

# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                       num_epochs=25)

And then I use the train_model, a method described here in PyTorch's docs.

However, when I run this I get an error.

Traceback (most recent call last):
  File "/Users/nirvair/Sites/pyTorch/TL.py",
    line 244, in <module>
        num_epochs=25)
      File "/Users/nirvair/Sites/pyTorch/TL.py", line 176, in train_model
        outputs = model(inputs)
      File "/Library/Python/2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
        result = self.forward(*input, **kwargs)
      File "/Library/Python/2.7/site-packages/torchvision/models/resnet.py", line 149, in forward
        x = self.avgpool(x)
      File "/Library/Python/2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
        result = self.forward(*input, **kwargs)
      File "/Library/Python/2.7/site-packages/torch/nn/modules/pooling.py", line 505, in forward
        self.padding, self.ceil_mode, self.count_include_pad)
      File "/Library/Python/2.7/site-packages/torch/nn/functional.py", line 264, in avg_pool2d
        ceil_mode, count_include_pad)
      File "/Library/Python/2.7/site-packages/torch/nn/_functions/thnn/pooling.py", line 360, in forward
        ctx.ceil_mode, ctx.count_include_pad)
    RuntimeError: Given input size: (512x2x2). Calculated output size: (512x0x0). Output size is too small at /Users/soumith/code/builder/wheel/pytorch-src/torch/lib/THNN/generic/SpatialAveragePooling.c:64

I can't seem to figure out what's going wrong here.


title: How to convert an list of image into Pytorch Tensor

body:
I have a list called wordImages. It contains images in np.array format with different width & height.

How Do I convert this into a tensor and use this instead of my_dataset in the below code?

Currently i am using this. But I need to save/read images

demo_data = RawDataset(root="output_craft/", opt=opt)

demo_loader = torch.utils.data.DataLoader(
                demo_data , batch_size=opt.batch_size,
                shuffle=False,
                num_workers=int(opt.workers),
                collate_fn=AlignCollate_demo, pin_memory=True)


title: PyTorch error loading saved nn.Module: object has no attribute 'to'

body:
I am using PyTorch 0.4.

I defined a PyTorch MyModel by inheriting from nn.Module, and saved an instance of it by calling

torch.save(my_model, my_path)

Then, when loading it again with torch.load(my_path), my program crashed with the following error:

AttributeError: 'MyModel' object has no attribute 'to'

But my program was able to run it in previous stages. What did go wrong?


title: Could not import pytorch in windows

body:
I am trying to install torch in windows to use with python. I have install torch and lua somehow.
Then I tried using:

conda install -c peterjc123 pytorch

from this answer here.
It seems to have succeeded and asks for a new package to be installed:

  The following NEW packages will be INSTALLED:

pytorch: 0.3.1-py36_cuda80_cudnn6he774522_2 peterjc123

I don't have a GPU but thought it might use CPU (I am not sure about that but I proceeded anyway).

The problem is that after the installation I cannot import it:

import PyTorch

  ModuleNotFoundError: No module named 'pytorch'

The code I tried was from here. Does this seem normal? I would expect an error regarding functionality or something not about importing the module.

P.S. My initial intention is to run this code which uses torch. I am not sure how torch and pytorch are connected but I was hoping to make it work.


title: How to use CUDA stream in Pytorch?

body:
I wanna use CUDA stream in Pytorch to parallel some computations, but I don't know how to do it.
For instance, if there's 2 tasks, A and B, need to be parallelized, I wanna do the following things:

stream0 = torch.get_stream()
stream1 = torch.get_stream()
with torch.now_stream(stream0):
    // task A
with torch.now_stream(stream1):
    // task B
torch.synchronize()
// get A and B's answer

How can I achieve the goal in real python code?


title: RuntimeError: DataLoader worker (pid 27351) is killed by signal: Killed

body:
I'm running the data loader below which applies a filter to a microscopy image prior to training. In order to count the red and green. This code filters the red cells. Since I have applied this to the code I keep on getting the error message above. I have tried increasing the memory allocation to the maximum allowance possible but that didn't help. Is there a way I could modify the filter so it isn't causing this issue, please? Many thanks in advance
import os

import numpy as np
import torch
from PIL import Image
from torch.utils.data import Dataset
from torchvision import transforms, utils
#from torchvision.transforms import Grayscalei
import pandas as pd
import pdb
import cv2

class CellsDataset(Dataset):
    # a very simple dataset

    def __init__(self, root_dir, transform=None, return_filenames=False):
        self.root = root_dir
        self.transform = transform
        self.return_filenames = return_filenames
        self.files = [os.path.join(self.root,filename) for filename in os.listdir(self.root)]
        self.files = [path for path in self.files
                      if os.path.isfile(path) and os.path.splitext(path)[1]=='.png']

    def __len__(self):
        return len(self.files)

    def __getitem__(self, idx):
        path = self.files[idx]
        image = cv2.imread(path)
        sample = image.copy()
        # set blue and green channels to 0
        sample[:, :, 0] = 0
        sample[:, :, 1] = 0

channel.
        if self.transform:
            sample = self.transform(sample)

        if self.return_filenames:
            return sample, path
        else:
            return sample


title: how to convert pytorch adaptive_avg_pool2d method to keras or tensorflow

body:
I don't know how to convert the PyTorch method adaptive_avg_pool2d to Keras or TensorFlow. Anyone can help?
PyTorch mehod is

adaptive_avg_pool2d(14,[14])

I tried to use the average pooling, the reshape the tensor in Keras, but got the error:

  ValueError: total size of new array must be unchanged


title: Torch installation results in not supported wheel on this platform

body:
Tried running pip3 install torch===1.4.0 torchvision===0.5.0 -f https://download.pytorch.org/whl/torch_stable.html first, taken from PyTorch website
which resulted in
No matching distribution found for torch===1.4.0
 and  Could not find a version that satisfies the requirement torch===1.4.0 (from versions: 0.1.2, 0.1.2.post1, 0.1.2.post2)

Finally downloaded the .whl file from downloads page and tried installing locally like so 'C:\Users\Raf\AppData\Local\Programs\Python\Python38\Scripts\pip.exe' install torch-1.4.0+cpu-cp37-cp37m-win_amd64.whl
after which I get torch-1.4.0+cpu-cp37-cp37m-win_amd64.whl is not a supported wheel on this platform.

using 64 Python 3.8, on 64 bit Windows


title: What does the underscore suffix in PyTorch functions mean?

body:
In PyTorch, many methods of a tensor exist in two versions - one with a underscore suffix, and one without. If I try them out, they seem to do the same thing:

In [1]: import torch

In [2]: a = torch.tensor([2, 4, 6])

In [3]: a.add(10)
Out[3]: tensor([12, 14, 16])

In [4]: a.add_(10)
Out[4]: tensor([12, 14, 16])

What is the difference between

torch.add and torch.add_
torch.sub and torch.sub_
...and so on?


Using below code I create 10 instances of each training data of which each has 100 dimensions.
Each of the 100 dimension contains 3 dimensions. Therefore it's shape is : (3, 100, 10). This emulates 10 instances of 100 pixels each with 3 channels to emulate an RGB value

I've set this model to just classify between 1 and 0.

When applying the softmax layer I receive the error :

  RuntimeError: Expected tensor for argument #1 'input' to have the same
  dimension as tensor for 'result'; but 4 does not equal 3 (while
  checking arguments for cudnn_convolution)

I'm using 0.4.0 (checked using print(torch.__version__) )
How to correctly set the dimension for the softmax layer? As I think my dimensions are correct?

%reset -f

import os
import torch
from skimage import io, transform
import numpy as np
import matplotlib.pyplot as plt
from torch.utils.data import Dataset, DataLoader
from torchvision import transforms, utils
import torch.utils.data as data_utils
import torchvision
import numpy as np
from sklearn.preprocessing import scale
import torch.nn.functional as F
import torch.nn as nn
import torch.nn.functional as F
from random import randint

batch_size_value = 10

train_dataset = []
mu, sigma = 0, 0.1 # mean and standard deviation
num_instances = 10

# Create 3000 instance and reshape to (3 , 100, 10) , this emulates 10 instances of 100 pixels
# each with 3 channels to emulate an RGB value

for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 3000).reshape((3 , 100, 10))
    train_dataset.append(image_x)

mu, sigma = 100, 0.80 # mean and standard deviation
for i in range(num_instances) :
    image = []
    image_x = np.random.normal(mu, sigma, 3000).reshape((3 , 100, 10))
    train_dataset.append(image_x)

labels_1 = [1 for i in range(num_instances)]
labels_0 = [0 for i in range(num_instances)]

labels = labels_1 + labels_0

print(labels)

x2 = torch.tensor(train_dataset).float()
y2 = torch.tensor(labels).long()

my_train2 = data_utils.TensorDataset(x2, y2)
train_loader2 = data_utils.DataLoader(my_train2, batch_size=batch_size_value, shuffle=False)

# print(x2)

# Device configuration
device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print('device' , device)
# device = 'cpu'

# Hyper parameters
num_epochs = 10
num_classes = 2
batch_size = 5
learning_rate = 0.001

# Convolutional neural network (two convolutional layers)
class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, kernel_size=5)
        self.conv2 = nn.Conv2d(6, 16, kernel_size=5)
        self.fc1   = nn.Linear(864, 120)
        self.fc2   = nn.Linear(120, 84)
        self.fc3   = nn.Linear(84, num_classes)

    def forward(self, x):
        out = F.relu(self.conv1(x))
        out = F.max_pool2d(out, 2)
        out = F.relu(self.conv2(out))
        out = F.max_pool2d(out, 2)
        out = out.view(out.size(0), -1)
        out = F.relu(self.fc1(out))
        out = F.relu(self.fc2(out))
        out = self.fc3(out)

model = ConvNet().to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

# Train the model
total_step = len(train_loader2)
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader2):
        images = images.to(device)
        labels = labels.to(device)

        # Forward pass
        outputs = model(images)
#         print(images)

        loss = criterion(outputs, labels)

        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i % 10) == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

Update :

Removing these lines :

out = F.relu(self.conv2(out))
out = F.max_pool2d(out, 2)

fixes issue of dimensions being smaller than kernel.


title: Runtimeerror: Cuda out of memory - problem in code or gpu?

body:
I am currently working on a computer vision project. I keep getting a runtime error that says "CUDA out of memory". I have tried all possible ways like reducing batch size and image resolution, clearing the cache, deleting variables after training starts, reducing image data and so on... Unfortunately, this error doesn't stop. I have a Nvidia Geforce 940MX graphics card on my HP Pavilion laptop. I have installed cuda 10.2 and cudNN from the pytorch installation page. My aim was to create a flask website out of this model but I am stuck with this issue. Any suggestions to this problem will be helpful.
This is my code
import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import os
import cv2
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
import albumentations as A
from torch.utils.data import TensorDataset, DataLoader,Dataset
from torchvision import models
from collections import defaultdict
from torch.utils.data.sampler import RandomSampler
import torch.optim as optim
from torch.optim import lr_scheduler
from sklearn import model_selection
from tqdm import tqdm
import gc

# generate data from csv file
class Build_dataset(Dataset):

    def __init__(self, csv, split, mode, transform=None):
        self.csv = csv.reset_index(drop=True)
        self.split = split
        self.mode = mode
        self.transform = transform

    def __len__(self):
        return self.csv.shape[0]

    def __getitem__(self, index):
        row = self.csv.iloc[index]

        image = cv2.imread(row.filepath)
        image = cv2.cvtColor(image, cv2.COLOR_RGB2BGR)

        if self.transform is not None:
            res = self.transform(image=image)
            image = res['image'].astype(np.float32)
        else:
            image = image.astype(np.float32)

        image = image.transpose(2, 0, 1)
        data = torch.tensor(image).float()

        if self.mode == 'test':
            return data
        else:
            return data, torch.tensor(self.csv.iloc[index].target).long()

# training epoch
def train_epoch(model, loader, optimizer,loss_fn,device, scheduler,n_examples):

model = model.train()

losses = []
correct_predictions = 0

for inputs, labels in tqdm(loader):
    inputs = inputs.to(device)
    labels = labels.to(device)

    outputs = model(inputs)

    _, preds = torch.max(outputs, dim=1)
    loss = loss_fn(outputs, labels)

    correct_predictions += torch.sum(preds == labels)
    losses.append(loss.item())

    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
# here you delete inputs and labels and then use gc.collect
    del inputs, labels
    gc.collect()

return correct_predictions.double() / n_examples, np.mean(losses)

# validation epoch
def val_epoch(model, loader,loss_fn, device,n_examples):

model = model.eval()

losses = []
correct_predictions = 0

with torch.no_grad():
    for inputs, labels in tqdm(loader):
        inputs = inputs.to(device)
        labels = labels.to(device)
        outputs = model(inputs)
        _, preds = torch.max(outputs, dim=1)
        loss = loss_fn(outputs, labels)
        correct_predictions += torch.sum(preds == labels)
        losses.append(loss.item())
        # here you delete inputs and labels and then use gc.collect
        del inputs, labels
        gc.collect()

return correct_predictions.double() / n_examples, np.mean(losses)

def train(model,device, num_epochs):
# generate data
dataset_train = Build_dataset(df_train,  'train', 'train', transform=transforms_train)
dataset_valid = Build_dataset(df_valid, 'train', 'val', transform=transforms_val)

#load data
train_loader = DataLoader(dataset_train, batch_size = 16,sampler=RandomSampler(dataset_train), num_workers=4)
valid_loader = DataLoader(dataset_valid, batch_size = 16,shuffle = True, num_workers= 4 )

dataset_train_size = len(dataset_train)

dataset_valid_size = len(dataset_valid)

optimizer = optim.Adam(model.parameters(), lr = 3e-5)

model = model.to(device)

scheduler = lr_scheduler.ReduceLROnPlateau(optimizer, patience = 3,threshold = 0.001, mode = 'max')

loss_fn = nn.CrossEntropyLoss().to(device)

history = defaultdict(list)

best_accuracy = 0.0

for epoch in range(num_epochs):
    print(f'Epoch {epoch+1} / {num_epochs}')
    print ('-'*30)

    train_acc, train_loss = train_epoch(model, train_loader, optimizer, loss_fn, device, scheduler, dataset_train_size)
    print(f'Train loss {train_loss} accuracy {train_acc}')
    valid_acc, valid_loss = val_epoch(model, valid_loader, loss_fn, device,dataset_valid_size)
    print(f'Val   loss {valid_loss} accuracy {valid_acc}')
    print()

    history['train_acc'].append(train_acc)
    history['train_loss'].append(train_loss)
    history['val_acc'].append(valid_acc)
    history['val_loss'].append(valid_loss)

    if valid_acc > best_accuracy:
        torch.save(model.state_dict(), 'best_model_state.bin')
        best_accuracy = valid_acc

print('Best Accuracy: {best_accuracy}')

model.load_state_dict(torch.load('best_model_state.bin'))

return model, history

if __name__ == '__main__':
#competition data -2020
data_dir = "C:\\Users\\Aniruddh\\Documents\\kaggle\\jpeg_melanoma_2020"
#competition data - 2019
data_dir2 = "C:\\Users\\Aniruddh\\Documents\\kaggle\\jpeg_melanoma_2019"
# device
device = torch.device("cuda")

# augmenting images

image_size = 384
transforms_train = A.Compose([
    A.Transpose(p=0.5),
    A.VerticalFlip(p=0.5),
    A.HorizontalFlip(p=0.5),
    A.RandomBrightness(limit=0.2, p=0.75),
    A.RandomContrast(limit=0.2, p=0.75),
    A.OneOf([
        A.MedianBlur(blur_limit=5),
        A.GaussianBlur(blur_limit=5),
        A.GaussNoise(var_limit=(5.0, 30.0)),
    ], p=0.7),

    A.OneOf([
        A.OpticalDistortion(distort_limit=1.0),
        A.GridDistortion(num_steps=5, distort_limit=1.),
        A.ElasticTransform(alpha=3),
    ], p=0.7),

    A.CLAHE(clip_limit=4.0, p=0.7),
    A.HueSaturationValue(hue_shift_limit=10, sat_shift_limit=20, val_shift_limit=10, p=0.5),
    A.ShiftScaleRotate(shift_limit=0.1, scale_limit=0.1, rotate_limit=15, border_mode=0, p=0.85),
    A.Resize(image_size, image_size),
    A.Cutout(max_h_size=int(image_size * 0.375), max_w_size=int(image_size * 0.375), num_holes=1, p=0.7),
    A.Normalize()
])

transforms_val = A.Compose([
    A.Resize(image_size, image_size),
    A.Normalize()
])
# create data
df_train = pd.read_csv("C:\\Users\\Aniruddh\\Documents\\kaggle\\jpeg_melanoma_2020\\train.csv")  #/kaggle/input/siim-isic-melanoma-classification/train.csv
df_train.head()

df_train['is_ext'] = 0
df_train['filepath'] = df_train['image_name'].apply(lambda x: os.path.join(data_dir, 'train', f'{x}.jpg'))

# dataset from 2020 data
df_train['diagnosis'] = df_train['diagnosis'].apply(lambda x: x.replace('seborrheic keratosis', 'BKL'))
df_train['diagnosis'] = df_train['diagnosis'].apply(lambda x: x.replace('lichenoid keratosis', 'BKL'))
df_train['diagnosis'] = df_train['diagnosis'].apply(lambda x: x.replace('solar lentigo', 'BKL'))
df_train['diagnosis'] = df_train['diagnosis'].apply(lambda x: x.replace('lentigo NOS', 'BKL'))
df_train['diagnosis'] = df_train['diagnosis'].apply(lambda x: x.replace('cafe-au-lait macule', 'unknown'))
df_train['diagnosis'] = df_train['diagnosis'].apply(lambda x: x.replace('atypical melanocytic proliferation', 'unknown'))

# dataset from 2019 data
df_train2 = pd.read_csv('/content/drive/My Drive/siim_melanoma images/train_2019.csv')
df_train2 = df_train2[df_train2['tfrecord'] >= 0].reset_index(drop=True)
#df_train2['fold'] = df_train2['tfrecord'] % 5
df_train2['is_ext'] = 1
df_train2['filepath'] = df_train2['image_name'].apply(lambda x: os.path.join(data_dir2, 'train', f'{x}.jpg'))

df_train2['diagnosis'] = df_train2['diagnosis'].apply(lambda x: x.replace('NV', 'nevus'))
df_train2['diagnosis'] = df_train2['diagnosis'].apply(lambda x: x.replace('MEL', 'melanoma'))

#concat both 2019 and 2020 data
df_train = pd.concat([df_train, df_train2]).reset_index(drop=True)

# shuffle data
df = df_train.sample(frac=1).reset_index(drop=True)

# creating 8 different target values
new_target = {d: idx for idx, d in enumerate(sorted(df.diagnosis.unique()))}
df['target'] = df['diagnosis'].map(new_target)
mel_idx = new_target['melanoma']

df = df[['filepath','diagnosis', 'target', 'is_ext']]

class_names = list(df['diagnosis'].unique())

# splitting train and validation data by 20%
df_valid = df[:11471]
df_train = df[11472:].reset_index()
df_train = df_train.drop(columns = ['index'])

# create model

def create_model(n_classes):
    model = models.resnet50(pretrained=True)

    n_features = model.fc.in_features
    model.fc = nn.Linear(n_features, n_classes)
    return model.to(device)

# model
base_model = create_model(len(class_names))

# train model
base_model, history = train(base_model, device, num_epochs = 15)

Code Objective
The purpose of the project is to classify skin cancer images by creating 8 different target variables from the given datasets (i.e the competition was about classifying benign and malignant images but I used the diagnosis column on the dataset as my target variable as the data was really skewed).  The model used is Resnet-50 from torchvision models.
These were the data used
skin images (this year competition): https://www.kaggle.com/cdeotte/jpeg-melanoma-384x384
skin images (last year competition): https://www.kaggle.com/cdeotte/jpeg-isic2019-384x384
I decided to create a Flask application out of this but, the CUDA memory was always causing a runtime error
RuntimeError: CUDA out of memory. Tried to allocate 144.00 MiB (GPU 0; 2.00 GiB total capacity; 1.21 GiB already allocated; 43.55 MiB free; 1.23 GiB reserved in total by PyTorch)

These are the details about my Nvidia GPU
Sun Sep 13 19:09:34 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 451.67       Driver Version: 451.67       CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name            TCC/WDDM | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce 940MX      WDDM  | 00000000:01:00.0 Off |                  N/A |
| N/A   63C    P8    N/A /  N/A |     37MiB /  2048MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

# more information about my GPU
==============NVSMI LOG==============

Timestamp                                 : Sun Sep 13 19:11:22 2020
Driver Version                            : 451.67
CUDA Version                              : 11.0

Attached GPUs                             : 1
GPU 00000000:01:00.0
Product Name                          : GeForce 940MX
Product Brand                         : GeForce
Display Mode                          : Disabled
Display Active                        : Disabled
Persistence Mode                      : N/A
Accounting Mode                       : Disabled
Accounting Mode Buffer Size           : 4000
Driver Model
    Current                           : WDDM
    Pending                           : WDDM
Serial Number                         : N/A
GPU UUID                              : GPU-9a8c69df-26f2-2a98-3712-ea22f6add038
Minor Number                          : N/A
VBIOS Version                         : 82.08.6D.00.8C
MultiGPU Board                        : No
Board ID                              : 0x100
GPU Part Number                       : N/A
Inforom Version
    Image Version                     : N/A
    OEM Object                        : N/A
    ECC Object                        : N/A
    Power Management Object           : N/A
GPU Operation Mode
    Current                           : N/A
    Pending                           : N/A
GPU Virtualization Mode
    Virtualization Mode               : None
    Host VGPU Mode                    : N/A
IBMNPU
    Relaxed Ordering Mode             : N/A
PCI
    Bus                               : 0x01
    Device                            : 0x00
    Domain                            : 0x0000
    Device Id                         : 0x134D10DE
    Bus Id                            : 00000000:01:00.0
    Sub System Id                     : 0x83F9103C
    GPU Link Info
        PCIe Generation
            Max                       : 3
            Current                   : 1
        Link Width
            Max                       : 4x
            Current                   : 4x
    Bridge Chip
        Type                          : N/A
        Firmware                      : N/A
    Replays Since Reset               : 0
    Replay Number Rollovers           : 0
    Tx Throughput                     : 0 KB/s
    Rx Throughput                     : 0 KB/s
Fan Speed                             : N/A
Performance State                     : P8
Clocks Throttle Reasons
    Idle                              : Not Active
    Applications Clocks Setting       : Not Active
    SW Power Cap                      : Not Active
    HW Slowdown                       : Not Active
        HW Thermal Slowdown           : N/A
        HW Power Brake Slowdown       : N/A
    Sync Boost                        : Not Active
    SW Thermal Slowdown               : Not Active
    Display Clock Setting             : Not Active
FB Memory Usage
    Total                             : 2048 MiB
    Used                              : 37 MiB
    Free                              : 2011 MiB
BAR1 Memory Usage
    Total                             : 256 MiB
    Used                              : 225 MiB
    Free                              : 31 MiB
Compute Mode                          : Default
Utilization
    Gpu                               : 0 %
    Memory                            : 0 %
    Encoder                           : N/A
    Decoder                           : N/A
Encoder Stats
    Active Sessions                   : 0
    Average FPS                       : 0
    Average Latency                   : 0
FBC Stats
    Active Sessions                   : 0
    Average FPS                       : 0
    Average Latency                   : 0
Ecc Mode
    Current                           : N/A
    Pending                           : N/A
ECC Errors
    Volatile
        Single Bit
            Device Memory             : N/A
            Register File             : N/A
            L1 Cache                  : N/A
            L2 Cache                  : N/A
            Texture Memory            : N/A
            Texture Shared            : N/A
            CBU                       : N/A
            Total                     : N/A
        Double Bit
            Device Memory             : N/A
            Register File             : N/A
            L1 Cache                  : N/A
            L2 Cache                  : N/A
            Texture Memory            : N/A
            Texture Shared            : N/A
            CBU                       : N/A
            Total                     : N/A
    Aggregate
        Single Bit
            Device Memory             : N/A
            Register File             : N/A
            L1 Cache                  : N/A
            L2 Cache                  : N/A
            Texture Memory            : N/A
            Texture Shared            : N/A
            CBU                       : N/A
            Total                     : N/A
        Double Bit
            Device Memory             : N/A
            Register File             : N/A
            L1 Cache                  : N/A
            L2 Cache                  : N/A
            Texture Memory            : N/A
            Texture Shared            : N/A
            CBU                       : N/A
            Total                     : N/A
Retired Pages
    Single Bit ECC                    : N/A
    Double Bit ECC                    : N/A
    Pending Page Blacklist            : N/A
Remapped Rows                         : N/A
Temperature
    GPU Current Temp                  : 60 C
    GPU Shutdown Temp                 : 99 C
    GPU Slowdown Temp                 : 94 C
    GPU Max Operating Temp            : 90 C
    Memory Current Temp               : N/A
    Memory Max Operating Temp         : N/A
Power Readings
    Power Management                  : N/A
    Power Draw                        : N/A
    Power Limit                       : N/A
    Default Power Limit               : N/A
    Enforced Power Limit              : N/A
    Min Power Limit                   : N/A
    Max Power Limit                   : N/A
Clocks
    Graphics                          : 405 MHz
    SM                                : 405 MHz
    Memory                            : 405 MHz
    Video                             : 396 MHz
Applications Clocks
    Graphics                          : 1006 MHz
    Memory                            : 1001 MHz
Default Applications Clocks
    Graphics                          : 1004 MHz
    Memory                            : 1001 MHz
Max Clocks
    Graphics                          : 1241 MHz
    SM                                : 1241 MHz
    Memory                            : 1001 MHz
    Video                             : 1216 MHz
Max Customer Boost Clocks
    Graphics                          : N/A
Clock Policy
    Auto Boost                        : N/A
    Auto Boost Default                : N/A
Processes                             : None

if I try running this on the CPU, the whole system freezes to the point where I have to manually restart the computer. Also if I try running the code with lower image resolution, lower batch sizes etc, each epoch takes around 12 hours to complete on a CPU which is definitely impractical.


title: exporting pytorch weights to keras

body:
I have a pretrained pytorch model whose weights I need to use in another keras model.

I am trying pytorch2keras github repository to convert pytorch weights .pth to keras .h5

my model looks like this:

import numpy as np
import torch.nn as nn
import torch.nn.functional as F

class BaseNetwork(nn.Module):
def __init__(self, name, channels=1):
    super(BaseNetwork, self).__init__()
    self._name = name
    self._channels = channels

def name(self):
    return self._name

def initialize_weights(self):
    for m in self.modules():
        if isinstance(m, nn.Conv2d):
            nn.init.xavier_uniform(m.weight, gain=np.sqrt(2))
            if m.bias is not None:
                m.bias.data.zero_()
        elif isinstance(m, nn.BatchNorm2d):
            m.weight.data.fill_(1)
            m.bias.data.zero_()
        elif isinstance(m, nn.Linear):
            m.weight.data.normal_(0, 0.01)
            m.bias.data.zero_()
class ImageWiseNetwork(BaseNetwork):
def __init__(self, channels=1):
    super(ImageWiseNetwork, self).__init__('iw' + str(channels), channels)

    self.features = nn.Sequential(
        # Block 1
        nn.Conv2d(in_channels=12 * channels, out_channels=64, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(64),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=64, out_channels=64, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(64),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=64, out_channels=64, kernel_size=2, stride=2),
        nn.BatchNorm2d(64),
        nn.ReLU(inplace=True),

        # Block 2
        nn.Conv2d(in_channels=64, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(128),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=128, out_channels=128, kernel_size=3, stride=1, padding=1),
        nn.BatchNorm2d(128),
        nn.ReLU(inplace=True),
        nn.Conv2d(in_channels=128, out_channels=128, kernel_size=2, stride=2),
        nn.BatchNorm2d(128),
        nn.ReLU(inplace=True),

        nn.Conv2d(in_channels=128, out_channels=1, kernel_size=1, stride=1),
    )

    self.classifier = nn.Sequential(
        nn.Linear(1 * 16 * 16, 128),
        nn.ReLU(inplace=True),
        nn.Dropout(0.5, inplace=True),

        nn.Linear(128, 128),
        nn.ReLU(inplace=True),
        nn.Dropout(0.5, inplace=True),

        nn.Linear(128, 64),
        nn.ReLU(inplace=True),
        nn.Dropout(0.5, inplace=True),

        nn.Linear(64, 4),
    )

    self.initialize_weights()

def forward(self, x):
    x = self.features(x)
    x = x.view(x.size(0), -1)
    x = self.classifier(x)
    x = F.log_softmax(x, dim=1)
    return x

created the object model with model = ImageWiseNetwork()
and then loaded the trained weights as :

model.load_state_dict(torch.load('Path\to\weights\weights_iw1.pth'))

followed by

input_np = np.random.uniform(0, 1, (3, 12, 512, 512))
input_var = Variable(torch.FloatTensor(input_np))and the

and then

from converter import pytorch_to_keras
k_model = pytorch_to_keras(model, input_var, (3, 512, 512,), verbose=True)

I am getting following error with traceback:

> > RuntimeError                              Traceback (most recent call last) <ipython-input-29-7c5d264109b9> in <module>()
>       1 from converter import pytorch_to_keras
> ----> 2 k_model = pytorch_to_keras(model, input_var, (3, 512, 512,), verbose=True)
>
> ~\Downloads\pytorch2keras-master\pytorch2keras-master\pytorch2keras\converter.py
> in pytorch_to_keras(model, args, input_shape, change_ordering,
> training, verbose)
>      84
>      85     with set_training(model, training):
> ---> 86         trace, torch_out = torch.jit.get_trace_graph(model, args)
>      87
>      88     if orig_state_dict_keys != _unique_state_dict(model).keys():
>
> ~\Anaconda3\lib\site-packages\torch\jit\__init__.py in
> get_trace_graph(f, args, kwargs, nderivs)
>     253     if not isinstance(args, tuple):
>     254         args = (args,)
> --> 255     return LegacyTracedModule(f, nderivs=nderivs)(*args, **kwargs)
>     256
>     257
>
> ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in
> __call__(self, *input, **kwargs)
>     487             hook(self, input)
>     488         if torch.jit._tracing:
> --> 489             result = self._slow_forward(*input, **kwargs)
>     490         else:
>     491             result = self.forward(*input, **kwargs)
>
> ~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in
> _slow_forward(self, *input, **kwargs)
>     465     def _slow_forward(self, *input, **kwargs):
>     466         input_vars = tuple(torch.autograd.function._iter_tensors(input))
> --> 467         tracing_state = torch.jit.get_tracing_state(input_vars)
>     468         if not tracing_state:
>     469             return self.forward(*input, **kwargs)
>
> ~\Anaconda3\lib\site-packages\torch\jit\__init__.py in
> get_tracing_state(args)
>      33     if not torch._C._is_tracing(args):
>      34         return None
> ---> 35     return torch._C._get_tracing_state(args)
>      36
>      37
>
> RuntimeError:
> C:\ProgramData\Miniconda3\conda-bld\pytorch_1524549877902\work\torch/csrc/jit/tracer.h:117:
> getTracingState: Assertion `var_state == state` failed.

I cannot figure out the error.  What could be the possible reason of this error.


title: What does "conda install pytorch torchvision cudatoolkit=10.2 -c pytorch" install?

body:
I tried installing PyTorch on my system with not just the pip install pytorch -c pytorch command but with conda install pytorch torchvision cudatoolkit=10.2 -c pytorch but I see a very long command prompt running since last 2 hours giving a very large outputs.
Is the process going good? I've CUDA 10.2 installed and also tensorflow-gpu and cuDNN too.
I see these arguments being shown.
Here's the long list of commands running since past 2 hours

https://drive.google.com/file/d/1D4p9bfxNHXZfe8PCjc45OPlPZfasxk_4/view?usp=sharing

Thanks in advance.


title: About pytorch learning rate scheduler

body:
here is my code

optimizer = optim.SGD(net.parameters(), lr=0.1)
scheduler = optim.lr_scheduler.StepLR(optimizer, step_size=5, gamma=0.5)

for i in range(15):
    lr = scheduler.get_lr()[0]
    lr1 = optimizer.param_groups[0]["lr"]
    print(i, lr, lr1)
    scheduler.step()

And here is the result

0 0.1 0.1
1 0.1 0.1
2 0.1 0.1
3 0.1 0.1
4 0.1 0.1
5 0.025 0.05
6 0.05 0.05
7 0.05 0.05
8 0.05 0.05
9 0.05 0.05
10 0.0125 0.025
11 0.025 0.025
12 0.025 0.025
13 0.025 0.025
14 0.025 0.025

We can see that the when scheduler.step() is applied, the learning rate first decreases 0.25 times, then bounces back to 0.5 times. Is it the problem of scheduler.get_lr() lr or the problem of scheduler.step()

About the envirioment

python=3.6.9
pytorch=1.1.0

In addition, I can't find this problem when pytorch=0.4.1 is used.


title: Pytorch runtimeError "matrices expected, got 1D, 2D tensors at"

body:
I begin to practice Pytorch ，try to use torch.mm() method。

Below is my code

import torch
import numpy as np
from torch.autograd import Variable
num_x = np.array([[1.0, 2.0]
                  ,[3.0,4.0]])

tensor_x = torch.from_numpy(num_x)
x = Variable(tensor_x,requires_grad = True)
s = Variable(torch.DoubleTensor([0.01,0.02]),requires_grad = True)
print(s)
s = s.mm(x)
print(s)

Unfortunately,there is a runtime error

*RuntimeError                              Traceback (most recent call last)
<ipython-input-58-e8a58ffb2545> in <module>()
      9 s = Variable(torch.DoubleTensor([0.01,0.02]),requires_grad = True)
     10 print(s)
---> 11 s = s.mm(x)
     12 print(s)
RuntimeError: matrices expected, got 1D, 2D tensors at /pytorch/aten/src/TH/generic/THTensorMath.cpp:131*

How can I fix this problem。
your reply is appreciated


title: PyTorch's dataloader "too many open files" error when no files should be open

body:
So this is a minimal code which illustrates the issue:

This is the Dataset:

class IceShipDataset(Dataset):
    BAND1='band_1'
    BAND2='band_2'
    IMAGE='image'

    @staticmethod
    def get_band_img(sample,band):
        pic_size=75
        img=np.array(sample[band])
        img.resize(pic_size,pic_size)
        return img

    def __init__(self,data,transform=None):
        self.data=data
        self.transform=transform

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):

        sample=self.data[idx]
        band1_img=IceShipDataset.get_band_img(sample,self.BAND1)
        band2_img=IceShipDataset.get_band_img(sample,self.BAND2)
        img=np.stack([band1_img,band2_img],2)
        sample[self.IMAGE]=img
        if self.transform is not None:
                sample=self.transform(sample)
        return sample

And this is the code which fails:

PLAY_BATCH_SIZE=4
#load data. There are 1604 examples.
with open('train.json','r') as f:
        data=f.read()
data=json.loads(data)

ds=IceShipDataset(data)
playloader = torch.utils.data.DataLoader(ds, batch_size=PLAY_BATCH_SIZE,
                                          shuffle=False, num_workers=4)
for i,data in enumerate(playloader):
        print(i)

It gives that weird open files error in the for loop…
My torch version is 0.3.0.post4

If you want the json file, it is available at Kaggle (https://www.kaggle.com/c/statoil-iceberg-classifier-challenge)

I should mention that the error has nothing to do with the state of my laptop:

yoni@yoni-Lenovo-Z710:~$ lsof | wc -l
89114
yoni@yoni-Lenovo-Z710:~$ cat /proc/sys/fs/file-max
791958

What am I doing wrong here?


title: Input_size error in LSTM of PyTorch: RuntimeError: shape ‘[10, 30, 1]’ is invalid for input of size 150

body:
everyone,
I am using LSTM to predict the stock index of someday using the ones of 30 days before it as the input only. I think in this example, the size of LSTM input should be [10,30,1],so I use t_x=x.view(10,30,1) to reshape the input. But there is an RuntimeError( shape '[10, 30, 1]' is invalid for input of size 150) when I run the code below, could you please help me find what’s the problem? Thank you:)

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.utils.data import Dataset, DataLoader
from torch.utils.data import TensorDataset

dataread_df=pd.read_csv('D:/Desktop/399300.csv')
dataread_series=pd.Series(dataread_df['value'].values)
plt.plot(dataread_series)
plt.show()

def generate_data_df(series, n):
    if len(series) <= n:
        raise Exception("The Length of series is %d, while affect by (n=%d)." % (len(series), n))
    df = pd.DataFrame()
    for i in range(n):
        df['x%d' % i] = series.tolist()[i:-(n - i)]
    df['y'] = series.tolist()[n:]
    return df
data_df = generate_data_df(dataread_series, 30)

data_numpy=np.array(data_df)
mean=np.mean(data_numpy)
std=np.std(data_numpy)
data_numpy = (data_numpy-mean)/std
train_size=int(len(data_numpy)*0.7)
test_size=len(data_numpy)-train_size
trainset_np=data_numpy[:train_size]
testset_np=data_numpy[train_size:]
train_x_np=trainset_np[:,:30]
train_y_np=trainset_np[:,30:]
test_x_np=testset_np[:,:30]
test_y_np=testset_np[:,30:]

train_x=torch.Tensor(train_x_np)
train_y=torch.Tensor(train_y_np)
test_x=torch.Tensor(test_x_np)
test_y=torch.Tensor(test_y_np)
trainset=TensorDataset(train_x,train_y)
testset=TensorDataset(test_x,test_y)
trainloader = DataLoader(trainset, batch_size=10, shuffle=True)
testloader=DataLoader(testset,batch_size=10,shuffle=True)

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()
        self.rnn=nn.LSTM(input_size=1,hidden_size=64,num_layers=1,batch_first=True)
        self.out=nn.Sequential(nn.Linear(64,1))
    def forward(self,x):
        r_out,(h_n,h_c)=self.rnn(x,None)
        out=self.out(r_out[:,-1,:])
        return out
rnn = Net()
print(rnn)

optimizer = torch.optim.Adam(rnn.parameters(), lr=0.0001)
criterion = nn.MSELoss()
train_correct=0
test_correct=0
train_total=0
test_total=0
prediction_list=[]

for epoch in range(10):
    running_loss_train=0
    running_loss_test=0
    for i,(x1,y1) in enumerate(trainloader):
        t_x1=x1.view(10,30,1)
        output=rnn(t_x1)
        loss_train=criterion(output,y1)
        optimizer.zero_grad()
        loss_train.backward()
        optimizer.step()
        running_loss_train+=loss_train.item()
    for i,(x2,y2) in enumerate(testloader):
        t_x2=x2.view(10,30,1)
        prediction=rnn(t_x2)
        loss_test=criterion(prediction,y2)
        running_loss_test+=loss_test.item()
        prediction_list.append(prediction)
    print('Epoch {} Train Loss:{}, Test Loss:{}'.format(epoch+1,running_loss_train,running_loss_test))
    prediction_list_plot=np.array(prediction_list)
    plt.plot(test_y_np.flatten(),'r-',linewidth=0.1,label='real data')
    plt.plot(prediction_list_plot.flatten(),'b-',linewidth=0.1,label='predicted data')
    plt.show()
print('Finish training')

RuntimeError:

RuntimeError                              Traceback (most recent call last)
<ipython-input-3-fb8cb4c93775> in <module>
     71     running_loss_test=0
     72     for i,(x1,y1) in enumerate(trainloader):
---> 73         t_x1=x1.view(10,30,1)
     74         output=rnn(t_x1)
     75         loss_train=criterion(output,y1)

RuntimeError: shape '[10, 30, 1]' is invalid for input of size 150


title: How to get the predict probability?

body:
This code get the 1 or 0 value from model.
If I want to get the probability of the prediction
Which line should I change?

from torch.autograd import Variable
results = []
#names = []
with torch.no_grad():
    model.eval()
    print('===============================================start')
    for num, data in enumerate(test_loader):
        #print(num)
        print("=====================================================")

        imgs, label = data
        imgs,labels = imgs.to(device), label.to(device)
        test = Variable(imgs)
        output = model(test)
        #print(output)
        ps = torch.exp(output)
        print(ps)
        top_p, top_class = ps.topk(1, dim = 1)
        results += top_class.cpu().numpy().tolist()

model = models.resnet50(pretrained=True)
model.fc = nn.Linear(2048, num_classes)
model.cuda()


Recently, I've run this project（https://github.com/milesial/Pytorch-UNet）and when I run predict.py, it came to this error:

Traceback (most recent call last):
  File "predict.py", line 131, in <module>
    device=device)
  File "predict.py", line 43, in predict_img
    transforms.Resize(full_img.shape[2]),
AttributeError: 'PngImageFile' object has no attribute 'shape'

Then I check the code and I find that this version is extremely similar to the version before

previous version:

img_height = full_img.size[1]
        tf = transforms.Compose(
            [
                transforms.ToPILImage(),
                transforms.Resize(img_height),
                transforms.ToTensor()
            ]
        )

present version:

    tf = transforms.Compose(
        [
            transforms.ToPILImage(),
            transforms.Resize(full_img.shape[1]),
            transforms.ToTensor()
        ]
    )

I don't know what had happened but if any friend know something about this question I will be great gratefull to you! Thanks!


title: Is it possible to split the training DataLoader (and dataset) into training and validation datasets?

body:
The torchvision package provides easy access to commonly used datasets. You would use them like this:

trainset = torchvision.datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                          shuffle=True, num_workers=2)

testset = torchvision.datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                         shuffle=False, num_workers=2)

Apparently, you can only switch between train=True and train=False. The docs explain:

  train (bool, optional) – If True, creates dataset from training.pt,
  otherwise from test.pt.

But this goes against the common practice of having a three-way split. For serious work, I need another DataLoader with a validation set. Also, it would be nice to specify the split proportions myself. They don't say what percentage of the dataset is reserved for testing, maybe I would like to change that.

I assume that this is a conscious design decision. Everyone working on one of these datasets is supposed to use the same testset. That makes results comparable. But I still need to get a validation set out of the trainloader. Is it possible to split a DataLoader into two separate streams of data?


title: BertTokenizer - when encoding and decoding sequences extra spaces appear

body:
When using Transformers from HuggingFace I am facing a problem with the encoding and decoding method.

I have a the following string:

test_string = 'text with percentage%'

Then I am running the following code:

import torch
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-cased')

test_string = 'text with percentage%'

# encode Converts a string in a sequence of ids (integer), using the tokenizer and vocabulary.
input_ids = tokenizer.encode(test_string)
output = tokenizer.decode(input_ids)

And the output looks like this:

'text with percentage %'

With an extra space before the %. I have tried the extra arguments like clean_up_tokenization_spaces  but this is for something different.

How what should I use in the decoding and encoding to get exactly the same text before and after. This also happens for other special signs.


title: pytorch nn.Sequential(*list) TypeError: list is not a Module subclass

body:
when I use pytorch to train a model, I tried to print the whole net structure

so I packed all the layers in a list
then I use nn.Sequential(*list)

but it doesn't work, and the TypeError: list is not a Module subclass


title: Taking the maximum values of each row in a tensor [PyTorch]

body:
Suppose I have a tensor of the form

[[-5, 0, -1],
 [3, 100, 87],
 [17, -34, 2],
 [45, 1, 25]]

I want to find the maximum value in each row and return a rank 1 tensor as follows:

[0,
 100,
 17,
 45]

How would I do this in PyTorch?


title: pytorch 0.4.1 - ‘LSTM’ object has no attribute ‘weight_ih_l’

body:
Simple question. I’d like to see the initialized parameter of LSTM. How do I see it?
Do I need to always put lstm in the model to see the params?

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

torch.manual_seed(1)
torch.__version__

lstm = nn.LSTM(3, 3)

lstm.weight_ih_l

  --------------------------------------------------------------------------- AttributeError                            Traceback (most recent call
  last)  in ()
        9 lstm = nn.LSTM(3, 3)
       10
  ---> 11 lstm.weight_ih_l

  ~/anaconda3/envs/pytorch0.41/lib/python3.6/site-packages/torch/nn/modules/module.py
  in getattr(self, name)
      516                 return modules[name]
      517         raise AttributeError("'{}' object has no attribute '{}'".format(
  --> 518             type(self).name, name))
      519
      520     def setattr(self, name, value):

  AttributeError: 'LSTM' object has no attribute 'weight_ih_l'


title: Loss doesn't decrease in Pytorch CNN

body:
I'm doing a CNN with Pytorch for a task, but it won't learn and improve the accuracy. I made a version working with the MNIST dataset so I could post it here. I'm just looking for an answer as to why it's not working. The architecture is fine, I implemented it in Keras and I had over 92% accuracy after 3 epochs. Note: I reshaped the MNIST into 60x60 pictures because that's how the pictures are in my "real" problem.

import numpy as np
from PIL import Image
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torch.utils.data import DataLoader
from torch.autograd import Variable
from keras.datasets import mnist

(x_train, y_train), (x_test, y_test) = mnist.load_data()

def resize(pics):
    pictures = []
    for image in pics:
        image = Image.fromarray(image).resize((dim, dim))
        image = np.array(image)
        pictures.append(image)
    return np.array(pictures)

dim = 60

x_train, x_test = resize(x_train), resize(x_test) # because my real problem is in 60x60

x_train = x_train.reshape(-1, 1, dim, dim).astype('float32') / 255
x_test = x_test.reshape(-1, 1, dim, dim).astype('float32') / 255
y_train, y_test = y_train.astype('float32'), y_test.astype('float32')

if torch.cuda.is_available():
    x_train = torch.from_numpy(x_train)[:10_000]
    x_test = torch.from_numpy(x_test)[:4_000]
    y_train = torch.from_numpy(y_train)[:10_000]
    y_test = torch.from_numpy(y_test)[:4_000]

class ConvNet(nn.Module):

    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 32, 3)
        self.conv2 = nn.Conv2d(32, 64, 3)
        self.conv3 = nn.Conv2d(64, 128, 3)

        self.fc1 = nn.Linear(5*5*128, 1024)
        self.fc2 = nn.Linear(1024, 2048)
        self.fc3 = nn.Linear(2048, 1)

    def forward(self, x):
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv2(x)), (2, 2))
        x = F.max_pool2d(F.relu(self.conv3(x)), (2, 2))

        x = x.view(x.size(0), -1)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.dropout(x, 0.5)
        x = torch.sigmoid(self.fc3(x))
        return x

net = ConvNet()

optimizer = optim.Adam(net.parameters(), lr=0.03)

loss_function = nn.BCELoss()

class FaceTrain:

    def __init__(self):
        self.len = x_train.shape[0]
        self.x_train = x_train
        self.y_train = y_train

    def __getitem__(self, index):
        return x_train[index], y_train[index].unsqueeze(0)

    def __len__(self):
        return self.len

class FaceTest:

    def __init__(self):
        self.len = x_test.shape[0]
        self.x_test = x_test
        self.y_test = y_test

    def __getitem__(self, index):
        return x_test[index], y_test[index].unsqueeze(0)

    def __len__(self):
        return self.len

train = FaceTrain()
test = FaceTest()

train_loader = DataLoader(dataset=train, batch_size=64, shuffle=True)
test_loader = DataLoader(dataset=test, batch_size=64, shuffle=True)

epochs = 10
steps = 0
train_losses, test_losses = [], []
for e in range(epochs):
    running_loss = 0
    for images, labels in train_loader:
        optimizer.zero_grad()
        log_ps = net(images)
        loss = loss_function(log_ps, labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    else:
        test_loss = 0
        accuracy = 0

        with torch.no_grad():
            for images, labels in test_loader:
                log_ps = net(images)
                test_loss += loss_function(log_ps, labels)
                ps = torch.exp(log_ps)
                top_p, top_class = ps.topk(1, dim=1)
                equals = top_class.type('torch.LongTensor') == labels.type(torch.LongTensor).view(*top_class.shape)
                accuracy += torch.mean(equals.type('torch.FloatTensor'))
        train_losses.append(running_loss/len(train_loader))
        test_losses.append(test_loss/len(test_loader))
        print("[Epoch: {}/{}] ".format(e+1, epochs),
              "[Training Loss: {:.3f}] ".format(running_loss/len(train_loader)),
              "[Test Loss: {:.3f}] ".format(test_loss/len(test_loader)),
              "[Test Accuracy: {:.3f}]".format(accuracy/len(test_loader)))


title: Running through a dataloader in Pytorch using Google Colab

body:
I am trying to use Pytorch to run classification on a dataset of images of cats and dogs. In my code I am so far downloading the data and going into the folder train which has two folders in it called "cats" and "dogs." I am then trying to load this data into a dataloader and iterate through batches, but it is giving me some error I don't understand in the iteration step.

Since it is Google Colabs I have code in there for downloading data and installing libraries. Any other advice on my code so far would be appreciated as well.

!pip install torch
!pip install torchvision

from __future__ import print_function, division
import os
import torch
import pandas as pd
import numpy as np
# For showing and formatting images
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

# For importing datasets into pytorch
import torchvision.datasets as dataset

# Used for dataloaders
import torch.utils.data as data

# For pretrained resnet34 model
import torchvision.models as models

# For optimisation function
import torch.nn as nn
import torch.optim as optim

!wget http://files.fast.ai/data/dogscats.zip
!unzip dogscats.zip

batch_size = 256

train_raw = dataset.ImageFolder(PATH+"train", transform=transforms.ToTensor())
train_loader = data.DataLoader(train_raw, batch_size=batch_size, shuffle=True)

for batch_idx, (data, target) in enumerate(train_loader):
  print("Data: ", batch_idx)

The error comes up on the last lines and is below:

RuntimeErrorTraceback (most recent call last)
<ipython-input-66-c32dd0c1b880> in <module>()
----> 1 for batch_idx, (data, target) in enumerate(train_loader):
      2   print("Data: ", batch_idx)
      3

/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.pyc in __next__(self)
    257         if self.num_workers == 0:  # same-process loading
    258             indices = next(self.sample_iter)  # may raise StopIteration
--> 259             batch = self.collate_fn([self.dataset[i] for i in indices])
    260             if self.pin_memory:
    261                 batch = pin_memory_batch(batch)

/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.pyc in default_collate(batch)
    133     elif isinstance(batch[0], collections.Sequence):
    134         transposed = zip(*batch)
--> 135         return [default_collate(samples) for samples in transposed]
    136
    137     raise TypeError((error_msg.format(type(batch[0]))))

/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.pyc in default_collate(batch)
    110             storage = batch[0].storage()._new_shared(numel)
    111             out = batch[0].new(storage)
--> 112         return torch.stack(batch, 0, out=out)
    113     elif elem_type.__module__ == 'numpy' and elem_type.__name__ != 'str_' \
    114             and elem_type.__name__ != 'string_':

/usr/local/lib/python2.7/dist-packages/torch/functional.pyc in stack(sequence, dim, out)
     62     inputs = [t.unsqueeze(dim) for t in sequence]
     63     if out is None:
---> 64         return torch.cat(inputs, dim)
     65     else:
     66         return torch.cat(inputs, dim, out=out)

RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 400 and 487 in dimension 2 at /pytorch/torch/lib/TH/generic/THTensorMath.c:2897

Thanks


title: How to convert list of str into Pytorch tensor

body:
How do I convert a list of strings into a string/character tensor in Pytorch?

Related example with numpy:

import numpy as np

mylist = ["this","is","my","list"]

np.array([mylist])

Returns:

array([['this', 'is', 'my', 'list']], dtype='<U4')

However, in pytorch:

torch.tensor(mylist)

Returns:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-156-36722d81da09> in <module>
----> 1 torch.tensor(mylist)

ValueError: too many dimensions 'str'

A tensor is a multi-dimensional array, so I'm assuming this is possible pytorch.

Note: this post does not answer my question


title: Partial slices in pytorch / numpy with arbitrary and variable number of dimensions

body:
Given an 2-dimensional tensor in numpy (or in pytorch), I can partially slice along all dimensions at once as follows:

>>> import numpy as np
>>> a = np.arange(2*3).reshape(2,3)
array([[ 0,  1,  2,  3],
       [ 4,  5,  6,  7],
       [ 8,  9, 10, 11]])
>>> a[1:,1:]
array([[ 5,  6,  7],
       [ 9, 10, 11]])

How can I achieve the same slicing pattern regardless of the number of dimensions in the tensor if I do not know the number of dimensions at implementation time? (i.e. I want a[1:] if a has only one dimension, a[1:,1:] for two dimensions, a[1:,1:,1:] for three dimensions, and so on)

It would be nice if I could do it in a single line of code like the following, but this is invalid:

a[(1:,) * len(a.shape)]  # SyntaxError: invalid syntax

I am specifically interested in a solution that works for pytorch tensors (just substitute torch for numpy above and the example is the same), but I figure it is likely and best if the solution works for both numpy and pytorch.


title: How to ensure all PyTorch code fully utilises GPU on Google Colab

body:
I am new to PyTorch and have been doing some tutorial on CIFAR10, specifically with Google Colab since I personally do not have a GPU to experiment on it yet.

I have successfully trained my neural network but I'm not sure whether my code is using the GPU from Colab, because the training time taken with Colab is not significantly faster than my 2014 MacBook Pro (without GPU).

I checked and my notebook is indeed running Tesla K80 but somehow the training speed is slow. So I think perhaps my code is not equipped with GPU syntax but I couldn't figure out which part is that.

# install PyTorch
from os import path
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'
!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.0-{platform}-linux_x86_64.whl torchvision

import torch
import torch.nn as nn
from torch.optim import Adam
from torchvision import transforms
from torch.autograd import Variable
import torchvision.datasets as datasets
from torch.utils.data import DataLoader, TensorDataset
import matplotlib.pyplot as plt

device = torch.device('cuda:0' if torch.cuda.is_available() else 'cpu')
print(device)

# hyperparameters
n_epochs = 50
n_batch_size = 200
n_display_step = 200
n_learning_rate = 1e-3
n_download_cifar = True

# import cifar
# more about cifar https://www.cs.toronto.edu/~kriz/cifar.html

transform = transforms.Compose(
    [transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))])

train_dataset = datasets.CIFAR10(
                    root="../datasets/cifar",
                    train=True,
                    transform=transform,
                    download=n_download_cifar)
test_dataset = datasets.CIFAR10(
                    root="../datasets/cifar",
                    train=False,
                    transform=transform)

# create data loader
train_loader = DataLoader(train_dataset, batch_size=n_batch_size, shuffle=True, num_workers=2)
test_loader = DataLoader(test_dataset, batch_size=n_batch_size, shuffle=False)

# build CNN
class CNN(nn.Module):

    def __init__(self):
        super(CNN, self).__init__()

        # (3, 32, 32)
        self.conv1 = nn.Sequential(
            nn.Conv2d(3, 32, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2, 2))

        # (32, 16, 16)
        self.conv2 = nn.Sequential(
            nn.Conv2d(32, 16, 5, 1, 2),
            nn.ReLU(),
            nn.MaxPool2d(2, 2))

        # (16, 8, 8)
        self.out = nn.Linear(16 * 8 * 8, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = x.view(x.size(0), -1)
        out = self.out(x)
        return out

net = CNN()
net.to(device)
criterion = nn.CrossEntropyLoss()
optimizer = Adam(net.parameters(), lr=n_learning_rate)

def get_accuracy(model, loader):
    model.eval()
    n_samples = 0
    n_correct = 0

    with torch.no_grad():
        for step, (x, y) in enumerate(loader):
            x, y = Variable(x).to(device), Variable(y).to(device)
            out = model(x)
            _, pred = torch.max(out, 1)
            n_samples += y.size(0)
            n_correct += (pred == y).sum().item()

    return n_correct / n_samples

def train(model, criterion, optimizer, epochs, train_loader, test_loader):
    for epoch in range(epochs):
        for step, (x, y) in enumerate(train_loader):
            model.train()
            x, y = Variable(x).to(device), Variable(y).to(device)
            out = model(x)
            loss = criterion(out, y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if step % n_display_step == 0:
                print("Epoch {:2d} Loss {:.4f} Accuracy (Train | Test) {:.4f} {:.4f}".format(epoch, loss.item(), get_accuracy(model, train_loader), get_accuracy(model, test_loader)))

train(net, criterion, optimizer, n_epochs, train_loader, test_loader)


title: How to wrap PyTorch functions and implement autograd?

body:
I'm working through the PyTorch tutorial on Defining new autograd functions. The autograd function I want to implement is a wrapper around torch.nn.functional.max_pool1d. Here is what I have so far:

import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.autograd as tag

class SquareAndMaxPool1d(tag.Function):

    @staticmethod
    def forward(ctx, input, kernel_size, stride=None, padding=0, dilation=1, \
                return_indices=False, ceil_mode=False):
        ctx.save_for_backward( input )

        inputC = input.clone() #copy input
        inputC *= inputC

        output = F.max_pool1d(inputC, kernel_size, stride=stride, \
                              padding=padding, dilation=dilation, \
                              return_indices=return_indices, \
                              ceil_mode=ceil_mode)

        return output

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_tensors
        grad_input = get_max_pool1d_grad_somehow(grad_output)
        return 2.0*input*grad_input

My question is: how to I get the gradient of the wrapped function? I know that there are probably other ways to do this given how simple the example I present is, but what I want to do fits this framework and requires me to implement an autograd function.

Edit: After examining this blog post I decided to try the following for backward:

def backward(ctx, grad_output):
    input, output = ctx.saved_tensors
    grad_input = output.backward(grad_output)
    return 2.0*input*grad_input

with output added to the saved variables. I then run the following code:

x = np.random.randn(1,1,5)
xT = torch.from_numpy(x)
xT.requires_grad=True
f = SquareAndMaxPool1d.apply
s = torch.sum(f(xT,2))
s.backward()

and I get Bus error: 10.

Say, xT is tensor([[[ 1.69533562, -0.21779421,  2.28693953, -0.86688095, -1.01033497]]], dtype=torch.float64), then I would expect to find that xT.grad is  tensor([[[ 3.39067124, -0.        ,  9.14775812, -0.        , -2.02066994]]], dtype=torch.float64) after calling s.backward() (that is 2*x*grad_of_max_pool, with grad_of_max_pool containing tensor([[[1., 0., 2., 0., 1.]]], dtype=torch.float64)).

I've figured out why I get a Bus error: 10. It appears that the above code leads to a recursive call of my backward at grad_input = output.backward(grad_output). So I need to find some other way to get the gradient of max_pool1d. I know how to implement this in pure Python, but the result would be much slower than if I could wrap the library code.


title: torch.argmax() fails to find a maximum value in a tensor containing data

body:
I have a tensor of shape [batch_size, channel, depth, height, width]:

torch.Size([1, 1, 32, 64, 64])

with data:

tensor([[[[[-1.8540, -2.8068, -2.7348,  ..., -1.9074, -1.8227, -1.4540],
           [-2.7012, -4.2785, -3.7421,  ..., -3.1961, -2.7786, -1.8042],
           [-2.1924, -4.2202, -4.4361,  ..., -3.1203, -2.9282, -2.3800],
           ...,
           [-2.7429, -4.3133, -4.4029,  ..., -4.4971, -5.3288, -2.8659],
           [-3.0169, -4.0198, -3.6886,  ..., -3.7542, -4.5010, -2.4040],
           [-1.6174, -2.5340, -2.3974,  ..., -1.9249, -2.4107, -1.2664]],

          [[-2.7840, -3.2442, -3.6118,  ..., -3.1365, -2.8342, -1.9516],
           [-3.5764, -4.9253, -5.9196,  ..., -4.8373, -4.2233, -3.3809],
           [-3.1701, -5.0826, -5.6424,  ..., -5.2955, -4.6438, -3.4820],
           ...,
           [-4.0111, -6.1946, -5.6582,  ..., -6.7947, -6.5305, -4.2866],
           [-4.2103, -6.6177, -6.0420,  ..., -5.8076, -6.2128, -3.2093],
           [-2.3174, -4.1081, -3.7369,  ..., -3.5552, -3.1871, -1.9736]],

          [[-2.8441, -4.1575, -3.8233,  ..., -3.5065, -3.4313, -2.3030],
           [-4.0076, -5.4939, -6.2451,  ..., -4.6663, -4.9835, -3.1530],
           [-3.4737, -5.6347, -6.0232,  ..., -5.6191, -5.2626, -3.6109],
           ...,
           [-3.8026, -5.3676, -6.1460,  ..., -7.6695, -6.7640, -4.1681],
           [-4.4012, -6.1293, -6.1859,  ..., -6.0011, -6.1012, -3.5307],
           [-2.7917, -4.2264, -4.1388,  ..., -4.2080, -3.5555, -1.6384]],

          ...,

          [[-2.2204, -3.5705, -4.3114,  ..., -4.2249, -3.9628, -2.9190],
           [-3.6343, -5.3445, -6.1638,  ..., -6.3998, -6.7561, -4.8491],
           [-3.4870, -5.5835, -5.6436,  ..., -6.8527, -7.2536, -4.8143],
           ...,
           [-2.4492, -3.7896, -5.4344,  ..., -6.2853, -6.0766, -3.7538],
           [-2.4723, -3.8393, -4.8480,  ..., -5.6503, -5.0375, -3.5580],
           [-1.6161, -2.9843, -3.2865,  ..., -3.2627, -3.2887, -2.5750]],

          [[-2.1509, -3.8303, -4.2807,  ..., -3.7945, -3.7561, -3.0863],
           [-3.1012, -5.1321, -6.1387,  ..., -6.5191, -6.3268, -4.4283],
           [-2.8346, -5.0640, -5.4868,  ..., -6.6515, -6.5529, -4.3672],
           ...,
           [-2.7278, -4.2538, -4.9776,  ..., -6.4153, -6.0100, -3.9929],
           [-2.8002, -4.0473, -4.7455,  ..., -5.4203, -4.7286, -3.4111],
           [-1.7964, -3.2307, -3.6329,  ..., -3.2750, -2.3952, -1.9714]],

          [[-1.4447, -2.1572, -2.4487,  ..., -2.3859, -2.9540, -1.8451],
           [-1.8075, -2.8380, -3.5621,  ..., -3.8641, -3.5828, -2.7304],
           [-1.7862, -2.9849, -3.8364,  ..., -4.3380, -4.4745, -2.8476],
           ...,
           [-1.8043, -2.5662, -2.7296,  ..., -4.2772, -3.9882, -2.8654],
           [-1.2364, -2.5228, -2.7190,  ..., -4.1142, -3.6160, -2.2325],
           [-1.0395, -1.7621, -2.5738,  ..., -2.0349, -1.5140, -1.1625]]]]]

Now to get the prediction from this I use

torch.argmax(data, 1)

which should give me the location of maximum values in the channel dimension, but instead I get a tensor containing only zeros. Even max(torch.argmax()) produces 0.

How can this be, the tensor is only a single dimension and a single batch, how can it return a 0?

To get rid of the negative values I applied torch.nn.Sigmoid() on it, but still argmax failed to find a maximum value. Which I dont understand, how can there not be a maximum value?

numpy.argmax(output.detach().numpy(), 1) gives the same output, all 0.

Am I not using argmax correctly?


I want to run a fastai deep learning model on my pc. Not train, just run the pre-trained model on my PC. I have the .pth file. I tried to import the fastai module that I installed and I recieved the error :

ImportError: cannot import name 'mobilenet_v2' from 'torchvision.models' (C:\file_path\__init__.py)

The Code I tried to execute:

#From the fastai library
from fastai import *
from torchvision.models import *
from fastai.vision import *

I can't find any solutions as to why I am getting this error.

Im running this code in anaconda, to be specific, the spyder IDE connected to my anaconda environment. I will re-edit this if anyone needs more specifications. Thank you.


title: How to perform sum pooling in PyTorch

body: How to perform sum pooling in PyTorch. Specifically, if we have input (N, C, W_in, H_in) and want output (N, C, W_out, H_out) using a particular kernel_size and stride just like nn.Maxpool2d ?


title: Truncated Backpropagation Through Time (BPTT) in Pytorch

body:
In pytorch, I train a RNN/GRU/LSTM network by starting the Backpropagation (Through Time) with :

loss.backward()

When the sequence is long, I'd like to do a Truncated Backpropagation Through Time instead of a normal Backpropagation Through Time where the whole sequence is used.

But I can't find in the Pytorch API any parameters or functions to set up the truncated BPTT. Did I miss it? Am I supposed to code it myself in Pytorch ?


title: Finding non-intersection of two pytorch tensors

body:
Thanks everyone in advance for your help! What I'm trying to do in PyTorch is something like numpy's setdiff1d. For example given the below two tensors:

t1 = torch.tensor([1, 9, 12, 5, 24]).to('cuda:0')
t2 = torch.tensor([1, 24]).to('cuda:0')

The expected output should be (sorted or unsorted):

torch.tensor([9, 12, 5])

Ideally the operations are done on GPU and no back and forth between GPU and CPU. Much appreciated!