title: How to speed up the 'Adding visible gpu devices' process in tensorflow with a 30 series card?

body:
I get stuck with that for ~2 minute every time I run the code. Many people on the Internet said that it would only take a long time in the first run, but that's not my case. Although it doesn't make anything go wrong, it's pretty annoying. When I'm stuck, the system is under pretty low usage, including the CPU, system RAM, GPU, video memory.
I'm using Nvidia Geforce RTX 3070, Windows 10 x64 20H2.Here's my environment:
# Name                    Version                   Build  Channel
blas                      1.0                         mkl    defaults
boto3                     1.16.47                  pypi_0    pypi
botocore                  1.19.47                  pypi_0    pypi
ca-certificates           2020.12.8            haa95532_0    defaults
certifi                   2020.12.5        py38haa95532_0    defaults
click                     7.1.2                    pypi_0    pypi
cudatoolkit               11.0.221             h74a9793_0    defaults
freetype                  2.10.4               hd328e21_0    defaults
intel-openmp              2020.2                      254    defaults
jmespath                  0.10.0                   pypi_0    pypi
joblib                    1.0.0                    pypi_0    pypi
jpeg                      9b                   hb83a4c4_2    defaults
keras                     2.4.3                    pypi_0    pypi
libpng                    1.6.37               h2a8f88b_0    defaults
libtiff                   4.1.0                h56a325e_1    defaults
libuv                     1.40.0               he774522_0    defaults
lz4-c                     1.9.2                hf4a77e7_3    defaults
mkl                       2020.2                      256    defaults
mkl-service               2.3.0            py38h196d8e1_0    defaults
mkl_fft                   1.2.0            py38h45dec08_0    defaults
mkl_random                1.1.1            py38h47e9c7a_0    defaults
ninja                     1.10.2           py38h6d14046_0    defaults
numpy                     1.19.2           py38hadc3359_0    defaults
numpy-base                1.19.2           py38ha3acd2a_0    defaults
olefile                   0.46                       py_0    defaults
openssl                   1.1.1i               h2bbff1b_0    defaults
pillow                    8.0.1            py38h4fa10fc_0    defaults
pip                       20.3.3           py38haa95532_0    defaults
python                    3.8.5                h5fd99cc_1    defaults
pytorch                   1.7.1           py3.8_cuda110_cudnn8_0    pytorch
regex                     2020.11.13               pypi_0    pypi
s3transfer                0.3.3                    pypi_0    pypi
sacremoses                0.0.43                   pypi_0    pypi
scikit-learn              0.24.0                   pypi_0    pypi
scipy                     1.6.0                    pypi_0    pypi
sentencepiece             0.1.94                   pypi_0    pypi
setuptools                51.0.0           py38haa95532_2    defaults
six                       1.15.0           py38haa95532_0    defaults
sklearn                   0.0                      pypi_0    pypi
sqlite                    3.33.0               h2a8f88b_0    defaults
tb-nightly                2.5.0a20210101           pypi_0    pypi
threadpoolctl             2.1.0                    pypi_0    pypi
thulac                    0.2.1                    pypi_0    pypi
tk                        8.6.10               he774522_0    defaults
torchaudio                0.7.2                      py38    pytorch
torchvision               0.8.2                py38_cu110    pytorch
transformers              2.1.1                    pypi_0    pypi
typing_extensions         3.7.4.3                    py_0    defaults
vc                        14.2                 h21ff451_1    defaults
vs2015_runtime            14.27.29016          h5e58377_2    defaults
wheel                     0.36.2             pyhd3eb1b0_0    defaults
wincertstore              0.2                      py38_0    defaults
xz                        5.2.5                h62dcd97_0    defaults
zlib                      1.2.11               h62dcd97_4    defaults
zstd                      1.4.5                h04227a9_0    defaults

Although I'm using PyTorch, it's tensorflow rather than PyTorch to blame(according to the logs). I got the same issue with pure tensorflow 2.3
2021-01-03 01:17:50.516100: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-01-03 01:17:52.622054: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library nvcuda.dll
2021-01-03 01:17:52.645796: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2021-01-03 01:17:52.645998: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-01-03 01:17:52.649575: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-01-03 01:17:52.649707: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2021-01-03 01:17:52.649827: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-01-03 01:17:52.649928: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-01-03 01:17:52.651954: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-01-03 01:17:52.660165: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2021-01-03 01:17:52.660416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-01-03 01:17:52.660971: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-03 01:17:52.668967: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x19659fe67d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2021-01-03 01:17:52.669132: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2021-01-03 01:17:52.669395: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: GeForce RTX 3070 computeCapability: 8.6
coreClock: 1.725GHz coreCount: 46 deviceMemorySize: 8.00GiB deviceMemoryBandwidth: 417.29GiB/s
2021-01-03 01:17:52.669576: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
2021-01-03 01:17:52.669683: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cublas64_10.dll
2021-01-03 01:17:52.669790: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cufft64_10.dll
2021-01-03 01:17:52.669896: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library curand64_10.dll
2021-01-03 01:17:52.670072: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusolver64_10.dll
2021-01-03 01:17:52.670201: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cusparse64_10.dll
2021-01-03 01:17:52.670365: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudnn64_7.dll
2021-01-03 01:17:52.670542: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
2021-01-03 01:18:37.097681: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-01-03 01:18:37.097876: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]      0
2021-01-03 01:18:37.098025: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1276] 0:   N
2021-01-03 01:18:37.098301: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1402] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 6591 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3070, pci bus id: 0000:0a:00.0, compute capability: 8.6)
2021-01-03 01:18:37.101296: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x1960330d0d0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2021-01-03 01:18:37.101474: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce RTX 3070, Compute Capability 8.6
args:
Namespace(articles_per_title=10, device='0,1,2,3', length=1000, model_config='config/model_config_small.json', model_path='model/final_model', no_wordpiece=False, repetition_penalty=1.0, save_path='generated/', segment=False, temperature=2.0, titles='我', titles_file='', tokenizer_path='cache/vocab_small.txt', topk=10, topp=0)

I noticed that the tensorflow installation guide for GPU users said that GPUs with Ampere architecture may encounter this issue and can solve that by using export CUDA_CACHE_MAXSIZE=2147483648 to expand the default JIT cache. It doesn't work with Windows. I searched my environment variables and none of them namedCUDA_CACHE_MAXSIZE.
I tried adding that on my own, but it still takes a long time to pass Adding Visible Devices 0.
What should I do?


title: Pytorch N - Beats model throwing error: 'str' object has no attribute '__name__'

body:
I'm trying to replicate pytorch's N - Beats model in colab. I copied the same code from https://pytorch-forecasting.readthedocs.io/en/stable/tutorials/ar.html to a colab notebook. There is an error showing up at training cell.
import os
import warnings
warnings.filterwarnings("ignore")
os.chdir("../../..")

import pandas as pd
import pytorch_lightning as pl
from pytorch_lightning.callbacks import EarlyStopping
import torch

from pytorch_forecasting import Baseline, NBeats, TimeSeriesDataSet
from pytorch_forecasting.data import NaNLabelEncoder
from pytorch_forecasting.data.examples import generate_ar_data
from pytorch_forecasting.metrics import SMAPE

data = generate_ar_data(seasonality=10.0, timesteps=400, n_series=100, seed
 = 42)
data["static"] = 2
data["date"] = pd.Timestamp("2020-01-01") + pd.to_timedelta(data.time_idx, "D")
data.head()

# create dataset and dataloaders
max_encoder_length = 60
max_prediction_length = 20

training_cutoff = data["time_idx"].max() - max_prediction_length

context_length = max_encoder_length
prediction_length = max_prediction_length

training = TimeSeriesDataSet(
data[lambda x: x.time_idx <= training_cutoff],
time_idx="time_idx",
target="value",
categorical_encoders={"series": NaNLabelEncoder().fit(data.series)},
group_ids=["series"],
# only unknown variable is "value" - and N-Beats can also not take any additional variables
 time_varying_unknown_reals=["value"],
 max_encoder_length=context_length,
 max_prediction_length=prediction_length,
)

validation = TimeSeriesDataSet.from_dataset(training, data,     min_prediction_idx=training_cutoff + 1)
batch_size = 128
train_dataloader = training.to_dataloader(train=True,     batch_size=batch_size, num_workers=0)
val_dataloader = validation.to_dataloader(train=False, batch_size=batch_size, num_workers=0)

Error is:
---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-67-db4b0ef13391> in <module>()
     25     net,
     26     train_dataloader=train_dataloader,
---> 27     val_dataloaders=val_dataloader,
     28 )

30 frames

/usr/local/lib/python3.7/dist-packages/yaml/representer.py in     represent_object(self, data)
    329         if dictitems is not None:
    330             dictitems = dict(dictitems)
--> 331         if function.__name__ == '__newobj__':
    332             function = args[0]
    333             args = args[1:]

 AttributeError: 'str' object has no attribute '__name__'


title: Is this method of calculating the top-5 accuracy in pytorch correct?

body:
I am trying to validate the findings of a paper by testing it on the same model architecture as well as the same dataset reported by the paper. I have been using the imagenet script provided in the official pytorch repository's examples section to do the same.
class AverageMeter(object):
    """Computes and stores the average and current value
       Imported from https://github.com/pytorch/examples/blob/master/imagenet/main.py#L247-L262
    """
    def init(self):
        self.reset()

    def reset(self):
        self.val = 0
        self.avg = 0
        self.sum = 0
        self.count = 0

    def update(self, val, n=1):
        self.val = val
        self.sum += val * n
        self.count += n
        self.avg = self.sum / self.count

def accuracy(output, target, topk=(1,)):
    """Computes the precision@k for the specified values of k"""
    maxk = max(topk)
    batchsize = target.size(0)

    , pred = output.topk(maxk, 1, True, True)
    pred = pred.t()
    correct = pred.eq(target.view(1, -1).expand_as(pred))

    res = []
    for k in topk:
        correct_k = correct[:k].reshape(-1).float().sum(0)
        res.append(correctk.mul(100.0 / batch_size))
    return res
top1 = AverageMeter()
top5 = AverageMeter()

# switch to evaluate mode
model.eval()
with torch.no_grad():
    for batch_idx, (inputs, targets) in enumerate(test_loader):
        # measure data loading time
        print(f"Processing {batch_idx+1}/{len(test_loader)}")
        inputs, targets = inputs.cuda(), targets.cuda()
        inputs, targets = torch.autograd.Variable(inputs, volatile=True), torch.autograd.Variable(targets)

        # compute output
        outputs = model(inputs)

        # measure accuracy and record loss
        prec1, prec5 = accuracy(outputs.data, targets.data, topk=(1, 5))
        print(prec1,prec5)
        top1.update(prec1.item(), inputs.size(0))
        top5.update(prec5.item(), inputs.size(0))

print(top1)
print(top5)

However the top 5 error which I am getting by using this script is not matching with the one in the paper. Can anyone tell me what is wrong in this particular snippet?


title: Error while executing RNN code on pyTorch?

body:
I am running a code on Binary Addition of two strings using PyTorch.

However, while training the model I am getting the following error:

can't convert np.ndarray of type numpy.object.
The only supported types are: double, float, float16, int64, int32, and uint8.

Can anyone help me? Here is my code:

featDim=2 # two bits each from each of the String
outputDim=1 # one output node which would output a zero or 1

lstmSize=10

lossFunction = nn.MSELoss()
model =Adder(featDim, lstmSize, outputDim)
print ('model initialized')
#optimizer = optim.SGD(model.parameters(), lr=3e-2, momentum=0.8)
optimizer=optim.Adam(model.parameters(),lr=0.001)
epochs=500
### epochs ##
totalLoss= float("inf")
while totalLoss > 1e-5:
  print(" Avg. Loss for last 500 samples = %lf"%(totalLoss))
  totalLoss=0
  for i in range(0,epochs): # average the loss over 200 samples

    stringLen=4
    testFlag=0
    x,y=getSample(stringLen, testFlag)

    model.zero_grad()

    x_var=autograd.Variable(torch.from_numpy(x).unsqueeze(1).float()) #convert to torch tensor and variable
    # unsqueeze() is used to add the extra dimension since
    # your input need to be of t*batchsize*featDim; you cant do away with the batch in pytorch
    seqLen=x_var.size(0)
    #print (x_var)
    x_var= x_var.contiguous()
    y_var=autograd.Variable(torch.from_numpy(y).float()) ##ERROR ON THIS LINE
    finalScores = model(x_var)
    #finalScores=finalScores.

    loss=lossFunction(finalScores,y_var)
    totalLoss+=loss.data[0]
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

  totalLoss=totalLoss/epochs


I am using python 2.7 and I use the

from hausdorff import hausdorff

I get this error although I am using pip2 install hausdorff and it syas requirements already satisfied.

    from hausdorff import hausdorff
  File "/usr/local/lib/python2.7/dist-packages/hausdorff/__init__.py", line 1, in <module>
    from .hausdorff import *
  File "/usr/local/lib/python2.7/dist-packages/hausdorff/hausdorff.py", line 3, in <module>
    import hausdorff.distances as distances
ImportError: No module named distances


title: Non LSTM: Trying to backward through the graph a second time, but the buffers have already been freed

body:
Note that unlike other questions, this is not about any RNN structure. I wish to create a model that has changing gradients, and will look like below. The breakpoints are manually supplied.

The model that I have created is as follows:
class Trend(nn.Module):
    """
    Broken Trend model, with breakpoints as defined by user.
    """
    def __init__(self, breakpoints):
        super().__init__()
        self.bpoints = breakpoints[None, :]
        self.init_layer = nn.Linear(1,1) # first linear bit
        # extract gradient and bias
        w = self.init_layer.weight
        b = self.init_layer.bias
        self.params = [[w,b]] # save it to buffer

        if len(breakpoints>0):
            # create deltas which is how the gradient will change
            deltas = torch.randn(len(breakpoints)) / len(breakpoints) # initialisation
            self.deltas = nn.Parameter(deltas) # make it a parameter

            for d, x1 in zip(self.deltas, breakpoints):
                y1 = w *x1 + b # find the endpoint of line segment (x1, y1)
                w = w + d # add on the delta to gradient
                b = y1 - w * x1 # find new bias of line segment
                self.params.append([w,b]) # add to buffer

        # create buffer
        self.wb = torch.zeros(len(self.params), len(self.params[0]))

    def __copy2array(self):
        """
        Saves parameters into wb
        """
        for i in range(self.wb.shape[0]):
            for j in range(self.wb.shape[1]):
                self.wb[i,j] = self.params[i][j]

    def forward(self, x):
        # get the line segment area (x_sec) for each x
        x_sec = x >= self.bpoints
        x_sec = x_sec.sum(1)
        self.__copy2array() # copy across parameters into matrix

        # get final prediction y = mx +b for relevant section
        return x*self.wb[x_sec][:,:1] + self.wb[x_sec][:,1:]

However, once I attempt to train it I get the error RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.
I obtained the above plot by doing:
time = torch.arange(700).float()[:,None]
y_pred = model(time)
plt.plot(time, y_pred.detach().numpy())
plt.show()

So we know the forward pass is working as expected. However the backward pass is not quite working. Was wondering what I need to change to get it working.
If you're wondering why __copy2array is being used, when I tried to use torch.Tensor(self.params) it destroyed the gradients in those parameters. Thanks in advance.


title: Get matrix dimensions from pytorch layers

body:
Here is an autoencoder I created from Pytorch tutorials :

epochs = 1000
from pylab import plt
plt.style.use('seaborn')
import torch.utils.data as data_utils
import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available()
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
import numpy as np
import pandas as pd
import datetime as dt

features = torch.tensor(np.array([ [1,2,3],[1,2,3],[100,200,500] ]))

print(features)

batch = 10
data_loader = torch.utils.data.DataLoader(features, batch_size=2, shuffle=False)

encoder = nn.Sequential(nn.Linear(3,batch), nn.Sigmoid())
decoder = nn.Sequential(nn.Linear(batch,3), nn.Sigmoid())
autoencoder = nn.Sequential(encoder, decoder)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.001)

encoded_images = []
for i in range(epochs):
    for j, images in enumerate(data_loader):
    #     images = images.view(images.size(0), -1)
        images = Variable(images).type(FloatTensor)
        optimizer.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()

#     encoded_images.append(encoder(images))

# print(decoder(torch.tensor(np.array([1,2,3])).type(FloatTensor)))

encoded_images = []
for j, images in enumerate(data_loader):
    images = images.view(images.size(0), -1)
    images = Variable(images).type(FloatTensor)

    encoded_images.append(encoder(images))

I can see the encoded images do have newly created dimension of 10. In order to understand the matrix operations going on under the hood I'm attempting to print the matrix dimensions of encoder and decoder but shape is not available on nn.Sequential

How to print the matrix dimensions of nn.Sequential ?


title: How to move axis on simple numpy array

body:
I'm having trouble moving the 3 axis to the 1 position. I would like to move the 3 to the first 69 position. This is for a machine learning dataset and PyTorch will only accept the data if it's in a 3x69x69 format. Thanks for any help!

 # To get the images and labels from file
with h5py.File(r"C:\Users\ajbur\Downloads\Galaxy10.h5", 'r') as F:
    images = np.array(F['images'])
    labels = np.array(F['ans'])
np.shape(images)
np.moveaxis(images,0,-1).shape
np.shape(images)

output is [20000, 69, 69, 3]
I want it to be [20000, 3, 69, 69]


title: Testing an implementation of an LSTM in Pytorch

body:
I'm trying to use the Pytorch implementation of an LSTM here. I'm including it here for reference. It consists of two classes, LSTMCell and LSTM, where LSTMCell is just a single unit and LSTM puts stacks multiple units together to create a full LSTM model

import math
import torch as th
import torch.nn as nn

class LSTMCell(nn.Module):

    def __init__(self, input_size, hidden_size, bias=True):
        super(LSTM, self).__init__()
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bias = bias
        self.i2h = nn.Linear(input_size, 4 * hidden_size, bias=bias)
        self.h2h = nn.Linear(hidden_size, 4 * hidden_size, bias=bias)
        self.reset_parameters()

    def reset_parameters(self):
        std = 1.0 / math.sqrt(self.hidden_size)
        for w in self.parameters():
            w.data.uniform_(-std, std)

    def forward(self, x, hidden):
        if hidden is None:
            hidden = self._init_hidden(x)

        h, c = hidden
        h = h.view(h.size(1), -1)
        c = c.view(c.size(1), -1)
        x = x.view(x.size(1), -1)

        # Linear mappings
        preact = self.i2h(x) + self.h2h(h)

        # activations
        gates = preact[:, :3 * self.hidden_size].sigmoid()
        g_t = preact[:, 3 * self.hidden_size:].tanh()
        i_t = gates[:, :self.hidden_size]
        f_t = gates[:, self.hidden_size:2 * self.hidden_size]
        o_t = gates[:, -self.hidden_size:]

        c_t = th.mul(c, f_t) + th.mul(i_t, g_t)

        h_t = th.mul(o_t, c_t.tanh())

        h_t = h_t.view(1, h_t.size(0), -1)
        c_t = c_t.view(1, c_t.size(0), -1)
        return h_t, (h_t, c_t)

    @staticmethod
    def _init_hidden(input_):
        h = th.zeros_like(input_.view(1, input_.size(1), -1))
        c = th.zeros_like(input_.view(1, input_.size(1), -1))
        return h, c

class LSTM(nn.Module):

    def __init__(self, input_size, hidden_size, bias=True):
        super().__init__()
        self.lstm_cell = LSTMCell(input_size, hidden_size, bias)

    def forward(self, input_, hidden=None):
        # input_ is of dimensionalty (1, time, input_size, ...)

        outputs = []
        for x in torch.unbind(input_, dim=1):
            hidden = self.lstm_cell(x, hidden)
            outputs.append(hidden[0].clone())

        return torch.stack(outputs, dim=1)

I'm doing the following simple test:

x = torch.randn(1, 3, 2, 4)
model = LSTM(4, 5, False)
model(x)

and I get the following error. What exactly is the problem here?

TypeError                                 Traceback (most recent call last)
<ipython-input-33-09e5544a61fc> in <module>
----> 1 model = LSTM(4, 5, False)

<ipython-input-30-9ad06cd4b768> in __init__(self, input_size, hidden_size, bias)
      3     def __init__(self, input_size, hidden_size, bias=True):
      4         super().__init__()
----> 5         self.lstm_cell = LSTMCell(input_size, hidden_size, bias)
      6
      7     def forward(self, input_, hidden=None):

<ipython-input-29-c91ddfb9dfae> in __init__(self, input_size, hidden_size, bias)
      6
      7     def __init__(self, input_size, hidden_size, bias=True):
----> 8         super(LSTM, self).__init__()
      9         self.input_size = input_size
     10         self.hidden_size = hidden_size

TypeError: super(type, obj): obj must be an instance or subtype of type


title: RNN model with one-hot encoding data for binary classification in PyTorch

body:
I have the video stream on top of which I run the Amazon Rekognition on second frames. Rekognition outputs the probability of the 2000 classes (wood, zebra, fence, car, etc.).
I need to develop my classifier for 3 classes on top of these predictions.
So, for a video with a length of 1600 seconds, I have a 1600x2002 dataset (Rekognition classes, time offset videos, and a label of my class). I came up with the idea to capture the context of Rekognition classes.

I tried to make a simple RNN model but the loss seems weird It seems that he was mistaken somewhere in the dataset preparation

I create a separate dataframe, where I put the ohe-hot array into the column.

df_for_torch = pd.DataFrame(columns=['inputs', 'label'])
df_for_torch['label'] = df['label']
df_for_torch['inputs'] = df[input_labels].applymap(lambda x: 1 if x > 0 else 0).values.tolist()
df_for_torch.to_csv(root_path+'data_torch.csv')

Then I load it through torchtext

tv_datafields = [("label", LABEL), ("inputs", TEXT)]
train = data.TabularDataset.splits(
               path=root_path,
               train='data_torch.csv',
               format='csv',
               skip_header=True,
               fields=tv_datafields)[0]

Create train и valid

SEED = 18
train_data, valid_data = train.split(random_state = random.seed(SEED))

Batching

BATCH_SIZE = 64
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
train_iterator, valid_iterator = data.BucketIterator.splits(
    (train_data, valid_data),
    batch_size = BATCH_SIZE,
    sort=False,
    device = device)

Create vocab

TEXT.build_vocab(train_data, max_size = MAX_VOCAB_SIZE)
LABEL.build_vocab(train_data)

Build a model

class RNN(nn.Module):
    def __init__(self, input_dim, embedding_dim, hidden_dim, output_dim):
        super().__init__()
        self.embedding = nn.Embedding(input_dim, embedding_dim)
        self.rnn = nn.RNN(embedding_dim, hidden_dim)
        self.fc = nn.Linear(hidden_dim, output_dim)
    def forward(self, input_labels):
        #input_labels = [sent len, batch size]
        embedded = self.embedding(input_labels)
        #embedded = [sent len, batch size, emb dim]
        output, hidden = self.rnn(embedded)
        #output = [sent len, batch size, hid dim]
        #hidden = [1, batch size, hid dim]
        assert torch.equal(output[-1,:,:], hidden.squeeze(0))
        return self.fc(hidden.squeeze(0))

Init it

INPUT_DIM = len(input_labels)
EMBEDDING_DIM = 500
HIDDEN_DIM = 256
OUTPUT_DIM = 1
model = RNN(INPUT_DIM, EMBEDDING_DIM, HIDDEN_DIM, OUTPUT_DIM)
optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)
criterion = torch.nn.BCEWithLogitsLoss()
model = model.to(device)
criterion = criterion.to(device)

Train and eval methods

def train(model, iterator, optimizer, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.train()
    for batch in iterator:
        optimizer.zero_grad()
        predictions = model(batch.inputs).squeeze(1)
        loss = criterion(predictions, batch.label)
        acc = binary_accuracy(predictions, batch.label)
        loss.backward()
        optimizer.step()
        epoch_loss += loss.item()
        epoch_acc += acc.item()
    return epoch_loss / len(iterator), epoch_acc / len(iterator)
def evaluate(model, iterator, criterion):
    epoch_loss = 0
    epoch_acc = 0
    model.eval()
    with torch.no_grad():
        for batch in iterator:
            predictions = model(batch.inputs).squeeze(1)
            loss = criterion(predictions, batch.label)
            acc = binary_accuracy(predictions, batch.label)
            epoch_loss += loss.item()
            epoch_acc += acc.item()
    return epoch_loss / len(iterator), epoch_acc / len(iterator)

Run everything

N_EPOCHS = 20
best_valid_loss = float('inf')
for epoch in range(N_EPOCHS):
    start_time = time.time()
    train_loss, train_acc = train(model, train_iterator, optimizer, criterion)
    valid_loss, valid_acc = evaluate(model, valid_iterator, criterion)
    end_time = time.time()
    epoch_mins, epoch_secs = epoch_time(start_time, end_time)
    if valid_loss < best_valid_loss:
        best_valid_loss = valid_loss
        torch.save(model.state_dict(), 'tut1-model.pt')
    print(f'Epoch: {epoch+1:02} | Epoch Time: {epoch_mins}m {epoch_secs}s')
    print(f'\tTrain Loss: {train_loss:.3f} | Train Acc: {train_acc*100:.2f}%')
    print(f'\t Val. Loss: {valid_loss:.3f} |  Val. Acc: {valid_acc*100:.2f}%')

But loss seems strange

Epoch: 01 | Epoch Time: 1m 43s
    Train Loss: -2230969.595 | Train Acc: 0.06%
     Val. Loss: 5658.502 |  Val. Acc: 0.00%
Epoch: 02 | Epoch Time: 1m 36s
    Train Loss: -7322002.296 | Train Acc: 0.06%
     Val. Loss: 11616.960 |  Val. Acc: 0.00%
....


title: Neural Network : Epoch and Batch Size

body:
I am trying to train a neural network to classify words into different categories.
I notice two things:
When I use a smaller batch_size (like 8,16,32) the loss is not decreasing, but rather sporadically varying. When I use a larger batch_size (like 128, 256), the loss is going going down, but very slowly.
More importantly, when I use a larger EPOCH value, my model does a good job at reducing the loss. However I'm using a really large value (EPOCHS = 10000).
Question:
How to get the optimal EPOCH and batch_size values?


title: Installing Python dependencies in Heroku

body:
I wanted to make an Ai chatbot hosted on Heroku, but there are some problems with installing the requirements.
The chatbot needs the following packages:

discord
pytorch
spaCy

I already figured out putting discord into the requirements.txt, but I got no clue on how to do the other two dependencies.
The problem with them is that they aren't just installed with
pip install NAME_OF_THE_PACKAGE .
Pytorch for example needs:
pip install torch==1.7.0+cpu torchvision==0.8.1+cpu torchaudio==0.7.0 -f https://download.pytorch.org/whl/torch_stable.html

and spaCy needs
pip install -U spacy
pip install -U spacy-lookups-data
python -m spacy download en_core_web_sm.
I tried different variations of the commands above, but none worked.
Now I even reached the built limit on my account.
Please help me if you know how to solve the problem.


title: conda install pytorch torchvision torchaudio cpuonly -c pytorch doesnt install pytorch

body:
everytime i attempt to run or modify this command I get the same errors.
(this command: conda install pytorch torchvision torchaudio cpuonly -c pytorch)
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.

PackagesNotFoundError: The following packages are not available from current channels:

  - pytorch

Current channels:

  - https://conda.anaconda.org/pytorch/win-32
  - https://conda.anaconda.org/pytorch/noarch
  - https://repo.anaconda.com/pkgs/main/win-32
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/win-32
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://repo.anaconda.com/pkgs/msys2/win-32
  - https://repo.anaconda.com/pkgs/msys2/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

(note this starts the line under where I ran the command) and when this happens I dont get anything to actually install and therefore cant use pytorch.


title: Pytorch: Why batch is the second dimension in the default LSTM?

body:
In the PyTorch LSTM documentation it is written:

batch_first – If True, then the input and output tensors are provided
as (batch, seq, feature). Default: False

I'm wondering why they chose the default batch dimension as the second one and not the first one. for me, it is easier to imaging my data as [batch, seq, feature] than [seq, batch, feature]. The first one is more intuitive for me and the second one is counterintuitive.
I'm asking here to know if the is any reason behind this and if you can help me to have some understanding about it.


title: How to create a tensor of given shape and interval?

body:
I am going through Pytorch and want to create a random tensor of shape 5X3 in the interval [3,7)
torch.rand(5,3) will return a random tensor of shape 5 X 3, however, I could not figure to set the given interval.
Please guide.


title: Multi label classification with unbalanced labels

body:
I am building multi label classification network.
My GTs are vectors of length 512 [0,0,0,1,0,1,0,...,0,0,0,1]
Most of the time they are zeroes, each vector has about 5 ones, and rest are zeros .
I am thinking to do:
Use sigmoid for activation for output layer.
Use binary_crossentropy for loss function.
But how I can solve the unbalance issue ?
Network can learn to predict always zeros and still have really low learning loss score.
How I can make it actually learn to predict ones...


title: How to debug source code written in pytorch?

body:
This is a general question regarding how do you people debug pytorch source code? Via print statements? Any particular IDE-plugin combination?
I am a beginner and I think this question would help beginners like me.


title: Focal loss implementation

body:
In the paper introducing focal loss, they state that the loss function is formulated as such:

Where

I found an implementation of it on a Github page from another author who used it in their paper. I tried the function out on a segmentation problem dataset I have and it seems to work quite well.
Below is the implementation:
def binary_focal_loss(pred, truth, gamma=2., alpha=.25):
    eps = 1e-8
    pred = nn.Softmax(1)(pred)
    truth = F.one_hot(truth, num_classes = pred.shape[1]).permute(0,3,1,2).contiguous()

    pt_1 = torch.where(truth == 1, pred, torch.ones_like(pred))
    pt_0 = torch.where(truth == 0, pred, torch.zeros_like(pred))

    pt_1 = torch.clamp(pt_1, eps, 1. - eps)
    pt_0 = torch.clamp(pt_0, eps, 1. - eps)

    out1 = -torch.mean(alpha * torch.pow(1. - pt_1, gamma) * torch.log(pt_1))
    out0 = -torch.mean((1 - alpha) * torch.pow(pt_0, gamma) * torch.log(1. - pt_0))

    return out1 + out0

The part that I don't understand is the calculation of pt_0 and pt_1. I created a small example for myself to try and figure it out but it still confuses me a bit.
# one hot encoded prediction tensor
pred = torch.tensor([
                     [
                      [.2, .7, .8], # probability
                      [.3, .5, .7], # of
                      [.2, .6, .5]  # background class
                     ],
                     [
                      [.8, .3, .2], # probability
                      [.7, .5, .3], # of
                      [.8, .4, .5]  # class 1
                     ]
                    ])

# one-hot encoded ground truth labels
truth = torch.tensor([
                      [1, 0, 0],
                      [1, 1, 0],
                      [1, 0, 0]
                     ])
truth = F.one_hot(truth, num_classes = 2).permute(2,0,1).contiguous()

print(truth)
# gives me:
# tensor([
#         [
#          [0, 1, 1],
#          [0, 0, 1],
#          [0, 1, 1]
#         ],
#         [
#          [1, 0, 0],
#          [1, 1, 0],
#          [1, 0, 0]
#         ]
#       ])

pt_0 = torch.where(truth == 0, pred, torch.zeros_like(pred))
pt_1 = torch.where(truth == 1, pred, torch.ones_like(pred))

print(pt_0)
# gives me:
# tensor([[
#         [0.2000, 0.0000, 0.0000],
#         [0.3000, 0.5000, 0.0000],
#         [0.2000, 0.0000, 0.0000]
#         ],
#        [
#         [0.0000, 0.3000, 0.2000],
#         [0.0000, 0.0000, 0.3000],
#         [0.0000, 0.4000, 0.5000]
#        ]
#      ])

print(pt_1)
# gives me:
# tensor([[
#          [1.0000, 0.7000, 0.8000],
#          [1.0000, 1.0000, 0.7000],
#          [1.0000, 0.6000, 0.5000]
#         ],
#         [
#          [0.8000, 1.0000, 1.0000],
#          [0.7000, 0.5000, 1.0000],
#          [0.8000, 1.0000, 1.0000]
#         ]
#       ])

What I don't understand is why in pt_0 we are placing zeros where the torch.where statement is false, and in pt_1 we place ones. From how I understood the paper, I would have thought that instead of placing zeros or ones, you would place 1-p.
Can anyone help explain this to me?


title: AttributeError: 'dict' object has no attribute 'shape'Does anyone know why this error occurs?

body:
I want to read my folder as a test data set folder. But I am very troubled by this problem.
My environment is python3.6+cuda9+pytorch1.4.0.
code:
# read images
# --------------------------------
test_results_ave = OrderedDict()
test_results_ave['psnr_sf_k'] = []

for sf in test_sf:
    for k_index in range(kernels.shape[1]):
        test_results = OrderedDict()
        test_results['psnr'] = []
        kernel = kernels[0, k_index].astype(np.float64)

error：
Traceback (most recent call last):

  File "E:\USRNet-master\USRNet-master\main_test_table1.py", line 225, in <module>

    main()

  File "E:\USRNet-master\USRNet-master\main_test_table1.py", line 143, in main

    for k_index in range(kernels.shape[1]):

**AttributeError: 'dict' object has no attribute 'shape'**


title: How to use pytorch lightning Accuracy with ignore class?

body:
I have some training pipeline which uses CrossEntropyLoss with an ignore class.
The model outputs log_probs of shape (150, 3) - meaning 3 possible classes in batches of 150.
The label_batch is of shape 150, and torch.max(label_batch) == tensor(3, device='cuda:0'), meaning there is an extra class labeled 3, which is the ignore class.
The loss handles it fine:
self._criterion = nn.CrossEntropyLoss(
    reduction='mean',
    ignore_index=3
)

But the accuracy metric thinks class 3 is valid and gives very wrong results:
self.train_acc = pl.metrics.Accuracy()
wrong result with self.train_acc.update(log_probs, label_batch) because of the 3 label should be ignored.

How to correctly use pl.metrics.Accuracy() with an ignore class?


title: RuntimeError: CUDA error: device-side assert triggered - Resnet18 on PyTorch

body:
I'm trying to use PyTorch's Resnet18 model with my image data. Given the complexity of the model as well as the size of the data, I'd like to run it using CUDA. I'm doing the follow:
resnet_cnn = models.resnet18(pretrained = True)
num_ftrs = resnet_cnn.fc.in_features
resnet_cnn.fc = nn.Linear(num_ftrs, 8)

criterion = nn.CrossEntropyLoss().cuda()
optimizer_ft = optim.SGD(resnet_cnn.parameters(), lr=0.001, momentum=0.9)
exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=5, gamma=0.1)

After this, I attempt to train and test my model with the following loop:
count = 0
loss_list = []
iteration_list = []
accuracy_list = []
epochs = 30

for epoch in range(epochs):
    for i, (images, labels) in enumerate(trainloader):
            resnet_cnn = resnet_cnn.cuda()
            images.cuda()
            labels.cuda()

            optimizer_ft.zero_grad()
            outputs = resnet_cnn(images.cuda())
            loss = criterion(outputs.cuda(), labels.cuda())
            loss.backward()
            optimizer_ft.step()

            count += 1

            if count % 50 == 0:
                correct = 0
                total = 0

                for i, (images, labels) in enumerate(testloader):
                    # images.to(device)
                    # labels.to(device)

                    outputs = resnet_cnn(images.cuda())
                    predicted = torch.max(outputs.data, 1)[1]
                    total += len(labels)
                    correct += (predicted == labels.cuda()).sum()
                accuracy = 100 * correct / float(total)

                loss_list.append(loss.data)
                iteration_list.append(count)
                accuracy_list.append(accuracy)

                if count % 500 == 0:
                    print("Iteration: {} Loss: {} Accuracy: {} %".format(count, loss.data, accuracy))

But I am met with the following error:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-48-cb669e8d47c0> in <module>()
      7 for epoch in range(epochs):
      8     for i, (images, labels) in enumerate(trainloader):
----> 9             resnet_cnn = resnet_cnn.cuda()
     10             images.cuda()
     11             labels.cuda()

3 frames
/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in cuda(self, device)
    489             Module: self
    490         """
--> 491         return self._apply(lambda t: t.cuda(device))
    492
    493     def xpu(self: T, device: Optional[Union[int, device]] = None) -> T:

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    385     def _apply(self, fn):
    386         for module in self.children():
--> 387             module._apply(fn)
    388
    389         def compute_should_use_set_data(tensor, tensor_applied):

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _apply(self, fn)
    407                 # `with torch.no_grad():`
    408                 with torch.no_grad():
--> 409                     param_applied = fn(param)
    410                 should_use_set_data = compute_should_use_set_data(param, param_applied)
    411                 if should_use_set_data:

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in <lambda>(t)
    489             Module: self
    490         """
--> 491         return self._apply(lambda t: t.cuda(device))
    492
    493     def xpu(self: T, device: Optional[Union[int, device]] = None) -> T:

RuntimeError: CUDA error: device-side assert triggered

I can't figure what I'm doing wrong as I've trained a manually defined CNN in the same way. Thank you in advance.


title: Error reading images with pandas (+pyTorch,scikit)

body:
I'm trying to read images to work with a CNN, but I'm getting a pandas error while trying to load the images.
This is some of the code (omitted imports and irrelevant nn class for clarity):

file_name = "annotation.csv"
image_files = pd.read_csv(file_name)

class SimpsonsDataset(Dataset):

    def __init__(self, csv_file, root_dir, transform=None):
        self.image_file = pd.read_csv(csv_file)
        self.root_dir = root_dir
        self.transform = transform

    def __len__(self):
        return len(self.image_file)

    def __getitem__(self, idx):
        img_name = os.path.join(self.root_dir, self.image_file.iloc[idx,0][1:])
        image = io.imread(img_name)
        sample = {'image': image}
        if self.transform:
            sample = self.transform(sample)
        return sample

simpsons = SimpsonsDataset(csv_file=image_files,root_dir="folder/")

I use the iloc[idx,0][1:] to format the filepath, and the filepath is joined, with the folders and filenames matching.

However, when I try to run the file, I get the following error:

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 710, in runfile
    execfile(filename, namespace)

  File "C:\ProgramData\Anaconda3\lib\site-packages\spyder\utils\site\sitecustomize.py", line 101, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "C:/.../image_extractor.py", line 41, in <module>
    simpsons = SimpsonsDataset(csv_file=image_files,root_dir="folder/")

  File "C:/.../image_extractor.py", line 26, in __init__
    self.image_file = pd.read_csv(csv_file)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 655, in parser_f
    return _read(filepath_or_buffer, kwds)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\parsers.py", line 392, in _read
    filepath_or_buffer, encoding, compression)

  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\io\common.py", line 210, in get_filepath_or_buffer
    raise ValueError(msg.format(_type=type(filepath_or_buffer)))

ValueError: Invalid file path or buffer object type: <class 'pandas.core.frame.DataFrame'>

Would love some insights on why this is happening. Thanks!


title: In pytorch, why does sum(tensor) return a fault result instead of torch.sum(tensor) sometimes?

body:
When I'm debugging , I meet a strange problem.

>>> cos
>>>
tensor([ 0.3869,  0.2857,  0.4931,  0.5086,  0.6757,  0.6417,  0.3773,  0.4084,
         0.2496,  0.7558,  0.4305,  0.3839,  0.1892,  0.8675,  0.3392,  0.5415,
         0.4421,  0.2782,  0.5187,  0.2672,  0.2896,  0.5031,  0.4791,  0.3528,
         0.2577,  0.3932,  0.2554,  0.4925,  0.8496,  0.1264,  0.5594,  0.8667,
         ···
         0.4919,  0.4073,  0.6890,  0.3976,  0.5691,  0.0741,  0.6420,  0.4249,
         0.2785])

I want to calculate how many element is greater than zero, so I try this:

>>> sum(cos>0)
>>> tensor(181, dtype=torch.uint8)  # fault answer
>>> torch.sum(cos>0)
>>> tensor(437)                     # correct answer

I also repeat the above operations, but the both answers turn to be right.

>>> sum(cos[:100]>0)
>>> tensor(100, dtype=torch.uint8)
>>> torch.sum(cos[:100]>0)
>>> tensor(100)

I run this in pytorch 1.0.2 .


title: How to take multi-GPU support to the OpenNMT-py (pytorch)?

body:
I used python-2.7 version to run the PyTorch with GPU support. I used this command to train the dataset using multi-GPU.
Can someone please tell me how can I fix this error with PyTorch in OpenNMT-py or is there a way to take pytorch support for multi-GPU using python 2.7?

Here is the command that I tried.

    CUDA_VISIBLE_DEVICES=1,2
    python train.py -data data/demo -save_model demo-model -world_size 2 -gpu_ranks 0 1

This is the error:

  Traceback (most recent call last):
   File "train.py", line 200, in
    main(opt)
  File "train.py", line 60, in main
  mp = torch.multiprocessing.get_context('spawn')
  AttributeError: 'module' object has no attribute 'get_context'


title: U-Net, loss does not decrease beyond a point

body:
I have been trying U-Net on a custom dataset for road segmentation. But the loss does not decrease beyond a point.
link to the image and corresponding mask

The loss stops decreasing at around 1.
The output of the trained model is a completely white image.
The architecture is as

import torch.nn as nn

class Unet(nn.Module):
    '''U-Net Architecture'''

    def __init__(self,inp,out):
        super(Unet,self).__init__()
        self.c1=self.contracting_block(inp,64)
        self.c2=self.contracting_block(64,128)
        self.c3=self.contracting_block(128,256)
        self.maxpool=nn.MaxPool2d(2)
        self.upsample=nn.Upsample(scale_factor=2,mode="bilinear",align_corners=True)
        self.c4=self.contracting_block(128+256,128)
        self.c5=self.contracting_block(64+128,64)
        self.c6=nn.Conv2d(64,out,1)

    def contracting_block(self,inp,out,k=3):
        block =nn.Sequential(
            nn.Conv2d(inp,out,k,padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(out),
            nn.Conv2d(out,out,k,padding=1),
            nn.ReLU(),
            nn.BatchNorm2d(out)
        )
        return block

    def forward(self,x):
        #280,280,1
        conv1=self.c1(x)
        #280,280,128
        x=self.maxpool(conv1)
        #140,140,128
        conv2=self.c2(x)
        #140,140,256
        x=self.maxpool(conv2)
        #70,70,256
        conv3=self.c3(x)
        #70,70,512
        x=self.upsample(conv3)
        #140,140,512
        x=torch.cat([conv2,x],axis=1)
        #140,140,768
        x=self.c4(x)
        #140,140,256
        x=self.upsample(x)
        #280,280,256
        x=torch.cat([conv1,x],axis=1)
        #280,280,384
        x=self.c5(x)
        #280,280,128
        x=self.c6(x)
        #280,280,3
        return x

if __name__=="__main__":
    x=torch.ones(1,3,280,280)
    net=unet(3,3)
    print(net(x).shape)

I am using Adam as my optimizer and BCEWithLogitsLoss as my loss function. I am not sure about my choice of loss function as well. The input dimensions are 280,280,3.
What should I do to decrease loss ?


title: Pixel color slightly changed after PyTorch, OpenCV and NumPy

body:
I've been trying to run a pre-trained model which uses PyTorch, NumPy and OpenCV to upscale pixelated images. Problem is, after I save the image, it looks the same but when I insert them on the spriting software, it detects some color differences on the background.

I've tried changing the background color to something like Magenta, since I'm working with BMP and no image will have that color, but after running on the neural network I get some differences in pixels values. The magenta still looks like magenta, but the pixel values are not the same.

I've also tried calculating if the new color is close to magenta so I could replace it with the original background but no luck as well, I got pretty close to be honest, but the pixels right next to the subject were still in the old color.

Am I doing something wrong or do you have any suggestions?

Here's the code

import sys
import os.path
import glob
import cv2
import numpy as np
import torch
import architecture as arch

model_path = 'models/model.pth'  # models/RRDB_ESRGAN_x4.pth OR models/RRDB_PSNR_x4.pth
device = torch.device('cuda')  # if you want to run on CPU, change 'cuda' -> cpu
# device = torch.device('cpu')

test_img_folder = 'LR/*'

model = arch.RRDB_Net(3, 3, 64, 23, gc=32, upscale=4, norm_type=None, act_type='leakyrelu', \
                        mode='CNA', res_scale=1, upsample_mode='upconv')
model.load_state_dict(torch.load(model_path), strict=True)
model.eval()
for k, v in model.named_parameters():
    v.requires_grad = False
model = model.to(device)

print('Model path {:s}. \nTesting...'.format(model_path))

idx = 0
for path in glob.glob(test_img_folder):
    idx += 1
    base = os.path.splitext(os.path.basename(path))[0]
    print(idx, base)

    original_background = [209, 237, 212]
    magenta = [255,0,255]

    # read image
    img = cv2.imread(path, cv2.COLOR_RGB2BGR)
    img[np.where((img == original_background).all(axis=2))] = magenta
    img = img * 1.0 / 255
    img = torch.from_numpy(np.transpose(img[:, :, [2, 1, 0]], (2, 0, 1))).float()
    img_LR = img.unsqueeze(0)
    img_LR = img_LR.to(device)

    output = model(img_LR).data.squeeze().float().cpu().clamp_(0, 1).numpy()
    output = np.transpose(output[[2, 1, 0], :, :], (1, 2, 0))
    output = (output * 255).round()

    cv2.imwrite('results/{:s}.png'.format(base), output)

And here's the images, pre-processing and post

The image pre-processing:

[[[255   0 255]
  [255   0 255]
  [255   0 255]
  ...
  [255   0 255]
  [255   0 255]
  [255   0 255]]

And post-processing:

[[[176.  73. 220.]
  [228.  62. 252.]
  [255.  45. 244.]
  ...
  [255.  34. 255.]
  [255.  58. 252.]
  [219. 100. 204.]]


title: Pytorch custom randomcrop for semantic segmentation

body:
I am trying to implement a custom dataset loader. Firstly I resize the images and labels with the same ratio between (0.98, 1,1) then I randomly crop both images and labels with same parameters so that I can feed them into NN. However, I am getting an error from pytorch functional. Here is my code:
class RandomCrop(object):

    def __init__(self, size, padding=None, pad_if_needed=True, fill=0, padding_mode='constant'):

        self.size = size
        self.padding = padding
        self.pad_if_needed = pad_if_needed
        self.fill = fill
        self.padding_mode = padding_mode

    @staticmethod
    def get_params(img, output_size):

        w, h = img.size
        th, tw = output_size
        if w == tw and h == th:
            return 0, 0, h, w

        i = random.randint(0, h - th)
        j = random.randint(0, w - tw)
        return i, j, th, tw

    def __call__(self, data):

        img,mask = data["image"],data["mask"]

        # pad the width if needed
        if self.pad_if_needed and img.size[0] < self.size[1]:
            img = F.pad(img, (self.size[1] - img.size[0], 0), self.fill, self.padding_mode)
            mask = F.pad(mask, (self.size[1] - mask.size[0], 0), self.fill, self.padding_mode)
        # pad the height if needed
        if self.pad_if_needed and img.size[1] < self.size[0]:
            img = F.pad(img, (0, self.size[0] - img.size[1]), self.fill, self.padding_mode)
            mask = F.pad(mask, (0, self.size[0] - mask.size[1]), self.fill, self.padding_mode)

        i, j, h, w = self.get_params(img, self.size)
        crop_image = transforms.functional.crop(img, i, j, h, w)
        crop_mask = transforms.functional.crop(mask, i, j, h, w)

        return{"image": crop_image, "mask": crop_mask }

Here is the error:
AttributeError: 'Image' object has no attribute 'dim'


title: import torch :> ImportError: numpy.core.multiarray failed to import

body:
I am facing an issue regarding torch importing. here is the image error1 attached.ImportError: numpy.core.multiarray failed to import
I searched about it and got some solutions like i have to update numpy or uninstall the numpy and agin install the numpy etc with these commands

pip install numpy -I
or
pip install numpy --upgrade

when i tried to use these commands, I got one more error which is very serious and i didn't ever see it. Yy system is showing neither conda nor pip.
One more image is attached here to see second error UnicodeDecodeError: 'charmap' codec can't decode byte 0x90 in position 623: character maps to
Note:" Every thing was working very fine before upgrading my pycharm. As i upgraded my pycharm i got errors."
your help would be appreciated!
Thank you in advance


title: Where to implement pre-processing in PyTorch Lightning (e.g. tokenizing input text)

body:
Is there a convention to implement some kind of predict() method in PyTorch Lightning that does pre-processing before performing the actual prediction using forward()?
In my case, I have a text classifier consisting of an embedding layer and a few fully connected layers. The text needs to be tokenized before being passed to the embedding layer. During training and evaluation the LightningDataModule's setup() method does the job.
Now, I'm wondering what the best practice for inference during production is. I could add a predict() method to my LightningModule where I could write the same pre-processing code as in LightningDataModule.setup(). But, of course, I do not want to duplicate the code.
In this community example project linked in the official PyTorch Lightning docs, the authors define a prepare_sample() function in the LightningModule that is used by their predict() function, and is also passed to the LightningDataModule.
Is this the right way to handle pre-processing? Also, why is there no prepare_sample() or predict() in LightningModule? To me, this seems like a common use case, for example:
model = load_model('data/model.ckpt')  # load pre-trained model, analyzes user reviews

user_input = input('Your movie review > ')

predicted_rating = model.predict(user_input)  # e.g. "I liked the movie pretty much." -> 4 stars

print('Predicted rating: %s/5 stars' % predicted_rating)

Now that I think about it, predict() should also process the result from forward() the same way the evaluation code does, like selecting the class with the highest output or selecting all classes with outputs larger than some threshold - some more code that should not be duplicated.


I have this neural net from the repo: https://github.com/ayooshkathuria/YOLO_v3_tutorial_from_scratch/blob/master/darknet.py.
class Darknet(nn.Module):
    def __init__(self, cfgfile):
        super(Darknet, self).__init__()
        self.blocks = parse_cfg(cfgfile)
        self.net_info, self.module_list = create_modules(self.blocks)

    def forward(self, x, CUDA):
        modules = self.blocks[1:]
        outputs = {}   #We cache the outputs for the route layer

        write = 0
        for i, module in enumerate(modules):
            module_type = (module["type"])

            if module_type == "convolutional" or module_type == "upsample":
                x = self.module_list[i](x)

            elif module_type == "route":
                layers = module["layers"]
                layers = [int(a) for a in layers]

                if (layers[0]) > 0:
                    layers[0] = layers[0] - i

                if len(layers) == 1:
                    x = outputs[i + (layers[0])]

                else:
                    if (layers[1]) > 0:
                        layers[1] = layers[1] - i

                    map1 = outputs[i + layers[0]]
                    map2 = outputs[i + layers[1]]
                    x = torch.cat((map1, map2), 1)

            elif  module_type == "shortcut":
                from_ = int(module["from"])
                x = outputs[i-1] + outputs[i+from_]

            elif module_type == 'yolo':
                anchors = self.module_list[i][0].anchors
                #Get the input dimensions
                inp_dim = int (self.net_info["height"])

                #Get the number of classes
                num_classes = int (module["classes"])

                #Transform
                x = x.data
                x = predict_transform(x, inp_dim, anchors, num_classes, CUDA)
                if not write:              #if no collector has been intialised.
                    detections = x
                    write = 1

                else:
                    detections = torch.cat((detections, x), 1)

            outputs[i] = x

        return detections

    def load_weights(self, weightfile):
        #Open the weights file
        fp = open(weightfile, "rb")

        #The first 5 values are header information
        # 1. Major version number
        # 2. Minor Version Number
        # 3. Subversion number
        # 4,5. Images seen by the network (during training)
        header = np.fromfile(fp, dtype = np.int32, count = 5)
        self.header = torch.from_numpy(header)
        self.seen = self.header[3]

        weights = np.fromfile(fp, dtype = np.float32)

        ptr = 0
        for i in range(len(self.module_list)):
            module_type = self.blocks[i + 1]["type"]

            #If module_type is convolutional load weights
            #Otherwise ignore.

            if module_type == "convolutional":
                model = self.module_list[i]
                try:
                    batch_normalize = int(self.blocks[i+1]["batch_normalize"])
                except:
                    batch_normalize = 0

                conv = model[0]

                if (batch_normalize):
                    bn = model[1]

                    #Get the number of weights of Batch Norm Layer
                    num_bn_biases = bn.bias.numel()

                    #Load the weights
                    bn_biases = torch.from_numpy(weights[ptr:ptr + num_bn_biases])
                    ptr += num_bn_biases

                    bn_weights = torch.from_numpy(weights[ptr: ptr + num_bn_biases])
                    ptr  += num_bn_biases

                    bn_running_mean = torch.from_numpy(weights[ptr: ptr + num_bn_biases])
                    ptr  += num_bn_biases

                    bn_running_var = torch.from_numpy(weights[ptr: ptr + num_bn_biases])
                    ptr  += num_bn_biases

                    #Cast the loaded weights into dims of model weights.
                    bn_biases = bn_biases.view_as(bn.bias.data)
                    bn_weights = bn_weights.view_as(bn.weight.data)
                    bn_running_mean = bn_running_mean.view_as(bn.running_mean)
                    bn_running_var = bn_running_var.view_as(bn.running_var)

                    #Copy the data to model
                    bn.bias.data.copy_(bn_biases)
                    bn.weight.data.copy_(bn_weights)
                    bn.running_mean.copy_(bn_running_mean)
                    bn.running_var.copy_(bn_running_var)

                else:
                    #Number of biases
                    num_biases = conv.bias.numel()

                    #Load the weights
                    conv_biases = torch.from_numpy(weights[ptr: ptr + num_biases])
                    ptr = ptr + num_biases

                    #reshape the loaded weights according to the dims of the model weights
                    conv_biases = conv_biases.view_as(conv.bias.data)

                    #Finally copy the data
                    conv.bias.data.copy_(conv_biases)

                #Let us load the weights for the Convolutional layers
                num_weights = conv.weight.numel()

                #Do the same as above for weights
                conv_weights = torch.from_numpy(weights[ptr:ptr+num_weights])
                ptr = ptr + num_weights

                conv_weights = conv_weights.view_as(conv.weight.data)
                conv.weight.data.copy_(conv_weights)

When I try to run inference with the module using this detector I wrote I get the above error?
import cv2
from darknet import Darknet
from util import *
from torch.autograd import Variable
import torch

class ObjectDetector:
    def __init__(self, confidence=0.5, nms_thresh=0.4, reso=416):

        #Load Darknet model for object detection
        print("Loading network...")
        self.model = Darknet("cfg/yolov3.cfg")
        self.model.load_weights("yolov3.weights")
        print("Network successfully loaded")

        self.model.net_info["height"] = reso
        self.inp_dim = int(self.model.net_info["height"])
        if torch.cuda.is_available():
            self.model.cuda()

        self.model.eval()

        #Set detector parameters
        self.confidence = confidence
        self.nms_thresh = nms_thresh
        self.reso = reso

        #Load class file
        self.num_classes = 80
        self.classes = load_classes("data/coco.names")

    def detect(self, img):
        img_ = prep_image(img, self.inp_dim)

        if torch.cuda.is_available():
            img_.cuda()

        pred = self.model(img_, torch.cuda.is_available())
        res = write_results(pred, self.confidence, self.num_classes)[:,[1,2,3,4,7]] # only take the 4 corner coordinate points and class index

        # Create a python list for all the objects with the format [x1,y1,x2,y2,class_name]
        objs = []
        for obj in res:
            _ = obj[:4].tolist()
            _.append(self.classes[int(obj[4])])
            objs.append(_)

        return objs

if __name__=="__main__":
    detector = ObjectDetector()
    img = cv2.imread(input("Input image path: "))
    detector.detect(img)

Can anyone help me find the problem, because I can't see it. I set up an issue on the repo but there hasn't been a reply so if someone here can help, that would be great.
This is the full error trace:
Traceback (most recent call last):
  File "engine.py", line 133, in <module>
    system.run()
  File "engine.py", line 107, in run
    res = self.detector.detect(img)
  File "/home/greatman/code/vics/describe/detector.py", line 39, in detect
    pred = self.model(img_, torch.cuda.is_available())
  File "/home/greatman/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/greatman/code/vics/describe/darknet.py", line 171, in forward
    x = self.module_list[i](x)
  File "/home/greatman/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/greatman/.local/lib/python3.6/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/home/greatman/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/home/greatman/.local/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "/home/greatman/.local/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 420, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _thnn_conv2d_forward


title: Custom CNN does not have classification report

body:
I am building a model to predict a dataset and my accuracy is really good but I would like to test this by using a classification report but I am really not sure how to fit that into my code. I did not initially think about it but I am in a trouble. Could someone help? Usually, how can we generate the classification report for the custom model?
lb = joblib.load('/content/drive/MyDrive/Colab_Notebooks/outputs/lb.pkl')

class CustomCNN(nn.Module):
    def __init__(self):
        super(CustomCNN, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, 5)
        self.conv2 = nn.Conv2d(16, 32, 5)
        self.conv3 = nn.Conv2d(32, 64, 3)
        self.conv4 = nn.Conv2d(64, 128, 5)
        self.fc1 = nn.Linear(128, 256)
        self.fc2 = nn.Linear(256, len(lb.classes_))
        self.pool = nn.MaxPool2d(2, 2)
    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = self.pool(F.relu(self.conv3(x)))
        x = self.pool(F.relu(self.conv4(x)))
        bs, _, _, _ = x.shape
        x = F.adaptive_avg_pool2d(x, 1).reshape(bs, -1)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x

# learning_parameters
lr = 1e-3
batch_size = 32
device = 'cuda' if torch.cuda.is_available() else 'cpu'

print(f"Computation device: {device}\n")

# read the data.csv file and get the image paths and labels
df = pd.read_csv('/content/drive/MyDrive/Colab_Notebooks/input/data.csv')
X = df.image_path.values # image paths
y = df.target.values # targets
(xtrain, xtest, ytrain, ytest) = train_test_split(X, y,
    test_size=0.10, random_state=42)
print(f"Training instances: {len(xtrain)}")
print(f"Validation instances: {len(xtest)}")

class ImageDataset(Dataset):
    def __init__(self, images, labels=None, tfms=None):
        self.X = images
        self.y = labels
        # apply augmentations
        if tfms == 0:  # if validating
            self.aug = albumentations.Compose([
                albumentations.Resize(224, 224, always_apply=True),
            ])
        else:  # if training
            self.aug = albumentations.Compose([
                albumentations.Resize(224, 224, always_apply=True),
                albumentations.HorizontalFlip(p=0.5),
                albumentations.ShiftScaleRotate(
                    shift_limit=0.3,
                    scale_limit=0.3,
                    rotate_limit=15,
                    p=0.5
                ),
            ])

    def __len__(self):
        return (len(self.X))

    def __getitem__(self, i):
        image = Image.open(self.X[i])
        image = image.convert('RGB')
        image = self.aug(image=np.array(image))['image']
        image = np.transpose(image, (2, 0, 1)).astype(np.float32)
        label = self.y[i]
        return (torch.tensor(image, dtype=torch.float), torch.tensor(label, dtype=torch.long))

train_data = ImageDataset(xtrain, ytrain, tfms=1)
test_data = ImageDataset(xtest, ytest, tfms=0)
# dataloaders
trainloader = DataLoader(train_data, batch_size=batch_size, shuffle=True)
testloader = DataLoader(test_data, batch_size=batch_size, shuffle=False)

model = CustomCNN().to(device)
print(model)
# total parameters and trainable parameters
total_params = sum(p.numel() for p in model.parameters())
print(f"{total_params:,} total parameters.")
total_trainable_params = sum(
    p.numel() for p in model.parameters() if p.requires_grad)
print(f"{total_trainable_params:,} training parameters.")

# optimizer
optimizer = optim.Adam(model.parameters(), lr=lr)
# loss function
criterion = nn.CrossEntropyLoss()
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
    optimizer,
    mode='min',
    patience=5,
    factor=0.5,
    min_lr=1e-6,
    verbose=True
)

# validation function
def validate(model, test_dataloader):
    print('Validating')
    model.eval()
    val_running_loss = 0.0
    val_running_correct = 0
    with torch.no_grad():
        for i, data in tqdm(enumerate(test_dataloader), total=int(len(test_data) / test_dataloader.batch_size)):
            data, target = data[0].to(device), data[1].to(device)
            outputs = model(data)
            loss = criterion(outputs, target)

            val_running_loss += loss.item()
            _, preds = torch.max(outputs.data, 1)
            val_running_correct += (preds == target).sum().item()

        val_loss = val_running_loss / len(test_dataloader.dataset)
        val_accuracy = 100. * val_running_correct / len(test_dataloader.dataset)
        print(f'Val Loss: {val_loss:.4f}, Val Acc: {val_accuracy:.2f}')

        return val_loss, val_accuracy

# training function
def fit(model, train_dataloader):
    print('Training')
    model.train()
    train_running_loss = 0.0
    train_running_correct = 0
    for i, data in tqdm(enumerate(train_dataloader), total=int(len(train_data) / train_dataloader.batch_size)):
        data, target = data[0].to(device), data[1].to(device)
        optimizer.zero_grad()
        outputs = model(data)
        loss = criterion(outputs, target)
        train_running_loss += loss.item()
        _, preds = torch.max(outputs.data, 1)
        train_running_correct += (preds == target).sum().item()
        loss.backward()
        optimizer.step()

    train_loss = train_running_loss / len(train_dataloader.dataset)
    train_accuracy = 100. * train_running_correct / len(train_dataloader.dataset)

    print(f"Train Loss: {train_loss:.4f}, Train Acc: {train_accuracy:.2f}")

---I tried to include classification report here but yes, my custom CNN does not have the classification report! the commented part below was my try to see if I can generate a report but I get an error.
'''
    predictions = model.predict(x=xtest.astype("float32"), batch_size=32)
    print(classification_report(ytest.argmax(axis=1),
      predictions.argmax(axis=1), target_names=lb.classes_))
    return train_loss, train_accuracy
'''

train_loss, train_accuracy = [], []
val_loss, val_accuracy = [], []
start = time.time()
for epoch in range(5):
    print(f"Epoch {epoch + 1} of {epoch + 1}")
    train_epoch_loss, train_epoch_accuracy = fit(model, trainloader)
    val_epoch_loss, val_epoch_accuracy = validate(model, testloader)
    train_loss.append(train_epoch_loss)
    train_accuracy.append(train_epoch_accuracy)
    val_loss.append(val_epoch_loss)
    val_accuracy.append(val_epoch_accuracy)
    scheduler.step(val_epoch_loss)
end = time.time()

print(f"{(end - start) / 60:.3f} minutes")

##
# accuracy plots
plt.figure(figsize=(10, 7))
plt.plot(train_accuracy, color='green', label='train accuracy')
plt.plot(val_accuracy, color='blue', label='validataion accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.legend()
plt.savefig('/content/drive/MyDrive/Colab_Notebooks/outputs/accuracy.png')
plt.show()

# loss plots
plt.figure(figsize=(10, 7))
plt.plot(train_loss, color='orange', label='train loss')
plt.plot(val_loss, color='red', label='validataion loss')
plt.xlabel('Epochs')
plt.ylabel('Loss')
plt.legend()
plt.savefig('/content/drive/MyDrive/Colab_Notebooks/outputs/loss.png')
plt.show()

# serialize the model to disk
print('Saving model...')
torch.save(model.state_dict(), '/content/drive/MyDrive/Colab_Notebooks/outputs/sports.pth')

print('TRAINING COMPLETE')

Error:
--------------------------------------------------------------------------
ModuleAttributeError                      Traceback (most recent call last)
<ipython-input-5-ad111802746a> in <module>()
    157 for epoch in range(5):
    158     print(f"Epoch {epoch + 1} of {epoch + 1}")
--> 159     train_epoch_loss, train_epoch_accuracy = fit(model, trainloader)
    160     val_epoch_loss, val_epoch_accuracy = validate(model, testloader)
    161     train_loss.append(train_epoch_loss)

1 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __getattr__(self, name)
    777                 return modules[name]
    778         raise ModuleAttributeError("'{}' object has no attribute '{}'".format(
--> 779             type(self).__name__, name))
    780
    781     def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

ModuleAttributeError: 'CustomCNN' object has no attribute 'predict'


title: How to use glove with spacy tokenizer? (python, error)

body:
I am attempting some sentiment analysis (The following links are essentially the same as what I'm working on) and need to build a a vocab from amazon reviews that have associated good/bad/ok labels. My code fails at build_vocab and returns:  AttributeError: 'spacy.tokens.token.Token' object has no attribute 'strip'. I think this implies that build_vocab wont accept tokens, which never have attribute 'strip' and instead it wants strings? However I've seen others use build_vocab on spacy tokenized data before including here:
https://github.com/bentrevett/pytorch-sentiment-analysis/blob/master/5%20-%20Multi-class%20Sentiment%20Analysis.ipynb,
and here:
PyTorch: Loading word vectors into Field vocabulary vs. Embedding layer.
I cannot understand why I am recieving this error! Here is my code:
import torch
import torch.nn as nn
import torch.optim as optim
import time
from torchtext import data
from torchtext import datasets
from torchtext import vocab
import glove
from torchtext.vocab import GloVe

SEED = 1234
torch.manual_seed(SEED)
torch.backends.cudnn.deterministic = True

# Section 1: Spacy

import spacy
nlp = spacy.load("en_core_web_sm")
all_stopwords = nlp.Defaults.stop_words

TEXT = data.Field(tokenize = nlp, include_lengths = True, stop_words = all_stopwords)
LABEL = data.LabelField(dtype = torch.long)
fields = {'reviewText': ('t', TEXT), 'sentiment': ('l', LABEL)}

suffix = 'small-'
dataFolder = #normally this is where the following files are stored, removed for stackoverflow  # keep all your data files in a single folder"

#here, we see clean(not "noisy") .json data files for train and validate being called on consisting of an amazon product review of text (one column)
# and whether the review is good/bad/ok (the other column)
train_data, valid_data, test_data = data.TabularDataset.splits(path = dataFolder, train = suffix + 'clean-train.json',
                                                               validation = suffix + 'clean-val.json', test = suffix + 'clean-val.json',
                                                               format = 'json', fields = fields)

MAX_VOCAB_SIZE = 25_000
#GloVe vocabulary
TEXT.build_vocab(train_data, max_size = MAX_VOCAB_SIZE, vectors = 'glove.6B.100d', unk_init = torch.Tensor.normal_)


I want to convert Prototypical Networks for Few-shot Learning to onnx.
Export onnx:
torch.onnx.export(model,(example_query_images, example_query_labels, x_pred), "super_resolution.onnx")

And it raise error 'RuntimeError: Exporting the operator cdist to ONNX opset version 9 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub.'


title: Why is the output to my CPU pytorch model not repeatable?

body:
I have recently started working with pytorch and I noticed that I was not getting repeatable/deterministic results when evaluating a pre-trained model on new inputs.

I have boiled the problem down to this minimum example which shows that repeatedly applying the same simple convolution model does not yield identical results:

import numpy as np
import matplotlib.pyplot as plt
import torch

device = torch.device('cpu')

# function to get all the params from a pytorch model
def getParams(model):
    a = list(model.parameters())
    b = [a[i].detach().cpu().numpy() for i in range(len(a))]
    c = [b[i].flatten() for i in range(len(b))]
    d = np.hstack(c)

    return d

# set up a simple model (9 params)
testModule = torch.nn.Conv2d(1, 1, kernel_size = (3, 3), bias = False, stride = 1, padding = 1).double()
torch.nn.init.normal_(testModule.weight, mean=0, std=1)
testModule = testModule.eval()

# set up a dummy input
patch = torch.from_numpy(np.random.randn(1,1,80,80).astype('double')).to(device)

# apply the model 100 times
testVals = []
testParams = []
testModuleOut = []
for ii in range(100):
    testParams.append(getParams(testModule))
    testModuleOut.append(testModule(patch).cpu().detach()[0,:,:,:].numpy())

testParams = np.stack(testParams)
testModuleOut = np.stack(testModuleOut)

# view the variation of the model parameters and the output values
plt.figure()
plt.plot(np.std(testParams,axis=0))
plt.xlabel('Parameter index')
plt.ylabel('Standard deviation over runs')

plt.figure()
plt.plot(np.std(testModuleOut,axis=0).ravel())
plt.xlabel('Output index')
plt.ylabel('Standard deviation over runs')

If re-running the network were repeatable, I would expect the standard deviation graphs to show flat lines at SD = 0. But I do not get this, instead I get some random looking graph lines, that change with every run of the script (sometimes the module parameters have SD = 0, but the network output never seems to).

What is the problem with my code? The SDs seem to be around machine precision but why would repeatedly pulling params from the module cause them to change in this way? Would we not just be pulling the exact same value from memory?


title: Is there a way to reduce the time needed to find the indices of the top k largest elements from a 1D vector?

body:
I am trying to obtain the indices of the of the largest n and smallest n elements of an array in Pytorch, and then concatenate these elements into a return array, but the time needed to do so is extremely unreasonable as the size of the array increases exponentially.

I have tried the code that is included, as well as sorting the vector so I can just take the first and last n elements, but time is not reduced and I lose the indices of the vector, which is what I need.

def draw(n, distr):
    return np.concatenate((np.array(distr.topk(k=int(n), largest=True).indices),
                           np.array(distr.topk(k=int(n), largest=False).indices)),
                          axis=0)

For a series of 1 dimensional arrays, varying in size from length 10 to length 2359296; furthermore, the variable n is an integer valued at 1/10th of the length of the array. My computer can compute the indices of the largest and smallest n elements of all the arrays in about 0.5 seconds. I would prefer to minimize this time as much as possible, preferably to less than 0.2 seconds


title: PyTorch equivalent of tf.dynamic_partition

body: I need to use an equivalent of tf.dynamic_partition in PyTorch. Is there anything with similar functionality in PyTorch or other library or is there a simple and clever way to code it for PyTorch and work fast?


title: What does the `model.parameters()` include?

body:
In Pytorch,

What will be registered into the model.parameters().

As far as now, what I know are as belows:

1.  Conv layer: weight  bias
2.  BN layers: weight(gamma)  bias(beta)
3.  nn.Parameter()
    such as:   self.alpha = nn.Parameter(torch.rand(10))  defined in the model.

My question is:
And are there some parameters else that are registered in the model.parameters() ?

PS. The most common case for the model.parameters() is in the optimizer,
e.g.  pytorch resnet example

optimizer = torch.optim.SGD(model.parameters(), args.lr,
                                momentum=args.momentum,
                                weight_decay=args.weight_decay)

Thank you in advance.


title: A gradient clipping in Chainer

body:
Can I get a Gradient Clipping function in Chainer?

I found some codes in Pytorch Documentation : https://pytorch.org/docs/stable/_modules/torch/nn/utils/clip_grad.html

Is there anything like alternative function in Chainer?
I just found chainer.optimizer_hooks.GradientClipping, but this is hard to utilize.

Thanks in advance.


title: What is Reference Facial Points and how are they created?

body:
I'm trying to use face.evoLVe library, which is high-performance Face Recognition Library in PyTorch. Going through the codes, I encountered a list of coordinates named REFERENCE_FACIAL_POINTS which is :

REFERENCE_FACIAL_POINTS = [        # default reference facial points for crop_size = (112, 112); should adjust REFERENCE_FACIAL_POINTS accordingly for other crop_size
    [30.29459953,  51.69630051],
    [65.53179932,  51.50139999],
    [48.02519989,  71.73660278],
    [33.54930115,  92.3655014],
    [62.72990036,  92.20410156]
]

Further down the code, these numbers are converted into numpy arrays and used heavily everywhere in align_trans.py

I have some questions:

What exactly are these numbers? Reading the comment, I'm certain they are locations for eyes, lips, etc but what exactly do they represent and how are they calculated?
It seems they are tightly coupled with the size of the input image [used in training (at the very least)]. Knowing this, how can one calculate new reference points for newer image sizes?
Are these points for frontal pose only? or will they work on profiles, etc as well?  If they don't, how can we add reference points for profiles or other random facial poses?


title: PyTorch Python && C++ inconsistent results from shared model

body:
I am running a Python3 trained, torch_script exported CNN with C++.
However, I can not reproduce results on testing data.

The class labels the classifier returns in respective language are opposite for the identical data patch. Not sure if some memory is corrupted in the C++ implementation, but I was able to validate at least the Python implementation gives correct results. How else shall I debug? Right now I am trying to compare the weight matrices of the conv kernels to validate the model is consistent among both implementations.

Check the last lines in the code to see the difference.

Note - patch stats are identical between both languages, could be that the memory is misaligned in C++?

Anything else I can check?

Python export script:

    cnn = CNN1()

    cnn.load_state_dict(torch.load(args.cnn))

    torch_script_nn = torch.jit.script(cnn)

    script_file_name = args.cnn.replace('.torch', '.pytorch_script')
    torch_script_nn.save(script_file_name)

    print('{} CONVERTED TO {} '.format(args.cnn, script_file_name))

Python test script

    cnn = CNN1()
    cnn.load_state_dict(torch.load(cnn_file))

    i = 100
    j = 100
    patch_size = 14
    img = np.load(f)['arr_0']
    # plt.imshow(img, cmap='gray');plt.show()

    patch = img[i: i+ patch_size, j:j+patch_size]
    print('Patch max : {}, min {}'.format(np.max(patch), np.min(patch)))

    plt.imshow(patch, cmap='gray');plt.show()
    patch = np.expand_dims(patch, axis=2)
    normalized_patch = (patch - np.min(patch)) / np.max(patch - np.min(patch))
    print('Normalized Patch max : {}, min {}'.format(np.max(normalized_patch), np.min(normalized_patch)))
    torch_stack = torch.stack([torch.Tensor(normalized_patch)])
    print(torch_stack.shape)
    res = cnn(torch_stack.permute(0, 3, 1, 2))

    print('Class label {}'.format(res))

Prints

Patch max : 16.035221099853516, min 11.432785987854004
Normalized Patch max : 1.0, min 0.0
torch.Size([1, 14, 14, 1])
tensor([[9.9995e-01, 5.4011e-05]], grad_fn=<SoftmaxBackward>)

C++ test

std::cout << "TESTING CNN" << std::endl;
    std::size_t i = 100;
    std::size_t j = 100;

    const cv::Rect roi(i, j, patchSize, patchSize);
    cv::Mat amplitudePatch = amplitude(roi);
    double min;
    double max;
    cv::minMaxLoc(amplitudePatch, &min, &max);
    std::cout << "Patch Max : " << max << std::endl;
    std::cout << "Patch Min " << min << std::endl;

    cv::imshow("Patch", amplitudePatch);
    cv::Mat normalizedPatch = normalizeAmplitudePatch(amplitudePatch);
    cv::imshow("NPatch", normalizedPatch);
    cv::minMaxLoc(normalizedPatch, &min, &max);
    std::cout << "Normalized Patch Max : " << max << std::endl;
    std::cout << "Normalized Patch Min " << min << std::endl;

    cv::waitKey(0);
    auto options = torch::TensorOptions().dtype(torch::kF32);

    std::cout << 1 << " " << amplitudePatch.size().height<< " " <<
        amplitudePatch.size().width<< " " << amplitudePatch.channels() << std::endl;

    torch::Tensor patchTensor = torch::from_blob(
      amplitudePatch.data,
      {1, amplitudePatch.size().height,
        amplitudePatch.size().width, amplitudePatch.channels()},
      options);

    std::vector<c10::IValue> inputs;
    inputs.push_back(patchTensor.permute({0, 3, 1, 2}));
    torch::Tensor result = cnn.forward(inputs).toTensor();

    std::cout << result << std::endl;

Prints

Patch Max : 16.0352
Patch Min : 11.4328
Could not initialize OpenGL for RasterGLSurface, reverting to RasterSurface.
Could not initialize OpenGL for RasterGLSurface, reverting to RasterSurface.
Normalized Patch Max : 1
Normalized Patch Min 0
1 14 14 1
 0  1
[ Variable[CPUFloatType]{1,2} ]


title: My Pytorch CNN computes but does not show training results

body:
I am writing a CNN to classify some images. when I run the training part of my code, the only output I receive is the print "Training finished". This tells me that my code was iterated through but did not successfully compute - I have no errors and have really tried to identify the error.

The fc1's (First connected layer) input size is supposed to be the flattened previous layer.

Data

train_split = 0.70 # Defines the ratio of train/valid/test data.
valid_split = 0.10

train_size = int(len(data_df)*train_split)
valid_size = int(len(data_df)*valid_split)

ins_dataset_train = ImageNet10(
   df=data_df[:train_size],
   transform=data_transform,
)

ins_dataset_valid = ImageNet10(
   df=data_df[train_size:(train_size +
   valid_size)].reset_index(drop=True),
   transform=data_transform,
)

ins_dataset_test = ImageNet10(
   df=data_df[(train_size + valid_size):].reset_index(drop=True),
   transform=data_transform,
)

Data Loaders

train_loader = torch.utils.data.DataLoader(
   ins_dataset_train,
   batch_size=16,
   shuffle=True,
   num_workers=2
)
valid_loader = torch.utils.data.DataLoader(
   ins_dataset_valid,
   batch_size=16,
   shuffle=True,
   num_workers=2
)
test_loader = torch.utils.data.DataLoader(
   ins_dataset_test,
   batch_size=24, # Forward pass only so batch size can be larger
   shuffle=False,
   num_workers=2
)

Convolutional neural network

class ConvNet(nn.Module):

def __init__(self, num_classes=10):
    super(ConvNet, self).__init__()

    # Add network layers here
    # Layer 1
    self.conv1 = nn.Conv2d(3,16, (3,3))
    self.pool = nn.MaxPool2d(2,2)
    self.relu = nn.ReLU()
    self.dropout = nn.Dropout(p=0.3)

        # Layer 2
    self.conv2 = nn.Conv2d(16,24, (4,4))
    self.pool = nn.MaxPool2d(2,2)
    self.relu = nn.ReLU()
    self.dropout = nn.Dropout(p=0.3)

        # Layer 3
    self.conv3 = nn.Conv2d(24,32, (4,4))
    self.pool = nn.MaxPool2d(2,2)
    self.relu = nn.ReLU()
    self.dropout = nn.Dropout(p=0.3)

        # Layer 4 - Fully connected
    self.fc1 = nn.Linear(32 * 8 * 8, 512)

    self.fc2 = nn.Linear(512, num_classes)
    self.final = nn.Softmax(dim=1)

def forward(self, x):
   out = x.reshape(x.size(0), -1) # TODO what does this do? Why do we need it?
   out = self.fc1(out)
   return out

Training Model

def train_model_epochs(num_epochs):
  for epoch in range(10):

    running_loss = 0.0

    for i, data in enumerate(test_loader, 0):
      images, labels = data
      print("Batch", i, "size:", len(images))

      optimizer.zero_grad()

      outputs = net(images)

      loss = loss_function(outputs, labels)

      loss.backward()

      optimizer.step()

      running_loss =+ loss.item()
      if i%1000 == 999:
         print('Epoch / Batch [%d / %d] - Loss: %.3f' %
         (epoch + 1, i + 1, running_loss / 1000))
         running_loss = 0.0

  print("Training finished")


title: Attribute warnings for PyTorch classes in PyCharm

body:
I'm getting warnings for certain instructions.

Example 1:

Code:

input_a = Variable(torch.stack(a1, dim=0), requires_grad=False).to(self.device)

Warnings:

  Unexpected argument(s)

  Possible callees:

  object(self: object)

  object.__new__(cls: object)

  Unresolved attribute reference 'to' for class 'Variable'

Example 2:

Code:

optimizer = torch.optim.Adam(model.parameters(), lr=self.lr, betas=(0.9, 0.999), eps=1e-8)
...
print('%d minutes: epoch %d loss = %.3f, learning rate: %.4f'
              % ((time.time() - start_time) // 60, epoch + 1, avg_loss, optimizer.param_groups[0]['lr']))

Warning:

  Type 'Adam' doesn't gave expected attribute 'param_groups'

What do they mean and how should I rewrite my code to remove them?


title: Can my PyTorch forward function do additional operations?

body:
Typically a forward function strings together a bunch of layers and returns the output of the last one. Can I do some additional processing after that last layer before returning? For example, some scalar multiplication and reshaping via .view?

I know that the autograd somehow figures out gradients. So I don’t know if my additional processing will somehow screw that up. Thanks.


title: How does loss.backward() work for batches?

body:
So I am training a DDQN to play connect four at the moment. At each state, the network predicts the action the best action and moves accordingly. The code looks basically like follows:

for epoch in range(num_epochs):
        for i in range(batch_size):
                while game is not finished:
                        action = select_action(state)
                        new_state = play_move(state, action)
                        pred, target = get_pred_target(state, new_state, action)
                        preds = torch.cat([preds, pred])
                        targets = torch.cat([targets, target)]
        loss = loss_fn(preds, targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

While training, the network is getting a little bit better, but nowhere as good as I would expect. Thinking about it, I am wondering now, whether I have actually correctly implemented the loss.backward() call. The point is, I am saving all the predictions and targets for each move in the tensors preds and targets. However, I am not tracking the states, that have led to these predictions and targets. But isn't that necessary for the backward propagation, or is this information somehow saved?

Thank you very much!


title: import matplotlib failed while deploying my model in AWS sagemaker

body:
I have deployed my AWS model successfully.
but while testing i am getting runtime Error: "import matplotlib.pyplot as plt" . I think it is due to pytorch framework version i used(framework_version=1.2.0). I am facing the same issue when i use higher versions as well.
PyTorchModel(model_data=model_artifact,
                 role = role,
                 framework_version=1.2.0,
                 entry_point='predict.py',
                 predictor_cls=ImagePredictor)

I have other issue when i use version=1.0.0. i.e i am not able to import libraries from sub directories and deployment itself is failing.
Eg: i have some code files in "Code" directory.
 from Code.CTModel import NetWork  ---> **this line will fail as "No module named Code" when i use version=1.0.0**

Ultimately i want to how to use/import libraries which are written under sub-directories.