title: How to run inference of a pytorch model on pyspark dataframe (create new column with prediction) using pandas_udf?

body:
Is there a way to run the inference of pytorch model over a pyspark dataframe in vectorized way (using pandas_udf?).

One row udf is pretty slow since the model state_dict() needs to be loaded for each row. I'm trying to use pandas_udf to speed this up, since all the operations can be vectorized efficiently in pandas/pytorch.

I've looked at this databricks post for inspiration, but it's doesn't correspond exactly to my use case since I want to run prediction on an existing pyspark dataframe.

I can get it to work using one row udf in this simple example:

import torch
import torch.nn as nn
from pyspark.sql.functions import col, pandas_udf, PandasUDFType, udf
import pyspark.sql.functions as F
from pyspark.sql import SparkSession
from pyspark.sql.types import ArrayType, FloatType, DoubleType
import pandas as pd
import numpy as np

spark = SparkSession.builder.master('local[*]') \
    .appName("model_training") \
    .getOrCreate()

class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.w = nn.Linear(5, 1)

    def forward(self, x):
        return self.w(x)

net = Net()
bc_model_state = spark.sparkContext.broadcast(net.state_dict())

df = spark.sparkContext.parallelize([[np.random.rand() for i in range(5)] for j in range(10)]).toDF()
df = df.withColumn('features', F.array([F.col(f"_{i}") for i in range(1, 6)]))

def get_model_for_eval():
  # Broadcast the model state_dict
  net.load_state_dict(bc_model_state.value)
  net.eval()
  return net

def one_row_predict(x):
    model = get_model_for_eval()
    t = torch.tensor(x, dtype=torch.float32)
    prediction = model(t).cpu().detach().item()
    return prediction

one_row_udf = udf(one_row_predict, FloatType())
df = df.withColumn('pred_one_row', one_row_udf(col('features')))
df.show()

Output:

+--------------------+-------------------+-------------------+-------------------+-------------------+--------------------+------------+
|                  _1|                 _2|                 _3|                 _4|                 _5|            features|pred_one_row|
+--------------------+-------------------+-------------------+-------------------+-------------------+--------------------+------------+
|  0.8447505355266759| 0.3938414671838497|0.46347383092447003| 0.7694022276208854| 0.6152606009215115|[0.84475053552667...| 0.025048971|
|0.023782157504950607| 0.6434186254505012| 0.4090423037706754| 0.5466917794921007| 0.7855157903802007|[0.02378215750495...|  0.19694215|
|  0.5057589877333257| 0.7186078182786649| 0.9123361330966105|  0.601837718628886| 0.0773272396167538|[0.50575898773332...|    0.278222|
|  0.2815336141913932| 0.5196112020157087| 0.9646444599173869|0.04844988843812004|0.35445251642633047|[0.28153361419139...|  0.10699606|
|  0.3896101050146765|0.38732747821339863| 0.8516864705178889| 0.2500977280156421| 0.7781221754566505|[0.38961010501467...| -0.08206403|
|  0.8223344715797269| 0.9089425281658239|0.10088026161623431| 0.9920995834835098|0.40665125930441104|[0.82233447157972...|   0.3565607|
| 0.31167413110257425| 0.9778009876605741| 0.4717549025588036|0.24563879994222826| 0.7594244867194454|[0.31167413110257...|  0.18897778|
|  0.5667657426129576| 0.5383639427018171| 0.2983527299596511|0.18914810241640534|0.47854422807435326|[0.56676574261295...|  0.17796803|
|  0.6419824467244137|0.03992370080139418|0.38462617679839173|  0.709487894249459|0.23020927682221126|[0.64198244672441...|  0.15635887|
|  0.7972928622000178| 0.7700992684264264| 0.4387404431803098| 0.1340696629092989| 0.7072213018683782|[0.79729286220001...|   0.0500246|
+--------------------+-------------------+-------------------+-------------------+-------------------+--------------------+------------+

Trying to do the same thing with in a vectorized way, this works:

def batch_predict(x):
    model = get_model_for_eval()
    xp = np.vstack(x)
    t = torch.tensor(xp, dtype=torch.float32)
    prediction = model(t).cpu().detach().numpy().flatten()
    return pd.Series(prediction)

df_pd = df.toPandas()
x = df_pd['features']
print(batch_predict(x))

But running it inside a pandas_udf fails:

batch_udf = pandas_udf(batch_predict, FloatType())
df = df.withColumn('pred_batch', batch_udf(col('features')))
df.show()

with:

20/02/11 10:13:01 ERROR Executor: Exception in task 2.0 in stage 1.0 (TID 3)
java.lang.IllegalArgumentException
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:334)
    at org.apache.arrow.vector.ipc.message.MessageSerializer.readMessage(MessageSerializer.java:543)
    at org.apache.arrow.vector.ipc.message.MessageChannelReader.readNext(MessageChannelReader.java:58)
    at org.apache.arrow.vector.ipc.ArrowStreamReader.readSchema(ArrowStreamReader.java:132)
    at org.apache.arrow.vector.ipc.ArrowReader.initialize(ArrowReader.java:181)
    at org.apache.arrow.vector.ipc.ArrowReader.ensureInitialized(ArrowReader.java:172)
    at org.apache.arrow.vector.ipc.ArrowReader.getVectorSchemaRoot(ArrowReader.java:65)
    at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:162)
    at org.apache.spark.sql.execution.python.ArrowPythonRunner$$anon$1.read(ArrowPythonRunner.scala:122)
    at org.apache.spark.api.python.BasePythonRunner$ReaderIterator.hasNext(PythonRunner.scala:410)
    at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
    at org.apache.spark.sql.execution.python.ArrowEvalPythonExec$$anon$2.<init>(ArrowEvalPythonExec.scala:98)
    at org.apache.spark.sql.execution.python.ArrowEvalPythonExec.evaluate(ArrowEvalPythonExec.scala:96)
    at org.apache.spark.sql.execution.python.EvalPythonExec$$anonfun$doExecute$1.apply(EvalPythonExec.scala:127)
    at org.apache.spark.sql.execution.python.EvalPythonExec$$anonfun$doExecute$1.apply(EvalPythonExec.scala:89)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:801)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:123)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Thanks for the help


title: How do I implement a PyTorch Dataset for use with AWS SageMaker?

body:
I have implemented a PyTorch Dataset that works locally (on my own desktop), but when executed on AWS SageMaker, it breaks. My Dataset implementation is as follows.

class ImageDataset(Dataset):
    def __init__(self, path='./images', transform=None):
        self.path = path
        self.files = [join(path, f) for f in listdir(path) if isfile(join(path, f)) and f.endswith('.jpg')]
        self.transform = transform
        if transform is None:
            self.transform = transforms.Compose([
                transforms.Resize((128, 128)),
                transforms.ToTensor(),
                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
            ])

    def __len__(self):
        return len(files)

    def __getitem__(self, idx):
        img_name = self.files[idx]

        # we may infer the label from the filename
        dash_idx = img_name.rfind('-')
        dot_idx = img_name.rfind('.')
        label = int(img_name[dash_idx + 1:dot_idx])

        image = Image.open(img_name)

        if self.transform:
            image = self.transform(image)

        return image, label

I am following this example and this one too, and I run the estimator as follows.

inputs = {
 'train': 'file://images',
 'eval': 'file://images'
}
estimator = PyTorch(entry_point='pytorch-train.py',
                            role=role,
                            framework_version='1.0.0',
                            train_instance_count=1,
                            train_instance_type=instance_type)
estimator.fit(inputs)

I get the following error.

  FileNotFoundError: [Errno 2] No such file or directory: './images'

In the example that I am following, they upload the CFAIR dataset (which is downloaded locally) to S3.

inputs = sagemaker_session.upload_data(path='data', bucket=bucket, key_prefix='data/cifar10')

If I take a peek at inputs, it is just a string literal s3://sagemaker-us-east-3-184838577132/data/cifar10. The code to create a Dataset and a DataLoader is shown here, which does not help unless I track down the source and step through the logic.

I think what needs to happen inside my ImageDataset is to supply the S3 path and use the AWS CLI or something to query the files and acquire their content. I do not think the AWS CLI is the right approach as this relies on the console and I will have to execute some sub-process commands and then parse through.

There must be a recipe or something to create a custom Dataset backed by S3 files, right?


title: How to add L1 Regularization to PyTorch NN Model?

body:
When searching for ways to implement L1 regularization in PyTorch Models, I came across this question, which is now 2 years old so i was wondering if theres anything new on this topic?
I also found this recent approach of dealing with the missing l1 function. However I don't understand how to use it for a basic NN as shown below.
class FFNNModel(nn.Module):
    def __init__(self, input_dim, output_dim, hidden_dim, dropout_rate):
        super(FFNNModel, self).__init__()
        self.input_dim = input_dim
        self.output_dim = output_dim
        self.hidden_dim = hidden_dim
        self.dropout_rate = dropout_rate
        self.drop_layer = nn.Dropout(p=self.dropout_rate)
        self.fully = nn.ModuleList()
        current_dim = input_dim
        for h_dim in hidden_dim:
            self.fully.append(nn.Linear(current_dim, h_dim))
            current_dim = h_dim
        self.fully.append(nn.Linear(current_dim, output_dim))

    def forward(self, x):
        for layer in self.fully[:-1]:
            x = self.drop_layer(F.relu(layer(x)))
        x = F.softmax(self.fully[-1](x), dim=0)
        return x

I was hoping simply putting this before training would work:
model = FFNNModel(30,5,[100,200,300,100],0.2)
regularizer = _Regularizer(model)
regularizer = L1Regularizer(regularizer, lambda_reg=0.1)

with
out = model(inputs)
loss = criterion(out, target) + regularizer.__add_l1()

Does anyone understand how to apply these 'ready to use' classes?


title: Cartpole-v0 loss increasing using DQN

body:
Hi I'm trying to train a DQN to solve gym's Cartpole problem.
For some reason the Loss looks like this (orange line). Can y'all take a look at my code and help with this? I've played around with the hyperparameters a decent bit so I don't think they're the issue here.

class DQN(nn.Module):
    def __init__(self, input_dim, output_dim):
        super(DQN, self).__init__()
        self.linear1 = nn.Linear(input_dim, 16)
        self.linear2 = nn.Linear(16, 32)
        self.linear3 = nn.Linear(32, 32)
        self.linear4 = nn.Linear(32, output_dim)

    def forward(self, x):
        x = F.relu(self.linear1(x))
        x = F.relu(self.linear2(x))
        x = F.relu(self.linear3(x))
        return self.linear4(x)

final_epsilon = 0.05
initial_epsilon = 1
epsilon_decay = 5000
global steps_done
steps_done = 0

def select_action(state):
    global steps_done
    sample = random.random()
    eps_threshold = final_epsilon + (initial_epsilon - final_epsilon) * \
                    math.exp(-1. * steps_done / epsilon_decay)
    if sample > eps_threshold:
        with torch.no_grad():
            state = torch.Tensor(state)
            steps_done += 1
            q_calc = model(state)
            node_activated = int(torch.argmax(q_calc))
            return node_activated
    else:
        node_activated = random.randint(0,1)
        steps_done += 1
        return node_activated

class ReplayMemory(object): # Stores [state, reward, action, next_state, done]

    def __init__(self, capacity):
        self.capacity = capacity
        self.memory = [[],[],[],[],[]]

    def push(self, data):
        """Saves a transition."""
        for idx, point in enumerate(data):
            #print("Col {} appended {}".format(idx, point))
            self.memory[idx].append(point)

    def sample(self, batch_size):
        rows = random.sample(range(0, len(self.memory[0])), batch_size)
        experiences = [[],[],[],[],[]]
        for row in rows:
            for col in range(5):
                experiences[col].append(self.memory[col][row])
        return experiences

    def __len__(self):
        return len(self.memory[0])

input_dim, output_dim = 4, 2
model = DQN(input_dim, output_dim)
target_net = DQN(input_dim, output_dim)
target_net.load_state_dict(model.state_dict())
target_net.eval()
tau = 2
discount = 0.99

learning_rate = 1e-4
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

memory = ReplayMemory(65536)
BATCH_SIZE = 128

def optimize_model():
    if len(memory) < BATCH_SIZE:
        return 0
    experiences = memory.sample(BATCH_SIZE)
    state_batch = torch.Tensor(experiences[0])
    action_batch = torch.LongTensor(experiences[1]).unsqueeze(1)
    reward_batch = torch.Tensor(experiences[2])
    next_state_batch = torch.Tensor(experiences[3])
    done_batch = experiences[4]

    pred_q = model(state_batch).gather(1, action_batch)

    next_state_q_vals = torch.zeros(BATCH_SIZE)

    for idx, next_state in enumerate(next_state_batch):
        if done_batch[idx] == True:
            next_state_q_vals[idx] = -1
        else:
            # .max in pytorch returns (values, idx), we only want vals
            next_state_q_vals[idx] = (target_net(next_state_batch[idx]).max(0)[0]).detach()

    better_pred = (reward_batch + next_state_q_vals).unsqueeze(1)

    loss = F.smooth_l1_loss(pred_q, better_pred)
    optimizer.zero_grad()
    loss.backward()
    for param in model.parameters():
        param.grad.data.clamp_(-1, 1)
    optimizer.step()
    return loss

points = []
losspoints = []

#save_state = torch.load("models/DQN_target_11.pth")
#model.load_state_dict(save_state['state_dict'])
#optimizer.load_state_dict(save_state['optimizer'])

env = gym.make('CartPole-v0')
for i_episode in range(5000):
    observation = env.reset()
    episode_loss = 0
    if episode % tau == 0:
        target_net.load_state_dict(model.state_dict())
    for t in range(1000):
        #env.render()
        state = observation
        action = select_action(observation)
        observation, reward, done, _ = env.step(action)

        if done:
            next_state = [0,0,0,0]
        else:
            next_state = observation

        memory.push([state, action, reward, next_state, done])
        episode_loss = episode_loss + float(optimize_model(i_episode))
        if done:
            points.append((i_episode, t+1))
            print("Episode {} finished after {} timesteps".format(i_episode, t+1))
            print("Avg Loss: ", episode_loss / (t+1))
            losspoints.append((i_episode, episode_loss / (t+1)))
            if (i_episode % 100 == 0):
                eps = final_epsilon + (initial_epsilon - final_epsilon) * \
                    math.exp(-1. * steps_done / epsilon_decay)
                print(eps)
            if ((i_episode+1) % 5001 == 0):
                save = {'state_dict': model.state_dict(), 'optimizer': optimizer.state_dict()}
                torch.save(save, "models/DQN_target_" + str(i_episode // 5000) + ".pth")
            break
env.close()

x = [coord[0] * 100 for coord in points]
y = [coord[1] for coord in points]

x2 = [coord[0] * 100 for coord in losspoints]
y2 = [coord[1] for coord in losspoints]

plt.plot(x, y)
plt.plot(x2, y2)
plt.show()

I basically followed the tutorial pytorch has, except using the state returned by the env rather than the pixels. I also changed the replay memory because I was having issues there. Other than that, I left everything else pretty much the same.

Edit:

I tried overfitting on a small batch and the Loss looks like this without updating the target net and this when updating it

Edit 2:

This is definitely an issue with the target net, I tried removing it and loss seemed to not increase exponentially


title: PyTorch having trouble detecting CUDA

body:
I am running CNN on PyTorch. The torch.cuda.is_available() function returned false and no GPU is detected. However, I can run Keras model with GPU. Here is my system information:

OS: Ubuntu 18.04.3
Python 3.7.3 (Conda)
GPU: GTX1080Ti
Nvidia driver: 430.50

When I check nvidia-smi, the output said that the CUDA version is 10.1. However, the nvcc -V command tells me that it is CUDA 9.1.

I downloaded NVIDIA-Linux-x86_64-430.50.run from the official site and install it with command line. I installed CUDA 10.1 using these following command line recommended by the official site:

wget http://developer.download.nvidia.com/compute/cuda/10.1/Prod/local_installers/cuda_10.1.243_418.87.00_linux.run
sudo sh cuda_10.1.243_418.87.00_linux.run

I installed PyTorch through pip install. What is wrong? Thanks in advance!


title: RMSE loss for multi output regression problem in PyTorch

body:
I'm training a CNN architecture to solve a regression problem using PyTorch where my output is a tensor of 20 values. I planned to use RMSE as my loss function for the model and tried to use PyTorch's nn.MSELoss() and took the square root for it using torch.sqrt() for that but got confused after obtaining the results.I'll try my best to explain why. It's obvious that for a batch-size bs my output tensor's dimensions would be [bs , 20].I tried to implement and RMSE function of my own :

   def loss_function (predicted_x , target ):
        loss = torch.sum(torch.square(predicted_x - target) , axis= 1)/(predicted_x.size()[1]) #Taking the mean of all the squares by dividing it with the number of outputs i.e 20 in my case
        loss = torch.sqrt(loss)
        loss = torch.sum(loss)/predicted_x.size()[0]  #averaging out by batch-size
        return loss

But the output of my loss_function() and how PyTorch implements it with nn.MSELoss() differed . I'm not sure whether my implementation is wrong or am I using nn.MSELoss() in the wrong way.


I am using models.vgg16(pretrained=True) model for image classification, where number of classes = 3.

Batch size is 12 trainloader = torch.utils.data.DataLoader(train_data, batch_size=12, shuffle=True) since error says Target size (torch.Size([12])) must be the same as input size (torch.Size([12, 1000]))

I have changed last fc layer parameters and got last FC layer as  Linear(in_features=1000, out_features=3, bias=True)

Loss function is BCEWithLogitsLoss()

criterion = nn.BCEWithLogitsLoss()
optimizer = optim.SGD(vgg16.parameters(), lr=0.001, momentum=0.9)

Training code is

        # zero the parameter gradients
        optimizer.zero_grad()
        outputs = vgg16(inputs)               #----> forward pass
        loss = criterion(outputs, labels)   #----> compute loss #error occurs here
        loss.backward()                     #----> backward pass
        optimizer.step()                    #----> weights update

While computing loss, I get this error Target size (torch.Size([12])) must be the same as input size (torch.Size([12, 1000]))

code is available at: code


title: Error in loading model in PyTorch

body:
I Have the following code snippet

from train import predict
import random
import torch

ann=torch.load('ann.pt') #importing trained model

while True:
      k=raw_input("User:")
      intent,top_value,top_index = predict(str(k),ann)
      print(intent)

when I run the script it is throwing the error as below:

Traceback (most recent call last):
  File "test.py", line 6, in <module>
    ann=torch.load('ann.pt') #importing trained model
  File "/home/local/ZOHOCORP/raghav-5305/miniconda2/lib/python2.7/site-packages/torch/serialization.py", line 261, in load
    return _load(f, map_location, pickle_module)
  File "/home/local/ZOHOCORP/raghav-5305/miniconda2/lib/python2.7/site-packages/torch/serialization.py", line 409, in _load
    result = unpickler.load()
AttributeError: 'module' object has no attribute 'ANN'

I have ann.pt file in the same folder as my script is.
Kindly help me identify fix the error and load the model.
Thanks in advance.


title: How to prune weights less than a threshold in PyTorch?

body:
How to prune weights of a CNN (convolution neural network) model which is less than a threshold value (let's consider prune all weights which are <= 1).

How we can achieve that for a weight file saved in .pth format in pytorch?


title: RuntimeError: CUDA out of memory. Tried to allocate 2.86 GiB (GPU 0; 10.92 GiB total capacity;...9.06 GiB reserved in total by PyTorch)

body:
What does it mean by 9.06 GiB reserved in total by PyTorch.

If i use a GPU of smaller size 7.80 GiB total capacity for the same script, it says 6.20 GiB reserved in total by PyTorch
How does reservation in Pytorch work and why does the reserved memory change depending on the GPU size?

To solve the error message, RuntimeError: CUDA out of memory. Tried to allocate 2.86 GiB (GPU 0; 10.92 GiB total capacity; 9.02 GiB already allocated; 1.29 GiB free; 9.06 GiB reserved in total by PyTorch) I have tried reducing the batch size from 10 to 5 to 3.
I have tried deleting unused tensors with del x_train1. I have also tried using torch.cuda.empty_cache(). I have also used with torch.no_grad() when applying the pretrained model at x_train1 = bert_model(train_indices)[2] and also when training and validating the new model. But none of them work.

This is the trace:

cuda:0
    x_train1 = bert_model(train_indices)[2]  # Models outputs are tuples
  File "/home/kosimadukwe/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/kosimadukwe/miniconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 783, in forward
    input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
  File "/home/kosimadukwe/miniconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/kosimadukwe/miniconda3/lib/python3.7/site-packages/transformers/modeling_bert.py", line 177, in forward
    embeddings = inputs_embeds + position_embeddings + token_type_embeddings
RuntimeError: CUDA out of memory. Tried to allocate 2.86 GiB (GPU 0; 10.92 GiB total capacity; 9.02 GiB already allocated; 1.29 GiB free; 9.06 GiB reserved in total by PyTorch)

and nvidia-smi out put

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.36       Driver Version: 440.36       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:3B:00.0 Off |                  N/A |
| 54%   79C    P2   233W / 250W |   8613MiB / 11178MiB |    100%   E. Process |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 108...  Off  | 00000000:AF:00.0 Off |                  N/A |
| 58%   79C    P2   247W / 250W |   4545MiB / 11178MiB |      0%   E. Process |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 108...  Off  | 00000000:D8:00.0 Off |                  N/A |
| 23%   29C    P0    56W / 250W |      0MiB / 11178MiB |      2%   E. Process |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0   1025219      C   /usr/pkg/bin/python3.8                      8601MiB |
|    1   1024440      C   /usr/pkg/bin/python3.8                      4535MiB |

with

os.environ['CUDA_VISIBLE_DEVICES'] = '2'


title: how to calculate loss over a number of images and then back propagate the average loss and update network weight

body:
I am doing a task where the batch size is 1, i.e, each batch contains only 1 image. So I have to do manual batching: when the the number of accumulated losses reach a number, average the loss and then do back propagation.
My original code is:

real_batchsize = 200

for epoch in range(1, 5):
    net.train()

    total_loss = Variable(torch.zeros(1).cuda(), requires_grad=True)

    iter_count = 0
    for batch_idx, (input, target) in enumerate(train_loader):

        input, target = Variable(input.cuda()), Variable(target.cuda())
        output = net(input)

        loss = F.nll_loss(output, target)

        total_loss = total_loss + loss

        if batch_idx % real_batchsize == 0:
            iter_count += 1

            ave_loss = total_loss/real_batchsize
            ave_loss.backward()
            optimizer.step()

            if iter_count % 10 == 0:
                print("Epoch:{}, iteration:{}, loss:{}".format(epoch,
                                                           iter_count,
                                                           ave_loss.data[0]))
            total_loss.data.zero_()
            optimizer.zero_grad()

This code will give the error message

  RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

I have tried the following way,

First way (failed)

I read some post about this error message, but can not understand it fully. Change ave_loss.backward() to ave_loss.backward(retain_graph=True) prevent the error message, but the loss doesn't improve the soon becomes nan.

Second way (failed)

I also tried to change total_loss = total_loss + loss.data[0], this will also prevent the error message. But the loss are always the same. So there must be something wrong.

Third way (success)

Following the instruction in this post, for each image's loss,I divide the loss by real_batchsize and backprop it. When the number of input image reach the real_batchsize, I do one update of parameter using optimizer.step().  The loss is slowly decreasing as the training process goes. But the training speed is really slow, because we backprop for each image.

My question

What does the error message mean in my case? Also, why doesn't first way and second way work?  How to write the code correctly so that we can backprop gradient every real_batchsize images and update gradient once so that the training speed a faster?  I know my code is nearly correct, but I just do not know how to change it.


title: Heroku: slug size too large after installing Pytorch

body:
I've been getting the slug size too large warning (Compiled slug size: 789.8M is too large (max is 500M)) from Heroku and I can't figure out why, as my model size (cnn.pth below) is fairly small and my file directory is only 1.1mb in total: screenshot of directory.

It seems like the size increase is caused by running pipenv install torch, as the slug size was 89.1mb before installing torch and 798.8mb after.

My Pipfile currently has these packages installed:

[packages]
flask = "*"
flask-sqlalchemy = "*"
psycopg2 = "*"
psycopg2-binary = "*"
requests = "*"
numpy = "*"
gunicorn = "*"
pillow = "*"
torch = "*"

Is there any workaround for this?

Edit: I'm running Mac OSX 10.10.5, using Flask and pipenv.


title: Can we use pytorch scatter_ on GPU

body:
I'm trying to do one hot encoding on some data with pyTorch on GPU mode, however, it keeps giving me an exception. Can anybody help me?

Here's one example:

def char_OneHotEncoding(x):
    coded = torch.zeros(x.shape[0], x.shape[1], 101)
    for i in range(x.shape[1]):
        coded[:,i] = scatter(x[:,i])
    return coded

def scatter(x):
    return torch.zeros(x.shape[0], 101).scatter_(1, x.view(-1,1), 1)

So if I give it an tensor on GPU, it shows like this:

x_train = [[ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [ 0,  0,  0,  0,  0],
       [14, 13, 83, 18, 14],
       [ 0,  0,  0,  0,  0]]
print(char_OneHotEncoding(torch.tensor(x_train, dtype=torch.long).cuda()).shape)

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-62-95c0c4ade406> in <module>()
      4        [14, 13, 83, 18, 14],
      5        [ 0,  0,  0,  0,  0]]
----> 6 print(char_OneHotEncoding(torch.tensor(x_train, dtype=torch.long).cuda()).shape)
      7 x_train[:5, maxlen:maxlen+5]

<ipython-input-53-055f1bf71306> in char_OneHotEncoding(x)
      2     coded = torch.zeros(x.shape[0], x.shape[1], 101)
      3     for i in range(x.shape[1]):
----> 4         coded[:,i] = scatter(x[:,i])
      5     return coded
      6

<ipython-input-53-055f1bf71306> in scatter(x)
      7
      8 def scatter(x):
----> 9     return torch.zeros(x.shape[0], 101).scatter_(1, x.view(-1,1), 1)

RuntimeError: Expected object of backend CPU but got backend CUDA for argument #3 'index'

BTW, if we simply remove the .cuda() here, everything goes one well

print(char_OneHotEncoding(torch.tensor(x_train, dtype=torch.long)).shape)

torch.Size([5, 5, 101])


title: Pytorch cuda get_device_name and current_device() hang and are killed?

body:
I've just installed a new GPU (RTX 2070) in my machine alongside the old GPU. I wanted to see if PyTorch picked up it, so following the instructions here: How to check if pytorch is using the GPU?, I ran the following commands (Python3.6.9, Linux Mint Tricia 19.3)

>>> import torch
>>> torch.cuda.is_available()
True
>>> torch.cuda.current_device()
Killed
>>> torch.cuda.get_device_name(0)
Killed

Both of the two Killed processes took some time and one of them froze the machine for half a minute or so. Does anyone have any experience with this? Are there some setup steps I'm missing?


title: plt.imshow() gives TypeError: Image data of dtype object cannot be converted to float using PyTorch

body:
I am trying to create a custom Dataset Processor for a set of images. However, when I try to view the images in my dataset, I get hit with the TypeError: Image data of dtype object cannot be converted to float.

I tried to check if I am passing in a PIL Image into the plt.imshow() function and I am.

class DatasetProcessing(Dataset):

    def __init__(self, input_data, output_data, transform=None):
        self.transform = transform
        self.input_data =
        input_data.reshape((-1,64,64)).astype(np.float32)[:,:,:,None]
        self.output_data = output_data

    def __getitem__(self, index):
        return self.transform(self.input_data[index]), self.output_data[index]

    def __len__(self):
        return len(list(self.input_data))

transform = transforms.Compose([transforms.ToPILImage()])

dset_train = DatasetProcessing(X_slices_train, Y_train, transform)

train_loader = torch.utils.data.DataLoader(dset_train, batch_size=4,
                                      shuffle=True, num_workers=4)

plt.figure(figsize = (16, 4))
for num, x in enumerate(dset_train):
    plt.subplot(1,6,num+1)
    plt.axis('off')
    print(x)
    plt.imshow(np.asarray(x))
    plt.title(y_train[num])

I expected to get pictures of my dataset, but instead I get the following error message:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-10-8b8caac49d97> in <module>
      4     plt.axis('off')
      5     print(x)
----> 6     plt.imshow(np.asarray(x))
      7     plt.title(y_train[num])

~/anaconda3/lib/python3.7/site-packages/matplotlib/pyplot.py in imshow(X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, data, **kwargs)
   2675         filternorm=filternorm, filterrad=filterrad, imlim=imlim,
   2676         resample=resample, url=url, **({"data": data} if data is not
-> 2677         None else {}), **kwargs)
   2678     sci(__ret)
   2679     return __ret

~/anaconda3/lib/python3.7/site-packages/matplotlib/__init__.py in inner(ax, data, *args, **kwargs)
   1587     def inner(ax, *args, data=None, **kwargs):
   1588         if data is None:
-> 1589             return func(ax, *map(sanitize_sequence, args), **kwargs)
   1590
   1591         bound = new_sig.bind(ax, *args, **kwargs)

~/anaconda3/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py in wrapper(*args, **kwargs)
    367                 f"%(removal)s.  If any parameter follows {name!r}, they "
    368                 f"should be pass as keyword, not positionally.")
--> 369         return func(*args, **kwargs)
    370
    371     return wrapper

~/anaconda3/lib/python3.7/site-packages/matplotlib/cbook/deprecation.py in wrapper(*args, **kwargs)
    367                 f"%(removal)s.  If any parameter follows {name!r}, they "
    368                 f"should be pass as keyword, not positionally.")
--> 369         return func(*args, **kwargs)
    370
    371     return wrapper

~/anaconda3/lib/python3.7/site-packages/matplotlib/axes/_axes.py in imshow(self, X, cmap, norm, aspect, interpolation, alpha, vmin, vmax, origin, extent, shape, filternorm, filterrad, imlim, resample, url, **kwargs)
   5658                               resample=resample, **kwargs)
   5659
-> 5660         im.set_data(X)
   5661         im.set_alpha(alpha)
   5662         if im.get_clip_path() is None:

~/anaconda3/lib/python3.7/site-packages/matplotlib/image.py in set_data(self, A)
    676                 not np.can_cast(self._A.dtype, float, "same_kind")):
    677             raise TypeError("Image data of dtype {} cannot be converted to "
--> 678                             "float".format(self._A.dtype))
    679
    680         if not (self._A.ndim == 2

TypeError: Image data of dtype object cannot be converted to float


title: PyTorch Lightning move tensor to correct device in validation_epoch_end

body:
I would like to create a new tensor in a validation_epoch_end method of a LightningModule. From the official docs (page 48) it is stated that we should avoid direct .cuda() or .to(device) calls:

There are no .cuda() or .to() calls. . . Lightning does these for you.

and we are encouraged to use type_as method to transfer to the correct device.
new_x = new_x.type_as(x.type())
However, in a step validation_epoch_end I do not have any tensor to copy device from(by type_as method) in a clean way.
My question is what should I do if I want to create a new tensor in this method and transfer it to the device where is the model?
The only thing I can think of is to find a tensor in the outputs dictionary but it feels kinda messy:
avg_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
output = self(self.__test_input.type_as(avg_loss))

Is there any clean way to achieve that?


title: RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2

body:
I have a PyTorch LSTM model and my forward function looks like:

    def forward(self, x, hidden):
        print('in forward', x.dtype, hidden[0].dtype, hidden[1].dtype)
        lstm_out, hidden = self.lstm(x, hidden)
        return lstm_out, hidden

All of the print statements show torch.float64, which I believe is a double. So then why am I getting this issue?

I've cast to double in all of the relevant places already.


title: Understanding the softmax output layer of RNN

body:
Here's a simple LSTM model in Keras:

input = Input(shape=(max_len,))
model = Embedding(input_dim=input_dim, output_dim=embed_dim, input_length=max_len)(input)
model = Dropout(0.1)(model)
model = Bidirectional(LSTM(units=blstm_dim, return_sequences=True, recurrent_dropout=0.1))(model)
out =Dense(label_dim, activation="softmax")(model)

Here's my attempt to translate it to a Pytorch model:

class RNN(nn.Module):
    def __init__(self, input_dim, embed_dim, blstm_dim, label_dim):
        super(RNN, self).__init__()
        self.embed = nn.Embedding(input_dim, embed_dim)
        self.blstm = nn.LSTM(embed_dim, blstm_dim,  bidirectional=True, batch_first=True)
        self.fc = nn.Linear(2*blstm_dim, label_dim)

    def forward(self, x):
        h0 = torch.zeros(2, x.size(0), blstm_dim).to(device)
        c0 = torch.zeros(2, x.size(0), blstm_dim).to(device)
        x = self.embed(x)
        x = F.dropout(x, p=0.1, training=self.training)
        x,_ = self.blstm(x, (h0, c0))
        x = self.fc(x)
        return F.softmax(x, dim=1)
#        return x

Now running the Keras model gives the following:

Epoch 5/5
38846/38846 [==============================] - 87s 2ms/step - loss: 0.0374 - acc: 0.9889 - val_loss: 0.0473 - val_acc: 0.9859

But running the PyTorch model gives this:

Train Epoch: 10/10 [6400/34532 (19%)]   Loss: 2.788933
Train Epoch: 10/10 [12800/34532 (37%)]  Loss: 2.788880
Train Epoch: 10/10 [19200/34532 (56%)]  Loss: 2.785547
Train Epoch: 10/10 [25600/34532 (74%)]  Loss: 2.796180
Train Epoch: 10/10 [32000/34532 (93%)]  Loss: 2.790446
Validation: Average loss: 0.0437, Accuracy: 308281/431600 (71%)

I've made sure the loss & optimiser are the same (cross entropy & RMSprop). Now interestingly if I remove the softmax from the PyTorch model (i.e. use the hashed output in the code, I get what seems to be right:

Train Epoch: 10/10 [32000/34532 (93%)]  Loss: 0.022118
Validation: Average loss: 0.0009, Accuracy: 424974/431600 (98%)

So here are my questions:

1) Are the two models I've printed above equivalent (let's ignore the recurrent_dropout since I haven't figure out how to do that in PyTorch)?

2) What am I doing wrong with the softmax output layer in PyTorch?

Thanks a lot!


title: How to fix "RuntimeError: Function AddBackward0 returned an invalid gradient at index 1 - expected type torch.FloatTensor but got torch.LongTensor"

body:
I encountered this bug when I am running the machine translation code.

  RuntimeError                              Traceback (most recent call
  last)  in
        5 decoder = Decoder(len(out_vocab), embed_size, num_hiddens, num_layers,
        6                   attention_size, drop_prob)
  ----> 7 train(encoder, decoder, dataset, lr, batch_size, num_epochs)

   in train(encoder, decoder, dataset,
  lr, batch_size, num_epochs)
       13             dec_optimizer.zero_grad()
       14             l = batch_loss(encoder, decoder, X, Y, loss)
  ---> 15             l.backward()
       16             enc_optimizer.step()
       17             dec_optimizer.step()

  /usr/lib64/python3.6/site-packages/torch/tensor.py in backward(self,
  gradient, retain_graph, create_graph)
      105                 products. Defaults to False.
      106         """
  --> 107         torch.autograd.backward(self, gradient, retain_graph, create_graph)
      108
      109     def register_hook(self, hook):

  /usr/lib64/python3.6/site-packages/torch/autograd/init.py in
  backward(tensors, grad_tensors, retain_graph, create_graph,
  grad_variables)
       91     Variable._execution_engine.run_backward(
       92         tensors, grad_tensors, retain_graph, create_graph,
  ---> 93         allow_unreachable=True)  # allow_unreachable flag
       94
       95

  RuntimeError: Function AddBackward0 returned an invalid gradient at
  index 1 - expected type torch.FloatTensor but got torch.LongTensor

I think the bug is in the batch_loss function. But I don't know why and can't fix it.

def batch_loss(encoder, decoder, X, Y, loss):
    batch_size = X.shape[0]
    enc_state = None
    enc_outputs, enc_state = encoder(X, enc_state)
    # 初始化解码器的隐藏状态
    dec_state = decoder.begin_state(enc_state)
    # 解码器在最初时间步的输入是BOS
    dec_input = torch.tensor([out_vocab.stoi[BOS]] * batch_size)
    # 我们将使用掩码变量mask来忽略掉标签为填充项PAD的损失
    mask, num_not_pad_tokens = torch.ones(batch_size), 0
    l = torch.tensor([0])
    for y in Y.t():
        dec_output, dec_state = decoder(dec_input, dec_state, enc_outputs)
        l = l + (mask * loss(dec_output, y)).sum()
        dec_input = y  # 使用强制教学
        num_not_pad_tokens += mask.sum().item()
        # 当遇到EOS时，序列后面的词将均为PAD，相应位置的掩码设成0
        mask = mask * (y != out_vocab.stoi[EOS]).float()
    return l / num_not_pad_tokens

def train(encoder, decoder, dataset, lr, batch_size, num_epochs):
    d2lt.params_init(encoder, init=nn.init.xavier_uniform_)
    d2lt.params_init(decoder, init=nn.init.xavier_uniform_)

    enc_optimizer = optim.Adam(encoder.parameters(), lr=lr)
    dec_optimizer = optim.Adam(decoder.parameters(), lr=lr)
    loss = nn.CrossEntropyLoss(reduction='none')
    data_iter = tdata.DataLoader(dataset, batch_size, shuffle=True)
    for epoch in range(num_epochs):
        l_sum = 0.0
        for X, Y in data_iter:
            enc_optimizer.zero_grad()
            dec_optimizer.zero_grad()
            l = batch_loss(encoder, decoder, X, Y, loss)
            l.backward()
            enc_optimizer.step()
            dec_optimizer.step()
            l_sum += l.item()
        if (epoch + 1) % 10 == 0:
            print("epoch %d, loss %.3f" % (epoch + 1, l_sum / len(data_iter)))

Looking forward to a positive reply.


title: Pytorch: can we use nn.Module layers directly in forward() function?

body:
Generally,
In the constructor, we declare all the layers we want to use.
In the forward function, we define how the model is going to be run, from input to output.

My question is what if calling those predefined/built-in nn.Modules directly in forward() function? Is this Keras function API style legal for Pytorch? If not, why?

Update: TestModel constructed in this way did run successfully, without an alarm. But the training loss will descend slowly compared with the conventional way.

import torch.nn as nn
from cnn import CNN

class TestModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.num_embeddings = 2020
        self.embedding_dim = 51

    def forward(self, input):
        x = nn.Embedding(self.num_embeddings, self.embedding_dim)(input)
        # CNN is a customized class and nn.Module subclassed
        # we will ignore the arguments for its instantiation
        x = CNN(...)(x)
        x = nn.ReLu()(x)
        x = nn.Dropout(p=0.2)(x)
        return output = x


I am fairly new to machine learning. I learned to write this code from youtube tutorials but I keep getting this error

Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/aniket/Desktop/DeepLearning/PythonLearningPyCharm/CatVsDogs.py", line 109, in <module>
    optimizer = optim.Adam(net.parameters(), lr=0.001) # tweaks the weights from what I understand
AttributeError: 'Net' object has no attribute 'parameters'

this is the Net class

class Net():
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1,32,5)
        self.conv2 = nn.Conv2d(32,64,5)
        self.conv3 = nn.Conv2d(64,128,5)
        self.to_linear = None
        x = torch.randn(50,50).view(-1,1,50,50)
        self.Conv2d_Linear_Link(x)
        self.fc1 = nn.Linear(self.to_linear, 512)
        self.fc2 = nn.Linear(512, 2)

    def Conv2d_Linear_Link(self , x):
        x = F.max_pool2d(F.relu(self.conv1(x)),(2,2))
        x = F.max_pool2d(F.relu(self.conv2(x)),(2,2))
        x = F.max_pool2d(F.relu(self.conv3(x)),(2,2))

        if self.to_linear is None :
            self.to_linear = x[0].shape[0]*x[0].shape[1]*x[0].shape[2]
        return x

    def forward(self, x):
        x = self.Conv2d_Linear_Link(x)
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return F.softmax(x, dim=1)

and this is the function train

def train():
    for epoch in range(epochs):
        for i in tqdm(range(0,len(X_train), batch)):
            batch_x = train_X[i:i + batch].view(-1, 1, 50, 50)
            batch_y = train_y[i:i + batch]
            net.zero_grad() # i don't understand why we do this but we do we don't want the probabilites adding up
            output = net(batch_x)
            loss = loss_function(output, batch_y)
            loss.backward()
            optimizer.step()
        print(loss)

and the optimizer and loss functions and data

optimizer = optim.Adam(net.parameters(), lr=0.001) # tweaks the weights from what I understand
loss_function = nn.MSELoss() # gives the loss


title: huggingface bert showing poor accuracy / f1 score [pytorch]

body:
I am trying BertForSequenceClassification for a simple article classification task.

No matter how I train it (freeze all layers but the classification layer, all layers trainable, last k layers trainable), I always get an almost randomized accuracy score. My model doesn't go above 24-26% training accuracy (I only have 5 classes in my dataset).

I'm not sure what did I do wrong while designing/training the model. I tried the model with multiple datasets, every time it gives the same random baseline accuracy.

Dataset I used: BBC Articles (5 classes)

https://github.com/zabir-nabil/pytorch-nlp/tree/master/bbc

  Consists of 2225 documents from the BBC news website corresponding to
  stories in five topical areas from 2004-2005. Natural Classes: 5
  (business, entertainment, politics, sport, tech)

I added the model part and the training part which are the most important portion (to avoid any irrelevant details). I added the full source-code + data too if that's useful for reproducibility.

My guess is there is something wrong with the I way I designed the network or the way I'm passing the attention_masks/ labels to the model. Also, the token length 512 should not be a problem as most of the texts has length < 512 (the mean length is < 300).

Model code:

import torch
from torch import nn

class BertClassifier(nn.Module):
    def __init__(self):
        super(BertClassifier, self).__init__()
        self.bert = BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels = 5)
        # as we have 5 classes

        # we want our output as probability so, in the evaluation mode, we'll pass the logits to a softmax layer
        self.softmax = torch.nn.Softmax(dim = 1) # last dimension
    def forward(self, x, attn_mask = None, labels = None):

        if self.training == True:
            # print(x.shape)
            loss = self.bert(x, attention_mask = attn_mask, labels = labels)
            # print(x[0].shape)

            return loss

        if self.training == False: # in evaluation mode
            x = self.bert(x)
            x = self.softmax(x[0])

            return x
    def freeze_layers(self, last_trainable = 1):
        # we freeze all the layers except the last classification layer + few transformer blocks
        for layer in list(self.bert.parameters())[:-last_trainable]:
            layer.requires_grad = False

# create our model

bertclassifier = BertClassifier()

Training code:

device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # cuda for gpu acceleration

# optimizer

optimizer = torch.optim.Adam(bertclassifier.parameters(), lr=0.001)

epochs = 15

bertclassifier.to(device) # taking the model to GPU if possible

# metrics

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

train_losses = []

train_metrics = {'acc': [], 'f1': []}
test_metrics = {'acc': [], 'f1': []}

# progress bar

from tqdm import tqdm_notebook

for e in tqdm_notebook(range(epochs)):
    train_loss = 0.0
    train_acc = 0.0
    train_f1 = 0.0
    batch_cnt = 0

    bertclassifier.train()

    print(f'epoch: {e+1}')

    for i_batch, (X, X_mask, y) in tqdm_notebook(enumerate(bbc_dataloader_train)):
        X = X.to(device)
        X_mask = X_mask.to(device)
        y = y.to(device)

        optimizer.zero_grad()

        loss, y_pred = bertclassifier(X, X_mask, y)

        train_loss += loss.item()
        loss.backward()
        optimizer.step()

        y_pred = torch.argmax(y_pred, dim = -1)

        # update metrics
        train_acc += accuracy_score(y.cpu().detach().numpy(), y_pred.cpu().detach().numpy())
        train_f1 += f1_score(y.cpu().detach().numpy(), y_pred.cpu().detach().numpy(), average = 'micro')
        batch_cnt += 1

    print(f'train loss: {train_loss/batch_cnt}')
    train_losses.append(train_loss/batch_cnt)
    train_metrics['acc'].append(train_acc/batch_cnt)
    train_metrics['f1'].append(train_f1/batch_cnt)

    test_loss = 0.0
    test_acc = 0.0
    test_f1 = 0.0
    batch_cnt = 0

    bertclassifier.eval()
    with torch.no_grad():
        for i_batch, (X, y) in enumerate(bbc_dataloader_test):
            X = X.to(device)
            y = y.to(device)

            y_pred = bertclassifier(X) # in eval model we get the softmax output so, don't need to index

            y_pred = torch.argmax(y_pred, dim = -1)

            # update metrics
            test_acc += accuracy_score(y.cpu().detach().numpy(), y_pred.cpu().detach().numpy())
            test_f1 += f1_score(y.cpu().detach().numpy(), y_pred.cpu().detach().numpy(), average = 'micro')
            batch_cnt += 1

    test_metrics['acc'].append(test_acc/batch_cnt)
    test_metrics['f1'].append(test_f1/batch_cnt)

Full source-code with the dataset is available here: https://github.com/zabir-nabil/pytorch-nlp/blob/master/bert-article-classification.ipynb

Update:

After observing the prediction, it seems model almost always predicts 0:

bertclassifier.eval()
with torch.no_grad():
    for i_batch, (X, y) in enumerate(bbc_dataloader_test):
        X = X.to(device)
        y = y.to(device)

        y_pred = bertclassifier(X) # in eval model we get the softmax output so, don't need to index

        y_pred = torch.argmax(y_pred, dim = -1)

        print(y)
        print(y_pred)
        print('--------------------')

tensor([4, 2, 2, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 3, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 0, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 4, 4, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 3, 2, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 3, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 1, 4, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 0, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 1, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 2, 4, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 1, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 1, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 0, 1, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 3, 1, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 2, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 1, 2, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 4, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 0, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 3, 2, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 1, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 2, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 3, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 4, 2, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 4, 4, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 1, 3, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 2, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 0, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 1, 4, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 4, 3, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 2, 1, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 4, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 1, 1, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 2, 4, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 2, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 1, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 2, 2, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 2, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 3, 2, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 0, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 1, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 4, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 3, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 2, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 2, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 2, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 2, 3, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 3, 0, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 0, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 2, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 4, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 0, 4, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 2, 0, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 3, 1, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 1, 3, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 3, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 3, 0, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 2, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 0, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 0, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 1, 1, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 1, 0, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 4, 1, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 2, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 3, 4, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([3, 0, 4, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 1, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 4, 3, 1], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 3, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 3, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 0, 3, 4], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 1, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([1, 2, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([2, 0, 4, 2], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([4, 2, 4, 0], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
tensor([0, 0, 3, 3], device='cuda:0')
tensor([0, 0, 0, 0], device='cuda:0')
--------------------
...
...

Actually, the model is always predicting the same output [0.2270, 0.1855, 0.2131, 0.1877, 0.1867] for any input, it's like it didn't learn anything at all.

It's weird because my dataset is not imbalanced.

Counter({'politics': 417,
         'business': 510,
         'entertainment': 386,
         'tech': 401,
         'sport': 511})


title: PyTorch's Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same but my data has been pushed to GPU

body:
I receive the error:

RuntimeError: Input type (torch.FloatTensor) and weight type (torch.cuda.FloatTensor) should be the same

yet I made sure to send my data as well as my model on the GPU. Anyone can help ?

My code is:

net.cuda()

'''
print('pyTorch style  summay: ',net)
print('Keras style summary:\n')
summary(net,(2,128,128))
'''

criterion=nn.MSELoss()
#optimizer = optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9)
learning_rate = 1e-4
optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)
print('\nLossFun=',str(criterion))

hf=h5py.File(fn,'r')
print(hf['trainingset'])
tr=np.array(hf['trainingset'])
trtg=np.array(hf['targetsTraining'])
hf.close()

tr = np.moveaxis(tr,3,2)
trtg = np.moveaxis(trtg,3,2)

tr = torch.FloatTensor(tr)
trtg = torch.FloatTensor(trtg)

tr.cuda()
trtg.cuda()

batch_size=16
epochs=2
# run the main training loop
for epoch in range(epochs):
    for batch_idx in range(batch_size):#batch_idx, (data, target) in enumerate(train_loader):
        data =  tr[batch_idx:batch_idx+batch_size-1,:,:,:]
        target =  trtg[batch_idx:batch_idx+batch_size-1,:,:,:]
        data, target = Variable(data), Variable(target)
        # resize data from (batch_size, 1, 28, 28) to (batch_size, 28*28)
        #data = data.view(-1, 28*28)
        optimizer.zero_grad()
        net_out = net(data)
        loss = criterion(net_out, target)
        loss.backward()
        optimizer.step()
        batch_idx += 1
        if batch_idx % log_interval == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(epoch, batch_idx * len(data), len(train_loader.dataset),100. * batch_idx / len(train_loader), loss.data[0]))

I don't understand why tr.cuda() and trtg.cuda() is not enough! How can I force them to go to cuda?


title: torch.optim returns "ValueError: can't optimize a non-leaf Tensor" for multidimensional tensor

body:
I am trying to optimize the translations of the vertices of a scene with torch.optim.adam. It is a code piece from the redner tutorial series, which works fine with the initial setting. It tries to optimize a scene with shifting all the vertices by the same value called translation. Here is the original code:

vertices = []
for obj in base:
    vertices.append(obj.vertices.clone())

def model(translation):
    for obj, v in zip(base, vertices):
        obj.vertices = v + translation
    # Assemble the 3D scene.
    scene = pyredner.Scene(camera = camera, objects = objects)
    # Render the scene.
    img = pyredner.render_albedo(scene)
    return img

# Initial guess
# Set requires_grad=True since we want to optimize them later

translation = torch.tensor([10.0, -10.0, 10.0], device = pyredner.get_device(), requires_grad=True)

init = model(translation)
# Visualize the initial guess

t_optimizer = torch.optim.Adam([translation], lr=0.5)

I tried to modify the code such that it calculates an individual translation for each of the  vertices. For this I applied the following modifications to the code above, that makes the shape of the translation from torch.Size([3]) to torch.Size([43380, 3]):

# translation = torch.tensor([10.0, -10.0, 10.0], device = pyredner.get_device(), requires_grad=True)
translation = base[0].vertices.clone().detach().requires_grad_(True)
translation[:] = 10.0

This introduces the ValueError: can't optimize a non-leaf Tensor. Could you please help me work around the problem.

PS: I am sorry for the long text, I am very new to this subject, and I wanted to state the problem as comprehensive as possible.


title: PyTorch model gives IndexError: index 0 is out of bounds for dimension 0 with size 0

body:
I followed the following tutorial on the pytorch website:
https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html

I have trained my model on an eye database to detect the pupil. I save the trained model and open it to evaluate.
The problem is that if there is no eye in the picture the model will crash.

To reproduce I first printed the model on a working picture:

 [{'boxes': tensor([[126.9488, 112.5852, 141.5698, 127.4215]], device='cuda:0'), 'labels': tensor([1], device='cuda:0'), 'scores': tensor([1.0000], device='cuda:0'), 'masks': tensor([[[[0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          ...,
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.],
          [0., 0., 0.,  ..., 0., 0., 0.]]]], device='cuda:0')}]

Then on a picture where the eye is closed: (so nothing is detected)

[{'boxes': tensor([], device='cuda:0', size=(0, 4)), 'labels': tensor([], device='cuda:0', dtype=torch.int64), 'scores': tensor([], device='cuda:0'), 'masks': tensor([], device='cuda:0', size=(0, 1, 255, 255))}]

The last one should look like the first one.

The error I get is this:

img1 = Image.fromarray(prediction[0]['masks'][0, 0].mul(255).byte().cpu().numpy())
IndexError: index 0 is out of bounds for dimension 0 with size 0

I get that I'm trying to reach somewhere outside the list but I can't really initiate a list of zeros of  prediction because it is a loaded model. The model is a .pt file

If you need something else please tell me

Kind regards


title: Equivalence of slicing tensor in Pytorch/ATen C++

body:
In Python given a 2-D tensor, we can use tensor[:,:2] to slice the a 2x2 matrix of the first two elements in the top left of the matrix, e.g. :

x = torch.tensor([[-1.4673,  0.9980, -2.1427, -1.1798, -0.0646, -0.2635, -2.8930, -0.2563,
          0.4559, -0.7947, -0.4540,  3.3224,  0.2295,  5.5568, -8.0451, -2.4529,
          4.8724,  2.1640,  3.3255,  0.6693, -1.2362,  4.4713, -3.5547, -0.0528,
          0.1031, -1.2472, -1.6014,  1.8134],
        [ 2.1636, -1.1497, -5.0298,  2.8261, -0.5684,  0.6389,  2.9009, -5.1609,
          1.7358, -3.1819, -0.9877,  5.5598,  6.7142,  4.5704, -1.2683, -5.3046,
          3.0454,  3.2757, -3.2541,  3.6619, -3.6391, -0.2002,  5.7175,  5.7130,
          0.6632, -0.0744, -0.3502,  4.8116]])

y, z = x[:,:2].chunk(2,1)

print(y)

print(z)

[out]:

tensor([[-1.4673],
        [ 2.1636]])
tensor([[ 0.9980],
        [-1.1497]])

What is right way to do it in C++ for PyTorch's ATen particularly?

For e.g. in the LSTM, there is the gate.chunk(4,1) function at https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/RNN.cpp#L253

If I want to do a gate[:,:2].chunk(2,1) to extract different parts of the gates, e.g. auto partial_gates = gates[:,:2].chunk(4, 1);, how can it be done?

template <typename cell_params>
struct LSTMCell : Cell<std::tuple<Tensor, Tensor>, cell_params> {
  using hidden_type = std::tuple<Tensor, Tensor>;
  hidden_type operator()(const Tensor& input, const hidden_type& hidden, const cell_params& params) const override {
    auto hx = std::get<0>(hidden);
    auto cx = std::get<1>(hidden);

    if (input.is_cuda()) {
      auto igates = params.matmul_ih(input);
      auto hgates = params.matmul_hh(hx);
      auto result = at::_thnn_fused_lstm_cell(igates, hgates, cx, params.b_ih, params.b_hh);
      // Slice off the workspace argument (it's needed only for AD).
      return std::make_tuple(std::get<0>(result), std::get<1>(result));
    }

    auto gates = params.linear_ih(input) + params.linear_hh(hx);
    auto chunked_gates = gates.chunk(4, 1);

    auto partial_gates = gates[:,:2].chunk(4, 1);

    auto ingate = chunked_gates[0].sigmoid();
    auto forgetgate = chunked_gates[1].sigmoid();
    auto cellgate = chunked_gates[2].tanh();
    auto outgate = chunked_gates[3].sigmoid();

    auto cy = (forgetgate * cx) + (ingate * cellgate);
    auto hy = outgate * cy.tanh();

    return std::make_tuple(hy, cy);
  }
};


title: Regarding odd image dimensions in Pytorch

body:
So I am currently building a 2-channel (also called double channel) convolution neural network for measuring the similarity between 2 (binary) images.

The problem I am having is the following:

My input images are 40 x 50, and after 1 conv and 1 pooling layer (for example), the output size is 18 x 23. So how does one do more pooling without ending up with non-integer output sizes? For example, pooling a 18 x 23 image with size 2 x 2, the output size is given by 9 x 11.5.

I cannot seem to find any suitable kernel sizes to avoid such a problem, which in my opinion is a result of the fact that the original input image dimensions are not powers of 2. For example, input images of size 64 x 64 doesn't have this issue with the correct padding size and so on.

Any help is much appreciated.


title: Spyder just "kernel died,restarting" when I run net.forward in pytorch

body:
I am using PYTHON IDE spyder3.2.1 in anaconda2, with python2.7, ubuntu14.04

code is just simple as follows:

import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

"""
input:[batch_size,in_channel,height,width]
kernel:[out_channel,in_channel,kh,kw]
"""
class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        # 1 input image channel, 6 output channels, 5x5 square convolution
        # kernel
        self.conv1 = nn.Conv2d(1, 6, 5)
        #(28-5+1)/2=12
        self.conv2 = nn.Conv2d(6, 16, 5)
        #(12-5+1)/2=4
        # an affine operation: y = Wx + b
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        # Max pooling over a (2, 2) window
        x = F.max_pool2d(F.relu(self.conv1(x)), (2, 2))
        # If the size is a square you can only specify a single number
        print "after conv1 size is {}".format(x.size())
        x = F.max_pool2d(F.relu(self.conv2(x)), 2)
        print "after conv2 size is {}".format(x.size())
        x = x.view(-1, self.num_flat_features(x))
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]  # all dimensions except the batch dimension
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

net = Net()
print(net)

print "hello wrold"
input = Variable(torch.Tensor(np.random.randint(1,10,size=(1,1,32,32))))
print net.forward(input)

when I use jupyter notebook or console, it runs normally, it means the code has no error.
But when I use it from Spyder, it just hangs likes this:

As I am not allowed to embed the error image: it says "Kernel died, restarting"

I just try to shrink my code to find where the problem is.

The problem is absolutely around the very start F.max_pool2d(), in the forward method function.

But after I searched a lot, I still have no idea how to fix it.

conda list:

alabaster                 0.7.10                   py27_0
anaconda                  4.4.0               np112py27_0
anaconda-client           1.6.3                    py27_0
anaconda-navigator        1.6.2                    py27_0
anaconda-project          0.6.0                    py27_0
asn1crypto                0.22.0                   py27_0
astroid                   1.4.9                    py27_0
astroid                   1.5.3                     <pip>
astropy                   1.3.2               np112py27_0
Babel                     2.5.0                     <pip>
babel                     2.4.0                    py27_0
backports                 1.0                      py27_0
backports.functools-lru-cache 1.4                       <pip>
backports.weakref         1.0rc1                    <pip>
backports_abc             0.5                      py27_0
beautifulsoup4            4.6.0                    py27_0
bitarray                  0.8.1                    py27_0
blaze                     0.10.1                   py27_0
bleach                    1.5.0                    py27_0
bleach                    2.0.0                     <pip>
bokeh                     0.12.5                   py27_1
boto                      2.46.1                   py27_0
bottleneck                1.2.1               np112py27_0
cairo                     1.14.8                        0
cdecimal                  2.3                      py27_2
certifi                   2017.7.27.1               <pip>
cffi                      1.10.0                   py27_0
chardet                   3.0.3                    py27_0
chardet                   3.0.4                     <pip>
click                     6.7                      py27_0
cloudpickle               0.2.2                    py27_0
clyent                    1.2.2                    py27_0
colorama                  0.3.9                    py27_0
conda                     4.3.21                   py27_0
conda-env                 2.6.0                         0
configparser              3.5.0                    py27_0
contextlib2               0.5.5                    py27_0
cryptography              1.8.1                    py27_0
curl                      7.52.1                        0
cycler                    0.10.0                   py27_0
cython                    0.25.2                   py27_0
cytoolz                   0.8.2                    py27_0
dask                      0.14.3                   py27_1
datashape                 0.5.4                    py27_0
dbus                      1.10.10                       0
decorator                 4.0.11                   py27_0
distributed               1.16.3                   py27_0
docutils                  0.14                      <pip>
docutils                  0.13.1                   py27_0
entrypoints               0.2.2                    py27_1
entrypoints               0.2.3                     <pip>
enum34                    1.1.6                    py27_0
et_xmlfile                1.0.1                    py27_0
expat                     2.1.0                         0
fastcache                 1.0.2                    py27_1
flask                     0.12.2                   py27_0
flask-cors                3.0.2                    py27_0
fontconfig                2.12.1                        3
freetype                  2.5.5                         2
funcsigs                  1.0.2                    py27_0
functools32               3.2.3.2                  py27_0
futures                   3.1.1                    py27_0
get_terminal_size         1.0.0                    py27_0
gevent                    1.2.1                    py27_0
glib                      2.50.2                        1
greenlet                  0.4.12                   py27_0
grin                      1.2.1                    py27_3
gst-plugins-base          1.8.0                         0
gstreamer                 1.8.0                         0
h5py                      2.7.0               np112py27_0
harfbuzz                  0.9.39                        2
hdf5                      1.8.17                        1
heapdict                  1.0.0                    py27_1
html5lib                  0.999999999               <pip>
html5lib                  0.999                    py27_0
icu                       54.1                          0
idna                      2.5                      py27_0
idna                      2.6                       <pip>
imagesize                 0.7.1                    py27_0
ipaddress                 1.0.18                   py27_0
ipykernel                 4.6.1                    py27_0
ipython                   5.3.0                    py27_0
ipython_genutils          0.2.0                    py27_0
ipywidgets                6.0.0                    py27_0
isort                     4.2.5                    py27_0
isort                     4.2.15                    <pip>
itsdangerous              0.24                     py27_0
jbig                      2.1                           0
jdcal                     1.3                      py27_0
jedi                      0.10.2                   py27_2
jinja2                    2.9.6                    py27_0
jpeg                      9b                            0
jsonschema                2.6.0                    py27_0
jupyter                   1.0.0                    py27_3
jupyter-client            5.1.0                     <pip>
jupyter_client            5.0.1                    py27_0
jupyter_console           5.1.0                    py27_0
jupyter_core              4.3.0                    py27_0
lazy-object-proxy         1.2.2                    py27_0
lazy-object-proxy         1.3.1                     <pip>
libffi                    3.2.1                         1
libgcc                    4.8.5                         2
libgfortran               3.0.0                         1
libiconv                  1.14                          0
libpng                    1.6.27                        0
libsodium                 1.0.10                        0
libtiff                   4.0.6                         3
libtool                   2.4.2                         0
libxcb                    1.12                          1
libxml2                   2.9.4                         0
libxslt                   1.1.29                        0
llvmlite                  0.18.0                   py27_0
locket                    0.2.0                    py27_1
lxml                      3.7.3                    py27_0
Markdown                  2.6.8                     <pip>
markupsafe                0.23                     py27_2
MarkupSafe                1.0                       <pip>
matplotlib                2.0.2               np112py27_0
mccabe                    0.6.1                     <pip>
mistune                   0.7.4                    py27_0
mkl                       2017.0.1                      0
mkl-service               1.1.2                    py27_3
mock                      2.0.0                     <pip>
mpmath                    0.19                     py27_1
msgpack-python            0.4.8                    py27_0
multipledispatch          0.4.9                    py27_0
navigator-updater         0.1.0                    py27_0
nbconvert                 5.2.1                     <pip>
nbconvert                 5.1.1                    py27_0
nbformat                  4.4.0                     <pip>
nbformat                  4.3.0                    py27_0
networkx                  1.11                     py27_0
nltk                      3.2.3                    py27_0
nose                      1.3.7                    py27_1
notebook                  5.0.0                    py27_0
numba                     0.33.0              np112py27_0
numexpr                   2.6.2               np112py27_0
numpy                     1.12.1                   py27_0
numpy                     1.13.1                    <pip>
numpydoc                  0.6.0                    py27_0
numpydoc                  0.7.0                     <pip>
odo                       0.5.0                    py27_1
olefile                   0.44                     py27_0
opencv                    1.0.1                     <pip>
opencv-python             3.3.0.9                   <pip>
openpyxl                  2.4.7                    py27_0
openssl                   1.0.2l                        0
packaging                 16.8                     py27_0
pandas                    0.20.1              np112py27_0
pandocfilters             1.4.1                    py27_0
pandocfilters             1.4.2                     <pip>
pango                     1.40.3                        1
partd                     0.3.8                    py27_0
path.py                   10.3.1                   py27_0
pathlib2                  2.3.0                     <pip>
pathlib2                  2.2.1                    py27_0
patsy                     0.4.1                    py27_0
pbr                       3.1.1                     <pip>
pcre                      8.39                          1
pep8                      1.7.0                    py27_0
pexpect                   4.2.1                    py27_0
pickleshare               0.7.4                    py27_0
pillow                    4.1.1                    py27_0
pip                       9.0.1                    py27_1
pixman                    0.34.0                        0
ply                       3.10                     py27_0
prompt_toolkit            1.0.14                   py27_0
protobuf                  3.3.0                     <pip>
psutil                    5.2.2                    py27_0
ptyprocess                0.5.1                    py27_0
py                        1.4.33                   py27_0
pycairo                   1.10.0                   py27_0
pycodestyle               2.3.1                     <pip>
pycosat                   0.6.2                    py27_0
pycparser                 2.17                     py27_0
pycrypto                  2.6.1                    py27_6
pycurl                    7.43.0                   py27_2
pyflakes                  1.5.0                    py27_0
pyflakes                  1.6.0                     <pip>
pygments                  2.2.0                    py27_0
pylint                    1.6.4                    py27_1
pylint                    1.7.2                     <pip>
pylzma                    0.4.9.post0               <pip>
pyodbc                    4.0.16                   py27_0
PyOpenGL                  3.1.0                     <pip>
pyopenssl                 17.0.0                   py27_0
pyparsing                 2.1.4                    py27_0
pyqt                      5.6.0                    py27_2
pytables                  3.3.0               np112py27_0
pytest                    3.0.7                    py27_0
python                    2.7.13                        0
python-dateutil           2.6.1                     <pip>
python-dateutil           2.6.0                    py27_0
pytz                      2017.2                   py27_0
pywavelets                0.5.2               np112py27_0
pyyaml                    3.12                     py27_0
pyzmq                     16.0.2                   py27_0
qt                        5.6.2                         4
qtawesome                 0.4.4                    py27_0
qtconsole                 4.3.0                    py27_0
qtconsole                 4.3.1                     <pip>
qtpy                      1.2.1                    py27_0
QtPy                      1.3.1                     <pip>
readline                  6.2                           2
requests                  2.18.4                    <pip>
requests                  2.14.2                   py27_0
rope                      0.10.7                    <pip>
rope                      0.9.4                    py27_1
ruamel_yaml               0.11.14                  py27_1
scandir                   1.5                      py27_0
scikit-image              0.13.0              np112py27_0
scikit-learn              0.18.1              np112py27_1
scipy                     0.19.0              np112py27_0
seaborn                   0.7.1                    py27_0
setuptools                27.2.0                   py27_0
setuptools                36.2.7                    <pip>
simplegeneric             0.8.1                    py27_1
singledispatch            3.4.0.3                  py27_0
sip                       4.18                     py27_0
six                       1.10.0                   py27_0
snowballstemmer           1.2.1                    py27_0
sortedcollections         0.5.3                    py27_0
sortedcontainers          1.5.7                    py27_0
Sphinx                    1.6.3                     <pip>
sphinx                    1.5.6                    py27_0
sphinxcontrib-websupport  1.0.1                     <pip>
spyder                    3.2.1                     <pip>
spyder                    3.1.4                    py27_0
sqlalchemy                1.1.9                    py27_0
sqlite                    3.13.0                        0
ssl_match_hostname        3.4.0.2                  py27_1
statsmodels               0.8.0               np112py27_0
subprocess32              3.2.7                    py27_0
sympy                     1.0                      py27_0
tblib                     1.3.2                    py27_0
tensorflow                1.2.1                     <pip>
tensorlayer               1.6.0                     <pip>
terminado                 0.6                      py27_0
testpath                  0.3.1                     <pip>
testpath                  0.3                      py27_0
tk                        8.5.18                        0
toolz                     0.8.2                    py27_0
torch-0.2.0               1                         <pip>
torchvision               0.1.9                     <pip>
tornado                   4.5.1                    py27_0
tqdm                      4.15.0                    <pip>
traitlets                 4.3.2                    py27_0
typing                    3.6.2                     <pip>
unicodecsv                0.14.1                   py27_0
unixodbc                  2.3.4                         0
urllib3                   1.22                      <pip>
wcwidth                   0.1.7                    py27_0
webencodings              0.5.1                     <pip>
werkzeug                  0.12.2                   py27_0
wheel                     0.29.0                   py27_0
widgetsnbextension        2.0.0                    py27_0
wrapt                     1.10.11                   <pip>
wrapt                     1.10.10                  py27_0
xlrd                      1.0.0                    py27_0
xlsxwriter                0.9.6                    py27_0
xlwt                      1.2.0                    py27_0
xz                        5.2.2                         1
yaml                      0.1.6                         0
zeromq                    4.1.5                         0
zict                      0.1.2                    py27_0
zlib                      1.2.8                         3


title: PyTorch 2d Convolution with sparse filters

body:
I am trying to perform a spatial convolution (e.g. on an image) in pytorch on dense input using a sparse filter matrix.

Sparse Tensors are implemented in PyTorch. I tried to use a sparse Tensor, but it ends up with a segmentation fault.

import torch
from torch.autograd import Variable
from torch.nn import functional as F

# build sparse filter matrix
i = torch.LongTensor([[0, 1, 1],[2, 0, 2]])
v = torch.FloatTensor([3, 4, 5])
filter =  Variable(torch.sparse.FloatTensor(i, v, torch.Size([3,3])))

inputs = Variable(torch.randn(1,1,6,6))

F.conv2d(inputs, filter)

Can anyone just give me a hint how to do that?

Thanks in advance!
dymat


title: MultiLabel Soft Margin Loss in PyTorch

body:
I want to implement a classifier which can have 1 of 10 possible classes. I am trying to use the MultiClass Softmax Loss Function to do this. Going through the documentation I'm not clear with what input is required for the function.

The documentation says it needs two matrices of [N, C] of which one is input and the other is target. As much as I understand, input matrix would be the one my Neural Network would calculate, which would have probabilities given by the neural network to each of the 10 classes. The target is the one that I have from my dataset.

The documentation says - "Target(N, C) - label targets padded by -1 ensuring same shape as the input." What does this mean? Do I pass zeros in incorrect classes and -1 for the correct one?

It would be great if someone could elaborate on this and show even a sample 2d matrix which could be passed as a target matrix.


title: Pytorch LSTM: Target Dimension in Calculating Cross Entropy Loss

body:
I've been trying to get an LSTM (LSTM followed by a linear layer in a custom model), working in Pytorch, but was getting the following error when calculating the loss:

Assertion cur_target >= 0 && cur_target < n_classes' failed.

I defined the loss function with:

criterion = nn.CrossEntropyLoss()

and then called with

loss += criterion(output, target)

I was giving the target with dimensions [sequence_length, number_of_classes], and output has dimensions [sequence_length, 1, number_of_classes].

The examples I was following seemed to be doing the same thing, but it was different on the Pytorch docs on cross entropy loss.

The docs say the target should be of dimension (N), where each value is  0 ≤ targets[i] ≤ C−1 and C is the number of classes. I changed the target to be in that form, but now I'm getting an error saying (The sequence length is 75, and there are 55 classes):

Expected target size (75, 55), got torch.Size([75])

I've tried looking at solutions for both errors, but still can't get this working properly. I'm confused as to the proper dimensions of target, as well as the actual meaning behind the first error (different searches gave very different meanings for the error, none of the fixes worked).

Thanks


title: Accuracy for every epoch in Pytorch

body:
I'm struggling to calculate accuracy for every epoch in my training function for CNN classifier in Pytorch.

After I run this script, it always prints out 0, 0.25 or 0.75 which is obviously wrong. I'm guessing the problem are the inputs of the get_accuracy function (outputs and labels) as they are not accumulated for the entire epoch but not sure how to fix that.

Ideally, I'd like to print out both train and test accuracy for every epoch.

def get_accuracy(pred, actual):
  assert len(pred) == len(actual)

  total = len(actual)
  _, predicted = torch.max(pred.data, 1)
  correct = (predicted == actual).sum().item()
  return correct / total

def train_model(model):
    for epoch in range(epochs):
        running_loss = 0.00
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data
            inputs, labels = inputs.to(device), labels.to(device)

            optimizer.zero_grad()

            outputs = cnn(inputs)

            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            running_loss += loss.item() #calculates loss of the batch in average

        running_loss /= len(trainloader)

        training_accuracy = get_accuracy(outputs, labels)
        test_accuracy = 'todo'

        print('='*10,'iteration: {0}'.format(epoch+1),'='*10,)
        print('\n Loss: {0} \n Training accuracy:{1}% \n Test accuracy: {2}%'.format(running_loss, training_accuracy, test_accuracy))

    print('Finished Training')


title: How to install CUDA enabled PyTorch in a Docker container?

body:
I am trying to build a Docker container on a server within which a conda environment is built. All the other requirements are satisfied except for CUDA enabled PyTorch (I can get PyTorch working without CUDA however, no problem). How do I make sure PyTorch is using CUDA?
This is the Dockerfile :
# Use nvidia/cuda image
FROM nvidia/cuda:10.2-cudnn7-devel-ubuntu18.04

# set bash as current shell
RUN chsh -s /bin/bash

# install anaconda
RUN apt-get update
RUN apt-get install -y wget bzip2 ca-certificates libglib2.0-0 libxext6 libsm6 libxrender1 git mercurial subversion && \
        apt-get clean
RUN wget --quiet https://repo.anaconda.com/archive/Anaconda3-2020.02-Linux-x86_64.sh -O ~/anaconda.sh && \
        /bin/bash ~/anaconda.sh -b -p /opt/conda && \
        rm ~/anaconda.sh && \
        ln -s /opt/conda/etc/profile.d/conda.sh /etc/profile.d/conda.sh && \
        echo ". /opt/conda/etc/profile.d/conda.sh" >> ~/.bashrc && \
        find /opt/conda/ -follow -type f -name '*.a' -delete && \
        find /opt/conda/ -follow -type f -name '*.js.map' -delete && \
        /opt/conda/bin/conda clean -afy

# set path to conda
ENV PATH /opt/conda/bin:$PATH

# setup conda virtual environment
COPY ./requirements.yaml /tmp/requirements.yaml
RUN conda update conda \
    && conda env create --name camera-seg -f /tmp/requirements.yaml \
    && conda install -y -c conda-forge -n camera-seg flake8

# From the pythonspeed tutorial; Make RUN commands use the new environment
SHELL ["conda", "run", "-n", "camera-seg", "/bin/bash", "-c"]

# PyTorch with CUDA 10.2
RUN conda activate camera-seg && conda install pytorch torchvision cudatoolkit=10.2 -c pytorch

RUN echo "conda activate camera-seg" > ~/.bashrc
ENV PATH /opt/conda/envs/camera-seg/bin:$PATH

This gives me the following error when I try to build this container ( docker build -t camera-seg . ):
.....

Step 10/12 : RUN conda activate camera-seg && conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
 ---> Running in e0dd3e648f7b
ERROR conda.cli.main_run:execute(34): Subprocess for 'conda run ['/bin/bash', '-c', 'conda activate camera-seg && conda install pytorch torchvision cudatoolkit=10.2 -c pytorch']' command failed.  (See above for error)

CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

The command 'conda run -n camera-seg /bin/bash -c conda activate camera-seg && conda install pytorch torchvision cudatoolkit=10.2 -c pytorch' returned a non-zero code: 1

This is the requirements.yaml:
name: camera-seg
channels:
  - defaults
  - conda-forge
dependencies:
  - python=3.6
  - numpy
  - pillow
  - yaml
  - pyyaml
  - matplotlib
  - jupyter
  - notebook
  - tensorboardx
  - tensorboard
  - protobuf
  - tqdm

When I put pytorch, torchvision and cudatoolkit=10.2 within the requirements.yaml, then PyTorch is successfully installed but it cannot recognize CUDA ( torch.cuda.is_available() returns False ).
I have tried various solutions, for example, this, this and this and some different combinations of them but all to no avail.
Any help is much appreciated. Thanks.


title: Detectron2 - Extract region features at a threshold for object detection

body:
I am trying to extract region features where class detection is higher than some threshold using the detectron2 framework. I will be using these features later in my pipeline (similar to: VilBert section 3.1 Training ViLBERT) So far I have trained a Mask R-CNN with this config and fine-tuned it on some custom data. It performs well. What I would like to do is extract the features from my trained model for the produced bounding box.

EDIT: I looked at what the users who closed my post wrote and tried to refine it. Although the reader needs context as to what I am doing. If you have any idea on how I can make the question better or if you have some insight as to how to do what I am trying to do your feedback is welcome!

I have a question:

Why am I only getting one prediction instance, but when I look
at the prediction CLS scores there are more than 1 which passes the
threshold?

I believe this is the correct way of producing the ROI features:

images = ImageList.from_tensors(lst[:1], size_divisibility=32).to("cuda")  # preprocessed input tensor
#setup config
cfg = get_cfg()
cfg.merge_from_file(model_zoo.get_config_file("COCO-InstanceSegmentation/mask_rcnn_R_101_FPN_3x.yaml"))
cfg.MODEL.WEIGHTS = os.path.join(cfg.OUTPUT_DIR, "model_final.pth")
cfg.SOLVER.IMS_PER_BATCH = 1
cfg.MODEL.ROI_HEADS.NUM_CLASSES = 1  # only has one class (pnumonia)
#Just run these lines if you have the trained model im memory
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7   # set the testing threshold for this model
#build model
model = build_model(cfg)
DetectionCheckpointer(model).load("output/model_final.pth")
model.eval()#make sure its in eval mode

#run model
with torch.no_grad():
    features = model.backbone(images.tensor.float())
    proposals, _ = model.proposal_generator(images, features)
    instances = model.roi_heads._forward_box(features, proposals)

Then

pred_boxes = [x.pred_boxes for x in instances]
rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], pred_boxes)

This should be my ROI features.

What I am very confused about is instead of using the bounding boxes produced at inference I could use the proposals and the proposal_boxes with their class scores to get the top n features for this image. Cool so I have tried the following:

proposal_boxes = [x.proposal_boxes for x in proposals]
proposal_rois = model.roi_heads.box_pooler([features[f] for f in model.roi_heads.in_features], proposal_boxes)
#found here: https://detectron2.readthedocs.io/_modules/detectron2/modeling/roi_heads/roi_heads.html
box_features = model.roi_heads.box_head(proposal_rois)
predictions = model.roi_heads.box_predictor(box_features)
pred_instances, losses = model.roi_heads.box_predictor.inference(predictions, proposals)

Where I should be getting my proposal box features and its cls in my predictions object. Inspecting this predictions object I see the scores for each box:

CLS Scores in Predictions object

(tensor([[ 0.6308, -0.4926],
         [-1.6662,  1.5430],
         [-0.2080,  0.4856],
         ...,
         [-6.9698,  6.6695],
         [-5.6361,  5.4046],
         [-4.4918,  4.3899]], device='cuda:0', grad_fn=<AddmmBackward>),

After softmaxing and placing these cls scores in a dataframe and setting a threshold of 0.6 I get:

pred_df = pd.DataFrame(predictions[0].softmax(-1).tolist())
pred_df[pred_df[0] > 0.6]
    0           1
0   0.754618    0.245382
6   0.686816    0.313184
38  0.722627    0.277373

and in my predictions object I get the same top score, but only 1 instance rather than 2 (I set cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7):

Prediction Instances:

[Instances(num_instances=1, image_height=800, image_width=800, fields=[pred_boxes: Boxes(tensor([[548.5992, 341.7193, 756.9728, 438.0507]], device='cuda:0',
        grad_fn=<IndexBackward>)), scores: tensor([0.7546], device='cuda:0', grad_fn=<IndexBackward>), pred_classes: tensor([0], device='cuda:0')])]

The predictions also contain Tensor: Nx4 or Nx(Kx4) bounding box regression deltas. which I don't exactly know what they do and look like:

Bounding box regression deltas in Predictions object

tensor([[ 0.2502,  0.2461, -0.4559, -0.3304],
        [-0.1359, -0.1563, -0.2821,  0.0557],
        [ 0.7802,  0.5719, -1.0790, -1.3001],
        ...,
        [-0.8594,  0.0632,  0.2024, -0.6000],
        [-0.2020, -3.3195,  0.6745,  0.5456],
        [-0.5542,  1.1727,  1.9679, -2.3912]], device='cuda:0',
       grad_fn=<AddmmBackward>)

Something else strange is that my proposal boxes and my prediction boxes are different but similar:

Proposal bounding boxes

[Boxes(tensor([[532.9427, 335.8969, 761.2068, 438.8086],#this box vs the instance box
         [102.7041, 352.5067, 329.4510, 440.7240],
         [499.2719, 317.9529, 764.1958, 448.1386],
         ...,
         [ 25.2890, 379.3329,  28.6030, 429.9694],
         [127.1215, 392.6055, 328.6081, 489.0793],
         [164.5633, 275.6021, 295.0134, 462.7395]], device='cuda:0'))]