title: encoding and decoding pictures pytorch

body:
Task: Using the example of the "fetch_lfw_people" dataset to write and train an autocoder.
Write an iteration code by epoch. Write code to visualize the learning process and count the metrics for validation after each epoch.
Train auto encoder. Achieve low loss on validation.

My code:

from sklearn.datasets import fetch_lfw_people
import numpy as np
import torch
from torch.utils.data import TensorDataset, DataLoader
from sklearn.model_selection import train_test_split

Data preparation:

lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)
X = lfw_people['images']

X_train, X_test = train_test_split(X, test_size=0.1)

X_train = torch.tensor(X_train, dtype=torch.float32, requires_grad=True)
X_test = torch.tensor(X_test, dtype=torch.float32, requires_grad=False)
dataset_train = TensorDataset(X_train, torch.zeros(len(X_train)))
dataset_test = TensorDataset(X_test, torch.zeros(len(X_test)))

batch_size = 32

train_loader = DataLoader(dataset_train, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(dataset_test, batch_size=batch_size, shuffle=False)

Сreate a network with encoding and decoding functions:

class Autoencoder(torch.nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()
        self.encoder = torch.nn.Sequential(
            torch.nn.Conv2d(in_channels=1, out_channels=32, kernel_size=3, stride=2),
            torch.nn.ReLU(),

            torch.nn.Conv2d(in_channels=32, out_channels=64, stride=2, kernel_size=3),
            torch.nn.ReLU(),

            torch.nn.Conv2d(in_channels=64, out_channels=64, stride=2, kernel_size=3),
            torch.nn.ReLU(),

            torch.nn.Conv2d(in_channels=64, out_channels=64, stride=2, kernel_size=3)
        )

        self.decoder = torch.nn.Sequential(
            torch.nn.ConvTranspose2d(in_channels=64, out_channels=64, kernel_size=3, stride=2),

            torch.nn.ConvTranspose2d(in_channels=64, out_channels=64, kernel_size=(3,4), stride=2),

            torch.nn.ConvTranspose2d(in_channels=64, out_channels=32, kernel_size=4, stride=2),

            torch.nn.ConvTranspose2d(in_channels=32, out_channels=1, kernel_size=(4,3), stride=2)
        )

    def encode(self, X):
        encoded_X = self.encoder(X)
        batch_size = X.shape[0]
        return encoded_X.reshape(batch_size, -1)

    def decode(self, X):
        pre_decoder = X.reshape(-1, 64, 2, 1)
        return self.decoder(pre_decoder)

I check the work of the model before learning by one example:

model = Autoencoder()

sample = X_test[:1]
sample = sample[:, None]
result = model.decode(model.encode(sample))  # before train

fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2)
ax1.imshow(sample[0][0].detach().numpy(), cmap=plt.cm.Greys_r)
ax2.imshow(result[0][0].detach().numpy(), cmap=plt.cm.Greys_r)
plt.show()

The result is unsatisfactory. I start training:

model = Autoencoder()
loss = torch.nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

history_train = []
history_test = []

for i in range(5):
    for x, y in train_loader:
        x = x[:, None]

        model.train()

        decoded_x = model.decode(model.encode(x))
        mse_loss = loss(torch.tensor(decoded_x, dtype=torch.float), x)

        optimizer.zero_grad()
        mse_loss.backward()
        optimizer.step()

        history_train.append(mse_loss.detach().numpy())

    model.eval()
    with torch.no_grad():
        for x, y in train_loader:
            x = x[:, None]

            result_x = model.decode(model.encode(x))
            loss_test = loss(torch.tensor(result_x, dtype=torch.float), x)

            history_test.append(loss_test.detach().numpy())

plt.subplot(1, 2, 1)
plt.plot(history_train)
plt.title("Optimization process for train data")

plt.subplot(1, 2, 2)
plt.plot(history_test)
plt.title("Loss for test data")

plt.show

A huge loss on the training data and on the test.

Аfter training nothing has changed:

with torch.no_grad():
    model.eval()
    res1 = model.decode(model.encode(sample))

fig, (ax1, ax2) = plt.subplots(nrows=1, ncols=2)
ax1.imshow(sample[0][0].detach().numpy(), cmap=plt.cm.Greys_r)
ax2.imshow(res1[0][0].detach().numpy(), cmap=plt.cm.Greys_r)
plt.show()

Why such a big loss? Reducing the input to the interval [-1, 1] does not help. I did it like this: (value / 255) * 2 - 1
Why do not change the parameters of the model after training?
Why does not change the decoded sample?

Result: before train, after train, loss
https://i.stack.imgur.com/OhdrJ.jpg


title: Pytorch Error, RuntimeError: expected scalar type Long but found Double

body:
I have run into the following error while training a BERT classifier. The

type(b_input_mask) = type(b_labels) = torch.Tensor

type(b_labels[i]) = tensor(1., dtype=torch.float64)

type(b_input_masks[i]) = class'torch.Tensor'

What could be the possible data type error here since I have not typecasted any variable to either long or double?

Thanks in advance!


title: Memory error when iterate over two dataloaders simultaneously in pytorch

body:
I am trying to train my model using 2 dataloaders from 2 different datasets.

I found how to set up this by using cycle() and zip() because my datasets are not the same length from here: How to iterate over two dataloaders simultaneously using pytorch?

  File "/home/Desktop/example/train.py", line 229, in train_2
    for i, (x1, x2) in enumerate(zip(cycle(train_loader_1), train_loader_2)):
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 346, in __next__
    data = self.dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 80, in default_collate
    return [default_collate(samples) for samples in transposed]
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 80, in <listcomp>
    return [default_collate(samples) for samples in transposed]
  File "/home/.conda/envs/3dcnn/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 154140672 bytes. Error code 12 (Cannot allocate memory)

I tried to solve that by setting num_workers=0, decreasing the batch size, using pinned_memory=False and shuffle=False...
But none of it worked... I am having 256GB of RAM and 4 NVIDIA TESLA V100 GPUs.

I tried to run it just by not training in 2 dataloaders simultaneously but individually and it worked. However for my project I need this parallel training with 2 datasets...


title: Expected more than 1 value per channel when training, got input size torch.Size([1, xx])

body:
Consider the following network snippet:

def __init__(self, model, n_class, dropout_rate,device):
    super(NewModel, self).__init__()

    self.bert = model
    self.linear = nn.Linear(self.bert.config.hidden_size, 2)
    self.linear_1 = nn.Linear(self.bert.config.hidden_size, self.bert.config.hidden_size)

    self.dropout_rate = dropout_rate
    self.dropout_1 = nn.Dropout(p = self.dropout_rate)

    self.activation = nn.LeakyReLU()

    self.bn = nn.BatchNorm1d(num_features = self.bert.config.hidden_size)

def forward(self, batch):
    outputs = self.bert(
        input_ids = batch[0].to(self.device),
        attention_mask = batch[1].to(self.device),
        token_type_ids = None,
        position_ids = None,
        head_mask = None,
        inputs_embeds = None,
    )

    output = outputs[0]
    pooled_output=output[:,0]
    pooled_output = pooled_output.unsqueeze(0)

    pooled_output_1 = self.dropout_1(self.bn(pooled_output))

    logits = self.linear(F.leaky_relu(self.linear_1(pooled_output_1)))

My batch size is 16 and during training I get this error:

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 16])

I already set drop_last=True in the DataLoader, but the error persists.

Any help would be greatly appreciated.


title: Order of layers in hidden states in PyTorch GRU return

body:
This is the API I am looking at, https://pytorch.org/docs/stable/nn.html#gru

It outputs:

output of shape (seq_len, batch, num_directions * hidden_size)
h_n of shape (num_layers * num_directions, batch, hidden_size)

For GRU with more than one layers, I wonder how to fetch the hidden state of the last layer, should it be h_n[0] or h_n[-1]?

What if it's bidirectional, how to do the slicing to obtain the last hidden layer states of GRUs in both directions?


can someone please help me out with this UserWarning in torchaudio?
ErrorMessage:
C:\Users\anaconda3\lib\site-packages\torchaudio\extension\extension.py:14:
UserWarning: torchaudio C++ extension is not available.
  warnings.warn('torchaudio C++ extension is not available.')

Thank's in advance!


title: Pytorch: why print(model) does not show the activation functions?

body:
I need to extract weights, bias and at least the type of activation function from a trained NN in pytorch.

I know that to extract the weights and biases the command is:

model.parameters()

but I can't figure out how to extract also the activation function used on the layers.Here is my network

class NetWithODE(torch.nn.Module):
    def __init__(self, n_feature, n_hidden, n_output, sampling_interval, scaler_features):
        super(NetWithODE, self).__init__()
        self.hidden = torch.nn.Linear(n_feature, n_hidden)  # hidden layer
        self.predict = torch.nn.Linear(n_hidden, n_output)  # output layer
        self.sampling_interval = sampling_interval
        self.device = torch.device("cpu")
        self.dtype = torch.float
        self.scaler_features = scaler_features

    def forward(self, x):
        x0 = x.clone().requires_grad_(True)
        # activation function for hidden layer
        x = F.relu(self.hidden(x))
        # linear output, here r should be the output
        r = self.predict(x)
        # Now the r enters the integrator
        x = self.integrate(r, x0)

        return x

    def integrate(self, r, x0):
        # RK4 steps per interval
        M = 4
        DT = self.sampling_interval / M
        X = x0

        for j in range(M):
            k1 = self.ode(X, r)
            k2 = self.ode(X + DT / 2 * k1, r)
            k3 = self.ode(X + DT / 2 * k2, r)
            k4 = self.ode(X + DT * k3, r)
            X = X + DT / 6 * (k1 + 2 * k2 + 2 * k3 + k4)

        return X

    def ode(self, x0, r):
        qF = r[0, 0]
        qA = r[0, 1]
        qP = r[0, 2]
        mu = r[0, 3]

        FRU = x0[0, 0]
        AMC = x0[0, 1]
        PHB = x0[0, 2]
        TBM = x0[0, 3]

        fFRU = qF * TBM
        fAMC = qA * TBM
        fPHB = qP - mu * PHB
        fTBM = mu * TBM

        return torch.stack((fFRU, fAMC, fPHB, fTBM), 0)

if I run the command

print(model)

I get

NetWithODE(
  (hidden): Linear(in_features=4, out_features=10, bias=True)
  (predict): Linear(in_features=10, out_features=4, bias=True)
)

But where can I get the activation function (in this case Relu)?

I have pytorch 1.4.


title: lstm pytorch RuntimeError: Expected hidden[0] size (1, 1, 256), got (1, 611, 256)

body:
I am trying to do batch processing with nn.lstm
From the documentation https://pytorch.org/docs/master/generated/torch.nn.LSTM.html I get that h0 and c0 should be of dimension:(num_layers * num_directions, batch, hidden_size).
But when I am trying to give input tensor with batch size>1 and h0 , c0 batch size>1. It is giving me error stating: "RuntimeError: Expected hidden[0] size (1, 1, 256), got (1, 611, 256)"
Here is my code:
it contains 1 memory buffer, Actor, Critic, TD3, ENV classes and main training is in TD3 which has actor and critic objects.
Can someone please help a check what am i missing here.
import os
import time
import random
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sklearn.preprocessing import StandardScaler
import torch
import torch.nn as nn
import torch.nn.functional as F
from random import random as rndm
from torch.autograd import Variable
from collections import deque
import pandas_datareader.data as pdr
import datetime

os.chdir('C:\\Users\\granthjain\\Desktop\\startup_code')

torch.set_default_tensor_type('torch.DoubleTensor')

f = open('lstm_with_noise_batch.txt', 'w+')

class ReplayBuffer(object):

    def __init__(self, max_size=1e6):
        self.storage = []
        self.max_size = max_size
        self.ptr = 0

    def add(self, transition):
        if len(self.storage) == self.max_size:
            self.storage[int(self.ptr)] = transition
        else:
            self.storage.append(transition)
        self.ptr = (self.ptr + 1) % self.max_size

    def sample(self, batch_size):

        ind = np.random.randint(0, self.ptr, size=batch_size)
        ind = np.random.randint(self.ptr)
        (batch_states, batch_next_states, batch_actions, batch_rewards,
         batch_dones) = ([], [], [], [], [])
        for i in range(ind - batch_size, ind):
            (state, next_state, action, reward, done) = self.storage[i]

            if state is None:
                continue
            elif next_state is None:
                continue
            elif action is None:
                continue
            elif reward is None:
                continue
            elif done is None:
                continue

            batch_states.append(np.array(state, copy=False))
            batch_next_states.append(np.array(next_state, copy=False))
            batch_actions.append(np.array(action, copy=False))
            batch_rewards.append(np.array(reward, copy=False))
            batch_dones.append(np.array(done, copy=False))

        return (np.array(batch_states, dtype=object).astype(float),
                np.array(batch_next_states,
                dtype=object).astype(float), np.array(batch_actions,
                dtype=object).astype(float), np.array(batch_rewards,
                dtype=object).astype(float), np.array(batch_dones,
                dtype=object).astype(float))

class Actor(nn.Module):

    def __init__(
        self,
        state_dim,
        action_dim,
        max_action,
        ):
        super(Actor, self).__init__()
        self.lstm = nn.LSTM(state_dim, 256)
        self.layer_1 = nn.Linear(256, 400)
        self.layer_2 = nn.Linear(400, 300)
        self.layer_3 = nn.Linear(300, action_dim)
        self.max_action = max_action

    def forward(self, x, hx):
        (hx, cx) = hx
        (output, (hx, cx)) = self.lstm(x, (hx, cx))
        x = F.relu(self.layer_1(output))
        x = F.relu(self.layer_2(x))
        x = self.max_action * torch.tanh(self.layer_3(x))

    # print("inside forward type cx:",len(output))

        return (x, hx, cx)

class Critic(nn.Module):

    def __init__(self, state_dim, action_dim):
        super(Critic, self).__init__()

    # Defining the first Critic neural network

        self.lstm1 = nn.LSTM(state_dim + action_dim, 256)
        self.layer_1 = nn.Linear(256, 400)
        self.layer_2 = nn.Linear(400, 300)
        self.layer_3 = nn.Linear(300, 1)

    # Defining the second Critic neural network

        self.lstm2 = nn.LSTM(state_dim + action_dim, 256)
        self.layer_4 = nn.Linear(256, 400)
        self.layer_5 = nn.Linear(400, 300)
        self.layer_6 = nn.Linear(300, 1)

    def forward(
        self,
        x,
        u,
        hx,
        ):
        xu = torch.cat([x, u], 1)

    # Forward-Propagation on the first Critic Neural Network

        xu = torch.reshape(xu, (xu.shape[0], 1, 6))
        (hx1, cx1) = hx
        (hx2, cx2) = hx
        (output, (hx1, cx1)) = self.lstm1(xu, (hx1, hx2))
        x1 = F.relu(self.layer_1(output))
        x1 = F.relu(self.layer_2(x1))
        x1 = self.layer_3(x1)

    # Forward-Propagation on the second Critic Neural Network

        (output, (hx2, cx2)) = self.lstm2(xu, (hx2, cx2))
        x2 = F.relu(self.layer_4(output))
        x2 = F.relu(self.layer_5(x2))
        x2 = self.layer_6(x2)
        return (
            x1,
            x2,
            hx1,
            hx2,
            cx1,
            cx2,
            )

    def Q1(
        self,
        x,
        u,
        hx1,
        ):
        xu = torch.cat([x, u], 1)
        xu = torch.reshape(xu, (xu.shape[0], 1, 6))
        (hx1, cx1) = hx1
        (output, (hx1, cx1)) = self.lstm1(xu, (hx1, cx1))
        x1 = F.relu(self.layer_1(output))
        x1 = F.relu(self.layer_2(x1))
        x1 = self.layer_3(x1)
        return (x1, hx1, cx1)

class TD3(object):

    def __init__(
        self,
        state_dim,
        action_dim,
        max_action,
        ):
        self.actor = Actor(state_dim, action_dim, max_action).to(device)
        self.actor_target = Actor(state_dim, action_dim,
                                  max_action).to(device)
        self.actor_target.load_state_dict(self.actor.state_dict())
        self.actor_optimizer = torch.optim.Adam(self.actor.parameters())
        self.critic = Critic(state_dim, action_dim).to(device)
        self.critic_target = Critic(state_dim, action_dim).to(device)
        self.critic_target.load_state_dict(self.critic.state_dict())
        self.critic_optimizer = \
            torch.optim.Adam(self.critic.parameters())
        self.max_action = max_action

    def select_action(self, state, hx1):
        (hx, cx) = hx1
        x = self.actor(state, hx1)
        return x

    def train(
        self,
        replay_buffer,
        iterations,
        batch_size=50,
        discount=0.99,
        tau=0.005,
        policy_noise=0.2,
        noise_clip=0.5,
        policy_freq=2,
        ):

        b_state = torch.Tensor([])
        b_next_state = torch.Tensor([])
        b_done = torch.Tensor([])
        b_reward = torch.Tensor([])
        b_action = torch.Tensor([])

        for it in range(iterations):

            # print ('it: ', it, ' iterations: ', iterations)

      # Step 4: We sample a batch of transitions (s, s’, a, r) from the memory

            (batch_states, batch_next_states, batch_actions,
             batch_rewards, batch_dones) = \
                replay_buffer.sample(batch_size)

            batch_states = batch_states.astype(float)
            batch_next_states = batch_next_states.astype(float)
            batch_actions = batch_actions.astype(float)
            batch_rewards = batch_rewards.astype(float)
            batch_dones = batch_dones.astype(float)

            state = torch.from_numpy(batch_states)
            next_state = torch.from_numpy(batch_next_states)
            action = torch.from_numpy(batch_actions)
            reward = torch.from_numpy(batch_rewards)
            done = torch.from_numpy(batch_dones)

            b_size = 1
            seq_len = state.shape[0]
            batch = b_size
            input_size = state_dim

            state = torch.reshape(state, (seq_len, 1, state_dim))
            next_state = torch.reshape(next_state, (seq_len, 1,
                    state_dim))
            done = torch.reshape(done, (seq_len, 1, 1))
            reward = torch.reshape(reward, (seq_len, 1, 1))
            action = torch.reshape(action, (seq_len, 1, action_dim))

            b_state = torch.cat((b_state, state),dim=1)
            b_next_state = torch.cat((b_next_state, next_state),dim=1)
            b_done = torch.cat((b_done, done),dim=1)
            b_reward = torch.cat((b_reward, reward),dim=1)
            b_action = torch.cat((b_action, action),dim=1)

        print("dim state:",b_state.shape)
        print("dim next_state:",b_next_state.shape)
        print("dim done:",b_done.shape)
        print("dim reward:",b_reward.shape)
        print("dim action:",b_action.shape)

      # for h and c shape (num_layers * num_directions, batch, hidden_size)

        h0 = torch.zeros(1, b_state.shape[1], 256)
        c0 = torch.zeros(1, b_state.shape[1], 256)
      # Step 5: From the next state s’, the Actor target plays the next action a’

        next_action = self.actor_target(next_state, (h0, c0))
        next_action = next_action[0]

      # Step 6: We add Gaussian noise to this next action a’ and we clamp it in a range of values supported by the environment

        noise = torch.Tensor(next_action).data.normal_(0,
                policy_noise).to(device)
        noise = noise.clamp(-noise_clip, noise_clip)
        next_action = (next_action + noise).clamp(-self.max_action,
                self.max_action)

      # Step 7: The two Critic targets take each the couple (s’, a’) as input and return two Q-values Qt1(s’,a’) and Qt2(s’,a’) as outputs

        result = self.critic_target(next_state, next_action, (h0,
                c0))
        target_Q1 = result[0]
        target_Q2 = result[1]

      # Step 8: We keep the minimum of these two Q-values: min(Qt1, Qt2)

        target_Q = torch.min(target_Q1, target_Q2).double()

      # Step 9: We get the final target of the two Critic models, which is: Qt = r + γ * min(Qt1, Qt2), where γ is the discount factor

        target_Q = reward + (1 - done) * discount * target_Q

      # Step 10: The two Critic models take each the couple (s, a) as input and return two Q-values Q1(s,a) and Q2(s,a) as outputs

        action = torch.reshape(action, next_action.shape)
        result = self.critic(state, action, (h0, c0))
        current_Q1 = result[0]
        current_Q2 = result[1]

      # Step 11: We compute the loss coming from the two Critic models: Critic Loss = MSE_Loss(Q1(s,a), Qt) + MSE_Loss(Q2(s,a), Qt)

        critic_loss = F.mse_loss(current_Q1, target_Q) \
            + F.mse_loss(current_Q2, target_Q)

      # Step 12: We backpropagate this Critic loss and update the parameters of the two Critic models with a SGD optimizer

        self.critic_optimizer.zero_grad()
        critic_loss.backward()
        self.critic_optimizer.step()

      # Step 13: Once every two iterations, we update our Actor model by performing gradient ascent on the output of the first Critic model

        out = self.actor(state, (h0, c0))
        out = out[0]
        (actor_loss, hx, cx) = self.critic.Q1(state, out, (h0,
                c0))
        actor_loss = -1 * actor_loss.mean()
        self.actor_optimizer.zero_grad()
        actor_loss.backward()
        self.actor_optimizer.step()

      # Step 14: Still once every two iterations, we update the weights of the Actor target by polyak averaging

        for (param, target_param) in zip(self.actor.parameters(),
                self.actor_target.parameters()):
            target_param.data.copy_(tau * param.data + (1 - tau)
                    * target_param.data)

      # Step 15: Still once every two iterations, we update the weights of the Critic target by polyak averaging

        for (param, target_param) in zip(self.critic.parameters(),
                self.critic_target.parameters()):
            target_param.data.copy_(tau * param.data + (1 - tau)
                    * target_param.data)

  # Making a save method to save a trained model

    def save(self, filename, directory):
        torch.save(self.actor.state_dict(), '%s/%s_actor.pth'
                   % (directory, filename))
        torch.save(self.critic.state_dict(), '%s/%s_critic.pth'
                   % (directory, filename))

  # Making a load method to load a pre-trained model

    def load(self, filename, directory):
        self.actor.load_state_dict(torch.load('%s/%s_actor.pth'
                                   % (directory, filename)))
        self.critic.load_state_dict(torch.load('%s/%s_critic.pth'
                                    % (directory, filename)))

class ENV:

    def __init__(
        self,
        state_dim,
        action_dim,
        data,
        ):
        self.state_dim = state_dim
        self.state = torch.zeros(self.state_dim)
        self.state[state_dim - 1] = 100000.0
        self.next_state = torch.zeros(self.state_dim)
        self.next_state[state_dim - 1] = 100000.0
        self.action_dim = action_dim
        self.data = data
        self.idx = 0
        self._max_episode_steps = 200
        self.state[1] = self.data[self.idx]
        self.next_state[1] = self.data[self.idx]
        self.buy = 0

    def reset(self):
        self.next_state = torch.zeros(self.state_dim)
        self.next_state[state_dim - 1] = 100000.0
        self.state = torch.zeros(self.state_dim)
        self.state[state_dim - 1] = 100000.0
        self.state[1] = self.data[self.idx]
        self.next_state[1] = self.data[self.idx]

        ch = self.state[0]
        cp = self.state[1]
        cc = self.state[2]
        st = torch.tensor([ch, cp, cc])
        self.buy = 0
        return st

    def step(self, action):
        done = False
        act_t = torch.argmax(action)
        self.idx += 1
        if act_t == 0:
            cp = 1.0003 * self.state[1]
            num_s = int(self.state[2] / cp)

            self.next_state[0] += num_s
            self.next_state[2] = self.state[2] % cp

            self.next_state[1] = self.data[self.idx]
            self.buy = 1
        elif act_t == 1:
            self.next_state[1] = self.data[self.idx]
        elif act_t == 2:
            self.next_state[2] = self.state[2] + self.state[1] * (1
                    - 0.0023) * self.state[0]
            self.next_state[0] = 0
            self.next_state[1] = self.data[self.idx]

            if self.buy == 1:
                done = True
                self.buy = 0

        reward = self.next_state[2] - self.state[2] \
            + self.next_state[1] * self.next_state[0] - self.state[1] \
            * self.state[0] - 1

        self.state[0] = self.next_state[0]
        self.state[1] = self.next_state[1]
        self.state[2] = self.next_state[2]

        ch = self.state[0]
        cp = self.state[1]
        cc = self.state[2]

        st = torch.tensor([ch, cp, cc])

        return (st, reward, done)

# Selecting the device (CPU or GPU)

device = torch.device(('cuda' if torch.cuda.is_available() else 'cpu'))

# set the parameters

start_timesteps = 1e3  # Number of iterations/timesteps before which the model randomly chooses an action, and after which it starts to use the policy network
eval_freq = 5e1  # How often the evaluation step is performed (after how many timesteps)
max_timesteps = 5e3  # Total number of iterations/timesteps
save_models = True  # Boolean checker whether or not to save the pre-trained model
expl_noise = 0.1  # Exploration noise - STD value of exploration Gaussian noise
batch_size = 200  # Size of the batch
discount = 0.99  # Discount factor gamma, used in the calculation of the total discounted reward
tau = 0.005  # Target network update rate
policy_noise = 0.2  # STD of Gaussian noise added to the actions for the exploration purposes
noise_clip = 0.5  # Maximum value of the Gaussian noise added to the actions (policy)
policy_freq = 2  # Number of iterations to wait before the policy network (Actor model) is updated

state_dim = 3
action_dim = 3
max_action = 1
idx = 0

# instantiate policy

policy = TD3(state_dim, action_dim, max_action)

indices = pd.read_csv('nifty_test.csv')
indices = indices['0']

indices = pd.read_csv('EQUITY_L.csv')
indices = indices['SYMBOL']

# Create the environment for each ticker
# data = pd.read_csv('PAGEIND.csv')

for ticker in indices:
    print(ticker)

    ohlcv = pd.read_csv(ticker + '.csv')
    data = ohlcv.copy()
    data = data['Close']
    data = np.array(data).reshape(-1, 1)
    count = 0
    max_timesteps = data.shape[0]

    data = torch.DoubleTensor(data)
    env = ENV(state_dim, action_dim, data)

    replay_buffer = ReplayBuffer()

    # init training variables

    total_timesteps = 0
    timesteps_since_eval = 0
    episode_num = 0
    done = True
    t0 = time.time()
    obs = env.reset()
    hx = torch.zeros(1, 1, 256)
    cx = torch.zeros(1, 1, 256)

    # Set rewards and episode timesteps to zero

    episode_reward = 0
    episode_timesteps = 0
    episode_num = 0

    # We start the main loop over max_timesteps

    while total_timesteps < max_timesteps:

      # If the episode is done

        if done | (total_timesteps == max_timesteps - 2) \
            & (episode_timesteps > 200):
            count = count + 1
            if (count % 100 == 0) & (count >= 100) \
                | (total_timesteps == max_timesteps - 2) \
                & (episode_timesteps > 200):

            # If we are not at the very beginning, we start the training process of the model

                if total_timesteps != 0:
                    print('Total Timesteps: {} Episode Num: {} Reward: {}'.format(total_timesteps,
                            episode_num, episode_reward))
                    policy.train(
                        replay_buffer,
                        episode_timesteps,
                        batch_size,
                        discount,
                        tau,
                        policy_noise,
                        noise_clip,
                        policy_freq,
                        )

                    if total_timesteps > 0.6 * max_timesteps + 1:
                        print('model output: Total Timesteps: {} Episode Num: {} Reward: {}'.format(total_timesteps,
                                episode_num, episode_reward))
                        f.write('model output: Total Timesteps: '
                                + str(total_timesteps)
                                + ' episode_num '
                                + str(episode_num)
                                + ' episode_reward '
                                + str(episode_reward))

            # When the training step is done, we reset the state of the environment

                obs = env.reset()

            # Set the Done to False

                done = False

            # Set rewards and episode timesteps to zero

                episode_reward = 0
                episode_timesteps = 0
                episode_num += 1
                hx = torch.zeros(1, 1, 256)
                cx = torch.zeros(1, 1, 256)

      # Before 1000 timesteps, we play random actions

        if total_timesteps < 0.6 * max_timesteps:

    # random action

            actn = torch.randn(action_dim)
            action = torch.zeros(action_dim)
            action[torch.argmax(actn)] = 1
        else:

            # After 1000 timesteps, we switch to the model

    #    input of shape (seq_len, batch, input_size)

            obs1 = torch.reshape(obs, (1, 1, state_dim))
            action = policy.select_action(obs1, (hx, cx))

            actn = action[0]
            hx = action[1]
            cx = action[2]

        # If the explore_noise parameter is not 0, we add noise to the action and we clip it

            if expl_noise != 0:
                print ('policy action:', actn)
                actn = actn + torch.randn(action_dim)
                action = torch.zeros(action_dim)
                action[torch.argmax(actn)] = 1

      # The agent performs the action in the environment, then reaches the next state and receives the reward

        (new_obs, reward, done) = env.step(action)

      # We check if the episode is done

        done_bool = (0 if episode_timesteps + 1
                     == env._max_episode_steps else float(done))

      # We increase the total reward

        episode_reward += reward

      # We store the new transition into the Experience Replay memory (ReplayBuffer)

        replay_buffer.add((obs, new_obs, action, reward, done_bool))

      # We update the state, the episode timestep, the total timesteps, and the timesteps since the evaluation of the policy

        obs = new_obs
        episode_timesteps += 1
        total_timesteps += 1
        timesteps_since_eval += 1

f.close()

and below is the output:
20MICRONS
Total Timesteps: 611 Episode Num: 0 Reward: -53044.2697380831
dim state: torch.Size([200, 611, 3])
dim next_state: torch.Size([200, 611, 3])
dim done: torch.Size([200, 611, 1])
dim reward: torch.Size([200, 611, 1])
dim action: torch.Size([200, 611, 3])
Traceback (most recent call last):

  File "C:\Users\granthjain\Desktop\try_lstm.py", line 538, in <module>
    policy_freq,

  File "C:\Users\granthjain\Desktop\try_lstm.py", line 279, in train
    next_action = self.actor_target(next_state, (h0, c0))

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)

  File "C:\Users\granthjain\Desktop\try_lstm.py", line 106, in forward
    (output, (hx, cx)) = self.lstm(x, (hx, cx))

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 567, in forward
    self.check_forward_args(input, hx, batch_sizes)

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 523, in check_forward_args
    'Expected hidden[0] size {}, got {}')

  File "C:\Users\granthjain\anaconda3\lib\site-packages\torch\nn\modules\rnn.py", line 187, in check_hidden_size
    raise RuntimeError(msg.format(expected_hidden_size, tuple(hx.size())))

RuntimeError: Expected hidden[0] size (1, 1, 256), got (1, 611, 256)


title: How to delete permanently from mounted Drive folder?

body:
I wrote a script to upload my models and training examples to Google Drive after every iteration in case of crashes or anything that stops the notebook from running, which looks something like this:

drive_path = 'drive/My Drive/Colab Notebooks/models/'
if path.exists(drive_path):
    shutil.rmtree(drive_path)
shutil.copytree('models', drive_path)

Whenever I check my Google Drive, a few GBs is taken up by dozens of deleted models folder in the Trash, which I have to manually delete them.

The only function in google.colab.drive seems to be mount and that's it.

According to this tutorial, shutil.rmtree() removes a directory permanently but apparently it doesn't work for Drive.


title: How to sort a tensor by first dimension in pytorch?

body:
I have a 2D tensor and I would like to sort by the first dimension like this example:

a = torch.FloatTensor(
   [[5, 5],
    [5, 3],
    [3, 5],
    [6, 4],
    [3, 7]])

And I expected this result after sorting:

a = torch.FloatTensor(
   [[3, 5],
    [3, 7],
    [5, 3],
    [5, 5],
    [6, 4]])

Is it possible to do this in pytorch? I know that is possible to do it in numpy, but I want do it in GPU using torch.


title: How to load dataset from pickle files into PyTorch?

body: I have X_train(inputs) and Y_train(labels) in separate pickle files in form of integer matrices. Now, I need to load them and train using PyTorch. I tried torch.utils.data.DataLoader and torchvision.datasets.DatasetFolder but nothing worked or I might be getting wrong somewhere. Please suggest a proper way for the same.


title: pytorch sliding window with unfold & fold

body:
I'm trying to use a simple ssd that was trained on 300x300 data with annotated bounding boxes. If I crop the images manually, it works correctly but with full size images it fails (obviously) due to resizing large images to 300x300 removes many visual features.
I figured the good old sliding window will work here, but I have some problems rebuilding the images with detection and must admit I'm a bit clueless on how to approach, What I have so far is:
at first, I tried this:
chips = F.unfold(img_t.data, kernel_size=300)

following some examples from stack overflow, but this gives me error Input Error: Only 4D input Tensors are supported (got 3D)
so following some more googling I found something that works:
        patch_w = 300
        patch_h = 300
        patches = img_t.data.unfold(0, 3, 3).unfold(1, patch_w, patch_h).unfold(2, patch_w, patch_h)
#Visualise small part:
        fig = plt.figure(figsize=(4, 4))
        fig.tight_layout()
        plt.subplots_adjust(left=0.1, bottom=0.1, right=0.9, top=0.9, wspace=0.01, hspace=0.01)
        for i in range(4):
            for j in range(4):
                inp = transp(patches[0][i][j])
                inp = np.array(inp)

                ax = fig.add_subplot(4, 4, ((i*4)+j)+1, xticks=[], yticks=[])
                plt.imshow(inp)
        plt.show()

I then feed the patches to my detector and it looks more or less ok, but there's no overlap (object can be cut into pieces and missed) and more importantly, I can't reverse the process with unfold without getting drowned in exceptions.
I'm not adamant on using the fold/unfold combination for the task as what I really want is to be able to feed large image into the network in a way that will preserve as much information as possible, mark down the detections and rebuild image with bounding boxes from the smaller patches.
What I came up with is this:
new_im = Image.new("RGB", (300*dims[1], 300*dims[0]))
idx = 0
for i in range(dims[0]):
    for j in range(dims[1]):
        new_im.paste(tiles[idx], (j*300, i*300))
        idx += 1
new_im.show()

which rebuilds the image back, but in a very artificial way, where the detector annotates image cropping and returns them as image list which I here rebuild which is both ugly and inefficient.
After a bit of fiddling, I got it to work, but there comes a peculiarity of pytorch.It adds overlapping parts of patches instead of averaging them (see image). How can I fix it? I realise normalisation won't do anything here since it would normalise the good pixels as well, so it needs to just average overlapping pixels.

Also, please note the image was cropped erroneously. Simple code to reproduce:
def fold_unfold(img_path):
    transt = transforms.Compose([transforms.ToTensor(),
    # transforms.Normalize(mean=[0.5,0.5, 0.5], std=[0.5, 0.5, 0.5])
        ])

    transp = transforms.Compose([
        # transforms.Normalize(mean=[0.5,0.5, 0.5], std=[0.5, 0.5, 0.5]),
        transforms.ToPILImage()
    ])
    img_t = transt(Image.open(img_path))
    img_t = img_t.unsqueeze(0)
    kernel = 300
    stride = 200
    img_shape = img_t.shape
    B, C, H, W = img_shape
    # number of pixels missing:
    pad_w = W % kernel
    pad_h = H % kernel
    # Padding the **INPUT** image with missing pixels:
    img_t = F.pad(input=img_t, pad=(pad_w//2, pad_w-pad_w//2, pad_h//2, pad_h-pad_h//2), mode='constant', value=0)
    img_shape = img_t.shape
    B, C, H, W = img_shape
    print("\n-----input shape: ", img_shape)

    patches = img_t.unfold(3, kernel, stride).unfold(2, kernel, stride).permute(0,1,2,3,5,4)

    print("\n-----patches shape:", patches.shape)
    # reshape output to match F.fold input
    patches = patches.contiguous().view(B, C, -1, kernel*kernel)
    print("\n", patches.shape) # [B, C, nb_patches_all, kernel_size*kernel_size]
    patches = patches.permute(0, 1, 3, 2)
    print("\n", patches.shape) # [B, C, kernel_size*kernel_size, nb_patches_all]
    patches = patches.contiguous().view(B, C*kernel*kernel, -1)
    print("\n", patches.shape) # [B, C*prod(kernel_size), L] as expected by Fold
    # https://pytorch.org/docs/stable/nn.html#torch.nn.Fold

    output = F.fold(patches, output_size=(H, W), kernel_size=kernel, stride=stride)
    # mask that mimics the original folding:
    recovery_mask = F.fold(torch.ones_like(patches), output_size=(H,W), kernel_size=kernel, stride=stride)
    output = output/recovery_mask

    print(output.shape) # [B, C, H, W]
    aspil = transp(output[0])
    aspil.show()

still, the image is cropped quite a lot so something is still wrong:

Finally, getting the cropping done Code updated to the working version
The problem is coming from the way pytorch does the unfolding. The unfold from tensor method doesn't zero pad automatically, but rather stops the cropping by cutting off the part that didn't fit. I solved it by zero padding the tensor before cropping it.


title: Pytorch torch.load ModuleNotFoundError: No module named 'utils'

body:
I'm trying to load a pretrained model with torch.load.
I get the following error:
ModuleNotFoundError: No module named 'utils'

I've checked that the path I am using is correct by opening it from the command line. What could be causing this?
Here's my code:
import torch
import sys

PATH = './gan.pth'

model = torch.load(PATH)
model.eval()

EDIT:
Entire error stack:
Traceback (most recent call last):
  File "load.py", line 6, in <module>
    model = torch.load(PATH)
  File "C:\Users\user\anaconda3\envs\pytorch-flask\lib\site-packages\torch\serialization.py", line 595, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\user\anaconda3\envs\pytorch-flask\lib\site-packages\torch\serialization.py", line 774, in _legacy_load
    result = unpickler.load()
ModuleNotFoundError: No module named 'utils'


title: Subsetting A Pytorch Tensor Using Square-Brackets

body:
I came across a line of code used to reduce a 3D Tensor to a 2D Tensor in PyTorch. The 3D tensor x is of size torch.Size([500, 50, 1]) and this line of code:

x = x[lengths - 1, range(len(lengths))]

was used to reduce x to a 2D tensor of size torch.Size([50, 1]). lengths is also a tensor of shape torch.Size([50]) containing values.

Please can anyone explain how this works? Thank you.


title: Pytorch loss function error in the last batch

body:
Assume that I have 77 samples to train my CNN, and my batch size is 10. Then the last batch has a batch size of 7 instead of 10. Somehow when I pass it to the loss function such as nn.MSELoss(), it gives me the error:

  RuntimeError: The size of tensor a (10) must match the size of tensor
  b (7) at non-singleton dimension 1

So pytorch doesn't support batches with different sizes?

My code in doubt:

import numpy as np
import torch
from torch import nn
import torchvision
import torch.nn.functional as F
import torch.optim as optim

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, (5,4))
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(64, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, x.shape[1] * x.shape[2] * x.shape[3])
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

model = Net()

batch_size = 10

# Generating Artifical data
x_train = torch.randn((77,1,20,20))
y_train = torch.randint(0,10,size=(77,),dtype=torch.float)

trainset = torch.utils.data.TensorDataset(x_train,y_train)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=batch_size, shuffle=True, num_workers=0)
# testloader = torch.utils.data.DataLoader(testset, batch_size=batch_size, shuffle=False, num_workers=0)

criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

for epoch in range(20):  # loop over the dataset multiple times

    running_loss = 0.0
    for i, data in enumerate(trainloader, 0):
        # get the inputs
        inputs, labels = data
        # zero the parameter gradients
        optimizer.zero_grad()

        # forward + backward + optimize
        outputs = model(inputs)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()
        if i%10==0:
            print('epoch{}, step{}, loss: {}'.format(epoch + 1, i + 1, running_loss))
#             print("frac post = {}".format(frac_post))
            running_loss = 0.0


title: How I can swap 3 dimensions with each other in Pytorch?

body:
I have a a= torch.randn(28, 28, 8) and I want to swap the dimensions of the tensor and move the third dimension to the first place, first one to the second place and the second one to the third place. I used  b = a.transpose(2, 0, 1) , but I received this error:

TypeError: transpose() received an invalid combination of arguments - got (int, int, int), but expected one of:
 * (name dim0, name dim1)
 * (int dim0, int dim1)

Should I use transpose several times, each time only to swap two dimensions? Is there any way that I can swap all at once?

Thanks.


title: PyTorch calculate MSE and MAE

body:
I would like to calculate the MSE and MAE of the model below. The model is calculating the MSE after each Epoch. What do I need to do to get the overall MSE value, please? Can I use the same code to calculate the MAE? Many Thanks in advance
model.eval()
for images, paths in tqdm(loader_test):
    images = images.to(device)
    targets = torch.tensor([metadata['count'][os.path.split(path)[-1]] for path in paths]) # B
    targets = targets.float().to(device)

    # forward pass:
    output = model(images) # B x 1 x 9 x 9 (analogous to a heatmap)
    preds = output.sum(dim=[1,2,3]) # predicted cell counts (vector of length B)

    # logging:
    loss = torch.mean((preds - targets)**2)
    count_error = torch.abs(preds - targets).mean()
    mean_test_error += count_error
    writer.add_scalar('test_loss', loss.item(), global_step=global_step)
    writer.add_scalar('test_count_error', count_error.item(), global_step=global_step)

    global_step += 1

average_accuracy = 0
mean_test_error = mean_test_error / len(loader_test)
writer.add_scalar('mean_test_error', mean_test_error.item(), global_step=global_step)
average_accuracy += mean_test_error
average_accuracy = average_accuracy /len(loader_test)
print("Average accuracy: %f" % average_accuracy)
print("Test count error: %f" % mean_test_error)
if mean_test_error < best_test_error:
    best_test_error = mean_test_error
    torch.save({'state_dict':model.state_dict(),
                'optimizer_state_dict':optimizer.state_dict(),
                'globalStep':global_step,
                'train_paths':dataset_train.files,
                'test_paths':dataset_test.files},checkpoint_path)


title: How to fill in the blank using bidirectional RNN and pytorch?

body:
I am trying to fill in the blank using a bidirectional RNN and pytorch.

The input will be like: The dog is _____, but we are happy he is okay.

The output will be like:

1. hyper (Perplexity score here)
2. sad (Perplexity score here)
3. scared (Perplexity score here)

I discovered this idea here: https://medium.com/@plusepsilon/the-bidirectional-language-model-1f3961d1fb27

import torch, torch.nn as nn
from torch.autograd import Variable

text = ['BOS', 'How', 'are', 'you', 'EOS']
seq_len = len(text)
batch_size = 1
embedding_size = 1
hidden_size = 1
output_size = 1

random_input = Variable(
    torch.FloatTensor(seq_len, batch_size, embedding_size).normal_(), requires_grad=False)

bi_rnn = torch.nn.RNN(
    input_size=embedding_size, hidden_size=hidden_size, num_layers=1, batch_first=False, bidirectional=True)

bi_output, bi_hidden = bi_rnn(random_input)

# stagger
forward_output, backward_output = bi_output[:-2, :, :hidden_size], bi_output[2:, :, hidden_size:]
staggered_output = torch.cat((forward_output, backward_output), dim=-1)

linear = nn.Linear(hidden_size * 2, output_size)

# only predict on words
labels = random_input[1:-1]

# for language models, use cross-entropy :)
loss = nn.MSELoss()
output = loss(linear(staggered_output), labels)

I am trying to reimplement the code above found at the bottom of the blog post. I am new to pytorch and nlp, and can't understand what the input and output to the code is.

Question about the input: I am guessing the input are the few words that are given. Why does one need beginning of sentence and end of sentence tags in this case? Why don't I see the input being a corpus on which the model is trained like other classic NLP problems? I would like to use the Enron email corpus to train the RNN.

Question about the output: I see the output is a tensor. My understanding is the tensor is a vector, so maybe a word vector in this case. How can you use the tensor to output the words themselves?


title: Why Tensor.clone().detach() is recommended when copying a tensor?

body:
What is the difference between copying a tensor using torch.tensor(sourcetensor) and tensor.clone().detach()?

Like torch.tensor(srctsr) always copies data, tensor.clone().detach() copies data too.
x = torch.tensor([1, 2, 3])
y1 = x.clone().detach()
y2 = torch.tensor(x)
x[0] = 0
print(y1, y2) # both are same

So they seem to be exactly the same. Below is the explanation given in the PyTorch documentation about torch.tensor() and torch.clone().detach()

Therefore torch.tensor(x) is equivalent to x.clone().detach() and torch.tensor(x, requires_grad=True) is equivalent to x.clone().detach().requires_grad_(True). The equivalents using clone() and detach() are recommended.

So if they are equivalent to each other, why is .clone().detach() more preferred than the other?


title: PyTorch : GPU execution time

body:
I am following different tutorials on PyTorch. And I'm trying to use GPU speed but I am encountering a problem between the execution time announced on the web site and my reality!
https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

You can find here the code at the end of the page, and we can read that the program finished in 1min30 against 6/7min on my computer! Can you try it and tell me how many time have you got please! I am very confuse to see a high difference using the same file code than the web site!
This will allow me to understand if the problem comes from my GPU or not :)

My config :
GTX 1080Ti
Windows10
Cuda 9.1
Pytorch 0.4.0

The code:

if __name__ == "__main__":

    plt.ion()   # interactive mode

    ######################################################################
    # Load Data
    # ---------
    #
    # We will use torchvision and torch.utils.data packages for loading the
    # data.
    #
    # The problem we're going to solve today is to train a model to classify
    # **ants** and **bees**. We have about 120 training images each for ants and bees.
    # There are 75 validation images for each class. Usually, this is a very
    # small dataset to generalize upon, if trained from scratch. Since we
    # are using transfer learning, we should be able to generalize reasonably
    # well.
    #
    # This dataset is a very small subset of imagenet.
    #
    # .. Note ::
    #    Download the data from
    #    `here <https://download.pytorch.org/tutorial/hymenoptera_data.zip>`_
    #    and extract it to the current directory.

    # Data augmentation and normalization for training
    # Just normalization for validation
    data_transforms = {
        'train': transforms.Compose([
            transforms.RandomResizedCrop(224),
            transforms.RandomHorizontalFlip(),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),
        'val': transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ]),
    }

    data_dir = 'hymenoptera_data'
    image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                              data_transforms[x])
                      for x in ['train', 'val']}
    dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                                 shuffle=True, num_workers=4)
                  for x in ['train', 'val']}
    dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
    class_names = image_datasets['train'].classes

    device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")

    ######################################################################
    # Visualize a few images
    # ^^^^^^^^^^^^^^^^^^^^^^
    # Let's visualize a few training images so as to understand the data
    # augmentations.

    def imshow(inp, title=None):
        """Imshow for Tensor."""
        inp = inp.numpy().transpose((1, 2, 0))
        mean = np.array([0.485, 0.456, 0.406])
        std = np.array([0.229, 0.224, 0.225])
        inp = std * inp + mean
        inp = np.clip(inp, 0, 1)
        plt.imshow(inp)
        if title is not None:
            plt.title(title)
        plt.pause(0.001)  # pause a bit so that plots are updated

    # Get a batch of training data
    inputs, classes = next(iter(dataloaders['train']))

    # Make a grid from batch
    out = torchvision.utils.make_grid(inputs)

    imshow(out, title=[class_names[x] for x in classes])

    ######################################################################
    # Training the model
    # ------------------
    #
    # Now, let's write a general function to train a model. Here, we will
    # illustrate:
    #
    # -  Scheduling the learning rate
    # -  Saving the best model
    #
    # In the following, parameter ``scheduler`` is an LR scheduler object from
    # ``torch.optim.lr_scheduler``.

    def train_model(model, criterion, optimizer, scheduler, num_epochs=25):
        since = time.time()

        best_model_wts = copy.deepcopy(model.state_dict())
        best_acc = 0.0

        for epoch in range(num_epochs):
            print('Epoch {}/{}'.format(epoch, num_epochs - 1))
            print('-' * 10)

            # Each epoch has a training and validation phase
            for phase in ['train', 'val']:
                if phase == 'train':
                    scheduler.step()
                    model.train()  # Set model to training mode
                else:
                    model.eval()   # Set model to evaluate mode

                running_loss = 0.0
                running_corrects = 0

                # Iterate over data.
                for inputs, labels in dataloaders[phase]:
                    inputs = inputs.to(device)
                    labels = labels.to(device)

                    # zero the parameter gradients
                    optimizer.zero_grad()

                    # forward
                    # track history if only in train
                    with torch.set_grad_enabled(phase == 'train'):
                        outputs = model(inputs)
                        _, preds = torch.max(outputs, 1)
                        loss = criterion(outputs, labels)

                        # backward + optimize only if in training phase
                        if phase == 'train':
                            loss.backward()
                            optimizer.step()

                    # statistics
                    running_loss += loss.item() * inputs.size(0)
                    running_corrects += torch.sum(preds == labels.data)

                epoch_loss = running_loss / dataset_sizes[phase]
                epoch_acc = running_corrects.double() / dataset_sizes[phase]

                print('{} Loss: {:.4f} Acc: {:.4f}'.format(
                    phase, epoch_loss, epoch_acc))

                # deep copy the model
                if phase == 'val' and epoch_acc > best_acc:
                    best_acc = epoch_acc
                    best_model_wts = copy.deepcopy(model.state_dict())

            print()

        time_elapsed = time.time() - since
        print('Training complete in {:.0f}m {:.0f}s'.format(
            time_elapsed // 60, time_elapsed % 60))
        print('Best val Acc: {:4f}'.format(best_acc))

        # load best model weights
        model.load_state_dict(best_model_wts)
        return model

    ######################################################################
    # Visualizing the model predictions
    # ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    #
    # Generic function to display predictions for a few images
    #

    def visualize_model(model, num_images=6):
        was_training = model.training
        model.eval()
        images_so_far = 0
        fig = plt.figure()

        with torch.no_grad():
            for i, (inputs, labels) in enumerate(dataloaders['val']):
                inputs = inputs.to(device)
                labels = labels.to(device)

                outputs = model(inputs)
                _, preds = torch.max(outputs, 1)

                for j in range(inputs.size()[0]):
                    images_so_far += 1
                    ax = plt.subplot(num_images//2, 2, images_so_far)
                    ax.axis('off')
                    ax.set_title('predicted: {}'.format(class_names[preds[j]]))
                    imshow(inputs.cpu().data[j])

                    if images_so_far == num_images:
                        model.train(mode=was_training)
                        return
            model.train(mode=was_training)

    ######################################################################
    # Finetuning the convnet
    # ----------------------
    #
    # Load a pretrained model and reset final fully connected layer.
    #

    model_ft = models.resnet18(pretrained=True)
    num_ftrs = model_ft.fc.in_features
    model_ft.fc = nn.Linear(num_ftrs, 2)

    model_ft = model_ft.to(device)

    criterion = nn.CrossEntropyLoss()

    # Observe that all parameters are being optimized
    optimizer_ft = optim.SGD(model_ft.parameters(), lr=0.001, momentum=0.9)

    # Decay LR by a factor of 0.1 every 7 epochs
    exp_lr_scheduler = lr_scheduler.StepLR(optimizer_ft, step_size=7, gamma=0.1)

    ######################################################################
    # Train and evaluate
    # ^^^^^^^^^^^^^^^^^^
    #
    # It should take around 15-25 min on CPU. On GPU though, it takes less than a
    # minute.
    #

    model_ft = train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,
                           num_epochs=25)

    ######################################################################
    #

    visualize_model(model_ft)

    ######################################################################
    # ConvNet as fixed feature extractor
    # ----------------------------------
    #
    # Here, we need to freeze all the network except the final layer. We need
    # to set ``requires_grad == False`` to freeze the parameters so that the
    # gradients are not computed in ``backward()``.
    #
    # You can read more about this in the documentation
    # `here <http://pytorch.org/docs/notes/autograd.html#excluding-subgraphs-from-backward>`__.
    #

    model_conv = torchvision.models.resnet18(pretrained=True)
    for param in model_conv.parameters():
        param.requires_grad = False

    # Parameters of newly constructed modules have requires_grad=True by default
    num_ftrs = model_conv.fc.in_features
    model_conv.fc = nn.Linear(num_ftrs, 2)

    model_conv = model_conv.to(device)

    criterion = nn.CrossEntropyLoss()

    # Observe that only parameters of final layer are being optimized as
    # opoosed to before.
    optimizer_conv = optim.SGD(model_conv.fc.parameters(), lr=0.001, momentum=0.9)

    # Decay LR by a factor of 0.1 every 7 epochs
    exp_lr_scheduler = lr_scheduler.StepLR(optimizer_conv, step_size=7, gamma=0.1)

    ######################################################################
    # Train and evaluate
    # ^^^^^^^^^^^^^^^^^^
    #
    # On CPU this will take about half the time compared to previous scenario.
    # This is expected as gradients don't need to be computed for most of the
    # network. However, forward does need to be computed.
    #

    model_conv = train_model(model_conv, criterion, optimizer_conv,
                             exp_lr_scheduler, num_epochs=25)

    ######################################################################
    #

    visualize_model(model_conv)

    plt.ioff()
    plt.show()

Thanks every body!


Each time I am getting this error while resizing the images.
Here is the code.

why I am getting this error?

import models as m
import densenet as x
from torch.autograd import Variable
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader
from torchvision.datasets import ImageFolder
from torchvision.transforms import transforms
#import matplotlib.pyplot as plt
import torch

from sklearn.metrics import confusion_matrix
import numpy as np

tr1=  transforms.Compose([transforms.Resize((224,224))])

train_dataset = ImageFolder('../../cat_dog_dataset /training_set',transform=tr1)
train_loader = DataLoader(train_dataset,
                      batch_size = 1,
                      shuffle = True,
                      num_workers = 1)

test_dataset = ImageFolder('../../cat_dog_dataset/test_set',transform=tr1)
test_loader = DataLoader(test_dataset,
                      batch_size = 1,
                      shuffle = False,
                      num_workers = 1)

# GLOBAL VARIABLES
noe = 25
noc = 2

# MODEL INSTANCE 1
net = m.resnet50()
net.fc = nn.Linear(2048,noc)
#net=net.cuda()

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters())

def train(epoch):
    net.train()

    train_loss_epoch=0.0
    #    running_loss=0
    for batch in train_loader:

        inputs, labels = batch
        inputs, labels = Variable(inputs), Variable(labels)
        optimizer.zero_grad()
        #        print(inputs)
        # FORWARD PASS

        out = net (inputs)
        # CALCULATE LOSS
        loss = criterion(out, labels)
        # BACK PROPAGATION
        loss.backward()
        # WEIGHT UPDATION
        optimizer.step()
#        print(loss.data[0])

        train_loss_epoch+=loss.item()
#    total_loss_epoch+=t
    print(epoch,train_loss_epoch)

def test():
    final_updated_cm=np.zeros((2,2),dtype=int)
#    confusion_matrix = meter.ConfusionMeter(3)
    net.eval()
    test_loss=0.0
    correct=0.0
    for inputs,labels in test_loader:
 #           updated_cm=torch.zeros(23,23)
        inputs, labels = Variable(inputs), Variable(labels)
        out = net (inputs)
        #print(out)
        loss=criterion(out,labels)
        test_loss+=loss.item()

#        pred=out.data.max(1, keepdim=True)[1]
        pred=torch.max(out.data)
        #print(pred)
#        pred=torch.max(out.data,1)[1]
        correct+=pred.eq(labels.data.view_as(pred)).cpu().sum()
#        confusion_matrix.add(out.data.squeeze(), labels)
        cm=confusion_matrix(labels.data,pred.data,labels=[0,1])
        final_updated_cm=final_updated_cm+cm
    test_loss /=len(test_loader.dataset)
    print('\nTest Set: Average loss: {:.4f},Accuracy: {}/{}  ({:.0f}%)\n'.format(test_loss,correct,len(test_loader.dataset),
       100.0 * correct / len(test_loader.dataset)))

#    t=(100.0*correct/len(test_loader.dataset)).item()
#    f=open('2.txt','w')
#    f.write(str(t))
#    f.close()
#    print(t)

    print(final_updated_cm)

for epoch in range(noe):
    train(epoch)
test()

Error I am getting is:

AttributeError: Traceback (most recent call last):
    File "/home/ankur/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 57, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ankur/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 57, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/ankur/projects/Codes/Building_classifier_model/torchvision/datasets/folder.py", line 118, in __getitem__
    sample = self.transform(sample)
  File "/home/ankur/projects/Codes/Building_classifier_model/torchvision/transforms/transforms.py", line 49, in __call__
    img = t(img)
  File "/home/ankur/projects/Codes/Building_classifier_model/torchvision/transforms/transforms.py", line 175, in __call__
    return F.resize(img, self.size, self.interpolation)
AttributeError: module 'torch.nn.functional' has no attribute 'resize'


title: Copy a chunk of one tensor into another one in C++ API

body:
I need to copy a row of one tensor (in c++ API) into some part of another tensor, form which the begin and end indexes are available. Within C++ we can use something like:

int myints[] = {10, 20, 30, 40, 50, 60, 70};
std::vector<int> myvector(18);

std::copy(myints, myints + 3, myvector.begin() + 4);

to copy three values from myints into myvector, starting at the fourth index.
I was wondering if there is a similar API in libtorch (i.e., C++)?


title: Google Colab become slower with the same code sometimes. What is the possible reasons?

body:
I am training a CNN model with Google Colab's GPU through pytorch.  My question is, even though running with the same code, it gets about three times slower sometimes(30s -> 90s in my case).

I've tried restart runtime(it clears all local variable but keep files), it doesn't work

I have seen this    post, however, I've checked my GPU status, it works well with 11.5GB.

Sometimes it goes normal after disconnect it for a while though, I still want to figure out what can be the possible reason of it.


title: Concatenating two torch tensors of different shapes in pytorch

body:
I have two torch tensors. One with shape [64, 4, 300], and one with shape [64, 300]. How can I concatenate these two tensors to obtain the resultant tensor of shape [64, 5, 300]. I'm aware about the tensor.cat function used for this, but in order to use that function, I need to reshape the second tensor in order to match the number of dimensions of the tensor. I've heard that reshaping of the tensors should not be done, as it might mess up the data in the tensor. How can I do this concatenation?

I've tried reshaping, but following part makes me more doubtful about such reshaping.

a = torch.rand(64,300)

a1 = a.reshape(64,1,300)

list(a1[0]) == list(a)
Out[32]: False


title: 3D tensors expect 2 values for padding Torch

body:
I have succesfully trained and tested my model using this tutorial, and I want to test my model using single images. Here is my code

import torchvision.datasets as dset
import torchvision.transforms as transforms
from torch.utils.data import DataLoader,Dataset
import matplotlib.pyplot as plt
import torchvision.utils
import numpy as np
import random
from PIL import Image
import torch
from torch.autograd import Variable
import PIL.ImageOps
import torch.nn as nn
from torch import optim
import torch.nn.functional as F
import os

class Config():
    training_dir = "./data/faces/training/"
    testing_dir = "./data/faces/testing/"
    train_batch_size = 80
    train_number_epochs = 100

class SiameseNetworkDataset(Dataset):
    ...

class SiameseNetwork(nn.Module):
    ...

class ContrastiveLoss(torch.nn.Module):
    ...

if __name__=='__main__':
    net = SiameseNetwork().cuda()
    net.load_state_dict(torch.load("model.pt"))
    img0 = Image.open(os.path.join('data', 'faces', 'testing', 's5', '2.png'))
    img1 = Image.open(os.path.join('data', 'faces', 'testing', 's5', '1.png'))
    img0 = img0.convert("L")
    img1 = img1.convert("L")

    img0 = PIL.ImageOps.invert(img0)
    img1 = PIL.ImageOps.invert(img1)
    transform=transforms.Compose([transforms.Resize((100,100)),
                                    transforms.ToTensor()
                                    ])
    img0 = transform(img0)
    img1 = transform(img1)
    img0 = img0.cuda()
    img1 = img1.cuda()
    output1,output2 = net(Variable(img0).cuda(),Variable(img1).cuda()) //The error occurred here
    euclidean_distance = F.pairwise_distance(output1, output2)
    print(euclidean_distance.cpu().data.numpy())

And I have the following error: AssertionError: 3D tensors expect 2 values for padding. I can't understand, what is wrong, because I use the same preprocessing, which is used in SiameseNetworkDataset


title: Copying a PyTorch Variable to a Numpy array

body:
Suppose I have a PyTorch Variable in GPU:

var = Variable(torch.rand((100,100,100))).cuda()

What's the best way to copy (not bridge) this variable to a NumPy array?

var.clone().data.cpu().numpy()

or

var.data.cpu().numpy().copy()

By running a quick benchmark, .clone() was slightly faster than .copy(). However, .clone() + .numpy() will create a PyTorch Variable plus a NumPy bridge, while .copy() will create a NumPy bridge + a NumPy array.


title: How to apply bounds on a variable when performing optimisation in Pytorch?

body:
I am trying to use Pytorch for non-convex optimisation, trying to maximise my objective (so minimise in SGD). I would like to bound my dependent variable x > 0, and also have the sum of my x values be less than 1000.

I think I have the penalty implemented correctly in the form of a ramp penalty, but am struggling with the bounding of the x variable. In Pytorch you can set the bounds using clamp but it doesn't seem appropriate in this case. I think this is because optim needs the gradients free under the hood. Full working example:

import torch
from torch.autograd import Variable
import numpy as np

def objective(x, a, b, c):   # Want to maximise this quantity (so minimise in SGD)
    d = 1 / (1 + torch.exp(-a * (x)))

    # Checking constraint
    exceeded_limit = constraint(x).item()
    #print(exceeded_limit)

    obj = torch.sum(d * (b * c - x))

    # If overlimit add ramp penalty
    if  exceeded_limit < 0:
        obj = obj - (exceeded_limit * 10)
        print("Exceeded limit")

    return - obj

def constraint(x, limit = 1000): # Must be > 0
    return limit - x.sum()

N = 1000

# x is variable to optimise for
x = Variable(torch.Tensor([1 for ii in range(N)]), requires_grad=True)
a = Variable(torch.Tensor(np.random.uniform(0,100,N)), requires_grad=True)
b = Variable(torch.Tensor(np.random.rand(N)), requires_grad=True)
c = Variable(torch.Tensor(np.random.rand(N)), requires_grad=True)

# Would like to include the clamp
# x = torch.clamp(x, min=0)

# Non-convex methodf
opt = torch.optim.SGD([x], lr=.01)

for i in range(10000):
    # Zeroing gradients
    opt.zero_grad()

    # Evaluating the objective
    obj = objective(x, a, b, c)

    # Calculate gradients
    obj.backward()
    opt.step()
    if i%1000==0:  print("Objective: %.1f" % -obj.item())

print("\nObjective: {}".format(-obj))
print("Limit: {}".format(constraint(x).item()))

if torch.sum(x<0) > 0: print("Bounds not met")
if  constraint(x).item() < 0: print("Constraint not met")

Any suggestions as to how to impose the bounds would be appreciated, either using clamp or otherwise. Or generally advice on non-convex optimisation using Pytorch. This is a much simpler and scaled down version of the problem I'm working so am trying to find a lightweight solution if possible. I am considering using a workaround such as transforming the x variable using an exponential function but then you'd have to scale the function to avoid the positive values becoming infinite, and I want some flexibility with being able to set the constraint.


title: How do you use next_functions[0][0] on grad_fn correctly in pytorch?

body:
I was given this nn structure in the offical pytorch tutorial:

  input
  -> conv2d -> relu -> maxpool2d -> conv2d -> relu -> maxpool2d
       -> view -> linear -> relu -> linear -> relu -> linear
       -> MSELoss
       -> loss

then an example of how to follow the grad backwards using built-in .grad_fn from Variable.

# Eg:
print(loss.grad_fn)  # MSELoss
print(loss.grad_fn.next_functions[0][0])  # Linear
print(loss.grad_fn.next_functions[0][0].next_functions[0][0])  # ReLU

So I thought I can reach the grad object for Conv2d by pasting next_function[0][0] 9 times because of the given examples but I got the error tuple out of index. So how can I index these backprop objects correctly?


title: ImportError: DLL load failed: The specified module could not be found. (Torch issues)

body:
I am pretty new to python (not so new to java). I am very interested in the AI/Machine learning aspect of python.. So I went to Udemy and got two courses.. One on python and another on AI. I fooled around with the python course then went to the AI course. One of the imports needed was (torch) So I installed the import via the anaconda prompt.. I tried to run the import and I got the error

Traceback (most recent call last):
  File "C:\Users\lukem\Anaconda3\lib\site-packages\IPython\core\interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-eb42ca6e4af3>", line 1, in <module>
    import torch
  File "C:\Program Files\JetBrains\PyCharm 2018.3.4\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
  File "C:\Users\lukem\Anaconda3\lib\site-packages\torch\__init__.py", line 84, in <module>
    from torch._C import *
  File "C:\Program Files\JetBrains\PyCharm 2018.3.4\helpers\pydev\_pydev_bundle\pydev_import_hook.py", line 21, in do_import
    module = self._system_import(name, *args, **kwargs)
ImportError: DLL load failed: The specified module could not be found.

I then tried MANY versions of python and different ways of installing torch (pip3/cond) nothing worked.. I then uninstalled everything and all of a sudden the base import torch worked but things like (import torch.n or random) didnt work. Then all of a sudden the import torch wasn't there. Makes sense since I did uninstall everything involving torch. Current situation is I have torch downloaded

pip3 install https://download.pytorch.org/whl/cpu/torch-1.0.1-cp37-cp37m-win_amd64.whl

and that gives me the error listed above.. I am just very stuck and I've literal wasted around 3 days searching and trying to figure everything out in general (Had issues with tensor and python versions ect)

Hopefully this all makes sense.. I've only posted on stackoverflow once before.


title: Making torch data set error "cannot import name 'read_data_sets' from 'dataloader'"

body:
I'm trying to create my own torch dataset class and I ran into this problem

Traceback (most recent call last):
  File "us_name_train.py", line 10, in <module>
    from dataloader.usname_dl import NameDataset
  File "C:\ProgramData\Anaconda3\lib\site-packages\dataloader\__init__.py", line 1, in <module>
    from dataloader import read_data_sets
ImportError: cannot import name 'read_data_sets' from 'dataloader' (C:\ProgramData\Anaconda3\lib\site-packages\dataloader\__init__.py)

I've seen people post about this problem, but I think mine is extra odd, because the solution is to change 'dataloader' to 'DataLoader' cause it's a typo that was supposedly fixed in 2018, but my file is actually called 'dataloader.py' in the torch library and when I look through the file I do see the 'read_data_sets' function.

Also when I do change 'import dataloader' to 'import DataLoader' it says it can't find the module, but with 'import dataloader' it finds the module just can't find the function 'read_data_sets.' Other people had this problem cause they created their own module called dataloader, but I definitely don't have anything named 'dataloader' in my project dir. Anyone else deal with this issue?


title: Pytorch,How to feed output of CNN into input of RNN?

body:
I am new to CNN, RNN and deep learning.
I am trying to make architecture that will combine CNN and RNN.
input image size = [20,3,48,48]
a CNN output size = [20,64,48,48]
and now i want cnn ouput to be RNN input
but as I know the input of RNN must be 3-dimension only which is [seq_len, batch, input_size]
How can I make 4-dimensional [20,64,48,48] tensor into 3-dimensional for RNN input?

and another question how do I initiate the first hidden state with

torch.zeros()

I don't know what exact information I should pass in this function. the only thing that I know it is

[layer_dim, batch, hidden_dim]

Thank you.


title: Batch-wise beam search in pytorch

body:
I'm trying to implement a beam search decoding strategy in a text generation model. This is the function that I am using to decode the output probabilities.
def beam_search_decoder(data, k):
    sequences = [[list(), 0.0]]
    # walk over each step in sequence
    for row in data:
        all_candidates = list()
        for i in range(len(sequences)):
            seq, score = sequences[i]
            for j in range(len(row)):
                candidate = [seq + [j], score - torch.log(row[j])]
                all_candidates.append(candidate)
        # sort candidates by score
        ordered = sorted(all_candidates, key=lambda tup:tup[1])
        sequences = ordered[:k]
    return sequences

Now you can see this function is implemented with batch_size 1 in mind. Adding another loop for batch size would make the algorithm O(n^4). It is slow as it is now. Is there any way to improve the speed of this function. My model output is usually of the size (32, 150, 9907) which follows the format (batch_size, max_len, vocab_size)


title: Incompletable PyTorch with any CUDA version (module 'torch' has no attribute 'cuda')

body:
I have NVidia 1080TI, Ubuntu x64, and Python 3.6.9 installed.
I was trying to launch PyTorch with command
import torch
print(torch.cuda.is_available)

and expected to see 'True' but met the error:
AttributeError: module 'torch' has no attribute 'cuda'

I tried to update PyTorch and install the last version 1.7.0 with CUDA 11.0 support. After that, I noticed some version discrepancies. nvidia-smi shows CUDA version 11.0 but nvcc -V shows 9.1. Also, I used cat /usr/local/cuda/version.txt to check CUDA version but got the error: cat: /usr/local/cuda/version.txt: No such file or directory
I installed CUDA driver 450.33 after fully nvidia purging but the error remains and nvcc -V still shows 9.1 version (after reboot also).
One more option I addressed to is conda installation but it didn't help.
What I can do to resolve the problem?


title: Cannot deploy trained model to Google Cloud Ai-Platform with custom prediction routine: Model requires more memory than allowed

body:
I am trying to deploy a pretrained pytorch model to AI Platform with a custom prediction routine. After following the instructions described here the deployment fails with the following error:

ERROR: (gcloud.beta.ai-platform.versions.create) Create Version failed. Bad model detected with error: Model requires more memory than allowed. Please try to decrease the model size and re-deploy. If you continue to have error, please contact Cloud ML.

The contents of the model folder are 83.89 MB large and are below the 250 MB limit that's described in the documentation. The only files in the folder are the checkpoint file (.pth) for the model and the tarball required for the custom prediction routine.

Command to create the model:

gcloud beta ai-platform versions create pose_pytorch --model pose --runtime-version 1.15 --python-version 3.5 --origin gs://rcg-models/pytorch_pose_estimation --package-uris gs://rcg-models/pytorch_pose_estimation/my_custom_code-0.1.tar.gz --prediction-class predictor.MyPredictor

Changing the runtime version to 1.14 leads to the same error.
I have tried changing the --machine-type argument to mls1-c4-m2 like Parth suggested but I still get the same error.

The setup.py file that generates my_custom_code-0.1.tar.gz looks like this:

setup(
    name='my_custom_code',
    version='0.1',
    scripts=['predictor.py'],
    install_requires=["opencv-python", "torch"]
)

Relevant code snippet from the predictor:

    def __init__(self, model):
        """Stores artifacts for prediction. Only initialized via `from_path`.
        """
        self._model = model
        self._client = storage.Client()

    @classmethod
    def from_path(cls, model_dir):
        """Creates an instance of MyPredictor using the given path.

        This loads artifacts that have been copied from your model directory in
        Cloud Storage. MyPredictor uses them during prediction.

        Args:
            model_dir: The local directory that contains the trained Keras
                model and the pickled preprocessor instance. These are copied
                from the Cloud Storage model directory you provide when you
                deploy a version resource.

        Returns:
            An instance of `MyPredictor`.
        """

        net = PoseEstimationWithMobileNet()
        checkpoint_path = os.path.join(model_dir, "checkpoint_iter_370000.pth")
        checkpoint = torch.load(checkpoint_path, map_location='cpu')
        load_state(net, checkpoint)

        return cls(net)

Additionally I have enabled logging for the model in AI Platform and I get the following outputs:

2019-12-17T09:28:06.208537Z OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
2019-12-17T09:28:13.474653Z WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/google/cloud/ml/prediction/frameworks/tf_prediction_lib.py:48: The name tf.saved_model.tag_constants.SERVING is deprecated. Please use tf.saved_model.SERVING instead.
2019-12-17T09:28:13.474680Z {"textPayload":"","insertId":"5df89fad00073e383ced472a","resource":{"type":"cloudml_model_version","labels":{"project_id":"rcg-shopper","region":"","version_id":"lightweight_pose_pytorch","model_id":"pose"}},"timestamp":"2019-12-17T09:28:13.474680Z","logName":"projects/rcg-shopper/logs/ml.googleapis…
2019-12-17T09:28:13.474807Z WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/google/cloud/ml/prediction/frameworks/tf_prediction_lib.py:50: The name tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY is deprecated. Please use tf.saved_model.DEFAULT_SERVING_SIGNATURE_DEF_KEY instead.
2019-12-17T09:28:13.474829Z {"textPayload":"","insertId":"5df89fad00073ecd4836d6aa","resource":{"type":"cloudml_model_version","labels":{"project_id":"rcg-shopper","region":"","version_id":"lightweight_pose_pytorch","model_id":"pose"}},"timestamp":"2019-12-17T09:28:13.474829Z","logName":"projects/rcg-shopper/logs/ml.googleapis…
2019-12-17T09:28:13.474918Z WARNING:tensorflow:
2019-12-17T09:28:13.474927Z The TensorFlow contrib module will not be included in TensorFlow 2.0.
2019-12-17T09:28:13.474934Z For more information, please see:
2019-12-17T09:28:13.474941Z   * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
2019-12-17T09:28:13.474951Z   * https://github.com/tensorflow/addons
2019-12-17T09:28:13.474958Z   * https://github.com/tensorflow/io (for I/O related ops)
2019-12-17T09:28:13.474964Z If you depend on functionality not listed there, please file an issue.
2019-12-17T09:28:13.474999Z {"textPayload":"","insertId":"5df89fad00073f778735d7c3","resource":{"type":"cloudml_model_version","labels":{"version_id":"lightweight_pose_pytorch","model_id":"pose","project_id":"rcg-shopper","region":""}},"timestamp":"2019-12-17T09:28:13.474999Z","logName":"projects/rcg-shopper/logs/ml.googleapis…
2019-12-17T09:28:15.283483Z ERROR:root:Failed to import GA GRPC module. This is OK if the runtime version is 1.x
2019-12-17T09:28:16.890923Z Copying gs://cml-489210249453-1560169483791188/models/pose/lightweight_pose_pytorch/15316451609316207868/user_code/my_custom_code-0.1.tar.gz...
2019-12-17T09:28:16.891150Z / [0 files][    0.0 B/  8.4 KiB]
2019-12-17T09:28:17.007684Z / [1 files][  8.4 KiB/  8.4 KiB]
2019-12-17T09:28:17.009154Z Operation completed over 1 objects/8.4 KiB.
2019-12-17T09:28:18.953923Z Processing /tmp/custom_code/my_custom_code-0.1.tar.gz
2019-12-17T09:28:19.808897Z Collecting opencv-python
2019-12-17T09:28:19.868579Z   Downloading https://files.pythonhosted.org/packages/d8/38/60de02a4c9013b14478a3f681a62e003c7489d207160a4d7df8705a682e7/opencv_python-4.1.2.30-cp37-cp37m-manylinux1_x86_64.whl (28.3MB)
2019-12-17T09:28:21.537989Z Collecting torch
2019-12-17T09:28:21.552871Z   Downloading https://files.pythonhosted.org/packages/f9/34/2107f342d4493b7107a600ee16005b2870b5a0a5a165bdf5c5e7168a16a6/torch-1.3.1-cp37-cp37m-manylinux1_x86_64.whl (734.6MB)
2019-12-17T09:28:52.401619Z Collecting numpy>=1.14.5
2019-12-17T09:28:52.412714Z   Downloading https://files.pythonhosted.org/packages/9b/af/4fc72f9d38e43b092e91e5b8cb9956d25b2e3ff8c75aed95df5569e4734e/numpy-1.17.4-cp37-cp37m-manylinux1_x86_64.whl (20.0MB)
2019-12-17T09:28:53.550662Z Building wheels for collected packages: my-custom-code
2019-12-17T09:28:53.550689Z   Building wheel for my-custom-code (setup.py): started
2019-12-17T09:28:54.212558Z   Building wheel for my-custom-code (setup.py): finished with status 'done'
2019-12-17T09:28:54.215365Z   Created wheel for my-custom-code: filename=my_custom_code-0.1-cp37-none-any.whl size=7791 sha256=fd9ecd472a6a24335fd24abe930a4e7d909e04bdc4cf770989143d92e7023f77
2019-12-17T09:28:54.215482Z   Stored in directory: /tmp/pip-ephem-wheel-cache-i7sb0bmb/wheels/0d/6e/ba/bbee16521304fc5b017fa014665b9cae28da7943275a3e4b89
2019-12-17T09:28:54.222017Z Successfully built my-custom-code
2019-12-17T09:28:54.650218Z Installing collected packages: numpy, opencv-python, torch, my-custom-code


title: Error while running Convolutional Autoencoder RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

body:
I am a noob and am creating a model in PyTorch for the first time. I am trying to create a convolutional autoencoder and am getting the error while running the model. The code I am using is:

class MyDataset(Dataset):
    def __init__(self, image_paths, target_paths, train=True):
        self.image_paths = image_paths
        self.target_paths = target_paths

    def transform(self, image, target):
        # Transform to tensor
        resize = transforms.Resize(size=(2350,1650))
        image = resize(image)
        target = resize(target)
        grayscale = transforms.Grayscale(1)
        image = grayscale(image)
        target = grayscale(target)
        image = TF.to_tensor(image)
        target = TF.to_tensor(target)
        return image, target

    def __getitem__(self, index):
        image = Image.open(self.image_paths[index])
        target = Image.open(self.target_paths[index])
        x, y = self.transform(image, target)
        return x, y

    def __len__(self):
        return len(self.image_paths)

traindata = MyDataset(image_paths=train_data, target_paths=target_data, train=True)
testdata = MyDataset(image_paths=test_data, target_paths=None, train=False)

train_loader = DataLoader(traindata, batch_size=1, shuffle=True, num_workers=4)
test_loader = DataLoader(testdata, batch_size=1, shuffle=False, num_workers=4)

class ConvolutionalAutoEncoder(nn.Module):
    def __init__(self):
        super(ConvolutionalAutoEncoder, self).__init__()
        self.encoder_block1 = nn.Sequential(
            nn.Conv2d(1, 64, 3, stride=1, padding=1),
            nn.ReLU(True),
            nn.Conv2d(64, 64, 3, stride=1, padding=1),
            nn.ReLU(True)
        )
        self.decoder_block1 = nn.Sequential(
            nn.ConvTranspose2d(64, 64, 3, stride=1, padding=1),
            nn.ReLU(True),
            nn.ConvTranspose2d(64, 64, 3, stride=1, padding=1),
            nn.ReLU(True)
         )
        self.decoder_block0 = nn.Sequential(
            nn.ConvTranspose2d(64, 1, 3, stride=1, padding=1),
            nn.Sigmoid()
        )
    def forward(self, x):
        x1 = self.encoder_block1(x)
        y1 = self.decoder_block1(x1)
        y0 = self.decoder_block0(y1)
        return x

device = torch.device("cuda:2" if torch.cuda.is_available() else "cpu")
print(device)

model = ConvolutionalAutoEncoder().to(device)
# Loss and optimizer
learning_rate = 0.001
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=learning_rate)

params = list(model.parameters())
print(len(params))
print(params[0].size())  # conv1's .weight

num_epochs = 30
total_step = len(train_loader)
for epoch in range(num_epochs):
    for batch_idx, data in enumerate(train_loader):
        inp, targ = data
        inp = inp.to(device)
        targ = targ.to(device)

        output = model(inp)
        loss = criterion(output, targ)

        model.zero_grad()
        loss.backward()
        optimizer.step()

        # print statistics
        running_loss += loss.item()

        if (batch_idx+1) % 10 == 0:
            print ('Epoch [{}/{}], Step [{}/{}], Loss: {:.4f}'
                   .format(epoch+1, num_epochs, i+1, total_step, loss.item()))

The full error I am getting is:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-17-28fa0c94d845> in <module>
     13
     14         model.zero_grad()
---> 15         loss.backward()
     16         optimizer.step()
     17

~/anaconda3/envs/gautam_new/lib/python3.6/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
     91                 products. Defaults to ``False``.
     92         """
---> 93         torch.autograd.backward(self, gradient, retain_graph, create_graph)
     94
     95     def register_hook(self, hook):

~/anaconda3/envs/gautam_new/lib/python3.6/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     88     Variable._execution_engine.run_backward(
     89         tensors, grad_tensors, retain_graph, create_graph,
---> 90         allow_unreachable=True)  # allow_unreachable flag
     91
     92

RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn

Please help. Also, if possible also advice on how I can make my model deeper. I keep getting CUDA  out of memory error.

Thanks.


title: Measuring uncertainty using MC Dropout on pytorch

body:
I am trying to implement Bayesian CNN using Mc Dropout on Pytorch,
the main idea is that by applying dropout at test time and running over many forward passes , you get predictions from a variety of different models.
I’ve found an application of the Mc Dropout and I really did not get how they applied this method and how exactly they did choose the correct prediction from the list of predictions
here is the code

 def mcdropout_test(model):
    model.train()
    test_loss = 0
    correct = 0
    T = 100
    for data, target in test_loader:
        if args.cuda:
            data, target = data.cuda(), target.cuda()
        data, target = Variable(data, volatile=True), Variable(target)
        output_list = []
        for i in xrange(T):
            output_list.append(torch.unsqueeze(model(data), 0))
        output_mean = torch.cat(output_list, 0).mean(0)
        test_loss += F.nll_loss(F.log_softmax(output_mean), target, size_average=False).data[0]  # sum up batch loss
        pred = output_mean.data.max(1, keepdim=True)[1]  # get the index of the max log-probability
        correct += pred.eq(target.data.view_as(pred)).cpu().sum()

    test_loss /= len(test_loader.dataset)
    print('\nMC Dropout Test set: Average loss: {:.4f}, Accuracy: {}/{} ({:.2f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

    train()
    mcdropout_test()

I have replaced

data, target = Variable(data, volatile=True), Variable(target)

by adding

with torch.no_grad():  at the beginning

And this is how I have defined my CNN
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 192, 5, padding=2)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(192, 192, 5, padding=2)
        self.fc1 = nn.Linear(192 * 8 * 8, 1024)
        self.fc2 = nn.Linear(1024, 256)
        self.fc3 = nn.Linear(256, 10)
        self.dropout = nn.Dropout(p=0.3)

        nn.init.xavier_uniform_(self.conv1.weight)
        nn.init.constant_(self.conv1.bias, 0.0)
        nn.init.xavier_uniform_(self.conv2.weight)
        nn.init.constant_(self.conv2.bias, 0.0)
        nn.init.xavier_uniform_(self.fc1.weight)
        nn.init.constant_(self.fc1.bias, 0.0)
        nn.init.xavier_uniform_(self.fc2.weight)
        nn.init.constant_(self.fc2.bias, 0.0)
        nn.init.xavier_uniform_(self.fc3.weight)
        nn.init.constant_(self.fc3.bias, 0.0)

    def forward(self, x):
        x = self.pool(F.relu(self.dropout(self.conv1(x))))  # recommended to add the relu
        x = self.pool(F.relu(self.dropout(self.conv2(x))))  # recommended to add the relu
        x = x.view(-1, 192 * 8 * 8)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(self.dropout(x)))
        x = self.fc3(self.dropout(x))  # no activation function needed for the last layer
        return x

Can anyone help me to get the right implementation of the Monte Carlo Dropout method on CNN?


title: Pytorch NLP model doesn’t use GPU when making inference

body:
I have a NLP model trained on Pytorch to be run in Jetson Xavier. I installed Jetson stats to monitor usage of CPU and GPU. When I run the Python script, only CPU cores work on-load, GPU bar does not increase. I have searched on Google about that with keywords of " How to check if pytorch is using the GPU?" and checked results on stackoverflow.com etc. According to their advices to someone else facing similar issue, cuda is available and there is cuda device in my Jetson Xavier. However, I don’t understand why GPU bar does not change, CPU core bars go to the ends.
I don’t want to use CPU, it takes so long to compute. In my opinion, it uses CPU, not GPU. How can I be sure and if it uses CPU, how can I change it to GPU?
Note: Model is taken from huggingface transformers library. I have tried to use cuda() method on the model. (model.cuda()) In this scenario, GPU is used but I can not get an output from model and raises exception.
Here is the code:
from transformers import AutoTokenizer, AutoModelForQuestionAnswering, pipeline
import torch

BERT_DIR = "savasy/bert-base-turkish-squad"

tokenizer = AutoTokenizer.from_pretrained(BERT_DIR)
model = AutoModelForQuestionAnswering.from_pretrained(BERT_DIR)
nlp=pipeline("question-answering", model=model, tokenizer=tokenizer)

def infer(question,corpus):
    try:
        ans = nlp(question=question, context=corpus)
        return ans["answer"], ans["score"]
    except:
        ans = None
        pass

    return None, 0


I have tried installing CUDA on Windows 10 to train nerual networks in my GPU(NVIDIA GeForce 710) but I get the following error when I tried to load an initial model.
Here's the code I am running:
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

device
"""# 1. Create the classifier"""

C = nn.Sequential(Flatten(), nn.Linear(784,200), nn.ReLU(),
                            nn.Linear(200,100), nn.ReLU(),
                            nn.Linear(100,100), nn.ReLU(),
                            nn.Linear(100,10))

"""Upload the trained model"""

C.load_state_dict(torch.load("C.pt",map_location='cuda'))

And this is the error I am getting:
C.load_state_dict(torch.load("C.pt",map_location='cuda'))
Traceback (most recent call last):

  File "<ipython-input-2-358f76f483ed>", line 1, in <module>
    C.load_state_dict(torch.load("C.pt",map_location='cuda'))

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\serialization.py", line 368, in load
    return _load(f, map_location, pickle_module)

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\serialization.py", line 542, in _load
    result = unpickler.load()

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\serialization.py", line 505, in persistent_load
    data_type(size), location)

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\serialization.py", line 385, in restore_location
    return default_restore_location(storage, map_location)

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\serialization.py", line 114, in default_restore_location
    result = fn(storage, location)

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\serialization.py", line 96, in _cuda_deserialize
    return obj.cuda(device)

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\_utils.py", line 68, in _cuda
    with torch.cuda.device(device):

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\cuda\__init__.py", line 229, in __enter__
    _lazy_init()

  File "C:\Users\usuario\anaconda3\lib\site-packages\torch\cuda\__init__.py", line 162, in _lazy_init
    torch._C._cuda_init()

RuntimeError: cuda runtime error (30) : unknown error at ..\aten\src\THC\THCGeneral.cpp:87

I already installed cudNN and these are the versions I am using.
Python 3.7.6
CUDA 8.0
Pytorch 1.0.1


title: PyTorch - applying attention efficiently

body:
I have build a RNN language model with attention and I am creating context vector for every element of the input by attending all the previous hidden states (only one direction).

The most straight forward solution in my opinion is using a for-loop over the RNN output, such that each context vector is computed one after another.

import torch
import torch.nn as nn
import torch.nn.functional as F

class RNN_LM(nn.Module):
    def __init__(self, hidden_size, vocab_size, embedding_dim=None, droprate=0.5):
        super().__init__()
        if not embedding_dim:
            embedding_dim = hidden_size
        self.embedding_matrix = nn.Embedding(vocab_size, embedding_dim)

        self.lstm = nn.LSTM(input_size=embedding_dim, hidden_size=hidden_size, batch_first=False)
        self.attn = nn.Linear(hidden_size, hidden_size)
        self.vocab_dist = nn.Linear(hidden_size, vocab_size)
        self.dropout = nn.Dropout(droprate)

    def forward(self, x):
        x = self.dropout(self.embedding_matrix(x.view(-1, 1)))
        x, states = self.lstm(x)
        #print(x.size())
        x = x.squeeze()
        content_vectors = [x[0].view(1, -1)]
        # for-loop over hidden states and attention
        for i in range(1, x.size(0)):
            prev_states = x[:i]
            current_state = x[i].view(1, -1)

            attn_prod = torch.mm(self.attn(current_state), prev_states.t())
            attn_weights = F.softmax(attn_prod, dim=1)
            context = torch.mm(attn_weights, prev_states)
            content_vectors.append(context)

        return self.vocab_dist(self.dropout(torch.cat(content_vectors)))

Note: The forward method here is only used for training.

However this solution is not very efficient as the code is not well parallelizable with computing each context vector sequently. But since the context vectors are not dependent on each other, I wonder if there is a non-sequential way of calculating them.

So is there is a way to compute the context vectors without for-loop so that more of computation can be parallelized?