title: How to use scripting to convert pytorch transformer?

body:
I am trying to compile pytorch transformer to run it in C++:

from torch.nn import TransformerEncoder, TransformerEncoderLayer
encoder_layers = TransformerEncoderLayer(1000, 8, 512, 0.1)
transf = TransformerEncoder(encoder_layers, 6)
sm = torch.jit.script(transf)

But I am getting an error:

RuntimeError:  Expected a default value of type Tensor on parameter
"src_mask":   File "C:\Program Files (x86)\Microsoft Visual
Studio\Shared\Python36_64\lib\site-packages\torch\nn\modules\transformer.py",
line 271
    def forward(self, src, src_mask=None, src_key_padding_mask=None):
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~... <--- HERE
        r"""Pass the input through the encoder layer.

It looks like something wrong with pytorch transformer module.

Is there any way to run pytorch transformer in C++ ?


title: Implementing WNGrad in Pytorch?

body:
I'm trying to implement the WNGrad (technically WN-Adam, algorithm 4 in the paper) optimizier (WNGrad) in pytorch.  I've never implemented an optimizer in pytorch before so I don't know if I've done it correctly (I started from the adam implementation).  The optimizer does not make much progress and falls down like I would expect (bj values can only monotonically increase, which happens quickly so no progress is made) but I'm guessing I have a bug.  Standard optimizers (Adam, SGD) work fine on the same model I'm trying to optimize.

Does this implementation look correct?

from torch.optim import Optimizer

class WNAdam(Optimizer):
    """Implements WNAdam algorithm.

    It has been proposed in `WNGrad: Learn the Learning Rate in Gradient Descent`_.

    Arguments:
        params (iterable): iterable of parameters to optimize or dicts defining
            parameter groups
        lr (float, optional): learning rate (default: 0.1)
        beta1 (float, optional): exponential smoothing coefficient for gradient.
                                When beta=0 this implements WNGrad.

    .. _WNGrad\: Learn the Learning Rate in Gradient Descent:
        https://arxiv.org/abs/1803.02865
    """

    def __init__(self, params, lr=0.1, beta1=0.9):
        if not 0.0 <= beta1 < 1.0:
            raise ValueError("Invalid beta1 parameter: {}".format(beta1))
        defaults = dict(lr=lr, beta1=beta1)
        super().__init__(params, defaults)

    def step(self, closure=None):
        """Performs a single optimization step.

        Arguments:
            closure (callable, optional): A closure that reevaluates the model
                and returns the loss.
        """
        loss = None
        if closure is not None:
            loss = closure()

        for group in self.param_groups:
            for p in group['params']:
                if p.grad is None:
                    continue
                grad = p.grad.data
                state = self.state[p]

                # State initialization
                if len(state) == 0:
                    state['step'] = 0
                    # Exponential moving average of gradient values
                    state['exp_avg'] = torch.zeros_like(p.data)
                    # Learning rate adjustment
                    state['bj'] = 1.0

                exp_avg = state['exp_avg']
                beta1 = group['beta1']

                state['step'] += 1
                state['bj'] += (group['lr']**2)/(state['bj'])*grad.pow(2).sum()

                # update exponential moving average
                exp_avg.mul_(beta1).add_(1 - beta1, grad)

                bias_correction = 1 - beta1 ** state['step']
                p.data.sub_(group['lr'] / state['bj'] / bias_correction, exp_avg)
        return loss


title: Calling pytorch 0.4 from Matlab

body:
I'm trying to run a python a code that uses Pytorch 0.4 from Matlab, by using a system call (Matlab's "system" or "unix" commands), and I am getting the following error:

  Traceback (most recent call last):   File
  "/home/gil/PycharmProjects/test123/AugmentedTrain.py", line 3, in

      import torch   File "/home/gil/anaconda2/envs/pytorch/lib/python3.6/site-packages/torch/init.py",
  line 78, in
      from torch._C import * RuntimeError: stoi

According to what I see over the internet, it is a problem of pytorch 0.4 and indeed it does not occur in pytorch 0.3. Unfortunately, porting my code to 0.3 is a lot of work and does not feel like the right thing to do (going backward).

Can anyone help?


title: Is there a way to get batches with continuous examples in Pytorch

body:
I have a dataset with 10,000+ examples and using Dataloader, I create batches of size 50. I'm trying to find a way to have batch 1 start at example 1 and end at example 50 then have batch 2 start at example 2 and end at example 51 and so on.

This is a snip of where I use DataLoader:

train_loader = torch.utils.data.DataLoader(train, batch_size=batch_size, drop_last=True, shuffle=False)

    for epoch in range(num_epochs):
        totalEpochs += 1

        for X, y in train_loader:

            train = X.view(-1, 1, X.shape[1]).float()


title: How to multiply a tensor with a vector?

body:
I have 2 tensors

a = torch.tensor([1,2])
b = torch.tensor([[[10,20],
                   [30,40]],
                  [[1,2],
                   [3,4]]])

and I would like to combine them in such a way that

a ? b = tensor([[[10,20],
                 [30,40]],
                [[ 2, 4],
                 [ 6, 8]]])

(and then sum over the 0th dimension, in the end I want to do a weighted sum)

I've tried:

""" no idea how to interpret that """
a @ b
tensor([[ 70, 100],
        [  7,  10]])

b @ a
tensor([[ 50, 110],
        [  5,  11]])

for i in range(b.size()[0]): # works but I don't think this will work with autograd
      b[i] *= a[i]

a * b              # multiplies right side by 2
tensor([[[10, 40],
         [30, 80]],

        [[ 1,  4],
         [ 3,  8]]])

a.unsqueeze(1)     # multiplies bottom side by 2
tensor([[[10, 20],
         [60, 80]],

        [[ 1,  2],
         [ 6,  8]]])

a.unsqueeze(2) * b # dimension out of range


title: Why does torch.gt function turn requires_grad into False?

body:
The requires_grad of tensor b and c are True. But the requires_grad of tensor d is False. I am curious why this change happens because all the requires_grad of inputs are True.

However, the requires_grad of tensor e is True. I can still do backward() on e. But is there an error in this way?

I am using Python3.7 and Pytorch1.1.

import torch
import torch.nn as nn

net = nn.Conv2d(1, 1, 3, padding=1)
a = torch.randn(1, 1, 10, 10)
b = net(a)
c = net(b)

d = torch.gt(b, c)

e = b - c
e[e > 0] = 1.0
e[e < 0] = 0.0


title: In 2020 what is the optimal way to train a model in Pytorch on more than one GPU on one computer?

body:
What are the best practices for training one neural net on more than one GPU on one machine?
I'm a little confused by the different options from nn.DataParallel vs putting different layers on different GPUs with .to('cuda:0') and .to('cuda:1'). I see in the Pytorch docs the latter method the date was 2017. Is there a standard or does it depend on preference or the type of model?
Method 1
class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = torch.nn.Linear(10, 10)
        self.relu = torch.nn.ReLU()
        self.net2 = torch.nn.Linear(10, 5)

    def forward(self, x):
        x = self.relu(self.net1(x))
        return self.net2(x)

model = ToyModel().to('cuda')
model = nn.DataParallel(model)

Method 2
class ToyModel(nn.Module):
    def __init__(self):
        super(ToyModel, self).__init__()
        self.net1 = torch.nn.Linear(10, 10).to('cuda:0')
        self.relu = torch.nn.ReLU()
        self.net2 = torch.nn.Linear(10, 5).to('cuda:1')

    def forward(self, x):
        x = self.relu(self.net1(x.to('cuda:0')))
        return self.net2(x.to('cuda:1'))

I'm not sure there aren't more ways Pytorch provides to train on more than one GPU.
Both of these methods seem to cause my system to freeze depending on what model I use them. In Jupyter the cell stays at a [*] and if I don't restart the kernel the screen freezes and I have to do a hard reset. A few tutorials on multi-gpu cause my system to hang and freeze like this.


title: From 2D to 3D using convolutional autoencoder

body:
I’d like to reconstruct 3D object from 2D images.
For that, I try to use convolutional auto encoder. However, in which layer should I lift the dimensionality?

I wrote a code below, however, it shows an error:

  “RuntimeError: invalid argument 2: size ‘[1 x 1156 x 1156]’ is invalid for input of with 2312 elements at pytorch-src/torch/lib/TH/THStorage.c:41”

class dim_lifting(nn.Module):
    def __init__(self):
        super(dim_lifting, self).__init__()
        self.encode = nn.Sequential(
            nn.Conv2d(1, 34, kernel_size=5, padding=2),
            nn.MaxPool2d(2),
            nn.Conv2d(34, 16, kernel_size=5, padding=2),
            nn.MaxPool2d(2),
            nn.Conv2d(16, 8, kernel_size=5, padding=2),
            nn.MaxPool2d(2),
            nn.LeakyReLU()
        )

        self.fc1 = nn.Linear(2312, 2312)
        self.decode = nn.Sequential(
            nn.ConvTranspose3d(1, 16, kernel_size=5, padding=2),
            nn.LeakyReLU(),
            nn.ConvTranspose3d(16, 32, kernel_size=5, padding=2),
            nn.LeakyReLU(),
            nn.MaxPool2d(2))

    def forward(self, x):
        out = self.encode(x)
        out = out.view(out.size(0), -1)
        out = self.fc1(out)
        out = out.view(1, 1156, 1156)
        out = self.decode(out)
        return out

Error happens here

out = out.view(1, 1156, 1156)


title: Compile error when building doom_py on Ubuntu 18.04 LTS

body:
I've been trying to build doom_py on Ubuntu to no avail
Whenever I run python setup.py build command inside doom_py directory, and get the following errors.

Setup
- Ubuntu 18.04 LTS
- Python 2.7.14 (Anaconda 2)
- OpenAI Gym 0.9.5
- Vizdoom 1.1.4
- doom-py 0.0.14
- ppaquette/gym-doom

Errors

Scanning dependencies of target libvizdoom_static
[  0%] Building CXX object CMakeFiles/libvizdoom_static.dir/src/lib/ViZDoomController.cpp.o
[  0%] Building CXX object CMakeFiles/libvizdoom_static.dir/src/lib/ViZDoomExceptions.cpp.o
[  0%] Building CXX object CMakeFiles/libvizdoom_static.dir/src/lib/ViZDoomGame.cpp.o
/tmp/pip-build-fqKs1A/doom-py/doom_py/src/lib/ViZDoomGame.cpp: In member function ‘bool vizdoom::DoomGame::loadConfig(std::__cxx11::string)’:
/tmp/pip-build-fqKs1A/doom-py/doom_py/src/lib/ViZDoomGame.cpp:743:22: error: ‘cerr’ is not a member of ‘std’
                  std::cerr<<"WARNING! Loading config from: \""<<filename<<"\". Syntax erorr in line #"<<line_number<<". Line ignored.\n";
                      ^~~~
/tmp/pip-build-fqKs1A/doom-py/doom_py/src/lib/ViZDoomGame.cpp:743:22: note: suggested alternative: ‘erf’
                  std::cerr<<"WARNING! Loading config from: \""<<filename<<"\". Syntax erorr in line #"<<line_number<<". Line ignored.\n";
                      ^~~~
                      erf
/tmp/pip-build-fqKs1A/doom-py/doom_py/src/lib/ViZDoomGame.cpp:757:22: error: ‘cerr’ is not a member of ‘std’
                  std::cerr<<"WARNING! Loading config from: \""<<filename<<"\". Empty key in line #"<<line_number<<". Line ignored.\n";
                      ^~~~
/tmp/pip-build-fqKs1A/doom-py/doom_py/src/lib/ViZDoomGame.cpp:757:22: note: suggested alternative: ‘erf’
                  std::cerr<<"WARNING! Loading config from: \""<<filename<<"\". Empty key in line #"<<line_number<<". Line ignored.\n";
                      ^~~~
                      erf

                  erf
CMakeFiles/libvizdoom_static.dir/build.make:88: recipe for target 'CMakeFiles/libvizdoom_static.dir/src/lib/ViZDoomGame.cpp.o' failed
make[2]: *** [CMakeFiles/libvizdoom_static.dir/src/lib/ViZDoomGame.cpp.o] Error 1
CMakeFiles/Makefile2:178: recipe for target 'CMakeFiles/libvizdoom_static.dir/all' failed
make[1]: *** [CMakeFiles/libvizdoom_static.dir/all] Error 2
Makefile:83: recipe for target 'all' failed
make: *** [all] Error 2
[1m
Could not build doom-py: Command '['make', '-j', '1']' returned non-zero exit status 2. (HINT: are you sure cmake is installed? You might also be missing a library. Try running 'apt-get install -y python-numpy cmake zlib1g-dev libjpeg-dev libboost-all-dev gcc libsdl2-dev wget unzip'

[0mTraceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/tmp/pip-build-fqKs1A/doom-py/setup.py", line 88, in <module>
    include_package_data=True,
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/site-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/site-packages/setuptools/command/install.py", line 61, in run
    return orig.install.run(self)
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/distutils/command/install.py", line 563, in run
    self.run_command('build')
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/distutils/cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/tmp/pip-build-fqKs1A/doom-py/setup.py", line 63, in run
    build_func()
  File "/tmp/pip-build-fqKs1A/doom-py/setup.py", line 41, in build_linux
    build_common('so')
  File "/tmp/pip-build-fqKs1A/doom-py/setup.py", line 29, in build_common
    subprocess.check_call(['make', '-j', str(cores_to_use)], cwd='doom_py')
  File "/home/hybridsyntax/anaconda2/envs/doomenv/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['make', '-j', '1']' returned non-zero exit status 2

----------------------------------------

Thanks in advance


title: pytorch - modify embedding backpropagation

body:
I would like to modify the back-propagation for the embedding layer but I don't understand where the definition is.

In the definition available in https://pytorch.org/docs/stable/_modules/torch/nn/functional.html in the embedding function, they call torch.embedding and here there should be defined how the weights are updated.

So my question is:
Where can I find the documentation of torch.embedding?


title: How to view the weight_decay loss in pytoch during the training?

body:
optimizer = optim.Adam(model.parameters(), lr=args.lr, weight_decay=1.0)
optimizer.zero_grad()
loss.backward()
optimizer.step()

I train the model with Adam optimizer in pytorch and set the weight_decay parameter to 1.0.
If I want to compare the number of the weight_decay loss and the model loss, how to view the value of the loss caused by the weight_decay?


title: Pytorch - going back and forth between eval() and train() modes

body:
I'm studying "Deep Reinforcement Learning" and build my own example after pytorch's REINFORCEMENT LEARNING (DQN) TUTORIAL.

I'm implement actor's strategy as follows:
1. model.eval()
2. get best action from a model
3. self.net.train()

The question is:
Does going back and forth between eval() and train() modes cause any damage to optimization process?

The model includes only Linear and BatchNorm1d layers.
As far as I know when using BatchNorm1d one must perform model.eval() to use a model, because there is different results in eval() and train() modes.

When training Classification Neural Network the model.eval() performed only after training is finished, but in case of "Deep Reinforcement Learning" it is usual to use strategy and then continue the optimization process.

I'm wondering if going back and forth between modes is "harmless" to optimization process?

def strategy(self, state):
    # Explore or Exploit
    if self.epsilon > random():
        action = choice(self.actions)
    else:
        self.net.eval()
        action = self.net(state.unsqueeze(0)).max(1)[1].detach()
        self.net.train()


title: Runtime error when reading data from a training dataset PyTorch

body:
I have a sample of data in my training dataset which I am able to view if I print the data, but when accessing it to train the data, I keep getting RuntimeError: Expected object of scalar type Double but got scalar type Float for argument #2 'weight' in call to _thnn_conv2d_forward. I am unable to figure out why this is happening. I have also attached an image at the end to give a better understanding of the error message.

The labels.txt files looks as follows (name of image linking to another folder with corresponding images, center points (x, y) and radius)

0000,   0.67 ,   0.69 ,   0.26
0001,   0.69 ,   0.33 ,   0.3
0002,   0.16 ,   0.27 ,   0.15
0003,   0.54 ,   0.33 ,   0.17
0004,   0.32 ,   0.45 ,   0.3
0005,   0.78 ,   0.26 ,   0.17
0006,   0.44 ,   0.49 ,   0.19

EDIT: This is the loss function and optimizer I am using optimizer =

optim.Adam(model.parameters(), lr=0.001)
nn.CrossEntropyLoss()

My validate model function is as follows:

def validate_model(model, loader):
    model.eval() # eval mode (batchnorm uses moving mean/variance instead of mini-batch mean/variance)
                  # (dropout is set to zero)

    val_running_loss = 0.0
    val_running_correct = 0

    for int, data in enumerate(loader):
        data, target = data['image'].to(device), data['labels'].to(device)
        output = model(data)
        loss = my_loss(output, target)

        val_running_loss = val_running_loss + loss.item()
        _, preds = torch.max(output.data, 1)

        val_running_correct = val_running_correct+ (preds == target).sum().item()

    avg_loss = val_running_loss/len(loader.dataset)
    val_accuracy = 100. * val_running_correct/len(loader.dataset)

    #----------------------------------------------
    # implementation needed here
    #----------------------------------------------
    return avg_loss, val_accuracy

I have a fit function for working out training loss:

def fit(model, train_dataloader):
    model.train()
    train_running_loss = 0.0
    train_running_correct = 0
    for i, data in enumerate(train_dataloader):
        print(data)
        #I believe this is causing the error, but not sure why.
        data, target = data['image'].to(device), data['labels'].to(device)
        optimizer.zero_grad()
        output = model(data)
        loss = my_loss(output, target)
        train_running_loss = train_running_loss + loss.item()
        _, preds = torch.max(output.data, 1)
        train_running_correct =  train_running_correct + (preds == target).sum().item()
        loss.backward()
        optimizer.step()
    train_loss = train_running_loss/len(train_dataloader.dataset)
    train_accuracy = 100. * train_running_correct/len(train_dataloader.dataset)

    print(f'Train Loss: {train_loss:.4f}, Train Acc: {train_accuracy:.2f}')

    return train_loss, train_accuracy

and the following train_model function which stores the losses and accuracy in a list:

train_losses , train_accuracy = [], []
validation_losses , val_accuracy = [], []

def train_model(model,
                optimizer,
                train_loader,
                validation_loader,
                train_losses,
                validation_losses,
                epochs=1):

    """
    Trains a neural network.
    Args:
        model               - model to be trained
        optimizer           - optimizer used for training
        train_loader        - loader from which data for training comes
        validation_loader   - loader from which data for validation comes (maybe at the end, you use test_loader)
        train_losses        - adding train loss value to this list for future analysis
        validation_losses   - adding validation loss value to this list for future analysis
        epochs              - number of runs over the entire data set
    """

    #----------------------------------------------
    # implementation needed here
    #----------------------------------------------

    for epoch in range(epochs):
        train_epoch_loss, train_epoch_accuracy = fit(model, train_loader)
        val_epoch_loss, val_epoch_accuracy = validate_model(model, validation_loader)
        train_losses.append(train_epoch_loss)
        train_accuracy.append(train_epoch_accuracy)
        validation_losses.append(val_epoch_loss)
        val_accuracy.append(val_epoch_accuracy)

    return

And when I run the following code I get the Runtime Error:

train_model(model,
            optimizer,
            train_loader,
            validation_loader,
            train_losses,
            validation_losses,
            epochs=2)

  ERROR:  RuntimeError: Expected object of scalar type Double but got
  scalar type Float for argument #2 'weight' in call to
  _thnn_conv2d_forward

Here is a screenshot of the error message as well:
ERROR

EDIT: this is how my model looks like, I am supposed to detect circles in an image with the given centers and radius in the labels.txt file and paint over them - the painting function was given, I had create the model and the training and validation.

class CircleNet(nn.Module):    # nn.Module is parent class
    def __init__(self):
        super(CircleNet, self).__init__()  #calls init of parent class
        #----------------------------------------------
        # implementation needed here
        #----------------------------------------------
        #keep dimensions of input image: (I-F+2P)/S +1= (128-3+2)/1 + 1 = 128

        #RGB image = input channels = 3. Use 12 filters for first 2 convolution layers, then double
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=12, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(in_channels=12, out_channels=12, kernel_size=3, stride=1, padding=1)
        self.conv3 = nn.Conv2d(in_channels=12, out_channels=24, kernel_size=3, stride=1, padding=1)
        self.conv4 = nn.Conv2d(in_channels=24, out_channels=32, kernel_size=3, stride=1, padding=1)

        #Pooling to reduce sizes, and dropout to prevent overfitting
        self.pool = nn.MaxPool2d(kernel_size=2)
        self.relu = nn.ReLU()

        self.drop = nn.Dropout2d(p=0.25)
        self.norm1 = nn.BatchNorm2d(12)
        self.norm2 = nn.BatchNorm2d(24)

        # There are 2 pooling layers, each with kernel size of 2. Output size: 128/(2*2) = 32
        # Have 3 output features, corresponding to x-pos, y-pos, radius.
        self.fc = nn.Linear(in_features=32 * 32 * 32, out_features=3)

    def forward(self, x):
        """
        Feed forward through network
        Args:
            x - input to the network

        Returns "x", which is the network's output
        """

        #----------------------------------------------
        # implementation needed here
        #----------------------------------------------
        #Conv1
        out = self.conv1(x)
        out = self.pool(out)
        out = self.relu(out)
        out = self.norm1(out)
        #Conv2
        out = self.conv2(out)
        out = self.pool(out)
        out = self.relu(out)
        out = self.norm1(out)
        #Conv3
        out = self.conv3(out)
        out = self.drop(out)
        #Conv4
        out = self.conv4(out)
        out = F.dropout(out, training=self.training)
        out = out.view(-1, 32 * 32 * 32)
        out = self.fc(out)

        return out

EDIT: Should it help here is my custom loss function:

criterion = nn.CrossEntropyLoss()

def my_loss(outputs, labels):

    """
    Args:
        outputs - output of network ([batch size, 3])
        labels  - desired labels  ([batch size, 3])
    """

    loss = torch.zeros(1, dtype=torch.float, requires_grad=True)
    loss = loss.to(device)

    loss = criterion(outputs, labels)

    #----------------------------------------------
    # implementation needed here
    #----------------------------------------------

    # Observe: If you need to iterate and add certain values to loss defined above
    # you cannot write: loss +=... because this will raise the error:
    # "Leaf variable was used in an inplace operation"
    # Instead, to avoid this error write: loss = loss + ...

    return loss

Train Loader (given to me):

train_dir      = "./train/"
validation_dir = "./validation/"
test_dir       = "./test/"

train_dataset = ShapesDataset(train_dir)

train_loader = DataLoader(train_dataset,
                          batch_size=32,
                          shuffle=True)

validation_dataset = ShapesDataset(validation_dir)

validation_loader = DataLoader(validation_dataset,
                               batch_size=1,
                               shuffle=False)

test_dataset = ShapesDataset(test_dir)

test_loader = DataLoader(test_dataset,
                          batch_size=1,
                          shuffle=False)

print("train loader examples     :", len(train_dataset))
print("validation loader examples:", len(validation_dataset))
print("test loader examples      :", len(test_dataset))

EDIT: This view image, target circle labels and network output was also given:

"""
View first image of a given number of batches assuming that model has been created.
Currently, lines assuming model has been creatd, are commented out. Without a model,
you can view target labels and the corresponding images.
This is given to you so that you may see how loaders and model can be used.
"""

loader = train_loader # choose from which loader to show images
bacthes_to_show = 2
with torch.no_grad():
    for i, data in enumerate(loader, 0): #0 means that counting starts at zero
        inputs = (data['image']).to(device)   # has shape (batch_size, 3, 128, 128)
        labels = (data['labels']).to(device)  # has shape (batch_size, 3)
        img_fnames = data['fname']            # list of length batch_size

        #outputs = model(inputs.float())
        img = Image.open(img_fnames[0])

        print ("showing image: ", img_fnames[0])

        labels_str = [ float(("{0:.2f}".format(x))) for x in labels[0]]#labels_np_arr]

        #outputs_np_arr = outputs[0] # using ".numpy()" to convert tensor to numpy array
        #outputs_str = [ float(("{0:.2f}".format(x))) for x in outputs_np_arr]
        print("Target labels :", labels_str )
        #print("network coeffs:", outputs_str)
        print()
        #img.show()

        if (i+1) == bacthes_to_show:
            break

Here is the output I'm getting, it is supposed to cover the full circle:
Output I am getting
Any idea will be helpful.


title: For semantic segmentation how to perform data augmentation in Pytorch?

body:
I am using PyTorch for semantic segmentation, But I am facing a problem, because I am use images, and their labels. I want to perform data augmentation such as RandomHorizontalFlip, and RandomCrop, etc.
Here is my code, please check and let me know, how I can embed the following operations in the provided code.
import torchvision.transforms.functional as F

class ToTensor(object):
    def __call__(self, sample):
        image, label = sample['image'], sample['label']
        return {'image': F.to_tensor(image), 'label': F.to_tensor(label)}

my_transform = transforms.Compose([ ToTensor() ])

dataset = Mydataset(image_dir, label_dir, transform = my_transform)

# Print dataset output
dataset[1]

Output
{'image': tensor([[[0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          ...,
          [0.0902, 0.0902, 0.0902,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0745, 0.0745, 0.0745,  ..., 0.0824, 0.0824, 0.0824]],

         [[0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          ...,
          [0.0902, 0.0902, 0.0902,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0745, 0.0745, 0.0745,  ..., 0.0824, 0.0824, 0.0824]],

         [[0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          ...,
          [0.0902, 0.0902, 0.0902,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0745, 0.0745, 0.0745,  ..., 0.0824, 0.0824, 0.0824]]]),
 'label': tensor([[[0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          ...,
          [0.0902, 0.0902, 0.0902,  ..., 0.0824, 0.0824, 0.0824],
          [0.0824, 0.0824, 0.0824,  ..., 0.0824, 0.0824, 0.0824],
          [0.0745, 0.0745, 0.0745,  ..., 0.0824, 0.0824, 0.0824]]])}


title: Expected more than 1 value per channel when training, got input size torch.Size([1, **])

body:
I met an error when I use BatchNorm1d, code:
##% first I set a model
class net(nn.Module):
    def __init__(self, max_len, feature_linear, rnn, input_size, hidden_size, output_dim, num__rnn_layers, bidirectional, batch_first=True, p=0.2):
        super(net, self).__init__()
        self.max_len = max_len
        self.feature_linear = feature_linear
        self.input_size = input_size
        self.hidden_size = hidden_size
        self.bidirectional = bidirectional
        self.num_directions = 2 if bidirectional == True else 1
        self.p = p
        self.batch_first = batch_first
        self.linear1 = nn.Linear(max_len, feature_linear)
        init.kaiming_normal_(self.linear1.weight, mode='fan_in')
        self.BN1 = BN(feature_linear)

    def forward(self, xb, seq_len_crt):
        rnn_input = torch.zeros(xb.shape[0], self.feature_linear, self.input_size)
        for i in range(self.input_size):
            out = self.linear1(xb[:, :, i]) # xb[:,:,i].shape:(1,34), out.shape(1,100)
            out = F.relu(out) # 输入：out.shape(1,100), 输出：out.shape(1,100)
            out = self.BN1(out) # 输入：out.shape(1,100)，输出：out.shape(1,100)

        return y_hat.squeeze(-1)

##% make the model as a function and optimize it
input_size = 5
hidden_size = 32
output_dim = 1
num_rnn_layers = 2
bidirectional = True
rnn = nn.LSTM
batch_size = batch_size
feature_linear = 60
BN = nn.BatchNorm1d

model = net(max_len, feature_linear, rnn, input_size, hidden_size, output_dim, num_rnn_layers, bidirectional, p=0.1)
loss_func = nn.MSELoss(reduction='none')
# optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)
# optimizer = optim.Adam(model.parameters(), lr=0.01)
optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.05)

##% use this model to predict data
def predict(xb, model, seq_len):
    # xb's shape should be (batch_size, seq_len, n_features)
    if xb.ndim == 2:  # suitable for both ndarray and Tensor
       # add a {batch_size} dim
       xb = xb[None, ]

    if not isinstance(xb, torch.Tensor):
       xb = torch.Tensor(xb)
    return model(xb, seq_len) # xb.shape(1,34,5)

##% create training/valid/test data
seq_len_train_iter = []
for i in range(0, len(seq_len_train), batch_size):
    if i + batch_size <= len(seq_len_train):
       seq_len_train_iter.append(seq_len_train[i:i+batch_size])
    else:
       seq_len_train_iter.append(seq_len_train[i:])

seq_len_valid_iter = []
for i in range(0, len(seq_len_valid), batch_size):
    if i + batch_size <= len(seq_len_valid):
        seq_len_valid_iter.append(seq_len_valid[i:i+batch_size])
    else:
        seq_len_valid_iter.append(seq_len_valid[i:])

seq_len_test_iter = []
for i in range(0, len(seq_len_test), batch_size):
    if i + batch_size <= len(seq_len_test):
        seq_len_test_iter.append(seq_len_test[i:i+batch_size])
    else:
        seq_len_test_iter.append(seq_len_test[i:])

##% fit model
def fit(epochs, model, loss_func, optimizer, train_dl, valid_dl, valid_ds, seq_len_train_iter, seq_len_valid_iter):
    train_loss_record = []
    valid_loss_record = []
    mean_pct_final = []
    mean_abs_final = []

    is_better = False
    last_epoch_abs_error = 0
    last_epoch_pct_error = 0

    mean_pct_final_train = []
    mean_abs_final_train = []

    for epoch in range(epochs):
        # seq_len_crt: current batch seq len
        for batches, ((xb, yb), seq_len_crt) in enumerate(zip(train_dl, seq_len_train_iter)):
            if isinstance(seq_len_crt, np.int64):
                seq_len_crt = [seq_len_crt]
            y_hat = model(xb, seq_len_crt)
            packed_yb = nn.utils.rnn.pack_padded_sequence(yb, seq_len_crt, batch_first=True, enforce_sorted=False)
            final_yb, input_sizes = nn.utils.rnn.pad_packed_sequence(packed_yb)
            final_yb = final_yb.permute(1, 0)
            # assert torch.all(torch.tensor(seq_len_crt).eq(input_sizes))
            loss = loss_func(y_hat, final_yb)

            batch_size_crt = final_yb.shape[0]
            loss = (loss.sum(-1) / input_sizes).sum() / batch_size_crt

            loss.backward()
            optimizer.step()
            # scheduler.step()
            optimizer.zero_grad()
#             print(i)

            with torch.no_grad():
                train_loss_record.append(loss.item())
                if batches % 50 == 0 and epoch % 1 == 0:
#                     print(f'Epoch {epoch}, batch {i} training loss: {loss.item()}')

                    y_hat = predict(xb[0], model, torch.tensor([seq_len_crt[0]])).detach().numpy().squeeze() # xb[0].shape(34,5)
                    label = yb[0][:len(y_hat)]
                    # plt.ion()
                    plt.plot(y_hat, label='predicted')
                    plt.plot(label, label='label')
                    plt.legend(loc='upper right')
                    plt.title('training mode')
                    plt.text(len(y_hat)+1, max(y_hat.max(), label.max()), f'Epoch {epoch}, batch {batches} training loss: {loss.item()}')
                    plt.show()
    return train_loss_record

but I met:Expected more than 1 value per channel when training, got input size torch.Size([1, 60])
the error message is:
ValueError                                Traceback (most recent call last)
<ipython-input-119-fb062ad3f20e> in <module>
----> 1 fit(500, model, loss_func, optimizer, train_dl, valid_dl, valid_ds, seq_len_train_iter, seq_len_valid_iter)

<ipython-input-118-2eb946c379bf> in fit(epochs, model, loss_func, optimizer, train_dl, valid_dl, valid_ds, seq_len_train_iter, seq_len_valid_iter)
     38 #                     print(f'Epoch {epoch}, batch {i} training loss: {loss.item()}')
     39
---> 40                     y_hat = predict(xb[0], model, torch.tensor([seq_len_crt[0]])).detach().numpy().squeeze() # xb[0].shape(34,5)
     41                     label = yb[0][:len(y_hat)]
     42                     # plt.ion()

<ipython-input-116-28afce77e325> in predict(xb, model, seq_len)
      7     if not isinstance(xb, torch.Tensor):
      8         xb = torch.Tensor(xb)
----> 9     return model(xb, seq_len) # xb.shape(None,34,5)

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

<ipython-input-114-3e9c30d20ed6> in forward(self, xb, seq_len_crt)
     50             out = self.linear1(xb[:, :, i]) # xb[:,:,i].shape:(None,34), out.shape(None,100)
     51             out = F.relu(out) # 输入：out.shape(None,100), 输出：out.shape(None,100)
---> 52             out = self.BN1(out) # 输入：out.shape(None,100)，输出：out.shape(None,100)
     53
     54             out = self.linear2(out)

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\modules\module.py in _call_impl(self, *input, **kwargs)
    725             result = self._slow_forward(*input, **kwargs)
    726         else:
--> 727             result = self.forward(*input, **kwargs)
    728         for hook in itertools.chain(
    729                 _global_forward_hooks.values(),

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\modules\batchnorm.py in forward(self, input)
    129         used for normalization (i.e. in eval mode when buffers are not None).
    130         """
--> 131         return F.batch_norm(
    132             input,
    133             # If buffers are not to be tracked, ensure that they won't be updated

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)
   2052                 bias=bias, training=training, momentum=momentum, eps=eps)
   2053     if training:
-> 2054         _verify_batch_size(input.size())
   2055
   2056     return torch.batch_norm(

D:\Anaconda3\envs\LSTM\lib\site-packages\torch\nn\functional.py in _verify_batch_size(size)
   2035         size_prods *= size[i + 2]
   2036     if size_prods == 1:
-> 2037         raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
   2038
   2039

ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 60])

I have checked and I found that in out = self.BN1(out),out.shape = (1,60),it seems that batchsize=1 is not permitted in BatchNorm1d .But I don't know how to modify it.


title: Pytorch Spell Check Character RNN not outputting end tokens

body:
I’m trying to implement a character RNN for the purpose of spell correction and tokenization. The model is based on the practical pytorch GRU-RNN implementation of a seq2seq model - the loss function is masked cross entropy like they use here and I’m using a 2 layer bidirectional GRU for my encoder/decoder. I am also using scheduled sampling with a teacher forcing ratio of 0.5.

The problem I’m having is that the system is struggling to output end tokens. Sentences such as ‘do3s t4is w0rk’ are being correctly changed to ‘does this work’, but then random tokens are outputted until either the system reaches it’s max output length or seemingly randomly puts down an EOS token, making the output something like ‘does this workkkkkk orwk EOS’.

Does anyone have any insight into this problem? I tried unmasking my loss function but this lead to the system outputting PAD tokens now. The system is able to overfit very easy sets so I don’t think there’s a bug in the training. Any input would be appreciated!


title: How to compute gradient of power function wrt exponent in PyTorch?

body:
I am trying to compute the gradient of

out = x.sign()*torch.pow(x.abs(), alpha)

with respect to alpha.

I tried the following so far:

class Power(nn.Module):
  def __init__(self, alpha=2.):
    super(Power, self).__init__()
    self.alpha = nn.Parameter(torch.tensor(alpha))

  def forward(self, x):
    return x.sign()*torch.abs(x)**self.alpha

but this class keeps giving me nan in training of my network. I expect to see something like grad=out*torch.log(x) but cannot get to it. This code, for instance, returns nothing:

alpha_rooting = Power()
x = torch.randn((1), device='cpu', dtype=torch.float)
out = (alpha_rooting(x)).sum()
out.backward()
print(out.grad)

I am trying to use autograd for this by no luck either. How should I go about solving this? Thanks.


title: Assigning a parameter to the GPU sets is_leaf as false

body:
If I create a Parameter in PyTorch, then it is automatically assigned as a leaf variable:

x = torch.nn.Parameter(torch.Tensor([0.1]))
print(x.is_leaf)

This prints out True. From what I understand, if x is a leaf variable, then it will be updated by the optimiser.

But if I then assign x to the GPU:

x = torch.nn.Parameter(torch.Tensor([0.1]))
x = x.cuda()
print(x.is_leaf)

This prints out False. So now I cannot assign x to the GPU and keep it as a leaf node.

Why does this happen?


title: Getting range of values from Pytorch Tensor

body:
I am trying to get a specific range of values from my pytorch tensor.
tensor=torch.tensor([0,1,2,3,4,5,6,7,8,9])
new_tensor=tensor[tensor>2]
print(new_tensor)

This will give me a tensor with scalars of 3-9
new_tensor2=tensor[tensor<8]
print(new_tensor2)

This will give me a tensor with scalars of 0-7
new_tensor3=tensor[tensor>2 and tensor<8]
print(new_tensor3)

However this raises an error. Would I be able to get a tensor with the values of 3-7 using something like this? I am trying to edit the tensor directly, and do not wish to change the order of the tensor itself.
grad[x<-3]=0.1
grad[x>2]=1
grad[(x>=-3 and x<=2)]=siglrelu(grad[(x>=-3 and x<=2)])*(1.0-siglrelu(grad[(x>=-3 and x<=2)]))

This is what I am really going for, and I am not exactly sure of how to go about this. Any help is appreciated, thank you!


title: How does one fix a `Exception: process 0 terminated with signal SIGSEGV` error and if the single gpu code works fine?

body:
I start 2 processes because I only have 2 gpus but then it gives me a Exception: process 0 terminated with signal SIGSEGV. This code does work with multiple cpus (or at least no error is thrown). Also, it works with a single GPU. Besides that is fails when world_size > 0 and multiple cuda/gpus are present.
My error message this this:
(automl-meta-learning) miranda9~/ML4Coq $ python playground/multiprocessing_playground/ddp_hello_world.py

world_size=2

Traceback (most recent call last):
  File "playground/multiprocessing_playground/ddp_hello_world.py", line 49, in <module>
    main()
  File "playground/multiprocessing_playground/ddp_hello_world.py", line 43, in main
    mp.spawn(example,
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 199, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 157, in start_processes
    while not context.join():
  File "/home/miranda9/miniconda3/envs/automl-meta-learning/lib/python3.8/site-packages/torch/multiprocessing/spawn.py", line 105, in join
    raise Exception(
Exception: process 0 terminated with signal SIGSEGV

This is the code that gives the error:
import os

import torch
import torch.distributed as dist
import torch.multiprocessing as mp
import torch.nn as nn
import torch.optim as optim
from torch.nn.parallel import DistributedDataParallel as DDP

def example(rank, world_size):
    # create default process group
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '8888'
    dist.init_process_group("gloo", rank=rank, world_size=world_size)

    # create local model
    model = nn.Linear(10, 10).to(rank)
    # construct DDP model
    ddp_model = DDP(model, device_ids=[rank])
    # define loss function and optimizer
    loss_fn = nn.MSELoss()
    optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

    # forward pass
    outputs = ddp_model(torch.randn(20, 10).to(rank))
    labels = torch.randn(20, 10).to(rank)
    # backward pass
    loss_fn(outputs, labels).backward()
    # update parameters
    optimizer.step()

def main():
    # world_size = 2
    world_size = torch.cuda.device_count()
    mp.spawn(example,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    main()
    print('Done\n\a')

[Optional] Larger self-contained example (gives same error)
Note however, that this slightly more complete example (only missing a distributed dataloader) also gives me the same issue:
"""
Based on: https://pytorch.org/tutorials/intermediate/ddp_tutorial.html

Correctness of code: https://stackoverflow.com/questions/66226135/how-to-parallelize-a-training-loop-ever-samples-of-a-batch-when-cpu-is-only-avai

Note: as opposed to the multiprocessing (torch.multiprocessing) package, processes can use
different communication backends and are not restricted to being executed on the same machine.
"""
import time

from typing import Tuple

import torch
from torch import nn, optim
import torch.distributed as dist
import torch.multiprocessing as mp
from torch.nn.parallel import DistributedDataParallel as DDP

import os

num_epochs = 5
batch_size = 8
Din, Dout = 10, 5
data_x = torch.randn(batch_size, Din)
data_y = torch.randn(batch_size, Dout)
data = [(i*data_x, i*data_y) for i in range(num_epochs)]

class PerDeviceModel(nn.Module):
    """
    Toy example for a model ran in parallel but not distributed accross gpus
    (only processes with their own gpu or hardware)
    """
    def __init__(self):
        super().__init__()
        self.net1 = nn.Linear(Din, Din)
        self.relu = nn.ReLU()
        self.net2 = nn.Linear(Din, Dout)

    def forward(self, x):
        return self.net2(self.relu(self.net1(x)))

def setup_process(rank, world_size, backend='gloo'):
    """
    Initialize the distributed environment (for each process).

    gloo: is a collective communications library (https://github.com/facebookincubator/gloo). My understanding is that
    it's a library/API for process to communicate/coordinate with each other/master. It's a backend library.
    """
    # set up the master's ip address so this child process can coordinate
    # os.environ['MASTER_ADDR'] = '127.0.0.1'
    os.environ['MASTER_ADDR'] = 'localhost'
    os.environ['MASTER_PORT'] = '12355'

    # - use NCCL if you are using gpus: https://pytorch.org/tutorials/intermediate/dist_tuto.html#communication-backends
    if torch.cuda.is_available():
        backend = 'nccl'
    # Initializes the default distributed process group, and this will also initialize the distributed package.
    dist.init_process_group(backend, rank=rank, world_size=world_size)

def cleanup():
    """ Destroy a given process group, and deinitialize the distributed package """
    dist.destroy_process_group()

def get_batch(batch: Tuple[torch.Tensor, torch.Tensor], rank):
    x, y = batch
    if torch.cuda.is_available():
        x, y = x.to(rank), y.to(rank)
    else:
        x, y = x.share_memory_(), y.share_memory_()
    return x, y

def get_ddp_model(model: nn.Module, rank):
    """
    Moves the underlying storage to shared memory.

        This is a no-op if the underlying storage is already in shared memory
        and for CUDA tensors. Tensors in shared memory cannot be resized.

    :return:

    TODO: does this have to be done outside or inside the process? my guess is that it doesn't matter because
    1) if its on gpu once it's on the right proc it moves it to cpu with id rank via mdl.to(rank)
    2) if it's on cpu then mdl.share_memory() or data.share_memory() is a no op if it's already in shared memory o.w.
    """
    # if gpu avail do the standard of creating a model and moving the model to the GPU with id rank
    if torch.cuda.is_available():
    # create model and move it to GPU with id rank
        model = model.to(rank)
        ddp_model = DDP(model, device_ids=[rank])
    else:
    # if we want multiple cpu just make sure the model is shared properly accross the cpus with shared_memory()
    # note that op is a no op if it's already in shared_memory
        model = model.share_memory()
        ddp_model = DDP(model)  # I think removing the devices ids should be fine...?
    return ddp_model
    # return OneDeviceModel().to(rank) if torch.cuda.is_available() else OneDeviceModel().share_memory()

def run_parallel_training_loop(rank, world_size):
    """
    Distributed function to be implemented later.

    This is the function that is actually ran in each distributed process.

    Note: as DDP broadcasts model states from rank 0 process to all other processes in the DDP constructor,
    you don’t need to worry about different DDP processes start from different model parameter initial values.
    """
    setup_process(rank, world_size)
    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')

    # get ddp model
    model = PerDeviceModel()
    ddp_model = get_ddp_model(model, rank)

    # do training
    for batch_idx, batch in enumerate(data):
        x, y = get_batch(batch, rank)
        loss_fn = nn.MSELoss()
        optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

        optimizer.zero_grad()
        outputs = ddp_model(x)
        # Gradient synchronization communications take place during the backward pass and overlap with the backward computation.
        loss_fn(outputs, y).backward()  # When the backward() returns, param.grad already contains the synchronized gradient tensor.
        optimizer.step()  # TODO how does the optimizer know to do the gradient step only once?

    print()
    print(f"Start running DDP with model parallel example on rank: {rank}.")
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # Destroy a given process group, and deinitialize the distributed package
    cleanup()

def main():
    print()
    print('running main()')
    print(f'current process: {mp.current_process()}')
    print(f'pid: {os.getpid()}')
    # args
    if torch.cuda.is_available():
        world_size = torch.cuda.device_count()
    else:
        world_size = mp.cpu_count()
    print(f'world_size={world_size}')
    mp.spawn(run_parallel_training_loop, args=(world_size,), nprocs=world_size)

if __name__ == "__main__":
    print('starting __main__')
    start = time.time()
    main()
    print(f'execution length = {time.time() - start}')
    print('Done!\a\n')

cross posted: https://discuss.pytorch.org/t/why-is-mp-spawn-spawning-4-processes-when-i-only-want-2/112299


title: Gcloud flask app error while installing requirements.txt during deploying

body:
I am trying to deploy a flask app on gcloud that uses pytorch. when i ran server locally it worked fine. but when i deployed same code using gcloud app deploy, it gives following error.

Step #1: Step 6/9 : ADD requirements.txt /app/
Step #1:  ---> eed90d6f59c6
Step #1: Step 7/9 : RUN pip install -r requirements.txt
Step #1:  ---> Running in e6f627cd6ee4
Step #1: Collecting torch (from -r requirements.txt (line 1))
Step #1:   Downloading
https://files.pythonhosted.org/packages/df/a4/7f5ec6e9df1bf13
f1881353702aa9713fcd997481b26018f35e0be85faf7/torch-0.4.0-cp27-cp27mu-
manylinux1_x8
6_64.whl (484.0MB)
Step #1: Exception:
Step #1: Traceback (most recent call last):
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/basecommand.py", line 215, in main
Step #1:     status = self.run(options, args)
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/commands/install.py", line 335, in run
Step #1:     wb.build(autobuilding=True)
Step #1:   File "/env/local/lib/python2.7/site-packages/pip/wheel.py",
line 749, in build
Step #1:     self.requirement_set.prepare_files(self.finder)
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/req/req_set.py", line 380, in prepare_files
Step #1:     ignore_dependencies=self.ignore_dependencies))
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/req/req_set.py", line 620, in _prepare_file
Step #1:     session=self.session, hashes=hashes)
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/download.py", line 821, in unpack_url
Step #1:     hashes=hashes
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/download.py", line 659, in unpack_http_url
Step #1:     hashes)
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/download.py", line 882, in _download_http_url
Step #1:     _download_url(resp, link, content_file, hashes)
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/download.py", line 603, in _download_url
Step #1:     hashes.check_against_chunks(downloaded_chunks)
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/utils/hashes.py", line 46, in check_against_chunks
Step #1:     for chunk in chunks:
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/download.py", line 571, in written_chunks
Step #1:     for chunk in chunks:
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/utils/ui.py", line 139, in iter
Step #1:     for x in it:
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/download.py", line 560, in resp_read
Step #1:     decode_content=False):
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/_vendor/urllib3/response.py", line 436, in stream
Step #1:     data = self.read(amt=amt, decode_content=decode_content)
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/_vendor/urllib3/response.py", line 384, in read
Step #1:     data = self._fp.read(amt)
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/_vendor/cachecontrol/filewrapper.py", line 63, in read
Step #1:     self._close()
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/_vendor/cachecontrol/filewrapper.py", line 50, in _close
Step #1:     self.__callback(self.__buf.getvalue())
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/_vendor/cachecontrol/controller.py", line 275, in
cache_response
Step #1:     self.serializer.dumps(request, response, body=body),
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/_vendor/cachecontrol/serialize.py", line 55, in dumps
Step #1:     "body": _b64_encode_bytes(body),
Step #1:   File "/env/local/lib/python2.7/site-
packages/pip/_vendor/cachecontrol/serialize.py", line 12, in
_b64_encode_bytes
Step #1:     return base64.b64encode(b).decode("ascii")
Step #1: MemoryError
Step #1: You are using pip version 9.0.3, however version 10.0.1 is
available.
Step #1: You should consider upgrading via the 'pip install --upgrade
pip' command.
Step #1: The command '/bin/sh -c pip install -r requirements.txt'
returned a non-zero code: 2
Finished Step #1
ERROR
ERROR: build step 1 "gcr.io/cloud-

builders/docker@sha256:5836cd0f940f2f1a912b8fee4c74e5
f1a84df92aab557c608a94e171e9df8d46" failed: exit status 2
Step #1:
----------------------------------------------------------------------
----------------------------------------------------------------------
---------------------------

ERROR: (gcloud.app.deploy) Cloud build failed. Check logs at
https://console.cloud.google.com/gcr/builds/d1e646e5-53f4-43d0-8504-
98226e97de78?project=1091101138750 Fai
lure status: UNKNOWN: Error Response: [2] Build failed; check build
logs
for details

My requirements.txt looks like this

torch

torchvision

jsonpickle

Flask==0.12.2

gunicorn==19.7.1

numpy

matplotlib

and app.yaml looks like this.

runtime: python
env: flex
entrypoint: gunicorn -b :$PORT main:app
runtime_config:
  python_version: 2
# This sample incurs costs to run on the App Engine flexible
environment.
# The settings below are to reduce costs during testing and are not
appropriate
# for production use. For more information, see:
# https://cloud.google.com/appengine/docs/flexible/python/configuring-
your-app-with-app-yaml
manual_scaling:
  instances: 1
resources:
  cpu: 1
  memory_gb: 1.5
  disk_size_gb: 10


title: PIL image resize change the value of pixel?

body:
I want to resize an PIL image without changing the pixel value range.
I have tried the Image.resize() but it changes my pixel value range from [0,255] to [79,179]

I'm using Python and PyTorch, in PyTorch, the transforms.resize() will implement Image.resize()

Here is the test code I used

a = torch.randint(0,255,(500,500), dtype=torch.uint8)
print(a.size())
print(torch.max(a))
a = torch.unsqueeze(a, dim =0)
print(a.size())
compose = transforms.Compose([transforms.ToPILImage(),transforms.Resize((128,128))])
a_trans = compose(a)
print(a_trans.size)
print(a_trans.getextrema())

The output of the code is:

torch.Size([500, 500])
tensor(254, dtype=torch.uint8)
torch.Size([1, 500, 500])
(128, 128)
(79, 179)

So someone could explain why the output range is [79,179]?
If I want to do the resize without changing the value range, what should I do?
Thank you in advance


title: May the same NN converge in Keras and not in PyTorch?

body:
I am trying to implement a CNN for regression on images in PyTorch. I have a working model already implemented in Keras and I would like to translate it in PyTorch, but I am facing many issues.

Essentially, in Keras the model converges, whereas in PyTorch it doesn't. In PyTorch I always have a constant training loss and the trained model always outputs the same value for any image. I initialized all the layers as in the working Keras model, I added l2 regularization as in Keras, I also implemented the same learning rate decay. Everything looks exactly the same, but in PyTorch my model doesn't converge.

Is it possible that, with everything initialized the same way, a model does not converge in PyTorch while converging in Keras? If so, what would you suggest me to do in my specific case (constant training loss after one epoch and constant predictions)?

I have already tried to clip gradients and change learning rate, at the beginning I used Adam with lr = 0.001, then I tried 0.1 and 0.0001, always emulating Keras time-based decay.

The task consists in regressing steering angles from this Udacity dataset.

Preprocessing:
- get grayscale version of images in [0,1] interval and crop them to 200x200 size with the following procedure:

half_the_width = int(img.shape[1] / 2)
img = img[img.shape[0] - crop_heigth: img.shape[0],      half_the_width - int(crop_width / 2):
half_the_width + int(crop_width / 2)]

Thus I have input images with size 200x200 and target of dimension 1 (steering angle).

Loss used is MSEloss

This is the KERAS model that I want to emulate, it uses Adam with learning rate decay 1e-5 and is taken from DroNet repo:

def resnet8(img_width, img_height, img_channels, output_dim):
    """
    Define model architecture.

    # Arguments
       img_width: Target image widht.
       img_height: Target image height.
       img_channels: Target image channels.
       output_dim: Dimension of model output.

    # Returns
       model: A Model instance.
    """

    # Input
    img_input = Input(shape=(img_height, img_width, img_channels))

    x1 = Conv2D(32, (5, 5), strides=[2,2], padding='same')(img_input)
    x1 = MaxPooling2D(pool_size=(3, 3), strides=[2,2])(x1)

    # First residual block
    x2 = keras.layers.normalization.BatchNormalization()(x1)
    x2 = Activation('relu')(x2)
    x2 = Conv2D(32, (3, 3), strides=[2,2], padding='same',
                kernel_initializer="he_normal",
                kernel_regularizer=regularizers.l2(1e-4))(x2)

    x2 = keras.layers.normalization.BatchNormalization()(x2)
    x2 = Activation('relu')(x2)
    x2 = Conv2D(32, (3, 3), padding='same',
                kernel_initializer="he_normal",
                kernel_regularizer=regularizers.l2(1e-4))(x2)

    x1 = Conv2D(32, (1, 1), strides=[2,2], padding='same')(x1)
    x3 = add([x1, x2])

    # Second residual block
    x4 = keras.layers.normalization.BatchNormalization()(x3)
    x4 = Activation('relu')(x4)
    x4 = Conv2D(64, (3, 3), strides=[2,2], padding='same',
                kernel_initializer="he_normal",
                kernel_regularizer=regularizers.l2(1e-4))(x4)

    x4 = keras.layers.normalization.BatchNormalization()(x4)
    x4 = Activation('relu')(x4)
    x4 = Conv2D(64, (3, 3), padding='same',
                kernel_initializer="he_normal",
                kernel_regularizer=regularizers.l2(1e-4))(x4)

    x3 = Conv2D(64, (1, 1), strides=[2,2], padding='same')(x3)
    x5 = add([x3, x4])

    # Third residual block
    x6 = keras.layers.normalization.BatchNormalization()(x5)
    x6 = Activation('relu')(x6)
    x6 = Conv2D(128, (3, 3), strides=[2,2], padding='same',
                kernel_initializer="he_normal",
                kernel_regularizer=regularizers.l2(1e-4))(x6)

    x6 = keras.layers.normalization.BatchNormalization()(x6)
    x6 = Activation('relu')(x6)
    x6 = Conv2D(128, (3, 3), padding='same',
                kernel_initializer="he_normal",
                kernel_regularizer=regularizers.l2(1e-4))(x6)

    x5 = Conv2D(128, (1, 1), strides=[2,2], padding='same')(x5)
    x7 = add([x5, x6])

    x = Flatten()(x7)
    x = Activation('relu')(x)
    x = Dropout(0.5)(x)

    # Steering channel
    steer = Dense(output_dim)(x)

    # Collision channel
    coll = Dense(output_dim)(x)
    coll = Activation('sigmoid')(coll)

    # Define steering-collision model
    model = Model(inputs=[img_input], outputs=[steer, coll])
    print(model.summary())

    return model

In PyTorch I am trying to implement only steering angle prediction, the paper mentions that steering angles and collision predictions are uncorrelated.

This is my PyTorch implementation:
MODEL

def init_kernel(m):
    if isinstance(m, nn.Conv2d):
        # Initialize kernels of Conv2d layers as kaiming normal
        nn.init.kaiming_normal_(m.weight, mode='fan_in', nonlinearity='relu')
        # Initialize biases of Conv2d layers at 0
        nn.init.zeros_(m.bias)

def __init__(self, img_channels, in_height, in_width, output_dim):
        super(resnet8, self).__init__()

        self.layer1 = nn.Sequential(
            nn.Conv2d(in_channels=img_channels,out_channels=32,
                      kernel_size=[5,5], stride=[2,2], padding=[5//2,5//2]),
            nn.MaxPool2d(kernel_size=[3,3], stride=[2,2]))

        self.residual_block_1a = nn.Sequential(
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(in_channels=32,out_channels=32, kernel_size=[3,3],
                      stride=[2,2], padding=[3//2,3//2]),
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(in_channels=32,out_channels=32, kernel_size=[3,3],
                      padding=[3//2,3//2]))

        self.parallel_conv_1 = nn.Conv2d(in_channels=32,out_channels=32,
                                         kernel_size=[1,1], stride=[2,2],
                                         padding=[1//2,1//2])

        self.residual_block_2a = nn.Sequential(
            nn.BatchNorm2d(32),
            nn.ReLU(),
            nn.Conv2d(in_channels=32,out_channels=64, kernel_size=[3,3],
                      stride=[2,2], padding=[3//2,3//2]),
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(in_channels=64,out_channels=64, kernel_size=[3,3],
                      padding=[3//2,3//2]))

        self.parallel_conv_2 = nn.Conv2d(in_channels=32,out_channels=64,
                                         kernel_size=[1,1], stride=[2,2],
                                         padding=[1//2,1//2])

        self.residual_block_3a = nn.Sequential(
            nn.BatchNorm2d(64),
            nn.ReLU(),
            nn.Conv2d(in_channels=64,out_channels=128, kernel_size=[3,3],
                      stride=[2,2], padding=[3//2,3//2]),
            nn.BatchNorm2d(128),
            nn.ReLU(),
            nn.Conv2d(in_channels=128,out_channels=128, kernel_size=[3,3],
                      padding=[3//2,3//2]))

        self.parallel_conv_3 = nn.Conv2d(in_channels=64,out_channels=128,
                                         kernel_size=[1,1], stride=[2,2],
                                         padding=[1//2,1//2])

        self.output_dim = output_dim

        self.last_block = nn.Sequential(
            nn.ReLU(),
            nn.Dropout2d(),
            nn.Linear(6272,self.output_dim))

        # Initialize layers exactly as in Keras
        for m in self.modules():
            if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
                nn.init.xavier_uniform_(m.weight, gain=nn.init.calculate_gain('relu'))
                nn.init.zeros_(m.bias)
            elif isinstance(m, nn.BatchNorm2d):
                # Initialize kernels of Conv2d layers as kaiming normal
                nn.init.constant_(m.weight, 1)
                nn.init.constant_(m.bias, 0)

        self.residual_block_1a.apply(init_kernel)
        self.residual_block_2a.apply(init_kernel)
        self.residual_block_3a.apply(init_kernel)

    def forward(self, x):
        x1 = self.layer1(x)
        # First residual block
        x2 = self.residual_block_1a(x1)
        x1 = self.parallel_conv_1(x1)
        x3 = x1.add(x2)
        # Second residual block
        x4 = self.residual_block_2a(x3)
        x3 = self.parallel_conv_2(x3)
        x5 = x3.add(x4)
        # Third residual block
        x6 = self.residual_block_3a(x5)
        x5 = self.parallel_conv_3(x5)
        x7 = x5.add(x6)

        out = x7.view(x7.size(0), -1) # Flatten
        out = self.last_block(out)

        return out

TRAINING LOOP

def compute_l2_reg(model,model_name):
    # Function that sets weight_decay only for weights and not biases and only
    # for conv layers inside residual layers
    lambda_ = FLAGS.weight_decay
    params_dict = dict(model.named_parameters())
    l2_reg=[]
    if model_name == 'resnet8':
        for key, value in params_dict.items():
            if ((key[-8:] == '2.weight' or key[-8:] == '5.weight') and key[0:8]=='residual'):
                l2_reg += [lambda_*torch.norm(value.view(value.size(0),-1),2)]

    l2_reg = sum(l2_reg)
    return l2_reg

def train_model(model, num_epochs, learning_rate, train_loader, valid_loader,
                patience, model_name):

    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

    # To track the training loss as the model trains
    train_losses = []
    # To track the validation loss as the model trains
    valid_losses = []
    # To track the average training loss per epoch as the model trains
    avg_train_losses = []
    # To track the average validation loss per epoch as the model trains
    avg_valid_losses = []

    # Initialize the early_stopping object
    early_stopping = EarlyStopping(patience=patience, verbose=True)

    # Training loop
    decay = FLAGS.decay # Default 1e-5
    fcn = lambda step: 1./(1. + decay*step)
    scheduler = LambdaLR(optimizer, lr_lambda=fcn)

    for epoch in range(1, num_epochs+1):
        ###################
        # TRAIN the model #
        ###################
        model.train() # prep model for training
        for batch, (images, targets) in enumerate(train_loader, 1):
            # Load images and targets to device
            images = images.to(device)
            targets = targets.to(device)
            # Clear gradients
            optimizer.zero_grad()
            # Forward pass
            outputs = model(images)
            # Calculate loss
            l2_reg = compute_l2_reg(model,model_name)
            loss = F.mse_loss(outputs, targets) + l2_reg
            # Backward pass
            loss.backward()
            # Update weights
            optimizer.step()
            # Decay Learning Rate
            scheduler.step()
            # Record training loss
            train_losses.append(loss.item())

        ######################
        # VALIDATE the model #
        ######################
        model.eval() # prep model for evaluation
        for images, targets in valid_loader:
            images = images.to(device)
            targets = targets.to(device)
            # Forward pass:
            outputs = model(images)
            # Calculate loss
            loss = F.mse_loss(outputs, targets)
            # Record validation loss
            valid_losses.append(loss.item())

        # Print training/validation statistics
        # Calculate average loss over an epoch
        train_loss = np.average(train_losses)
        valid_loss = np.average(valid_losses)
        avg_train_losses.append(train_loss)
        avg_valid_losses.append(valid_loss)

        epoch_len = len(str(num_epochs))

        print_msg = (f'[{epoch:>{epoch_len}}/{num_epochs:>{epoch_len}}] ' +
                     f'train_loss: {train_loss:.5f} ' +
                     f'valid_loss: {valid_loss:.5f}')

        print(print_msg)

        # Clear lists to track next epoch
        train_losses = []
        valid_losses = []

        # Early_stopping needs the validation loss to check if it has decresed,
        # and if it has, it will make a checkpoint of the current model
        early_stopping(valid_loss, model)

        if early_stopping.early_stop:
            print("Early stopping")
            break

    # Load the last checkpoint with the best model
    # (returned by early_stopping call)
    model.load_state_dict(torch.load('checkpoint.pt'))
    print('Training completed and model saved.')

    return  model, avg_train_losses, avg_valid_losses

where weight_decay = 1e-4  and Early Stop is a function that checks for validation loss and stores the model with least validation error.

I tried to emulate in PyTorch same manual and default initializations as in Keras, you can see it in my Model code.

I also tried to exactly reproduce the same regularization as in Keras kernel regularizer.


title: Is there a way sort eigenvalues and associated eigenvectors in pytorch?

body:
Is there a way to sort a matrix based on another matrix or vector in pytorch? Given eigenvalues and eigenvectors as shown below:

eigenvalues, eigenvectors = torch.eig(matrix,eigenvectors=True) #compute the eigenvalues & vectors

sortedEig, indices=torch.sort(w, dim=1, descending=True)
eigenvectors_sorted = lamb.gather(dim=1, index=indices)

I need to sort eigenvectors by decreasing eigenvalues. The code above shows what I have tried out. However the output, eigenvectors_sorted is a N x 2 tensor instead of N x N tensor  which eigenvectors actually is.

Note that in pytorch, eigenvalues is a N x 2 tensor.


title: The use of meshgrid in pytorch/numpy

body:
Below code is a snippet taken from https://blog.paperspace.com/how-to-implement-a-yolo-v3-object-detector-from-scratch-in-pytorch-part-3/, and I am confused as to what it is trying to achieve.
    grid = np.arange(grid_size)
    a,b = np.meshgrid(grid, grid)

    x_offset = torch.FloatTensor(a).view(-1,1)
    y_offset = torch.FloatTensor(b).view(-1,1)

    if CUDA:
        x_offset = x_offset.cuda()
        y_offset = y_offset.cuda()

    x_y_offset = torch.cat((x_offset, y_offset), 1).repeat(1,num_anchors).view(-1,2).unsqueeze(0)

I tried the case when grid_size = 3, and it outputed:
tensor([[0., 1.],
        [2., 0.],
        [1., 2.],
        [0., 1.],
        [2., 0.],
        [0., 0.],
        [1., 1.],
        [1., 2.],
        [2., 2.],
        [0., 1.],
        [2., 0.],
        [1., 2.],
        [0., 1.],
        [2., 0.],
        [0., 0.],
        [1., 1.],
        [1., 2.],
        [2., 2.],
        [0., 1.],
        [2., 0.],
        [1., 2.],
        [0., 1.],
        [2., 0.],
        [0., 0.],
        [1., 1.],
        [1., 2.],
        [2., 2.]])

I cannot quite see what is the pattern here. According to the description in the given link I think I should really expect something like:
tensor([[0,0],
        [0,0],
        [0,0],
        [0,1],
        [0,1],
        [0,1],
        [0,2],
        [0,2],
        ...]])


title: How to handle RuntimeError: CUDA error: an illegal memory access was encountered in PyTorch?

body:
I am running some code with pytorch 1.7 and cuda 11.0 on my gpu enabled machine. I have a few classes that contain the code for a few different neural networks. My first class runs smoothly and the following print statements work:
Class A:
print("Device resnet", torch.cuda.current_device()) # Prints 0
print("Boxes resnet", boxes) # Prints the array of bounding boxes

Class A creates an instance of class b, passing in the boxes to the constructor. But when I do similar print statements in class b it crashes:
print("Device fastrcnn", torch.cuda.current_device()) # Prints 0
print("BOXES in fastrcnn", boxes) # crashes and prints the following error

This crashes and prints:
RuntimeError: CUDA error: an illegal memory access was encountered

How do I go about debugging this, I have already tried adding .cpu() and .cuda() to boxes to transfer them to the respective device but I get the exact same crash.


title: Unable to load weights from pytorch checkpoint after splitting pytorch_model.bin into chunks

body:
I need to transfer a pytorch_model.bin of a pretrained deeppavlov ruBERT model but I have a file size limit. So I split it into chunks using python, transferred and reassembled in the correct order. However, the size of the file increased, and when I tried to load the resulting file using BertModel.from_pretrained(pytorch_model.bin) I received an error:
During handling of the above exception, another exception occurred:
OSError: Unable to load weights from pytorch checkpoint <...>

So my question is: is it actually possible to split the file like that? I could possibly have a mistake in the way I split and reassemble the file. However, this could also be some version mismatch.
My python code to get chunks:
chunk_size = 40000000
file_num = 1
with open("pytorch_model.bin", "rb") as f:
    chunk = f.read(chunk_size)
    while chunk:
        with open("chunk_" + str(file_num), "wb") as chunk_file:
            chunk_file.write(chunk)
        file_num += 1
        chunk = f.read(chunk_size)

Code to reassemble one file:
chunks = !ls | grep chunk_
chunks = sorted(chunks, key=lambda x: int(x.split("_")[-1]))

for chunk in chunks:
    with open(chunk, "rb") as f:
        contents = f.read()
    if chunk == chunks[0]:
        write_mode = "wb"
    else:
        write_mode = "ab"
    with open("pytorch_model.bin", write_mode) as f:
        f.write(contents)

python 3.7.0, torch 1.5.1, transformers 4.2.2. I have no way to move files bigger than 40 MB.
TIA for your help!


title: RNN parameters not updating?

body:
I'm very new to PyTorch, and fairly new to neural networks in general.
I was trying to build a neural net that can guess gender given name, and I based off of the PyTorch RNN tutorial that judges nationality.
I got the code to run without errors, but the loss hardly changes, making me think the weights aren't updating...
Is this a problem with my input/output/target tensor setup? Or perhaps something wrong with my training function? I'm very lost, and any help would be appreciated :cold_sweat:
Here's my code:

from __future__ import unicode_literals, print_function, division
from io import open
import glob
import unicodedata
import string
import torch
import torchvision
import torch.nn as nn
import torch.optim as optim
import random
from torch.autograd import Variable

"""------GLOBAL VARIABLES------"""

all_letters = string.ascii_letters + " .,;'"
num_letters = len(all_letters)
all_names = {}
genders = ["Female", "Male"]

"""-------DATA EXTRACTION------"""

def findFiles(path):
    return glob.glob(path)

def unicodeToAscii(s):
    return ''.join(
        c for c in unicodedata.normalize('NFD', s)
        if unicodedata.category(c) != 'Mn'
        and c in all_letters
    )

# Read a file and split into lines
def readLines(filename):
    lines = open(filename, encoding='utf-8').read().strip().split('\n')
    return [unicodeToAscii(line) for line in lines]

for file in findFiles("/home/andrew/PyCharm/PycharmProjects/CantStop/data/names/*.txt"):
    gender = file.split("/")[-1].split(".")[0]
    names = readLines(file)
    all_names[gender] = names

"""-----DATA INTERPRETATION-----"""

def nameToTensor(name):
    tensor = torch.zeros(len(name), 1, num_letters)
    for index, letter in enumerate(name):
        tensor[index][0][all_letters.find(letter)] = 1
    return tensor

def outputToGender(output):
    gender, gender_index = output.data.topk(1)
    if gender_index[0][0] == 0:
        return "Female"
    return "Male"

"""------NETWORK SETUP------"""

class Net(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(Net, self).__init__()
        self.hidden_size = hidden_size
        #Layer 1
        self.Lin1 = nn.Linear(input_size+hidden_size, int((input_size+hidden_size)/2))
        self.ReLu1 = nn.ReLU()
        self.Batch1 = nn.BatchNorm1d(int((input_size+hidden_size)/2))
        #Layer 2
        self.Lin2 = nn.Linear(int((input_size+hidden_size)/2), output_size)
        self.ReLu2 = nn.ReLU()
        self.Batch2 = nn.BatchNorm1d(output_size)
        self.softMax = nn.LogSoftmax()
        #Hidden layer
        self.HidLin = nn.Linear(input_size+hidden_size, hidden_size)
        self.HidReLu = nn.ReLU()
        self.HidBatch = nn.BatchNorm1d(hidden_size)

    def forward(self, input, hidden):
        comb = torch.cat((input, hidden), 1)
        hidden = self.HidBatch(self.HidReLu(self.HidLin(comb)))
        output1 = self.Batch1(self.ReLu1(self.Lin1(comb)))
        output2 = self.softMax(self.Batch2(self.ReLu2(self.Lin2(output1))))
        return output2, hidden

    def initHidden(self):
        return Variable(torch.zeros(1, self.hidden_size))

NN = Net(num_letters, 128, 2)

"""------TRAINING------"""

def getRandomTrainingEx():
    gender = genders[random.randint(0, 1)]
    name = all_names[gender][random.randint(0, len(all_names[gender])-1)]
    gender_tensor = Variable(torch.LongTensor([genders.index(gender)]))
    name_tensor = Variable(nameToTensor(name))
    return gender_tensor, name_tensor, gender

def train(input, target):
    hidden = NN.initHidden()

    loss_func = nn.NLLLoss()

    alpha = 0.01

    NN.zero_grad()

    for i in range(input.size()[0]):
        output, hidden = NN(input[i], hidden)

    loss = loss_func(output, target)
    loss.backward()
    for w in NN.parameters():
        w.data.add_(-alpha, w.grad.data)

    return output, loss

for i in range(5000):
    gender_tensor, name_tensor, gender = getRandomTrainingEx()
    output, loss = train(name_tensor, gender_tensor)

    if i%500 == 0:
        print("Guess: %s, Correct: %s, Loss: %s" % (outputToGender(output), gender, loss.data[0]))

And here's the output:

Guess: Male, Correct: Male, Loss: 0.6931471824645996
Guess: Male, Correct: Female, Loss: 0.7400936484336853
Guess: Male, Correct: Male, Loss: 0.6755779385566711
Guess: Female, Correct: Female, Loss: 0.6648257374763489
Guess: Male, Correct: Male, Loss: 0.6765623688697815
Guess: Female, Correct: Male, Loss: 0.7330614924430847
Guess: Female, Correct: Female, Loss: 0.6565149426460266
Guess: Male, Correct: Female, Loss: 0.6946508884429932
Guess: Female, Correct: Female, Loss: 0.6621525287628174
Guess: Male, Correct: Male, Loss: 0.6662092804908752

Process finished with exit code 0


title: Debugging GAN covergence error

body:
Building a GAN to generate images. The images have 3 color channels, 96 x 96.

The images that are generated by the generator at the beginning are all black, which is an issue given that is statistically highly unlikely.

Also, the loss for both networks is not improving.

I have posted the entire code below, and commented to allow it to be easily read. This is my first time building a GAN and I am new to Pytorch so any help is very appreciated!

Thanks.

import torch
from torch.optim import Adam
from torch.utils.data import DataLoader
from torch.autograd import Variable

import numpy as np
import os
import cv2
from collections import deque

# training params
batch_size = 100
epochs = 1000

# loss function
loss_fx = torch.nn.BCELoss()

# processing images
X = deque()
for img in os.listdir('pokemon_images'):
    if img.endswith('.png'):
        pokemon_image = cv2.imread(r'./pokemon_images/{}'.format(img))
        if pokemon_image.shape != (96, 96, 3):
            pass
        else:
            X.append(pokemon_image)

# data loader for processing in batches
data_loader = DataLoader(X, batch_size=batch_size)

# covert output vectors to images if flag is true, else input images to vectors
def images_to_vectors(data, reverse=False):
    if reverse:
        return data.view(data.size(0), 3, 96, 96)
    else:
        return data.view(data.size(0), 27648)

# Generator model
class Generator(torch.nn.Module):
    def __init__(self):
        super(Generator, self).__init__()
        n_features = 1000
        n_out = 27648

        self.model = torch.nn.Sequential(
                torch.nn.Linear(n_features, 128),
                torch.nn.ReLU(),
                torch.nn.Linear(128, 256),
                torch.nn.ReLU(),
                torch.nn.Linear(256, 512),
                torch.nn.ReLU(),
                torch.nn.Linear(512, 1024),
                torch.nn.ReLU(),
                torch.nn.Linear(1024, n_out),
                torch.nn.Tanh()
        )

    def forward(self, x):
        img = self.model(x)
        return img

    def noise(self, s):
       x = Variable(torch.randn(s, 1000))
       return x

# Discriminator model
class Discriminator(torch.nn.Module):
    def __init__(self):
        super(Discriminator, self).__init__()
        n_features = 27648
        n_out = 1

        self.model = torch.nn.Sequential(
                torch.nn.Linear(n_features, 512),
                torch.nn.ReLU(),
                torch.nn.Linear(512, 256),
                torch.nn.ReLU(),
                torch.nn.Linear(256, n_out),
                torch.nn.Sigmoid()
        )

    def forward(self, img):
        output = self.model(img)
        return output

# discriminator training
def train_discriminator(discriminator, optimizer, real_data, fake_data):
    N = real_data.size(0)
    optimizer.zero_grad()

    # train on real
    # get prediction
    pred_real = discriminator(real_data)

    # calculate loss
    error_real = loss_fx(pred_real, Variable(torch.ones(N, 1)))

    # calculate gradients
    error_real.backward()

    # train on fake
    # get prediction
    pred_fake = discriminator(fake_data)

    # calculate loss
    error_fake = loss_fx(pred_fake, Variable(torch.ones(N, 0)))

    # calculate gradients
    error_fake.backward()

    # update weights
    optimizer.step()

    return error_real + error_fake, pred_real, pred_fake

# generator training
def train_generator(generator, optimizer, fake_data):
    N = fake_data.size(0)

    # zero gradients
    optimizer.zero_grad()

    # get prediction
    pred = discriminator(generator(fake_data))

    # get loss
    error = loss_fx(pred, Variable(torch.ones(N, 0)))

    # compute gradients
    error.backward()

    # update weights
    optimizer.step()

    return error

# Instance of generator and discriminator
generator = Generator()
discriminator = Discriminator()

# optimizers
g_optimizer = torch.optim.Adam(generator.parameters(), lr=0.001)
d_optimizer = torch.optim.Adam(discriminator.parameters(), lr=0.001)

# training loop
for epoch in range(epochs):
     for n_batch, batch in enumerate(data_loader, 0):
         N = batch.size(0)

         # Train Discriminator

         # REAL
         real_images = Variable(images_to_vectors(batch)).float()

         # FAKE
         fake_images = generator(generator.noise(N)).detach()

         # TRAIN
         d_error, d_pred_real, d_pred_fake = train_discriminator(
                 discriminator,
                 d_optimizer,
                 real_images,
                 fake_images
         )

         # Train Generator

         # generate noise
         fake_data = generator.noise(N)

         # get error based on discriminator
         g_error = train_generator(generator, g_optimizer, fake_data)

         # convert generator output to image and preprocess to show
         test_img = np.array(images_to_vectors(generator(fake_data), reverse=True).detach())
         test_img = test_img[0, :, :, :]
         test_img = test_img[..., ::-1]

         # show example of generated image
         cv2.imshow('GENERATED', test_img[0])
         if cv2.waitKey(1) & 0xFF == ord('q'):
             break

     print('EPOCH: {0}, D error: {1}, G error: {2}'.format(epoch, d_error, g_error))

cv2.destroyAllWindows()

# save weights
# torch.save('weights.pth')


title: convert sess.run to pytorch

body:
I am trying to convert a code from tf to pytorch.
The part of the code where I am stuck is this sess.run.  As fas as I know, pytorch doesn't need it, but I don't find the way to replicate it. I attach you the code.

TF:

ebnos_db = np.linspace(1,6, 6)
bers_no_training = np.zeros(shape=[ebnos_db.shape[0]])
for j in range(epochs):
    for i in range(ebnos_db.shape[0]):
        ebno_db = ebnos_db[i]
        bers_no_training[i] += sess.run(ber, feed_dict={
            batch_size: samples,
            noise_var: ebnodb2noisevar(ebno_db, coderate)
        })
bers_no_training /= epochs

samples is a int32 and ebnodb2noisevar() returns a float32.

BER in TF is calculated as:

ber = tf.reduce_mean(tf.cast(tf.not_equal(x, x_hat), dtype=tf.float32))

and in PT:

wrong_bits = ( torch.eq(x, x_hat).type(torch.float32) * -1 ) + 1
ber = torch.mean(wrong_bits)

I think BER is well computed, but the main problem is that I don't know how to convert sess.run into PyTorch, nor I completely understand its function.

Can anybody help me?

Thanks


title: I cant import the ucf 101 dataset (torchvision), 'list index out of range' error

body:
dataset = torchvision.datasets.UCF101(r'my_directory', annotation_path=r'my_directory2', frames_per_clip=16, step_between_clips=1, frame_rate=None, fold=1, train=True, transform=transforms.Compose([transforms.ToTensor()]), _precomputed_metadata=None, num_workers=1, _video_width=64, _video_height=64, _video_min_dimension=0, _audio_samples=0)

This line works, the problem is when I try to do any kind of operation using 'dataset', in particular this one:

data = torch.utils.data.DataLoader(dataset, batch_size=512, shuffle=True)

I get the error, so i can't work on the video data because I can't use the dataLoader, the error is:

IndexError: list index out of range

Complete error message:

    ---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-16-dc3631481cb3> in <module>
----> 1 data = torch.utils.data.DataLoader(dataset, batch_size=512, shuffle=True)

~\anaconda3\lib\site-packages\torch\utils\data\dataloader.py in __init__(self, dataset, batch_size, shuffle, sampler, batch_sampler, num_workers, collate_fn, pin_memory, drop_last, timeout, worker_init_fn, multiprocessing_context)
    211             else:  # map-style
    212                 if shuffle:
--> 213                     sampler = RandomSampler(dataset)
    214                 else:
    215                     sampler = SequentialSampler(dataset)

~\anaconda3\lib\site-packages\torch\utils\data\sampler.py in __init__(self, data_source, replacement, num_samples)
     90                              "since a random permute will be performed.")
     91
---> 92         if not isinstance(self.num_samples, int) or self.num_samples <= 0:
     93             raise ValueError("num_samples should be a positive integer "
     94                              "value, but got num_samples={}".format(self.num_samples))

~\anaconda3\lib\site-packages\torch\utils\data\sampler.py in num_samples(self)
     98         # dataset size might change at runtime
     99         if self._num_samples is None:
--> 100             return len(self.data_source)
    101         return self._num_samples
    102

~\anaconda3\lib\site-packages\torchvision\datasets\ucf101.py in __len__(self)
     96
     97     def __len__(self):
---> 98         return self.video_clips.num_clips()
     99
    100     def __getitem__(self, idx):

~\anaconda3\lib\site-packages\torchvision\datasets\video_utils.py in num_clips(self)
    241         Number of subclips that are available in the video list.
    242         """
--> 243         return self.cumulative_sizes[-1]
    244
    245     def get_clip_location(self, idx):

IndexError: list index out of range


title: Updating based on two different loss functions, but with a different optimizer learning rate after each one (pytorch)?

body:
I have a set up as follows, where I have an outer for loop iterating over epochs, and an inner for loop iterating over batches.

In the inner for loop, over batches, I'm usin a cross entropy loss, and using an Adam optimizer with a certain learning rate.

After the inner for loop (after all batches are evaluated), I'm then calculating another loss function based off of the output (a custom loss function), and optimizing.

However, I notice that when I define a different optimizer with a different learning rate, it doesn't seem to be training. When I keep the same optimizer, it seems like things change, but when I replace it, it doesn't. Example as follows:

net = <my defined model, from another function>

optimizer_1 = torch.optim.Adam(net.parameters(), lr=0.1)
optimizer_2 = torch.optim.Adam(net.parameters(), lr=0.01)

for epoch in range(num_epochs):
    for data in training_data: # these are the batches
        <get output here>
        loss1 = <compute loss function>
        optimizer_1.zero_grad()
        loss1.backward()
        optimizer_1.step()

    loss2 = <compute a different loss function here>
    optimizer_2.zero_grad() #use a second optimizer with a different learning rate
    loss.backward()
    loss.step()

When I do this, it seems like it doesn't actually carry through with the second optimization on the second loss function. Why is this? I want the second optimization to have a different learning rate than the first one. However, it seems like only continuing to use the first optimizer, optimizer_1, with its respective learning rate seems to work.


title: Tensorboard resume training plot

body: I ran a reinforcement learning training script which used Pytorch and logged data to tensorboardX and saved checkpoints. Now I want to continue training. How do I tell tensorboardX to continue from where I left off? Thank you!


title: Pytorch second derivative returns None

body:
I am unable to take a second derivative of the following function. When I want the second derivative with respect to u_s it works but with x_s it doesn't work.

Does anyone know what I have done wrong here?

def cost(xt, x_goal, u, Q, R):
        return (xt - x_goal).matmul(Q).matmul((xt - x_goal).transpose(0,1)) + u.matmul(R).matmul(u)

x_s = tr.tensor([ 0.0000, -1.0000,  0.0000], dtype=torch.float64,  requires_grad=True)
u_s = tr.tensor([-0.2749], dtype=torch.float64, requires_grad=True)
c = cost(x_s, x_Goal, u_s, tr.tensor(Q), tr.tensor(R))

c
   output:
   tensor([[4.0076]], dtype=torch.float64, grad_fn=<ThAddBackward>)

Cu = grad(c, u_s, create_graph=True)[0]
Cu
   output:
   tensor([-0.0550], dtype=torch.float64, grad_fn=<ThAddBackward>)

Cuu = grad(Cu, u_s, allow_unused=True)[0]
Cuu
   output:
   tensor([0.2000], dtype=torch.float64)

Cux = grad(Cu, x_s, allow_unused=True)
Cux
    output:
    (None,)

I am guessing the Cu itself is completely independent of x_s, but then the derivative should be zero at least, not None!


title: The training loss of vgg16 implemented in pytorch does not decrease

body:
I want to try some toy examples in pytorch, but the training loss does not decrease in the training.

Some info is provided here:

The model is vgg16, consisted of 13 conv layers and 3 dense layers.
The data is cifar100 in pytorch.
I choose cross entropy as the loss function.

The code is as follows

# encoding: utf-8

import torch
import torch.optim as optim
import torch.nn as nn
import torch.nn.functional as F
import torchvision.transforms as transforms
import torchvision

import numpy as np

class VGG16(torch.nn.Module):
    def __init__(self, n_classes):
        super(VGG16, self).__init__()

        # construct model
        self.conv1_1 = nn.Conv2d(3, 64, 3, padding=1)
        self.conv1_2 = nn.Conv2d(64, 64, 3, padding=1)
        self.conv2_1 = nn.Conv2d(64, 128, 3, padding=1)
        self.conv2_2 = nn.Conv2d(128, 128, 3, padding=1)
        self.conv3_1 = nn.Conv2d(128, 256, 3, padding=1)
        self.conv3_2 = nn.Conv2d(256, 256, 3, padding=1)
        self.conv3_3 = nn.Conv2d(256, 256, 3, padding=1)
        self.conv4_1 = nn.Conv2d(256, 512, 3, padding=1)
        self.conv4_2 = nn.Conv2d(512, 512, 3, padding=1)
        self.conv4_3 = nn.Conv2d(512, 512, 3, padding=1)
        self.conv5_1 = nn.Conv2d(512, 512, 3, padding=1)
        self.conv5_2 = nn.Conv2d(512, 512, 3, padding=1)
        self.conv5_3 = nn.Conv2d(512, 512, 3, padding=1)

        self.fc6 = nn.Linear(512, 512)
        self.fc7 = nn.Linear(512, 512)
        self.fc8 = nn.Linear(512, n_classes)

    def forward(self, x):
        x = F.relu(self.conv1_1(x))
        x = F.relu(self.conv1_2(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv2_1(x))
        x = F.relu(self.conv2_2(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv3_1(x))
        x = F.relu(self.conv3_2(x))
        x = F.relu(self.conv3_3(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv4_1(x))
        x = F.relu(self.conv4_2(x))
        x = F.relu(self.conv4_3(x))
        x = F.max_pool2d(x, (2, 2))

        x = F.relu(self.conv5_1(x))
        x = F.relu(self.conv5_2(x))
        x = F.relu(self.conv5_3(x))
        x = F.max_pool2d(x, (2, 2))

        x = x.view(-1, self.num_flat_features(x))

        x = F.relu(self.fc6(x))
        x = F.relu(self.fc7(x))
        x = self.fc8(x)
        return x

    def num_flat_features(self, x):
        size = x.size()[1:]

        num_features = 1
        for s in size:
            num_features *= s
        return num_features

if __name__ == '__main__':

    BATCH_SIZE = 128
    LOG_INTERVAL = 5

    # data
    transform = transforms.Compose([
        transforms.ToTensor()
    ])

    trainset = torchvision.datasets.CIFAR100(
        root='./data',
        train=True,
        download=True,
        transform=transform
    )

    testset = torchvision.datasets.CIFAR100(
        root='./data',
        train=False,
        download=True,
        transform=transform
    )

    trainloader = torch.utils.data.DataLoader(trainset, batch_size=BATCH_SIZE, shuffle=True)
    testloader = torch.utils.data.DataLoader(testset, batch_size=BATCH_SIZE, shuffle=False)

    # model
    vgg16 = VGG16(100)
    vgg16.cuda()

    # optimizer
    optimizer = optim.SGD(vgg16.parameters(), lr=0.01)

    # loss
    criterion = nn.CrossEntropyLoss()

    print('———— Train Start —————')
    for epoch in range(20):
        running_loss = 0.
        for step, (batch_x, batch_y) in enumerate(trainloader):
            batch_x, batch_y = batch_x.cuda(), batch_y.cuda()

            #
            optimizer.zero_grad()

            output = vgg16(batch_x)
            loss = criterion(output, batch_y)
            loss.backward()
            optimizer.step()

            running_loss += loss.item()

            if step % LOG_INTERVAL == 0:
                print('[%d, %4d] loss: %.4f' % (epoch, step, running_loss / LOG_INTERVAL))
                running_loss = 0.

        def test():
            print('———— Test Start ————')
            correct = 0
            total = 0

            #
            with torch.no_grad():
                for test_x, test_y in testloader:
                    images, labels = test_x.cuda(), test_y.cuda()
                    output = vgg16(images)
                    _, predicted = torch.max(output.data, 1)
                    total += labels.size(0)
                    correct += (predicted == labels).sum().item()

            accuracy = 100 * correct / total
            print('Accuracy of the network is: %.4f %%' % accuracy)
            print('———— Test Finish ————')

        test()

    print('———— Train Finish —————')

The loss stays around 4.6060 and never decrease. I have tried different learning rate but does not work.


title: Does tf.math.reduce_max allows gradient flow like torch.max?

body:
I am trying to build a multi-label binary classification model in Tensorflow. The model has a tf.math.reduce_max operator between two layers (It is not Max Pooling, it's for a different purpose).
And the number of classes is 3.
I am using Binary Cross Entropy loss and using Adam optimizer.
Even after hours of training, when I check the predictions, all the predictions are in the range 0.49 to 0.51.
It seems that the model is not learning anything and is making random predictions, which is making me think that using a tf.math.reduce_max function may be causing the problems.
However, I read on the web that the torch.max function allows back propagation of gradients through it.
When I checked the Graph in Tensorboard, I saw that the graph is showing unconnected at the tf.math.reduce_max operator.
SO, does this operator allows gradients ot back propagate through it?
EDIT :
Addin the code
input_tensor = Input(shape=(256, 256, 3))
base_model_toc = VGG16(input_tensor=input_tensor,weights='imagenet',pooling=None, include_top=False)

x = base_model.output

x = GlobalAveragePooling2D()(x)

x = tf.math.reduce_max(x,axis=0,keepdims=True)

x = Dense(1024,activation='relu')(x)

output_1 = Dense(3, activation='sigmoid')(x)

model_a = Model(inputs=base_model_toc.input, outputs=output_1)

for layer in base_model.layers:
    layer.trainable = True

THe tf.math.reduce_max is done along axis = 0 becasue that is what needs to be done in this model
Optimizer that I am using is Adam with initial learning rate 0.00001


title: pytorch Neural Network training using GPU 2x slower than CPU on colab

body:
I'm training a neural network on 2000 data points using pytorch on google's colab.
Using their CPU I timed my NN training to take 13 [s].
Using GPUs, the same model training took 23[s].
I ensured colab is using GPUs and I initiated each tensor to be manipulated using GPUs.
I think the "for" loop is slowing me down.
How can I speed up my code? Thank you!
My code:
#initialize tensors to GPU processor
X_train = torch.from_numpy(np.array(X_train)).float().cuda()
X_test = torch.from_numpy(np.array(X_test)).float().cuda()
y_train = torch.from_numpy(np.array(y_train)).float().cuda()
y_test = torch.from_numpy(np.array(y_test)).float().cuda()

X = X_train
y = y_train
in_size = len(X[0])

torch.manual_seed(0)
model = NNet(in_size)

#initialize GPU as model processor
model.cuda()

criterion = torch.nn.L1Loss(reduction='mean')
optimizer = torch.optim.Adam(model.parameters(), lr=1e-4)
stats = {'train_err':[],'test_err':[]}

start = time.time()

#train the NN model
for epoch in range(10000):
    y_out = model(X)
    loss = criterion(y_out,y)
    if epoch % 100 == 99:
        stats['train_err'].append(loss.item()) # train MAE
        y_tem = model(X_test).detach()
        stats['test_err'].append(erroranalysis(y_tem, y_test,y_len)[1]) # test MAE
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

end = time.time()
print(end - start)

The "erroranalysis" function:
def erroranalysis(y,fx,len_y):
    y = y.flatten()
    fx = fx.flatten()

    d = torch.abs(y - fx)
    dif = torch.sum(d)
    dif2 = torch.sum(d*d)
    RMSE = torch.sqrt(dif2/len_y)
    MAE =  (dif/len_y)
    k = len_y
    covar = (k*torch.dot(y,fx)-torch.sum(y)*torch.sum(fx))
    vary = (k*torch.sum(y*y)-(torch.sum(y))**2)
    varfx = (k*torch.sum(fx*fx)-(torch.sum(fx))**2)
    R2 = (covar**2/vary/varfx)
    return RMSE,MAE,R2

Output:
22.847734451293945
RMSE,MAE,R2 (tensor(0.3591, device='cuda:0'), tensor(0.0881, device='cuda:0'), tensor(0.9818, device='cuda:0'))