title: Calculating shape of conv1d layer in Pytorch

body:
How we can calculate the shape of conv1d layer in PyTorch. IS there any command to calculate size and shape of these layers in PyTorch.

            nn.Conv1d(depth_1, depth_2, kernel_size=kernel_size_2, stride=stride_size),
            nn.ReLU(),
            nn.MaxPool1d(kernel_size=2, stride=stride_size),
            nn.Dropout(0.25)```


title: Computing matrix derivatives with torch.autograd.grad (PyTorch)

body:
I am trying to compute matrix derivatives in PyTorch using torch.autograd.grad however I am running into few issues. Here is a minimal working example to reproduce the error.
theta = torch.tensor(np.random.uniform(low=-np.pi, high=np.pi), requires_grad=True)
rot_mat = torch.tensor([[torch.cos(theta), torch.sin(theta), 0],
                        [-torch.sin(theta), torch.cos(theta), 0]],
                        dtype=torch.float, requires_grad=True)
torch.autograd.grad(outputs=rot_mat,
                    inputs=theta, grad_outputs=torch.ones_like(rot_mat),
                    create_graph=True, retain_graph=True)

This code results in the error "One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior."
I tried using allow_unused=True but the gradients are returned as None. I am not sure what is causing the graph to be disconnected here.


title: Keras/TensorFlow equivalent of PyTorch Conv1d

body:
I am currently in the process of converting a PyTorch code to TensorFlow (Keras). One of the layers used is Conv1d and the description of how to use it in PyTorch is given as
torch.nn.Conv1d(in_channels: int, out_channels: int, kernel_size: Union[T, Tuple[T]], stride: Union[T, Tuple[T]] = 1, padding: Union[T, Tuple[T]] = 0, dilation: Union[T, Tuple[T]] = 1, groups: int = 1, bias: bool = True, padding_mode: str = 'zeros')

where as in Keras (TF1.15) the description is given as
tf.keras.layers.Conv1D(filters, kernel_size, strides=1, padding='valid', data_format='channels_last', dilation_rate=1, activation=None, use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros', kernel_regularizer=None, bias_regularizer=None, activity_regularizer=None, kernel_constraint=None, bias_constraint=None, **kwargs)

I have not been able to reproduce the same output I get in PyTorch in TensorFlow. i.e. for the sample code in PyTorch
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np

B, K, L, N = 4, 10, 128, 64
mixture = torch.randint(3, (B, K, L), dtype=torch.float32)
# L2 Norm along L axis
EPS = 1e-8
norm_coef = torch.norm(mixture, p=2, dim=2, keepdim=True)  # B x K x 1
norm_mixture = mixture / (norm_coef + EPS)  # B x K x L
# 1-D gated conv
norm_mixture = torch.unsqueeze(norm_mixture.view(-1, L), 2)  # B*K x L x 1
conv1d_U = nn.Conv1d(L, N, kernel_size=1, stride=1, bias=False)
conv_out = conv1d_U(norm_mixture)
conv_out = F.relu(conv_out)  # B*K x N x 1
mixture_w = conv_out.view(B, K, N)  # B x K x N

weights = conv1d_U.weight.data

In TensorFlow to get similar dimensions for the output the code can be found below
import tensorflow as tf
import numpy as np

def build_net():
    # Encoder
    mixture = tf.keras.layers.Input(shape=(10, 128), name='mixture', batch_size=4)  # [B,K,L]
    norm_coef = tf.keras.backend.sqrt(tf.keras.backend.sum(mixture ** 2, axis=2, keepdims=True) + 1e-8)  # [B,K,1]
    norm_mixture = mixture / norm_coef  # [B, K, L]
    norm_mixture = tf.keras.backend.expand_dims(tf.keras.backend.reshape(norm_mixture, [-1, 128]), axis=2)  # [B*K,L,1]
    conv = tf.keras.layers.Conv1D(filters=64, kernel_size=1, activation='relu', use_bias=False, name='conv')(norm_mixture)  # [B*K,N,1]
    mixture_w = tf.keras.backend.reshape(conv, [4, -1, 64])  # [B, K, N]
    return tf.keras.models.Model(inputs=mixture, outputs=mixture_w)

model = build_net()
weights = model.get_weights()
inp = np.random.randn(4, 10, 128)
out = model.predict(inp)

Comparing the dimensions of the weights in either case the Conv1d operation is clearly different than TensorFlow (Keras), how should the TF code be changed to reflect the same operation?


title: PyTorch error isDifferentiableType(variable.scalar_type()) for calculating det of a complex matrix

body:
Following up this post
When I want to use complex_det function to calculate det of a complex matrix I face this error:
RuntimeError: isDifferentiableType(variable.scalar_type()) INTERNAL ASSERT FAILED at "/pytorch/torch/csrc/autograd/functions/utils.h":59, please report a bug to PyTorch.

any idea how I can fix it?
<ipython-input-76-246d142f8871> in complex_det(A)
      3         return torch.view_as_complex(torch.stack((A.real.diag(), A.imag.diag()),dim=1))
      4     #Perform LU decomposition to matrix A:
----> 5     A_LU, pivots = A.lu()
      6     P, A_L, A_U = torch.lu_unpack(A_LU, pivots)
      7     #Det. of multiplied matrices is multiplcation of det.:

/usr/local/lib/python3.6/dist-packages/torch/tensor.py in lu(self, pivot, get_infos)
    332         r"""See :func:`torch.lu`"""
    333         # If get_infos is True, then we don't need to check for errors and vice versa
--> 334         LU, pivots, infos = torch._lu_with_info(self, pivot=pivot, check_errors=(not get_infos))
    335         if get_infos:
    336             return LU, pivots, infos


title: Error when trying to send neural net generated image in flask

body:
I'm making a basic api to request for generated images from a generator model from pytorch. I've done this using flask and I'm running it locally on MacOS. Everything works and the image returns but then python quits unexpectedly. Here is the code and the error:

Error:

   2019-03-25 16:21:23.514 Python[78776:1407049] WARNING: NSWindow drag regions should only be invalidated on the Main Thread! This will throw an exception in the future. Called from (
    0   AppKit                              0x00007fff4f96fccc -[NSWindow(NSWindow_Theme) _postWindowNeedsToResetDragMarginsUnlessPostingDisabled] + 386
    1   AppKit                              0x00007fff4f96d07c -[NSWindow _initContent:styleMask:backing:defer:contentView:] + 1488
    2   AppKit                              0x00007fff4f96caa6 -[NSWindow initWithContentRect:styleMask:backing:defer:] + 45
    3   _macosx.cpython-37m-darwin.so       0x000000010fe634c0 -[Window initWithContentRect:styleMask:backing:defer:withManager:] + 80
    4   _macosx.cpython-37m-darwin.so       0x000000010fe66a17 FigureManager_init + 327
    5   Python                              0x00000001023780bc wrap_init + 12
    6   Python                              0x000000010232fe09 wrapperdescr_call + 121
    7   Python                              0x0000000102328ae1 _PyObject_FastCallKeywords + 433
    8   Python                              0x00000001023e76d4 call_function + 420
    9   Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    10  Python                              0x0000000102329100 function_code_fastcall + 128
    11  Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    12  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    13  Python                              0x0000000102378001 slot_tp_init + 145
    14  Python                              0x0000000102373959 type_call + 297
    15  Python                              0x0000000102328ae1 _PyObject_FastCallKeywords + 433
    16  Python                              0x00000001023e76d4 call_function + 420
    17  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    18  Python                              0x0000000102329100 function_code_fastcall + 128
    19  Python                              0x00000001023e7812 call_function + 738
    20  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    21  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    22  Python                              0x000000010232886b _PyFunction_FastCallDict + 523
    23  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    24  Python                              0x0000000102328df7 PyObject_Call + 135
    25  Python                              0x00000001023e4ae7 _PyEval_EvalFrameDefault + 25975
    26  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    27  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    28  Python                              0x00000001023e7812 call_function + 738
    29  Python                              0x00000001023e4877 _PyEval_EvalFrameDefault + 25351
    30  Python                              0x0000000102329100 function_code_fastcall + 128
    31  Python                              0x00000001023e7812 call_function + 738
    32  Python                              0x00000001023e4877 _PyEval_EvalFrameDefault + 25351
    33  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    34  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    35  Python                              0x00000001023e7812 call_function + 738
    36  Python                              0x00000001023e4877 _PyEval_EvalFrameDefault + 25351
    37  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    38  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    39  Python                              0x00000001023e7812 call_function + 738
    40  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    41  Python                              0x0000000102329100 function_code_fastcall + 128
    42  Python                              0x00000001023e7812 call_function + 738
    43  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    44  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    45  Python                              0x000000010232886b _PyFunction_FastCallDict + 523
    46  Python                              0x00000001023e4ae7 _PyEval_EvalFrameDefault + 25975
    47  Python                              0x0000000102329100 function_code_fastcall + 128
    48  Python                              0x00000001023e7812 call_function + 738
    49  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    50  Python                              0x0000000102329100 function_code_fastcall + 128
    51  Python                              0x00000001023e7812 call_function + 738
    52  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    53  Python                              0x0000000102329100 function_code_fastcall + 128
    54  Python                              0x00000001023e7812 call_function + 738
    55  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    56  Python                              0x0000000102329100 function_code_fastcall + 128
    57  Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    58  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    59  Python                              0x0000000102376bc6 slot_tp_call + 150
    60  Python                              0x0000000102328ae1 _PyObject_FastCallKeywords + 433
    61  Python                              0x00000001023e76d4 call_function + 420
    62  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    63  Python                              0x00000001023370de gen_send_ex + 206
    64  Python                              0x00000001023e3fb8 _PyEval_EvalFrameDefault + 23112
    65  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    66  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    67  Python                              0x00000001023e7812 call_function + 738
    68  Python                              0x00000001023e4877 _PyEval_EvalFrameDefault + 25351
    69  Python                              0x00000001023e8336 _PyEval_EvalCodeWithName + 2422
    70  Python                              0x0000000102328c91 _PyFunction_FastCallKeywords + 257
    71  Python                              0x00000001023e7812 call_function + 738
    72  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    73  Python                              0x0000000102329100 function_code_fastcall + 128
    74  Python                              0x00000001023e7812 call_function + 738
    75  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    76  Python                              0x0000000102329100 function_code_fastcall + 128
    77  Python                              0x00000001023e7812 call_function + 738
    78  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    79  Python                              0x0000000102329100 function_code_fastcall + 128
    80  Python                              0x00000001023e7812 call_function + 738
    81  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    82  Python                              0x0000000102329100 function_code_fastcall + 128
    83  Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    84  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    85  Python                              0x0000000102378001 slot_tp_init + 145
    86  Python                              0x0000000102373959 type_call + 297
    87  Python                              0x0000000102328ae1 _PyObject_FastCallKeywords + 433
    88  Python                              0x00000001023e76d4 call_function + 420
    89  Python                              0x00000001023e47d6 _PyEval_EvalFrameDefault + 25190
    90  Python                              0x0000000102329100 function_code_fastcall + 128
    91  Python                              0x00000001023e7812 call_function + 738
    92  Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    93  Python                              0x0000000102329100 function_code_fastcall + 128
    94  Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    95  Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    96  Python                              0x0000000102328df7 PyObject_Call + 135
    97  Python                              0x00000001023e4ae7 _PyEval_EvalFrameDefault + 25975
    98  Python                              0x0000000102329100 function_code_fastcall + 128
    99  Python                              0x00000001023e7812 call_function + 738
    100 Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    101 Python                              0x0000000102329100 function_code_fastcall + 128
    102 Python                              0x00000001023e7812 call_function + 738
    103 Python                              0x00000001023e47bc _PyEval_EvalFrameDefault + 25164
    104 Python                              0x0000000102329100 function_code_fastcall + 128
    105 Python                              0x00000001023286f4 _PyFunction_FastCallDict + 148
    106 Python                              0x0000000102329b3f _PyObject_Call_Prepend + 143
    107 Python                              0x0000000102328df7 PyObject_Call + 135
    108 Python                              0x000000010246fbd7 t_bootstrap + 71
    109 Python                              0x0000000102426819 pythread_wrapper + 25
    110 libsystem_pthread.dylib             0x00007fff7f7ce305 _pthread_body + 126
    111 libsystem_pthread.dylib             0x00007fff7f7d126f _pthread_start + 70
    112 libsystem_pthread.dylib             0x00007fff7f7cd415 thread_start + 13
)
127.0.0.1 - - [25/Mar/2019 16:21:23] "GET /sdfa HTTP/1.1" 200 -
Assertion failed: (NSViewIsCurrentlyBuildingLayerTreeForDisplay() != currentlyBuildingLayerTree), function NSViewSetCurrentlyBuildingLayerTreeForDisplay, file /BuildRoot/Library/Caches/com.apple.xbs/Sources/AppKit/AppKit-1671.20.108/AppKit.subproj/NSView.m, line 14143

Main:

from flask import Flask, request, send_file
import loadModel

app = Flask(__name__)

@app.route('/')
def index():
    return 'this is the homepage'

@app.route('/<ganType>')
def generate(ganType):
    loadModel.loadModel()
    return send_file('fakes.jpg')

if __name__ == "__main__":
    app.run(debug=True)

Load Model:

import torch
import torch.nn as nn
import torchvision.utils as vutils
import matplotlib.pyplot as plt
import numpy as np

batch_size = 100
image_size = 64
nc = 3
nz = 100
ngf = 64
ndf = 64
num_epochs = 1
lr = 0.0002
beta1 = 0.5

class Generator(nn.Module): # nn.Module is the base class for all neural net modules.
    def __init__(self):
        super(Generator, self).__init__() # Calls the parent's initialization method

        self.main = nn.Sequential(
             # input is Z, going into a convolution
            nn.ConvTranspose2d( nz, ngf * 8, 4, 1, 0, bias=False),
            nn.BatchNorm2d(ngf * 8),
            nn.ReLU(True),
            # state size. (ngf*8) x 4 x 4
            nn.ConvTranspose2d(ngf * 8, ngf * 4, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            # state size. (ngf*4) x 8 x 8
            nn.ConvTranspose2d( ngf * 4, ngf * 2, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            # state size. (ngf*2) x 16 x 16
            nn.ConvTranspose2d( ngf * 2, ngf, 4, 2, 1, bias=False),
            nn.BatchNorm2d(ngf),
            nn.ReLU(True),
            # state size. (ngf) x 32 x 32
            nn.ConvTranspose2d( ngf, nc, 4, 2, 1, bias=False),
            nn.Tanh()
            # state size. (nc) x 64 x 64
        )

    def forward(self, input):
        return self.main(input)

def loadModel():

    # device = torch.device("cpu")
    model = Generator()
    model.load_state_dict(torch.load('generator.pt', map_location='cpu'))

    model.eval()
    first_list = []
    img_list = []
    # Generate some images to test the model.

    noise = torch.randn(64, nz, 1, 1)
    fake = model(noise).detach().cpu()
    # inception_img_list.append(np.transpose(fake[0],(1,2,0)))
    img_list.append(vutils.make_grid(fake, padding=2, normalize=True))

    # img_list.append(first_list)
    # plt.subplot(1,2,2)
    plt.axis("off")
    plt.title("Fake Images")
    plt.imsave('fakes.jpg', np.transpose(img_list[-1],(1,2,0)))

    return 'hello'

if __name__ == "__main__":

    loadModel()

As I said, it actually works and returns the generated image. It just also crashes python. I'm not sure if I am doing something incorrectly or if its a MacOs problem. Any help would be appreciated.


title: How to solve "RuntimeError: 1D target tensor expected, multi-target not supported" in multi-class classification?

body:
Our goal is to build a multi-class image classifier using Pytorch and based on the EMNIST dataset (black and white pictures of letters).
The shape of our training data X_train is (124800, 28, 28).
The shape of the original target variables y_train is (124800, 1), however, we created a one-hot encoding so that now the shape is (124800, 26).
The model that we are building should have 26 output variables, each representing the probability of one letter.
We read in our data as follows:
import scipy .io
emnist = scipy.io.loadmat(DATA_DIR + '/emnist-letters.mat')
data = emnist ['dataset']
X_train = data ['train'][0, 0]['images'][0, 0]
X_train = X_train.reshape((-1,28,28), order='F')

y_train = data ['train'][0, 0]['labels'][0, 0]

Then, we created a one-hot-encoding as follows:
y_train_one_hot = np.zeros([len(y_train), 27])

for i in range (0, len(y_train)):
    y_train_one_hot[i, y_train[i][0]] = 1

y_train_one_hot = np.delete(y_train_one_hot, 0, 1)

We create the dataset with:
train_dataset = torch.utils.data.TensorDataset(torch.from_numpy(X_train), torch.from_numpy(y_train_one_hot))

batch_size = 128
n_iters = 3000
num_epochs = n_iters / (len(train_dataset) / batch_size)
num_epochs = int(num_epochs)

train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
                                           batch_size=batch_size,
                                           shuffle=True)

And then we build our model as follows:
class CNNModel(nn.Module):

    def __init__(self):
        super(CNNModel, self).__init__()

        # Convolution 1
        self.cnn1 = nn.Conv2d(in_channels=1, out_channels=16, kernel_size=5, stride=1, padding=0)
        self.relu1 = nn.ReLU()

        # Max pool 1
        self.maxpool1 = nn.MaxPool2d(2,2)

        # Convolution 2
        self.cnn2 = nn.Conv2d(in_channels=16, out_channels=32, kernel_size=5, stride=1, padding=0)
        self.relu2 = nn.ReLU()

        # Max pool 2
        self.maxpool2 = nn.MaxPool2d(kernel_size=2)

        # Fully connected 1 (readout)
        self.fc1 = nn.Linear(32 * 4 * 4, 26)

    def forward(self, x):
        # Convolution 1
        out = self.cnn1(x.float())
        out = self.relu1(out)

        # Max pool 1
        out = self.maxpool1(out)

        # Convolution 2
        out = self.cnn2(out)
        out = self.relu2(out)

        # Max pool 2
        out = self.maxpool2(out)

        # Resize
        # Original size: (100, 32, 7, 7)
        # out.size(0): 100
        # New out size: (100, 32*7*7)
        out = out.view(out.size(0), -1)

        # Linear function (readout)
        out = self.fc1(out)

        return out

model = CNNModel()

criterion = nn.CrossEntropyLoss()

learning_rate = 0.01

optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)

And then we train the model as follows:
iter = 0
for epoch in range(num_epochs):
    for i, (images, labels) in enumerate(train_loader):

        # Add a single channel dimension
        # From: [batch_size, height, width]
        # To: [batch_size, 1, height, width]
        images = images.unsqueeze(1)

        # Forward pass to get output/logits
        outputs = model(images)

        # Clear gradients w.r.t. parameters
        optimizer.zero_grad()

        # Forward pass to get output/logits
        outputs = model(images)

        # Calculate Loss: softmax --> cross entropy loss
        loss = criterion(outputs, labels)

        # Getting gradients w.r.t. parameters
        loss.backward()

        # Updating parameters
        optimizer.step()

        iter += 1

        if iter % 500 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Iterate through test dataset
            for images, labels in test_loader:

                images = images.unsqueeze(1)

                # Forward pass only to get logits/output
                outputs = model(images)

                # Get predictions from the maximum value
                _, predicted = torch.max(outputs.data, 1)

                # Total number of labels
                total += labels.size(0)

                correct += (predicted == labels).sum()

            accuracy = 100 * correct / total

            # Print Loss
            print('Iteration: {}. Loss: {}. Accuracy: {}'.format(iter, loss.data[0], accuracy))

However, when we run this, we get the following error:
    ---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-c26c43bbc32e> in <module>()
     21
     22         # Calculate Loss: softmax --> cross entropy loss
---> 23         loss = criterion(outputs, labels)
     24
     25         # Getting gradients w.r.t. parameters

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    930     def forward(self, input, target):
    931         return F.cross_entropy(input, target, weight=self.weight,
--> 932                                ignore_index=self.ignore_index, reduction=self.reduction)
    933
    934

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2315     if size_average is not None or reduce is not None:
   2316         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2317     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2318
   2319

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2113                          .format(input.size(0), target.size(0)))
   2114     if dim == 2:
-> 2115         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)
   2116     elif dim == 4:
   2117         ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

RuntimeError: 1D target tensor expected, multi-target not supported

We expect that we do something wrong when we initialize/use our loss function. What can we do so that we can start training our model?


title: How to convert sparse to dense adjacency matrix?

body:
I am trying to convert a sparse adjacency matrix/list that only contains the indices of the non-zero elements ([[rows], [columns]]) to a dense matrix that contains 1s at the indices and otherwise 0s. I found a solution using to_dense_adj from Pytorch geometric (Documentation). But this does not exactly what I want, since the shape of the dense matrix is not as expected. Here is an example:
sparse_adj = torch.tensor([[0, 1, 2, 1, 0], [0, 1, 2, 3, 4]])

So the dense matrix should be of size 5x3 (the second array "stores" the columns; with non-zero elements at (0,0), (1,1), (2,2),(1,3) and (0,4)) because the elements in the first array are lower or equal than 2.
However,
dense_adj = to_dense(sparse_adj)[0]

outputs a dense matrix, but of shape (5,5). Is it possible to define the output shape or is there a different solution to get what I want?
Edit: I have a solution to convert it back to the sparse representation now that works
dense_adj = torch.sparse.FloatTensor(sparse_adj, torch.ones(5), torch.Size([3,5])).to_dense()
ind = dense_adj.nonzero(as_tuple=False).t().contiguous()
sparse_adj = torch.stack((ind[1], ind[0]), dim=0)

Or is there any alternative way that is better?


title: PyTorch, simple char level RNN, can't overfit one example

body:
I'm new to the PyTorch framework (coming from Theano and Tensorflow mainly):
I've followed the introduction tutorial and read the Classifying Names with a Character-Level RNN one.
I now try to adapt it to a char level LSTM model in order to gain some practical experience with the framework.
Basically I feed in the model sequences of char indices and give as target to the model the same sequence but shifted by one in the future.
However I can't overfit a simple training example and I don't see what I did wrong.
If someone can spot my mistake it would be very helpful.
Here is my code:

class LSTMTxtGen(nn.Module):
def __init__(self, hidden_dim, n_layer, vocab_size):
    super(LSTMTxtGen, self).__init__()
    self.n_layer = n_layer
    self.hidden_dim = hidden_dim
    self.vocab_size = vocab_size

    self.lstm = nn.LSTM(vocab_size, hidden_dim, n_layer, batch_first=True)

    # The linear layer that maps from hidden state space to tag space
    #self.hidden = self.init_hidden()

def init_hidden(self, batch_size):
    # Before we've done anything, we dont have any hidden state.
    # Refer to the Pytorch documentation to see exactly
    # why they have this dimensionality.
    # The axes semantics are (num_layers, minibatch_size, hidden_dim)
    return (autograd.Variable(torch.zeros(self.n_layer, batch_size,
                                          self.hidden_dim)),
            autograd.Variable(torch.zeros(self.n_layer, batch_size,
                                          self.hidden_dim)))

def forward(self, seqs):
    self.hidden = self.init_hidden(seqs.size()[0])
    lstm_out, self.hidden = self.lstm(seqs, self.hidden)
    lstm_out = lstm_out.contiguous().view(-1, self.hidden_dim)
    lstm_out = nn.Linear(lstm_out.size(1), self.vocab_size)(lstm_out)
    return lstm_out

model = LSTMTxtGen (
hidden_dim = 50,
n_layer = 3,
vocab_size = 44,
)

print(Model)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adamax(model.parameters())

G = Data.batch_generator(5,100)
batch_per_epoch, to_idx, to_char = next(G)
X, Y = next(G)
for epoch in range(10):
    losses = []
    for batch_count in range(batch_per_epoch):
        model.zero_grad()

        #mode.hidden = model.init_hidden()

        #X, Y = next(G)
        X = autograd.Variable(torch.from_numpy(X))
        Y = autograd.Variable(torch.from_numpy(Y))

       preds = model(X)

       loss = criterion(preds.view(-1, model.vocab_size), Y.view(-1))
       loss.backward()
       optimizer.step()
       losses.append(loss)

       if (batch_count % 20 == 0):
           print('Loss: ', losses[-1])


title: How do I modify this PyTorch convolutional neural network to accept a 64 x 64 image and properly output predictions?

body:
I took this convolutional neural network (CNN) from here. It accepts 32 x 32 images and defaults to 10 classes. However, I have 64 x 64 images with 500 classes. When I pass in 64 x 64 images (batch size held constant at 32), I get the following error.

ValueError: Expected input batch_size (128) to match target batch_size (32).

The stack trace starts at the line loss = loss_fn(outputs, labels). The outputs.shape is [128, 500] and the labels.shape is [32].

The code is listed here for completeness.

class Unit(nn.Module):
    def __init__(self,in_channels,out_channels):
        super(Unit,self).__init__()
        self.conv = nn.Conv2d(in_channels=in_channels,kernel_size=3,out_channels=out_channels,stride=1,padding=1)
        self.bn = nn.BatchNorm2d(num_features=out_channels)
        self.relu = nn.ReLU()

    def forward(self,input):
        output = self.conv(input)
        output = self.bn(output)
        output = self.relu(output)
        return output

class SimpleNet(nn.Module):
    def __init__(self,num_classes=10):
        super(SimpleNet,self).__init__()

        self.unit1 = Unit(in_channels=3,out_channels=32)
        self.unit2 = Unit(in_channels=32, out_channels=32)
        self.unit3 = Unit(in_channels=32, out_channels=32)

        self.pool1 = nn.MaxPool2d(kernel_size=2)

        self.unit4 = Unit(in_channels=32, out_channels=64)
        self.unit5 = Unit(in_channels=64, out_channels=64)
        self.unit6 = Unit(in_channels=64, out_channels=64)
        self.unit7 = Unit(in_channels=64, out_channels=64)

        self.pool2 = nn.MaxPool2d(kernel_size=2)

        self.unit8 = Unit(in_channels=64, out_channels=128)
        self.unit9 = Unit(in_channels=128, out_channels=128)
        self.unit10 = Unit(in_channels=128, out_channels=128)
        self.unit11 = Unit(in_channels=128, out_channels=128)

        self.pool3 = nn.MaxPool2d(kernel_size=2)

        self.unit12 = Unit(in_channels=128, out_channels=128)
        self.unit13 = Unit(in_channels=128, out_channels=128)
        self.unit14 = Unit(in_channels=128, out_channels=128)

        self.avgpool = nn.AvgPool2d(kernel_size=4)

        self.net = nn.Sequential(self.unit1, self.unit2, self.unit3, self.pool1, self.unit4, self.unit5, self.unit6
                                 ,self.unit7, self.pool2, self.unit8, self.unit9, self.unit10, self.unit11, self.pool3,
                                 self.unit12, self.unit13, self.unit14, self.avgpool)

        self.fc = nn.Linear(in_features=128,out_features=num_classes)

    def forward(self, input):
        output = self.net(input)
        output = output.view(-1,128)
        output = self.fc(output)
        return output

Any ideas on how to modify this CNN to accept and properly return outputs?


title: PyTorch did not compute gradient and update parameters for 'masking' tensors?

body:
I am coding a LeNet in PyTorch for MNIST dataset; I add a tensor self.mask_fc1\2\3 to mask some certain connections for the full connection layers. The code is like this:

import torch
import torchvision
import torchvision.transforms as transforms
import matplotlib.pyplot as plt
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

def loadMNIST():
    transform = transforms.Compose([transforms.ToTensor()])

    trainset = torchvision.datasets.MNIST(root='./data', train=True,
                                          download=True, transform=transform)
    trainloader = torch.utils.data.DataLoader(trainset, batch_size=4,
                                              shuffle=True, num_workers=2)

    testset = torchvision.datasets.MNIST(root='./data', train=False,
                                         download=True, transform=transform)
    testloader = torch.utils.data.DataLoader(testset, batch_size=4,
                                             shuffle=False, num_workers=2)
    return trainloader, testloader

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(1, 6, 5, 1, 2)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

        self.mask_fc1 = torch.ones(16 * 5 * 5, 120, requires_grad=True)
        self.mask_fc2 = torch.ones(120, 84, requires_grad=True)
        self.mask_fc3 = torch.ones(84, 10, requires_grad=True)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, self.num_flat_features(x))

        # first layer
        x = x.matmul(self.fc1.weight.t() * self.mask_fc1)
        if self.fc1.bias is not None:
            x += torch.jit._unwrap_optional(self.fc1.bias)
        x = F.relu(x)

        # second layer
        x = x.matmul(self.fc2.weight.t() * self.mask_fc2)
        if self.fc2.bias is not None:
            x += torch.jit._unwrap_optional(self.fc2.bias)
        x = F.relu(x)

        # third layer
        x = x.matmul(self.fc3.weight.t() * self.mask_fc3)
        if self.fc3.bias is not None:
            x += torch.jit._unwrap_optional(self.fc3.bias)

        return x

    def num_flat_features(self, x):
        size = x.size()[1:]
        num_features = 1
        for s in size:
            num_features *= s
        return num_features

if __name__ == '__main__':
    trainloader, testloader = loadMNIST()

    net = Net()

    # train
    criterion = nn.CrossEntropyLoss()
    optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)

    for epoch in range(2):
        running_loss = 0.0
        for i, data in enumerate(trainloader, 0):
            inputs, labels = data

            optimizer.zero_grad()

            outputs = net(inputs)
            loss = criterion(outputs, labels)
            loss.backward()
            optimizer.step()

            # print
            running_loss += loss.item()
            if i % 2000 == 1999:
                # mean loss
                print('[%d, %5d] loss: %.3f' % (epoch + 1, i + 1, running_loss / 2000))
                running_loss = 0.0

    print('Finished Training')

    # print the mask
    print(net.mask_fc1)

I achieve the masking in the forward function and implement linear layer by myself rather than call x = F.relu(self.fc1(x)), and the model performs normally (at last in terms of the loss and accuracy).

However, when I print self.mask_fc1/2/3, the tensor does not change during the training. Since the tensor was set requires_grad=True in function __init__, I cannot understand why it does not change. Maybe it is because the tensor multiplication?


title: How to create embeddedings for a column that is a list of categorical values

body:
I am having some trouble deciding how to create embeddings for a categorical feature for my DNN model. The feature consists of a non fixed set of tags.

The feature is like:

column = [['Adventure','Animation','Comedy'],
          ['Adventure','Comedy'],
          ['Adventure','Children','Comedy']

I would like to do this with tensorflow so I know the tf.feature_column module should work, I just don't know which version to use.

Thanks!


title: Pytorch setting part of the training value into zero

body:
When I am training network, I want to assign a specific part of the output into zero. For example, temp = nn.conv2d(3,6) in 3-channel / 16 batch size
would have tensor value like this.

temp = tensor(16,6,36,36)

I want half of the channel value into zero.

temp[:,:3,:,:] = 0

But I got this error.

RuntimeError: one of the variables needed for gradient computation
has been modified by an inplace operation: [torch.cuda.FloatTensor
[1, 128, 15, 15]], which is output 0 of ReluBackward1, is at version
2; expected version 1 instead. Hint: the backtrace further above
shows the operation that failed to compute its gradient. The variable
in question was changed in there or anywhere later. Good luck!

How can I approach this problem?

I've tried with this code

def weight_init(self,x,label):
    if label.data[0]:
        x[:,:64,:,:] =0
        return x
    else:
        x[:,64:,:,:] =0
        return x


title: Why override Dataset instead of directly pass in input and labels, pytorch

body:
Sorry if what I say here is wrong -- new to pytorch.

From what I can tell there are two main ways of getting training data and passing through a network. One is to override Dataset and the other is to just prepare your data correctly and then iterate over it, like shown in this example: pytorch classification example

which does something like

rnn(input, hidden, output)
for i in range(input.size()[0]):
    output, hidden = rnn(input[i], hidden)

The other way would be to do something like

for epoch in range(epochs):
    for data, target in trainloader:
        computer model etc

where in this method, trainloader is from doing something like

trainloader = DataLoader(my_data)

after overriding getitem and len

My question here, is what are the differences between these methods, and why would you use one over the other? Also, it seems to me that overriding Dataset doesn't work for something that has lets say an input layer of size 100 nodes with an output of 10 nodes, since when you return getitem it needs a pair of (data, label). This seems like a case where I probably don't understand how to use Dataset very well, but that is why I'm asking in the first place. I think I read something about a collate function which might help in this scenario?


title: Convert Pytorch Yolact (.pth) model for deployment in C++

body:
I am using YOLACT++ and I want to deploy it using C++. Originally the model is saved as yolact.pth file and I realized from the issues that I couldn't directly convert .pth to .pt file that can be called in C++ see this issue and I would need to do convert it first to ONNX which gives out yolact.onnx as the output. I converted it to ONNX using this solution in the issues and in the comments below this one you can see the terminal commands he uses to achieve the yolact.onnx converted model from yolact.pth.
So my issue is I don't really understand what to do next to deploy it using C++. Should I be looking for solutions to convert from yolact.onnx to yolact.pt or can yolact.onnx be called from C++? There's an interesting direction given in one of the issues here for this work flow: Pytorch->ONNX->NCNN which he tested with C++ inference on ARM device. Is this what I am looking for. I am not very familiar with C++ so I dont know the direction.
I also tried adding this to end of of eval.py where model is being called:
    sm = torch.jit.script(net)
    sm.save("Yolact.pt")

and I got error:
    torch.jit.frontend.UnsupportedNodeError: with statements aren't
    supported: (line 570 in yolact.py)

and line 570 is:
     with timer.env('backbone'):

I commented it out and it gave error on the next "timer.env" in line 574 and so on!


title: What is the meaning of "x:" and of the following line?

body:
What is the meaning of x: and of the following line?

image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])

Can somebody explain the syntax of this line?

It is from PyTorch tutorial:
https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html

# Data augmentation and normalization for training
# Just normalization for validation
data_transforms = {
    'train': transforms.Compose([
        transforms.RandomResizedCrop(224),
        transforms.RandomHorizontalFlip(),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
    'val': transforms.Compose([
        transforms.Resize(256),
        transforms.CenterCrop(224),
        transforms.ToTensor(),
        transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
    ]),
}

data_dir = 'data/hymenoptera_data'
image_datasets = {x: datasets.ImageFolder(os.path.join(data_dir, x),
                                          data_transforms[x])
                  for x in ['train', 'val']}
dataloaders = {x: torch.utils.data.DataLoader(image_datasets[x], batch_size=4,
                                             shuffle=True, num_workers=4)
              for x in ['train', 'val']}
dataset_sizes = {x: len(image_datasets[x]) for x in ['train', 'val']}
class_names = image_datasets['train'].classes

device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")


title: Getting truth value of array with more than one element in ambigous for albuementation transform

body:
I am using albumentations for applying transform to a Pytorch model but getting this error and I m not getting any clue of what this error is about. Only thing I know is this is occuring due to transform which is being applied but not sure what is wrong with that.

ValueError: Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/opt/conda/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "<ipython-input-23-119ea6bc360e>", line 24, in __getitem__
    image = self.transform(image)
  File "/opt/conda/lib/python3.6/site-packages/albumentations/core/composition.py", line 164, in __call__
    need_to_run = force_apply or random.random() < self.p
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

This is are the snippets of code.
Dataloader getitem() method:

        image = cv2.imread(p_path)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        image = crop_image_from_gray(image)
        image = cv2.resize(image, (IMG_SIZE, IMG_SIZE))
        image = cv2.addWeighted ( image,4, cv2.GaussianBlur( image , (0,0) , 10) ,-4 ,128)
        print(image.shape)
        image = self.transform(image)

transforms applied :

val_transform = albumentations.Compose([
            Normalize(
                mean=[0.485, 0.456, 0.406],
                std=[0.229, 0.224, 0.225],
            ),
            ToTensor()
        ])

The class is called by:

valset       = MyDataset(val_df, transform = val_transform)


title: How to calculate geometric mean in a differentiable way?

body: How to calculate goemetric mean along a dimension using Pytorch? Some numbers can be negative. The function must be differentiable.


title: Compute gradient of bias b in pytorch for Softmax Regression

body:
I am learning pytorch materials and I encountered a problem that while using backpropagation using autograd moudule of pytorch, I was able to get the gradient of weights W, but not the bias b for the SOFTMAX REGRRESSION.

The code is presented below.

The problem is that I am getting b.grad= None and I was expecting not None values.

How can we fix this error?

Help will be appreciated.

import torch
from torch.autograd import Variable

dtype = torch.FloatTensor
dtype2 = torch.LongTensor

X_data = 10 * torch.rand(2, 300).type(dtype)
y_labels = torch.zeros(300).type(dtype2)
y_labels[0:100] = 1
y_labels[100:200] = 2
y_labels = y_labels.type(dtype2)
print(y_labels)

X = Variable(X_data, requires_grad = False)
y = Variable(y_labels, requires_grad = False)

N = X.shape[1]
decay = 0.01

W = Variable(torch.randn(3, 2).type(dtype), requires_grad = True)
b = Variable(torch.zeros(3, 1).type(dtype), requires_grad = True)
XT = X.t()
WT = W.t()

ones = Variable(torch.ones(1,N).type(dtype), requires_grad = False)

# print("XT.shape = {}".format(XT.shape)) # 300,2
# print("WT.shape = {}".format(WT.shape)) # 2,3
# print("b.shape = {}".format(b.shape)) # 3,1

# scores
z = torch.mm(XT, WT)  + torch.mm(b,ones).t() # 300,3

# Get exponentials and probabilities
exp_z = torch.exp(z - torch.max(z))
probs = exp_z / torch.sum(exp_z) # 300,3

# loss
correct_probs = probs.data[0:N,y.data.type(torch.LongTensor)] # 300,
logprobs = torch.log(correct_probs)
loss_data = -1/N * torch.sum(logprobs)

# regularization
loss_reg = 0.5 * decay * torch.sum(W**2)
loss = loss_data + loss_reg

# gradients
loss.backward()
print("W = {}".format(W))
print("b = {}".format(b))
print("W.grad = {}".format(W.grad))
print("b.grad = {}".format(b.grad)) # this is None


title: Reading multiple images as custom dataset for PyTorch?

body:
I want to read in multiple images for the main_image set and blur_image set. For example, 5 main images and 5 blurred images. The goal is determine what values for the kernel in the convolutional layer convert the main images to the blurred images. The assumption is that the same kernel is used to blur each of the 5 original images to produce the 5 blurred images

My code is available at: https://pastebin.com/PWf7rjd4 and https://pastebin.com/VxryDb7g

However, it seems to only be processing the first image, that is "1.png" for the main and blurred images. It is not processing images 2.png, 3.png, 4.png, and 5.png

How can I fix this?


title: Cuda Runtime/Driver incompatibility in docker container

body:
I'm trying to run this simple line of code in a docker container that comes with Pytorch.

import torch
torch.cuda.set_device(0)

I get this error:

RuntimeError: cuda runtime error (35) : CUDA driver version is insufficient for CUDA runtime version at torch/csrc/cuda/Module.cpp:32

Running torch.cuda.is_available() returns False.

The host machine has the most up-to-date Nvidia drivers. Pytorch ships with Cuda, so there should be no incompatibility issues.

What could cause this problem?

Edit:
@Patel Sunil's answer to this question answers my question, but I didn't come across this question in my search because their question is broad, while my question is specific to the cuda runtime/driver error. I posted this as a separate question for those who come across this error but don't know what it is a symptom of (namely, forgetting to use nvidia-docker).


title: Pytorch code error.... ' NoneType' object has no attribute 'zero_' 'unsupported operand type(s) for *: 'float' and 'NoneType''

body:
lr=0.001

x=np.linspace(-6,6,120)
y=0.5+3*x-x**2+np.exp(-0.4*x)
x=torch.from_numpy(x)
y=torch.from_numpy(y)

w0=torch.tensor(0.1,requires_grad=True)
w1=torch.tensor(0.1,requires_grad=True)
w2=torch.tensor(0.1,requires_grad=True)
w3=torch.tensor(0.1,requires_grad=True)

Y=w0+w1*x+w2*x**2+torch.exp(w3*x)
#opt=torch.optim.Adam([w0,w1,w2,w3],0.001)

L=nn.MSELoss()

for i in range(20):
  opt.zero_grad()
  err=L(Y,y)
  print(err)
  err.backward(retain_graph=True)
  with torch.no_grad():
    w0=w0-lr*w0.grad
    w1=w1-lr*w1.grad
    w2=w2-lr*w2.grad
    w3=w3-lr*w3.grad

    w0.grad.zero_()
    w1.grad.zero_()
    w2.grad.zero_()
    w3.grad.zero_()

from the line w0 = w0 - lr * w0.grad:
===> unsupported operand type(s) for *: 'float' and 'NoneType'
from the line  w0.grad.zero_():
===>  'NoneType' object has no attribute 'zero_'
comes up.
How should I fix it?
And

x=np.linspace(-6,6,120)
y=0.5+3*x-x**2+np.exp(-0.4*x)
x=torch.from_numpy(x)
y=torch.from_numpy(y)

w0=torch.tensor(0.1,requires_grad=True)
w1=torch.tensor(0.1,requires_grad=True)
w2=torch.tensor(0.1,requires_grad=True)
w3=torch.tensor(0.1,requires_grad=True)

Y=w0+w1*x+w2*x**2+torch.exp(w3*x)
opt=torch.optim.Adam([w0,w1,w2,w3],0.001)

L=nn.MSELoss()

for i in range(20):
  opt.zero_grad()
  err=L(Y,y)
  print(err)
  err.backward(retain_graph=True)
  opt.step()

If I try this code, then err is not updated. What is the problem? and how should I fix it

Further,  should the input of nn.MSELoss() be torch.double? Sometimes I get expected dtype Double but got dtype Float error.

What should be the type of parameters w0, w1, ...?


title: Resnet50 to classify human action on videos in PyTorch

body: I am trying to use a Resnet50 to classify human actions following this repo. I’ve implemented my own DataLoader to produce my batches from this dataset. Unlike the repo, I am not using the 3D CNN, but a simple PyTorch Resnet50. Thus, I need a 4D tensor input to feed the net, instead I have a 5D (Batch size, channels size, stacked images, Height, Width), where the stacked images are frames from the video in different time steps. Should I stop stacking this images and just iterate over the list of 4D tensors?


title: PyTorch grid_sample returns zero array

body:
I want to sample rectangular patch from my image by affine_grid/grid_sample
I created array which contains only 255 values

canvas1 = np.zeros((128, 128), dtype=np.uint8)
canvas1[:] = 255

Also i created grid

theta = torch.FloatTensor([[
    [11/2, 0, 63],
    [0, 11/2, 63],
]])
grid = F.affine_grid(theta, (1, 1, 11, 11))

Grid contains values like

     [[57.5000, 57.5000],
      [58.6000, 57.5000],
      [59.7000, 57.5000],
      [60.8000, 57.5000],
      [61.9000, 57.5000],
      [63.0000, 57.5000],
      [64.1000, 57.5000],
      [65.2000, 57.5000],
      [66.3000, 57.5000],
      [67.4000, 57.5000],
      [68.5000, 57.5000]],
...............

After that i called grid_sample

canvas1_torch = torch.FloatTensor(canvas1.astype(np.float32))
canvas1_torch = canvas1_torch.unsqueeze(0).unsqueeze(0)
sampled = F.grid_sample(canvas1_torch, grid, mode="bilinear")

Unfortunately sampled contains zero values (but canvas1_torch[0, 0, 63, 65]) is 255

What i am doing wrong?


title: Extracting fixed vectors from BioBERT without using terminal command?

body:
If we want to use weights from pretrained BioBERT model, we can execute following terminal command after downloading all the required BioBERT files.

os.system('python3 extract_features.py \
      --input_file=trial.txt \
      --vocab_file=vocab.txt \
      --bert_config_file=bert_config.json \
      --init_checkpoint=biobert_model.ckpt \
      --output_file=output.json')

The above command actually reads individual file containing the text, reads the textual content from it, and then writes the extracted vectors to another file. So, the problem with this is that it could not be scaled easily for very large data-sets containing thousands of sentences/paragraphs.

Is there is a way to extract these features on the go (using an embedding layer) like it could be done for the word2vec vectors in PyTorch or TF1.3?

Note: BioBERT checkpoints do not exist for TF2.0, so I guess there is no way it could be done with TF2.0 unless someone generates TF2.0 compatible checkpoint files.

I will be grateful for any hint or help.


title: How do I make custom pytorch datasets structured like the torchvision datasets?

body:
I'm new to pytorch and I'm trying to reuse a Fashion MNIST CNN (from deeplizard) to categorize my timeseries data. I'm finding it hard to understand the structure of datasets, because following this official tutorial and this SO question as best I can, I'm getting something too simple. I think this is because I don't understand OOP very well. The dataset I've made works fine in my CNN for training but then trying to analyse the results with their code I get stuck.
So I create a dataset from two pytorch tensors called features [4050, 1, 150, 6] and targets[4050]:
train_dataset = TensorDataset(features,targets) # create your datset
train_dataloader = DataLoader(train_dataset, batch_size=50, shuffle=False) # create your dataloader
print(train_dataset.__dict__.keys()) # list the attributes

I get this printed output from inspecting the attributes

dict_keys(['tensors'])

But in the Fashion MNIST tutorial they access the data like this:
train_set = torchvision.datasets.FashionMNIST(
    root='./data'
    ,train=True
    ,download=True
    ,transform=transforms.Compose([
        transforms.ToTensor()
    ])
)

train_loader = torch.utils.data.DataLoader(train_set, batch_size=1000, shuffle=True)
print(train_set.__dict__.keys()) # list the attributes

And you get this printed output from inspecting the attributes

dict_keys(['root', 'transform', 'target_transform', 'transforms',
'train', 'data', 'targets'])

My dataset works fine for training but when I get to later analysis parts of the tutorial, they want me to access parts of the dataset and I get an error:
# Analytics
prediction_loader = torch.utils.data.DataLoader(train_dataset, batch_size=50)
train_preds = get_all_preds(network, prediction_loader)
preds_correct = train_preds.argmax(dim=1).eq(train_dataset.targets).sum().item()

print('total correct:', preds_correct)
print('accuracy:', preds_correct / len(train_set))

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-73-daa87335a92a> in <module>
      4 prediction_loader = torch.utils.data.DataLoader(train_dataset, batch_size=50)
      5 train_preds = get_all_preds(network, prediction_loader)
----> 6 preds_correct = train_preds.argmax(dim=1).eq(train_dataset.targets).sum().item()
      7
      8 print('total correct:', preds_correct)

AttributeError: 'TensorDataset' object has no attribute 'targets'

Can anyone tell me what's going on here? Is this something I need to change in how I make the datasets, or can I rewrite the analysis code somehow to access the right part of the dataset?


title: How can I solve "torch.utils.ffi is deprecated. Please use cpp extensions instead" without downgrade pytorch version?

body:
When I run the code below it shows me the error.
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.
I have been searching solution on the online. The problem is the code below working in old version of torch (0.4.1). I want to know whether it is possible to modify or replace this code for working in the new version of pytorch.
from torch.utils.ffi import _wrap_function
from ._nms import lib as _lib, ffi as _ffi

__all__ = []
def _import_symbols(locals):
    for symbol in dir(_lib):
        fn = getattr(_lib, symbol)
        if callable(fn):
            locals[symbol] = _wrap_function(fn, _ffi)
        else:
            locals[symbol] = fn
        __all__.append(symbol)

_import_symbols(locals())


title: Keras: Train different parts of a network on different losses simultaneously

body:
For example, if I want to train something like a GAN, the discriminator D part will be trying to minimize crossentropy loss, and the generator G part will be trying to maximize the crossentropy loss.

The GAN would be feeding noise into G, G's output into D, and D's output being the final output. For a single input into the GAN, I want D's weights to update with their loss, and G's weights to update with their own loss (which happens to be the negative of D's loss). I want these updates to happen in one shot. I know that you can alternating between training D and then freezing the training of D to train G. But in that case, you always have half of the network being run without its weights being updated. I want to be able to run the network once, and have the different parts of it update with respect to their own losses.

Is this possible in Keras? Or is this even a reasonable thing to ask for?

If not possible in Keras, if it can be done in tensorflow, or even pytorch, that would be alright too.


title: Accessing reduced dimensionality of trained autoencoder

body:
Here is a autoencoder trained on mnist using PyTorch :

import torch
import torchvision
import torch.nn as nn
from torch.autograd import Variable

cuda = torch.cuda.is_available() # True if cuda is available, False otherwise
FloatTensor = torch.cuda.FloatTensor if cuda else torch.FloatTensor
print('Training on %s' % ('GPU' if cuda else 'CPU'))

# Loading the MNIST data set
transform = torchvision.transforms.Compose([torchvision.transforms.ToTensor(),
                torchvision.transforms.Normalize((0.1307,), (0.3081,))])
mnist = torchvision.datasets.MNIST(root='../data/', train=True, transform=transform, download=True)

# Loader to feed the data batch by batch during training.
batch = 100
data_loader = torch.utils.data.DataLoader(mnist, batch_size=batch, shuffle=True)

autoencoder = nn.Sequential(
                # Encoder
                nn.Linear(28 * 28, 512),
                nn.PReLU(512),
                nn.BatchNorm1d(512),

                # Low-dimensional representation
                nn.Linear(512, 128),
                nn.PReLU(128),
                nn.BatchNorm1d(128),

                # Decoder
                nn.Linear(128, 512),
                nn.PReLU(512),
                nn.BatchNorm1d(512),
                nn.Linear(512, 28 * 28))

autoencoder = autoencoder.type(FloatTensor)

optimizer = torch.optim.Adam(params=autoencoder.parameters(), lr=0.005)

epochs = 10
data_size = int(mnist.train_labels.size()[0])

for i in range(epochs):
    for j, (images, _) in enumerate(data_loader):
        images = images.view(images.size(0), -1) # from (batch 1, 28, 28) to (batch, 28, 28)
        images = Variable(images).type(FloatTensor)

        autoencoder.zero_grad()
        reconstructions = autoencoder(images)
        loss = torch.dist(images, reconstructions)
        loss.backward()
        optimizer.step()
    print('Epoch %i/%i loss %.2f' % (i + 1, epochs, loss.data[0]))

print('Optimization finished.')

I'm attempting to compare the lower dimensionality representation of each image.

Printing the dimensionality of each layer :

for l in autoencoder.parameters() :
    print(l.shape)

displays :

torch.Size([512, 784])
torch.Size([512])
torch.Size([512])
torch.Size([512])
torch.Size([512])
torch.Size([128, 512])
torch.Size([128])
torch.Size([128])
torch.Size([128])
torch.Size([128])
torch.Size([512, 128])
torch.Size([512])
torch.Size([512])
torch.Size([512])
torch.Size([512])
torch.Size([784, 512])
torch.Size([784])

So appears the dimensionality is not stored in learned vectors ?

In other words if I have 10000 images each containing 100 pixels, executing above autoencoder that reduces dimensionality to 10 pixels should allow to access the 10 pixel dimensionality of all 10000 images ?


title: How to use Adam optim considering of its adaptive learning rate?

body:
In the Adam optimization algorithm, the learning speed is adjusted according to the number of iterations. I don't quite understand Adam's design, especially when using batch training. When using batch training, if there are 19,200 pictures, each time 64 pictures are trained, it is equivalent to 300 iterations. If our epoch has 200 times, there are a total of 60,000 iterations. I don't know if such multiple iterations will reduce the learning speed to a very small size. So when we are training, shall we initialize the optim after each epoch, or do nothing throughout the process?

Using pytorch. I have tried to initialize the optim after each epoch if I use batch train, and I do nothing when the number of data is small.

For expample, I don't know whether the two pieces of code is correct:

optimizer = optim.Adam(model.parameters(), lr=0.1)
for epoch in range(100):
    ###Some code
    optim.step()

Another piece of code:

for epoch in range(100):
    optimizer = optim.Adam(model.parameters(), lr=0.1)
    ###Some code
    optim.step()


title: How to slice a torch tensor without using python indexing

body:
My pytorch code below keeps getting jit tracer warning (in pytorch 1.1.0 environment) complaining that "Pytorch 1.0 Tracer Warning: Converting a tensor to a Python index might ..."

Is there a way to implement the code line marked (A) below without using python indexing?

N,C,H,W = input.size()
Cout=4*C
Hout=H//2
Wout=W//2
downsampled=torch.zeros([N,Cout,Hout,Wout], dtype= torch.FloatTensor)
downsampled[:,1:Cout:4,:,:]=input[:,:,0::2,1::2] ---- (A)


title: Opening tensorboard saved scalars on windows

body:
I am using Pytorch on Windows 10 OS, and having trouble understanding the correct use of Pytorch TensorboardX.

After instantiating a writer (writer = SummaryWriter()), and adding the value of the loss function (in every iteration) to it (write.add_scalar('data/loss_func', loss.data[0].item(), iteration)), I have a folder which contains the saved run.

My questions are:

1) As far as I understand, after the training is complete, I need to write in the terminal (which corresponds to the command line prompt in Windows):

tensorboard --logdir=C:\Users\DrJohn\Documents\runs

where this is the folder which contains the file created by tensorboardX. What is the valid syntax in Windows command prompt? I couldn't understand this from the online tutorials

2) Is it possible to see the learning during the training, by using tensorboardX? (i.e. to plot the learning curve during the iterations?)Is the only option is to see everything once the training ends?

Thanks in advance


title: 2D Gaussian distribution over a grid in pytorch

body:
I am trying to convert the following numpy code to its pytorch equivalent so that I can use it during a model's forward pass and auto grad. Basically, I would like to have the distribution as a tensor instead of a numpy array.

import torch
import itertools
import numpy as np
import math

from scipy.stats import multivariate_normal
xx_np, yy_np = np.mgrid[0 : 224 : 1, 0 : 298 : 1]
pos_np = np.empty(xx_np.shape + (2,))
pos_np[:, :, 0] = xx_np
pos_np[:, :, 1] = yy_np

var_np = 4900
mean_np = [70, 100]

rv = multivariate_normal(mean_np, [[var_np, 0], [0, var_np]])
gaussian_np = rv.pdf(pos_np)

Surprisingly, the following code in pytorch gives equivalent output (gaussian_np ~= gaussian_torch). Even thought I should ideally be using 1/math.sqrt(2*math.pi*var_torch) instead of 1/(2*math.pi*var_torch) when I compute gaussian_torch. What am I missing? :(

x_range = torch.arange(0, 224, 1)
y_range = torch.arange(0, 298, 1)
xx, yy = torch.meshgrid(x_range, y_range)
pos = torch.empty(xx.shape + (2,))
pos[:, :, 0] = xx
pos[:, :, 1] = yy

var_torch = torch.tensor([4900], dtype=torch.float32)
mean_torch = torch.tensor([70,100])

gaussian_torch = (1./(2.*math.pi*var_torch)) * \
torch.exp(-torch.sum((pos - mean_torch)**2., dim=-1) \
/(2*var_torch))


title: can I train(optimize) on f1 score loss with pytorch

body:
I am building a binary classifier like below. Can I replace the BCELoss to optimize f1 score?
criterion = nn.BCELoss()

preds = model(inputs)
loss = criterion(preds , labels)


title: pythorch-lightning train_dataloader runs out of data

body:
I started to use pytorch-lightning and faced a problem of my custom data loaders:

Im using an own dataset and a common torch.utils.data.DataLoader. Basically the dataset takes a path and loads the data corresponding to an given index the dataloader loads its.

def train_dataloader(self):
    train_set = TextKeypointsDataset(parameters...)
    train_loader = torch.utils.data.DataLoader(train_set, batch_size, num_workers)
    return train_loader

When I use the pytorch-lightning modules train_dataloader and training_step everything runs fine. When I add val_dataloader and validation_step Im facing this error:

Epoch 1:  45%|████▌     | 10/22 [00:02<00:03,  3.34it/s, loss=5.010, v_num=131199]
ValueError: Expected input batch_size (1500) to match target batch_size (5)

In this case my dataset is really small (to test functionality) of 84 samples, my batch size is 8. The dataset for training and validation has the same length (just for testing purposes again).

So in total its 84 * 2 = 168 and 168 / 8 (batchsize) = 21, which are roughly the total steps (22) shown above. This means that after running on the training dataset for 10 times (10 * 8 = 80) the loader expects a new full sample of 8, but since there are only 84 samples I get an error (at least this is my current understanding).

I faced a similar problem in my own implementation (not using pytorch-lighntning) and used this pattern to solve it. Basically I am resetting the iterator, when running out of data:

try:
    data = next(data_iterator)
    source_tensor = data[0]
    target_tensor = data[1]

except StopIteration:  # reinitialize data loader if num_iteration > amount of data
    data_iterator = iter(data_loader)

Right now it seems like Im facing sth similar? I dont know how to reset/reinitialize the data loader in pytorch-lightning when my training_dataloader is running out of data. I guess there must be another sophisticated way Im not familar with. Thank you


title: Training with Pytorch: error due to CUDA memory issue

body:
I am trying to train a model on the Cityscapes dataset, for segmentation. I use torchvision deeplabv3_resnet50 model and it's Cityscapes dataset class and transforms. In case it matters, I am running the code in Jupyter notebook.

The datasets are working, as are the dataloaders. When I attempt to train, I always get this error, at the point when the first batch is trying to be put thru the network (y_ = net(xb) in one_epoch function).

RuntimeError: CUDA out of memory. Tried to allocate 128.00 MiB (GPU 0; 6.00 GiB total capacity; 4.20 GiB already allocated; 6.87 MiB free; 4.20 GiB reserved in total by PyTorch)

What is strange, is that no matter what the batch size (bs) is, the the amount of memory free according to the error is a value a little less than the amount of memory that is trying to be allocated, e.g. for bs=16 I get:

RuntimeError: CUDA out of memory. Tried to allocate 2.00 GiB (GPU 0; 6.00 GiB total capacity; 2.90 GiB already allocated; 1.70 GiB free; 2.92 GiB reserved in total by PyTorch)

I have a much more complicated model running, that will work with bs=16. This model builds everything from scratch. But I really want to be able to use the simplicity that torchvision seems to have with it's model zoo and datasets.

My code is below, not much more than the bare essentials, enough to show if it is running ok on the GPU.

def one_epoch(net, loss, dl, opt=None, metric=None):

if opt:
    net.train()  # only affects some layers
else:
    net.eval()
    rq_stored = []
    for p in net.parameters():
        rq_stored.append(p.requires_grad)
        p.requires_grad = False

L, M = [], []
dl_it = iter(dl)
for xb, yb in tqdm(dl_it, leave=False):
    xb, yb = xb.cuda(), yb.cuda()
    y_ = net(xb)
    l = loss(y_, yb)
    if opt:
        opt.zero_grad()
        l.backward()
        opt.step()
    L.append(l.detach().cpu().numpy())
    if metric: M.append(metric(y_, yb).cpu().numpy())

if not opt:
    for p,rq in zip(net.parameters(), rq_stored): p.requires_grad = rq

return L, M

accuracy = lambda y_,yb: (y_.max(dim=1)[1] == yb).float().mean()

def fit(net, tr_dl, val_dl, loss=nn.CrossEntropyLoss(), epochs=3, lr=3e-3, wd=1e-3):

opt = optim.Adam(net.parameters(), lr=lr, weight_decay=wd)

Ltr_hist, Lval_hist = [], []
for epoch in trange(epochs):
    Ltr,  _    = one_epoch(net, loss, tr_dl,  opt)
    Lval, Aval = one_epoch(net, loss, val_dl, None, accuracy)
    Ltr_hist.append(np.mean(Ltr))
    Lval_hist.append(np.mean(Lval))
    print(f'epoch: {epoch+1}\ttraining loss: {np.mean(Ltr):0.4f}\tvalidation loss: {np.mean(Lval):0.4f}\tvalidation accuracy: {np.mean(Aval):0.2f}')

return Ltr_hist, Lval_hist

class To3ch(object):
def __call__(self, pic):
    if pic.shape[0]==1: pic = pic.repeat(3,1,1)
    return pic

bs = 1
imagenet_stats = ([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])

transf = transforms.Compose([
    transforms.ToTensor(),
    To3ch(),
    transforms.Normalize(*imagenet_stats)
])

train_ds = datasets.Cityscapes('C:/cityscapes_ds', split='train', target_type='semantic', transform=transf, target_transform=transf)
val_ds = datasets.Cityscapes('C:/cityscapes_ds', split='val', target_type='semantic', transform=transf, target_transform=transf)

train_dl  = DataLoader(train_ds,  batch_size=bs,   shuffle=True,  num_workers=0)
val_dl = DataLoader(val_ds, batch_size=2*bs, shuffle=False, num_workers=0)

net = models.segmentation.deeplabv3_resnet50(num_classes=20)
fit(net.cuda(), train_dl, val_dl, loss=nn.CrossEntropyLoss(), epochs=1, lr=1e-4, wd=1e-4, plot=True)


title: pyTorch 1.5.1 - Cuda 11

body:
Can I use Cuda 11 with Pytorch?  pytorch stable version is working with Cuda 10.2
Do I get any errors or is it stable with Cuda 11?


title: How to convert dataset of images to tensor?

body:
I've got a dataset of images that looks like this:
array([[[[0.35980392, 0.26078431, 0.14313725],
     [0.38137255, 0.26470588, 0.15196078],
     [0.51960784, 0.3745098 , 0.26176471],
     ...,
     [0.34313725, 0.22352941, 0.15      ],
     [0.30784314, 0.2254902 , 0.15686275],
     [0.28823529, 0.22843137, 0.16862745]],

    [[0.38627451, 0.28235294, 0.16764706],
     [0.45098039, 0.32843137, 0.21666667],
     [0.62254902, 0.47254902, 0.36470588],
     ...,
     [0.34607843, 0.22745098, 0.15490196],
     [0.30686275, 0.2245098 , 0.15588235],
     [0.27843137, 0.21960784, 0.16176471]],

    [[0.41568627, 0.30098039, 0.18431373],
     [0.51862745, 0.38529412, 0.27352941],
     [0.67745098, 0.52058824, 0.40980392],
     ...,
     [0.34901961, 0.22941176, 0.15588235],
     [0.29901961, 0.21666667, 0.14901961],
     [0.26078431, 0.20098039, 0.14313725]],

    ...,

What I need is convert it to tensor so that I could pass it to CNN.
I'm trying to do it like that:
from torchvision import transforms as transforms
transform = transforms.Compose([
            transforms.ToTensor(),
            transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
        ])

How can I apply this transform to my dataset? Thanks for any help.


title: Using "apex" and "nn.Embedding" weights matrix without specify float16 reports type error

body:
I have an embedding layer

  self.embedding = nn.Embedding(n,m)

I need the whole embedding weights to join computation,

    logits = torch.einsum('bd,nd->bn', [over_states, self.embedding.weight.half()])

If I remove.half() in the line above, when optimizing using apex, I will get the data type error,

Original Traceback (most recent call last):
  File "/home/mhtan/anaconda3/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/home/mhtan/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/mhtan/gitee/modeling_bert.py", line 551, in forward
    over_logits = self.vocab(self.over_linear(sentiment_states))
  File "/home/mhtan/gitee/modeling_bert.py", line 520, in vocab
    c_mo_logits = torch.einsum('bd,nd->bn', [over_states, self.embedding.weight])  # (b, 256, 10)
  File "/home/mhtan/anaconda3/lib/python3.7/site-packages/torch/functional.py", line 201, in einsum
    return torch._C._VariableFunctions.einsum(equation, operands)
RuntimeError: Expected object of scalar type Half but got scalar type Float for argument #2 'mat2' in call to _th_bmm

I have used

        if args.fp16:
            try:
                from apex import amp
            except ImportError:
                raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
            self.model, self.optimizer = amp.initialize(self.model, self.optimizer, opt_level=args.fp16_opt_level)

and

            if self.args.fp16:
                try:
                    from apex import amp
                except ImportError:
                    raise ImportError(
                        "Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
                with amp.scale_loss(loss, self.optimizer) as scaled_loss:
                    scaled_loss.backward()

in my training process.

It's strange to use float16 .half() explicitly, which package does this error may relate to, pytorch or apex?


title: Weight & Biases Detectron2 Google Colab - wandb: ERROR Unable to log event [Errno 95] Operation not supported

body:
I am training a Faster-RCNN model by Detectron2 in Google Colab. I would like to track my experiments with Weights and Biases (WandB).
My dataset is uploaded to Google Drive and mounted to the session via:
from google.colab import drive
drive.mount('/content/gdrive')

Following the suggestion from https://github.com/facebookresearch/detectron2/issues/774
I am trying to link WandB via Tensorboard with:
import wandb
wandb.init(sync_tensorboard=True)

Once the training starts I get the following error repeatedly:
wandb: ERROR Unable to log event [Errno 95] Operation not supported: '/content/gdrive/My Drive/Data/output/events.out.tfevents.1598810231.3dc4616192b5.103.0' -> '/content/gdrive/My Drive/Data/wandb/run-20200830_175618-3fp3tyhs/events.out.tfevents.1598810231.3dc4616192b5.103.0'

In this case, in my WandB account, I can see that there is an active experiment running but there are no logs of losses, learning rate, etc., only hardware info like the specs of the GPU show up.
Interestingly, when I add the linking between Tensorboard and WandB in the Demo Colab Notebook of Detectron2 (https://colab.research.google.com/drive/16jcaJoc6bCFAQ96jDe2HwtXj7BMD_-m5), it works perfectly: the logging of losses, learning rate, etc shows up in my WandB account.
Can I get some tips regarding what is going wrong in my case?


title: How Solve 'RuntimeError: CUDA error: out of memory'?

body:
I import torch and run simple code using cuda, but RuntimeError: CUDA error: out of memory error occurred.
My code and error is:
>>> import torch
>>> print(torch.rand(1, device="cuda"))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
RuntimeError: CUDA error: out of memory


title: How does GPU utilization work in the context of neural network training?

body:
I am using an AWS p3.2xlarge instance with the Deep Learning AMI (DLAMI). This instance has a single Tesla V100 (640 Tensor Cores and 5,120 CUDA Cores). When I run the PyTorch Seq2Seq Jupyter Notebook, I noticed that only 25% of the GPU is used. I monitor the GPU usage with the following command watch -n 1 nvidia-smi.

My question is, what determines GPU usage? Or, why is the GPU usage not 100%? The reason behind this question is related not only to inefficiency that may be a result of code but also cost ($3.06/hour). I am wondering if there is anything more that I can do to maximize the GPU usage.

Of course, this is a deep learning model that is being learned, and the training code sends one sample at a time through the network for learning. I am thinking that mini-batch learning may not be appropriate (e.g. sending a couple of samples through before backpropagating). I am also wondering if the network architecture (the number of layers, their parameters, their input tensor dimensions, etc.) constrains how GPU is being used. For example, if I add more layers or add more hidden nodes, should I expect GPU usage to go up?


title: IndexError: Dimension out of range - PyTorch dimension expected to be in range of [-1, 0], but got 1

body:
Despite already numerous answers on this very topic, failing to see in the example below (extract from https://gist.github.com/lirnli/c16ef186c75588e705d9864fb816a13c on Variational Recurrent Networks) which input and output dimensions trigger the error.
Having tried to change dimensions in torch.cat and also suppress the call to squeeze(),  the error persists,
<ipython-input-51-cdc928891ad7> in generate(self, hidden, temperature)
     56         x_sample = x = x_out.div(temperature).exp().multinomial(1).squeeze()
     57         x = self.phi_x(x)
---> 58         tc = torch.cat([x,z], dim=1)
     59
     60         hidden_next = self.rnn(tc,hidden)

IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

Thus how to shape the dimensions in x and z in tc = torch.cat([x,z], dim=1)?
Note the code as follows,
import torch
from torch import nn, optim
from torch.autograd import Variable

class VRNNCell(nn.Module):
    def __init__(self):
        super(VRNNCell,self).__init__()
        self.phi_x = nn.Sequential(nn.Embedding(128,64), nn.Linear(64,64), nn.ELU())
        self.encoder = nn.Linear(128,64*2) # output hyperparameters
        self.phi_z = nn.Sequential(nn.Linear(64,64), nn.ELU())
        self.decoder = nn.Linear(128,128) # logits
        self.prior = nn.Linear(64,64*2) # output hyperparameters
        self.rnn = nn.GRUCell(128,64)

    def forward(self, x, hidden):
        x = self.phi_x(x)
        # 1. h => z
        z_prior = self.prior(hidden)
        # 2. x + h => z
        z_infer = self.encoder(torch.cat([x,hidden], dim=1))
        # sampling
        z = Variable(torch.randn(x.size(0),64))*z_infer[:,64:].exp()+z_infer[:,:64]
        z = self.phi_z(z)
        # 3. h + z => x
        x_out = self.decoder(torch.cat([hidden, z], dim=1))
        # 4. x + z => h
        hidden_next = self.rnn(torch.cat([x,z], dim=1),hidden)
        return x_out, hidden_next, z_prior, z_infer

    def calculate_loss(self, x, hidden):
        x_out, hidden_next, z_prior, z_infer = self.forward(x, hidden)
        # 1. logistic regression loss
        loss1 = nn.functional.cross_entropy(x_out, x)
        # 2. KL Divergence between Multivariate Gaussian
        mu_infer, log_sigma_infer = z_infer[:,:64], z_infer[:,64:]
        mu_prior, log_sigma_prior = z_prior[:,:64], z_prior[:,64:]
        loss2 = (2*(log_sigma_infer-log_sigma_prior)).exp() \
                + ((mu_infer-mu_prior)/log_sigma_prior.exp())**2 \
                - 2*(log_sigma_infer-log_sigma_prior) - 1
        loss2 = 0.5*loss2.sum(dim=1).mean()
        return loss1, loss2, hidden_next

    def generate(self, hidden=None, temperature=None):
        if hidden is None:
            hidden=Variable(torch.zeros(1,64))
        if temperature is None:
            temperature = 0.8
        # 1. h => z
        z_prior = self.prior(hidden)
        # sampling
        z = Variable(torch.randn(z_prior.size(0),64))*z_prior[:,64:].exp()+z_prior[:,:64]
        z = self.phi_z(z)
        # 2. h + z => x
        x_out = self.decoder(torch.cat([hidden, z], dim=1))
        # sampling
        x_sample = x = x_out.div(temperature).exp().multinomial(1).squeeze()
        x = self.phi_x(x)
        # 3. x + z => h
        # hidden_next = self.rnn(torch.cat([x,z], dim=1),hidden)
        tc = torch.cat([x,z], dim=1)
        hidden_next = self.rnn(tc,hidden)
        return x_sample, hidden_next

    def generate_text(self, hidden=None,temperature=None, n=100):
        res = []
        hidden = None
        for _ in range(n):
            x_sample, hidden = self.generate(hidden,temperature)
            res.append(chr(x_sample.data[0]))
        return "".join(res)

# Test
net = VRNNCell()
x = Variable(torch.LongTensor([12,13,14]))
hidden = Variable(torch.rand(3,64))
output, hidden_next, z_infer, z_prior = net(x, hidden)
loss1, loss2, _ = net.calculate_loss(x, hidden)
loss1, loss2

hidden = Variable(torch.zeros(1,64))
net.generate_text()


title: Why do we need clone the grad_output and assign it to grad_input when defining a ReLU autograd function?

body:
I'm walking through the autograd part of pytorch tutorials. I have two questions:

Why do we need clone the grad_output and assign it to grad_input other than simple assignment during backpropagation?
What's the purpose of grad_input[input < 0] = 0? Does it mean we don't update the gradient when input less than zero?

Here's the code:

class MyReLU(torch.autograd.Function):
    @staticmethod
    def forward(ctx, input):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input)
        return input.clamp(min=0)

    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        input, = ctx.saved_tensors
        grad_input = grad_output.clone()
        grad_input[input < 0] = 0
        return grad_input

Link here:
https://pytorch.org/tutorials/beginner/pytorch_with_examples.html#pytorch-defining-new-autograd-functions

Thanks a lot in advance.


title: PyTorch Adding src_key_padding_mask in TransformerEncoder leads to inf loss

body:
I'm using this code as base to build an own transformer model, using different inputs as presented there. The part in class TransformerModel(nn.Module): or in the following my own implementation shows some problems:
def make_len_mask(self, inp):
    return (inp == 0).transpose(0, 1)

class TransformerModel(nn.Module):
    def __init__(self):
        encoder_layer = TransformerEncoderLayer(ninp, nhead, nhid, dropout)
        encoder_norm = LayerNorm(ninp)
        self.encoder = TransformerEncoder(encoder_layer, nlayers, encoder_norm)

    def forward(self, src, trg):
        src.shape # (x, y, z)
        trg.shape # (x, y)
        # eliminate last dimension of source tensor, which is (batch_size, samples, features) to compute mask
        # resulting in a [true, false]-Vector indicating which elements are padding elements
        padding_tensor = src.mean(2) # padding_tensor.shape: (x,y)
        src_pad_mask = self.make_len_mask(padding_tensor)
        # self.src.mask = None
        output = self.encoder(src, mask=self.src_mask, src_key_padding_mask=src_pad_mask)

Using that src_pad_mask results in a ValueError: The loss returned in training_step is nan or inf.. If not using that mask in EncoderLayer there are results.
My inputs are
model(source, target)

where source are continuous and target are words structured as follows:
target = [1] + [4,2,3,8] + [99]  # 0 and 99 are start and end of sentence tokens

I tried to remove them with target_tensor[target_tensor == 1] = 0 or target_tensor[target_tensor == 99] = 0 this unfortunately lead to a RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: ... so I'm providing the sequence with sos and eos tokens, which doesnt feel right, maybe the problem arises there? But the sequence is already padded there so its not possible to remove the last or first index as suggested in the example.
If using nn.Transformer() instead of single EncoderLayer () the results are strongly overfitting without a mask or the same error arises using said mask. Using just a target_mask doesnt produce correct inputs.
Is there any possibility to find out where this error arises or if my mask is calculated wrongly? More discussion on github. Is it necessary to provide masks during training or inference? If so Im not doing it, maybe someone could help or point to a source if its true?
# Values != 0 => False
# Values == 0 => True
src_pad_mask:
tensor([[False],
        [False],
        ...
        [ True],
        [ True]], device='cuda:0')


title: How to specify different layer sizes in Pytorch LSTM/GRU/RNN

body:
so I know how to work with LSTMs in general with Pytorch. But it bugs me, that you can only specify ONE hidden_size for all your layers in the LSTM. Like this:
lstm = nn.LSTM(input_size=26, hidden_size=128, num_layers=3, dropout=dropout_chance, batch_first=True)

So for all three layers, the size will be 128. But is there really no way to say, for example, that the first layer should be 128, the second 32 and the third 128?
If I missed something in the documentation or you know a work-around, please let me know, thank ya!


title: what does offsets mean in pytorch nn.EmbeddingBag?

body:
I know offsets meaning when it has two numbers, but what does it mean when more than two numbers,for example:
weight = torch.FloatTensor([[1, 2, 3], [4, 5, 6]])
embedding_sum = nn.EmbeddingBag.from_pretrained(weight, mode='sum')
print(list(embedding_sum.parameters()))
input = torch.LongTensor([0,1])
offsets = torch.LongTensor([0,1,2,1])

print(embedding_sum(input, offsets))

the result is :
[Parameter containing:
tensor([[1., 2., 3.],
        [4., 5., 6.]])]
tensor([[1., 2., 3.],
        [4., 5., 6.],
        [0., 0., 0.],
        [0., 0., 0.]])

who can help me?


title: Forward Jacobian Of Neural Network in Pytorch is Slow

body:
I am computing the forward jacobian (derivative of outputs with respect to inputs) of a 2 layer feedforward neural network in pytorch, and my results are correct but relatively slow. Given the nature of the calculation I would expect it to be approximately as fast as a forward pass through the network (or maybe 2-3x as long), but it takes ~12x as long to run an optimization step on this routine (in my test example I just want the jacobian=1 at all points) vs the standard mean squared error so I assume I am doing something in an un-optimal manner. I'm just wondering if anyone knew a faster way to code this. My test network has 2 input nodes, followed by 2 hidden layers of 5 nodes each and an output layer of 2 nodes, and uses tanh activation functions on the hidden layers, with a linear output layer.

The Jacobian calculations are based on the paper The Limitations of Deep Learning in Adversarial Settings which gives a basic recursive definition of the forward derivative (basically you end up multiplying the derivative of your activation functions with the weights and previous partial derivatives of each layer). This is very similar to forward propagation, which is why I would expect it to be faster than it is. Then the determinant of the 2x2 jacobian at the end is pretty straightforward.

Below is the code for the network and the jacobian

class Network(torch.nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.h_1_1 = torch.nn.Linear(input_1, hidden_1)
        self.h_1_2 = torch.nn.Linear(hidden_1, hidden_2)
        self.out = torch.nn.Linear(hidden_2, out_1)

    def forward(self, x):
        x = F.tanh(self.h_1_1(x))
        x = F.tanh(self.h_1_2(x))
        x = (self.out(x))
        return x

    def jacobian(self, x):
        a = self.h_1_1.weight
        x = F.tanh(self.h_1_1(x))
        tanh_deriv_tensor = 1 - (x ** 2)
        expanded_deriv = tanh_deriv_tensor.unsqueeze(-1).expand(-1, -1, input_1)
        partials = expanded_deriv * a.expand_as(expanded_deriv)

        a = torch.matmul(self.h_1_2.weight, partials)
        x = F.tanh(self.h_1_2(x))
        tanh_deriv_tensor = 1 - (x ** 2)
        expanded_deriv = tanh_deriv_tensor.unsqueeze(-1).expand(-1, -1, out_1)
        partials = expanded_deriv*a

        partials = torch.matmul(self.out.weight, partials)

        determinant = partials[:, 0, 0] * partials[:, 1, 1] - partials[:, 0, 1] * partials[:, 1, 0]
        return determinant

and here are the two error functions being compared. Note that the first one requires an extra forward call through the network, to get the output values (labeled action) while the second function does not since it works on the input values.

def actor_loss_fcn1(action, target):
    loss = ((action-target)**2).mean()
    return loss

def actor_loss_fcn2(input): # 12x slower
    jacob = model.jacobian(input)
    loss = ((jacob-1)**2).mean()
    return loss

Any insight on this would be greatly appreciated


title: Saving Numpy Array as Lab Image via PIL

body:
Hello everyone!

I am writing a function which gets two Pytorch-Tensors as input and merges parts of both tensors to a new array which will be converted to a Lab-image afterwards.

net_input is a tensor with 3 channels (L, a, b) and output is a tensor with 2 channels (a, b).
Now, I want to take the L-channel from net_input and the a-channel and the b-channel from output.

The first tensor, net_input, has values from -1 to 1. The second tensor, output, has values from 0 - 1.

Thus, both of them must be mapped to values from -128 to 127. I do this, using interp1d

Apparently the saved image has some unwanted patterns and is incorrect.

original:  . grayscale:

from PIL import Image as pil
import numpy as np
from PIL import Image
from scipy.interpolate import interp1d
from datetime import datetime

def output_to_image(net_input, output, index):
    # net_input.size()  -->     torch.Size([1, 3, 225, 225]) # l, a, b channel
    # output.size()     -->     torch.Size([1, 2, 225, 225]) # a, b channel

    # A - B - Channel
    map_a_b = interp1d([0.0, 1.0], [-128, 127])

    # L-Channel
    map_l = interp1d([-1.0, 1.0], [-128, 127])

    height = output.size()[2]
    width = output.size()[3]

    l = net_input.detach().numpy()[0][0]    # l channel
    a = output.detach().numpy()[0][0]   # a channel
    b = output.detach().numpy()[0][1]   # b channel

    # pdb.set_trace()

    img_arr = np.empty([height, width, 3])  # lab img array
    l_arr = np.empty([height, width])   # grayscale img array

    for h in range(0, height-1):
        for w in range(0, width-1):
            img_arr[h][w] = [map_l(l[h][w]), map_a_b(a[h][w]), map_a_b(b[h][w])]
            l_arr[h][w] = map_l(l[h][w])

    now = datetime.now()

    img = Image.fromarray(img_arr, "LAB")
    img.save(f"../images/output/{now.strftime('%d-%m-%Y %H:%M:%S')}-lab-img{index}.tiff")

    gray = Image.fromarray(l_arr, "L")
    gray.save(f"../images/output/{now.strftime('%d-%m-%Y %H:%M:%S')}-gray-img{index}.jpg")

This is how L, a and b look like:

(Pdb) >? print(l)
[[-1. -1. -1. ... -1. -1. -1.]
 [-1. -1. -1. ... -1. -1. -1.]
 [-1. -1. -1. ... -1. -1. -1.]
 ...
 [-1. -1. -1. ... -1. -1. -1.]
 [-1. -1. -1. ... -1. -1. -1.]
 [-1. -1. -1. ... -1. -1. -1.]]
(Pdb) >? print(a)
[[0.51877767 0.5208904  0.5310791  ... 0.5340722  0.51334995 0.522657  ]
 [0.5142181  0.50250506 0.5197009  ... 0.51169556 0.5332947  0.5155644 ]
 [0.53288984 0.51795006 0.5224927  ... 0.51704454 0.53655064 0.50311136]
 ...
 [0.5270468  0.5071506  0.52318716 ... 0.5217321  0.53424454 0.5011423 ]
 [0.5216123  0.53247094 0.5254119  ... 0.53089285 0.5259453  0.532716  ]
 [0.53135234 0.5184961  0.51334924 ... 0.5131047  0.51930845 0.51474   ]]
(Pdb) >? print(b)
[[0.4812223  0.47910962 0.46892092 ... 0.46592775 0.48665005 0.47734302]
 [0.48578197 0.49749494 0.4802992  ... 0.4883045  0.46670532 0.48443565]
 [0.46711013 0.48204994 0.47750723 ... 0.4829555  0.46344927 0.4968886 ]
 ...
 [0.47295317 0.49284932 0.47681284 ... 0.47826794 0.46575552 0.49885774]
 [0.47838774 0.46752912 0.47458804 ... 0.46910718 0.47405463 0.46728405]
 [0.46864766 0.48150396 0.48665074 ... 0.48689532 0.48069155 0.48526   ]]
(Pdb)

Edit:

After adjusting the interpolation range to 0..255, like Mark suggested, the output looks a little bit different:

import numpy as np
from PIL import Image
from datetime import datetime

def output_to_image(net_input, output, index):
    # net_input.size()  -->     torch.Size([1, 3, 225, 225])
    # output.size()     -->     torch.Size([1, 3, 225, 225])

    height = output.size()[2]
    width = output.size()[3]

    l = net_input.detach().numpy()[0][0]    # l channel
    a = output.detach().numpy()[0][0]   # a channel
    b = output.detach().numpy()[0][1]   # b channel

    #pdb.set_trace()

    img_arr = np.empty([height, width, 3])  # lab img array
    l_arr = np.empty([height, width])   # grayscale img array

    for h in range(0, height-1):
        for w in range(0, width-1):
            img_arr[h][w] = [map_l(l[h][w]), map_a_b(a[h][w]), map_a_b(b[h][w])]
            l_arr[h][w] = map_l(l[h][w])

    now = datetime.now()

    img = Image.fromarray(img_arr, "LAB")
    img.save(f"../images/output/{now.strftime('%d-%m-%Y %H:%M:%S')}-lab-img{index}.tiff")

    gray = Image.fromarray(l_arr, "L")
    gray.save(f"../images/output/{now.strftime('%d-%m-%Y %H:%M:%S')}-gray-img{index}.jpg")

def map_l(x):
    return (x+1)*127

def map_a_b(x):
    return x*127


title: Can not import fastai [WinError 126] The specified module could not be foun

body:
Firstly I run the following command: conda install -c pytorch -c fastai fastai. After finishing install. I import this:

from fastai.imports import *

I got an error like this:

Traceback (most recent call last):

  File "C:\Users\acer\Desktop\Reddit Bot\demo.py", line 70, in <module>
    from fastai.imports import *

  File "C:\Users\acer\anaconda3\lib\site-packages\fastai\imports\__init__.py", line 2, in <module>
    from .torch import *

  File "C:\Users\acer\anaconda3\lib\site-packages\fastai\imports\torch.py", line 1, in <module>
    import torch, torch.nn.functional as F

  File "C:\Users\acer\anaconda3\lib\site-packages\torch\__init__.py", line 81, in <module>
    ctypes.CDLL(dll)

  File "C:\Users\acer\anaconda3\lib\ctypes\__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)

OSError: [WinError 126] The specified module could not be found

I watch some post on stackoverflow but can't solve this. How can I solve this? Please help me.


title: Dequantize values to their original prior to quantization

body:
The paper "Natural Language Processing with Small Feed-Forward Networks" https://arxiv.org/pdf/1708.00214.pdf states:

I've implemented quantization as per the above equations in python:
b = 128

embedding_matrix = [[20000,3000,1000],[1999999,20000,1999999], [20000,3000,1000]]

scaled = [ abs(round( (1 / (b - 1) * max(e)) , 3)) for e in embedding_matrix]

print(scaled)

i = 0

quantized = []
for e in embedding_matrix :
    for v in e :
        quantized.append((v , math.floor(.5 + ( (v / scaled[i]) + b) )))
    i = i + 1

quantized

Running this code quantized is set to :
[(20000, 255),
 (3000, 147),
 (1000, 134),
 (1999999, 255),
 (20000, 129),
 (1999999, 255),
 (20000, 255),
 (3000, 147),
 (1000, 134)]

How to de-quantize back to the original values prior to quantization ?
Reading https://www.tensorflow.org/api_docs/python/tf/quantization/dequantize describes :
tf.quantization.dequantize(
    input, min_range, max_range, mode='MIN_COMBINED', name=None, axis=None,
    narrow_range=False, dtype=tf.dtypes.float32
)

[min_range, max_range] are scalar floats that specify the range for the output. The 'mode' attribute controls exactly which calculations are used to convert the float values to their quantized equivalents.

and the PyTorch docs: https://pytorch.org/docs/stable/quantization.html
Seems to implement quantize differently to above implementation ?


So I am trying to make a seq to seq model for translating german to english using pytorch on online notebook like kaggle notebook and google colab
import torch
import torch.nn as nn
import torch.optim as optim
from torchtext.datasets import Multi30k
from torchtext.data import Field, BucketIterator
import numpy as np
import spacy
import random
from torch.utils.tensorboard import SummaryWriter  # to print to tensorboard

Libraries imported, when i load dataset using the function with spacy, as below,
spacy_ger = spacy.load("de")
spacy_eng = spacy.load("en")

This error comes. :
OSError: [E050] Can't find model 'de'. It doesn't seem to be a shortcut link, a Python package or a valid path to a data directory.
Everywhere, an explanation is given for 'en', but not for 'de'. If anyone could help with this.
Specification:
Package : Version

Spacy : 2.3.1

pytorch-crf : 0.7.0

torch : 1.5.1

torchnlp : 0.0.0.1

torchtext : 0.4.0

torchvision : 0.6.1

jupyter-tensorboard : 0.2.0

tensorboard : 2.2.2

tensorboard-plugin-wit : 1.7.0

Thanks in advance for helping.


title: PyTorch print output to 2dp

body:
Is there a way to print the output of a CNN model evaluation to 2dp given that the output is a multi element FloatTensor?

eg.

prediction = torch.exp(model(image2))
print(prediction)

Out:

Variable containing:
 2.84e-01  1.68e-07  7.16e-01
[torch.FloatTensor of size 1x3]

It would be better if I could output the value as:

Variable containing:
 0.28  0.00  0.72
[torch.FloatTensor of size 1x3]

I've tried:

print("%.2f" % prediction)

and using:

torch.set_printoptions(precision=2)

But neither give the desired effect.

I had a look on the documentation page:

http://pytorch.org/docs/master/torch.html#creation-ops

...under 'torch.set_printoptions' but I can't see how any of the arguments might help in this situation.

Many thanks in advance!


title: segmentation fault when return loss error during pytorch training

body:
I encountered a segmentation fault issue when return loss during training.
I use gdb to find out what's wrong, but only get the following lines:

Epoch 0,    step 0,   Current loss 518.903503418

Thread 1 "Python" received signal SIGSEGV, Segmentation fault.

_int_free (av=0x7ffff7bb4b20 <main_arena>, p=0xd767b30, have_lock=0) at

malloc.c:3841
3841

malloc.c: No such file or directory.

Then I debug the program line by line, and find out when return the loss, it would be broken.

The code snippets as following:

class ContrastiveLoss(torch.nn.Module):
    def __init__(self, margin=1):
        super(ContrastiveLoss, self).__init__()
        self.margin = margin

    def forward(self, output1, output2, label):
        output1 = output1 / torch.sqrt(torch.sum(torch.pow(output1, 2), 1).view(output1.size()[0], 1))
        output2 = output2 / torch.sqrt(torch.sum(torch.pow(output2, 2), 1).view(output2.size()[0], 1))
        dist = torch.sum( torch.pow(output1 - output2, 2),1)
        dist_sq = torch.sqrt(dist)
        loss = (1-label) * dist + label * torch.pow(torch.clamp(self.margin - dist_sq, min=0.0), 2)
        loss = torch.sum(loss) / loss.size()[0]
        return loss

Any idea of what could cause it?
It's interesting when calling the loss function firstly, it can run properly. But when calling it again, it will break.