title: How to predict an adjacency matrix for graph matching using a deep learning model

body:
I need to build a model that solve a graph matching problem having as output an adjacency matrix.
Each node of the graph should be connected to AT MOST one node (even exactly-one if is easier to implement).
Till now I implemented a convolutional model that has as output an adjacency matrix, but the prediction never meet the constraint of one node connected with at most another node.

Here are few examples of good outputs:

Y = [[0,1,0,0],
     [1,0,0,0],
     [0,0,0,1],
     [0,0,1,0]]  edges:[0,1],[2,3]

Y = [[0,0,1,0],
     [0,0,0,1],
     [1,0,0,0],
     [0,1,0,0]] edges:[0,2], [1,3]

Y = [[0,1,0,0],
     [1,0,0,0],
     [0,0,0,0],
     [0,0,0,0]] edges:[0,1]

Thanks


title: How do I represent a PyTorch LSTM 3D Tensor?

body:
As per the docs, I see that Pytorch’s LSTM expects all of its inputs to be 3D tensors. I am trying to do a simple sequence-to-sequence LSTM and I have:

class BaselineLSTM(nn.Module):
    def __init__(self):
        super(BaselineLSTM, self).__init__()

        self.lstm = nn.LSTM(input_size=100, hidden_size=100)

    def forward(self, x):
        print('x', x)
        x = self.lstm(x)

        return x

My x.size() is torch.Size([100, 1]). I expect I need a third dimension somehow, but I'm unsure as to what it actually means. Any help would be greatly appreciated.


title: Getting error when trying to torch stack a list of tensors

body:
I am trying to get cuda to work but I need to change my training input into a tensor. When I tried to do that, I am getting an error when I tried to stack a list of tensor into one tensor.

Code

for epoch in:
        alst = []
        for x, y in loader:
            x = torch.stack(x)
            #x = torch.Tensor(x)
            #x = torch.stack(x).to(device,dtype=float)

Shape of x: List of tensors

[tensor([[[0.325],
     [ 0.1257],
     [ 0.1149],
     ...,
     [-1.572],
     [-1.265],
     [-3.574]],
]), tensor([1,2,3,4,5]), tensor(6,5,4,3,2])]

Error I got

     22             alst = []
     23             for x, y in loader:
---> 24                 x_list = torch.stack(x)
     25 #                 x = torch.Tensor(x)
     26 #                 x = torch.stack(x).to(device,dtype=float)

RuntimeError: Expected object of scalar type Float but got scalar type Long for sequence element 1 in sequence argument at position #1 'tensors'

Not sure what I am doing wrong. I tried x = torch.stack(x).to(device,dtype=float) as well but it still didn't work.


title: Autoencoder for image reconstruction produces gray image with a weird grid

body:
I am trying to create a Deep Fake using an autoencoder. I use one encoder and two decoders: one for the target image, and another for the source image (the target-face is the face I want to “paste” on the sources head). So first I am trying to train the encoder and both decoders to reconstruct the input faces (300, 300, 3). But for both decoders, the output is not a colorful image but its gray, because for each pixel the Red, Green and Blue value are almost the same. And additionally to that, there is a weird 3x3 grid on the output images:

I am using a batch size of 1 because I dont know how to do it with minibatches in that case (but thats another problem). Im also using residual connections, which improved the quality. The last layer has a sigmoid activation (which could be wrong). My loss is Binary-Cross-Entropy and optimizer is Adam. Learning rate is 0.001 (I also tried 0.0001, and 0.00075).

Here is my model:

import matplotlib.pyplot as plt
import torchvision
import torch
import torch.nn as nn
import torch.nn.functional as F

class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()

        """ encoder """
        self.conv1 = nn.Conv2d(3, 32, kernel_size=(4, 4))
        self.batchnorm1 = nn.BatchNorm2d(32)

        self.conv2 = nn.Conv2d(32, 64, kernel_size=(4, 4))
        self.batchnorm2 = nn.BatchNorm2d(64)

        self.conv3 = nn.Conv2d(64, 128, kernel_size=(3, 3))
        self.batchnorm3 = nn.BatchNorm2d(128)

        self.conv4 = nn.Conv2d(128, 256, kernel_size=(4, 4))
        self.batchnorm4 = nn.BatchNorm2d(256)

        self.maxpool3x3 = nn.MaxPool2d(3)
        self.maxpool2x2 = nn.MaxPool2d(2)

        """ target-decoder """
        self.targetDeconv1 = nn.ConvTranspose2d(256, 128, kernel_size=(4, 4))
        self.targetBatchnorm1 = nn.BatchNorm2d(128)

        self.targetDeconv2 = nn.ConvTranspose2d(128, 64, kernel_size=(3, 3))
        self.targetBatchnorm2 = nn.BatchNorm2d(64)

        self.targetDeconv3 = nn.ConvTranspose2d(64, 32, kernel_size=(4, 4))
        self.targetBatchnorm3 = nn.BatchNorm2d(32)

        self.targetDeconv4 = nn.ConvTranspose2d(32, 3, kernel_size=(4, 4))

        self.upsample3x3 = nn.Upsample(scale_factor=3)
        self.upsample2x2 = nn.Upsample(scale_factor=2)

        """ source-decoder """
        self.sourceDeconv1 = nn.ConvTranspose2d(256, 128, kernel_size=(4, 4))
        self.sourceBatchnorm1 = nn.BatchNorm2d(128)

        self.sourceDeconv2 = nn.ConvTranspose2d(128, 64, kernel_size=(3, 3))
        self.sourceBatchnorm2 = nn.BatchNorm2d(64)

        self.sourceDeconv3 = nn.ConvTranspose2d(64, 32, kernel_size=(4, 4))
        self.sourceBatchnorm3 = nn.BatchNorm2d(32)

        self.sourceDeconv4 = nn.ConvTranspose2d(32, 3, kernel_size=(4, 4))

        self.upsample3x3 = nn.Upsample(scale_factor=3)
        self.upsample2x2 = nn.Upsample(scale_factor=2)

    def _visualize_features(self, feature_maps, dim: tuple=(), title: str=""):
        try:
            x, y = dim
            fig, axs = plt.subplots(x, y)
            c = 0
            for i in range(x):
                for j in range(y):
                    axs[i][j].matshow(feature_maps.detach().cpu().numpy()[0][c])
                    c += 1

            fig.suptitle(title)
            plt.show()

        except:
            pass

    def forward(self, x, label: str="0", visualize: bool=False):
        """ encoder """
        x = self.conv1(x)
        x = self.batchnorm1(x)
        x = F.relu(x)
        x_1 = self.maxpool3x3(x)

        if visualize: print(x_1.shape); self._visualize_features(x_1, dim=(4, 4))

        x = self.conv2(x_1)
        x = self.batchnorm2(x)
        x = F.relu(x)
        x_2 = self.maxpool3x3(x)

        if visualize: print(x_2.shape); self._visualize_features(x_2, dim=(4, 4))

        x = self.conv3(x_2)
        x = self.batchnorm3(x)
        x = F.relu(x)
        x_3 = self.maxpool2x2(x)

        if visualize: print(x_3.shape); self._visualize_features(x_3, dim=(4, 4))

        x = self.conv4(x_3)
        x = self.batchnorm4(x)
        x = F.relu(x)
        x = self.maxpool2x2(x)

        if visualize: print(x.shape); self._visualize_features(x, dim=(4, 4))

        """ target-decoder """
        if label == "0":
            x = self.upsample2x2(x)
            x = self.targetDeconv1(x)
            x += x_3
            x = self.targetBatchnorm1(x)
            x = F.relu(x)

            if visualize: print(x.shape); self._visualize_features(x, dim=(4, 4))

            x = self.upsample2x2(x)
            x = self.targetDeconv2(x)
            x += x_2
            x = self.targetBatchnorm2(x)
            x = F.relu(x)

            if visualize: print(x.shape); self._visualize_features(x, dim=(4, 4))

            x = self.upsample3x3(x)
            x = self.targetDeconv3(x)
            x += x_1
            x = self.targetBatchnorm3(x)
            x = F.relu(x)

            if visualize: print(x.shape); self._visualize_features(x, dim=(4, 4))

            x = self.upsample3x3(x)
            x = self.targetDeconv4(x)
            x = torch.sigmoid(x)

            if visualize: print(x.shape); self._visualize_features(x, dim=(3, 1))

            return x

        """ source-decoder """
        if label == "1":
            x = self.upsample2x2(x)
            x = self.sourceDeconv1(x)
            x += x_3
            x = self.sourceBatchnorm1(x)
            x = F.relu(x)

            if visualize: print(x.shape); self._visualize_features(x, dim=(4, 4))

            x = self.upsample2x2(x)
            x = self.sourceDeconv2(x)
            x += x_2
            x = self.sourceBatchnorm2(x)
            x = F.relu(x)

            if visualize: print(x.shape); self._visualize_features(x, dim=(4, 4))

            x = self.upsample3x3(x)
            x = self.sourceDeconv3(x)
            x += x_1
            x = self.sourceBatchnorm3(x)
            x = F.relu(x)

            if visualize: print(x.shape); self._visualize_features(x, dim=(4, 4))

            x = self.upsample3x3(x)
            x = self.sourceDeconv4(x)
            x = torch.sigmoid(x)

            if visualize: print(x.shape); self._visualize_features(x, dim=(3, 1))

            return x

So my problems are: Why is the output image gray? What is that grid on my output image? Are both problems related? I checked if it has anything to do with rgb vs. bgr but it doesnt seem like it.
I hope anyone can fix my problem, thanks in advance :)


title: trouble importing Pytorch in Jupyter notebook

body:
Iam new to deep learning and Iam trying to import Pytorch on Jupyter Notebook.
I installed Pytorch with the following lines of code in Anaconda Prompt.

conda create -n pytorch_p37 python=3.7
conda activate pytorch_p37
conda install pytorch torchvision -c pytorch
conda install jupyter
conda list

it all executed well.
but on importing pytorch it shows errors.

import torch

this error below:-

OSError: [WinError 126] The specified module could not be found

error showing image


title: Tensor(1.0).item() vs float(Tensor(1.0))

body: If x is a torch.Tensor of dtype torch.float then are the operations x.item() and float(x) exactly the same?


I'm trying to run MNIST dataset in PyTorch, I'm using cross-entropy loss and simple Neural network of neurons(748,512,128,10) for the process. But I'm getting this error:
ValueError: Expected input batch_size (1) to match target batch_size (10).

My Model:
class Netz(nn.Module):
  def __init__(self,n_input_features):
    super(Netz,self).__init__()
    self.linear=nn.Linear(784,512,bias=True)
    self.l1=nn.Linear(512,128,bias=True)
    self.l2=nn.Linear(128,10,bias=True)
  def forward(self,x):
    x=F.relu(self.linear(x))
    x=F.relu(self.l1(x))
    return F.softmax(self.l2(x),dim=1)
model=Netz(784)

Data preprocessing:
mnist = keras.datasets.mnist
#Copying data
(x_train, y_train),(x_test, y_test) = mnist.load_data()
#One-hot encoding the labels
y_train = keras.utils.to_categorical(y_train)
y_test = keras.utils.to_categorical(y_test)
#Flattening the images
x_train_reshaped = x_train.reshape((60000,784))
x_test_reshaped = x_test.reshape((10000,784))
#Normalizing the inputs
x_train_nml = x_train_reshaped/255.0
x_test_nml = x_test_reshaped/255.0
comb_data = np.hstack([x_train_nml,y_train])
#np.random.shuffle(comb_data)
x_train = comb_data[:,:-10]
y_train = comb_data[:,-10:]
x_train=torch.from_numpy(x_train.astype(np.float32))
x_test=torch.from_numpy(x_test.astype(np.float32))
y_train=torch.from_numpy(y_train.astype(np.float32))
y_test=torch.from_numpy(y_test.astype(np.float32))

Data Loading:
class Data(Dataset):
    def __init__(self):
        self.x=x_train
        self.y=y_train
        self.len=self.x.shape[0]
    def __getitem__(self,index):
      return self.x[index],self.y[index]
    def __len__(self):
        return self.len

Main:
criterion=nn.CrossEntropyLoss()
print(criterion)
optimizer=torch.optim.SGD(model.parameters(),lr=0.05)
dataset=Data()
train_data=DataLoader(dataset=dataset,batch_size=1,shuffle=False)
num_epochs=5
for epoch in range(num_epochs):
  for x,y in train_data:
    x = x.view(x.shape[0], -1)
    y_pred=model(x)
    #y=y.squeeze_()
    print(y_pred)
    loss=criterion(y_pred,y.flatten())
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Full error:
ValueError                                Traceback (most recent call last)
<ipython-input-85-142a974fa5cd> in <module>()
     10     #y=y.squeeze_()
     11     print(y_pred)
---> 12     loss=criterion(y_pred,y.flatten())
     13     loss.backward()
     14     optimizer.step()

3 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/loss.py in forward(self, input, target)
    946     def forward(self, input: Tensor, target: Tensor) -> Tensor:
    947         return F.cross_entropy(input, target, weight=self.weight,
--> 948                                ignore_index=self.ignore_index, reduction=self.reduction)
    949
    950

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in cross_entropy(input, target, weight, size_average, ignore_index, reduce, reduction)
   2420     if size_average is not None or reduce is not None:
   2421         reduction = _Reduction.legacy_get_string(size_average, reduce)
-> 2422     return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
   2423
   2424

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2214     if input.size(0) != target.size(0):
   2215         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 2216                          .format(input.size(0), target.size(0)))
   2217     if dim == 2:
   2218         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (1) to match target batch_size (10).

Can someone please tell me how to fix this error?


I tried to load csv file into tensor dataset for vertical federated learning.
The reference website is https://github.com/OpenMined/PyVertical/blob/master/examples/PyVertical%20Example.ipynb
Below is how I loaded the file but failed
train = pd.read_csv('datatrain.csv')   # load data

cols = ["a","b,"c"]   # select feature columns

train_feature = train[cols]   # create dataset with features
train_target = train['result']   # the dataset with result

# turn them in to torch.tensor data
train_feature_tensor = torch.tensor(train_feature.values)
train_target_tensor = torch.tensor(train_target.values)

# Put them into a TensorDataset
train_tensor = data_utils.dataset.TensorDataset(train_feature_tensor, train_target_tensor)

# them put them in to add_ids()
temp = add_ids(data_utils.dataset.TensorDataset)
temp.data = train_tensor
traindata_ft = temp(train_tensor)

Output:
'TensorDataset' object has no attribute 'size'

They pointed out the problem is from:
assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)

which is in:
class TensorDataset(Dataset):
    r"""Dataset wrapping tensors.

    Each sample will be retrieved by indexing tensors along the first dimension.

    Arguments:
        *tensors (Tensor): tensors that have the same size of the first dimension.
    """

    def __init__(self, *tensors):
        assert all(tensors[0].size(0) == tensor.size(0) for tensor in tensors)
        self.tensors = tensors

    def __getitem__(self, index):
        return tuple(tensor[index] for tensor in self.tensors)

    def __len__(self):
        return self.tensors[0].size(0)

About add_ids(), it is a function that generates unique id to each data row.
The original code is below
def add_ids(cls):
    """Decorator to add unique IDs to a dataset
    Args:
        cls (torch.utils.data.Dataset) : dataset to generate IDs for
    Returns:
        VerticalDataset : A class which wraps cls to add unique IDs as an attribute,
            and returns data, target, id when __getitem__ is called
    """

    class VerticalDataset(cls):
        def __init__(self, *args, **kwargs):
            super().__init__(*args, **kwargs)

            self.ids = np.array([uuid4() for _ in range(len(self))])

        def __getitem__(self, index):
            if self.data is None:
                img = None
            else:
                img = self.data[index]
                img = Image.fromarray(img.numpy(), mode="L")

                if self.transform is not None:
                    img = self.transform(img)

            if self.targets is None:
                target = None
            else:
                target = int(self.targets[index]) if self.targets is not None else None

                if self.target_transform is not None:
                    target = self.target_transform(target)

            id = self.ids[index]

            # Return a tuple of non-None elements
            return (*filter(lambda x: x is not None, (img, target, id)),)

        def __len__(self):
            if self.data is not None:
                return self.data.size(0)
            else:
                return len(self.targets)

        def get_ids(self) -> List[str]:
            """Return a list of the ids of this dataset."""
            return [str(id_) for id_ in self.ids]

        def sort_by_ids(self):
            """
            Sort the dataset by IDs in ascending order
            """
            ids = self.get_ids()
            sorted_idxs = np.argsort(ids)

            if self.data is not None:
                self.data = self.data[sorted_idxs]

            if self.targets is not None:
                self.targets = self.targets[sorted_idxs]

            self.ids = self.ids[sorted_idxs]

    return VerticalDataset


title: Why is this tensorflow training taking so long?

body:
I'm learning DRL with the book Deep Reinforcement Learning in Action. In chapter 3, they present the simple game Gridworld (instructions here, in the rules section) with the corresponding code in PyTorch.
I've experimented with the code and it takes less than 3 minutes to train the network with 89% of wins (won 89 of 100 games after training).

As an exercise, I have migrated the code to tensorflow. All the code is here.
The problem is that with my tensorflow port it takes near 2 hours to train the network with a win rate of 84%. Both versions are using the only CPU to train (I don't have GPU)

Training loss figures seem correct and also the rate of a win (we have to take into consideration that the game is random and can have impossible states). The problem is the performance of the overall process.
I'm doing something terribly wrong, but what?
The main differences are in the training loop, in torch is this:
        loss_fn = torch.nn.MSELoss()
        learning_rate = 1e-3
        optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
        ....
        Q1 = model(state1_batch)
        with torch.no_grad():
            Q2 = model2(state2_batch) #B

        Y = reward_batch + gamma * ((1-done_batch) * torch.max(Q2,dim=1)[0])
        X = Q1.gather(dim=1,index=action_batch.long().unsqueeze(dim=1)).squeeze()
        loss = loss_fn(X, Y.detach())
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

and in the tensorflow version:
        loss_fn = tf.keras.losses.MSE
        learning_rate = 1e-3
        optimizer = tf.keras.optimizers.Adam(learning_rate)
        ...
        Q2 = model2(state2_batch) #B
        with tf.GradientTape() as tape:
            Q1 = model(state1_batch)
            Y = reward_batch + gamma * ((1-done_batch) * tf.math.reduce_max(Q2, axis=1))
            X = [Q1[i][action_batch[i]] for i in range(len(action_batch))]
            loss = loss_fn(X, Y)
        grads = tape.gradient(loss, model.trainable_variables)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))

Why is the training taking so long?


title: Why does multi layer perceprons outperform RNN in CartPole?

body:
Recently, I compared two models for a DQN on CartPole-v0 environment. One of them is a multilayer perceptron with 3 layers and the other is an RNN built up from an LSTM and 1 fully connected layer. I have an experience replay buffer of size 200000 and the training doesn't start until it is filled up.
Although MLP has solved the problem under a reasonable amount of training steps  (this means to achieve a mean reward of 195 for the last 100 episodes), the RNN model could not converge as quickly and its maximum mean reward did not even reach 195 too!

I have already tried to increase batch size, add more neurons to the LSTM'S hidden state, increase the RNN'S sequence length and making the fully connected layer more complex - but every attempt failed as I saw enormous fluctuations in mean reward so the model hardly converged at all. May these are the sings of early overfitting?

class DQN(nn.Module):
    def __init__(self, n_input, output_size, n_hidden, n_layers, dropout=0.3):
        super(DQN, self).__init__()

        self.n_layers = n_layers
        self.n_hidden = n_hidden

        self.lstm = nn.LSTM(input_size=n_input,
            hidden_size=n_hidden,
            num_layers=n_layers,
            dropout=dropout,
            batch_first=True)

        self.dropout= nn.Dropout(dropout)

        self.fully_connected = nn.Linear(n_hidden, output_size)

    def forward(self, x, hidden_parameters):
        batch_size = x.size(0)

        output, hidden_state = self.lstm(x.float(), hidden_parameters)

        seq_length = output.shape[1]

        output1 = output.contiguous().view(-1, self.n_hidden)
        output2 = self.dropout(output1)
        output3 = self.fully_connected(output2)

        new = output3.view(batch_size, seq_length, -1)
        new = new[:, -1]

        return new.float(), hidden_state

    def init_hidden(self, batch_size, device):
        weight = next(self.parameters()).data

        hidden = (weight.new(self.n_layers, batch_size, self.n_hidden).zero_().to(device),
            weight.new(self.n_layers, batch_size, self.n_hidden).zero_().to(device))

        return hidden

Contrarily to what I expected, the simpler model gave a much better result than the other; even though RNN is supposed to be better in processing time series data.

Can anybody tell me what's the reason for this?

Also, I have to state that I applied no feature engineering and both DQN's worked with raw data. Could RNN outperform the MLP on using normalized features? (I mean feeding both models with normalized data)

Is there anything you can recommend me to improve training efficiency on RNN's to achieve the best results?


title: Hidden state tensors have a different order than the returned tensors

body:
As part of GRU training, I want to retrieve the hidden state tensors.

I have defined a GRU with two layers:

self.lstm = nn.GRU(params.vid_embedding_dim, params.hidden_dim , 2)

The forward function is defined as follows (the following is just a part of the implementation):

    def forward(self, s, order, batch_size, where, anchor_is_phrase = False):
    """
    Forward prop.
    """
      # s is of shape [128 , 1 , 300] , 128 is batch size
      output, (a,b) = self.lstm(s.cuda())
      output.data.contiguous()

And out is of shape: [128 , 400] (128 is the number of samples which each one is embedded in 400 dimensional vector).

I understand that out is the output of the last hidden state and thus I expect it to be equal to b. However, after I checked the values I saw that it's indeed equal but b contains the tensor in a different order, that is for example output[0] is b[49]. Am I missing something here ?

Thanks.


title: SyntaxError: invalid character in identifier - Pytorch 1.3.1

body:
When executing the following method:

def flatten(t):
    t = t.reshape(1, −1)
    t = t.squeeze()
    return t

Python complains about the second argument.

  File "pytorch.py", line 16
t = t.reshape(1, −1)
                  ^       SyntaxError: invalid character in identifier

My pytorch version is 1.3.1.
I already tried to remove the space before the argument with no effect. Any ideas?


title: a problem when i use cross-entropy loss as a loss function

body:
when i using cross-entropy loss as a loss function, i get this Dimension out of range error

Traceback (most recent call last):
  File "e:\testcode\cnn.py", line 122, in <module>
    loss = loss_func(output, b_y)   # cross entropy loss
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\module.py", line 532, in __call__
    result = self.forward(*input, **kwargs)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\modules\loss.py", line 916, in forward
    ignore_index=self.ignore_index, reduction=self.reduction)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 2021, in cross_entropy
    return nll_loss(log_softmax(input, 1), target, weight, None, ignore_index, None, reduction)
  File "D:\Anaconda3\envs\pytorch\lib\site-packages\torch\nn\functional.py", line 1317, in log_softmax
    ret = input.log_softmax(dim)
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)

and the loss function is

loss = loss_func(output, b_y)

the value of output is

tensor([-0.3507,  0.2214,  0.3781,  0.3057], grad_fn=<SelectBackward>)

the value of b_y is

tensor([3])


title: How to implement a 3D sparse_tensor_dense_matmul operation in pytorch (or tf)?

body:
If I have two tensor, a sparse tensor A and a dense tensor B, A.shape is [batch_size, m, n], B.shape is [batch_size, n, k], how can I implement a function f that can perform the following task efficiently:C = f(A, B), C.shape is [batch_size, m, k], and for any batch < batch_size, C[batch] = matmul(A[batch], B[batch]), this function should support backward method.
I try to use for loop and torch.sparse.mm, however, this method does not get the most of GPU. How can I parallelize these operations? (I don't mean torch.nn.DataParallel or something like that.)
I am searching for a long time on net. But no use. Please help or try to give some ideas how to achieve this.
Thanks in advance.


title: How do I fix the Dataset to return desired output (pytorch)

body:
I am trying to use information from the outside functions to decide which data to return. Here, I have added a simplified code to demonstrate the problem. When I use num_workers = 0, I get the desired behavior (The output after 3 epochs is 18). But, when I increase the value of num_workers, the output after each epoch is the same. And the global variable remains unchanged.
from torch.utils.data import Dataset, DataLoader

x = 6
def getx():
    global x
    x+=1
    print("x: ", x)
    return x

class MyDataset(Dataset):
    def __init__(self):
        pass

    def __getitem__(self, index):
        global x
        x = getx()
        return x

    def __len__(self):
        return 3

dataset = MyDataset()
loader = DataLoader(
    dataset,
    num_workers=0,
    shuffle=False
)

for epoch in range(4):
    for idx, data in enumerate(loader):
        print('Epoch {}, idx {}, val: {}'.format(epoch, idx, data))

The final output when num_workers=0 is 18 as expected. But when num_workers>0, x remains unchanged (The final output is 6).
How can I get a similar behavior as num_workers=0 using num_workers>0(i.e.How to ensure the __getitem__ function of dataloader changes the global variable x's value )?


title: Pytorch, how to extend a tensor

body:
I want to extend a tensor in PyTorch in the following way:
Let C be a 3x4 tensor which requires_grad = True. I want to have a new C which is 3x5 tensor and C = [C, ones(3,1)]
(the last column is a one-vector, and others are the old C)
Moreover, I need requires_grad = True for new C.
What is an efficient way to do this?


title: How to calculate KL Divergence between two batches of distributions in Pytroch?

body:
Given a batch of distributions, which represented as a pytorch tensor:
A = torch.tensor([[0., 0., 0., 0., 1., 5., 1., 2.],
        [0., 0., 1., 0., 4., 2., 1., 1.],
        [0., 0., 1., 1., 0., 5., 1., 1.],
        [0., 1., 1., 0., 2., 3., 1., 1.],
        [0., 0., 2., 1., 3., 1., 1., 0.],
        [0., 0., 2., 0., 5., 0., 1., 0.],
        [0., 2., 1., 4., 0., 0., 1., 0.],
        [0., 0., 2., 4., 1., 0., 1., 0.]], device='cuda:0')

A is a batch of distributions, which consists of eight distributions.
Now, given another batch of distributions B:
B = torch.tensor([[0., 0., 1., 4., 2., 1., 1., 0.],
        [0., 0., 0., 5., 1., 2., 1., 0.],
        [0., 0., 0., 4., 2., 3., 0., 0.],
        [0., 0., 1., 7., 0., 0., 1., 0.],
        [0., 0., 1., 2., 4., 0., 1., 1.],
        [0., 0., 1., 3., 1., 3., 0., 0.],
        [0., 0., 1., 4., 1., 0., 2., 0.],
        [1., 0., 1., 5., 0., 1., 0., 0.],
        [0., 1., 5., 1., 0., 0., 1., 0.],
        [0., 0., 3., 2., 2., 0., 1., 0.],
        [0., 2., 4., 0., 1., 0., 1., 0.],
        [1., 0., 4., 1., 1., 1., 0., 0.]], device='cuda:0')

B has 12 distributions. I want to calculate the KL Divergence between each distribution in A and each distribution in B, and then obtain a KL Distance Matrix, of which shape is 12*8. I know to use loop structure and torch.nn.functional.kl_div() to reach it. Is there any other methods in pytorch to implement it without using for-loop?
Here is my implementation using for-loop:
p_1 = F.softmax(A, dim = -1)
p_2 = F.softmax(B, dim = -1)
C = torch.empty(size = (A.shape[0], B.shape[0]), dtype = torch.float)

for i,a in enumerate(p_1):
    for j,b in enumerate(p_2):
        C[i][j] =  torch.nn.functional.kl_div(a.log(), b)
print(C)

Output is :
tensor([[0.4704, 0.5431, 0.3422, 0.6284, 0.3985, 0.2003, 0.4925, 0.5739, 0.5793,
         0.3992, 0.5007, 0.4934],
        [0.3416, 0.4518, 0.2950, 0.5263, 0.0218, 0.2254, 0.3786, 0.4747, 0.3626,
         0.1823, 0.2960, 0.2937],
        [0.3845, 0.4306, 0.2722, 0.5022, 0.4769, 0.1500, 0.3964, 0.4556, 0.4609,
         0.3396, 0.4076, 0.3933],
        [0.2862, 0.3752, 0.2116, 0.4520, 0.1307, 0.1116, 0.3102, 0.3990, 0.2869,
         0.1464, 0.2164, 0.2225],
        [0.1829, 0.2674, 0.1763, 0.3227, 0.0244, 0.1481, 0.2067, 0.2809, 0.1675,
         0.0482, 0.1271, 0.1210],
        [0.4359, 0.5615, 0.4427, 0.6268, 0.0325, 0.4160, 0.4749, 0.5774, 0.3492,
         0.2093, 0.3015, 0.3014],
        [0.0235, 0.0184, 0.0772, 0.0286, 0.3462, 0.1461, 0.0142, 0.0162, 0.3524,
         0.1824, 0.2844, 0.2988],
        [0.0097, 0.0171, 0.0680, 0.0284, 0.2517, 0.1374, 0.0082, 0.0148, 0.2403,
         0.1058, 0.2100, 0.1978]], device='cuda:0')


if torch._six.PY3:

AttributeError: module 'torch._six' has no attribute 'PY3'


title: Setting device for pytorch version 0.3.1.post2

body: In newer versions of pytorch, you can set the device by using torch.device. I need to use torch version 0.3.1.post2 for some legacy code. How do I set the device for this version?


title: How to use TensorDataset in Pytorch for batching word into correct sequence?

body:
I got this function to do

Implement the batch_data function to batch words data into chunks of size batch_size using the TensorDataset and DataLoader classes.

For example, say we have these as input:

words = [1, 2, 3, 4, 5, 6, 7]
  sequence_length = 4
  Your first feature_tensor should contain the values:

[1, 2, 3, 4]
 And the corresponding target_tensor should just be the next "word"/tokenized word value:

5
This should continue with the second feature_tensor, target_tensor being:

[2, 3, 4, 5]  # features
6             # target

So, I tried using

def batch_data(words, sequence_length, batch_size):
    """
    Batch the neural network data using DataLoader
    :param words: The word ids of the TV scripts
    :param sequence_length: The sequence length of each batch
    :param batch_size: The size of each batch; the number of sequences in a batch
    :return: DataLoader with batched data
    """

    # TODO: Implement function

    for i in range(batch_size):
        feature_tensor = words[:sequence_length]
        target_tensor = words[sequence_length]
        sequence_length+=1

    data = TensorDataset(torch.from_numpy(feature_tensor),torch.from_numpy(target_tensor))
    data_loader = DataLoader(data,batch_size=batch_size,shuffle =True)

    # return a dataloader
    return data_loader

But I am getting the error

---> 23     data = TensorDataset(torch.from_numpy(feature_tensor),torch.from_numpy(target_tensor))
     24     data_loader = DataLoader(data,batch_size=batch_size,shuffle =True)
     25     #print(feature_tensor)

TypeError: expected np.ndarray (got range)


title: Displaying image with PIL causes weird color patches

body:
I'm working on a pytorch model which outputs an image.
The problem is in displaying this image as the output is a torch tensor with values between 0-1.
If I display it like this it works fine

out=model.forward(x)
out=torch.squeeze(out)
out=out.permute(1,2,0)
plt.imshow(out)

EDIT:
Here is what this output looked like, this is not the actual output as I lost the weights now but same for all purposes of this question

However if I convert it into a PIL image before displaying it produces weird color patches like shown

out=model.forward(x)
out=torch.squeeze(out)
out=transforms.ToPILImage()(out)
plt.imshow(out)

I'm not using the first method as I have a few other steps after this which require me to have an image with integer values in 0-255.


title: PyTorch extracting tensor elements with boolean mask (retaining dimensions)

body:
Say, I have a PyTorch 2x2 tensor, and I also generated a boolean tensor of the same dimension (2x2). I want to use this as a mask.

For example, if I have a tensor:

tensor([[1, 3],
        [4, 7]])

And if my mask is:

tensor([[ True, False],
        [False,  True]])

I want to use that mask to get a tensor where elements corresponding to True from my original tensor are retained, whereas elements corresponding to False are set to zero in the output tensor.

Expected Output:

tensor([[1, 0],
        [0, 7]])

Any help is appreciated. Thanks!


title: Pytorch Autograd: Can't disable profiler when it's not running

body:
I have an issue when trying to profile my model. It showed that it can't disable profiler when it is not running.
I have tried many ways, but I could not find how to fix that error. Could someone give me a hint on could I proceed with this?
P/s: Below is the trace of my error. I used fastai library for training the neural network.
RuntimeError                              Traceback (most recent call last)
<ipython-input-11-5fc263f447e9> in <module>
----> 1 lrn.lr_find()
      2 lrn.recorder.plot(suggestion=True)

~\Anaconda3\lib\site-packages\fastai\train.py in lr_find(learn, profiling, start_lr, end_lr, num_it, stop_div, wd)
     41     cb = LRFinder(learn, start_lr, end_lr, num_it, stop_div)
     42     epochs = int(np.ceil(num_it/len(learn.data.train_dl))) * (num_distrib() or 1)
---> 43     learn.fit(epochs, profiling, start_lr, callbacks=[cb], wd=wd)
     44
     45 def to_fp16(learn:Learner, loss_scale:float=None, max_noskip:int=1000, dynamic:bool=True, clip:float=None,

~\Anaconda3\lib\site-packages\fastai\basic_train.py in fit(self, epochs, profiling, lr, wd, callbacks)
    209         else: self.opt.lr,self.opt.wd = lr,wd
    210         callbacks = [cb(self) for cb in self.callback_fns + listify(defaults.extra_callback_fns)] + listify(callbacks)
--> 211         fit(epochs, profiling, self, metrics=self.metrics, callbacks=self.callbacks+callbacks)
    212
    213     def create_opt(self, lr:Floats, wd:Floats=0.)->None:

~\Anaconda3\lib\site-packages\fastai\basic_train.py in fit(epochs, profiling, learn, callbacks, metrics)
    108                     print(epoch)
    109                     with torch.autograd.profiler.profile(use_cuda=False) as prof:
--> 110                         loss = loss_batch(learn.model, xb, yb, learn.loss_func, learn.opt, cb_handler)
    111                 else:
    112                     loss = loss_batch(learn.model, xb, yb, learn.loss_func, learn.opt, cb_handler)

~\Anaconda3\lib\site-packages\torch\autograd\profiler.py in __exit__(self, exc_type, exc_val, exc_tb)
    260         if not self.enabled:
    261             return
--> 262         records = torch.autograd._disable_profiler()
    263         self.function_events = EventList(parse_cpu_trace(records))
    264         return False

RuntimeError: can't disable profiler when it's not running```


title: How to convert keras pre-trained model from NCHW to NHWC format

body:
Same problem in 2 Questions, 1 using Keras, 1 using PyTorch -
1st question
Using a pre-trained PyTorch model (NCHW format), I converted it to Keras through ONNX. But the acceleration platform I want to use, supports Keras with channels_last (NHWC) format only. Is there a way to convert the keras model easily into channels_last formatting?

I have tried using ONNX (change_ordering parameter in onnx2keras package) - It doesn't support some ops used in model like slice.
Inserting a permute layer is also not a solution as I need the whole model in NHWC format.

If no easy implementation exists, I believe - I'll need to manually define the model structure in Keras with correct format and manually port the weights. If that's the case, may I please know an easy way to recreate/define the model from weights (.h5 file) please?
2nd question
Other than this, I also think converting PyTorch model itself to NHWC is possible - but I have minimal knowledge of this framework. I have permuted the weights by fetching weights from PyTorch's state_dict() method like -
        if('conv' in str(key)):
            params[key] = value.permute(0,2,3,1)

But am unable to repopulate the model with permuted dictionary -
model.load_state_dict(Updated_params)  gives -
size mismatch for stage4.2.branches.2.3.conv1.weight: copying a param with shape torch.Size([128, 3,
3, 128]) from checkpoint, the shape in current model is torch.Size([128, 128, 3, 3]).

To resolve this, how can I define layers in the new model in NHWC format in PyTorch.


title: Loading the pre-trained model of torch and sentence_transformers when running in a docker container failing

body:
I am getting below error while loading the pre-trained model of torch and sentence_transformers("distilbert-base-nli-stsb-mean-tokens") when trying to run in a docker container.
Error: Invalid value for '-A' / '--app':
 Unable to load celery application.
 While trying to load the module app.celery the following error occurred:
 Traceback (most recent call last):
   File "/usr/local/lib/python3.8/site-packages/celery/bin/celery.py", line 53, in convert
     return find_app(value)
   File "/usr/local/lib/python3.8/site-packages/celery/app/utils.py", line 384, in find_app
     sym = symbol_by_name(app, imp=imp)
   File "/usr/local/lib/python3.8/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
     module = imp(module_name, package=package, **kwargs)
   File "/usr/local/lib/python3.8/site-packages/celery/utils/imports.py", line 100, in import_from_cwd
     return imp(module, package=package)
   File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
     return _bootstrap._gcd_import(name[level:], package, level)
   File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
   File "<frozen importlib._bootstrap>", line 991, in _find_and_load
   File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
   File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
   File "<frozen importlib._bootstrap_external>", line 783, in exec_module
   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
   File "/code/app.py", line 997, in <module>
     load_model()
   File "/code/app.py", line 255, in load_model
     embedder = SentenceTransformer('distilbert-base-nli-stsb-mean-tokens')
   File "/usr/local/lib/python3.8/site-packages/sentence_transformers/SentenceTransformer.py", line 48, in __init__
     os.makedirs(model_path, exist_ok=True)
   File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
     makedirs(head, exist_ok=exist_ok)
   File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
     makedirs(head, exist_ok=exist_ok)
   File "/usr/local/lib/python3.8/os.py", line 213, in makedirs
     makedirs(head, exist_ok=exist_ok)
   [Previous line repeated 1 more time]
   File "/usr/local/lib/python3.8/os.py", line 223, in makedirs
     mkdir(name, mode)
 PermissionError: [Errno 13] Permission denied: '/nonexistent'

Here it is saying permission denied error while creating the folder. But I have tried providing USER root in the Dockerfile. Stuck with this issue for long time. Please anyone help me here.
Updated:
My Dockerfile:
FROM python:3.8.5-slim

WORKDIR /code

ENV ENVIRONMENT='LOCAL'
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0
ENV PYTHONDONTWRITEBYTECODE 1
ENV PYTHONUNBUFFERED 1

RUN apt-get update && apt-get install -y sudo netcat apt-utils
RUN apt-get install -y python3-dev  build-essential python3-pip

COPY ./requirements_local.txt /code/requirements_local.txt
RUN pip install -r /code/requirements_local.txt

EXPOSE 8000
COPY . /code/

CMD [ "gunicorn", "app:app", "-b", "0.0.0.0:8000","--timeout","7200"]

Docker-compose:
services:
  web:
    build:
      context: .
      dockerfile: ./Dockerfile.prod
    hostname: flaskapp
    env_file:
      - ./.env.prod
    links:
      - redis
      - celery
    depends_on:
      - redis
    volumes:
      - data:/code
      - type: bind
        source: /home/ubuntu/models
        target: /mnt/models


title: DiagonalBiLSTM RuntimeError: bool value of Tensor with more than one value is ambiguous

body:
I am trying to implement Diagonal BiLSTM using a torch. I have CBAM model prepared, also spatial gate as mentioned in this article. My code for defining conv2d layer is as follows.

def Conv2D(name, input_dim, output_dim, filter_size, inputs, mask_type=None, he_init=False):
    """
    inputs.shape: (batch size, height, width, input_dim)
    mask_type: None, 'a', 'b'
    output.shape: (batch size, height, width, output_dim)
    """
    def uniform(stdev, size):
        return np.random.uniform(
            low=-stdev * np.sqrt(3),
            high=stdev * np.sqrt(3),
            size=size
        ).astype(numpy.float32)

    filters_init = uniform(
        1./np.sqrt(input_dim * filter_size * filter_size),
        # output dim, input dim, height, width
        (output_dim, input_dim, filter_size, filter_size)
    )

    if he_init:
        filters_init *= lib.floatX(np.sqrt(2.))

    if mask_type is not None:
        filters_init *= lib.floatX(np.sqrt(2.))

    filters = lib.param(
        name+'.Filters',
        filters_init
    )

    if mask_type is not None:
        mask = np.ones(
            (output_dim, input_dim, filter_size, filter_size),
            dtype=np.float32
        )
        center = filter_size//2
        for i in range(filter_size):
            for j in range(filter_size):
                    if (j > center) or (j==center and i > center):
                        mask[:, :, j, i] = 0.
        for i in range(N_CHANNELS):
            for j in range(N_CHANNELS):
                if (mask_type=='a' and i >= j) or (mask_type=='b' and i > j):
                    mask[
                        j::N_CHANNELS,
                        i::N_CHANNELS,
                        center,
                        center
                    ] = 0.

        filters = filters * mask

    inputs = inputs.permute(0, 3, 1, 2)
    #print(inputs)
    inps = torch.cat((torch.max(inputs, 1)[0].unsqueeze(1), torch.mean(inputs, 1).unsqueeze(1)), dim=1)
    print(inps)
    result = torch.nn.Conv2d(inps, 1, 7, stride=1)
    biases = lib.param(
        name+'.Biases',
        np.zeros(output_dim, dtype=np.float32)
    )
    result = result + biases[None, :, None, None]

    return result.permute(0, 2, 3, 1)

But when I try to pass the tensors to conv2d layer from torch.nn I get the following error.

I tried to print the tensors in the inps variable & they look something like this

tensor([[[[ 2.1475e+00,  2.2656e+00,  2.2285e+00,  ...,  2.1634e+00,
            1.9802e+00,  2.0768e+00],
          [ 1.1065e-01,  9.3942e-02,  1.3884e-01,  ...,  6.6712e-02,
            9.9830e-02,  1.4429e-01]],

         [[ 1.6910e+00,  1.5110e+00,  1.5579e+00,  ...,  1.0768e+00,
            1.5984e+00,  1.6736e+00],
          [-8.2758e-02, -1.3184e-02,  5.0098e-02,  ...,  2.2589e-03,
            5.8106e-02,  7.3571e-03]]],

        [[[ 2.2599e+00,  2.3655e+00,  2.2511e+00,  ...,  2.5398e+00,
            2.0128e+00,  1.9834e+00],
          [ 9.4436e-02,  9.2293e-02,  1.5296e-01,  ...,  5.4408e-02,
            9.8131e-02,  1.3627e-01]],

         [[ 1.6812e+00,  1.4765e+00,  1.5793e+00,  ...,  1.1819e+00,
            1.6332e+00,  1.6482e+00],
          [-8.4564e-02, -1.1611e-02,  5.6016e-02,  ...,  1.8283e-03,
            5.0835e-02,  1.7487e-02]]],

        [[[ 2.2437e+00,  2.4011e+00,  2.4019e+00,  ...,  2.4042e+00,
            2.0544e+00,  2.2374e+00],
          [ 8.5540e-02,  9.6436e-02,  1.3502e-01,  ...,  7.4034e-02,
            1.0697e-01,  1.4066e-01]],

         [[ 1.6898e+00,  1.4664e+00,  1.5747e+00,  ...,  1.0820e+00,
            1.6203e+00,  1.7650e+00],
          [-7.5721e-02, -1.1245e-02,  5.4568e-02,  ..., -5.5246e-03,
            5.6962e-02,  9.4589e-03]]],

        ...,

        [[[ 2.1059e+00,  2.3265e+00,  2.4311e+00,  ...,  2.4247e+00,
            1.9939e+00,  2.0536e+00],
          [ 9.6550e-02,  1.0173e-01,  1.4329e-01,  ...,  6.5086e-02,
            1.0176e-01,  1.3881e-01]],

         [[ 1.6375e+00,  1.5037e+00,  1.5442e+00,  ...,  1.1553e+00,
            1.6443e+00,  1.6747e+00],
          [-8.6096e-02, -1.2071e-02,  5.6651e-02,  ..., -1.4265e-03,
            5.6373e-02,  1.3945e-02]]],

        [[[ 2.3196e+00,  2.1638e+00,  2.1018e+00,  ...,  2.3779e+00,
            1.9471e+00,  1.9064e+00],
          [ 9.8306e-02,  9.1398e-02,  1.5183e-01,  ...,  6.4398e-02,
            1.0629e-01,  1.4232e-01]],

         [[ 1.6620e+00,  1.5137e+00,  1.5810e+00,  ...,  1.0661e+00,
            1.5326e+00,  1.6870e+00],
          [-8.2823e-02, -8.9255e-03,  5.7103e-02,  ..., -3.1661e-03,
            5.9011e-02,  6.9708e-03]]],

        [[[ 2.2586e+00,  2.3127e+00,  2.1126e+00,  ...,  2.3692e+00,
            2.0004e+00,  2.0361e+00],
          [ 8.9194e-02,  9.7919e-02,  1.4491e-01,  ...,  8.3260e-02,
            1.0935e-01,  1.4324e-01]],

         [[ 1.7048e+00,  1.5188e+00,  1.6082e+00,  ...,  1.1044e+00,
            1.5624e+00,  1.7280e+00],
          [-8.0997e-02, -8.0655e-03,  6.0712e-02,  ..., -4.3486e-03,
            5.7900e-02,  8.8437e-03]]]], grad_fn=<CatBackward>)

RuntimeError: bool value of Tensor with more than one value is ambiguous

Do help me with this, please. I am new to torch & would appreciate any answers.


title: How the standard normal distribution works in practice in NumPy and PyTorch?

body:
I have two points to ask about:

1)
I would like to understand what is precisely returned from the np.random.randn from NumPy and torch.randn from PyTorch. They both return a tensor with random numbers from a normal distribution with mean 0 and std 1, hence, a standard normal distribution. However, it is not the same thing as puting x  values in the standard normal distribution function here and getting its respective image values y. The values returned by PyTorch and NumPy does not seem like this.

For me, it seems that both np.random.randn and torch.randn from these libraries returns the x values from the functions, not the image y as I calculated below. Is that correct?

normal = np.array([(1/np.sqrt(2*np.pi))*np.exp(-(1/2)*(i**2)) for i in range(-38,39)])

Printing the normal variable shows me something like this.

array([1.10e-314, 2.12e-298, 1.51e-282, 3.94e-267, 3.79e-252, 1.34e-237,
       1.75e-223, 8.36e-210, 1.47e-196, 9.55e-184, 2.28e-171, 2.00e-159,
       6.45e-148, 7.65e-137, 3.34e-126, 5.37e-116, 3.17e-106, 6.90e-097,
       5.52e-088, 1.62e-079, 1.76e-071, 7.00e-064, 1.03e-056, 5.53e-050,
       1.10e-043, 8.00e-038, 2.15e-032, 2.12e-027, 7.69e-023, 1.03e-018,
       5.05e-015, 9.13e-012, 6.08e-009, 1.49e-006, 1.34e-004, 4.43e-003,
       5.40e-002, 2.42e-001, 3.99e-001, 2.42e-001, 5.40e-002, 4.43e-003,
       1.34e-004, 1.49e-006, 6.08e-009, 9.13e-012, 5.05e-015, 1.03e-018,
       7.69e-023, 2.12e-027, 2.15e-032, 8.00e-038, 1.10e-043, 5.53e-050,
       1.03e-056, 7.00e-064, 1.76e-071, 1.62e-079, 5.52e-088, 6.90e-097,
       3.17e-106, 5.37e-116, 3.34e-126, 7.65e-137, 6.45e-148, 2.00e-159,
       2.28e-171, 9.55e-184, 1.47e-196, 8.36e-210, 1.75e-223, 1.34e-237,
       3.79e-252, 3.94e-267, 1.51e-282, 2.12e-298, 1.10e-314])

2) Also, if we ask these libraries that I want a matrix of values from a standard normal distribution, it means that all rows and columns are draw from the same standard distribution? If I want i.i.d distributions in every row, I would need to call np.random.randn over a for loop for each row and then vstack them?


title: Learner.export() problem: Can’t pickle  at 0x……..>: it’s not the same object as __main__.

body:
I am new to fast-ai. I define my own transformation pad3d for databunch and create and train a learner using my own autoencoder models. But when we want to export the learner, I ran into an error:

PicklingError: Can’t pickle <function pad3d at 0x7fd07dee8d90>: it’s not the same object as main.pad3d
The traceback looks like this:
----> 2 learner.export(‘testexport.pkl’)

~/anaconda2/envs/my37/lib/python3.7/site-packages/fastai/basic_train.py in export(self, file, destroy)
240 state[‘data’] = self.data.valid_ds.get_state(**xtra)
241 state[‘cls’] = self.class
–> 242 try_save(state, self.path, file)
243 if destroy: self.destroy()
244

~/anaconda2/envs/my37/lib/python3.7/site-packages/fastai/torch_core.py in try_save(state, path, file)
404 def try_save(state:Dict, path:Path=None, file:PathLikeOrBinaryStream=None):
405 target = open(path/file, ‘wb’) if is_pathlike(file) else file
–> 406 try: torch.save(state, target)
407 except OSError as e:
408 raise Exception(f"{e}\n Can’t write {path/file}. Pass an absolute writable pathlib obj fname.")

~/anaconda2/envs/my37/lib/python3.7/site-packages/torch/serialization.py in save(obj, f, pickle_module, pickle_protocol)
222 >>> torch.save(x, buffer)
223 “”"
–> 224 return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))
225
226

~/anaconda2/envs/my37/lib/python3.7/site-packages/torch/serialization.py in _with_file_like(f, mode, body)
147 f = open(f, mode)
148 try:
–> 149 return body(f)
150 finally:
151 if new_fd:

~/anaconda2/envs/my37/lib/python3.7/site-packages/torch/serialization.py in (f)
222 >>> torch.save(x, buffer)
223 “”"
–> 224 return _with_file_like(f, “wb”, lambda f: _save(obj, f, pickle_module, pickle_protocol))
225
226

~/anaconda2/envs/my37/lib/python3.7/site-packages/torch/serialization.py in _save(obj, f, pickle_module, pickle_protocol)
295 pickler = pickle_module.Pickler(f, protocol=pickle_protocol)
296 pickler.persistent_id = persistent_id
–> 297 pickler.dump(obj)
298
299 serialized_storage_keys = sorted(serialized_storages.keys())

PicklingError: Can’t pickle <function pad3d at 0x7fd07dee8d90>: it’s not the same object as main.pad3d

I have seen these two posts

https://forums.fast.ai/t/learn-export-not-working-with-gans/42195/12

https://github.com/fastai/fastai/issues/1540

In the latter post, I feel like it is because my code containing the customized function as follow:

@faiv.TfmPixel
@singledispatch
def pad3d(x, padding)->torch.Tensor:
  #s = x.shape
  npad = (padding[0],padding[0],padding[1],padding[1],padding[2],padding[2])
  m = nn.ConstantPad3d(npad, 0)
  r = m(x)
  return r.contiguous()
  tfms1 = pad3d(padding=(50,20,50))

I rerun the definition of the function before export, but it does work as well.
There it seems none of them will help me solve my problem.

Can any one help with this thanks very much.


title: How to perform Kneser-Ney smoothing in NLTK at word-level for tri-gram language model?

body:
I am trying to train a tri-gram language model on a text corpus and want to perform KN smoothing. Apparently, the 'nltk.trigrams' does this at character-level. I was wondering how I would be able to do this at word-level and also perform KN smoothing. Here is a piece of code that I wrote and doesn't work:

    with open('file.txt',"r",encoding = "ISO-8859-1") as ff:
        text = ff.read()

    word_tok = tknzr.tokenize(text)
    ngrams = nltk.trigrams(word_tok)
    freq_dist = nltk.FreqDist(ngrams)
    kneser_ney = nltk.KneserNeyProbDist(freq_dist)
    print(kneser_ney.prob('you go to'))

I get the error:

    Expected an iterable with 3 members.


title: Using an MLMultiArray as an Output of a CoreML model

body:
I trained a YOLOv3-SPP model using PyTorch. I then saved the model in onnx format and then converted my onnx model to CoreML using onnx-coreml. I cannot seem to figure out how to use my model since the outputs are MLMultiArrays.

This is what my outputs look like:

I am new to Machine Learning and don't know where to begin when it comes to trying to use this model. I don't know what information each MultiArray contains and cannot figure out how to access it. Given that my model is an object detector trained on 3 classes, can anyone tell me what information each MultiArray holds and how I can get access to it? Some sample code with explanation would go a long way. If anybody knows of any Github project that I can simply drop this model into to test it, that would also work.


title: How to split a dataset into a custom training set and a custom validation set with pytorch?

body:
I'm using a non-torchvision dataset and I have extracted it with the ImageFolder method. I'm trying to split the dataset into 20% validation set and 80% training set. I can only find this method (random_split) from PyTorch library which allows splitting dataset. However, this is random every time. I'm wondering is there a way to split the dataset with a specific amount in the PyTorch library?

This is my code for extracting the dataset and split it randomly.

transformations = transforms.Compose([
    transforms.Resize(255),
    transforms.CenterCrop(224),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
])

TrafficSignSet = datasets.ImageFolder(root='./train/', transform=transformations)

####### split data
train_size = int(0.8 * len(TrafficSignSet))
test_size = len(TrafficSignSet) - train_size
train_dataset_split, test_dataset_split = torch.utils.data.random_split(TrafficSignSet, [train_size, test_size])

#######put into a Dataloader
train_dataset = torch.utils.data.DataLoader(train_dataset_split, batch_size=32, shuffle=True)
test_dataset = torch.utils.data.DataLoader(test_dataset_split, batch_size=32, shuffle=True)


title: Pytorch: accessing a subtensor using lists of indices

body:
I have a pair of tensors S and T of dimensions (s1,...,sm) and (t1,...,tn) with si < ti. I want to specify a list of indices in each dimensions of T to "embed" S in T. If I1 is a list of s1 indices in (0,1,...,t1) and likewise for I2 up to In, I would like to do something like
T.select(I1,...,In)=S
that will have the effect that now T has entries equal to the entries of S over the indices (I1,...,In).
for example

`S=
[[1,1],
[1,1]]

T=
[[0,0,0],
[0,0,0],
[0,0,0]]

T.select([0,2],[0,2])=S

T=
[[1,0,1],
[0,0,0],
[1,0,1]]`


title: PyTorch "modified by inplace operation" error

body:
The error message indicates that [torch.cuda.FloatTensor [256, 1, 4, 4]] is at version 2; expected version 1 instead, and execution breaks on d_loss.backward() — i.e., the backward call on my Discriminator.

UPDATE: Okay, I tracked it down to an optimizer.step() for my Generator that was happening before running .backward() on my Discriminator.

UPDATE 2: So once I got the model running on PyTorch 1.5 (by moving G's optimizer to after the d_loss.backward() call, as above), I noticed that losses were suddenly much higher during training. I let the model run for a few epochs and the images were basically noise. So, out of curiosity I switched back to my PyTorch 1.4 environment and ran the original for a few epochs, and the images were good again. It's a ClusterGAN that I'm training — so not the standard routine — and I'm wondering why this change is so detrimental to the output. Also, how can I get the model to run in PyTorch 1.5 without the degradation in performance? Presumably I have to keep the optimizer update where it was originally (right after ge_loss.backward(retain_graph=True)), but somehow avoid the error PyTorch 1.5 reports when we hit d_loss.backward() later in the code. I suppose I have to clone() something, but I'm not clear what... ?

[...]
# main training block
for epoch in range(n_epochs):
        for i, (imgs, itruth_label) in enumerate(dataloader):
            iter_count += 1
            # Ensure generator/encoder are trainable
            generator.train()
            encoder.train()
            # Zero gradients for models
            generator.zero_grad()
            encoder.zero_grad()
            discriminator.zero_grad()

            # Configure input
            real_imgs = Variable(imgs.type(Tensor))

            # ---------------------------
            #  Train Generator + Encoder
            # ---------------------------

            optimizer_GE.zero_grad()

            # Sample random latent variables
            zn, zc, zc_idx = sample_z(shape=imgs.shape[0],
                                      latent_dim=latent_dim,
                                      n_c=n_c)

            # Generate a batch of images
            gen_imgs = generator(zn, zc)

            # Discriminator output from real and generated samples
            D_gen = discriminator(gen_imgs)
            D_real = discriminator(real_imgs)

            # Step for Generator & Encoder, n_skip_iter times less than for discriminator
            did_update = False
            if (i % n_skip_iter == 0):
                # Encode the generated images
                enc_gen_zn, enc_gen_zc, enc_gen_zc_logits = encoder(gen_imgs)

                # Calculate losses for z_n, z_c
                zn_loss = mse_loss(enc_gen_zn, zn)
                zc_loss = xe_loss(enc_gen_zc_logits, zc_idx)

                # additional top-k step (from Sinha et al, 2020)
                if top_k <= D_gen.size()[0]:
                    top_k_gen = torch.topk(D_gen, top_k, 0)
                else:
                    top_k_gen = torch.topk(D_gen, D_gen.size()[0], 0)

                # Check requested metric
                if wass_metric:
                    # Wasserstein GAN loss

                    ge_loss = torch.mean(top_k_gen[0]) + betan * zn_loss + betac * zc_loss
                else:
                    # Vanilla GAN loss
                    valid = Variable(Tensor(gen_imgs.size(0), 1).fill_(1.0), requires_grad=False)
                    v_loss = bce_loss(D_gen, valid)
                    ge_loss = v_loss + betan * zn_loss + betac * zc_loss

                ge_loss.backward(retain_graph=True)
                # ---- ORIGINAL OPTIMIZER UPDATE ---- #
                optimizer_GE.step()
                scheduler.step(epoch + i / iters)
                did_update = True

            # ---------------------
            #  Train Discriminator
            # ---------------------

            optimizer_D.zero_grad()

            # Measure discriminator's ability to classify real from generated samples
            if wass_metric:
                # Gradient penalty term
                grad_penalty = calc_gradient_penalty(discriminator, real_imgs, gen_imgs)

                # Wasserstein GAN loss w/gradient penalty
                d_loss = torch.mean(D_real) - torch.mean(D_gen) + grad_penalty

            else:
                # Vanilla GAN loss
                fake = Variable(Tensor(gen_imgs.size(0), 1).fill_(0.0), requires_grad=False)
                real_loss = bce_loss(D_real, valid)
                fake_loss = bce_loss(D_gen, fake)
                d_loss = (real_loss + fake_loss) / 2

            d_loss.backward()
            # --- REVISED OPTIMIZER UPDATE FOR PyTorch 1.5 ------ #
            # if did_update:
            #     optimizer_GE.step()
            optimizer_D.step()
            # scheduler.step(epoch + i / iters)
[...]


title: Error Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

body:
I have the following code taken directly from here with some pretty little modifications:
import pandas as pd
import torch
import json
from transformers import T5Tokenizer, T5ForConditionalGeneration, T5Config
from torch import cuda

df = pd.read_pickle('df_final.pkl')

model = T5ForConditionalGeneration.from_pretrained('t5-base')
tokenizer = T5Tokenizer.from_pretrained('t5-base')
device = 'cuda' if cuda.is_available() else 'cpu'

text = ''.join(df[(df['col1'] == 'type') & (df['col2'] == 2)].col3.to_list())

preprocess_text = text.strip().replace("\n","")
t5_prepared_Text = "summarize: "+preprocess_text
#print ("original text preprocessed: \n", preprocess_text)

tokenized_text = tokenizer.encode(t5_prepared_Text, return_tensors="pt", max_length = 500000).to(device)

# summmarize
summary_ids = model.generate(tokenized_text,
                                    num_beams=4,
                                    no_repeat_ngram_size=2,
                                    min_length=30,
                                    max_length=100,
                                    early_stopping=True)

output = tokenizer.decode(summary_ids[0], skip_special_tokens=True)

print ("\n\nSummarized text: \n",output)

When executing the model_generate() part i get an error like this:
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-12-e8e9819a85dc> in <module>
     12                                     min_length=30,
     13                                     max_length=100,
---> 14                                     early_stopping=True).to(device)
     15
     16 output = tokenizer.decode(summary_ids[0], skip_special_tokens=True)

~\Anaconda3\lib\site-packages\torch\autograd\grad_mode.py in decorate_no_grad(*args, **kwargs)
     47         def decorate_no_grad(*args, **kwargs):
     48             with self:
---> 49                 return func(*args, **kwargs)
     50         return decorate_no_grad
     51

~\Anaconda3\lib\site-packages\transformers\generation_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id, use_cache, **model_specific_kwargs)
    383             encoder = self.get_encoder()
    384
--> 385             encoder_outputs: tuple = encoder(input_ids, attention_mask=attention_mask)
    386
    387         # Expand input ids if num_beams > 1 or num_return_sequences > 1

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\transformers\modeling_t5.py in forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, inputs_embeds, head_mask, past_key_value_states, use_cache, output_attentions, output_hidden_states, return_dict)
    701         if inputs_embeds is None:
    702             assert self.embed_tokens is not None, "You have to intialize the model with valid token embeddings"
--> 703             inputs_embeds = self.embed_tokens(input_ids)
    704
    705         batch_size, seq_length = input_shape

~\Anaconda3\lib\site-packages\torch\nn\modules\module.py in __call__(self, *input, **kwargs)
    539             result = self._slow_forward(*input, **kwargs)
    540         else:
--> 541             result = self.forward(*input, **kwargs)
    542         for hook in self._forward_hooks.values():
    543             hook_result = hook(self, input, result)

~\Anaconda3\lib\site-packages\torch\nn\modules\sparse.py in forward(self, input)
    112         return F.embedding(
    113             input, self.weight, self.padding_idx, self.max_norm,
--> 114             self.norm_type, self.scale_grad_by_freq, self.sparse)
    115
    116     def extra_repr(self):

~\Anaconda3\lib\site-packages\torch\nn\functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1482         # remove once script supports set_grad_enabled
   1483         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1484     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1485
   1486

RuntimeError: Expected object of device type cuda but got device type cpu for argument #1 'self' in call to _th_index_select

I've searched this error and fouund some other threads like this one and this one but they didn't help me much since their case seems to be completely different. In my case there are no custom instances or classes created, so i don't know how to fix this or where the error come from.
Could you please tell me where is the error coming from and how could i fix it?
Thank you very much in advance.


title: Why is the implementation of cross entropy different in Pytorch and Tensorflow?

body:
I am going through the documentation of Cross Entropy in Pytorch and Tensorflow. I understand that they are modifying the naive implementation of Cross Entropy to solve for the potential numeric over/underflows. However, I am unable to understand as to how these modifications are helping at all.
The implementation of Cross Entropy in Pytorch follows the following logic -

where  is the softmax score and  is the raw score.
This doesn't seem to solve the problem because  also leads to numeric overflow.
Now, we contrast it with Tensorflow's implementation (I got it from a discussion in Github. This might be completely wrong) -
Let  is the vector of all k raw logit scores.

While this solves the problem of overflow but it runs into problems of underflow because it is possible that  which would lead to an even smaller
Can someone please help me in understanding what's going on here?


title: PyTorch list slicing on GPU slower than on CPU

body:
I would like to optimize ML code (SSD in PyTorch) on NVIDIA Jetson Xavier NX (development kit). One of the bottlenecks seems to be list slicing on PyTorch (1.6.0) tensors on GPU device.
The same problem occured on NVIDIA GeForce GTX 1050 Ti (GP107), CPU was ~2 times faster.
Let me create the variables first
import torch
from time import time

cuda0 = torch.device('cuda:0')

probs = torch.ones([3000], dtype=torch.float64, device=cuda0)
mask = torch.ones([3000], dtype=torch.bool, device=cuda0)

probs_cpu = probs.cpu()
mask_cpu = mask.cpu()

Then run the logic (Approximately same results occurred every run)
before = time()
probs[mask]
print(f'GPU {time() - before:.5f}') # output: GPU 0.00263

before = time()
probs_cpu[mask_cpu]
print(f'CPU {time() - before:.5f}') # output: CPU 0.00066

Why is the list slicing ~4 times slower on GPU compared to CPU using PyTorch library vesrion 1.6.0 on NVIDIA Jetson Xavier NX Developer kit according to the code above? How to speed it up?
Code details: see line 51 in predictor.py which is part of SSD Implementation in PyTorch
Run it on CPU?: Whole algorithm will not be faster if I run it on the CPU since the downloading from GPU takes too long (~0.00805 s).


title: How set custom class weights in Detectron2

body:
I use https://detectron2.readthedocs.io/tutorials/install.html for my dataset with another classes and objects.
And my dataset is unbalanced. I wish set different weights for each class. How can I do that?


title: What could be reason of "ValueError: axes don't match array error" for Pytorch U-net segmentation model?

body:
I'm trying to implement a segmentation model (which i used for another dataset succesfully before) for kaggle dataset called "Carvana Image Masking Challange".
I searched a lot, but still could not figured out what is the reason i am getting this error. There were some suggestion to check image dimension which could be grayscale format but it seems i have 3 channel for both original and mask images.I am grateful for all your support
My code is following:
Libraries
import os
os.environ['CUDA_VISIBLE_DEVICES'] = '0'
from PIL import Image
import numpy as np
import cv2
import matplotlib.pyplot as plt
from torch.utils.data import DataLoader
from torch.utils.data import Dataset as BaseDataset
import albumentations as albu
import torch
import numpy as np
import segmentation_models_pytorch as smp

Data path
DATA_DIR = 'D:/Users/eugur/Belgeler/Jupyter/Segmentation_Kaggle'

x_train_dir = os.path.join(DATA_DIR, 'train')
y_train_dir = os.path.join(DATA_DIR, 'train_masks')

x_valid_dir = os.path.join(DATA_DIR, 'valid')
y_valid_dir = os.path.join(DATA_DIR, 'valid_masks')

x_test_dir = os.path.join(DATA_DIR, 'test')

helper function for data visualization
def visualize(**images):
    """PLot images in one row."""
    n = len(images)
    plt.figure(figsize=(16, 5))
    for i, (name, image) in enumerate(images.items()):
        plt.subplot(1, n, i + 1)
        plt.xticks([])
        plt.yticks([])
        plt.title(' '.join(name.split('_')).title())
        plt.imshow(image)
    plt.show()

Dataset Class
class Dataset(BaseDataset):
    """

    Args:
        images_dir (str): path to images folder
        masks_dir (str): path to segmentation masks folder
        class_values (list): values of classes to extract from segmentation mask
        augmentation (albumentations.Compose): data transfromation pipeline
            (e.g. flip, scale, etc.)
        preprocessing (albumentations.Compose): data preprocessing
            (e.g. noralization, shape manipulation, etc.)

    """

    CLASSES = ['car']

    def __init__(
            self,
            images_dir,
            masks_dir,
            classes=None,
            augmentation=None,
            preprocessing=None,
    ):
        self.ids = os.listdir(images_dir)
        self.images_fps = [os.path.join(images_dir, image_id) for image_id in self.ids]
        self.masks_fps = [os.path.join(masks_dir, image_id.split('.')[0]+'_mask.gif') for image_id in self.ids]

        # convert str names to class values on masks
        self.class_values = [self.CLASSES.index(cls.lower()) for cls in classes]

        self.augmentation = augmentation
        self.preprocessing = preprocessing

    def __getitem__(self, i):

        # read data
        image = cv2.imread(self.images_fps[i])
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
#         mask = cv2.imread(self.masks_fps[i], 0)
        mask = cv2.VideoCapture(self.masks_fps[i],0)
        ret,mask = mask.read()
        mask = mask/255

        # extract certain classes from mask (e.g. cars)
        masks = [(mask == v) for v in self.class_values]
        mask = np.stack(masks, axis=-1).astype('float')

        # apply augmentations
        if self.augmentation:
            sample = self.augmentation(image=image, mask=mask)
            image, mask = sample['image'], sample['mask']

        # apply preprocessing
        if self.preprocessing:
            sample = self.preprocessing(image=image, mask=mask)
            image, mask = sample['image'], sample['mask']

        return image, np.squeeze(mask,axis=3)

    def __len__(self):
        return len(self.ids)

Preprocessing and Augmentation
def get_training_augmentation():
    train_transform = [

        albu.HorizontalFlip(p=0.5),

        albu.ShiftScaleRotate(scale_limit=0.5, rotate_limit=0, shift_limit=0.1, p=1, border_mode=0),

        albu.PadIfNeeded(min_height=320, min_width=320, always_apply=True, border_mode=0),
        albu.RandomCrop(height=320, width=320, always_apply=True),

        albu.IAAAdditiveGaussianNoise(p=0.2),
        albu.IAAPerspective(p=0.5),

        albu.OneOf(
            [
                albu.CLAHE(p=1),
                albu.RandomBrightness(p=1),
                albu.RandomGamma(p=1),
            ],
            p=0.9,
        ),

        albu.OneOf(
            [
                albu.IAASharpen(p=1),
                albu.Blur(blur_limit=3, p=1),
                albu.MotionBlur(blur_limit=3, p=1),
            ],
            p=0.9,
        ),

        albu.OneOf(
            [
                albu.RandomContrast(p=1),
                albu.HueSaturationValue(p=1),
            ],
            p=0.9,
        ),
    ]
    return albu.Compose(train_transform)

def get_validation_augmentation():
    """Add paddings to make image shape divisible by 32"""
    test_transform = [
        albu.PadIfNeeded(384, 480)
    ]
    return albu.Compose(test_transform)

def to_tensor(x, **kwargs):

    return x.transpose(0,2,1).astype('float32')

def get_preprocessing(preprocessing_fn):
    """Construct preprocessing transform

    Args:
        preprocessing_fn (callbale): data normalization function
            (can be specific for each pretrained neural network)
    Return:
        transform: albumentations.Compose

    """

    _transform = [
        albu.Lambda(image=preprocessing_fn),
        albu.Lambda(image=to_tensor, mask=to_tensor),
    ]
    return albu.Compose(_transform)

Model Definition
ENCODER = 'se_resnext50_32x4d'
ENCODER_WEIGHTS = 'imagenet'
CLASSES = ['car']
ACTIVATION = 'sigmoid' # could be None for logits or 'softmax2d' for multicalss segmentation
DEVICE = 'cuda'

# create segmentation model with pretrained encoder
model = smp.FPN(
    encoder_name=ENCODER,
    encoder_weights=ENCODER_WEIGHTS,
    classes=len(CLASSES),
    in_channels=3,
    activation=ACTIVATION,
)

preprocessing_fn = smp.encoders.get_preprocessing_fn(ENCODER, ENCODER_WEIGHTS)

Data Loader
train_dataset = Dataset(
    x_train_dir,
    y_train_dir,
    preprocessing=get_preprocessing(preprocessing_fn),
    classes=CLASSES,
)

valid_dataset = Dataset(
    x_valid_dir,
    y_valid_dir,
    preprocessing=get_preprocessing(preprocessing_fn),
    classes=CLASSES,
)

train_loader = DataLoader(train_dataset, batch_size=8, shuffle=True, num_workers=0)
valid_loader = DataLoader(valid_dataset, batch_size=1, shuffle=False, num_workers=0)

Optimer Definition
loss = smp.utils.losses.DiceLoss()
metrics = [
    smp.utils.metrics.IoU(threshold=0.5),
]

optimizer = torch.optim.Adam([
    dict(params=model.parameters(), lr=0.0001),
])

Training
train_epoch = smp.utils.train.TrainEpoch(
    model,
    loss=loss,
    metrics=metrics,
    optimizer=optimizer,
    device=DEVICE,
    verbose=True,
)

valid_epoch = smp.utils.train.ValidEpoch(
    model,
    loss=loss,
    metrics=metrics,
    device=DEVICE,
    verbose=True,
)

max_score = 0

for i in range(0, 20):

    print('\nEpoch: {}'.format(i))
    train_logs = train_epoch.run(train_loader)
    valid_logs = valid_epoch.run(valid_loader)

    # do something (save model, change lr, etc.)
    if max_score < valid_logs['iou_score']:
        max_score = valid_logs['iou_score']
        torch.save(model, './best_model.pth')
        print('Model saved!')

    if i == 25:
        optimizer.param_groups[0]['lr'] = 1e-5
        print('Decrease decoder learning rate to 1e-5!')

Error
> Epoch: 0 train:   0%|          | 0/510 [00:00<?, ?it/s]
>
> --------------------------------------------------------------------------- ValueError                                Traceback (most recent call
> last) <ipython-input-208-d2306c5ca0ea> in <module>
>       6
>       7     print('\nEpoch: {}'.format(i))
> ----> 8     train_logs = train_epoch.run(train_loader)
>       9     valid_logs = valid_epoch.run(valid_loader)
>      10
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\segmentation_models_pytorch\utils\train.py
> in run(self, dataloader)
>      43
>      44         with tqdm(dataloader, desc=self.stage_name, file=sys.stdout, disable=not (self.verbose)) as iterator:
> ---> 45             for x, y in iterator:
>      46                 x, y = x.to(self.device), y.to(self.device)
>      47                 loss, y_pred = self.batch_update(x, y)
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\tqdm\std.py
> in __iter__(self)    1169     1170         try:
> -> 1171             for obj in iterable:    1172                 yield obj    1173                 # Update and possibly print the
> progressbar.
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\torch\utils\data\dataloader.py
> in __next__(self)
>     433         if self._sampler_iter is None:
>     434             self._reset()
> --> 435         data = self._next_data()
>     436         self._num_yielded += 1
>     437         if self._dataset_kind == _DatasetKind.Iterable and \
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\torch\utils\data\dataloader.py
> in _next_data(self)
>     473     def _next_data(self):
>     474         index = self._next_index()  # may raise StopIteration
> --> 475         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
>     476         if self._pin_memory:
>     477             data = _utils.pin_memory.pin_memory(data)
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\torch\utils\data\_utils\fetch.py
> in fetch(self, possibly_batched_index)
>      42     def fetch(self, possibly_batched_index):
>      43         if self.auto_collation:
> ---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
>      45         else:
>      46             data = self.dataset[possibly_batched_index]
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\torch\utils\data\_utils\fetch.py
> in <listcomp>(.0)
>      42     def fetch(self, possibly_batched_index):
>      43         if self.auto_collation:
> ---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
>      45         else:
>      46             data = self.dataset[possibly_batched_index]
>
> <ipython-input-146-65256f8f536d> in __getitem__(self, i)
>      54         # apply preprocessing
>      55         if self.preprocessing:
> ---> 56             sample = self.preprocessing(image=image, mask=mask)
>      57             image, mask = sample['image'], sample['mask']
>      58
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\albumentations\core\composition.py
> in __call__(self, force_apply, *args, **data)
>     180                     p.preprocess(data)
>     181
> --> 182             data = t(force_apply=force_apply, **data)
>     183
>     184             if dual_start_end is not None and idx == dual_start_end[1]:
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\albumentations\core\transforms_interface.py
> in __call__(self, force_apply, *args, **kwargs)
>      87                     )
>      88                 kwargs[self.save_key][id(self)] = deepcopy(params)
> ---> 89             return self.apply_with_params(params, **kwargs)
>      90
>      91         return kwargs
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\albumentations\core\transforms_interface.py
> in apply_with_params(self, params, force_apply, **kwargs)
>     100                 target_function = self._get_target_function(key)
>     101                 target_dependencies = {k: kwargs[k] for k in self.target_dependence.get(key, [])}
> --> 102                 res[key] = target_function(arg, **dict(params, **target_dependencies))
>     103             else:
>     104                 res[key] = None
>
> C:\ProgramData\Anaconda3\envs\segmentation\lib\site-packages\albumentations\augmentations\transforms.py
> in apply_to_mask(self, mask, **params)    3068     def
> apply_to_mask(self, mask, **params):    3069         fn =
> self.custom_apply_fns["mask"]
> -> 3070         return fn(mask, **params)    3071     3072     def apply_to_bbox(self, bbox, **params):
>
> <ipython-input-186-4f194a842931> in to_tensor(x, **kwargs)
>      52
>      53
> ---> 54     return x.transpose(0,2,1).astype('float32')
>      55
>      56
>
> ValueError: axes don't match array


title: Pytorch normalize 2D tensor

body:
For more robustnes of my model I want to normalize my feature tensor.
I tried doing it the way that is to the best of my knowledge standard for pictures:
class Dataset(torch.utils.data.Dataset):
'Characterizes a dataset for PyTorch'
def __init__(self, input_tensor, transform = transforms.Normalize(mean= 0.5, std=0.5)):
    self.labels = input_tensor[:,:,-1]
    self.features = input_tensor[:,:,:-1]
    self.transform = transform

def __len__(self):
    return self.labels_planned.shape[0]

def __getitem__(self, index):

    # Load data and get label
    X = self.features[index]
    y = self.labelslabels[index]
    if self.transform:
        X = self.transform(X)
    return X, y

But receive this error message:
ValueError: Expected tensor to be a tensor image of size (C, H, W). Got tensor.size() = torch.Size([8, 25]).

Everywhere I looked people suggest that one should use .view to generate the third dimension in order to comply with the standard shape of pictures, but this seems very odd to me. Is there maybe a cleaner way to do this.
Also where should I best place the normalization? Just for the batch or for the entire train dataset?


title: what does colon wrapped class mean in python comment?

body:
What does colon wrapped class (:class:) mean in python comment?

For example,

class Optimizer(object):
    r"""Base class for all optimizers.
    Arguments:
        params (iterable): an iterable of :class:`torch.Tensor` s or
            :class:`dict` s. Specifies what Tensors should be optimized.
        defaults: (dict): a dict containing default values of optimization
            options (used when a parameter group doesn't specify them).
    """

    def __init__(self, params, defaults):
        self.defaults = defaults

Is it a pytorch-specific syntax, or what?

Source: https://github.com/pytorch/pytorch/blob/master/torch/optim/optimizer.py


title: Getting attention weights in opennmt-py

body:
Specifically in opennmt-py. Now there are many questions on this topic, such as Getting alignment/attention during translation in OpenNMT-py and the following thread on the opennmt forum https://github.com/OpenNMT/OpenNMT-py/issues/575. I use the code suggested by the latter. However none seem to address the problem I have. I try to run the following simple snippet of code.

import onmt
import onmt.inputters
import onmt.translate
import onmt.model_builder
from collections import namedtuple

Opt = namedtuple('Opt', ['models', 'data_type', 'reuse_copy_attn', "gpu"])

opt = Opt("/home/Desktop/hidden-att/model/hidden-2/seed-0/LSTMlang1_step_400.pt", "text",False, 0)
fields, model, model_opt =  onmt.model_builder.load_test_model(opt,{"reuse_copy_attn":False})

I get this error trace.

Traceback (most recent call last):

  File "<ipython-input-63-94c1f45c429f>", line 1, in <module>
    runfile('/home/Desktop/hidden-att/graph_hidden_exp.py', wdir='/home/Desktop/hidden-att')

  File "/home/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 786, in runfile
    execfile(filename, namespace)

  File "/home/anaconda3/lib/python3.7/site-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/home/Desktop/hidden-att/graph_hidden_exp.py", line 33, in <module>
    fields, model, model_opt =  onmt.model_builder.load_test_model(opt,{"reuse_copy_attn":False})

  File "../../Documents/NMT/OpenNMT-py/onmt/model_builder.py", line 85, in load_test_model
    map_location=lambda storage, loc: storage)

  File "/home/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 387, in load
    return _load(f, map_location, pickle_module, **pickle_load_args)

  File "/home/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 549, in _load
    _check_seekable(f)

  File "/home/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 194, in _check_seekable
    raise_err_msg(["seek", "tell"], e)

  File "/home/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 187, in raise_err_msg
    raise type(e)(msg)

AttributeError: 'dict' object has no attribute 'seek'. You can only torch.load from a file that is seekable. Please pre-load the data into a buffer like io.BytesIO and try to load from it instead.

So has anyone experienced and solved this problem? Or know where to look? I guess it is something about the loaded file, but it is trained with opennmt-py in a fairly standard way.


title: Image path from S3 to pytorch script for Sagemaker train

body:
I am building a AWS Pipeline to train a PyTorch model using Sagemaker. I have specified the s3 bucket (data/training)in Yaml file.

I usually upload a csv file in keras using,

dataframe = pd.read_csv(paths.input(channel='training', filename="ab.csv"))

In pytorch I am trying to use S3 bucket to get images in folders (labels as folder name and images inside the folder).

for the local compilation, the path is written as

datasource = "xyz/data/"

tr_data = ImageFolder(datasource, transform=tr_tf) in the working directory.

for using in sagemaker tha python bundle is specified as

datasource =  paths.input(channel='training') but AWS Sage-maker job fails.

Would be grateful if anyone help me to give a path of training images in S3 bucket.


title: Pytorch cut 2d array by lengths on first dimension

body:
I have a 2d array, let's say of size torch.tensor(batch_size, 1000).  The 1000 array from the second dimension is actually variable length.  I have a second array of size [batch_size] containing the length for each rows...

Here is an example code snippet:

# preds is the 2d array of size [batch_size, 1000]
# lengths is a 1d array containing the lengths of each row of preds
res_pred = []
for i in range(len(preds)):
    length = lengths[i].item()
    res_pred += [preds[i][:length]]

result = torch.cat(res_pred).flatten()

I do the same thing for my targets and then I can apply a loss function to both.

I was wondering if there was a single vectorized operation I could do to extract all batch_size vectors of variable lengths and torch.cat them together. Right now I am looping on the first dimension, but this feel slow.

Thanks,


I have got this error, can you help me please?
This is my error in utils.py file
 content/gdrive/My Drive/DeepFakeDetection/utils.py:21: UserWarning:
 volatile was removed and now has no effect. Use with torch.no_grad():
 instead. return Variable(x, volatile=volatile)

And this is my code in utils.py file
 def to_var(x, volatile=False):

     if torch.cuda.is_available():
        x = x.cuda()
     return Variable(x, volatile=volatile)

Thank you


title: PyTorch: Predicting future values with LSTM

body:
I'm currently working on building an LSTM model to forecast time-series data using PyTorch. I used lag features to pass the previous n steps as inputs to train the network. I split the data into three sets, i.e., train-validation-test split, and used the first two to train the model. My validation function takes the data from the validation data set and calculates the predicted valued by passing it to the LSTM model using DataLoaders and TensorDataset classes. Initially, I've got pretty good results with R2 values in the region of 0.85-0.95.
However, I have an uneasy feeling about whether this validation function is also suitable for testing my model's performance. Because the function now takes the actual X values, i.e., time-lag features, from the DataLoader to predict y^ values, i.e., predicted target values, instead of using the predicted y^ values as features in the next prediction. This situation seems far from reality where the model has no clue of the real values of the previous time steps, especially if you forecast time-series data for longer time periods, say 3-6 months.
I'm currently a bit puzzled about tackling this issue and defining a function to predict future values relying on the model's values rather than the actual values in the test set. I have the following function predict, which makes a one-step prediction, but I haven't really figured out how to predict the whole test dataset using DataLoader.
    def predict(self, x):
        # convert row to data
        x = x.to(device)
        # make prediction
        yhat = self.model(x)
        # retrieve numpy array
        yhat = yhat.to(device).detach().numpy()
        return yhat

You can find how I split and load my datasets, my constructor for the LSTM model, and the validation function below. If you need more information, please do not hesitate to reach out to me.
Splitting and Loading Datasets
def create_tensor_datasets(X_train_arr, X_val_arr, X_test_arr, y_train_arr, y_val_arr, y_test_arr):
    train_features = torch.Tensor(X_train_arr)
    train_targets = torch.Tensor(y_train_arr)
    val_features = torch.Tensor(X_val_arr)
    val_targets = torch.Tensor(y_val_arr)
    test_features = torch.Tensor(X_test_arr)
    test_targets = torch.Tensor(y_test_arr)

    train = TensorDataset(train_features, train_targets)
    val = TensorDataset(val_features, val_targets)
    test = TensorDataset(test_features, test_targets)

    return train, val, test

def load_tensor_datasets(train, val, test, batch_size=64, shuffle=False, drop_last=True):
    train_loader = DataLoader(train, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last)
    val_loader = DataLoader(val, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last)
    test_loader = DataLoader(test, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last)
    return train_loader, val_loader, test_loader

Class LSTM
class LSTMModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, layer_dim, output_dim, dropout_prob):
        super(LSTMModel, self).__init__()
        self.hidden_dim = hidden_dim
        self.layer_dim = layer_dim
        self.lstm = nn.LSTM(
            input_dim, hidden_dim, layer_dim, batch_first=True, dropout=dropout_prob
        )

        self.fc = nn.Linear(hidden_dim, output_dim)

    def forward(self, x, future=False):
        h0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()
        c0 = torch.zeros(self.layer_dim, x.size(0), self.hidden_dim).requires_grad_()

        out, (hn, cn) = self.lstm(x, (h0.detach(), c0.detach()))
        out = out[:, -1, :]
        out = self.fc(out)

        return out

Validation (defined within a trainer class)
    def validation(self, val_loader, batch_size, n_features):

        with torch.no_grad():
            predictions = []
            values = []
            for x_val, y_val in val_loader:
                x_val = x_val.view([batch_size, -1, n_features]).to(device)
                y_val = y_val.to(device)
                self.model.eval()
                yhat = self.model(x_val)
                predictions.append(yhat.cpu().detach().numpy())
                values.append(y_val.cpu().detach().numpy())

        return predictions, values


title: RNN can't learn integral function

body:
For studying deep learning, RNN, LSTM and so on I tried to make RNN fit integration function. I have put random signal from 0 to 1 as input to RNN and made integral from biased by -0.5 input signal, made the limit for integral between 0:1 and put it as RNN target to learn. Blue - random input, orange - integrated input

So I have time series with only one input (random) and one output (limited integral of input) and I want RNN to predict output by the input.
I used Pytorch and tried to use vanilla RNN, GRU cell, different sizes of hidden layers, stacking several RNN, putting dense connected layers to the RNN output, different deep in backpropagation through time (from 2 to 50 gradients rolling-back). And I can't get a good result at all! It works somehow, but I can't find a way to fit integral function precisely. Here is the best of my results:
green - RNN output. Green line (model output) does not fit orange line in many cases - that is the problem.

Here is my source code in jupyter.
My questions: is it possible - to learn a saturated integral function by RNN? Where is my problem? What can I try more to achieve good quality? Ideally I want to RNN output be equal desired output (integral function) through all time series.

PS:
My code in raw format:

import numpy as np
from scipy.stats import truncnorm
import random
import math
import copy
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
import matplotlib.cm as cm

def generate_data(num_of_data):
    input_data=[]
    output_data=[]
    current_input_value=0
    current_output_value=0
    for i in range(num_of_data):
        if (random.random()<0.1):
            current_input_value=random.random()
#            current_output_value=0
        current_input_value=current_input_value+(random.random()-0.5)*0
        current_output_value=current_output_value+0.0*(current_input_value-current_output_value)+(current_input_value-0.5)*0.1
        if (current_output_value<0):
            current_output_value=0
        if (current_output_value>1):
            current_output_value=1

        input_data.append(current_input_value)
        output_data.append(current_output_value)
    return input_data,output_data

%matplotlib inline
matplotlib.rcParams['figure.figsize'] = (20, 6)

input_data,output_data=generate_data(500)
plt.plot(input_data)
plt.plot(output_data)
plt.show()

import torch
import torch.nn as nn
from torch.autograd import Variable
from torch import optim

class RNN(nn.Module):
    def __init__(self, input_size, hidden_size, output_size):
        super(RNN, self).__init__()
        self.number_of_layers=1
        self.hidden_size = hidden_size
        self.gru = nn.GRU(input_size, hidden_size,self.number_of_layers)
        self.Dense1 = nn.Linear(hidden_size, hidden_size)
        self.Dense1A = nn.ReLU()
        self.Dense2 = nn.Linear(hidden_size, output_size)

    def forward(self, input, hidden):
        gru_output, hidden = self.gru(input, hidden)
        Dense1Out=self.Dense1(gru_output)
        Dense1OutAct=self.Dense1A(Dense1Out)
        output=self.Dense2(Dense1OutAct)
        return output, hidden

    def initHidden(self):
        return Variable(torch.zeros(self.number_of_layers,1,self.hidden_size))

import time
import math
import operator

def timeSince(since):
    now = time.time()
    s = now - since
    m = math.floor(s / 60)
    s -= m * 60
    return '%dm %ds' % (m, s)
rnn = RNN(1, 50, 1)

n_iters = 250000
print_every = 2000
plot_every = 2000
all_losses = []
total_loss_print = 0
total_loss_plot = 0

criterion=nn.L1Loss()

print("training...\n")

start = time.time()
optimizer = optim.Adam(rnn.parameters(), lr=0.0002)
rnn_hidden = rnn.initHidden()
rnn.zero_grad()
loss = 0
#for gata_q in range(int(n_iters/500)):
#    rnn_hidden = rnn.initHidden()

input_data,output_data=generate_data(n_iters)
for data_index in range(len(input_data)):
    input_tensor=torch.zeros(1, 1, 1)
    input_tensor[0][0][0]=input_data[data_index]

    output_tensor=torch.zeros(1, 1, 1)
    output_tensor[0][0][0]=output_data[data_index]

    rnn_output, rnn_hidden = rnn(Variable(input_tensor), rnn_hidden)
    loss += criterion(rnn_output, Variable(output_tensor))

    if data_index%2==0:
        loss.backward()

        total_loss_print += loss.data[0]
        total_loss_plot += loss.data[0]
        optimizer.step()
        rnn_hidden=Variable(rnn_hidden.data)
        rnn.zero_grad()
        loss = 0

    if data_index % print_every == 0:
        print('%s (%d %d%%) tl=%.4f' % (timeSince(start), data_index, data_index / n_iters * 100,total_loss_print/print_every))
        total_loss_print = 0

    if data_index % plot_every == 0:
        all_losses.append(total_loss_plot / plot_every)
        total_loss_plot = 0

import matplotlib.pyplot as plt
import matplotlib.ticker as ticker

plt.figure()
plt.plot(all_losses)
plt.show()

rnn_hidden = rnn.initHidden()

rnn.zero_grad()
loss = 0

rnn_output_data=[]
input_data,output_data=generate_data(1500)
for data_index in range(len(input_data)):
    input_tensor=torch.zeros(1, 1, 1)
    input_tensor[0][0][0]=input_data[data_index]
    rnn_output, rnn_hidden = rnn(Variable(input_tensor), rnn_hidden)
    rnn_output_data.append(rnn_output.data.numpy()[0][0][0])

plt.plot(input_data)#blue
plt.plot(output_data)#ogange
plt.plot(rnn_output_data)#green
plt.show()


title: "Expected Long but got Int" while running PyTorch script

body:
I am facing this error while running a pytorch script.

Can anyone help me?

Expected object of scalar type Long but got scalar type Int for argument #2 'target'


title: Is there any code torch.backends.cudnn.benchmark/torch.cuda.synchronize() similar in tensorflow?

body:
I am working on optimizing CUDA program's performance.
I used torch.backends.cudnn.benchmark to optimize performance and torch.cuda.synchronize() to synchronize CUDA applications in pytorch.
To do the same job in tensorflow I searched a lot time whether similar code is in tensorflow, however I could'nt find anything.

I wonder is there any equivalent code to
torch.cuda.synchronize() and torch.backends.cudnn.benchmark exists in tensorflow.


title: Basic Pytorch tensor multiplication and addition

body:
I just realize I lack some very basic pytorch tensor math. How do I do the following with a pytorch tensor?

lab_rs = (lab_rs * [100, 255, 255] - [0, 128, 128])

This works well in numpy. It's an image with shape (3, 512, 1024) and I want to multiply and subtract values from each color channel individually

The error I get trying the same with a tensor is:

TypeError: only integer tensors of a single element can be converted to an index