title: Vectorizing PyTorch operation in forward

body:
I wrote this PyTorch Module to take two groups of feature maps outputted by two separate convolution layers and average them. The inputs are two Tensors containing the same number of images with the same number of channels ([batch_size, channels, Height, Width]). The feature maps from the two images are of different sizes.
    class Avg_Combine(nn.Module):
        def __init__(self):
            super(Avg_Combine, self).__init__()

        def forward(self, a, b):
            if a.shape[2] < b.shape[2]:
                factor = b.shape[2] / a.shape[2]
                a = functional.interpolate(a, scale_factor=factor)
            elif a.shape[2] > b.shape[2]:
                factor = a.shape[2] / b.shape[2]
                b = functional.interpolate(b, scale_factor=factor)

            idx0 = 0
            for img in a:
                idx1 = 0
                for channel_a in img:
                    channel_a = channel_a.add(b[idx0][idx1]) / 2
                    idx1 += 1
                idx0 += 1

            return a

For example, a=[20, 128, 8, 8], b=[20, 128, 4, 4]. The code will upsample b to match the size of a. Now that a and b are all of the size [20, 128, 8, 8], they are averaged on the "channel" dimension to generate an output of size [20, 128, 8, 8].
However, during training, this function is like 50~100 times slower than a regular CNN with the same layers. I'm suspecting that the for-loop is the cause of the issue. How can I vectorize this operation? Are there any other suggestions on how I can possibly speed this thing up?
Thanks a lot! Any suggestion is appreciated!


title: How can I define sentence position embeddings?

body:
My goal is to generate sentence position embeddings like this article: Multiview Convolutional Neural Networks for Multidocument Extractive Summarization
As described in the above article, first we denote the position of a sentence si
according to the following equation:
p(si)={0 if si ∈ S1:3
p(si)={1 if si ∈ S-3:-1
p(si)={2 otherwise

S1:3 denotes the set containing the first three sentences of a document
S-3:-1 denotes the set containing the last three sentences of a document
Then, the three integers are mapped to a vector space h. The mapping procedure is done like that of word embedding (x = Lw, where w is a one hot vector and L is a word representation matrix).
My first problem is: (1) Why the authors use from 0, 1 and 2 values? Are these values important? or can I change them with any other number?
Second problem: (2) I think that the beginning and last sentences are more important in the text than middle ones. Is it reflect in 0, 1, and 2 values? Wouldn't it be better, for example, to use a value of 10 for the beginning and end sentences and a value of 0 for the middle sentences so that a larger value indicates more importance for the beginning and end sentences?
I want to use embedding layer in PyTorch to implement it. I know that embedding layer is like a lookup table that the input is a list of indices and the output is their embedding vectors. In the case of sentence position embedding,I should assign a unique index to each sentence (0 for the 0th sentence in the document, 1 for the 1th sentence and so on), but my original problem is:
Third problem: (3) I can't understand where should I use from p(si) values (0, 1 or 2)? Is the size of embedding matrix (3, embedding_dimension) or (number_of_sentences, embedding_dimension)?
Thanks in advance.


title: How to concat two tensors of size [B,C,13,18] and [B,C,14,18] respectively in Pytorch?

body:
I often met this problem when the height or width of an image or a tensor becomes odd.
For example, suppose the original tensor is of size [B,C,13,18]. After forwarding a strided-2 conv and several other conv layers, its size will become [B,C,7,9]. If we upsample the output by 2 and concat it with the original feature map as most cases, the error occurs.
I found that in many source codes, they use even sizes like (512,512) for training, so this kind of problem won't happen. But for test, I use the original image size to keep fine details and often met this problem.
What should I do? Do I need to change the network architecture?


title: Efficient occlusion-1 in pytorch

body:
I have a question on performing the task calculating occlusion-1 using PyTorch.The model is a convolution-based model, like the first several layers are convolution, pooling,or batchnormalization, followed by a fully connected layer.
I want to calculate occlusion-1, which is a mutation on a single pixel on one channel of the input image,and then calculating the difference of the output value and the original output(The output is a scalar). Since an image,or a 1d sequence input, contains tens of thousands of pixels, this is really time-consuming.
Is there any accelerating techniques to improve the time efficiency? Since the input tensor has a lot of components being the same, and exactly only one value on one channel, on one pixel will be changed(from the reference value of 0, or other values to 1).
The example code of the task to calculate occlusion-1 is given an input 'x' and 'model', the size of 'x', for a 1d sequence may be x.size()=torch.Size([1,3,100000]).And occlusion will calculate the difference
y0=model(x)
for i in range(1000000):
     for j in range(3):
        x1=x.clone()
        x1[:,j,i]=torch.tensor(1)
        diff.append(model(x1)-y0)

The forward operation could be done on GPU and it has been omitted in the code.Assume we could NOT use talyer expansion to approximate the results.


title: The Tutorials in Qiskit: Option Pricing with qGANs

body:
# Set upper and lower data values
bounds = np.array([0.,7.])
# Set number of qubits used in the uncertainty model
num_qubits = [3]

# Set entangler map
entangler_map = []
for i in range(sum(num_qubits)):
    entangler_map.append([i, int(np.mod(i+1, sum(num_qubits)))])

# Load the trained circuit parameters
g_params = [0.29399714, 0.38853322, 0.9557694, 0.07245791, 6.02626428, 0.13537225]

# Set an initial state for the generator circuit
init_dist = NormalDistribution(sum(num_qubits), mu=1., sigma=1., low=bounds[0], high=bounds[1])
init_distribution = np.sqrt(init_dist.probabilities)
init_distribution = Custom(num_qubits=sum(num_qubits), state_vector=init_distribution)

I want to know how to obtain the trained circuit parameters, which Attribute or Method should be used?


title: About pytorch the output is not what I expect

body:
import torch
a = torch.randn(2, 2)
a = ((a * 3) / (a - 1))
print(a)

The output is:
tensor([[  -0.7242,    2.0021],
        [-280.8320,    0.6750]])

But I think it should be:
tensor([[  6,    6],
        [  6,    6]])

Why I'm wrong?


title: Pytorch runtime error in semantic segmentation

body:
I am trying to reproduce semantic segmentation in this blog post. I have done some minor adjustments but still can't get the model to train. I always get the runtimeerror: Given groups=1, weight of size [512, 1024, 1, 1], expected input[4, 512, 188, 188] to have 1024 channels, but got 512 channels instead. I have tried to unsueeze the input but no change.
Here's the network
class ConvRelu(nn.Module):

    def __init__(self, in_depth, out_depth):
        super(ConvRelu, self).__init__()
        self.conv = torch.nn.Conv2d(in_depth, out_depth, kernel_size=3, stride=1, padding=1)
        self.activation = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv(x)
        x = self.activation(x)
        return x

class DecoderBlock(nn.Module, ABC):
    def __init__(self, in_depth, middle_depth, out_depth):
        super(DecoderBlock, self).__init__()
        self.conv_relu = ConvRelu(in_depth, middle_depth)
        self.conv_transpose = nn.ConvTranspose2d(middle_depth, out_depth, kernel_size=4, stride=2, padding=1)
        self.activation = nn.ReLU(inplace=True)

    def forward(self, x):
        x = self.conv_relu(x)
        x = self.conv_transpose(x)
        x = self.activation(x)
        return x

class UNetResNet(nn.Module):
    def __init__(self, num_classes):
        super(UNetResNet, self).__init__()
        self.encoder = torchvision.models.resnet101(pretrained=True)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv1 = nn.Sequential(self.encoder.conv1, self.encoder.bn1, self.encoder.relu, self.pool)
        self.conv2 = self.encoder.layer1
        self.conv3 = self.encoder.layer2
        self.conv4 = self.encoder.layer3
        self.conv4 = self.encoder.layer4

        self.pool = nn.MaxPool2d(2, 2)
        self.center = DecoderBlock(2048, 512, 256)

        self.dec5 = DecoderBlock(2048 + 256, 512, 256)
        self.dec4 = DecoderBlock(1024 + 256, 512, 256)
        self.dec3 = DecoderBlock(512 + 256, 256, 64)
        self.dec2 = DecoderBlock(256 + 64, 128, 128)
        self.dec1 = DecoderBlock(128, 128, 32)
        self.dec0 = ConvRelu(32, 32)
        self.final = nn.Conv2d(32, num_classes, kernel_size=1)

    def forward(self, x):
        conv1 = self.conv1(x)
        conv2 = self.conv2(conv1)
        conv3 = self.conv3(conv2)
        conv4 = self.conv4(conv3)
        conv5 = self.conv5(conv4)

        pool = self.pool(conv5)
        center = self.center(pool)

        dec5 = self.dec5(torch.cat([center, conv5], 1))
        dec4 = self.dec4(torch.cat([dec5, conv4], 1))
        dec3 = self.dec4(torch.cat([dec4, conv3], 1))
        dec2 = self.dec4(torch.cat([dec3, conv2], 1))
        dec1 = self.dec1(dec2)
        dec0 = self.dec0(dec1)

        return self.final(dec0)```

and here's the training loop

for epoch_idx in range(2):
    loss_batches = []
    for batch_idx, data in enumerate(tqdm(train_dataloader, desc = "training")):

        imgs = torch.autograd.Variable(data['sat_img'].cuda())
        #imgs = imgs.view(1, -1)
        #imgs.unsqueeze(0)
        #imgs.squeeze()
        masks = torch.autograd.Variable(data['map_img'].cuda())
        masks.unsqueeze_(0)
        #print(imgs.size())

        y = unet_resnet(imgs)
        loss = cross_entropy_loss(y, masks)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        loss_batches.append(loss.data.cpu().numpy())
    print('epoch: ' + str(epoch_idx) + ' training loss: ' + str(np.sum(loss_batches)))

model_file = './unet-' + str(epoch_idx)
unet_resnet = unet_resnet.cpu()
torch.save(unet_resnet.state_dict(), model_file)
unet_resnet = unet_resnet.cuda()
print('model saved') ```

Please I need some help. According to the error message, the error is on conv4 in the network. What do I need to change there or how can I adjust the input?
The input images are PIL images


title: Command Errored Exit Status 1: - Pytorch Object Detection

body:
I tried to follow this tutorial to learn how to run my own object detection, but I am running into an error that I can't seem to fix. I found a solution on some git hubs issues pages. They suggested running: !pip install git+https://github.com/philferriere/cocoapi.git, but I still get the same error. I am using Google Colab. Any suggestions?
input:
!pip installgit+https://github.com/cocodataset/cocoapi.git

output:
Collecting git+https://github.com/cocodataset/cocoapi.git
  Cloning https://github.com/cocodataset/cocoapi.git to /tmp/pip-req-build-u21a5wjs
  Running command git clone -q https://github.com/cocodataset/cocoapi.git /tmp/pip-req-build-u21a5wjs
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.


title: Pytorch couldn't build multi scaled kernel nested model

body:
I'm trying to create a modified MNIST model which takes input 1x28x28 MNIST tensor images, and it kind of branches into different models with different sized kernels, and accumulates at the end, so as to give a multi-scale-kerneled response in the spatial domain of the images. I'm worried about the model, since, I'm unable to construct it.

import numpy as np
import torch
import torch.nn as nn
import torch.optim as optim
import torch.utils.data as Data
from torchvision import datasets, transforms
import torch.nn.functional as F
import timeit
import unittest

torch.manual_seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
np.random.seed(0)

# check availability of GPU and set the device accordingly

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# define a transforms for preparing the dataset
transform = transforms.Compose([
        transforms.ToTensor(), # convert the image to a pytorch tensor
        transforms.Normalize((0.1307,), (0.3081,)) # normalise the images with mean and std of the dataset
        ])

# Load the MNIST training, test datasets using `torchvision.datasets.MNIST` using the transform defined above

train_dataset = datasets.MNIST('./data',train=True,transform=transform,download=True)
test_dataset =  datasets.MNIST('./data',train=False,transform=transform,download=True)

# create dataloaders for training and test datasets
# use a batch size of 32 and set shuffle=True for the training set

train_dataloader = Data.DataLoader(dataset=train_dataset, batch_size=32, shuffle=True)
test_dataloader = Data.DataLoader(dataset=test_dataset, batch_size=32, shuffle=True)

# My Net

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()

        # define a conv layer with output channels as 16, kernel size of 3 and stride of 1
        self.conv11 = nn.Conv2d(1, 16, 3, 1) # Input = 1x28x28  Output = 16x26x26
        self.conv12 = nn.Conv2d(1, 16, 5, 1) # Input = 1x28x28  Output = 16x24x24
        self.conv13 = nn.Conv2d(1, 16, 7, 1) # Input = 1x28x28  Output = 16x22x22

        # define a conv layer with output channels as 32, kernel size of 3 and stride of 1
        self.conv21 = nn.Conv2d(16, 32, 3, 1) # Input = 16x26x26 Output = 32x24x24
        self.conv22 = nn.Conv2d(16, 32, 5, 1) # Input = 16x24x24 Output = 32x20x20
        self.conv23 = nn.Conv2d(16, 32, 7, 1) # Input = 16x22x22 Output = 32x16x16

        # define a conv layer with output channels as 64, kernel size of 3 and stride of 1
        self.conv31 = nn.Conv2d(32, 64, 3, 1) # Input = 32x24x24 Output = 64x22x22
        self.conv32 = nn.Conv2d(32, 64, 5, 1) # Input = 32x20x20 Output = 64x16x16
        self.conv33 = nn.Conv2d(32, 64, 7, 1) # Input = 32x16x16 Output = 64x10x10

        # define a max pooling layer with kernel size 2
        self.maxpool = nn.MaxPool2d(2), # Output = 64x11x11
        # define dropout layer with a probability of 0.25
        self.dropout1 = nn.Dropout(0.25)
        # define dropout layer with a probability of 0.5
        self.dropout2 = nn.Dropout(0.5)
        # define a linear(dense) layer with 128 output features
        self.fc11 = nn.Linear(64*11*11, 128)
        self.fc12 = nn.Linear(64*8*8, 128)      # after maxpooling 2x2
        self.fc13 = nn.Linear(64*5*5, 128)

        # define a linear(dense) layer with output features corresponding to the number of classes in the dataset
        self.fc21 = nn.Linear(128, 10)
        self.fc22 = nn.Linear(128, 10)
        self.fc23 = nn.Linear(128, 10)

        self.fc33 = nn.Linear(30,10)

    def forward(self, x1):
        # Use the layers defined above in a sequential way (folow the same as the layer definitions above) and
        # write the forward pass, after each of conv1, conv2, conv3 and fc1 use a relu activation.

        x = F.relu(self.conv11(x1))
        x = F.relu(self.conv21(x))
        x = F.relu(self.maxpool(self.conv31(x)))
        #x = torch.flatten(x, 1)
        x = x.view(-1,64*11*11)
        x = self.dropout1(x)
        x = F.relu(self.fc11(x))
        x = self.dropout2(x)
        x = self.fc21(x)

        y = F.relu(self.conv12(x1))
        y = F.relu(self.conv22(y))
        y = F.relu(self.maxpool(self.conv32(y)))
        #x = torch.flatten(x, 1)
        y = y.view(-1,64*8*8)
        y = self.dropout1(y)
        y = F.relu(self.fc12(y))
        y = self.dropout2(y)
        y = self.fc22(y)

        z = F.relu(self.conv13(x1))
        z = F.relu(self.conv23(z))
        z = F.relu(self.maxpool(self.conv33(z)))
        #x = torch.flatten(x, 1)
        z = z.view(-1,64*5*5)
        z = self.dropout1(z)
        z = F.relu(self.fc13(z))
        z = self.dropout2(z)
        z = self.fc23(z)

        out = self.fc33(torch.cat((x, y, z), 0))

        output = F.log_softmax(out, dim=1)
        return output

import unittest

class TestImplementations(unittest.TestCase):

    # Dataloading tests
    def test_dataset(self):
        self.dataset_classes = ['0 - zero',
                                '1 - one',
                                '2 - two',
                                '3 - three',
                                '4 - four',
                                '5 - five',
                                '6 - six',
                                '7 - seven',
                                '8 - eight',
                                '9 - nine']
        self.assertTrue(train_dataset.classes == self.dataset_classes)
        self.assertTrue(train_dataset.train == True)

    def test_dataloader(self):
        self.assertTrue(train_dataloader.batch_size == 32)
        self.assertTrue(test_dataloader.batch_size == 32)

    def test_total_parameters(self):
        model = Net().to(device)
        #self.assertTrue(sum(p.numel() for p in model.parameters()) == 1015946)

suite = unittest.TestLoader().loadTestsFromModule(TestImplementations())
unittest.TextTestRunner().run(suite)

def train(model, device, train_loader, optimizer, epoch):
    model.train()
    for batch_idx, (data, target) in enumerate(train_loader):
        # send the image, target to the device
        data, target = data.to(device), target.to(device)
        # flush out the gradients stored in optimizer
        optimizer.zero_grad()
        # pass the image to the model and assign the output to variable named output
        output = model(data)
        # calculate the loss (use nll_loss in pytorch)
        loss = F.nll_loss(output, target)
        # do a backward pass
        loss.backward()
        # update the weights
        optimizer.step()

        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader), loss.item()))

def test(model, device, test_loader):
    model.eval()
    test_loss = 0
    correct = 0
    with torch.no_grad():
        for data, target in test_loader:

            # send the image, target to the device
            data, target = data.to(device), target.to(device)
            # pass the image to the model and assign the output to variable named output
            output = model(data)
            test_loss += F.nll_loss(output, target, reduction='sum').item() # sum up batch loss

            pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
            correct += pred.eq(target.view_as(pred)).sum().item()

    test_loss /= len(test_loader.dataset)

    print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(test_loader.dataset),
        100. * correct / len(test_loader.dataset)))

model = Net().to(device)

## Define Adam Optimiser with a learning rate of 0.01
optimizer =  torch.optim.Adam(model.parameters(),lr=0.01)

start = timeit.default_timer()
for epoch in range(1, 11):
  train(model, device, train_dataloader, optimizer, epoch)
  test(model, device, test_dataloader)
stop = timeit.default_timer()
print('Total time taken: {} seconds'.format(int(stop - start)) )

Here is my full code. I couldn't understand what could possibly go wrong...
It is giving
<ipython-input-72-194680537dcc> in forward(self, x1)
     46         x = F.relu(self.conv11(x1))
     47         x = F.relu(self.conv21(x))
---> 48         x = F.relu(self.maxpool(self.conv31(x)))
     49         #x = torch.flatten(x, 1)
     50         x = x.view(-1,64*11*11)

TypeError: 'tuple' object is not callable

Error.
P.S.: Pytorch Noob here.


title: Why does the same PyTorch code (different implementation) give different loss?

body:
I was tackling the Fashion MNIST data-set problem on Udacity. However my implementation of code is giving drastically different loss as compared to the solution shared by the Udacity team. I believe the only difference in my answer is the definition of the Neural Network and apart from that everything is the same. I am not able to figure out the reason for such a drastic difference in Loss.
Code 1: My solution:
import torch.nn as nn
from torch import optim

images, labels = next(iter(trainloader))
model = nn.Sequential(nn.Linear(784,256),
                 nn.ReLU(),
                 nn.Linear(256,128),
                 nn.ReLU(),
                 nn.Linear(128,64),
                 nn.ReLU(),
                 nn.Linear(64,10),
                 nn.LogSoftmax(dim=1))
# Flatten images
optimizer = optim.Adam(model.parameters(),lr=0.003)

criterion = nn.NLLLoss()

for i in range(10):
    running_loss = 0
    for images,labels in trainloader:
        images = images.view(images.shape[0], -1)

        output = model.forward(images)
        loss = criterion(output,labels)
        loss.backward()
        optimizer.step()
        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss}")

# Loss is coming around 4000

Code 2: Official Solution:
from torch import nn, optim
import torch.nn.functional as F

class Classifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(784, 256)
        self.fc2 = nn.Linear(256, 128)
        self.fc3 = nn.Linear(128, 64)
        self.fc4 = nn.Linear(64, 10)

    def forward(self, x):
        # make sure input tensor is flattened
        x = x.view(x.shape[0], -1)

        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = F.relu(self.fc3(x))
        x = F.log_softmax(self.fc4(x), dim=1)

        return x

model = Classifier()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)
epochs = 5

for e in range(epochs):
    running_loss = 0
    for images, labels in trainloader:
        log_ps = model(images)
        loss = criterion(log_ps, labels)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        running_loss += loss.item()
    else:
        print(f"Training loss: {running_loss}")
# Loss is coming around 200

Is there any explanation for the vast difference in loss ?


title: Getting a 'Field' object has no attribute 'vocab' error when trying to predict on a test set

body:
Ive been successfully able to train models with my train data, but am having some trouble now predicting on my test data. So I am loading in by test data with
id1 = ttd.LabelField(use_vocab =False)
test_dataset= ttd.TabularDataset(
    path= 'test_1.csv',
    format='csv',
    skip_header=True,
    fields=[('id', id1),('text', TEXT)]
)
test_iter= ttd.BucketIterator(
    test_dataset_kaggle,
    batch_size=64,
    sort_key=lambda x: len(x.text),
    device=device)

and use the following function for prediction
def get_predictions_test(test_iter, model):
  model.eval()
  with torch.no_grad():
    predictions= np.array([])
    id2= np.array([])

    for batch in test_iter:
      output=model(batch.data)
      ##print(np.shape(output))
      ##print(output)
      y_pred_tag = np.squeeze(torch.round((output)))
      #_,indices = torch.max(output,dim=1)
      ##print(np.shape(predictions))
      ##print(np.shape(y_pred_tag))
      predictions=np.concatenate((predictions,y_pred_tag.cpu().numpy()))
      ##print(predictions)
      ##y_test = np.concatenate((y_test,batch.label.cpu().numpy()))
      id2 = np.concatenate((id2,batch.id.cpu().numpy()))

  return id2, predictions

However, when I try to run
id4, predictions = get_predictions_test(test_iter,rnn_model)

I get the error 'Field' object has no attribute 'vocab'. I already built a vocab for TEXT so that shouldn't be the problem. Am I supposed to build a vocab for ID? Or is there something else I am doing wrong


title: Additional parameters in def__init__ for Optuna Trial

body:
class ConvolutionalNetwork(nn.Module):
    def __init__(self, in_features):
        super().__init__()
        self.in_features = in_features
        # this computes num features outputted from the two conv layers
        c1 = int(((self.in_features - 2)) / 64)  # this is to account for the loss due to conversion to int type
        c2 = int((c1-2)/64)
        self.n_conv = int(c2*16)
        #self.n_conv = int((( ( (self.in_features - 2)/4 ) - 2 )/4 ) * 16)
        self.conv1 = nn.Conv1d(1, 16, 3, 1)
        self.conv1_bn = nn.BatchNorm1d(16)
        self.conv2 = nn.Conv1d(16, 16, 3, 1)
        self.conv2_bn = nn.BatchNorm1d(16)
        self.dp = nn.Dropout(trial.suggest_uniform('dropout_rate',0,1.0))
        self.fc3 = nn.Linear(self.n_conv, 2)

As you can see, def __init__ already has self and in_features as the variable. I am thinking of adding another variable trial (which is part of the Optuna package) to accommodate
    self.dp = nn.Dropout(trial.suggest_uniform('dropout_rate',0,1.0))

in the above codes. Please advise how, most only sources only has def __init__ (self, trial), which is pretty straightforward, but for my case I have 3 variables to be passed in the objective.


title: ImportError about libstdc++.so.6 while excuting exe file packed by pyinstaller

body:
I have a python file using yolov5 and thinter. It runs normally in the cmd and pycharm. But after packed to exe with pyinstaller, running this exe returning error as below:
Traceback (most recent call last):
  File "train.py", line 14, in <module>
    import torch.distributed as dist
  File "/home/gph/anaconda3/envs/py38_pth16/lib/python3.8/site-packages/PyInstaller/loader/pyimod03_importers.py", line 493, in exec_module
    exec(bytecode, module.__dict__)
  File "torch/__init__.py", line 189, in <module>
ImportError: /tmp/_MEIebgYfG/./libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /tmp/_MEIebgYfG/torch/lib/libtorch_python.so)
[32356] Failed to execute script train

My environment is:

Ubuntu 16.04
python 3.8
pytorch 1.6
pyinstaller 4.1
conda 4.8.3
Please help!


title: Optuna Autotuning

body:
anyone tried Optuna autotuning before?
   #Setup optuna platform for autotuning, with TPESampler to minimize loss
    study = optuna.create_study(sampler=optuna.samplers.TPESampler(), direction="minimize")
    study.optimize(autotune, n_trials=300)
    pruned_trials = [t for t in study.trials if t.state == optuna.structs.TrialState.PRUNED]
    complete_trials = [t for t in study.trials if t.state == optuna.structs.TrialState.COMPLETE]

Tried setting number of trials to 100,200,300 but the results are still not comparable to manual tuning for CNN , RNN , BiRNN.
Anyone experienced this as well? I wonder if I should set no of trials to 1000 to try out more permutations of auto-tuning.
#Add in parameters to be optimized:lr, optimizer , weight_decay:
optimizer_name = trial.suggest_categorical("optimizer", ["Adam", "RMSprop", "SGD"])
lr = trial.suggest_loguniform("lr", 1e-5, 1e-1)
Weight_decay=trial.suggest_loguniform('Weight_decay',1e-10,1e-3)
#momentum=trial.suggest_loguniform('momentum',1e-5,1e-1)
optimizer = getattr(optim, optimizer_name)(model.parameters(), lr=lr,weight_decay=Weight_decay)


title: linear interpolation between checkpoints in pytorch - Batch Normalization

body:
I want to linearly interpolate between two PyTorch trained model checkpoints. For all layers except the batch normalization, I load the stated dict and simply do the linear inteporaltion as follow:
def interpolate_state_dicts(state_dict_1, state_dict_2, weight):
return {key: (1 - weight) * state_dict_1[key] + weight * state_dict_2[key]
        for key in state_dict_1.keys()}

I do not know if we can simply do the same for BN layer parameters (Weight, Bias, Running mean, running std) or not? I guess it is not that simple, as mean and std are calculated for a specific batch.


title: TPU freeze when trying to use PyTorch/XLA, pytorch_lightning and custom autograd function

body:
The complete code can be found here. This is done on Google Colab, using the PyTorch/XLA to run the TabNet model. I have tested other models without custom autograd function using the same setting, and it works ok. However, when I use the autograd function, the training process just freeze. In short, the trouble seems to stem from the following code fragments.
class SparsemaxFunction(Function):
    """
    An implementation of sparsemax (Martins & Astudillo, 2016). See
    :cite:`DBLP:journals/corr/MartinsA16` for detailed description.
    By Ben Peters and Vlad Niculae
    """

    @staticmethod
    def forward(ctx, input, dim=-1):
        """sparsemax: normalizing sparse transform (a la softmax)

        Parameters
        ----------
        ctx : torch.autograd.function._ContextMethodMixin
        input : torch.Tensor
            any shape
        dim : int
            dimension along which to apply sparsemax

        Returns
        -------
        output : torch.Tensor
            same shape as input

        """
        ctx.dim = dim
        max_val, _ = input.max(dim=dim, keepdim=True)
        input -= max_val  # same numerical stability trick as for softmax
        tau, supp_size = SparsemaxFunction._threshold_and_support(input, dim=dim)
        output = torch.clamp(input - tau, min=0)
        ctx.save_for_backward(supp_size, output)
        return output

    @staticmethod
    def backward(ctx, grad_output):
        supp_size, output = ctx.saved_tensors
        dim = ctx.dim
        grad_input = grad_output.clone()
        grad_input[output == 0] = 0

        v_hat = grad_input.sum(dim=dim) / supp_size.to(output.dtype).squeeze()
        v_hat = v_hat.unsqueeze(dim)
        grad_input = torch.where(output != 0, grad_input - v_hat, grad_input)
        return grad_input, None

    @staticmethod
    def _threshold_and_support(input, dim=-1):
        """Sparsemax building block: compute the threshold

        Parameters
        ----------
        input: torch.Tensor
            any dimension
        dim : int
            dimension along which to apply the sparsemax

        Returns
        -------
        tau : torch.Tensor
            the threshold value
        support_size : torch.Tensor

        """

        input_srt, _ = torch.sort(input, descending=True, dim=dim)
        input_cumsum = input_srt.cumsum(dim) - 1
        rhos = _make_ix_like(input, dim)
        support = rhos * input_srt > input_cumsum

        support_size = support.sum(dim=dim).unsqueeze(dim)
        tau = input_cumsum.gather(dim, support_size - 1)
        tau /= support_size.to(input.dtype)
        return tau, support_size

sparsemax = SparsemaxFunction.apply

class Sparsemax(nn.Module):

    def __init__(self, dim=-1):
        self.dim = dim
        super(Sparsemax, self).__init__()

    def forward(self, input):
        return sparsemax(input, self.dim)

When I remove the backward pass the whole code works perfectly fine. The updated version is this.
class Sparsemax(nn.Module):

    def __init__(self, dim=-1):
        self.dim = dim
        super(Sparsemax, self).__init__()

    def forward(self, input):
        max_val, _ = input.max(dim=self.dim, keepdim=True)
        input -= max_val  # same numerical stability trick as for softmax
        tau, supp_size = Sparsemax._threshold_and_support(input, dim=self.dim)
        output = torch.clamp(input - tau, min=0)
        return output

    @staticmethod
    def _threshold_and_support(input, dim=-1):
        """Sparsemax building block: compute the threshold

        Parameters
        ----------
        input: torch.Tensor
            any dimension
        dim : int
            dimension along which to apply the sparsemax

        Returns
        -------
        tau : torch.Tensor
            the threshold value
        support_size : torch.Tensor

        """

        input_srt, _ = torch.sort(input, descending=True, dim=dim)
        input_cumsum = input_srt.cumsum(dim) - 1
        rhos = _make_ix_like(input, dim)
        support = rhos * input_srt > input_cumsum

        support_size = support.sum(dim=dim).unsqueeze(dim)
        tau = input_cumsum.gather(dim, support_size - 1)
        tau /= support_size.to(input.dtype)
        return tau, support_size

Unfortunately, doing so means the training accuracy is greatly damaged. Although still in terms of speed, the TPU still prevails (even accounting for the different epochs needed).
I wonder whether this is a bug for PyTorch/XLA or I am making a mistake.


title: The ONNX model's constant layer tensor dimensions size must be 1, 2, 3 or 4 Snapchat Lens Studio Error

body:
I exported my Pytorch model to ONNX to be a SnapML model but when I tried to load it into lens studio I get the error message "The ONNX model's constant layer tensor dimensions size must be 1, 2, 3 or 4." What does this mean and what kinds of things might cause this error message? Does it have anything to do with the many constantofShape layers in my model. What generates those? I export to onnx using only:
torch.onnx.export(model.genA2B, random_input, 'model.onnx', verbose=False,
                  input_names=input_names, output_names=output_names,
                  opset_version=9)


title: find index of certain value in 3 dimension array in parallel way

body:
I made a zero_mask2 function (copied below) in the middle of my pytorch file. However, it is too slow. So, I'm looking for a better way.
First, Let me explain the gist of the function below.
Input

all_idx_before' (dimension of [batch_size, number of points, 20]): batch_size is a well-known definition of mini batch in deep learning society. In every batch, there are many points like 1024. And, in every point, a point has 20 values.

idx (dimension of [batch_size, number of points, 5]): batch and number of points are same as before. But, in this case, a point has only 5 values.

p: just an any number, for example, 7.

Toy example
Lets say, a point in all_idx_before has (9, 10, 11, 12, 6, 7, 8, 14, 2, 3, 4, 5, 18, 19, 15, 16, 17, 13, 1, 20).
And a point in idx has (10, 6, 2, 4, 18).
In this setting, I want to find the index of each value in idx. For example, (10) in idx is second element in all_idx_before, (6) in idx is fifth element in all_idx_before...
So, potential output of this point would be like (1, 5, 8, 10, 12), ('second' becomes '1' because of number system in python.).
I do this logic in every batch and every point. So, I used for iteration. But this is too slow. Is there any way of doing this in parallel? I use NumPy format. But I can use tensor if it is better.
def zero_mask2(idx, all_idx_before, p):

size_batch = len(idx)
size_points = len(idx[0])
size_neighbor = len(idx[0][0])

mask = torch.empty(size_batch, size_points, size_neighbor)

for i in range(size_batch):
 for j in range(size_points):
  for l in range(size_neighbor):
   mask[i][j][l] = np.where(all_idx_before[i][j] == idx[i][j][l])[0][0]

mask_t = mask <= p # smaller than p or same : true , bigger than p : false
return mask_t


title: Create a knn adjacent matrix in Pytorch

body:
In Pytorch, say I have a top-k indexing matrix P(B,N,k), a weight matrix W(B,N,N) and a target matrix A(B,N,N), I want to get a adjacent matrix that operates as the following loops:
for i in range(B):
     for ii in range(N):
         for j in range(k):
             if weighted:
                 A[i][ii][P[i][ii][j]] = W[i][ii][P[i][ii][j]]
             else:
                 A[i][ii][P[i][ii][j]] = 1

How to implement it more efficiently and concisely in Pytorch?


title: Can you use pytorch for a non-torch optimization problem?

body:
I would like to use Pytorch to learn the 6 parameters in this small CRF model. I can implement a forward method, loss function and optimizer - but it's unclear how to "tell" pytorch what the parameters of this model are:
import numpy as np
import pydensecrf.densecrf as dcrf
import pydensecrf.utils as utils

class DenseCRF(object):
    def __init__(self, iter_max, pos_w, pos_xy_std, bi_w, bi_xy_std, bi_rgb_std):
        self.iter_max = iter_max
        self.pos_w = pos_w
        self.pos_xy_std = pos_xy_std
        self.bi_w = bi_w
        self.bi_xy_std = bi_xy_std
        self.bi_rgb_std = bi_rgb_std

    def __call__(self, image, probmap):
        C, H, W = probmap.shape

        U = utils.unary_from_softmax(probmap)
        U = np.ascontiguousarray(U)

        image = np.ascontiguousarray(image)

        d = dcrf.DenseCRF2D(W, H, C)
        d.setUnaryEnergy(U)
        d.addPairwiseGaussian(sxy=self.pos_xy_std, compat=self.pos_w)
        d.addPairwiseBilateral(
            sxy=self.bi_xy_std, srgb=self.bi_rgb_std, rgbim=image, compat=self.bi_w
        )

        Q = d.inference(self.iter_max)
        Q = np.array(Q).reshape((C, H, W))

        return Q

Here is my attempt so far:
import torch.nn as nn

class TorchCRF(nn.Module):
    def __init__(self, iter_max, pos_w, pos_xy_std, bi_w, bi_xy_std, bi_rgb_std):
        super(TorchCRF, self).__init__()
        self.iter_max  = iter_max
        self.pos_w  = nn.Parameter(torch.tensor(pos_w, requires_grad=True))
        self.pos_xy_std  = nn.Parameter(torch.tensor(pos_xy_std, requires_grad=True))
        self.bi_w  = nn.Parameter(torch.tensor(bi_w, requires_grad=True))
        self.bi_xy_std  = nn.Parameter(torch.tensor(bi_xy_std, requires_grad=True))
        self.bi_rgb_std  = nn.Parameter(torch.tensor(bi_rgb_std, requires_grad=True))

    def forward(self, image, probmap):
        C, H, W = probmap.shape
        U = utils.unary_from_softmax(probmap)
        U = np.ascontiguousarray(U)
        image = np.ascontiguousarray(image)
        d = dcrf.DenseCRF2D(W, H, C)
        d.setUnaryEnergy(U)
        d.addPairwiseGaussian(sxy=self.pos_xy_std, compat=self.pos_w)
        d.addPairwiseBilateral(sxy=self.bi_xy_std, srgb=self.bi_rgb_std, rgbim=image, compat=self.bi_w )
        Q = d.inference(self.iter_max)
        Q = np.array(Q).reshape((C, H, W))
        return Q


title: Don't include an operation for gradient computation in PyTorch

body:
I have a custom layer. Let the layer be called 'Gaussian'
class Gaussian(nn.Module):
  def __init__():
    super(Gaussian, self).__init__()

 #@torch.no_grad
  def forward(self, x):
    _r = np.random.randint(0, x.shape[0], x.shape[0])
    _sample = x[_r]
    _d = (_sample - x)
    _number = int(self.k * x.shape[0])
    x[1: _number] = x[1: _number] + (self.n * _d[1: _number]).detach()

    return x

The above class will be used as below:
cnn_model = nn.Sequential(nn.Conv2d(1, 32, 5), Gaussian(), nn.ReLU(), nn.Conv2d(32, 32, 5))

If x is the input, I want the gradient of x to exclude operations that are present in the Gaussian module, but include the calculations in other layers of the neural network(nn.Conv2d etc).
In the end, my aim is to use the Gaussian module to perform calculations but that calculations should not be included in gradient computation.
I tried to do the following:

Used the @torch.no_grad above the forward method of the Gaussian

Using detach after every operation in the Gaussian module:
x[1: _number] = x[1: _number] + (self.n * _d[1: _number]).detach() and similarly for other operations

Use y = x.detach() in the forward method. Perform the operations on y and then x.data = y

Are the above methods correct?
P.S: Question edited


title: Perceptron on multi-dimensional tensor

body:
I'm trying to use Perceptron to reduce a tensor of size: [1, 24, 768] to another tensor with size of [1, 1, 768]. The only way I could use was to first reshape the input tensor to [1, 1, 24*768] and then pass it through linear layers. I'm wondering if there's a more elegant way of this transformation --other than using RNNs cause I do not want to use that. Are there other methods generally for the transformation that I want to make? Here is my code for doing the above operation:
lin = nn.Linear(24*768, 768)

# x is in shape of [1, 24, 768]
# out is in shape of [1, 1, 768]
x = x.view(1,1,-1)
out = lin(x)


title: PyTorch matrix factorization with fixed item matrix

body:
I estimate ratings in a user-item matrix by decomposing the matrix into two matrices P and Q using PyTorch Matrix Factorization. I got my loss function L(X-PQ).
Let's say rows of X correspond to users, and x is new user's row, so that new X is X concatenated with x.
Now I want to minimize L(X' - P'Q) = L(X - PQ) + L(x - x_pQ). Since I have already trained P and Q.
I want to train x_p that is the new user's row, but leave Q fixed.
So my question would be, is there a way in PyTorch to train MatrixFactorization model for P with fixed Q?
Code I'm working with:
class MatrixFactorizationWithBiasXavier(nn.Module):
def __init__(self, num_people, num_partners, bias=(-0.01, 0.01), emb_size=100):
    super(MatrixFactorizationWithBiasXavier, self).__init__()
    self.person_emb = nn.Embedding(num_people, emb_size)
    self.person_bias = nn.Embedding(num_people, 1)
    self.partner_emb = nn.Embedding(num_partners, emb_size)
    self.parnter_bias = nn.Embedding(num_partners, 1)
    torch.nn.init.xavier_uniform_(self.person_emb.weight)
    torch.nn.init.xavier_uniform_(self.partner_emb.weight)
    self.person_bias.weight.data.uniform_(bias[0], bias[1])
    self.parnter_bias.weight.data.uniform_(bias[0], bias[1])

def forward(self, u, v):
    u = self.person_emb(u)
    v = self.partner_emb(v)
    bias_u = self.person_bias(u).squeeze()
    bias_v = self.parnter_bias(v).squeeze()
    # calculate dot product
    # u*v is a element wise vector multiplication
    return torch.sigmoid((u*v).sum(1) + bias_u + bias_v)

def test(model, df_test, verbose=False):
model.eval()
# .to(dev) puts code on either gpu or cpu.
people = torch.LongTensor(df_test.id.values).to(dev)
partners = torch.LongTensor(df_test.pid.values).to(dev)
decision = torch.FloatTensor(df_test.decision.values).to(dev)
y_hat = model(people, partners)
loss = F.mse_loss(y_hat, decision)
if verbose:
    print('test loss %.3f ' % loss.item())
return loss.item()

def train(model, df_train, epochs=100, learning_rate=0.01, weight_decay=1e-5, verbose=False):
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=weight_decay)
model.train()

for epoch in range(epochs):
    # From numpy to PyTorch tensors.
    # .to(dev) puts code on either gpu or cpu.
    people = torch.LongTensor(df_train.id.values).to(dev)
    partners = torch.LongTensor(df_train.pid.values).to(dev)
    decision = torch.FloatTensor(df_train.decision.values).to(dev)

    # calls forward method of the model
    y_hat = model(people, partners)
    # Using mean squared errors loss function
    loss = F.mse_loss(y_hat, decision)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

    if verbose and epoch % 100 == 0:
        print(loss.item())


title: How to create dynamic Pytorch Dataloader

body:
I’m doing active learning for my segmentation project. I have an “IMG folder” containing about 5000 images. My task is: First, loading 500 images of that IMG folder for training a model X; Second, using some constraint ( which is defined by is_collected() function) to collect next data ( also in that IMG folder) for the next training phase, and so on…
What should I do to customize the loading Dataset?


title: GPU showing no speed up over CPU

body:
I'm training a neural network with 100*100 hidden nodes, four inputs/one output, and batch size of 32, and I am seeing no speed improvement in using the GPU vs. CPU. I only have a limited data set (1067 samples, copied all to the GPU at the beginning), but I would have thought the 33 batches could have run in parallel, more than making up for the time in copying to the GPU. Is my data set too small, or is there potentially some other issue? Here is my code snippet:
def train_for_regression(X, T):
    BATCH_SIZE = 32
    n_epochs = 1000
    learning_rate = 0.01
    device = torch.device("cuda:0")
    Xt = torch.from_numpy(X).float().to(device) #Training inputs are 4 * 1067 samples
    Tt = torch.from_numpy(T).float().to(device) #Training outputs are 1 * 1067 samples

    nnet = torch.nn.Sequential(torch.nn.Linear(4, 100),
                               torch.nn.Tanh(),
                               torch.nn.Linear(100, 100),
                               torch.nn.Tanh(),
                               torch.nn.Linear(100, 1))
    nnet.to(device)
    mse_f = torch.nn.MSELoss()
    optimizer = torch.optim.Adam(nnet.parameters(), lr=learning_rate)

    for epoch in range(n_epochs):
        for i in range(0, len(Xt), BATCH_SIZE):
            batch_Xt = Xt[i:i+BATCH_SIZE,:]
            batch_Tt = Tt[i:i+BATCH_SIZE,:]
            optimizer.zero_grad()
            Y = nnet(batch_Xt)
            mse = mse_f(Y, batch_Tt)
            mse.backward()
            optimizer.step()
    return nnet


title: Pytorch calculate hit ratio of predictions on batch level

body:
So I would like to calculate my hit ratio at K samples for a recommendation problem I'm dealing with. For those who doesnt know hitratio at K means if the true sample class was in the top K classes predicted from the model. I would like to do this on a batch level.
So If I had the following code
import torch
import torch.nn as nn
import numpy as np

# 5 samples of 5 dimensions
yhat = torch.tensor([[4, 9, 7, 4, 0],
        [8, 1, 3, 1, 0],
        [9, 8, 4, 4, 8],
        [0, 9, 4, 7, 8],
        [8, 8, 0, 1, 4]])

# I get the top 3 classes of each sample recommendation
values, indices = torch.topk(a, 3)
# print(indices)
# tensor([[1, 2, 0],
#        [0, 2, 1],
#        [0, 1, 4],
#        [1, 4, 3],
#        [1, 0, 4]])

# True sample classes
ytrue = torch.tensor([2], #hits on top 3
                    [5],  #Doesnt hit on top 3
                    [4],  #hits on top 3
                    [0],  #Doesnt hit on top 3
                    [1])  #hits on top 3

I would like to calculate the hitratio at top 3 (HR3) here. HR3==1 Means all my predictions contained the true sample class in their top3 while 0 means none did.
For the given example I hit 3 out of 5. Result should be 3/5
What is the most efficient way on pytorch to do this?


title: How reformat pytorch geometric DataLoader node feature output in batch format?

body:
How to reformat pytorch geometric torch_geometric.data.DataLoader batch_data's node features into a (batch_size, num_nodes_per_graph, feature_dim) shape ?
The torch_geometric.data.DataLoader produces each batches with batch_data.x (node features) in the shape of (total_num_nodes, feature_dim). But I want to reformat that in model.forward() function in the shape of (batch_size, num_nodes_per_each_sample_graph, feature_dim). Are there any ways to do this ?
For example, reformatting
[1605, 512] --> [64, 40, 512]
Here total num of nodes are 1605. But each samples does not have exactly 40 nodes, so we have to pad them with zeros. But each node features have 512 dimension.


title: pytorch best way to call unique_consecutive() on certain dimension of a tensor?

body:
What's the best way to call unique_consecutive() on certain dimension of a tensor, and pad the left-out with specified values?
For simplicity, we could use 2D tensor as an example:
input = tensor([[3, 3, 5, 5, 5],
                [3, 3, 2, 2, 3]])

if specifying padding value -1, what i hope to get is:
output = tensor([[3, 5, -1, -1, -1],
                 [3, 2, 3, -1, -1]])

Also, to save memory, i'd prefer to do so in-place if possible.
Thanks!


title: Loss of Conv-neural-network not decreasing, instead obsoleting

body:
I have a convolutional neural network in vgg architecture "style" (down below) to classify images if there is a cat on the picture, or a dog. My training set contains 25000 images cropped to 256px each side. I tried different learning rates, different loss functions and much more but my loss keeps fluctuating between 0.692 and 0.694, but it will not decrease...
normalize = transforms.Normalize(
    mean=[0.485, 0.456, 0.406],
    std=[0.229, 0.224, 0.225]
)

transform = transforms.Compose([
    transforms.Resize(256),
    transforms.CenterCrop(256),
    transforms.ToTensor(),
    normalize
])

# Output = [isDog, isCat]
train_data_list = []
train_data = []
target_list = []
plotlist = []
train_files = listdir("data/catsdogs/train/")

def loadTrainData():
    global train_data_list
    global train_data
    global target_list
    print("Loading data now...")

    amount = len(listdir("data/catsdogs/train/"))
    current = 0

    for i in range(amount):
        r = random.randint(0, len(train_files) - 1)
        file = train_files[r]
        train_files.remove(file)

        img = Image.open("data/catsdogs/train/" + file)
        img_tensor = transform(img)  # (3, 256, 256)

        isCat = 1 if 'cat' in file else 0
        isDog = 1 if 'dog' in file else 0
        target = [isCat, isDog]

        train_data_list.append(img_tensor)
        target_list.append(target)

        if len(train_data_list) >= 64:
            train_data.append((torch.stack(train_data_list), target_list))
            train_data_list = []
            target_list = []

        current = current + 1
        print("Loaded: {:.1f}%".format(current * 100 / amount))
    print("Loaded data successfully!")

class Network(nn.Module):
    def __init__(self):
        super(Network, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=3, stride=1, padding=1)
        self.conv2 = nn.Conv2d(16, 16, kernel_size=3, stride=1, padding=1)

        self.conv3 = nn.Conv2d(16, 32, kernel_size=3, stride=1, padding=1)
        self.conv4 = nn.Conv2d(32, 32, kernel_size=3, stride=1, padding=1)

        self.conv5 = nn.Conv2d(32, 64, kernel_size=3, stride=1, padding=1)
        self.conv6 = nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1)

        self.dropout = nn.Dropout2d()
        self.relu = nn.ReLU(inplace=True)
        self.pool = nn.MaxPool2d(kernel_size=3, stride=3, padding=0, dilation=1, ceil_mode=False)

        self.fc1 = nn.Linear(5184, 1296)
        self.fc2 = nn.Linear(1296, 2)

    def forward(self, x):
        # Block 1
        x = self.conv1(x)
        x = self.relu(x)

        x = self.conv2(x)
        x = self.relu(x)

        x = self.pool(x)

        # Block 2
        x = self.conv3(x)
        x = self.relu(x)

        x = self.conv4(x)
        x = self.relu(x)

        x = self.pool(x)

        # Block 3
        x = self.conv5(x)
        x = self.relu(x)

        x = self.conv6(x)
        x = self.relu(x)

        x = self.pool(x)

        x = x.view(-1, 5184)

        x = F.relu(self.fc1(x))
        x = self.dropout(x)
        x = self.fc2(x)

        return torch.sigmoid(x)

model = Network()
model = model.cuda()

optimizer = optim.SGD(model.parameters(), lr=0.0001, weight_decay=0.0016)

def train(epoch):
    global optimizer

    model.train()
    batch_id = 0
    for data, target in train_data:
        data = data.cuda()
        target = torch.Tensor(target).cuda()

        data = Variable(data)
        target = Variable(target)

        optimizer.zero_grad()

        out = model(data)
        criterion = F.binary_cross_entropy

        loss = criterion(out, target)
        loss.backward()

        optimizer.step()

        plotlist.append(loss)

        print('Train Epoch: {},  {:.0f}% ,\tLoss: {:.6f}'.format(
            epoch, 100. * batch_id / len(train_data), loss.item()
        ))
        batch_id = batch_id + 1

loadTrainData()

for epoch in range(25):
    train(epoch)

plt.plot(plotlist)
plt.show()
plt.ylabel("Loss")
plt.savefig("lossPlot.png")

Here is a plot of my loss over 5 iterations:

Also with a higher learning rate the fluctuation only gets stronger, for 0.1 lr between 0.5 and 0.7.


title: Problem using writer.add_graph() on model class with multiple networks and methods which takes different inputs

body:
I was trying to visualize a model, implemented as a Class, with several networks and methods, and each with different input. When I use writer.add_graph(), it returns NotImplementedError.
My model is for reinforcement learning, and it is something similar to the following pseudocode:

class MyModel():
  init
  self.network1 = torch.nn.DataParallel(some nn.Sequential)
  self.network2 = torch.nn.DataParallel(some nn.Sequential)
  self.network3 = torch.nn.DataParallel(some nn.Sequential)
  (... and more networks)

  def method1(self, x):
    v, r, p, e = self.network1(x)
    return v, r, p, e

  def method2(self, e):
    v, r, p, next_e = self.network2(e, a)
    return v, r, p, next_e

  def method3(self, x, y):
    pa = self.network3(x, y)
    return pa

# FYI: In training, loss = calculate_loss(array_of_vs, array_of_rs, array_of_ps, e, next_e, pa, a)
# but as far as I understand, the train part is independent of the writer.add_graph()

model_to_visualize = MyModel()

writer = SummaryWriter('some_dir')

dummyx = torch.ones(some_dims)
dummyy = torch.ones(some_dims)
dummye = torch.ones(some_dims)
dummya = torch.ones(some_dims)
dummydata = (dummyx, dummyy, dummye, dummya)

writer.add_graph(model_to_visualize, dummydata)
writer.close()

The question I wanted to ask is, is it actually not supported by writer.add_graph() to visualize these kind of structures?
If not, any recommendation on how I can do this?
For your information:

There is no error when I tried writer.add_graph(MyModel().network1, dummyx), but this is not the thing I planned. I wanted to visualize the overall flow of all data.

ray and DataParallel are used in the implementation. Not sure if it is related. I tried writer.add_graph in a separate script and the same error occurred.

Details of the error message:
Error occurs, No graph saved
Traceback (most recent call last):

  File "<ipython-input-35-d95ce6585e68>", line 1, in <module>
    runfile('/home/username/PybulletTest/auxs/testAddGraph/testAddGraph.py', wdir='/home/username/PybulletTest/auxs/testAddGraph')

  File "/usr/lib/python3/dist-packages/spyder_kernels/customize/spydercustomize.py", line 827, in runfile
    execfile(filename, namespace)

  File "/usr/lib/python3/dist-packages/spyder_kernels/customize/spydercustomize.py", line 110, in execfile
    exec(compile(f.read(), filename, 'exec'), namespace)

  File "/home/username/PybulletTest/auxs/testAddGraph/testAddGraph.py", line 196, in <module>
    writer.add_graph(model_to_visualize, dummydata)

  File "/usr/local/lib/python3.8/dist-packages/torch/utils/tensorboard/writer.py", line 724, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose))

  File "/usr/local/lib/python3.8/dist-packages/torch/utils/tensorboard/_pytorch_graph.py", line 292, in graph
    raise e

  File "/usr/local/lib/python3.8/dist-packages/torch/utils/tensorboard/_pytorch_graph.py", line 286, in graph
    trace = torch.jit.trace(model, args)

  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 733, in trace
    return trace_module(

  File "/usr/local/lib/python3.8/dist-packages/torch/jit/_trace.py", line 934, in trace_module
    module._c._create_method_from_trace(

  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 725, in _call_impl
    result = self._slow_forward(*input, **kwargs)

  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 709, in _slow_forward
    result = self.forward(*input, **kwargs)

  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 175, in _forward_unimplemented
    raise NotImplementedError


title: Difference between Xavier weights pytorch implementation

body: What is the difference between nn.init.xavier_uniform and nn.init.xavier_uniform_ when initialising weights?


title: Serialiazation of pytorch state_dict changes after loading into new model instance

body:
Why do the bytes obtained from serializing pytorch state_dicts change after loading the a state_dict into a new instance of the same model architecture?
Have a look:
import binascii
import torch.nn as nn
import pickle

lin1 = nn.Linear(1, 1, bias=False)
lin1s = pickle.dumps(lin1.state_dict())
print("--- original model ---")
print(f"hash of state dict: {hex(binascii.crc32(lin1s))}")
print(f"weight: {lin1.state_dict()['weight'].item()}")

lin2 = nn.Linear(1, 1, bias=False)
lin2.load_state_dict(pickle.loads(lin1s))
lin2s = pickle.dumps(lin2.state_dict())
print("\n--- model from deserialized state dict ---")
print(f"hash of state dict: {hex(binascii.crc32(lin2s))}")
print(f"weight: {lin2.state_dict()['weight'].item()}")

prints
--- original model ---
hash of state dict: 0x4806e6b6
weight: -0.30337071418762207

--- model from deserialized state dict ---
hash of state dict: 0xe2881422
weight: -0.30337071418762207

As you can see, the hashes of the (pickles of the) state_dicts are different whereas the weight is copied over correctly. I would assume that a state_dict from the new model equals the old one in every aspect. Seemingly, it does not, hence the different hashes.


title: values are slightly changed when converting Tensor to Numpy in PyTorch

body:
I ran the code below to convert tensor to numpy but values are slightly changed in spite of the same data type.
import torch
import numpy

a = torch.load("path to saved tensor")
print("tensor data type: ", a.dtype)
print("first 5 values: ", a[:5])
b = a.numpy()
print("numpy data type: ", b.dtype)
print("first 5 values: ", b[:5])

The result is here:
tensor data type:  torch.float32
first 5 values:  tensor([-0.5084, -0.2898,  0.0860,  1.3595,  0.5308])
numpy data type:  float32
first 5 values:  [-0.50837755 -0.28977072  0.08596492  1.3595085   0.5307576 ]

What happened? And how can I solve this?
version:
pytorch 1.8.1, numpy 1.19.2


title: PyTorch: compare three tensors?

body:
I have three boolean mask tensors that I want to create a boolean mask that if the value matches in three tensors then it is 1, else 0.
I tried torch.where(A == B == C, 1, 0), but it doesn't seem to support such.


title: Python(PyTorch): TypeError: string indices must be integers

body:
I have written the following code to train a bert model on my dataset but when I execute it I get an error at the part where I implement tqdm. I have written the entire training code below with full description of the error. How to fix this?
Code
Model
TRANSFORMERS = {
    "bert-multi-cased": (BertModel, BertTokenizer, "bert-base-uncased"),
}

class Transformer(nn.Module):
    def __init__(self, model, num_classes=1):
        """
        Constructor

    Arguments:
        model {string} -- Transformer to build the model on. Expects "camembert-base".
        num_classes {int} -- Number of classes (default: {1})
    """
    super().__init__()
    self.name = model

    model_class, tokenizer_class, pretrained_weights = TRANSFORMERS[model]

    bert_config = BertConfig.from_json_file(MODEL_PATHS[model] + 'bert_config.json')
    bert_config.output_hidden_states = True

    self.transformer = BertModel(bert_config)

    self.nb_features = self.transformer.pooler.dense.out_features

    self.pooler = nn.Sequential(
        nn.Linear(self.nb_features, self.nb_features),
        nn.Tanh(),
    )

    self.logit = nn.Linear(self.nb_features, num_classes)

def forward(self, tokens):
    """
    Usual torch forward function

    Arguments:
        tokens {torch tensor} -- Sentence tokens

    Returns:
        torch tensor -- Class logits
    """
    _, _, hidden_states = self.transformer(
        tokens, attention_mask=(tokens > 0).long()
    )

    hidden_states = hidden_states[-1][:, 0] # Use the representation of the first token of the last layer

    ft = self.pooler(hidden_states)

    return self.logit(ft)

Training
def fit(model, train_dataset, val_dataset, epochs=1, batch_size=8, warmup_prop=0, lr=5e-4):

    train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
    val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)

    optimizer = AdamW(model.parameters(), lr=lr)

    num_warmup_steps = int(warmup_prop * epochs * len(train_loader))
    num_training_steps = epochs * len(train_loader)

    scheduler = get_linear_schedule_with_warmup(optimizer, num_warmup_steps, num_training_steps)

    loss_fct = nn.BCEWithLogitsLoss(reduction='mean').cuda()

    for epoch in range(epochs):
        model.train()
        start_time = time.time()

        optimizer.zero_grad()
        avg_loss = 0

        for step, (x, y_batch) in tqdm(enumerate(train_loader), total=len(train_loader)):

            y_pred = model(x.to(device))

            loss = loss_fct(y_pred.view(-1).float(), y_batch.float().to(device))
            loss.backward()
            avg_loss += loss.item() / len(train_loader)

            xm.optimizer_step(optimizer, barrier=True)
            #optimizer.step()
            scheduler.step()
            model.zero_grad()
            optimizer.zero_grad()

        model.eval()
        preds = []
        truths = []
        avg_val_loss = 0.

        with torch.no_grad():
            for x, y_batch in tqdm(val_loader):
                y_pred = model(x.to(device))
                loss = loss_fct(y_pred.detach().view(-1).float(), y_batch.float().to(device))
                avg_val_loss += loss.item() / len(val_loader)

                probs = torch.sigmoid(y_pred).detach().cpu().numpy()
                preds += list(probs.flatten())
                truths += list(y_batch.numpy().flatten())
            score = roc_auc_score(truths, preds)

        dt = time.time() - start_time
        lr = scheduler.get_last_lr()[0]
        print(f'Epoch {epoch + 1}/{epochs} \t lr={lr:.1e} \t t={dt:.0f}s \t loss={avg_loss:.4f} \t val_loss={avg_val_loss:.4f} \t val_auc={score:.4f}')

Error
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<timed eval> in <module>

<ipython-input-19-e47eae808597> in fit(model, train_dataset, val_dataset, epochs, batch_size, warmup_prop, lr)
     22         for step, (x, y_batch) in tqdm(enumerate(train_loader), total=len(train_loader)):
     23
---> 24             y_pred = model(x.to(device))
     25
     26             loss = loss_fct(y_pred.view(-1).float(), y_batch.float().to(device))

/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    724             result = self._slow_forward(*input, **kwargs)
    725         else:
--> 726             result = self.forward(*input, **kwargs)
    727         for hook in itertools.chain(
    728                 _global_forward_hooks.values(),

<ipython-input-11-2002cc7ec843> in forward(self, tokens)
     41         )
     42
---> 43         hidden_states = hidden_states[-1][:, 0] # Use the representation of the first token of the last layer
     44
     45         ft = self.pooler(hidden_states)

TypeError: string indices must be integers


title: Is there any problem if install different version of cuda toolkit and pytorch?

body:
If my cuda version is 11.2 and  if install Pytorch cuda version 10.2 and after if i install NVIDIA cuda toolkit 11.3 .
Is this will cause any problem ?


title: What and where am I going wrong in this code for pytorch based object detection?

body:
I am using Yolov5 for this project
Here is my code
import numpy as np
import cv2
import torch
import torch.backends.cudnn as cudnn
from models.experimental import attempt_load
from utils.general import non_max_suppression

weights = '/Users/nidhi/Desktop/yolov5/best.pt'
device = torch.device('cpu')

model = attempt_load(weights, map_location=device)  # load FP32 model
stride = int(model.stride.max())  # model stride
cudnn.benchmark = True

# Capture with opencv and detect object
cap = cv2.VideoCapture('Pothole testing.mp4')
width, height = (352, 352) # quality
cap.set(3, width) # width
cap.set(4, height) # height

while(cap.isOpened()):
    time.sleep(0.2) # wait for 0.2 second
    ret, frame = cap.read()
    if ret ==True:
        now = time.time()
        img = torch.from_numpy(frame).float().to(device).permute(2, 0, 1)
        img /= 255.0  # 0 - 255 to 0.0 - 1.0

        if img.ndimension() == 3:
            img = img.unsqueeze(0)

        pred = model(img, augment=False)[0]
        pred = non_max_suppression(pred, 0.39, 0.45, classes=0, agnostic=True) # img, conf, iou, classes, ...
        print('time -> ', time.time()-now)
     else:
         break

cap.release()

The error I am getting:
  File "run.py", line 38, in <module>
    pred = model(img, augment=False)[0]
  File "/Users/nidhi/Library/Python/3.8/lib/python/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/nidhi/Desktop/yolov5/models/yolo.py", line 118, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "/Users/nidhi/Desktop/yolov5/models/yolo.py", line 134, in forward_once
    x = m(x)  # run
  File "/Users/nidhi/Library/Python/3.8/lib/python/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/Users/nidhi/Desktop/yolov5/models/common.py", line 152, in forward
    return torch.cat(x, self.d)
RuntimeError: Sizes of tensors must match except in dimension 1. Got 108 and 107 in dimension 3 (The offending index is 1)

Operating system: macOS Big Sur 11.2.3
Python version: 3.8.2
The model is used best.pt which I had trained on Google Colab, I used yolov5l model to train the dataset.


title: How to make prediction on pytorch emotion detection model

body:
I made a CNN model for emotion recognition on 5 emotions. I wanted to test it on an single image to get the individual class predictions for each emotion.
Evaluating the model works, but I can't seem to find how to make a prediction with a single image. How can I do that?
The Model
def conv_block(in_channels, out_channels, pool=False):
layers = [nn.Conv2d(in_channels, out_channels, kernel_size=3, padding=1),
          nn.BatchNorm2d(out_channels),
          nn.ELU(inplace=True)]
if pool: layers.append(nn.MaxPool2d(2))
return nn.Sequential(*layers)

class ResNet(ImageClassificationBase):
    def __init__(self, in_channels, num_classes):
        super().__init__()

        self.conv1 = conv_block(in_channels, 128)
        self.conv2 = conv_block(128, 128, pool=True)
        self.res1 = nn.Sequential(conv_block(128, 128), conv_block(128, 128))
        self.drop1 = nn.Dropout(0.5)

        self.conv3 = conv_block(128, 256)
        self.conv4 = conv_block(256, 256, pool=True)
        self.res2 = nn.Sequential(conv_block(256, 256), conv_block(256, 256))
        self.drop2 = nn.Dropout(0.5)

        self.conv5 = conv_block(256, 512)
        self.conv6 = conv_block(512, 512, pool=True)
        self.res3 = nn.Sequential(conv_block(512, 512), conv_block(512, 512))
        self.drop3 = nn.Dropout(0.5)

        self.classifier = nn.Sequential(nn.MaxPool2d(6),
                                        nn.Flatten(),
                                        nn.Linear(512, num_classes))

    def forward(self, xb):
        out = self.conv1(xb)
        out = self.conv2(out)
        out = self.res1(out) + out
        out = self.drop1(out)

        out = self.conv3(out)
        out = self.conv4(out)
        out = self.res2(out) + out
        out = self.drop2(out)

        out = self.conv5(out)
        out = self.conv6(out)
        out = self.res3(out) + out
        out = self.drop3(out)

        out = self.classifier(out)
        return out

The fit_one_cycle function is called to train the model
@torch.no_grad()
def evaluate(model, val_loader):
model.eval()
outputs = [model.validation_step(batch) for batch in val_loader]
return model.validation_epoch_end(outputs)

def get_lr(optimizer):
    for param_group in optimizer.param_groups:
       return param_group['lr']

def fit_one_cycle(epochs, max_lr, model, train_loader, val_loader,
              weight_decay=0, grad_clip=None, opt_func=torch.optim.SGD):
    torch.cuda.empty_cache()
    history = []

# Set up custom optimizer with weight decay
optimizer = opt_func(model.parameters(), max_lr, weight_decay=weight_decay)
# Set up one-cycle learning rate scheduler
sched = torch.optim.lr_scheduler.OneCycleLR(optimizer, max_lr, epochs=epochs,
                                            steps_per_epoch=len(train_loader))

    for epoch in range(epochs):
        # Training Phase
        model.train()
        train_losses = []
        lrs = []
        for batch in train_loader:
            loss = model.training_step(batch)
            train_losses.append(loss)
            loss.backward()

            # Gradient clipping
            if grad_clip:
                nn.utils.clip_grad_value_(model.parameters(), grad_clip)

            optimizer.step()
            optimizer.zero_grad()

            # Record & update learning rate
            lrs.append(get_lr(optimizer))
            sched.step()

        # Validation phase
        result = evaluate(model, val_loader)
        result['train_loss'] = torch.stack(train_losses).mean().item()
        result['lrs'] = lrs
        model.epoch_end(epoch, result)
        history.append(result)
    return history

This returns the accuracy and loss, i want to change this so it returns prediction percentages for each class.
    def accuracy(outputs, labels):
    _, preds = torch.max(outputs, dim=1)
    return torch.tensor(torch.sum(preds == labels).item() / len(preds))

class ImageClassificationBase(nn.Module):
    def training_step(self, batch):
        images, labels = batch
        out = self(images)
        loss = F.cross_entropy(out, labels)
        return loss

    def validation_step(self, batch):
        images, labels = batch
        out = self(images)
        loss = F.cross_entropy(out, labels)
        acc = accuracy(out, labels)
        return {'val_loss': loss, 'val_acc': acc}

    def validation_epoch_end(self, outputs):
        batch_losses = [x['val_loss'] for x in outputs]
        epoch_loss = torch.stack(batch_losses).mean()
        batch_accs = [x['val_acc'] for x in outputs]
        epoch_acc = torch.stack(batch_accs).mean()
        return {'val_loss': epoch_loss.item(), 'val_acc': epoch_acc.item()}

    def epoch_end(self, epoch, result):
        print("Epoch [{}], last_lr: {:.5f}, train_loss: {:.4f}, val_loss: {:.4f}, val_acc: {:.4f}".format(
            epoch, result['lrs'][-1], result['train_loss'], result['val_loss'], result['val_acc']))


title: installed pytorch1.0.1 for OS X with pip3 but cannot import, what can I do?

body:
I have already installed pytorch for MacOS 10.14 with pip3, but I can not import it in the python script. What should I do?

System: MacOS 10.14

Python3: v3.7

➜  ~ pip3 list
Package     Version
----------- -----------
numpy       1.16.2
Pillow      5.4.1
pip         18.1
pycairo     1.17.1
pygobject   3.28.3
setuptools  40.5.0
six         1.12.0
torch       1.0.1.post2
torchvision 0.2.2.post3
virtualenv  16.1.0
wheel       0.32.2
➜  ~ python3
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 26 2018, 23:26:24)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>>


title: Unexpected import behavior: non-existing module imported

body:
I have a torch extension (but I think the error I get is independent of torch), let's call it foo, that is built with setuptools. However, it depends on whether there is a GPU available in the system, in which case some CUDA code is compiled:

from torch.utils.cpp_extension import CppExtension, BuildExtension, CUDAExtension

if cuda_available:
    ext = CUDAExtension(
            'foo_gpu',
            prefixed(prefix, ['foo.cpp', 'foo_kernel.cu']),
            define_macros=[('COMPILE_CUDA', '1')])
else:
    ext = CppExtension(
            'foo_cpu',
            prefixed(prefix, ['foo.cpp']))

setup(name=ext.name,
      version='1.0.0',
      ext_modules=[ext_module],
      extra_compile_args=['-mmacosx-version-min=10.9'],
      cmdclass={'build_ext': BuildExtension})

Notice how the module is compiled to be either foo_gpu or foo_cpu.

Later, I import it as follows:

try:
    import foo_gpu as foo_backend
    CUDA_SUPPORTED = True
except ImportError:
    CUDA_SUPPORTED = False
    # Try importing the cpu version
    try:
        import foo_cpu as foo_backend
    except ImportError:
        ...

Let's say I compile it with CUDA support, so foo_gpu goes into PIP (actually goes into PIP as foo-gpu, not sure why?)

Now, I uninstalled it, pip uninstall foo-gpu, and compile it with CPU support only, and pip shows foo-cpu 1.0.0.

BUT, now, when I run the import code above, it still finds foo-gpu, i.e., the first import statement succeeds! Even though it does not show up in pip.

EDIT. I checked sys.path and found there is one folder that contains something with gpu in my conda env:

$ ls ~/miniconda3/envs/cpu_env/lib/python3.7/site-packages/foo_cpu-1.0.0-py3.7-linux-x86_64.egg/
 EGG-INFO
 __pycache__
 foo_cpu.cpython-37m-x86_64-linux-gnu.so
 foo_cpu.py
 foo_gpu.cpython-37m-x86_64-linux-gnu.so

But how should it get there? In this env (cpu_env), I never compiled with GPU support. Are there some caches that get invoked?


title: How to link labels to pictures and feed them into a training function in Pytorch?

body:
I'm trying to build a classifier that classifies sign language gesture images. have folder with sub folders as follows
1) Train
2) Validation
3) Test

Each sub folder has folders named A through I. Those folder names are the labels to pictures within those folders.

I'm not really sure how can link those labels to pictures and feed them into the model.

Most of the Pytorch documentation online uses built in datasets and for someone new to this area, it is difficult to understand what is happening behind the scene. I have read about dataloaders, datasets, ImageFolder. I just started deep learning recently and splitting data was easy in machine learning using built in functions and pandas columns, not as much with pictures..

All I was able to do was those two functions but not sure how to proceed from there:

The first function is to load training pictures

    data_path = 'C:/Users/.../Train/'
    train_dataset = torchvision.datasets.ImageFolder(
        root=data_path,
        transform=torchvision.transforms.ToTensor()
    )
    train_loader = torch.utils.data.DataLoader(
        train_dataset,
        batch_size=10,
        num_workers=0,
        shuffle=True
    )
    return train_loader

The second function is to extract the labels from the pictures names within one specific folder, filenames have the convention of uselessinfo_letter_uselessnumbers

import os
data_path = r"C:\Users...\small_sample"
labels = []
for img_filename in os.listdir(data_path):
    if img_filename.endswith(".jpg"):
        useless1, letter, useless2 = img_filename.split('.')[0].split('_')
        labels.append(letter)

I'm looking to create a train_loader and I guess that train_loader would have both the tensor format of pictures with their labels.

to feed into

for i, data in enumerate(train_loader, 0):
            # Get the inputs
            inputs, labels = data
            labels = normalize_label(labels) # Convert labels to 0/1
            # Zero the parameter gradients
            optimizer.zero_grad()
            # Forward pass, backward pass, and optimize
            outputs = net(inputs)
            loss = criterion(outputs, labels.float())
            loss.backward()
            optimizer.step()


title: Listing first and second values in a tensor

body:
I know this might be a simple problem for some of you. I am not able to get the output as expected.
The input tensor is given below.

X = tensor([[50.7500, 44.0000],
        [47.0000, 47.0000],
        [42.5000, 52.2500],
        [59.6163, 50.7097],
        [54.6682, 54.6682],
        [48.7304, 61.5956],
        [71.3156, 59.5631],
        [64.7864, 64.7864],
        [56.9515, 73.9272]])

I want two tensors A and B such that

A = tensor([[50.7500],
        [47.0000],
        [42.5000],
        [59.6163],
        [54.6682],
        [48.7304],
        [71.3156],
        [64.7864],
        [56.9515]]) and
B = tensor([[44.0000],
        [47.0000],
        [52.2500],
        [50.7097],
        [54.6682],
        [61.5956],
        [59.5631],
        [64.7864],
        [73.9272]])

This is what I tried till now.

idx = torch.LongTensor([0])
idx1 = torch.LongTensor([1])
A = X.index_select(1,idx)
B = X.index_select(1,idx1)

which is not giving me output as expected. Thanks in advance.


title: Training an autoencoder across different machines

body:
I am attempting to train an autoencoder to support wireless data transfer. The encoder portion will live on the transmitter side of the transceiver and the decoder will live on the receiver side. In general, the transmitter and receiver could be separated by miles and will exist on different computers.

The autoencoder must be trained using the real physical channel, therefore it will be necessary to perform backpropagation across two different computers (the transmitter and receiver computers). My question is, how can I start the process of back propagation on the receiver side, and finish it on the transmitter side?

To make this problem a little simpler, if you can help me perform backprop across two different files, that might be enough for me to extend it as needed. Imagine the encoder is defined by one file and the decoder is defined by another. How would I perform backprop across these two separate files?

I'm willing to use either pytorch or tensorflow, whichever is better suited to solve the problem. Pytorch would be my first choice if possible.

Here is pytorch code for a standard autoencoder that lives in one file and acts on CIFAR data. You can see how backpropagation is performed in one line loss.backward(). This is what won't work when the autoencoder is split between machines.

import torch
import torchvision as tv
import torchvision.transforms as transforms
import torch.nn as nn
from torch.autograd import Variable

# Loading and Transforming data
transform = transforms.Compose([transforms.ToTensor(),  transforms.Normalize((0.4914, 0.4822, 0.4466), (0.247,            0.243, 0.261))])
trainTransform  = tv.transforms.Compose([tv.transforms.ToTensor(), tv.transforms.Normalize((0.4914, 0.4822, 0.4466), (0.247, 0.243, 0.261))])
trainset = tv.datasets.CIFAR10(root='./data',  train=True,download=True, transform=transform)
dataloader = torch.utils.data.DataLoader(trainset, batch_size=32, shuffle=False, num_workers=4)
testset = tv.datasets.CIFAR10(root='./data', train=False, download=True, transform=transform)
classes = ('plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
testloader = torch.utils.data.DataLoader(testset, batch_size=4, shuffle=False, num_workers=2)

# Writing our model
class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder,self).__init__()

        self.encoder = nn.Sequential(
            nn.Conv2d(3, 6, kernel_size=5),
            nn.ReLU(True),
            nn.Conv2d(6,16,kernel_size=5),
            nn.ReLU(True))
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(16,6,kernel_size=5),
            nn.ReLU(True),
            nn.ConvTranspose2d(6,3,kernel_size=5),
            nn.ReLU(True),
            nn.Sigmoid())
    def forward(self,x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

#defining some params
num_epochs = 5 #you can go for more epochs, I am using a mac
batch_size = 128

model = Autoencoder().cpu()
distance = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(),weight_decay=1e-5)

for epoch in range(num_epochs):
    for data in dataloader:
        img, _ = data
        img = Variable(img).cpu()
        # ===================forward=====================
        output = model(img)
        loss = distance(output, img)
        # ===================backward====================
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
    # ===================log========================
    print('epoch [{}/{}], loss:{:.4f}'.format(epoch+1, num_epochs, loss.data.numpy()))


title: Pass Python object to C and back again

body:
I'm trying to get to grips with using embedded Python from a C++ application. Specifically I want my C++ to launch some PyTorch code.

I am making some initialization function in Python to perform the device (CPU or GPU) discovery and would like to pass this back to the C++ code. The C++ will call another Python function for inference which is when the C++ will pass the device to Python.

        pFunc_init = PyObject_GetAttrString(pModule, "torch_init");
        if (pFunc_init && PyCallable_Check(pFunc_init)) {
            pValue_device = PyObject_CallObject(pFunc_init, pArgs);

            if (pArgs != NULL)
                Py_DECREF(pArgs);

            if (pValue_device != NULL) {
                pFunc_infer = PyObject_GetAttrString(pModule, "torch_infer");
                if (pFunc_infer && PyCallable_Check(pFunc_infer)) {
                    //
                    // TODO put object pValue_device into pArgs_device.
                    //
                    pValue_infer = PyObject_CallObject(pFunc_infer, pArgs_device);
                    if (pValue_infer != NULL) {
                        printf("Result pValue_infer: %ld\n", PyLong_AsLong(pValue_infer));
                        Py_DECREF(pValue_infer);
                    }
                }
                Py_DECREF(pValue_device);
            }
            else {
                Py_DECREF(pFunc_init);
                Py_DECREF(pModule);
                PyErr_Print();
                fprintf(stderr, "Call failed\n");
                return 1;
            }
        }

The TODO marks where I would like to put this code. With simple Python objects I think I know what I need but how to deal with this custom Python object?


title: How to train only front part of a neural network?

body:
I'm using pytorch to train part of the network. For example, I have a model structure

hidden1 = Layer1(x)
hidden2 = Layer2(hidden1)
out = Layer3(hidden2)

If I want to train Layer3 only, I can use

hidden1 = Layer1(x)
hidden2 = Layer2(hidden1).detach()
out = Layer3(hidden2)

However, this time I want to train Layer1 only. How can I achieve this? Thanks.


title: Fluctuations in neural netwok inference time in PyTorch

body:
I have a neural network and I would like to test its inference time by running it on a CPU. It's important that the time would be measured as accurately as possible.

I use the following code to get the inference time of a network:

with torch.no_grad():
    for image, _ in testloader:
        start = time.time()
        net(image)
        total_time = time.time() - start

The problem is, that even if I pass the same image through the network (although in this case "image" is really a batch of images) and do that many times over, the inference times will fluctuate with a difference between a minimum and maximum times reaching 30ms or even higher.

Also, before I ran the tests, I had made sure that the OS would interfere as less as possible with the process - closed all applications.

I want to understand where this fluctuations come from exactly. It is in my understanding, that if the same image is passed through the network, it should take the same time to classify it no matter how many tests I make (at least theoretically speaking).

So what causes these fluctuations? Is there a way to eliminate them?


title: Finding indices with zeros from 4D Tensors in Pytorch

body:
I have an interesting question for you, if you are working with Pytorch.

I have a batch of images with shape (3, 224, 224). So, if my batch size is 64, let’s say, the final tensor I have has the shape of (64, 3, 224, 224).

Now, here is the question. Suppose that some of the images in this batch is filled with only zeros. What is the fastest way to find out which batch indices are with only zeros?

I don’t want to create for loop for that, since it is slow.

Thanks for your answer.


title: how to prevent my DNN / MLP converging to average

body:
I want to use available several features to predict a variable. It does not seem to be related to vision or NLP. Although I believe there are good reasons that the variable to be predicted is a non linear function of these features. So I just use normal MLP like following:

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.fc1 = nn.Linear(53, 200)
        self.fc2 = nn.Linear(200, 100)
        self.fc3 = nn.Linear(100, 36)
        self.fc4 = nn.Linear(36, 1)

    def forward(self, x):
        x = F.leaky_relu(self.fc1(x))
        x = F.leaky_relu(self.fc2(x))
        x = F.leaky_relu(self.fc3(x))
        x = self.fc4(x)
        return x
net = Net().to(device)
loss_function = nn.MSELoss()
optimizer = optim.Adam(net.parameters(), lr=0.001, weight_decay= 1e-6)

def train_normal(model, device, train_loader, optimizer, epoch):
    model.train ()
    for batch_idx, (data, target) in enumerate (train_loader):
        data = data.to (device)
        target = target.to (device)
        optimizer.zero_grad ()
        output = model (data)
        loss = loss_function (output, target)
        loss.backward ()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 100)
        optimizer.step ()
        if batch_idx % 100 == 0:
            print ('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format (
                epoch, batch_idx * len (data), len (train_loader.dataset),
                       100. * batch_idx / len (train_loader), loss.item ()))

At first it seems to work and did learn something:

Train Epoch: 9 [268800/276316 (97%)] Loss: 0.217219
Train Epoch: 9 [275200/276316 (100%)] Loss: 0.234965
predicted actual diff
-1.18 -1.11 -0.08
0.15 -0.15 0.31
0.19 0.27 -0.08
-0.49 -0.48 -0.01
-0.05 0.08 -0.14
0.44 0.50 -0.06
-0.17 -0.05 -0.12
1.81 1.92 -0.12
1.55 0.76 0.79
-0.05 -0.30 0.26

But when it kept learning, I saw the results seemingly to be close to each other's  average regardless the different input:

predicted actual diff
-0.16   -0.06   -0.10
-0.16   -0.55   0.39
-0.13   -0.26   0.14
-0.15   0.50    -0.66
-0.16   0.02    -0.18
-0.16   -0.12   -0.04
-0.16   -0.40   0.24
-0.01   1.20    -1.21
-0.07   0.33    -0.40
-0.09   0.02    -0.10

What technology / trick can prevent it? Also, how to increase the accuracy, shall I add more hidden layers or add more neurons of each layer?


title: How can I solve the pytorch mismatch error?

body:
I've got pytorch mismatch error in following code:

# Visualization
import matplotlib.pyplot as plt

digit_size = 28
z_sample = np.random.rand(1, latent_dim) # random
z_sample.dtype = 'float32'
z_sample = torch.tensor(z_sample)
plt.figure(figsize=(20, 1))

for i in range(10):
    c = np.zeros((1, cond_dim), dtype='float32')
    c[i] = 1
    c = torch.tensor(c)
    cvae.eval()
    with torch.no_grad():
        img = cvae.decoder(z_sample, c)
        # reshape (if needed)

    plt.subplot(1, i+1)
    plt.axis('off')
    plt.imshow(img, cmap='Greys_r',)
plt.show()
##### test #####
print(np.size(z_sample))
print(latent_dim)

I defined latent_dim equals to 2.
I know where the error is coming from.
it's come from these lines:

z_sample = np.random.rand(1, latent_dim) # random
z_sample.dtype = 'float32'
z_sample = torch.tensor(z_sample)

in first line, I defined a 1*2 random vector
but after I changed it from 'array' to 'tensor' it gives me 1*4 vector so I got this mismatch error.

anyone know how can I fix it?
it also still should be 'float32'.

EDIT:
the CVAE is:

class CVAE(nn.Module):
    def __init__(self, x_dim, h_dim1, h_dim2, z_dim, c_dim):
        super(CVAE, self).__init__()

        # encoder part
        self.fc1 = nn.Linear(x_dim + c_dim, h_dim1)
        self.fc2 = nn.Linear(h_dim1, h_dim2)
        self.fc31 = nn.Linear(h_dim2, z_dim)
        self.fc32 = nn.Linear(h_dim2, z_dim)
        # decoder part
        self.fc4 = nn.Linear(z_dim + c_dim, h_dim2)
        self.fc5 = nn.Linear(h_dim2, h_dim1)
        self.fc6 = nn.Linear(h_dim1, x_dim)

    def encoder(self, x, c):
        concat_input = torch.cat((x, c), 1)
        h = F.relu(self.fc1(concat_input))
        h = F.relu(self.fc2(h))
        return self.fc31(h), self.fc32(h)

    def sampling(self, mu, log_var):
        std = torch.exp(0.5*log_var)
        eps = torch.randn_like(std)
        return eps.mul(std).add(mu) # return z sample

    def decoder(self, z, c):
        concat_input = torch.cat((z, c), 1)
        h = F.relu(self.fc4(concat_input))
        h = F.relu(self.fc5(h))
        return F.sigmoid(self.fc6(h))

    def forward(self, x, c):
        mu, log_var = self.encoder(x.view(-1, 784), c)
        z = self.sampling(mu, log_var)
        return self.decoder(z, c), mu, log_var

the model is:

# Create Model (change None)
cond_dim = train_loader.dataset.train_labels.unique().size(0)
latent_dim = 2
cvae = CVAE(x_dim=28*28, h_dim1=512, h_dim2=256, z_dim=latent_dim, c_dim=cond_dim)

# Device setting
cvae = cvae.to(device)

and the error is:

RuntimeError: size mismatch, m1: [1 x 14], m2: [12 x 256] at /pytorch/aten/src/TH/generic/THTensorMath.cpp:197


title: Can someone explain this pytorch neural network code ? Are there two different neural networks here or one?

body:
class doubleNetwork(nn.Module):

def __init__(self, input_dim, output_dim):
    super(doubleNetwork, self).__init__()
    self.policy1 = nn.Linear(input_dim, 256)
    self.policy2 = nn.Linear(256, output_dim)

    self.value1 = nn.Linear(input_dim, 256)
    self.value2 = nn.Linear(256, 1)

def forward(self, state):
    logits = F.relu(self.policy1(state))
    logits = self.policy2(logits)

    value = F.relu(self.value1(state))
    value = self.value2(value)

    return logits, value

Are policy1, value1 in different networks or the same?
Are there two different neural networks here or single?
What is happening in the code here?


title: Jupyter notebook crashing server

body:
I just bought a GPU to put in a computer where I'm hosting a jupyter notebook. I tunnel the output of the jupyter notebook from the tower to my laptop via ssh. I'm running some code and the jupyter notebook freezes at the same line every time. Not only does the jupyter notebook freeze, but everything on the tower. If I have any other ssh connections, they freeze. If I use the GUI on the tower directly it's frozen. Nothing is responsive until I hit the power button to reset the computer.

The odd thing is, that while nothing is responding, nothing is timing out either. the ssh sessions keep their connections. the jupyter notebook homepage claims it's still connected. It's very odd, and I'm not sure this is a problem with the code or the tower somehow, so I'm not sure if I should post this here or somewhere else. But here's the code

def show_img(x):
    x = x.clone().detach().permute(1,2,0).numpy()
    print(x.shape)
    x = rio.convert_tensor_to_rgb(x)
    print(x.shape)
    plt.figure(figsize=(8, 8))
    plt.axis('off')
    _ = plt.imshow(x)

# define generator discriminator, dataloader, other stuff....

G.cuda()
D.cuda()
g_optim = optim.RMSprop(G.parameters(), lr=lr)
d_optim = optim.RMSprop(D.parameters(), lr=lr)
g_losses = []
d_losses = []
i_losses = []
for epoch in range(n_epochs):
    dataloader = DataLoader(train_dataset, batch_size=batch_size,
                            shuffle=True, pin_memory=True,
                            num_workers=num_workers)
    g_loss = 0.0 # g_loss is the generator's loss
    d_loss = 0.0 # d_loss is the discriminator's loss
    i_loss = 0.0 # i_loss is the generator's loss for not being invertable
    i_weight = 10 # prioritize being invertible ten times more than minimizing g_loss
    x,y,z = train_dataset[0]
    print("image")
    show_img(x)
    print("target")
    show_img(y)
    print("generated")
    x,y = x.cuda(), y.cuda()
    g = G(x.unsqueeze(0),y.unsqueeze(0))
    print(g.shape)
    show_img(g.squeeze().cpu())
    loop = tqdm(total=len(dataloader), position=0, file=sys.stdout)
    print("just to be sure") #prints this
    for minibatch, (image, batchImage, exp_batch) in enumerate(dataloader): #this is the line it freezes on?
        print("image ", image.shape, " batchImage ", batchImage.shape, " experiment batch ", exp_batch) # doesn't print this. already frozen

EDIT: GUI and ssh are responsive, but unusually slow. I guess the main problem is that the code is still freezing on said line of code, and I don't know why.


title: Why is my CNN program not improving in accuracy after I add a dropout?

body:
I am using a CNN to identify images in the CiFar10 data set and before adding a dropout, the cnn was hitting 58% accuracy, but after adding it, it went down to 52%. Is it the case that the network isn't overfitting? cause I doubt that's the case. After adding two more dropouts, the accuracy went up to 55%, but I'm still confused as to why it dipped in the first place. Here's my code:

class Net(nn.Module):
def __init__(self):
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(3, 12, 3, padding=1)
    self.pool = nn.MaxPool2d(2, 2)
    self.conv2 = nn.Conv2d(12, 24, 3, padding=1)
    self.conv3 = nn.Conv2d(24, 48, 3, padding=1)
    self.conv4 = nn.Conv2d(48, 48, 3, padding=1)
    self.dropout1 = nn.Dropout(p=0.2)
    self.dropout2 = nn.Dropout(p=0.2)
    self.dropout3 = nn.Dropout(p=0.3)
    self.fc1 = nn.Linear(48 * 2 * 2, 120)
    self.fc2 = nn.Linear(120, 84)
    self.fc3 = nn.Linear(84, 10)

def forward(self, x):
    x = self.pool(self.dropout1(F.relu(self.conv1(x))))
    x = self.pool(self.dropout2(F.relu(self.conv2(x))))
    x = self.pool(self.dropout3(F.relu(self.conv3(x))))
    x = self.pool(F.relu(self.conv4(x)))
    x = x.view(-1, 48 * 2 * 2)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    return x


title: Understanding the parameters of a simple neural network in Pytorch

body:
I have defined a simple NN as follows in Pytorch:

my_model = nn.Sequential(
            nn.Linear(1, 5),
            nn.Tanh(),
            nn.Linear(5, 1))

I then iterate through the parameters and check their sizes:

[parameter.shape for parameter in my_model.parameters()]

I get:

[torch.Size([5, 1]), torch.Size([5]), torch.Size([1, 5]), torch.Size([1])]

I'm confused as to why the last Size is 1. Shouldn't there be 5 bias values as well, going out from the hidden layer into the output layer?


title: Python Tkinter: Parsing dropdown menu value into a function as a variable?

body:
I am trying to create a Tkinter dropdown menu to add into the main GUI. The purpose of the menu is to give the user several options to choose the image segmentation neural network (currently there are only two options).

I am having trouble getting the value of the dropdown menu and parsing it through the function as a parameter (variable, not a string).

I have tried creating a staticmethod within the GUI window class to search for the value of the dropdown menu and parse the value through the image segmentation function when the segment button in the GUI is pressed. The segmentation function takes two parameters, net and path, where it is neural network and image file pathway respectively.

I am not too sure how to change the individual parameters of a variable so I just changed the whole variable depending on the value of the dropdown menu. The only thing being changed is the net parameter of the segment function bind to the Tkinter button.

The code:

@staticmethod
    def find_option():
        if menu1.get() == "fcn":
            App.btn2 = Button(App.btn_frame, text="Segment", width = 10, height = 1, cursor = "hand2", command=lambda: App.segment(net=App.fcn, path=App.path))
        else:
            App.btn2 = btn2 = Button(App.btn_frame, text="Segment", width = 10, height = 1, cursor = "hand2", command=lambda: App.segment(net=App.dlab, path=App.path))

The dropdown menu code:

fcn = models.segmentation.fcn_resnet101(pretrained=True).eval()
dlab = models.segmentation.deeplabv3_resnet101(pretrained=1).eval()
options = ["fcn", "dlab"]

variable = StringVar(btn_frame)
        variable.set(options[0]) # default value

        menu1 = OptionMenu(btn_frame, variable, *options)
        menu1.pack(side=LEFT)

btn_frame is the frame within the main window containing the buttons.

App is the main GUI class. menu1 needed to be referenced as App.menu1 but it would, for some reason, cause several errors (other class variables would become undefined and it would say App object has no attribute menu). I also tried referencing menu1 as App().menu1 but would initiate a new window each time I pressed the segment button, also it never actually showed the segmented image.


title: Implementation Issue: Deep ConvNet for Pattern Recognition

body:
I'm trying to implement a pattern recognition model using a fully convolutional network (fig 1 in https://www.sciencedirect.com/science/article/pii/S0031320318304370, I was able to get the full text without signing in or anything but if it's a problem I can attach a picture too!) but I'm getting a size error when moving from the final Conv2D layer to the first fc_layer.

Here is my error message:

RuntimeError: size mismatch, m1: [4 x 1024], m2: [4 x 1024] at /pytorch/aten/src/THC/generic/THCTensorMathBlas.cu:283

Originally, as in the figure, my first linear layer was:

nn.Linear(4*4*512, 1024)

but after getting the size mismatch, I changed it to:

nn.Linear(4,1024)

Now, I have a strange error message as written above.

For reference (if it helps), here is my code:

import torch.nn as nn
import torch.utils.model_zoo as model_zoo

class convnet(nn.Module):

    def __init__(self, num_classes=1000):
        super(convnet, self).__init__()
        self.features = nn.Sequential(
            nn.Conv2d(1, 64, kernel_size=3, stride=2, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(64, 64, kernel_size=3, padding=1),
            nn.MaxPool2d(kernel_size=1),
            nn.Conv2d(64, 128, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(128, 128, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2),# stride=2),
            nn.Conv2d(128, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(256, 256, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.MaxPool2d(kernel_size=2), #stride=2),
            nn.Conv2d(256, 512, kernel_size=3, padding=1),
            nn.ReLU(inplace=True),
            nn.Conv2d(512, 512, kernel_size=3, padding=1),
            nn.ReLU(inplace=True), #nn.Dropout(p=0.5)
        )

        self.classifier = nn.Sequential(
            nn.Linear(4, 1024),
            nn.Dropout(p=0.5),
            nn.ReLU(inplace=True),
            #nn.Dropout(p=0.5),
            nn.Linear(1024, 1024),
            nn.ReLU(inplace=True),
            nn.Linear(1024, num_classes),
        )

    def forward(self, x):
        x = self.features(x)
        x = torch.flatten(x,1)
        x = self.classifier(x)
        return x

I suspect it's an issue with the padding and stride.
Thanks!