title: Batch size and Training time

body:
Thank you for @Prune's critical comments on my questions.
I am trying to find the relationship between batch size and training time by using MNIST dataset.
By reading numerous questions in stackoverflow, such as this one:
How does batch size impact time execution in neural networks?
people said that the training time will be decreased when I use small batch size.
However, by trying out these two, I found that training with batch size == 1 takes way more time than batch size == 60,000. I set epoch as 10.
I split my MMIST dataset into 60k for the training and 10k for the testing.
This below is my code and results.
mnist_trainset = torchvision.datasets.MNIST(root=root_dir, train=True,
                                download=True,
                                transform=transforms.Compose([transforms.ToTensor()]))

mnist_testset  = torchvision.datasets.MNIST(root=root_dir,
                                train=False,
                                download=True,
                                transform=transforms.Compose([transforms.ToTensor()]))

train_dataloader = torch.utils.data.DataLoader(mnist_trainset,
                                               batch_size=1,
                                               shuffle=True)

test_dataloader  = torch.utils.data.DataLoader(mnist_testset,
                                               batch_size=50,
                                               shuffle=False)

# Define the model
class Model(torch.nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.linear_1 = torch.nn.Linear(784, 256)
        self.linear_2 = torch.nn.Linear(256, 10)
        self.sigmoid  = torch.nn.Sigmoid()

    def forward(self, x):
        x = x.reshape(x.size(0), -1)
        x = self.linear_1(x)
        x = self.sigmoid(x)
        pred = self.linear_2(x)

        return pred

# trainer
no_epochs = 10

def my_trainer(optimizer, model):

    criterion = torch.nn.CrossEntropyLoss()

    train_loss = list()
    test_loss  = list()
    test_acc   = list()
    best_test_loss = 1

    for epoch in range(no_epochs):

        # timer starts
        start = timer()

        total_train_loss = 0
        total_test_loss = 0

        # training
        # set up training mode
        model.train()

        for itr, (image, label) in enumerate(train_dataloader):

            optimizer.zero_grad()

            pred = model(image)

            loss = criterion(pred, label)
            total_train_loss += loss.item()

            loss.backward()
            optimizer.step()

        total_train_loss = total_train_loss / (itr + 1)
        train_loss.append(total_train_loss)

        # testing
        # change to evaluation mode
        model.eval()
        total = 0
        for itr, (image, label) in enumerate(test_dataloader):

            pred = model(image)

            loss = criterion(pred, label)
            total_test_loss += loss.item()

            # we now need softmax because we are testing.
            pred = torch.nn.functional.softmax(pred, dim=1)
            for i, p in enumerate(pred):
                if label[i] == torch.max(p.data, 0)[1]:
                    total = total + 1

        # caculate accuracy
        accuracy = total / len(mnist_testset)

        # append accuracy here
        test_acc.append(accuracy)

        # append test loss here
        total_test_loss = total_test_loss / (itr + 1)
        test_loss.append(total_test_loss)

        print('\nEpoch: {}/{}, Train Loss: {:.8f}, Test Loss: {:.8f}, Test Accuracy: {:.8f}'.format(epoch + 1, no_epochs, total_train_loss, total_test_loss, accuracy))

        if total_test_loss < best_test_loss:
            best_test_loss = total_test_loss
            print("Saving the model state dictionary for Epoch: {} with Test loss: {:.8f}".format(epoch + 1, total_test_loss))
            torch.save(model.state_dict(), "model.dth")

        # timer finishes
        end = timer()
        print(end - start)

    return no_epochs, test_acc, test_loss

model_sgd = Model()
optimizer_SGD = torch.optim.SGD(model_sgd.parameters(), lr=0.1)
sgd_no_epochs,      sgd_test_acc, sgd_test_loss           = my_trainer(optimizer=optimizer_SGD, model=model_sgd)

I calculated how much time did it took for each epoch.
And this below is the result.
Epoch: 1/10, Train Loss: 0.23193890, Test Loss: 0.12670580, Test Accuracy: 0.96230000
63.98903721500005 seconds

Epoch: 2/10, Train Loss: 0.10275097, Test Loss: 0.10111042, Test Accuracy: 0.96730000
63.97179028100004 seconds

Epoch: 3/10, Train Loss: 0.07269370, Test Loss: 0.09668248, Test Accuracy: 0.97150000
63.969843954 seconds

Epoch: 4/10, Train Loss: 0.05658571, Test Loss: 0.09841745, Test Accuracy: 0.97070000
64.24135530400008 seconds

Epoch: 5/10, Train Loss: 0.04183391, Test Loss: 0.09828428, Test Accuracy: 0.97230000
64.19695308500013 seconds

Epoch: 6/10, Train Loss: 0.03393899, Test Loss: 0.08982467, Test Accuracy: 0.97530000
63.96944059600014 seconds

Epoch: 7/10, Train Loss: 0.02808819, Test Loss: 0.08597597, Test Accuracy: 0.97700000
63.59837343000004 seconds

Epoch: 8/10, Train Loss: 0.01859330, Test Loss: 0.07529452, Test Accuracy: 0.97950000
63.591578820999985 seconds

Epoch: 9/10, Train Loss: 0.01383720, Test Loss: 0.08568452, Test Accuracy: 0.97820000
63.66664020100029

Epoch: 10/10, Train Loss: 0.00911216, Test Loss: 0.07377760, Test Accuracy: 0.98060000
63.92636473799985 seconds

After this I changed the batch size to 60000 and run the same program again.
train_dataloader = torch.utils.data.DataLoader(mnist_trainset,
                                               batch_size=60000,
                                               shuffle=True)

test_dataloader  = torch.utils.data.DataLoader(mnist_testset,
                                               batch_size=50,
                                               shuffle=False)

print("\n===== Entering SGD optimizer =====\n")
model_sgd = Model()
optimizer_SGD = torch.optim.SGD(model_sgd.parameters(), lr=0.1)
sgd_no_epochs,      sgd_test_acc, sgd_test_loss           = my_trainer(optimizer=optimizer_SGD, model=model_sgd)

I got this result for batch size == 60000
Epoch: 1/10, Train Loss: 2.32325006, Test Loss: 2.30074144, Test Accuracy: 0.11740000
6.54154992299982 seconds

Epoch: 2/10, Train Loss: 2.30010080, Test Loss: 2.29524792, Test Accuracy: 0.11790000
6.341824101999919 seconds

Epoch: 3/10, Train Loss: 2.29514933, Test Loss: 2.29183527, Test Accuracy: 0.11410000
6.161918789000083 seconds

Epoch: 4/10, Train Loss: 2.29196787, Test Loss: 2.28874513, Test Accuracy: 0.11450000
6.180891567999879 seconds

Epoch: 5/10, Train Loss: 2.28899717, Test Loss: 2.28571669, Test Accuracy: 0.11570000
6.1449509030003355 seconds

Epoch: 6/10, Train Loss: 2.28604794, Test Loss: 2.28270152, Test Accuracy: 0.11780000
6.311743144000047 seconds

Epoch: 7/10, Train Loss: 2.28307867, Test Loss: 2.27968731, Test Accuracy: 0.12250000
6.060618773999977 seconds

Epoch: 8/10, Train Loss: 2.28014660, Test Loss: 2.27666961, Test Accuracy: 0.12890000
6.171511712999745 seconds

Epoch: 9/10, Train Loss: 2.27718973, Test Loss: 2.27364607, Test Accuracy: 0.13930000
6.164125173999764 seconds

Epoch: 10/10, Train Loss: 2.27423453, Test Loss: 2.27061504, Test Accuracy: 0.15350000
6.077817454000069 seconds

As you can see it is clear that it took more time for each epoch when batch_size == 1 which is different from what I have seen.
Maybe I am confused with the training time per epoch vs the training time until convergence? Seems like my intuition is correct by looking at this webpage: https://medium.com/deep-learning-experiments/effect-of-batch-size-on-neural-net-training-c5ae8516e57
Can someone please explain what is happening?


title: How to measure each inference time with different batch size ? (using PyTorch on CPU)

body:
I Have a PyTorch CNN model and i want to measure each time inference for different batch sizes.
I've been looking for a while for a solution, i've tried using :

torch.profile() : which only gives the average time of th whole inferences of each batch of input Data.
thop : which i used for measuring number of mac computations of my model, but nothing for inference time.

Is there any way to measure each inference time for different batch sizes ?
Or even for any batch size ? (example a batch of 32 inputs)
NB : code of inference :
...

BATCH_SIZE = 10

input = randn((BATCH_SIZE, 1, 100))
with torch.no_grad():
        model = torch.load("cnnModel.pth")
        model.eval()
#inference
        start = timeit.default_timer()
        model(input)
        end = timeit.default_timer() - start
        print("time %f ms" %end)


title: Positive and negative edges in train_test_split_edges from Pytorch geometric package

body: I am trying to find the explanation for negative and positive edges in a graph as explained in the opening of the function train_test_split_edges Pytorch Geometric doc. According to the doc file it says the function is supposed to split a graph into "positive and negative train/val/test edges". What is the meaning of a positive edge or a negative edge for that matter. According to the code the positive edges "seems" to be the connections of edges in the upper triangle of the adjacency matrix of the graph and the negative edges are the edges in the lower triangle of the adjacency matrix. So if (1,0) is considered to be a positive edge then in an undirected graph (0,1) is a negative edge. Am I correct? I am not finding anywhere about the meaning of positive edge / negative edge where in comes to graphs.


title: RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED in Nvidia docker

body:
I use the nvidia docker ufoym/deepo:all-py36-cu90.
And I have tried use the solution https://discuss.pytorch.org/t/runtimeerror-cudnn-error-cudnn-status-not-initialized/115286 .But it doesn't work.
File "train.py", line 330, in <module>
    main()
  File "train.py", line 199, in main
    pos2_trans)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/pwclonet_latest/models.py", line 224, in forward
    l0_points_f1)
  File "/data/pwclonet_latest/pointconv_util.py", line 382, in forward
    new_points = conv(new_points)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/pwclonet_latest/pointconv_util.py", line 73, in forward
    outputs = self.conv(x.to(torch.float32))
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 399, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED


title: How to generate a new Tensor with different vectors in PyTorch?

body:
I want to generate new a○b vector with a and b (○ means element wise multiply). My code is below, but the performance looks bad because of for. Are there any efficient way?

a = torch.rand(batch_size, a_len, hid_dim)
b = torch.rand(batch_size, b_len, hid_dim)
# a_elmwise_mul_b = torch.zeros(batch_size, a_len, b_len, hid_dim)
for sample in range(batch_size):
    for ai in range(a_len):
        for bi in range(b_len):
            a_elmwise_mul_b[sample, ai, bi] = torch.mul(a[sample, ai], b[sample, bi])

Update

I updated my code refer to Ahmad! Thank you.

N = 16
hid_dim = 50
a_seq_len = 10
b_seq_len = 20
a = torch.randn(N, a_seq_len, hid_dim)
b = torch.randn(N, b_seq_len, hid_dim)
shape = (N, a_seq_len, b_seq_len, hid_dim)

a_dash = a.unsqueeze(2) # (N, a_len, 1,     hid_dim)
b_dash = b.unsqueeze(1) # (N, 1,     b_len, hid_dim)
a_dash = a_dash.expand(shape)
b_dash = b_dash.expand(shape)
print(a_dash.size(), b_dash.size())
mul = a_dash * b_dash
print(mul.size())
----------
torch.Size([16, 10, 20, 50]) torch.Size([16, 10, 20, 50])
torch.Size([16, 10, 20, 50])


title: PyTorch 0.40 optimization issue

body:
I’m trying to optimize the following parameters by keeping the L param always lower triangular with a positive diagonal and the noise param always with just positive diagonal during optimization, but they are not correctly updating in the forward pass. I guess I’m doing something wrong with the autograd mechanism. Any help appreciated. Here is a sample snippet illustrating the problem -using fictional calculation- using PyTorch 0.40.

import torch
from torch.autograd import Variable
from torch.nn.parameter import Parameter
from torch.nn import ParameterList

class Model(torch.nn.Module):
    def __init__(self, dim):
        """
        Constructor.
        """
        super(Model, self).__init__()

        self.noise_vector = Parameter(torch.tensor(torch.zeros(D).cuda(), requires_grad=True))
        self.noise = Parameter(torch.tensor(torch.diag(torch.exp(self.noise_vector.data)).cuda(), requires_grad=True))

        self.L_chol_cov_theta = Parameter(torch.tensor(torch.randn(dim, dim).cuda(), requires_grad=True))
        self.log_diag_L_chol_cov_theta = Parameter(torch.tensor(torch.randn(dim).cuda(), requires_grad=True))
        self.L = Parameter(torch.tensor(torch.randn(dim, dim).cuda(), requires_grad=True))
        self.L_chol_cov_theta.data = torch.tril(self.L_chol_cov_theta.data)
        self.L_chol_cov_theta.data -= torch.diag(torch.diag(self.L_chol_cov_theta.data))
        self.L.data = self.L_chol_cov_theta.data + torch.diag(torch.exp(self.log_diag_L_chol_cov_theta.data))

    def forward(self):
        # update parameters
        self.L_chol_cov_theta.data -= torch.diag(torch.diag(self.L_chol_cov_theta.data))
        self.L.data = self.L_chol_cov_theta.data + torch.diag(torch.exp(self.log_diag_L_chol_cov_theta.data))

        self.noise.data = torch.diag(torch.exp(self.noise_vector.data))

        return torch.mm(self.L, self.noise_vector.view(-1,-1))

 model = Model(5)
 optimizer = torch.optim.SGD(filter(lambda p: p.requires_grad, model.parameters()), lr=1e-3)

for epoch in range(10):
   optimizer.zero_grad()
   forward_pass_something = model()
   loss = calc_likelihood(samples, a_ground_truth) # calc a custom loss
   loss.backward()
   optimizer.step()


title: Why filters do not learn same features

body:
The result of the convolution operation is multiple subsets of data are generated per kernel. For example if 5 kernels are applies to an image of dimension WxDx1 (1 channel) then 5 convolutions are applied to the data which generates a 5 dimensional image representation. WxDx1 becomes W'xD'x5 where W' and D' are smaller in dimension that W * D

Is the fact that each kernel is initialised to different values prevent each kernel from learning the same parameters ? If not what prevents each kernel learning the same parameters ?

If the image is RGB instead of grayscale so dimension WxDx3 instead of WxDx1 does this impact how the kernels learns patterns ?


title: Deep learning model to clean documents

body:
I am trying to build a Convolutional Autoencoder which can remove pen marks such as circles, underlines, etc from official documents.

I have the original, clean soft copy of the document and the xeroxed copy with the pen marks.

The xeroxed copy would obviously not completely match up with the original doc and it would be slightly skewed or shifted, etc while scanning.

Additionally, since the doc size is huge (2360, 1650), I had to split the image into 4 halves of size (587, 412) to input into my model.

What I would like to know is would the above issues cause any problems while training and is there any way to rectify it?

Any help would be highly appreciated.

Thanks

EDIT:

As you can see (hopefully!!), the noisy image is slightly different because of skewness or translation while xeroxing.


title: Strange behavior of linear regression in PyTorch

body:
I am facing a peculiar problem and I was wondering if there is an explanation. I am trying to run a linear regression problem and test different optimization methods and two of them have a strange outcome when comparing to each other. I build a data set that satisfies y=2x+5 and I add a random noise to that.

 xtrain=np.range(0,50,1).reshape(50,1)
 ytrain=2*train+5+np.random.normal(0,2,(50,1))
 opt1=torch.optim.SGD(model.parameters(),lr=1e-5,momentum=0.8))
 opt2=torch.optim.Rprop(model.parameters(),lr=1e-5)
 F_loss=F.mse_loss
 from torch.utils.data import TensorDataset,DataLoader
 train_d=TensorDataset(xtrain,ytrain)
 train=DataLoader(train_d,50,shuffle=True)
 model1=nn.Linear(1,1)
 loss=F_loss(model1(xtrain),ytrain)

 def fit(nepoch, model1, F_loss, opt):
     for epoch in range(nepoch):
         for i,j in train:
         predict = model1(i)
         loss = F_loss(predict, j)
         loss.backward()
         opt.step()
         opt.zero_grad()

When i compare the results of the following commands:

        fit(500000, model1, F_loss, opt1)
        fit(500000, model1, F_loss, opt2)

In the last epoch for opt1:loss=2.86,weight=2.02,bias=4.46
 In the last epoch for opt2:loss=3.47,weight=2.02,bias=4.68

These results do not make sense to me, shouldn't opt2 have a smaller loss than opt1 since the weight and bias it finds is closer to the real value? opt2's method finds weights and biases to be closer to the real value (they are respectively 2 and 5). Am i doing something wrong?


title: Pytorch RNN HTML Generation

body:
I’ m stuck for a couple of days trying to make and RNN network to learn a basic HTML template.
I tried different approaches and I even overfit on the following data:

<!DOCTYPE html>

<title>Page Title</title>

This is a Heading

This is a paragraph.

Obtaining 100% accuracy on training and validation using Adam Optimizer and CrossEntropyLoss.

The problem is that when I try to sample from the network, the results are completely random and I don’t know whats the problem:

..<a<a<a<a<aa<ttp11111b11111b11111111b11b1bbbb<btttn111

My sampling function is the following:

def sample_sentence():
    words = list()
    count = 0
    modelOne.eval()
    with torch.no_grad():
        # Setup initial input state, and input word (we use "the").
        previousWord = torch.LongTensor(1, 1).fill_(trainData.vocabulary['letter2id']['[START]'])
        hidden =  Variable(torch.zeros(6, 1, 100).to(device))

        while True:
            # Predict the next word based on the previous hidden state and previous word.
            inputWord = torch.autograd.Variable(previousWord.to(device))

            predictions, newHidden = modelOne(inputWord, hidden)

            hidden = newHidden

            pred = torch.nn.functional.softmax(predictions.squeeze()).data.cpu().numpy().astype('float64')

            pred = pred/np.sum(pred)

            nextWordId = np.random.multinomial(1, pred, 1).argmax()

            if nextWordId == 0:
                continue

            words.append(trainData.vocabulary['id2letter'][nextWordId])
            # Setup the inputs for the next round.
            previousWord.fill_(nextWordId)

            # Keep adding words until the [END] token is generated.
            if nextWordId == trainData.vocabulary['letter2id']['[END]']:
                break

            if count>20000:
                break
            count += 1
        words.insert(0, '[START]')

        return words

And my network architecture is here:

class ModelOne(Model) :
    def __init__(self,
                vocabulary_size,
                hidden_size,
                num_layers,
                rnn_dropout,
                embedding_size,
                dropout,
                num_directions):
        super(Model, self).__init__()

        self.vocabulary_size = vocabulary_size
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.rnn_dropout = rnn_dropout
        self.dropout = dropout
        self.num_directions = num_directions
        self.embedding_size = embedding_size

        self.embeddings = nn.Embedding(self.vocabulary_size, self.embedding_size)
        self.rnn = nn.GRU(self.embedding_size,
                          self.hidden_size,
                          num_layers=self.num_layers,
                          bidirectional=True if self.num_directions==2 else False,
                          dropout=self.rnn_dropout,
                          batch_first=True)
        self.linear = nn.Linear(self.hidden_size*self.num_directions, self.vocabulary_size)

    def forward(self, paddedSeqs, hidden):

        batchSequenceLength = paddedSeqs.size(1)

        batchSize = paddedSeqs.size(0)

        lengths = paddedSeqs.ne(0).sum(dim=1)

        embeddingVectors = self.embeddings(paddedSeqs)

        x = torch.nn.utils.rnn.pack_padded_sequence(embeddingVectors, lengths, batch_first=True)

        self.rnn.flatten_parameters()

        x,hid = self.rnn(x, hidden)

        output, _ = torch.nn.utils.rnn.pad_packed_sequence(x, batch_first=True, padding_value=0, total_length=batchSequenceLength)

        predictions = self.linear(output)

        return predictions.view(batchSize, self.vocabulary_size, batchSequenceLength), hid

    def init_hidden(self, paddedSeqs):
        hidden = Variable(torch.zeros(self.num_layers*self.num_directions,
                                    1,
                                    self.hidden_size).to(device))
        return hidden

modelOne =ModelOne(vocabulary_size=vocabularySize,
                 hidden_size=100,
                 embedding_size=50,
                 num_layers=3,
                 rnn_dropout=0.0,
                 dropout=0,
                 num_directions=2).to(device)

If you have any idea of what needs to be changed, please let me know.
I added all the code to github repository here: https://github.com/OverclockRo/HTMLGeneration/blob/SamplingTestTemplate/Untitled.ipynb


title: Pytorch: turning a [1,x] sized tensor into an [x] sized tensor

body:
While trying to load in Pytorch 0.4.0 a model that has probably been produced by Pytorch 0.3.1, I keep getting such errors:

While copying the parameter named "conv1_7x7_s2_bn.bias", whose dimensions in the
model are torch.Size([64]) and whose dimensions in the checkpoint are torch.Size([1, 64]).

I thought that if I had applied transpose on each tensor, then it would work, but it is still failing, as the dimension turns into [64, 1], rather than [64], which I need.

I can I remove the redundant dimension and thus turn the 1-row matrix into a vector?

Note: When calling torch.flatten, I get:

AttributeError: module 'torch' has no attribute 'flatten'


title: plot function does not plot curves just plotted legend in PyCharm

body:
I am doing some homework about image classification using PyTorch. For this purpose I have visited the following link: https://github.com/Bjarten/early-stopping-pytorch/blob/master/MNIST_Early_Stopping_example.ipynb and obtained some results like this related to training losses and validation losses

[ 1/10] train_loss: 1.56952 valid_loss: 1.54557
Validation loss decreased (inf --> 1.545571).  Saving model ...
[ 2/10] train_loss: 1.52198 valid_loss: 1.50932
Validation loss decreased (1.545571 --> 1.509324).  Saving model ...
[ 3/10] train_loss: 1.48958 valid_loss: 1.48514
Validation loss decreased (1.509324 --> 1.485136).  Saving model ...
[ 4/10] train_loss: 1.46960 valid_loss: 1.46938
Validation loss decreased (1.485136 --> 1.469381).  Saving model ...

But the problem is that whenever I try to plot loss curves it does not show anything.

my plotted figure

I would expect my curve to be like the one in this picture.
Here is the code that I used to draw this curve:

  # visualize the loss as the network trained
    fig = plt.figure(figsize=(10, 8))
    plt.plot(range(1, len(train_loss) + 1), train_loss, label='Training Loss')
    plt.plot(range(1, len(valid_loss) + 1), valid_loss, label='Validation Loss')

        # find position of lowest validation loss
    minposs = valid_loss.index(min(valid_loss)) + 1
    plt.axvline(minposs, linestyle='--', color='r', label='Early Stopping Checkpoint')

    plt.xlabel('epochs')
    plt.ylabel('loss')
    plt.ylim(0, 0.5)  # consistent scale
    plt.xlim(0, len(train_loss) + 1)  # consistent scale
    plt.grid(True)
    plt.legend()
    plt.tight_layout()
    plt.show()
    fig.savefig('loss_plot.png', bbox_inches='tight')

Here is the value of length of the training loss and the testing loss

len of train loss: 1
train loss :[1.6059992909431458]


title: How to train features in different scales in deep learning model

body:
I'm new in deep learning and I built a very simple model to try to train my data. I have two features input: sex and age. sex is 0 or 1 and age is between 25 and 60. Output is just 0 means this person has no such disease and 1 means has such disease.

However, when I train my model, the training loss does not decrease at all. It looks like because my two features are very different in range. So how can I fix this? Any suggestions would be greatly appreciated.

My code is here:

class Net(nn.Module):
    def __init__(self):
        super(Net,self).__init__()

        self.fc1 = nn.Sequential(
            nn.Linear(2,50),
            nn.ReLU(),
            nn.Linear(50,2)
        )

    def forward(self,x):
        x = self.fc1(x)

        x = F.softmax(x, dim=1)
        return x

#Inputs
X = np.column_stack((sex,age))
X = torch.from_numpy(X).type(torch.FloatTensor)
y = torch.from_numpy(y).type(torch.LongTensor)

#Initialize the model
model = Net()
#Define loss criterion
criterion = nn.CrossEntropyLoss()
#Define the optimizer
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)

epochs = 1000

losses = []
for i in range(epochs):

    y_pred = model.forward(X)
    #Compute Cross entropy loss
    loss = criterion(y_pred,y)
    #Add loss to the list
    losses.append(loss.item())
    #Clear the previous gradients
    optimizer.zero_grad()
    #Compute gradients
    loss.backward()
    #Adjust weights
    optimizer.step()
    _, predicted = torch.max(y_pred.data, 1)

    if i % 50 == 0:
        print(loss.item())

And the train loss looks like this

0.9273738861083984
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618
0.6992899179458618

EDIT

Thank you for your comments. Sorry I didn't explain my question clearly. This is part of my network and my input data contains two parts: the first part is some signal data and I use CNN model to train it and it works well; the second part is what I mentioned above. My goal is to merge two models to improve my accuracy.
I've tried normalization and looks like it works. I want to know is it always necessary to do normalization when pre-processing data? Thank you!


title: Recommendation for Best Neural Network Type (in TensorFLow or PyTorch) For Fitting Problems

body:
I am looking to develop a simple Neural Network in PyTorch or TensorFlow to predict one numeric value based on several inputs.

For example, if one has data describing the interior comfort parameters for a building, the NN should predict the numeric value for the energy consumption.

Both PyTorch or TensorFlow documented examples and tutorials are generally focused on classification and time dependent series (which is not the case). Any idea on which NN available in those libraries is best for this kind of problems? I'm just looking for a hint about the type, not code.
Thanks!


title: How to find a good LR?

body:
I'm very new to PyTorch and working on a project of mine to solve a Sudoku board. What I'm doing is giving the network a Tensor that has the board (9x9) and another 2 values, the first being the row and the second being the column.
My network:

class Network(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(9 * 9 + 2, 32)  # board + row and col
        self.fc2 = nn.Linear(32, 32)
        self.fc3 = nn.Linear(32, 9)  # the number in that spot

    def forward(self, x):
        x = f.relu(self.fc1(x))
        x = f.relu(self.fc2(x))
        x = self.fc3(x)
        return f.log_softmax(x, dim=1)

I have a for loop that iterates over each board, and then iterate on each block in the 9x9 grid and input the board with the row and column to the network.

optimizer = optim.Adam(net.parameters(), lr=0.1)
scheduler = lr_scheduler.CosineAnnealingLR(optimizer, len(quizzes), eta_min=0)
for i, board in enumerate(quizzes):
    new_board = [[val for val in row] for row in board]  # Don't affect original board
    for row in range(9):
        for col in range(9):
            if new_board[row][col] != 0:
                continue
            row_col[0] = row  # the row value
            row_col[1] = col  # the col value

            final_tensor = board_tensor.view(-1, 9 * 9 + 2)
            output = net(final_tensor)  # type: Tensor
            solution_num = solutions[i][row][col]
            solution_tensor = torch.tensor(solution_num - 1, dtype=torch.long).reshape(-1)  # do -1 because it needs to match the node.
            optimizer.zero_grad()
            loss = f.nll_loss(output, solution_tensor)
            loss.backward()
            optimizer.step()
            scheduler.step(epoch=epoch)
            new_board[row][col] = solution_num  # Add the new value into the board.
            avg_loss += loss.item()
            count += 1
    if i % 100 == 0:
        print(f"Loss: {round(avg_loss / count, 3)}. {i} / {len(quizzes)}. {epoch} / {EPOCHS}")
        avg_loss = 0
        count = 0

Now, my issue is that the network is just guessing the same number over and over. Every time I reset the network it guesses a different number, but after a bit it just stays constant and doesn't guess any other number. This of course makes the accuracy 11.111% (1/9) and I don't know how to get past this. I tried using MSELoss instead of NLL_loss but it didn't change any result, and switched between optim.Adam to optim.SGE. I am quite new to this whole topic so I don't know which functions I should be using (log_softmax, Adam, SGE, and all those types of functions for loss / optimization).

Does anyone know where I messed up? I tried changing the learning rate and adding a weight decay too, but that didn't help


title: Python process never finishing when called from Java

body:
I've tried to setup an AI with PyTorch. Everything is fine when I call my script from the console. But when I call the script in a Java `ProcessBuildera, it will finish but never terminate...

Here is the ProcessBuilder code

String[] cmd = {"python3", "-i" , "AI/Home-System.py",
                data.getName().replace(".csv", ""),
                "true",
                "false"};
ProcessBuilder pb = new ProcessBuilder(cmd);
Process p = pb.start();

Hope that you can help me

Edit:

I found another solution. I call this script in a linux screen with

String[] cmd = {"screen", "-dmS", "AI-" + device,
                "python3", "AI/Home-System.py",
                data.getName().replace(".csv", ""),
                "true",
                "false"};
Runtime.getRuntime().exec(cmd);


title: pytorch train function Varibles and Tensors (read my introduction i dont know my problem as well it just dont work )

body:
I started learning pytorch and started with videos about MNIST handwriting and learnt it with an video but the video is 2 years old and some things have changen since then i guess because it dosent work as in the video and i seriously dont know anything so i dont know whats my error or what i do wrong i just type everything the dude says in the video and want to understand and learn it this way(maybe you know better ways how to learn machine learning/deep learning would appreciate it) my code looks like this:

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torch.optim as optim
import os
from torchvision import datasets, transforms

kwargs = {'num_workers': 1, 'pin_memory': True}
train_data = torch.utils.data.DataLoader(datasets.MNIST('data', train=True, download=True, transform=transforms.Compose([transforms.ToTensor(),
                                                    transforms.Normalize((0.1307,),(0.3081,))])),
                                                    batch_size=64, shuffle=True, **kwargs)

test_data = torch.utils.data.DataLoader(datasets.MNIST('data', train=False, transform=transforms.Compose([transforms.ToTensor(),
                                                    transforms.Normalize((0.1307,),(0.3081,))])),
                                                    batch_size=64, shuffle=True, **kwargs)

above everything works like in the video and i find the data in an folder now comes the class and it doesnt looks like theres an error but i dont know.

class Netz(nn.Module):
    def __init__(self):
        super(Netz, self).__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size= 4)
        self.conv2 = nn.Conv2d(10, 20, kernel_size= 4)
        self.conv_dropout = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 60)
        self.fc2 = nn.Linear(60, 10)

    def forward(self, x):
        x = self.conv1(x)
        x = F.max_pool2d(x, 4)
        x = F.relu(x)
        x = self.conv2(x)
        x = self.conv_dropout(x)
        x = F.max_pool2d(x, 4)
        x = F.relu(x)
        print(x.size())
        exit()

model = Netz()
model.cuda()

something with this Varibale function is wrong it just dont works and pycharm also shows me there has to be something wrong but i dont know what so i ask here maybe you can help i also googled abit about it and it looks like this varible thing got removed or so but i dont know what to write else

optimizer = optim.SGD(model.parameters(), lr=0.1, momentum=0.8)
def train(epoch):
    model.train()
    for batch_id, (data, target) in enumerate(train_data):
        data = data.cuda()
        target = target.cuda()
        data = Variable(data)
        target = Variable(target)
        optimizer.zero_grad()
        out = model(data)
        criterion = F.nll_loss
        loss = criterion(out, target)
        loss.backward()
        optimizer.step()

for epoch in range(1, 30):
    train(epoch)

the error code looks like this :

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\mnist handwriting.py", line 60, in <module>
    train(epoch)
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\mnist handwriting.py", line 46, in train
    for batch_id, (data, target) in enumerate(train_data):
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 279, in __iter__
    return _MultiProcessingDataLoaderIter(self)
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 719, in __init__
    w.start()
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\popen_spawn_win32.py", line 46, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "C:\Users\Finnw\AppData\Local\Programs\Python\Python37\lib\queue.py", line 178, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/Finnw/PycharmProjects/pytorch 3.7/mnist handwriting.py", line 60, in <module>
    train(epoch)
  File "C:/Users/Finnw/PycharmProjects/pytorch 3.7/mnist handwriting.py", line 46, in train
    for batch_id, (data, target) in enumerate(train_data):
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 798, in _get_data
    success, data = self._try_get_data()
  File "C:\Users\Finnw\PycharmProjects\pytorch 3.7\venv\lib\site-packages\torch\utils\data\dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 10444) exited unexpectedly

Process finished with exit code 1


title: Read multiple .gz file and return it in one tensor

body:
I am trying to read multiple .gz file and return its content in one tensor as follows:

with ReadHelper('ark: gunzip -c /home/mnabih/kaldi/egs/timit/s5/exp/mono_ali/*.gz|') as reader:
    for key, b in reader:
        #print(type(b))
        c = torch.from_numpy(b)
        labels = torch.cat(c)

Unfortunately, it gives me this error  cat(): argument 'tensors' (position 1) must be tuple of Tensors, not Tensor

Can any one explain Why or a solution


title: How to output the accuracy alongside with the loss when training the MNIST dataset after each epoch

body:
from __future__ import print_function
import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F
from tensorflow.examples.tutorials.mnist import input_data
import torch.optim as optim
import tensorflow.python.util.deprecation as deprecation
deprecation._PRINT_DEPRECATION_WARNINGS = False
import matplotlib.pyplot as plt
%matplotlib inline
from plot import plot_loss_and_acc

mnist = input_data.read_data_sets("MNIST_data", one_hot=False)
batch_size = 250
epoch_num = 10
lr = 0.0001
disp_freq = 20

def next_batch(train=True):
    # Reads the next batch of MNIST images and labels and returns them

    if train:
        batch_img, batch_label = mnist.train.next_batch(batch_size)
    else:
        batch_img, batch_label = mnist.test.next_batch(batch_size)

    batch_label = torch.from_numpy(batch_label).long()  # convert the numpy array into torch tensor
    batch_label = Variable(batch_label)         # create a torch variable

    batch_img = torch.from_numpy(batch_img).float()     # convert the numpy array into torch tensor
    batch_img = Variable(batch_img)             # create a torch variable
    return batch_img, batch_label

class MLP(nn.Module):
    def __init__(self, n_features, n_classes):
        super(MLP, self).__init__()
        self.layer1 = nn.Linear(n_features, 128)
        self.layer2 = nn.Linear(128, 128)
        self.layer3 = nn.Linear(128, n_classes)

    def forward(self, x, training=True):
        # a neural network with 2 hidden layers
        # x -> FC -> relu -> dropout -> FC -> relu -> dropout -> FC -> output
        x = F.relu(self.layer1(x))
        x = F.dropout(x, 0.5, training=training)
        x = F.relu(self.layer2(x))
        x = F.dropout(x, 0.5, training=training)
        x = self.layer3(x)
        return x

    def predict(self, x):
        # a function to predict the labels of a batch of inputs
        x = F.softmax(self.forward(x, training=False))
        return x

    def accuracy(self, x, y):
        # a function to calculate the accuracy of label prediction for a batch of inputs
        #   x: a batch of inputs
        #   y: the true labels associated with x
        prediction = self.predict(x)
        maxs, indices = torch.max(prediction, 1)
        acc = 100 * torch.sum(torch.eq(indices.float(), y.float()).float())/y.size()[0]
        print(acc.data)
        return acc.data

# define the neural network (multilayer perceptron)
net = MLP(784, 10)

# calculate the number of batches per epoch
batch_per_ep = mnist.train.num_examples // batch_size

# define the loss (criterion) and create an optimizer
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=lr)
print(' ')
print("__________Training__________________")

xArray = []
yLoss = []
yAcc = []
for ep in range(epoch_num):  # epochs loop
    for batch_n in range(batch_per_ep):  # batches loop
        features, labels = next_batch()

        # Reset gradients
        optimizer.zero_grad()

        # Forward pass
        output = net(features)
        loss = criterion(output, labels)
        # Backward pass and updates
        loss.backward()                 # calculate the gradients (backpropagation)
        optimizer.step()                # update the weights
        if batch_n % disp_freq == 0:
            print('epoch: {} - batch: {}/{} '.format(ep, batch_n, batch_per_ep))
    xArray.append(ep)
    yLoss.append(loss.data)
    #yAcc.append(acc.data)

    print('loss: ', loss.data)
    print('__________________________________')

# test the accuracy on a batch of test data
features, labels = next_batch(train=False)
print("Result")
print('Test accuracy: ', net.accuracy(features, labels))
print('loss: ', loss.data)
accuracy = net.accuracy(features, labels)

#Loss Plot

# plotting the points
plt.plot(xArray, yLoss)

# naming the x axis
plt.xlabel('epoch')
# naming the y axis
plt.ylabel('loss')

# giving a title to my graph
plt.title('Loss Plot')

# function to show the plot
plt.show()

#Accuracy Plot

# plotting the points
plt.plot(xArray, yAcc)

# naming the x axis
plt.xlabel('epoch')
# naming the y axis
plt.ylabel(' accuracy')

# giving a title to my graph
plt.title('Accuracy Plot ')

# function to show the plot
plt.show()

I want to display the accuracy of my training dataset. I have managed to display and plot the loss but I didn't manage to do it for accuracy. I know I am missing 1 or 2 lines of code and I don't know how to do it.

I mean if I can display the accuracy alongside each epoch like the loss I can do the plotting myself.


title: Does tracking the loss via lists affect the training?

body:
I wanted to plot the loss of my CNN, so created lists before starting to train with
test_loss_history = [] train_loss_history = [] and added the values after every epoch with train_loss_history.append(train_loss) test_loss_history.append(test_loss). I had done the same the same with the accuracy before, but when I add these lines for the loss, the accuracy drops around 40%. Does storing values affect the training process in any way?

I am using Google Colab and train a ResNet18 with a subset of MNIST.

My code looks like that:

    train_loss_history = []
    train_acc_history = []
   for epoch in range(epoch_resume, opt.max_epochs):
       ...
       for i, data in enumerate(trainloader, 0):
          train_loss     += imgs.size(0)*criterion(logits, labels).data
          ...
       train_loss     /= len(trainset)
       train_acc_history.append(train_acc)
       train_loss_history.append(train_loss)


title: Representing sentence as Graph Neural Networks - NLP

body: I am new to the field of NLP and I am looking to express a text sentence in terms of Graph Neural Network with Nodes and Edges where edges define the semantic relationship. Is there any library in Python on the same or is it purely subjective? I know Pytorch Geometric helps us to manipulate our GNNs. But my question is to seek help in bringing the text sentence to a form that can later be processed using Pytorch Geometric.


title: How is a minibatch processed by the GPU in PyTorch?

body:
I am trying to understand how PyTorch actually performs a forward pass over a minibatch. When a minibatch is processed by a network, is each example in the minibatch (e.g. each image) sent forwards individually, one after the other? Or are all examples in the minibatch sent forwards at the same time?
When an example is sent forwards through a network, the additional memory requirement is the activations at each layer. And as long as the network does not take up the entire GPU, then it seems that multiple instantiations of these activations could be stored at the same time. Each instantiation could then be used to store the activations for one example in the minibatch. And therefore, multiple examples could be sent through the network simultaneously. However, I'm unsure whether this is actually done in practice.
I have done some simple experiments, and the time for a forward pass is roughly proportional to the minibatch size. This suggests that the examples are sent through one after the other. If so, then why is it that people say that training is faster when the minibatch size is larger? It seems that the processing time for an entire epoch would not be dependent on the minibatch size.


title: STILL overfitting image classification for CheXpert dataset

body:
I try to classify images from CheXpert dataset on only one of the observations (Alectasis) and a 2 class classification problem (1 true, 0 false).
I preprocess the images by resizing them to 224x224 and normalizing them. I used 30,000 pictures for training (10% validation) and 7,500 test images.
As a model I am using a pretrained ResNet34 on ImageNet.
When running the model it is overfitting: training loss decreases to 0.043 wheareas the validation loss rises to 2.199 which leads to a test accuracy of 55.56%.
I tried following attempts to prevent the overfitting:
#Attempt 1: I used a classifier with dropout layers
-> Validation loss decreased in the beginning but after 6 epochs it was rising again
#Attempt 2: I added dropout layers in the whole model
-> Validation loss decreased in the beginning but model converged very slowly. After some epochs, validation loss increased and model also converged faster
#Attempt 3: I froze all nonlinear layers and fitted the model only on the last linear layer
-> Network did not seem to converge at all, not even after 50 epochs
Making the classification task a binary classification task also changed nothing. Trying the same things with a DenseNet121 also didn't change a lot.
class ResNet(FitModule):
    def __init__(self, num_classes=2):
        super(ResNet, self).__init__()
        self.net = torchvision.models.resnet34(pretrained=True)
        # Change classifier
        kernel_count = self.net.fc.in_features
        self.net.fc = nn.Sequential(nn.Linear(kernel_count, 500), nn.Linear(500, num_classes))
        self.dropout = nn.Dropout(p=0.5)
        # Attempt 1: use classifier with dropout layers
        '''
        self.net.fc = nn.Sequential(
            nn.BatchNorm1d(kernel_count),
            nn.Dropout(p=0.5),
            nn.Linear(in_features=kernel_count, out_features=500),
            nn.ReLU(),
            nn.BatchNorm1d(500),
            nn.Dropout(p=0.5),
            nn.Linear(in_features=500, out_features=num_classes))
        '''

    def freeze_nonlinear_layers(self):
        self._freeze_layer(self.net.conv1)
        self._freeze_layer(self.net.bn1)
        self._freeze_layer(self.net.relu)
        self._freeze_layer(self.net.maxpool)
        self._freeze_layer(self.net.layer1)
        self._freeze_layer(self.net.layer2)
        self._freeze_layer(self.net.layer3)
        self._freeze_layer(self.net.layer4)
        self._freeze_layer(self.net.avgpool)

    def _freeze_layer(self, layer, freeze=True):
        if freeze:
            for p in layer.parameters():
                p.requires_grad = False
        else:
            for p in layer.parameters():
                p.requires_grad = True

    def forward(self, inputs):
        # Attempt 2: build whole network with dropout layers
        '''
        out = self.net.conv1(inputs)
        out = self.net.bn1(out)
        out = self.net.relu(out)
        out = self.net.maxpool(out)
        out = self.dropout(out)
        out = self.net.layer1(out)
        out = self.dropout(out)
        out = self.net.layer2(out)
        out = self.dropout(out)
        out = self.net.layer3(out)
        out = self.dropout(out)
        out = self.net.layer4(out)
        out = self.dropout(out)
        out = self.net.avgpool(out)
        out = out.view(out.size(0), -1)
        out = self.net.fc(out)
        '''
        # Attempt 3: freeze nonlinear layers and only train last linear layer
        '''
        self.freeze_nonlinear_layers
        '''
        return out/self.net(inputs) # attempt 2/attempt 1,3

To sum things up: Either the network does not converge or the validation loss rises and the test accuracy is poor.


title: Copy large github file into existing repo without downloading/cloning

body:
How to i add bin file associated to this repo https://github.com/graykode/gpt-2-Pytorch to my existing private repo without downloading/cloning?  There is a 500mb file that needs downloading to get the codebase to work that I need in my repo and don't want to download into my small hard disk during the cloning process.
It's a noob question but i'm at a complete loss as to how to proceed. Please be nice.


title: Pytorch guide not using optim to train

body:
I am currently working through a guide on the Pytroch website here: https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html
I have done pytorch projects before and they have always made use of an optimizer. This guide instead uses the code here:
    # Add parameters' gradients to their values, multiplied by learning rate
    for p in rnn.parameters():
        p.data.add_(p.grad.data, alpha=-learning_rate)

I was confused by this and wanted to know why this works. Additionally I tried to rewrite the code using an optimizer and it was unable to learn. It is using a recurrent neural network which may be the reason, but I am unsure why. Thanks!


title: How to check if 'Optim' package on torch 7 is installed?

body:
I have some LUA code downloaded from GitHub for a neural network which I'm trying to run on my Ubuntu server following the instruction here (https://github.com/hezhangsprinter/ID-CGAN). But it says module 'optim' not found. How do I check if it's installed or not? The error message is given below:
shaurovd@fire:~/ID-CGAN-master/IDCGAN$ DATA_ROOT=/home/shaurovd/dataset/rain name=rain which_direction=BtoA th train.lua
/usr/bin/luajit: /usr/share/lua/5.1/trepl/init.lua:389: module 'optim' not found:
        no field package.preload['optim']
        no file './optim.lua'
        no file '/usr/share/luajit-2.1.0-beta3/optim.lua'
        no file '/usr/local/share/lua/5.1/optim.lua'
        no file '/usr/local/share/lua/5.1/optim/init.lua'
        no file '/usr/share/lua/5.1/optim.lua'
        no file '/usr/share/lua/5.1/optim/init.lua'
        no file './optim.so'
        no file '/usr/local/lib/lua/5.1/optim.so'
        no file '/usr/lib/x86_64-linux-gnu/lua/5.1/optim.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
        [C]: in function 'error'
        /usr/share/lua/5.1/trepl/init.lua:389: in function 'require'
        train.lua:8: in main chunk
        [C]: in function 'dofile'
        /usr/lib/torch-trepl/th:149: in main chunk
        [C]: at 0x55f3f24781d0


title: Why does `is_same` in the C++ PyTorch API fail when comparing with the same tensor that is read from a file?

body:
Why does torch::Tensor::is_same fail the following assertion? A tensor is written to a file using the C++ PyTorch API, then read again into another tensor, and is_same compares both tensors:
torch::Tensor x_sequence = torch::linspace(0, M_PI, 1000);
torch::save(x_sequence, "x_sequence.dat");
torch::Tensor x_read;
torch::load(x_read, "x_sequence.dat");
assert(x_read.is_same(x_sequence));

This results in:
int main(int, char**): Assertion `x_read.is_same(x_sequence)' failed.

using

python-pytorch, version 1.6.0-2 on Arch Linux
g++ (GCC) 10.1.0


title: PyTorch: difference between type(a), a.type, a.type()

body:
suppose a is a tensor, then what's the difference between:
type(a)
a.type
a.type()
I couldn't find a document differentiating these.


title: TVM fails compiling a pytorch model in dense_strategy_cpu

body:
I made and trained a pytorch v1.4 model that predicts a sin() value (based on an example found on the web). Inference works. I then tried to compile it with TVM v0.8dev0 and llvm 10 on Ubuntu with a x86 cpu. I followed the TVM setup guide and ran some tutorials for onnx that do work.
I mainly used existing tutorials on TVM to figure out the procedure below. Note that I'm not a ML nor DataScience engineer. These were my steps:
import tvm, torch, os
from tvm import relay
state = torch.load("/home/dude/tvm/tst_state.pt") # load the trained pytorch state
import tst
m = tst.Net()
m.load_state_dict(state) # init the model with its trained state
m.eval()

sm = torch.jit.trace(m, torch.tensor([3.1415 / 4]))  # convert to a scripted model

# the model only takes 1 input for inference hence [("input0", (1,))]
mod, params = tvm.relay.frontend.from_pytorch(sm, [("input0", (1,))])
mod.astext # outputs some small relay(?) script

with tvm.transform.PassContext(opt_level=1):
  lib = relay.build(mod, target="llvm", target_host="llvm", params=params)

The last line gives me this error that I don't know how to solve nor where I went wrong. I hope that someone can pinpoint my mistake ...
    ... removed some lines here ...
  [bt] (3) /home/dude/tvm/build/libtvm.so(TVMFuncCall+0x5f) [0x7f5cd65660af]
  [bt] (2) /home/dude/tvm/build/libtvm.so(+0xb4f8a7) [0x7f5cd5f318a7]
  [bt] (1) /home/dude/tvm/build/libtvm.so(tvm::GenericFunc::CallPacked(tvm::runtime::TVMArgs, tvm::runtime::TVMRetValue*) const+0x1ab) [0x7f5cd5f315cb]
  [bt] (0) /home/tvm/build/libtvm.so(+0x1180cab) [0x7f5cd6562cab]
  File "/home/tvm/python/tvm/_ffi/_ctypes/packed_func.py", line 81, in cfun
    rv = local_pyfunc(*pyargs)
  File "/home/tvm/python/tvm/relay/op/strategy/x86.py", line 311, in dense_strategy_cpu
    m, _ = inputs[0].shape
ValueError: not enough values to unpack (expected 2, got 1)


title: Building a deep neural network that produces output that is distributed as multivariate Standard normal distribution

body:
I'm looking for a way to Build a deep neural network that produces output that is distributed as multivariate Standard normal distribution ~N(0,1).
I can use Pytorch or TensorFlow, whichever is easier for this task.
I actually have some input X, which in terms of this question can be assumed to be just a matrix of values from the uniform distribution.
I put the input into the network, whose architecture can currently change.
And I want to get output, so in addition to other requirements I will have from it. I want that if we represent the values obtained by all the possible x's, we get that it looks like a multivariate standard normal distribution ~N(0,1).
What I think needs to be done for this to happen is to choose the right loss function.
To do this, I thought of two ways:

Use of statistical tests.
A loss that tests a large number of properties (mean, standard deviation, ..).

Realizing 2 sounds complicated, so I started with 1.
I was looking for statistical tests already implemented in one of the packages as a loss function, and I did not find anything like that.
I implemented statistical tests by myself to obtain output that is univariate standard normal distribution - and it seemed to work relatively well.
With the realization of multidimensional tests I became more entangled.
Do you know of any understandable tensorflow\pythorch functions that do something similar to what I'm trying to do?
Do you have another idea for the operation?
Do you have any comments regarding the methods I try to work with?
Thanks


title: Train a GCN without any feature?

body:
This is a kind of conceptual question :)
I was using Graph convolutional network (GCN) in PyTorch to train a graph of nodes that has a couple of features, but I'm wondering if I have some of the nodes with no features (these nodes are kind of knowledge/metadata); can I still do the training on the graph including these nodes?
If not, what would be an alternative solution to keep training on them?
Any ideas?


title: Why am I getting a low error before I did any optimization?

body:
I am using a model training program I have built for a toy example and trying to use it on another example.
The only difference is this model was used for regression, hence I was using MSE as the error criterion, and now it is used for binary classification, hence I am using BCEWithLogitsLoss.
The model is very simple:
class Model(nn.Module):
    def __init__(self, input_size, output_size):
        super(Model, self).__init__()
        self.fc1 = nn.Sequential(
            nn.Linear(input_size, 8*input_size),
            nn.PReLU() #parametric relu - same as leaky relu except the slope is learned
        )
        self.fc2 = nn.Sequential(
            nn.Linear(8*input_size, 80*input_size),
            nn.PReLU()
        )
        self.fc3 = nn.Sequential(
            nn.Linear(80*input_size, 32*input_size),
            nn.PReLU()
        )
        self.fc4 = nn.Sequential(
            nn.Linear(32*input_size, 4*input_size),
            nn.PReLU()
        )
        self.fc = nn.Sequential(
            nn.Linear(4*input_size, output_size),
            nn.PReLU()
        )

    def forward(self, x, dropout=dropout, batchnorm=batchnorm):
        x = self.fc1(x)
        x = self.fc2(x)
        x = self.fc3(x)
        x = self.fc4(x)
        x = self.fc(x)

        return x

And this is where I run it:
model = Model(input_size, output_size)

if (loss == 'MSE'):
    criterion = nn.MSELoss()
if (loss == 'BCELoss'):
    criterion = nn.BCEWithLogitsLoss()

optimizer = torch.optim.SGD(model.parameters(), lr = lr)

model.train()
for epoch in range(num_epochs):
    # Forward pass and loss
    train_predictions = model(train_features)
    print(train_predictions)
    print(train_targets)

    loss = criterion(train_predictions, train_targets)

    # Backward pass and update
    loss.backward()
    optimizer.step()

    # zero grad before new step
    optimizer.zero_grad()

    train_size = len(train_features)
    train_loss = criterion(train_predictions, train_targets).item()
    pred = train_predictions.max(1, keepdim=True)[1]
    correct = pred.eq(train_targets.view_as(pred)).sum().item()
    #train_loss /= train_size
    accuracy = correct / train_size
    print('\nTrain set: Loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        train_loss, correct, train_size,
        100. * accuracy))

However, when I print the loss, for some reason the loss already starts very low (around 0.6) before I have done any backwards pass! It remains this low all subsequent epochs.
The prediction vector, however, looks like random garbage...
tensor([[-0.0447],
        [-0.0640],
        [-0.0564],
        ...,
        [-0.0924],
        [-0.0113],
        [-0.0774]], grad_fn=<PreluBackward>)
tensor([[0.],
        [0.],
        [0.],
        ...,
        [0.],
        [0.],
        [1.]])
epoch: 1, loss = 0.6842

I have no clue why is it doing that, and would appriciate any help.
Thanks!
EDIT:
I added the params if they can help anyone figuring this out:
if (dataset == 'adult_train.csv'):
    input_size=9
    print_every = 1
    output_size = 1
    lr = 0.001
    num_epochs = 10
    loss='BCELoss'

EDIT2: Added accuracy calculation in the middle block


title: Is there a depthwise constant convolutional layer option in PyTorch?

body:
I'm interested in applying a convolutional kernel that's only got HxW parameters where (H, W) is kernel size. The kernel would still have dimensions CxHxW like a normal convolution, but the parameters are constant in the channel dimension.
Is there an inbuilt option for this in PyTorch?


I am trying to implement question answering model with a BERT transformer implemented by jugapuff.
Link to the code: https://github.com/jugapuff/BERT-for-bAbi-task
After executing the main.py file which is written below as well, I m getting this error: "for tokens_tensor, segments_tensors, att_mask, pos_id, trg in data_loader: NameError: name 'data_loader' is not defined"
from dataloader import bAbi_Dataset
import torch
import torch.nn as nn
from model import model
from pytorch_transformers import AdamW

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

if torch.cuda.is_available():
    print("GPU:" + str(torch.cuda.get_device_name(0)))

my_model = model()
my_model.to(device)

optimizer = AdamW(my_model.parameters())
criterion = nn.NLLLoss()

EPOCHS = 10
for epoch in range(1, EPOCHS+1):

    my_model.train()

    train_loss = 0
    length = 0
    for tokens_tensor, segments_tensors, att_mask, pos_id, trg in data_loader:
        output = my_model(tokens_tensor.to(device), segments_tensors.to(device), att_mask.to(device), pos_id.to(device))
        loss = criterion(output, trg.to(device))
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        length+=1
        train_loss += loss.item()
        if length % 10 == 0:
          print("\t\t{:3}/25000 : {}".format(length, train_loss / length))

    epoch_loss = train_loss / length
    print("##################")
    print("{} epoch Loss : {:.4f}".format(epoch, epoch_loss))

and data_loader.py is as
import os
import torch
import torch.utils.data as data
from pytorch_transformers import BertTokenizer

def _parse( file, only_supporting=False):
        data, story = [], []
        for line in file:
            tid, text = line.rstrip('\n').split(' ', 1)
            if tid == '1':
                story = []
            if text.endswith('.'):
                story.append(text[:])
            else:
                query, answer, supporting = (x.strip() for x in text.split('\t'))
                if only_supporting:
                    substory = [story[int(i) - 1] for i in supporting.split()]
                else:
                    substory = [x for x in story if x]
                data.append((substory, query[:-1], answer))
                story.append("")
        return data

def build_trg_dics(tenK=True, path="tasks_1-20_v1-2", train=True):

    if tenK:
        dirname = os.path.join(path, 'en-10k')
    else:
        dirname = os.path.join(path, 'en')

    for (dirpath, dirnames, filenames) in os.walk(dirname):
        filenames = filenames

    if train:
        filenames = [filename for filename in filenames if  "train.txt" in filename]
    else:
        filenames = [filename for filename in filenames if  "test.txt" in filename]

    temp = []
    for filename in filenames:
        f = open(os.path.join(dirname, filename), 'r')
        parsed =_parse(f)
        temp.extend([d[2] for d in parsed])
    temp = set(temp)

    trg_word2id = {word:i for i, word in enumerate(temp)}
    trg_id2word = {i:word for i, word in enumerate(temp)}
    return trg_word2id, trg_id2word

class bAbi_Dataset(data.Dataset):

    def __init__(self, trg_word2id, tenK=True, path = "tasks_1-20_v1-2", train=True):
        # joint is Default

        self.tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')

        if tenK:
            dirname = os.path.join(path, 'en-10k')
        else:
            dirname = os.path.join(path, 'en')

        for (dirpath, dirnames, filenames) in os.walk(dirname):
            filenames = filenames

        if train:
            filenames = [filename for filename in filenames if  "train.txt" in filename]
        else:
            filenames = [filename for filename in filenames if  "test.txt" in filename]

        self.src = []
        self.trg = []

        for filename in filenames:
            f = open(os.path.join(dirname, filename), 'r')
            parsed = _parse(f)
            self.src.extend([d[:2] for d in parsed])
            self.trg.extend([trg_word2id[d[2]] for d in parsed])
        self.trg = torch.tensor(self.trg)

    def __getitem__(self, index):
        src_seq = self.src[index]
        trg = self.trg[index]
        src_seq, seg_seq, att_mask, pos_id = self.preprocess_sequence(src_seq)

        return src_seq, seg_seq, att_mask, pos_id, trg

    def __len__(self):
        return len(self.trg)

    def preprocess_sequence(self, seq):

        text =  ["[CLS]"] + list(seq[0]) + ["[SEP]"] + [seq[1]] + ["[SEP]"]

        tokenized_text = self.tokenizer.tokenize(" ".join(text))
        indexed_text = self.tokenizer.convert_tokens_to_ids(tokenized_text)
        where_is_sep = indexed_text.index(102) + 1
        segment_ids = [0 ]* (where_is_sep) + [1] * (len(indexed_text)- where_is_sep)
        attention_mask = [1] *len(indexed_text)
        pos_id = [i for i in range(len(indexed_text))]

        return torch.tensor(indexed_text), torch.tensor(segment_ids), torch.tensor(attention_mask), torch.tensor(pos_id)

def collate_fn(data):
    def merge(sequences):
        lengths = [len(seq) for seq in sequences]
        padded_seqs = torch.zeros(len(sequences), 512).long()
        for i, seq in enumerate(sequences):

            end = lengths[i]
            if end <= 512:
                padded_seqs[i, :end] = seq[:end]
            else:
                padded_seqs[i] = seq[-512:]

        return padded_seqs

    def pos_merge(sequences):

        lengths = [len(seq) for seq in sequences]
        padded_seqs = torch.zeros(len(sequences), 512).long()
        for i, seq in enumerate(sequences):

            padded_seqs[i] = torch.tensor([i for i in range(512)])

        return padded_seqs

    src_seqs, seg_seqs, att_mask, pos_id, trgs = zip(*data)
    src_seqs = merge(src_seqs)
    seg_seqs = merge(seg_seqs)
    att_mask = merge(att_mask)
    pos_id = pos_merge(pos_id)
    trgs = torch.tensor(trgs)
    return src_seqs, seg_seqs, att_mask, pos_id, trgs

data_loader variable declaration in main.py is missing. So I tried to load data_loader as
for tokens_tensor, segments_tensors, att_mask, pos_id, trg in dataloader.collate_fn(bAbi_Dataset):
use collate_fn() function in data_loader.py, but it did not work. When I change it as above, it gives the following error:
Traceback (most recent call last):
  File "main.py", line 27, in <module>
  File "/content/BERT-for-bAbi-task/dataloader.py", line 133, in collate_fn
    src_seqs, seg_seqs, att_mask, pos_id, trgs = zip(*data)
  File "/usr/lib/python3.6/typing.py", line 682, in inner
    return func(*args, **kwds)
  File "/usr/lib/python3.6/typing.py", line 1107, in __getitem__
    params = tuple(_type_check(p, msg) for p in params)
  File "/usr/lib/python3.6/typing.py", line 1107, in <genexpr>
    params = tuple(_type_check(p, msg) for p in params)
  File "/usr/lib/python3.6/typing.py", line 374, in _type_check
    raise TypeError(msg + " Got %.100r." % (arg,))
TypeError: Parameters to generic types must be types. Got 0.

Could anyone please help me how to correct the error?


title: Numpy/PyTorch funny tensor product

body:
I've got a 4 dimensional torch tensor parameter defined like this :
nn.parameter.Parameter(data=torch.Tensor((13,13,13,13)), requires_grad=True)

and four tensors with dims (batch_size,13) (or one tensor with dims (batch_size,4,13)).
I'd like to get a tensor with dims (batch_size) equal to the formula at the end of this picture :
[EDIT: I made a mistake in the first pict, I've corrected it]

I've seen in the torch documentation the function tensordot, but I can't manage to make it work by myself.


title: Pytorch model training loss does not improve. Are the logistic regression model parameters/weights not updating?

body:
I am attempting to use the following model to classify images. The training loss does not appear to converge/improve. Can you check the code and see if this may be a model issue implementing logistic regression?
The results I get for a series of 10 training epochs are:
epoch: 1, loss= -16.0369
epoch: 2, loss= -23.3950
epoch: 3, loss= -23.4226
epoch: 4, loss= -18.7254
epoch: 5, loss= -29.8720
epoch: 6, loss= -29.2601
epoch: 7, loss= -21.3710
epoch: 8, loss= -28.2535
epoch: 9, loss= -33.8465
epoch: 10, loss= -27.8332

Model code with optimizer:
class LogisticRegression(nn.Module):
    def __init__(self):
        super(LogisticRegression, self).__init__()
        self.linear = []
        self.linear.append(nn.Linear(in_features=28*28, out_features=1))
        self.linear = nn.Sequential(*self.linear)
        self.activation = nn.ReLU()

    def forward(self, x):
        y = self.activation(torch.sigmoid(self.linear(x)))
        return y

Loss and optimizer:
learn_rate = 0.01
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(params = LR_model.parameters(), lr=learn_rate)

Data loader generates "images" and "labels"
Training loop segment:
#forward
        y_predicted = LR_model(images)
        total_loss = criterion(y_predicted, labels.unsqueeze(1))
        #backward
        total_loss.backward()
        #update
        optimizer.step()
        optimizer.zero_grad()
    # Print epoch result
    print(f'epoch: {epoch+1}, loss= {total_loss.item():.4f}')


title: The size of tensor a (42) must match the size of tensor b (100) at non-singleton dimension 0

body:
The error is coming at line  correct += (predicted == labels).sum() of the code below:
class CPillModel(nn.Module):
    def __init__(self, input_dim, hidden_dim, output_dim):
        super(CPillModel, self).__init__()
        self.fc1 = nn.Linear(input_dim, hidden_dim)
        self.relu1 = nn.ReLU()

        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.tanh2 = nn.Tanh()

        self.fc3 = nn.Linear(hidden_dim, hidden_dim)
        self.elu3 = nn.ELU()

        self.fc4 = nn.Linear(hidden_dim, hidden_dim)
        self.elu4 = nn.ELU()

        self.fc5 = nn.Linear(hidden_dim, hidden_dim)
        self.elu5 = nn.ELU()

        self.fc6 = nn.Linear(hidden_dim, hidden_dim)
        self.elu6 = nn.ELU()

        self.fc7 = nn.Linear(hidden_dim, output_dim)
        #self.act7 = Sigmoid()

    def forward(self, x):

        x = self.fc1(x)
        x = self.relu1(x)

        x = self.fc2(x)
        x = self.tanh2(x)

        x = self.fc3(x)
        x = self.elu3(x)

        x = self.fc4(x)
        x = self.elu4(x)

        x = self.fc5(x)
        x = self.elu5(x)

        x = self.fc6(x)
        x = self.elu6(x)

        x = self.fc7(x)
        #x = self.act7(x)

        return x

input_dim = 9
hidden_dim = 100
output_dim = 10

# Create ANN
model = CPillModel(input_dim, hidden_dim, output_dim)

# Cross Entropy Loss
error = nn.CrossEntropyLoss()
#error = nn.MSELoss()

# SGD Optimizer
learning_rate = 0.02
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

count = 0
loss_list = []
iteration_list = []
accuracy_list = []
for epoch in range(num_epochs):
# enumerate mini batches
    for i, (data_a, data_b) in enumerate(train_loader):
        train = Variable(data_a)
        labels = Variable(data_b)
        # clear the gradients
        optimizer.zero_grad()
        # compute the model output
        outputs = model(data_a)
        # calculate loss
        loss = error(outputs, data_b)
        # credit assignment
        loss.backward()
        # update model weights
        optimizer.step()
        count += 1

        if count % 50 == 0:
            # Calculate Accuracy
            correct = 0
            total = 0
            # Predict test dataset
            for data_a, data_b in test_loader:

                test = Variable(data_a)

                # Forward propagation
                outputs = model(test)

                # Get predictions from the maximum value
                predicted = torch.max(outputs.data, 1)[1]

                # Total number of labels
                total += len(labels)

                # Total correct predictions
                correct += (predicted == labels).sum()

            accuracy = 100 * correct / float(total)

            # store loss and iteration
            loss_list.append(loss.data)
            iteration_list.append(count)
            accuracy_list.append(accuracy)
            if count % 500 == 0:
                # Print Loss
                print('Iteration: {}  Loss: {}  Accuracy: {} %'.format(count, loss.data, accuracy))

Do I need to change the predicted variable.


title: How can I add more neurons / filters to a neural network model after training?

body:
I'm interested in training both a CNN model and a simple linear feed forward model in PyTorch, and after training to add more filters -- to the CNN layers, & neurons -- to the linear model layers and the outputs (e.g. from binary classification to multiclass classification) of both. By adding them I specifically mean to keep the weights that were trained constant, and to add random initialized weights to the new, incoming weights.
There's an example of a CNN model here, and an example of a simple linear feed forward model here


title: Get the internal hidden states of an LSTM in pytorch

body: I have an LSTM with N hidden units, and when I'm using pytorch's nn.LSTM I get the output of all of the units, but the hidden and cell state of only the last unit. I want to also use the intermediate hidden states of the intermediate hidden units, what's the simplest way of doing that?


title: PyTorch: load weights from another model without saving

body:
Assume that I have two models in PyTorch, how can I load the weights of model 1 by weights of model 2 without saving the weights?
Like this:
model1.weights = model2.weights

In TensorFlow I can do this:
variables1 = model1.trainable_variables
variables2 = model2.trainable_variables
for v1, v2 in zip(variables1, variables2):
    v1.assign(v2.numpy())


title: When Inference,How do I set parameter of target with EfficientDet-Pytorch?

body:
Im studying objective detection code with efficientdet-pytorch at the first time.
I used pretrained Weight model as reference:
https://www.kaggle.com/shonenkov/inference-efficientdet
https://github.com/toandaominh1997/EfficientDet.Pytorch
https://github.com/toandaominh1997/EfficientDet.Pytorch
I tried to check efficientdet model's output...

from effdet import get_efficientdet_config, EfficientDet, DetBenchTrain
from effdet.efficientdet import HeadNet

#load sample efficientdet code

config = get_efficientdet_config('tf_efficientdet_d0')
config.image_size = [512,512]
config.norm_kwargs=dict(eps=.001, momentum=.01)

net = EfficientDet(config, pretrained_backbone=False)
checkpoint = torch.load('efficientdet_d0-d92fd44f.pth')

net.load_state_dict(checkpoint)
#net.reset_head(num_classes=1)
#net.class_net = HeadNet(config, num_outputs=config.num_classes)

net=DetBenchTrain(net, config)
print("Loaded pretrained weights")
#>>>Loaded pretrained weights"

#img.shape:[3,512,512]

net.eval()
with torch.no_grad():
    detected=net(img.unsqueeze(0))

#error　of target is below.
TypeError: forward() missing 1 required positional argument: 'target'

And refering to https://www.kaggle.com/shonenkov/inference-efficientdet
I tried below code.

def make_predictions(images, score_threshold=0.22):
    predictions = []
    with torch.no_grad():
        det = net(images, torch.tensor([1]*images.shape[0]).float())
        print(det.shape)
        for i in range(images.shape[0]):
            boxes = det[i].detach().cpu().numpy()[:,:4]
            scores = det[i].detach().cpu().numpy()[:,4]
            indexes = np.where(scores > score_threshold)[0]
            boxes = boxes[indexes]
            boxes[:, 2] = boxes[:, 2] + boxes[:, 0]
            boxes[:, 3] = boxes[:, 3] + boxes[:, 1]
            predictions.append({
                'boxes': boxes[indexes],
                'scores': scores[indexes],
            })
    return [predictions]

output=make_predictions(img.unsqueeze(0))

#error is below...
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-412-e81ace805280> in <module>
----> 1 output=make_predictions(img.unsqueeze(0))

<ipython-input-407-a6aab7dca874> in make_predictions(images, score_threshold)
      3     predictions = []
      4     with torch.no_grad():
----> 5         det = net(images, torch.tensor([1]*images.shape[0]).float())
      6         print(det.shape)
      7         for i in range(images.shape[0]):

/opt/anaconda3/envs/yohenv/lib/python3.7/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

/opt/anaconda3/envs/yohenv/lib/python3.7/site-packages/effdet/bench.py in forward(self, x, target)
    117         else:
    118             cls_targets, box_targets, num_positives = self.anchor_labeler.batch_label_anchors(
--> 119                 target['bbox'], target['cls'])
    120
    121         loss, class_loss, box_loss = self.loss_fn(class_out, box_out, cls_targets, box_targets, num_positives)

IndexError: too many indices for tensor of dimension 1

When inference with TestData ,How do I set parameter of target?
Sorry for the inconvenience,Can you give me advice?


title: How to add two separate layers on the top of one layer using pytorch?

body: I want to add two separate layers on the top of one layer (or a pre-trained model) Is that possible for me to do using Pytorch?


title: How to use PyTorch to softmax only the upper triangular elements of a matrix?

body:
Given input like:
tensor([[[1.9392, -1.9266,  0.9664],
         [0.0000, -1.9266,  0.9664],
         [0.0000, -0.0000,  0.9664]]])

My desired output is:
tensor([[[0.4596,  0.0096, 0.1737],
         [0.0000,  0.0096, 0.1737],
         [0.0000, -0.0000, 0.1737]]])

I.e. just calculating the function over the upper triangular elements.


title: how to rotate images in a batch separately in pytorch

body:
I am randomly rotating 3D images in pytorch using torch.rot90 but this rotates all the images in the batch in the same way. I would like to find a differentiable way to randomly rotate each image in a different axis.
here is the code which rotates each image to the same orientation:
#x = next batch

k = torch.randint(0, 4, (1,)).item()
dims = [0,0]
dims[0] = dims[1] = torch.randint(2, 5, (1,))
while dims[0] == dims[1]:#this makes sure the two axes aren't the same
    dims[1] = torch.randint(2, 5, (1,))

x = torch.rot90(x, k, dims)

# x is now a batch of 3D images that have all been rotated in the same random orientation


title: Why is the Databricks notebook cell training my ML model "Running command..." forever?

body:
I am training PyTorch models on a Azure Databricks cluster (on a notebook), using PyTorch Lightning, and doing the tracking using mlflow.
I would like to store training metrics + artifacts on the Databricks-hosted tracking server.
To enable that, code is as follows:
mlflow.pytorch.autolog()
trainer = pl.Trainer(gpus=1, max_epochs=30, callbacks=[EarlyStopping(monitor='val_loss', patience = 6)], progress_bar_refresh_rate=0)
trainer.fit(classifier, train_dl, valid_dl)
print("Done")

However, the notebook cell gets stuck in a "Running command..." state for way too long:

even though in the driver logs execution seems to have ended:

and the experiment is marked as FINISHED in the mlflow UI:

Stopping execution manually doesn't solve either, as the cell would stay in a "Cancelling..." state forever. So the only option left is to clear the cluster state.
This is a problem because I can't execute further commands that would be useful for artifact logging:
mapping.to_json("/tmp/mapping.json", orient = "records")
mlflow.log_artifact("/tmp/mapping.json", "mapping")

torch.save(classifier.state_dict(), "/tmp/model.pt")
mlflow.log_artifact("/tmp/model.pt", "model.pt")

This problem seems to correlate with GC problems:

2021-04-22T07:58:58.025+0000: [GC (Allocation Failure) [PSYoungGen:
28399104K->100290K(28499456K)] 28696488K->397698K(85935104K),
0.0755720 secs] [Times: user=0.15 sys=0.06, real=0.08 secs]  2021-04-22T08:01:01.645+0000: [GC (System.gc()) [PSYoungGen:
4522724K->54360K(28561920K)] 4820132K->351776K(85997568K), 0.0237712
secs] [Times: user=0.09 sys=0.01, real=0.02 secs]
2021-04-22T08:01:01.669+0000: [Full GC (System.gc()) [PSYoungGen:
54360K->0K(28561920K)] [ParOldGen: 297416K->123173K(57435648K)]
351776K->123173K(85997568K), [Metaspace: 203356K->203325K(219136K)],
0.3513905 secs] [Times: user=0.99 sys=0.00, real=0.36 secs]

Am I doing something wrong? Should I track experiments + artifacts in another way? Running neither single-node nor cluster works, nor reducing the size of the training set.


title: Expected 4-dimensional input for 4-dimensional weight [64, 1, 3, 3], but got 2-dimensional input of size [18560, 79] instead

body:
I'm trying to train on TPSSpatialTransformationNetwork. My input image size is (32,128). After running this code below, I got an error:
enter image description here
I can not figure out the shape of the tensor in the error message. Here is the code:
class TPS_SpatialTransformerNetwork(nn.Module):
    def __init__(self):
        super(TPS_SpatialTransformerNetwork, self).__init__()
        self.conv1 = nn.Conv2d(1, 79, kernel_size=5)
        self.conv2 = nn.Conv2d(79, 256, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(256, 512)
        self.fc2 = nn.Linear(512, 79)

        # Spatial transformer localization-network
        self.localization = nn.Sequential(
            nn.Conv2d(1, 8, kernel_size=7),
            nn.MaxPool2d(2, stride=2),
            nn.ReLU(True),
            nn.Conv2d(8, 79, kernel_size=5),
            nn.MaxPool2d(2, stride=2),
            nn.ReLU(True)
        )

        # Regressor for the 3 * 2 affine matrix
        self.fc_loc = nn.Sequential(
            nn.Linear(79 * 4 * 28, 32),
            nn.ReLU(True),
            nn.Linear(32, 3 * 2)
        )

        # Initialize the weights/bias with identity transformation
        self.fc_loc[2].weight.data.zero_()
        self.fc_loc[2].bias.data.copy_(torch.tensor([1, 0, 0, 0, 1, 0], dtype=torch.float))

    # Spatial transformer network forward function
    def stn(self, x):
        xs = self.localization(x)
        xs = xs.view(-1, 79 * 4 * 28)
        theta = self.fc_loc(xs)
        theta = theta.view(-1, 2, 3)

        grid = F.affine_grid(theta, x.size())
        x = F.grid_sample(x, grid)

        return x

    def forward(self, x):
        # transform the input
        x = self.stn(x)

        # Perform the usual forward pass
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
        x = x.view(-1, 256)
        x = F.relu(self.fc1(x))
        x = F.dropout(x, training=self.training)
        x = self.fc2(x)
        return F.log_softmax(x, dim=1)


title: Only backpropagate up to a given variable

body:
I’m working on a GAN where the discriminator operates on the latent space vector produced by the encoder. The details aren’t important, but the paper describing the model is https://arxiv.org/abs/1706.00409 if you want to take a look.

Essentially, my problem is that my training code requires an unnecessary backward pass through the encoder and I’m not sure how to get around this such that it becomes optimal. Here is the relevant code, where E is the encoder and D is the discriminator.

latent_vec = E(input) #latent_vector will be a Variable with requires_grad=True
predictions = D(latent_vec)

e_loss = encoder_loss(predictions, ground_truth)
e_optimizer.zero_grad()
e_loss.backward() #backpropagates through both D and E, which is necessary
e_optimizer.step()

d_loss = discriminator_loss(predictions, ground_truth)
d_optimizer.zero_grad()
d_loss.backward() #backpropagates through D, but also through E unnecessarily
d_optimizer.step()

This still works because the optimizers are only modifying the parameters of their respective models, but it’s inefficient because d_loss.backward() unnecessarily backpropagates through E. I’m aware that I can recreate a version of latent_vec that will prevent backpropagation through E using
latent_vec_no_grad = Variable(latent_vec.data), but then I would be stuck with 2 forward passes through D (one with latent_vec so that e_loss backpropagates through E, and one with latent_vec_no_grad so that d_loss only backpropagates through D).

Ideally, a flag such as latent_vec.block_backprop = True could be set after optimizing E, but no such flag exists. Is there an elegant solution that would make training optimal?


title: The correct way to build a binary classifier for CNN

body:
I created a neural network on pytorch using the pretraining model VGG16 and added my own extra layer to define belonging to one of two classes. For example bee or ant.

model = models.vgg16(pretrained=True)
# Freeze early layers
for param in model.parameters():
    param.requires_grad = False
n_inputs = model.classifier[6].in_features
# Add on classifier
model.classifier[6] = nn.Sequential(
    nn.Linear(n_inputs, 256), nn.ReLU(), nn.Dropout(0.2),
    nn.Linear(256, 2), nn.LogSoftmax(dim=1))

The model works well with two classes, but if to upload a crocodile image into it, it is likely to take it for a bee)
Now I want to make a binary classifier based on this model, which defines for example a bee or not a bee (absolutely any image without a bee)
I am just starting to understand neural networks and I need advice on whether the correct way would be training in two groups of images in one of which will only bees and in the other several thousand random images. Or should it be done in another way?


title: How do I deploy my chatbot model on google assistant?

body: I have developed my chatbot in PyTorch framework for college purposes. The model is working fine with log loss value of 0.5 and is able to answer questions appropriately. I have seen few productionization suggestions like fast.ai, flask and Django. But I want the model to be deployed on Google Assistant so that my end users can utilize the service of bot without any external installations. How do I have to integrate my PyTorch model to the Google Assistant on Dialogflow ?


title: Vectorize to Apply Function to 3d Array

body:
I am trying to apply a function to 3d torch tensor while the function is applied to 2d tensor which is read through the axis 1 of the 3d torch tensor.

For example, I have a torch tensor of the shape (51, 128, 20100) (a variable with name autoencode_logprob) and the function(rawid2sentence) runs on the input of the shape (51, 20100).

Right now I wrote the code to run with naive for loop, looping one by one with range(128).

However, it’s too slow. Following is the code part that matters.

autoencode_logprobs is the 3d tensor and I need to apply rawids2sentence function along its second axis. Any help to vectorize it?

for i in range(128):
    output_sent = self.dictionary.rawids2sentence(
        autoencode_logprobs[:, i].max(1)[
            1].data.cpu().numpy(),
        oov_dicts[i],
    )
    output_sent_encoding = ifst_model.encode([output_sent])


title: Creating a memory efficient neural network

body:
I'm creating a neural network on python, my problem is a really simple classification problem, I'm using a Leaky ReLu network, that its main goal is to over-fit to a set data points. Before I was working with small data points, my network was really fast if I gave it all the data at once. Now ever now my computer is not able to take it all at once. So I have decided to feed it mini batches of the whole data set, and now it works. Now I have several questions:

If I used a loss function before, should I use the same one? (in my case I'm using BCEWithLogitsLoss). I ask this because I get really weird loss values in each batch. For example before I made this batches change the loss function goes from being really small, like 0.005 in the model without batches. To being really big.

My input data points size 310000 x 3 more or less. Is mini batches the correct approach? Or is there another trick to tackle this problem.

Now that I have trained the data, do I have evaluate the trained model in the same way(i.e by batches so I don't run out memory on my GPU)?

Here is my code :
class CustomDataset(Dataset):
   def __init__(self, x_tensor, y_tensor):
       self.x = x_tensor
       self.y = y_tensor
   def __getitem__(self, index):
       return (self.x[index], self.y[index])
   def __len__(self):
       return len(self.x)

class LabelData(Dataset):
   def __init__(self, y_tensor):
       self.y = y_tensor
   def __getitem__(self, index):
       return (self.y[index])
   def __len__(self):
       return len(self.y)

#X, order = common.loadplanes()
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
Samples, Ocupancy = common.load_samples()
x_test = torch.from_numpy(Samples.astype(np.float32)).to(device)
y_test = torch.from_numpy(Ocupancy.astype(np.float32)).to(device)
train_data = CustomDataset(x_test, y_test)
train_loader = DataLoader(dataset=train_data, batch_size= 1000, shuffle=False) # Make bigger batches
phi = common.MLP(3, 1).to(device)
criterion = torch.nn.BCEWithLogitsLoss()
optimizer = torch.optim.Adam(phi.parameters(), lr = 0.01)
epoch = 512
fit_start_time = time.time()
for epoch in range(epoch):
   for x_batch, y_batch in train_loader:
       optimizer.zero_grad()
       x_batch = x_batch.to(device)
       y_batch = y_batch.to(device)
       y_pred = phi(x_batch)
       # Compute Loss
       loss = criterion(y_pred.squeeze(), y_batch.squeeze())
       print('Epoch {}: train loss: {}'.format(epoch, loss.item()))    # Backward pass
   loss.backward()
   optimizer.step()

fit_end_time = time.time()
print("Total time = %f" % (fit_end_time - fit_start_time))
X,Y,Z  = np.mgrid[-3000:300:80,-3000:3000:80,-3000:3000:80]
xyz = torch.from_numpy(np.vstack([X.ravel(), Y.ravel(),Z.ravel()]).transpose().astype(np.float32))
# feed the network bit by bit?
xyz = LabelData(xyz)
labels = phi(xyz).to(device)
visualization_iso(X,Y,Z,labels)

I'm still running my code as I write this, but I can imagine that the there is going to be an error after the model is trained, that is going to say that I've run out of memory(that is why I'm asking the 3rd question) or that I'm evaluating my network incorrectly, edit it seems I'm evaluating the model wrongly, you can see my error below. Please let me know if I'm not using any common practices or if you spot any mayor flaws on my code.


title: NLP multi-class classifier loss can’t go down

body:
I'm building a classifier for a QA bot, and  have a dataset for 8k questions, and 149 different Answers.

I got some problems when training my model; the "loss" won't go down as I expected so I am asking for your help...

Here is my method:

I use word2vec to get a word's vector, then use a GRU-based network to get the vector of sentence The w2v model has been trained with wiki data, and works well on another of my NLP projects. The GRU code is from my senior, I think it works well, too.

# Part of the code for getting sentence vector
input_size = 400
hidden_dim = 400
num_layers = 1
gru = nn.GRU(input_size, hidden_dim,num_layers,batch_first = True)

h0 = torch.rand(num_layers, 7187, hidden_dim) # (num_layers, batch, hidden_dim)
# shape of input [dataset_len,max_sentence_len,input_feature]
inputSet = torch.tensor(x_train,dtype = torch.float)
sentenceVecs, hidden = gru(inputSet,h0)
sentenceVecs = sentenceVecs[:,-1, :]

and here is my classifier model

from argparse import Namespace
args = Namespace(
    dataset_file = 'dataset/waimai_10k_tw.pkl',
    model_save_path='torchmodel/pytorch_bce.model',
    # Training hyper parameters
    batch_size = 100,
    learning_rate = 0.002,
    min_learning_rate = 0.002,
    num_epochs=200,
)

class JWP(nn.Module):
    def __init__(self,
                 n_feature,
                 n_hidden,
                 n_hidden2,
                 n_hidden3,
                 n_output):

        super(JWP, self).__init__()
        self.hidden = nn.Linear(n_feature, n_hidden)
        self.hidden2 = nn.Linear(n_hidden, n_hidden2)
        self.hidden3 = nn.Linear(n_hidden2, n_hidden3)
        self.out = nn.Linear(n_hidden3, n_output)

    def forward(self, x, apply_softmax=False):
        x = F.relu(self.hidden(x).squeeze())
        x = F.relu(self.hidden2(x).squeeze())
        x = F.relu(self.hidden3(x).squeeze())
        #
        if(apply_softmax):
            x = torch.softmax(self.out(x))
        else:
            x = self.out(x)

        return x

training code

lr = args.learning_rate
min_lr = args.min_learning_rate
def adjust_learning_rate(optimizer, epoch):
    global lr
    if epoch % 10 == 0 and epoch != 0:
        lr = lr * 0.65
        if(lr < min_lr):
            lr = min_lr
        for param_group in optimizer.param_groups:
            param_group['lr'] = lr

if __name__ == "__main__":
    EPOCH = args.num_epochs
    net = JWP(400,325,275,225,149)
#     net = JWP(400,250,149)
#     net = JWP(400,149)
    print(net)

    optimizer = torch.optim.SGD(net.parameters(), lr=lr)
    loss_func = torch.nn.CrossEntropyLoss()

    for t in range(EPOCH):
        adjust_learning_rate(optimizer,t)

        """
        Train phase
        """
        net.train()
        TrainLoss = 0.0
        # Train batch
        for step,(batchData, batchTarget) in enumerate(trainDataLoader):
            optimizer.zero_grad()
            out = net(batchData)
            loss = loss_func(out,batchTarget)
            TrainLoss = TrainLoss + loss
            loss.backward()
            optimizer.step()
        TrainLoss = TrainLoss / (step+1) # epoch loss

        """
        Result
        """
        print(
            "epoch:",t+1 ,
            "train_loss:",round(TrainLoss.item(),3),
            "LR:",lr
        )

Is it that my model is too simple or do I simply use the wrong method?
The loss is always stuck at around 4.6 and I can't lower it any more...

epoch: 2898 train_loss: 4.643 LR: 0.002
epoch: 2899 train_loss: 4.643 LR: 0.002
epoch: 2900 train_loss: 4.643 LR: 0.002
epoch: 2901 train_loss: 4.643 LR: 0.002


title: Segmentation fault when using the custom activation function

body:
I am trying to implement a custom activation function (the codes attached below). Before using the custom activation function, everything works well. However, as long as it is used, the server would throw the error:

Segmentation fault

The error always appears at the first epoch.

I am using

Pytorch 1.1.0
Cuda compilation tools, release 9.2, V9.2.148
the codes

def mg(x):

    c = 1.33
    b = 0.4
    p = 6.88
    input_size = x.shape
    num = torch.numel(x) # the element number of the input tensor
    x = x.view(num)

    out = torch.zeros(len(x))

    for i in range(len(x)):
    if x[i] < 0:
            out[i] = 0
        else:
            out[i] = (c * x[i]) / (1 + torch.mul(b * p, torch.pow(x[i], p)))

    out = out.view(input_size[0], input_size[1], input_size[2], input_size[3])
    return out


title: I want to split the dataset in a such a way that class ratio is maintained in Pytorch

body:
I want to split the dataset in a such a way that class ratio is maintained.
Suppose the following:

Total Image: 500, Class A 300; Class B 200
----------
Train: 300 (Class A 180, Class B 120)
----------
Valid: 100 (Class A 60, Class B 40)
----------
Test:100  (Class A 60, Class B 40)

In my case Total Train, valid and test ratio is maintained, but not the class ratio


title: How do I make a minimal and reproducible example for neural networks?

body:
I would like to know how to make a minimal and reproducible deep learning example for Stack Overflow. I want to make sure that people have enough information to pinpoint the exact problem with my code. Is it enough to just provide the traceback?

    c:\users\samuel\appdata\local\programs\python\python35\lib\site-packages\keras\engine\training_utils.py
                         in standardize_input_data(data, names, shapes, check_batch_axis, exception_prefix)
        135                             ': expected ' + names[i] + ' to have shape ' +
        136                             str(shape) + ' but got array with shape ' +
    --> 137                             str(data_shape))
        138     return data
        139

Or should I simply post the error message?

  Value Error: Error when checking input: expected dense_1_input to have shape
  (4,) but got array with shape (1,)


title: Sequence labeling for sentences and not tokens

body:
I have sentences that belong to a paragraph. Each sentence has a label.
[s1,s2,s3,…], [l1,l2,l3,…]
I understand that I have to encode each sentence using an encoder, and then use sequence labeling. Could you guide me on how I could do that, combining them?


title: Why does a torch gradient increase linearly every time the function is backpropagated?

body:
I am trying to understand how PyTorch backpropagation works, using the following code.

import torch
import numpy
x = torch.tensor(numpy.e, requires_grad=True)
y = torch.log(x)
y.backward()
print(x.grad)

The result is tensor(0.3679), as expected, which is 1 / x, which is the derivative of log(x) w.r.t. x with x = numpy.e. However, if I run the last 3 lines again WITHOUT re-assigning x, i.e. do

y = torch.log(x)
y.backward()
print(x.grad)

then I will get tensor(0.7358), which is twice the previous result.
Why does this happen?