title: How does PyTorch compute the backward pass when optimizing triplet loss?

body:
I am implementing a triplet network in Pytorch where the 3 instances (sub-networks) share the same weights. Since the weights are shared, I implemented it as a single instance network that is called three times to produce the anchor, positive, and negative embeddings. The embeddings are learned by optimizing the triplet loss. Here is a small snippet for illustration:
from dependencies import *
model = SingleSubNet() # represents each instance in the triplet net

for epoch in epochs:
        for anch, pos, neg in enumerate(train_loader):
                optimizer.zero_grad()
                fa, fp, fn = model(anch), model(pos), model(neg)
                loss = triplet_loss(fa, fp, fn)
                loss.backward()
                optimizer.step()
                # Do more stuff ...

My complete code works as expected. However, I do not understand what does the loss.backward() compute the gradient(s) in this case. I am confused because there are 3 gradients of loss is in each learning step (the gradients formulas are here). I assume the gradients are summed before performing optimizer.step(). But then it looks from the equations that if the gradients are summed, they will cancel each other out and yield zero update term. Of course, this is not true as the network learns meaningful embeddings at the end.
Thanks in advance


title: How to draw multiple PR curves to one chart for PyTorch TensorBoard?

body:
Below is the code drawing PR curves taken from https://pytorch.org/tutorials/intermediate/tensorboard_tutorial.html#assessing-trained-models-with-tensorboard
from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter()

def add_pr_curve_tensorboard(class_index, test_probs, test_preds):
    tensorboard_preds = test_preds == class_index
    tensorboard_probs = test_probs[:, class_index]

    writer.add_pr_curve(classes[class_index],
                        tensorboard_preds,
                        tensorboard_probs,
                        global_step=0)
    writer.close()

for i in range(len(classes)):
    add_pr_curve_tensorboard(i, test_probs, test_preds)

I thought if I change the first parameter tag to same value 'all_pr_curves', it would help draw all PR curves on same chart.
writer.add_pr_curve(
    'all_pr_curves',
    tensorboard_preds,
    tensorboard_probs,
    global_step=0)

But it turns out the later one will overwrite the previous one. So at the end, only the last PR curve shows.
Is there a way to draw all PR curves on same chart?


title: How to parallelize running linear regressions on a GPU with pytorch

body:
I am working on my Senior College Thesis and I need to calculate the Mean Squared Error between all possible pairs of roughly 7000 companies every year for the past 60 years, i.e. I need to run an enormous amount of regressions. The server that I have available has multiple very powerful GPU's, so I have implemented my linear regression code in pytorch. However, I am not very experienced in how to optimize my code in order to utilize the GPU to its fullest power, in particular how to run the code in parallel while on the GPU. Here is the for loop that is iterating over the company data, and I would greatly appreciate any advise or tips on how to code it such that it splits the jobs into optimal sizes.
--Note: I am aware that I am double counting pairs because (x,y)=(y,x), I still need to figure out how to implement this as well.
for index_x,x in enumerate(unique_cusip_list):
    for index_y,y in enumerate(unique_cusip_list):

        #FIXME figure out how to not duplicate values

        #FIXME figure out how to run it on the gpu

        #setting up the model parameters and inputs
        #------------------------------------------------------------#

        #adding the corresponding cusip pairs to our list
        ids_list=[x,y]
        for val in ids_list:
            nested_list_outputs[total_iteration_counter].append(val)

        #preparing data
        x_values=nested_list_returns[index_x]
        y_values=nested_list_returns[index_y]

        #storing the number of ret variables given
        nested_list_outputs[total_iteration_counter].append(len(x_values))
        nested_list_outputs[total_iteration_counter].append(len(y_values))

        #if paired data doesnt match in length reduce larger dataset to fit the other
        if len(x_values)<len(y_values):
            y_values=y_values[:len(x_values)]
        if len(x_values)>len(y_values):
            x_values=x_values[:len(y_values)]

        #convserion to tensor variables
        x_values_np=np.array(x_values,dtype=np.float32)
        x_values_np=x_values_np.reshape(-1,1)
        x_values_tensor=Variable(torch.from_numpy(x_values_np))

        y_values_np=np.array(y_values,dtype=np.float32)
        y_values_np=y_values_np.reshape(-1,1)
        y_values_tensor=Variable(torch.from_numpy(y_values_np))

        #move tensors to device
        x_values_tensor=x_values_tensor.to(device)
        y_values_tensor=y_values_tensor.to(device)

        if args.print_info:
            print('\n')
            print('Tensor shapes:')
            print(x_values_tensor.size())
            print(y_values_tensor.size())

        #defining the model
        class LinearRegression(nn.Module):
            def __init__(self,input_size,output_size):
            # super function inherits from nn.Module so that we can access everything from nn.Module
                super(LinearRegression,self).__init__()
            # Linear function
                self.linear = nn.Linear(input_dim,output_dim)

            def forward(self,x):
                return self.linear(x)

        #defining model input and outputs:
        input_dim = 1
        output_dim = 1
        model = LinearRegression(input_dim, output_dim)

        #defining loss
        mse=nn.MSELoss()

        #defining optimzation
        learning_rate = 0.01
        optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

        loss_list=[]

        num_epochs=100

        #send model to gpu
        if dev=='cuda:0':
            model.cuda()

        #training loop
        #------------------------------------------------------------#

        for i in range(num_epochs):
            # perform optimization with zero gradient
            optimizer.zero_grad()

            results = model(x_values_tensor)
            loss = mse(results, y_values_tensor)

            # calculate derivative by stepping backward
            loss.backward()

            # Updating parameters
            optimizer.step()

            # store loss
            loss_list.append(loss.data)

            # print loss
            if args.print_info:
                if(i % 10 == 0):
                    print('epoch {}, loss {}'.format(i, loss.data))

        #save loss value
        nested_list_outputs[total_iteration_counter].append(loss.data.item())

        #incriment loop counter
        total_iteration_counter+=1

end=time.time()
if args.print_outputs:
    for val in nested_list_outputs:
        print(val)


title: What does calling self within a class do?

body:
I noticed in the documentation for Pytorch Lightning, it was mentioned you can call the forward method from another method in the same class just by calling self(x). I haven't been able to find any info about how this works. I always thought you would call the method using self.forward
Evidently, it calls the forward method but how? Is there any python documentation about what's going on?
I found this at the following URL:
https://pytorch-lightning.readthedocs.io/en/stable/new-project.html
The specific code fragment is this:
def training_step(self, batch, batch_idx):
    ...
    z = self(x)


title: Pytorch: NNAPI conversion for a CNN - node kind is not in ADDER_MAP

body:
I'm trying to use the new torch.backends._nnapi.prepare.convert_model_to_nnapi() function to prepare a PyTorch CNN model for NNAPI on an Android device. I'm basically following this tutorial. Here's my code:
#Prepare the trained model for NNAPI on Android device
import sys
import os
import torch
import torch.utils.bundled_inputs
import torch.utils.mobile_optimizer
import torch.backends._nnapi.prepare
import torchvision.models.quantization.mobilenet
from pathlib import Path

#import a function from a different module that returns the model in eval mode, loading the state dict from a checkpoint file (pretrained)
from common.model import trace_model

# This script supports 3 modes of quantization:
# - "none": Fully floating-point model.
# - "core": Quantize the core of the model, but wrap it a
#    quantizer/dequantizer pair, so the interface uses floating point.
# - "full": Quantize the model, and use quantized tensors
#   for input and output.
#
# "none" maintains maximum accuracy
# "core" sacrifices some accuracy for performance,
# but maintains the same interface.
# "full" maximized performance (with the same accuracy as "core"),
# but requires the application to use quantized tensors.
#
# There is a fourth option, not supported by this script,
# where we include the quant/dequant steps as NNAPI operators.
def make_mymodel_nnapi(output_dir_path, quantize_mode):
    quantize_core, quantize_iface = {
        "none": (False, False),
        "core": (True, False),
        "full": (True, True),
    }[quantize_mode]

#get the model. Ignore layers
    model, layers = trace_model()

#Commented this section out because model I'm using doesn't have fuse_model() fxn.
    # Fuse BatchNorm operators in the floating point model.
    # (Quantized models already have this done.)
    # Remove dropout for this inference-only use case.
    #if not quantize_core:
    #    model.fuse_model()

    #assert type(model.classifier[0]) == torch.nn.Dropout

    #model.classifier[0] = torch.nn.Identity()

#create some input based on my CNN's shape
    input_float = torch.zeros(2, 272, 17, 2)
    input_tensor = input_float

    # If we're doing a quantized model, we need to trace only the quantized core.
    # So capture the quantizer and dequantizer, use them to prepare the input,
    # and replace them with identity modules so we can trace without them.
    if quantize_core:
        quantizer = model.quant
        dequantizer = model.dequant
        model.quant = torch.nn.Identity()
        model.dequant = torch.nn.Identity()
        input_tensor = quantizer(input_float)

    # Many NNAPI backends prefer NHWC tensors, so convert our input to channels_last,
    # and set the "nnapi_nhwc" attribute for the converter.
    input_tensor = input_tensor.contiguous(memory_format=torch.channels_last)
    input_tensor.nnapi_nhwc = True

    # Trace the model.  NNAPI conversion only works with TorchScript models,
    # and traced models are more likely to convert successfully than scripted.
    with torch.no_grad():
        traced = torch.jit.trace(model, input_tensor)

#ERROR IN THIS FUNCTION
#Prepare model for NNAPI
    nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(traced, input_tensor)

if __name__ == "__main__":
    for quantize_mode in ["none", "core", "full"]:
        make_mymodel_nnapi(Path(os.environ["HOME"]) / "mobilenetv2-nnapi", quantize_mode)

What I get is an error in the convert_model_to_nnapi() function:
Traceback (most recent call last):
  File "trace_model_nnapi.py", line 131, in <module>
    make_mymodel_nnapi(Path(os.environ["HOME"]) / "mobilenetv2-nnapi", quantize_mode)
  File "trace_model_nnapi.py", line 89, in make_mymodel_nnapi
    nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(traced, input_tensor)
  File "/home/nodog/.local/lib/python3.8/site-packages/torch/backends/_nnapi/prepare.py", line 169, in convert_model_to_nnapi
    ser_model, used_weights, inp_mem_fmts, out_mem_fmts = serialize_model(model, inputs)
  File "/home/nodog/.local/lib/python3.8/site-packages/torch/backends/_nnapi/serializer.py", line 1363, in serialize_model
    return _NnapiSerializer(config).serialize_model(module, inputs)
  File "/home/nodog/.local/lib/python3.8/site-packages/torch/backends/_nnapi/serializer.py", line 520, in serialize_model
    self.add_node(node)
  File "/home/nodog/.local/lib/python3.8/site-packages/torch/backends/_nnapi/serializer.py", line 641, in add_node
    raise Exception("Unsupported node kind (%r) in node %r" % (node.kind(), node))
Exception: Unsupported node kind ('aten::contiguous') in node %x.2 : Tensor = aten::contiguous(%x.1, %39) # /home/nodog/docs/asp/mymodel/common/model.py:267:0

And if I comment out the "Many NNAPI backends prefer NHWC tensors" part of the code:
Traceback (most recent call last):
  File "trace_model_nnapi.py", line 131, in <module>
    make_mymodel_nnapi(Path(os.environ["HOME"]) / "mobilenetv2-nnapi", quantize_mode)
  File "trace_model_nnapi.py", line 89, in make_mymodel_nnapi
    nnapi_model = torch.backends._nnapi.prepare.convert_model_to_nnapi(traced, input_tensor)
  File "/home/nodog/.local/lib/python3.8/site-packages/torch/backends/_nnapi/prepare.py", line 169, in convert_model_to_nnapi
    ser_model, used_weights, inp_mem_fmts, out_mem_fmts = serialize_model(model, inputs)
  File "/home/nodog/.local/lib/python3.8/site-packages/torch/backends/_nnapi/serializer.py", line 1363, in serialize_model
    return _NnapiSerializer(config).serialize_model(module, inputs)
  File "/home/nodog/.local/lib/python3.8/site-packages/torch/backends/_nnapi/serializer.py", line 520, in serialize_model
    self.add_node(node)
  File "/home/nodog/.local/lib/python3.8/site-packages/torch/backends/_nnapi/serializer.py", line 641, in add_node
    raise Exception("Unsupported node kind (%r) in node %r" % (node.kind(), node))
Exception: Unsupported node kind ('aten::size') in node %40 : int = aten::size(%x.1, %39) # /home/nodog/docs/asp/mymodel/common/model.py:269:0

It appears that 'aten::size' and 'aten::contiguous' are not included in ADDER_MAP (what looks to be a list of node types) in serializer.py. Does anyone know if I'm doing something wrong, or if some functionality just hasn't been added to the NNAPI system yet?
By the way, the CNN model I use has Conv1d, Batchnorm, Dropout, and Relu layers.
Thanks!


error installing neural_renderer_pytorch in google colab
can someone help me with this, i try to install neural_renderer_pytorch
But I can't make it, I don't know what's wrong
GPU 0: Tesla P100-PCIE-16GB (UUID: GPU-415e05e7-6236-d11b-c02d-65f7fec9509f)
!pip install neural_renderer_pytorch

Collecting neural_renderer_pytorch
  Downloading https://files.pythonhosted.org/packages/15/28/dd59b5098e60e75948ae414ee65700597217199591dffa22797f47914702/neural_renderer_pytorch-1.1.3.tar.gz
Requirement already satisfied: torch in /usr/local/lib/python3.6/dist-packages (from neural_renderer_pytorch) (1.7.0 + cu101)
Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from neural_renderer_pytorch) (1.19.5)
Requirement already satisfied: scikit-image in /usr/local/lib/python3.6/dist-packages (from neural_renderer_pytorch) (0.16.2)
Requirement already satisfied: dataclasses in /usr/local/lib/python3.6/dist-packages (from torch-> neural_renderer_pytorch) (0.8)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.6/dist-packages (from torch-> neural_renderer_pytorch) (3.7.4.3)
Requirement already satisfied: future in /usr/local/lib/python3.6/dist-packages (from torch-> neural_renderer_pytorch) (0.16.0)
Requirement already satisfied: matplotlib! = 3.0.0,> = 2.0.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image-> neural_renderer_pytorch) (3.2.2)
Requirement already satisfied: scipy> = 0.19.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image-> neural_renderer_pytorch) (1.4.1)
Requirement already satisfied: pillow> = 4.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image-> neural_renderer_pytorch) (7.0.0)
Requirement already satisfied: imageio> = 2.3.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image-> neural_renderer_pytorch) (2.4.1)
Requirement already satisfied: networkx> = 2.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image-> neural_renderer_pytorch) (2.5)
Requirement already satisfied: PyWavelets> = 0.4.0 in /usr/local/lib/python3.6/dist-packages (from scikit-image-> neural_renderer_pytorch) (1.1.1)
Requirement already satisfied: kiwisolver> = 1.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib! = 3.0.0,> = 2.0.0-> scikit-image-> neural_renderer_pytorch) (1.3 .1)
Requirement already satisfied: pyparsing! = 2.0.4,! = 2.1.2,! = 2.1.6,> = 2.0.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib! = 3.0. 0,> = 2.0.0-> scikit-image-> neural_renderer_pytorch) (2.4.7)
Requirement already satisfied: cycler> = 0.10 in /usr/local/lib/python3.6/dist-packages (from matplotlib! = 3.0.0,> = 2.0.0-> scikit-image-> neural_renderer_pytorch) (0.10.0 )
Requirement already satisfied: python-dateutil> = 2.1 in /usr/local/lib/python3.6/dist-packages (from matplotlib! = 3.0.0,> = 2.0.0-> scikit-image-> neural_renderer_pytorch) (2.8 .1)
Requirement already satisfied: decorator> = 4.3.0 in /usr/local/lib/python3.6/dist-packages (from networkx> = 2.0-> scikit-image-> neural_renderer_pytorch) (4.4.2)
Requirement already satisfied: six in /usr/local/lib/python3.6/dist-packages (from cycler> = 0.10-> matplotlib! = 3.0.0,> = 2.0.0-> scikit-image-> neural_renderer_pytorch) ( 1.15.0)
Building wheels for collected packages: neural-renderer-pytorch
  Building wheel for neural-renderer-pytorch (setup.py) ... error
  ERROR: Failed building wheel for neural-renderer-pytorch
  Running setup.py clean for neural-renderer-pytorch
Failed to build neural-renderer-pytorch
Installing collected packages: neural-renderer-pytorch
    Running setup.py install for neural-renderer-pytorch ... error
ERROR: Command errored out with exit status 1: / usr / bin / python3 -u -c 'import sys, setuptools, tokenize; sys.argv [0] = '"'" '/tmp/pip-install-2e3k5ftk/neural-renderer-pytorch/setup.py' "'"'; __file __ = '"'" '/ tmp / pip-install-2e3k5ftk / neural-renderer-pytorch / setup.py' "'"'; f = getattr (tokenize, '"'" 'open' "'"', open ) (__ file __); code = f.read (). replace ('"'" '\ r \ n' "'"', '"'" '\ n' "'"'); f.close (); exec (compile (code, __file__, '"'" 'exec' "'"')) 'install --record /tmp/pip-record-bh0wf0zd/install-record.txt --single-version-externally-managed - -compile Check the logs for full command output.


title: How to split duplicate samples to train test with no overlapping?

body:
I have a nlp datasets (about 300K samples) where there exits duplicate data. I want to split it to train test （70%-30%）, and they should have no overlapping.
For instance:
|dataset:      |   train   |     test   |
|   a          |     a     |       c    |
|   a          |     a     |       c    |
|   b          |     b     |       c    |
|   b          |     b     |            |
|   b          |     b     |            |
|   c          |     d     |            |
|   c          |     d     |            |
|   c          |           |            |
|   d          |           |            |
|   d          |           |            |

I have tired exhaustively random sample, but it too time consuming.


title: Is it advisable to use the same torch Dataset class for training and predicting?

body:
I have recently started using PyTorch and I liked it for its object-oriented style. However, I wonder what’s the best and advised workflow when predicting the model. I wanted to use a custom Dataset class I wrote and which I use for training and validating my model. This class is a map-style dataset, therefore I implement __getitem__ method to return image and target:
class CustomDataset:

    def __init__(self, ...):
        ...

    def __getitem__(self, image_id):
        ....
        return (
            torch.tensor(image, dtype=torch.float),
            torch.tensor(target, dtype=torch.long),
       )

However, when I’m using this class for predicting I don’t have any targets to return. My current workaround is something like
def __getitem__(self, image_id):
    ....
    if predict:
        return (
            torch.tensor(image, dtype=torch.float),
            np.nan,
       )
   else:
        return (
             torch.tensor(image, dtype=torch.float),
             torch.tensor(target, dtype=torch.long),
       )

However, I wonder if there’s a better way to do it. And at the same time, as it feels a bit unnatural, I started wondering if it is even advisable to use the same class for training and predicting (it should be, but the clunkiness of my solutions makes me wonder). Of course, I could not return a tuple at all, but only a first element, but this still needs if-else.


title: Why the fixed order of vocabulary in word embedding matter?

body:
When I went through this Pytorch word embedding turtorial recently, I noticed that the order of vocabulary will have an influence on the predicting result.

Here's an example code to explain the problem, which is modified from Robert Guthrie's previous code.

import torch
import torch.autograd as autograd
import torch.nn as nn
import torch.nn.functional as functional
import torch.optim as optim

torch.manual_seed(1)

CONTEXT_SIZE = 2
EMBEDDING_DIM = 4

test_sentence = r"""<s> The mathematician ran . <\s>
<s> The mathematician ran to the store . <\s>
<s> The physicist ran to the store . <\s>
<s> The philosopher thought about it . <\s>
<s> The mathematician solved the open problem . <\s>""".split()

# build a list of tuples.  Each tuple is ([ word_i-2, word_i-1 ], target word)
trigrams = [([test_sentence[i], test_sentence[i + 1]], test_sentence[i + 2])
            for i in range(len(test_sentence) - 2)]

# fix the order of vocabulary
# if the sorted() is removed, the predicting result will be unstable.
vocab = sorted(list(set(test_sentence)))
word_to_ix = {word: i for i, word in enumerate(vocab)}

class NGramLanguageModeler(nn.Module):

    def __init__(self, vocab_size, embedding_dim, context_size):
        super(NGramLanguageModeler, self).__init__()
        self.embeddings = nn.Embedding(vocab_size, embedding_dim)
        self.linear1 = nn.Linear(context_size * embedding_dim, 128)
        self.linear2 = nn.Linear(128, vocab_size)

    def forward(self, inputs):
        embeds = self.embeddings(inputs).view((1, -1))
        out = functional.relu(self.linear1(embeds))
        out = self.linear2(out)
        log_probs = functional.log_softmax(out, dim=1)
        return log_probs

loss_function = nn.NLLLoss()
model = NGramLanguageModeler(len(vocab), EMBEDDING_DIM, CONTEXT_SIZE)
optimizer = optim.SGD(model.parameters(), lr=0.01)

# train the model
for epoch in range(20):
    for context, target in trigrams:
        context_indices = [word_to_ix[w] for w in context]
        context_var = autograd.Variable(torch.LongTensor(context_indices))

        model.zero_grad()
        log_probs = model(context_var)

        loss = loss_function(log_probs, autograd.Variable(torch.LongTensor([word_to_ix[target]])))

        loss.backward()
        optimizer.step()

context_tuple = ("<s>", "The")

context_indices = [word_to_ix[w] for w in context_tuple]
context_var = autograd.Variable(torch.LongTensor(context_indices))

model.zero_grad()
log_probs = model(context_var)

sims = []
probs = []
candidates = ["philosopher", "physicist"]

# to calculate which word is closer to mathematician according to cosine similarities
related_embedding = model.embeddings(autograd.Variable(torch.LongTensor([word_to_ix["mathematician"]])))

for word in candidates:
    # Probability
    probs.append(log_probs[0][word_to_ix[word]])
    # Cosine similarity
    embedding = model.embeddings(autograd.Variable(torch.LongTensor([word_to_ix[word]])))
    sims.append(functional.cosine_similarity(embedding, related_embedding))

print("Predicted word (probability): %s" % (candidates[0] if probs[0] > probs[1] else candidates[1]))
print("Predicted word (cosine similarity): %s" % (candidates[0] if sims[0] > sims[1] else candidates[1]))

If the sorted() function of the vocab variable is removed, the result will be different. Since I already fixed the Pytorch's random seed, why the result is irreproducible?


title: Sentiment analysis using LSTM on imbalanced citation dataset

body:
I have an extremely unbalanced dataset for sentiment classification. https://cl.awaisathar.com/citation-sentiment-corpus/

Class POSITIVE:829
Class NEGATIVE:280
Class NEUTRAL: 7627

Here is my network:

Sentiment_LSTM(
  (embedding): Embedding(5491, 400)
  (lstm): LSTM(400, 512, num_layers=2, batch_first=True, dropout=0.5)
  (dropout): Dropout(p=0.5, inplace=False)
  (fc): Linear(in_features=512, out_features=3, bias=True)
  (sig): Sigmoid()
)

Loss function:

lr=0.001

criterion = nn.BCELoss()
optimizer = torch.optim.Adam(net.parameters(), lr=lr)

My accuracy is low on the small classes. How can i improve it futher?


title: How to predict a single image with Skorch?

body:
I've just created a Neural Network with Skorch to detect aircrafts on a picture and I trained it with a train dataset with the shape (40000, 64, 64, 3).

Then I tested it with a test dataset of (15000, 64, 64, 3).

module = nn.Sequential(
    nn.Conv2d(3, 64, 3),
    nn.BatchNorm2d(64),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Conv2d(64, 64, 3),
    nn.BatchNorm2d(64),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Conv2d(64, 64, 3),
    nn.BatchNorm2d(64),
    nn.ReLU(),
    nn.MaxPool2d(2),
    nn.Flatten(),
    nn.Linear(6 * 6 * 64, 256),
    nn.Linear(256, 256),
    nn.ReLU(),
    nn.Linear(256, 2),
    nn.Softmax(),
)

early_stopping = EarlyStopping(monitor='valid_loss', lower_is_better=True)
net = NeuralNetClassifier(
    module,
    max_epochs=20,
    lr=1e-4,
    callbacks=[early_stopping],
    # Shuffle training data on each epoch
    iterator_train__shuffle=True,
    device="cuda" if torch.cuda.is_available() else "cpu",
    optimizer=optim.Adam
)
net.fit(
    train_images_balanced.transpose((0, 3, 1, 2)).astype(np.float32),
    train_labels_balanced
)

Now I need to test it on 512*512 pictures, so I have a new dataset of (30, 512, 512, 3).
So I took a sliding window code, that allowed me to divide the picture in 64*64 parts.

def sliding_window(image, stepSize, windowSize):
# slide a window across the image
for y in range(0, image.shape[0], stepSize):
    for x in range(0, image.shape[1], stepSize):
        # yield the current window
        yield (x, y, image[y:y + windowSize[1], x:x + windowSize[0]])

Now I wanna be able to predict if every single 64*64 image contains an aircraft, but I don't know how to do it, as net.predict() takes a dataset as an argument (arg : dim 4)


Reading PyTorch docs  BCEWithLogitsLoss:

  where c is the class number (c > 1 for multi-label binary classification, c = 1 for single-label binary classification), nn is the number of the sample in the batch and p_cp
  c is the weight of the positive answer for the class cc .

What is the multi-label binary classification? Binary assume only two labels AFIK


title: Pytorch - Getting loss 0.000 and keyerror while inferencing

body:
my train set contains 51000 (3 channel) images.i am trying to get those images for training from csv file / dataframe containing 2 columns images and label. for example : traindataset.loc[0][0] means '/kaggle/input/alaska2-image-steganalysis/UERD/00155.jpg' which is first image directory and traindataset.loc[0][1] means '1' which is label of that image , there exist 2 labels (1,0) so it's a binary classification problem
but i was unable to figure out if there exist any bug in my code or not.
here is my code :

class decode_images(Dataset):

    def __init__(self, file):

        self.data = file

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        #print(idx)
        img_name =  self.data.loc[idx][0]
        image = Image.open(img_name)
        image = image.resize((512, 512), resample=Image.BILINEAR)
        label = self.data.loc[idx][1] #torch.tensor(self.data.loc[idx, 'label'])
        return {'image': transforms.ToTensor()(image),
                'label': label
                }

train_dataset = decode_images(traindataset)  #traindataset is a dataframe containing images and labels(0,1)

# simple model

model = torchvision.models.resnet101(pretrained=False)
model.load_state_dict(torch.load("../input/pytorch-pretrained-models/resnet101-5d3b4d8f.pth"))
num_features = model.fc.in_features
model.fc = nn.Linear(2048, 1)
device = torch.device("cuda:0")
#device = torch.device("cpu")
model = model.to(device)

data_loader = torch.utils.data.DataLoader(train_dataset, batch_size=16, shuffle=True, num_workers=4)

plist = [
         {'params': model.layer4.parameters(), 'lr': 1e-4, 'weight': 0.001},
         {'params': model.fc.parameters(), 'lr': 1e-3}
         ]

optimizer = optim.Adam(plist, lr=0.001)
scheduler = lr_scheduler.StepLR(optimizer, step_size=10)

since = time.time()
criterion = torch.nn.CrossEntropyLoss()
num_epochs = 1
for epoch in range(num_epochs):
    print('Epoch {}/{}'.format(epoch, num_epochs - 1))
    print('-' * 10)
    scheduler.step()
    model.train()
    running_loss = 0.0
    tk0 = tqdm(data_loader, total=int(len(data_loader)))
    counter = 0
    for bi, d in enumerate(tk0):
        inputs = d["image"]
        labels = d["label"].view(-1, 1)
        inputs = inputs.to(device, dtype=torch.float)
        labels = labels.to(device, dtype=torch.float)
        optimizer.zero_grad()
        with torch.set_grad_enabled(True):
            outputs = model(inputs)
            #loss = criterion(outputs, labels)
            loss = criterion(outputs, torch.max(labels, 1)[1])
            #print(loss)
            loss.backward()
            optimizer.step()
        running_loss += loss.item() * inputs.size(0)
        counter += 1
        tk0.set_postfix(loss=(running_loss / (counter * data_loader.batch_size)))
    epoch_loss = running_loss / len(data_loader)
    print('Training Loss: {:.4f}'.format(epoch_loss))

time_elapsed = time.time() - since
print('Training complete in {:.0f}m {:.0f}s'.format(time_elapsed // 60, time_elapsed % 60))
torch.save(model.state_dict(), "model.bin")

with the code above i get this output :

Training Loss: 0.0000
Training complete in 34m 53s

so my question is why loss : 0.0000

then i tried to make test set prediction like this :

class decode_images(Dataset):

    def __init__(self, csv_file):

        self.data = csv_file#pd.read_csv(csv_file)

    def __len__(self):
        return len(self.data)

    def __getitem__(self, idx):
        #print(idx)
        img_name =  self.data.loc[idx][0]
        image = Image.open(img_name)
        image = image.resize((512, 512), resample=Image.BILINEAR)
        #label = self.data.loc[idx][1] #torch.tensor(self.data.loc[idx, 'label'])
        #image = self.transform(image)
        return {'image': image}

test_loader = torch.utils.data.DataLoader(test_set, batch_size=1, shuffle=False) # test_set contains only images directory

for param in model.parameters():
    param.requires_grad = False

tk0 = tqdm(test_loader)
for i, x_batch in enumerate(tk0):
    print(i)
    print(x_batch)
    x_batch = x_batch["image"]
    pred = model(x_batch.to(device))
    sub.Label[i] = pred

and now i get this error :

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/opt/conda/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2645             try:
-> 2646                 return self._engine.get_loc(key)
   2647             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 0

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
<ipython-input-41-fd6bbd63a0bb> in <module>
      1 tk0 = tqdm(test_loader)
----> 2 for i, x_batch in enumerate(tk0):
      3     print(i)
      4     print(x_batch)
      5     x_batch = x_batch["image"]

/opt/conda/lib/python3.7/site-packages/tqdm/notebook.py in __iter__(self, *args, **kwargs)
    216     def __iter__(self, *args, **kwargs):
    217         try:
--> 218             for obj in super(tqdm_notebook, self).__iter__(*args, **kwargs):
    219                 # return super(tqdm...) will not catch exception
    220                 yield obj

/opt/conda/lib/python3.7/site-packages/tqdm/std.py in __iter__(self)
   1106                 fp_write=getattr(self.fp, 'write', sys.stderr.write))
   1107
-> 1108         for obj in iterable:
   1109             yield obj
   1110             # Update and possibly print the progressbar.

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in __next__(self)
    343
    344     def __next__(self):
--> 345         data = self._next_data()
    346         self._num_yielded += 1
    347         if self._dataset_kind == _DatasetKind.Iterable and \

/opt/conda/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    383     def _next_data(self):
    384         index = self._next_index()  # may raise StopIteration
--> 385         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    386         if self._pin_memory:
    387             data = _utils.pin_memory.pin_memory(data)

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/opt/conda/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in <listcomp>(.0)
     42     def fetch(self, possibly_batched_index):
     43         if self.auto_collation:
---> 44             data = [self.dataset[idx] for idx in possibly_batched_index]
     45         else:
     46             data = self.dataset[possibly_batched_index]

/opt/conda/lib/python3.7/site-packages/pandas/core/frame.py in __getitem__(self, key)
   2798             if self.columns.nlevels > 1:
   2799                 return self._getitem_multilevel(key)
-> 2800             indexer = self.columns.get_loc(key)
   2801             if is_integer(indexer):
   2802                 indexer = [indexer]

/opt/conda/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)
   2646                 return self._engine.get_loc(key)
   2647             except KeyError:
-> 2648                 return self._engine.get_loc(self._maybe_cast_indexer(key))
   2649         indexer = self.get_indexer([key], method=method, tolerance=tolerance)
   2650         if indexer.ndim > 1 or indexer.size > 1:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: 0


title: Pytorch: Loading sample of images using DataLoader

body:
I use standard DataLoader from torch.utils.data. I create dataset class and then build DataLoader this way:

train_dataset = LandmarksDataset(os.path.join(args.data, 'train'), train_transforms, split="train")
train_dataloader = data.DataLoader(train_dataset, batch_size=args.batch_size, num_workers=2,
                                   pin_memory=True, shuffle=True, drop_last=True)

It works perfect, but dataset is big enough - 300k of images. So it takes a lot of time for reading images on using DataLoader. So it is really wretchedly to build such big DataLoader on debug stage! I just want to test some my hypothesis and want to do it fast! I don't need to load whole dataset for this.

I'm trying to find the way How to load just a small fixed part of dataset without building dataLoader on whole dataset?
At current moment all my ideas are just create another folder, copy some part of images here and use pipeline on it. But I suppose, Pytorch is clever enough to have some builtin methods for loading just a part of images from big dataset. Can you give me advice how to?


title: Interpretation of `hidden units` in an RNN Layer (tensorflow,Pytorch)

body:
This is a general question for any of the frameworks for both RNN and LSTM.

When we use a Vanilla  or plain networks, for a single layer, such as

current_layer = torch.nn.Linear(100,125)  means that there are 125 neurons
or a single weight vector of 125 units (for each neuron) which change the incoming 100 inputs to 125 outgoing units.

Similarly if current_layer = torch.nn.Linear(125,100) says that the incoming 125 inputs will be converted to 100 outgoing vectors.

Now mu question that if I have

previous_layer = torch.nn.Embedding(10000,100)
current_layer = torch.nn.RNN(100,125)

what does this mean?? What is 100 and 125 shows in the case or RNN? Will there be 100 timestamps as inputs for each time stamps? What does the 125 or hidden_size mean in this context then??

Is it going to be a weight vector of 125 units which will be multiplied by the single float (each of the incoming inputs for a total of 100 times?) to produce a hidden_state??


title: how to save logfile in torchvision mask rcnn to plot in tensorboard

body:
I'm trying to access/save the logfile to plot the losses and other metrics displayed while training my torchvision mask rcnn model.

model = torchvision.models.detection.maskrcnn_resnet50_fpn(pretrained=True)

Epoch: [0]  [0/4]  eta: 0:01:11  lr: 0.000334  loss: 1.4949 (1.4949)  loss_classifier: 0.4294 (0.4294)  loss_box_reg: 0.3279 (0.3279)  loss_mask: 0.5124 (0.5124)  loss_objectness: 0.0564 (0.0564)  loss_rpn_box_reg: 0.1689 (0.1689)  time: 17.8587  data: 10.8542  max mem: 5687

As far as I know, the metrics are produced by the metric_logger (engine.py) and I should be able to save them by adding a custom_callback arg to my model.train()

model.train(custom_callbacks = callbacks)
for epoch in range(num_epochs):
    print ('+++++++++++++++++++++++++++')
    # train for one epoch, printing every 10 iterations
    train_one_epoch(model, optimizer, data_loader, device, epoch, print_freq=10)

at this point, nothing is working, except showing a tensorboard page with no metrics to display.
I'll be happy by just appending the losses in a list...

thanks


title: Multi-Class U-Net Produces Mismatch on Labels and Outputs Dimensions

body:
I am using a U-Net from this github (https://github.com/usuyama/pytorch-unet) and modified it for num_clas = 5. The unlabeled/background is black. But I am getting an output of this size so when I get the BCELoss, input sizes are mismatched

Dimensions of Mask Target and Output

MASK TARGET: torch.Size([4, 1, 600, 900, 3])
OUTPUT: torch.Size([4, 5, 600, 900])

U-Net architecture

import torch
import torch.nn as nn
from torchvision import models

def match_shapes(x, y):

    if x.shape[-2:] != y.shape[-2:]:
        x = F.interpolate(x, y.shape[-2:], mode='nearest')

    return x

def convrelu(in_channels, out_channels, kernel, padding):
  return nn.Sequential(
      nn.Conv2d(in_channels, out_channels, kernel, padding=padding),
      nn.ReLU(inplace=True)
  )

class ResNetUNet(nn.Module):
  def __init__(self, n_class=1):
    super().__init__()

    self.base_model = models.resnet18(pretrained=True)
    self.base_layers = list(self.base_model.children())

    self.layer0 = nn.Sequential(*self.base_layers[:3]) # size=(N, 64, x.H/2, x.W/2)
    self.layer0_1x1 = convrelu(64, 64, 1, 0)
    self.layer1 = nn.Sequential(*self.base_layers[3:5]) # size=(N, 64, x.H/4, x.W/4)
    self.layer1_1x1 = convrelu(64, 64, 1, 0)
    self.layer2 = self.base_layers[5]  # size=(N, 128, x.H/8, x.W/8)
    self.layer2_1x1 = convrelu(128, 128, 1, 0)
    self.layer3 = self.base_layers[6]  # size=(N, 256, x.H/16, x.W/16)
    self.layer3_1x1 = convrelu(256, 256, 1, 0)
    self.layer4 = self.base_layers[7]  # size=(N, 512, x.H/32, x.W/32)
    self.layer4_1x1 = convrelu(512, 512, 1, 0)

    self.upsample = nn.Upsample(scale_factor=2, mode='bilinear', align_corners=True)

    self.conv_up3 = convrelu(256 + 512, 512, 3, 1)
    self.conv_up2 = convrelu(128 + 512, 256, 3, 1)
    self.conv_up1 = convrelu(64 + 256, 256, 3, 1)
    self.conv_up0 = convrelu(64 + 256, 128, 3, 1)

    self.conv_original_size0 = convrelu(3, 64, 3, 1)
    self.conv_original_size1 = convrelu(64, 64, 3, 1)
    self.conv_original_size2 = convrelu(64 + 128, 64, 3, 1)

    self.conv_last = nn.Conv2d(64, n_class, 1)

  def forward(self, input):
    x_original = self.conv_original_size0(input)
    x_original = self.conv_original_size1(x_original)

    layer0 = self.layer0(input)
    layer1 = self.layer1(layer0)
    layer2 = self.layer2(layer1)
    layer3 = self.layer3(layer2)
    layer4 = self.layer4(layer3)

    layer4 = self.layer4_1x1(layer4)
    x = self.upsample(layer4)
    layer3 = self.layer3_1x1(layer3)
    x = match_shapes(x, layer3)
    x = torch.cat([x, layer3], dim=1)
    x = self.conv_up3(x)

    x = self.upsample(x)
    layer2 = self.layer2_1x1(layer2)
    x = match_shapes(x, layer2)
    x = torch.cat([x, layer2], dim=1)
    x = self.conv_up2(x)

    x = self.upsample(x)
    layer1 = self.layer1_1x1(layer1)
    x = match_shapes(x, layer1)
    x = torch.cat([x, layer1], dim=1)
    x = self.conv_up1(x)

    x = self.upsample(x)
    layer0 = self.layer0_1x1(layer0)
    x = torch.cat([x, layer0], dim=1)
    x = self.conv_up0(x)

    x = self.upsample(x)
    x = torch.cat([x, x_original], dim=1)
    x = self.conv_original_size2(x)

    out = self.conv_last(x)

    return out

Current Loss Function

def calc_loss(pred, target, metrics, bce_weight=0.5):
    bce = F.binary_cross_entropy_with_logits(pred, target)

    pred = F.sigmoid(pred)
    dice = dice_loss(pred, target)

    loss = bce * bce_weight + dice * (1 - bce_weight)

    metrics['bce'] += bce.data.cpu().numpy() * target.size(0)
    metrics['dice'] += dice.data.cpu().numpy() * target.size(0)
    metrics['loss'] += loss.data.cpu().numpy() * target.size(0)

    return loss


title: Pytorch and Keras different behaviour

body:
I've implemented VGG16 from scratch on both Keras and Pytorch to make sure that the implementation is exactly the same. I'm training it on CIFAR10 and the results are too different.
Here's my implementation in keras:
(x_train,y_train),(x_test,y_test)=cifar10.load_data()
x_train = x_train / 255
x_test = x_test / 255

y_train=np_utils.to_categorical(y_train,10)
y_test=np_utils.to_categorical(y_test,10)

model = Sequential()
model.add(Conv2D(input_shape=(32,32,3),filters=64,kernel_size=(3,3),padding="same"))
model.add(ReLU())
model.add(Conv2D(filters=64,kernel_size=(3,3),padding="same"))
model.add(ReLU())
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))

model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(Conv2D(filters=128, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))

model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(Conv2D(filters=256, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))

model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))

model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same"))
model.add(ReLU())
model.add(Conv2D(filters=512, kernel_size=(3,3), padding="same"))
model.add(MaxPool2D(pool_size=(2,2),strides=(2,2)))

model.add(Flatten())

model.add(Dense(units=4096,activation="relu"))
model.add(Dense(units=4096,activation="relu"))
model.add(Dense(units=10, activation="softmax"))

opt = Adam(lr=0.0001, epsilon=1e-06)
model.compile(optimizer=opt, loss=keras.losses.categorical_crossentropy, metrics=['accuracy'])

hist = model.fit(x_train, y_train,
    epochs=50,
    batch_size=100,
    validation_data=(x_test, y_test)
    )

And the same in pytorch:
num_epochs = 50
num_classes = 10
batch_size = 100
learning_rate = 0.0001

trans = transforms.ToTensor()
train_dataset = torchvision.datasets.CIFAR10(root="./dataset_pytorch", train=True, download=True, transform=trans)
test_dataset = torchvision.datasets.CIFAR10(root="./dataset_pytorch", train=False, download=True, transform=trans)

train_loader = DataLoader(dataset=train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(dataset=test_dataset, batch_size=batch_size, shuffle=False)

class ConvNet(nn.Module):
    def __init__(self):
        super(ConvNet, self).__init__()
        self.layer1 = nn.Sequential(
            nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(64, 64, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer2 = nn.Sequential(
            nn.Conv2d(64, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(128, 128, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer3 = nn.Sequential(
            nn.Conv2d(128, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(256, 256, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer4 = nn.Sequential(
            nn.Conv2d(256, 512, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.layer5 = nn.Sequential(
            nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2))
        self.fc1 = nn.Sequential(
            nn.Linear(512, 4096),
            nn.ReLU())
        self.fc2 = nn.Sequential(
            nn.Linear(4096, 4096),
            nn.ReLU())
        self.fcout = nn.Sequential(
            nn.Linear(4096, 10),
            nn.Softmax())

    def forward(self, x):
        out = self.layer1(x)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        out = self.layer5(out)
        out = out.reshape(out.size(0), -1)
        out = self.fc1(out)
        out = self.fc2(out)
        out = self.fcout(out)
        return out

def weights_init(m):
    if isinstance(m, nn.Conv2d) or isinstance(m, nn.Linear):
        nn.init.xavier_uniform_(m.weight.data)
        nn.init.zeros_(m.bias.data)

model = ConvNet()
model.apply(weights_init)

device = torch.device("cuda:0")#"cuda:0" if torch.cuda.is_available() else "cpu")
model.to(device)

# Loss and optimizer
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, eps=1e-6)

# Train the model
total_step = len(train_loader)
for epoch in range(num_epochs):
    total = 0
    correct = 0
    for i, (images, labels) in enumerate(train_loader):
        images = images.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        # Run the forward pass
        outputs = model(images)
        loss = criterion(outputs, labels)

        # Backprop and perform Adam optimisation
        loss.backward()
        optimizer.step()

        # Track the accuracy
        total += labels.size(0)
        _, predicted = torch.max(outputs.data, 1)
        correct += (predicted == labels).sum().item()

    print("Train")
    print('Epoch [{}/{}], Accuracy: {:.2f}%'
          .format(epoch + 1, num_epochs, (correct / total) * 100))

    total_test = 0
    correct_test = 0
    for i, (images, labels) in enumerate(test_loader):
        images = images.to(device)
        labels = labels.to(device)

        # Run the forward pass
        outputs = model(images)

        # Track the accuracy
        total_test += labels.size(0)
        _, predicted = torch.max(outputs.data, 1)
        correct_test += (predicted == labels).sum().item()
    print("Test")
    print('Epoch [{}/{}], Accuracy: {:.2f}%'
          .format(epoch + 1, num_epochs, (correct_test / total_test) * 100))

As you can see, I've adjusted it to use the same initializers in the network and parameters in the optimizer. However, the results are too different, as you can see in the following image.

Here's another execution of these same networks

And I've tried also adding some batch normalization after every layer (between layer and activation). It kind of solves the problem, but not completely. Here's the plot:

Is there any reasonable explanation to this behavior or have I made any mistake in the code?


title: Unable to install apex on windows, can't find compiler

body:
According to apex doc:

Windows support is experimental, and Linux is recommended. pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" .

I have no choice  but to use Windows because Cuda on wsl2 is not in a stable windows version for now I followed the line and I got the following error :

C:\Users\user\anaconda3\envs\masktextspotter\lib\site-packages\torch\utils\cpp_extension.py:277:
UserWarning: Error checking compiler version for cl: [WinError 2] Le
fichier spécifié est introuvable
warnings.warn('Error checking compiler version for {}: {}'.format(compiler, error))

Then I look into the cpp_extension file, the try element was the following:
minimum_required_version = MINIMUM_MSVC_VERSION
        print('MINIMUM_MSVC_VERSION :' + str(MINIMUM_MSVC_VERSION))

        compiler_info = subprocess.check_output(compiler, stderr=subprocess.STDOUT)

        match = re.search(r'(\d+)\.(\d+)\.(\d+)', compiler_info.decode().strip())

And I found that the compiler_info declaration was giving the error,
I have C/C++ version 19.27.29111 for cl.exe but I'm thinking the fetching of the compiler is what goes wrong, has anyone experienced this kind of behavior, I'm not understanding well the versions of visual studio so I downloaded these:
MSVC v142 - VS 2019 C++ x64/x86 Build Tools (v14.27)

SDK Windows 10 (10.0.17763.0)

MSVC v141 - VS 2017 C++ x64/x86 Build Tools (v14.16)

SDK CRT (runtime C) universel pour Windows

MSVC v140 – VS 2015 C++ Build Tools (v14.00)

Btw,
cuda version: 10.2
torch version: 1.6
python version: 3.8


title: Understanding ‘backward()’: How to code the Pytorch function ‘.backward()’ from scratch?

body: I’m a newbie learning Deep Learning, I’m stuck trying to understand what ‘.backward()’ from Pytorch does since it does pretty much most the work there. Therefor, I’m trying to understand what backward function does in detail so, I’m going to try to code what the function does step by step. Any resource that you can recommend me (Book, video, GitHub repo) to start coding the function? Thank you for time and hopefully for your help.


title: How to use Docker to run this specific program from Github?

body:
I am working on a project that requires me to recreate the results of a paper. The code used by the authors, along with instructions on how to recreate the results, is here:
https://github.com/ElementAI/N-BEATS/tree/04f56c4ca4c144071b94089f7195b1dd606072b0#datasets
The authors suggest using Docker, which I have never used before.
Here is a summary of what I've tried:
I downloaded Docker and Python, and ran the following in the command prompt (at the suggestion of a redditor):
git clone https://github.com/ElementAI/N-BEATS.git
cd N-BEATS
docker build .
After running docker build . I receieved the following error:

[2/9] RUN apt-get update -y --fix-missing && apt-get install -y --no-install-recommends software-properties-common wget curl unrar unzip libnvinfer6=6.0.1-1+cuda10.2 libnvinfer-dev=6.0.1-1+cuda10.2 libnvinfer-plugin6=6.0.1-1+cuda10.2 git && apt-get clean -y: 5 1.325 Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 5 1.334 Get:2 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB] 5 1.351 Ign:3 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 InRelease 5 1.424 Ign:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 InRelease 5 1.440 Get:5 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release [697 B] 5 1.458 Get:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release [564 B] 5 1.473 Get:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Release.gpg [836 B] 5 1.497 Get:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Release.gpg [833 B] 5 1.691 Ign:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Packages 5 1.705 Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 Packages [334 kB] 5 1.791 Get:10 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 Packages [49.5 kB] 5 1.843 Get:11 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 5 2.019 Get:12 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 5 2.197 Get:13 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB] 5 2.210 Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB] 5 3.957 Get:15 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB] 5 4.195 Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB] 5 20.05 Reading package lists... 5 20.69 E: Release file for http://security.ubuntu.com/ubuntu/dists/bionic-security/InRelease is not valid yet (invalid for another 12h 5min 13s). Updates for this repository will not be applied. 5 20.69 E: Release file for http://archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease is not valid yet (invalid for another 12h 5min 52s). Updates for this repository will not be applied. 5 20.69 E: Release file for http://archive.ubuntu.com/ubuntu/dists/bionic-backports/InRelease is not valid yet (invalid for another 12h 6min 37s). Updates for this repository will not be applied.

failed to solve with frontend dockerfile.v0: failed to build LLB: executor failed running [/bin/sh -c apt-get update -y --fix-missing && apt-get install -y --no-install-recommends software-properties-common wget curl unrar unzip libnvinfer6=6.0.1-1+cuda10.2 libnvinfer-dev=6.0.1-1+cuda10.2 libnvinfer-plugin6=6.0.1-1+cuda10.2 git && apt-get clean -y]: runc did not terminate sucessfully

My OS: Windows 10 Home
A few people from Reddit have tried to help me with this problem, but I think the problem is that they weren't following the authors' instructions on the readme.


title: Get the CNN layer output size in def init PyTorch

body:
When defining our model architecture in PyTorch, we need to specify the size of CNN output layer to feed into the nn.Linear layer. How can we find the size of this layer in the def __init__ function (not in def forward())
class model(nn.Module):

    def __init__(self,word_count,img_channel,n_out):
        super(multimodal,self).__init__()

        # CNN image encoding hyperparameters
        conv1_channel_out = 8
        conv1_kernel = 5

        pool1_size = 2

        conv2_channel_out = 16
        conv2_kernel = 16

        pool2_size = 2

        conv3_channel_out = 32
        conv3_kernel = 4

        dropout_rate = 0.1

        cnn_fc_out = 512

        comb_fc1_out = 512
        comb_fc2_out = 128

        # FNN text encoding hyperparameters
        text_fc1_out = 4096
        text_fc2_out = 512

        # Text encoding
        self.text_fc1 = nn.Linear(word_count, text_fc1_out)
        self.text_fc2 = nn.Linear(text_fc1_out, text_fc2_out)

        # Image encoding
        self.conv1 = nn.Conv2d(img_channel, conv1_channel_out, conv1_kernel)
        self.max_pool1 = nn.MaxPool2d(pool1_size)
        self.conv2 = nn.Conv2d(conv1_channel_out, conv2_channel_out, conv2_kernel)
        self.max_pool2 = nn.MaxPool2d(pool2_size)
        self.conv3 = nn.Conv2d(conv2_channel_out, conv3_channel_out, conv3_kernel)

        self.cnn_dropout = nn.Dropout(dropout_rate)
        self.cnn_fc = nn.Linear(32*24*12, cnn_fc_out)

        #Concat layer
        concat_feat = cnn_fc_out + text_fc2_out
        self.combined_fc1 = nn.Linear(concat_feat, comb_fc1_out)
        self.combined_fc2 = nn.Linear(comb_fc1_out, comb_fc2_out)
        self.output_fc = nn.Linear(comb_fc2_out, n_out)

    def forward(self, text, img):
        # Image Encoding
        x = F.relu(self.conv1(img))
        x = self.max_pool1(x)
        x = F.relu(self.conv2(x))
        x = self.max_pool2(x)
        x = F.relu(self.conv3(x))
        x = x.view(-1, 32*24*12)
        x = self.cnn_dropout(x)
        img = F.relu(self.cnn_fc(x))

        # Text Encoding
        text = F.relu(self.text_fc1(text))
        text = F.relu(self.text_fc2(text))

        # Concat the features
        concat_inp = torch.cat((text, img), 1)
        out = F.relu(self.combined_fc1(concat_inp))
        out = F.relu(self.combined_fc2(out))
        return torch.sigmoid(self.output_fc(out))

If you see above, I define the size of CNN output layer as 322412 manually
self.cnn_fc = nn.Linear(32*24*12, cnn_fc_out)
How can I avoid this? I know we might be able to call [model_name].[layer_name].in_features in def forward(), but not in def __init__()


title: backpropagation implemention

body:
I am trying to implement the backpropagation algorithm with one hideen layer network.
If I don't run with batch I succeed and everything works fine.
But when I run with batch I get that the loss function is nan after number of iterations.
if I'll replace output layer softmax with sigmoid activation
Everything works as expected.
import numpy as np
import torch
import torchvision
import matplotlib.pyplot as plt
from time import time
import torch.nn.functional as F
from torchvision import datasets, transforms
from torch import nn, optim

train_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST('/files/', train=True, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (0.1307,), (0.3081,))
                             ])),
   batch_size=64,
    shuffle=True)

test_loader = torch.utils.data.DataLoader(
  torchvision.datasets.MNIST('/files/', train=False, download=True,
                             transform=torchvision.transforms.Compose([
                               torchvision.transforms.ToTensor(),
                               torchvision.transforms.Normalize(
                                 (0.1307,), (0.3081,))
                             ])),
  batch_size=1, shuffle=True)

def sigmoid(s):
    return 1 / (1 + torch.exp(-s))

def sigmoidPrime(s):
    # derivative of sigmoid
    # s: sigmoid output
    return s * (1 - s)

def softmax(x):
    expo = torch.exp(x)
    expo_sum = torch.sum(torch.exp(x))
    return expo/expo_sum

class Neural_Network:
    def __init__(self, input_size=28*28, output_size=10, hidden_size=64):
        # parameters
        self.inputSize = input_size
        self.outputSize = output_size
        self.hiddenSize = hidden_size

        # weights
        self.W1 = torch.randn(self.inputSize, self.hiddenSize)
        self.b1 = torch.zeros(self.hiddenSize)

        self.W2 = torch.randn(self.hiddenSize, self.outputSize)
        self.b2 = torch.zeros(output_size)

    def forward(self, X):
        self.z1 = torch.matmul(X, self.W1) + self.b1
        self.h = sigmoid(self.z1)
        self.z2 = torch.matmul(self.h, self.W2) + self.b2
        return softmax(self.z2)

    def backward(self, X, y, y_hat, lr=0.1)
        batch_size = y.size(0)
        dl_dz2 = (1/batch_size)*(y_hat - y)

        dl_dh = torch.matmul(dl_dz2, torch.t(self.W2))
        dl_dz1 = dl_dh * sigmoidPrime(self.h)

        self.W1 -= lr*torch.matmul(torch.t(X), dl_dz1)
        self.b1 -= lr*torch.matmul(torch.t(dl_dz1), torch.ones(batch_size))
        self.W2 -= lr*torch.matmul(torch.t(self.h), dl_dz2)
        self.b2 -= lr*torch.matmul(torch.t(dl_dz2), torch.ones(batch_size))

        size=0
        count=0
        for batch_idx, (data, target) in enumerate(test_loader):
            size=size+target.shape[0]
            output = NN.forward(data.view(-1,784))
            for i in range(output.shape[0]):
               pred = torch.argmax(output[i]).view(1)
               if target[i] == pred:
                   count=count+1
       print(f"the accuracy of the model is {count/size}")


title: The Loss of Siamese Network doesn't decrease and accuracy low

body:
I removed the classification part of a faster-rcnn model and then added another output layer to get the feature embeddings. I trained it with siamese network and contrastive loss with cosine distance.
I trained this contrastive loss with the losses of faster rcnn together, but the contrastive loss did not decrease while the other losses decreased.
Here is the link of faster rcnn model.
What I have tried so far:

Changed the learning from 1e-3 rate to up to 1e-5.
Only trained the contrastive loss after several epochs.

Still doesn't work.
This is my head net:
class HeadNet(nn.Module):
    def __init__(self, roi_size, spatial_scale,
                 classifier):
        super(HeadNet, self).__init__()

        self.classifier = classifier
        self.roi_size = roi_size
        self.spatial_scale = spatial_scale
        self.roi_pool = RoIPool((self.roi_size, self.roi_size), self.spatial_scale)

        # self.fc_4096 = nn.Sequential(
        #     nn.Linear(512 * 7 * 7, 4096),
        #     nn.ReLU(),
        #     nn.Linear(4096, 4096),
        #     nn.ReLU()
        # )

        self.fc_embedding = nn.Linear(4096, 128)
        self.fc_cls_loc = nn.Linear(4096, 8)  # For 2 classes, foreground and background
        self.fc_score = nn.Linear(4096, 2)

        normal_init(self.fc_cls_loc, 0, 0.001)
        normal_init(self.fc_score, 0, 0.01)
        normal_init(self.fc_embedding, 0, 0.01)

    def forward(self, x, rois, roi_indices, n_pos):
        """

        :param roi_indices: batch of image
        :param x: 4D image variable.
        :param rois:
        :return: roi_cls_locs, roi_scores, embeddings for filtered positive
                 proposals of each image, format:[N,8]
                 N is the number of pos.rois

        Tips: The pos_rois are only for training. It is different by testing
        """

        # precessing the input data, for all rois
        roi_indices = at.totensor(roi_indices).float()
        rois = at.totensor(rois).float()
        indices_and_rois = t.cat([roi_indices[:, None], rois], dim=1)

        xy_indices_and_rois = indices_and_rois[:, [0, 2, 1, 4, 3]]
        indices_and_rois = xy_indices_and_rois.contiguous()
        pool = self.roi_pool(x, indices_and_rois)

        if n_pos is not None:
            pool_pos = pool[:4]
            # Get the positive part
        else:
            pool_pos = pool

        pool = pool.view(pool.size(0), -1)
        # fc_4096 = self.fc_4096(pool)
        fc_4096 = self.classifier(pool)
        pool_pos = pool_pos.view(pool_pos.size(0), -1)  # Reformat to [N,25088]
        # fc_4096_pos = self.fc_4096(pool_pos)
        fc_4096_pos = self.classifier(pool_pos)

        roi_cls_locs = self.fc_cls_loc(fc_4096)
        roi_scores = self.fc_score(fc_4096)  # [N,2]
        embeddings = self.fc_embedding(fc_4096_pos)  # Format [n_pos,128]

        return roi_scores, roi_cls_locs, embeddings

This is the network structure:
class SiameseReID(nn.Module):
    feat_stride = 16

    def __init__(self):
        super(SiameseReID, self).__init__()
        self.extractor, self.cls_layers = vgg_16()
        self.rpn = RegionProposalNetwork(512, 512, ratios=[0.5, 1, 2], anchor_scales=[8, 16, 32],
                                         feat_stride=self.feat_stride)
        self.head = HeadNet(roi_size=7, spatial_scale=(1. / self.feat_stride), classifier=self.cls_layers)
        # self.proposal_target_layer = ProposalFilter()

    def forward_once(self, im):
        """
        :param im: 4D image
        :return:
        """
        im_size = im.shape[2:]
        features = self.extractor(im)

        rpn_locs, rpn_scores, rois, roi_indices, anchor = \
            self.rpn(features, im_size, scale=1)

        roi_scores, roi_locs_reg, embeddings = self.head(features, rois, roi_indices, None)

        return rois, embeddings, roi_locs_reg, roi_scores

This is the contrastive loss:(the rest of the losses are the same as used in the link I posted above)
    def forward(self, img1, img2, bbox1, bbox2, target, scale):
        """
        :param target: 1, when positive pair; 0, when negative pair
        """
        embedding1, losses1 = self.forward_once(img1, bbox1, scale)
        embedding2, losses2 = self.forward_once(img2, bbox2, scale)

        # -------------------- Similarity Measure Loss ------------------------- #
        n_embd1 = embedding1.shape[0]
        n_embd2 = embedding2.shape[0]
        embd1 = at.totensor(embedding1)
        embd2 = at.totensor(embedding2)
        # if n_embd1 > 3:
        #     embd1 = embd1[0:3, :]
        # if n_embd2 > 3:
        #     embd2 = embd2[0:3, :]

        embd1.requires_grad = True
        embd2.requires_grad = True
        # embd1 = Variable(embd1.data, requires_grad = True)
        # embd2 = Variable(embd2.data, requires_grad=True)

        # check = t.eq(embd1[0], embd2[0])
        # check = at.tonumpy(check)
        # print(at.tonumpy(embd1[0])[np.where(check == False)])
        embd1 = l2_norm(embd1)
        embd2 = l2_norm(embd2)

        # cos = pairwise.linear_kernel(embd1.cpu(), embd2.cpu())  # Output size (n_embd1, n_embd2)
        cos = t.einsum('ik,jk->ij', embd1, embd2)
        # cos = t.matmul(embd1, embd2)
        cos = t.clamp(cos, min=1e-6, max=1 - 1e-6)

        # cos_dist = 1 - np.arccos(cos) / np.pi   # [0,1]
        # ang_dist = 1 - t.div(t.acos(cos), math.pi)
        ang_dist = t.div(t.acos(cos), math.pi)

        # ang_dist = t.sort(ang_dist)
        # if len(ang_dist) > 2:
        #     if target.item() == 1.:
        #         # mask = np.where(ang_dist > thres_l)
        #         # cos_dist = ang_dist[mask]
        #         ang_dist =
        #     elif target.item() == 0.:
        #         mask = np.where(ang_dist < thres_h)
        #         cos_dist = ang_dist[mask]

        # if cos_dist.size == 0:
        #     sm_loss = t.tensor([1 - 1e-4]).cuda()
        # else:
        avg_ang_dist = t.mean(ang_dist)
        sm_loss = 0.5 * (target.float() * avg_ang_dist +
                         (1 + (-1 * target)).float() * F.relu(1 - (avg_ang_dist + 1e-7).sqrt()).pow(2))
        # log_ang = t.log(avg_ang_dist)
        # neg_log_ang = t.log(1 - avg_ang_dist)
        # sm_loss = -(target * log_ang + (1 - target) * neg_log_ang)

        # Task Importance parameter
        # alpha_rpn = 0.5
        # alpha_roi = 0.5
        # alpha_sm = 1.0

        rpn_loc_loss_total = (losses1[0] + losses2[0]) / 2
        rpn_cls_loss_total = (losses1[1] + losses2[1]) / 2
        roi_loc_loss_total = (losses1[2] + losses2[2]) / 2
        roi_cls_loss_total = (losses1[3] + losses2[3]) / 2

        Losses = [rpn_loc_loss_total, rpn_cls_loss_total, roi_loc_loss_total, roi_cls_loss_total, sm_loss]
        # Losses = [rpn_cls_loss_total, roi_cls_loss_total, sm_loss]
        # LossTotal = alpha_rpn * (Losses[0] + Losses[1]) + alpha_roi * (Losses[2] + Losses[3]) + alpha_sm * Losses[4]
        LossTotal = Losses[4]
        Losses = Losses + [LossTotal]

        return LossTuple(*Losses)

And finally the image of training losses from epoch 4 to 8:

sm_loss is the contrastive loss.


title: How to find the (Most important) responsible Words/ Tokens/ embeddings responsible for the label result of a text classification model in PyTorch

body:
Let us suppose I have a model like:
class BERT_Subject_Classifier(nn.Module):

    def __init__(self,out_classes,hidden1=128,hidden2=32,dropout_val=0.2):
      super(BERT_Subject_Classifier, self).__init__()

      self.hidden1 = hidden1
      self.hidden2 = hidden2
      self.dropout_val = dropout_val
      self.logits = logit
      self.bert = AutoModel.from_pretrained('bert-base-uncased')
      self.out_classes = out_classes
      self.unfreeze_n = unfreeze_n # make the last n layers trainable

      self.dropout = nn.Dropout(self.dropout_val)
      self.relu =  nn.ReLU()
      self.fc1 = nn.Linear(768,self.hidden1)
      self.fc2 = nn.Linear(self.hidden1,self.hidden2)
      self.fc3 = nn.Linear(self.hidden2,self.out_classes)

    def forward(self, sent_id, mask):
      _, cls_hs = self.bert(sent_id, attention_mask=mask)
      x = self.fc1(cls_hs)
      x = self.relu(x)
      x = self.dropout(x)
      x = self.fc2(x)
      x = self.dropout(x)
      return self.fc3(x)

I train my model and for a new data point x = ['My Name is Slim Shady'], I get my label result as  3.
My Question is that how can I check which of the words in the sentence were responsible for the the classification? I mean it could be any collection of words. Is there a library or way to check the functionality? Just like shown in the paper and Tensorflow Implementation of show Attend and Tell, you can get the areas of images where the model is paying attention to. How can I do it for the Text?


title: print the value of variables after each line of code

body:
I'm trying to understand how the following PyTorch code works. To know how each function works and what they output & to know the outputted variables value and size, I'm using print() after each line of code.
s = pc()
for _ in trange(max_length):
    if self.onnx:
        outputs = torch.tensor(self.decoder_with_lm_head.run(None, {"input_ids": generated.cpu().numpy(),
                                        "encoder_hidden_states": encoder_outputs_prompt})[0][0])
        print(f'decoder output -- {outputs}')
    else:
        outputs = self.decoder_with_lm_head(input_ids=generated, encoder_hidden_states=encoder_outputs_prompt)[0]
    next_token_logits = outputs[-1, :] / (temperature if temperature > 0 else 1.0)
    print(f'next_token_logits -- {next_token_logits}')
    if int(next_token_logits.argmax()) == 1:
        print(f'next token logits argmax  -- {int(next_token_logits.argmax())}')
        break
    new_logits.append(next_token_logits)
    print(f'new_logits  { i }   -- {new_logits}')
    print(f'generated     -- {generated}')
    print(f'generated  view list   -- {set(generated.view(-1).tolist())}')
    for _ in set(generated.view(-1).tolist()):
        next_token_logits[_] /= repetition_penalty
        print(f'ext_token_logits[_]   -- {next_token_logits[_]}')
    if temperature == 0:  # greedy sampling:
        next_token = torch.argmax(next_token_logits).unsqueeze(0)
        print(f'next_token   --   {next_token}')
    else:
        filtered_logits = top_k_top_p_filtering(next_token_logits, top_k=top_k, top_p=top_p)
        next_token = torch.multinomial(F.softmax(filtered_logits, dim=-1), num_samples=1)
    generated = torch.cat((generated, next_token.unsqueeze(0)), dim=1)
    print(f'generated end    --   {generated}')
    new_tokens = torch.cat((new_tokens, next_token), 0)
    print(f'new_tokens end  -- {new_tokens}')
    i += 1
    print(f'--------------------------\n')
e = pc()
ap = e-s
print(ap)
print(timedelta(ap))
return self.tokenizer.decode(new_tokens), new_logit

my question, is there a more efficient way of tracking these values & their shape, or are there any libraries that handle this task.


title: Pytorch model running out of memory on both CPU and GPU, can’t figure out what I’m doing wrong

body:
Trying to implement a simple multi-label image classifier using Pytorch Lightning. Here's the model definition:
import torch
from torch import nn

# creates network class
class Net(pl.LightningModule):
    def __init__(self):
        super().__init__()

        # defines conv layers
        self.conv_layer_b1 = nn.Sequential(
            nn.Conv2d(in_channels=3, out_channels=32,
                      kernel_size=3, padding=1),
            nn.ReLU(),
            nn.MaxPool2d(kernel_size=2, stride=2),
            nn.Flatten(),
        )

        # passes dummy x matrix to find the input size of the fc layer
        x = torch.randn(1, 3, 800, 600)
        self._to_linear = None
        self.forward(x)

        # defines fc layer
        self.fc_layer = nn.Sequential(
            nn.Linear(in_features=self._to_linear,
                      out_features=256),
            nn.ReLU(),
            nn.Linear(256, 5),
        )

        # defines accuracy metric
        self.accuracy = pl.metrics.Accuracy()
        self.confusion_matrix = pl.metrics.ConfusionMatrix(num_classes=5)

    def forward(self, x):
        x = self.conv_layer_b1(x)

        if self._to_linear is None:
            # does not run fc layer if input size is not determined yet
            self._to_linear = x.shape[1]
        else:
            x = self.fc_layer(x)
        return x

    def cross_entropy_loss(self, logits, y):
        criterion = nn.CrossEntropyLoss()

        return criterion(logits, y)

    def training_step(self, train_batch, batch_idx):
        x, y = train_batch
        logits = self.forward(x)

        train_loss = self.cross_entropy_loss(logits, y)
        train_acc = self.accuracy(logits, y)
        train_cm = self.confusion_matrix(logits, y)

        self.log('train_loss', train_loss)
        self.log('train_acc', train_acc)
        self.log('train_cm', train_cm)

        return train_loss

    def validation_step(self, val_batch, batch_idx):
        x, y = val_batch
        logits = self.forward(x)

        val_loss = self.cross_entropy_loss(logits, y)
        val_acc = self.accuracy(logits, y)

        return {'val_loss': val_loss, 'val_acc': val_acc}

    def validation_epoch_end(self, outputs):
        avg_val_loss = torch.stack([x['val_loss'] for x in outputs]).mean()
        avg_val_acc = torch.stack([x['val_acc'] for x in outputs]).mean()

        self.log("val_loss", avg_val_loss)
        self.log("val_acc", avg_val_acc)

    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=0.0008)

        return optimizer

The issue is probably not the machine since I'm using a cloud instance with 60 GBs of RAM and 12 GBs of VRAM. Whenever I run this model even for a single epoch, I get an out of memory error. On the CPU it looks like this:
RuntimeError: [enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 1966080000 bytes. Error code 12 (Cannot allocate memory)
and on the GPU it looks like this:
RuntimeError: CUDA out of memory. Tried to allocate 7.32 GiB (GPU 0; 11.17 GiB total capacity; 4.00 KiB already allocated; 2.56 GiB free; 2.00 MiB reserved in total by PyTorch)

Clearing the cache and reducing the batch size did not work. I'm a novice so clearly something here is exploding but I can't tell what. Any help would be appreciated.
Thank you!


title: Strange Cuda out of Memory behavior in Pytorch

body:
Edit: SOLVED- Problem relied on the number of workers, lowered them, problem solved
I am using a 24GB Titan RTX and I am using it for an image segmentation Unet with Pytorch,
it is always throwing Cuda out of Memory at different batch sizes, plus I have more free memory than it states that I need, and by lowering batch sizes, it INCREASES the memory it tries to allocate which doesn't make any sense.
here is what I tried:
Image size = 448, batch size = 8

"RuntimeError: CUDA error: out of memory"

Image size = 448, batch size = 6

"RuntimeError: CUDA out of memory. Tried to allocate 3.12 GiB (GPU 0; 24.00 GiB total capacity; 2.06 GiB already allocated; 19.66 GiB free; 2.31 GiB reserved in total by PyTorch)"

is says it tried to allocate 3.12GB and I have 19GB free and it throws an error??
Image size = 224, batch size = 8

"RuntimeError: CUDA out of memory. Tried to allocate 28.00 MiB (GPU 0; 24.00 GiB total capacity; 2.78 GiB already allocated; 19.15 GiB free; 2.82 GiB reserved in total by PyTorch)"

Image size = 224, batch size = 6

"RuntimeError: CUDA out of memory. Tried to allocate 344.00 MiB (GPU 0; 24.00 GiB total capacity; 2.30 GiB already allocated; 19.38 GiB free; 2.59 GiB reserved in total by PyTorch)"

reduced batch size but tried to allocate more ???
Image size = 224, batch size = 4

"RuntimeError: CUDA out of memory. Tried to allocate 482.00 MiB (GPU 0; 24.00 GiB total capacity; 2.21 GiB already allocated; 19.48 GiB free; 2.50 GiB reserved in total by PyTorch)"

Image size = 224, batch size = 2

"RuntimeError: CUDA out of memory. Tried to allocate 1.12 GiB (GPU 0; 24.00 GiB total capacity; 1.44 GiB already allocated; 19.88 GiB free; 2.10 GiB reserved in total by PyTorch)"

Image size = 224, batch size = 1

"RuntimeError: CUDA out of memory. Tried to allocate 1.91 GiB (GPU 0; 24.00 GiB total capacity; 894.36 MiB already allocated; 20.94 GiB free; 1.03 GiB reserved in total by PyTorch)"

Even with stupidly low image sizes and batch sizes...


title: How to pad zeros on Batch, PyTorch

body:
Is there a better way to do this? How to pad tensor with zeros, without creating new tensor object? I need inputs to be of the same batchsize all the time, so I want to pad inputs that are smaller than batchsize with zeros. Like padding zeros in NLP when sequence length is shorter, but this is padding for batch.
Currently, I create a new tensor, but because of that, my GPU will go out of memory. I don't want to reduce batchsize by half to handle this operation.
import torch
from torch import nn

class MyModel(nn.Module):
    def __init__(self, batchsize=16):
        super().__init__()
        self.batchsize = batchsize

    def forward(self, x):
        b, d = x.shape

        print(x.shape) # torch.Size([7, 32])

        if b != self.batchsize: # 2. I need batches to be of size 16, if batch isn't 16, I want to pad the rest to zero
            new_x = torch.zeros(self.batchsize,d) # 3. so I create a new tensor, but this is bad as it increase the GPU memory required greatly
            new_x[0:b,:] = x
            x = new_x
            b = self.batchsize

        print(x.shape) # torch.Size([16, 32])

        return x

model = MyModel()
x = torch.randn((7, 32)) # 1. shape's batch is 7, because this is last batch, and I dont want to "drop_last"
y = model(x)
print(y.shape)


title: Conda Environment Has Two Pythons Installed

body:
I followed the install instructions for PyTorch 1.0, in a Conda environment called pytorch1:

(pytorch1) me@comp:~$ conda install pytorch torchvision cudatoolkit=10.0 -c pytorch

It claimed to be a successful install, and the log showed pytorch-1.0.1 and python-3.7.2, but when I opened ipython I encountered the following:

Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import torch
In [2]: torch.__version__
Out[2]: '0.4.1.post2'

Which is, of course, the wrong version of PyTorch.

I can open python3.7 and the correct PyTorch is installed:

(pytorch1) me@comp:~$ python3.7
Python 3.7.2 (default, Dec 29 2018, 06:19:36)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.0.1.post2'

I'd like to use PyTorch with my current Jupyter notebook setup. Is there a way to get this working so that when I open iPython in the pytorch1 environment, I can import torch to make version 1 available?

UPDATE: I updated Anaconda and now the Python version is right but iPython can't find PyTorch


title: Pytorch - How to efficiently form this Transform tensor?

body:
I have a Set of Rotation Matrices Rs:

Rs.shape = [62x3x3]

And a Set of Translation Components Js:

Js.shape = [62x3]

I have been trying to find an efficient way to combine them into a [62x4x4] matrix which is 62 homogenous transform matrices. Currently I am doing it with a stupid for loop:

def make_A(R, t):
    R_homo = torch.cat([R, torch.zeros(1, 3).cuda()], dim = 0)
    t_homo = torch.cat([t.view(3,1), torch.ones(1, 1).cuda()], dim = 0)
    return torch.cat([R_homo, t_homo], dim=1)

transforms = self.NUM_JOINTS*[None]
for idj in range(0, self.NUM_JOINTS):
    transforms[idj] = make_A(Rs[idj, :], Js[idj,:])

FinalMatrix = torch.stack(transforms, dim=0)

This is highly inefficient, and takes almost 10ms to form. How can I tensorize this?


title: How to fix the error "no model named odoo"

body:
After creating a module, I am ready to create a model and when I put from odoo import models, api I see an error no model named odoo, could you help me to solve the error please.

I've been searching for possible solutions in Google for a long time and I can not find the solution.

from odoo import model, api


title: pytorch TabularDataset: can't load a large file?

body:
I have this training set of about 1.4 GB, but I can't load it in a TabularDataset.

>>> REF = data.Field(lower=True, tokenize=tokenize_char, init_token='<sos>',eos_token='<eos>')
>>> SRC = data.Field(lower=True, tokenize=tokenize_char)
>>> train = data.TabularDataset('./train.csv', format='csv', fields= [('src', SRC), ('ref', REF)])
Killed

For small dataset, this code works fine. I don't really understand why the process got killed, since the dataset is not enormous large. Any idea how I can load this file? thanks!


title: Running a python file to train GAN using reinforcement learning

body:
I'm reading the Deep Reinforcement Learning Hands-On book by Maxim Lapan but have trouble running code here. I get the following error when running this file. I'm new to ubuntu's command line and PyTorch but I'm familiar to Deep Learning. Please help me solve this problem. Thank you!

I tried to run this file using python 03_atari_gan.py

How do I pass 'mode' while running the program?
Thanks!

  INFO: Making new env: Breakout-v0 INFO: Making new env: AirRaid-v0
  Traceback (most recent call last):   File "03_atari_gan.py", line 138,
  in
      envs = [InputWrapper(gym.make(name)) for name in ('Breakout-v0', 'AirRaid-v0', 'Pong-v0')]   File "03_atari_gan.py", line 138, in

      envs = [InputWrapper(gym.make(name)) for name in ('Breakout-v0', 'AirRaid-v0', 'Pong-v0')]   File
  "/home/sandesh/rl/lib/python3.6/site-packages/gym/envs/registration.py",
  line 156, in make
      return registry.make(id, kwargs)   File "/home/sandesh/rl/lib/python3.6/site-packages/gym/envs/registration.py",
  line 101, in make
      env = spec.make(**kwargs)   File "/home/sandesh/rl/lib/python3.6/site-packages/gym/envs/registration.py",
  line 73, in make
      env = cls(_kwargs)   File "/home/sandesh/rl/lib/python3.6/site-packages/gym/envs/atari/atari_env.py",
  line 69, in init
      self.seed()   File "/home/sandesh/rl/lib/python3.6/site-packages/gym/envs/atari/atari_env.py",
  line 98, in seed
      ).format(self.game_mode, self.game, modes) AssertionError: Invalid game mode "0" for game air_raid. Available modes are: [1 2 3 4 5 6 7
  8]


title: CNN: taking the most confident prediction among many

body:
I'm training a CNN for image classification. The same object (with the same label then) is present in the test set twice (like two view-point). I'd like to take advantage of this when predicting the class.

Right now the final layer is a Linear layer (PyTorch) and I'm using cross-entropy as loss function. I was wondering what is the best way to take the most confident prediction for each object. Should I first compute the LogSoftMax and take the class with the highest probability (among both arrays of predictions), or should I take the logits directly?


title: Pytorch - multiple GPUs

body:
I have a rather large network that requires lots of GPU memory. I have 4 GPUs and I use nn.DataParallel. When I define a batch size of 4 (1 input for each GPU) my GPUs run out of memory. I tried torch.cuda.empty_cache() and removing any unnecessary data from the GPUs, but it was not enough.

I am looking for a way to treat the 4 GPUs as 2 GPUs with double memory, so that I could use a batch size of 2 (1 input for two GPUs). Is there any way to do so?

Thanks.


title: Running out of memory for a Fully Connected Layer for a 3D Tensor (PyTorch)

body:
I am new to Pytorch and am trying to implement a 3D neural network.

I have a Pytorch tensor of size 112 (filters) x 20 x 20 x 20. I must pass this through a Fully Connected Layer, consisting of 128 nodes. The output should therefore be a 128 (filters) x 20 x 20 x 20 tensor.

Naively, I would flatten the tensor and pass it through a Pytorch Linear layer. However, Pytorch wants to allocate 3417GB of RAM for this layer, which I (perhaps obviously) do not have.

The minimum working example is shown below:

import torch
import torch.nn as nn

test = torch.randn(112, 20, 20, 20)
test = test.view(-1, 112*20*20*20)

layer = nn.Linear(112*20*20*20, 128*20*20*20)

out = layer(test)

What I would like to do is apply a Fully Connected Layer to each 20x20x20 element. This would then consist of 20x20x20 fully connected layers, applied independently to each element. However, I have no idea how to implement this.

It is very likely I have misunderstood some aspect of this. How is it that I can build this layer?


title: Empty state_dict with vector or tuple of layers in nn.Module

body:
I switched to using a Version with a parametrized number of layers of torch.nn.Module like Net_par below, only to find out all the saved state_dicts were empty after quite some optimizing.^^

This method is the recommended saving operation (https://pytorch.org/docs/stable/notes/serialization.html#recommend-saving-models), still layers stored in a vector (or tuple, for that matter) are discarded when constructing the state_dict.

torch.save works properly in contrast, but adds to data and limits robustness. This feels a little like a bug, can anybody help with a workaround?

Minimal example for comparison between parametrized and fixed layer count:

import torch
import torch.nn as nn

class Net_par(nn.Module):
    def __init__(self,layer_dofs):
        super(Net_par, self).__init__()
        self.layers=[]
        for i in range(len(layer_dofs)-1):
            self.layers.append(nn.Linear(layer_dofs[i],layer_dofs[i+1]))
    def forward(self, x):
        for i in range(len(self.layers)-1):
            x = torch.tanh(self.layers[i](x))
        return torch.tanh(self.layers[len(self.layers)-1](x))

class Net_sta(nn.Module):
    def __init__(self,dof1,dof2):
        super(Net_sta, self).__init__()
        self.layer=nn.Linear(dof1,dof2)
    def forward(self, x):
        return torch.tanh(self.layer1(x))

if __name__=="__main__":
    net_par=Net_par((3,4))
    net_sta=Net_sta(3,4)
    print(str(net_par.state_dict()))
    #OrderedDict()                 <------Why?!
    print(str(net_sta.state_dict()))
    #OrderedDict([('layer.weight', tensor([[...
    #   ...]])), ('layer.bias', tensor([...   ...]))])


title: Understanding the mathematics of batch training in neural networks

body:
When I am doing batch training, say Mini-batch Gradient Descent for example, how does the network feed itself with the batch data? To be specific, at each step (when there is one batch as input), does the network

1) take the whole batch (a fixed number of images, i.e. a 4D input tensor) per step and process one example (an image) to calculate gradients which will be averaged after iterating the batch or

2) process the whole batch and calculate the gradients directly?

A simple example of CNN on MNIST job (with some unnecessary codes removed):

import torch
import torch.nn as nn
import torch.utils.data as Data     # Batch training
import torchvision                  # Dataset API
# import matplotlib.pyplot as plt

# Hyper parameters
EPOCH = 1
BATCH_SIZE = 50
LR = 0.001
DOWNLOAD_MNIST = False

# Download dataset
train_data = torchvision.datasets.MNIST(
    root='./mnist',
    train=True,  # False: test data
    transform=torchvision.transforms.ToTensor(),    # value(0,1)
    download=DOWNLOAD_MNIST
)

# Batch data loader
train_loader = Data.DataLoader(
    dataset=train_data,
    batch_size=BATCH_SIZE,
    shuffle=True,
    num_workers=2
)

# Test data
test_data = torchvision.datasets.MNIST(
    root='./mnist',
    train=False
)
test_x = torch.unsqueeze(test_data.data, dim=1).type(
    torch.FloatTensor)[:2000]/255
test_y = test_data.targets[:2000]

# CNN
class CNN(nn.Module):
    def __init__(self):
        super(CNN, self).__init__()
        self.conv1 = nn.Sequential(
            nn.Conv2d(  # (1,28,28)
                in_channels=1,  # Gray
                out_channels=16,    # Silmultaneous filters
                kernel_size=5,
                stride=1,
                # Filled with zeros if stride=1,padding=(kernel_size-1)/2
                padding=2,
            ),    # -> (16,28,28)
            nn.ReLU(),  # -> (16,28,28)
            nn.MaxPool2d(kernel_size=2),    # -> (16,14,14)
        )
        self.conv2 = nn.Sequential(   # (16,14,14)
            nn.Conv2d(16, 32, 5, 1, 2),  # -> (32,14,14)
            nn.ReLU(),  # -> (32,14,14)
            nn.MaxPool2d(2)  # -> (32,7,7)
        )
        self.out = nn.Linear(32*7*7, 10)  # Flattened -> 10 classes

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)   # (batch,32,7,7)
        x = x.view(x.size(0), -1)   # (batch,32*7*7)
        output = self.out(x)
        return output

if __name__ == '__main__':
    cnn = CNN()
    # print(cnn)
    optimizer = torch.optim.Adam(cnn.parameters(), lr=LR)
    loss_func = nn.CrossEntropyLoss()   # The target is not one-hot
    loss_his = []

    # Training
    for epoch in range(EPOCH):
        for step, (x, y) in enumerate(train_loader):  # Extract tensors
            output = cnn(x)
            loss = loss_func(output, y)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
            loss_his.append(loss.data)

Personally, I prefer the first description and I think the neural network can not take in a 2D tensor at a time. Am I correct in this stuff?


title: Pytorch Iterating over dataloader twice

body:
Let's say I have a dataloader (loader) which returns data of the following shape : [BatchSize , 1024]
I have 10000 samples and I want to build a score matrix of the shape (10000 , 10000) such that entry [i,j] denotes the score between sample i and sample j, so far I did it in the following way (setting batchsize = 1)

   d = numpy.zeros((10000 , 10000))
   for i, sample_i  in tqdm(enumerate(loader)):
     for i, sample_j  in tqdm(enumerate(loader)):
       d[i][j] = calculate_somre_score(sample_i , sample_j)

But this takes a lot of time and i'm sure there is some good way to do it whihc takes less time.
Thanks
Best


title: 'Sequential' object has no attribute 'classifier'

body: How to find in_features of a Pytorch model? model.classifier.in_features is working on densenet121 but nit on vgg18, is there any function which may work on all torchvision models?


title: how to get the input and final output of a pytorch network

body:
is there an API of how to get the intput and output node of a pytorch network ?
I tried model.features(), but this won't help.

Example: I get a pytorch network, and its structure in netron:
network

The Conv2d, MaxPool2d and Linear can be easily parsed. I get trouble with getting the information like name and size of the input node and output node.


title: Pytorch: Backpropogate more than one loss

body:
I want to backpropogate more than one sample. That means more than one loss in PyTorch. I want to do that at a specific timestamp.
I am trying to do that:

        losso = 0
        for g, logprob in zip(G, self.action_memory):
            losso += -g * logprob
        self.buffer.append(losso)

        if (self.game_counter > self.pre_training_games):
            for element in self.buffer:
                self.policy.optimizer.zero_grad()
                element.backward(retain_graph=True)
                self.policy.optimizer.step()

But I got a Runtime Error:

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [91, 9]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).


title: Why my deep nerual net descends slowly with softmax in the fully connected layer instead of without softmax in the fully connected layer?

body:
I am just building a deep nerual networks, and I find my network converge faster when there is no activation function (softmax) in fully connected layer. But when I add this softmax function, the convergence is really bad and even stop in really high loss. Btw. I use the cross-entropy loss as my loss function and rmsprop as my optimizer.
Nerual network without last softmax activation


title: How to save a model in PyTorch?

body:
Say we have a model that we can validate like this and that Model inherits torch.nn.Module
def validate(logger, config, valid_loader, model, criterion, epoch, main_proc):
    meters = AverageMeterGroup()
    model.eval()

    with torch.no_grad():
        for step, (x, y) in enumerate(valid_loader):
            x, y = x.cuda(non_blocking=True), y.cuda(non_blocking=True)
            logits, _ = model(x)
            loss = criterion(logits, y)
            prec1, prec5 = utils.accuracy(logits, y, topk=(1, 5))
            metrics = {"prec1": prec1, "prec5": prec5, "loss": loss}
            metrics = utils.reduce_metrics(metrics, config.distributed)
            meters.update(metrics)

            if main_proc and (step % config.log_frequency == 0 or step + 1 == len(valid_loader)):
                logger.info("Epoch [%d/%d] Step [%d/%d]  %s", epoch + 1, config.epochs, step + 1, len(valid_loader), meters)

    if main_proc:
        logger.info("Train: [%d/%d] Final Prec@1 %.4f Prec@5 %.4f", epoch + 1, config.epochs, meters.prec1.avg, meters.prec5.avg)
    return meters.prec1.avg, meters.prec5.avg

How to change validation procedure in order to save model after validation into file system so that it would be runnable on other data?


title: Extract tensor from string

body:
Is it possible to extract directly the tensor included in this string tensor([-1.6975e+00,  1.7556e-02, -2.4441e+00, -2.3994e+00, -6.2069e-01])? I'm looking for some tensorflow or pytorch function that can do it, like the ast.literal_eval function does for dictionaries and lists.
If not, could you provide a pythonic method, please?
I'm thinking about something like this:
tensor_list = "tensor([-1.6975e+00,  1.7556e-02, -2.4441e+00, -2.3994e+00, -6.2069e-01])"
str_list = tensor_list.replace("tensor(", "").replace(")", "")
l = ast.literal_eval(str_list)
torch.from_numpy(np.array(l))

But I'm not sure this is the best way.


title: PyTorch: 'ToTensor()' turns color image into 9 grayscale pictures

body:
I have found that when I use 'ToTensor' to a images, one image becomes 9 displayed.I checked the official documents but couldn't find the reason. so why a picture become 9 pictures???questioon as the following figure.
a = plt.imread('test.jpg')
plt.imshow(a)
plt.show()

transform = transforms.Compose([transforms.ToTensor()])
b = transform(a)
b = b.view(375,500,3)
plt.imshow(b)


title: Get half of rows of column - Array slicing with python/pandas

body:
I would like to get exactly half of all rows in the first column.
tensor([[0.8831, 0.5263],
        [0.5528, 0.3027],
        [0.4942, 0.0996],
        [0.5346, 0.8010],
        [0.1299, 0.0558]
        [0.3020, 0.2002]])

I would like to select the first column like:
print(x[(0:-1,0])

The -1 will leave out the last entry out. I would like to add a +1.
print(x[(0:(-1+1),0])

And now I would like to divide it by 2 so I get exactly the half of the column.
print(x[(0:(-1+1)/2,0])

So that im getting:
tensor([0.8831, 0.5528, 0.4942])


title: Do evaluation without pausing the main training process in Pytorch

body:
Usually, we train model and eval it in the same procedure.
...
for epoch_idx range(max_epoch):
    net.train()
    # do training with batchsize = N
    for train_img, label in train_loader:
        ...
        pred = net(train_img)
        loss = criterion(pred, label)
        optim.zero_grad()
        loss.backward()
        optim.step()
   # eval
   if epoch_idx % eval_step == 0:
       net.eval()
       # do evaluation with batchsize =1

However, for quite a lot tasks that are favor of cropped images with fixed size when performing training, yet when it comes to evaluation, each image varies with each other in size and so the batchsize has to be 1.
In my case, the evaluation time is about 10x longer than that of training, which makes the whole procedure slow. The model is large and I do not want to store it for each epoch.
I wonder whether it is practicable: create a new thread and perform the evaluation, so that the main thread continues trianing without suspending.
I am not familar with multi-processing, so any help will be appreciating.