Loss Overview

All multi-vector losses embed both sides into token-level embeddings and score them with late interaction inside the loss. The scoring is swappable: the ranking and KD losses accept a similarity_fct= kwarg that defaults to ColBERT-style MaxSim (colbert_scores()), and can be switched to XTR-style global top-k scoring by passing XTRScores (or XTRKDScores for the KD loss). MultiVectorMarginMSELoss scores matched pairs instead, with a pairwise similarity_fct (default colbert_scores_pairwise(), or xtr_scores_pairwise() for XTR).

Loss Table

Loss functions play a critical role in the performance of your fine-tuned model. Sadly, there is no “one size fits all” loss function. Ideally, this table should help narrow down your choice of loss function(s) by matching them to your data formats.

Note

You can often convert one training data format into another, allowing more loss functions to be viable for your scenario. For example, (anchor, positive) pairs can be extended into (anchor, positive, negative) triplets by mining hard negatives with a first-stage retriever.

Legend: Loss functions marked with ★ are commonly recommended default choices.

Inputs	Labels	Appropriate Loss Functions
`(anchor, positive) pairs`	`none`	`MultiVectorMultipleNegativesRankingLoss` ★ `CachedMultiVectorMultipleNegativesRankingLoss` ★
`(anchor, positive, negative) triplets`	`none`	`MultiVectorMultipleNegativesRankingLoss` ★ `CachedMultiVectorMultipleNegativesRankingLoss` ★
`(anchor, positive, negative_1, ..., negative_n)`	`none`	`MultiVectorMultipleNegativesRankingLoss` ★ `CachedMultiVectorMultipleNegativesRankingLoss` ★

CachedMultiVectorMultipleNegativesRankingLoss is a drop-in replacement for MultiVectorMultipleNegativesRankingLoss adopting GradCache: it computes and caches the embedding gradients in mini-batches, allowing much larger effective batch sizes without additional GPU memory. In-batch negatives losses benefit heavily from larger batch sizes, as they yield more negatives and a stronger training signal.

Distillation

These loss functions are specifically designed to be used when distilling the knowledge from one model into another. Distillation from a strong cross-encoder teacher is how the strongest late-interaction models (e.g. ColBERTv2, GTE-ModernColBERT) are trained.

Inputs	Labels	Appropriate Loss Functions
`(query, positive, negative)`	`gold_sim(query, positive) - gold_sim(query, negative)`	`MultiVectorMarginMSELoss`
`(query, positive, negative_1, ..., negative_n)`	`[gold_sim(query, positive) - gold_sim(query, negative_i) for i in 1..n]`	`MultiVectorMarginMSELoss`
`(query, positive, negative)`	`[gold_sim(query, positive), gold_sim(query, negative)]`	`MultiVectorMarginMSELoss`
`(query, doc_1, ..., doc_n)`	`[gold_sim(query, doc_i) for i in 1..n]`	`MultiVectorDistillKLDivLoss` ★ `MultiVectorMarginMSELoss`

For the n-way KD format with externally-stored query / document texts (e.g. lightonai/ms-marco-en-bge), you can use resolve_ids() to resolve IDs against the query and document datasets on the fly.

Commonly used Loss Functions

In practice, not all loss functions get used equally often. The most common scenarios are:

(anchor, positive) pairs without any labels: MultiVectorMultipleNegativesRankingLoss (a.k.a. InfoNCE or in-batch negatives loss) is cheap to obtain data for and generally very performant, especially combined with mined hard negatives. CachedMultiVectorMultipleNegativesRankingLoss extends it to very large effective batch sizes.
(query, doc_1, ..., doc_n) with teacher scores: MultiVectorDistillKLDivLoss implements the n-way knowledge distillation recipe behind the strongest late-interaction models.

Custom Loss Functions

Advanced users can create and train with their own loss functions. Custom loss functions only have a few requirements:

They must be a subclass of torch.nn.Module.
They must have model as the first argument in the constructor.
They must implement a forward method that accepts sentence_features and labels. The former is a list of tokenized batches, one element for each column. These tokenized batches can be fed directly to the model being trained to produce the token-level embeddings (read token_embeddings and the scoring attention_mask from the model output). The latter is an optional tensor of labels. The method must return a single loss value or a dictionary of loss components (component names to loss values) that will be summed to produce the final loss value. When returning a dictionary, the individual components will be logged separately in addition to the summed loss, allowing you to monitor the individual components of the loss.

To get full support with the automatic model card generation, you may also wish to implement:

a get_config_dict method that returns a dictionary of loss parameters.
a citation property so your work gets cited in all models that train with the loss.

Consider inspecting existing loss functions to get a feel for how loss functions are commonly implemented.