Pretrained Models
Several Multi-Vector Encoder models have been publicly released on the Hugging Face Hub:
Community models: All Multi-Vector Encoder models on Hugging Face.
Note
The sentence-transformers library tag is shared by all four model types, so pair it with the multi-vector tag to find only the models that load with MultiVectorEncoder. cross-encoder and sparse are the equivalents for Cross Encoder and Sparse Encoder models, while dense Sentence Transformer models make up the bulk of what the library tag returns on its own.
That community list is the one that stays current, and we are working to get the multi-vector tag onto every model that works with MultiVectorEncoder. The tables below are what we test against directly, so treat them as a starting point rather than the full set. For text retrieval in particular, any PyLate or Stanford-NLP ColBERT checkpoint loads whether or not it carries the tag yet.
Models integrate seamlessly with this simple interface:
from sentence_transformers import MultiVectorEncoder
# Download from the 🤗 Hub
model = MultiVectorEncoder("lightonai/LateOn")
# Run inference
queries = ["What is the capital of France?"]
documents = [
"Paris is the capital of France.",
"Berlin is the capital of Germany.",
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (10, 128) (9, 128) - one 128-dimensional vector per token
# Get the late-interaction (MaxSim) similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[9.1129, 8.8769]])
Text Retrieval Models
These load with their trained prefix tokens, query expansion, and punctuation skiplist recovered from the saved configuration. Where a revision is listed, pass it until the pull request on that repository is merged, after which the plain model name is enough.
The NanoBEIR column reports the mean NDCG@10 (higher is better) across the 13 NanoBEIR datasets, each a 50-query subsample of a BEIR dataset, as a fast proxy for English text retrieval quality. We used the MultiVectorNanoBEIREvaluator to compute the scores for the primarily-English models. A - means the model was not evaluated on it. Note that NanoBEIR is a small benchmark, and its scores aren’t a substitute for evaluating on your own data, which is always the right way to pick a model.
Visual Document Retrieval Models
ColPali-style models embed page images as documents and text as queries.
The NanoViDoRe column reports the mean NDCG@10 (higher is better) across NanoViDoRe v3, a compact visual document retrieval benchmark spanning 8 subsets (computer science, energy, finance in English and French, HR, industrial, pharmaceuticals, and physics). Like with NanoBEIR, NanoViDoRe is a small benchmark which shouldn’t replace evaluation on your own data.
Most of these are LoRA adapter repositories, with the adapter applied directly onto its base at load time. Some also have a -merged sibling on the Hub (e.g. vidore/colpali-v1.3-merged) with the adapter already folded into the weights.
The three -hf entries are the transformers-native *ForRetrieval ports. They load without any configuration, but use more modeling from transformers and less from sentence_transformers. Generally, it’s preferable to use the original models instead, as the ports score approximately the same.