Visual Document Retrieval

ColPali-style late interaction is the state of the art for visual document retrieval: text queries are matched directly against document page images, with every query token compared against every image patch embedding. This skips OCR, layout parsing, and chunking entirely, so tables, figures, and charts are searched just like body text. This page shows two ways to build such a model: training a fresh late-interaction model from a multimodal embedding backbone, or finetuning one of the already published ColVLM checkpoints.

There are pre-trained models available, which you can directly use without the need of training your own models. For more information, see Pretrained Models > Visual Document Retrieval Models.

Training from a multimodal embedding backbone

Training code: training_visual_document_retrieval.py

This is the recommended route for a new model. MultiVectorEncoder appends a freshly initialised 128-dim token projection to any multimodal embedding backbone, the same construction that turned VLMs into ColPali / ColQwen in the first place, and the resulting module stack is identical to the one a text backbone produces. This script builds a late-interaction model from Qwen/Qwen3-VL-Embedding-2B and trains it on (query, page image, hard negative image) triplets from the tomaarsen/llamaindex-vdr-en-train-preprocessed dataset. It mirrors the single-vector recipe from the Training and Finetuning Multimodal Embedding & Reranker Models blog post (training_visual_document_retrieval.py) on the same data, so the two families can be compared directly. CachedMultiVectorMultipleNegativesRankingLoss keeps the effective batch large while only encoding a few samples at a time.

The whole backbone is finetuned, with two training arguments doing the heavy lifting: learning_rate_mapping gives the freshly initialised projection 10x the backbone learning rate, and max_grad_norm=30 stops the fresh head’s large gradients from letting default clipping throttle every update.

Finetuning a pretrained ColVLM

Training code: finetuning_colqwen2.py

Use this route to adapt one of the already published ColVLM checkpoints to your own document collection. MultiVectorEncoder auto-detects transformers-native *ForRetrieval checkpoints such as vidore/colqwen2-v1.0-hf, and image documents are encoded exactly like text: pass PIL images, file paths, or URLs to encode_document or use them as a document column in your training dataset. The vidore/syntheticDocQA_energy_train dataset (one of the domain components of the original ColPali training mix) provides (query, page_image) pairs from scraped energy-domain PDFs, and MultiVectorMultipleNegativesRankingLoss optimizes the MaxSim score of each pair to be higher than the scores against all other in-batch pages.

The 2.2B parameter backbone is too large for full finetuning on a 24GB consumer GPU, so the script freezes everything except the top layers of the language model and the multi-vector projection. Keeping the vision tower frozen is standard ColVLM practice (the ViDoRe Col* checkpoints themselves were trained with a frozen vision tower and LoRA adapters on the language model), and the frozen bottom layers need no activation storage or optimizer state, keeping the peak VRAM usage well below 24GB with bf16 weights. Evaluation uses MultiVectorInformationRetrievalEvaluator with held-out queries against a corpus of held-out pages.