Training with PEFT Adapters

Sentence Transformers has been integrated with PEFT (Parameter-Efficient Fine-Tuning), allowing you to finetune multi-vector (late-interaction) models without fine-tuning all of the model parameters. Instead, with PEFT methods you are only finetuning a fraction of (extra) model parameters with only a minor hit in performance compared to full model finetuning.

Compatibility Methods

The MultiVectorEncoder supports the following methods for interacting with the PEFT Adapters:

Adding a New Adapter

Adding a new adapter to a model is as simple as calling add_adapter() with a (subclass of) PeftConfig on an initialized Multi-Vector Encoder model. In the following example, we use a LoraConfig instance.

The adapter is applied to the Transformer backbone only. The projection layer that maps token embeddings to the multi-vector dimension sits outside the backbone and remains fully trainable, which is convenient as it is randomly initialized when starting from a plain backbone like answerdotai/ModernBERT-base.

from peft import LoraConfig, TaskType

from sentence_transformers import MultiVectorEncoder

# 1. Load a model to finetune
model = MultiVectorEncoder("answerdotai/ModernBERT-base")

# 2. Create a LoRA adapter for the model & add it
peft_config = LoraConfig(
    task_type=TaskType.FEATURE_EXTRACTION,
    inference_mode=False,
    target_modules=["Wqkv", "Wo", "Wi"],  # ModernBERT attention and MLP linear layers
    r=64,
    lora_alpha=128,
    lora_dropout=0.1,
)
model.add_adapter(peft_config)

# Proceed as usual... See https://sbert.net/docs/multi_vector_encoder/training_overview.html

Training Script

See the following example file for a full example of how PEFT can be used with Multi-Vector Encoder models: