This page lists available pretrained Cross-Encoders. Cross-Encoders require the input of a text pair and output a score 0…1. They do not work for individual sentences and they don’t compute embeddings for individual texts.
The following models can be used like this:
from sentence_transformers import CrossEncoder model = CrossEncoder('model_name') scores = model.predict([('Sent A1', 'Sent B1'), ('Sent A2', 'Sent B2')])
They return a score 0…1 indicating the semantic similarity of the given sentence pair.
cross-encoder/stsb-TinyBERT-L-4 - STSbenchmark test performance: 85.50
cross-encoder/stsb-distilroberta-base - STSbenchmark test performance: 87.92
cross-encoder/stsb-roberta-base - STSbenchmark test performance: 90.17
cross-encoder/stsb-roberta-large - STSbenchmark test performance: 91.47
Quora Duplicate Questions¶
These models have been trained on the Quora duplicate questions dataset. They can used like the STSb models and give a score 0…1 indicating the probability that two questions are duplicate questions.
cross-encoder/quora-distilroberta-base - Average Precision dev set: 87.48
cross-encoder/quora-roberta-base - Average Precision dev set: 87.80
cross-encoder/quora-roberta-large - Average Precision dev set: 87.91
Note: The model don’t work for question similarity. The question How to learn Java and How to learn Python will get a low score, as these questions are not duplicates. For question similarity, the respective bi-encoder trained on the Quora dataset yields much more meaningful results.
The following models are trained for Information Retrieval: Given a query (like key-words or a question), and a paragraph, can the query be answered by the paragraph? The models have beend trained on MS Marco, a large dataset with real-user queries from Bing search engine.
The models can be used like this:
from sentence_transformers import CrossEncoder model = CrossEncoder('model_name', max_length=512) scores = model.predict([('Query1', 'Paragraph1'), ('Query2', 'Paragraph2')]) #For Example scores = model.predict([('How many people live in Berlin?', 'Berlin had a population of 3,520,031 registered inhabitants in an area of 891.82 square kilometers.'), ('What is the size of New York?', 'New York City is famous for the Metropolitan Museum of Art.')])
This returns a score 0…1 indicating if the paragraph is relevant for a given query.
For details on the usage, see Applications - Information Retrieval
MS MARCO Passage Retrieval is a large dataset with real user queries from Bing search engine with annotated relevant text passages.
cross-encoder/ms-marco-TinyBERT-L-2 - MRR@10 on MS Marco Dev Set: 30.15
cross-encoder/ms-marco-TinyBERT-L-4 - MRR@10 on MS Marco Dev Set: 34.50
cross-encoder/ms-marco-TinyBERT-L-6 - MRR@10 on MS Marco Dev Set: 36.13
cross-encoder/ms-marco-electra-base - MRR@10 on MS Marco Dev Set: 36.41
cross-encoder/qnli-distilroberta-base - Accuracy on QNLI dev set: 90.96
cross-encoder/qnli-electra-base - Accuracy on QNLI dev set: 93.21
cross-encoder/nli-distilroberta-base - Accuracy on MNLI mismatched set: 83.98
cross-encoder/nli-roberta-base - Accuracy on MNLI mismatched set: 87.47
cross-encoder/nli-deberta-base - Accuracy on MNLI mismatched set: 88.08
from sentence_transformers import CrossEncoder model = CrossEncoder('model_name') scores = model.predict([('A man is eating pizza', 'A man eats something'), ('A black race car starts up in front of a crowd of people.', 'A man is driving down a lonely road.')]) #Convert scores to labels label_mapping = ['contradiction', 'entailment', 'neutral'] labels = [label_mapping[score_max] for score_max in scores.argmax(axis=1)]