Evaluation

class sentence_transformers.base.evaluation.BaseEvaluator[source]

Base class for all evaluators. Notably, this class introduces the greater_is_better and primary_metric attributes. The former is a boolean indicating whether a higher evaluation score is better, which is used for choosing the best checkpoint if load_best_model_at_end is set to True in the training arguments.

The latter is a string indicating the primary metric for the evaluator. This has to be defined whenever the evaluator returns a dictionary of metrics, and the primary metric is the key pointing to the primary metric, i.e. the one that is used for model selection and/or logging.

Extend this class and implement __call__ for custom evaluators.

class sentence_transformers.base.evaluation.SequentialEvaluator(evaluators: ~collections.abc.Iterable[~sentence_transformers.base.evaluation.evaluator.BaseEvaluator], main_score_function=<function SequentialEvaluator.<lambda>>)[source]

This evaluator allows that multiple sub-evaluators are passed. When the model is evaluated, the data is passed sequentially to all sub-evaluators.

All scores are passed to ‘main_score_function’, which derives one final score value

Parameters:

  • evaluators (Iterable[BaseEvaluator]) – A collection of BaseEvaluator objects.

  • main_score_function (function, optional) – A function that takes a list of scores and returns the main score. Defaults to selecting the last score in the list.

Example

evaluator1 = BinaryClassificationEvaluator(...)
evaluator2 = InformationRetrievalEvaluator(...)
evaluator3 = MSEEvaluator(...)
seq_evaluator = SequentialEvaluator([evaluator1, evaluator2, evaluator3])