Custom Metrics in Evidently AI: Lift, Precision

Monitoring machine learning models in production is non-trivial. Out-of-the-box metrics like accuracy or RMSE tell you something went wrong, but rarely what or why. Evidently AI solves this by giving you a rich library of built-in metrics — and, crucially, the ability to define your own.

This guide walks through how Evidently’s metric system works, when to use Metric Presets versus individual metrics, and how to implement a custom metric from scratch — using lift as a practical example.

The tool generates interactive visual reports and JSON profiles from pandas DataFrame or csv files. There are currently 6 reports available:

• Data Drift – detects changes in feature distribution
• Numerical Target Drift – detects numerical target changes and feature behavior
• Categorical Target Drift – detects changes in categorical target and feature behavior
• Regression Model Performance – analyzes regression model performance and model errors
• Classification Model Performance – Analyzes the performance and errors of the classification model. Works for both binary and multiclass models.
• Probabilistic Classification Model Performance – Analyzes the performance of a probabilistic classification model, the quality of model calibration, and model errors. Works for both binary and multiclass models.

What Is Evidently AI?

Evidently AI is an open-source Python library for evaluating, testing, and monitoring ML models and data pipelines. It generates visual HTML reports and JSON snapshots that you can embed into dashboards or CI/CD pipelines.

The library ships with six built-in report types:

These presets cover 80% of use cases. The remaining 20% — domain-specific business metrics — require custom implementation.

Understanding Metrics vs Metric Presets

Before writing custom metrics, it helps to understand the two layers Evidently exposes.

Metrics

A Metric is the atomic unit of measurement. Each metric computes one thing: a value, a table, or a chart. Examples:

Python

from evidently.metrics import ColumnDriftMetric, DatasetMissingValuesMetric

report = Report(metrics=[
    DatasetMissingValuesMetric(),
    ColumnDriftMetric(column_name="score"),
])

You can mix and match any metrics into a single Report.

Metric Preset

A MetricPreset is a curated collection of metrics bundled for a specific task. Instead of listing 10 individual metrics, you import one preset:

Python

from evidently.metric_preset import ClassificationPreset

report = Report(metrics=[ClassificationPreset()])

Presets are opinionated — they include what Evidently’s team considers the most useful metrics for that task. When you need metrics outside this opinion, you write a custom one.

Metrics

A metric is a component that evaluates a specific aspect of data or model quality.

Metric Preset

A metrics preset is a pre-built report that aggregates metrics for a specific use case (for example, DataDriftPreset, RegressionPreset, etc.).

How it works?

Generate a report on reference and current datasets

• Reference dataset is the base dataset for comparison. This could be a training set or previous production data.
• Current dataset – the second dataset compared with the base one. It may contain the latest production data.

How Evidently Processes a Report

Understanding the execution model helps when debugging custom metrics.

Reference dataset → ┐ ├→ Report.run() → Metric.calculate() → MetricResult → HTML/JSON Current dataset → ┘

1. Report.run(reference_data=df_ref, current_data=df_cur) triggers all metrics
2. Each metric receives both datasets and a shared InputData context
3. Metrics return a MetricResult dataclass containing scalar values, tables, or figures
4. Evidently serialises results into a self-contained HTML file or JSON snapshot

Implementation of custom metrics

There are times in work when you need to monitor metrics that are not available in Evidently. For example, in telecom they really like the lift metric. Business loves her and understands her very well. You can read more about lift metrics here.

To add a new metric you need to do two things:

• Implement metric
• Add visualization to plotly – optional

We will take a more complicated route and implement the metric directly in Evidently in order to then make a pull request

Where to add:

• /src/evidently/calculations – add a metric depending on the task (classification, regression, etc.)
• /src/evidently/metrics — add code for calculating metrics depending on the task
• /src/evidently/renderers/html_widgets.py – visualization of metrics
• /src/evidently/metrics/init.py — initialize metrics
• /src/evidently/metric_results.py – add visualization

The metric code can be viewed in the already accepted pull request

Metric call

#probabilistic binary classification
classification_report = Report(metrics=[
    ClassificationLiftCurve(),
    ClassificationLiftTable(),
])

classification_report.run(reference_data=bcancer_ref, current_data=bcancer_cur)
classification_report

Common Pitfalls

Integrating with a Monitoring Pipeline

Custom metrics work seamlessly inside Evidently’s TestSuite and ColumnMapping:

Python

from evidently import TestSuite
from evidently.tests import TestValueRange

suite = TestSuite(tests=[
    TestValueRange(metric=LiftMetric(), value_name="lift_value", gte=1.5),
])
suite.run(reference_data=df_ref, current_data=df_cur)
suite.save_html("lift_test.html")

This fails the suite — and your CI pipeline — automatically if lift drops below 1.5.

When to Use Custom Metrics

Use a custom metric when:

• Your business KPI isn’t covered by built-in presets (lift, GINI, IoU, BLEU, etc.)
• You need to monitor a computed column derived from model output
• You want to enforce a domain-specific threshold in CI/CD

Stick to presets when:

• Standard classification / regression / drift monitoring is enough
• You want zero maintenance overhead

Summary

Evidently AI’s custom metric system is straightforward once you understand the two-layer architecture: MetricResult (the data container) and Metric (the computation). The lift example above is ~30 lines of code and plugs directly into any Evidently Report or TestSuite.

Key takeaways

1. Subclass MetricResult to define your output schema
2. Subclass Metric[YourResult] and implement calculate()
3. Return a fully populated MetricResult — never None
4. Combine custom and built-in metrics freely in one Report

Next step: try wrapping your custom metric in a TestSuite with a threshold and connecting it to your MLOps CI trigger.