Implementing Custom Metrics and Threshold Tests

Custom metrics offer added flexibility by extending the default metrics provided by ValidMind, enabling you to document any type of model or use case. Both metrics and threshold tests assess models but they differ in approach: metrics measure a range of dataset or model behaviors, while threshold tests yield a pass or fail result based on specific criteria. These instructions include the code required to:

Documentation components of a metric and threshold test

A metric is composed of the following documentation elements:

  • Title
  • Description
  • Results Table(s)
  • Plot(s)

A threshold test is composed of the following documentation elements:

  • Title
  • Description
  • Test Parameters
  • Results Table(s)
  • Plot(s)

Before you begin

New to ValidMind?

To access the ValidMind Platform UI, you’ll need an account.

Signing up is FREE — Create your account.

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

Install the client library

%pip install -q validmind

Initialize the client library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the client library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet:

  1. In a browser, log into the Platform UI.

  2. In the left sidebar, navigate to Model Inventory and click + Register new model.

  3. Enter the model details, making sure to select Binary classification as the template and Marketing/Sales - Attrition/Churn Management as the use case, and click Continue. (Need more help?)

  4. Go to Getting Started and click Copy snippet to clipboard.

Next, replace this placeholder with your own code snippet:

# Replace with your code snippet

import validmind as vm

vm.init(
  api_host = "...",
  api_key = "...",
  api_secret = "...",
  project = "..."
)

Create a metric class signature

In order to implement a custom metric or threshold test, you must create a class that inherits from the Metric or ThresholdTest class. The class signatures below show the different methods that need to be implemented in order to provide the required documentation elements:

@dataclass
class ExampleMetric(Metric):
    name = "mean_of_values"

    # Markdown compatible description of the metric
    def description(self):

    # Code to compute the metric and cache its results and Figures
    def run(self):

    # Code to build a list of ResultSummaries that form the results tables
    def summary(self, metric_values):

We’ll now implement a sample metric to illustrate their different documentation components.

Implement a custom metric

The following example shows how to implement a custom metric that calculates the mean of a list of numbers.

Basic metric implementation

At its most basic, a metric implementation requires a run() method that computes the metric and caches its results and Figures. The run() method is called by the ValidMind client when the metric is executed. The run() should return any value that can be serialized to JSON.

In the example below we also provide a simple description for the metric:

from dataclasses import dataclass
from validmind.vm_models import Metric


@dataclass
class MeanMetric(Metric):
    name = "mean_of_values"

    def description(self):
        return "Calculates the mean of the provided values"

    def run(self):
        if "values" not in self.params:
            raise ValueError("values must be provided in params")

        if not isinstance(self.params["values"], list):
            raise ValueError("values must be a list")

        values = self.params["values"]
        mean = sum(values) / len(values)
        return self.cache_results(metric_value={"Mean": mean})

Test the custom metric

We should run a metric first without running an entire test suite and test its behavior.

The only requirement to run a metric is build a TestContext object and pass it to the metric initializer. Test context objects allow metrics and tests to access data inside their class methods in a predictable way. By default, ValidMind provides support for the following special keys in a test context objects:

  • dataset
  • model
  • models

When a test context object is build with one of these keys, the corresponding value is automatically added to the object as an attribute. For example, if you build a test context object with the dataset key, you can access the dataset inside the metric’s run() method as self.dataset. We’ll illustrate this in detail in the next section.

In our simple example, we don’t need to pass any arguments to the TestContext initializer.

from validmind.vm_models.test_context import TestContext

test_context = TestContext()
mean_metric = MeanMetric(
    test_id="mean_of_values",
    context=test_context,
    params={
        "values": [1, 2, 3, 4, 5]
    }
)
mean_metric.run()

You can also inspect the results of the metric by accessing the result variable:

mean_metric.result.show()

Add a summary() method to the custom metric

The summary() method is used to build a ResultSummary object that can display the results of our test as a list of one or more summray tables. The ResultSummary class takes a results argument that is a list of ResultTable objects.

Each ResultTable object is composed of a data and metadata attribute. The data attribute is any valid Pandas tabular DataFrame and metadata is a ResultTableMetadata instance that takes title as the table description.

from dataclasses import dataclass

import pandas as pd
from validmind.vm_models import Metric, ResultSummary, ResultTable, ResultTableMetadata


@dataclass
class MeanMetric(Metric):
    name = "mean_of_values"

    def description(self):
        return "Calculates the mean of the provided values"

    def summary(self, metric_value):
        # Create a dataframe structure that can be rendered as a table
        simple_df = pd.DataFrame({"Mean of Values": [metric_value]})

        return ResultSummary(
            results=[
                ResultTable(
                    data=simple_df,
                    metadata=ResultTableMetadata(title="Example Table"),
                ),
            ]
        )

    def run(self):
        if "values" not in self.params:
            raise ValueError("values must be provided in params")

        if not isinstance(self.params["values"], list):
            raise ValueError("values must be a list")

        values = self.params["values"]
        mean = sum(values) / len(values)
        return self.cache_results(mean)
from validmind.vm_models.test_context import TestContext

test_context = TestContext()
mean_metric = MeanMetric(
    test_id="mean_of_values",
    context=test_context,
    params={
        "values": [1, 2, 3, 4, 5]
    }
)
mean_metric.run()
mean_metric.result.show()

Add figures to a metric

You can also add figures to a metric by passing a figures list to cache_results(). Each figure is a Figure object that takes the following arguments:

  • for_object: The name of the object that the figure is for. Usually defaults to self
  • figure: A Matplotlib or Plotly figure object
  • key: A unique key for the figure

The developer framework uses for_object and key to link figures to the corresponding metric or test.

from dataclasses import dataclass

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from validmind.vm_models import Figure, Metric, ResultSummary, ResultTable, ResultTableMetadata


@dataclass
class MeanMetric(Metric):
    name = "mean_of_values"

    def description(self):
        return "Calculates the mean of the provided values"

    def summary(self, metric_value):
        # Create a dataframe structure that can be rendered as a table
        simple_df = pd.DataFrame({"Mean of Values": [metric_value]})

        return ResultSummary(
            results=[
                ResultTable(
                    data=simple_df,
                    metadata=ResultTableMetadata(title="Example Table"),
                ),
            ]
        )

    def run(self):
        if "values" not in self.params:
            raise ValueError("values must be provided in params")

        if not isinstance(self.params["values"], list):
            raise ValueError("values must be a list")

        values = self.params["values"]
        mean = sum(values) / len(values)

        # Create a random histogram with matplotlib
        fig, ax = plt.subplots()
        ax.hist(np.random.randn(1000), bins=20, color="blue")
        ax.set_title("Histogram of random numbers")
        ax.set_xlabel("Value")
        ax.set_ylabel("Frequency")

        # Do this if you want to prevent the figure from being displayed
        plt.close("all")

        figure = Figure(
            for_object=self,
            key=self.key,
            figure=fig
        )

        return self.cache_results(mean, figures=[figure])
from validmind.vm_models.test_context import TestContext

test_context = TestContext()
mean_metric = MeanMetric(
    test_id="mean_of_values",
    context=test_context,
    params={
        "values": [1, 2, 3, 4, 5]
    }
)
mean_metric.run()
mean_metric.result.show()