Rendering more than one unique result for the same metric

Documentation templates facilitate the presentation of multiple unique metric results for a single metric. Consider various scenarios where you may intend to showcase results of the same metric with diverse inputs:

Comparing test results with varied parameter values: Illustrate metric performance by contrasting test results achieved with different parameter values to identify optimal settings.

Displaying test results with distinct datasets: Showcase metric versatility by presenting results on diverse datasets, such as providing confusion matrices for both training and test data.

Model comparison: Conduct a comprehensive model evaluation by comparing metrics like ROC curve and Accuracy to discern and select the superior-performing model.

This interactive notebook guides you through the process of documenting a model with the ValidMind Developer Framework. It uses the Bank Customer Churn Prediction sample dataset from Kaggle to train a simple classification model.

As part of the notebook, you will learn how to render more than one unique metric result for the same metric while exploring how the documentation process works:

ValidMind at a glance

ValidMind’s platform enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Developer Framework to automate documentation and validation tests, and then use the ValidMind AI Risk Platform UI to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.

If this is your first time trying out ValidMind, you can make use of the following resources alongside this notebook:

Before you begin

New to ValidMind?

For access to all features available in this notebook, create a free ValidMind account.

Signing up is FREE — Sign up now

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

Install the client library

The client library provides Python support for the ValidMind Developer Framework. To install it:

%pip install -q validmind

Initialize the client library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the client library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet:

  1. In a browser, log into the Platform UI.

  2. In the left sidebar, navigate to Model Inventory and click + Register new model.

  3. Enter the model details, making sure to select Binary classification as the template and Marketing/Sales - Attrition/Churn Management as the use case, and click Continue. (Need more help?)

  4. Go to Getting Started and click Copy snippet to clipboard.

Next, replace this placeholder with your own code snippet:

# Replace with your code snippet

import validmind as vm

vm.init(
    api_host="...",
    api_key="...",
    api_secret="...",
    project="..."
)

Update the customer churn demo template

Before you initialize the client library by running the notebook, edit the Binary classification template to make a copy of a metric of interest and update it with different result_id fields for each entry:

  • Go to Settings > Templates and click on the Binary classification template. Let’s say we want to show Skewness results for training and test datasets.

To do this we replace

- content_type: test
  content_id: validmind.data_validation.Skewness

with

- content_type: test
  content_id: validmind.data_validation.Skewness:training_data
- content_type: test
  content_id: validmind.data_validation.Skewness:test_data

This way, we can show two results of the same test in the model document. Here, the training_data and test_data could be any string. However, they should be unique for the same test.

  • Click on Prepare new version, provide some version notes and click on Save new version to save a new version of this template.
  • Next, we need to swap our model documentation to use this new version of the template. Follow the steps on this guide to swap the template of our customer churn model: https://docs.validmind.ai/guide/swap-documentation-templates.html.

In the following sections we provide more context on how these content_id fields mentioned earlier get mapped to the actual tests.

Initialize the Python environment

Next, let’s import the necessary libraries and set up your Python environment for data analysis:

import pandas as pd
import xgboost as xgb

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

%matplotlib inline

Preview the documentation template

A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.

You will upload documentation and test results into this template later on. For now, take a look at the structure that the template provides with the vm.preview_template() function from the ValidMind library and note the empty sections. You will see two blocks with different result IDs for skewness.

vm.preview_template()

Load the sample dataset

The sample dataset used here is provided by the ValidMind library, along with a second, different dataset (taiwan_credit) you can try as well.

To be able to use either sample dataset, you need to import the dataset and load it into a pandas DataFrame, a two-dimensional tabular data structure that makes use of rows and columns:

# Import the sample dataset from the library

from validmind.datasets.classification import customer_churn as demo_dataset

df = demo_dataset.load_data()

Initialize a ValidMind dataset object

Before you can run a test suite, which are a collection of tests, you must first initialize a ValidMind dataset object using the init_dataset function from the ValidMind (vm) module.

This function takes a number of arguments:

  • dataset — the raw dataset that you want to analyze
  • target_column — the name of the target column in the dataset
  • class_labels — the list of class labels used for classification model training
vm_dataset = vm.init_dataset(
    input_id="raw_dataset",
    dataset=df,
    target_column=demo_dataset.target_column,
    class_labels=demo_dataset.class_labels
)

Document the model

As part of documenting the model with the ValidMind Developer Framework, you need to preprocess the raw dataset, initialize some training and test datasets, initialize a model object you can use for testing, and then run the full suite of tests.

Prepare datasets

DataFrame (df) preprocessing is simplified by employing demo_dataset.preprocess to partition it into distinct datasets (train_df, validation_df, and test_df)

train_df, validation_df, test_df = demo_dataset.preprocess(df)

Initialize the training and test datasets

With the datasets ready, you can now initialize the training and test datasets (train_df and test_df) created earlier into their own dataset objects using vm.init_dataset():

vm_train_ds = vm.init_dataset(
    input_id="train_dataset",
    dataset=train_df,
    target_column=demo_dataset.target_column
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset",
    dataset=test_df,
    target_column=demo_dataset.target_column
)

Run documentation tests

Now specify inputs and params for individual tests using config parameter. The results for the both the datasets will be visible in the documentation. The inputs in the config get priority over global inputs in the run_documentation_tests.

config = {
    "validmind.data_validation.Skewness:training_data": {
        "params": {"max_threshold": 1},
        "inputs": {"dataset": vm_train_ds}
    },
    "validmind.data_validation.Skewness:test_data": {
        "params": {"max_threshold": 1.5},
        "inputs": {"dataset": vm_test_ds}
    },
}

tests_suite = vm.run_documentation_tests(
    inputs={
        "dataset": vm_dataset,
    },
    config=config,
    section=["data_preparation"]
)

Run the individual tests using the run_test

Now run the Skewness tests for training and test datasets. The results for the both the datasets will be visible in the documentation.

test = vm.tests.run_test(
    test_id="validmind.data_validation.Skewness:training_data",
    params={
        "max_threshold": 1
    },
    inputs={
        "dataset": vm_train_ds
    }
)
test.log()

test = vm.tests.run_test(
    test_id="validmind.data_validation.Skewness:test_data",
    params={
        "max_threshold": 1.5
    },
    inputs={
        "dataset": vm_test_ds,
    }
)
test.log()

Next steps

You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the test results as part of your model documentation right in the ValidMind Platform UI:

  1. In the Platform UI, go to the Documentation page for the model you registered earlier.

  2. Expand the following sections and take a look around:

    • 2. Data Preparation

    You can now see the skewness tests results of training and test datasets in the Data Preparation section.

If you want to learn more about where you are in the model documentation process, take a look at How do I use the framework?.