Integrate an External Test Provider

This notebook shows model developers how to integrate a custom test provider into the Validmind Developer Framework. The notebook loads two of sample test providers and registers them with the framework which enables you to run a template that utilizes the custom tests made available by the providers.

ValidMind at a glance

ValidMind’s platform enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Developer Framework to automate documentation and validation tests, and then use the ValidMind AI Risk Platform UI to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.

If this is your first time trying out ValidMind, we recommend going through the following resources first:

Before you begin

New to ValidMind?

For access to all features available in this notebook, create a free ValidMind account.

Signing up is FREE — Sign up now

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

Install the client library

The client library provides Python support for the ValidMind Developer Framework. To install it:

%pip install -q validmind

Before you initialize the client library: update the customer churn template template

First, let’s edit the Binary classification template and register test blocks for the demo test providers we will implement below.

  • Go to Settings > Templates and click on the Binary classification template. Let’s add a new top level section called test_providers_demo with some test driven content blocks like below:
- id: test_providers_demo
  title: Test providers demo
  contents:
    - content_type: metric
      content_id: my_local_provider.tests.MyCustomTest
    - content_type: metric
      content_id: my_inline_provider.tests.MyCustomTest
  • Click on Prepare new version, provide some version notes and click con Save new version to save a new version of this template
  • Now we need to swap our model documentation to this new version of the template. Follow the steps on this guide to swap the template of our customer churn model: https://docs.validmind.ai/guide/swap-documentation-templates.html

In the sections below we provide more context on how these content_ids above get mapped to the actual test providers and tests.

Initialize the client library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the client library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet:

  1. In a browser, log into the Platform UI.

  2. In the left sidebar, navigate to Model Inventory and click + Register new model.

  3. Enter the model details, making sure to select the Attrition/Churn Management use case, and click Continue. (Need more help?)

  4. Go to Getting Started and click Copy snippet to clipboard.

Next, replace this placeholder with your own code snippet:

# Replace with your code snippet

import validmind as vm

vm.init(
    api_host="https://api.prod.validmind.ai/api/v1/tracking",
    api_key="...",
    api_secret="...",
    project="...",
)

Preview the documentation template and validate that it contains the new test blocks

We should see two custom content blocks in the template whose IDs are under the namespaces registered below (my_inline_provider and my_local_provider).

preview_template() will show an error when loading the new tests since we haven’t registered our new test providers yet. This is expected.

vm.preview_template()

Register external test providers

We will now instantiate and register test provider classes that include the tests we included in the template.

We will build an inline test provider that allows creating tests directly in the notebook and a local filesystem test provider that allows loading tests from a local folder.

For the local filesystem provider, we just need to specify the root folder under which the provider class will look for tests. For this demo, it is the ./tests/ directory.

Import the Local File System Test Provider from the validmind.tests module

from validmind.tests import LocalTestProvider

Create an inline TestProvider Class that just returns a single test

import pandas as pd


class MySecondCustomTest(vm.vm_models.Metric):
    # The test name should match the content ID on the template
    name = "my_inline_provider.tests.MyCustomTest"

    def description(self):
        return "This is a custom test from an external test provider."

    def run(self):
        return self.cache_results([{"foo": "bar"}])

    def summary(self, results):
        return vm.vm_models.ResultSummary(
            results=[
                vm.vm_models.ResultTable(
                    data=pd.DataFrame(results),
                    metadata=vm.vm_models.ResultTableMetadata(
                        title="Results from Test Provider Inside Notebook"
                    ),
                )
            ]
        )


class TestProviderInline:
    def load_test(self, test_id):
        # ignore the test_id and just return the single test above
        return MySecondCustomTest
# instantiate the test provider
inline_test_provider = TestProviderInline()
local_test_provider = LocalTestProvider(root_folder=".")

# register the test providers
vm.tests.register_test_provider(
    namespace="my_inline_provider",
    test_provider=inline_test_provider,
)  # validmind will now call the `TestProviderInline.load_test` method whenever it encounters a test ID that starts with `my_inline_provider`

vm.tests.register_test_provider(
    namespace="my_local_provider",
    test_provider=local_test_provider,
)  # validmind will now call the `LocalTestProvider.load_test` method whenever it encounters a test ID that starts with `my_local_provider`

Verify that preview_template() now loads the tests from the test providers

After registering the test providers with vm.tests.register_test_provider(), the developer framework can now locate the code that will execute the tests when we run the documentation tests on the template. We can verify this by running preview_template() again and seeing that the tests are now loaded correctly.

vm.preview_template()

Running the template

Now we can run the template as usual and it will use the external test providers to load the appropriate tests. Note that we’re not passing any inputs such as dataset and model to run_documentation_tests(). This is because our demo test providers do not have any required inputs and we’re scoping the template execution to the test_providers_demo section by using the section keyword argument.

suite_results = vm.run_documentation_tests(section="test_providers_demo")

Next steps

You can now view the results of the external test providers in the ValidMind Platform UI:

  1. In the Platform UI, go to the Documentation page for the model you registered earlier.

  2. Expand the Test Providers Demo section as defined in the template.

What you can see now is a more easily consumable version of the test results that were uploaded by the ValidMind Developer Framework, along with other parts of your documentation.