Get started with the ValidMind Developer Framework

As a model developer, you use the ValidMind Developer Framework to automate the generation of model documentation by uploading documentation artifacts and test results to the ValidMind platform.

What is the developer framework?

ValidMind’s developer framework provides a rich collection of documentation tools and test suites, from documenting descriptions of your dataset to validation testing your models for weak spots and overfit areas.

ValidMind offers two primary methods for documenting model risk:

• By generating model documentation: Through automation, the framework extracts metadata from associated datasets and models for you and generates model documentation. You can also add more documentation and tests manually using the documentation editing capabilities in the ValidMind UI.

• By running pre-built validation tests: The framework provides a suite of validation tests for common financial services use cases. For cases where these tests do not cover everything you need, you can also extend existing test suites with your own proprietary tests or testing providers.

The developer framework is designed to be model agnostic. If your model is built in Python, ValidMind’s Python client library already provides all the standard functionality you might need without requiring your developers to rewrite any functions.

Key ValidMind concepts

Model documentation

A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. Within the realm of model risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.

Validation report

A formal document prepared as part of the model validation process, outlining the findings, assessments, and recommendations related to a specific model’s performance, appropriateness, and limitations. Provides a comprehensive review of the model’s conceptual framework, data sources and integrity, calibration methods, and performance outcomes. Within model risk management, the validation report is crucial for ensuring transparency, demonstrating regulatory compliance, and offering actionable insights for model refinement or adjustments.

Documentation Template

Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. ValidMind templates come with pre-defined sections, similar to test placeholders, including boilerplates and spaces designated for documentation and test results. You can edit these templates to suit your needs.

Model documentation is populated by the ValidMind Developer Framework when the tests contained in a template run successfully, completing the test suite. This process ensures that the model meets all the specified requirements. At the same time, template placeholders get populated with content that documents the model.

Tests

A function contained in the ValidMind Developer Framework, which is designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind Platform to generate the model documentation according to the relevant templates.

Test suites

A collection of tests which are meant to run together to automate generate model documentation end-to-end for specific use-cases.

Example: the classifier_full_suite test suite runs the tests from the tabular_dataset and classifier test suites to fully document the data and model sections for binary classification model use-cases.

How do I use the framework?

A typical high-level workflow for model developers consists of four major steps:

graph LR
    A[Develop<br>model] --> B[Generate model<br>documentation]
    B --> C[Refine model<br>documentation]
    C --> D[Submit for review]
    C --> B

Develop model

In your existing developer environment, build one or more candidate models that need to be validated. This step includes all the usual activities you already follow as a model developer.

Generate model documentation

With the ValidMind Developer Framework, generate automated model documentation and run validation tests. This step includes making use of the automation and testing functionality provided by the framework and uploading the output to the Platform UI. You can iteratively regenerate the documentation as you work though the next step of refining your documentation.

Refine model documentation

In the ValidMind Platform UI, review the generated documentation and test output. Iterate over the documentation and test output to refine your model documentation. Collaborate with other developers and model validators to finalize the model documentation and get it ready for review.

Submit for review

In the ValidMind Platform UI, you submit the model documentation for review which moves the documentation workflow moves to the next phase where a model validator will review it.

How do I generate model documentation?

Before you can use the developer framework, you need to verify that the current documentation template contains all the necessary tests for the model you are developing:

• The template might already be sufficient and you only need to run the template within the developer framework to populate documentation.

• Or, more likely, the template might need additional tests that you can add these tests via the developer framework.

This process of verifying the suitability of the the current documentation template and adding more tests to the template is an iterative process:

graph LR
    A[Verify template] --> B[Build template]
    B --> D[Add tests and<br>content blocks]
    D --> E[Add external<br>test providers]
    E --> C[Run template]
    C --> B

Build the template

When the documentation template requires more tests to be added, or if the documentation template does not include a specific content or test block you need:

• For functionality provided by the developer framework: Add the relevant tests or content blocks for the model use case.

• For tests not provided by the framework: Add your own external test provider.

Run the template : When you have registered all the required tests as content blocks in the documentation template, populate the necessary model documentation by adding this call to your model:

run_documentation_tests()

ValidMind may not support all potential use cases or provide a universally applicable documentation template. Typically, you initiate the process of putting ValidMind into production by constructing a template specific for your own use case and then refine your model documentation.

End-to end workflow

In your modeling environment

1. Build your model.

2. Export the datasets and model.

Next, go to With the developer framework, Step 2.

With the developer framework

1. Create a notebook to select and build the relevant tests.

2. From your modeling environment, load the trained datasets and models.

3. Use the instructions from In the Platform UI, Step 3, initialize the ValidMind Developer Framework.

4. Select the relevant tests.

5. Review if all tests are covered by ValidMind or your external test provider:

   • If all tests are NOT covered: Create and register additional tests.
   • If all tests are covered:
     1. Run the selected tests.
     2. Review your test results.

Next, go to In the ValidMind Platform UI, Step 5.

In the ValidMind Platform UI

1. Register a new model.

2. Review the template structure.

3. Locate the framework integration instructions.

4. Go to With the developer framework, Step 3.

5. After With the developer framework, Step 6, add content blocks to your model documentation:

   Select the block type:

   • For test-driven blocks: Select from available test provider results

   • For text blocks:

     • For new block:
       • Add new editable text content block
       • Review and collaborate on the content block
     • For existing blocks: Select from available texts from content provider

6. Submit your model documentation for review.

What’s next

• Run tests & test suites
• Code samples
• ValidMind Developer Framework Reference