Run tests and test suites

This topic provides an overview for what tests and test suites are in the ValidMind Developer Framework, when to use tests and test suites, and typical scenarios.

What are tests and test suites?

Tests

Tests are designed to run a specific quantitative test on the dataset or model. Test results are sent to the ValidMind Platform to generate the model documentation according to the relevant templates.

Test suites

Test suites are collections of tests which are run together to automate generating model documentation end-to-end for specific use-cases.

Example: the classifier_full_suite test suite runs the tests from the tabular_dataset and classifier test suites to fully document the data and model sections for binary classification model use-cases.

When do I use tests and test suites?

ValidMind provides many built-in tests and test suites which make it easy for a model developer to document their work at any point during the model development lifecycle when they need to validate that their work satisfies model risk management requirements.

While you, as a model developer, have the flexibility to decide when to use which ValidMind tests, we have identified a few typical scenarios with their own characteristics and needs:

Dataset testing

To document and validate your dataset:

• For generic tabular datasets: use the tabular_dataset test suite.
• For time-series datasets: use the time_series_dataset test suite.

Model testing

To document and validate your model:

• For binary classification models: use the classifier test suite.
• For time series models: use the timeseries test suite.

End-to-end testing

To document a binary classification model and the relevant dataset end-to-end:

Use the classifier_full_suite test suite.

Can I use my own tests?

Yes, ValidMind supports custom tests that you develop yourself or that are provided by third-party test libraries, also referred to as test providers. We provide instructions with code examples that you can adapt:

• Implement Custom Metrics and Threshold Tests
• Integrate an External Test Provider

Is there an API reference available?

See the ValidMind Developer Framework API Reference for a list of all of the built-in tests and test suites for datasets and models.

How-tos for testing