Glossary

Published

March 6, 2024

This glossary of terms provides short definitions for technical terms you find commonly used in our product documentation grouped by terms related to:

ValidMind platform

client library, Python client library
Enables the interaction of your development environment with the ValidMind platform as part of the developer framework.
documentation automation
A core benefit of the ValidMind platform that allows for the automatic creation of model documentation using predefined templates and test suites.
model inventory
A feature of the ValidMind platform where you can track, manage, and oversee the lifecycle of models. Covers the full model lifecycle, including customizable documentation and approval workflows for different user roles, status and activity tracking, and periodic revalidation.
template, documentation template
Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. ValidMind templates come with pre-defined sections, similar to test placeholders, including boilerplates and spaces designated for documentation and test results. When rendered, produces a document that model developers can use for model validation.
test
Runs a specific quantitative test provided by the ValidMind Developer Framework on the dataset or model. Test results are sent to the ValidMind platform to generate the model documentation according to the template that is associated with the documentation.
test suite
A collection of tests which are run together to generate model documentation end-to-end for specific use-cases. For example, the classifier_full_suite test suite runs the tabular_dataset and classifier test suites to fully document the data and model sections for binary classification model use-cases.
ValidMind Developer Framework
A suite of documentation tools and test suites designed to document models, test models for weaknesses, and identify overfit areas. Enables automating the generation of model documentation by uploading documentation, metrics, and test results to the ValidMind platform.
ValidMind Platform UI
A hosted multi-tenant architecture that includes the cloud-based web interface, APIs, databases, documentation and validation engine, and various internal services.

Developer tools

pip
A package manager for Python, used to install and manage software packages written in the Python programming language. ValidMind uses the pip command to install the Python client library that is part of the ValidMind Developer Framework so that model developers can make use of its features.
JupyterHub
A multi-user server provides a platform for users to interactively work with data science and scientific computing tools in a collaborative environment. ValidMind uses JupyterHub to share live code, how-to instructions, and visualizations via notebooks as part of our getting started experience for new users.
Jupyter notebook
Allows users to create and share documents containing live code, data visualizations, and narrative text. Supports various programming languages, most notably Python, and is widely used for data analysis, machine learning, scientific research, and educational purposes. ValidMind uses notebooks to share sample code and how-to instructions with users that you can adapt to your own use case.
GitHub
A cloud-based platform that provides hosting for software development and version control using Git. GitHub offers collaboration tools such as bug tracking, feature requests, task management, and continuous integration pipelines. ValidMind uses GitHub to share open-source software with you.

AI and model risk management

1st line of defense
In the context of model risk, the business unit(s) responsible for model development, validation, and implementation during the model lifecycle. As the 1st line of defense, model developers must document and test models to ensure that they are accurate, robust, and fit for purpose.
2nd line of defense
In the context of model risk, an independent oversight function that provides a governance framework for the model lifecycle. As the 2nd line of defense, model validators must independently validate and challenge models created by model developers to ensure that model risk management principles are followed.
model developer
Responsible for the design, implementation, and maintenance of models to ensure they are fit-for-purpose, accurate, and aligned with business requirements. As subject matter experts, they collaborate with model validators and other business units, ensuring the models are conceptually sound and robust.
model documentation
A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. Within the realm of model risk management, this documentation serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.
model inventory
A systematic and organized record of all quantitative and qualitative models used within an organization. This inventory facilitates oversight, tracking, and assessment by listing each model’s purpose, characteristics, owners, validation status, and associated risks. Also see ValidMind model inventory.
model risk management (MRM)
A structured approach to identifying, assessing, mitigating, and monitoring risks arising from the use of quantitative and qualitative models within an organization. Ensures that models are developed, validated, and used appropriately, with robust controls in place. Encompasses practices such as maintaining a model inventory, conducting periodic validations, and ensuring proper documentation.
model risk
The potential for financial loss, incorrect decisions, or unintended consequences resulting from errors or inaccuracies in AI or machine learning models. Model risk typically arises from incorrect or inappropriate use of models, inaccurate assumptions, or limitations in data quality. Consequences of unmitigated model risk can include adverse outcomes such as financial loss, damage to reputation, and regulatory penalties, for example.
model governance
A framework of policies, procedures, and standards established to oversee the lifecycle of models within an organization. Ensures that models are developed, validated, implemented, and retired in a controlled and consistent manner, promoting accountability, transparency, and adherence to regulatory requirements.
model validation
A systematic process to evaluate and verify that a model is performing as intended, accurately represents the phenomena it is designed to capture, and is appropriate for its specified purpose. This assessment encompasses a review of the model’s conceptual soundness, data integrity, calibration, and performance outcomes, as well as testing against out-of-sample datasets. Within model risk management, model validation ensures that potential risks associated with model errors, misuse, or misunderstanding are identified and mitigated.
model validator
responsible for conducting independent assessments of models to ensure their accuracy, reliability, and appropriateness for intended purposes. The role involves evaluating a model’s conceptual soundness, data integrity, calibration methods, and overall performance, typically using out-of-sample datasets. Model validators identify potential risks and weaknesses, ensuring that models within an organization meet established standards and regulatory requirements, and provide recommendations to model developers for improvements or modifications.
three lines of defense
A structured approach to model risk management, consisting of three independent functions. The first line consists of business units responsible for model development, validation, and implementation. They ensure that models are accurate, robust, and fit for purpose. The second line is an independent model risk oversight function that provides a governance framework and guidance for model risk management. The third line is the internal or external audit function, which assesses the effectiveness of model risk management practices and controls.
validation report
A formal document produced after a model validation process, outlining the findings, assessments, and recommendations related to a specific model’s performance, appropriateness, and limitations. Provides a comprehensive review of the model’s conceptual framework, data sources and integrity, calibration methods, and performance outcomes. Within model risk management, the validation report is crucial for ensuring transparency, demonstrating regulatory compliance, and offering actionable insights for model refinement or adjustments.