Running an Individual Test

This notebook shows how to run individual metrics or thresholds tests that is part of the ValidMind Developer Framework. These instructions include the code required to:

Before you begin

New to ValidMind?

For access to all features available in this notebook, create a free ValidMind account.

Signing up is FREE — Sign up now

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

Install the client library

%pip install -q validmind

Initialize the client library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the client library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet:

  1. In a browser, log into the Platform UI.

  2. In the left sidebar, navigate to Model Inventory and click + Register new model.

  3. Enter the model details, making sure to select Binary classification as the template and Marketing/Sales - Attrition/Churn Management as the use case, and click Continue. (Need more help?)

  4. Go to Getting Started and click Copy snippet to clipboard.

Next, replace this placeholder with your own code snippet:

# Replace with your code snippet

import validmind as vm

vm.init(
    api_host="https://api.prod.validmind.ai/api/v1/tracking",
    api_key="...",
    api_secret="...",
    project="...",
)
import pandas as pd
import xgboost as xgb

from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split

%matplotlib inline

Load the demo dataset

from validmind.datasets.classification import customer_churn as demo_dataset

# You can also import customer_churn like this:
# from validmind.datasets.classification import taiwan_credit as demo_dataset

df = demo_dataset.load_data()

Prepocess the raw dataset

train_df, validation_df, test_df = demo_dataset.preprocess(df)

Train a model for testing

We train a simple customer churn model for our test.

x_train = train_df.drop(demo_dataset.target_column, axis=1)
y_train = train_df[demo_dataset.target_column]
x_val = validation_df.drop(demo_dataset.target_column, axis=1)
y_val = validation_df[demo_dataset.target_column]

model = xgb.XGBClassifier(early_stopping_rounds=10)
model.set_params(
    eval_metric=["error", "logloss", "auc"],
)
model.fit(
    x_train,
    y_train,
    eval_set=[(x_val, y_val)],
    verbose=False,
)

Set up test inputs and run the test

Initialize ValidMind objects

vm_train_ds = vm.init_dataset(
    input_id="train_dataset",
    dataset=train_df,
    type="generic",
    target_column=demo_dataset.target_column,
)

vm_test_ds = vm.init_dataset(
    input_id="test_dataset",
    dataset=test_df,
    type="generic",
    target_column=demo_dataset.target_column,
)

vm_model = vm.init_model(model, input_id="model")

Assign predictions to the datasets

We can now use the assign_predictions() method from the Dataset object to link existing predictions to any model. If no prediction values are passed, the method will compute predictions automatically:

vm_train_ds.assign_predictions(
    model=vm_model,
)
vm_test_ds.assign_predictions(
    model=vm_model,
)

Run the test

Individual tests can be easily run by calling the run_test function provided by the validmind.tests module. The function takes the following arguments:

  • test_id: The ID of the test to run. To find a particular test and get its, refer to the explore_tests.ipynb notebook
  • params: A dictionary of parameters for the test. These will override any default_params set in the test definition. Refer to the explore_tests.ipynb notebook to find the default parameters for a test.

You can then pass in any inputs for the test as keyword arguments. Most likely, these will be dataset and model objects. Again, you may refer to the explore_tests.ipynb notebook to find the required inputs for a test.

test = vm.tests.run_test(
    test_id="validmind.model_validation.sklearn.TrainingTestDegradation",
    params={},  # can be used to set overrides to the test's default parameters
    inputs={"model": vm_model, "datasets": (vm_train_ds, vm_test_ds)},
)

Log the test results to ValidMind

After the test has been run, you can save the results to ValidMind by calling the log method of the test object returned after running the test:

test.log()