Time Series Forecasting Model Tutorial

This notebook guides model developers through the process of automatically documenting and testing time series forecasting models. It shows you how to use the ValidMind Developer Framework to import and prepare data and before running a data validation test suite, followed by loading a pre-trained model and running a model validation test suite.

As part of the notebook, you will learn how to:

ValidMind at a glance

ValidMind’s platform enables organizations to identify, document, and manage model risks for all types of models, including AI/ML models, LLMs, and statistical models. As a model developer, you use the ValidMind Developer Framework to automate documentation and validation tests, and then use the ValidMind AI Risk Platform UI to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators.

If this is your first time trying out ValidMind, we recommend going through the following resources first:

Before you begin

New to ValidMind?

To access the ValidMind Platform UI, you’ll need an account.

Signing up is FREE — Create your account.

If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.

Install the client library

The client library provides Python support for the ValidMind Developer Framework. To install it:

%pip install -q validmind

Initialize the client library

ValidMind generates a unique code snippet for each registered model to connect with your developer environment. You initialize the client library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.

Get your code snippet:

  1. In a browser, log into the Platform UI.

  2. In the left sidebar, navigate to Model Inventory and click + Register new model.

  3. Enter the model details, making sure to select Time Series Forecasting as the template and Credit Risk - Underwriting - Loan as the use case, and click Continue. (Need more help?)

  4. Go to Getting Started and click Copy snippet to clipboard.

Next, replace this placeholder with your own code snippet:

# Replace with your code snippet

import validmind as vm

vm.init(
  api_host = "...",
    api_host="https://api.prod.validmind.ai/api/v1/tracking",
    api_key="...",
    api_secret="...",
    project="..."
)

Explore available test suites

In this notebook we will run a collection of test suites that are available in the ValidMind Developer Framework. Test suites group together a collection of tests that are relevant for a specific use case. In our case, we will run test different test suites for time series forecasting models. Once a test suite runs successfully, its results will be automatically uploaded to the ValidMind platform.

vm.test_suites.list_suites()

For our example use case we will run the following test suites:

  • time_series_dataset
  • time_series_model_validation

Step 1: Import raw data

Import FRED dataset

Federal Reserve Economic Data, or FRED, is a comprehensive database maintained by the Federal Reserve Bank of St. Louis. It offers a wide array of economic data from various sources, including U.S. government agencies and international organizations. The dataset encompasses numerous economic indicators across various categories such as employment, consumer price indices, money supply, and gross domestic product, among others.

FRED provides a valuable resource for researchers, policymakers, and anyone interested in understanding economic trends and conducting economic analysis. The platform also includes tools for data visualization, which can help users interpret complex economic data and identify trends over time.

The following code snippet imports a sample FRED dataset into a Pandas dataframe:

from validmind.datasets.regression import fred as demo_dataset

target_column = demo_dataset.target_column
feature_columns = demo_dataset.feature_columns

df = demo_dataset.load_data()
df.tail(10)

Step 2: Run data validation test suite on raw data

Explore the time series dataset test suites

Let’s see what tests are included on each test suite:

vm.test_suites.describe_suite("time_series_data_quality")
vm.test_suites.describe_suite("time_series_univariate")

Initialize the dataset

Use the ValidMind Developer Framework to initialize the dataset object:

vm_dataset = vm.init_dataset(
    input_id="raw_dataset",
    dataset=df,
    target_column=demo_dataset.target_column,
)

Run time series dataset test suite on raw dataset

Next, use the ValidMind Developer Framework to run the test suite for time series datasets:

config = {
    # TIME SERIES DATA QUALITY PARAMS
    "validmind.data_validation.TimeSeriesOutliers": {
        "params": {
            "zscore_threshold": 3
        }
    },
    "validmind.data_validation.TimeSeriesMissingValues": {
        "params": {
            "min_threshold": 2
        }
    },

    # TIME SERIES UNIVARIATE PARAMS
    "validmind.data_validation.RollingStatsPlot": {
        "params": {
            "window_size": 12
        }
    },
    "validmind.data_validation.SeasonalDecompose": {
        "params": {
            "seasonal_model": "additive"
        }
    },
    "validmind.data_validation.AutoSeasonality": {
        "params": {
            "min_period": 1,
            "max_period": 3
        }
    },
    "validmind.data_validation.AutoStationarity": {
        "params": {
            "max_order": 3,
            "threshold": 0.05
        }
    },
    "validmind.data_validation.AutoAR": {
        "params": {
            "max_ar_order": 2
        }
    },
    "validmind.data_validation.AutoMA": {
        "params": {
            "max_ma_order": 2
        }
    },

    # TIME SERIES MULTIVARIATE PARAMS
    "validmind.data_validation.LaggedCorrelationHeatmap": {
        "params": {
            "target_col": demo_dataset.target_column,
            "independent_vars": demo_dataset.feature_columns
        }
    },
    "validmind.data_validation.EngleGrangerCoint": {
        "params": {
            "threshold": 0.05
        }
    },
}

full_suite = vm.run_test_suite(
    "time_series_dataset",
    inputs = {
        "dataset": vm_dataset,
    },
    config=config,
)

Step 3: Preprocess data

Handle frequencies, missing values and stationairty

# Sample frequencies to Monthly
resampled_df = df.resample("MS").last()

#  Remove all missing values
nona_df = resampled_df.dropna()

#  Take the first different across all variables
preprocessed_df = nona_df.diff().dropna()

Step 4: Run data validation test suite on processed data

vm_dataset = vm.init_dataset(
    input_id="preprocess_dataset",
    dataset=preprocessed_df,
    target_column=demo_dataset.target_column,
)

full_suite = vm.run_test_suite(
    "time_series_dataset",
    inputs={"dataset": vm_dataset},
    config=config,
)

Step 5: Load pre-trained models

Load pre-trained models

from validmind.datasets.regression import fred as demo_dataset

model_A, train_df_A, test_df_A = demo_dataset.load_model('fred_loan_rates_model_3')
model_B, train_df_B, test_df_B = demo_dataset.load_model('fred_loan_rates_model_4')

Initialize Validmind models

# Initialize training and testing datasets for model A
vm_train_ds_A = vm.init_dataset(
    input_id="train_a_dataset",
    dataset=train_df_A,
    target_column=demo_dataset.target_column
)
vm_test_ds_A = vm.init_dataset(
    input_id="test_a_dataset",
    dataset=test_df_A,
    target_column=demo_dataset.target_column
)

# Initialize training and testing datasets for model B
vm_train_ds_B = vm.init_dataset(
    input_id="train_b_dataset",
    dataset=train_df_B,
    target_column=demo_dataset.target_column
)
vm_test_ds_B = vm.init_dataset(
    input_id="test_b_dataset",
    dataset=test_df_B,
    target_column=demo_dataset.target_column
)

# Initialize model A
vm_model_A = vm.init_model(
    input_id="model_a",
    model=model_A,

)

vm_train_ds_A.assign_predictions(model=vm_model_A)
vm_test_ds_A.assign_predictions(model=vm_model_A)

# Initialize model B
vm_model_B = vm.init_model(
    input_id="model_b",
    model=model_B,
)
vm_train_ds_B.assign_predictions(model=vm_model_B)
vm_test_ds_B.assign_predictions(model=vm_model_B)

models = [vm_model_A, vm_model_B]

Step 6: Run model validation test suite on models

Explore the time series model validation test suite

vm.test_suites.describe_test_suite("time_series_model_validation")

Run model validation test suite on a list of models

config = {
    "validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels": {
        "params":{
            "transformation": "integrate",
        }
    },
    "validmind.model_validation.statsmodels.RegressionModelSensitivityPlot": {
        "params":{
            "transformation": "integrate",
            "shocks": [0.3],
        }
    },
    "validmind.model_validation.statsmodels.RegressionModelsPerformance": {
        "inputs":{
            "in_sample_datasets": (vm_train_ds_A, vm_train_ds_B),
            "out_of_sample_datasets": (vm_test_ds_A, vm_test_ds_B),
            "models": models,
        },
        "params":{
            "transformation": "integrate",
            "shocks": [0.3],
        }
    },
    "validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels": {
        "inputs":{
            "datasets": (vm_train_ds_A, vm_test_ds_A),
            "models": [vm_model_A],
        },
    },
    "validmind.model_validation.statsmodels.RegressionModelSensitivityPlot": {
        "inputs":{
            "datasets": (vm_train_ds_A, vm_test_ds_A),
            "models": [vm_model_A],
        },
    }
}

full_suite = vm.run_test_suite(
    "time_series_model_validation",
    inputs = {
        "dataset": vm_train_ds_A,
        "datasets": (vm_train_ds_A, vm_test_ds_A),
        "model": vm_model_A,
        "models": models,
    },
    config=config,
)

Next steps

You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation right in the ValidMind Platform UI:

  1. In the Platform UI, go to the Documentation page for the model you registered earlier.

  2. Expand 3. Model Development to review all test results.

What you can see now is a more easily consumable version of the prompt validation testing you just performed, along with other parts of your model documentation that still need to be completed.

If you want to learn more about where you are in the model documentation process, take a look at How do I use the framework?.