Test descriptions

Published

April 3, 2024

This topic describes the tests that are available as part of the ValidMind Developer Framework, grouped by type of validation test.

Try the test sandbox (BETA)

Explore our interactive sandbox to see what tests are available in the ValidMind Developer Framework.

ACFandPACFPlot
Analyzes time series data using Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots to reveal trends and correlations.
ANOVAOneWayTable
Applies one-way ANOVA (Analysis of Variance) to identify statistically significant numerical features in the dataset.
AutoAR
Automatically identifies the optimal Autoregressive (AR) order for a time series using BIC and AIC criteria.
AutoMA
Automatically selects the optimal Moving Average (MA) order for each variable in a time series dataset based on minimal BIC and AIC values.
AutoSeasonality
Automatically identifies and quantifies optimal seasonality in time series data to improve forecasting model performance.
AutoStationarity
Automates Augmented Dickey-Fuller test to assess stationarity across multiple time series in a DataFrame.
BivariateFeaturesBarPlots
Generates visual bar plots to analyze the relationship between paired features within categorical data in the model.
BivariateHistograms
Generates bivariate histograms for paired features, aiding in visual inspection of categorical variables distributions and correlations.
BivariateScatterPlots
Generates bivariate scatterplots to visually inspect relationships between pairs of predictor variables in machine learning classification tasks.
ChiSquaredFeaturesTable
Executes Chi-Squared test for each categorical feature against a target column to assess significant association.
ClassImbalance
Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.
CommonWords
Identifies and visualizes the 40 most frequent non-stopwords in a specified text column within a dataset.
DatasetDescription
Provides comprehensive analysis and statistical summaries of each field in a machine learning model’s dataset.
DatasetSplit
Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML model.
DefaultRatesbyRiskBandPlot
Generates a bar plot showcasing the distribution of default rates across different risk bands in a dataset.
DescriptiveStatistics
Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model’s dataset.
Duplicates
Tests dataset for duplicate entries, ensuring model reliability via data quality verification.
EngleGrangerCoint
Validates co-integration in pairs of time series data using the Engle-Granger test and classifies them as Cointegrated’ or ‘Not cointegrated’.
FeatureTargetCorrelationPlot
Visualizes the correlation between input features and model’s target output in a color-coded horizontal bar plot.
Hashtags
Assesses hashtag frequency in a text column, highlighting usage trends and potential dataset bias or spam.
HeatmapFeatureCorrelations
Creates a heatmap to visually represent correlation patterns between pairs of numerical features in a dataset.
HighCardinality
Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.
HighPearsonCorrelation
Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.
IQROutliersBarPlot
Visualizes outlier distribution across percentiles in numerical data using Interquartile Range (IQR) method.
IQROutliersTable
Determines and summarizes outliers in numerical features using Interquartile Range method.
IsolationForestOutliers
Detects outliers in a dataset using the Isolation Forest algorithm and visualizes results through scatter plots.
LaggedCorrelationHeatmap
Assesses and visualizes correlation between target variable and lagged independent variables in a time-series dataset.
Mentions
Calculates and visualizes frequencies of ‘@’ prefixed mentions in a text-based dataset for NLP model analysis.
MissingValues
Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold.
MissingValuesBarPlot
Creates a bar plot showcasing the percentage of missing values in each column of the dataset with risk categorization based on a user-defined threshold.
MissingValuesRisk
Assesses and quantifies the risk related to missing values in a dataset used for training an ML model.
PearsonCorrelationMatrix
Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.
PiTCreditScoresHistogram
Generates a histogram visualization for observed and predicted credit default scores.
PiTPDHistogram
Assesses credit risk prediction accuracy of a model by comparing actual and predicted defaults at a chosen point in time.
Punctuations
Analyzes and visualizes the frequency distribution of punctuation usage in a given text dataset.
RollingStatsPlot
This test evaluates the stationarity of time series data by plotting its rolling mean and standard deviation.
ScatterPlot
Creates a scatter plot matrix to visually analyze feature relationships, patterns, and outliers in a dataset.
SeasonalDecompose
Decomposes dataset features into observed, trend, seasonal, and residual components to identify patterns and validate dataset.
Skewness
Evaluates the skewness of numerical data in a machine learning model and checks if it falls below a set maximum threshold.
SpreadPlot
Visualizes the spread relationship between pairs of time-series variables in a dataset, thereby aiding in identification of potential correlations.
StopWords
Evaluates and visualizes the frequency of English stop words in a text dataset against a defined threshold.
TabularCategoricalBarPlots
Generates and visualizes bar plots for each category in categorical features to evaluate dataset’s composition.
TabularDateTimeHistograms
Generates histograms to provide graphical insight into the distribution of time intervals in model’s datetime data.
TabularDescriptionTables
Summarizes key descriptive statistics for numerical, categorical, and datetime variables in a dataset.
TabularNumericalHistograms
Generates histograms for each numerical feature in a dataset to provide visual insights into data distribution and detect potential issues.
TargetRateBarPlots
Generates bar plots visualizing the default rates of categorical features for a classification machine learning model.
TextDescription
Performs comprehensive textual analysis on a dataset using NLTK, evaluating various parameters and generating visualizations.
TimeSeriesFrequency
Evaluates consistency of time series data frequency and generates a frequency plot.
TimeSeriesHistogram
Visualizes distribution of time-series data using histograms and Kernel Density Estimation (KDE) lines.
TimeSeriesLinePlot
Generates and analyses time-series data through line plots revealing trends, patterns, anomalies over time.
TimeSeriesMissingValues
Validates time-series data quality by confirming the count of missing values is below a certain threshold.
TimeSeriesOutliers
Identifies and visualizes outliers in time-series data using z-score method.
TooManyZeroValues
Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold percentage.
UniqueRows
Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.
WOEBinPlots
Generates visualizations of Weight of Evidence (WoE) and Information Value (IV) for understanding predictive power of categorical variables in a data set.
WOEBinTable
Calculates and assesses the Weight of Evidence (WoE) and Information Value (IV) of each feature in a ML model.
No matching items
ADF
Assesses the stationarity of a time series dataset using the Augmented Dickey-Fuller (ADF) test.
ADFTest
Assesses the stationarity of time series data using the Augmented Dickey-Fuller (ADF) test.
AdjustedMutualInformation
Evaluates clustering model performance by measuring mutual information between true and predicted labels, adjusting for chance.
AdjustedRandIndex
Measures the similarity between two data clusters using the Adjusted Rand Index (ARI) metric in clustering machine learning models.
AutoARIMA
Evaluates ARIMA models for time-series forecasting, ranking them using Bayesian and Akaike Information Criteria.
BertScore
Evaluates text generation models’ performance by calculating precision, recall, and F1 score based on BERT contextual embeddings.
BertScoreAggregate
Evaluates the aggregate performance of text generation models by computing the average precision, recall, and F1 score based on BERT contextual embeddings.
BleuScore
Assesses translation quality by comparing machine-translated sentences with human-translated ones using BLEU score.
BoxPierce
Detects autocorrelation in time-series data through the Box-Pierce test to validate model performance.
ClassifierPerformance
Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.
ClusterCosineSimilarity
Measures the intra-cluster similarity of a clustering model using cosine similarity.
ClusterDistribution
Assesses the distribution of text embeddings across clusters produced by a model using KMeans clustering.
ClusterPerformance
Evaluates and compares a clustering model’s performance on training and testing datasets using multiple defined metrics.
ClusterPerformanceMetrics
Evaluates the performance of clustering machine learning models using multiple established metrics.
ClusterSizeDistribution
Compares and visualizes the distribution of cluster sizes in model predictions and actual data for assessing clustering model performance.
CompletenessScore
Evaluates a clustering model’s capacity to categorize instances from a single class into the same cluster.
ConfusionMatrix
Evaluates and visually represents the classification ML model’s predictive performance using a Confusion Matrix heatmap.
ContextualRecall
Evaluates a Natural Language Generation model’s ability to generate contextually relevant and factually correct text.
CosineSimilarityDistribution
Assesses the similarity between predicted text embeddings from a model using a Cosine Similarity distribution histogram.
DFGLSArch
Executes Dickey-Fuller GLS metric to determine order of integration and check stationarity in time series data.
DescriptiveAnalytics
Evaluates statistical properties of text embeddings in an ML model via mean, median, and standard deviation histograms.
DurbinWatsonTest
Assesses autocorrelation in time series data features using the Durbin-Watson statistic.
EmbeddingsVisualization2D
Visualizes 2D representation of text embeddings generated by a model using t-SNE technique.
FeatureImportanceAndSignificance
Evaluates and visualizes the statistical significance and feature importance using regression and decision tree models.
FowlkesMallowsScore
Evaluates the similarity between predicted and actual cluster assignments in a model using the Fowlkes-Mallows score.
GINITable
Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets.
HomogeneityScore
Assesses clustering homogeneity by comparing true and predicted labels, scoring from 0 (heterogeneous) to 1 (homogeneous).
HyperParametersTuning
Exerts exhaustive grid search to identify optimal hyperparameters for the model, improving performance.
JarqueBera
Assesses normality of dataset features in an ML model using the Jarque-Bera test.
KMeansClustersOptimization
Optimizes the number of clusters in K-means models using Elbow and Silhouette methods.
KPSS
Executes KPSS unit root test to validate stationarity of time-series data in machine learning model.
KolmogorovSmirnov
Executes a feature-wise Kolmogorov-Smirnov test to evaluate alignment with normal distribution in datasets.
LJungBox
Assesses autocorrelations in dataset features by performing a Ljung-Box test on each feature.
Lilliefors
Assesses the normality of feature distributions in an ML model’s training dataset using the Lilliefors test.
LogRegressionConfusionMatrix
Generates a confusion matrix for logistic regression model performance, utilizing thresholded probabilities for classification assessments.
LogisticRegCumulativeProb
Visualizes cumulative probabilities of positive and negative classes for both training and testing in logistic regression models.
LogisticRegPredictionHistogram
Generates and visualizes histograms of the Probability of Default predictions for both positive and negative classes in training and testing datasets.
MinimumAccuracy
Checks if the model’s prediction accuracy meets or surpasses a specified threshold.
MinimumF1Score
Evaluates if the model’s F1 score on the validation set meets a predefined minimum threshold.
MinimumROCAUCScore
Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.
ModelMetadata
Extracts and summarizes critical metadata from a machine learning model instance for comprehensive analysis.
ModelsPerformanceComparison
Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy, precision, recall, and F1 score.
OverfitDiagnosis
Detects and visualizes overfit regions in an ML model by comparing performance on training and test datasets.
PDRatingClassPlot
Assesses and visualizes credit risk distribution across different rating classes within a dataset via default probabilities.
PermutationFeatureImportance
Assesses the significance of each feature in a model by evaluating the impact on model performance when feature values are randomly rearranged.
PhillipsPerronArch
Executes Phillips-Perron test to assess the stationarity of time series data in each ML model feature.
PopulationStabilityIndex
Evaluates the Population Stability Index (PSI) to quantify the stability of an ML model’s predictions across different datasets.
PrecisionRecallCurve
Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.
ROCCurve
Evaluates binary classification model performance by generating and plotting the Receiver Operating Characteristic (ROC) curve and calculating the Area Under Curve (AUC) score.
RegardHistogram
Purpose: The RegardHistogram metric offers a histogram-based visualization of regard scores across different text…
RegardScore
Purpose: The RegardScore metric assesses the degree of regard—positive, negative, neutral, or other—present in the given text, whether it’s a classification or summarization…
RegressionCoeffsPlot
Visualizes regression coefficients with 95% confidence intervals to assess predictor variables’ impact on response variable.
RegressionErrors
Purpose: This metric is used to measure the performance of a regression model. It gauges the model’s…
RegressionFeatureSignificance
Assesses and visualizes the statistical significance of features in a set of regression models.
RegressionModelForecastPlot
Generates plots to visually compare the forecasted outcomes of one or more regression models against actual observed values over a specified date range.
RegressionModelForecastPlotLevels
Compares and visualizes forecasted and actual values of regression models on both raw and transformed datasets.
RegressionModelInsampleComparison
Evaluates and compares in-sample performance of multiple regression models using R-Squared, Adjusted R-Squared, MSE, and RMSE.
RegressionModelOutsampleComparison
Computes MSE and RMSE for multiple regression models using out-of-sample test to assess model’s prediction accuracy on unseen data.
RegressionModelSensitivityPlot
Tests the sensitivity of a regression model to variations in independent variables by applying shocks and visualizing the effects.
RegressionModelSummary
Evaluates regression model performance using metrics including R-Squared, Adjusted R-Squared, MSE, and RMSE.
RegressionModelsCoeffs
Compares feature importance by evaluating and contrasting coefficients of different regression models.
RegressionModelsPerformance
Evaluates and compares regression models’ performance using R-squared, Adjusted R-squared, and MSE metrics.
RegressionModelsPerformanceComparison
Compares and evaluates the performance of multiple regression models using five different metrics: MAE, MSE, RMSE, MAPE, and MBD.
RegressionR2Square
Purpose: The purpose of the RegressionR2Square Metric test is to measure the overall goodness-of-fit of a regression model. Specifically, this…
ResidualsVisualInspection
Provides a comprehensive visual analysis of residuals for regression models utilizing various plot types.
RobustnessDiagnosis
Evaluates the robustness of a machine learning model by injecting Gaussian noise to input data and measuring performance.
RougeMetrics
Evaluates the quality of machine-generated text using various ROUGE metrics, and visualizes the results.
RougeMetricsAggregate
Evaluates the average quality of machine-generated text using various ROUGE metrics and visualizes the aggregated results.
RunsTest
Executes Runs Test on ML model to detect non-random patterns in output data sequence.
SHAPGlobalImportance
Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.
ScorecardBucketHistogram
Evaluates and visualizes distribution of risk categories in a classification model’s scores, useful in credit risk assessment.
ScorecardHistogram
Creates histograms of credit scores, from both default and non-default instances, generated by a credit-risk model.
ScorecardProbabilitiesHistogram
Evaluates risk classification of a model by visualizing the distribution of default probability across score buckets.
ShapiroWilk
Evaluates feature-wise normality of training data using the Shapiro-Wilk test.
SilhouettePlot
Calculates and visualizes Silhouette Score, assessing degree of data point suitability to its cluster in ML models.
StabilityAnalysis
Base class for embeddings stability analysis tests
StabilityAnalysisKeyword
Evaluate robustness of embeddings models to keyword swaps on the test dataset
StabilityAnalysisRandomNoise
Evaluate robustness of embeddings models to random noise introduced by using a probability parameter to choose random locations in the text to apply random perturbations. These perturbations include:
StabilityAnalysisSynonyms
Evaluates the stability of text embeddings models when words in test data are replaced by their synonyms randomly.
StabilityAnalysisTranslation
Evaluate robustness of embeddings models to noise introduced by translating the original text to another language and back.
TokenDisparity
Assess and visualize token count disparity between model’s predicted and actual dataset.
ToxicityHistogram
Purpose: The ToxicityHistogram metric visualizes and analyzes the toxicity scores of various texts. Through histograms, it provides insights into the distribution and nature of toxicity present in the evaluated text segments.
ToxicityScore
Purpose: The…
TrainingTestDegradation
Tests if model performance degradation between training and test datasets exceeds a predefined threshold.
VMeasure
Evaluates homogeneity and completeness of a clustering model using the V Measure Score.
WeakspotsDiagnosis
Identifies and visualizes weak spots in a machine learning model’s performance across various sections of the feature space.
ZivotAndrewsArch
Evaluates the order of integration and stationarity of time series data using Zivot-Andrews unit root test.
No matching items