Test descriptions

Published

June 13, 2024

This topic describes the tests that are available as part of the ValidMind Developer Framework, grouped by type of validation test.

Try the test sandbox (BETA)

Explore our interactive sandbox to see what tests are available in the ValidMind Developer Framework.

ACFandPACFPlot
Analyzes time series data using Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots to reveal trends and correlations.
ADF
Assesses the stationarity of a time series dataset using the Augmented Dickey-Fuller (ADF) test.
ANOVAOneWayTable
Applies one-way ANOVA (Analysis of Variance) to identify statistically significant numerical features in the dataset.
AutoAR
Automatically identifies the optimal Autoregressive (AR) order for a time series using BIC and AIC criteria.
AutoMA
Automatically selects the optimal Moving Average (MA) order for each variable in a time series dataset based on minimal BIC and AIC values.
AutoSeasonality
Automatically identifies and quantifies optimal seasonality in time series data to improve forecasting model performance.
AutoStationarity
Automates Augmented Dickey-Fuller test to assess stationarity across multiple time series in a DataFrame.
BivariateFeaturesBarPlots
Generates visual bar plots to analyze the relationship between paired features within categorical data in the model.
BivariateHistograms
Generates bivariate histograms for paired features, aiding in visual inspection of categorical variables distributions and correlations.
BivariateScatterPlots
Generates bivariate scatterplots to visually inspect relationships between pairs of predictor variables in machine learning classification tasks.
ChiSquaredFeaturesTable
Executes Chi-Squared test for each categorical feature against a target column to assess significant association.
ClassImbalance
Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.
CommonWords
Identifies and visualizes the 40 most frequent non-stopwords in a specified text column within a dataset.
DFGLSArch
Executes Dickey-Fuller GLS metric to determine order of integration and check stationarity in time series data.
DatasetDescription
Provides comprehensive analysis and statistical summaries of each field in a machine learning model’s dataset.
DatasetSplit
Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML model.
DescriptiveStatistics
Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model’s dataset.
Duplicates
Tests dataset for duplicate entries, ensuring model reliability via data quality verification.
EngleGrangerCoint
Validates co-integration in pairs of time series data using the Engle-Granger test and classifies them as Cointegrated’ or ‘Not cointegrated’.
FeatureTargetCorrelationPlot
Visualizes the correlation between input features and model’s target output in a color-coded horizontal bar plot.
Hashtags
Assesses hashtag frequency in a text column, highlighting usage trends and potential dataset bias or spam.
HeatmapFeatureCorrelations
Creates a heatmap to visually represent correlation patterns between pairs of numerical features in a dataset.
HighCardinality
Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.
HighPearsonCorrelation
Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.
IQROutliersBarPlot
Visualizes outlier distribution across percentiles in numerical data using Interquartile Range (IQR) method.
IQROutliersTable
Determines and summarizes outliers in numerical features using Interquartile Range method.
IsolationForestOutliers
Detects outliers in a dataset using the Isolation Forest algorithm and visualizes results through scatter plots.
KPSS
Executes KPSS unit root test to validate stationarity of time-series data in machine learning model.
LaggedCorrelationHeatmap
Assesses and visualizes correlation between target variable and lagged independent variables in a time-series dataset.
LanguageDetection
Mentions
Calculates and visualizes frequencies of ‘@’ prefixed mentions in a text-based dataset for NLP model analysis.
MissingValues
Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold.
MissingValuesBarPlot
Creates a bar plot showcasing the percentage of missing values in each column of the dataset with risk categorization based on a user-defined threshold.
MissingValuesRisk
Assesses and quantifies the risk related to missing values in a dataset used for training an ML model.
PearsonCorrelationMatrix
Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.
PhillipsPerronArch
Executes Phillips-Perron test to assess the stationarity of time series data in each ML model feature.
PolarityAndSubjectivity
Punctuations
Analyzes and visualizes the frequency distribution of punctuation usage in a given text dataset.
RollingStatsPlot
This test evaluates the stationarity of time series data by plotting its rolling mean and standard deviation.
ScatterPlot
Creates a scatter plot matrix to visually analyze feature relationships, patterns, and outliers in a dataset.
SeasonalDecompose
Decomposes dataset features into observed, trend, seasonal, and residual components to identify patterns and validate dataset.
Sentiment
Skewness
Evaluates the skewness of numerical data in a machine learning model and checks if it falls below a set maximum threshold.
SpreadPlot
Visualizes the spread relationship between pairs of time-series variables in a dataset, thereby aiding in identification of potential correlations.
StopWords
Evaluates and visualizes the frequency of English stop words in a text dataset against a defined threshold.
TabularCategoricalBarPlots
Generates and visualizes bar plots for each category in categorical features to evaluate dataset’s composition.
TabularDateTimeHistograms
Generates histograms to provide graphical insight into the distribution of time intervals in model’s datetime data.
TabularDescriptionTables
Summarizes key descriptive statistics for numerical, categorical, and datetime variables in a dataset.
TabularNumericalHistograms
Generates histograms for each numerical feature in a dataset to provide visual insights into data distribution and detect potential issues.
TargetRateBarPlots
Generates bar plots visualizing the default rates of categorical features for a classification machine learning model.
TextDescription
Performs comprehensive textual analysis on a dataset using NLTK, evaluating various parameters and generating visualizations.
TimeSeriesDescription
TimeSeriesDescriptiveStatistics
TimeSeriesFrequency
Evaluates consistency of time series data frequency and generates a frequency plot.
TimeSeriesHistogram
TimeSeriesLinePlot
Generates and analyses time-series data through line plots revealing trends, patterns, anomalies over time.
TimeSeriesMissingValues
Validates time-series data quality by confirming the count of missing values is below a certain threshold.
TimeSeriesOutliers
Identifies and visualizes outliers in time-series data using z-score method.
TooManyZeroValues
Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold percentage.
Toxicity
UniqueRows
Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.
WOEBinPlots
Generates visualizations of Weight of Evidence (WoE) and Information Value (IV) for understanding predictive power of categorical variables in a data set.
WOEBinTable
Calculates and assesses the Weight of Evidence (WoE) and Information Value (IV) of each feature in a ML model.
ZivotAndrewsArch
Evaluates the order of integration and stationarity of time series data using Zivot-Andrews unit root test.
No matching items
AdjustedMutualInformation
Evaluates clustering model performance by measuring mutual information between true and predicted labels, adjusting for chance.
AdjustedRandIndex
Measures the similarity between two data clusters using the Adjusted Rand Index (ARI) metric in clustering machine learning models.
AnswerCorrectness
AnswerRelevance
AnswerSimilarity
AspectCritique
AutoARIMA
Evaluates ARIMA models for time-series forecasting, ranking them using Bayesian and Akaike Information Criteria.
BertScore
BleuScore
BoxPierce
Detects autocorrelation in time-series data through the Box-Pierce test to validate model performance.
ClassifierPerformance
Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy, and ROC AUC scores.
ClusterCosineSimilarity
Measures the intra-cluster similarity of a clustering model using cosine similarity.
ClusterDistribution
Assesses the distribution of text embeddings across clusters produced by a model using KMeans clustering.
ClusterPerformance
Evaluates and compares a clustering model’s performance on training and testing datasets using multiple defined metrics.
ClusterPerformanceMetrics
Evaluates the performance of clustering machine learning models using multiple established metrics.
ClusterSizeDistribution
Compares and visualizes the distribution of cluster sizes in model predictions and actual data for assessing clustering model performance.
CompletenessScore
Evaluates a clustering model’s capacity to categorize instances from a single class into the same cluster.
ConfusionMatrix
Evaluates and visually represents the classification ML model’s predictive performance using a Confusion Matrix heatmap.
ContextEntityRecall
ContextPrecision
ContextRecall
ContextRelevancy
ContextualRecall
CosineSimilarityComparison
CosineSimilarityDistribution
Assesses the similarity between predicted text embeddings from a model using a Cosine Similarity distribution histogram.
CosineSimilarityHeatmap
CumulativePredictionProbabilities
Visualizes cumulative probabilities of positive and negative classes for both training and testing in logistic regression models.
DescriptiveAnalytics
Evaluates statistical properties of text embeddings in an ML model via mean, median, and standard deviation histograms.
DurbinWatsonTest
Assesses autocorrelation in time series data features using the Durbin-Watson statistic.
EmbeddingsVisualization2D
Visualizes 2D representation of text embeddings generated by a model using t-SNE technique.
EuclideanDistanceComparison
EuclideanDistanceHeatmap
Faithfulness
FeatureImportanceComparison
FeaturesAUC
Evaluates the discriminatory power of each individual feature within a binary classification model by calculating the Area Under the Curve (AUC) for each feature separately.
FowlkesMallowsScore
Evaluates the similarity between predicted and actual cluster assignments in a model using the Fowlkes-Mallows score.
GINITable
Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets.
HomogeneityScore
Assesses clustering homogeneity by comparing true and predicted labels, scoring from 0 (heterogeneous) to 1 (homogeneous).
HyperParametersTuning
Exerts exhaustive grid search to identify optimal hyperparameters for the model, improving performance.
JarqueBera
Assesses normality of dataset features in an ML model using the Jarque-Bera test.
KMeansClustersOptimization
Optimizes the number of clusters in K-means models using Elbow and Silhouette methods.
KolmogorovSmirnov
Executes a feature-wise Kolmogorov-Smirnov test to evaluate alignment with normal distribution in datasets.
LJungBox
Assesses autocorrelations in dataset features by performing a Ljung-Box test on each feature.
Lilliefors
Assesses the normality of feature distributions in an ML model’s training dataset using the Lilliefors test.
MeteorScore
MinimumAccuracy
Checks if the model’s prediction accuracy meets or surpasses a specified threshold.
MinimumF1Score
Evaluates if the model’s F1 score on the validation set meets a predefined minimum threshold.
MinimumROCAUCScore
Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.
ModelMetadata
Extracts and summarizes critical metadata from a machine learning model instance for comprehensive analysis.
ModelMetadataComparison
ModelPredictionResiduals
ModelsPerformanceComparison
Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy, precision, recall, and F1 score.
OverfitDiagnosis
Detects and visualizes overfit regions in an ML model by comparing performance on training and test datasets.
PCAComponentsPairwisePlots
PermutationFeatureImportance
Assesses the significance of each feature in a model by evaluating the impact on model performance when feature values are randomly rearranged.
PopulationStabilityIndex
Evaluates the Population Stability Index (PSI) to quantify the stability of an ML model’s predictions across different datasets.
PrecisionRecallCurve
Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.
PredictionProbabilitiesHistogram
Generates and visualizes histograms of the Probability of Default predictions for both positive and negative classes in training and testing datasets.
ROCCurve
Evaluates binary classification model performance by generating and plotting the Receiver Operating Characteristic (ROC) curve and calculating the Area Under Curve (AUC) score.
RegardScore
RegressionCoeffsPlot
Visualizes regression coefficients with 95% confidence intervals to assess predictor variables’ impact on response variable.
RegressionErrors
Purpose: This metric is used to measure the performance of a regression model. It gauges the model’s accuracy by computing several error metrics such as Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Mean…
RegressionErrorsComparison
RegressionFeatureSignificance
Assesses and visualizes the statistical significance of features in a set of regression models.
RegressionModelForecastPlot
Generates plots to visually compare the forecasted outcomes of one or more regression models against actual observed values over a specified date range.
RegressionModelForecastPlotLevels
Compares and visualizes forecasted and actual values of regression models on both raw and transformed datasets.
RegressionModelSensitivityPlot
Tests the sensitivity of a regression model to variations in independent variables by applying shocks and visualizing the effects.
RegressionModelSummary
Evaluates regression model performance using metrics including R-Squared, Adjusted R-Squared, MSE, and RMSE.
RegressionModelsCoeffs
Compares feature importance by evaluating and contrasting coefficients of different regression models.
RegressionModelsPerformanceComparison
Compares and evaluates the performance of multiple regression models using five different metrics: MAE, MSE, RMSE, MAPE, and MBD.
RegressionPermutationFeatureImportance
Assesses the significance of each feature in a model by evaluating the impact on model performance when feature values are randomly rearranged. Specifically designed for use with statsmodels, this metric offers insight into the importance of…
RegressionR2Square
Purpose: The purpose of the RegressionR2Square Metric test is to measure the overall goodness-of-fit of a regression model. Specifically, this Python-based test evaluates the R-squared (R2) and Adjusted R-squared (Adj R2) scores: two statistical…
RegressionR2SquareComparison
RegressionResidualsPlot
Evaluates regression model performance using residual distribution and actual vs. predicted plots.
RobustnessDiagnosis
Evaluates the robustness of a machine learning model by injecting Gaussian noise to input data and measuring performance.
RougeScore
RunsTest
Executes Runs Test on ML model to detect non-random patterns in output data sequence.
SHAPGlobalImportance
Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.
ScorecardHistogram
Creates histograms of credit scores, from both default and non-default instances, generated by a credit-risk model.
ShapiroWilk
Evaluates feature-wise normality of training data using the Shapiro-Wilk test.
SilhouettePlot
Calculates and visualizes Silhouette Score, assessing degree of data point suitability to its cluster in ML models.
StabilityAnalysis
Base class for embeddings stability analysis tests
StabilityAnalysisKeyword
Evaluate robustness of embeddings models to keyword swaps on the test dataset
StabilityAnalysisRandomNoise
Evaluate robustness of embeddings models to random noise introduced by using a probability parameter to choose random locations in the text to apply random perturbations. These perturbations include:
StabilityAnalysisSynonyms
Evaluates the stability of text embeddings models when words in test data are replaced by their synonyms randomly.
StabilityAnalysisTranslation
Evaluate robustness of embeddings models to noise introduced by translating the original text to another language and back.
TSNEComponentsPairwisePlots
TimeSeriesPredictionWithCI
TimeSeriesPredictionsPlot
TimeSeriesR2SquareBySegments
TokenDisparity
ToxicityScore
TrainingTestDegradation
Tests if model performance degradation between training and test datasets exceeds a predefined threshold.
VMeasure
Evaluates homogeneity and completeness of a clustering model using the V Measure Score.
WeakspotsDiagnosis
Identifies and visualizes weak spots in a machine learning model’s performance across various sections of the feature space.
No matching items