This notebook shows model developers how they can learn more about the test suites and tests that are available in the ValidMind Developer Framework. These instructions include the code required to get:
If you encounter errors due to missing modules in your Python environment, install the modules with pip install, and then re-run the notebook. For more help, refer to Installing Python Modules.
To get the list of all test suites available in the ValidMind Developer Framework:
| ID | Name | Description | Tests |
|---|---|---|---|
| classifier_model_diagnosis | ClassifierDiagnosis | Test suite for sklearn classifier model diagnosis tests | validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
| classifier_full_suite | ClassifierFullSuite | Full test suite for binary classification models. | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
| classifier_metrics | ClassifierMetrics | Test suite for sklearn classifier metrics | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance |
| classifier_model_validation | ClassifierModelValidation | Test suite for binary classification models. | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
| classifier_validation | ClassifierPerformance | Test suite for sklearn classifier models | validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison |
| cluster_full_suite | ClusterFullSuite | Full test suite for clustering models. | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.HomogeneityScore, validmind.model_validation.sklearn.CompletenessScore, validmind.model_validation.sklearn.VMeasure, validmind.model_validation.sklearn.AdjustedRandIndex, validmind.model_validation.sklearn.AdjustedMutualInformation, validmind.model_validation.sklearn.FowlkesMallowsScore, validmind.model_validation.sklearn.ClusterPerformanceMetrics, validmind.model_validation.sklearn.ClusterCosineSimilarity, validmind.model_validation.sklearn.SilhouettePlot, validmind.model_validation.ClusterSizeDistribution, validmind.model_validation.sklearn.HyperParametersTuning, validmind.model_validation.sklearn.KMeansClustersOptimization |
| cluster_metrics | ClusterMetrics | Test suite for sklearn clustering metrics | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.HomogeneityScore, validmind.model_validation.sklearn.CompletenessScore, validmind.model_validation.sklearn.VMeasure, validmind.model_validation.sklearn.AdjustedRandIndex, validmind.model_validation.sklearn.AdjustedMutualInformation, validmind.model_validation.sklearn.FowlkesMallowsScore, validmind.model_validation.sklearn.ClusterPerformanceMetrics, validmind.model_validation.sklearn.ClusterCosineSimilarity, validmind.model_validation.sklearn.SilhouettePlot |
| cluster_performance | ClusterPerformance | Test suite for sklearn cluster performance | validmind.model_validation.ClusterSizeDistribution |
| embeddings_full_suite | EmbeddingsFullSuite | Full test suite for embeddings models. | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.embeddings.DescriptiveAnalytics, validmind.model_validation.embeddings.CosineSimilarityDistribution, validmind.model_validation.embeddings.ClusterDistribution, validmind.model_validation.embeddings.EmbeddingsVisualization2D, validmind.model_validation.embeddings.StabilityAnalysisRandomNoise, validmind.model_validation.embeddings.StabilityAnalysisSynonyms, validmind.model_validation.embeddings.StabilityAnalysisKeyword, validmind.model_validation.embeddings.StabilityAnalysisTranslation |
| embeddings_metrics | EmbeddingsMetrics | Test suite for embeddings metrics | validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.embeddings.DescriptiveAnalytics, validmind.model_validation.embeddings.CosineSimilarityDistribution, validmind.model_validation.embeddings.ClusterDistribution, validmind.model_validation.embeddings.EmbeddingsVisualization2D |
| embeddings_model_performance | EmbeddingsPerformance | Test suite for embeddings model performance | validmind.model_validation.embeddings.StabilityAnalysisRandomNoise, validmind.model_validation.embeddings.StabilityAnalysisSynonyms, validmind.model_validation.embeddings.StabilityAnalysisKeyword, validmind.model_validation.embeddings.StabilityAnalysisTranslation |
| hyper_parameters_optimization | KmeansParametersOptimization | Test suite for sklearn hyperparameters optimization | validmind.model_validation.sklearn.HyperParametersTuning, validmind.model_validation.sklearn.KMeansClustersOptimization |
| llm_classifier_full_suite | LLMClassifierFullSuite | Full test suite for LLM classification models. | validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis, validmind.prompt_validation.Bias, validmind.prompt_validation.Clarity, validmind.prompt_validation.Conciseness, validmind.prompt_validation.Delimitation, validmind.prompt_validation.NegativeInstruction, validmind.prompt_validation.Robustness, validmind.prompt_validation.Specificity |
| prompt_validation | PromptValidation | Test suite for prompt validation | validmind.prompt_validation.Bias, validmind.prompt_validation.Clarity, validmind.prompt_validation.Conciseness, validmind.prompt_validation.Delimitation, validmind.prompt_validation.NegativeInstruction, validmind.prompt_validation.Robustness, validmind.prompt_validation.Specificity |
| nlp_classifier_full_suite | NLPClassifierFullSuite | Full test suite for NLP classification models. | validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
| regression_metrics | RegressionMetrics | Test suite for performance metrics of regression metrics | validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.sklearn.PermutationFeatureImportance |
| regression_model_description | RegressionModelDescription | Test suite for performance metric of regression model of statsmodels library | validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata |
| regression_models_evaluation | RegressionModelsEvaluation | Test suite for metrics comparison of regression model of statsmodels library | validmind.model_validation.statsmodels.RegressionModelsCoeffs, validmind.model_validation.statsmodels.RegressionModelsPerformance |
| regression_models_comparison | RegressionModelsComparison | Test suite for regression models performance comparison | validmind.model_validation.sklearn.RegressionModelsPerformanceComparison |
| regression_full_suite | RegressionFullSuite | Full test suite for regression models. | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues, validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.RegressionErrors, validmind.model_validation.sklearn.RegressionR2Square, validmind.model_validation.sklearn.RegressionModelsPerformanceComparison |
| regression_performance | RegressionPerformance | Test suite for regression model performance | validmind.model_validation.sklearn.RegressionErrors, validmind.model_validation.sklearn.RegressionR2Square |
| summarization_metrics | SummarizationMetrics | Test suite for Summarization metrics | validmind.model_validation.RougeMetrics, validmind.model_validation.TokenDisparity, validmind.model_validation.BleuScore, validmind.model_validation.BertScore, validmind.model_validation.ContextualRecall |
| tabular_dataset | TabularDataset | Test suite for tabular datasets. | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues |
| tabular_dataset_description | TabularDatasetDescription | Test suite to extract metadata and descriptive statistics from a tabular dataset | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix |
| tabular_data_quality | TabularDataQuality | Test suite for data quality on tabular datasets | validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues |
| text_data_quality | TextDataQuality | Test suite for data quality on text data | validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.nlp.StopWords, validmind.data_validation.nlp.Punctuations, validmind.data_validation.nlp.CommonWords, validmind.data_validation.nlp.TextDescription |
| time_series_data_quality | TimeSeriesDataQuality | Test suite for data quality on time series datasets | validmind.data_validation.TimeSeriesOutliers, validmind.data_validation.TimeSeriesMissingValues, validmind.data_validation.TimeSeriesFrequency |
| time_series_dataset | TimeSeriesDataset | Test suite for time series datasets. | validmind.data_validation.TimeSeriesOutliers, validmind.data_validation.TimeSeriesMissingValues, validmind.data_validation.TimeSeriesFrequency, validmind.data_validation.TimeSeriesLinePlot, validmind.data_validation.TimeSeriesHistogram, validmind.data_validation.ACFandPACFPlot, validmind.data_validation.SeasonalDecompose, validmind.data_validation.AutoSeasonality, validmind.data_validation.AutoStationarity, validmind.data_validation.RollingStatsPlot, validmind.data_validation.AutoAR, validmind.data_validation.AutoMA, validmind.data_validation.ScatterPlot, validmind.data_validation.LaggedCorrelationHeatmap, validmind.data_validation.EngleGrangerCoint, validmind.data_validation.SpreadPlot |
| time_series_model_validation | TimeSeriesModelValidation | Test suite for time series model validation. | validmind.data_validation.DatasetSplit, validmind.model_validation.ModelMetadata, validmind.model_validation.statsmodels.RegressionModelsCoeffs, validmind.model_validation.statsmodels.RegressionModelsPerformance, validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels, validmind.model_validation.statsmodels.RegressionModelSensitivityPlot |
| time_series_multivariate | TimeSeriesMultivariate | This test suite provides a preliminary understanding of the features and relationship in multivariate dataset. It presents various multivariate visualizations that can help identify patterns, trends, and relationships between pairs of variables. The visualizations are designed to explore the relationships between multiple features simultaneously. They allow you to quickly identify any patterns or trends in the data, as well as any potential outliers or anomalies. The individual feature distribution can also be explored to provide insight into the range and frequency of values observed in the data. This multivariate analysis test suite aims to provide an overview of the data structure and guide further exploration and modeling. | validmind.data_validation.ScatterPlot, validmind.data_validation.LaggedCorrelationHeatmap, validmind.data_validation.EngleGrangerCoint, validmind.data_validation.SpreadPlot |
| time_series_sensitivity | TimeSeriesSensitivity | This test suite performs sensitivity analysis on a statsmodels OLS linear regression model by applying distinct shocks to each input variable individually and then computing the model's predictions. The aim of this test suite is to investigate the model's responsiveness to variations in its inputs. By juxtaposing the model's predictions under baseline and shocked conditions, users can visually evaluate the sensitivity of the model to changes in each variable. This kind of analysis can also shed light on potential model limitations, including over-reliance on specific variables or insufficient responsiveness to changes in inputs. As a result, this test suite can provide insights that may be beneficial for refining the model structure, improving its robustness, and ensuring a more reliable prediction performance. | validmind.model_validation.statsmodels.RegressionModelSensitivityPlot |
| time_series_univariate | TimeSeriesUnivariate | This test suite provides a preliminary understanding of the target variable(s) used in the time series dataset. It visualizations that present the raw time series data and a histogram of the target variable(s). The raw time series data provides a visual inspection of the target variable's behavior over time. This helps to identify any patterns or trends in the data, as well as any potential outliers or anomalies. The histogram of the target variable displays the distribution of values, providing insight into the range and frequency of values observed in the data. | validmind.data_validation.TimeSeriesLinePlot, validmind.data_validation.TimeSeriesHistogram, validmind.data_validation.ACFandPACFPlot, validmind.data_validation.SeasonalDecompose, validmind.data_validation.AutoSeasonality, validmind.data_validation.AutoStationarity, validmind.data_validation.RollingStatsPlot, validmind.data_validation.AutoAR, validmind.data_validation.AutoMA |
To get the list of tests available in a given test suite:
| ID | Name | Description | Tests |
|---|---|---|---|
| classifier_full_suite | ClassifierFullSuite | Full test suite for binary classification models. | validmind.data_validation.DatasetDescription, validmind.data_validation.DescriptiveStatistics, validmind.data_validation.PearsonCorrelationMatrix, validmind.data_validation.ClassImbalance, validmind.data_validation.Duplicates, validmind.data_validation.HighCardinality, validmind.data_validation.HighPearsonCorrelation, validmind.data_validation.MissingValues, validmind.data_validation.Skewness, validmind.data_validation.UniqueRows, validmind.data_validation.TooManyZeroValues, validmind.model_validation.ModelMetadata, validmind.data_validation.DatasetSplit, validmind.model_validation.sklearn.ConfusionMatrix, validmind.model_validation.sklearn.ClassifierPerformance, validmind.model_validation.sklearn.PermutationFeatureImportance, validmind.model_validation.sklearn.PrecisionRecallCurve, validmind.model_validation.sklearn.ROCCurve, validmind.model_validation.sklearn.PopulationStabilityIndex, validmind.model_validation.sklearn.SHAPGlobalImportance, validmind.model_validation.sklearn.MinimumAccuracy, validmind.model_validation.sklearn.MinimumF1Score, validmind.model_validation.sklearn.MinimumROCAUCScore, validmind.model_validation.sklearn.TrainingTestDegradation, validmind.model_validation.sklearn.ModelsPerformanceComparison, validmind.model_validation.sklearn.OverfitDiagnosis, validmind.model_validation.sklearn.WeakspotsDiagnosis, validmind.model_validation.sklearn.RobustnessDiagnosis |
To get the details for a given test:
To get more comprehensive details for test suites and tests:
| Test Suite ID | Test Suite Name | Test Suite Section | Test ID | Test Name | Test Type |
|---|---|---|---|---|---|
| classifier_full_suite | ClassifierFullSuite | tabular_dataset_description | validmind.data_validation.DatasetDescription | Dataset Description | Metric |
| classifier_full_suite | ClassifierFullSuite | tabular_dataset_description | validmind.data_validation.DescriptiveStatistics | Descriptive Statistics | Metric |
| classifier_full_suite | ClassifierFullSuite | tabular_dataset_description | validmind.data_validation.PearsonCorrelationMatrix | Pearson Correlation Matrix | Metric |
| classifier_full_suite | ClassifierFullSuite | tabular_data_quality | validmind.data_validation.ClassImbalance | Class Imbalance | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | tabular_data_quality | validmind.data_validation.Duplicates | Duplicates | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | tabular_data_quality | validmind.data_validation.HighCardinality | High Cardinality | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | tabular_data_quality | validmind.data_validation.HighPearsonCorrelation | High Pearson Correlation | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | tabular_data_quality | validmind.data_validation.MissingValues | Missing Values | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | tabular_data_quality | validmind.data_validation.Skewness | Skewness | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | tabular_data_quality | validmind.data_validation.UniqueRows | Unique Rows | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | tabular_data_quality | validmind.data_validation.TooManyZeroValues | Too Many Zero Values | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.model_validation.ModelMetadata | Model Metadata | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.data_validation.DatasetSplit | Dataset Split | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.model_validation.sklearn.ConfusionMatrix | Confusion Matrix | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.model_validation.sklearn.ClassifierPerformance | Classifier Performance | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.model_validation.sklearn.PermutationFeatureImportance | Permutation Feature Importance | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.model_validation.sklearn.PrecisionRecallCurve | Precision Recall Curve | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.model_validation.sklearn.ROCCurve | ROC Curve | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.model_validation.sklearn.PopulationStabilityIndex | Population Stability Index | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_metrics | validmind.model_validation.sklearn.SHAPGlobalImportance | SHAP Global Importance | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_validation | validmind.model_validation.sklearn.MinimumAccuracy | Minimum Accuracy | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | classifier_validation | validmind.model_validation.sklearn.MinimumF1Score | Minimum Score | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | classifier_validation | validmind.model_validation.sklearn.MinimumROCAUCScore | Minimum ROCAUC Score | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | classifier_validation | validmind.model_validation.sklearn.TrainingTestDegradation | Training Test Degradation | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | classifier_validation | validmind.model_validation.sklearn.ModelsPerformanceComparison | Models Performance Comparison | Metric |
| classifier_full_suite | ClassifierFullSuite | classifier_model_diagnosis | validmind.model_validation.sklearn.OverfitDiagnosis | Overfit Diagnosis | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | classifier_model_diagnosis | validmind.model_validation.sklearn.WeakspotsDiagnosis | Weakspots Diagnosis | ThresholdTest |
| classifier_full_suite | ClassifierFullSuite | classifier_model_diagnosis | validmind.model_validation.sklearn.RobustnessDiagnosis | Robustness Diagnosis | ThresholdTest |
To get the list of tests and their purpose:
| Test Type | Name | Description | ID |
|---|---|---|---|
| ThresholdTest | Bias | Evaluates bias in a Large Language Model based on the order and distribution of exemplars in a prompt.... | validmind.prompt_validation.Bias |
| ThresholdTest | Clarity | Evaluates and scores the clarity of prompts in a Large Language Model based on specified guidelines.... | validmind.prompt_validation.Clarity |
| ThresholdTest | Specificity | Evaluates and scores the specificity of prompts provided to a Large Language Model (LLM), based on clarity,... | validmind.prompt_validation.Specificity |
| ThresholdTest | Robustness | Assesses the robustness of prompts provided to a Large Language Model under varying conditions and contexts.... | validmind.prompt_validation.Robustness |
| ThresholdTest | Negative Instruction | Evaluates and grades the use of affirmative, proactive language over negative instructions in LLM prompts.... | validmind.prompt_validation.NegativeInstruction |
| ThresholdTest | Conciseness | Analyzes and grades the conciseness of prompts provided to a Large Language Model.... | validmind.prompt_validation.Conciseness |
| ThresholdTest | Delimitation | Evaluates the proper use of delimiters in prompts provided to Large Language Models.... | validmind.prompt_validation.Delimitation |
| Metric | Bert Score | Evaluates text generation models' performance by calculating precision, recall, and F1 score based on BERT... | validmind.model_validation.BertScore |
| Metric | Regard Score | **Purpose:**... | validmind.model_validation.RegardScore |
| Metric | Bleu Score | Assesses translation quality by comparing machine-translated sentences with human-translated ones using BLEU score.... | validmind.model_validation.BleuScore |
| Metric | Contextual Recall | Evaluates a Natural Language Generation model's ability to generate contextually relevant and factually correct... | validmind.model_validation.ContextualRecall |
| Metric | Regard Histogram | **Purpose:**... | validmind.model_validation.RegardHistogram |
| Metric | Toxicity Histogram | **Purpose:**... | validmind.model_validation.ToxicityHistogram |
| Metric | Rouge Metrics | Evaluates the quality of machine-generated text using various ROUGE metrics, and visualizes the results.... | validmind.model_validation.RougeMetrics |
| Metric | Model Metadata | Extracts and summarizes critical metadata from a machine learning model instance for comprehensive analysis.... | validmind.model_validation.ModelMetadata |
| Metric | Bert Score Aggregate | Evaluates the aggregate performance of text generation models by computing the average precision, recall,... | validmind.model_validation.BertScoreAggregate |
| Metric | Cluster Size Distribution | Compares and visualizes the distribution of cluster sizes in model predictions and actual data for assessing... | validmind.model_validation.ClusterSizeDistribution |
| Metric | Token Disparity | Assess and visualize token count disparity between model's predicted and actual dataset.... | validmind.model_validation.TokenDisparity |
| Metric | Toxicity Score | **Purpose:**... | validmind.model_validation.ToxicityScore |
| Metric | Rouge Metrics Aggregate | Evaluates the average quality of machine-generated text using various ROUGE metrics and visualizes the aggregated results.... | validmind.model_validation.RougeMetricsAggregate |
| Metric | Embeddings Visualization D | Visualizes 2D representation of text embeddings generated by a model using t-SNE technique.... | validmind.model_validation.embeddings.EmbeddingsVisualization2D |
| ThresholdTest | Stability Analysis Random Noise | Evaluate robustness of embeddings models to random noise introduced by using... | validmind.model_validation.embeddings.StabilityAnalysisRandomNoise |
| Metric | Cosine Similarity Distribution | Assesses the similarity between predicted text embeddings from a model using a Cosine Similarity distribution... | validmind.model_validation.embeddings.CosineSimilarityDistribution |
| ThresholdTest | Stability Analysis Translation | Evaluate robustness of embeddings models to noise introduced by translating... | validmind.model_validation.embeddings.StabilityAnalysisTranslation |
| Metric | Cluster Distribution | Assesses the distribution of text embeddings across clusters produced by a model using KMeans clustering.... | validmind.model_validation.embeddings.ClusterDistribution |
| ThresholdTest | Stability Analysis | Base class for embeddings stability analysis tests | validmind.model_validation.embeddings.StabilityAnalysis |
| ThresholdTest | Stability Analysis Keyword | Evaluate robustness of embeddings models to keyword swaps on the test dataset... | validmind.model_validation.embeddings.StabilityAnalysisKeyword |
| ThresholdTest | Stability Analysis Synonyms | Evaluates the stability of text embeddings models when words in test data are replaced by their synonyms randomly.... | validmind.model_validation.embeddings.StabilityAnalysisSynonyms |
| Metric | Descriptive Analytics | Evaluates statistical properties of text embeddings in an ML model via mean, median, and standard deviation... | validmind.model_validation.embeddings.DescriptiveAnalytics |
| Metric | Regression Models Performance Comparison | Compares and evaluates the performance of multiple regression models using five different metrics: MAE, MSE, RMSE,... | validmind.model_validation.sklearn.RegressionModelsPerformanceComparison |
| Metric | Adjusted Mutual Information | Evaluates clustering model performance by measuring mutual information between true and predicted labels, adjusting... | validmind.model_validation.sklearn.AdjustedMutualInformation |
| Metric | Silhouette Plot | Calculates and visualizes Silhouette Score, assessing degree of data point suitability to its cluster in ML models.... | validmind.model_validation.sklearn.SilhouettePlot |
| ThresholdTest | Robustness Diagnosis | Evaluates the robustness of a machine learning model by injecting Gaussian noise to input data and measuring... | validmind.model_validation.sklearn.RobustnessDiagnosis |
| Metric | Adjusted Rand Index | Measures the similarity between two data clusters using the Adjusted Rand Index (ARI) metric in clustering machine... | validmind.model_validation.sklearn.AdjustedRandIndex |
| Metric | SHAP Global Importance | Evaluates and visualizes global feature importance using SHAP values for model explanation and risk identification.... | validmind.model_validation.sklearn.SHAPGlobalImportance |
| Metric | Confusion Matrix | Evaluates and visually represents the classification ML model's predictive performance using a Confusion Matrix... | validmind.model_validation.sklearn.ConfusionMatrix |
| Metric | Homogeneity Score | Assesses clustering homogeneity by comparing true and predicted labels, scoring from 0 (heterogeneous) to 1... | validmind.model_validation.sklearn.HomogeneityScore |
| Metric | Completeness Score | Evaluates a clustering model's capacity to categorize instances from a single class into the same cluster.... | validmind.model_validation.sklearn.CompletenessScore |
| ThresholdTest | Overfit Diagnosis | Detects and visualizes overfit regions in an ML model by comparing performance on training and test datasets.... | validmind.model_validation.sklearn.OverfitDiagnosis |
| Metric | Cluster Performance Metrics | Evaluates the performance of clustering machine learning models using multiple established metrics.... | validmind.model_validation.sklearn.ClusterPerformanceMetrics |
| Metric | Permutation Feature Importance | Assesses the significance of each feature in a model by evaluating the impact on model performance when feature... | validmind.model_validation.sklearn.PermutationFeatureImportance |
| Metric | Fowlkes Mallows Score | Evaluates the similarity between predicted and actual cluster assignments in a model using the Fowlkes-Mallows... | validmind.model_validation.sklearn.FowlkesMallowsScore |
| ThresholdTest | Minimum ROCAUC Score | Validates model by checking if the ROC AUC score meets or surpasses a specified threshold.... | validmind.model_validation.sklearn.MinimumROCAUCScore |
| Metric | Cluster Cosine Similarity | Measures the intra-cluster similarity of a clustering model using cosine similarity.... | validmind.model_validation.sklearn.ClusterCosineSimilarity |
| Metric | Precision Recall Curve | Evaluates the precision-recall trade-off for binary classification models and visualizes the Precision-Recall curve.... | validmind.model_validation.sklearn.PrecisionRecallCurve |
| Metric | Classifier Performance | Evaluates performance of binary or multiclass classification models using precision, recall, F1-Score, accuracy,... | validmind.model_validation.sklearn.ClassifierPerformance |
| Metric | V Measure | Evaluates homogeneity and completeness of a clustering model using the V Measure Score.... | validmind.model_validation.sklearn.VMeasure |
| ThresholdTest | Minimum Score | Evaluates if the model's F1 score on the validation set meets a predefined minimum threshold.... | validmind.model_validation.sklearn.MinimumF1Score |
| Metric | ROC Curve | Evaluates binary classification model performance by generating and plotting the Receiver Operating Characteristic... | validmind.model_validation.sklearn.ROCCurve |
| Metric | Regression Square | **Purpose**: The purpose of the RegressionR2Square Metric test is to measure the overall goodness-of-fit of a... | validmind.model_validation.sklearn.RegressionR2Square |
| Metric | Regression Errors | **Purpose**: This metric is used to measure the performance of a regression model. It gauges the model's accuracy... | validmind.model_validation.sklearn.RegressionErrors |
| Metric | Cluster Performance | Evaluates and compares a clustering model's performance on training and testing datasets using multiple defined... | validmind.model_validation.sklearn.ClusterPerformance |
| ThresholdTest | Training Test Degradation | Tests if model performance degradation between training and test datasets exceeds a predefined threshold.... | validmind.model_validation.sklearn.TrainingTestDegradation |
| Metric | Hyper Parameters Tuning | Exerts exhaustive grid search to identify optimal hyperparameters for the model, improving performance.... | validmind.model_validation.sklearn.HyperParametersTuning |
| Metric | K Means Clusters Optimization | Optimizes the number of clusters in K-means models using Elbow and Silhouette methods.... | validmind.model_validation.sklearn.KMeansClustersOptimization |
| Metric | Models Performance Comparison | Evaluates and compares the performance of multiple Machine Learning models using various metrics like accuracy,... | validmind.model_validation.sklearn.ModelsPerformanceComparison |
| ThresholdTest | Weakspots Diagnosis | Identifies and visualizes weak spots in a machine learning model's performance across various sections of the... | validmind.model_validation.sklearn.WeakspotsDiagnosis |
| Metric | Population Stability Index | Evaluates the Population Stability Index (PSI) to quantify the stability of an ML model's predictions across... | validmind.model_validation.sklearn.PopulationStabilityIndex |
| ThresholdTest | Minimum Accuracy | Checks if the model's prediction accuracy meets or surpasses a specified threshold.... | validmind.model_validation.sklearn.MinimumAccuracy |
| Metric | Regression Models Coeffs | Compares feature importance by evaluating and contrasting coefficients of different regression models.... | validmind.model_validation.statsmodels.RegressionModelsCoeffs |
| Metric | Box Pierce | Detects autocorrelation in time-series data through the Box-Pierce test to validate model performance.... | validmind.model_validation.statsmodels.BoxPierce |
| Metric | Regression Coeffs Plot | Visualizes regression coefficients with 95% confidence intervals to assess predictor variables' impact on response... | validmind.model_validation.statsmodels.RegressionCoeffsPlot |
| Metric | Regression Model Sensitivity Plot | Tests the sensitivity of a regression model to variations in independent variables by applying shocks and... | validmind.model_validation.statsmodels.RegressionModelSensitivityPlot |
| Metric | Regression Models Performance | Evaluates and compares regression models' performance using R-squared, Adjusted R-squared, and MSE metrics.... | validmind.model_validation.statsmodels.RegressionModelsPerformance |
| Metric | Zivot Andrews Arch | Evaluates the order of integration and stationarity of time series data using Zivot-Andrews unit root test.... | validmind.model_validation.statsmodels.ZivotAndrewsArch |
| Metric | Regression Model Outsample Comparison | Computes MSE and RMSE for multiple regression models using out-of-sample test to assess model's prediction accuracy... | validmind.model_validation.statsmodels.RegressionModelOutsampleComparison |
| Metric | Regression Model Forecast Plot Levels | Compares and visualizes forecasted and actual values of regression models on both raw and transformed datasets.... | validmind.model_validation.statsmodels.RegressionModelForecastPlotLevels |
| Metric | Log Regression Confusion Matrix | Generates a confusion matrix for logistic regression model performance, utilizing thresholded probabilities for... | validmind.model_validation.statsmodels.LogRegressionConfusionMatrix |
| Metric | PD Rating Class Plot | Assesses and visualizes credit risk distribution across different rating classes within a dataset via default... | validmind.model_validation.statsmodels.PDRatingClassPlot |
| Metric | Scorecard Histogram | Creates histograms of credit scores, from both default and non-default instances, generated by a credit-risk model.... | validmind.model_validation.statsmodels.ScorecardHistogram |
| Metric | Feature Importance And Significance | Evaluates and visualizes the statistical significance and feature importance using regression and decision tree... | validmind.model_validation.statsmodels.FeatureImportanceAndSignificance |
| Metric | L Jung Box | Assesses autocorrelations in dataset features by performing a Ljung-Box test on each feature.... | validmind.model_validation.statsmodels.LJungBox |
| Metric | Logistic Reg Prediction Histogram | Generates and visualizes histograms of the Probability of Default predictions for both positive and negative... | validmind.model_validation.statsmodels.LogisticRegPredictionHistogram |
| Metric | Jarque Bera | Assesses normality of dataset features in an ML model using the Jarque-Bera test.... | validmind.model_validation.statsmodels.JarqueBera |
| Metric | Phillips Perron Arch | Executes Phillips-Perron test to assess the stationarity of time series data in each ML model feature.... | validmind.model_validation.statsmodels.PhillipsPerronArch |
| Metric | Kolmogorov Smirnov | Executes a feature-wise Kolmogorov-Smirnov test to evaluate alignment with normal distribution in datasets.... | validmind.model_validation.statsmodels.KolmogorovSmirnov |
| Metric | Residuals Visual Inspection | Provides a comprehensive visual analysis of residuals for regression models utilizing various plot types.... | validmind.model_validation.statsmodels.ResidualsVisualInspection |
| Metric | Shapiro Wilk | Evaluates feature-wise normality of training data using the Shapiro-Wilk test.... | validmind.model_validation.statsmodels.ShapiroWilk |
| Metric | Scorecard Bucket Histogram | Evaluates and visualizes distribution of risk categories in a classification model's scores, useful in credit risk... | validmind.model_validation.statsmodels.ScorecardBucketHistogram |
| Metric | Regression Model Insample Comparison | Evaluates and compares in-sample performance of multiple regression models using R-Squared, Adjusted R-Squared,... | validmind.model_validation.statsmodels.RegressionModelInsampleComparison |
| Metric | Regression Feature Significance | Assesses and visualizes the statistical significance of features in a set of regression models.... | validmind.model_validation.statsmodels.RegressionFeatureSignificance |
| Metric | Regression Model Summary | Evaluates regression model performance using metrics including R-Squared, Adjusted R-Squared, MSE, and RMSE.... | validmind.model_validation.statsmodels.RegressionModelSummary |
| Metric | KPSS | Executes KPSS unit root test to validate stationarity of time-series data in machine learning model.... | validmind.model_validation.statsmodels.KPSS |
| Metric | Lilliefors | Assesses the normality of feature distributions in an ML model's training dataset using the Lilliefors test.... | validmind.model_validation.statsmodels.Lilliefors |
| Metric | Logistic Reg Cumulative Prob | Visualizes cumulative probabilities of positive and negative classes for both training and testing in logistic... | validmind.model_validation.statsmodels.LogisticRegCumulativeProb |
| Metric | Runs Test | Executes Runs Test on ML model to detect non-random patterns in output data sequence.... | validmind.model_validation.statsmodels.RunsTest |
| Metric | Scorecard Probabilities Histogram | Evaluates risk classification of a model by visualizing the distribution of default probability across score... | validmind.model_validation.statsmodels.ScorecardProbabilitiesHistogram |
| Metric | DFGLS Arch | Executes Dickey-Fuller GLS metric to determine order of integration and check stationarity in time series data.... | validmind.model_validation.statsmodels.DFGLSArch |
| Metric | Auto ARIMA | Evaluates ARIMA models for time-series forecasting, ranking them using Bayesian and Akaike Information Criteria.... | validmind.model_validation.statsmodels.AutoARIMA |
| ThresholdTest | ADF Test | Assesses the stationarity of time series data using the Augmented Dickey-Fuller (ADF) test.... | validmind.model_validation.statsmodels.ADFTest |
| Metric | GINI Table | Evaluates classification model performance using AUC, GINI, and KS metrics for training and test datasets.... | validmind.model_validation.statsmodels.GINITable |
| Metric | Regression Model Forecast Plot | Generates plots to visually compare the forecasted outcomes of one or more regression models against actual... | validmind.model_validation.statsmodels.RegressionModelForecastPlot |
| Metric | ADF | Assesses the stationarity of a time series dataset using the Augmented Dickey-Fuller (ADF) test.... | validmind.model_validation.statsmodels.ADF |
| Metric | Durbin Watson Test | Assesses autocorrelation in time series data features using the Durbin-Watson statistic.... | validmind.model_validation.statsmodels.DurbinWatsonTest |
| Metric | Missing Values Risk | Assesses and quantifies the risk related to missing values in a dataset used for training an ML model.... | validmind.data_validation.MissingValuesRisk |
| Metric | IQR Outliers Table | Determines and summarizes outliers in numerical features using Interquartile Range method.... | validmind.data_validation.IQROutliersTable |
| Metric | Bivariate Features Bar Plots | Generates visual bar plots to analyze the relationship between paired features within categorical data in the model.... | validmind.data_validation.BivariateFeaturesBarPlots |
| ThresholdTest | Skewness | Evaluates the skewness of numerical data in a machine learning model and checks if it falls below a set maximum... | validmind.data_validation.Skewness |
| ThresholdTest | Duplicates | Tests dataset for duplicate entries, ensuring model reliability via data quality verification.... | validmind.data_validation.Duplicates |
| Metric | Missing Values Bar Plot | Creates a bar plot showcasing the percentage of missing values in each column of the dataset with risk... | validmind.data_validation.MissingValuesBarPlot |
| Metric | Dataset Description | Provides comprehensive analysis and statistical summaries of each field in a machine learning model's dataset.... | validmind.data_validation.DatasetDescription |
| Metric | Scatter Plot | Creates a scatter plot matrix to visually analyze feature relationships, patterns, and outliers in a dataset.... | validmind.data_validation.ScatterPlot |
| ThresholdTest | Time Series Outliers | Identifies and visualizes outliers in time-series data using z-score method.... | validmind.data_validation.TimeSeriesOutliers |
| Metric | Tabular Categorical Bar Plots | Generates and visualizes bar plots for each category in categorical features to evaluate dataset's composition.... | validmind.data_validation.TabularCategoricalBarPlots |
| Metric | Auto Stationarity | Automates Augmented Dickey-Fuller test to assess stationarity across multiple time series in a DataFrame.... | validmind.data_validation.AutoStationarity |
| Metric | Descriptive Statistics | Performs a detailed descriptive statistical analysis of both numerical and categorical data within a model's... | validmind.data_validation.DescriptiveStatistics |
| Metric | ANOVA One Way Table | Applies one-way ANOVA (Analysis of Variance) to identify statistically significant numerical features in the... | validmind.data_validation.ANOVAOneWayTable |
| Metric | Target Rate Bar Plots | Generates bar plots visualizing the default rates of categorical features for a classification machine learning... | validmind.data_validation.TargetRateBarPlots |
| Metric | Pearson Correlation Matrix | Evaluates linear dependency between numerical variables in a dataset via a Pearson Correlation coefficient heat map.... | validmind.data_validation.PearsonCorrelationMatrix |
| Metric | Feature Target Correlation Plot | Visualizes the correlation between input features and model's target output in a color-coded horizontal bar plot.... | validmind.data_validation.FeatureTargetCorrelationPlot |
| Metric | Tabular Numerical Histograms | Generates histograms for each numerical feature in a dataset to provide visual insights into data distribution and... | validmind.data_validation.TabularNumericalHistograms |
| Metric | Isolation Forest Outliers | Detects outliers in a dataset using the Isolation Forest algorithm and visualizes results through scatter plots.... | validmind.data_validation.IsolationForestOutliers |
| Metric | Chi Squared Features Table | Executes Chi-Squared test for each categorical feature against a target column to assess significant association.... | validmind.data_validation.ChiSquaredFeaturesTable |
| ThresholdTest | High Cardinality | Assesses the number of unique values in categorical columns to detect high cardinality and potential overfitting.... | validmind.data_validation.HighCardinality |
| ThresholdTest | Missing Values | Evaluates dataset quality by ensuring missing value ratio across all features does not exceed a set threshold.... | validmind.data_validation.MissingValues |
| Metric | Default Ratesby Risk Band Plot | Generates a bar plot showcasing the distribution of default rates across different risk bands in a dataset.... | validmind.data_validation.DefaultRatesbyRiskBandPlot |
| Metric | Rolling Stats Plot | This test evaluates the stationarity of time series data by plotting its rolling mean and standard deviation.... | validmind.data_validation.RollingStatsPlot |
| Metric | Tabular Description Tables | Summarizes key descriptive statistics for numerical, categorical, and datetime variables in a dataset.... | validmind.data_validation.TabularDescriptionTables |
| Metric | Auto MA | Automatically selects the optimal Moving Average (MA) order for each variable in a time series dataset based on... | validmind.data_validation.AutoMA |
| ThresholdTest | Unique Rows | Verifies the diversity of the dataset by ensuring that the count of unique rows exceeds a prescribed threshold.... | validmind.data_validation.UniqueRows |
| ThresholdTest | Too Many Zero Values | Identifies numerical columns in a dataset that contain an excessive number of zero values, defined by a threshold... | validmind.data_validation.TooManyZeroValues |
| ThresholdTest | High Pearson Correlation | Identifies highly correlated feature pairs in a dataset suggesting feature redundancy or multicollinearity.... | validmind.data_validation.HighPearsonCorrelation |
| Metric | AC Fand PACF Plot | Analyzes time series data using Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots to... | validmind.data_validation.ACFandPACFPlot |
| Metric | Bivariate Histograms | Generates bivariate histograms for paired features, aiding in visual inspection of categorical variables'... | validmind.data_validation.BivariateHistograms |
| Metric | WOE Bin Table | Calculates and assesses the Weight of Evidence (WoE) and Information Value (IV) of each feature in a ML model.... | validmind.data_validation.WOEBinTable |
| Metric | Heatmap Feature Correlations | Creates a heatmap to visually represent correlation patterns between pairs of numerical features in a dataset.... | validmind.data_validation.HeatmapFeatureCorrelations |
| ThresholdTest | Time Series Frequency | Evaluates consistency of time series data frequency and generates a frequency plot.... | validmind.data_validation.TimeSeriesFrequency |
| Metric | Dataset Split | Evaluates and visualizes the distribution proportions among training, testing, and validation datasets of an ML... | validmind.data_validation.DatasetSplit |
| Metric | Spread Plot | Visualizes the spread relationship between pairs of time-series variables in a dataset, thereby aiding in... | validmind.data_validation.SpreadPlot |
| Metric | Time Series Line Plot | Generates and analyses time-series data through line plots revealing trends, patterns, anomalies over time.... | validmind.data_validation.TimeSeriesLinePlot |
| Metric | Pi T Credit Scores Histogram | Generates a histogram visualization for observed and predicted credit default scores.... | validmind.data_validation.PiTCreditScoresHistogram |
| Metric | Auto Seasonality | Automatically identifies and quantifies optimal seasonality in time series data to improve forecasting model... | validmind.data_validation.AutoSeasonality |
| Metric | Bivariate Scatter Plots | Generates bivariate scatterplots to visually inspect relationships between pairs of predictor variables in machine... | validmind.data_validation.BivariateScatterPlots |
| Metric | Engle Granger Coint | Validates co-integration in pairs of time series data using the Engle-Granger test and classifies them as... | validmind.data_validation.EngleGrangerCoint |
| ThresholdTest | Time Series Missing Values | Validates time-series data quality by confirming the count of missing values is below a certain threshold.... | validmind.data_validation.TimeSeriesMissingValues |
| Metric | Time Series Histogram | Visualizes distribution of time-series data using histograms and Kernel Density Estimation (KDE) lines.... | validmind.data_validation.TimeSeriesHistogram |
| Metric | Lagged Correlation Heatmap | Assesses and visualizes correlation between target variable and lagged independent variables in a time-series... | validmind.data_validation.LaggedCorrelationHeatmap |
| Metric | Seasonal Decompose | Decomposes dataset features into observed, trend, seasonal, and residual components to identify patterns and... | validmind.data_validation.SeasonalDecompose |
| Metric | WOE Bin Plots | Generates visualizations of Weight of Evidence (WoE) and Information Value (IV) for understanding predictive power... | validmind.data_validation.WOEBinPlots |
| ThresholdTest | Class Imbalance | Evaluates and quantifies class distribution imbalance in a dataset used by a machine learning model.... | validmind.data_validation.ClassImbalance |
| Metric | IQR Outliers Bar Plot | Visualizes outlier distribution across percentiles in numerical data using Interquartile Range (IQR) method.... | validmind.data_validation.IQROutliersBarPlot |
| Metric | Pi TPD Histogram | Assesses credit risk prediction accuracy of a model by comparing actual and predicted defaults at a chosen point in... | validmind.data_validation.PiTPDHistogram |
| Metric | Auto AR | Automatically identifies the optimal Autoregressive (AR) order for a time series using BIC and AIC criteria.... | validmind.data_validation.AutoAR |
| Metric | Tabular Date Time Histograms | Generates histograms to provide graphical insight into the distribution of time intervals in model's datetime data.... | validmind.data_validation.TabularDateTimeHistograms |
| Metric | Punctuations | Analyzes and visualizes the frequency distribution of punctuation usage in a given text dataset.... | validmind.data_validation.nlp.Punctuations |
| Metric | Common Words | Identifies and visualizes the 40 most frequent non-stopwords in a specified text column within a dataset.... | validmind.data_validation.nlp.CommonWords |
| ThresholdTest | Hashtags | Assesses hashtag frequency in a text column, highlighting usage trends and potential dataset bias or spam.... | validmind.data_validation.nlp.Hashtags |
| ThresholdTest | Mentions | Calculates and visualizes frequencies of '@' prefixed mentions in a text-based dataset for NLP model analysis.... | validmind.data_validation.nlp.Mentions |
| Metric | Text Description | Performs comprehensive textual analysis on a dataset using NLTK, evaluating various parameters and generating... | validmind.data_validation.nlp.TextDescription |
| ThresholdTest | Stop Words | Evaluates and visualizes the frequency of English stop words in a text dataset against a defined threshold.... | validmind.data_validation.nlp.StopWords |