diff --git a/site/about/overview-model-documentation.qmd b/site/about/overview-model-documentation.qmd index a57598481b..59e462e4aa 100644 --- a/site/about/overview-model-documentation.qmd +++ b/site/about/overview-model-documentation.qmd @@ -146,7 +146,7 @@ How the {{< var validmind.developer >}} works: [^1]: [Model risk management](overview-model-risk-management.qmd) -[^2]: [Supported models](/developer/supported-models.qmd) +[^2]: [Supported models and frameworks](/developer/supported-models-and-frameworks.qmd) [^3]: [Customize document templates](/guide/templates/customize-document-templates.qmd) diff --git a/site/developer/_sidebar.yaml b/site/developer/_sidebar.yaml index 1215f1cf41..a4c11adc38 100644 --- a/site/developer/_sidebar.yaml +++ b/site/developer/_sidebar.yaml @@ -10,7 +10,7 @@ website: # USING THE VARIABLE IN THE LINK TEXT MESSES UP THE MOBILE VIEW - text: "ValidMind Library" file: developer/validmind-library.qmd - - developer/supported-models.qmd + - developer/supported-models-and-frameworks.qmd - text: "---" - text: "Quickstart" - notebooks/quickstart/quickstart_model_documentation.ipynb diff --git a/site/developer/supported-models-and-frameworks.qmd b/site/developer/supported-models-and-frameworks.qmd new file mode 100644 index 0000000000..d8e05e2586 --- /dev/null +++ b/site/developer/supported-models-and-frameworks.qmd @@ -0,0 +1,467 @@ +--- +# Copyright © 2023-2026 ValidMind Inc. All rights reserved. +# Refer to the LICENSE file in the root of this repository for details. +# SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial +title: "Supported models and frameworks" +date: last-modified +aliases: + - /guide/supported-models.html + - /developer/model-documentation/supported-models.html + - /developer/supported-models.html + - /developer/supported-model-frameworks.html +listing: + - id: next-models + type: grid + grid-columns: 2 + max-description-length: 250 + sort: false + fields: [title, description] + contents: + - /how-to/testing-overview.qmd + - test-descriptions.qmd + - /how-to/feature-overview.qmd + - samples-jupyter-notebooks.qmd +--- + +Learn about the wide range of model frameworks for testing and documentation supported by the {{< var validmind.developer >}}. Understand which frameworks are supported, what your model needs to provide for tests to run, and how to work with models that don't fit standard patterns. + +::: {.attn} + +## What does _supported_ mean? + +_Supported_ here means the {{< var validmind.developer >}} provides dedicated wrappers, install options, and guidance for these models and frameworks. + +You can also use other code with the {{< var vm.developer >}} for models that don't fit standard framework patterns. For example, you can wrap any Python callable with `FunctionModel` or pass precomputed predictions when you don't have a model object. + +:::: {.columns} + +::: {.column width="50%" .pr3} +Framework-agnostic support +: Compatible with any Python object that implements a `predict()` method. Framework-specific wrappers provide extra features and convenience, but they are not required to use generic Python models. +::: + +::: {.column width="50%" .pl3} +Vendor model compatibility +: Works with both first-party models you create and third-party models from external vendors, enabling flexible integration with both proprietary and external sources. +::: + +:::: + +::: + +## AI systems + +::: {.panel-tabset} +## LLM classifiers +LLM-based classification models via OpenAI-compatible APIs. + +- Test suite: [`LLMClassifierFullSuite`](/validmind/validmind/test_suites/llm.qmd#llmclassifierfullsuite) +- Includes prompt validation, text data quality, classifier metrics + +## NLP classifiers +Hugging Face and other NLP classification models. + +- Test suite: [`NLPClassifierFullSuite`](/validmind/validmind/test_suites/nlp.qmd#nlpclassifierfullsuite) +- Text classification, sentiment analysis, named entity recognition + +## Summarization models +Models that generate concise summaries from longer texts. + +- Test suite: [`SummarizationMetrics`](/validmind/validmind/test_suites/summarization.qmd#summarizationmetrics) +- BERT, BLEU, METEOR, ROUGE score evaluation + +## Embeddings models +Text and vector embedding models. + +- Test suite: [`EmbeddingsFullSuite`](/validmind/validmind/test_suites/embeddings.qmd#embeddingsfullsuite) +- Similarity metrics, stability analysis, visualization + +## RAG systems +Retrieval-augmented generation pipelines. + +- Tests: RAGAS integration +- Refer to [RAG evaluation](#rag-evaluation) for dataset requirements and available tests +::: + +## Machine learning models + +::: {.panel-tabset} +## Hugging Face-compatible +NLP and tabular models using Hugging Face Transformers. + +- Test suite: [`NLPClassifierFullSuite`](/validmind/validmind/test_suites/nlp.qmd#nlpclassifierfullsuite) (for NLP), [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) (for tabular) +- Text classification, tabular classification/regression + +## PyTorch +Any PyTorch neural network architecture, including LSTM, RNN, CNN, transformers, and custom architectures. + +- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) (depending on task) +- Model performance, feature importance, diagnosis + +## Tree-based +Gradient boosting and ensemble tree methods (XGBoost, CatBoost, random forest). + +- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) +- Confusion matrix, ROC/AUC, feature importance, SHAP values + +## K-nearest neighbors (KNN) +Distance-based classification and regression. + +- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) or [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) +- Performance metrics, model comparison + +## Clustering +Any sklearn-compatible clustering algorithm, including K-means, DBSCAN, and hierarchical clustering. + +- Test suite: [`ClusterFullSuite`](/validmind/validmind/test_suites/cluster.qmd#clusterfullsuite) +- Silhouette score, homogeneity, completeness, cluster distribution +::: + + +## Traditional statistical models + +::: {.panel-tabset} +## Linear regression +Models the relationship between a scalar response and one or more explanatory variables. + +- Test suite: [`RegressionFullSuite`](/validmind/validmind/test_suites/regression.qmd#regressionfullsuite) +- R² score, regression errors, feature importance + +## Logistic regression +Models the probability of a binary outcome based on one or more predictor variables. + +- Test suite: [`ClassifierFullSuite`](/validmind/validmind/test_suites/classifier.qmd#classifierfullsuite) +- Confusion matrix, ROC/AUC, precision/recall, feature importance + +## Timeseries and forecasting +ARIMA, VAR, and other statsmodels time series models. + +- Test suite: [`TimeSeriesModelValidation`](/validmind/validmind/test_suites/time_series.qmd#timeseriesmodelvalidation) +- Stationarity tests, autocorrelation analysis, forecast evaluation +::: + + +## Frameworks + +The {{< var validmind.developer >}} provides wrapper classes for common frameworks. When you call `vm.init_model()`, the library automatically detects your model's framework from its module, such as `sklearn`, `torch`, or `transformers`, and selects the appropriate wrapper. You don't need to specify the wrapper class manually. + +::: {.panel-tabset} +## CatBoost + +Gradient boosting on decision trees with categorical feature support. + +- **Wrapper:** `CatBoostModel` +- **Install:** `pip install validmind` + +## Hugging Face Transformers + +Pre-trained models for text classification, summarization, translation, and generation from the Hugging Face Transformers library. + +- **Wrapper:** `HFModel` +- **Install:** `pip install validmind[huggingface]` +- **Supported tasks:** `text_classification`, `text2text_generation` (summarization), `feature_extraction` (embeddings) + +## LLMs via API + +Access to OpenAI and Azure OpenAI models for text generation and analysis via OpenAI-compatible APIs. + +- **Wrapper:** `FoundationModel` +- **Install:** `pip install validmind[llm]` + +## PyTorch + +An open-source machine learning library for computer vision and natural language processing. + +- **Wrapper:** `PyTorchModel` +- **Install:** `pip install validmind[pytorch]` + +## R models via rpy2 + +Integrates R's statistical computing with Python via rpy2. + +- **Wrapper:** `RModel` +- **Install:** `pip install validmind rpy2` +- **Supported model types:** `glm` (including logistic regression), `lm`, `xgb.Booster` +- **Initialization:** Use `vm.init_r_model(model_path, input_id)` to load R model objects + +## scikit-learn + +A Python library for machine learning, offering supervised and unsupervised learning algorithms. + +- **Wrapper:** `SKlearnModel` +- **Install:** `pip install validmind` + +## statsmodels + +A Python module for statistical models, tests, and data exploration. + +- **Wrapper:** `StatsModelsModel` +- **Install:** `pip install validmind[stats]` + +## XGBoost + +An optimized gradient boosting library for classification and regression. + +- **Wrapper:** `XGBoostModel` +- **Install:** `pip install validmind[xgboost]` + +## Python callable + +Wrap custom prediction functions or models from otherwise unsupported frameworks. + +- **Wrapper:** `FunctionModel` +- **Install:** `pip install validmind` +::: + +To install all optional dependencies: + +```bash +pip install validmind[all] +``` + +### Test input requirements + +Different tests require different inputs from your model and dataset. Understanding these requirements helps you run the right tests for your use case.[^1] + +::: {.panel-tabset} + +#### When `predict()` is required + +Most model tests call your model's `predict()` method to generate predictions. This includes: + +- Performance metrics (accuracy, precision, recall, F1) +- Error analysis tests +- Robustness tests + +#### When `predict_proba()` is needed + +Classification metrics that evaluate probability outputs require `predict_proba()`: + +- ROC-AUC score +- Precision-recall curves +- Calibration tests +- Probability distribution analysis + +If your model doesn't have `predict_proba()`, these tests will be skipped or return an error. + +::: + +### Test input flow + +```{mermaid} +flowchart LR + subgraph Inputs + Model[Model Object] + Dataset[Dataset] + Precomputed[Precomputed Predictions] + end + + Model --> predict["predict()"] + Model --> predict_proba["predict_proba()"] + Precomputed --> Dataset + + subgraph TestTypes[Test Types] + ModelTests[Model Tests] + DatasetTests[Dataset Tests] + ClassificationTests[Classification Metrics] + end + + predict --> ModelTests + predict --> DatasetTests + predict_proba --> ClassificationTests + Dataset --> DatasetTests +``` + +### Using precomputed predictions + +If you can't provide a model object because your model runs in a separate environment, you can pass precomputed predictions directly to the dataset: + +```python +vm_dataset = vm.init_dataset( + dataset=df, + target_column="target", + prediction_values=predictions, # numpy array of predictions + probability_values=probabilities # optional: for classification +) +``` + +Alternatively, if predictions are already in your dataframe: + +```python +vm_dataset = vm.init_dataset( + dataset=df, + target_column="target", + prediction_column="predicted", # column name in df + probability_column="probability" # optional: for classification +) +``` + +### Dataset-only tests + +Some tests analyze data quality and don't require a model at all: + +- Missing value analysis +- Class imbalance detection +- Feature correlation +- Outlier detection +- Data drift tests + +### Supported dataset formats + +The library accepts multiple data formats when initializing datasets with `vm.init_dataset()`: + +| Format | Notes | +|--------|-------| +| Pandas DataFrame | Primary format, used internally | +| Polars DataFrame | Converted to pandas internally | +| NumPy ndarray | Requires column names to be specified | +| PyTorch TensorDataset | Requires `pip install validmind[pytorch]` | + +## Custom model wrappers + +For models that don't fit standard framework patterns, use these flexible wrappers: + +::: {.panel-tabset} +### FunctionModel + +Wrap any Python callable as a model: + +```python +from validmind.models import FunctionModel + +def my_predict(X): + # Your prediction logic here + return predictions + +vm_model = vm.init_model( + model=FunctionModel(predict_fn=my_predict), + input_id="my_model" +) +``` + +### PipelineModel + +Chain multiple models or processing steps: + +```python +from validmind.models import PipelineModel + +vm_model = vm.init_model( + model=PipelineModel(models=[preprocessor, model]), + input_id="my_pipeline" +) +``` + +### MetadataModel + +Use when you have model metadata but no inference capability — for example, when documenting a model deployed in an external system: + +```python +from validmind.models import MetadataModel + +vm_model = vm.init_model( + model=MetadataModel(), + input_id="external_model" +) +``` + +With `MetadataModel`, you can still run dataset-only tests and use precomputed predictions. + +## Setting up for LLMs and GenAI + +This section covers how to configure and test large language models and generative AI systems. + +### FoundationModel + +Use `FoundationModel` for LLM endpoints: + +```python +from validmind.models import FoundationModel +from validmind.prompt import Prompt + +prompt = Prompt( + template="Classify the sentiment: {text}", + variables=["text"] +) + +vm_model = vm.init_model( + model=FoundationModel( + prompt=prompt, + model="gpt-4", # or your model endpoint + ), + input_id="sentiment_classifier" +) +``` +::: + +### LLM test suites + +Install LLM dependencies: + +```bash +pip install validmind[llm] +``` + +Available test suites for LLMs include: + +- Prompt injection detection +- Output consistency +- Hallucination detection +- Toxicity analysis +- Bias evaluation + +Tests such as bias, clarity, robustness, and toxicity can be applied to any text-generating model, regardless of type. + +## RAG evaluation + +For retrieval-augmented generation (RAG) systems, the {{< var validmind.developer >}} integrates with [RAGAS](https://docs.ragas.io/) for comprehensive evaluation. + +### Dataset requirements + +RAG evaluation requires specific fields in your dataset: + +| Field | Type | Description | +|-------|------|-------------| +| `user_input` | str | The user's query | +| `response` | str | Model output | +| `retrieved_contexts` | List[str] | Retrieved context chunks | +| `reference` | str | Ground truth (required for some tests) | + +### Available RAG tests + +- **Faithfulness** — Measures how well the response is grounded in retrieved contexts +- **Context Recall** — Evaluates if relevant information was retrieved +- **Context Precision** — Measures relevance of retrieved contexts +- **Answer Relevancy** — Assesses if the response addresses the query + +## Python and dependency compatibility + +The {{< var validmind.developer >}} requires: + +- **Python:** >=3.9, <3.13 +- **Core dependencies:** pandas, numpy, scikit-learn + +Optional dependencies for specific frameworks: + +| Extra | Includes | +|-------|----------| +| `pytorch` | PyTorch, torchvision | +| `llm` | OpenAI, langchain, ragas, deepeval | +| `xgboost` | XGBoost | +| `huggingface` | Transformers, sentencepiece | +| `nlp` | langdetect, nltk, textblob, evaluate, rouge, bert-score | +| `stats` | scipy, statsmodels, arch | +| `explainability` | SHAP | +| `credit_risk` | scorecardpy | +| `datasets` | Hugging Face datasets | +| `pii-detection` | presidio-analyzer, presidio-structured | +| `all` | All optional dependencies | + +## What's next + +:::{#next-models} +::: + + + +[^1]: [How to run tests and test suites](/developer/how-to/testing-overview.qmd) diff --git a/site/developer/supported-models.qmd b/site/developer/supported-models.qmd deleted file mode 100644 index 930db0533f..0000000000 --- a/site/developer/supported-models.qmd +++ /dev/null @@ -1,160 +0,0 @@ ---- -# Copyright © 2023-2026 ValidMind Inc. All rights reserved. -# Refer to the LICENSE file in the root of this repository for details. -# SPDX-License-Identifier: AGPL-3.0 AND ValidMind Commercial -title: "Supported models" -date: last-modified -aliases: - - /guide/supported-models.html - - /developer/model-documentation/supported-models.html -listing: - - id: next-models - type: grid - grid-columns: 2 - max-description-length: 250 - sort: false - fields: [title, description] - contents: - - /how-to/testing-overview.qmd - - test-descriptions.qmd - - /how-to/feature-overview.qmd - - samples-jupyter-notebooks.qmd ---- - -The {{< var validmind.developer >}} provides out-of-the-box support for testing and documentation for an array of model types and modeling packages. - -## What is a supported model? - -A _supported model_ refers to a model for which predefined testing or documentation functions exist in the {{< var validmind.developer >}}, provided that the model you are developing is documented using a supported version of our {{< var vm.developer >}}. These model types cover a very large portion of the models used in commercial and retail banking. - -::: {.callout} -## {{< var vm.product >}} does not limit our users to specific model types. - -- The {{< var validmind.developer >}} is extensible to support future model types or modeling packages to accomodate rapid developments in the AI space, including the advent of large language models (LLMs). -- You always have the flexibility to implement custom tests and integrate external test providers.[^1] -::: - - - -## Supported model types - -::: {.column-margin} -::: {.feature} -Vendor models -: {{< var vm.product >}} offers support for both first-party models and [third-party vendor models](/about/glossary/glossary.qmd#vendor-model). - -::: - -::: - -### Traditional statistical models - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-30-ns} - -#### Linear regression -Models relationship between a scalar response and one or more explanatory variables. - -::: - -::: {.w-30-ns} - -#### Logistic regression -Models relationship between a scalar response and one or more explanatory variables. - -::: - -::: {.w-30-ns} -#### Time series -Analyzes data points collected or sequenced over time. - -::: - -:::: - -### Machine learning models - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns .pr2} -#### Hugging Face-compatible models -- Natural language processing (NLP) text classification — Categorizes text into predefined classes. -- Tabular classification — Assigns categories to tabular dataset entries. -- Tabular regression — Predicts continuous outcomes from tabular data. - -#### Neural networks -- Long short-term memory (LSTM) — Processes sequences of data, remembering inputs over long periods. -- Recurrent neural network (RNN) — Processes sequences by maintaining a state that reflects the history of processed elements. -- Convolutional neural network (CNN) — Primarily used for processing grid-like data such as images. - - -::: - -::: {.w-50-ns .pl2 .pr2} - -#### Tree-based models
(XGBoost / CatBoost / random forest) -- Classification — Predicts categorical outcomes using decision trees. -- Regression — Predicts continuous outcomes using decision trees. - -#### K-nearest neighbors (KNN) -- Classification — Assigns class by majority vote of the k-nearest neighbors. -- Regression — Predicts value based on the average of the k-nearest neighbors. - -#### Clustering -- K-means — Partitions _n_ observations into _k_ clusters in which each observation belongs to the cluster with the nearest mean. - - -::: - -:::: - -### Generative AI models - -#### Large language models (LLMs) -- Classification — Categorizes input into predefined classes. -- Text summarization — Generates concise summaries from longer texts. - -## Supported modeling libraries and other tools - -:::: {.flex .flex-wrap .justify-around} - -::: {.w-50-ns} - -- **[scikit-learn](https://scikit-learn.org/stable/)** — A Python library for machine learning, offering a range of supervised and unsupervised learning algorithms. - -- **[statsmodels](https://www.statsmodels.org/stable/index.html)** — A Python module that provides classes and functions for the estimation of many different statistical models, as well as for conducting statistical tests and exploring data. - -- **[PyTorch](https://pytorch.org/)** — An open-source machine learning library based on the Torch library, used for applications such as computer vision and natural language processing. - -- **[Hugging Face Transformers](https://huggingface.co/docs/transformers/en/index)** — Provides thousands of pre-trained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, and text generation. - -- **[XGBoost](https://xgboost.readthedocs.io/en/stable/)** — An optimized distributed gradient boosting library designed to be highly efficient, flexible, and portable, implementing machine learning algorithms under the Gradient Boosting framework. - -::: - -::: {.w-50-ns} - -- **[CatBoost](https://catboost.ai/)** — An open-source gradient boosting on decision trees library with categorical feature support out of the box, for ranking, classification, regression, and other ML tasks. - -- **[LightGBM](https://lightgbm.readthedocs.io/en/stable/)** — A fast, distributed, high-performance gradient boosting (GBDT, GBRT, GBM, or MART) framework based on decision tree algorithms, used for ranking, classification, and many other machine learning tasks. - -- **R models, via [rpy2 - R in Python](https://rpy2.github.io/)** — Facilitates the integration of R's statistical computing and graphics capabilities with Python, allowing for R models to be called from Python. - -- **Large language models (LLMs), via [OpenAI-compatible APIs](https://platform.openai.com/docs/introduction)** — Access to advanced AI models trained by OpenAI for a variety of natural language tasks, including text generation, translation, and analysis, through a compatible API interface. This support includes both the OpenAI API and the Azure OpenAI Service via API. -::: - -:::: - -## What's next - -:::{#next-models} -::: - - - - -[^1]: - - - [Implement custom tests](/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb) - - [Integrate external test providers](/notebooks/how_to/tests/custom_tests/integrate_external_test_providers.ipynb) \ No newline at end of file diff --git a/site/faq/faq-integrations.qmd b/site/faq/faq-integrations.qmd index 8c3d7ac9ea..a8c86a0238 100644 --- a/site/faq/faq-integrations.qmd +++ b/site/faq/faq-integrations.qmd @@ -14,7 +14,7 @@ listing: sort: false fields: [title, description] contents: - - ../developer/supported-models.qmd + - ../developer/supported-models-and-frameworks.qmd - ../about/overview-llm-features.qmd - ../about/deployment/deployment-options.qmd categories: ["supported libraries", "supported languages", "integrations", "images", "large language models", "explainability", "deployment options", "validmind library"] @@ -82,7 +82,7 @@ We will be implementing connector interfaces allowing extraction of relationship [^1]: [{{< var validmind.developer >}}](/developer/validmind-library.qmd) -[^2]: [Supported modeling libraries and other tools](/developer/supported-models.qmd#supported-modeling-libraries-and-other-tools) +[^2]: [Supported models and frameworks](/developer/supported-models-and-frameworks.qmd) [^3]: [Matplotlib](https://matplotlib.org/) diff --git a/site/releases/2024/2024-mar-27/highlights.qmd b/site/releases/2024/2024-mar-27/highlights.qmd index 5d499af572..162ef46335 100644 --- a/site/releases/2024/2024-mar-27/highlights.qmd +++ b/site/releases/2024/2024-mar-27/highlights.qmd @@ -458,7 +458,7 @@ We improved our supported models documentation with additional information about ::: ::: {.w-30-ns .tc} -[Supported models](/developer/supported-models.qmd){.button} +[Supported models and frameworks](/developer/supported-models-and-frameworks.qmd){.button} :::