diff --git a/notebooks/code_sharing/clustering/quickstart_cluster_demo.ipynb b/notebooks/code_sharing/clustering/quickstart_cluster_demo.ipynb index 963bbedb9..e9f30a1fc 100644 --- a/notebooks/code_sharing/clustering/quickstart_cluster_demo.ipynb +++ b/notebooks/code_sharing/clustering/quickstart_cluster_demo.ipynb @@ -34,23 +34,23 @@ ] }, { - "attachments": {}, "cell_type": "markdown", - "metadata": { - "id": "ZNcbDRubSPc1" - }, + "id": "setup-intro", + "metadata": {}, "source": [ - "## Before you begin\n", - "\n", - "To use the ValidMind Library with a Jupyter notebook, you need to install and initialize the ValidMind Library first, along with getting your Python environment ready. When running this notebook locally, this includes installing any missing prerequisite modules that you discover with `pip install`. " + "## Setting up" ] }, { - "attachments": {}, "cell_type": "markdown", + "id": "setup-install", "metadata": {}, "source": [ - "## Install the ValidMind Library\n", + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", "\n", "To install the library:" ] @@ -58,44 +58,76 @@ { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "6G5-kHOZ7YWk" - }, + "id": "install-code", + "metadata": {}, "outputs": [], "source": [ "%pip install -q validmind" ] }, { - "attachments": {}, "cell_type": "markdown", - "metadata": { - "id": "_ZqI8W5jSPc1" - }, + "id": "setup-initialize", + "metadata": {}, "source": [ - "## Initialize the ValidMind Library\n", - "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "id": "register-model", + "metadata": {}, + "source": [ + "#### Register sample model\n", "\n", - "### Get your code snippet\n", + "Let's first register a sample model for use with this notebook.\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", - "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", "\n", - "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", "\n", - "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "id": "apply-template", + "metadata": {}, + "source": [ + "#### Apply documentation template\n", + "\n", + "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "id": "code-snippet", + "metadata": {}, + "source": [ + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, - "metadata": { - "id": "5hqGn9jHSPc2" - }, + "id": "initialize-code", + "metadata": {}, "outputs": [], "source": [ "# Load your model identifier credentials from an `.env` file\n", @@ -112,10 +144,32 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " #document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, + { + "cell_type": "markdown", + "id": "preview-template", + "metadata": {}, + "source": [ + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, { "attachments": {}, "cell_type": "markdown", @@ -390,24 +444,43 @@ ] }, { - "attachments": {}, "cell_type": "markdown", + "id": "next-steps", "metadata": {}, "source": [ "## Next steps\n", "\n", - "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the test results as part of your model documentation right in the ValidMind Platform: \n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation." + ] + }, + { + "cell_type": "markdown", + "id": "next-document", + "metadata": {}, + "source": [ + "### Work with your model documentation\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + ] + }, + { + "cell_type": "markdown", + "id": "next-resources", + "metadata": {}, + "source": [ + "### Discover more learning resources\n", "\n", - "2. Expand the following sections and take a look around:\n", - " \n", - " - **2. Data Preparation**\n", - " - **3. Model Development**\n", + "We offer many interactive notebooks to help you automate testing, documenting, validating, and more:\n", "\n", - "What you can see now is a much more easily consumable version of the documentation, including the results of the tests you just performed, along with other parts of your model documentation that still need to be completed. There is a wealth of information that gets uploaded when you run the full test suite, so take a closer look around, especially at test results that might need attention (hint: some of the tests in 2.1 Data description look like they need some attention).\n", + "- [Run tests & test suites](https://docs.validmind.ai/developer/how-to/testing-overview.html)\n", + "- [Use ValidMind Library features](https://docs.validmind.ai/developer/how-to/feature-overview.html)\n", + "- [Code samples by use case](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", "\n", - "If you want to learn more about where you are in the model documentation process, take a look at our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html).\n" + "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind." ] }, { diff --git a/notebooks/code_sharing/credit_risk/assign_prediction_probabilities.ipynb b/notebooks/code_sharing/credit_risk/assign_prediction_probabilities.ipynb index c031fd556..7c763fa57 100644 --- a/notebooks/code_sharing/credit_risk/assign_prediction_probabilities.ipynb +++ b/notebooks/code_sharing/credit_risk/assign_prediction_probabilities.ipynb @@ -68,12 +68,17 @@ "\n", "### Install the ValidMind Library\n", "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", "To install the library:" ] }, { "cell_type": "code", "execution_count": null, + "id": "install-code", "metadata": {}, "outputs": [], "source": [ @@ -97,7 +102,7 @@ "\n", "#### Register sample model\n", "\n", - "Let's first register a sample model for use with this notebook:\n", + "Let's first register a sample model for use with this notebook.\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", @@ -120,7 +125,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -135,15 +140,17 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -161,7 +168,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -173,7 +180,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -181,6 +188,7 @@ { "cell_type": "code", "execution_count": null, + "id": "template-code", "metadata": {}, "outputs": [], "source": [ @@ -736,7 +744,6 @@ }, { "cell_type": "markdown", - "id": "copyright-0871b365e3924d0995a276dbbbcf62c1", "metadata": {}, "source": [ "\n", diff --git a/notebooks/code_sharing/deepeval_integration_demo.ipynb b/notebooks/code_sharing/deepeval_integration_demo.ipynb index 504b2a402..d877e4c59 100644 --- a/notebooks/code_sharing/deepeval_integration_demo.ipynb +++ b/notebooks/code_sharing/deepeval_integration_demo.ipynb @@ -2,11 +2,7 @@ "cells": [ { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ "# DeepEval Integration with ValidMind\n", "\n", @@ -17,89 +13,97 @@ " 2. Create a dataset with source texts and generated summaries\n", " 3. Use ValidMind's Summarization scorer to evaluate summary quality\n", " 4. Analyze the evaluation results and reasons\n", - " 5. Apply the evaluation pipeline to multiple examples\n" + " 5. Apply the evaluation pipeline to multiple examples" ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ + "::: {.content-hidden when-format=\"html\"}\n", "## Contents \n", - "- [Introduction](#toc1_) \n", - "- [About DeepEval Integration](#toc2_) \n", - " - [Before you begin](#toc2_1_) \n", - " - [Key concepts](#toc2_2_) \n", - "- [Setting up](#toc3_) \n", - " - [Install required packages](#toc3_1_) \n", - " - [Initialize ValidMind](#toc3_2_) \n", - "- [Basic Usage - Simple Q&A Evaluation](#toc4_) \n", - "- [RAG System Evaluation](#toc5_) \n", - " - [Create test cases](#toc5_1_) \n", - " - [Build dataset](#toc5_2_) \n", - " - [Evaluation metrics](#toc5_3_) \n", - " - [Contextual Relevancy](#toc5_3_1_) \n", - " - [Contextual Precision](#toc5_3_2_) \n", - " - [Contextual Recall](#toc5_3_3_) \n", - "- [LLM Agent Evaluation](#toc6_) \n", - " - [Create test cases](#toc6_1_) \n", - " - [Build dataset](#toc6_2_) \n", - " - [Evaluation metrics](#toc6_3_) \n", - " - [Faithfulness](#toc6_3_1_) \n", - " - [Hallucination](#toc6_3_2_) \n", - " - [Summarization](#toc6_3_3_) \n", - " - [AI Agent Evaluation Metrics](#toc6_3_4_) \n", - "- [In summary](#toc10_) \n", - "- [Next steps](#toc11_) \n", - "\n" + "- [Contents](#toc1__) \n", + "- [Introduction](#toc2__) \n", + "- [About DeepEval Integration](#toc3__) \n", + " - [Before you begin](#toc3_1__) \n", + " - [Key concepts](#toc3_2__) \n", + "- [Setting up](#toc4__) \n", + " - [Install the ValidMind Library](#toc4_1__) \n", + " - [Initialize the ValidMind Library](#toc4_2__) \n", + " - [Register sample model](#toc4_2_1__) \n", + " - [Apply documentation template](#toc4_2_2__) \n", + " - [Get your code snippet](#toc4_2_3__) \n", + " - [Preview the documentation template](#toc4_3__) \n", + "- [Basic Usage - Simple Q&A Evaluation](#toc5__) \n", + " - [Create a simple LLM test case](#toc5_1__) \n", + " - [Create LLMAgentDataset from the test case](#toc5_2__) \n", + " - [Compute metrics using ValidMind scorer interface](#toc5_3__) \n", + "- [RAG System Evaluation](#toc6__) \n", + " - [Create test cases](#toc6_1__) \n", + " - [Build dataset](#toc6_2__) \n", + " - [Evaluation metrics](#toc6_3__) \n", + " - [Contextual Relevancy](#toc6_3_1__) \n", + " - [Contextual Precision](#toc6_3_2__) \n", + " - [Contextual Recall](#toc6_3_3__) \n", + "- [LLM Agent Evaluation](#toc7__) \n", + " - [Build dataset](#toc7_1__) \n", + " - [Faithfulness](#toc7_1_1__) \n", + " - [Hallucination](#toc7_1_2__) \n", + " - [Summarization](#toc7_1_3__) \n", + " - [AI Agent Evaluation Metrics](#toc7_2__) \n", + " - [**Reasoning Layer**](#toc7_2_1__) \n", + " - [PlanQualityMetric](#toc7_2_2__) \n", + " - [PlanAdherenceMetric](#toc7_2_3__) \n", + " - [**Action Layer**](#toc7_2_4__) \n", + " - [ToolCorrectnessMetric](#toc7_2_5__) \n", + " - [ArgumentCorrectnessMetric](#toc7_2_6__) \n", + " - [**Execution Layer**](#toc7_2_7__) \n", + " - [TaskCompletionMetric](#toc7_2_8__) \n", + "- [In summary](#toc8__) \n", + "- [Next steps](#toc9__) \n", + "\n", + ":::\n", + "\n", + "" ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## Introduction\n", "\n", "Large Language Model (LLM) evaluation is critical for understanding model performance across different tasks and scenarios. This notebook demonstrates how to integrate DeepEval's comprehensive evaluation framework with ValidMind's testing infrastructure to create a robust LLM evaluation pipeline.\n", "\n", - "DeepEval provides over 30 evaluation metrics specifically designed for LLMs, covering scenarios from simple Q&A to complex agent interactions. By integrating with ValidMind, you can leverage these metrics within a structured testing framework that supports documentation, collaboration, and compliance requirements.\n" + "DeepEval provides over 30 evaluation metrics specifically designed for LLMs, covering scenarios from simple Q&A to complex agent interactions. By integrating with ValidMind, you can leverage these metrics within a structured testing framework that supports documentation, collaboration, and compliance requirements." ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## About DeepEval Integration\n", "\n", "DeepEval is a comprehensive evaluation framework for LLMs that provides metrics for various scenarios including hallucination detection, answer relevancy, faithfulness, and custom evaluation criteria. ValidMind is a platform for managing model risk and documentation through automated testing.\n", "\n", - "Together, these tools enable comprehensive LLM evaluation within a structured, compliant framework.\n" + "Together, these tools enable comprehensive LLM evaluation within a structured, compliant framework." ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Before you begin\n", "\n", @@ -109,18 +113,14 @@ "- Access to OpenAI API (for DeepEval metrics evaluation)\n", "- ValidMind account and model registration\n", "\n", - "If you encounter errors due to missing modules, install them with `pip install` and re-run the notebook.\n" + "If you encounter errors due to missing modules, install them with `pip install` and re-run the notebook." ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Key concepts\n", "\n", @@ -130,35 +130,31 @@ "\n", "**RAG Evaluation**: Testing retrieval-augmented generation systems that combine document retrieval with generation.\n", "\n", - "**Agent Evaluation**: Testing LLM agents that can use tools and perform multi-step reasoning.\n" + "**Agent Evaluation**: Testing LLM agents that can use tools and perform multi-step reasoning." ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", - "## Setting up\n" + "## Setting up" ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", + "\n", + "### Install the ValidMind Library\n", "\n", - "### Install required packages\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", "\n", - "First, let's install the required packages and set up our environment.\n" + "To install the library:" ] }, { @@ -172,21 +168,64 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, + "source": [ + "\n", + "\n", + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, "source": [ - "\n", + "\n", "\n", - "### Initialize ValidMind\n", + "#### Register sample model\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Let's first register a sample model for use with this notebook.\n", "\n", - "
For access to all features available in this notebook, you'll need access to a ValidMind account.\n", - "

\n", - "Register with ValidMind
\n" + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Apply documentation template\n", + "\n", + "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -196,21 +235,46 @@ "outputs": [], "source": [ "# Load your model identifier credentials from an `.env` file\n", + "\n", "%load_ext dotenv\n", "%dotenv .env\n", "\n", "# Or replace with your code snippet\n", + "\n", "import validmind as vm\n", "\n", "vm.init(\n", - " api_host=\"...\",\n", - " api_key=\"...\",\n", - " api_secret=\"...\",\n", - " model=\"...\",\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", " document=\"documentation\",\n", ")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, { "cell_type": "code", "execution_count": null, @@ -228,23 +292,21 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## Basic Usage - Simple Q&A Evaluation\n", "\n", - "Let's start with the simplest use case: evaluating a basic question-and-answer interaction with an LLM. This demonstrates how to create LLMTestCase objects and integrate them with ValidMind's dataset infrastructure.\n" + "Let's start with the simplest use case: evaluating a basic question-and-answer interaction with an LLM. This demonstrates how to create LLMTestCase objects and integrate them with ValidMind's dataset infrastructure." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ + "\n", + "\n", "### Create a simple LLM test case" ] }, @@ -312,6 +374,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", + "\n", "### Create LLMAgentDataset from the test case\n", "Let's create ValidMind dataset from Deepeval's test cases" ] @@ -351,9 +415,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", + "\n", "### Compute metrics using ValidMind scorer interface\n", "\n", - "Now we'll compute metrics on our dataset using ValidMind's scorer interface. This will help us evaluate how well our model is performing by calculating various metrics like answer relevancy. The scorer interface provides a standardized way to assess model outputs against expected results.\n" + "Now we'll compute metrics on our dataset using ValidMind's scorer interface. This will help us evaluate how well our model is performing by calculating various metrics like answer relevancy. The scorer interface provides a standardized way to assess model outputs against expected results." ] }, { @@ -386,24 +452,20 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## RAG System Evaluation\n", "\n", - "Now let's evaluate a more complex use case: a Retrieval-Augmented Generation (RAG) system that retrieves relevant documents and generates responses based on them. RAG systems combine document retrieval with text generation, requiring specialized evaluation approaches.\n" + "Now let's evaluate a more complex use case: a Retrieval-Augmented Generation (RAG) system that retrieves relevant documents and generates responses based on them. RAG systems combine document retrieval with text generation, requiring specialized evaluation approaches." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Create test cases" ] @@ -468,7 +530,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Build dataset\n", "\n", @@ -484,7 +546,7 @@ "- Retrieved context passages\n", "\n", "This structured format enables detailed analysis of the RAG system's performance\n", - "across multiple evaluation dimensions.\n" + "across multiple evaluation dimensions." ] }, { @@ -510,7 +572,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Evaluation metrics" ] @@ -519,7 +581,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "#### Contextual Relevancy\n", "The Contextual Relevancy metric evaluates how well the retrieved context aligns with the input query.\n", @@ -527,7 +589,7 @@ "A high relevancy score indicates that the retrieved context is highly relevant and contains the key information needed.\n", "This helps validate that the RAG system is retrieving appropriate context for the given queries.\n", "\n", - "\n", + "\n", "\n", "#### Contextual Precision\n", "The Contextual Precision metric evaluates how well a RAG system ranks retrieved context nodes by relevance to the input query. \n", @@ -535,7 +597,7 @@ "A high precision score indicates that the retrieved context is highly relevant to the query and properly ranked.\n", "This is particularly useful for evaluating RAG systems and ensuring they surface the most relevant information first.\n", "\n", - "\n", + "\n", "\n", "#### Contextual Recall\n", "The Contextual Recall metric evaluates how well the retrieved context covers all the information needed to generate the expected output.\n", @@ -543,7 +605,7 @@ "A high recall score indicates that the retrieved context contains all the key information needed to generate the expected response.\n", "This helps ensure the RAG system retrieves comprehensive context that covers all aspects of the expected answer.\n", "\n", - "Now we'll evaluate the RAG system's performance using multiple metrics at once. The `assign_scores()` method accepts a list of metrics to evaluate different aspects of the system's behavior. The metrics will add score and reason columns to the dataset, providing quantitative and qualitative feedback on the system's performance. This multi-metric evaluation gives us comprehensive insights into the strengths and potential areas for improvement.\n" + "Now we'll evaluate the RAG system's performance using multiple metrics at once. The `assign_scores()` method accepts a list of metrics to evaluate different aspects of the system's behavior. The metrics will add score and reason columns to the dataset, providing quantitative and qualitative feedback on the system's performance. This multi-metric evaluation gives us comprehensive insights into the strengths and potential areas for improvement." ] }, { @@ -565,13 +627,9 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## LLM Agent Evaluation\n", "\n", @@ -582,8 +640,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", - "### Create test cases\n" + "### Create test cases" ] }, { @@ -700,9 +757,9 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", - "### Build dataset\n" + "### Build dataset" ] }, { @@ -748,13 +805,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", - "\n", "### Evaluation metrics\n", - "\n", + "\n", + "\n", "\n", "#### Faithfulness\n", - "The Faithfulness metric evaluates whether the model's output contains any contradictions or hallucinations compared to the provided context. It ensures that the model's response is grounded in and consistent with the given information, rather than making up facts or contradicting the context. A high faithfulness score indicates that the model's output aligns well with the source material.\n" + "The Faithfulness metric evaluates whether the model's output contains any contradictions or hallucinations compared to the provided context. It ensures that the model's response is grounded in and consistent with the given information, rather than making up facts or contradicting the context. A high faithfulness score indicates that the model's output aligns well with the source material." ] }, { @@ -776,10 +832,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "#### Hallucination\n", - "The Hallucination metric evaluates whether the model's output contains information that is not supported by or contradicts the provided context. It helps identify cases where the model makes up facts or includes details that aren't grounded in the source material. A low hallucination score indicates that the model's response stays faithful to the given context without introducing unsupported information.\n" + "The Hallucination metric evaluates whether the model's output contains information that is not supported by or contradicts the provided context. It helps identify cases where the model makes up facts or includes details that aren't grounded in the source material. A low hallucination score indicates that the model's response stays faithful to the given context without introducing unsupported information." ] }, { @@ -801,10 +857,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "#### Summarization\n", - "The Summarization metric evaluates how well a model's output summarizes the given context by generating assessment questions to check if the summary is factually aligned with and sufficiently covers the source text. It helps ensure that summaries are accurate, complete, and maintain the key information from the original content without introducing unsupported details or omitting critical points. A high summarization score indicates that the model effectively condenses the source material while preserving its essential meaning.\n" + "The Summarization metric evaluates how well a model's output summarizes the given context by generating assessment questions to check if the summary is factually aligned with and sufficiently covers the source text. It helps ensure that summaries are accurate, complete, and maintain the key information from the original content without introducing unsupported details or omitting critical points. A high summarization score indicates that the model effectively condenses the source material while preserving its essential meaning." ] }, { @@ -825,7 +881,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "### AI Agent Evaluation Metrics\n", "\n", @@ -856,8 +912,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", "\n", "#### **Reasoning Layer**\n", + "\n", + "\n", "#### PlanQualityMetric\n", "Let's measures how well the agent generates a plan before acting. A high score means the plan is logical, complete, and efficient." ] @@ -881,6 +940,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", + "\n", "#### PlanAdherenceMetric\n", "Let's checks whether the agent follows the plan it created. Deviations lower this score and indicate gaps between reasoning and execution." ] @@ -905,8 +966,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", "\n", "#### **Action Layer**\n", + "\n", + "\n", "#### ToolCorrectnessMetric\n", "Let's evaluates if the agent selects the appropriate tool for the task. Choosing the wrong tool reduces performance even if reasoning was correct." ] @@ -932,6 +996,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", + "\n", "#### ArgumentCorrectnessMetric\n", "Let's assesses whether the agent provides correct inputs or arguments to the selected tool. Incorrect arguments can lead to failed or unexpected results." ] @@ -955,8 +1021,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", "\n", "#### **Execution Layer**\n", + "\n", + "\n", "#### TaskCompletionMetric\n", "Let's measures whether the agent successfully completes the overall task. This is the ultimate indicator of success." ] @@ -980,13 +1049,9 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## In summary\n", "\n", @@ -1003,18 +1068,14 @@ "- **Flexibility**: Support for custom metrics and domain-specific evaluation criteria\n", "- **Production Ready**: Handle real-world LLM evaluation scenarios at scale\n", "\n", - "The `LLMAgentDataset` class provides a seamless bridge between DeepEval's evaluation capabilities and ValidMind's testing infrastructure, enabling robust LLM evaluation within a structured, compliant framework.\n" + "The `LLMAgentDataset` class provides a seamless bridge between DeepEval's evaluation capabilities and ValidMind's testing infrastructure, enabling robust LLM evaluation within a structured, compliant framework." ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## Next steps\n", "\n", @@ -1033,12 +1094,11 @@ "- Integrate with your existing ML infrastructure and workflows\n", "- Explore multi-modal evaluation scenarios (text, code, images)\n", "\n", - "Start building comprehensive LLM evaluation workflows that combine the power of DeepEval's specialized metrics with ValidMind's structured testing and documentation framework.\n" + "Start building comprehensive LLM evaluation workflows that combine the power of DeepEval's specialized metrics with ValidMind's structured testing and documentation framework." ] }, { "cell_type": "markdown", - "id": "copyright-0bacca5c16774a5bb77c86f9b25602da", "metadata": {}, "source": [ "\n", diff --git a/notebooks/code_sharing/embeddings/quickstart_embeddings_demo.ipynb b/notebooks/code_sharing/embeddings/quickstart_embeddings_demo.ipynb index efcda264e..992ea2132 100644 --- a/notebooks/code_sharing/embeddings/quickstart_embeddings_demo.ipynb +++ b/notebooks/code_sharing/embeddings/quickstart_embeddings_demo.ipynb @@ -27,24 +27,22 @@ }, { "cell_type": "markdown", - "id": "0b6f02be", + "id": "setup-intro", "metadata": {}, "source": [ - "## Before you begin\n", - "\n", - "
For access to all features available in this notebook, you'll need access to a ValidMind account.\n", - "

\n", - "Register with ValidMind
\n", - "\n", - "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)." + "## Setting up" ] }, { "cell_type": "markdown", - "id": "4b27fcda", + "id": "setup-install", "metadata": {}, "source": [ - "## Install the ValidMind Library\n", + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", "\n", "To install the library:" ] @@ -52,7 +50,7 @@ { "cell_type": "code", "execution_count": null, - "id": "d1365a88", + "id": "install-code", "metadata": {}, "outputs": [], "source": [ @@ -61,30 +59,66 @@ }, { "cell_type": "markdown", - "id": "03369f9e", + "id": "setup-initialize", "metadata": {}, "source": [ - "## Initialize the ValidMind Library\n", - "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "id": "register-model", + "metadata": {}, + "source": [ + "#### Register sample model\n", "\n", - "### Get your code snippet\n", + "Let's first register a sample model for use with this notebook.\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", - "2. In the left sidebar, navigate to **Model Inventory** and click **+ Register Model**.\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "id": "apply-template", + "metadata": {}, + "source": [ + "#### Apply documentation template\n", + "\n", + "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", "\n", - "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "id": "code-snippet", + "metadata": {}, + "source": [ + "#### Get your code snippet\n", "\n", - "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, - "id": "ab825078", + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -102,10 +136,32 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, + { + "cell_type": "markdown", + "id": "preview-template", + "metadata": {}, + "source": [ + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, { "cell_type": "markdown", "id": "e4978450", diff --git a/notebooks/code_sharing/geval_deepeval_integration_demo.ipynb b/notebooks/code_sharing/geval_deepeval_integration_demo.ipynb index 8a393e082..2f1828c73 100644 --- a/notebooks/code_sharing/geval_deepeval_integration_demo.ipynb +++ b/notebooks/code_sharing/geval_deepeval_integration_demo.ipynb @@ -2,11 +2,7 @@ "cells": [ { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ "# G-Eval Integration for DeepEval within ValidMind\n", "\n", @@ -16,44 +12,50 @@ "To integrate DeepEval with ValidMind, we'll:\n", " 1. Set up both frameworks and install required dependencies\n", " 2. Create a dataset with source texts and generated summaries\n", - " 3. Analyze the evaluation results using G-eval custom metrics\n" + " 3. Analyze the evaluation results using G-eval custom metrics" ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ + "::: {.content-hidden when-format=\"html\"}\n", "## Contents \n", - "- [Introduction](#toc1_) \n", - " - [Before you begin](#toc2_1_) \n", - " - [Key concepts](#toc2_2_) \n", - "- [Setting up](#toc3_) \n", - " - [Install required packages](#toc3_1_) \n", - " - [Initialize ValidMind](#toc3_2_) \n", - "- [Custom Metrics with G-Eval](#toc4_) \n", - " - [Technical accuracy](#toc4_1_) \n", - " - [Clarity and Comprehensiveness](#toc4_2_) \n", - " - [Business Context Appropriateness](#toc4_3_) \n", - " - [Tool Usage Appropriateness](#toc4_4_) \n", - " - [Coherence Evaluation](#toc4_5_) \n", - "- [In summary](#toc5_) \n", - "- [Next steps](#toc6_) \n", - "\n" + "- [Introduction](#toc1__) \n", + " - [Before you begin](#toc1_1__) \n", + " - [Key concepts](#toc1_2__) \n", + "- [Setting up](#toc2__) \n", + " - [Install the ValidMind Library](#toc2_1__) \n", + " - [Initialize the ValidMind Library](#toc2_2__) \n", + " - [Register sample model](#toc2_2_1__) \n", + " - [Apply documentation template](#toc2_2_2__) \n", + " - [Get your code snippet](#toc2_2_3__) \n", + " - [Preview the documentation template](#toc2_3__) \n", + "- [Create test cases](#toc3__) \n", + "- [Scorers in ValidMind](#toc4__) \n", + "- [Custom Metrics with G-Eval](#toc5__) \n", + " - [Technical accuracy](#toc5_1__) \n", + " - [Clarity and Comprehensiveness](#toc5_2__) \n", + " - [Business Context Appropriateness](#toc5_3__) \n", + " - [Conciseness Evaluation](#toc5_4__) \n", + "- [Next steps](#toc6__) \n", + "\n", + ":::\n", + "\n", + "" ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## Introduction\n", "**G-Eval** is a framework that uses large language models (LLMs) as evaluators—essentially treating an LLM as a “judge” to assess the quality of other LLM outputs. Instead of relying on traditional metrics like BLEU or ROUGE, G-Eval enables natural-language evaluation criteria (e.g., “rate how factual this summary is”). The framework guides the judge model through structured reasoning steps, producing more consistent, transparent, and interpretable scoring results. It is particularly effective for subjective or open-ended tasks such as summarization, dialogue generation, and content evaluation.\n", @@ -69,13 +71,9 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Before you begin\n", "\n", @@ -85,53 +83,44 @@ "- Access to OpenAI API (for DeepEval metrics evaluation)\n", "- ValidMind account and model registration\n", "\n", - "If you encounter errors due to missing modules, install them with `pip install` and re-run the notebook.\n" + "If you encounter errors due to missing modules, install them with `pip install` and re-run the notebook." ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Key concepts\n", "\n", "**LLMTestCase**: A DeepEval object that represents a single test case with input, expected output, actual output, and optional context.\n", "\n", - "**G-Eval**: Generative evaluation using LLMs to assess response quality based on custom criteria.\n", - "\n" + "**G-Eval**: Generative evaluation using LLMs to assess response quality based on custom criteria." ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", - "## Setting up\n" + "## Setting up" ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", - "### Install required packages\n", + "### Install the ValidMind Library\n", "\n", - "First, let's install the required packages and set up our environment.\n" + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" ] }, { @@ -145,21 +134,64 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", - "### Initialize ValidMind\n", + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "#### Register sample model\n", "\n", - "
For access to all features available in this notebook, you'll need access to a ValidMind account.\n", - "

\n", - "Register with ValidMind
\n" + "Let's first register a sample model for use with this notebook.\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Apply documentation template\n", + "\n", + "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -169,21 +201,46 @@ "outputs": [], "source": [ "# Load your model identifier credentials from an `.env` file\n", + "\n", "%load_ext dotenv\n", "%dotenv .env\n", "\n", - "# # Or replace with your code snippet\n", + "# Or replace with your code snippet\n", + "\n", "import validmind as vm\n", "\n", "vm.init(\n", - " api_host=\"...\",\n", - " api_key=\"...\",\n", - " api_secret=\"...\",\n", - " model=\"...\",\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", " document=\"documentation\",\n", ")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, { "cell_type": "code", "execution_count": null, @@ -210,6 +267,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", "\n", "## Create test cases\n", "\n", @@ -250,6 +308,8 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "\n", + "\n", "## Scorers in ValidMind\n", "\n", "Scorers are evaluation metrics that analyze model outputs and store their results in the dataset. When using `assign_scores()`:\n", @@ -264,13 +324,12 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "## Custom Metrics with G-Eval\n", "One of DeepEval's most powerful features is the ability to create custom evaluation metrics using G-Eval (Generative Evaluation). This enables domain-specific evaluation criteria tailored to your use case.\n", "\n", - "\n", - "\n", + "\n", "\n", "### Technical accuracy" ] @@ -303,11 +362,10 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Clarity and Comprehensiveness\n", - "This evaluation assesses the clarity and comprehensiveness of responses, focusing on how well-structured and understandable they are. The criteria examines whether responses are logically organized, address all aspects of questions thoroughly, and maintain an appropriate level of detail without being overly verbose.\n", - " \n" + "This evaluation assesses the clarity and comprehensiveness of responses, focusing on how well-structured and understandable they are. The criteria examines whether responses are logically organized, address all aspects of questions thoroughly, and maintain an appropriate level of detail without being overly verbose." ] }, { @@ -342,11 +400,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Business Context Appropriateness\n", "\n", - "This evaluation assesses whether responses are appropriate for a business context, considering factors like professional tone, business relevance, and actionable insights. The criteria focuses on ensuring content would be valuable and applicable for business users.\n" + "This evaluation assesses whether responses are appropriate for a business context, considering factors like professional tone, business relevance, and actionable insights. The criteria focuses on ensuring content would be valuable and applicable for business users." ] }, { @@ -378,7 +436,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Conciseness Evaluation\n", "This evaluation assesses how well the responses flow and connect logically. It examines whether the content builds naturally from sentence to sentence to form a coherent narrative, rather than just being a collection of related but disconnected information. The evaluation considers factors like fluency, logical progression, and overall readability." @@ -457,13 +515,9 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "metadata": {}, "source": [ - "\n", + "\n", "\n", "## Next steps\n", "\n", @@ -481,12 +535,11 @@ "- Integrate with your existing ML infrastructure and workflows\n", "- Explore multi-modal evaluation scenarios (text, code, images)\n", "\n", - "Start building comprehensive LLM evaluation workflows that combine the power of DeepEval's specialized metrics with ValidMind's structured testing and documentation framework.\n" + "Start building comprehensive LLM evaluation workflows that combine the power of DeepEval's specialized metrics with ValidMind's structured testing and documentation framework." ] }, { "cell_type": "markdown", - "id": "copyright-66949a76c825474d92b8082056a7202d", "metadata": {}, "source": [ "\n", diff --git a/notebooks/code_sharing/insurance_mortality/insurance_validation_demo.ipynb b/notebooks/code_sharing/insurance_mortality/insurance_validation_demo.ipynb index f01727d9d..ab97a1749 100644 --- a/notebooks/code_sharing/insurance_mortality/insurance_validation_demo.ipynb +++ b/notebooks/code_sharing/insurance_mortality/insurance_validation_demo.ipynb @@ -20,6 +20,7 @@ " - [Register sample model](#toc1_3_1__) \n", " - [Apply documentation template](#toc1_3_2__) \n", " - [Get your code snippet](#toc1_3_3__) \n", + " - [Preview the documentation template](#toc1_4__) \n", "- [Load the Demo Dataset](#toc2__) \n", " - [GLM modeling 101](#toc2_1__) \n", " - [Model 1: Poisson distribution with log link on count](#toc2_2__) \n", @@ -77,12 +78,17 @@ "\n", "### Install the ValidMind Library\n", "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", "To install the library:" ] }, { "cell_type": "code", "execution_count": null, + "id": "install-code", "metadata": {}, "outputs": [], "source": [ @@ -106,7 +112,7 @@ "\n", "#### Register sample model\n", "\n", - "Let's first register a sample model for use with this notebook:\n", + "Let's first register a sample model for use with this notebook.\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", @@ -129,7 +135,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -144,15 +150,17 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -170,10 +178,33 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -501,7 +532,6 @@ }, { "cell_type": "markdown", - "id": "copyright-2442ef8075fe4d588c08432329f35b19", "metadata": {}, "source": [ "\n", diff --git a/notebooks/code_sharing/market_basket_analysis/mba_poc.ipynb b/notebooks/code_sharing/market_basket_analysis/mba_poc.ipynb index 50e53d53f..04384cbab 100644 --- a/notebooks/code_sharing/market_basket_analysis/mba_poc.ipynb +++ b/notebooks/code_sharing/market_basket_analysis/mba_poc.ipynb @@ -23,9 +23,100 @@ "%pip install -q mlxtend networkx\n" ] }, + { + "cell_type": "markdown", + "id": "setup-intro", + "metadata": {}, + "source": [ + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "setup-install", + "metadata": {}, + "source": [ + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-code", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "id": "setup-initialize", + "metadata": {}, + "source": [ + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "id": "register-model", + "metadata": {}, + "source": [ + "#### Register sample model\n", + "\n", + "Let's first register a sample model for use with this notebook.\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "id": "apply-template", + "metadata": {}, + "source": [ + "#### Apply documentation template\n", + "\n", + "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "id": "code-snippet", + "metadata": {}, + "source": [ + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + ] + }, { "cell_type": "code", "execution_count": null, + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -43,9 +134,32 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", + " document=\"documentation\",\n", ")" ] }, + { + "cell_type": "markdown", + "id": "preview-template", + "metadata": {}, + "source": [ + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, { "cell_type": "code", "execution_count": null, diff --git a/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb b/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb index b360d1eb6..d653521c1 100644 --- a/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb +++ b/notebooks/code_sharing/operational_deposit/operational_deposit_poc.ipynb @@ -18,10 +18,12 @@ " - [New to ValidMind?](#toc1_2__) \n", "- [Pre-requisites](#toc2__) \n", "- [Setting up](#toc3__) \n", - " - [Initialize the ValidMind Library](#toc3_1__) \n", - " - [Register sample model](#toc3_1_1__) \n", - " - [Apply documentation template](#toc3_1_2__) \n", - " - [Get your code snippet](#toc3_1_3__) \n", + " - [Install the ValidMind Library](#toc3_1__) \n", + " - [Initialize the ValidMind Library](#toc3_2__) \n", + " - [Register sample model](#toc3_2_1__) \n", + " - [Apply documentation template](#toc3_2_2__) \n", + " - [Get your code snippet](#toc3_2_3__) \n", + " - [Preview the documentation template](#toc3_3__) \n", "- [Data preparation](#toc4__) \n", "- [Data validation](#toc5__) \n", "- [ValidMind objects](#toc6__) \n", @@ -146,6 +148,31 @@ "source": [ "\n", "\n", + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-code", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", "### Initialize the ValidMind Library" ] }, @@ -153,11 +180,11 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "#### Register sample model\n", "\n", - "Let's first register a sample model for use with this notebook:\n", + "Let's first register a sample model for use with this notebook.\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", @@ -174,13 +201,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "#### Apply documentation template\n", "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Time Series Forecasting with ML`.\n", "\n", @@ -191,19 +218,21 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -221,10 +250,33 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, { "cell_type": "markdown", "metadata": {}, @@ -1133,7 +1185,6 @@ }, { "cell_type": "markdown", - "id": "copyright-25518ea001774e8895e7a51fa032bfe1", "metadata": {}, "source": [ "\n", diff --git a/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb b/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb index 7908df0e7..79e375446 100644 --- a/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb +++ b/notebooks/code_sharing/output_templates/customizing_tests_with_output_templates.ipynb @@ -39,7 +39,8 @@ " - [Register sample model](#toc2_2_1__) \n", " - [Apply documentation template](#toc2_2_2__) \n", " - [Get your code snippet](#toc2_2_3__) \n", - " - [Initialize the Python environment](#toc2_3__) \n", + " - [Preview the documentation template](#toc2_3__) \n", + " - [Initialize the Python environment](#toc2_4__) \n", "- [Load the sample dataset](#toc3__) \n", "- [Document the model](#toc4__) \n", " - [Prepocess the raw dataset](#toc4_1__) \n", @@ -236,12 +237,17 @@ "\n", "### Install the ValidMind Library\n", "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", "To install the library:" ] }, { "cell_type": "code", "execution_count": null, + "id": "install-code", "metadata": {}, "outputs": [], "source": [ @@ -265,7 +271,7 @@ "\n", "#### Register sample model\n", "\n", - "Let's first register a sample model for use with this notebook:\n", + "Let's first register a sample model for use with this notebook.\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", @@ -288,7 +294,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -303,15 +309,17 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -329,7 +337,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -339,6 +347,29 @@ "source": [ "\n", "\n", + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", "### Initialize the Python environment\n", "\n", "Next, let's import the necessary libraries and set up your Python environment for data analysis:" @@ -745,26 +776,33 @@ }, { "cell_type": "markdown", + "id": "next-steps", "metadata": {}, "source": [ - "\n", - "\n", "## Next steps\n", "\n", - "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation.\n", - "\n", - "\n", - "\n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation." + ] + }, + { + "cell_type": "markdown", + "id": "next-document", + "metadata": {}, + "source": [ "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", - "\n", - "2. Click and expand the **Model Development** section.\n", - "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + ] + }, + { + "cell_type": "markdown", + "id": "next-resources", + "metadata": {}, + "source": [ "### Discover more learning resources\n", "\n", "We offer many interactive notebooks to help you automate testing, documenting, validating, and more:\n", @@ -818,7 +856,6 @@ }, { "cell_type": "markdown", - "id": "copyright-9714145fb42f4245b8b0b51d499c007f", "metadata": {}, "source": [ "\n", diff --git a/notebooks/code_sharing/plots_and_stats_demo.ipynb b/notebooks/code_sharing/plots_and_stats_demo.ipynb index 8ebadbce7..569592865 100644 --- a/notebooks/code_sharing/plots_and_stats_demo.ipynb +++ b/notebooks/code_sharing/plots_and_stats_demo.ipynb @@ -58,11 +58,6 @@ "Each test is highly configurable and can be adapted to different datasets and use cases.\n" ] }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [] - }, { "cell_type": "markdown", "metadata": { @@ -78,42 +73,98 @@ }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "id": "setup-intro", + "metadata": {}, "source": [ - "## Setting up\n", + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "setup-install", + "metadata": {}, + "source": [ + "### Install the ValidMind Library\n", "\n", - "### Install the ValidMind Library\n" + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" ] }, { "cell_type": "code", "execution_count": null, + "id": "install-code", "metadata": {}, "outputs": [], "source": [ - "%pip install -q validmind\n" + "%pip install -q validmind" ] }, { "cell_type": "markdown", - "metadata": { - "vscode": { - "languageId": "raw" - } - }, + "id": "setup-initialize", + "metadata": {}, "source": [ - "### Initialize the ValidMind Library\n", + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "id": "register-model", + "metadata": {}, + "source": [ + "#### Register sample model\n", + "\n", + "Let's first register a sample model for use with this notebook.\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", - "For this demonstration, we'll initialize ValidMind in demo mode.\n" + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "id": "apply-template", + "metadata": {}, + "source": [ + "#### Apply documentation template\n", + "\n", + "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "id": "code-snippet", + "metadata": {}, + "source": [ + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -126,18 +177,37 @@ "\n", "import validmind as vm\n", "\n", - "# Note: You need valid API credentials for this to work\n", - "# If you don't have credentials, use the standalone script: test_outlier_detection_standalone.py\n", - "\n", "vm.init(\n", - " api_host=\"...\",\n", - " api_key=\"...\",\n", - " api_secret=\"...\",\n", - " model=\"...\",\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", " document=\"documentation\",\n", ")" ] }, + { + "cell_type": "markdown", + "id": "preview-template", + "metadata": {}, + "source": [ + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, { "cell_type": "markdown", "metadata": { diff --git a/notebooks/code_sharing/regression/regression_unit_metrics.ipynb b/notebooks/code_sharing/regression/regression_unit_metrics.ipynb index 7a8dad352..bcca78635 100644 --- a/notebooks/code_sharing/regression/regression_unit_metrics.ipynb +++ b/notebooks/code_sharing/regression/regression_unit_metrics.ipynb @@ -29,7 +29,8 @@ " - [Register sample model](#toc3_2_1__) \n", " - [Apply documentation template](#toc3_2_2__) \n", " - [Get your code snippet](#toc3_2_3__) \n", - " - [Initialize the Python environment](#toc3_3__) \n", + " - [Preview the documentation template](#toc3_3__) \n", + " - [Initialize the Python environment](#toc3_4__) \n", "- [Load the sample dataset](#toc4__) \n", "- [Build the model](#toc5__) \n", " - [Prepocess the raw dataset](#toc5_1__) \n", @@ -113,12 +114,17 @@ "\n", "### Install the ValidMind Library\n", "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", "To install the library:" ] }, { "cell_type": "code", "execution_count": null, + "id": "install-code", "metadata": {}, "outputs": [], "source": [ @@ -142,7 +148,7 @@ "\n", "#### Register sample model\n", "\n", - "Let's first register a sample model for use with this notebook:\n", + "Let's first register a sample model for use with this notebook.\n", "\n", "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", "\n", @@ -165,7 +171,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -180,15 +186,17 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -206,7 +214,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -216,6 +224,29 @@ "source": [ "\n", "\n", + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", "### Initialize the Python environment\n", "\n", "Next, let's import the necessary libraries and set up your Python environment for data analysis:" @@ -760,7 +791,6 @@ }, { "cell_type": "markdown", - "id": "copyright-46bd32efda67419684f8a989076192f8", "metadata": {}, "source": [ "\n", diff --git a/notebooks/code_sharing/test_configuration_updates_demo.ipynb b/notebooks/code_sharing/test_configuration_updates_demo.ipynb index d5952d8cd..7ee092d88 100644 --- a/notebooks/code_sharing/test_configuration_updates_demo.ipynb +++ b/notebooks/code_sharing/test_configuration_updates_demo.ipynb @@ -1,11 +1,100 @@ { "cells": [ + { + "cell_type": "markdown", + "id": "setup-intro", + "metadata": {}, + "source": [ + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "setup-install", + "metadata": {}, + "source": [ + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" + ] + }, { "cell_type": "code", "execution_count": null, - "metadata": { - "scrolled": false - }, + "id": "install-code", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "id": "setup-initialize", + "metadata": {}, + "source": [ + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "id": "register-model", + "metadata": {}, + "source": [ + "#### Register sample model\n", + "\n", + "Let's first register a sample model for use with this notebook.\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "id": "apply-template", + "metadata": {}, + "source": [ + "#### Apply documentation template\n", + "\n", + "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "id": "code-snippet", + "metadata": {}, + "source": [ + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "initialize-code", + "metadata": {}, "outputs": [], "source": [ "# Load your model identifier credentials from an `.env` file\n", @@ -22,10 +111,20 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", - ")\n", + " document=\"documentation\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "preview-template", + "metadata": {}, + "source": [ + "### Preview the documentation template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "vm.preview_template()" + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] }, { diff --git a/notebooks/how_to/data_and_datasets/dataset_inputs/configure_dataset_features.ipynb b/notebooks/how_to/data_and_datasets/dataset_inputs/configure_dataset_features.ipynb index fc15f4d37..d8f086ea9 100644 --- a/notebooks/how_to/data_and_datasets/dataset_inputs/configure_dataset_features.ipynb +++ b/notebooks/how_to/data_and_datasets/dataset_inputs/configure_dataset_features.ipynb @@ -173,7 +173,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -188,10 +188,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -214,7 +215,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -371,11 +372,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/data_and_datasets/dataset_inputs/load_datasets_predictions.ipynb b/notebooks/how_to/data_and_datasets/dataset_inputs/load_datasets_predictions.ipynb index 5353b326d..fbefddafd 100644 --- a/notebooks/how_to/data_and_datasets/dataset_inputs/load_datasets_predictions.ipynb +++ b/notebooks/how_to/data_and_datasets/dataset_inputs/load_datasets_predictions.ipynb @@ -193,7 +193,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -208,10 +208,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -234,7 +235,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -246,7 +247,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -956,11 +957,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/data_and_datasets/use_dataset_model_objects.ipynb b/notebooks/how_to/data_and_datasets/use_dataset_model_objects.ipynb index 28543ac9a..eb2ece93c 100644 --- a/notebooks/how_to/data_and_datasets/use_dataset_model_objects.ipynb +++ b/notebooks/how_to/data_and_datasets/use_dataset_model_objects.ipynb @@ -44,9 +44,9 @@ " - [Assign predictions to the datasets](#toc5_6__) \n", " - [Using VM Model and Dataset objects as arguments in Custom tests](#toc5_7__) \n", " - [Log the test results](#toc5_8__) \n", - "- [Where to go from here](#toc6__) \n", - " - [Discover more learning resources](#toc6_1__) \n", - "- [Upgrade ValidMind](#toc7__) \n", + "- [In summary](#toc6__) \n", + "- [Discover more learning resources](#toc7__) \n", + "- [Upgrade ValidMind](#toc8__) \n", "\n", ":::\n", "\n", diff --git a/notebooks/how_to/metrics/log_metrics_over_time.ipynb b/notebooks/how_to/metrics/log_metrics_over_time.ipynb index 31f9f1dcf..4722eeed6 100644 --- a/notebooks/how_to/metrics/log_metrics_over_time.ipynb +++ b/notebooks/how_to/metrics/log_metrics_over_time.ipynb @@ -188,7 +188,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Credit Risk Scorecard`.\n", "\n", @@ -203,10 +203,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -229,7 +230,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -858,11 +859,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/metrics/run_unit_metrics.ipynb b/notebooks/how_to/metrics/run_unit_metrics.ipynb index 3112b8a91..1b8dc8ce6 100644 --- a/notebooks/how_to/metrics/run_unit_metrics.ipynb +++ b/notebooks/how_to/metrics/run_unit_metrics.ipynb @@ -181,7 +181,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -196,10 +196,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -222,7 +223,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -695,11 +696,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/scoring/assign_scores_complete_tutorial.ipynb b/notebooks/how_to/scoring/assign_scores_complete_tutorial.ipynb index 5dad199c0..586817850 100644 --- a/notebooks/how_to/scoring/assign_scores_complete_tutorial.ipynb +++ b/notebooks/how_to/scoring/assign_scores_complete_tutorial.ipynb @@ -183,7 +183,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -198,10 +198,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -224,7 +225,7 @@ " api_key=\"...\",\n", " api_secret=\"...\",\n", " model=\"...\",\n", - " document=\"documentation\",\n", + " # document=\"documentation\",\n", ")\n" ] }, @@ -714,11 +715,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "The scores you've assigned using `assign_scores()` become part of your model's documentation and can be used in ongoing monitoring workflows. You can view these metrics over time, set up alerts for performance drift, and compare models systematically. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb b/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb index 35b75b457..8e4be82ac 100644 --- a/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb +++ b/notebooks/how_to/tests/custom_tests/implement_custom_tests.ipynb @@ -185,7 +185,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -200,10 +200,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -995,11 +996,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/tests/custom_tests/integrate_external_test_providers.ipynb b/notebooks/how_to/tests/custom_tests/integrate_external_test_providers.ipynb index f2411481b..eaff3a579 100644 --- a/notebooks/how_to/tests/custom_tests/integrate_external_test_providers.ipynb +++ b/notebooks/how_to/tests/custom_tests/integrate_external_test_providers.ipynb @@ -198,7 +198,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -213,10 +213,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -239,7 +240,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -895,11 +896,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/tests/run_tests/1_run_dataset_based_tests.ipynb b/notebooks/how_to/tests/run_tests/1_run_dataset_based_tests.ipynb index 3f5ea875a..150a488cf 100644 --- a/notebooks/how_to/tests/run_tests/1_run_dataset_based_tests.ipynb +++ b/notebooks/how_to/tests/run_tests/1_run_dataset_based_tests.ipynb @@ -211,7 +211,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -227,10 +227,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -254,7 +255,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -267,7 +268,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -635,7 +636,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", "3. Locate the Data Preparation section and click on **2.1. Data Description** to expand that section.\n", "\n", diff --git a/notebooks/how_to/tests/run_tests/2_run_comparison_tests.ipynb b/notebooks/how_to/tests/run_tests/2_run_comparison_tests.ipynb index 689cfae9b..3cdc64cff 100644 --- a/notebooks/how_to/tests/run_tests/2_run_comparison_tests.ipynb +++ b/notebooks/how_to/tests/run_tests/2_run_comparison_tests.ipynb @@ -189,7 +189,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -204,10 +204,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -230,6 +231,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -664,7 +666,7 @@ "\n", "You will log these results in the `model_evaluation` section of the documentation. After logging the results, you can view the updated documentation in the ValidMind Platform following these steps:\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "1. In the ValidMind Platform, click **Development** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", "\n", "2. Expand the **3.2. Model Evaluation** section.\n", "\n", diff --git a/notebooks/how_to/tests/run_tests/configure_tests/customize_test_result_descriptions.ipynb b/notebooks/how_to/tests/run_tests/configure_tests/customize_test_result_descriptions.ipynb index a6f446934..68c659a45 100644 --- a/notebooks/how_to/tests/run_tests/configure_tests/customize_test_result_descriptions.ipynb +++ b/notebooks/how_to/tests/run_tests/configure_tests/customize_test_result_descriptions.ipynb @@ -136,7 +136,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -151,10 +151,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -177,7 +178,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, diff --git a/notebooks/how_to/tests/run_tests/configure_tests/enable_pii_detection.ipynb b/notebooks/how_to/tests/run_tests/configure_tests/enable_pii_detection.ipynb index 45de1ac66..e0ae28fff 100644 --- a/notebooks/how_to/tests/run_tests/configure_tests/enable_pii_detection.ipynb +++ b/notebooks/how_to/tests/run_tests/configure_tests/enable_pii_detection.ipynb @@ -193,15 +193,11 @@ "\n", "#### Get your code snippet\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", - "\n", - "3. Enter the model details and click **Continue**. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", - "\n", - "4. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", - "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -225,7 +221,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -556,7 +552,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier.\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", "3. Click on any section heading to expand that section to add a new test-driven block ([Need more help?](https://docs.validmind.ai/developer/model-documentation/work-with-test-results.html)).\n", "\n", diff --git a/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.ipynb b/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.ipynb index c01a11ace..1904b90cc 100644 --- a/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.ipynb +++ b/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.ipynb @@ -185,7 +185,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -200,10 +200,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -226,7 +227,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -238,7 +239,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -471,11 +472,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/tests/run_tests/documentation_tests/document_multiple_results_for_the_same_test.ipynb b/notebooks/how_to/tests/run_tests/documentation_tests/document_multiple_results_for_the_same_test.ipynb index 395a1a57a..0ba0f67f7 100644 --- a/notebooks/how_to/tests/run_tests/documentation_tests/document_multiple_results_for_the_same_test.ipynb +++ b/notebooks/how_to/tests/run_tests/documentation_tests/document_multiple_results_for_the_same_test.ipynb @@ -190,7 +190,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -205,10 +205,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -231,6 +232,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -304,7 +306,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -517,9 +519,9 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", "3. Expand the **2. Data Preparation** section and take a look around.\n", "\n", diff --git a/notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_sections.ipynb b/notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_sections.ipynb index cfdce6bdb..e5ed5728e 100644 --- a/notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_sections.ipynb +++ b/notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_sections.ipynb @@ -183,7 +183,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -198,10 +198,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -224,6 +225,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -246,7 +248,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -493,11 +495,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_tests_with_config.ipynb b/notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_tests_with_config.ipynb index 44b7666c5..fd5b82017 100644 --- a/notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_tests_with_config.ipynb +++ b/notebooks/how_to/tests/run_tests/documentation_tests/run_documentation_tests_with_config.ipynb @@ -187,7 +187,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -202,10 +202,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -228,6 +229,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -239,7 +241,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -618,11 +620,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/quickstart/quickstart_model_documentation.ipynb b/notebooks/quickstart/quickstart_model_documentation.ipynb index b2565f20d..32678a32c 100644 --- a/notebooks/quickstart/quickstart_model_documentation.ipynb +++ b/notebooks/quickstart/quickstart_model_documentation.ipynb @@ -242,7 +242,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -258,10 +258,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -281,10 +282,10 @@ "import validmind as vm\n", "\n", "vm.init(\n", - " api_host=\"https://api.validmind.ai/api/v1/tracking\",\n", - " api_key=\"\",\n", - " api_secret=\"\",\n", - " model=\"\",\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", " document=\"documentation\",\n", ")" ] @@ -335,7 +336,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -365,7 +366,7 @@ "\n", "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this notebook.\n", "\n", - "3. Click **Documentation** under Documents for your model and note how the structure of the documentation matches our preview above." + "3. Click **Development** under Documents for your model and note how the structure of the documentation matches our preview above." ] }, { @@ -778,9 +779,9 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - " What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" ] }, { diff --git a/notebooks/quickstart/quickstart_model_validation.ipynb b/notebooks/quickstart/quickstart_model_validation.ipynb index a3f6b1a25..640e64015 100644 --- a/notebooks/quickstart/quickstart_model_validation.ipynb +++ b/notebooks/quickstart/quickstart_model_validation.ipynb @@ -239,9 +239,9 @@ "\n", "#### Apply validation report template\n", "\n", - "Once you've registered your model, let's select a validation report template. A template predefines sections for your report and provides a general outline to follow, making the validation process much easier.\n", + "Next, let's select a validation report template. A template predefines sections for your report and provides a general outline to follow, making the validation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Validation**.\n", "\n", "2. Under **TEMPLATE**, select `Generic Validation Report`.\n", "\n", @@ -281,9 +281,7 @@ "source": [ "\n", "\n", - "### Initialize the ValidMind Library\n", - "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your validation environment. You initialize the ValidMind Library with this code snippet, which ensures that your test results are uploaded to the correct model when you run the notebook." + "### Initialize the ValidMind Library" ] }, { @@ -295,8 +293,11 @@ "\n", "#### Get your code snippet\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Validation` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -320,7 +321,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"validation-report\",\n", + " document=\"validation-report\",\n", ")" ] }, @@ -368,7 +369,7 @@ "\n", "### Preview the validation report template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for model validation.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for model validation. A template predefines sections for your validation report and provides a general outline to follow, making the validation process much easier.\n", "\n", "You will attach evidence to this template in the form of risk assessment notes, artifacts, and test results later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library:" ] @@ -398,7 +399,7 @@ "\n", "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this notebook.\n", "\n", - "3. Click **Validation Report** under Documents for your model and note:\n", + "3. Click **Validation** under Documents for your model and note:\n", "\n", " - [x] The risk assessment compliance summary at the top of the report (screenshot below)\n", " - [x] How the structure of the validation report reflects the previewed template\n", @@ -1097,11 +1098,13 @@ "\n", "### Work with your validation report\n", "\n", - "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "Now that you've logged all your test results and verified the work done by the model development team, head to the ValidMind Platform to wrap up your validation report:\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n", "\n", - "2. In the left sidebar that appears for your model, click **Validation Report** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", "\n", - " From here, you can link test results as evidence, add model artifacts, assess compliance, and submit your validation report for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html)" + "Include your logged test results as evidence, create risk assessment notes, add artifacts, and assess compliance, then submit your report for review when it's ready. **Learn more:** [Preparing validation reports](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html)" ] }, { diff --git a/notebooks/templates/README.md b/notebooks/templates/README.md index e90a1c39d..f2d53cd83 100644 --- a/notebooks/templates/README.md +++ b/notebooks/templates/README.md @@ -12,9 +12,9 @@ make notebook The template generation script/notebook draws from the following mini-templates, should you need to revise them or grab the information from them manually: -- [`_about-validmind.ipynb`](_about-validmind.ipynb): Conceptual overview of ValidMind & prerequisites. -- [`_install-initialize-validmind.ipynb`](_install-initialize-validmind.ipynb): ValidMind Library installation & initialization instructions. -- [`_next-steps.ipynb`](_next-steps.ipynb): Directions to review the generated documentation within the ValidMind Platform & additional learning resources. +- [`about-validmind/`](about-validmind/): Conceptual overview of ValidMind & prerequisites. +- [`install-validmind/`](install-validmind/): ValidMind Library installation & initialization instructions. +- [`next-steps/`](next-steps/): Directions to review the generated documentation within the ValidMind Platform & additional learning resources. - [`_upgrade-validmind.ipynb`](_upgrade-validmind.ipynb): Instructions for comparing & upgrading versions of the ValidMind Library. - [`_copyright.ipynb`](_copyright.ipynb): Copyright and licensing info — mandatory at the bottom of every notebook owned by ValidMind. diff --git a/notebooks/templates/_about-validmind.ipynb b/notebooks/templates/about-validmind/_about-validmind-developers.ipynb similarity index 100% rename from notebooks/templates/_about-validmind.ipynb rename to notebooks/templates/about-validmind/_about-validmind-developers.ipynb diff --git a/notebooks/templates/about-validmind/_about-validmind-monitoring.ipynb b/notebooks/templates/about-validmind/_about-validmind-monitoring.ipynb new file mode 100644 index 000000000..bb3decbc8 --- /dev/null +++ b/notebooks/templates/about-validmind/_about-validmind-monitoring.ipynb @@ -0,0 +1,80 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "about-intro", + "metadata": {}, + "source": [ + "## About ValidMind\n", + "\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models. \n", + "\n", + "You use the ValidMind Library to automate documentation and validation tests, and then use the ValidMind Platform to collaborate on model documentation. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model validators." + ] + }, + { + "cell_type": "markdown", + "id": "about-begin", + "metadata": {}, + "source": [ + "### Before you begin\n", + "\n", + "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", + "\n", + "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)." + ] + }, + { + "cell_type": "markdown", + "id": "about-signup", + "metadata": {}, + "source": [ + "### New to ValidMind?\n", + "\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", + "\n", + "
For access to all features available in this notebook, you'll need access to a ValidMind account.\n", + "

\n", + "Register with ValidMind
" + ] + }, + { + "cell_type": "markdown", + "id": "about-concepts", + "metadata": {}, + "source": [ + "### Key concepts\n", + "\n", + "**Model documentation**: A structured and detailed record pertaining to a model, encompassing key components such as its underlying assumptions, methodologies, data sources, inputs, performance metrics, evaluations, limitations, and intended uses. It serves to ensure transparency, adherence to regulatory requirements, and a clear understanding of potential risks associated with the model’s application.\n", + "\n", + "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "\n", + "**Model monitoring report**: A comprehensive and structured record of a production model, including key elements such as data sources, inputs, performance metrics, and periodic evaluations. This documentation ensures transparency and visibility of the model's performance in the production environment.\n", + "\n", + "**Monitoring report template**: Similar to documentation template, The monitoring report template functions as a test suite and lays out the structure of model monitoring, segmented into various sections and sub-sections. Monitoring report templates define the structure of your model monitoring report, specifying the tests that should be run, and how the results should be displayed.\n", + "\n", + "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", + "\n", + "**Custom tests**: Custom tests are functions that you define to evaluate your model or dataset. These functions can be registered via the ValidMind Library to be used with the ValidMind Platform.\n", + "\n", + "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n", + "\n", + " - **model**: A single model that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n", + " - **dataset**: Single dataset that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n", + " - **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom test.\n", + " - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom test. See this [example](https://docs.validmind.ai/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.html) for more information.\n", + "\n", + "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a test, customize its behavior, or provide additional context.\n", + "\n", + "**Outputs**: Custom tests can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/templates/about-validmind/_about-validmind-validators.ipynb b/notebooks/templates/about-validmind/_about-validmind-validators.ipynb new file mode 100644 index 000000000..263d3ebf3 --- /dev/null +++ b/notebooks/templates/about-validmind/_about-validmind-validators.ipynb @@ -0,0 +1,78 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "about-intro", + "metadata": {}, + "source": [ + "## About ValidMind\n", + "\n", + "ValidMind is a suite of tools for managing model risk, including risk associated with AI and statistical models.\n", + "\n", + "You use the ValidMind Library to automate comparison and other validation tests, and then use the ValidMind Platform to submit compliance assessments of champion models via comprehensive validation reports. Together, these products simplify model risk management, facilitate compliance with regulations and institutional standards, and enhance collaboration between yourself and model developers." + ] + }, + { + "cell_type": "markdown", + "id": "about-begin", + "metadata": {}, + "source": [ + "### Before you begin\n", + "\n", + "This notebook assumes you have basic familiarity with Python, including an understanding of how functions work. If you are new to Python, you can still run the notebook but we recommend further familiarizing yourself with the language. \n", + "\n", + "If you encounter errors due to missing modules in your Python environment, install the modules with `pip install`, and then re-run the notebook. For more help, refer to [Installing Python Modules](https://docs.python.org/3/installing/index.html)." + ] + }, + { + "cell_type": "markdown", + "id": "about-signup", + "metadata": {}, + "source": [ + "### New to ValidMind?\n", + "\n", + "If you haven't already seen our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html), we recommend you begin by exploring the available resources in this section. There, you can learn more about documenting models and running tests, as well as find code samples and our Python Library API reference.\n", + "\n", + "
For access to all features available in this notebook, you'll need access to a ValidMind account.\n", + "

\n", + "Register with ValidMind
" + ] + }, + { + "cell_type": "markdown", + "id": "about-concepts", + "metadata": {}, + "source": [ + "### Key concepts\n", + "\n", + "**Validation report**: A comprehensive and structured assessment of a model’s development and performance, focusing on verifying its integrity, appropriateness, and alignment with its intended use. It includes analyses of model assumptions, data quality, performance metrics, outcomes of testing procedures, and risk considerations. The validation report supports transparency, regulatory compliance, and informed decision-making by documenting the validator’s independent review and conclusions.\n", + "\n", + "**Validation report template**: Serves as a standardized framework for conducting and documenting model validation activities. It outlines the required sections, recommended analyses, and expected validation tests, ensuring consistency and completeness across validation reports. The template helps guide validators through a systematic review process while promoting comparability and traceability of validation outcomes.\n", + "\n", + "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets.\n", + "\n", + "**Metrics**: A subset of tests that do not have thresholds. In the context of this notebook, metrics and tests can be thought of as interchangeable concepts.\n", + "\n", + "**Custom metrics**: Custom metrics are functions that you define to evaluate your model or dataset. These functions can be registered with the ValidMind Library to be used in the ValidMind Platform.\n", + "\n", + "**Inputs**: Objects to be evaluated and documented in the ValidMind Library. They can be any of the following:\n", + "\n", + " - **model**: A single model that has been initialized in ValidMind with [`vm.init_model()`](https://docs.validmind.ai/validmind/validmind.html#init_model).\n", + " - **dataset**: Single dataset that has been initialized in ValidMind with [`vm.init_dataset()`](https://docs.validmind.ai/validmind/validmind.html#init_dataset).\n", + " - **models**: A list of ValidMind models - usually this is used when you want to compare multiple models in your custom metric.\n", + " - **datasets**: A list of ValidMind datasets - usually this is used when you want to compare multiple datasets in your custom metric. (Learn more: [Run tests with multiple datasets](https://docs.validmind.ai/notebooks/how_to/tests/run_tests/configure_tests/run_tests_that_require_multiple_datasets.html))\n", + "\n", + "**Parameters**: Additional arguments that can be passed when running a ValidMind test, used to pass additional information to a metric, customize its behavior, or provide additional context.\n", + "\n", + "**Outputs**: Custom metrics can return elements like tables or plots. Tables may be a list of dictionaries (each representing a row) or a pandas DataFrame. Plots may be matplotlib or plotly figures." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/templates/e2e-notebook.ipynb b/notebooks/templates/e2e-notebook.ipynb index f19fc436e..e165dd628 100644 --- a/notebooks/templates/e2e-notebook.ipynb +++ b/notebooks/templates/e2e-notebook.ipynb @@ -13,20 +13,24 @@ "cell_type": "markdown", "metadata": {}, "source": [ + "::: {.content-hidden when-format=\"html\"}\n", "## Contents \n", "- [Setting up](#toc1__) \n", " - [Import the notebook creation script](#toc1_1__) \n", " - [Designate the filename](#toc1_2__) \n", " - [Set the title](#toc1_3__) \n", + " - [Set a role / document type](#toc1_4__) \n", "- [Single-sourcing from mini-templates](#toc2__) \n", " - [Insert information about ValidMind](#toc2_1__) \n", " - [Insert setup instructions](#toc2_2__) \n", " - [Insert next steps](#toc2_3__) \n", " - [Insert upgrade information](#toc2_4__) \n", + " - [Insert copyright information](#toc2_5__) \n", "- [Next steps](#toc3__) \n", " - [Show files to commit](#toc3_1__) \n", " - [Wrap it up](#toc3_2__) \n", "\n", + ":::\n", "\n", diff --git a/notebooks/templates/e2e_template.py b/notebooks/templates/e2e_template.py index feaf16268..77fd9e5ad 100644 --- a/notebooks/templates/e2e_template.py +++ b/notebooks/templates/e2e_template.py @@ -8,6 +8,20 @@ import platform from typing import Callable, Dict, Iterable, Optional, Set, Tuple +DOCUMENT_TYPES = { + "1": "development", + "2": "validation", + "3": "monitoring", +} + +INSTALL_CHOICE = { + "1": "install_only", + "2": "install_and_initialize", +} + +_selected_document: Optional[str] = None +_selected_install: Optional[str] = None + def ensure_ids(notebook): """Ensure every cell in the notebook has a unique id.""" for cell in notebook.cells: @@ -273,9 +287,52 @@ def set_title(filepath): except Exception as e: print(f"Error updating notebook: {e}") +def select_document(): + """Requests the user to select a role/document type for template application used by add_about, add_install, and next_steps. + + Stores the selection in the module-level `_selected_document` variable for + downstream functions to reference. + + Returns: + The selected document type string, or None if the selection was invalid. + """ + global _selected_document + + choice = input( + "Select a role/document type — " + "[1: Developer/Development], " + "[2: Validator/Validation], " + "[3: Developer/Monitoring:] " + ).strip() + + if choice not in DOCUMENT_TYPES: + print(f"Invalid selection: '{choice}'. Please enter 1, 2, or 3.") + _selected_document = None + return None + + _selected_document = DOCUMENT_TYPES[choice] + labels = { + "development": "Developer / Development", + "validation": "Validator / Validation", + "monitoring": "Developer / Monitoring", + } + print(f"Selected: {labels[_selected_document]}") + return _selected_document + def add_about(filepath): - """Appends the contents of '_about-validmind.ipynb' to the specified notebook if the user agrees.""" - source_notebook_path = os.path.join(os.path.dirname(__file__), "_about-validmind.ipynb") + """Appends an about-validmind notebook based on the document type selected via select_document().""" + about_files = { + "development": "about-validmind/_about-validmind-developers.ipynb", + "validation": "about-validmind/_about-validmind-validators.ipynb", + "monitoring": "about-validmind/_about-validmind-monitoring.ipynb", + } + + if _selected_document is None: + print("No document type selected. Run select_document() first.") + return + + relative_path = about_files[_selected_document] + source_notebook_path = os.path.join(os.path.dirname(__file__), relative_path) if not os.path.exists(source_notebook_path): print(f"Source notebook '{source_notebook_path}' does not exist") @@ -283,7 +340,7 @@ def add_about(filepath): user_input = input("Do you want to include information about ValidMind? (yes/no): ").strip().lower() if user_input not in ("yes", "y"): - print("Skipping appending '_about-validmind.ipynb'") + print(f"Skipping appending '{relative_path}'") return try: @@ -307,7 +364,7 @@ def add_about(filepath): try: with open(filepath, "w") as target_file: nbformat.write(target_notebook, target_file) - print(f"'_about-validmind.ipynb' appended to '{filepath}'") + print(f"'{relative_path}' appended to '{filepath}'") except Exception as e: print(f"Error appending notebooks: {e}") @@ -315,9 +372,9 @@ def add_about(filepath): # Prefix -> allowed cell types to remove (None/empty => any type) DEFAULT_SKIP_PREFIX_RULES: Dict[str, Optional[Iterable[str]]] = { - "install-template": {"markdown"}, - "install-preview": {"markdown"}, - "install-preview-template": {"code"}, + "apply-template": {"markdown"}, + "preview-template": {"markdown"}, + "template-code": {"code"}, } @@ -487,18 +544,44 @@ def replace_variables( except Exception as e: print_func(f"Error replacing variables in file: {e}") -def add_install(filepath): - """Appends the contents of '_install-initialize-validmind.ipynb' to the specified notebook if the user agrees.""" - source_notebook_path = os.path.join(os.path.dirname(__file__), "_install-initialize-validmind.ipynb") +def select_install(): + """Requests the user to select an installation type template used by add_install. + + Stores the selection in the module-level `_selected_install` variable for + downstream functions to reference. + + Returns: + The selected install choice string, or None if the selection was invalid. + """ + global _selected_install + + choice = input( + "Select an installation option — " + "[1: Installation Only], " + "[2: Both Install & Initialize:] " + ).strip() + + if choice not in INSTALL_CHOICE: + print(f"Invalid selection: '{choice}'. Please enter 1 or 2.") + _selected_install = None + return None + + _selected_install = INSTALL_CHOICE[choice] + labels = { + "install_only": "Installation Only", + "install_and_initialize": "Both Install & Initialize", + } + print(f"Selected: {labels[_selected_install]}") + return _selected_install + + +def _append_notebook(filepath, relative_path): + """Helper to append a source notebook's cells to the target notebook.""" + source_notebook_path = os.path.join(os.path.dirname(__file__), relative_path) if not os.path.exists(source_notebook_path): print(f"Source notebook '{source_notebook_path}' does not exist") - return - - user_input = input("Do you want to include installation and initialization instructions? (yes/no): ").strip().lower() - if user_input not in ("yes", "y"): - print("Skipping appending '_install-initialize-validmind.ipynb'") - return + return False try: with open(filepath, "r") as target_file: @@ -508,7 +591,7 @@ def add_install(filepath): source_notebook = nbformat.read(source_file, as_version=4) except Exception as e: print(f"Error reading notebooks: {e}") - return + return False for cell in source_notebook.cells: original_id = cell.get("id", f"cell-{uuid.uuid4()}") @@ -521,49 +604,59 @@ def add_install(filepath): try: with open(filepath, "w") as target_file: nbformat.write(target_notebook, target_file) - print(f"'_install-initialize-validmind.ipynb' appended to '{filepath}'") + print(f"'{relative_path}' appended to '{filepath}'") except Exception as e: print(f"Error appending notebooks: {e}") + return False - replace_variables(filepath) + return True -def next_steps(filepath): - """Appends the contents of '_next-steps.ipynb' to the specified notebook if the user agrees.""" - source_notebook_path = os.path.join(os.path.dirname(__file__), "_next-steps.ipynb") - if not os.path.exists(source_notebook_path): - print(f"Source notebook '{source_notebook_path}' does not exist") +def add_install(filepath): + """Appends install notebook(s) based on the install choice and document type selections.""" + if _selected_install is None: + print("No install choice selected. Run select_install() first.") return - user_input = input("Do you want to include next steps? (yes/no): ").strip().lower() - if user_input not in ("yes", "y"): - print("Skipping appending '_next-steps.ipynb'") + if _selected_install == "install_only": + relative_path = "install-validmind/_install-only.ipynb" + _append_notebook(filepath, relative_path) return - try: - with open(filepath, "r") as target_file: - target_notebook = nbformat.read(target_file, as_version=4) + install_files = { + "development": "install-validmind/_install-initialize-with-development.ipynb", + "validation": "install-validmind/_install-initialize-with-validation.ipynb", + "monitoring": "install-validmind/_install-initialize-with-monitoring.ipynb", + } - with open(source_notebook_path, "r") as source_file: - source_notebook = nbformat.read(source_file, as_version=4) - except Exception as e: - print(f"Error reading notebooks: {e}") + if _selected_document is None: + print("No document type selected. Run select_document() first.") return - for cell in source_notebook.cells: - original_id = cell.get("id", f"cell-{uuid.uuid4()}") - new_id = f"{original_id}-{uuid.uuid4()}" - cell["id"] = new_id + relative_path = install_files[_selected_document] + _append_notebook(filepath, relative_path) + replace_variables(filepath) - target_notebook.cells.extend(source_notebook.cells) - target_notebook = ensure_ids(target_notebook) +def next_steps(filepath): + """Appends a next-steps notebook based on the document type selected via select_document().""" + next_steps_files = { + "development": "next-steps/_next-steps-development.ipynb", + "validation": "next-steps/_next-steps-validation.ipynb", + "monitoring": "next-steps/_next-steps-monitoring.ipynb", + } - try: - with open(filepath, "w") as target_file: - nbformat.write(target_notebook, target_file) - print(f"'_next-steps.ipynb' appended to '{filepath}'") - except Exception as e: - print(f"Error appending notebooks: {e}") + if _selected_document is None: + print("No document type selected. Run select_document() first.") + return + + relative_path = next_steps_files[_selected_document] + + user_input = input("Do you want to include next steps? (yes/no): ").strip().lower() + if user_input not in ("yes", "y"): + print(f"Skipping appending '{relative_path}'") + return + + _append_notebook(filepath, relative_path) def add_upgrade(filepath): """Appends the contents of '_upgrade-validmind.ipynb' to the specified notebook if the user agrees.""" @@ -650,7 +743,9 @@ def add_copyright(filepath: str) -> None: filepath = create_notebook() if filepath: set_title(filepath) + select_document() add_about(filepath) + select_install() add_install(filepath) next_steps(filepath) add_upgrade(filepath) diff --git a/notebooks/templates/_install-initialize-validmind.ipynb b/notebooks/templates/install-validmind/_install-initialize-with-development.ipynb similarity index 80% rename from notebooks/templates/_install-initialize-validmind.ipynb rename to notebooks/templates/install-validmind/_install-initialize-with-development.ipynb index 29d9be91e..b6b9cc74c 100644 --- a/notebooks/templates/_install-initialize-validmind.ipynb +++ b/notebooks/templates/install-validmind/_install-initialize-with-development.ipynb @@ -2,6 +2,7 @@ "cells": [ { "cell_type": "markdown", + "id": "setup-intro", "metadata": {}, "source": [ "## Setting up" @@ -9,6 +10,7 @@ }, { "cell_type": "markdown", + "id": "setup-install", "metadata": {}, "source": [ "### Install the ValidMind Library\n", @@ -23,6 +25,7 @@ { "cell_type": "code", "execution_count": null, + "id": "install-code", "metadata": {}, "outputs": [], "source": [ @@ -31,6 +34,7 @@ }, { "cell_type": "markdown", + "id": "setup-initialize", "metadata": {}, "source": [ "### Initialize the ValidMind Library" @@ -38,6 +42,7 @@ }, { "cell_type": "markdown", + "id": "register-model", "metadata": {}, "source": [ "#### Register sample model\n", @@ -57,13 +62,14 @@ }, { "cell_type": "markdown", + "id": "apply-template", "metadata": {}, "source": [ "#### Apply documentation template\n", "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `{template}`.\n", "\n", @@ -72,19 +78,22 @@ }, { "cell_type": "markdown", + "id": "code-snippet", "metadata": {}, "source": [ "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { "cell_type": "code", "execution_count": null, + "id": "initialize-code", "metadata": {}, "outputs": [], "source": [ @@ -102,17 +111,18 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, { "cell_type": "markdown", + "id": "preview-template", "metadata": {}, "source": [ "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -120,6 +130,7 @@ { "cell_type": "code", "execution_count": null, + "id": "template-code", "metadata": {}, "outputs": [], "source": [ diff --git a/notebooks/templates/install-validmind/_install-initialize-with-monitoring.ipynb b/notebooks/templates/install-validmind/_install-initialize-with-monitoring.ipynb new file mode 100644 index 000000000..ad33eef97 --- /dev/null +++ b/notebooks/templates/install-validmind/_install-initialize-with-monitoring.ipynb @@ -0,0 +1,148 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "setup-intro", + "metadata": {}, + "source": [ + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "setup-install", + "metadata": {}, + "source": [ + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-code", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "id": "setup-initialize", + "metadata": {}, + "source": [ + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "id": "register-model", + "metadata": {}, + "source": [ + "#### Register sample model\n", + "\n", + "Let's first register a sample model for use with this notebook.\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "id": "apply-template", + "metadata": {}, + "source": [ + "#### Apply monitoring report template\n", + "\n", + "Once you've registered your model, let's select a monitoring report template. A template predefines sections for your monitoring report and provides a general outline to follow, making the monitoring process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Monitoring**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "id": "code-snippet", + "metadata": {}, + "source": [ + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Monitoring` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "initialize-code", + "metadata": {}, + "outputs": [], + "source": [ + "# Load your model identifier credentials from an `.env` file\n", + "\n", + "%load_ext dotenv\n", + "%dotenv .env\n", + "\n", + "# Or replace with your code snippet\n", + "\n", + "import validmind as vm\n", + "\n", + "vm.init(\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", + " document=\"monitoring\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "preview-template", + "metadata": {}, + "source": [ + "### Preview the monitoring report template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your monitoring report and provides a general outline to follow, making the monitoring process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/templates/install-validmind/_install-initialize-with-validation.ipynb b/notebooks/templates/install-validmind/_install-initialize-with-validation.ipynb new file mode 100644 index 000000000..cfa93dfc3 --- /dev/null +++ b/notebooks/templates/install-validmind/_install-initialize-with-validation.ipynb @@ -0,0 +1,178 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "setup-intro", + "metadata": {}, + "source": [ + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "register-model", + "metadata": {}, + "source": [ + "### Register a sample model\n", + "\n", + "In a usual model lifecycle, a champion model will have been independently registered in your model inventory and submitted to you for validation by your model development team as part of the effective challenge process. (**Learn more:** [Submit for approval](https://docs.validmind.ai/guide/model-documentation/submit-for-approval.html))\n", + "\n", + "For this notebook, we'll have you register a dummy model in the ValidMind Platform inventory and assign yourself as the validator to familiarize you with the ValidMind interface and circumvent the need for an existing model:\n", + "\n", + "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", + "\n", + "2. In the left sidebar, navigate to **Inventory** and click **+ Register Model**.\n", + "\n", + "3. Enter the model details and click **Next >** to continue to assignment of model stakeholders. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/register-models-in-inventory.html))\n", + "\n", + "4. Select your own name under the **MODEL OWNER** drop-down — don’t worry, we’ll adjust these permissions next for validation.\n", + "\n", + "5. Click **Register Model** to add the model to your inventory." + ] + }, + { + "cell_type": "markdown", + "id": "validator-credentials", + "metadata": {}, + "source": [ + "#### Assign validator credentials\n", + "\n", + "In order to log tests as a validator instead of as a developer, on the model details page that appears after you've successfully registered your sample model:\n", + "\n", + "1. Remove yourself as a model owner: \n", + "\n", + " - Click on the **OWNERS** tile.\n", + " - Click the **x** next to your name to remove yourself from that model's role.\n", + " - Click **Save** to apply your changes to that role.\n", + "\n", + "2. Remove yourself as a developer: \n", + "\n", + " - Click on the **DEVELOPERS** tile.\n", + " - Click the **x** next to your name to remove yourself from that model's role.\n", + " - Click **Save** to apply your changes to that role.\n", + "\n", + "3. Add yourself as a validator: \n", + "\n", + " - Click on the **VALIDATORS** tile.\n", + " - Select your name from the drop-down menu.\n", + " - Click **Save** to apply your changes to that role." + ] + }, + { + "cell_type": "markdown", + "id": "apply-template", + "metadata": {}, + "source": [ + "#### Apply validation report template\n", + "\n", + "Next, let's select a validation report template. A template predefines sections for your report and provides a general outline to follow, making the validation process much easier.\n", + "\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Validation**.\n", + "\n", + "2. Under **TEMPLATE**, select `{template}`.\n", + "\n", + "3. Click **Use Template** to apply the template." + ] + }, + { + "cell_type": "markdown", + "id": "setup-install", + "metadata": {}, + "source": [ + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "install-code", + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + }, + { + "cell_type": "markdown", + "id": "setup-initialize", + "metadata": {}, + "source": [ + "### Initialize the ValidMind Library" + ] + }, + { + "cell_type": "markdown", + "id": "code-snippet", + "metadata": {}, + "source": [ + "#### Get your code snippet\n", + "\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", + "\n", + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Validation` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "initialize-code", + "metadata": {}, + "outputs": [], + "source": [ + "# Load your model identifier credentials from an `.env` file\n", + "\n", + "%load_ext dotenv\n", + "%dotenv .env\n", + "\n", + "# Or replace with your code snippet\n", + "\n", + "import validmind as vm\n", + "\n", + "vm.init(\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", + " document=\"validation-report\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "preview-template", + "metadata": {}, + "source": [ + "### Preview the validation report template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for model validation. A template predefines sections for your validation report and provides a general outline to follow, making the validation process much easier.\n", + "\n", + "You will attach evidence to this template in the form of risk assessment notes, artifacts, and test results later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "template-code", + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/templates/install-validmind/_install-only.ipynb b/notebooks/templates/install-validmind/_install-only.ipynb new file mode 100644 index 000000000..3d5bf8ea0 --- /dev/null +++ b/notebooks/templates/install-validmind/_install-only.ipynb @@ -0,0 +1,43 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "setup-intro", + "metadata": {}, + "source": [ + "## Setting up" + ] + }, + { + "cell_type": "markdown", + "id": "setup-install", + "metadata": {}, + "source": [ + "### Install the ValidMind Library\n", + "\n", + "
Recommended Python versions\n", + "

\n", + "Python 3.8 <= x <= 3.11
\n", + "\n", + "To install the library:" + ] + }, + { + "cell_type": "code", + "id": "install-code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%pip install -q validmind" + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/templates/_next-steps.ipynb b/notebooks/templates/next-steps/_next-steps-development.ipynb similarity index 96% rename from notebooks/templates/_next-steps.ipynb rename to notebooks/templates/next-steps/_next-steps-development.ipynb index 320a3a66b..203b4c587 100644 --- a/notebooks/templates/_next-steps.ipynb +++ b/notebooks/templates/next-steps/_next-steps-development.ipynb @@ -12,14 +12,14 @@ }, { "cell_type": "markdown", - "id": "next-docs", + "id": "next-document", "metadata": {}, "source": [ "### Work with your model documentation\n", "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)" ] diff --git a/notebooks/templates/next-steps/_next-steps-monitoring.ipynb b/notebooks/templates/next-steps/_next-steps-monitoring.ipynb new file mode 100644 index 000000000..fe8001c03 --- /dev/null +++ b/notebooks/templates/next-steps/_next-steps-monitoring.ipynb @@ -0,0 +1,51 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "next-steps", + "metadata": {}, + "source": [ + "## Next steps\n", + "\n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your monitoring report." + ] + }, + { + "cell_type": "markdown", + "id": "next-document", + "metadata": {}, + "source": [ + "### Work with your monitoring report\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "\n", + "2. In the left sidebar that appears for your model, click **Monitoring** under Documents.\n", + "\n", + "What you see is the full draft of your monitoring report in a more easily consumable version. From here, you can make qualitative edits to monitoring reports, view guidelines, review monitoring results, and submit your monitoring report for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/monitoring/ongoing-monitoring.html)" + ] + }, + { + "cell_type": "markdown", + "id": "next-resources", + "metadata": {}, + "source": [ + "### Discover more learning resources\n", + "\n", + "We offer many interactive notebooks to help you automate testing, documenting, monitoring, and more:\n", + "\n", + "- [Run tests & test suites](https://docs.validmind.ai/developer/how-to/testing-overview.html)\n", + "- [Use ValidMind Library features](https://docs.validmind.ai/developer/how-to/feature-overview.html)\n", + "- [Code samples by use case](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", + "\n", + "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/templates/next-steps/_next-steps-validation.ipynb b/notebooks/templates/next-steps/_next-steps-validation.ipynb new file mode 100644 index 000000000..004176d27 --- /dev/null +++ b/notebooks/templates/next-steps/_next-steps-validation.ipynb @@ -0,0 +1,53 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "next-steps", + "metadata": {}, + "source": [ + "## Next steps\n", + "\n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your validation report." + ] + }, + { + "cell_type": "markdown", + "id": "next-document", + "metadata": {}, + "source": [ + "### Work with your validation report\n", + "\n", + "Now that you've logged all your test results and verified the work done by the model development team, head to the ValidMind Platform to wrap up your validation report:\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n", + "\n", + "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", + "\n", + "Include your logged test results as evidence, create risk assessment notes, add artifacts, and assess compliance, then submit your report for review when it's ready. **Learn more:** [Preparing validation reports](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html)" + ] + }, + { + "cell_type": "markdown", + "id": "next-resources", + "metadata": {}, + "source": [ + "### Discover more learning resources\n", + "\n", + "We offer many interactive notebooks to help you automate testing, documenting, validating, and more:\n", + "\n", + "- [Run tests & test suites](https://docs.validmind.ai/developer/how-to/testing-overview.html)\n", + "- [Use ValidMind Library features](https://docs.validmind.ai/developer/how-to/feature-overview.html)\n", + "- [Code samples by use case](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", + "\n", + "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind." + ] + } + ], + "metadata": { + "language_info": { + "name": "python" + } + }, + "nbformat": 4, + "nbformat_minor": 2 +} diff --git a/notebooks/tutorials/model_development/1-set_up_validmind.ipynb b/notebooks/tutorials/model_development/1-set_up_validmind.ipynb index 97a1e9dc1..ae99e7d7c 100644 --- a/notebooks/tutorials/model_development/1-set_up_validmind.ipynb +++ b/notebooks/tutorials/model_development/1-set_up_validmind.ipynb @@ -231,7 +231,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -247,10 +247,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -274,7 +275,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -297,7 +298,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -327,7 +328,7 @@ "\n", "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n", "\n", - "3. Click **Documentation** under Documents for your model and note how the structure of the documentation matches our preview above." + "3. Click **Development** under Documents for your model and note how the structure of the documentation matches our preview above." ] }, { diff --git a/notebooks/tutorials/model_development/2-start_development_process.ipynb b/notebooks/tutorials/model_development/2-start_development_process.ipynb index 217df37d1..b01cb3536 100644 --- a/notebooks/tutorials/model_development/2-start_development_process.ipynb +++ b/notebooks/tutorials/model_development/2-start_development_process.ipynb @@ -93,13 +93,9 @@ "\n", "First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", - "\n", - "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n", - "\n", - "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", - "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -126,7 +122,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -682,7 +678,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", "3. Locate the Data Preparation section and click on **2.3. Correlations and Interactions** to expand that section.\n", "\n", diff --git a/notebooks/tutorials/model_development/3-integrate_custom_tests.ipynb b/notebooks/tutorials/model_development/3-integrate_custom_tests.ipynb index f5bbf7f53..b4b990346 100644 --- a/notebooks/tutorials/model_development/3-integrate_custom_tests.ipynb +++ b/notebooks/tutorials/model_development/3-integrate_custom_tests.ipynb @@ -108,13 +108,9 @@ "\n", "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", - "\n", - "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n", - "\n", - "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", - "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -141,7 +137,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -909,7 +905,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", "3. Locate the Data Preparation section and click on **3.2. Model Evaluation** to expand that section.\n", "\n", diff --git a/notebooks/tutorials/model_development/4-finalize_testing_documentation.ipynb b/notebooks/tutorials/model_development/4-finalize_testing_documentation.ipynb index b6064a005..086ccc202 100644 --- a/notebooks/tutorials/model_development/4-finalize_testing_documentation.ipynb +++ b/notebooks/tutorials/model_development/4-finalize_testing_documentation.ipynb @@ -105,13 +105,9 @@ "\n", "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", - "\n", - "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model development\" series of notebooks.\n", - "\n", - "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", - "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -138,7 +134,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, diff --git a/notebooks/tutorials/model_validation/1-set_up_validmind_for_validation.ipynb b/notebooks/tutorials/model_validation/1-set_up_validmind_for_validation.ipynb index 60bf8373d..db4b30d98 100644 --- a/notebooks/tutorials/model_validation/1-set_up_validmind_for_validation.ipynb +++ b/notebooks/tutorials/model_validation/1-set_up_validmind_for_validation.ipynb @@ -243,7 +243,7 @@ "\n", "Next, let's select a validation report template. A template predefines sections for your report and provides a general outline to follow, making the validation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Validation**.\n", "\n", "2. Under **TEMPLATE**, select `Generic Validation Report`.\n", "\n", @@ -283,9 +283,7 @@ "source": [ "\n", "\n", - "### Initialize the ValidMind Library\n", - "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your validation environment. You initialize the ValidMind Library with this code snippet, which ensures that your test results are uploaded to the correct model when you run the notebook." + "### Initialize the ValidMind Library" ] }, { @@ -297,13 +295,11 @@ "\n", "#### Get your code snippet\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", - "\n", - "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model validation\" series of notebooks.\n", - "\n", - "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Validation` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -327,7 +323,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"validation-report\",\n", + " document=\"validation-report\",\n", ")" ] }, @@ -380,7 +376,7 @@ "\n", "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model validation\" series of notebooks.\n", "\n", - "3. Click **Validation Report** under Documents for your model and note:\n", + "3. Click **Validation** under Documents for your model and note:\n", "\n", " - [x] The risk assessment compliance summary at the top of the report (screenshot below)\n", " - [x] How the structure of the validation report reflects the previewed template\n", diff --git a/notebooks/tutorials/model_validation/2-start_validation_process.ipynb b/notebooks/tutorials/model_validation/2-start_validation_process.ipynb index e0618c24a..7fafa5ab5 100644 --- a/notebooks/tutorials/model_validation/2-start_validation_process.ipynb +++ b/notebooks/tutorials/model_validation/2-start_validation_process.ipynb @@ -96,13 +96,9 @@ "\n", "First, let's connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", - "\n", - "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model validation\" series of notebooks.\n", - "\n", - "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", - "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Validation` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -129,7 +125,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"validation-report\",\n", + " document=\"validation-report\",\n", ")" ] }, @@ -693,7 +689,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n", "\n", - "2. In the left sidebar that appears for your model, click **Validation Report** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", "\n", "3. Locate the Data Preparation section and click on **2.2.1. Data Quality** to expand that section.\n", "\n", diff --git a/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb b/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb index cc3f25c4d..e86fde85d 100644 --- a/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb +++ b/notebooks/tutorials/model_validation/3-developing_challenger_model.ipynb @@ -103,13 +103,9 @@ "\n", "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", - "\n", - "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model validation\" series of notebooks.\n", - "\n", - "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", - "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Validation` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -136,7 +132,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"validation-report\",\n", + " document=\"validation-report\",\n", ")" ] }, @@ -627,7 +623,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n", "\n", - "2. In the left sidebar that appears for your model, click **Validation Report** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", "\n", "3. Locate the Data Preparation section and click on **2.2.2. Model Performance** to expand that section.\n", "\n", diff --git a/notebooks/tutorials/model_validation/4-finalize_validation_reporting.ipynb b/notebooks/tutorials/model_validation/4-finalize_validation_reporting.ipynb index 3eaee5a3a..854ccde76 100644 --- a/notebooks/tutorials/model_validation/4-finalize_validation_reporting.ipynb +++ b/notebooks/tutorials/model_validation/4-finalize_validation_reporting.ipynb @@ -112,13 +112,9 @@ "\n", "As usual, let's first connect up the ValidMind Library to our model we previously registered in the ValidMind Platform:\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", - "\n", - "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this \"ValidMind for model validation\" series of notebooks.\n", - "\n", - "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", - "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Validation` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -145,7 +141,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"validation-report\",\n", + " document=\"validation-report\",\n", ")" ] }, diff --git a/notebooks/use_cases/agents/document_agentic_ai.ipynb b/notebooks/use_cases/agents/document_agentic_ai.ipynb index aa49a9a31..151602ba6 100644 --- a/notebooks/use_cases/agents/document_agentic_ai.ipynb +++ b/notebooks/use_cases/agents/document_agentic_ai.ipynb @@ -252,7 +252,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Agentic AI`.\n", "\n", @@ -283,10 +283,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -310,7 +311,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -321,9 +322,9 @@ "source": [ "\n", "\n", - "#### Preview the documentation template\n", + "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -2047,7 +2048,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", " What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", diff --git a/notebooks/use_cases/capital_markets/quickstart_option_pricing_models.ipynb b/notebooks/use_cases/capital_markets/quickstart_option_pricing_models.ipynb index 175b02a59..5f24cf553 100644 --- a/notebooks/use_cases/capital_markets/quickstart_option_pricing_models.ipynb +++ b/notebooks/use_cases/capital_markets/quickstart_option_pricing_models.ipynb @@ -203,7 +203,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Capital markets`.\n", "\n", @@ -219,10 +219,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -246,7 +247,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -287,7 +288,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -2042,11 +2043,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/use_cases/capital_markets/quickstart_option_pricing_models_quantlib.ipynb b/notebooks/use_cases/capital_markets/quickstart_option_pricing_models_quantlib.ipynb index 3dadf2ca1..97085a038 100644 --- a/notebooks/use_cases/capital_markets/quickstart_option_pricing_models_quantlib.ipynb +++ b/notebooks/use_cases/capital_markets/quickstart_option_pricing_models_quantlib.ipynb @@ -260,7 +260,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Capital Markets`.\n", "\n", @@ -291,10 +291,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -318,7 +319,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")\n" ] }, @@ -361,7 +362,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -1287,11 +1288,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/use_cases/code_explainer/quickstart_code_explainer_demo.ipynb b/notebooks/use_cases/code_explainer/quickstart_code_explainer_demo.ipynb index afd5e19f6..ceba6ab67 100644 --- a/notebooks/use_cases/code_explainer/quickstart_code_explainer_demo.ipynb +++ b/notebooks/use_cases/code_explainer/quickstart_code_explainer_demo.ipynb @@ -199,7 +199,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Model Source Code Documentation`.\n", "\n", @@ -228,10 +228,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -254,7 +255,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -266,7 +267,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] diff --git a/notebooks/use_cases/credit_risk/application_scorecard_executive.ipynb b/notebooks/use_cases/credit_risk/application_scorecard_executive.ipynb index 40388556c..cd6e4f1a1 100644 --- a/notebooks/use_cases/credit_risk/application_scorecard_executive.ipynb +++ b/notebooks/use_cases/credit_risk/application_scorecard_executive.ipynb @@ -178,7 +178,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **[template]{.smallcaps}**, select `Credit Risk Scorecard`.\n", "\n", @@ -193,10 +193,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -208,10 +209,10 @@ "import validmind as vm\n", "\n", "vm.init(\n", - " api_host = \"https://api.prod.validmind.ai/api/v1/tracking\",\n", - " api_key = \"...\",\n", - " api_secret = \"...\",\n", - " model = \"...\",\n", + " # api_host = \"...\",\n", + " # api_key = \"...\",\n", + " # api_secret = \"...\",\n", + " # model = \"...\",\n", " document=\"documentation\",\n", ")" ] @@ -279,14 +280,16 @@ "\n", "### Work with your model documentation\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Expand the following sections and take a look around:\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", + "\n", + "3. Expand the following sections and take a look around:\n", "\n", " - **2. Data Preparation**\n", " - **3. Model Development**\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation (hint: some of the tests in **2.3. Feature Selection and Engineering** look like they need some attention), view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready.\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation (hint: some of the tests in **2.3. Feature Selection and Engineering** look like they need some attention), view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/use_cases/credit_risk/application_scorecard_full_suite.ipynb b/notebooks/use_cases/credit_risk/application_scorecard_full_suite.ipynb index e64a206e4..fee561844 100644 --- a/notebooks/use_cases/credit_risk/application_scorecard_full_suite.ipynb +++ b/notebooks/use_cases/credit_risk/application_scorecard_full_suite.ipynb @@ -192,7 +192,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Credit Risk Scorecard`.\n", "\n", @@ -207,10 +207,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -229,10 +230,10 @@ "import validmind as vm\n", "\n", "vm.init(\n", - " api_host = \"https://api.prod.validmind.ai/api/v1/tracking\",\n", - " api_key = \"...\",\n", - " api_secret = \"...\",\n", - " model = \"...\",\n", + " # api_host = \"...\",\n", + " # api_key = \"...\",\n", + " # api_secret = \"...\",\n", + " # model = \"...\",\n", " document=\"documentation\",\n", ")" ] @@ -270,7 +271,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -799,14 +800,16 @@ "\n", "### Work with your model documentation\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Expand the following sections and take a look around:\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", + "\n", + "3. Expand the following sections and take a look around:\n", "\n", " - **2. Data Preparation**\n", " - **3. Model Development**\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation (hint: some of the tests in **2.3. Feature Selection and Engineering** look like they need some attention), view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready.\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation (hint: some of the tests in **2.3. Feature Selection and Engineering** look like they need some attention), view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/use_cases/credit_risk/application_scorecard_with_bias.ipynb b/notebooks/use_cases/credit_risk/application_scorecard_with_bias.ipynb index 41769972a..44907d374 100644 --- a/notebooks/use_cases/credit_risk/application_scorecard_with_bias.ipynb +++ b/notebooks/use_cases/credit_risk/application_scorecard_with_bias.ipynb @@ -193,7 +193,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Credit Risk Scorecard`.\n", "\n", @@ -208,10 +208,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -286,7 +287,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -1444,14 +1445,16 @@ "\n", "### Work with your model documentation\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Expand the following sections and take a look around:\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", + "\n", + "3. Expand the following sections and take a look around:\n", "\n", " - **2. Data Preparation**\n", " - **3. Model Development**\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation (hint: some of the tests in **2.3. Feature Selection and Engineering** look like they need some attention), view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready.\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation (hint: some of the tests in **2.3. Feature Selection and Engineering** look like they need some attention), view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/use_cases/credit_risk/application_scorecard_with_ml.ipynb b/notebooks/use_cases/credit_risk/application_scorecard_with_ml.ipynb index cdbb02b0e..88ba2e554 100644 --- a/notebooks/use_cases/credit_risk/application_scorecard_with_ml.ipynb +++ b/notebooks/use_cases/credit_risk/application_scorecard_with_ml.ipynb @@ -205,7 +205,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Credit Risk Scorecard`.\n", "\n", @@ -220,10 +220,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -242,10 +243,10 @@ "import validmind as vm\n", "\n", "vm.init(\n", - " api_host = \"https://api.prod.validmind.ai/api/v1/tracking\",\n", - " api_key = \"...\",\n", - " api_secret = \"...\",\n", - " model = \"...\",\n", + " # api_host = \"...\",\n", + " # api_key = \"...\",\n", + " # api_secret = \"...\",\n", + " # model = \"...\",\n", " document=\"documentation\",\n", ")" ] @@ -284,7 +285,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -1894,14 +1895,16 @@ "\n", "### Work with your model documentation\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Expand the following sections and take a look around:\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", + "\n", + "3. Expand the following sections and take a look around:\n", "\n", " - **2. Data Preparation**\n", " - **3. Model Development**\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation (hint: some of the tests in **2.3. Feature Selection and Engineering** look like they need some attention), view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready.\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation (hint: some of the tests in **2.3. Feature Selection and Engineering** look like they need some attention), view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n", diff --git a/notebooks/use_cases/credit_risk/document_excel_application_scorecard.ipynb b/notebooks/use_cases/credit_risk/document_excel_application_scorecard.ipynb index 252fb1a60..e77675c13 100644 --- a/notebooks/use_cases/credit_risk/document_excel_application_scorecard.ipynb +++ b/notebooks/use_cases/credit_risk/document_excel_application_scorecard.ipynb @@ -194,7 +194,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Credit Risk Scorecard`.\n", "\n", @@ -209,10 +209,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -235,7 +236,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -275,7 +276,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -883,7 +884,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. In the left sidebar that appears for your model, click **Documentation** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", " What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready.\n", "\n", diff --git a/notebooks/use_cases/model_validation/validate_application_scorecard.ipynb b/notebooks/use_cases/model_validation/validate_application_scorecard.ipynb index 3e20fa1eb..f3df8a617 100644 --- a/notebooks/use_cases/model_validation/validate_application_scorecard.ipynb +++ b/notebooks/use_cases/model_validation/validate_application_scorecard.ipynb @@ -228,9 +228,9 @@ "\n", "#### Apply validation report template\n", "\n", - "Once you've registered your model, let's select a validation report template. A template predefines sections for your report and provides a general outline to follow, making the validation process much easier.\n", + "Next, let's select a validation report template. A template predefines sections for your report and provides a general outline to follow, making the validation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Validation**.\n", "\n", "2. Under **TEMPLATE**, select `Generic Validation Report`.\n", "\n", @@ -267,9 +267,7 @@ "source": [ "\n", "\n", - "### Initialize the ValidMind Library\n", - "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your validation environment. You initialize the ValidMind Library with this code snippet, which ensures that your test results are uploaded to the correct model when you run the notebook." + "### Initialize the ValidMind Library" ] }, { @@ -280,13 +278,11 @@ "\n", "#### Get your code snippet\n", "\n", - "1. In a browser, [log in to ValidMind](https://docs.validmind.ai/guide/configuration/log-in-to-validmind.html).\n", - "\n", - "2. In the left sidebar, navigate to **Inventory** and select the model you registered for this notebook.\n", - "\n", - "3. Go to **Getting Started** and click **Copy snippet to clipboard**.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Validation` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -309,7 +305,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"validation-report\",\n", + " document=\"validation-report\",\n", ")" ] }, @@ -1783,7 +1779,7 @@ "\n", "1. From the **Inventory** in the ValidMind Platform, go to the model you connected to earlier.\n", "\n", - "2. In the left sidebar that appears for your model, click **Validation Report** under Documents.\n", + "2. In the left sidebar that appears for your model, click **Validation** under Documents.\n", "\n", "Include your logged test results as evidence, create risk assessment notes, add artifacts, and assess compliance, then submit your report for review when it's ready. **Learn more:** [Preparing validation reports](https://docs.validmind.ai/guide/model-validation/preparing-validation-reports.html)" ] diff --git a/notebooks/use_cases/nlp_and_llm/foundation_models_integration_demo.ipynb b/notebooks/use_cases/nlp_and_llm/foundation_models_integration_demo.ipynb index 813acd5d9..09cc86b95 100644 --- a/notebooks/use_cases/nlp_and_llm/foundation_models_integration_demo.ipynb +++ b/notebooks/use_cases/nlp_and_llm/foundation_models_integration_demo.ipynb @@ -11,6 +11,11 @@ "This interactive notebook shows you how to set up the ValidMind Library, initializes the library, and uses a specific prompt template for analyzing the sentiment of sentences in a dataset. The notebook also includes example data to test the model's ability to correctly identify sentiment as positive, negative, or neutral." ] }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [] + }, { "cell_type": "markdown", "metadata": {}, @@ -145,7 +150,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `LLM-based Text Classification`.\n", "\n", @@ -160,10 +165,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -186,7 +192,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -198,7 +204,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -407,7 +413,7 @@ "\n", "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation in the ValidMind Platform:\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html))\n", + "1. In the ValidMind Platform, click **Development** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html))\n", "\n", "2. Expand **2. Data Preparation** or **3. Model Development** to review all test results.\n", "\n", diff --git a/notebooks/use_cases/nlp_and_llm/foundation_models_summarization_demo.ipynb b/notebooks/use_cases/nlp_and_llm/foundation_models_summarization_demo.ipynb index cfb50ce82..e0957d894 100644 --- a/notebooks/use_cases/nlp_and_llm/foundation_models_summarization_demo.ipynb +++ b/notebooks/use_cases/nlp_and_llm/foundation_models_summarization_demo.ipynb @@ -146,7 +146,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `LLM-based Text Summarization`.\n", "\n", @@ -161,10 +161,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -187,7 +188,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -199,7 +200,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -456,7 +457,7 @@ "\n", "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation in the ValidMind Platform:\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "1. In the ValidMind Platform, click **Development** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", "\n", "2. Expand **2. Data Preparation** or **3. Model Development** to review all test results.\n", "\n", diff --git a/notebooks/use_cases/nlp_and_llm/hugging_face_integration_demo.ipynb b/notebooks/use_cases/nlp_and_llm/hugging_face_integration_demo.ipynb index 082b4fb6e..fef6db81a 100644 --- a/notebooks/use_cases/nlp_and_llm/hugging_face_integration_demo.ipynb +++ b/notebooks/use_cases/nlp_and_llm/hugging_face_integration_demo.ipynb @@ -147,7 +147,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `NLP-based Text Classification`.\n", "\n", @@ -162,10 +162,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -188,7 +189,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -200,7 +201,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -391,7 +392,7 @@ "\n", "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation in the ValidMind Platform:\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "1. In the ValidMind Platform, click **Development** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", "\n", "2. Expand **2. Data Preparation** or **3. Model Development** to review all test results.\n", "\n", diff --git a/notebooks/use_cases/nlp_and_llm/hugging_face_summarization_demo.ipynb b/notebooks/use_cases/nlp_and_llm/hugging_face_summarization_demo.ipynb index fc4eefd75..d9732979c 100644 --- a/notebooks/use_cases/nlp_and_llm/hugging_face_summarization_demo.ipynb +++ b/notebooks/use_cases/nlp_and_llm/hugging_face_summarization_demo.ipynb @@ -144,7 +144,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `NLP-based Text Classification`.\n", "\n", @@ -159,10 +159,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -185,7 +186,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -197,7 +198,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -412,7 +413,7 @@ "\n", "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation in the ValidMind Platform:\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "1. In the ValidMind Platform, click **Development** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", "\n", "2. Expand **2. Data Preparation** or **3. Model Development** to review all test results.\n", "\n", diff --git a/notebooks/use_cases/nlp_and_llm/llm_summarization_demo.ipynb b/notebooks/use_cases/nlp_and_llm/llm_summarization_demo.ipynb index 69f5dcd4c..3792a0a78 100644 --- a/notebooks/use_cases/nlp_and_llm/llm_summarization_demo.ipynb +++ b/notebooks/use_cases/nlp_and_llm/llm_summarization_demo.ipynb @@ -178,7 +178,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `LLM-based Text Summarization`.\n", "\n", @@ -193,10 +193,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -219,7 +220,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -253,7 +254,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -929,7 +930,7 @@ "\n", "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the prompt validation test results as part of your model documentation right in the ValidMind Platform:\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "1. In the ValidMind Platform, click **Development** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", "\n", "2. Expand **2. Data Preparation** or **3. Model Development** to review all test results.\n", "\n", diff --git a/notebooks/use_cases/nlp_and_llm/prompt_validation_demo.ipynb b/notebooks/use_cases/nlp_and_llm/prompt_validation_demo.ipynb index 1f0114a3c..51fb1fbfa 100644 --- a/notebooks/use_cases/nlp_and_llm/prompt_validation_demo.ipynb +++ b/notebooks/use_cases/nlp_and_llm/prompt_validation_demo.ipynb @@ -174,7 +174,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `LLM-based Text Classification`.\n", "\n", @@ -189,10 +189,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -215,7 +216,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -227,7 +228,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] diff --git a/notebooks/use_cases/nlp_and_llm/rag_benchmark_demo.ipynb b/notebooks/use_cases/nlp_and_llm/rag_benchmark_demo.ipynb index bb0a89b5b..1e3eb07b6 100644 --- a/notebooks/use_cases/nlp_and_llm/rag_benchmark_demo.ipynb +++ b/notebooks/use_cases/nlp_and_llm/rag_benchmark_demo.ipynb @@ -213,7 +213,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Gen AI RAG`.\n", "\n", @@ -242,10 +242,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -264,10 +265,10 @@ "import validmind as vm\n", "\n", "vm.init(\n", - " api_host = \"https://api.prod.validmind.ai/api/v1/tracking\",\n", - " api_key = \"...\",\n", - " api_secret = \"...\",\n", - " model = \"...\",\n", + " # api_host = \"...\",\n", + " # api_key = \"...\",\n", + " # api_secret = \"...\",\n", + " # model = \"...\",\n", " document=\"documentation\",\n", ")" ] diff --git a/notebooks/use_cases/nlp_and_llm/rag_documentation_demo.ipynb b/notebooks/use_cases/nlp_and_llm/rag_documentation_demo.ipynb index 7e606f031..a57ebed3b 100644 --- a/notebooks/use_cases/nlp_and_llm/rag_documentation_demo.ipynb +++ b/notebooks/use_cases/nlp_and_llm/rag_documentation_demo.ipynb @@ -187,7 +187,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Gen AI RAG`.\n", "\n", @@ -216,10 +216,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -242,7 +243,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, diff --git a/notebooks/use_cases/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb b/notebooks/use_cases/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb index b822bbcde..09d94a6f1 100644 --- a/notebooks/use_cases/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb +++ b/notebooks/use_cases/ongoing_monitoring/application_scorecard_ongoing_monitoring.ipynb @@ -25,10 +25,11 @@ " - [Install the ValidMind Library](#toc2_1__) \n", " - [Initialize the ValidMind Library](#toc2_2__) \n", " - [Register sample model](#toc2_2_1__) \n", - " - [Apply documentation template](#toc2_2_2__) \n", + " - [Apply monitoring report template](#toc2_2_2__) \n", " - [Get your code snippet](#toc2_2_3__) \n", - " - [Initialize the Python environment](#toc2_3__) \n", - " - [Preview the monitoring template](#toc2_4__) \n", + " - [Preview the monitoring report template](#toc2_3__) \n", + " - [Initialize the Python environment](#toc2_4__) \n", + " - [Preview the monitoring template](#toc2_5__) \n", "- [Load the reference and monitoring datasets](#toc3__) \n", "- [Train the model](#toc4__) \n", " - [Initialize the ValidMind datasets](#toc4_1__) \n", @@ -95,9 +96,9 @@ "\n", "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", "\n", - "**Model monitoring documentation**: A comprehensive and structured record of a production model, including key elements such as data sources, inputs, performance metrics, and periodic evaluations. This documentation ensures transparency and visibility of the model's performance in the production environment.\n", + "**Model monitoring report**: A comprehensive and structured record of a production model, including key elements such as data sources, inputs, performance metrics, and periodic evaluations. This documentation ensures transparency and visibility of the model's performance in the production environment.\n", "\n", - "**Monitoring documentation template**: Similar to documentation template, The monitoring documentation template functions as a test suite and lays out the structure of model monitoring documentation, segmented into various sections and sub-sections. Monitoring documentation templates define the structure of your model monitoring documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "**Monitoring report template**: Similar to documentation template, The monitoring report template functions as a test suite and lays out the structure of model monitoring, segmented into various sections and sub-sections. Monitoring report templates define the structure of your model monitoring report, specifying the tests that should be run, and how the results should be displayed.\n", "\n", "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", "\n", @@ -180,13 +181,13 @@ "source": [ "\n", "\n", - "#### Apply documentation template\n", + "#### Apply monitoring report template\n", "\n", - "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "Once you've registered your model, let's select a monitoring report template. A template predefines sections for your monitoring report and provides a general outline to follow, making the monitoring process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Monitoring**.\n", "\n", - "2. Under **TEMPLATE**, select `Binary classification`.\n", + "2. Under **TEMPLATE**, select `Ongoing Monitoring for Classification Models`.\n", "\n", "3. Click **Use Template** to apply the template." ] @@ -199,10 +200,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Monitoring` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -221,12 +223,12 @@ "import validmind as vm\n", "\n", "vm.init(\n", - " api_host = \"https://api.prod.validmind.ai/api/v1/tracking\",\n", - " api_key = \"...\",\n", - " api_secret = \"...\",\n", - " model = \"...\",\n", - " monitoring = True,\n", - " document=\"monitoring\",\n", + " # api_host=\"...\",\n", + " # api_key=\"...\",\n", + " # api_secret=\"...\",\n", + " # model=\"...\",\n", + " document=\"monitoring\",\n", + " monitoring = True,\n", ")" ] }, @@ -236,6 +238,28 @@ "source": [ "\n", "\n", + "### Preview the monitoring report template\n", + "\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your monitoring report and provides a general outline to follow, making the monitoring process much easier.\n", + "\n", + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "vm.preview_template()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "\n", + "\n", "### Initialize the Python environment\n", "\n", "Next, let's import the necessary libraries and set up your Python environment for data analysis:" @@ -266,7 +290,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "\n", + "\n", "\n", "### Preview the monitoring template\n", "\n", @@ -1319,7 +1343,6 @@ }, { "cell_type": "markdown", - "id": "copyright-9b204f07d6ef4508a249d0c7d5e7d44b", "metadata": {}, "source": [ "\n", diff --git a/notebooks/use_cases/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb b/notebooks/use_cases/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb index 8ca30216e..109773a63 100644 --- a/notebooks/use_cases/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb +++ b/notebooks/use_cases/ongoing_monitoring/quickstart_customer_churn_ongoing_monitoring.ipynb @@ -27,10 +27,10 @@ " - [Install the ValidMind Library](#toc2_1__) \n", " - [Initialize the ValidMind Library](#toc2_2__) \n", " - [Register sample model](#toc2_2_1__) \n", - " - [Apply documentation template](#toc2_2_2__) \n", + " - [Apply monitoring report template](#toc2_2_2__) \n", " - [Get your code snippet](#toc2_2_3__) \n", " - [Initialize the Python environment](#toc2_3__) \n", - " - [Preview the monitoring template](#toc2_4__) \n", + " - [Preview the monitoring report template](#toc2_4__) \n", "- [Load the reference and monitoring datasets](#toc3__) \n", " - [Load the production model](#toc3_1__) \n", " - [Initialize the ValidMind datasets](#toc3_2__) \n", @@ -41,7 +41,8 @@ " - [Feature drift tests](#toc3_6_1__) \n", " - [Model performance monitoring tests](#toc3_7__) \n", "- [Next steps](#toc4__) \n", - " - [Discover more learning resources](#toc4_1__) \n", + " - [Work with your monitoring report](#toc4_1__) \n", + " - [Discover more learning resources](#toc4_2__) \n", "- [Upgrade ValidMind](#toc5__) \n", "\n", ":::\n", @@ -93,9 +94,9 @@ "\n", "**Documentation template**: Functions as a test suite and lays out the structure of model documentation, segmented into various sections and sub-sections. Documentation templates define the structure of your model documentation, specifying the tests that should be run, and how the results should be displayed.\n", "\n", - "**Model monitoring documentation**: A comprehensive and structured record of a production model, including key elements such as data sources, inputs, performance metrics, and periodic evaluations. This documentation ensures transparency and visibility of the model's performance in the production environment.\n", + "**Model monitoring report**: A comprehensive and structured record of a production model, including key elements such as data sources, inputs, performance metrics, and periodic evaluations. This documentation ensures transparency and visibility of the model's performance in the production environment.\n", "\n", - "**Monitoring documentation template**: Similar to documentation template, The monitoring documentation template functions as a test suite and lays out the structure of model monitoring documentation, segmented into various sections and sub-sections. Monitoring documentation templates define the structure of your model monitoring documentation, specifying the tests that should be run, and how the results should be displayed.\n", + "**Monitoring report template**: Similar to documentation template, The monitoring report template functions as a test suite and lays out the structure of model monitoring, segmented into various sections and sub-sections. Monitoring report templates define the structure of your model monitoring report, specifying the tests that should be run, and how the results should be displayed.\n", "\n", "**Tests**: A function contained in the ValidMind Library, designed to run a specific quantitative test on the dataset or model. Tests are the building blocks of ValidMind, used to evaluate and document models and datasets, and can be run individually or as part of a suite defined by your model documentation template.\n", "\n", @@ -178,13 +179,13 @@ "source": [ "\n", "\n", - "#### Apply documentation template\n", + "#### Apply monitoring report template\n", "\n", - "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", + "Once you've registered your model, let's select a monitoring report template. A template predefines sections for your monitoring report and provides a general outline to follow, making the monitoring process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Monitoring**.\n", "\n", - "2. Under **TEMPLATE**, select `Binary classification`.\n", + "2. Under **TEMPLATE**, select `Ongoing Monitoring for Classification Models`.\n", "\n", "3. Click **Use Template** to apply the template." ] @@ -197,10 +198,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Monitoring` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -223,8 +225,8 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " monitoring = True,\n", " document=\"monitoring\",\n", + " monitoring = True,\n", ")" ] }, @@ -262,11 +264,11 @@ "source": [ "\n", "\n", - "### Preview the monitoring template\n", + "### Preview the monitoring report template\n", "\n", - "A template predefines sections for your model monitoring documentation and provides a general outline to follow, making the documentation process much easier.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your monitoring report and provides a general outline to follow, making the monitoring process much easier.\n", "\n", - "You will upload documentation and test results into this template later on. For now, take a look at the structure that the template provides with the `vm.preview_template()` function from the ValidMind library and note the empty sections:" + "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] }, { @@ -512,7 +514,7 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "To view the results of the model metadata and data quality tests, select **Ongoing Monitoring** under Documents in the left sidebar of the model in the ValidMind Platform and click on the following sections:\n", + "To view the results of the model metadata and data quality tests, select **Monitoring** under Documents in the left sidebar of the model in the ValidMind Platform and click on the following sections:\n", "\n", "- 1. Model Monitoring Overview > **1.2. Model Details**\n", "- 2. Data Quality & Drift Assessment > **2.1. Prediction Data Description**" @@ -785,14 +787,19 @@ "\n", "## Next steps\n", "\n", - "You can now review all the ongoing monitoring results in the ValidMind Platform.\n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your monitoring report.\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", - "2. In the left sidebar that appears for your model, click **Ongoing Monitoring** under Documents.\n", + "\n", "\n", - "What you see is the full draft of your model monitoring documentation in a more easily consumable version. From here, you can make qualitative edits to model monitoring documentation, view guidelines, collaborate with validators, and submit your model monitoring documentation for approval when it's ready.\n", + "### Work with your monitoring report\n", "\n", - "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "\n", + "2. In the left sidebar that appears for your model, click **Monitoring** under Documents.\n", + "\n", + "What you see is the full draft of your monitoring report in a more easily consumable version. From here, you can make qualitative edits to monitoring reports, view guidelines, review monitoring results, and submit your monitoring report for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/monitoring/ongoing-monitoring.html)\n", + "\n", + "\n", "\n", "### Discover more learning resources\n", "\n", @@ -847,7 +854,6 @@ }, { "cell_type": "markdown", - "id": "copyright-a5f8f86b12d74c72a867a01b3aeb6da2", "metadata": {}, "source": [ "\n", diff --git a/notebooks/use_cases/regression/quickstart_regression_full_suite.ipynb b/notebooks/use_cases/regression/quickstart_regression_full_suite.ipynb index 078fc73f0..88c758cd3 100644 --- a/notebooks/use_cases/regression/quickstart_regression_full_suite.ipynb +++ b/notebooks/use_cases/regression/quickstart_regression_full_suite.ipynb @@ -39,6 +39,8 @@ " - [Assign predictions to the datasets](#toc5_4__) \n", " - [Run the full suite of tests](#toc5_5__) \n", "- [Next steps](#toc6__) \n", + " - [Work with your model documentation](#toc6_1__) \n", + " - [Discover more learning resources](#toc6_2__) \n", "- [Upgrade ValidMind](#toc7__) \n", "\n", ":::\n", @@ -152,7 +154,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -167,10 +169,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -193,7 +196,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -228,7 +231,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -491,18 +494,34 @@ "\n", "## Next steps\n", "\n", - "You can look at the results of this test suite right in the notebook where you ran the code, as you would expect. But there is a better way: view the test results as part of your model documentation right in the ValidMind Platform:\n", + "You can look at the output produced by the ValidMind Library right in the notebook where you ran the code, as you would expect. But there is a better way — use the ValidMind Platform to work with your model documentation.\n", "\n", - "1. In the ValidMind Platform, click **Documentation** under Documents for the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "\n", "\n", - "2. Expand the following sections and take a look around:\n", + "### Work with your model documentation\n", + "\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", + "\n", + "3. Expand the following sections and take a look around:\n", "\n", " - **2. Data Preparation**\n", " - **3. Model Development**\n", "\n", - "What you can see now is a much more easily consumable version of the documentation, including the results of the tests you just performed, along with other parts of your model documentation that still need to be completed. There is a wealth of information that gets uploaded when you run the full test suite, so take a closer look around, especially at test results that might need attention (hint: some of the tests in **2.1 Data description** look like they need some attention).\n", + "What you can see now is a much more easily consumable version of the documentation, including the results of the tests you just performed, along with other parts of your model documentation that still need to be completed. There is a wealth of information that gets uploaded when you run the full test suite, so take a closer look around, especially at test results that might need attention (hint: some of the tests in **2.1 Data description** look like they need some attention). From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", + "\n", + "\n", + "\n", + "### Discover more learning resources\n", + "\n", + "We offer many interactive notebooks to help you automate testing, documenting, validating, and more:\n", + "\n", + "- [Run tests & test suites](https://docs.validmind.ai/developer/how-to/testing-overview.html)\n", + "- [Use ValidMind Library features](https://docs.validmind.ai/developer/how-to/feature-overview.html)\n", + "- [Code samples by use case](https://docs.validmind.ai/guide/samples-jupyter-notebooks.html)\n", "\n", - "If you want to learn more about where you are in the model documentation process, take a look our documentation on the [ValidMind Library](https://docs.validmind.ai/developer/validmind-library.html)." + "Or, visit our [documentation](https://docs.validmind.ai/) to learn more about ValidMind." ] }, { @@ -547,7 +566,6 @@ }, { "cell_type": "markdown", - "id": "copyright-e2b543078a614a038f27fa63a6c74297", "metadata": {}, "source": [ "\n", diff --git a/notebooks/use_cases/time_series/quickstart_time_series_full_suite.ipynb b/notebooks/use_cases/time_series/quickstart_time_series_full_suite.ipynb index d130403e3..a54148045 100644 --- a/notebooks/use_cases/time_series/quickstart_time_series_full_suite.ipynb +++ b/notebooks/use_cases/time_series/quickstart_time_series_full_suite.ipynb @@ -186,7 +186,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Binary classification`.\n", "\n", @@ -201,10 +201,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -227,7 +228,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -264,7 +265,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] diff --git a/notebooks/use_cases/time_series/quickstart_time_series_high_code.ipynb b/notebooks/use_cases/time_series/quickstart_time_series_high_code.ipynb index a09cf81d7..8274f5443 100644 --- a/notebooks/use_cases/time_series/quickstart_time_series_high_code.ipynb +++ b/notebooks/use_cases/time_series/quickstart_time_series_high_code.ipynb @@ -187,7 +187,7 @@ "\n", "Once you've registered your model, let's select a documentation template. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", - "1. In the left sidebar that appears for your model, click **Documents** and select **Documentation**.\n", + "1. In the left sidebar that appears for your model, click **Documents** and select **Development**.\n", "\n", "2. Under **TEMPLATE**, select `Time Series Forecasting with ML`.\n", "\n", @@ -202,10 +202,11 @@ "\n", "#### Get your code snippet\n", "\n", - "ValidMind generates a unique _code snippet_ for each registered model to connect with your developer environment. You initialize the ValidMind Library with this code snippet, which ensures that your documentation and tests are uploaded to the correct model when you run the notebook.\n", + "Initialize the ValidMind Library with the *code snippet* unique to each model per document, ensuring your test results are uploaded to the correct model and automatically populated in the right document in the ValidMind Platform when you run this notebook.\n", "\n", - "1. On the left sidebar that appears for your model, select **Getting Started** and click **Copy snippet to clipboard**.\n", - "2. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" + "1. On the left sidebar that appears for your model, select **Getting Started** and select `Development` from the **DOCUMENT** drop-down menu.\n", + "2. Click **Copy snippet to clipboard**.\n", + "3. Next, [load your model identifier credentials from an `.env` file](https://docs.validmind.ai/developer/model-documentation/store-credentials-in-env-file.html) or replace the placeholder with your own code snippet:" ] }, { @@ -228,7 +229,7 @@ " # api_key=\"...\",\n", " # api_secret=\"...\",\n", " # model=\"...\",\n", - " # document=\"documentation\",\n", + " document=\"documentation\",\n", ")" ] }, @@ -264,7 +265,7 @@ "\n", "### Preview the documentation template\n", "\n", - "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model.\n", + "Let's verify that you have connected the ValidMind Library to the ValidMind Platform and that the appropriate *template* is selected for your model. A template predefines sections for your model documentation and provides a general outline to follow, making the documentation process much easier.\n", "\n", "You will upload documentation and test results unique to your model based on this template later on. For now, **take a look at the default structure that the template provides with [the `vm.preview_template()` function](https://docs.validmind.ai/validmind/validmind.html#preview_template)** from the ValidMind library and note the empty sections:" ] @@ -907,11 +908,11 @@ "\n", "### Work with your model documentation\n", "\n", - "1. From the **Model Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", + "1. From the **Inventory** in the ValidMind Platform, go to the model you registered earlier. ([Need more help?](https://docs.validmind.ai/guide/model-inventory/working-with-model-inventory.html))\n", "\n", - "2. Click and expand the **Model Development** section.\n", + "2. In the left sidebar that appears for your model, click **Development** under Documents.\n", "\n", - "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/model-documentation/working-with-model-documentation.html)\n", + "What you see is the full draft of your model documentation in a more easily consumable version. From here, you can make qualitative edits to model documentation, view guidelines, collaborate with validators, and submit your model documentation for approval when it's ready. [Learn more ...](https://docs.validmind.ai/guide/working-with-model-documentation.html)\n", "\n", "\n", "\n",