Users/robrandao/integration overhaul #315

rodrigobr-msft · 2026-02-06T23:50:20Z

Motivation

Without this framework

To test an echo agent you need to: get a token, start a callback server,
build the activity JSON, send it, wait, and check the response.

import aiohttp
import asyncio
import json
from aiohttp import web

APP_ID, APP_SECRET, TENANT_ID = "...", "...", "..."

async def get_token() -> str:
    async with aiohttp.ClientSession() as session:
        async with session.post(
            f"https://login.microsoftonline.com/{TENANT_ID}/oauth2/v2.0/token",
            data={
                "grant_type": "client_credentials",
                "client_id": APP_ID,
                "client_secret": APP_SECRET,
                "scope": f"{APP_ID}/.default",
            },
        ) as resp:
            return (await resp.json())["access_token"]

collected = []

async def _callback_handler(request):
    collected.append(await request.json())
    return web.Response(text="OK")

callback_app = web.Application()
callback_app.router.add_post("/v3/conversations/{path:.*}", _callback_handler)
runner = web.AppRunner(callback_app)

activity = {
    "type": "message",
    "text": "Hello!",
    "channelId": "test",
    "conversation": {"id": "test-conv-123"},
    "from": {"id": "user-1", "name": "Test User"},
    "recipient": {"id": "bot-1", "name": "My Bot"},
    "serviceUrl": "http://localhost:9378/v3/conversations/",
}

async def test_echo():
    await runner.setup()
    site = web.TCPSite(runner, "localhost", 9378)
    await site.start()

    token = await get_token()

    async with aiohttp.ClientSession() as session:
        async with session.post(
            "http://localhost:3978/api/messages",
            json=activity,
            headers={"Authorization": f"Bearer {token}"},
        ) as resp:
            assert resp.status == 200

    await asyncio.sleep(2)  # hope the callback arrives in time
    await runner.cleanup()

    assert any("Hello" in json.dumps(r) for r in collected)

~60 lines to send one message and check for a substring.

With this framework

import pytest
from microsoft_agents.testing import AiohttpScenario, AgentEnvironment

async def init_echo(env: AgentEnvironment):
    @env.agent_application.activity("message")
    async def on_message(context, state):
        await context.send_activity(f"Echo: {context.activity.text}")

scenario = AiohttpScenario(init_agent=init_echo, use_jwt_middleware=False)

@pytest.mark.agent_test(scenario)
class TestEcho:
    async def test_echoes(self, agent_client):
        await agent_client.send("Hello!", wait=0.2)
        agent_client.expect().that_for_any(text="Echo: Hello!")

Or without pytest:

...
async with scenario.client() as client:
    await client.send("Hello!", wait=0.2)
    client.expect().that_for_any(text="Echo: Hello!")

What moved where

Before	After
`get_token()`, env vars, `Authorization` header	Scenario reads `.env` and handles auth
`web.Application()`, `AppRunner`, `TCPSite`, cleanup	Callback server runs inside the scenario
Hand-built activity dict	`ActivityTemplate` fills required fields
`asyncio.sleep(2)`	`wait=0.2` on `send()`
`assert any("Hello" in json.dumps(r) for r in collected)`	`expect().that_for_any(text="Echo: Hello!")`
No record of the conversation	`Transcript` captures every exchange

How it's structured

Scenario.run()  →  ClientFactory  →  AgentClient

Scenario owns lifecycle (servers, auth, teardown).
ClientFactory creates clients — call it multiple times for multi-user tests.
AgentClient is the API you write tests against.

Swap AiohttpScenario for ExternalScenario and your assertions stay the same.

The core has no dependency on pytest. The pytest plugin
(@pytest.mark.agent_test) is an optional layer that wires scenarios into
fixtures.

Accessing agent internals via fixtures

With external testing you can only observe what an agent says.
With AiohttpScenario the agent runs in-process, so the pytest plugin
exposes its internals as fixtures you can inject into any test:

Fixture	Type	What it gives you
`agent_environment`	`AgentEnvironment`	The full environment dataclass (config, app, storage, …)
`agent_application`	`AgentApplication`	The running application — register handlers, inspect middleware
`authorization`	`Authorization`	Auth handler — swap or inspect auth behaviour
`storage`	`Storage`	State storage (default `MemoryStorage`) — read/write agent state directly
`adapter`	`ChannelServiceAdapter`	The channel adapter — useful for proactive-message tests
`connection_manager`	`Connections`	Connection manager — mock or verify external-service calls

Because these are plain pytest fixtures, you can combine them freely:

@pytest.mark.agent_test(scenario)
class TestStatePersistence:
    async def test_remembers_name(self, agent_client, storage):
        await agent_client.send("I want to order a cherry soda.", wait=0.3)
        # reach into storage to verify the agent wrote state correctly
        order_store = await storage.read(["drinks"], target_cls=OrderStore)
        assert order_store.size() > 0
        assert "cherry soda" in order_store

None of this is possible with raw HTTP tests against a deployed agent — you
would need to add debug endpoints or parse logs after the fact.

Complexity the framework reduces: response collection

Without the framework, collecting responses from an agent is not easy. Agents can send replies within the HTTP response body and through separate POSTs to a service URL, depending on the delivery mode of the activity.

The framework here facilitates response handling by unifying it under the Transcript abstraction that is managed by Scenarios and AgentClients to automatically record every exchange that takes place with the target agent.

# Framework handles callback server, response routing, and timing
# after sending the request, wait 2 seconds for any incoming activites.
replies = await agent_client.send(activity, wait=2.0)

Complexity the framework reduces: assertions

Without the framework, asserting on a list of Activity objects requires
defensive coding: null-guards before accessing nested fields, or ""
wrappers to avoid TypeError on None text, and manual iteration.
When the assertion fails, pytest can only tell you the generator returned
False — not which activities were checked or which conditions didn't hold.

Without the framework:

import re

assert any(
    a.type == "message"
    and a.channel_id == "msteams"
    and a.locale == "en-US"
    and "order confirmed" in (a.text or "")          # guard against None
    and re.search(r"Order #\d{6}", a.text or "")     # guard again
    and a.from_property is not None                    # null-check before
    and a.from_property.name == "OrderBot"             #   accessing .name
    and a.conversation is not None                     # null-check before
    and a.conversation.id.startswith("thread-")        #   accessing .id
    for a in replies
)

Failure output:

E   assert False
E    +  where False = any(<generator object test_raw_assertion.<locals>.<genexpr> at 0x...>)

The framework's Expect API handles missing/None fields automatically
(no crash, just a non-match) and uses dot-notation to reach into nested
objects. When the assertion fails it reports per item which fields
didn't match, what was expected, and what was actually there:

With the framework:

import re

agent_client.expect().that_for_any({
    "type": "message",
    "channel_id": "msteams",
    "locale": "en-US",
    "text": lambda x: "order confirmed" in x and re.search(r"Order #\d{6}", x),
    "from.name": "OrderBot",                  # dot-notation — no null-guard needed
    "conversation.id": "~thread-",            # '~' prefix = substring match
})

Failure output:

E   AssertionError: Expectation failed:
E     ✗ Expected at least one item to match, but none did. 0/2 items matched.
E     Details:
E     Item 0: failed on keys ['channel_id', 'text']
E       channel_id:
 E       expected: 'msteams'
E         actual: 'webchat'
E       text:
E         actual: 'Hello there!'
E     Item 1: failed on keys ['from.name']
E       from.name:
E         expected: 'OrderBot'
E         actual: 'HelperBot'

…it tests

…into users/robrandao/integration-overhaul

rodrigobr-msft · 2026-02-06T23:55:42Z

dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_callback_server.py

+            exchange = Exchange(error=str(e), response_at=response_at)
+            response = Response(
+                status=500,
+                text=str(e)


Generally, to fix this problem you should avoid returning raw exception messages (or stack traces) to the client. Instead, log or otherwise record the detailed error server-side, and send the client a generic, non-sensitive message such as “Internal server error”.

For this specific code, the best fix that preserves functionality is:

Keep recording the detailed error message inside the Exchange object (as already done via Exchange(error=str(e), ...)) so that test infrastructure can inspect it.

Change the HTTP 500 response body to a generic message that does not include str(e).

Optionally, include a very generic, static message indicating an error occurred when processing the activity, without exposing implementation details.

Concretely, in AiohttpCallbackServer._handle_request in aiohttp_callback_server.py, replace:

response = Response( status=500, text=str(e) )

with something like:

response = Response( status=500, content_type="application/json", text='{"message": "An internal error occurred while processing the activity."}', )

(or a similarly generic plain-text message). No new imports are required, as Response is already imported and we can use a simple string literal. All existing behavior of recording the error in the transcript remains unchanged; only what is exposed to the external caller changes.

I'm open to having this discussion with the team. I'm not super committed against this, but I also wonder if this is acceptable for a testing tool?

Copilot

Pull request overview

This PR replaces the legacy integration/DDT/assertion tooling with a new scenario-based testing framework, introducing a transport layer (Sender/CallbackServer), a hierarchical Transcript/Exchange model, a fluent Select/Expect API, and a CLI + pytest plugin entry point. It also removes older integration/benchmark utilities and updates packaging to pyproject.toml.

Changes:

Introduces core scenario/transport/transcript abstractions (ExternalScenario, callback server, sender, transcript).
Adds fluent backend types/utilities (Unset, SafeObject, quantifiers, predicates, Select) and new unit tests for readonly/unset.
Adds CLI scaffolding + sample scripts, and removes legacy integration/DDT/assertions/benchmark code.

Reviewed changes

Copilot reviewed 197 out of 274 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
dev/microsoft-agents-testing/tests/core/fluent/backend/types/test_unset.py	Adds unit tests for the new `Unset` sentinel singleton behavior
dev/microsoft-agents-testing/tests/core/fluent/backend/types/test_readonly.py	Adds unit tests for the `Readonly` mixin immutability rules
dev/microsoft-agents-testing/tests/assertions/_common.py	Removes old assertion test fixtures tied to legacy assertions package
dev/microsoft-agents-testing/setup.py	Removes legacy `setup.py` packaging in favor of `pyproject.toml`
dev/microsoft-agents-testing/pytest.ini	Updates pytest warnings filters and adds a marker for intentional failure demos
dev/microsoft-agents-testing/pyproject.toml	Defines runtime deps + adds CLI script and pytest plugin entry point
dev/microsoft-agents-testing/microsoft_agents/testing/utils/populate.py	Removes legacy activity-population helpers
dev/microsoft-agents-testing/microsoft_agents/testing/utils/misc.py	Removes legacy URL/model normalization helpers
dev/microsoft-agents-testing/microsoft_agents/testing/utils/init.py	Removes legacy utils package exports
dev/microsoft-agents-testing/microsoft_agents/testing/utils.py	Adds new convenience helpers for quick sends + scenario resolution
dev/microsoft-agents-testing/microsoft_agents/testing/sdk_config.py	Removes legacy SDKConfig wrapper (replaced by config/env handling elsewhere)
dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/load_ddts.py	Removes legacy data-driven test loader
dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/ddt.py	Removes legacy `ddt()` decorator system
dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/data_driven_test.py	Removes legacy data-driven test runner
dev/microsoft-agents-testing/microsoft_agents/testing/integration/data_driven/init.py	Removes legacy data-driven package exports
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/sample.py	Removes legacy integration sample base class
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/integration.py	Removes legacy pytest fixture-based integration harness
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/environment.py	Removes legacy Environment abstraction for integration tests
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/client/response_client.py	Removes legacy callback response collector
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/client/auto_client.py	Removes unused/placeholder legacy auto client
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/client/agent_client.py	Removes legacy HTTP client implementation
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/client/init.py	Removes legacy integration client exports
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/application_runner.py	Removes legacy thread-based aiohttp runner base
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/aiohttp/aiohttp_runner.py	Removes legacy aiohttp runner implementation
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/aiohttp/aiohttp_environment.py	Removes legacy in-process aiohttp environment wiring
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/aiohttp/init.py	Removes legacy aiohttp integration exports
dev/microsoft-agents-testing/microsoft_agents/testing/integration/core/init.py	Removes legacy integration core exports
dev/microsoft-agents-testing/microsoft_agents/testing/integration/init.py	Removes legacy integration package exports
dev/microsoft-agents-testing/microsoft_agents/testing/core/utils.py	Adds new core helpers (token generation, exchange→activities helpers)
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/transcript/transcript.py	Adds hierarchical Transcript implementation
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/transcript/init.py	Exposes Transcript/Exchange from transport transcript package
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/sender.py	Adds Sender abstraction for outbound activity sending
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/callback_server.py	Adds CallbackServer abstraction for async inbound activities
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_sender.py	Implements Sender using `aiohttp`
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_callback_server.py	Implements CallbackServer using `aiohttp` TestServer
dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/init.py	Exposes transport primitives + aiohttp implementations
dev/microsoft-agents-testing/microsoft_agents/testing/core/scenario.py	Adds `Scenario` base and `ClientFactory` protocol
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/utils.py	Adds normalize/flatten helpers for fluent assertion backend
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/select.py	Adds `Select` fluent selection/filtering API
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/unset.py	Adds `Unset` singleton sentinel type
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/safe_object.py	Adds `SafeObject` for safe chained access returning `Unset`
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/readonly.py	Adds `Readonly` mixin preventing mutation
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/init.py	Exposes fluent backend type utilities
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/quantifier.py	Adds quantifier functions (all/any/none/one/n)
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/model_predicate.py	Adds predicate evaluation plumbing for matching models
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/init.py	Exposes backend primitives for fluent API
dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/init.py	Exposes public fluent API surface
dev/microsoft-agents-testing/microsoft_agents/testing/core/external_scenario.py	Adds `ExternalScenario` for testing hosted agents
dev/microsoft-agents-testing/microsoft_agents/testing/core/config.py	Adds ClientConfig/ScenarioConfig dataclasses
dev/microsoft-agents-testing/microsoft_agents/testing/core/_aiohttp_client_factory.py	Adds aiohttp-based client factory used by scenarios
dev/microsoft-agents-testing/microsoft_agents/testing/core/init.py	Defines core package exports for new framework
dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/basic_scenario.py	Adds built-in CLI scenario for a basic echo agent
dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/auth_scenario.py	Adds built-in CLI scenario for auth handler testing
dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/init.py	Registers predefined scenarios list for CLI
dev/microsoft-agents-testing/microsoft_agents/testing/cli/main.py	Adds CLI entry point + scenario registration hook
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/utils.py	Adds CLI scenario resolution logic
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/output.py	Adds reusable CLI output formatting helper
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/cli_config.py	Adds CLI config loader for .env/process env
dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/init.py	Exposes core CLI helpers/decorators
dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/env.py	Adds CLI `env` diagnostic command
dev/microsoft-agents-testing/microsoft_agents/testing/cli/commands/init.py	Registers available CLI commands
dev/microsoft-agents-testing/microsoft_agents/testing/cli/init.py	Exposes CLI `main()`
dev/microsoft-agents-testing/microsoft_agents/testing/auth/generate_token.py	Removes legacy auth token helpers module
dev/microsoft-agents-testing/microsoft_agents/testing/auth/init.py	Removes legacy auth package exports
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/type_defs.py	Removes legacy assertions type system
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/model_selector.py	Removes legacy selector-based assertion tooling
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/model_assertion.py	Removes legacy ModelAssertion implementation
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/check_model.py	Removes legacy recursive model comparison logic
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/check_field.py	Removes legacy field assertion evaluation
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/assertions.py	Removes legacy assertion entry points
dev/microsoft-agents-testing/microsoft_agents/testing/assertions/init.py	Removes legacy assertions package exports
dev/microsoft-agents-testing/microsoft_agents/testing/init.py	Re-exports new public API surface for scenarios/fluent/transport/CLI helpers
dev/microsoft-agents-testing/docs/samples/test_motivation_assertions.py	Adds a sample test for comparing failure outputs (intentional failures)
dev/microsoft-agents-testing/docs/samples/scenario_registry_demo.py	Adds runnable demo for scenario registry usage
dev/microsoft-agents-testing/docs/samples/quickstart.py	Adds runnable quickstart sample
dev/microsoft-agents-testing/docs/samples/pytest_plugin_usage.py	Adds runnable documentation sample for pytest plugin usage
dev/microsoft-agents-testing/docs/samples/interactive.py	Adds runnable REPL demo using in-process scenario + transcript formatting
dev/microsoft-agents-testing/docs/samples/init.py	Adds index/guide docstring for samples package
dev/microsoft-agents-testing/docs/SAMPLES.md	Adds markdown index describing available samples
dev/microsoft-agents-testing/_manual_test/env.TEMPLATE	Removes legacy manual env template
dev/integration/tests/test_expect_replies.py	Removes legacy integration tests tied to removed harness
dev/integration/tests/quickstart/test_quickstart_sample.py	Removes legacy DDT-driven quickstart integration tests
dev/integration/tests/quickstart/directline/send_hi.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/quickstart/directline/send_hello.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/quickstart/directline/conversation_update.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/quickstart/directline/_parent.yaml	Removes legacy DDT YAML parent defaults
dev/integration/tests/basic_agent/webchat/SendStreamActivity_SendStreamMessage_ExpectStreamResponses.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendInvoke_SendsInvokeActivityToAcExecute_ReturnsValidAdaptiveCardInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendInvoke_SendBasicInvokeActivity_ReceiveInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendInvoke_SelectItem_ReceiveItem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendInvoke_QueryPackage_ReceiveInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendInvoke_QueryLink_ReturnsText.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendExpectedRepliesActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendExpectedRepliesActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_SimulateMessageLoop_ExpectQuestionAboutTimeAndReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_SendsMessageActivityToAcSubmit_ReturnValidResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_SendsHi5_Returns5HiActivities.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_SendsHelloWorld_ReturnsHelloWorld.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_SendHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_RemoveHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_EndConversation_DeleteConversation.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/webchat/SendActivity_ConversationUpdate_ReturnsWelcomeMessage.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/test_basic_agent.py	Removes legacy integration test entry point
dev/integration/tests/basic_agent/msteams/SendStreamActivity_SendStreamMessage_ExpectStreamResponses.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendInvoke_SendsInvokeActivityToAcExecute_ReturnsValidAdaptiveCardInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendInvoke_SendBasicInvokeActivity_ReceiveInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendInvoke_SelectItem_ReceiveItem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendInvoke_QueryPackage_ReceiveInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendInvoke_QueryLink_ReturnsText.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendExpectedRepliesActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendExpectedRepliesActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_StartTeamsMeeting_ExpectMessage.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_SimulateMessageLoop_ExpectQuestionAboutTimeAndReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_SendsMessageActivityToAcSubmit_ReturnValidResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_SendsHi5_Returns5HiActivities.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_SendsHelloWorld_ReturnsHelloWorld.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_SendHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_RemoveHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_ParticipantJoinsTeamMeeting_ExpectMessage.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_EndTeamsMeeting_ExpectMessage.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_EndConversation_DeleteConversation.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_EditMessage_ReceiveUpdate.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/msteams/SendActivity_ConversationUpdate_ReturnsWelcomeMessage.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendStreamActivity_SendStreamMessage_ExpectStreamResponses.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendInvoke_SendsInvokeActivityToAcExecute_ReturnsValidAdaptiveCardInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendInvoke_SendBasicInvokeActivity_ReceiveInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendInvoke_SelectItem_ReceiveItem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendInvoke_QueryPackage_ReceiveInvokeResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendInvoke_QueryLink_ReturnsText.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendExpectedRepliesActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendExpectedRepliesActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_SimulateMessageLoop_ExpectQuestionAboutTimeAndReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_SendsText_ReturnsPoem.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_SendsSeattleTodayWeather_ReturnsWeather.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_SendsMessageActivityToAcSubmit_ReturnValidResponse.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_SendsHi5_Returns5HiActivities.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_SendsHelloWorld_ReturnsHelloWorld.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_SendHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_RemoveHeartMessageReaction_ReturnsMessageReactionHeart.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_EndConversation_DeleteConversation.yaml	Removes legacy DDT YAML test flow
dev/integration/tests/basic_agent/directline/SendActivity_ConversationUpdate_ReturnsWelcomeMessage.yaml	Removes legacy DDT YAML test flow
dev/integration/samples/quickstart_sample.py	Removes legacy integration sample
dev/integration/samples/init.py	Removes legacy integration sample exports
dev/benchmark/src/payload_sender.py	Removes legacy benchmark helper
dev/benchmark/src/output.py	Removes legacy benchmark output helper
dev/benchmark/src/main.py	Removes legacy benchmark CLI
dev/benchmark/src/generate_token.py	Removes legacy benchmark token helper
dev/benchmark/src/executor/thread_executor.py	Removes legacy benchmark thread executor
dev/benchmark/src/executor/executor.py	Removes legacy benchmark executor base
dev/benchmark/src/executor/execution_result.py	Removes legacy benchmark result model
dev/benchmark/src/executor/coroutine_executor.py	Removes legacy benchmark coroutine executor
dev/benchmark/src/executor/init.py	Removes legacy benchmark executor exports
dev/benchmark/src/config.py	Removes legacy benchmark config
dev/benchmark/src/aggregated_results.py	Removes legacy benchmark aggregation
dev/benchmark/requirements.txt	Removes legacy benchmark requirements
dev/benchmark/env.template	Removes legacy benchmark env template
dev/benchmark/README.md	Removes legacy benchmark documentation
dev/README.md	Simplifies dev README to just the editable install command

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/types/safe_object.py

dev/microsoft-agents-testing/microsoft_agents/testing/core/utils.py

dev/microsoft-agents-testing/microsoft_agents/testing/core/_aiohttp_client_factory.py

dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_callback_server.py

dev/microsoft-agents-testing/microsoft_agents/testing/cli/core/utils.py

dev/microsoft-agents-testing/microsoft_agents/testing/cli/scenarios/auth_scenario.py

dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/sender.py

dev/microsoft-agents-testing/microsoft_agents/testing/core/transport/aiohttp_sender.py

dev/microsoft-agents-testing/pyproject.toml

dev/microsoft-agents-testing/microsoft_agents/testing/core/fluent/backend/quantifier.py

rodrigobr-msft added 30 commits January 31, 2026 10:10

Revamping assertion system

a1b3848

Revamped check system

75d2cbe

Adding more test cases

7bb29a1

Draft implementation of overhauled Checking engine with supporting un…

68086cd

…it tests

Added doc

e724353

Adding back old work

b040e3f

Redoing testing decorator flow

da0fc77

Removing unused files

d3785f8

Updating utils modules

c4e4ad7

Reworked response collection

aa19061

Improved separation of concerns for AgentClient and SenderClient

f1b2901

Removing unused imports

c9504a1

Data and model utils sophistication

134869f

Moving over token generation work

8871eeb

Adding fn.py integration for Check

6c8af5e

Changing Check quantifier selection methods to be properties

7cb736b

Removed functional utils, saving for later

7e65065

Adding updated utils and check tests

01b39cd

Added copyright comments and created tests for AgentClient subcomponents

c6cfdac

Fixed bug in ModelTemplate

d756032

Progress in testing AiohttpAgentScenario

f621053

Bug fix for aiohttp scenario and provided more tests

463467e

Added agent_test tests

2edbdce

Adding underscore utilities

d4ee441

Adding more tests and documenting limitations

2d48688

Updating docs

e7233f8

Update to Check API

2fecf56

Moving things around

baf3d36

Implement CLI commands

3a6bfb3

Cleaning up CLI imports/exports

9a83b2b

rodrigobr-msft added 22 commits January 31, 2026 10:10

Updated to pyproject.toml

955fdaa

Reorganizing configuration

3a479b5

Adding docstrings

5e3e912

Pytest plugin integration

29677a0

Adding transcript formatter variants

99761bc

Adding streaming support

161a5b9

Porting old integration tests

9be27e4

Redoing test cases

89d7a7a

Scenario registry and cli reorg

bd52e23

Agent resolution in CLI and scenario registry

bb28b4d

Chat and post CLI commands

f20acd5

Adding scenario run

375c68f

Env command

8d16299

Improving docs

db222a3

More tests and documentation

5385d2a

Addressed test cases

e765887

Adding more documentation via markdown files

dabb93c

Cleaning up files

667a79d

Merge branch 'main' of https://github.com/microsoft/Agents-for-python …

27fc2eb

…into users/robrandao/integration-overhaul

Tweaks to pyproject.toml and docs

5aca7cc

Revising MOTIVATION.md doc

3f1a928

Revising samples

129c0de

Copilot AI review requested due to automatic review settings February 6, 2026 23:50

rodrigobr-msft marked this pull request as ready for review February 6, 2026 23:51

rodrigobr-msft requested a review from a team as a code owner February 6, 2026 23:51

github-advanced-security bot found potential problems Feb 6, 2026

View reviewed changes

Copilot AI reviewed Feb 6, 2026

View reviewed changes

Change to SafeObject.__getitem__

d8230fa

rodrigobr-msft closed this Feb 9, 2026

rodrigobr-msft reopened this Feb 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Users/robrandao/integration overhaul #315

Users/robrandao/integration overhaul #315

Uh oh!

rodrigobr-msft commented Feb 6, 2026

Uh oh!

Check warning

Copilot Autofix

rodrigobr-msft Feb 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Users/robrandao/integration overhaul #315

Are you sure you want to change the base?

Users/robrandao/integration overhaul #315

Uh oh!

Conversation

rodrigobr-msft commented Feb 6, 2026

Motivation

Without this framework

With this framework

What moved where

How it's structured

Accessing agent internals via fixtures

Complexity the framework reduces: response collection

Complexity the framework reduces: assertions

Uh oh!

Check warning

Uh oh!

Copilot Autofix

rodrigobr-msft Feb 6, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant