feat(cosmos/workloads): add performance metrics collection for DR drill testing

[tvaron3]

Unified Workload with Performance Metrics for Cosmos DB DR Drill Testing

Summary

Consolidates 8 separate workload scripts into a single workload.py controlled entirely by environment variables, and adds a performance metrics
collection layer that reports latency, throughput, errors, and resource
utilization to a Cosmos DB results account — enabling Grafana dashboards and ADX-powered analysis for DR drill testing.

Motivation

Previously, testing different workload configurations (read-only, write-only, proxy, sync, etc.) required copying and modifying entire workload files.
Adding observability meant manually instrumenting each file. This PR solves
both problems:

1. One workload, many configurations — env vars control operations, proxy, sync, and client behavior
2. Built-in metrics — every operation is timed and reported automatically

Architecture

workload.py ├── WORKLOAD_OPERATIONS → read, write, query (comma-separated) ├── WORKLOAD_USE_PROXY → route through Envoy proxy ├── WORKLOAD_USE_SYNC → sync
vs async client ├── WORKLOAD_SKIP_CLOSE → simulate applications that
don't close the client └── PerfReporter (daemon thread, configurable interval) ├── HdrHistogram per operation (p50/p90/p99, O(1) record) ├── psutil for
CPU/memory metrics ├── Error status code + sub-status code tracking └──
Upserts PerfResult docs to results Cosmos DB account

New Files

File	Description
workload.py	Single unified workload — replaces all 8 operation-specific files
perf_stats.py	Per-operation latency tracking via HdrHistogram — O(1) record, O(1) percentile,
fixed 40KB memory per histogram
perf_reporter.py	Background daemon thread that upserts PerfResult documents to a results Cosmos DB account at a configurable interval
perf_config.py	All metrics configuration from environment variables

Modified Files

File	Change
workload_configs.py	All SDK configs now driven by environment variables (no manual editing)
workload_utils.py	Added timed operation wrappers using time.perf_counter_ns() with error status code capture

Deleted Files

r_workload.py, w_workload.py, r_proxy_workload.py, w_proxy_workload.py, r_w_q_workload.py, r_w_q_proxy_workload.py, r_w_q_sync_workload.py,
r_w_q_with_incorrect_client_workload.py, run_workloads.sh, dev.md

All replaced by workload.py with environment variable configuration. Infrastructure/orchestration scripts (run_workloads.sh, dev.md) moved to the
cosmos-sdk-copilot-toolkit repo.

Environment Variables

Workload behavior:

Variable	Default	Description
WORKLOAD_OPERATIONS	read,write,query	Comma-separated operations to run
WORKLOAD_USE_PROXY	false	Route through Envoy proxy
WORKLOAD_USE_SYNC	false	Use sync client instead of async
WORKLOAD_SKIP_CLOSE	false	Don't close client (simulate incorrect usage)

Cosmos DB client:

Variable	Default	Description
COSMOS_URI	(required)	Workload account endpoint
COSMOS_PREFERRED_LOCATIONS	""	Comma-separated preferred regions
COSMOS_CLIENT_EXCLUDED_LOCATIONS	""	Comma-separated excluded regions
COSMOS_USE_MULTIPLE_WRITABLE_LOCATIONS	false	Enable multi-region writes
AZURE_COSMOS_ENABLE_CIRCUIT_BREAKER	false	Per-partition circuit breaker
COSMOS_CONCURRENT_REQUESTS	100	Concurrent operations per process
COSMOS_THROUGHPUT	1000000	Container provisioned throughput

Metrics reporting:

Variable	Default	Description
RESULTS_COSMOS_URI	""	Results account endpoint (enables metrics when set)
PERF_ENABLED	true	Toggle metrics collection
PERF_REPORT_INTERVAL	300	Seconds between metric reports

PerfResult Document Schema

Each document matches the Rust SDK perf tool schema, enabling
both SDKs to feed the same ADX → Grafana pipeline:

{
  "operation": "ReadItem",
  "count": 12345,
  "errors": 0,
  "min_ms": 1.2, "max_ms": 50.3, "mean_ms": 5.4,
  "p50_ms": 4.8, "p90_ms": 8.2, "p99_ms": 15.1,
  "cpu_percent": 45.2,
  "memory_bytes": 104857600,
  "system_cpu_percent": 62.1,
  "sdk_language": "python",
  "sdk_version": "4.15.0",
  "config_concurrency": 100,
  "config_ppcb_enabled": false
}

Usage Examples

# All operations, direct connection
WORKLOAD_OPERATIONS=read,write,query python3 workload.py

# Read-only via Envoy proxy
WORKLOAD_OPERATIONS=read WORKLOAD_USE_PROXY=true python3 workload.py

# Write-only with metrics reporting
WORKLOAD_OPERATIONS=write \
  RESULTS_COSMOS_URI=https://my-results.documents.azure.com:443/ \
  python3 workload.py

# Simulate unclosed client
WORKLOAD_SKIP_CLOSE=true python3 workload.py

# Scale to 5 processes
for i in $(seq 5); do
  WORKLOAD_OPERATIONS=read,write,query nohup python3 workload.py &
done

[Copilot]

Pull request overview

This PR modernizes the Cosmos DB DR drill workload harness by consolidating multiple per-scenario scripts into a single environment-variable-driven workload runner, and adds an optional performance metrics collection/reporting layer that can upsert results to a separate Cosmos DB account for dashboarding and analysis.

Changes:

• Consolidates prior workload entrypoints into a single workload.py controlled by environment variables.
• Adds performance statistics collection (per-operation latency + errors) and a background reporter that periodically upserts PerfResult documents.
• Refactors workload configuration and operation helpers to be environment-variable-driven and to support timed operation wrappers.

Reviewed changes

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sdk/cosmos/azure-cosmos/tests/workloads/workload.py	New unified workload entrypoint (sync/async + proxy + perf hooks).
sdk/cosmos/azure-cosmos/tests/workloads/workload_utils.py	Adds timed operation wrappers and Cosmos error status/substatus extraction for perf stats.
sdk/cosmos/azure-cosmos/tests/workloads/workload_configs.py	Moves workload/client configuration to environment variables with parsing/validation.
sdk/cosmos/azure-cosmos/tests/workloads/perf_stats.py	New thread-safe per-operation latency histograms + error aggregation (HdrHistogram-backed).
sdk/cosmos/azure-cosmos/tests/workloads/perf_reporter.py	New daemon-thread reporter to drain stats and upsert PerfResult/Error docs to a results Cosmos account.
sdk/cosmos/azure-cosmos/tests/workloads/perf_config.py	New perf reporter configuration builder from environment variables (includes git SHA + defaults).
sdk/cosmos/azure-cosmos/cspell.json	Adds workload/perf-specific dictionary exceptions for spell checking.
sdk/cosmos/azure-cosmos/tests/workloads/w_workload.py	Removes legacy write-only workload script (replaced by workload.py).
sdk/cosmos/azure-cosmos/tests/workloads/w_proxy_workload.py	Removes legacy write-via-proxy workload script (replaced by workload.py).
sdk/cosmos/azure-cosmos/tests/workloads/r_workload.py	Removes legacy read/query workload script (replaced by workload.py).
sdk/cosmos/azure-cosmos/tests/workloads/r_w_q_workload.py	Removes legacy mixed workload script (replaced by workload.py).
sdk/cosmos/azure-cosmos/tests/workloads/r_w_q_with_incorrect_client_workload.py	Removes legacy “incorrect client usage” script (replaced by WORKLOAD_SKIP_CLOSE).
sdk/cosmos/azure-cosmos/tests/workloads/r_w_q_sync_workload.py	Removes legacy sync mixed workload script (replaced by workload.py).
sdk/cosmos/azure-cosmos/tests/workloads/r_w_q_proxy_workload.py	Removes legacy mixed workload + proxy script (replaced by workload.py).
sdk/cosmos/azure-cosmos/tests/workloads/r_proxy_workload.py	Removes legacy read/query-via-proxy workload script (replaced by workload.py).
sdk/cosmos/azure-cosmos/tests/workloads/setup_env.sh	Removes local environment bootstrap script (moved out of repo per PR description).
sdk/cosmos/azure-cosmos/tests/workloads/run_workloads.sh	Removes legacy multi-script launcher (superseded by env-var-driven workload).
sdk/cosmos/azure-cosmos/tests/workloads/dev.md	Removes legacy local runbook doc (moved out of repo per PR description).