Skip to content

Improve end-to-end test reliability, single-run execution, and configurable performance profiles#864

Merged
tjementum merged 16 commits intomainfrom
pp-1090-improve-end-to-end-test-cli
Apr 2, 2026
Merged

Improve end-to-end test reliability, single-run execution, and configurable performance profiles#864
tjementum merged 16 commits intomainfrom
pp-1090-improve-end-to-end-test-cli

Conversation

@tjementum
Copy link
Copy Markdown
Member

@tjementum tjementum commented Apr 2, 2026

Summary & Motivation

Overhaul the end-to-end test CLI (pp e2e) to run all self-contained systems in a single Playwright invocation, default to reliable settings, and support machine-specific performance tuning.

  • Run end-to-end tests for all self-contained systems in a single Playwright invocation instead of separate runs, significantly reducing total execution time by eliminating redundant Playwright startup, browser launch, and authentication overhead per system (most noticeable when running smoke tests across all systems)
  • Make wait-for-Aspire the default behavior with a 3-minute retry window using SSL-tolerant HTTP checks, replacing the previous opt-in flag with --no-wait-for-aspire to skip
  • Default to chromium browser instead of running all browsers, significantly reducing test time for local development
  • Add configurable performance profiles (pp e2e config) with interactive setup for workers, assertion timeout, and test timeout, stored in .workspace/developer-cli/end-to-end-tests/e2e-defaults.json
  • Add retries, workers, stop-on-first-failure, repeat-each, and last-failed parameters to both the CLI and EndToEnd MCP tool
  • Suppress all console output in quiet mode (used by MCP tool) including Playwright browser installation
  • Increase default Playwright test timeout to 3 minutes and assertion timeout to 20 seconds, both configurable per machine via pp e2e config
  • Show test failures in red and prevent stderr deadlock in the test runner
  • Move CSP nonce checks from a separate spec file into the global UI flows test to reduce test file count
  • Initialize .workspace as a git sub-repository on every CLI invocation to protect developer-local state from git clean -xdf
  • Add fix-e2e-tests skill for systematically diagnosing and fixing failing end-to-end tests, replacing the unused update-flaky-tests skill
  • Skip subscription e2e tests when Stripe is not configured, and remove billing-specific navigation steps from the federated navigation test so the full suite passes in environments without Stripe

General housekeeping:

  • Generalize .tsbuildinfo gitignore pattern to match all projects instead of only account
  • Fix ACR resource group name check in deploy command after adding -global to the shared resource group
  • Add generated API types to turbo dev:setup outputs to prevent stale cache
  • Filter AppHost process check by source folder to allow multiple Aspire instances on the same machine

Checklist

  • I have added tests, or done manual regression tests
  • I have updated the documentation, if necessary

@tjementum tjementum requested a review from a team as a code owner April 2, 2026 17:51
@linear
Copy link
Copy Markdown

linear bot commented Apr 2, 2026

@tjementum tjementum self-assigned this Apr 2, 2026
@tjementum tjementum added the Enhancement New feature or request label Apr 2, 2026
@sonarqubecloud
Copy link
Copy Markdown

sonarqubecloud bot commented Apr 2, 2026

@tjementum tjementum merged commit 310887b into main Apr 2, 2026
35 of 40 checks passed
@tjementum tjementum deleted the pp-1090-improve-end-to-end-test-cli branch April 2, 2026 18:04
@github-project-automation github-project-automation bot moved this to ✅ Done in Kanban board Apr 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Enhancement New feature or request

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

1 participant