Open
Conversation
- Seed reusable perf fixtures for server scenarios - Add perf provider adapter and web perf tests - Capture artifacts for virtualization and websocket runs
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
github-actions
bot
added
size:XXL
- Refresh virtualization and websocket perf snapshots - Keep perf baselines in sync with latest run
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
Contributor
ApprovabilityVerdict: Needs human review 1 blocking correctness issue found. This PR introduces a comprehensive performance regression test harness with ~3000+ lines of new code, modifies production server startup to conditionally load perf provider layers, and has an open review comment identifying a potential logic bug in the env var handling. The scope and the unresolved issue warrant human review. You can customize Macroscope's approvability policy. Learn more. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 3 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Reset creates duplicate animation frame loops corrupting metrics
- Stored the rAF handle and added cancelAnimationFrame before starting a new loop in reset() to prevent duplicate concurrent animation frame chains.
- ✅ Fixed: Generated test artifacts accidentally committed to repository
- Removed the committed artifact JSON files via git rm --cached and added artifacts/ to .gitignore to prevent future accidental commits.
- ✅ Fixed: Percentile returns null for zero target value
- Added Math.max(0, ...) to clamp the computed index so that target=0 returns sorted[0] (the minimum) instead of sorted[-1] (undefined).
Or push these changes by commenting:
@cursor push 0e9b05e984
Preview (0e9b05e984)
diff --git a/.gitignore b/.gitignore
--- a/.gitignore
+++ b/.gitignore
@@ -21,3 +21,4 @@
.vitest-*
__screenshots__/
.tanstack
+artifacts/
diff --git a/artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json b/artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json
deleted file mode 100644
--- a/artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json
+++ /dev/null
@@ -1,141 +1,0 @@
-{
- "suite": "virtualization",
- "scenarioId": "large_threads",
- "startedAt": "2026-04-01T20:57:00.745Z",
- "completedAt": "2026-04-01T20:57:01.524Z",
- "thresholds": {
- "maxMountedTimelineRows": 140,
- "threadSwitchP50Ms": 250,
- "threadSwitchP95Ms": 500,
- "maxLongTaskMs": 120,
- "maxRafGapMs": 120,
- "burstCompletionMs": 5000,
- "longTasksOver50MsMax": 2
- },
- "summary": {
- "maxMountedTimelineRows": 23,
- "threadSwitchP50Ms": 49.5,
- "threadSwitchP95Ms": 117,
- "maxLongTaskMs": 0,
- "longTasksOver50Ms": 0,
- "maxRafGapMs": 25,
- "burstCompletionMs": null
- },
- "browserMetrics": {
- "actions": [
- {
- "name": "thread-switch-warmup-a",
- "durationMs": 117,
- "startedAtMs": 591,
- "endedAtMs": 708
- },
- {
- "name": "thread-switch-1",
- "durationMs": 46.5,
- "startedAtMs": 709.7999999523163,
- "endedAtMs": 756.2999999523163
- },
- {
- "name": "thread-switch-2",
- "durationMs": 49.5,
- "startedAtMs": 787.6000000238419,
- "endedAtMs": 837.1000000238419
- },
- {
- "name": "thread-switch-3",
- "durationMs": 70.69999992847443,
- "startedAtMs": 840.7000000476837,
- "endedAtMs": 911.3999999761581
- },
- {
- "name": "thread-switch-4",
- "durationMs": 45.89999997615814,
- "startedAtMs": 916,
- "endedAtMs": 961.8999999761581
- },
- {
- "name": "thread-switch-5",
- "durationMs": 46.699999928474426,
- "startedAtMs": 964.6000000238419,
- "endedAtMs": 1011.2999999523163
- },
- {
- "name": "thread-switch-6",
- "durationMs": 49.89999997615814,
- "startedAtMs": 1047.8999999761581,
- "endedAtMs": 1097.7999999523163
- }
- ],
- "longTasks": [],
- "rafGapsMs": [
- 0, 8, 0, 17, 0, 7.300000000000068, 0, 8.799999999999955, 0, 8.200000000000045, 0,
- 9.099999999999909, 0, 7.7000000000000455, 0, 8.399999999999977, 0, 8.800000000000068, 0, 16,
- 0, 8.399999999999977, 0, 9, 0, 8.299999999999955, 0, 7.600000000000023, 0, 17.100000000000023,
- 0, 8.600000000000023, 0, 8.299999999999955, 0, 16.200000000000045, 0, 7.899999999999977, 0,
- 9.299999999999955, 0, 7.399999999999977, 0, 9.200000000000045, 0, 16.700000000000045, 0, 8, 0,
- 8.299999999999955, 0, 7.899999999999977, 0, 8.700000000000045, 0, 17.100000000000023, 0,
- 8.199999999999932, 0, 16, 0, 16.600000000000023, 0, 8.399999999999977, 0, 8.899999999999977,
- 0, 7.800000000000068, 0, 17.399999999999977, 0, 8.299999999999955, 0, 8.300000000000068, 0,
- 7.699999999999932, 0, 16.700000000000045, 0, 8.100000000000023, 0, 17.399999999999977, 0,
- 7.7999999999999545, 0, 8.600000000000136, 0, 8.199999999999818, 0, 8.900000000000091, 0,
- 8.099999999999909, 0, 16.200000000000045, 0, 9.100000000000136, 0, 7.399999999999864, 0, 8.5,
- 0, 9, 0, 25, 0, 7.600000000000136, 0, 9.199999999999818, 0
- ],
- "mountedRowSamples": [
- {
- "label": "heavy-a-open",
- "count": 17,
- "capturedAtMs": 708.5
- },
- {
- "label": "thread-switch-1-rows",
- "count": 17,
- "capturedAtMs": 782
- },
- {
- "label": "thread-switch-2-rows",
- "count": 17,
- "capturedAtMs": 837.7000000476837
- },
- {
- "label": "thread-switch-3-rows",
- "count": 17,
- "capturedAtMs": 914.2000000476837
- },
- {
- "label": "thread-switch-4-rows",
- "count": 17,
- "capturedAtMs": 962.5
- },
- {
- "label": "thread-switch-5-rows",
- "count": 17,
- "capturedAtMs": 1034.1000000238419
- },
- {
- "label": "thread-switch-6-rows",
- "count": 17,
- "capturedAtMs": 1098.3999999761581
- },
- {
- "label": "scroll-start",
- "count": 17,
- "capturedAtMs": 1113.3999999761581
- },
- {
- "label": "scroll-top",
- "count": 23,
- "capturedAtMs": 1148.3999999761581
- },
- {
- "label": "scroll-bottom",
- "count": 17,
- "capturedAtMs": 1164.2000000476837
- }
- ]
- },
- "serverMetrics": null,
- "metadata": {
- "heavyThreadMessageCount": 2000
- }
-}
\ No newline at end of file
diff --git a/artifacts/perf/websocket-application-burst_base-1775077022750/websocket-application-dense_assistant_stream.json b/artifacts/perf/websocket-application-burst_base-1775077022750/websocket-application-dense_assistant_stream.json
deleted file mode 100644
--- a/artifacts/perf/websocket-application-burst_base-1775077022750/websocket-application-dense_assistant_stream.json
+++ /dev/null
@@ -1,139 +1,0 @@
-{
- "suite": "websocket-application",
- "scenarioId": "dense_assistant_stream",
- "startedAt": "2026-04-01T20:57:03.910Z",
- "completedAt": "2026-04-01T20:57:07.151Z",
- "thresholds": {
- "maxMountedTimelineRows": 140,
- "threadSwitchP50Ms": 250,
- "threadSwitchP95Ms": 500,
- "maxLongTaskMs": 120,
- "maxRafGapMs": 120,
- "burstCompletionMs": 5000,
- "longTasksOver50MsMax": 2
- },
- "summary": {
- "maxMountedTimelineRows": 0,
- "threadSwitchP50Ms": 51.199999928474426,
- "threadSwitchP95Ms": 52.39999997615814,
- "maxLongTaskMs": 0,
- "longTasksOver50Ms": 0,
- "maxRafGapMs": 16.200000000000045,
- "burstCompletionMs": 3135
- },
- "browserMetrics": {
- "actions": [
- {
- "name": "thread-switch-burst-nav",
- "durationMs": 51.199999928474426,
- "startedAtMs": 1348.3999999761581,
- "endedAtMs": 1399.5999999046326
- },
- {
- "name": "thread-switch-burst-return",
- "durationMs": 52.39999997615814,
- "startedAtMs": 1401.1999999284744,
- "endedAtMs": 1453.5999999046326
- },
- {
- "name": "burst-completion",
- "durationMs": 3135,
- "startedAtMs": 332.2999999523163,
- "endedAtMs": 3467.2999999523163
- }
- ],
- "longTasks": [],
- "rafGapsMs": [
- 0, 7.899999999999977, 0, 9, 0, 7.300000000000011, 0, 8.899999999999977, 0, 7.800000000000011,
- 0, 9.300000000000011, 0, 8.100000000000023, 0, 7.699999999999989, 0, 9.199999999999989, 0,
- 7.800000000000011, 0, 7.899999999999977, 0, 9.300000000000011, 0, 7.300000000000011, 0,
- 9.099999999999966, 0, 7.7000000000000455, 0, 8.199999999999989, 0, 8.399999999999977, 0,
- 8.300000000000011, 0, 8.399999999999977, 0, 8.5, 0, 9.100000000000023, 0, 7.7000000000000455,
- 0, 8.199999999999932, 0, 8.600000000000023, 0, 8.799999999999955, 0, 7.7000000000000455, 0,
- 8.200000000000045, 0, 8.399999999999977, 0, 9.100000000000023, 0, 7.5, 0, 8.199999999999932,
- 0, 9.300000000000068, 0, 7.399999999999977, 0, 8.600000000000023, 0, 8.399999999999977, 0,
- 8.899999999999977, 0, 8.299999999999955, 0, 7.800000000000068, 0, 8, 0, 8.399999999999977, 0,
- 8.799999999999955, 0, 7.7000000000000455, 0, 9.299999999999955, 0, 7.400000000000091, 0,
- 9.299999999999955, 0, 8.299999999999955, 0, 7.7000000000000455, 0, 8.100000000000023, 0,
- 8.899999999999977, 0, 8.700000000000045, 0, 7.2999999999999545, 0, 9.399999999999977, 0,
- 8.200000000000045, 0, 8.100000000000023, 0, 7.699999999999932, 0, 9.300000000000068, 0,
- 8.199999999999932, 0, 8.300000000000068, 0, 7.699999999999932, 0, 8.700000000000045, 0,
- 8.699999999999932, 0, 7.600000000000023, 0, 8.100000000000023, 0, 8.399999999999977, 0,
- 8.700000000000045, 0, 8, 0, 8.699999999999932, 0, 8.200000000000045, 0, 8.799999999999955, 0,
- 8.5, 0, 7.800000000000068, 0, 8.5, 0, 8.600000000000023, 0, 8.399999999999977, 0,
- 7.899999999999977, 0, 8.100000000000023, 0, 8.100000000000023, 0, 8.399999999999977, 0,
- 9.299999999999955, 0, 8.100000000000023, 0, 7.5, 0, 9.200000000000045, 0, 8.5, 0,
- 7.899999999999864, 0, 8.800000000000182, 0, 8.199999999999818, 0, 7.800000000000182, 0, 8.5,
- 0, 8.199999999999818, 0, 8.900000000000091, 0, 8.299999999999955, 0, 7.900000000000091, 0,
- 7.899999999999864, 0, 9.300000000000182, 0, 7.599999999999909, 0, 8.400000000000091, 0, 8, 0,
- 8.399999999999864, 0, 9.200000000000045, 0, 7.400000000000091, 0, 9.299999999999955, 0,
- 8.200000000000045, 0, 7.5, 0, 9.299999999999955, 0, 8.099999999999909, 0, 7.600000000000136,
- 0, 9.299999999999955, 0, 8.200000000000045, 0, 8.399999999999864, 0, 7.5, 0,
- 8.400000000000091, 0, 8.599999999999909, 0, 7.7999999999999545, 0, 8.5, 0, 9.300000000000182,
- 0, 7.2999999999999545, 0, 9.399999999999864, 0, 7.5, 0, 8.100000000000136, 0, 8.5, 0,
- 8.099999999999909, 0, 9.100000000000136, 0, 8.599999999999909, 0, 7.5, 0, 9.099999999999909,
- 0, 7.600000000000136, 0, 8.399999999999864, 0, 8.100000000000136, 0, 8.399999999999864, 0,
- 8.200000000000045, 0, 9.299999999999955, 0, 8, 0, 16.200000000000045, 0, 9.200000000000045, 0,
- 8.299999999999955, 0, 8.299999999999955, 0, 8.400000000000091, 0, 7.900000000000091, 0,
- 8.599999999999909, 0, 8.5, 0, 8.099999999999909, 0, 7.7000000000000455, 0, 8.700000000000045,
- 0, 8.5, 0, 8.5, 0, 7.599999999999909, 0, 8.900000000000091, 0, 8.599999999999909, 0,
- 8.400000000000091, 0, 8.299999999999955, 0, 7.7999999999999545, 0, 8.600000000000136, 0,
- 8.200000000000045, 0, 8.799999999999955, 0, 7.599999999999909, 0, 8.100000000000136, 0,
- 8.599999999999909, 0, 8.299999999999955, 0, 8.100000000000136, 0, 8.599999999999909, 0,
- 9.099999999999909, 0, 8.200000000000045, 0, 7.400000000000091, 0, 8.899999999999864, 0, 8.5,
- 0, 8.5, 0, 7.900000000000091, 0, 8.799999999999955, 0, 7.400000000000091, 0,
- 9.299999999999955, 0, 7.400000000000091, 0, 9.299999999999955, 0, 8.299999999999955, 0,
- 8.299999999999955, 0, 7.400000000000091, 0, 8.899999999999864, 0, 8.800000000000182, 0,
- 7.599999999999909, 0, 8.099999999999909, 0, 8.200000000000045, 0, 9.400000000000091, 0,
- 7.7000000000000455, 0, 8.899999999999864, 0, 7.400000000000091, 0, 8.399999999999864, 0,
- 8.900000000000091, 0, 8, 0, 8.5, 0, 8.799999999999955, 0, 7.7000000000000455, 0,
- 8.200000000000045, 0, 8.700000000000045, 0, 7.899999999999864, 0, 8.799999999999955, 0,
- 8.400000000000091, 0, 8.099999999999909, 0, 8.200000000000045, 0, 8.200000000000045, 0,
- 8.200000000000045, 0, 8.799999999999955, 0, 7.7999999999999545, 0, 8.600000000000136, 0,
- 8.199999999999818, 0, 9.200000000000045, 0, 7.500000000000227, 0, 8.199999999999818, 0,
- 9.099999999999909, 0, 7.700000000000273, 0, 8.899999999999636, 0, 7.600000000000364, 0,
- 8.399999999999636, 0, 9.300000000000182, 0, 7.400000000000091, 0, 8.299999999999727, 0,
- 9.300000000000182, 0, 8.300000000000182, 0, 7.799999999999727, 0, 8.900000000000091, 0,
- 7.699999999999818, 0, 8.800000000000182, 0, 8.300000000000182, 0, 7.599999999999909, 0,
- 8.799999999999727, 0, 7.900000000000091, 0, 9.099999999999909, 0, 8.200000000000273, 0, 8, 0,
- 8.799999999999727, 0, 7.600000000000364, 0, 9.299999999999727, 0, 8.200000000000273, 0,
- 7.699999999999818, 0, 8, 0, 8.599999999999909, 0, 9.099999999999909, 0, 7.300000000000182, 0,
- 8.900000000000091, 0, 8.799999999999727, 0, 8.100000000000364, 0, 7.899999999999636, 0, 8, 0,
- 8.400000000000091, 0, 9.099999999999909, 0, 7.5, 0, 8.800000000000182, 0, 7.900000000000091,
- 0, 9.299999999999727, 0, 7.5, 0, 9.100000000000364, 0, 8.099999999999909, 0,
- 7.799999999999727, 0, 8.700000000000273, 0, 7.799999999999727, 0, 9.300000000000182, 0,
- 8.300000000000182, 0, 8, 0, 8.699999999999818, 0, 7.599999999999909, 0, 9.099999999999909, 0,
- 8.200000000000273, 0, 7.599999999999909, 0, 9.199999999999818, 0, 8.200000000000273, 0,
- 8.400000000000091, 0, 8.299999999999727, 0, 8.200000000000273, 0, 8.5, 0, 8.299999999999727,
- 0, 8.400000000000091, 0, 8.300000000000182, 0, 7.5, 0, 8.5, 0, 8.099999999999909, 0,
- 9.299999999999727, 0, 7.400000000000091, 0, 9, 0, 8, 0, 8.900000000000091, 0,
- 8.300000000000182, 0, 8.299999999999727, 0, 7.5, 0, 8.599999999999909, 0, 8, 0,
- 9.300000000000182, 0, 7.400000000000091, 0, 9.299999999999727, 0, 8.300000000000182, 0,
- 7.599999999999909, 0, 8.800000000000182, 0, 8.099999999999909, 0, 8, 0, 8.800000000000182, 0,
- 8.5, 0, 8.5, 0, 8.099999999999909, 0, 8.5, 0, 8, 0, 8.199999999999818, 0, 7.900000000000091,
- 0, 8.599999999999909, 0, 8.599999999999909, 0, 8, 0, 9.100000000000364, 0, 8.199999999999818,
- 0, 8.5, 0, 8, 0, 7.800000000000182, 0, 8.899999999999636, 0, 7.800000000000182, 0,
- 9.099999999999909, 0, 8.300000000000182, 0, 7.799999999999727, 0, 9, 0, 8.300000000000182, 0,
- 7.5, 0, 9.199999999999818, 0, 7.400000000000091, 0, 9.300000000000182, 0, 8, 0,
- 7.799999999999727, 0, 9.200000000000273, 0, 8, 0, 7.599999999999909, 0, 8.400000000000091, 0,
- 9.299999999999727, 0, 7.300000000000182, 0, 8.5, 0, 9.199999999999818, 0, 8.200000000000273,
- 0, 8.400000000000091, 0, 7.5, 0, 9.199999999999818, 0, 8.300000000000182, 0,
- 7.399999999999636, 0, 8.400000000000091, 0, 9, 0, 8.599999999999909, 0, 8.200000000000273, 0,
- 7.5, 0, 9.299999999999727, 0, 8.200000000000273, 0, 8.400000000000091, 0, 7.399999999999636,
- 0, 8.300000000000182, 0, 9.400000000000091, 0, 8, 0, 8.5, 0, 8.099999999999909, 0, 8, 0, 8, 0,
- 8.900000000000091, 0, 7.900000000000091, 0, 8.299999999999727, 0, 8.300000000000182, 0, 8.5,
- 0, 8.699999999999818, 0, 8.099999999999909, 0, 9, 0, 8.300000000000182, 0, 7.400000000000091,
- 0, 9.099999999999909, 0, 7.800000000000182, 0, 9.099999999999909, 0, 8.199999999999818, 0,
- 8.400000000000091, 0, 8.300000000000182, 0, 7.899999999999636, 0, 8.700000000000273, 0, 8.5,
- 0, 7.400000000000091, 0, 8.299999999999727, 0, 8.300000000000182, 0, 9.299999999999727, 0,
- 8.300000000000182, 0, 8.199999999999818, 0
- ],
- "mountedRowSamples": []
- },
- "serverMetrics": null,
- "metadata": {
- "burstSeedThreadId": "perf-thread-burst",
- "navigationThreadId": "perf-thread-light-01",
- "sentinelText": "PERF_STREAM_SENTINEL:dense_assistant_stream:completed"
- }
-}
\ No newline at end of file
diff --git a/test/perf/support/artifact.ts b/test/perf/support/artifact.ts
--- a/test/perf/support/artifact.ts
+++ b/test/perf/support/artifact.ts
@@ -68,7 +68,10 @@
}
const sorted = values.toSorted((left, right) => left - right);
const clampedTarget = Math.min(Math.max(target, 0), 1);
- const index = Math.min(sorted.length - 1, Math.ceil(sorted.length * clampedTarget) - 1);
+ const index = Math.min(
+ sorted.length - 1,
+ Math.max(0, Math.ceil(sorted.length * clampedTarget) - 1),
+ );
return sorted[index] ?? null;
}
diff --git a/test/perf/support/browserMetrics.ts b/test/perf/support/browserMetrics.ts
--- a/test/perf/support/browserMetrics.ts
+++ b/test/perf/support/browserMetrics.ts
@@ -30,15 +30,16 @@
const rafGapsMs: number[] = [];
const mountedRowSamples: Array<BrowserPerfMetrics["mountedRowSamples"][number]> = [];
let previousAnimationFrameTs = 0;
+ let rafHandle = 0;
const animationFrameLoop = (timestampMs: number) => {
if (previousAnimationFrameTs > 0) {
rafGapsMs.push(timestampMs - previousAnimationFrameTs);
}
previousAnimationFrameTs = timestampMs;
- window.requestAnimationFrame(animationFrameLoop);
+ rafHandle = window.requestAnimationFrame(animationFrameLoop);
};
- window.requestAnimationFrame(animationFrameLoop);
+ rafHandle = window.requestAnimationFrame(animationFrameLoop);
if (typeof PerformanceObserver !== "undefined") {
try {
@@ -103,7 +104,8 @@
rafGapsMs.length = 0;
mountedRowSamples.length = 0;
previousAnimationFrameTs = 0;
- window.requestAnimationFrame(animationFrameLoop);
+ window.cancelAnimationFrame(rafHandle);
+ rafHandle = window.requestAnimationFrame(animationFrameLoop);
},
};
}You can send follow-ups to this agent here.
artifacts/perf/virtualization-large_threads-1775077017735/virtualization-large_threads.json
Outdated
Show resolved
Hide resolved
- Expand perf fixtures with multi-thread live stream events and namespaced IDs - Harden stream getters and update web perf tests for the heavier workload
juliusmarminge
force-pushed
the
t3code/performance-regression-tests
branch
from
b9a8795 to
18f19a6
Compare
macroscopeapp
bot
dismissed
their stale review
Dismissing prior approval to re-evaluate 18f19a6
- Keep `bun run test` focused on non-perf suites
- Add README entry for perf benchmarks - Document scenarios, commands, artifacts, and env vars
- Persist and remove the seeded run parent directory after perf runs - Add regression coverage for seed cleanup, percentile clamping, and RAF reset
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Perf provider always enabled even without provider scenario
- Moved PERF_PROVIDER_ENV inside the providerScenarioId conditional so it is only set when a provider scenario is specified, matching the behavior in open-perf-app.ts.
Or push these changes by commenting:
@cursor push e26c2abd97
Preview (e26c2abd97)
diff --git a/apps/web/test/perf/appHarness.ts b/apps/web/test/perf/appHarness.ts
--- a/apps/web/test/perf/appHarness.ts
+++ b/apps/web/test/perf/appHarness.ts
@@ -268,8 +268,12 @@
const env = {
...process.env,
T3CODE_AUTO_BOOTSTRAP_PROJECT_FROM_CWD: "false",
- [PERF_PROVIDER_ENV]: "1",
- ...(options.providerScenarioId ? { [PERF_SCENARIO_ENV]: options.providerScenarioId } : {}),
+ ...(options.providerScenarioId
+ ? {
+ [PERF_PROVIDER_ENV]: "1",
+ [PERF_SCENARIO_ENV]: options.providerScenarioId,
+ }
+ : {}),
};
let stdoutBuffer = "";You can send follow-ups to this agent here.
- Seed large_threads across 5 projects with bounded heavy turns - Add shared perf server env setup and update harness assertions - Co-authored-by: codex <codex@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 2 potential issues.
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Failed template creation cached permanently as rejected promise
- Added a .catch handler on the createTemplateDir promise that evicts the entry from templateDirPromises on rejection, allowing subsequent calls to retry.
- ✅ Fixed: Duplicated utility functions across harness and CLI script
- Extracted pickFreePort, waitForServerReady, stopChildProcess, cleanupPerfRunDir, verifyBuiltArtifacts, and parsePerfSeededState into a shared test/perf/support/perfProcess.ts module and updated both consumers to import from it.
Or push these changes by commenting:
@cursor push 89497fed15
Preview (89497fed15)
diff --git a/apps/server/integration/perf/seedPerfState.ts b/apps/server/integration/perf/seedPerfState.ts
--- a/apps/server/integration/perf/seedPerfState.ts
+++ b/apps/server/integration/perf/seedPerfState.ts
@@ -546,7 +546,10 @@
if (existing) {
return existing;
}
- const created = createTemplateDir(scenarioId);
+ const created = createTemplateDir(scenarioId).catch((error: unknown) => {
+ templateDirPromises.delete(scenarioId);
+ throw error;
+ });
templateDirPromises.set(scenarioId, created);
return created;
}
diff --git a/apps/web/test/perf/appHarness.ts b/apps/web/test/perf/appHarness.ts
--- a/apps/web/test/perf/appHarness.ts
+++ b/apps/web/test/perf/appHarness.ts
@@ -1,6 +1,5 @@
-import { spawn, type ChildProcess } from "node:child_process";
-import { access, mkdir, rm, writeFile } from "node:fs/promises";
-import { createServer } from "node:net";
+import { spawn } from "node:child_process";
+import { mkdir, writeFile } from "node:fs/promises";
import { join, resolve } from "node:path";
import { fileURLToPath } from "node:url";
import { once } from "node:events";
@@ -18,6 +17,14 @@
installBrowserPerfCollector,
PERF_BROWSER_GLOBAL,
} from "../../../../test/perf/support/browserMetrics";
+import {
+ pickFreePort,
+ waitForServerReady,
+ stopChildProcess,
+ cleanupPerfRunDir,
+ verifyBuiltArtifacts,
+ parsePerfSeededState,
+} from "../../../../test/perf/support/perfProcess";
import type { PerfThresholdProfile } from "../../../../test/perf/support/thresholds";
import type {
PerfProviderScenarioId,
@@ -35,8 +42,6 @@
const serverClientIndexPath = resolve(repoRoot, "apps/server/dist/client/index.html");
const PERF_ARTIFACT_DIR_ENV = "T3CODE_PERF_ARTIFACT_DIR";
const PERF_HEADFUL_ENV = "T3CODE_PERF_HEADFUL";
-const PERF_SEED_JSON_START = "__T3_PERF_SEED_JSON_START__";
-const PERF_SEED_JSON_END = "__T3_PERF_SEED_JSON_END__";
interface PerfSeedThreadSummary {
readonly id: string;
@@ -104,84 +109,6 @@
}>;
}
-async function pickFreePort(): Promise<number> {
- return await new Promise<number>((resolvePort, reject) => {
- const server = createServer();
- server.on("error", reject);
- server.listen(0, "127.0.0.1", () => {
- const address = server.address();
- if (!address || typeof address === "string") {
- reject(new Error("Unable to resolve a free localhost port."));
- return;
- }
- const { port } = address;
- server.close((closeError) => {
- if (closeError) {
- reject(closeError);
- return;
- }
- resolvePort(port);
- });
- });
- });
-}
-
-async function waitForServerReady(url: string, process: ChildProcess): Promise<void> {
- const startedAt = Date.now();
- const timeoutMs = 45_000;
- const requestTimeoutMs = 1_000;
-
- while (Date.now() - startedAt < timeoutMs) {
- if (process.exitCode !== null) {
- throw new Error(`Perf server exited early with code ${process.exitCode}.`);
- }
- try {
- const response = await fetch(url, {
- redirect: "manual",
- signal: AbortSignal.timeout(requestTimeoutMs),
- });
- if (response.ok) {
- return;
- }
- } catch {
- // Ignore connection races while the server is still starting.
- }
- await new Promise((resolveDelay) => setTimeout(resolveDelay, 200));
- }
-
- throw new Error(`Timed out waiting for perf server readiness at ${url}.`);
-}
-
-async function verifyBuiltArtifacts(): Promise<void> {
- await Promise.all([access(serverBinPath), access(serverClientIndexPath)]).catch(() => {
- throw new Error(
- `Built perf artifacts are missing. Expected ${serverBinPath} and ${serverClientIndexPath}. Run bun run test:perf:web or build the app first.`,
- );
- });
-}
-
-async function stopChildProcess(process: ChildProcess): Promise<void> {
- if (process.exitCode !== null) {
- return;
- }
-
- process.kill("SIGTERM");
- const exited = await new Promise<boolean>((resolveExited) => {
- const timer = setTimeout(() => resolveExited(false), 5_000);
- process.once("exit", () => {
- clearTimeout(timer);
- resolveExited(true);
- });
- });
-
- if (!exited && process.exitCode === null) {
- process.kill("SIGKILL");
- await new Promise<void>((resolveExited) => {
- process.once("exit", () => resolveExited());
- });
- }
-}
-
async function ensureArtifactDir(suite: string, scenarioId: string): Promise<string> {
const baseArtifactDir = resolve(
process.env[PERF_ARTIFACT_DIR_ENV] ?? join(repoRoot, "artifacts/perf"),
@@ -192,10 +119,6 @@
return artifactDir;
}
-async function cleanupPerfRunDir(runParentDir: string): Promise<void> {
- await rm(runParentDir, { recursive: true, force: true });
-}
-
async function writeServerLogs(
artifactDir: string,
stdout: string,
@@ -231,22 +154,10 @@
);
}
-function parsePerfSeededState(stdout: string): PerfSeededState {
- const startIndex = stdout.lastIndexOf(PERF_SEED_JSON_START);
- const endIndex = stdout.lastIndexOf(PERF_SEED_JSON_END);
-
- if (startIndex !== -1 && endIndex !== -1 && endIndex > startIndex) {
- const payload = stdout.slice(startIndex + PERF_SEED_JSON_START.length, endIndex).trim();
- return JSON.parse(payload) as PerfSeededState;
- }
-
- return JSON.parse(stdout) as PerfSeededState;
-}
-
export async function startPerfAppHarness(
options: StartPerfAppHarnessOptions,
): Promise<PerfAppHarness> {
- await verifyBuiltArtifacts();
+ await verifyBuiltArtifacts([serverBinPath, serverClientIndexPath]);
const seededState = await (async () => {
const seedProcess = spawn(
@@ -270,7 +181,7 @@
if (exitCode !== 0) {
throw new Error(`Perf seed command failed with code ${exitCode ?? "unknown"}.\n${stderr}`);
}
- return parsePerfSeededState(stdout);
+ return parsePerfSeededState<PerfSeededState>(stdout);
})();
const artifactDir = await ensureArtifactDir(options.suite, options.seedScenarioId);
const port = await pickFreePort();
diff --git a/apps/web/tsconfig.json b/apps/web/tsconfig.json
--- a/apps/web/tsconfig.json
+++ b/apps/web/tsconfig.json
@@ -27,6 +27,7 @@
"test",
"../../test/perf/support/artifact.ts",
"../../test/perf/support/browserMetrics.ts",
+ "../../test/perf/support/perfProcess.ts",
"../../test/perf/support/serverSampler.ts",
"../../test/perf/support/thresholds.ts"
]
diff --git a/scripts/open-perf-app.ts b/scripts/open-perf-app.ts
--- a/scripts/open-perf-app.ts
+++ b/scripts/open-perf-app.ts
@@ -1,17 +1,22 @@
-import { spawn, type ChildProcess } from "node:child_process";
+import { spawn } from "node:child_process";
import { once } from "node:events";
-import { access, rm } from "node:fs/promises";
-import { createServer } from "node:net";
import { resolve } from "node:path";
import { fileURLToPath } from "node:url";
+import {
+ pickFreePort,
+ waitForServerReady,
+ stopChildProcess,
+ cleanupPerfRunDir,
+ verifyBuiltArtifacts,
+ parsePerfSeededState,
+} from "../test/perf/support/perfProcess";
+
const repoRoot = fileURLToPath(new URL("../", import.meta.url));
const serverBinPath = resolve(repoRoot, "apps/server/dist/bin.mjs");
const serverClientIndexPath = resolve(repoRoot, "apps/server/dist/client/index.html");
const PERF_PROVIDER_ENV = "T3CODE_PERF_PROVIDER";
const PERF_SCENARIO_ENV = "T3CODE_PERF_SCENARIO";
-const PERF_SEED_JSON_START = "__T3_PERF_SEED_JSON_START__";
-const PERF_SEED_JSON_END = "__T3_PERF_SEED_JSON_END__";
type PerfSeedScenarioId = "large_threads" | "burst_base";
type PerfProviderScenarioId = "dense_assistant_stream";
@@ -151,47 +156,6 @@
};
}
-async function pickFreePort(): Promise<number> {
- return await new Promise<number>((resolvePort, reject) => {
- const server = createServer();
- server.on("error", reject);
- server.listen(0, "127.0.0.1", () => {
- const address = server.address();
- if (!address || typeof address === "string") {
- reject(new Error("Unable to resolve a free localhost port."));
- return;
- }
- server.close((closeError) => {
- if (closeError) {
- reject(closeError);
- return;
- }
- resolvePort(address.port);
- });
- });
- });
-}
-
-async function verifyBuiltArtifacts(): Promise<void> {
- await Promise.all([access(serverBinPath), access(serverClientIndexPath)]).catch(() => {
- throw new Error(
- `Built perf artifacts are missing. Expected ${serverBinPath} and ${serverClientIndexPath}. Run bun run test:perf:web or build the app first.`,
- );
- });
-}
-
-function parsePerfSeededState(stdout: string): PerfSeededState {
- const startIndex = stdout.lastIndexOf(PERF_SEED_JSON_START);
- const endIndex = stdout.lastIndexOf(PERF_SEED_JSON_END);
-
- if (startIndex === -1 || endIndex === -1 || endIndex <= startIndex) {
- throw new Error(`Perf seed command did not emit the expected JSON markers.\n${stdout}`);
- }
-
- const payload = stdout.slice(startIndex + PERF_SEED_JSON_START.length, endIndex).trim();
- return JSON.parse(payload) as PerfSeededState;
-}
-
async function seedPerfState(scenarioId: PerfSeedScenarioId): Promise<PerfSeededState> {
const seedProcess = spawn("bun", ["run", "apps/server/scripts/seedPerfState.ts", scenarioId], {
cwd: repoRoot,
@@ -213,61 +177,9 @@
throw new Error(`Perf seed command failed with code ${exitCode ?? "unknown"}.\n${stderr}`);
}
- return parsePerfSeededState(stdout);
+ return parsePerfSeededState<PerfSeededState>(stdout);
}
-async function waitForServerReady(url: string, process: ChildProcess): Promise<void> {
- const startedAt = Date.now();
- const timeoutMs = 45_000;
- const requestTimeoutMs = 1_000;
-
- while (Date.now() - startedAt < timeoutMs) {
- if (process.exitCode !== null) {
- throw new Error(`Perf server exited early with code ${process.exitCode}.`);
- }
- try {
- const response = await fetch(url, {
- redirect: "manual",
- signal: AbortSignal.timeout(requestTimeoutMs),
- });
- if (response.ok) {
- return;
- }
- } catch {
- // Ignore connection races during startup.
- }
- await new Promise((resolveDelay) => setTimeout(resolveDelay, 200));
- }
-
- throw new Error(`Timed out waiting for perf server readiness at ${url}.`);
-}
-
-async function stopChildProcess(process: ChildProcess): Promise<void> {
- if (process.exitCode !== null) {
- return;
- }
-
- process.kill("SIGTERM");
- const exited = await new Promise<boolean>((resolveExited) => {
- const timer = setTimeout(() => resolveExited(false), 5_000);
- process.once("exit", () => {
- clearTimeout(timer);
- resolveExited(true);
- });
- });
-
- if (!exited && process.exitCode === null) {
- process.kill("SIGKILL");
- await new Promise<void>((resolveExited) => {
- process.once("exit", () => resolveExited());
- });
- }
-}
-
-async function cleanupPerfRunDir(runParentDir: string): Promise<void> {
- await rm(runParentDir, { recursive: true, force: true });
-}
-
function openUrl(url: string): void {
const command: [string, ...string[]] =
process.platform === "darwin"
@@ -320,7 +232,7 @@
async function main(): Promise<void> {
const options = parseArgs(process.argv.slice(2));
- await verifyBuiltArtifacts();
+ await verifyBuiltArtifacts([serverBinPath, serverClientIndexPath]);
const seededState = await seedPerfState(options.scenarioId);
const port = options.port === 0 ? await pickFreePort() : options.port;
diff --git a/scripts/tsconfig.json b/scripts/tsconfig.json
--- a/scripts/tsconfig.json
+++ b/scripts/tsconfig.json
@@ -12,5 +12,5 @@
}
]
},
- "include": ["**/*.ts"]
+ "include": ["**/*.ts", "../test/perf/support/perfProcess.ts"]
}
diff --git a/test/perf/support/perfProcess.ts b/test/perf/support/perfProcess.ts
new file mode 100644
--- /dev/null
+++ b/test/perf/support/perfProcess.ts
@@ -1,0 +1,100 @@
+import { type ChildProcess } from "node:child_process";
+import { access, rm } from "node:fs/promises";
+import { createServer } from "node:net";
+
+const PERF_SEED_JSON_START = "__T3_PERF_SEED_JSON_START__";
+const PERF_SEED_JSON_END = "__T3_PERF_SEED_JSON_END__";
+
+export async function pickFreePort(): Promise<number> {
+ return await new Promise<number>((resolvePort, reject) => {
+ const server = createServer();
+ server.on("error", reject);
+ server.listen(0, "127.0.0.1", () => {
+ const address = server.address();
+ if (!address || typeof address === "string") {
+ reject(new Error("Unable to resolve a free localhost port."));
+ return;
+ }
+ const { port } = address;
+ server.close((closeError) => {
+ if (closeError) {
+ reject(closeError);
+ return;
+ }
+ resolvePort(port);
+ });
+ });
+ });
+}
+
+export async function waitForServerReady(url: string, process: ChildProcess): Promise<void> {
+ const startedAt = Date.now();
+ const timeoutMs = 45_000;
+ const requestTimeoutMs = 1_000;
+
+ while (Date.now() - startedAt < timeoutMs) {
+ if (process.exitCode !== null) {
+ throw new Error(`Perf server exited early with code ${process.exitCode}.`);
+ }
+ try {
+ const response = await fetch(url, {
+ redirect: "manual",
+ signal: AbortSignal.timeout(requestTimeoutMs),
+ });
+ if (response.ok) {
+ return;
+ }
+ } catch {
+ // Ignore connection races while the server is still starting.
+ }
+ await new Promise((resolveDelay) => setTimeout(resolveDelay, 200));
+ }
+
+ throw new Error(`Timed out waiting for perf server readiness at ${url}.`);
+}
+
+export async function stopChildProcess(process: ChildProcess): Promise<void> {
+ if (process.exitCode !== null) {
+ return;
+ }
+
+ process.kill("SIGTERM");
+ const exited = await new Promise<boolean>((resolveExited) => {
+ const timer = setTimeout(() => resolveExited(false), 5_000);
+ process.once("exit", () => {
+ clearTimeout(timer);
+ resolveExited(true);
+ });
+ });
+
+ if (!exited && process.exitCode === null) {
+ process.kill("SIGKILL");
+ await new Promise<void>((resolveExited) => {
+ process.once("exit", () => resolveExited());
+ });
+ }
+}
+
+export async function cleanupPerfRunDir(runParentDir: string): Promise<void> {
+ await rm(runParentDir, { recursive: true, force: true });
+}
+
+export async function verifyBuiltArtifacts(paths: ReadonlyArray<string>): Promise<void> {
+ await Promise.all(paths.map((p) => access(p))).catch(() => {
+ throw new Error(
+ `Built perf artifacts are missing. Expected ${paths.join(" and ")}. Run bun run test:perf:web or build the app first.`,
+ );
+ });
+}
+
+export function parsePerfSeededState<T>(stdout: string): T {
+ const startIndex = stdout.lastIndexOf(PERF_SEED_JSON_START);
+ const endIndex = stdout.lastIndexOf(PERF_SEED_JSON_END);
+
+ if (startIndex !== -1 && endIndex !== -1 && endIndex > startIndex) {
+ const payload = stdout.slice(startIndex + PERF_SEED_JSON_START.length, endIndex).trim();
+ return JSON.parse(payload) as T;
+ }
+
+ return JSON.parse(stdout) as T;
+}You can send follow-ups to this agent here.
| const created = createTemplateDir(scenarioId); | ||
| templateDirPromises.set(scenarioId, created); | ||
| return created; | ||
| } |
Contributor
There was a problem hiding this comment.
Failed template creation cached permanently as rejected promise
Low Severity
getTemplateDir caches the promise returned by createTemplateDir in the module-level templateDirPromises map before it resolves. If createTemplateDir fails (e.g., git not found, disk error), the rejected promise is permanently cached, so every subsequent call for the same scenarioId will immediately return the same rejection instead of retrying. This could cause confusing cascading failures when running multiple perf tests in the same process.
| process.once("exit", () => resolveExited()); | ||
| }); | ||
| } | ||
| } |
Contributor
There was a problem hiding this comment.
Duplicated utility functions across harness and CLI script
Low Severity
pickFreePort, waitForServerReady, stopChildProcess, cleanupPerfRunDir, verifyBuiltArtifacts, and parsePerfSeededState are duplicated nearly verbatim between scripts/open-perf-app.ts and apps/web/test/perf/appHarness.ts. These could be extracted to a shared module to avoid divergent fixes over time.
Additional Locations (1)
| ); | ||
| const runtime = ManagedRuntime.make(seedLayer); | ||
|
|
||
| const snapshot = await runtime.runPromise( |
Contributor
There was a problem hiding this comment.
🟡 Medium perf/seedPerfState.ts:503
If runtime.runPromise(...) throws (e.g., from eventStore.append or projectionPipeline.projectEvent), runtime.dispose() on line 540 is never called. This leaks the ManagedRuntime resources including the SQLite connection, and because templateDirPromises caches the promise, the leaked runtime persists for the process lifetime. Consider using Effect.tapError or a try/finally pattern to ensure dispose() runs even on failure.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file apps/server/integration/perf/seedPerfState.ts around line 503:
If `runtime.runPromise(...)` throws (e.g., from `eventStore.append` or `projectionPipeline.projectEvent`), `runtime.dispose()` on line 540 is never called. This leaks the `ManagedRuntime` resources including the SQLite connection, and because `templateDirPromises` caches the promise, the leaked runtime persists for the process lifetime. Consider using `Effect.tapError` or a `try/finally` pattern to ensure `dispose()` runs even on failure.
Evidence trail:
apps/server/integration/perf/seedPerfState.ts lines 473-541 (createTemplateDir function): Line 502 creates runtime via `ManagedRuntime.make(seedLayer)`, line 504-522 calls `runtime.runPromise(...)`, line 540 calls `runtime.dispose()`. No try/finally block exists. Line 46 shows module-level cache `templateDirPromises = new Map<PerfSeedScenarioId, Promise<string>>();`. Lines 543-552 show `getTemplateDir` caches the promise before resolution.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing
PerfProviderAdapterevent emission.Note
Medium Risk
Adds new server startup branches and provider/registry implementations (gated by
T3CODE_PERF_PROVIDER) plus large test harnesses that seed real persistence/projections and drive Playwright; while mostly isolated, it touches core provider wiring and could affect runtime behavior if the env flag or stream semantics are misused.Overview
Introduces a local performance regression harness that runs against the built server/web app, seeds deterministic fixture state, drives Playwright, collects browser responsiveness metrics, and writes JSON/log artifacts under
artifacts/(plus new docs and root scripts liketest:perf:webandperf:open).Adds a shared perf scenario catalog (
@t3tools/shared/perf/scenarioCatalog) and server-side tooling to seed realistic orchestration state via the real event store + projection pipeline (seedPerfState) for scenarios likelarge_threadsandburst_base.Adds a perf-only mock provider (
PerfProviderAdapter+ perf provider registry/layer) that replays deterministic runtime events (e.g.dense_assistant_stream) over the normal websocket path, and wires server startup to swap in these perf layers whenT3CODE_PERF_PROVIDER=1.Adds dedicated web perf tests/config (
vitest.perf.config.ts,test/perf/**/*.perf.test.ts) to assert budgets for virtualization/thread switching and websocket burst application, and updates several adapters/tests/buses to exposestream/streamEventsvia getters plus aChatViewdata-testidhook for perf navigation waits.Written by Cursor Bugbot for commit 7a1b982. This will update automatically on new commits. Configure here.
Note
Add performance regression test harnesses for virtualization and WebSocket streaming
PerfProviderAdapter(PerfProviderAdapter.ts) that replays deterministic scenario fixtures (assistant text deltas, turn events) with configurable delays; the server conditionally wires this adapter whenT3CODE_PERF_PROVIDER=1.seedPerfStateworkflow (seedPerfState.ts) that projects up to thousands of thread events into a SQLite snapshot, memoizes the template directory per scenario, and copies it for each isolated test run.T3CODE_PERF_PROVIDER=1is set, the server replaces the real provider adapter and registry with perf-specific in-memory implementations.Macroscope summarized 7a1b982.