Skip to content

Update result analyzer and remaining changes#23

Open
eerxuan wants to merge 8 commits intodocumentdb:mainfrom
eerxuan:result-analyzer-updates-2
Open

Update result analyzer and remaining changes#23
eerxuan wants to merge 8 commits intodocumentdb:mainfrom
eerxuan:result-analyzer-updates-2

Conversation

@eerxuan
Copy link
Copy Markdown
Contributor

@eerxuan eerxuan commented Mar 31, 2026

Update result_analyzer with improved categorization and reporting.

This has dependency on #20.

Demo of the report output:

Before the PR in console:

============================================================
Test Results Summary
============================================================
Total:   44
Passed:  39 (88.64%)
Failed:  4
Skipped: 1
============================================================

Before the PR in output file:

================================================================================
DocumentDB Functional Test Results
================================================================================
Generated: 2026-04-02 23:36:18 UTC

SUMMARY
--------------------------------------------------------------------------------
Total Tests:  44
Passed:       39 (88.64%)
Failed:       4
Skipped:      1

RESULTS BY TAG
--------------------------------------------------------------------------------
No tags found in test results.

FAILED TESTS
--------------------------------------------------------------------------------

documentdb_tests/compatibility/tests/core/query-and-write/commands/find/test_find_basic_queries.py::test_fake_mismatch
  Tags:
  Duration: 0.00s
  Error: AssertionError: [RESULT_MISMATCH] Expected [1,2,3] but got [1,2]...

documentdb_tests/compatibility/tests/core/operator/stages/group/test_group_stage.py::test_fake_connection_error
  Tags:
  Duration: 0.00s
  Error: ConnectionError: Cannot connect to mongodb at localhost:27017...

documentdb_tests/compatibility/tests/core/query-and-write/commands/insert/test_insert_operations.py::test_fake_unexpected
  Tags:
  Duration: 0.00s
  Error: AssertionError: [UNEXPECTED_ERROR] Expected success but got exception...

documentdb_tests/compatibility/tests/core/query-and-write/commands/find/test_find_basic_queries.py::test_fake_error_mismatch
  Tags:
  Duration: 0.00s
  Error: AssertionError: [ERROR_MISMATCH] Expected code 11000 but got 26...

================================================================================

After the PR in console:

============================================================
Test Results Summary
============================================================
Total:   44
Passed:  39 (88.64%)
Failed:  4
Skipped: 1
============================================================

Failed Tests (4):
------------------------------------------------------------
  ERROR_MISMATCH: 1
  INFRA_ERROR: 1
  RESULT_MISMATCH: 1
  UNEXPECTED_ERROR: 1
============================================================

After the PR in output file:

================================================================================
DocumentDB Functional Test Results
================================================================================
Generated: 2026-04-02 23:48:20 UTC

SUMMARY
--------------------------------------------------------------------------------
Total Tests:  44
Passed:       39 (88.64%)
Failed:       4
Skipped:      1

RESULTS BY TAG
--------------------------------------------------------------------------------

aggregate:
  Total:   11
  Passed:  9 (81.82%)
  Failed:  1
  Skipped: 1

smoke:
  Total:   7
  Passed:  6 (85.71%)
  Failed:  1
  Skipped: 0

insert:
  Total:   8
  Passed:  7 (87.5%)
  Failed:  1
  Skipped: 0

find:
  Total:   24
  Passed:  22 (91.67%)
  Failed:  2
  Skipped: 0

collection_mgmt:
  Total:   1
  Passed:  1 (100.0%)
  Failed:  0
  Skipped: 0

FAILED TESTS
--------------------------------------------------------------------------------

  ERROR_MISMATCH (1):

    documentdb_tests/compatibility/tests/core/query-and-write/commands/find/test_find_basic_queries.py::test_fake_error_mismatch
      Tags: find
      Duration: 0.00s
      Error: AssertionError: [ERROR_MISMATCH] Expected code 11000 but got 26...

  INFRA_ERROR (1):

    documentdb_tests/compatibility/tests/core/operator/stages/group/test_group_stage.py::test_fake_connection_error
      Tags: aggregate
      Duration: 0.00s
      Error: ConnectionError: Cannot connect to mongodb at localhost:27017...

  RESULT_MISMATCH (1):

    documentdb_tests/compatibility/tests/core/query-and-write/commands/find/test_find_basic_queries.py::test_fake_mismatch
      Tags: find, smoke
      Duration: 0.00s
      Error: AssertionError: [RESULT_MISMATCH] Expected [1,2,3] but got [1,2]...

  UNEXPECTED_ERROR (1):

    documentdb_tests/compatibility/tests/core/query-and-write/commands/insert/test_insert_operations.py::test_fake_unexpected
      Tags: insert
      Duration: 0.00s
      Error: AssertionError: [UNEXPECTED_ERROR] Expected success but got exception...

SKIPPED TESTS
--------------------------------------------------------------------------------
  documentdb_tests/compatibility/tests/core/operator/stages/match/test_match_stage.py::test_fake_skipped

================================================================================

Copy link
Copy Markdown

@xgerman xgerman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM as a human.

AI felt this could be done:
Issues Found

🟡 Medium — Import at function scope inside print_summary() report_generator.py line ~158: from collections
import Counter is imported inside the function body. This should be a top-level import per PEP 8. It works, but
is inconsistent with the rest of the file.

🟡 Medium — failure_type not set for non-FAIL outcomes analyzer.py line ~340-345: failure_type is only added to
the test detail dict when test_outcome == FAIL. But report_generator.py calls test.get("failure_type", "UNKNOWN")
— this works due to .get() with default, but the field's conditional presence is fragile. Consider always setting
it (e.g., "N/A" for passed/skipped).

🟢 Minor — Trailing whitespace cleanup is noise Many hunks are whitespace-only changes (trailing spaces removed).
Not a problem, but inflates the diff. Could have been a separate commit.

🟢 Minor — extract_failure_tag regex is broad

match = re.search(r'[([A-Z_]+)]', crash_message)

This matches any [UPPER_CASE] bracket pattern. If a crash message happens to contain something like
[SOME_BSON_KEY], it could false-match. Not likely in practice but worth noting.

🟢 Minor — Empty init.py tests/collection/init.py is added with just a docstring. Fine for package
recognition, but if PR #20 already added test files under tests/collection/, this should have been part of that
PR.

Update result_analyzer with improved categorization and reporting.

Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
@eerxuan eerxuan force-pushed the result-analyzer-updates-2 branch from 650a516 to e515b94 Compare April 2, 2026 00:17
eerxuan and others added 4 commits April 2, 2026 16:00
@eerxuan eerxuan force-pushed the result-analyzer-updates-2 branch from a28c1f9 to 6134aee Compare April 2, 2026 23:28
Yunxuan Shi added 2 commits April 2, 2026 16:47
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
@eerxuan eerxuan force-pushed the result-analyzer-updates-2 branch from 6134aee to d7fef3c Compare April 2, 2026 23:52
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
@eerxuan eerxuan requested a review from vtarassov April 2, 2026 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants