Add test format validation, error codes framework, and $divide tests#22
Add test format validation, error codes framework, and $divide tests#22eerxuan wants to merge 3 commits intodocumentdb:mainfrom
Conversation
cad86fe to
54c1a08
Compare
|
The [PR Tests / Tests (MongoDB) (pull_request)] will pass after the previous PR is merged. #20 |
xgerman
left a comment
There was a problem hiding this comment.
AI:
Issues Found
🔴 High — Duplicate imports in utils/init.py
from dataclasses import dataclass # duplicated
from typing import Any, Optional # duplicated
Lines 1-2 and 4-5 are identical. Copy-paste error.
🔴 High — Wildcard imports in utils/init.py
from documentdb_tests.framework.test_constants import *
from documentdb_tests.framework.error_codes import *
Wildcard imports pollute the namespace and make it unclear what's available. Should use explicit imports. This is
especially problematic since test_constants.py exports 80+ symbols.
🟡 Medium — test_structure_validator.py validation is now split but redundant The old check
test_{parent_folder}_*.py is now split into two separate checks:
- File starts with test_
- File name contains parent folder name
But this means test_smoke_expression_divide.py in folder divide/ passes check 2 because it contains "divide".
However test_operator_divide.py also passes. The original stricter pattern (test_divide_*.py) was more
prescriptive. The new looser check could allow my_test_that_mentions_divide_once.py. Not a blocker but worth
noting.
🟡 Medium — CONTRIBUTING.md has syntax error in example
execute_command(collection, {"find": collection.name, filter: {"a": 1}})
filter should be "filter" (string key). This will cause a NameError in Python.
🟡 Medium — test_format_validator.py missing newline at EOF Minor but will cause linter warnings.
🟡 Medium — Smoke test expected values may be wrong
expected = [{"_id": 1, "quotient": 5}, {"_id": 2, "quotient": 6}]
$divide returns doubles: 20/4 = 5.0 not 5, 30/5 = 6.0 not 6. The assertion may fail depending on how
assertSuccess compares types.
🟢 Minor — TEST_COVERAGE.md is 355 lines — very thorough but dense Great as a reference doc but could overwhelm
new contributors. Consider adding a "start here" callout at the top pointing to TEST_FORMAT.md and the $divide
example.
🟢 Minor — MISSING = "$missing" sentinel Using the string "$missing" as a sentinel works but is fragile — if any
test legitimately needs to test the literal string "$missing", it would collide. A _MISSING = object() sentinel
would be safer.
🟢 Minor — DOUBLE_PRECISION_LOSS = 9007199254740993 This is 2^53 + 1, which can't be exactly represented as a
float64. The Python literal 9007199254740993 is an int, but when converted to float it becomes 9007199254740992.0
. This is intentional (testing precision loss) but worth a comment.
Documentation Quality — Compared to pgcosmos/AGENTS.md
The pgcosmos AGENTS.md is excellent at providing actionable task aliases (build, test, start commands) and
institutional knowledge (which gateway is production, config gotchas). The functional-tests docs are strong on
test methodology but could borrow:
- A "Quick Start" section — how to run the test suite in one command
- Task aliases table — like pgcosmos does ("run tests" → pytest ..., "add a new operator test" → step-by-step)
- Known limitations section — what doesn't work yet, known flaky tests
09cf53e to
0094b7d
Compare
In compatibility test we won't fail on trailing zeros, as it will be flaky and not top priority to catch, we will have separate test cases to verify return type with $type in command. They will be add to the $divide test later to have full coverage. |
6d4c703 to
96b9658
Compare
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com> # Conflicts: # documentdb_tests/conftest.py # documentdb_tests/framework/test_structure_validator.py
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
Signed-off-by: Yunxuan Shi <yunxuan@amazon.com>
07a00fd to
00c3d9e
Compare
|
A bunch of checks are failing. Can you review those? |
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class DivideTest(BaseTestCase): |
There was a problem hiding this comment.
so then should we in the util class as per the guidance you have in the rules file?
| Result from execute_command with structure: | ||
| {"cursor": {"firstBatch": [{"result": <value>}]}} | ||
| """ | ||
| collection.insert_one(document) |
There was a problem hiding this comment.
shouldn't we separate out the arrange and act part of the test?
Adding test format auto validation in pytest test pickup hook.
Adding error code invariants.
Adding $divide test as a sample to show case the test format.