feat: add AGENTS.md awareness to assessment pipeline#362
Draft
sbauza wants to merge 1 commit intoambient-code:mainfrom
Draft
feat: add AGENTS.md awareness to assessment pipeline#362sbauza wants to merge 1 commit intoambient-code:mainfrom
sbauza wants to merge 1 commit intoambient-code:mainfrom
Conversation
Parse AGENTS.md/CLAUDE.md content and use it to produce more accurate assessment scores. Three user stories implemented: - US1: AGENTS.md content boosts scores for ADR, logging, and layout assessors (verified claims = full credit, unverified = 60% cap) - US2: README alternative format detection (.md > .rst > .txt) - US3: Evidence attribution with [AGENTS.md] prefix for transparency New files: AgentContext model, AgentContextParser service, integration tests. All assessor signatures updated to accept optional agent_context parameter. 94 tests passing, 42.5-point score improvement on Nova-like repos. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
Warning
|
| Cohort / File(s) | Summary |
|---|---|
Core Models src/agentready/models/agent_context.py |
Introduced immutable dataclass hierarchy: LoggingInfo, ADRInfo, DocumentationInfo, and AgentContext to represent parsed metadata from agent configuration files. |
Parser Service src/agentready/services/agent_context_parser.py |
New parser that extracts structured fields (test directories, logging info, ADR details, documentation metadata, directory structures) from AGENTS.md/CLAUDE.md via regex-based pattern matching and Markdown section splitting. |
Base Assessor Interface src/agentready/assessors/base.py |
Updated abstract assess() method signature to accept optional agent_context parameter. |
Signature-Only Updates src/agentready/assessors/containers.py, src/agentready/assessors/dbt.py, src/agentready/assessors/security.py, src/agentready/assessors/stub_assessors.py, src/agentready/assessors/testing.py |
Added optional agent_context parameter to assess() methods without modifying internal logic. |
Code Quality Assessors src/agentready/assessors/code_quality.py |
Extended StructuredLoggingAssessor to detect logging frameworks from agent_context.logging_info, verify against dependency files, and produce scores based on verified vs. unverified frameworks; all other assessors in this file accept but do not use agent_context. |
Documentation Assessors src/agentready/assessors/documentation.py |
Refactored README detection to support .md/.rst/.txt extensions; updated ArchitectureDecisionsAssessor to consult agent_context.adr_info for external repos and local ADR paths with verification logic; added helper method _find_readme() and _assess_from_agent_context(). |
Structure Assessors src/agentready/assessors/structure.py |
Enhanced StandardLayoutAssessor to verify test directories from agent_context.test_directories against the filesystem; updated OneCommandSetupAssessor to use flexible README detection; imports _find_readme() from documentation module. |
Scanner Integration src/agentready/services/scanner.py |
Updated scan() to parse AGENTS.md/CLAUDE.md once via AgentContextParser and pass the resulting context to each assessor via _execute_assessor(). |
Integration Tests tests/integration/test_agents_md_score_improvement.py |
New test suite validating that assessor scores improve by at least 15 points when agent_context is provided; verifies evidence attribution, ensures scores do not decrease, and measures performance overhead. |
Unit Tests tests/unit/test_agent_context_parser.py, tests/unit/test_assessors_code_quality.py, tests/unit/test_assessors_documentation.py, tests/unit/test_assessors_structure.py |
Comprehensive test coverage for parser behavior, structured logging detection with agent context, documentation assessor README format handling and ADR verification, and standard layout test directory discovery. |
Sequence Diagram
sequenceDiagram
actor User
participant Scanner
participant AgentContextParser
participant Repository as Repository<br/>(AGENTS.md/CLAUDE.md)
participant Assessor
participant Finding
User->>Scanner: scan(repository_path)
Scanner->>AgentContextParser: parse(repo_path)
AgentContextParser->>Repository: read AGENTS.md/CLAUDE.md
Repository-->>AgentContextParser: file content (or None)
AgentContextParser->>AgentContextParser: extract test dirs,<br/>logging info, ADR info, etc.
AgentContextParser-->>Scanner: AgentContext | None
loop For each assessor
Scanner->>Assessor: assess(repository,<br/>agent_context=ctx)
Assessor->>Assessor: check agent_context<br/>for metadata
alt agent_context provided
Assessor->>Assessor: verify/cross-reference<br/>with filesystem
end
Assessor-->>Scanner: Finding
end
Scanner-->>User: Assessment results
Estimated code review effort
🎯 4 (Complex) | ⏱️ ~45 minutes
🚥 Pre-merge checks | ✅ 2 | ❌ 1
❌ Failed checks (1 warning)
| Check name | Status | Explanation | Resolution |
|---|---|---|---|
| Docstring Coverage | Docstring coverage is 53.19% which is insufficient. The required threshold is 80.00%. | Write docstrings for the functions missing them to satisfy the coverage threshold. |
✅ Passed checks (2 passed)
| Check name | Status | Explanation |
|---|---|---|
| Title check | ✅ Passed | The title 'feat: add AGENTS.md awareness to assessment pipeline' clearly and concisely describes the main feature addition: integrating AGENTS.md parsing into the assessment pipeline. |
| Description check | ✅ Passed | The description is directly related to the changeset, detailing the AGENTS.md parsing implementation, assessor updates, new files, and comprehensive test results that align with the actual changes shown in the raw summary. |
✏️ Tip: You can configure your own custom pre-merge checks in the settings.
✨ Finishing Touches
🧪 Generate unit tests (beta)
- Create PR with unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands and usage tips.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/agentready/assessors/documentation.py`:
- Around line 651-656: The _assess_from_agent_context method is missing a type
for the adr_info parameter; update its signature to annotate adr_info with the
ADRInfo type (e.g., adr_info: ADRInfo) and add the corresponding import (from
agent_context import ADRInfo) or use a forward reference if needed; ensure any
references in the method body still type-check after the change and run
linters/IDE to confirm the annotation is recognized for
_assess_from_agent_context in documentation.py.
In `@src/agentready/assessors/structure.py`:
- Around line 467-468: The README fallback to .rst/.txt causes
_is_setup_prominent() to still split only on Markdown '##' headers; update the
flow so that when you call _find_readme(repository.path) and get readme_path,
also detect the README type (extension) and pass that to _is_setup_prominent()
(or make _is_setup_prominent() detect type from the content/path). Change
_is_setup_prominent() to split/scan using Markdown headings for .md,
reStructuredText header patterns (underlines like ===/--- or ^-~) for .rst, and
a simple heuristic for .txt (look for explicit section titles like
"installation", "setup", or blank-line delimited paragraphs) so later install
mentions aren’t treated as the first section for non-Markdown READMEs.
In `@src/agentready/models/agent_context.py`:
- Around line 6-49: The dataclasses (AgentContext, LoggingInfo, ADRInfo,
DocumentationInfo) claim immutability but expose mutable list/dict fields; fix
by converting all list and dict fields to immutable containers in a frozen
dataclass __post_init__ using object.__setattr__ — e.g. convert frameworks,
conventions, test_directories, local_paths, external_repos to tuples and wrap
directory_structure and sections in types.MappingProxyType (or another read-only
mapping) inside each class's __post_init__; update field type annotations
accordingly (tuple[...] and Mapping[...] or dict -> Mapping) so the objects are
truly immutable after construction and prevent assessor state leakage.
In `@src/agentready/services/agent_context_parser.py`:
- Around line 277-285: The current local_adr_patterns entry that matches a bare
"adr" is too permissive and produces false positives from prose; update the
pattern in local_adr_patterns (used in the for pattern in re.finditer loop that
scans content and appends to local_paths) to only match explicit path tokens
such as "adr/" or ".adr" or "docs/adr", code-spans/backticked tokens, or
list-item prefixed paths (i.e., require a path delimiter, dot, slash,
surrounding backticks, or list marker before/after the token) so ordinary words
like "ADRs" are not captured and only real local paths are appended to
local_paths.
- Around line 46-60: The structured extractor currently only parses
primary_content (agents_content) when both agents_content and claude_content
exist, dropping unique CLAUDE.md data; update the branch where both exist so
that primary_content used by AgentContextParser._extract_sections is the merged
combined_content (or call _extract_sections with combined_content) and ensure
source_file still reflects the merge (e.g., "both") so all non-overlapping
sections from agents_content and claude_content are included; check the symbols
agents_content, claude_content, combined_content, primary_content and
AgentContextParser._extract_sections when making this change.
In `@src/agentready/services/scanner.py`:
- Around line 220-223: The _execute_assessor method's agent_context parameter is
missing a type annotation; update the signature of _execute_assessor(self,
assessor, repository: Repository, verbose: bool = False, agent_context:
Optional[AgentContext] = None) -> Finding (or use AgentContext | None if using
Python 3.10+), import AgentContext (and typing.Optional if needed) or enable
from __future__ import annotations for forward references so IDEs and type
checkers recognize the type.
In `@tests/integration/test_agents_md_score_improvement.py`:
- Around line 65-94: The fixture _create_nova_agents_md_context currently
constructs an AgentContext directly, bypassing the new parsing and scanning
code; change the test to create actual AGENTS.md/CLAUDE.md files in a temporary
repo directory and then call AgentContextParser.parse() (or the repo scanner
entry point used in production) to produce the AgentContext so the test
exercises the scanner-to-assessor wiring and AgentContextParser.parse() path
instead of instantiating AgentContext manually.
- Around line 248-258: Replace the lone absolute assertion with a relative
slowdown check using baseline_duration and enhanced_duration: compute
relative_slowdown = (enhanced_duration - baseline_duration) / baseline_duration
and assert relative_slowdown < 0.10, and optionally keep the existing absolute
cap on overhead_per_run_ms < 1.0 if you want both guarantees; update the
assertion block that currently references overhead_per_run_ms to validate
relative_slowdown instead (and combine both conditions with an informative
message mentioning baseline_duration, enhanced_duration, iterations, and
overhead_per_run_ms).
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: d7386f2d-5015-4ba1-aab9-4d4dfaa78aae
📒 Files selected for processing (17)
src/agentready/assessors/base.pysrc/agentready/assessors/code_quality.pysrc/agentready/assessors/containers.pysrc/agentready/assessors/dbt.pysrc/agentready/assessors/documentation.pysrc/agentready/assessors/security.pysrc/agentready/assessors/structure.pysrc/agentready/assessors/stub_assessors.pysrc/agentready/assessors/testing.pysrc/agentready/models/agent_context.pysrc/agentready/services/agent_context_parser.pysrc/agentready/services/scanner.pytests/integration/test_agents_md_score_improvement.pytests/unit/test_agent_context_parser.pytests/unit/test_assessors_code_quality.pytests/unit/test_assessors_documentation.pytests/unit/test_assessors_structure.py
Comment on lines
+651
to
+656
| def _assess_from_agent_context(self, repository: Repository, adr_info) -> Finding: | ||
| """Assess ADRs based on AGENTS.md content. | ||
|
|
||
| Verified local paths get full credit. | ||
| Unverified external repos get 60% cap. | ||
| """ |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Add type annotation for adr_info parameter.
The method signature lacks a type annotation for adr_info, making it harder to understand the expected interface and reducing IDE support.
Suggested fix
- def _assess_from_agent_context(self, repository: Repository, adr_info) -> Finding:
+ def _assess_from_agent_context(self, repository: Repository, adr_info: "ADRInfo") -> Finding:You'll need to import ADRInfo from the agent_context module or use a forward reference.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _assess_from_agent_context(self, repository: Repository, adr_info) -> Finding: | |
| """Assess ADRs based on AGENTS.md content. | |
| Verified local paths get full credit. | |
| Unverified external repos get 60% cap. | |
| """ | |
| def _assess_from_agent_context(self, repository: Repository, adr_info: "ADRInfo") -> Finding: | |
| """Assess ADRs based on AGENTS.md content. | |
| Verified local paths get full credit. | |
| Unverified external repos get 60% cap. | |
| """ |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agentready/assessors/documentation.py` around lines 651 - 656, The
_assess_from_agent_context method is missing a type for the adr_info parameter;
update its signature to annotate adr_info with the ADRInfo type (e.g., adr_info:
ADRInfo) and add the corresponding import (from agent_context import ADRInfo) or
use a forward reference if needed; ensure any references in the method body
still type-check after the change and run linters/IDE to confirm the annotation
is recognized for _assess_from_agent_context in documentation.py.
Comment on lines
+467
to
+468
| # Check if README exists (support .md, .rst, .txt) | ||
| readme_path, _ = _find_readme(repository.path) |
There was a problem hiding this comment.
The new .rst/.txt fallback over-scores setup prominence.
Once _find_readme() starts returning non-Markdown files, _is_setup_prominent() still splits only on Markdown ## headers. For .rst and .txt, the whole README becomes the “first sections”, so any later installation/setup mention still gets the 30-point prominence bonus.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agentready/assessors/structure.py` around lines 467 - 468, The README
fallback to .rst/.txt causes _is_setup_prominent() to still split only on
Markdown '##' headers; update the flow so that when you call
_find_readme(repository.path) and get readme_path, also detect the README type
(extension) and pass that to _is_setup_prominent() (or make
_is_setup_prominent() detect type from the content/path). Change
_is_setup_prominent() to split/scan using Markdown headings for .md,
reStructuredText header patterns (underlines like ===/--- or ^-~) for .rst, and
a simple heuristic for .txt (look for explicit section titles like
"installation", "setup", or blank-line delimited paragraphs) so later install
mentions aren’t treated as the first section for non-Markdown READMEs.
Comment on lines
+6
to
+49
| @dataclass(frozen=True) | ||
| class LoggingInfo: | ||
| """Extracted logging framework and convention details.""" | ||
|
|
||
| frameworks: list[str] = field(default_factory=list) | ||
| conventions: list[str] = field(default_factory=list) | ||
| has_structured_logging: bool = False | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class ADRInfo: | ||
| """Architecture Decision Record locations and format.""" | ||
|
|
||
| local_paths: list[str] = field(default_factory=list) | ||
| external_repos: list[str] = field(default_factory=list) | ||
| format: str = "unknown" | ||
| directory_pattern: str | None = None | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class DocumentationInfo: | ||
| """Documentation format and location hints.""" | ||
|
|
||
| readme_format: str | None = None | ||
| docs_directory: str | None = None | ||
| external_docs_url: str | None = None | ||
|
|
||
|
|
||
| @dataclass(frozen=True) | ||
| class AgentContext: | ||
| """Parsed representation of AGENTS.md and/or CLAUDE.md content. | ||
|
|
||
| Immutable after construction. Created once per assessment run | ||
| by AgentContextParser and passed to assessors as supplementary evidence. | ||
| """ | ||
|
|
||
| source_file: str | ||
| raw_content: str | ||
| test_directories: list[str] = field(default_factory=list) | ||
| logging_info: LoggingInfo | None = None | ||
| adr_info: ADRInfo | None = None | ||
| documentation_info: DocumentationInfo | None = None | ||
| directory_structure: dict[str, str] = field(default_factory=dict) | ||
| sections: dict[str, str] = field(default_factory=dict) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
python - <<'PY'
from dataclasses import dataclass, field
`@dataclass`(frozen=True)
class Context:
items: list[int] = field(default_factory=list)
ctx = Context()
ctx.items.append(1)
print(ctx.items)
try:
ctx.items = []
except Exception as exc:
print(type(exc).__name__)
PYRepository: ambient-code/agentready
Length of output: 90
🏁 Script executed:
# First, verify the file and code at the specified lines
head -60 src/agentready/models/agent_context.py | tail -55Repository: ambient-code/agentready
Length of output: 1582
🏁 Script executed:
# Check if src/agentready/services/scanner.py exists and examine how AgentContext is passed
find . -name "scanner.py" -path "*/services/*" | head -1 | xargs wc -lRepository: ambient-code/agentready
Length of output: 107
🏁 Script executed:
# Search for how AgentContext is instantiated and shared across assessors
rg -A 5 "AgentContext" src/agentready/services/ --max-count 10Repository: ambient-code/agentready
Length of output: 4669
🏁 Script executed:
# Find where agent_context is used after being created - who are the assessors?
rg -B 3 -A 10 "agent_context" src/agentready/services/scanner.py | head -80Repository: ambient-code/agentready
Length of output: 2586
🏁 Script executed:
# Check if any defensive copies are made or if it's passed directly to assessors
rg "def.*assess" src/agentready/assessors/ --max-count 5Repository: ambient-code/agentready
Length of output: 3482
🏁 Script executed:
# Look at the actual assessor calls in scanner.py
grep -n "\.assess\|\.run" src/agentready/services/scanner.py | head -20Repository: ambient-code/agentready
Length of output: 195
These dataclasses are not actually immutable despite the docstring claim.
frozen=True prevents field reassignment but allows in-place mutation of list and dict fields. Since the same AgentContext instance is shared sequentially across multiple assessors in scanner.py, one assessor's accidental mutation leaks state into subsequent assessors, violating the immutability contract.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agentready/models/agent_context.py` around lines 6 - 49, The dataclasses
(AgentContext, LoggingInfo, ADRInfo, DocumentationInfo) claim immutability but
expose mutable list/dict fields; fix by converting all list and dict fields to
immutable containers in a frozen dataclass __post_init__ using
object.__setattr__ — e.g. convert frameworks, conventions, test_directories,
local_paths, external_repos to tuples and wrap directory_structure and sections
in types.MappingProxyType (or another read-only mapping) inside each class's
__post_init__; update field type annotations accordingly (tuple[...] and
Mapping[...] or dict -> Mapping) so the objects are truly immutable after
construction and prevent assessor state leakage.
Comment on lines
+46
to
+60
| if agents_content is not None and claude_content is not None: | ||
| source_file = "both" | ||
| # AGENTS.md takes precedence; append CLAUDE.md for supplementary info | ||
| combined_content = agents_content + "\n\n" + claude_content | ||
| primary_content = agents_content | ||
| elif agents_content is not None: | ||
| source_file = "AGENTS.md" | ||
| combined_content = agents_content | ||
| primary_content = agents_content | ||
| else: | ||
| source_file = "CLAUDE.md" | ||
| combined_content = claude_content | ||
| primary_content = claude_content | ||
|
|
||
| sections = AgentContextParser._extract_sections(primary_content) |
There was a problem hiding this comment.
CLAUDE.md is dropped from structured extraction when both files exist.
source_file="both" suggests merged context, but every extracted field still comes from primary_content = agents_content. Any non-overlapping test/logging/ADR hints that live only in CLAUDE.md are silently lost.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agentready/services/agent_context_parser.py` around lines 46 - 60, The
structured extractor currently only parses primary_content (agents_content) when
both agents_content and claude_content exist, dropping unique CLAUDE.md data;
update the branch where both exist so that primary_content used by
AgentContextParser._extract_sections is the merged combined_content (or call
_extract_sections with combined_content) and ensure source_file still reflects
the merge (e.g., "both") so all non-overlapping sections from agents_content and
claude_content are included; check the symbols agents_content, claude_content,
combined_content, primary_content and AgentContextParser._extract_sections when
making this change.
Comment on lines
+277
to
+285
| local_adr_patterns = [ | ||
| r"(?:docs/adr|\.adr|adr|docs/decisions)/?", | ||
| r"specs?/[a-zA-Z0-9_<>]+/(?:approved|implemented|backlog|abandoned)/?", | ||
| ] | ||
| for pattern in local_adr_patterns: | ||
| for match in re.finditer(pattern, content, re.IGNORECASE): | ||
| path = match.group(0) | ||
| if path not in local_paths: | ||
| local_paths.append(path) |
There was a problem hiding this comment.
The bare adr pattern will manufacture local paths from prose.
This regex matches ordinary text like ADRs, so a sentence such as “ADRs are tracked externally” can yield a fake local path and push ArchitectureDecisionsAssessor into its 30/60-point branches. Restrict this to explicit path tokens like adr/, docs/adr/, code spans, or list items.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agentready/services/agent_context_parser.py` around lines 277 - 285, The
current local_adr_patterns entry that matches a bare "adr" is too permissive and
produces false positives from prose; update the pattern in local_adr_patterns
(used in the for pattern in re.finditer loop that scans content and appends to
local_paths) to only match explicit path tokens such as "adr/" or ".adr" or
"docs/adr", code-spans/backticked tokens, or list-item prefixed paths (i.e.,
require a path delimiter, dot, slash, surrounding backticks, or list marker
before/after the token) so ordinary words like "ADRs" are not captured and only
real local paths are appended to local_paths.
Comment on lines
220
to
223
| def _execute_assessor( | ||
| self, assessor, repository: Repository, verbose: bool = False | ||
| self, assessor, repository: Repository, verbose: bool = False, | ||
| agent_context=None, | ||
| ) -> Finding: |
There was a problem hiding this comment.
🧹 Nitpick | 🔵 Trivial
Add type annotation for agent_context parameter.
The agent_context parameter lacks a type annotation, which reduces code clarity and IDE support.
🔧 Add type annotation
def _execute_assessor(
- self, assessor, repository: Repository, verbose: bool = False,
- agent_context=None,
+ self, assessor, repository: Repository, verbose: bool = False,
+ agent_context: "AgentContext | None" = None,
) -> Finding:Note: You may need to import AgentContext at the top if not already imported for type checking, or use from __future__ import annotations for forward reference support.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| def _execute_assessor( | |
| self, assessor, repository: Repository, verbose: bool = False | |
| self, assessor, repository: Repository, verbose: bool = False, | |
| agent_context=None, | |
| ) -> Finding: | |
| def _execute_assessor( | |
| self, assessor, repository: Repository, verbose: bool = False, | |
| agent_context: "AgentContext | None" = None, | |
| ) -> Finding: |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/agentready/services/scanner.py` around lines 220 - 223, The
_execute_assessor method's agent_context parameter is missing a type annotation;
update the signature of _execute_assessor(self, assessor, repository:
Repository, verbose: bool = False, agent_context: Optional[AgentContext] = None)
-> Finding (or use AgentContext | None if using Python 3.10+), import
AgentContext (and typing.Optional if needed) or enable from __future__ import
annotations for forward references so IDEs and type checkers recognize the type.
Comment on lines
+65
to
+94
| def _create_nova_agents_md_context() -> AgentContext: | ||
| """Create AgentContext matching what AgentContextParser would produce | ||
| from a Nova-like AGENTS.md file.""" | ||
| return AgentContext( | ||
| source_file="AGENTS.md", | ||
| raw_content=textwrap.dedent("""\ | ||
| # AGENTS.md | ||
|
|
||
| ## Architecture Decisions | ||
| Architecture decisions are tracked in the external `openstack/nova-specs` | ||
| repository using RST-format specs. | ||
|
|
||
| ## Logging | ||
| Nova uses oslo.log for structured logging across all services. | ||
|
|
||
| ## Directory Structure | ||
| - nova/tests/ — Unit and functional tests | ||
| """), | ||
| test_directories=["nova/tests/"], | ||
| logging_info=LoggingInfo( | ||
| frameworks=["oslo.log"], | ||
| conventions=[], | ||
| has_structured_logging=True, | ||
| ), | ||
| adr_info=ADRInfo( | ||
| local_paths=[], | ||
| external_repos=["openstack/nova-specs"], | ||
| format="rst", | ||
| ), | ||
| ) |
There was a problem hiding this comment.
This fixture bypasses the code path the PR is adding.
By constructing AgentContext directly instead of writing an AGENTS.md/CLAUDE.md file and scanning the repo, these tests will stay green even if AgentContextParser.parse() or the scanner-to-assessor wiring regresses.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@tests/integration/test_agents_md_score_improvement.py` around lines 65 - 94,
The fixture _create_nova_agents_md_context currently constructs an AgentContext
directly, bypassing the new parsing and scanning code; change the test to create
actual AGENTS.md/CLAUDE.md files in a temporary repo directory and then call
AgentContextParser.parse() (or the repo scanner entry point used in production)
to produce the AgentContext so the test exercises the scanner-to-assessor wiring
and AgentContextParser.parse() path instead of instantiating AgentContext
manually.
Comment on lines
+248
to
+258
| # Calculate absolute overhead per assessment | ||
| overhead_total = enhanced_duration - baseline_duration | ||
| overhead_per_run_ms = (overhead_total / iterations) * 1000 | ||
|
|
||
| print(f" Baseline: {baseline_duration:.4f}s ({iterations} iterations)") | ||
| print(f" Enhanced: {enhanced_duration:.4f}s ({iterations} iterations)") | ||
| print(f" Overhead per run: {overhead_per_run_ms:.3f}ms") | ||
|
|
||
| # Absolute overhead must be <1ms per assessment run. | ||
| # A full assessment takes 2-10s, so <1ms is well within 10%. | ||
| assert overhead_per_run_ms < 1.0, ( |
There was a problem hiding this comment.
The assertion doesn't validate the stated 10% slowdown budget.
An absolute <1ms cap can still pass when the AGENTS-aware path is many times slower than baseline on fast fixtures. Assert on relative slowdown here, and keep the absolute cap only if you want both guarantees.
Suggested assertion change
overhead_total = enhanced_duration - baseline_duration
overhead_per_run_ms = (overhead_total / iterations) * 1000
+ relative_overhead = (
+ overhead_total / baseline_duration if baseline_duration > 0 else 0.0
+ )
@@
- assert overhead_per_run_ms < 1.0, (
- f"AGENTS.md overhead {overhead_per_run_ms:.3f}ms/run exceeds 1ms "
+ assert relative_overhead <= 0.10, (
+ f"AGENTS.md slowdown {relative_overhead:.1%} exceeds 10% "
f"(baseline={baseline_duration:.4f}s, "
f"enhanced={enhanced_duration:.4f}s)"
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
Suggested change
| # Calculate absolute overhead per assessment | |
| overhead_total = enhanced_duration - baseline_duration | |
| overhead_per_run_ms = (overhead_total / iterations) * 1000 | |
| print(f" Baseline: {baseline_duration:.4f}s ({iterations} iterations)") | |
| print(f" Enhanced: {enhanced_duration:.4f}s ({iterations} iterations)") | |
| print(f" Overhead per run: {overhead_per_run_ms:.3f}ms") | |
| # Absolute overhead must be <1ms per assessment run. | |
| # A full assessment takes 2-10s, so <1ms is well within 10%. | |
| assert overhead_per_run_ms < 1.0, ( | |
| # Calculate absolute overhead per assessment | |
| overhead_total = enhanced_duration - baseline_duration | |
| overhead_per_run_ms = (overhead_total / iterations) * 1000 | |
| relative_overhead = ( | |
| overhead_total / baseline_duration if baseline_duration > 0 else 0.0 | |
| ) | |
| print(f" Baseline: {baseline_duration:.4f}s ({iterations} iterations)") | |
| print(f" Enhanced: {enhanced_duration:.4f}s ({iterations} iterations)") | |
| print(f" Overhead per run: {overhead_per_run_ms:.3f}ms") | |
| # Absolute overhead must be <1ms per assessment run. | |
| # A full assessment takes 2-10s, so <1ms is well within 10%. | |
| assert relative_overhead <= 0.10, ( | |
| f"AGENTS.md slowdown {relative_overhead:.1%} exceeds 10% " | |
| f"(baseline={baseline_duration:.4f}s, " | |
| f"enhanced={enhanced_duration:.4f}s)" | |
| ) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@tests/integration/test_agents_md_score_improvement.py` around lines 248 -
258, Replace the lone absolute assertion with a relative slowdown check using
baseline_duration and enhanced_duration: compute relative_slowdown =
(enhanced_duration - baseline_duration) / baseline_duration and assert
relative_slowdown < 0.10, and optionally keep the existing absolute cap on
overhead_per_run_ms < 1.0 if you want both guarantees; update the assertion
block that currently references overhead_per_run_ms to validate
relative_slowdown instead (and combine both conditions with an informative
message mentioning baseline_duration, enhanced_duration, iterations, and
overhead_per_run_ms).
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| result = AgentContextParser.parse(tmp_repo) | ||
| assert result is not None | ||
| assert result.documentation_info is not None | ||
| assert "docs.openstack.org" in result.documentation_info.external_docs_url |
Check failure
Code scanning / CodeQL
Incomplete URL substring sanitization High test
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[AGENTS.md]prefix for transparencyResults
New files
src/agentready/models/agent_context.py— AgentContext, LoggingInfo, ADRInfo dataclassessrc/agentready/services/agent_context_parser.py— Section extraction and keyword matchingtests/unit/test_agent_context_parser.py— 21 unit tests for parsertests/integration/test_agents_md_score_improvement.py— Integration + performance testsTest plan
🤖 Generated with Claude Code