Skip to content

Add 'curated' sampling strategy, tag scoring output, etc.#19

Merged
dacharyc merged 6 commits intomainfrom
add-custom-url-list-support
Apr 8, 2026
Merged

Add 'curated' sampling strategy, tag scoring output, etc.#19
dacharyc merged 6 commits intomainfrom
add-custom-url-list-support

Conversation

@dacharyc
Copy link
Copy Markdown
Member

@dacharyc dacharyc commented Apr 8, 2026

Closes #18.

Summary

Adds a curated sampling strategy that lets users specify exact pages to check, with optional per-page tags for grouped scoring output. This addresses the requested --urls-file mechanism by extending the existing config and sampling system.

What's new

  • curated sampling strategy — skips page discovery and checks only the specified URLs
  • --urls CLI flag — comma-separated URLs for quick one-off curated runs without a config file
  • pages config field — list of URLs (plain strings or { url, tag } objects) in agent-docs.config.yml
  • Auto-inference — when pages is present in config without an explicit samplingStrategy, defaults to curated
  • Per-tag scoring — when pages have tags, the scorecard shows a Tag Scores section with per-check breakdowns showing which checks are failing and how many pages are affected
  • JSON tag detail — full per-page, per-check breakdown in JSON output for programmatic consumers
  • Vitest helper supportdescribeAgentDocs and describeAgentDocsPerCheck pass through curated pages and auto-infer the strategy

CLI examples

# Quick one-off: check specific URLs
afdocs check https://docs.example.com --urls https://docs.example.com/quickstart,https://docs.example.com/api/auth

# Config-based with tags
afdocs check --format scorecard
# agent-docs.config.yml
url: https://docs.example.com
pages:
  - url: https://docs.example.com/quickstart
    tag: getting-started
  - url: https://docs.example.com/api/auth
    tag: api-reference

Scorecard output with tags

  Tag Scores:
    api-reference                         66 / 100 (D) · 3 pages
      FAIL  page-size-markdown             2 fail, 1 pass
      FAIL  http-status-codes              3 fail
      WARN  redirect-behavior              3 warn
    getting-started                       72 / 100 (C) · 2 pages
      FAIL  page-size-html                 2 fail
      WARN  redirect-behavior              2 warn

Tag scoring implementation

computeTagScores() in src/scoring/tag-scores.ts walks all per-page check results, filters by each tag's URLs, and computes a weighted aggregate using the same proportion logic and per-check warn coefficients as the main scoring pipeline. Single-resource checks (llms-txt-*) are excluded since they don't have per-page data. Tags where no checks produced data are omitted rather than receiving a misleading score of 0.

Each TagScore includes a checks array with TagCheckBreakdown objects containing the check ID, category, weight, proportion, and per-page statuses, giving full visibility into why a tag scored the way it did.

Backward compatibility

All changes are additive. Existing consumers are unaffected:

  • ScoreResult.tagScores and ReportResult.urlTags are optional fields, only present when curated pages have tags
  • computeScore() signature is unchanged
  • SamplingStrategy adds 'curated' to the union; existing values work as before
  • New types (TagScore, TagCheckBreakdown, CuratedPageEntry, PageConfigEntry) are additive exports

Files changed (25)

Core types and pipeline:
src/types.ts, src/index.ts, src/runner.ts, src/helpers/get-page-urls.ts, src/helpers/config.ts

CLI:
src/cli/commands/check.ts, src/cli/formatters/scorecard.ts

Scoring:
src/scoring/types.ts, src/scoring/tag-scores.ts (new), src/scoring/score.ts, src/scoring/index.ts

Vitest integration:
src/helpers/vitest-runner.ts

Docs:
docs/quick-start.md, docs/ci-integration.md, docs/reference/cli.md, docs/reference/config-file.md, docs/reference/scoring-api.md, docs/reference/programmatic-api.md, examples/agent-docs.config.yml

Tests (25 new, 666 total):
test/unit/scoring/tag-scores.test.ts (new, 10 tests), test/unit/helpers/config.test.ts (+4), test/unit/helpers/get-page-urls.test.ts (+4), test/unit/runner.test.ts (+2), test/unit/scoring/score.test.ts (+2), test/unit/cli/scorecard-formatter.test.ts (+3)

@dacharyc dacharyc merged commit 73d99ae into main Apr 8, 2026
2 checks passed
@dacharyc dacharyc deleted the add-custom-url-list-support branch April 8, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support custom URL list input for targeted scoring

1 participant