Add 'curated' sampling strategy, tag scoring output, etc.#19
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #18.
Summary
Adds a
curatedsampling strategy that lets users specify exact pages to check, with optional per-page tags for grouped scoring output. This addresses the requested--urls-filemechanism by extending the existing config and sampling system.What's new
curatedsampling strategy — skips page discovery and checks only the specified URLs--urlsCLI flag — comma-separated URLs for quick one-off curated runs without a config filepagesconfig field — list of URLs (plain strings or{ url, tag }objects) inagent-docs.config.ymlpagesis present in config without an explicitsamplingStrategy, defaults tocurateddescribeAgentDocsanddescribeAgentDocsPerCheckpass through curated pages and auto-infer the strategyCLI examples
Scorecard output with tags
Tag scoring implementation
computeTagScores()insrc/scoring/tag-scores.tswalks all per-page check results, filters by each tag's URLs, and computes a weighted aggregate using the same proportion logic and per-check warn coefficients as the main scoring pipeline. Single-resource checks (llms-txt-*) are excluded since they don't have per-page data. Tags where no checks produced data are omitted rather than receiving a misleading score of 0.Each
TagScoreincludes achecksarray withTagCheckBreakdownobjects containing the check ID, category, weight, proportion, and per-page statuses, giving full visibility into why a tag scored the way it did.Backward compatibility
All changes are additive. Existing consumers are unaffected:
ScoreResult.tagScoresandReportResult.urlTagsare optional fields, only present when curated pages have tagscomputeScore()signature is unchangedSamplingStrategyadds'curated'to the union; existing values work as beforeTagScore,TagCheckBreakdown,CuratedPageEntry,PageConfigEntry) are additive exportsFiles changed (25)
Core types and pipeline:
src/types.ts,src/index.ts,src/runner.ts,src/helpers/get-page-urls.ts,src/helpers/config.tsCLI:
src/cli/commands/check.ts,src/cli/formatters/scorecard.tsScoring:
src/scoring/types.ts,src/scoring/tag-scores.ts(new),src/scoring/score.ts,src/scoring/index.tsVitest integration:
src/helpers/vitest-runner.tsDocs:
docs/quick-start.md,docs/ci-integration.md,docs/reference/cli.md,docs/reference/config-file.md,docs/reference/scoring-api.md,docs/reference/programmatic-api.md,examples/agent-docs.config.ymlTests (25 new, 666 total):
test/unit/scoring/tag-scores.test.ts(new, 10 tests),test/unit/helpers/config.test.ts(+4),test/unit/helpers/get-page-urls.test.ts(+4),test/unit/runner.test.ts(+2),test/unit/scoring/score.test.ts(+2),test/unit/cli/scorecard-formatter.test.ts(+3)