fix: chunk byline IN clauses to stay within D1 SQL variable limit#223
Open
baezor wants to merge 4 commits intoemdash-cms:mainfrom
Open
fix: chunk byline IN clauses to stay within D1 SQL variable limit#223baezor wants to merge 4 commits intoemdash-cms:mainfrom
baezor wants to merge 4 commits intoemdash-cms:mainfrom
Conversation
Fixes emdash-cms#219. hydrateEntryBylines builds unbounded IN (?, ?, …) clauses that exceed Cloudflare D1's bound-parameter limit on large collections. Adds a chunks() utility and applies it defense-in-depth at the repository level: getContentBylinesMany, findByUserIds, and getAuthorIds now batch IDs in groups of 50.
Deduplicates contentIds in getContentBylinesMany to prevent duplicate credits when the same ID appears across chunk boundaries. Adds tests for the duplication edge case and an end-to-end getBylinesForEntries test spanning both explicit and inferred byline paths.
🦋 Changeset detectedLatest commit: fddf1ee The changes in this PR will be included in the next version bump. This PR includes changesets to release 9 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
|
All contributors have signed the CLA ✍️ ✅ |
Author
|
I have read the CLA Document and I hereby sign the CLA |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes unbounded
IN (?, ?, …)clauses in byline hydration that exceed Cloudflare D1's SQL bound-parameter limit when querying large collections.Adds a
chunks()utility (utils/chunks.ts) and applies it defense-in-depth at the repository level so any caller is protected:BylineRepository.getContentBylinesMany— deduplicates IDs, then chunkscontent_id IN (…)BylineRepository.findByUserIds— chunksuser_id IN (…)getAuthorIds(bylines/index.ts) — chunksid IN (…)raw SQLEach batch is capped at 50 IDs, well within D1's limit. Content IDs are also deduplicated before chunking to prevent duplicate credits when the same ID spans multiple chunks.
Closes #219
Type of change
Checklist
pnpm typecheckpassespnpm --silent lint:json | jq '.diagnostics | length'returns 0pnpm testpasses (or targeted tests for my change)pnpm formathas been runAI-generated code disclosure
Screenshots / test output
All 26 tests pass across 3 test files (10 new):