-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
priority:lowNice to have, do when possibleNice to have, do when possiblesize:mMedium — 4 to 8 hoursMedium — 4 to 8 hoursstatus:readyRefined and ready for sprint selectionRefined and ready for sprint selectiontype:featureNew functionalityNew functionality
Description
Part of #69
Depends on #91 (disk-backed storage), #93 (streaming query validation)
Description
Implement external merge sort to support large ORDER BY operations in streaming mode without loading the full sorted dataset into memory.
Acceptance Criteria
-
ORDER BYqueries in--streammode produce correctly sorted output for datasets larger than available memory - External sort uses temp files (respects
--memory-limitfor merge buffer size) - Temp files are cleaned up after execution (including on error/interrupt)
- Performance is reasonable: sorting 1GB CSV should complete in under 5 minutes on typical hardware
- Query validation (Streaming query validation: detect full-table-scan requirements #93) no longer blocks ORDER BY when external sort is available
Notes
- SQLite disk-backed temp store (Disk-backed large dataset support via SQLite temp storage #91) may already handle this via its own sort implementation — confirm before implementing custom sort
- If SQLite handles it, this issue may just be removing the validation restriction from Streaming query validation: detect full-table-scan requirements #93 and verifying correctness
- Only implement custom external sort if SQLite's built-in approach is insufficient
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
priority:lowNice to have, do when possibleNice to have, do when possiblesize:mMedium — 4 to 8 hoursMedium — 4 to 8 hoursstatus:readyRefined and ready for sprint selectionRefined and ready for sprint selectiontype:featureNew functionalityNew functionality