-
Notifications
You must be signed in to change notification settings - Fork 0
Open
Labels
priority:mediumShould be done soonShould be done soonsize:mMedium — 4 to 8 hoursMedium — 4 to 8 hoursstatus:readyRefined and ready for sprint selectionRefined and ready for sprint selectiontype:featureNew functionalityNew functionality
Description
Part of #69
Depends on #69-spike (evaluate streaming approaches)
Description
Implement the simpler streaming approach: configure SQLite to use disk-backed temp storage for large datasets. This allows all SQL operations to continue working on datasets larger than RAM, without changing query semantics.
Acceptance Criteria
-
--memory-limit <size>flag sets a hint for SQLite temp storage threshold (e.g.256MB,1GB) - SQLite configured with
PRAGMA temp_store = FILEwhen memory limit is set - Temp files are cleaned up after execution
- Queries that previously ran out of memory on 1GB+ files now complete successfully
- Existing behavior unchanged when flag is not set
- Error message if temp directory is not writable
Notes
- Use
PRAGMA temp_store_directoryto configure temp location - No changes to query parsing or output — only storage backend changes
- Follow spike recommendation on exact PRAGMA configuration
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
priority:mediumShould be done soonShould be done soonsize:mMedium — 4 to 8 hoursMedium — 4 to 8 hoursstatus:readyRefined and ready for sprint selectionRefined and ready for sprint selectiontype:featureNew functionalityNew functionality