Skip to content

feat: add disk-backed component storage with Bitcask implementation (On Hold due to changes to core arch in cardinal)#884

Draft
winton-library wants to merge 1 commit intomainfrom
winton/extend-cardinal-storage
Draft

feat: add disk-backed component storage with Bitcask implementation (On Hold due to changes to core arch in cardinal)#884
winton-library wants to merge 1 commit intomainfrom
winton/extend-cardinal-storage

Conversation

@winton-library
Copy link
Copy Markdown
Contributor

@winton-library winton-library commented Apr 6, 2026

TL;DR

Added disk-backed component storage using a Bitcask-style append-only file system, allowing components to be stored on disk instead of memory with configurable compaction and snapshot integration.

What changed?

  • New disk storage system: Implemented diskStore and diskColumn[T] for Bitcask-style append-only component storage
  • Component naming convention: Components ending with @disk suffix are automatically stored on disk
  • Configuration options: Added DiskStoragePath and CompactionRate to WorldOptions with corresponding environment variables
  • Tick lifecycle integration: Added disk flush and compaction operations to the world tick cycle
  • Snapshot support: Extended snapshot format to include disk state with SHA-256 checksums for integrity verification
  • ECS operations: Updated Get/Set/Has operations to work transparently with both memory and disk components
  • Error handling: Disk compaction failures are treated as fatal errors that trigger shutdown

How to test?

  1. Set CARDINAL_DISK_STORAGE_PATH environment variable to a directory path
  2. Create components with names ending in @disk (e.g., match_history@disk)
  3. Register and use these components normally with ecs.Set(), ecs.Get(), etc.
  4. Configure CARDINAL_COMPACTION_RATE to control how often disk cleanup occurs
  5. Verify snapshots include disk state and restore correctly
  6. Test that disk components persist across world restarts

Why make this change?

This enables storing large or infrequently accessed component data on disk rather than in memory, reducing memory usage for games with substantial persistent data. The Bitcask design provides fast reads/writes while the configurable compaction system manages disk space efficiently. Integration with the existing snapshot system ensures disk components are properly backed up and restored.

Copy link
Copy Markdown
Contributor Author


How to use the Graphite Merge Queue

Add the label graphite/merge to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

An organization admin has enabled the Graphite Merge Queue in this repository.

Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.

This stack of pull requests is managed by Graphite. Learn more about stacking.

@winton-library winton-library requested a review from rmrt1n April 6, 2026 23:59
@winton-library winton-library self-assigned this Apr 6, 2026
@winton-library winton-library added the enhancement New feature or request label Apr 6, 2026 — with Graphite App
@winton-library winton-library marked this pull request as ready for review April 6, 2026 23:59
@winton-library winton-library marked this pull request as draft April 9, 2026 13:31
@winton-library winton-library changed the title feat(cardinal): extend cardinal storage into disk feat: add disk-backed component storage with Bitcask implementation (On Hold due to changes to core arch in cardinal) Apr 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant