Skip to content

Add entity manager foundation for ETL indexer#147

Open
raymondjacobson wants to merge 4 commits intomainfrom
rj/etl-em-foundation
Open

Add entity manager foundation for ETL indexer#147
raymondjacobson wants to merge 4 commits intomainfrom
rj/etl-em-foundation

Conversation

@raymondjacobson
Copy link
Contributor

@raymondjacobson raymondjacobson commented Mar 16, 2026

Scaffold the entity manager processing framework that will bring
discovery-provider parity to the ETL indexer. This PR establishes the
handler pattern, test infrastructure, database migrations, and local
runner -- no entity processing logic yet.

  • processors/entity_manager/handler.go: Handler interface, Dispatcher,
    Params, EntityType/Action constants, ValidationError
  • processors/entity_manager/validate.go: shared validators
    (ValidateSigner, ValidateHandle, ValidateBio, etc.) -- WILL BE REFACTORED!
  • processors/entity_manager/testutil_test.go: setupTestDB, seeders,
    buildParams, mustHandle/mustReject assertion helpers
  • processors/entity_manager/handler_test.go: dispatcher unit tests
  • db/sql/migrations/0002: domain tables matching discovery-provider
    schema (users, tracks, playlists, follows, saves, reposts, routes,
    developer_apps, grants)
  • indexer.go: wire dispatcher after raw ManageEntity append, add
    debug logging for tx payloads and completion
  • examples/etl-local/main.go: standalone runner for local testing
    against production RPC + local Postgres
  • FEATURE.md: updated architecture docs and entity roadmap

Testing

docker run -d --name etl-postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=etl_local \
  -p 5432:5432 \
  postgres:16

go run ./examples/etl-local \
  --rpc https://rpc.audius.co \
  --db "postgres://postgres:postgres@localhost:5432/etl_local?sslmode=disable" \
  --start 21343648

Made-with: Cursor

Scaffold the entity manager processing framework that will bring
discovery-provider parity to the ETL indexer. This PR establishes the
handler pattern, test infrastructure, database migrations, and local
runner -- no entity processing logic yet.

- processors/entity_manager/handler.go: Handler interface, Dispatcher,
  Params, EntityType/Action constants, ValidationError
- processors/entity_manager/validate.go: shared validators
  (ValidateSigner, ValidateHandle, ValidateBio, etc.)
- processors/entity_manager/testutil_test.go: setupTestDB, seeders,
  buildParams, mustHandle/mustReject assertion helpers
- processors/entity_manager/handler_test.go: dispatcher unit tests
- db/sql/migrations/0002: domain tables matching discovery-provider
  schema (users, tracks, playlists, follows, saves, reposts, routes,
  developer_apps, grants)
- indexer.go: wire dispatcher after raw ManageEntity append, add
  debug logging for tx payloads and completion
- examples/etl-local/main.go: standalone runner for local testing
  against production RPC + local Postgres
- FEATURE.md: updated architecture docs and entity roadmap

Made-with: Cursor
Copy link
Contributor Author

raymondjacobson commented Mar 16, 2026

@raymondjacobson raymondjacobson marked this pull request as ready for review March 16, 2026 23:56
- Add UNIQUE constraint on blocks.number (required for FK references)
- Align all primary keys with production 01_schema.sql
- Add ON DELETE CASCADE to FK references to blocks(number)
- Expand tracks/playlists schemas with missing production columns
- Fix test seeds to omit blocknumber (avoid FK violation on fresh DB)

Made-with: Cursor
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant