Skip to content

WIP initial steps for distributing compaction coordination#6217

Draft
keith-turner wants to merge 1 commit intoapache:mainfrom
keith-turner:dist-coord
Draft

WIP initial steps for distributing compaction coordination#6217
keith-turner wants to merge 1 commit intoapache:mainfrom
keith-turner:dist-coord

Conversation

@keith-turner
Copy link
Contributor

Some initial steps for distributing compaction coordinator. Currently only contains the following.

  • Creates a map in ZK that maps compactor resource group to a manager process. Currently maps all RGs to the primary manager.
  • Modified the compactor processes to consult this map to find which coordinator to use.
  • A periodic task running in the primary manager that updates this map in ZK

Still need to do the following

  • Delegate compactor resource groups to assistant managers. Will require refactoring existing code and a new RPC.
  • Modify TGW to stream compaction jobs to remote assistant managers.
  • Refactor all the cleanup code that runs in the coordinator to only run in the primary manager
  • Potentially move the in memory tracking the coordinator currently does to the monitor.
  • Correctly create the map of compactor RGs to assistant managers in ZK.

Plan to continue experimenting with this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant