Skip to content

WIP - Refactor External Compactions monitor view to use snapshot-backed Top 50 data#6211

Merged
DomGarguilo merged 1 commit intoapache:mainfrom
DomGarguilo:ecMonitorChanges
Mar 13, 2026
Merged

WIP - Refactor External Compactions monitor view to use snapshot-backed Top 50 data#6211
DomGarguilo merged 1 commit intoapache:mainfrom
DomGarguilo:ecMonitorChanges

Conversation

@DomGarguilo
Copy link
Member

Fixes #6174

This PR:

  • moves EC monitor data gathering into InformationFetcher/SystemInformation snapshot flow
  • serves the active compaction data from top 50 longest running compactions view
  • removes the old/competing rest-v2/compactions/detail path
  • makes no intended changes to the active compactions page and just refactors the data collection on the backend

@DomGarguilo DomGarguilo added this to the 4.0.0 milestone Mar 12, 2026
@DomGarguilo DomGarguilo requested a review from dlmarion March 12, 2026 17:09
@DomGarguilo DomGarguilo self-assigned this Mar 12, 2026
@DomGarguilo
Copy link
Member Author

I don't think we should keep the full list of external compactions in memory in the Monitor for two reasons

  1. It could eat up a lot of memory
  2. The information may not be useful or accessed for very quick compactions

Instead I think we can do the following to show the longest running compactions in the cluster:

  1. Move the TimeOrderedRunningCompactionSet object from the Coordinator to the Monitor
  2. As the Monitor is building the next version of the SystemInformation object, populate the TimeOrderedRunningCompactionSet set with information from each Compactors getActiveCompactions result.
  3. Allow the size of TimeOrderedRunningCompactionSet to be configured by the user so they can decide how much memory to use for this feature.
  4. Provide views of the data in TimeOrderedRunningCompactionSet by table, by resource group, and a global view.

Originally posted by @dlmarion in #6174

This branch still keeps the full running compactions set internally so ec/details can continue to work. A follow-on could drop that details support and keep only a Top N set in memory.

Another follow-on would be to move Top N collection fully into the Monitor by polling each Compactor while building SystemInformation, instead of asking the manager/coordinator for that info. That would reduce reliance on the manager/coordinator, but would add more collection logic complexity so we would need to consider that.

@DomGarguilo DomGarguilo merged commit 8d01714 into apache:main Mar 13, 2026
9 checks passed
@DomGarguilo DomGarguilo deleted the ecMonitorChanges branch March 13, 2026 15:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Competing views of External Compactions in new Monitor

2 participants