Skip to content

Alerts verbosity fix#9362

Open
MohamedBilelBesbes wants to merge 1 commit intomozilla:masterfrom
MohamedBilelBesbes:VotingFix
Open

Alerts verbosity fix#9362
MohamedBilelBesbes wants to merge 1 commit intomozilla:masterfrom
MohamedBilelBesbes:VotingFix

Conversation

@MohamedBilelBesbes
Copy link
Copy Markdown
Contributor

@MohamedBilelBesbes MohamedBilelBesbes commented Apr 1, 2026

Upon deploying the voting system into the testing environment, it gave so many alerts. Particularly, it gave too many alerts related to the priority voting system and few ones related to the equal voting system. Upon investigating why this is happening, it boiled down to these factors:

  • The hyper parameters used in the methods in the voting system are the default hyper parameters and not the ones that yields the best performance, so it makes sense to have so many alerts because such hyper parameters configuration is too sensitive.
  • Here, the alert generation function is being called twice, the first time for the priority voting and the sdecond time for the equal voting. So chronologically, priority voting check is occurring and producing the (verbose) alerts. Once the equal voting occurs, it gets limited visiblity on the time series to analyze for change detection because it is constrained to use measurements of revisions pushed after the creation of the latest alert (the ones that were created ealier by the priority voting) because of this part of the code.

In order to fix these issues, we do the following:

  • Methods hyper parameter fixing: The methods were first tuned individually by testing different hyperparameters, such as the sizes of backward and forward windows, to achieve the best performance for each one. However, upon incorporating those into the voting system, we need to select a set of hyper-parameter configuration that has a fixed forward and back windows across all methods. This is because having different windows values for different methods makes them fundamentally not evaluate the same set of data, which is inconsistent. To address this, the backward and forward window values from the Student T Test were fixed and applied to all methods, while only the confidence and magnitude-related hyperparameters were further tuned. Upon incorporating such setup, Levene method has been deteriorating the quality of the voting system by throwing false alerts. Indeed, initial investigation showed that it is the worse performing method on both precision (minimizing false alerts) and recall (minimizing missed alerts). So we ditch it from the voting system for now. However, we keep its implementation in case further tuning is done on it, maybe it is just constrained by the spoace of the hyper parameters we used in our experimentation and that is why it is giving bad perofrmance.
  • Adding the detection methods as part of the PerformanceAlertTesting lookup: This will allow multiple voting systems to detect alerts at the same time without affecting each other's workflow.

Copy link
Copy Markdown
Collaborator

@gmierz gmierz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a couple minor fixes :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants