Exclude LILO labeling trials from trials_expecting_data (#5144)#5144
Open
ItsMrLin wants to merge 2 commits intofacebook:mainfrom
Open
Exclude LILO labeling trials from trials_expecting_data (#5144)#5144ItsMrLin wants to merge 2 commits intofacebook:mainfrom
ItsMrLin wants to merge 2 commits intofacebook:mainfrom
Conversation
meta-cla
bot
added
the
CLA Signed
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5144 +/- ##
=======================================
Coverage 96.39% 96.39%
=======================================
Files 613 613
Lines 68295 68345 +50
=======================================
+ Hits 65830 65879 +49
- Misses 2465 2466 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
meta-codesync
bot
changed the title
ItsMrLin
added a commit
to ItsMrLin/Ax
that referenced
this pull request
Apr 5, 2026
Summary: LILO labeling trials have their pairwise preference data fetched inline during the labeling loop (`_run_lilo_labeling_loop`), not via the normal data refetch paths (orchestrator `poll_and_process_results`, PTSClient `refetch_data`). Including them in `trials_expecting_data` causes unnecessary data fetch attempts for metrics (e.g., Deltoid) that don't exist on these trials, producing noisy errors and wasting time. This filters LILO labeling trials (`trial_type == Keys.LILO_LABELING`) from the `trials_expecting_data` property on `Experiment`, which is the centralized source used by all downstream data refetch consumers. Differential Revision: D99571562
ItsMrLin
force-pushed
the
export-D99571562
branch
from
0583eef to
c9434bd
Compare
ItsMrLin
added a commit
to ItsMrLin/Ax
that referenced
this pull request
Apr 5, 2026
Summary: LILO labeling trials have their pairwise preference data fetched inline during the labeling loop (`_run_lilo_labeling_loop`), not via the normal data refetch paths (orchestrator `poll_and_process_results`, PTSClient `refetch_data`). Including them in `trials_expecting_data` causes unnecessary data fetch attempts for metrics (e.g., Deltoid) that don't exist on these trials, producing noisy errors and wasting time. This filters LILO labeling trials (`trial_type == Keys.LILO_LABELING`) from the `trials_expecting_data` property on `Experiment`, which is the centralized source used by all downstream data refetch consumers. Differential Revision: D99571562
ItsMrLin
force-pushed
the
export-D99571562
branch
from
c9434bd to
2a9c7b9
Compare
ItsMrLin
added a commit
to ItsMrLin/Ax
that referenced
this pull request
Apr 6, 2026
Summary: LILO labeling trials have their pairwise preference data fetched inline during the labeling loop (`_run_lilo_labeling_loop`), not via the normal data refetch paths (orchestrator `poll_and_process_results`, PTSClient `refetch_data`). Including them in `trials_expecting_data` causes unnecessary data fetch attempts for metrics (e.g., Deltoid) that don't exist on these trials, producing noisy errors and wasting time. This filters LILO labeling trials (`trial_type == Keys.LILO_LABELING`) from the `trials_expecting_data` property on `Experiment`, which is the centralized source used by all downstream data refetch consumers. Differential Revision: D99571562
…acebook#5145) Summary: `InSampleUniformGenerator` selects existing arms for LILO labeling by drawing from the `generated_points` pool constructed in `RandomAdapter._gen()`. This pool started from `arms_by_signature_for_deduplication`, which excludes arm signatures from *any* FAILED trial. Because LILO labeling trials borrow arms from regular BO trials (same signatures), a FAILED labeling trial incorrectly removes the original arm from the selection pool — even though it still exists in a non-FAILED trial. Within a single LILO labeling loop run, failed iterations accumulate and progressively poison the pool until no arms remain, crashing with: ValueError: Cannot select 2 arms: only 0 eligible arms available The fix: for in-sample generators, start from `arms_by_signature` (all arms) instead of `arms_by_signature_for_deduplication`. The existing `expecting_sigs` filter already handles the real restriction (only data-expecting, non-abandoned arms), so the FAILED-arm exclusion was just an accidental side effect of piggybacking on the dedup infrastructure. Reviewed By: bletham Differential Revision: D99611303
Summary: LILO labeling trials have their pairwise preference data fetched inline during the labeling loop (`_run_lilo_labeling_loop`), not via the normal data refetch paths (orchestrator `poll_and_process_results`, PTSClient `refetch_data`). Including them in `trials_expecting_data` causes unnecessary data fetch attempts for metrics (e.g., Deltoid) that don't exist on these trials, producing noisy errors and wasting time. This filters LILO labeling trials (`trial_type == Keys.LILO_LABELING`) from the `trials_expecting_data` property on `Experiment`, which is the centralized source used by all downstream data refetch consumers. Differential Revision: D99571562
ItsMrLin
force-pushed
the
export-D99571562
branch
from
2a9c7b9 to
d6a975d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
LILO labeling trials have their pairwise preference data fetched inline
during the labeling loop (
_run_lilo_labeling_loop), not via the normaldata refetch paths (orchestrator
poll_and_process_results,PTSClient
refetch_data). Including them intrials_expecting_datacauses unnecessary data fetch attempts for metrics (e.g., Deltoid) that
don't exist on these trials, producing noisy errors and wasting time.
This filters LILO labeling trials (
trial_type == Keys.LILO_LABELING)from the
trials_expecting_dataproperty onExperiment, which is thecentralized source used by all downstream data refetch consumers.
Differential Revision: D99571562