Fix in-sample arm pool exhaustion from FAILED LILO labeling trials (#5145)#5145
Closed
ItsMrLin wants to merge 1 commit intofacebook:mainfrom
Closed
Fix in-sample arm pool exhaustion from FAILED LILO labeling trials (#5145)#5145ItsMrLin wants to merge 1 commit intofacebook:mainfrom
ItsMrLin wants to merge 1 commit intofacebook:mainfrom
Conversation
meta-cla
bot
added
the
CLA Signed
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5145 +/- ##
=======================================
Coverage 96.39% 96.39%
=======================================
Files 613 613
Lines 68295 68321 +26
=======================================
+ Hits 65830 65860 +30
+ Misses 2465 2461 -4 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
meta-codesync
bot
changed the title
ItsMrLin
added a commit
to ItsMrLin/Ax
that referenced
this pull request
Apr 6, 2026
…acebook#5145) Summary: `InSampleUniformGenerator` selects existing arms for LILO labeling by drawing from the `generated_points` pool constructed in `RandomAdapter._gen()`. This pool started from `arms_by_signature_for_deduplication`, which excludes arm signatures from *any* FAILED trial. Because LILO labeling trials borrow arms from regular BO trials (same signatures), a FAILED labeling trial incorrectly removes the original arm from the selection pool — even though it still exists in a non-FAILED trial. Within a single LILO labeling loop run, failed iterations accumulate and progressively poison the pool until no arms remain, crashing with: ValueError: Cannot select 2 arms: only 0 eligible arms available The fix: for in-sample generators, start from `arms_by_signature` (all arms) instead of `arms_by_signature_for_deduplication`. The existing `expecting_sigs` filter already handles the real restriction (only data-expecting, non-abandoned arms), so the FAILED-arm exclusion was just an accidental side effect of piggybacking on the dedup infrastructure. Reviewed By: bletham Differential Revision: D99611303
ItsMrLin
force-pushed
the
export-D99611303
branch
from
5f02c9a to
c4c2c57
Compare
ItsMrLin
added a commit
to ItsMrLin/Ax
that referenced
this pull request
Apr 6, 2026
…acebook#5145) Summary: Pull Request resolved: facebook#5145 Pull Request resolved: https://github.com/facebook/Ax/pull/XXXX `InSampleUniformGenerator` selects existing arms for LILO labeling by drawing from the `generated_points` pool constructed in `RandomAdapter._gen()`. This pool started from `arms_by_signature_for_deduplication`, which excludes arm signatures from *any* FAILED trial. Because LILO labeling trials borrow arms from regular BO trials (same signatures), a FAILED labeling trial incorrectly removes the original arm from the selection pool — even though it still exists in a non-FAILED trial. Within a single LILO labeling loop run, failed iterations accumulate and progressively poison the pool until no arms remain, crashing with: ValueError: Cannot select 2 arms: only 0 eligible arms available The fix: for in-sample generators, start from `arms_by_signature` (all arms) instead of `arms_by_signature_for_deduplication`. The existing `expecting_sigs` filter already handles the real restriction (only data-expecting, non-abandoned arms), so the FAILED-arm exclusion was just an accidental side effect of piggybacking on the dedup infrastructure. Reviewed By: bletham Differential Revision: D99611303
ItsMrLin
force-pushed
the
export-D99611303
branch
from
c4c2c57 to
2a9354f
Compare
ItsMrLin
added a commit
to ItsMrLin/Ax
that referenced
this pull request
Apr 6, 2026
…acebook#5145) Summary: `InSampleUniformGenerator` selects existing arms for LILO labeling by drawing from the `generated_points` pool constructed in `RandomAdapter._gen()`. This pool started from `arms_by_signature_for_deduplication`, which excludes arm signatures from *any* FAILED trial. Because LILO labeling trials borrow arms from regular BO trials (same signatures), a FAILED labeling trial incorrectly removes the original arm from the selection pool — even though it still exists in a non-FAILED trial. Within a single LILO labeling loop run, failed iterations accumulate and progressively poison the pool until no arms remain, crashing with: ValueError: Cannot select 2 arms: only 0 eligible arms available The fix: for in-sample generators, start from `arms_by_signature` (all arms) instead of `arms_by_signature_for_deduplication`. The existing `expecting_sigs` filter already handles the real restriction (only data-expecting, non-abandoned arms), so the FAILED-arm exclusion was just an accidental side effect of piggybacking on the dedup infrastructure. Reviewed By: bletham Differential Revision: D99611303
ItsMrLin
added a commit
to ItsMrLin/Ax
that referenced
this pull request
Apr 6, 2026
…acebook#5145) Summary: `InSampleUniformGenerator` selects existing arms for LILO labeling by drawing from the `generated_points` pool constructed in `RandomAdapter._gen()`. This pool started from `arms_by_signature_for_deduplication`, which excludes arm signatures from *any* FAILED trial. Because LILO labeling trials borrow arms from regular BO trials (same signatures), a FAILED labeling trial incorrectly removes the original arm from the selection pool — even though it still exists in a non-FAILED trial. Within a single LILO labeling loop run, failed iterations accumulate and progressively poison the pool until no arms remain, crashing with: ValueError: Cannot select 2 arms: only 0 eligible arms available The fix: for in-sample generators, start from `arms_by_signature` (all arms) instead of `arms_by_signature_for_deduplication`. The existing `expecting_sigs` filter already handles the real restriction (only data-expecting, non-abandoned arms), so the FAILED-arm exclusion was just an accidental side effect of piggybacking on the dedup infrastructure. Reviewed By: bletham Differential Revision: D99611303
…acebook#5145) Summary: `InSampleUniformGenerator` selects existing arms for LILO labeling by drawing from the `generated_points` pool constructed in `RandomAdapter._gen()`. This pool started from `arms_by_signature_for_deduplication`, which excludes arm signatures from *any* FAILED trial. Because LILO labeling trials borrow arms from regular BO trials (same signatures), a FAILED labeling trial incorrectly removes the original arm from the selection pool — even though it still exists in a non-FAILED trial. Within a single LILO labeling loop run, failed iterations accumulate and progressively poison the pool until no arms remain, crashing with: ValueError: Cannot select 2 arms: only 0 eligible arms available The fix: for in-sample generators, start from `arms_by_signature` (all arms) instead of `arms_by_signature_for_deduplication`. The existing `expecting_sigs` filter already handles the real restriction (only data-expecting, non-abandoned arms), so the FAILED-arm exclusion was just an accidental side effect of piggybacking on the dedup infrastructure. Reviewed By: bletham Differential Revision: D99611303
ItsMrLin
force-pushed
the
export-D99611303
branch
from
2a9354f to
d330824
Compare
|
This pull request has been merged in e942453. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary:
InSampleUniformGeneratorselects existing arms for LILO labeling bydrawing from the
generated_pointspool constructed inRandomAdapter._gen(). This pool started fromarms_by_signature_for_deduplication, which excludes arm signaturesfrom any FAILED trial.
Because LILO labeling trials borrow arms from regular BO trials (same
signatures), a FAILED labeling trial incorrectly removes the original
arm from the selection pool — even though it still exists in a
non-FAILED trial. Within a single LILO labeling loop run, failed
iterations accumulate and progressively poison the pool until no arms
remain, crashing with:
The fix: for in-sample generators, start from
arms_by_signature(all arms) instead of
arms_by_signature_for_deduplication. Theexisting
expecting_sigsfilter already handles the real restriction(only data-expecting, non-abandoned arms), so the FAILED-arm exclusion
was just an accidental side effect of piggybacking on the dedup
infrastructure.
Reviewed By: bletham
Differential Revision: D99611303