Skip to content

FIX Include subdirectory jailbreak templates in listings#1498

Open
biefan wants to merge 3 commits intomicrosoft:mainfrom
biefan:include-subdirectory-jailbreak-templates
Open

FIX Include subdirectory jailbreak templates in listings#1498
biefan wants to merge 3 commits intomicrosoft:mainfrom
biefan:include-subdirectory-jailbreak-templates

Conversation

@biefan
Copy link
Copy Markdown
Contributor

@biefan biefan commented Mar 17, 2026

Summary

  • List jailbreak templates from the cached recursive template scan instead of only the top-level directory
  • Add regression coverage for subdirectory template discovery in both TextJailBreak and the AIRT jailbreak scenario
  • Fix random.choices (with replacement) to random.sample (without replacement) in get_jailbreak_templates() to avoid returning duplicate template names
  • Return templates in sorted order for cross-platform determinism

Problem

TextJailBreak already resolves template names from subdirectories such as pliny/amazon/nova.yaml, but get_jailbreak_templates() only scans templates/*.yaml at the top level.

This creates an inconsistent API surface: a template name can be loaded successfully via template_file_name, but the same name is missing from the advertised template list and is rejected by Jailbreak(..., jailbreak_names=[...]) validation.

Changes

text_jailbreak.py

  • Replace JAILBREAK_TEMPLATES_PATH.glob('*.yaml') with cls._get_template_cache().keys() in get_jailbreak_templates() to include subdirectory templates (72 previously-invisible templates now appear, 162 total vs 90 top-level)
  • Wrap result in sorted() for deterministic cross-platform ordering (previously depended on filesystem glob order)
  • Fix random.choices to random.sample so num_templates returns distinct templates without duplicates

Tests

  • Add structural regression test verifying subdirectory templates appear in the listing (checks count exceeds top-level-only count, not hardcoded to a specific file)
  • Add scenario-level test verifying Jailbreak(jailbreak_names=[...]) accepts subdirectory template names

Testing

pytest tests/unit/datasets/test_jailbreak_text.py tests/unit/scenarios/test_jailbreak.py -q

@hannahwestra25 hannahwestra25 self-assigned this Mar 24, 2026
@romanlutz romanlutz changed the title Include subdirectory jailbreak templates in listings FIX Include subdirectory jailbreak templates in listings Apr 12, 2026
romanlutz and others added 2 commits April 11, 2026 18:17
- Use random.sample instead of random.choices in get_jailbreak_templates
  to avoid returning duplicate template names when num_templates is set
- Replace hardcoded 'nova.yaml' assertions with structural checks that
  verify subdirectory templates are included without depending on a
  specific file
- Add JAILBREAK_TEMPLATES_PATH import to test files for structural
  assertions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
# Conflicts:
#	tests/unit/scenarios/test_jailbreak.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants