Cedric/better dx

[ViaDézo1er / cedric (viadezo1er)]

bt setup asks 0 and 2 questions: project and coding agent (and needs authentication)

org choice:

if --api-key is passed, use the provided api key
else if --prefer-profile, use the already registered profile (probably registered with OAuth previously)
else if the user has an api key in the env (variable BRAINTRUST_API_KEY), use it
else if there is already a registered profile, use it. If a profile is registered but the access token isn't accessible (for example storing the access token in plain text in ~/.config/bt/secrets.json but the file was deleted), just force re authentication
else same behavior as a user picking OAuth: give a url to paste in the browser

project choice:

if -p/--project is passed, use the specified project. If the project does not exist, stop the setup.
else if BRAINTRUST_DEFAULT_PROJECT exist in the env, same as for -p/--project
else the user has exactly one project, the default 'My Project' projects. Create a project in the user's organization named '{whoami}-test' where {whoami} is the result of the whoami command and use this project
else prompt the user to choose the project to use (creating a project is possible so the user isn't locked even with 0 projects)

skills, mcp, local, global:

by default, choose skills, no mcp and global scope
--no-skill, --no-mcp unselect skill/mcp
--local selects local instead of global
--no-mcp/skills don't remove pre-existing mcp/skills.

instrument:

by default, instrument
--(no-)instrument (un)select instrumentation
--no-instrument still configures mcp and skills, the only difference is that the agent isn't launched

agent:

if --agent AGENT flag is passed, use the specified agent
else if only one exists in [claude code, codex, cursor, opencode] (ie only one exist in PATH), use it
otherwise, prompt the user
If AGENT does not exist in PATH, try to launch it anyway.
Opencode support for starting with a prompt is lackluster as it can only start in the background.

workflow docs:

if one or multiple --workflow WORKFLOW flags are passed, fetch these workflows
otherwise, fetch all workflows

language:

try to identify the language based on file like pyproject.toml in the project root/sub folders (in current folder and subfolder with depth 1)
if nothing is found, alter the start prompt of the agent to find which language to instrument
conflicts with --no-instrument as the choice of language is only relevant if the agent is launched

TUI/background:

TUI by default
if --background is passed, launch the agent in the background
--tui exist only to conflict with --background (and --no-instrument)
Highly recommended to use --background with --yolo, as the agent won't be able to ask the user to allow it to run commands (and fail on step 1, checking if BRAINTRUST_API_KEY is in the env)

yolo mode

self explanatory for claude, codex, cursor
opencode is in yolo mode by default for the directory it is launched in
Besides, --yolo or equivalent don't exist in opencode yet (don't trust your LLM about this, there are many closed/merged/open PRs about this but just check opencode -h and you'll see it's not there).

interactive mode

-i/--interactive
Prompt the user to choose the configuration instead of passing options. Does not ask questions which are already answered by options or env variables.

$ bt setup -i
✔ Install scope · global (user-wide)
Set up coding-agent skills? yes
Set up MCP server? yes
Run instrumentation agent? yes
✔ Select agent to instrument this repo · claude
✔ Select additional workflow docs to prefetch (instrument is always included) · observe, annotate, evaluate, deploy
✔ How do you want to run the agent? · Interactive (TUI)
Grant agent full permissions? (bypass permission prompts) yes

Bugs

--json is very buggy
workflows and scope broken

Untested/poorly tested:

env variables
no-color, api/app-url, env-file, no-fetch-docs, refresh-docs, workers

TODO

Better help message in the interactive wizard
better json objects, they are not consistent
only make the available agents selectable when multiple are on PATH, print the other ones but in grey or something
if in a folder, don't look at the parent folder when detecting language (both prompt and auto detect)

[github-actions]

Latest downloadable build artifacts for this PR commit 2f4495e812e1:

• Workflow run: https://github.com/braintrustdata/bt/actions/runs/24259104835
• Download all artifacts (GitHub CLI): gh run download 24259104835 --repo braintrustdata/bt
• Installers are published from main automatically. To publish one for a PR branch, run release-canary manually via workflow_dispatch.

Available artifact names

• ``artifacts-build-global
• ``artifacts-build-local-x86_64-pc-windows-msvc
• ``artifacts-build-local-x86_64-apple-darwin
• ``artifacts-build-local-x86_64-unknown-linux-musl
• ``artifacts-build-local-x86_64-unknown-linux-gnu
• ``artifacts-build-local-aarch64-unknown-linux-musl
• ``artifacts-build-local-aarch64-apple-darwin
• ``artifacts-build-local-aarch64-unknown-linux-gnu
• ``artifacts-plan-dist-manifest
• ``cargo-dist-cache

[ViaDézo1er / cedric (viadezo1er)]

Not really sure how to test this but there is definitely a lack of tests.
OAuth link
TUIs
interactive wizard

[ViaDézo1er / cedric (viadezo1er)]

bt auth profiles does nothing

[Abhijeet Prasad (AbhiPrasad)]

Ideally we can split this file up a bit, getting unwieldy to review

[Abhijeet Prasad (AbhiPrasad)]

could we maybe split this PR up so the changes are a bit easier to review and land? Let me know if you need advice on how that should be. Claude told me this:

Based on the diff, here's how I'd split it into 4-5 independent PRs that can land in order:

PR 1: Flag restructuring + deprecation

The mechanical rename/removal of CLI flags. Smallest blast radius, easiest to review.

• --quiet → --verbose (invert default)
• --agents (repeatable) → --agent (single), remove AgentArg::All
• --no-mcp-skill → --no-skills + --no-mcp
• Add --no-instrument, --tui/--background, --no-workflow
• Remove --no-input, move --yes to #[arg(skip)]
• set_quiet(true) as global default
• All the BaseArgs field renames (quiet→verbose) and test fixups
• Consider adding hidden deprecation aliases so existing scripts don't break

PR 2: TTY fallback + agent detection

Self-contained infra changes that the wizard depends on.

• tty_term() function in src/ui/select.rs (/dev/tty fallback)
• fuzzy_select using interact_on(&term)
• detect_runnable_agents() (PATH-only check)
• detect_agents simplification (drop config-dir heuristics)
• resolve_unambiguous_instrument_agent
• /dev/tty stdin fallback in run_agent_invocation for interactive mode

PR 3: Auth + API key creation

The OAuth-related changes, isolated from the wizard.

• ensure_auth rewrite (credential recovery flow, is_missing_credential_error)
• maybe_create_api_key_for_oauth (auto-create key, print-once)
• login_interactive_oauth made pub(crate)
• #[allow(dead_code)] on login_interactive_api_key (or just remove it)
• Remove login_interactive (the old method-chooser)
• Auth output tweaks (warning→info, spacing)

PR 4: MCP key embedding

Depends on PR 3 (needs the API key to embed).

• merge_mcp_config takes api_key + mcp_url params
• install_mcp_for_claude_user (uses claude mcp add -s user)
• mcp_url_from_api_url helper
• run_mcp_setup becomes async (needs auth resolution)
• Updated tests for merge_mcp_config / install_mcp_for_agent

PR 5: Wizard UX + template changes

The big UX PR, but now much smaller because infra landed in PRs 1-4.

• --interactive/-i flag and wizard flow
• ASCII logo + welcome banner
• select_project with auto-create {whoami}-test and + Create new project
• Consolidate with select_project_interactive (deduplicate)
• prompt_instrument_language_selection, prompt_instrument_workflow_selection in wizard
• Agent invocation rework (Claude --permission-mode/--settings, Codex/Cursor TUI vs background split)
• instrument-task.md and braintrust-url-formats.md template changes
• LanguageArg::doc_filename()
• render_instrument_task rework
• SSE strip_prefix cleanup (could also go in PR 1)

---

Dependency order: PR 1 → PR 2 → PR 3 → PR 4 → PR 5 (though 2 and 3 are independent of each other and could land in parallel).

The biggest win is separating the flag restructuring (PR 1) from the wizard logic (PR 5) — those are the two hardest things to review when interleaved.

[Parker Henderson (parkerhendo)]

I don't know if we should remove this. there should be an explicit wait for users to tell the cli that the current session is not interactive/to not require user input

[Parker Henderson (parkerhendo)]

see other comment. i don't think we should remove this, but it's possible I'm missing something.