[BOT ISSUE] Google GenAI: Video generation API (`models.generate_videos()`) not instrumented

[braintrust-bot]

Summary

The Google GenAI video generation API (models.generate_videos() / async_models.generate_videos()) is not instrumented. Calls to generate videos using Veo models produce zero Braintrust tracing. This is the video counterpart to models.generate_images(), which IS instrumented in this repo.

What is missing

Google GenAI Method	Instrumented?
models.generate_content()	Yes
models.generate_content_stream()	Yes
models.embed_content()	Yes
models.generate_images()	Yes
models.generate_videos()	No
async_models.generate_videos()	No

The generate_videos() method is the SDK entry point for text-to-video and image-to-video generation using Google's Veo models (Veo 2 is stable; Veo 3.1 is preview). It returns an asynchronous long-running Operation that must be polled until completion.

At minimum, instrumentation should create a span capturing:

• Input: prompt text, model name, image reference (for image-to-video), generation config
• Output: video metadata (duration, format, number of videos generated)
• Metrics: latency (total operation time from submission to completion)
• Metadata: model, aspect ratio, resolution, person generation setting

The pattern would mirror the existing ModelsGenerateImagesPatcher / AsyncModelsGenerateImagesPatcher, adapted for the video response format and long-running operation semantics.

Braintrust docs status

not_found — The Gemini integration page documents generate_content, streaming, function calling, structured outputs, thinking tokens, and context caching. Image generation is supported via the existing generate_images patcher. Video generation is not mentioned.

Upstream sources

• Google GenAI video generation guide: https://ai.google.dev/gemini-api/docs/video
• Google GenAI Python SDK: client.models.generate_videos(model="veo-2-generate-001", ...) — stable Veo 2 model
• SDK API reference: https://googleapis.github.io/python-genai/
• Veo 2 model is marked "Stable" in Google's model listing; Veo 3.1 models are "Preview"

Local files inspected

• py/src/braintrust/integrations/google_genai/patchers.py — defines ModelsGenerateImagesPatcher and AsyncModelsGenerateImagesPatcher but no video generation patchers; zero references to generate_videos
• py/src/braintrust/integrations/google_genai/tracing.py — contains _generate_images_wrapper and _async_generate_images_wrapper but no video generation wrappers
• py/src/braintrust/integrations/google_genai/integration.py — integration class registers image generation patchers but no video patchers
• py/src/braintrust/integrations/google_genai/test_google_genai.py — no video generation test cases
• py/noxfile.py — test_google_genai session exists but no video-specific coverage

Relationship to existing issues

This is the same class of gap as #124 (OpenAI Images API not instrumented) — a stable generative media API that the wrapper silently skips. The difference is that Google GenAI generate_images() IS already instrumented, making generate_videos() an adjacent gap in the same integration.

[starfolkai]

Drop into this session: sfk devbox session 79

Both cases work correctly. Here's the full summary:

---

Summary

Issue

braintrust.framework._call_user_fn_args (framework.py:508-510) silently reassigned a BT-provided kwarg (e.g. metadata) to any scorer parameter whose name wasn't found in BT's kwargs — including DI/configuration parameters with declared defaults. This caused AttributeError: 'dict' object has no attribute 'threshold' at runtime with no warning.

Root Cause

# BEFORE (buggy)
else:
    next_arg = list(kwargs.keys())[0]       # pops first available kwarg
    final_kwargs[name] = kwargs.pop(next_arg)  # assigns it regardless of type

The positional fallback fired unconditionally — even for parameters with default values that should never receive a BT kwarg.

Fix (framework.py:508-514)

Only apply the positional fallback for required parameters (those with param.default is inspect.Parameter.empty). Parameters with defaults are skipped; Python uses the declared default when the function is called.

# AFTER (fixed)
elif param.default is inspect.Parameter.empty and kwargs:
    # Required parameter not found by name: fall back to positional binding
    # (backward compat for scorers using non-standard names like `input_value`)
    next_arg = list(kwargs.keys())[0]
    final_kwargs[name] = kwargs.pop(next_arg)
# else: has a default — skip, Python uses it

Verification

Real execution (pre-fix): Braintrust API experiment main-1775720090 showed my_scorer scoring 0.0 because config received {'key': 'val'} (the metadata dict) and .threshold raised AttributeError.

Before/After (unit):

• Case 1 (DI param with default config: Config = DEFAULT_CONFIG): BEFORE → config = {'k': 'v'} (dict, bug). AFTER → config not injected, Python uses Config(threshold=0.5). ✅
• Case 2 (required param input_value without default): BEFORE and AFTER → input_value = 'hello' (positional fallback preserved). ✅

Tests: 3 new tests that were RED before fix are all GREEN after. Full test_framework.py suite: 30/30 passing. No regressions introduced (1 pre-existing devserver failure unrelated to this change).