Add whisper-large-v3-turbo contrib model

[jimburtoft]

Description

Adds whisper-large-v3-turbo (809M params) as a contrib model for NxD Inference. This is an encoder-decoder speech-to-text model built on the OpenAI Whisper base class, with four performance optimizations and batched inference (BS>1) support.
Key optimizations:

• Cross-attention K/V cache — eliminates redundant K/V projections during decode (~2.5x decode speedup, saves 19.7B FLOPs/token)
• Fused QKV projections — 3 separate Q/K/V matmuls replaced with a single fused operation in self-attention
• NKI flash attention in all 32 encoder layers (bidirectional)
• NKI fused Conv1D+GELU encoder frontend kernel (optional, graceful fallback if nkilib unavailable)

Model Information

Model Name: whisper-large-v3-turbo
Model Architecture: Encoder-Decoder transformer (32 encoder layers + 4 decoder layers, 1280 hidden, 20 heads)
Purpose: Speech-to-text (automatic speech recognition)

Checklist

Please ensure your PR includes the following items. Refer to the contrib/CONTRIBUTING.md for detailed guidelines.

Required Components

• Accuracy Test (ex. test/integration/test_model.py)
  • Integration test that compiles, loads, and transcribes reference audio
  • Validates non-empty transcription output and measures latency
  • Test can compile and run the model on Neuron
• README.md with the following sections:
  • Usage Example: Clear code example showing how to use the model
  • Compatibility Matrix: Table showing tested Neuron SDK versions and instance types (Trn2)
  • Example Checkpoints: Links to compatible model checkpoints (HuggingFace Hub)
  • Testing Instructions: Command to run the test suite for the model
• Source Code (src/)
  • Modeling code following NxD Inference patterns
  • Properly structured in the contrib folder hierarchy

Optional Components

• Unit Tests (CPU or Neuron-based)
  • Tests for individual modeling components
  • Located in test/unit/ directory

Folder Structure

Confirm your contribution follows this structure:

/contrib/models/whisper-large-v3-turbo/
  README.md
  /src
    __init__.py
    modeling_whisper.py
    whisper_encoder_megakernel.py
    /utils
      __init__.py
      config.py
      decoding.py
      state_dict.py
  /test
    /unit (empty)
    /integration
      test_model.py

Testing

How did you test this change?
End-to-end validation on a fresh trn2.3xlarge instance using DLAMI Neuron (Ubuntu 24.04) 20260227 (SDK 2.28). Installed NxDI from the contrib/whisper branch via pip, ran the full notebook covering BS=1 compile/load/transcribe/benchmark and BS=4 compile/load/decode/benchmark. All 37 notebook cells executed without errors.
Test Results:
Measured on 7.3s audio (single 30s segment):

Config	Batch Latency	Audio/s	Audio-sec/s/core
BS=1, 1 core	195ms	5.1	37.2
BS=1, DP=4	195ms/core	20.4	37.2
BS=4, 1 core	487ms	8.2	59.9
BS=4, DP=4	512ms/core	31.2	56.9

• BS=4 achieves 1.6x the hardware utilization (audio-sec/s/core) of BS=1
• 0% WER vs CPU baseline on short and medium audio
• 29x faster than CPU (whisper-large-v3-turbo on 12 vCPUs) for short audio

Compatibility

Tested with:

• Neuron SDK Version(s): 2.28
• Instance Type(s): trn2.3xlarge
• PyTorch Version: 2.9.0
• Python Version: 3.12.3

Additional Information

• This is the first encoder-decoder (non-CausalLM) contrib model. It uses the OpenAI Whisper package as its base class rather than HuggingFace Transformers.
• ffmpeg is a system dependency required by openai-whisper for audio decoding.
• Each batch size requires separate compilation (BS is baked into the traced graph).
• BS=4 is the optimal batch size on trn2. BS=8 causes a 10x+ decoder latency explosion.
• The NKI fused Conv1D+GELU kernel requires nkilib; it gracefully falls back to standard ops if unavailable.

Related Issues

N/A

vLLM Integration

• This model/feature is intended for use with vLLM
• Documentation includes vLLM registration instructions

---

By submitting this PR, I confirm that:

• I have read and followed the contributing guidelines
• This is a community contribution and may have limited testing compared to officially-supported models
• The code follows best practices and is well-documented
• All required components listed above are included