Add live audio transcription streaming support to Foundry Local C# SDK

[rui-ren]

Here's the cleaned version:

---

Description:

Adds real-time audio streaming support to the Foundry Local C# SDK, enabling live microphone-to-text transcription via ONNX Runtime GenAI's StreamingProcessor API (Nemotron ASR).

The existing OpenAIAudioClient only supports file-based transcription. This PR introduces LiveAudioTranscriptionSession that accepts continuous PCM audio chunks (e.g., from a microphone) and returns partial/final transcription results as an async stream.

What's included

New files

• src/OpenAI/LiveAudioTranscriptionClient.cs — Streaming session with StartAsync(), AppendAsync(), GetTranscriptionStream(), StopAsync()
• src/OpenAI/LiveAudioTranscriptionTypes.cs — LiveAudioTranscriptionResponse (extends AudioCreateTranscriptionResponse) and CoreErrorResponse types
• test/FoundryLocal.Tests/LiveAudioTranscriptionTests.cs — Unit tests for deserialization, settings, state guards

Modified files

• src/OpenAI/AudioClient.cs — Added CreateLiveTranscriptionSession() factory method
• src/Detail/ICoreInterop.cs — Added StreamingRequestBuffer struct, StartAudioStream, PushAudioData, StopAudioStream interface methods
• src/Detail/CoreInterop.cs — Routes audio commands through existing execute_command / execute_command_with_binary native entry points
• src/Detail/JsonSerializationContext.cs — Registered LiveAudioTranscriptionResponse for AOT compatibility
• README.md — Added live audio transcription documentation

API surface

var audioClient = await model.GetAudioClientAsync();
var session = audioClient.CreateLiveTranscriptionSession();

session.Settings.SampleRate = 16000;
session.Settings.Channels = 1;
session.Settings.Language = "en";

await session.StartAsync();

// Push audio from microphone callback
await session.AppendAsync(pcmBytes);

// Read results as async stream
await foreach (var result in session.GetTranscriptionStream())
{
    Console.Write(result.Text);
}

await session.StopAsync();

Design highlights

• Output type alignment — LiveAudioTranscriptionResponse extends AudioCreateTranscriptionResponse for consistent output format with file-based transcription
• Internal push queue — Bounded Channel<T> serializes audio pushes from any thread (safe for mic callbacks) with backpressure
• Fail-fast on errors — Push loop terminates immediately on any native error (no retry logic)
• Settings freeze — Audio format settings are snapshot-copied at StartAsync() and immutable during the session
• Cancellation-safe stop — StopAsync always calls native stop even if cancelled, preventing native session leaks
• Dedicated session CTS — Push loop uses its own CancellationTokenSource, decoupled from the caller's token
• Routes through existing exports — StartAudioStream and StopAudioStream route through execute_command; PushAudioData routes through execute_command_with_binary — no new native entry points required

Core integration (neutron-server)

The Core side (AudioStreamingSession.cs) uses StreamingProcessor + Generator + Tokenizer + TokenizerStream from onnxruntime-genai to perform real-time RNNT decoding. The native commands (audio_stream_start/push/stop) are handled as cases in NativeInterop.ExecuteCommandManaged / ExecuteCommandWithBinaryManaged.

Verified working

• ✅ SDK build succeeds (0 errors, 0 warnings)
• ✅ Unit tests for JSON deserialization, type inheritance, settings, state guards
• ✅ GenAI StreamingProcessor pipeline verified with WAV file (correct transcript)
• ✅ Core TranscribeChunk byte[] PCM path matches reference float[] path exactly
• ✅ Full E2E simulation: SDK Channel + JSON serialization + session management
• ✅ Live microphone test: real-time transcription through SDK → Core → GenAI

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Mar 30, 2026 9:08pm

[Copilot]

Pull request overview

Adds a new C# SDK API for live/streaming audio transcription sessions (push PCM chunks, receive incremental/final text results) and includes a Windows microphone demo sample.

Changes:

• Introduces LiveAudioTranscriptionSession + result/error types for streaming ASR over Core interop.
• Extends Core interop to support audio stream start/push/stop (including binary payload routing).
• Adds a samples/cs/LiveAudioTranscription demo project and updates the audio client factory API.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
sdk_v2/cs/test/FoundryLocal.Tests/Utils.cs	Replaced prior test utilities with ad-hoc top-level streaming harness code (currently breaks test build).
sdk_v2/cs/test/FoundryLocal.Tests/ModelTests.cs	Adds trailing blank lines (formatting noise).
sdk_v2/cs/src/OpenAI/LiveAudioTranscriptionTypes.cs	Adds LiveAudioTranscriptionResult and a structured Core error type.
sdk_v2/cs/src/OpenAI/LiveAudioTranscriptionClient.cs	Adds LiveAudioTranscriptionSession implementation (channels, retry, stop semantics).
sdk_v2/cs/src/OpenAI/AudioClient.cs	Adds CreateLiveTranscriptionSession() and removes the public file streaming transcription API.
sdk_v2/cs/src/Detail/JsonSerializationContext.cs	Registers new audio streaming types for source-gen JSON.
sdk_v2/cs/src/Detail/ICoreInterop.cs	Adds interop structs + methods for audio stream start/push/stop.
sdk_v2/cs/src/Detail/CoreInterop.cs	Implements binary command routing via execute_command_with_binary and start/stop routing via execute_command.
sdk_v2/cs/src/AssemblyInfo.cs	Adds InternalsVisibleTo("AudioStreamTest").
samples/cs/LiveAudioTranscription/README.md	Documentation for the live transcription demo sample.
samples/cs/LiveAudioTranscription/Program.cs	Windows microphone demo using NAudio + new session API.
samples/cs/LiveAudioTranscription/LiveAudioTranscription.csproj	Adds sample project dependencies and references the SDK project (path currently incorrect).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.