Learning by Patrik

Develop an Azure Speech Voice Live agent in Microsoft Foundry | AI-103 | Episode 20

Azure Voice Live provides a managed speech-to-speech layer for building low-latency voice agents. Instead of manually chaining speech-to-text → LLM → text-to-speech, Voice Live exposes a real-time, bidirectional session that can listen and respond with streaming audio.

How it works

Client ⇄ WebSocket ⇄ Voice Live ⇄ Foundry Agent / Model

Connect to Voice Live and authenticate.
Configure the session — model/agent, modalities, voice, audio format and turn detection.
Stream microphone audio into the session.
Handle server events and immediately play streamed response audio.
Use VAD (Voice Activity Detection) to detect when the user starts/stops speaking.

# Conceptual SDK flow
connection = await connect(endpoint, credential, agent)

await connection.session.update(
    modalities=["text", "audio"],
    input_audio_format="pcm16",
    turn_detection="server_vad"
)

async for event in connection:
    if event.type == "input_audio_buffer.speech_started":
        stop_playback()          # user interrupts / barge-in

    elif event.type == "response.audio.delta":
        play(event.delta)        # streamed agent audio

Key concepts to remember

Full-duplex / real-time: audio flows bidirectionally through a persistent connection.
Authentication: Microsoft Entra ID is recommended; Voice Live also supports API-key authentication.
PCM16 vs G.711: PCM16 = uncompressed/high fidelity; G.711 = smaller, speech-oriented encoding.
input_audio_buffer.speech_started is especially important: with server VAD it signals detected user speech and can be used to stop current playback when the user interrupts.
Voice Live can additionally provide noise suppression, echo cancellation, turn detection, voices, avatars and function calling.