AssemblyAI Voice Agent API vs Ultravox Realtime API

AssemblyAI Voice Agent API

One $0.075 a minute rate covers speech-to-text, the managed model, speech output, recordings and hosting, and the socket has an AsyncAPI file, typed error codes and a 30-second resume window. No concurrent-session limit is published, the status page has no Voice Agent component, and phone calls are inbound only through a Twilio trunk the customer owns.

Ultravox Realtime API

Flat $0.05 a minute including model and built-in voices. No MCP server.

AssemblyAI Voice Agent API

1. Send session.end before closing the socket. Without it the session stays resumable, and billed, for 30 seconds
2. Retry only at_capacity, concurrency_exceeded and internal_error, with backoff. Every other session error code is fatal
3. Send tool.result inside the reply.done handler, not straight after tool.call
4. Mint a one-time token with GET /v1/token for browser clients and never ship the API key to a page
5. Deduplicate webhooks on event_id, and verify X-AAI-Signature against the raw body bytes because a 401 is not retried

Ultravox Realtime API

1. Honour Retry-After on 429 and 503 when creating calls
2. Pin the model string (ultravox-v0.7 or ultravox-v0.6) rather than relying on the default
3. Hand clients the joinUrl, never the API key
4. Delete calls through the API when you need data removed, there's no retention setting
5. Use call stages for multi-step flows instead of one long prompt