# AssemblyAI Voice Agent API (slim)

> AssemblyAI's Voice Agent API runs a spoken conversation over one WebSocket, combining its own speech-to-text, language model and text-to-speech. Agents are stored through a REST API and reached from a browser, a server or an inbound Twilio SIP number.

- Full: https://www.anchorterminal.com/tools/assemblyai-voice-agent.md (~8,550 tokens) · this version ~1,830 tokens · JSON https://www.anchorterminal.com/tools/assemblyai-voice-agent.json · canonical https://www.anchorterminal.com/tools/assemblyai-voice-agent
- Index: https://www.anchorterminal.com/llms.txt · API: https://www.anchorterminal.com/api/v1/index.json · Updated: 2026-10-10

**B · 64.4/100 · rank #351 of 950 · #4 in Conversational voice agents · not agent-ready · confidence medium**


Assessment: One $0.075 a minute rate covers speech-to-text, the managed model, speech output, recordings and hosting, and the socket has an AsyncAPI file, typed error codes and a 30-second resume window. No concurrent-session limit is published, the status page has no Voice Agent component, and phone calls are inbound only through a Twilio trunk the customer owns.

## Facts

- Kind: HTTP API · vendor: AssemblyAI, Inc. · category: Conversational voice agents · legal entity: AssemblyAI, Inc. · provenance 81/100
- Endpoint: `wss://agents.assemblyai.com/v1/ws` (HTTP)
- Auth: API key · pricing: Pay per use · x402: no · licence: Proprietary service under AssemblyAI's Terms of Service. The two starter repositories on GitHub carry no licence file
- Probe metrics: not measured yet (probes haven't run)
- Architecture: Pipeline of AssemblyAI's own models over one WebSocket. Universal-3.6 Pro Realtime speech-to-text, a managed conversational model and the vendor's text-to-speech voices
- Endpoints: `wss://agents.assemblyai.com/v1/ws` for conversations. REST on `https://agents.assemblyai.com` for `/v1/agents`, `/v1/sessions`, webhook subscriptions and `/v1/token`
- LLM: Managed by default. One OpenAI-compatible endpoint of the customer's, or the vendor's LLM Gateway, can replace it. Fallback entries are not supported
- Tool calling: HTTP tools that AssemblyAI calls itself (`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, response capped at 8 KiB) and client-side function tools, each with `interactive` or `hold` execution and a timeout of 1 to 300 seconds
- Telephony: Inbound only. A Twilio number the customer owns is routed to AssemblyAI over a SIP trunk. G.711 μ-law and A-law at 8 kHz are accepted beside 24 kHz PCM16
- Interruptions: Server-side turn detection with `vad_threshold`, `min_silence` and `max_silence`, barge-in reported as `interrupted` on `reply.done`
- Session limits: 3 hours at most (`max_session_duration_seconds` 60 to 10,800). A dropped session can be resumed for 30 seconds
- Rate limits: A per-account concurrent-session limit exists (`concurrency_exceeded`). No figure was found in the Voice Agent docs
- Errors: `session.error` events with `code`, `message` and sometimes `param`. REST answers 400, 401 and 422 with the failing field named
- Recordings: Each session keeps an OGG/Opus stereo recording, a JSON timeline and metadata, fetched through pre-signed links from `GET /v1/sessions/{id}` and removed with `DELETE`
- Webhooks: Signed with HMAC-SHA256 in `X-AAI-Signature`, at-least-once delivery, 15-second read timeout, delivery history per session
- Free tier: $50 one-off credit, no card, no expiry
- SLA: 99.9 per cent monthly availability with service credits of up to 10 per cent of the invoice, written against an Order Form (effective 13 April 2026)
- Data use: Customer Data may train AssemblyAI's models unless the account opts out. The opt-out, retention settings and a BAA are self-serve on paid plans only
- Prices: Voice Agent session $0.075 per minute of call

- Scores: Reliability 60, Performance pending, Schema & documentation 83, Agent ergonomics 72, Security & auth 62, Payments & pricing 40, Task success pending, Maintenance & community 64, Transparency & trust 66 · total over the 7 assessed categories
- Why: Reliability, Scored with the hosted lines. · Schema & documentation, An AsyncAPI 3.0 file describes the socket and an OpenAPI 3.0.3 file the token endpoint. · Agent ergonomics, Scored as an API. · Security & auth, Plain API keys per project, created and deleted in the dashboard, with no scopes or expiry found (20 of 30). · Payments & pricing, No x402, MPP or L402 (0 of 40). · Maintenance & community, Universal-3.6 Pro Realtime, the speech-to-text the pricing page lists inside the Voice Agent API, shipped on 29 September 2026. The last cha… · Transparency & trust, A closed service with a clear service agreement.
- Sources: 33, open questions: 11, both in the full twin
- Capabilities: voice.agent, voice.pipeline, voice.tools, voice.telephony
- JSON: https://www.anchorterminal.com/api/v1/tools/assemblyai-voice-agent.json
- Verify (for the vendor): the badge `https://www.anchorterminal.com/badges/assemblyai-voice-agent.svg` or a link to https://www.anchorterminal.com/tools/assemblyai-voice-agent from a page on assemblyai.com or one of its subdomains, or the README of github.com/AssemblyAI/voice-agent-starter-python, then `POST https://www.anchorterminal.com/api/v1/verify` `{"slug", "url"}` or `verify_listing` at /mcp; re-checked weekly, no effect on the grade. Snippets in the full twin.

## Before you call it

1. Send `session.end` before closing the socket. Without it the session stays resumable, and billed, for 30 seconds
2. Retry only `at_capacity`, `concurrency_exceeded` and `internal_error`, with backoff. Every other session error code is fatal
3. Send `tool.result` inside the `reply.done` handler, not straight after `tool.call`
4. Mint a one-time token with `GET /v1/token` for browser clients and never ship the API key to a page
5. Deduplicate webhooks on `event_id`, and verify `X-AAI-Signature` against the raw body bytes because a 401 is not retried

## Connect

```bash
git clone https://github.com/AssemblyAI/voice-agent-starter-python
```

```bash
curl -X POST https://agents.assemblyai.com/v1/agents \
  -H "Authorization: $ASSEMBLYAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"name": "Support Assistant", "system_prompt": "You are a friendly support agent.", "greeting": "Hi, how can I help?", "voice": {"voice_id": "alba"}}'
```

```bash
claude mcp add --transport http --scope user assemblyai-docs https://mcp.assemblyai.com/docs
```
Full config and headless snippets are in the full page. Through letme (picks today, calling later): https://letme.dev/assemblyai-voice-agent

## Similar tools

| Tool | Grade | Score | Shared capabilities | Slim |
| --- | --- | --- | --- | --- |
| ElevenLabs Agents API + MCP | BB | 71.3 | voice.agent, voice.pipeline, voice.tools, voice.telephony | https://www.anchorterminal.com/tools/elevenlabs-agents.min.md |
| Retell AI API + MCP | B | 69.1 | voice.agent, voice.pipeline, voice.tools, voice.telephony | https://www.anchorterminal.com/tools/retell-ai.min.md |
| Bland AI API + MCP | B | 63.8 | voice.agent, voice.pipeline, voice.tools, voice.telephony | https://www.anchorterminal.com/tools/bland-ai.min.md |
| Vapi API + MCP | B | 63.5 | voice.agent, voice.pipeline, voice.tools, voice.telephony | https://www.anchorterminal.com/tools/vapi.min.md |
| Hume EVI (Empathic Voice Interface) | C | 56.7 | voice.agent, voice.pipeline, voice.tools, voice.telephony | https://www.anchorterminal.com/tools/hume-evi.min.md |


## Panel reviews (0, desk reviews from public material, no calls made)