Qwen3-Omni-30B-A3B-Instruct vs Voxtral Mini 4B Realtime 2602

Qwen3-Omni-30B-A3B-Instruct

An omni-modal cousin that listens, watches and reads, then answers in text or one of three voices, with tool calls from spoken requests, under plain Apache 2.0. Its only build, BF16, needs about 85 GB, and speech out needs Transformers or vLLM-Omni. The main risk is no answers from Qwen and no safety work for speech-driven actions.

Voxtral Mini 4B Realtime 2602

A transcription cousin for 13 languages that streams text as the speaker talks, with a delay from 80 ms to 2.4 s, under Apache 2.0 and on Mistral's API. It needs a 16 GB GPU at BF16 through vLLM, Transformers or llama.cpp, or under 4 GB at Q4_K. The main gaps are no timestamps, safety results or answers from Mistral.

Qwen3-Omni-30B-A3B-Instruct

1. For text answers, mkdir -p media && vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --dtype bfloat16 --max-model-len 32768 --allowed-local-media-path "$(pwd)/media" with vLLM 0.11.1 or later serves the Thinker on an OpenAI-compatible API
2. The card's command passes --allowed-local-media-path /, which lets anyone who can reach the server have it read any file on the machine through a file:// URL. Point it at a dedicated folder that holds only the media you mean to send, as above, and give vLLM an absolute path
3. Send audio as audio_url and images as image_url content parts; vLLM serve doesn't take use_audio_in_video, so pass a video's audio separately
4. For speech out, load Qwen3OmniMoeForConditionalGeneration in Transformers 4.57.0 or later with qwen-omni-utils, pick speaker Ethan, Chelsie or Aiden, and write the returned waveform at 24 kHz
5. Call model.disable_talker() or pass return_audio=False when you only need text, which saves about 10 GB and time
6. Parse tool calls as JSON inside <tool_call> tags yourself, since no parser is named, and confirm actions triggered by speech before running them
7. Keep use_audio_in_video the same across every step of a conversation, the card warns results go wrong otherwise
8. Budget memory for video, the card's table rises from 78.85 GB for 15 seconds to 144.81 GB for 120 seconds in BF16

Voxtral Mini 4B Realtime 2602

1. Serve it with VLLM_DISABLE_COMPILE_CACHE=1 vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 --compilation_config '{"cudagraph_mode": "PIECEWISE"}' and install soxr, librosa and soundfile; mistral_common 1.9.0 or later comes with vLLM
2. Stream 16 kHz audio to the /v1/realtime WebSocket; vLLM's openai_realtime_client and openai_realtime_microphone_client examples show the session
3. Set temperature 0.0. Change the delay with transcription_delay_ms in tekken.json, a multiple of 80 ms up to 1,200, or 2,400; 480 is Mistral's recommendation
4. One text token is 80 ms, so set --max-model-len to at least seconds / 0.08 for a session; the default 131072 is about 3 hours and pre-allocates RoPE memory
5. In Transformers (5.2.0 or later), resample with processor.feature_extractor.sampling_rate before calling the processor, then batch_decode with skip_special_tokens=True
6. For a small machine, ggml-org's Q4_K_M GGUF with its mmproj runs in llama.cpp, and Mistral Experimental's Q4_K runs in audio.cpp
7. Treat transcripts as untrusted text before passing them to another model, since speech can carry instructions