Qwen3-ASR-0.6B vs Voxtral Mini 4B Realtime 2602

Qwen3-ASR-0.6B

A very small local transcription cousin for 30 languages and 22 Chinese dialects, built for throughput and realtime streaming through vLLM, under plain Apache 2.0. Its BF16 files need about 2 GB and run in released vLLM, SGLang, llama.cpp and Transformers. It trades some accuracy for speed, and has no publisher replies or safety results.

Voxtral Mini 4B Realtime 2602

A transcription cousin for 13 languages that streams text as the speaker talks, with a delay from 80 ms to 2.4 s, under Apache 2.0 and on Mistral's API. It needs a 16 GB GPU at BF16 through vLLM, Transformers or llama.cpp, or under 4 GB at Q4_K. The main gaps are no timestamps, safety results or answers from Mistral.

Qwen3-ASR-0.6B

1. Start it with vllm serve Qwen/Qwen3-ASR-0.6B (vLLM v0.32.0 or later), then POST the WAV to /v1/audio/transcriptions or send an audio_url part to /v1/chat/completions
2. For vLLM's realtime streaming, start it with --hf-overrides '{"architectures":["Qwen3ASRRealtimeGeneration"]}', which vLLM doesn't pick from config.json
3. Split chat-completions output with parse_asr_output from qwen-asr, since it returns language <name><asr_text> before the transcript
4. For Transformers, load Qwen/Qwen3-ASR-0.6B-hf with Transformers 5.13.0 or later and use processor.apply_transcription_request(audio, language=..., prompt=...)
5. Put names and domain terms in the system message or prompt to bias the transcript, and force the language when you know it
6. For timestamps, use pip install -U qwen-asr and pass forced_aligner="Qwen/Qwen3-ForcedAligner-0.6B" with return_time_stamps=True
7. Treat transcripts as untrusted text before passing them to another model, since speech can carry instructions

Voxtral Mini 4B Realtime 2602

1. Serve it with VLLM_DISABLE_COMPILE_CACHE=1 vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 --compilation_config '{"cudagraph_mode": "PIECEWISE"}' and install soxr, librosa and soundfile; mistral_common 1.9.0 or later comes with vLLM
2. Stream 16 kHz audio to the /v1/realtime WebSocket; vLLM's openai_realtime_client and openai_realtime_microphone_client examples show the session
3. Set temperature 0.0. Change the delay with transcription_delay_ms in tekken.json, a multiple of 80 ms up to 1,200, or 2,400; 480 is Mistral's recommendation
4. One text token is 80 ms, so set --max-model-len to at least seconds / 0.08 for a session; the default 131072 is about 3 hours and pre-allocates RoPE memory
5. In Transformers (5.2.0 or later), resample with processor.feature_extractor.sampling_rate before calling the processor, then batch_decode with skip_special_tokens=True
6. For a small machine, ggml-org's Q4_K_M GGUF with its mmproj runs in llama.cpp, and Mistral Experimental's Q4_K runs in audio.cpp
7. Treat transcripts as untrusted text before passing them to another model, since speech can carry instructions