Qwen3-ASR-1.7B vs Voxtral Mini 4B Realtime 2602

Qwen3-ASR-1.7B

A small local transcription cousin for 30 languages and 22 Chinese dialects that identifies the language, takes context to get names right and streams through vLLM, under plain Apache 2.0. Its BF16 files need about 5 GB and run in released vLLM, SGLang, llama.cpp and Transformers. The main risk is upkeep, with no publisher replies or safety results.

Voxtral Mini 4B Realtime 2602

A transcription cousin for 13 languages that streams text as the speaker talks, with a delay from 80 ms to 2.4 s, under Apache 2.0 and on Mistral's API. It needs a 16 GB GPU at BF16 through vLLM, Transformers or llama.cpp, or under 4 GB at Q4_K. The main gaps are no timestamps, safety results or answers from Mistral.

Qwen3-ASR-1.7B

1. Start it with vllm serve Qwen/Qwen3-ASR-1.7B (vLLM v0.32.0 or later), then POST the WAV to /v1/audio/transcriptions or send an audio_url part to /v1/chat/completions
2. Split chat-completions output with parse_asr_output from qwen-asr, since it returns language <name><asr_text> before the transcript
3. For Transformers, load Qwen/Qwen3-ASR-1.7B-hf with Transformers 5.13.0 or later and use processor.apply_transcription_request(audio, language=..., prompt=...), then decode(..., return_format="transcription_only")
4. Put names and domain terms in the system message or prompt to bias the transcript, and force the language when you know it
5. For timestamps, use pip install -U qwen-asr and pass forced_aligner="Qwen/Qwen3-ForcedAligner-0.6B" with return_time_stamps=True
6. Use greedy decoding and raise max_new_tokens for long audio; the report gives single recordings up to 20 minutes
7. Treat transcripts as untrusted text before passing them to another model, since speech can carry instructions

Voxtral Mini 4B Realtime 2602

1. Serve it with VLLM_DISABLE_COMPILE_CACHE=1 vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 --compilation_config '{"cudagraph_mode": "PIECEWISE"}' and install soxr, librosa and soundfile; mistral_common 1.9.0 or later comes with vLLM
2. Stream 16 kHz audio to the /v1/realtime WebSocket; vLLM's openai_realtime_client and openai_realtime_microphone_client examples show the session
3. Set temperature 0.0. Change the delay with transcription_delay_ms in tekken.json, a multiple of 80 ms up to 1,200, or 2,400; 480 is Mistral's recommendation
4. One text token is 80 ms, so set --max-model-len to at least seconds / 0.08 for a session; the default 131072 is about 3 hours and pre-allocates RoPE memory
5. In Transformers (5.2.0 or later), resample with processor.feature_extractor.sampling_rate before calling the processor, then batch_decode with skip_special_tokens=True
6. For a small machine, ggml-org's Q4_K_M GGUF with its mmproj runs in llama.cpp, and Mistral Experimental's Q4_K runs in audio.cpp
7. Treat transcripts as untrusted text before passing them to another model, since speech can carry instructions