Ming-3.0-asr-tiny vs Voxtral Mini 4B Realtime 2602

Ming-3.0-asr-tiny

A local transcription cousin for Chinese, English and Chinese dialects that takes hotwords or dialogue context to get names right, under MIT. It needs one CUDA GPU with about 21 GB and runs only through the publisher's remote code or SGLang-Omni's main branch. The main risk is running the publisher's Python on load, with no safety results.

Voxtral Mini 4B Realtime 2602

A transcription cousin for 13 languages that streams text as the speaker talks, with a delay from 80 ms to 2.4 s, under Apache 2.0 and on Mistral's API. It needs a 16 GB GPU at BF16 through vLLM, Transformers or llama.cpp, or under 4 GB at Q4_K. The main gaps are no timestamps, safety results or answers from Mistral.

Ming-3.0-asr-tiny

1. Start it with hf download inclusionAI/Ming-3.0-asr-tiny --local-dir Ming-3.0-asr-tiny, then pip install -r requirements.txt in Python 3.12 with CUDA (Transformers 5.12.1, fla-core 0.5.2, PyTorch 2.11.0), then python examples/transcribe.py --model . --audio speech.wav --language-hint en
2. To serve it, sgl-omni serve --model-path inclusionAI/Ming-3.0-asr-tiny from SGLang-Omni's main branch, then POST the WAV to /v1/audio/transcriptions with response_format json and read the text field
3. Parse raw Transformers output with decode_transcript from asr_request.py, which strips the language label before the tab
4. Pass language zh or en when known, anything else is rejected, and append a topic or hotwords to the base instruction with no newline
5. Load with dtype torch.bfloat16 rather than casting the model, use greedy decoding, max_new_tokens 256 and no_repeat_ngram_size 10, and don't apply the chat template
6. Split long recordings yourself or use --audio-chunk-frames, since input and output share 8,192 tokens
7. Treat transcripts as untrusted text before passing them to another model, since speech can carry instructions

Voxtral Mini 4B Realtime 2602

1. Serve it with VLLM_DISABLE_COMPILE_CACHE=1 vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 --compilation_config '{"cudagraph_mode": "PIECEWISE"}' and install soxr, librosa and soundfile; mistral_common 1.9.0 or later comes with vLLM
2. Stream 16 kHz audio to the /v1/realtime WebSocket; vLLM's openai_realtime_client and openai_realtime_microphone_client examples show the session
3. Set temperature 0.0. Change the delay with transcription_delay_ms in tekken.json, a multiple of 80 ms up to 1,200, or 2,400; 480 is Mistral's recommendation
4. One text token is 80 ms, so set --max-model-len to at least seconds / 0.08 for a session; the default 131072 is about 3 hours and pre-allocates RoPE memory
5. In Transformers (5.2.0 or later), resample with processor.feature_extractor.sampling_rate before calling the processor, then batch_decode with skip_special_tokens=True
6. For a small machine, ggml-org's Q4_K_M GGUF with its mmproj runs in llama.cpp, and Mistral Experimental's Q4_K runs in audio.cpp
7. Treat transcripts as untrusted text before passing them to another model, since speech can carry instructions