Ming-3.0-audio-tiny vs Voxtral Mini 4B Realtime 2602

Ming-3.0-audio-tiny

A small MIT-licensed audio-language model that transcribes Chinese and English, answers questions about audio and replies in a built-in or cloned voice. It needs about 25 GB and SGLang-Omni from main or inclusionAI's own vLLM-Omni and SGLang branches. The main risk is that every loading path runs repository code, with nothing published on safety.

Voxtral Mini 4B Realtime 2602

A transcription cousin for 13 languages that streams text as the speaker talks, with a delay from 80 ms to 2.4 s, under Apache 2.0 and on Mistral's API. It needs a 16 GB GPU at BF16 through vLLM, Transformers or llama.cpp, or under 4 GB at Q4_K. The main gaps are no timestamps, safety results or answers from Mistral.

Ming-3.0-audio-tiny

1. Fastest start is SGLang-Omni from main, sgl-omni serve --model-path inclusionAI/Ming-3.0-audio-tiny --talker.gpu 0, which exposes OpenAI-style /v1/audio/transcriptions, /v1/audio/speech and /v1/chat/completions
2. For transcripts use greedy decoding with 10-gram blocking and the validated ASR prompt, and expect a language label, a tab, then the text
3. Parse tool calls yourself from <tool_call>name with <arg_key> and <arg_value> pairs, since no runtime parser is named
4. Thinking is off by default, turn it on with chat_template_kwargs enable_thinking true and strip the <think> block from the reply yourself
5. In vLLM-Omni pick a voice with audio.voice id from docs/voices.md, and don't combine it with a reference recording
6. Review the repository's Python before passing trust_remote_code, and consider deleting the __pycache__ folders under talker/ so only readable source runs

Voxtral Mini 4B Realtime 2602

1. Serve it with VLLM_DISABLE_COMPILE_CACHE=1 vllm serve mistralai/Voxtral-Mini-4B-Realtime-2602 --compilation_config '{"cudagraph_mode": "PIECEWISE"}' and install soxr, librosa and soundfile; mistral_common 1.9.0 or later comes with vLLM
2. Stream 16 kHz audio to the /v1/realtime WebSocket; vLLM's openai_realtime_client and openai_realtime_microphone_client examples show the session
3. Set temperature 0.0. Change the delay with transcription_delay_ms in tekken.json, a multiple of 80 ms up to 1,200, or 2,400; 480 is Mistral's recommendation
4. One text token is 80 ms, so set --max-model-len to at least seconds / 0.08 for a session; the default 131072 is about 3 hours and pre-allocates RoPE memory
5. In Transformers (5.2.0 or later), resample with processor.feature_extractor.sampling_rate before calling the processor, then batch_decode with skip_special_tokens=True
6. For a small machine, ggml-org's Q4_K_M GGUF with its mmproj runs in llama.cpp, and Mistral Experimental's Q4_K runs in audio.cpp
7. Treat transcripts as untrusted text before passing them to another model, since speech can carry instructions