Cosmos-Reason2-2B vs Qwen3-Omni-30B-A3B-Instruct

Cosmos-Reason2-2B

A small local cousin for reasoning over video and images in robotics, driving and warehouse scenes, returning points, boxes and timed events as JSON. The 4.9 GB file runs in Transformers or vLLM, though NVIDIA asks for 24 GB of GPU memory. Development has moved to Cosmos 3, and the gated licence is revocable.

Qwen3-Omni-30B-A3B-Instruct

An omni-modal cousin that listens, watches and reads, then answers in text or one of three voices, with tool calls from spoken requests, under plain Apache 2.0. Its only build, BF16, needs about 85 GB, and speech out needs Transformers or vLLM-Omni. The main risk is no answers from Qwen and no safety work for speech-driven actions.

Cosmos-Reason2-2B

1. Accept the licence on the Hugging Face card with a signed-in account and set HF_TOKEN before downloading
2. Serve it with vllm serve nvidia/Cosmos-Reason2-2B --max-model-len 16384 --media-io-kwargs '{"video": {"num_frames": -1}}' --reasoning-parser qwen3, keeping max-model-len between 8,192 and 16,384 as NVIDIA advises
3. For local video or images, make a folder for them (mkdir -p media) and pass --allowed-local-media-path "$(pwd)/media", so the server can read only that folder, not everything under the directory you start it from
4. Sample video at 4 fps and allow at least 4,096 output tokens so the reasoning isn't cut off
5. For reasoning, add the card's instruction to answer with a think block and then the answer; leave it out for a direct reply
6. Use NVIDIA's prompt files (prompts/*.yaml in nvidia-cosmos/cosmos-reason2) when you want JSON boxes, events or trajectories, and validate the JSON yourself
7. Check its plans before a robot acts on them, since NVIDIA's own safety card warns of collision and manipulation risks

Qwen3-Omni-30B-A3B-Instruct

1. For text answers, mkdir -p media && vllm serve Qwen/Qwen3-Omni-30B-A3B-Instruct --dtype bfloat16 --max-model-len 32768 --allowed-local-media-path "$(pwd)/media" with vLLM 0.11.1 or later serves the Thinker on an OpenAI-compatible API
2. The card's command passes --allowed-local-media-path /, which lets anyone who can reach the server have it read any file on the machine through a file:// URL. Point it at a dedicated folder that holds only the media you mean to send, as above, and give vLLM an absolute path
3. Send audio as audio_url and images as image_url content parts; vLLM serve doesn't take use_audio_in_video, so pass a video's audio separately
4. For speech out, load Qwen3OmniMoeForConditionalGeneration in Transformers 4.57.0 or later with qwen-omni-utils, pick speaker Ethan, Chelsie or Aiden, and write the returned waveform at 24 kHz
5. Call model.disable_talker() or pass return_audio=False when you only need text, which saves about 10 GB and time
6. Parse tool calls as JSON inside <tool_call> tags yourself, since no parser is named, and confirm actions triggered by speech before running them
7. Keep use_audio_in_video the same across every step of a conversation, the card warns results go wrong otherwise
8. Budget memory for video, the card's table rises from 78.85 GB for 15 seconds to 144.81 GB for 120 seconds in BF16