Gemma 4 31B IT vs Qwen3-VL-32B-Instruct

Gemma 4 31B IT

A dense multimodal cousin with documented tool calling, optional thinking and a 256K context, for general agent work and image reading on one 24 GB GPU through Google's 4-bit build. It runs in every major runtime under plain Apache 2.0. The main risk is that Google publishes no safety results and has gone quiet on the repository since July.

Qwen3-VL-32B-Instruct

The largest dense Qwen3-VL cousin, for harder work on screenshots, documents, charts and video, needing about 25 GB from Alibaba's Q4_K_M GGUF with its vision file, in every major runtime under plain Apache 2.0. It has no thinking mode or published safety work, and only one provider was found serving it, so plan to run it yourself.

Gemma 4 31B IT

1. On the reference machine run llama-server -hf google/gemma-4-31B-it-qat-q4_0-gguf --hf-file gemma-4-31B_q4_0-it.gguf --jinja -c 32768, adding gemma-4-31B-it-mmproj.gguf for images
2. Or ollama run gemma4:31b for the 20 GB Q4_K_M build
3. In vLLM serve with --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4, as vLLM's recipe does
4. Thinking is off by default, so pass enable_thinking: true in chat_template_kwargs to turn it on, and expect the reasoning in the thought channel block
5. Drop earlier thoughts from the history except on tool-call turns, as the card asks
6. Use the card's sampling, temperature 1.0, top_p 0.95, top_k 64
7. Validate function names and arguments before running a call, and check JSON output yourself or use the runtime's guided decoding
8. The w4a16 build relies on Marlin INT4 kernels that vLLM marks unsupported on AMD, so use the GGUF on the reference machine

Qwen3-VL-32B-Instruct

1. Start it with llama-server -hf Qwen/Qwen3-VL-32B-Instruct-GGUF:Q4_K_M --jinja -c 32768, or ollama run qwen3-vl:32b-instruct
2. In Ollama the plain qwen3-vl:32b tag is the Thinking edition, so name 32b-instruct
3. With the GGUF, load the mmproj vision file too, or images won't be read
4. Parse tool calls as JSON inside <tool_call> tags, or use SGLang's --tool-call-parser qwen
5. Use the card's sampling, temperature=0.7, top_p=0.8, top_k=20, presence_penalty=1.5 for vision and temperature=1.0, top_p=1.0, top_k=40, presence_penalty=2.0 for text
6. Validate JSON output yourself, since structured output isn't documented
7. vLLM needs 0.11.0 or later and Transformers 4.57.0 or later
8. If you pulled the FP8 build before 22 October 2025, pull it again, since the earlier files were the Thinking edition's