Tiny Vision Models Compared II: WebBrain, Gemma 4, and Larger Qwen 3.5 Models Join the Race

A 450M model trained for one narrow job can compete with much larger general-purpose vision models on that job. WebBrain VL 2 450M passed 44 of our 100 browser-vision cases in its deployed ONNX/WebGPU configuration. Qwen 3.5 2B passed 39 and Gemma 4 E4B passed 41. Their mean rubric scores remain higher, so this is not a claim that 450M broadly beats either model. It is evidence that task-specific fine-tuning can move a genuinely tiny model into the same useful browser-observation band.

The expanded table

Our first tiny-vision comparison established the baseline: the off-the-shelf LFM2.5-VL models were compact but weak under WebBrain's exact screenshot contract, while Qwen 3.5 0.8B was unexpectedly capable for its size.

This follow-up adds four rows: WebBrain VL 2 450M, Qwen 3.5 2B, Qwen 3.5 4B, and Gemma 4 E4B IT. Every model saw the same 100 screenshots, the same production six-section prompt, and the same deterministic rubric.

One benchmark, one production prompt, and 100 held-out browser screenshots. A strict pass requires the complete six-section observation contract.

The complete benchmark source, screenshots, expected facts, scoring code, and committed result files are on GitHub.

We intentionally leave LFM2.5-VL-3B out of this comparison. Its run did not follow the numbered six-section output contract reliably, so including it in a compact-model leaderboard would mostly measure formatting failure rather than useful visual understanding. We discuss that result separately in the fine-tuning story.

Strict pass and mean rubric measure different things

A strict pass is a complete answer under WebBrain's production contract. The model must return six numbered sections covering page purpose, exact visible text, inputs, state signals, blockers, and unknowns. It must recover the weighted facts for that case without a critical contradiction or confident guess.

The mean rubric score gives partial credit. A model can correctly read a page title, two controls, and an error message yet fail the case because it missed the modal blocking the entire viewport. That answer contributes to the mean score but not to the strict-pass count.

This is why the middle three rows need careful language:

• WebBrain V2 has more complete passes than Gemma E4B and Qwen 2B.
• Gemma and Qwen have higher average rubric scores, meaning they recover more of the expected evidence across all 100 cases.
• The fair conclusion is that all three occupy a similar task-performance band—not that the 450M model is universally better.

What fine-tuning bought

WebBrain V2 uses the same 452,847,616-parameter LFM2.5-VL-450M foundation as the one-pass base row. Only 4,128,768 LoRA parameters—about 0.91% of the model—were trainable. The merged checkpoint does not become a multi-billion-parameter model after training; it remains a 453M model whose behavior has been redirected toward browser screenshots and WebBrain's response contract.

The deployed package uses FP16 embeddings and vision encoder weights with a symmetric-Q4 merged decoder. Its complete browser package is about 0.81 GB. For scale, the local artifacts used in the new comparison were approximately 1.75 GB for Qwen 2B MLX 4-bit, 3.06 GB for Qwen 4B MLX 4-bit, and 6.86 GB for Gemma E4B MLX 4-bit. Those byte counts are not perfectly interchangeable—WebBrain uses ONNX while the other rows use MLX—but they describe the practical download and storage difference.

By nominal parameter count, WebBrain V2 has roughly one-quarter as many parameters as Qwen 2B and one-ninth as many as Gemma E4B. That is the efficiency result worth celebrating.

With parameters on X and strict passes on Y, WebBrain V2 is the clear upper-left efficiency outlier. Gemma's E4B label denotes its effective 4B class.

Qwen 4B still leads this table

Qwen 3.5 4B is the strongest row among the models collected here: 55 strict passes and an 83.4% mean rubric score. Its advantage persists into harder cases, and it is more reliable on multilingual OCR than the smaller rows.

If your deployment can comfortably host it, Qwen 4B is the higher-quality general-purpose choice in this group. The WebBrain result addresses a different constraint: how much browser-specific vision can fit into an extension-friendly local sidecar that runs through WebGPU.

Why WebBrain V2 becomes the default

WebBrain normally reads the browser through structured sources first: page text, DOM state, accessibility information, URLs, and tool results. Vision is called when pixels contain information those sources missed—charts, canvas content, selected states, overlays, low-contrast labels, or a screenshot after a meaningful state change.

That architecture rewards a compact specialist. The model does not need to replace the planner or solve every multimodal problem. It needs to turn a browser viewport into reliable evidence often enough to unblock the main agent, while keeping screenshots local and the download reasonable.

For that role, V2 is a material upgrade over the raw 450M fallback:

• 44 strict passes instead of one in the comparable deployed/off-the-shelf table;
• no API or per-image token cost after download;
• a real Transformers.js/WebGPU smoke pass, not only server-side ONNX inference;
• an open-weight checkpoint, public dataset, evaluation artifacts, and release manifests.

We are therefore making WebBrain VL 2 450M the default local vision model in WebBrain, replacing the off-the-shelf LFM2.5-VL-450M behavior for this fallback slot. As always, the agent should corroborate high-impact visual claims before clicking, submitting, paying, or changing account state.

The power of fine-tuning

WebBrain VL 2 450M is an open-weight tiny vision model focused on browser tasks and runnable on WebGPU. In this test it performed similarly to Qwen 3.5 2B and Gemma 4 E4B while using far fewer parameters—roughly one-quarter to one-ninth as many.

That does not shrink every vision problem to 450M parameters. It shows that a narrow production contract, a held-out benchmark, targeted data, and disciplined fine-tuning can move a small model much further than its raw checkpoint suggests.

The power of fine-tuning, actually.

Tags: #WebBrain #WebBrainVL #Qwen35 #Gemma4 #LFM25VL #WebGPU #FineTuning #BrowserAgent #LocalAI