Evaluation

Open benchmarks

Benchmarks built from WebBrain’s real work: the same 100 browser-agent prompts, frozen to one system prompt and tool schema, run against every new model, plus a 100-screenshot vision suite and a compact routing test.

All benchmark posts

100frozen planner prompts

100held-out browser screenshots

77.0%best Sonnet match so far

$0.00977per 100 calls, cheapest strong row (Ling 3.0 Flash VL)

Why it matters

Model launches come with general benchmarks, but a browser agent needs to know one thing: will this model pick the right action on a real page? Freezing the test makes every new model comparable with every old one.

The frozen planner test

100 single-turn Chrome browser-agent prompts, frozen to the May 23, 2026 Claude Sonnet 4.6 system prompt and its 41-tool schema. The runner captures the first response only and does not execute it. Scores: parsed tool calls, strict exact first call, ideal tool name, and agreement with Sonnet’s choice.

A peer-consensus variant scores each model against all the others, with no single judge: DeepSeek V4 Flash 0731 leads at 51.8% for $0.050 a replay.

Vision

Each model gets 100 browser screenshots and WebBrain’s six-section vision prompt. MiMo-V2.6-Flash leads the hosted models at 81/100; among tiny models Qwen 3.5 4B scores 55 and WebBrain VL 2 450M scores 44.

Frozen planner test: current leaders

Model	Sonnet match	Notes
Gemma 4 31B QAT	77.0%	0.55 s median
Qwen 3.6 27B	77.0%
MiniMax M2.7	77.0%	23% exact
MiniMax M3	75.0%	$1.06 a run
WebBrain Compass 1.0	73.0%	$0.12 a run
Ling 3.0 Flash VL	72.0%	$0.00977 per 100 calls

Reference-based and consensus scores are not ground truth. First action only: no multi-turn recovery and no executed actions. Results are dated snapshots.

Caveats

Frozen-table and consensus percentages should not be mixed.
Latencies from different hosts are not controlled.