HLE

HLE sits at the opposite end of the search spectrum from BrowseComp. Its questions were written by subject-matter experts to be unambiguous but extremely hard, so a model's baseline score is mostly a function of what it already knows. Adding web search turns that into a different question: how much expert-level knowledge can a search configuration retrieve on demand?

We run it with the model held fixed because that isolates the variables OpenRouter users actually control. Those are which engine handles the searches, whether search runs as a server tool or a plugin, and how many agent turns the loop is allowed. Those knobs are exactly what you can set on a request today.

These scores compare search configurations, not agent products. The model reads search result excerpts only, with no full-page fetching and no code tools, so absolute numbers sit below published agent leaderboards, which allow both. Compare configurations rather than raw levels.

Because HLE leans on expertise rather than browsing depth, differences between search configurations are smaller here than on BrowseComp: many questions are answered (or missed) the same way at every budget. Engine gaps with overlapping confidence ranges are treated as unresolved here, not as proof of equality. A configuration's cost is as real as its score, so read quality and efficiency together.

We use the text-only subset of the public dataset (multi-modal questions are excluded), so scores are not directly comparable to full-HLE leaderboards. The questions are public, which means memorization can inflate absolute scores over time; differences between search configurations remain meaningful because every configuration shares the same model.

Each task is one question with a short reference answer. The model answers in a fixed format (explanation, exact answer, stated confidence), and a judge model grades whether the extracted answer is semantically equivalent to the reference — the same answer-equivalence grading BrowseComp uses. The grade is binary with no partial credit, and failed or refused tasks score zero.

reward = judge(extracted_answer ≡ reference_answer)   // ∈ {0, 1}
judge  = gpt-4.1 at temperature 0, strict json_schema verdict
empty or refused answers skip the judge and score 0

A search configuration is one model, search engine, request format, and maximum search budget. Scores are weighted by the number of questions in each verified run; small smoke tests are excluded.

Runs execute on OpenRouter's native benchmark harness against production endpoints. Server-tool configurations set a request-level tool-call budget on the web search server tool. Plugin configurations make one pre-inference search of the question with the web search plugin. Engines use the same default configurations that serve production traffic.

Every run persists its exact model, engine, request format, search budget, cost, and available timing telemetry. Missing configurations stay missing in the comparison table, and absent or zero telemetry is not treated as free or instantaneous performance.