CentralGauge

Score

Average score per attempt row (0–1). Rewards partial credit.

Formula: Mean of all attempt scores across all results rows: SUM(score) / COUNT(*) over the results table.

Ranks models that make consistent partial progress on hard tasks. A model that scores 0.5 on every task beats one that passes half and fails the rest on this metric.

Pass Rate

Fraction of distinct tasks solved in any attempt across all runs.

Formula: (tasks_passed_attempt_1 + tasks_passed_attempt_2_only) / tasks_attempted_distinct

Primary ranking metric. Compare models here first: it directly measures how often the model delivers working code.

HumanEval paper (Chen et al., 2021) ↗

Cost / run

Average total LLM cost per benchmark run in USD.

Formula: SUM(cost_usd) / run_count across all runs for this model.

Use to compare operating cost across models with similar pass rates. Does not account for quality. Combine with $/Pass for a cost-efficiency view.

$/Pass

Total cost divided by number of distinct tasks passed. Lower is better.

Formula: SUM(cost_usd) / tasks_passed_distinct across all runs.

Best single cost-efficiency metric. Penalises expensive models that pass few tasks and rewards cheap models with high pass rates.

Latency p95

95th-percentile per-task wall time. Captures tail latency.

Formula: 95th percentile of per-task duration_ms across all tasks in all runs.

Use p95 to understand worst-case latency. A low p95 means the model rarely stalls, relevant for automated pipelines with timeouts.