DevDay grading: Sol wins on cost, not benchmarks

Independent grading of OpenAI's DevDay releases is converging on a practitioner story rather than a benchmark one. Artificial Analysis, at max effort, scores Opus 5.5 at 57.6, Astra at 52.7, and Sol at 51.8 — but Sol wins at medium effort and costs $0.72 per task versus $5.98 for Opus. For teams actually running agents on production codebases, the cost gap is the headline: roughly eight times the work per dollar.

The numbers cut against the launch-week hype cycle — and against the leaderboard framing in general. What matters to working engineers is hidden-test pass rates, regressions, token burn, and dollars per completed task, not who tops a chart at maximum effort.

Sources: Monolith