36 AI models against 20 real small-business tasks

Success rate is the share of runs that passed, with its 95% interval drawn under it: a model’s true rate could sit anywhere on the bar. Cost per pass divides the cost of every run, failures included, by the runs that passed. An asterisk marks a cost worked from the maker’s list price, where the provider reported none.

Pass or fail

An answer passes only if every fixed check passes. On the tasks that also carry a rubric, it has to meet most of its criteria too, and the criteria that matter most are mandatory. There is no partial credit.

Fixed checks

Structured answers are read field by field against the right values, with a tolerance on money and hours. Written answers are checked for what they must say, what they must not say and how long they may be. The same code marks every model.

The rubric grader

Rubric criteria are marked by gpt-5-6-sol at temperature zero, each with a reason. Every answer is marked twice, and a third mark settles any criterion the two disagree on. Its own verdict is ignored: code applies the pass rule to the criteria it marked. Before the run it was tested twice on answers whose verdict is known.

One protocol

Each run is one instruction message and one message with the material: no tools, no web search, no JSON mode, each model’s default sampling and up to 4,000 tokens of output, reasoning included. An answer cut off at that limit is marked as it stands. 1 run a model on each task.

Audited rules

Before an edition is published, every failure on a deciding rule is read. A rule that fails a correct answer for leaving out a detail the instructions never asked for is corrected, and those answers are marked again. A rule that catches a real mistake stays, however many models it fails.

Cost and time

A run costs what the provider billed for it, or, where the provider reports no cost, its tokens at the maker’s list price. Time runs from the request to the last word of the answer; calls that failed are left out of it.

Intervals

Every success rate carries a Wilson 95% interval. With a few dozen runs a model, a gap of several points between two models can be chance, and the bars say so.

The data

Tasks, checks and results are published under CC BY 4.0: cite AIGROW and link this page. Every task page carries the edition’s canary string, so the tasks can be kept out of training sets.