Every model comparison ends with the same advice: test it on your own data. Almost none explain how. This guide is the how.
The goal is not to find the “best” model. It is to find the cheapest model that clears your quality bar on the work you actually do.
For the current line-up, see latest AI models. For the benchmark caveats, see AI model benchmarks explained.
Step 1: Build the task set
Use 40 tasks, not four. A small set overfits to whatever you happened to test first.
| Type | Count | Purpose |
|---|---|---|
| Easy, common | 10 | Confirms the model is not broken |
| Normal production | 15 | The real workload |
| Hard or ambiguous | 10 | Separates good from great |
| Adversarial or edge | 5 | Reveals failure modes |
Pull them from real tickets, documents or requests. Remove customer names and anything sensitive. Keep the expected answer next to each one.
Step 2: Define what “correct” means
A five-point scale is enough:
| Score | Meaning |
|---|---|
| 4 | Correct, usable as-is |
| 3 | Correct with small edits |
| 2 | Partly correct, needs rewriting |
| 1 | Wrong but harmless |
| 0 | Wrong and dangerous |
The 0 matters. A model that is slightly worse on average but never produces a harmful output can be the better business choice.
Step 3: Score blind
Remove model names from the outputs before anyone reviews them. Reviewers should not know which model produced which answer, and ideally two people should score the same 10 tasks to check consistency.
Vendor benchmark charts are useful context. Your blind scores are the decision.
Step 4: Measure the right cost
Cost per token is not the number to compare. Cost per successful task is.
cost per success = total cost ÷ number of tasks scored 3 or 4
Include retries, failed calls and the human time to fix a score-2 output. A cheap model that needs more editing can lose on this metric.
| Model | Cost per task | Success rate | Cost per success |
|---|---|---|---|
| A | $0.004 | 92% | $0.0043 |
| B | $0.008 | 96% | $0.0083 |
| C | $0.012 | 98% | $0.0122 |
The cheapest model wins here only if the editing cost is close to zero. Add human minutes and the ranking often changes.
Step 5: Test latency and reliability
Quality alone is not enough. Record:
- p50 and p95 response time.
- Error and timeout rate.
- Behaviour under rate limits.
- Output stability across repeated runs.
A model that is 2% better but three times slower can be the wrong choice for a customer-facing feature.
Step 6: Test the failure behaviour
Run the adversarial five. Check whether the model:
- Refuses when it should.
- Refuses when it should not.
- Invents a field instead of saying “not found”.
- Follows an instruction hidden in the input.
- Produces a confident wrong answer.
This is where cheap models separate fastest.
Step 7: Decide, then re-test
Pick the cheapest model that clears the bar, and route the hard cases to a stronger one. Re-run the same task set after every model release. The ranking moves more often than most teams expect.
Keep the task set in version control. It becomes your regression suite for AI.
Common mistakes
- Testing only easy tasks. Every model passes those and tells you nothing.
- Scoring with the model name visible. Bias is immediate.
- Comparing token prices. Cost per success is the only fair number.
- Ignoring p95 latency. Averages hide the bad requests.
- Never re-testing. A model change can silently break a workflow.
- Using one reviewer. Single-rater scores are noisy.
A minimal template
Task ID | Input | Expected | Model | Output | Score (0-4) | Cost | LatencyForty rows, five models, one afternoon. That is enough to make a defensible decision and to justify it later.
FAQ
How many test cases do I need?
Forty is a good starting point: enough variety to separate models, small enough to score in a day.
Can I just use public benchmarks?
They are useful for shortlisting, not for deciding. Your data, tools and review standards change the ranking.
Should I test the most expensive model first?
Yes. It sets the quality ceiling. Then test cheaper models and see how close they get.
How often should I re-run the evaluation?
After every model release you adopt, and at least once a quarter.
Next step
If you need help building an evaluation for a product decision, contact 24Bit System. Our AI app development cost guide covers the build side, and the model comparison covers the current picks.
