Same model, with and without SignalPilot.
SignalPilot96.9%
Snowflake CoCo
65.1%
dbt Wizard
58.1%
Claude Code alone
39.5%
Both runs: Claude Code with Sonnet 4.6 · every transcript public →
Solutions · Model & vendor bakeoffJudge every model and vendor on your data.
We build the evals once from your stack, then run every candidate through them: Claude, GPT, Gemini, warehouse agents, BI agents. You get pass rates per question, evidence for every answer, and a decision you can defend.
Bakeoff · Brightwater Goods
grading24 graded questions · 4 candidates · same context, same sandbox
QuestionClaude + SignalPilotClaude Code aloneCodex + SignalPilotWarehouse agent
- net_revenue_q2_excl_refundsMonth-end✓×✓✓
- refund_rate_by_segmentMonth-end✓×✓×
- gross_margin_by_channelMonth-end✓✓✓✓
- fan_out_join_on_order_idFailure case✓×✓×
- pipeline_slipped_q3Pipeline✓✓✓×
- wrong_time_window_fiscalFailure case✓×××
- active_customers_trailing_90dRetention✓✓✓✓
- refunds_left_in_revenueFailure case✓×✓×
Passed · of 240/240/240/240/24
Every cell links to the transcript and the evidence graph. The context and the evals stay in your repo, whichever candidate you pick.
Why a bakeoff needs evals
Public benchmarks do not run on your data.
Our public results, for reference. Every team we talk to asks the second question, and only your own graded questions answer it.
#1
on both public benchmarks
Spider 2.0-DBT and ADE-Bench
96.9%
ADE-Bench, tasks passed
Claude Code alone: 39.5%, same model
75%
Spider 2.0-DBT, tasks passed
next best 60.29%
SignalPilotThe hardest public data-engineering benchmark.
SignalPilot*75%
Databao · JetBrains
60.29%
Shadowfax
41.18%
Spider Ext · GPT-5
39.71%
*75% submitted in August; official leaderboard shows 65.6% (May).
What you get
The same evals, every candidate.
- Evals from your own dbt models and dashboards, including the failure cases.
- Each candidate run in the same governed sandbox with the same context.
- A report: pass and fail per question per candidate, with the evidence graph for every answer.
- The context layer and evals stay with you afterward, whichever vendor you pick.
Model routing
The best model per task, not one model for everything.
The report gives you pass rates per task group and cost per task for every candidate. Route each group to the cheapest model that clears the bar and the expensive model only runs where it earns its keep.
Bakeoff · pass rate by task group · cost per taskbar: 90% · 72 tasks a month
| Model | Month-end close24 tasks | Pipeline review18 tasks | Refunds12 tasks | Retention10 tasks | Failure cases8 tasks | Cost / task |
|---|---|---|---|---|---|---|
| Claude Opus | 100% | 96% | 100% | 100% | 92% | $1.20 |
| GPT-5.6 | 92% | 92% | 96% | 96% | 79% | $0.60 |
| Claude Sonnet | 96% | 92% | 100% | 100% | 83% | $0.40 |
| GLM 5.3 | 83% | 88% | 96% | 92% | 58% | $0.12 |
| Gemini Flash | 75% | 79% | 96% | 92% | 42% | $0.08 |
| DeepSeek | 79% | 83% | 92% | 90% | 50% | $0.06 |
✓ marks the cheapest model that clears the bar for that group. Where no cheaper model clears it, the top model keeps the slot.
Top model everywhere
$1.20 / task
98.1% of tasks pass · $86.40 a month
Best model per task
$0.39 / task
93.1% of tasks pass · $27.72 a month
Saved
68%
of model spend, with every task group above the bar. Rerun when the next model ships.
Illustrative figures for a fictional project. Your bakeoff produces your own table, on your questions, with your token prices.
SAME QUESTIONS, EVERY MODEL
Every candidate answers the same graded questions in the same sandbox with the same context, so the pass rates are comparable and the cost per task is real.
A BAR, NOT A WINNER
You set the pass rate a task group must clear. The cheapest model above it takes the slot. Failure cases usually stay with the strongest model; routine groups rarely need it.
RERUN WHEN MODELS SHIP
The evals and the context stay in your repo. When a new model lands, rerun the grid and move the slots that changed.
TIMELINE
About two weeks from access to report for a scoped set of questions.
WHAT STAYS
The evals and the context layer are yours. Rerun the bakeoff when the next model ships.