Same model, with and without SignalPilot.
SignalPilot96.9%
Snowflake CoCo
65.1%
dbt Wizard
58.1%
Claude Code alone
39.5%
Both runs: Claude Code with Sonnet 4.6 · every transcript public →
Solutions · Model & Vendor BakeoffJudge every model and vendor on your data
Your Context Factory already has the evals, built from your metrics. Run every model and vendor against them: Claude, GPT, Gemini, warehouse agents, BI agents. You get pass rates per question, evidence for every answer, and a decision you can defend.
Bakeoff · Brightwater Goods
grading24 graded questions · 4 candidates · same context, same sandbox
QuestionClaude + SignalPilotClaude Code aloneCodex + SignalPilotWarehouse agent
- net_revenue_q2_excl_refundsMonth-end✓×✓✓
- refund_rate_by_segmentMonth-end✓×✓×
- gross_margin_by_channelMonth-end✓✓✓✓
- fan_out_join_on_order_idFailure case✓×✓×
- pipeline_slipped_q3Pipeline✓✓✓×
- wrong_time_window_fiscalFailure case✓×××
- active_customers_trailing_90dRetention✓✓✓✓
- refunds_left_in_revenueFailure case✓×✓×
Passed · of 240/240/240/240/24
Every cell links to the transcript and the evidence graph. The context and the evals stay in your repo, whichever candidate you pick.
Why a bakeoff needs evals
Public benchmarks do not run on your data.
Our public results, for reference. Only your own graded questions tell you which model is right for your data.
#1
on both public benchmarks
Spider 2.0-DBT and ADE-Bench
96.9%
ADE-Bench, tasks passed
Claude Code alone: 39.5%, same model
75%
Spider 2.0-DBT, tasks passed
next best 60.29%
SignalPilotThe hardest public data-engineering benchmark.
SignalPilot*75%
Databao · JetBrains
60.29%
Shadowfax
41.18%
Spider Ext · GPT-5
39.71%
*75% submitted in August; official leaderboard shows 65.6% (May).
What you get
The same evals, every candidate.
- Evals from your own dbt models and dashboards, including the failure cases.
- Each candidate run in the same SignalPilot sandbox with the same context.
- A report: pass and fail per question per candidate, with the evidence graph for every answer.
- Your context and evals stay with you afterward, whichever vendor you pick.
Model routing
The best model per task, not one model for everything.
The report gives you pass rates per task group and cost per task for every candidate. Route each group to the cheapest model that clears the bar and the expensive model only runs where it earns its keep.
Claude Opus 5.5
$1.20 per run
$1,200
GLM 5.3 on SignalPilot
$0.12 per run
$120
Same thread, same evals, same SignalPilot sandbox.
Bakeoff · pass rate by task group · cost per taskbar: 90% · 72 tasks a month
| Model | CloseMonth-end close24 tasks | PipelinePipeline review18 tasks | RefundsRefunds12 tasks | RetentionRetention10 tasks | FailuresFailure cases8 tasks | CostCost / task |
|---|---|---|---|---|---|---|
| Opus 5.5 highClaude Opus 5.5 (high) | 100% | 96% | 100% | 100% | 92% | $1.20 |
| GPT-6.1 SolGPT-6.1 Sol | 98% | 96% | 100% | 100% | 90% | $0.60 |
| Opus 5.5 lowClaude Opus 5.5 (low) | 96% | 92% | 96% | 96% | 83% | $0.40 |
| GLM 5.3GLM 5.3Pareto pick | 98% | 94% | 100% | 100% | 90% | $0.12 |
| Gemini FlashGemini Flash | 75% | 79% | 96% | 92% | 42% | $0.08 |
| DeepSeekDeepSeek | 79% | 83% | 92% | 90% | 50% | $0.06 |
✓ marks the cheapest model that clears the bar for that group. Where no cheaper model clears it, the top model keeps the slot.
Top model everywhere
$1.20 / task
98.1% of tasks pass · $86.40 a month
Best model per task
$0.10 / task
94.0% of tasks pass · $7.32 a month
Saved
92%
of model spend, with every task group above the bar. Rerun when the next model ships.
Pareto pick: GLM 5.3 sits in GPT-6.1 Sol's class and above Claude Opus 5.5 at low effort in every task group, at $0.12 a task against $1.20. Pass rates are illustrative, for a fictional project; costs are list token prices. Your bakeoff produces your own table, on your questions, with your token prices.
WHAT STAYS
The evals and the context are yours. Rerun the bakeoff when the next model ships.