SignalPilot vs Claude Code
39 real dbt tasks Claude Code got wrong and the same agent on SignalPilot got right, with every side-by-side transcript public.
Failure mechanisms
01 · fan-before-aggregate
shape: wrong shape
trigger
a join on a non-unique lookup key, then a SUM downstream.
claude code
joins first, aggregates second. Every row is multiplied by its match count on the other side.
signalpilot rule
grain drift on any monetary or count aggregate must produce a red diff, not a green check.
02 · green on empty
shape: confidently wrong
trigger
the model built and downstream tests passed; the target table is empty.
claude code
reads pass/pass/pass, ships the run, blames the test fixture in the write-up.
signalpilot rule
a passing suite over a zero-row target is a failure, not a pass.
03 · silent filter drop
shape: silent row loss
trigger
the raw stream logs the same exposure under three kind labels; staging keeps only one.
claude code
returns a clean number with no warning. 60% of the population has quietly left the mart.
signalpilot rule
row-count drift between landing and mart above a threshold must be surfaced, not swallowed.
Public case archive
Filter the full archive, open any case, and inspect the source project and result.
39 of 39 cases
Handed the CFO $267 billion in total billed charges for the board deck. The real number is $71 billion. It verified the wrong figure twice.
truth $71.08B · claude $267.00B (3.76x)
Shipped an empty table (0 of 4,686 rows), then called the model correct and blamed the test data.
truth 4,686 rows · claude 0
Reported $2,090,694 in total revenue on the canonical dbt demo project. The real number is $763,290, nearly 3x lower.
truth $763,290 · claude $2,090,694 (2.74x)
Built one of the two required models, saw "34 models passed," and quit. The second table never existed.
truth 2 models · claude 1 (0 correct)
Read the buggy starter model, declared it already correct, and shipped a 21x under-count in 83 seconds.
truth 354,321 rows · claude 16,646 rows
Reported $32.27 billion in billed charges for Bayview Medical. The real number is $7.18 billion, 4.5x lower.
truth $7.18B · claude $32.27B (4.5x)
The payer-mix review said $20.14 billion billed to commercial payers. The real figure is $4.48 billion.
truth $4.48B · claude $20.14B (4.5x)
Plains Regional's QBR number came back as $22.69 billion. The real number is $5.04 billion.
truth $5.04B · claude $22.69B (4.5x)
Metro Children's board-deck total: $29.54 billion reported, $6.57 billion real.
truth $6.57B · claude $29.54B (4.5x)
The denials write-up came back with $3.57 billion in denied charges. The real number is $795 million.
truth $795.2M · claude $3.57B (4.49x)
Philadelphia store revenue reported as $1,387,892. The real number is $505,644.
truth $505,644 · claude $1,387,892 (2.74x)
2019 revenue for the annual review: $1,797,108 reported, $656,056 real.
truth $656,056 · claude $1,797,108 (2.74x)
Brooklyn store revenue reported as $702,802. The real number is $257,646.
truth $257,646 · claude $702,802 (2.73x)
Average customer lifetime value reported as $2,250.48. The real number is $821.63.
truth $821.63 · claude $2,250.48 (2.74x)
Average order value reported as $28.93. The real number is $10.83, nearly 3x lower.
truth $10.83 · claude $28.93 (2.67x)
May 2019 revenue reported as $276,011. The real number is $99,025.
truth $99,025 · claude $276,011 (2.79x)
The Philadelphia vs Brooklyn revenue gap came back as $685,090. The real gap is $247,998.
truth $247,998 · claude $685,090 (2.76x)
2018 revenue reported as $293,586. The real number is $107,234.
truth $107,234 · claude $293,586 (2.74x)
One customer's lifetime spend reported as $2,897. Steven Williams actually spent $1,475.
truth $1,475 · claude $2,897 (1.96x)
Audited a client's SDK volume at 230,760 logged impressions. The raw landing table holds 573,648. Sixty percent of the data had silently vanished below the mart.
truth 573,648 rows · claude 230,760 rows (60% missing)
Counted 89,944 experiment enrollments for drift_labs. The real number is 185,216: a filter on a missing variant label had silently discarded over half of them.
truth 185,216 rows · claude 89,944 rows (51% missing)
Reported 186,432 enrollments for halcyon_bank. The real count is 232,980: every enrollment missing an SDK version label had been silently dropped.
truth 232,980 rows · claude 186,432 rows (20% missing)
Reported 101,411 enrollments for northwind_marketplace, all time. The real number is 179,496: a date-window filter had quietly truncated history.
truth 179,496 rows · claude 101,411 rows (44% missing)
granite_gaming's capacity check came back at 103,670 enrollments. The real number is 183,216, cut nearly in half by the same date-window truncation.
truth 183,216 rows · claude 103,670 rows (43% missing)
Reported 167,779 enrollments for ember_commerce. The real number is 217,101: an over-aggressive deduplication step had collapsed distinct enrollments together.
truth 217,101 rows · claude 167,779 rows (23% missing)
A 3-row month-over-month summary came back as 11,135 rows: a 3,700x duplication across five years of history.
truth 3 rows · claude 11,135 rows
Collapsed 3,430 intraday price bars into 15 rows, then cascaded into a book-value table with exactly 1 row instead of 3,185.
truth 3,430 & 3,185 rows · claude 15 & 1
A single-column join on a non-unique lookup table silently multiplied the sales table 2.5x: 25,667 rows where the answer was 9,994.
truth 9,994 rows · claude 25,667 rows
Merged rows from three source tables into the report and more than doubled the output: 37 rows where the answer is 17.
truth 17 rows · claude 37 rows
One row per champion-season instead of one per champion: 75 rows where the truth is 34.
truth 34 rows · claude 75 rows
1,770 phantom customers whose purchase totals were fabricated entirely from returned goods.
truth 75,007 rows · claude 76,777 rows
Well-intentioned cleaning destroyed 2,060 genuinely empty post bodies and silently deleted 475 real comments.
truth 0 rows dropped · claude 475 rows dropped (2,060 empties overwritten)
Right shape, right columns, 40 rows, and every single date wrong: it aggregated the wrong upstream table.
truth 40 rows · claude 40 rows (every date wrong)
One incomplete labeling rule tagged 7 dormant customer-months as churn: a 13% churn over-count.
truth 55 rows · claude 62 rows (13% overstated)
The entire failure was a sign: reported account spend as -0.158 where the ledger wants +0.16.
truth +0.16 · claude -0.158
The project's own specification listed 17 required columns including account_ordinal. Claude code built 21 columns from the wrong upstream and never noticed the missing one.
truth 17 columns · claude 21 (1 required missing)
The date range reached back before each customer existed: 30 phantom days of history.
truth 122 rows · claude 152 rows
An invented filter dropped exactly one real row; a loose join admitted 5,237 unmatched trips. Both silent.
truth 4,436 rows · claude 9,673 rows (off by −1 and +5,237)
Methodology
same model
claude-opus-4 on both sides. Identical capability, no handicap.
independently graded
Truth answers derived from source data, not from either agent's output.
view truth · northwindtranscripts public
Every tool call, every file written, unedited. Nothing cherry-picked.
view the case archiveraw log view northwind_org_charges/session.jsonl (unedited)
39 of 39 · same model
Governance catches what capability misses.
