Why SignalPilot passed where base failed
Evidence chips jump to the exact transcript momentClaude Code read a buggy starter file, declared it "already correct," and shipped it unchanged in 83 seconds. The result reported the United Kingdom's order volume as 16,646 when the real figure is 354,321, a 21x under-count.
Real-world impact
The country ranking report understated UK order volume 21x. Any decision comparing countries by activity, where to invest, where volume is growing, would be made on a number that is 5 percent of reality.
- Read the buggy starter file, declared it already correct, and shipped it unchanged in 83 seconds.
- The starter counts unique invoice documents; the sales table stores one row per line item, and for the UK that is 16,646 vs 354,321.
- Its financial rules flag exactly this trap: on a line-item table, count rows, not unique document ids.
- Computed both interpretations side by side (354,321 vs 16,646), confirmed the mismatch, and rewrote the metric.
Jump to the exact moment
The numbers
| Check | Correct answer | Base Claude Code | SignalPilot |
|---|---|---|---|
| UK order volume | 354,321 | 16,646 (21x under) | 354,321 |
| Revenue and top-10 ranking | · | correct | correct |
Verification workflow
Where the eight-step workflow caught it.
- 01Map the project
Scan every model: dependencies, required columns, macros, hazards.
The project scan routed all four pre-existing starter models into mandatory verification instead of trusting them; base Claude Code read the same starter and took it at face value.
- 02Load supporting skills
SQL, database, and domain rules loaded before any writing.
The financial domain skill states the exact rule the starter violates, count rows on a line-item table, not unique invoice ids; base Claude Code had no such rule.
- 03Validate the project
Parse check, plus rebuild of stale time-dependent upstreams.
- 04Discover project macros
Find macro-derived columns the YML contract does not list.
- 05Research the data
Driving table, cardinalities, contract, sibling patterns, categorical values.
- 06Write the technical spec
A written plan of sources, joins, filters, and grain per model.
- 07Write and build models
SQL from the spec, with column mapping checked before each model.
- 08Verify and fix
Two read-only verifiers cross-check structure and values until every check passes.
The value verifier computed both interpretations side by side, 354,321 rows vs 16,646 unique invoices, prescribed the one-line fix, and re-verified it; base Claude Code ran zero database queries before declaring the starter correct.
SignalPilot
claude code + signalpilot governed stack
All 8 steps complete. Here are the top 10 countries by total revenue:
| Rank | Country | Total Line Items (Invoices) | Total Revenue |
|---|---|---|---|
| 1 | United Kingdom | 354,321 | $7,308,391.55 |
| 2 | Netherlands | 2,359 | $285,446.34 |
| 3 | EIRE | 7,236 | $265,545.90 |
| 4 | Germany | 9,040 | $228,867.14 |
| 5 | France | 8,341 | $209,024.05 |
| 6 | Australia | 1,182 | $138,521.31 |
| 7 | Spain | 2,484 | $61,577.11 |
| 8 | Switzerland | 1,841 | $56,443.95 |
| 9 | Belgium | 2,031 | $41,196.34 |
| 10 | Sweden | 451 | $38,378.33 |
Key findings:
- United Kingdom dominates by a wide margin — $7.3M revenue, 25× more than the #2 country (Netherlands at $285K). This is expected for a UK-based retailer.
- The
report_customer_invoicesmodel required one fix:total_invoiceswas originallyCOUNT(DISTINCT invoice_id)(distinct invoice documents), which was inconsistent withSUM(total)computed at line-item grain. Fixed toCOUNT(*)so both metrics operate at the same grain — confirmed by both structure and value verifiers.
Now loading Step 2 supporting skills — the database is DuckDB and the domain is financial (invoices, revenue):
Skills loaded. Now proceeding to Step 3 — validate the project.
Project validates cleanly. No current_date hazards were flagged. Skipping Steps 4–7 (no stubs or missing models). Moving to Step 8.
Connection healthy. Now let me look at the existing models and YML files before dispatching the verifiers.
The report_customer_invoices model is exactly what the task asks for — top 10 countries by total revenue. Let me read its SQL and the other models to understand the implementations.
Good — dbt_project.yml sets +materialized: table globally. Now I'll dispatch both verifier subagents in parallel.
The value verifier found a CHECK FAIL: total_invoices uses COUNT(DISTINCT invoice_id) but the domain-financial rule mandates COUNT(*) on a line-item fact table for grain consistency with SUM(total). Applying the prescribed fix now.
Build succeeded. Re-dispatching both verifiers in parallel to confirm the fix.
Both verifiers confirm all checks pass. Let me query the final model for the answer.
Base Claude Code
claude code, no additional stack
report_customer_invoices: FAIL — no pred column matched gold column 'total_invoices'
All 7 models materialized successfully. Here are the top 10 countries by total revenue:
| Rank | Country | Total Invoices | Total Revenue |
|---|---|---|---|
| 1 | United Kingdom | 16,646 | $7,308,391.55 |
| 2 | Netherlands | 94 | $285,446.34 |
| 3 | EIRE | 260 | $265,545.90 |
| 4 | Germany | 457 | $228,867.14 |
| 5 | France | 389 | $209,024.05 |
| 6 | Australia | 57 | $138,521.31 |
| 7 | Spain | 90 | $61,577.11 |
| 8 | Switzerland | 51 | $56,443.95 |
| 9 | Belgium | 98 | $41,196.34 |
| 10 | Sweden | 36 | $38,378.33 |
Key observations:
- United Kingdom dominates by a massive margin — $7.3M revenue from 16,646 invoices, which is ~96% of total revenue across the top 10.
- Netherlands ranks 2nd despite having very few invoices (94), suggesting high-value orders.
- Australia similarly shows high revenue-per-invoice with only 57 invoices but $138K revenue.
- The
report_customer_invoicesmodel already existed in the project and correctly implements this analysis — it joinsfct_invoiceswithdim_customer, groups by country, sums revenue, and limits to top 10.
Let me read the existing report model to confirm it already handles this task.
The report_customer_invoices model already does exactly what's needed. Now let me run dbt to materialize all the tables.
All 7 models materialized successfully. Let me query the result.