Why SignalPilot passed where base failed
Evidence chips jump to the exact transcript momentBoth tables were the exact same size. Claude Code still overstated churn by 13 percent, reporting 62 churned customer-months where the truth is 55, because it kept re-labeling customers who had already left months earlier. SignalPilot labeled all 410 rows correctly.
Real-world impact
The churn figure going into revenue reporting was 13 percent too high. A team reading it would believe customer loss is meaningfully worse than it is, in a metric boards and investors watch closely.
- Its churn rule only checked "inactive this month", so customers already gone for months kept getting stamped churn again and again.
- Seven rows that should carry no label were labeled churn: 62 instead of 55.
Jump to the exact moment
- Wrote all six labeling rules in full before any code, including the missed case: an inactive month after another inactive month gets no label.
- Required "was active last month" before anything is called churn, with the rules confirmed against the 410-row reference snapshot during research.
Jump to the exact moment
The numbers
| Check | Correct answer | Base Claude Code | SignalPilot |
|---|---|---|---|
| Churned customer-months | 55 | 62 (13% overstated) | 55 |
| All other categories | · | correct | correct |
Verification workflow
Where the eight-step workflow caught it.
- 01Map the project
Scan every model: dependencies, required columns, macros, hazards.
- 02Load supporting skills
SQL, database, and domain rules loaded before any writing.
- 03Validate the project
Parse check, plus rebuild of stale time-dependent upstreams.
- 04Discover project macros
Find macro-derived columns the YML contract does not list.
- 05Research the data
Driving table, cardinalities, contract, sibling patterns, categorical values.
Research opened the 410-row reference snapshot and traced both upstream tables before any code; base Claude Code never consulted a reference at all.
- 06Write the technical spec
A written plan of sources, joins, filters, and grain per model.
The spec wrote out all six labeling rules before any code, including the case base Claude Code missed: an inactive month after another inactive month gets no label.
- 07Write and build models
SQL from the spec, with column mapping checked before each model.
- 08Verify and fix
Two read-only verifiers cross-check structure and values until every check passes.
SignalPilot
claude code + signalpilot governed stack
All 8 steps complete. Here's what was built and verified:
✅ MRR Project — Complete
What was done
- Scanned the project — found 1 stub (
mrr) depending on 2 pre-existing models - Loaded dbt-write, DuckDB SQL, and domain-financial skills
- Fixed the
current_datehazard inutil_monthsand rebuilt downstream models (customer_revenue_by_month,customer_churn_month) - Researched via blueprint — 358 rows from active months + 52 churn rows = 410 total
- Completed
mrr.sql— added thefinalCTE with:mrr_change = mrr - previous_month_mrrchange_categoryCASE WHEN logic for all 6 categories (new/reactivation/upgrade/downgrade/churn/NULL)
- Verified — both structure and value verifiers returned all PASS:
- 410 rows ✓ · all 12 YML columns present ✓
SUM(mrr) = 17,020matches upstream exactly ✓mrr_changearithmetic correct for all 410 rows ✓change_categoryonly contains valid values ✓- Reference sample (customer 35: new → downgrade → churn) matches exactly ✓
I'll follow the 8-step dbt workflow. Let me start by checking the knowledge base and scanning the project simultaneously.
Now I have a clear picture. Let me set up tasks and load all supporting skills in parallel.
All 3 skills loaded. Now let me create the task list and run Step 3 validation in parallel with reading the existing models.
Project parses cleanly. Now I need to rebuild the util_months model (current_date hazard) and read all existing models. Let me do these in parallel.
Now I need to rebuild util_months (current_date hazard) and read the YML files in parallel.
util_months rebuilt. Now rebuild the downstream pre-existing models and read all YML files in parallel.
Now let me find the docs for change_category and explore the source data in parallel.
Let me read the reference snapshot and also mark Step 3 complete, then move to Steps 4-6.
I have all the information needed. Now let me load the knowledge-base skill and update tasks.
Now I'll write the technical spec (Step 6) and then complete the mrr model SQL (Step 7).
Spec written. Now Step 7 — complete the mrr stub SQL.
Now build the model:
Build successful. Now Step 8 — verify with both verifiers in parallel.
DB is live. Dispatching both verifiers in parallel:
Both verifiers return all PASS. The model is correct.
Base Claude Code
claude code, no additional stack
mrr: FAIL — no pred column matched gold column 'change_category'
All 4 models are materialized successfully. Here's a summary of what was done:
Completed mrr.sql by adding the missing final CTE and SELECT statement with:
mrr_change— difference between current and previous month's MRRchange_category— categorized as:new— customer's first active monthchurn— customer went inactive (mrr = 0)reactivation— previously inactive customer became active againupgrade— MRR increased from prior monthdowngrade— MRR decreased but still activenull— no change (stable)
Results in the mrr table (410 rows):
| change_category | count |
|---|---|
| (no change) | 236 |
| churn | 62 |
| new | 52 |
| upgrade | 30 |
| downgrade | 27 |
| reactivation | 3 |
Let me read the current mrr.sql file and the other model files to understand the complete picture before making changes.
The mrr.sql is incomplete — it has the CTEs but is missing the final SELECT with mrr_change and change_category. Let me complete it.
Now run dbt to materialize all models:
All 4 models materialized successfully. Let me verify the final mrr table looks correct: