Why SignalPilot passed where base failed
Evidence chips jump to the exact transcript momentAsked for halcyon_bank's total logged exposure impressions, Claude Code reported 230,760. The raw landing table holds 573,648: 60% of the rows had silently vanished below the mart. SignalPilot reconciled the count down to the raw data and recovered the full population.
Real-world impact
Capacity planning, billing, and experiment power calculations built on this count all start from a number 60% too low, and nothing in the pipeline flags it: the shrunken table has clean grain, clean uniqueness, and an authoritative name.
- Below the published mart, the staging model keeps only one of the three kind labels the raw stream logs exposures under, discarding the rest.
- Row counts and uniqueness checks still pass; only the total is wrong.
- Counted the authoritative-looking table and reported 230,760 with confidence; the raw landing table was never consulted.
- Started from the client's own staging and raw landing data instead of the published rollup, counting the population at every layer.
- Reported 573,648 from the source, matching the independently derived truth exactly.
The numbers
| Check | Correct answer | Base Claude Code | SignalPilot |
|---|---|---|---|
| Halcyon_bank's total logged exposure impressions | 573,648 | 230,760 (60% missing) | 573,648 |
Verification workflow
Where the eight-step workflow caught it.
- 01Map the project
Scan every model: dependencies, required columns, macros, hazards.
The project scan surfaced a 2.0 million row raw frames stream sitting below the exposures model; base Claude Code never mapped anything beneath the staging table it counted.
- 02Load supporting skills
SQL, database, and domain rules loaded before any writing.
- 03Validate the project
Parse check, plus rebuild of stale time-dependent upstreams.
- 04Discover project macros
Find macro-derived columns the YML contract does not list.
- 05Research the data
Driving table, cardinalities, contract, sibling patterns, categorical values.
Research traced the count to the raw frames and found the same events logged under three kind labels totaling 573,648, of which staging kept only one; base Claude Code cross-checked three pipeline layers that all inherit the same filter.
- 06Write the technical spec
A written plan of sources, joins, filters, and grain per model.
- 07Write and build models
SQL from the spec, with column mapping checked before each model.
- 08Verify and fix
Two read-only verifiers cross-check structure and values until every check passes.
SignalPilot
claude code + signalpilot governed stack
reported 573,648 · matches truth
Answer: 573,648 exposure impressions logged for halcyon_bank
⚠️ But note: the dbt pipeline currently reports only 230,760 — it's under-counting by 342,888 (≈60% missing). Here's why, and why 573,648 is the correct figure.
What the data shows
halcyon_bank's SDK emits exposure impressions into a raw frame stream (raw.client_blob → stg_halcyon__frames). Across the SDK's schema versions, the same exposure-impression event was renamed three times:
Frame kind | schema_version | Rows (distinct IDs) | Payload |
|---|---|---|---|
expose | 1.0 | 230,760 | {at, id, surface} |
exposure | 2.0 | 228,393 | {at, id, surface} |
imp (impression) | 3.0 | 114,495 | {at, id, surface} |
| Total | 573,648 |
I verified against the raw source that these are the true grain — 573,648 distinct exposure IDs, zero duplicate IDs, and zero overlap across the three labels (every ID appears under exactly one kind). They are disjoint events, not re-emissions, so they sum cleanly.
Why the pipeline number is wrong
models/staging/halcyon_bank/stg_halcyon__exposures.sql hardcodes:
where kind = 'expose'
It captures only the v1.0 label and silently drops the v2.0 (exposure) and v3.0 (imp) events. Every downstream model (int_halcyon__exposures, fct_halcyon__daily_metrics, org.int_all__exposures, etc.) inherits this filter — which is exactly why they all "agree" on 230,760 while all being wrong. They share the broken upstream, so reconciliation between them proves nothing.
This is a real dbt defect
It also corrupts every derived exposure metric for halcyon_bank (exposure rates, surface engagement, exposed_enrollments, org-level rollups).
Would you like me to fix stg_halcyon__exposures.sql to accept all three kinds (kind in ('expose','exposure','imp')) and rebuild the affected downstream models? I'd run it through the standard workflow (spec + verification) before finalizing.
I'll start by loading the required workflow skill, as mandated by the project instructions.
This is a business-outcome question. Per the workflow, I'll still run the exploration steps. Let me start Step 1: scan the project and load the SignalPilot MCP tools I'll need.
Now let me load the tools I need for exploration.
Now let me run the knowledge base check, the project scan, and DB analysis in parallel.
This is a multi-tenant experimentation platform. Let me probe the knowledge base and inspect the database schema. Let me run several single-keyword knowledge searches plus list the tables.
The halcyon codename likely maps to a client_id. Note halcyon has an unusual frame-based ingestion (stg_halcyon__frames, 2.0M rows) that likely feeds the exposures. Per the grain-verification rule, I must trace exposure counts back to the raw source, not trust a staging/mart number. Let me inspect the lineage and the raw data.
Critical finding. The raw stg_halcyon__frames stream logs exposure-type events under three different kind labels — expose (230,760), exposure (228,393), and imp (impression, 114,495) — but the staging model stg_halcyon__exposures has exactly 230,760 rows, matching only the expose kind. The staging/int/mart numbers all agree only because they inherit the same upstream filter. I need to determine whether these three kinds are genuinely the same "exposure impression" event or distinct things. Let me inspect the model SQL and the raw frame payloads.
The three kinds map cleanly to three schema_versions: expose=v1.0, exposure=v2.0, imp=v3.0 — the SDK renamed the same exposure-impression event across schema versions. The staging model captured only v1.0. Let me read the model SQL and inspect the v2/v3 payloads and check for ID overlap across versions (to avoid double-counting).
Confirmed the bug: stg_halcyon__exposures hardcodes where kind = 'expose', capturing only schema v1.0 and silently dropping v2.0 (exposure) and v3.0 (imp) — the same exposure-impression event under renamed labels. Now I'll compute the true total, deduplicated by the frame's exposure id, and check for any ID overlap across versions plus confirm the imp payload shape.
Base Claude Code
claude code, no additional stack
reported 230,760 · truth 573,648 (60% of rows silently dropped)
Each row is one exposure impression (one unique exposure_id, no per-row multiplier column), so the count is a direct row count.
Answer: 230,760 exposure impressions
For halcyon_bank (the halcyon client codename in the dbt project), we've logged 230,760 exposure impressions in total.
This figure is solid — it's identical and fully consistent across all three pipeline layers:
staging.stg_halcyon__exposures→ 230,760intermediate.int_halcyon__exposures→ 230,760org.int_all__exposures(filtered toclient_id = 'halcyon_bank') → 230,760
All 230,760 exposure_id values are distinct, so there's no duplication inflating the number, and no merging/dedup loss between layers.
One thing worth flagging for the SDK audit: halcyon_bank's exposure and enrollment activity appears to stop around 2026-04-24 (per fct_org_sdk_adoption's last_seen_ts), whereas other clients show activity through late June 2026 — so this is a lifetime total for a client that seems to have gone quiet ~3 months ago. Let me know if you'd like the count broken down by experiment, surface, platform, or time period.
I'll explore the warehouse to find exposure impression data for halcyon_bank's team.
halcyon_bank is the halcyon client. Let me count exposures across the pipeline layers to find the right "total logged" figure.
All three pipeline layers agree exactly, and every exposure_id is unique. Let me confirm the staging table doesn't carry a separate impression-count column (in case "impression" ≠ one row).