Why SignalPilot passed where base failed
Evidence chips jump to the exact transcript momentAsked for halcyon_bank's total enrollments, Claude Code reported 186,432. The raw landing table holds 232,980: 20% of the rows had silently vanished below the mart. SignalPilot reconciled the count down to the raw data and recovered the full population.
Real-world impact
Capacity planning, billing, and experiment power calculations built on this count all start from a number 20% too low, and nothing in the pipeline flags it: the shrunken table has clean grain, clean uniqueness, and an authoritative name.
- Below the published mart, a WHERE sdk_version IS NOT NULL filter drops every enrollment whose assign event carried no SDK version.
- Row counts and uniqueness checks still pass; only the total is wrong.
- Counted the authoritative-looking table and reported 186,432 with confidence; the raw landing table was never consulted.
- Started from the client's own staging and raw landing data instead of the published rollup, counting the population at every layer.
- Reported 232,980 from the source, matching the independently derived truth exactly.
The numbers
| Check | Correct answer | Base Claude Code | SignalPilot |
|---|---|---|---|
| Halcyon_bank's total enrollments | 232,980 | 186,432 (20% missing) | 232,980 |
Verification workflow
Where the eight-step workflow caught it.
- 01Map the project
Scan every model: dependencies, required columns, macros, hazards.
The scan mapped the staging and intermediate layers under the published mart; base Claude Code listed warehouse tables and stopped at the org layer.
- 02Load supporting skills
SQL, database, and domain rules loaded before any writing.
- 03Validate the project
Parse check, plus rebuild of stale time-dependent upstreams.
- 04Discover project macros
Find macro-derived columns the YML contract does not list.
- 05Research the data
Driving table, cardinalities, contract, sibling patterns, categorical values.
Research counted staging at 232,980 against the intermediate's 186,432 and read the model SQL, exposing a WHERE sdk_version IS NOT NULL filter silently dropping 46,548 enrollments; base Claude Code never looked beneath the mart.
- 06Write the technical spec
A written plan of sources, joins, filters, and grain per model.
- 07Write and build models
SQL from the spec, with column mapping checked before each model.
- 08Verify and fix
Two read-only verifiers cross-check structure and values until every check passes.
SignalPilot
claude code + signalpilot governed stack
reported 232,980 · matches truth
Answer: 232,980 enrollments (all time, halcyon_bank)
How I verified it
Grain confirmed at the raw source. Enrollments originate as kind='assign' frames in the raw event stream (stg_halcyon__frames). I checked the grain directly:
| Check | Value |
|---|---|
| Assign frames (raw) | 232,980 |
Distinct enrollment_id in frames | 232,980 |
stg_halcyon__enrollments rows | 232,980 |
Distinct enrollment_id in staging | 232,980 |
The enrollment table is 1:1 with assignment events — one enrollment_id per subject-experiment assignment, no fan-out. The project's own grain test (assert_halcyon_enrollments_grain) enforces this. So 232,980 is the true all-time count of subject-experiment assignments.
⚠️ One thing you should know
If you query the analytics layer (int_halcyon__enrollments and the fct_halcyon__* models built on it), you'll get 186,432, not 232,980. The intermediate model applies where sdk_version is not null, which silently drops 46,548 enrollments (20% of all assignments) whose assign frame carried no SDK version.
- 232,980 — every subject-experiment assignment ever recorded (what you asked for)
- 186,432 — assignments with a known SDK version (what the metrics/results models actually count)
So the honest all-time enrollment figure is 232,980. Just be aware that any downstream conversion/revenue metric for halcyon is computed on the 186,432 subset — that filter isn't documented as intentional, so it's worth confirming whether the NULL-SDK exclusion is deliberate or an upstream data-quality issue.
A minor related note: those 232,980 enrollments cover 227,494 distinct subject-experiment pairs — i.e. ~5,486 cases where a subject was re-assigned to an experiment they were already in. Since each assignment has its own enrollment_id, all 232,980 count as enrollments; I did not dedupe to the pair level.
This is a data question about halcyon_bank enrollments. Per the project workflow, let me load the dbt-workflow skill which owns the exploration steps.
I'll investigate this. Let me start by scanning the dbt project and checking the knowledge base in parallel.
Let me scan the project, check the knowledge base, and get a schema overview in parallel.
Let me search the knowledge base for the key concepts and explore the halcyon tables.
Found the relevant tables. There are both enrollments and exposures tables. The question defines enrollments as "subject-experiment assignments". Let me examine the enrollment tables' structure and grain.
The enrollments table (enrollment_id, subject_key, experiment_key, variant, assigned_ts) is exactly the subject-experiment assignment. Now let me verify grain against the raw source. Let me find the raw table and compare counts.
Staging has 232,980 rows, grain = enrollment_id (no fan-out). Note distinct (subject, experiment) pairs = 227,494, slightly fewer — some subjects re-enrolled. I must verify against the raw source per project rules. The raw table isn't named with "enroll" — let me find what staging reads from.
The project folder is literally halcyon_bank — confirming halcyon = halcyon_bank. Let me read the staging model to find its raw source, and check the grain test.
Key finding: staging derives enrollments from stg_halcyon__frames where kind='assign' (each assign frame = one enrollment). But int_halcyon__enrollments applies a filter where sdk_version is not null, dropping some. Let me verify the raw grain at the frames level and quantify the filter's effect.
Base Claude Code
claude code, no additional stack
reported 186,432 · truth 232,980 (20% of rows silently dropped)
Answer: 186,432 enrollments for halcyon_bank, all time.
This is from org.int_all__enrollments (the cross-client enrollment model), where each row is one enrollment record with a unique enrollment_id.
One nuance worth flagging, since you defined enrollments as subject-experiment assignments:
| Grain | Count |
|---|---|
Enrollment records (enrollment_id) | 186,432 |
| Distinct (subject, experiment) pairs | 182,936 |
The ~3,500 gap comes from subjects who were enrolled into the same experiment more than once — often into a different variant on the repeat assignment (e.g. card_activate_push, apy_banner_test). So:
- If you want the count of enrollment/assignment events, it's 186,432.
- If you want unique subject↔experiment pairs, it's 182,936.
I checked the canonical model (int_org__enrollments_canonical); its canonical_experiment_key doesn't collapse halcyon any further, so the gap is genuinely re-assignments, not experiment-key aliasing.
Want me to break the 186,432 down (by experiment, or excluding is_internal_subject), or dig into those repeat/variant-switch assignments as a data-quality check?
I'll explore the warehouse schema to find the relevant tables for enrollments.
I'll look at the marts and staging tables since those are typically the query-ready layer.
There's an org.int_all__enrollments table that looks cross-client. Let me inspect it and understand how clients are identified.
Let me verify the grain (no duplicate enrollment rows) and cross-check against the canonical table.
186,432 enrollment records, but only 182,936 distinct subject-experiment pairs. Since the user defined enrollments as subject-experiment assignments, that gap matters. Let me investigate the duplicates.
The canonical table adds a canonical_experiment_key — suggesting some experiment keys are aliases. Let me check how that affects the subject-experiment grain.