SignalPilot is #1 on both public data-agent benchmarksSee the results →
All cases

Poisoned money sums

Finance needs our total billed charges (USD) across all active reporting clients for the board deck. What's the org-wide number?

Handed the CFO $267 billion in total billed charges for the board deck. The real number is $71 billion. It verified the wrong figure twice.

Case 1 of 39Δ $195.9Bnorthwind

Why SignalPilot passed where base failed

Evidence chips jump to the exact transcript moment

Asked for the org-wide total billed charges, Claude Code reported $267 billion. The real number is $71 billion. It ran two cross-checks, watched them agree, and called the wrong figure verified. SignalPilot reconciled the mart against its source and reported the correct $71.08 billion.

Real-world impact

A 3.76x overstatement of the company's headline financial number, delivered to the board deck with confidence. Claude Code read both financial models but never ran the one reconciliation that would have exposed the gap.

Where base Claude Code went wrong
  • A hidden diagnoses join inflates 7 of the 9 clients' charge sums roughly 4.5x before aggregation; row counts and uniqueness checks stay clean.
  • Summed the official rollup, then cross-checked it against the per-client mart.
  • Both tables are built from the same poisoned layer, so they agreed to the cent, and that agreement convinced it the number was right.
What SignalPilot did differently
  • Did not accept agreement between two published tables as proof.
  • Rebuilt the total from three independent source paths that all agree to the cent: the mart was 3.76x higher.
  • Traced the gap to the hidden diagnoses join and reported the corrected $71,081,442,304.

The numbers

CheckCorrect answerBase Claude CodeSignalPilot
Org-wide billed charges$71,081,442,304$266,999,033,390 (3.76x)$71,081,442,304

Verification workflow

Where the eight-step workflow caught it.

Read the workflow docs
  1. 01
    Map the project

    Scan every model: dependencies, required columns, macros, hazards.

    The project scan mapped the model lineage and surfaced the clean service-line layer to reconcile against; base Claude Code went straight to the published mart and summed it.

  2. 02
    Load supporting skills

    SQL, database, and domain rules loaded before any writing.

    The financial domain skill's grain-consistency rules framed the reconciliation before any query ran; base Claude Code loaded no rules and accepted the mart's aggregate at face value.

  3. 03
    Validate the project

    Parse check, plus rebuild of stale time-dependent upstreams.

  4. 04
    Discover project macros

    Find macro-derived columns the YML contract does not list.

  5. 05
    Research the data

    Driving table, cardinalities, contract, sibling patterns, categorical values.

    Research rebuilt the total from three independent source paths that all agree at $71,081,442,304 and traced the inflation to a diagnoses join hitting 7 of the 9 clients about 4.5x; base Claude Code cross-checked two marts built from the same poisoned layer and called $267 billion verified.

  6. 06
    Write the technical spec

    A written plan of sources, joins, filters, and grain per model.

  7. 07
    Write and build models

    SQL from the spec, with column mapping checked before each model.

  8. 08
    Verify and fix

    Two read-only verifiers cross-check structure and values until every check passes.

SignalPilot

SignalPilot

claude code + signalpilot governed stack

pass
reported $71,081,442,304 · matches truth
34 turns31 tool calls$$1.69answer pinned · full chat below
final answer pass

Bottom line for the board deck: org-wide total billed charges = $71,081,442,303.60 (≈ $71.08B USD) across all 9 active reporting clients — not the ~$267B the published fct_org_charges_by_client mart reports, which is inflated ~3.75× by a diagnoses fan-out. Let me know if you'd like me to fix the underlying models.

full chat, from the start
Finance needs our total billed charges (USD) across all active reporting clients for the board deck. What's the org-wide number?

I'll start by loading the dbt-workflow skill, which owns the workflow for this project.

This is a business-outcome question. Per the workflow, I'll run the exploration steps. Let me first load the SignalPilot MCP tools and check the knowledge base, while scanning the project.

Now let me load the actual tool schemas I need.

Now let me run the knowledge base check and project scans in parallel (Step 1).

Healthcare domain. There's a directly relevant mart fct_org_charges_by_client. Let me read the full scan and load the domain skill. First, the full scan output.

This is a multi-client healthcare RCM project. There's a directly relevant mart: fct_org_charges_by_client. The question has important qualifiers — "active reporting clients" and "USD" (currency normalization). Let me examine the relevant models rather than blindly summing. Let me load domain skills and read the key model files.

Now let me read the key model files: the charges-by-client mart, its YML, and the normalization assertion.

The mart sums total_charge_usd from fct_org_claims, which comes from int_all__claim_financials. The qualifier "active reporting clients" and "USD" normalization are key. Let me look at the union model and search for any client-status/active concept and currency handling.

Nine clients are unioned with no "active reporting" filter here. Two things need investigation: (1) what "active reporting clients" means (some clients may be excluded), and (2) whether total_charge_usd is genuinely normalized to USD per client. Let me probe the knowledge base and read the normalization assertion plus per-client financials.

Knowledge base is empty. The assert_org_charges_normalized check (flagging clients whose avg charge exceeds 150k) is a strong signal that currency/scale normalization is a real risk here. Let me inspect how each client derives total_charge_usd.

Now I see structurally important differences. Six clients (northstar, bayview, plains, harbor, riverside, metro) and summit compute total_charge_usd by joining service_lines to diagnoses on encounter_id and then SUM(charge_usd) — that join fans out charges by the number of diagnoses per encounter. Meanwhile cedar and valley take total_charge_usd directly from the claims table. This is exactly a grain hazard that would silently inflate the published mart. I must verify grain against raw data before trusting any total. Let me look at what's materialized and check the raw tables.

fct_org_claims is 2.0M rows (≈ sum of the 9 clients' claim counts), so its grain is one row per claim — the row count is fine. The hazard is in the value: for the 7 clients whose charges CTE joins service_lines to diagnoses on encounter_id, SUM(charge_usd) is multiplied by the diagnosis-count per encounter. That inflates total_charge_usd in the mart without changing row counts — a silent value corruption.

There's a clean union — int_all__service_lines — and int_all__encounters.service_line_charge_usd. Let me verify grain and quantify the inflation. Let me pull the naive mart number, the clean service-line total, and a grain check.

This is decisive. int_all__service_lines has clean grain (n_rows == n_distinct_claim_line), and comparing its charge totals to the mart:

  • cedar and valley: mart total exactly equals the clean service-line sum (ratio 1.00×).
  • The other 7 clients: mart is ~4.5× inflated — precisely the diagnosis fan-out I traced (avg ~4.5 diagnoses/encounter multiplying each service line).

The two clients whose charge comes from an independent claim-header field agree perfectly with the clean service-line sum, which strongly validates the clean number. Let me triple-confirm by cross-checking the source claim-header total_charge_usd (present in staging for 8 clients) against the clean service-line sums, and check the encounter-level clean path. I'll also check Notion for any "active reporting clients" definition.

The cross-checks are conclusive. For all 8 clients carrying a source claim-header total_charge_usd, the header sum equals the clean service-line sum to the penny — two fully independent source representations agree. Summit (no header; charge only at line level) is covered by its clean line sum. And a third independent path (int_all__encounters.service_line_charge_usd) sums to exactly $71,081,442,303.60, matching my per-client clean totals.

Let me do due diligence on "active reporting clients" via Notion before finalizing.

Org-wide total billed charges: ≈ $71.08 billion USD ($71,081,442,303.60)

across all 9 active reporting clients (bayview, cedar, harbor, metro, northstar, plains, riverside, summit, valley).

⚠️ Do not use the number the published mart gives — it is inflated ~3.75× to ~$267.0B. Here's why, and how I verified the correct figure.

The mart is silently broken by an upstream fan-out

fct_org_charges_by_client sums total_charge_usd from fct_org_claims. For 7 of the 9 clients, that charge is computed in int_<client>__claim_financials by joining service_lines to diagnoses on encounter_id before SUM(charge_usd). That join multiplies every service line by the number of diagnoses on its encounter (~4.5 on average), so each claim's charge is counted ~4.5×. It doesn't add rows — the claim count stays correct — so the corruption is invisible unless you check the value against source grain.

ClientMart (naive)Correct billed chargeInflation
bayview32.27B7,175,420,144.874.50×
cedar7.88B7,877,937,544.101.00× ✓
harbor46.57B10,355,678,750.444.50×
metro29.54B6,567,401,322.574.50×
northstar39.31B8,734,413,027.694.50×
plains22.69B5,039,433,956.754.50×
riverside24.93B5,544,542,046.324.50×
summit56.59B12,557,516,112.524.51×
valley7.23B7,229,099,398.341.00× ✓
Org total≈267.0B71,081,442,303.60

cedar and valley are correct because they take the charge from the claim header instead of the fanned-out join.

How I confirmed the $71.08B figure (three independent source paths agree)
  1. Clean service-line grainint_all__service_lines has one row per (claim, line) (row count = distinct claim+line), summing to $71.08B.
  2. Source claim headers — for all 8 clients that carry a source-provided total_charge_usd header, the header sum equals the clean service-line sum to the penny (two independent representations of the same charge).
  3. Encounter pathint_all__encounters.service_line_charge_usd sums to exactly $71,081,442,303.60.

Summit has no claim-header charge (line-level only), so its clean line sum is authoritative.

There is no active/inactive flag anywhere in the models, seeds, knowledge base, or Notion, and all 9 clients carry full claim data and are explicitly unioned — so "active reporting clients" = all 9.


This is a genuine dbt defect: the same fan-out inflates fct_org_charges_by_client, fct_org_claims, fct_org_monthly_charges, and every fct_<client>__claims mart for the 7 affected clients — anything downstream reporting charges is overstated. Would you like me to fix it? The fix is to source each claim's charge from the claim-header total_charge_usd (or a de-duplicated service-line sum) instead of the service_lines×diagnoses join in the seven int_<client>__claim_financials models, then rebuild.

Let me record this finding so it isn't rediscovered from scratch.

Claude Code

Base Claude Code

claude code, no additional stack

fail
reported $266,999,033,390 · truth $71,081,442,304 (3.76x inflated)
? turns19 tool callsanswer pinned · full chat below
final answer fail

Org-wide total billed charges: $266,999,033,390.31 (≈ $267.0B USD)

Across the 9 active reporting clients, spanning 2,046,163 claims. This is gross billed charges (total_charge_usd), independent of what was paid or denied — the right basis for a "billed charges" line.

Source: analytics_marts.fct_org_claims (the org roll-up that unions all client pipelines).

ClientClaimsBilled charges (USD)
summit_ortho360,61856,588,150,856.86
harbor_system298,45546,571,939,513.28
northstar_health251,47139,309,380,344.30
bayview_medical206,90532,267,692,366.77
metro_childrens189,27229,537,823,474.82
riverside_health159,14724,930,109,587.95
plains_regional144,98922,686,900,303.89
cedar_clinic226,6137,877,937,544.10
valley_care208,6937,229,099,398.34
Total2,046,163266,999,033,390.31
⚠️ One thing to confirm before it goes in the deck

There's a 10th client in the raw warehouse that this number excludes: lakeside_health — 12,000 claims, ~$11.3M in billed charges. I left it out because it is deliberately not part of the reporting pipeline:

  • No dbt model consumes it — int_all__claim_financials hardcodes only the 9 clients above.
  • Its raw rows have NULL load_ids (every other client has sequential load IDs), i.e. it landed via a one-off/pilot batch rather than production ingestion.

So by the pipeline's own definition of "reporting clients," it's excluded, and at $11.3M it's immaterial to a $267B total (0.004%). But "active reporting client" is a Finance/commercial designation I can't verify from the data — if Lakeside is in fact a live, contracted client, the number becomes $267,010,353,043.31 and someone should also file that it's missing from the pipeline.

Want me to include Lakeside, or leave the deck number at the 9-client $266,999,033,390.31?

full chat, from the start
Finance needs our total billed charges (USD) across all active reporting clients for the board deck. What's the org-wide number?

I'll explore the dbt project to understand the data model, then query the warehouse for the number.

There's a fct_org_charges_by_client mart — directly relevant. Let me read it and the org-level financials model.

The org number comes from fct_org_claims. But the question specifies "active reporting clients" — I need to check whether all 9 clients are actually active/reporting. Let me look for client metadata.

No "active/reporting" client flag exists in the dbt models — all 9 clients are unioned unconditionally in int_all__claim_financials. The phrase "active reporting clients" suggests not every client should count. Let me check the warehouse itself for a client registry, and confirm what's actually materialized.

raw.client_blob looks like a client registry. Let me inspect it.

2.1M rows — that's not a registry, it's a raw landing table. Let me look at its distinct clients and structure, and also check the dbt sources to understand the raw layer.

10 clients in raw — but the dbt pipeline only models 9. lakeside_health (12,000 rows) exists in raw but is not built into any staging/intermediate/mart model — it's not part of the reporting pipeline. Let me confirm the modeled clients and pull the actual charge totals from the mart.

Org-wide total across the 9 modeled clients is $266,999,033,390.31. Before I hand Finance a number, let me look at the one loose end — lakeside_health exists in raw but isn't in the reporting pipeline. I'll check what it looks like to judge whether it's a real active client that's been omitted.

Lakeside has real billable claim data, but note its load_id is NULL for all rows (every other client has sequential load IDs) — it looks like a pilot/off-pipeline batch, not part of production ingestion, which is why no dbt model consumes it. Let me quantify its charges so I can tell Finance exactly what's excluded.

<task-notification> <task-id>bcpirjn5f</task-id> <tool-use-id>toolu_014zxpo7V2pddiCJgPXzUE9p</tool-use-id> <output-file>/tmp/claude-1000/-work/1d6c839a-257f-4240-bd58-adcf9d1357a2/tasks/bcpirjn5f.output</output-file> <status>killed</status> <summary>Background command "Dump client_blob rows" was stopped</summary> </task-notification>

(That notification is just the old jsonb_pretty dump command being cleaned up — not relevant.) My answer above stands; let me know whether to include Lakeside or keep the 9-client total.

Case #1 · Poisoned money sums · SignalPilot