SignalPilot is #1 on both public data-agent benchmarksSee the results →
All cases

Wrong source, ignored contract

Pull a table with all balance sheet entries for asset, liability, and equity accounts. Make sure it includes account details, class, parent information, and the monthly period balances.

The project's own specification listed 17 required columns including account_ordinal. Claude code built 21 columns from the wrong upstream and never noticed the missing one.

Case 21 of 39Δ 4quickbooks

Why SignalPilot passed where base failed

Evidence chips jump to the exact transcript moment

The project's own specification listed exactly 17 required columns for the balance sheet, including the one that orders accounts as Asset, Liability, Equity. Claude Code shipped 21 columns and never included that one. SignalPilot followed the written contract exactly.

Real-world impact

The balance sheet arrived without the column that puts accounts in statement order. A financial statement that cannot arrange Assets before Liabilities before Equity is not usable as a balance sheet.

Where base Claude Code went wrong
  • Built the balance sheet from a different upstream than the specification prescribes; that upstream lacks the account ordering column.
  • Shipped 21 columns with the required one missing, while the 17-column specification sat in the project ignored.

Jump to the exact moment

What SignalPilot did differently
  • Built exactly what the specification prescribes: the prescribed upstream first, then the balance sheet with the 17 listed columns in order.
  • Ran map-columns against the written contract before each model; that check is the guard that catches a missing required column.

Jump to the exact moment

The numbers

CheckCorrect answerBase Claude CodeSignalPilot
Columns17, incl. account ordering21, ordering missing17
Rows276276276

Verification workflow

Where the eight-step workflow caught it.

Read the workflow docs
  1. 01
    Map the project

    Scan every model: dependencies, required columns, macros, hazards.

    The project scan surfaced the balance sheet's required 17-column list, including account_ordinal, before any SQL; base Claude Code never mentioned that column once.

  2. 02
    Load supporting skills

    SQL, database, and domain rules loaded before any writing.

  3. 03
    Validate the project

    Parse check, plus rebuild of stale time-dependent upstreams.

  4. 04
    Discover project macros

    Find macro-derived columns the YML contract does not list.

  5. 05
    Research the data

    Driving table, cardinalities, contract, sibling patterns, categorical values.

    Research pinned the prescribed upstream, the general ledger by period, and confirmed the ordinal semantics; base Claude Code pulled from an intermediate table that has no ordering column.

  6. 06
    Write the technical spec

    A written plan of sources, joins, filters, and grain per model.

    The spec recorded the exact CASE expression that orders accounts as Asset, Liability, Equity.

  7. 07
    Write and build models

    SQL from the spec, with column mapping checked before each model.

    map-columns ran before each model, checking the output against the written contract so account_ordinal made it into the build; base Claude Code hand-picked 21 columns and dropped the required one.

  8. 08
    Verify and fix

    Two read-only verifiers cross-check structure and values until every check passes.

    The structure verifier confirmed complete and exact column schemas across all five models; base Claude Code dispatched no verifier and shipped the missing column unchecked.

SignalPilot

SignalPilot

claude code + signalpilot governed stack

pass
78 turns131 tool calls$$2.63answer pinned · full chat below
final answer pass

All steps complete. Here's what was built and verified:


Summary

Models Built (5 stubs rewritten)
ModelRowsRole
int_quickbooks__cash_flow_classifications276Intermediate: balance_sheet period balances with cash flow type/ordinal
quickbooks__general_ledger_by_period759Mart: all accounts across all periods with account_ordinal
quickbooks__balance_sheet276Target: balance sheet entries for Asset/Liability/Equity
quickbooks__profit_and_loss0Mart: income statement (no data in this dataset)
quickbooks__cash_flow_statement276Mart: cash flow period balances with beginning/net
quickbooks__balance_sheet columns

The balance sheet includes exactly what was requested — account details, class, parent info, and monthly period balances:

  • Period dates: calendar_date, period_first_day, period_last_day
  • Account details: account_id, account_name, account_number, account_type, account_sub_type, account_class
  • Class & parent info: class_id, is_sub_account, parent_account_number, parent_account_name
  • Tenant: source_relation
  • Monthly balances: amount (period ending balance), converted_amount (exchange-rate adjusted)
  • Ordering: account_ordinal (Asset=1, Liability=2, Equity=3)
Key design decisions
  • Sources from quickbooks__general_ledger_by_period filtered to financial_statement_helper = 'balance_sheet'
  • amount = cumulative period_ending_balance (not net change — balance sheets are cumulative)
  • account_ordinal derived from account_class (Asset→1, Liability→2, Equity→3) per the financial_statement_ordinal default
full chat, from the start
Pull a table with all balance sheet entries for asset, liability, and equity accounts. Make sure it includes account details, class, parent information, and the monthly period balances.

I'll follow the 8-step dbt workflow. Let me start by checking the knowledge base and scanning the project simultaneously.

Good, no prior knowledge. Now let me create tasks and load supporting skills.

Now let me run Step 3 validation and also fix the current_date hazard:

Good, project parses cleanly. Now fix the current_date hazard and materialize the unmaterialized model:

Good. Now update tasks and proceed to Step 4 (macros already discovered in scan) and Step 5 (blueprint):

The get_enabled_unioned_models macro is for assembling double-entry transaction models — used by upstream models already. Now Step 5 — blueprint research:

Excellent! Key insights from blueprints:

  • quickbooks__balance_sheet sources from quickbooks__general_ledger_by_period filtered by financial_statement_helper = 'balance_sheet'
  • quickbooks__general_ledger_by_period uses int_quickbooks__cash_flow_classifications as driving table (276 rows)
  • Build order: int_quickbooks__cash_flow_classificationsquickbooks__general_ledger_by_periodquickbooks__balance_sheet

Let me read the existing intermediate models that feed these stubs:

Let me read the stub files and the YML schema to understand the full picture:

Now let me query the existing materialized tables to understand their structure:

The reference snapshot is critical. I can see exactly what's needed:

  • quickbooks__balance_sheet = 276 rows, sourced from quickbooks__general_ledger_by_period filtered to financial_statement_helper = 'balance_sheet'
  • int_quickbooks__cash_flow_classifications = 276 rows (driving table, provides account_ordinal)
  • quickbooks__general_ledger_by_period = 759 rows (UNION of balances + retained earnings + ordinal join)

Let me check variables/config for cash flow classification logic:

Key finding: 276 balance_sheet rows (all Bank/Asset) and 483 NULL-detail rows. Let me check dbt_packages for cash flow classification logic:

Let me verify the account_unique_id MD5 formula and understand the build order better:

I now have all the research needed. Let me run map-columns for the stubs and then move to Step 6 (write spec):

Writing the technical spec now:

Now Step 7 — write and build all models:

Now write all 5 SQL models:

Now write all four model files:

Now build all models in dependency order:

All 5 models built successfully. Now Step 8 — verify with both subagents in parallel:

Structure checks all PASS. The value verifier needs to continue — let me send it a follow-up:

Both verifiers return all PASS. Let me mark the workflow complete:

Claude Code

Base Claude Code

claude code, no additional stack

fail
quickbooks__balance_sheet: FAIL — no pred column matched gold column 'account_ordinal'
  quickbooks__general_ledger_by_period: PASS
20 turns37 tool calls$$0.36answer pinned · full chat below
final answer fail

The model is materialized and working. Here's what was built:

quickbooks__balance_sheet — 276 rows across 4 bank/asset accounts, with one row per account per monthly period.

Columns included:

GroupColumns
Account detailsaccount_id, account_name, account_number, account_type, account_sub_type, account_class
Classificationfinancial_statement_helper, date_year
Classclass_id
Parent infois_sub_account, parent_account_number, parent_account_name
Monthly balancesperiod_first_day, period_last_day, period_net_change, period_beginning_balance, period_ending_balance
Converted balancesperiod_net_converted_change, period_beginning_converted_balance, period_ending_converted_balance

The model filters int_quickbooks__general_ledger_balances to balance sheet accounts (financial_statement_helper = 'balance_sheet' or account_class IN ('Asset', 'Liability', 'Equity')). In this dataset only Asset/Bank accounts have transactions, but the filter will correctly include Liability and Equity accounts if they appear in the data.

full chat, from the start
Pull a table with all balance sheet entries for asset, liability, and equity accounts. Make sure it includes account details, class, parent information, and the monthly period balances.

Let me read the key existing models to understand the current state before building the balance sheet model.

No balance sheet model exists yet. Let me check the general_ledger model and what columns are in the DB for the intermediate tables.

Now I have everything I need. The int_quickbooks__general_ledger_balances model has all required columns. I'll create quickbooks__balance_sheet.sql filtering to balance sheet accounts with full account details, class, parent info, and monthly period balances.

Model materialized successfully. Let me verify the output.

Case #21 · Wrong source, ignored contract · SignalPilot