Context & EvalsEveryone sells you a context layer, but nobody maintains it.
We parse your dbt models, context layers, and dashboards to author the definitions your agents need, generate the evals that prove them, and rerun those evals on every change.
What we read, what we author
Schemas, dbt models and tests, trusted dashboards, and the queries behind them. That is where the calculations, filters, and source choices already live.
- Metric definitions: meaning, calculation, sources, exceptions.
- Required changes to dbt models, proposed as pull requests.
- Business context in Markdown, next to the models.
- Metric definitions in the Apache Ossie format so your BI tools can read them too.
You approve. We come back only where the evidence is ambiguous.
Evals anchored in your approved definitions
Graded questions around the workflows you scoped. Each one is anchored twice: a gold value and the approved definition.
- Gold value: the number we agreed is right, reconciled against your warehouse, never against the agent's own answer.
- Approved definition: the rubric checks what the definition says, the join grain, the exclusions, the time window.
- Failure cases are questions too: a double-counted join, refunds left in, the wrong time window.
- The system never passes because our agent agrees with itself.
gross_amount − refunded_amount · fiscal quarter · one row per order
What was net revenue last quarter, excluding refunds?
- ✓join grain 1:1 on order_id1 : 1
- ✓refunds excluded2,318 rows
- ✓time window2026-04-01 → 06-30
Gate every change through GitHub
Every change to your dbt project is a pull request, and the evals it touches decide whether it merges.
- A change lands as a pull request, yours or one SignalPilot proposed.
- Only the affected evals run: 4 of 24, picked from the blast radius, not the whole suite.
- Failures name the models: two red, two green, and the fix required.
- A reviewer approves, the affected evals rerun, and the merge unlocks only when they pass.
feat: refund reason on int_orders_with_refunds #212
signalpilotproposed · feat/refund-reason → main
- add refund_reason to int_orders_with_refunds
- fix: keep 1:1 grain on order_id · signalpilot
- ✓Dana R. approved these changes
Keeping it honest
Coverage is the health metric.
Two numbers on one dashboard: how many of your models have graded questions, and how often those questions passed in the last 30 days. Together they tell you whether the context can be trusted and what to cover next. The more models with graded questions, the better the answers.
Thumbs-downs, gaps, and new questions are clustered daily and delivered as recommendations, usually as a pull request that adds the missing evals or context rules.
Bring your dbt project. We will show you what it already knows.
Open source · Apache 2.0 · GitHub · Slack