Analysis rules are knowledge objects, not prompt text

Here is a question with a correct answer that no competent analyst would publish.

Do married customers churn less than single customers?

Run it against a customer table and you get a clean result. Married customers churn at 4.1 percent, single customers at 7.8 percent. Nearly double. The SQL is right, the join is right, the numbers reconcile.

The finding is close to worthless, and a senior analyst knows why in about two seconds: marital status is heavily confounded with age, age drives tenure, and tenure drives churn. You have not discovered anything about marriage. You have rediscovered that older customers stay longer. Report it as a marital-status effect and someone will build a campaign on it.

The correct move is to control for age band first, then look at whether any marital-status difference survives. Usually most of it does not.

Now the important part: that is not a fact about your data. It is a rule about your method. And almost every AI analytics product on the market has nowhere to put it.

The rules a senior analyst is actually carrying

Ask an experienced analyst what they know that a new hire does not, and you rarely get “SQL”. You get a body of accumulated caution:

• Suppress any cell computed from fewer than 30 records. Below that the rate is noise, and publishing it invites someone to act on it.
• Marital status, job title and postcode are all proxies for age or income. Control before comparing.
• Never compare this month to last month on a business with weekly seasonality. Compare to the same period last year, or use a trailing four-week window.
• Revenue excludes intercompany unless the question is explicitly about gross bookings.
• Anything sourced from the pre-migration warehouse before March is not comparable to anything after it.
• If the denominator moved more than the numerator, lead with the denominator.

None of these are metric definitions in the semantic-layer sense. A semantic layer will tell you what revenue means. It will not tell you that comparing this March to last March is the only honest way to look at it, or that a 40 percent churn rate computed from five customers should never leave the building.

These are analysis rules, and they are the actual difference between a fast answer and a defensible one.

Where these rules currently live

In three places, all of them bad.

In people’s heads. Works until that person is on holiday, or leaves, or is simply not in the room when someone else runs the query.

In a prompt. This is what most current tools offer: a large text box labelled “instructions” or “context”, into which you paste your conventions and hope the model applies them. It is genuinely better than nothing and it fails in specific, predictable ways.

In a wiki nobody reads. The analytics handbook that was written once, in 2023, and is now wrong in about a third of its claims.

Why the prompt box is not enough

A prompt is a string. That is the whole problem.

You cannot review a string. There is no diff on “we changed the small-cell threshold from 30 to 50 in March”. There is no approver, no comment, no record of who decided and why.

You cannot test a string. A rule like small-cell suppression has an obvious test: ask a question whose answer depends on a cell of 12, and check that the system suppressed it. Rules stored as knowledge objects can carry their test cases. Rules stored as prose cannot be exercised at all.

It does not scale past a page. A hundred rules will not fit in a prompt without crowding out the actual question, and the ones at the bottom get applied least reliably. Retrieval solves this — but only if the rules are discrete objects that can be retrieved individually, which means they cannot be one blob of text.

It cannot be scoped. The suppression threshold in a clinical context is not the suppression threshold in a marketing context. A prompt applies to everything or nothing. An object can be attached to a domain, a dataset, a team.

It leaves no audit trail. When someone asks six months from now why the March report suppressed a segment, “it was in the prompt at the time” is not an answer. The prompt has been edited nine times since.

What a rule looks like as an object

Give the rule the same treatment you would give a metric definition — an identity, a scope, a rationale, a version, and something to test it against:

rule: small-cell-suppression
scope: [customer-analytics, hr-analytics]
applies_to: any rate or percentage
condition: denominator < 30
action: suppress cell, report "n too small" instead of the value
rationale: >
  Below n=30 the confidence interval on a proportion is wider than
  almost any effect we would act on. Reporting the point estimate
  invites decisions the data cannot support.
owner: analytics-governance
version: 3
supersedes: 2 (threshold was 20 before 2026-03-01)
test:
  question: What is churn for enterprise accounts in the Nordics?
  expect: suppressed, n = 12

Now it is reviewable, because version 3 has a diff against version 2. It is testable, because the test case is attached. It is scoped, so it does not fire in a context where n=12 is a legitimate population rather than a sample. And when the March report comes up in an audit, the rule that governed it is retrievable at the version that was live in March.

The confounding rule gets the same treatment, and it is more interesting because it changes the shape of the answer rather than hiding a cell:

rule: control-for-age-before-demographic-comparison
scope: [customer-analytics]
applies_to: any comparison across marital status, job title or postcode
action: >
  Stratify by age band and report the within-band comparison.
  If the effect does not survive stratification, say so explicitly
  rather than reporting the unadjusted difference.
rationale: >
  These attributes are proxies for age in our customer base.
  The unadjusted comparison is arithmetically correct and
  directionally misleading.
owner: analytics-governance
version: 1

A system holding that object does not answer the marital-status question with 4.1 versus 7.8. It answers with the stratified comparison, and it says why it did that.

This is the part that is hard to copy

There are good products that read documents. There are good products that generate governed SQL. What we have not seen anyone else build is the third thing: the analytical discipline itself, stored as versioned, retrievable, testable knowledge that both halves are subject to.

It is unglamorous, which is part of why it is uncontested. It does not demo as well as a chart appearing from a sentence. It is also the entire difference between a tool that produces answers and a tool that produces answers you can put in front of a regulator.

And it compounds in a way the other two do not. Every time an analyst corrects an output — no, control for age; no, that cell is too small; no, compare year over year — that correction can become an object, scoped and versioned, that applies to every future question. The system gets more like your best analyst rather than more like the average of the internet.

What to ask a vendor

If you are evaluating anything in this space, these four questions separate the categories quickly:

1. Where does a rule like “suppress cells under n=30” live, and can I see its version history?
2. If I change that threshold, what breaks, and can I test it before it ships?
3. Can a rule be scoped to one dataset or team, or is it global?
4. Six months from now, can you show me the exact rule set that governed a specific answer?

If the answer to all four is “it’s in the system prompt”, you are buying a fast junior analyst. That is a real thing to buy. It is not the same thing as a senior one.