How to set up a metric catalog for your AI agents in PostHog
2026-10-21
What we built and what you get
Our weekly KPI report used the word "connected" with three different rules. One column counted an account in the week of its first-ever connection. Another counted people who triggered the event that week. The Zapier column counted people who subscribed to any Zapier trigger that week, repeat subscribers included. The rules lived in the report's scripts, where no agent could see them. Each agent asked about connections wrote its own query and got its own number.
PostHog's Data Catalog is a list of number definitions that every agent must follow. PostHog's docs call it the semantic layer.
What we built
Claude Code put 8 numbers of our weekly report into the catalog as 9 metrics in two working sessions. On 12 October it read PostHog's docs and proposed the first metric; on 20 October it built the rest. 7 are SQL queries over PostHog's own events and over our Stripe data synced into PostHog. The other 2 have their data only in our app database, so they are step definitions that tell the agent where the number lives. Claude Code also created one saved view with the internal-accounts list, proposed trust marks on that view and on three Stripe tables, and proposed three table joins with measured match rates. Then it wrote the rules for the agent and 12 golden questions in the team's words.
For three recent weeks we compared each of the 7 queries with the report:
| Metric | Against the report |
|---|---|
| New seats | exact |
| New connections, CRM integration 1 | exact |
| New connections, CRM integration 2 | exact |
| People subscribing Zapier triggers | exact |
| Website visitors | two people apart in two of the weeks |
| New paying customers | one extra account in one week |
| New Zapier connections | different on purpose: it counts accounts connecting for the first time, the report's column counts people with any trigger subscription that week, repeats included |
The agent passed 10 of the 12 golden questions on the first run. After two changes our script scored 12 of 12, but one of those answers still quoted a sheet the agent could not open (step 7).
Nothing is approved yet. All 9 metrics, the 4 trust marks and the 3 joins wait for the owner, and the agent still runs on the owner's key. Steps 2 and 8 describe the part we have not done.
How an agent answers once the catalog is on
- The agent searches the catalog for a metric that matches the question.
- If exactly one approved metric matches, the agent runs it and says the number comes from the catalog.
- If several approved metrics match, the agent asks which one you mean.
- If none matches, the agent computes the number itself and says it is not from the catalog.
A proposed metric runs too. Our agent opened such answers with "Proposed definition in your data catalog", the line the PostHog MCP server gives it.
What you get
Copy the posthog-metric-catalog-starter template repository, replace the template's example company, Brightdesk, with your numbers, and follow the steps below. Every number in the template's evals/ files is Brightdesk's, so replace those too.
posthog-metric-catalog-starter/
├── catalog.json the list: views, metrics, trust marks, joins
├── views/catalog_accounts.sql the internal-accounts list
├── metrics/*.sql metrics PostHog computes
├── metrics/*.md metrics whose data lives elsewhere
├── agent/NUMBERS.md rules for the agent
├── evals/golden-questions.json the golden questions
├── evals/report-weeks.csv the report's numbers to compare against
├── checks/ the lists behind a number
├── scripts/ list sources, push, compare, check a key, test, weekly queue
└── docs/checklist.md the launch checklist
Before you start
- Node 20 or newer and a terminal. The scripts have no dependencies, so there is nothing to install.
- A PostHog project that already holds the data your report reads: product events, and outside data such as Stripe synced into PostHog.
- A
.envfile, filled in before step 1, because the scripts that call PostHog read it: copy.env.exampleto.env. The key is a personal API key from PostHog settings under Personal API keys, not the project key. Make it with the builder scopes from step 2 (data_catalog:read,data_catalog:write,query:read,warehouse_view:write, plusinsight:readif you will make metrics from saved insights) and withoutdata_catalog_approval:write. A key keeps the scopes it was made with.POSTHOG_HOSTishttps://us.posthog.comorhttps://eu.posthog.com;POSTHOG_PROJECT_IDis the number in your project URL,/project/<id>/. Runnode scripts/check-key.mjs builderto confirm the key. - Claude Code for the golden questions:
npm install -g @anthropic-ai/claude-code, then runclaudeonce and log in.
A filled .env looks like this:
POSTHOG_API_KEY=phx_...
POSTHOG_HOST=https://us.posthog.com
POSTHOG_PROJECT_ID=12345
Step 1: List your report's numbers and where each one lives
Start with the report your team already reads, not a list of every chart. For each number, write one line that says what it counts and where its data sits today. PostHog tells agents to propose a definition only when someone has asked for it or the number has been reused at least twice. A number in your weekly report is reused every week.
Find your events and tables
Run node scripts/list-sources.mjs to see what PostHog already holds. It prints every event of the last 30 days with its count, then every synced table and saved view. For a company like the template's Brightdesk, the output looks like this (shortened, made-up numbers):
$ node scripts/list-sources.mjs
Events in the last 30 days: 38
912406 $pageview
388150 $autocapture
52117 message_sent
1904 signed_up
596 mailbox_connected
64 hubspot_connected
38 zap_turned_on
...
Synced tables and saved views: 15
data_warehouse stripe.customer
data_warehouse stripe.invoice
data_warehouse stripe.subscription
...
Write the event or table next to each number on your list. A number with nothing to write next to it lives outside PostHog. Each synced table appears twice, as stripe.invoice and stripe_invoice; the second name is an alias of the first.
Our list
We took every column of the report whose data is already in PostHog, plus the two numbers outside PostHog that people ask about most. The Zapier column became two metrics (see below), so 8 report numbers gave 9 metrics.
| Number | Where the data is | In the catalog as |
|---|---|---|
| New paying customers | Stripe, synced into PostHog | a query |
| New seats | Stripe, synced into PostHog | a query |
| Website visitors | PostHog web analytics events | a query |
| New connections of two CRM integrations | PostHog events | a query each |
| Zapier connections | PostHog events | two queries (see below) |
| Signups | our app database only | steps that say where the number lives |
| First connections of the product's main channel | our app database only | steps that say where the number lives |
Before writing a metric, have Claude Code read the report's scripts and its method file, then copy the report's rule into the metric instead of inventing one.
One word, three rules
That reading found the three rules for "connected" from the start of this guide. Only one CRM column followed the method file's rule: an account counts once, in the week of its first-ever connection. The other CRM column removed staff by their email addresses. The Zapier column removed nothing and showed two to three times the number of new connections.
PostHog advises creating two metrics with honest names when one word has two meanings, rather than combining both meanings in one disputed metric. Its example is activation_signup and activation_first_insight. Our catalog has zapier_new_connections for accounts connecting for the first time. It has zapier_trigger_subscribers for people subscribing to triggers, including repeats. The report owner decides which one the report should show. The catalog makes the difference visible to every agent.
Step 2: Keys: who proposes, who approves
The agent reaches the catalog through the PostHog MCP server at https://mcp.posthog.com/mcp, with the personal API key sent as Authorization: Bearer <key>. The template's .mcp.json sends Bearer ${POSTHOG_API_KEY}, and Claude Code fills that in from the terminal's environment, not from .env. PostHog's wizard, npx @posthog/wizard mcp add, connects through your PostHog login instead; we use keys because a key's scopes are listed and can be checked, so the agent's key can leave out approval.
To ask the agent yourself, set the key in the terminal first (export POSTHOG_API_KEY=...; in PowerShell, $env:POSTHOG_API_KEY = "...") and run claude in the template folder. The scripts read .env, but a key set in the terminal wins over it, and run-golden.mjs hands Claude Code whichever key won. That is how you check the owner's key without editing .env: POSTHOG_API_KEY=phx_... node scripts/check-key.mjs owner (in PowerShell, set $env:POSTHOG_API_KEY first). It also means a terminal still holding the owner's key, for example after the approval conversation in step 8, runs every script on that key, run-golden.mjs and push-catalog.mjs included. Set the builder key back or open a new terminal before you run them.
Three roles
Give each role only the scopes it needs.
| Role | Scopes | What it can do |
|---|---|---|
| Agent | data_catalog:read, data_catalog:write, query:read; insight:read if a metric is made from a saved insight | The agent can search, run and propose metrics. The agent can propose trust marks and joins. |
| Builder (the push script) | The agent's scopes plus warehouse_view:write | The builder can also create the catalog_accounts saved view that holds the internal-accounts list. |
| Owner | data_catalog:read, data_catalog_approval:write; query:read and warehouse_view:write to accept a join | The owner can approve metrics, certify or deprecate tables, and accept or reject joins. |
The agent role fits an agent that answers questions and proposes metrics. When the agent also runs the push script, as in this guide, give it one builder key. PostHog advises that the agent's key should never have the approval scope; your own key carries it, so approvals go through you.
Check the key first
Run scripts/check-key.mjs before deciding that a key cannot perform an action. The script asks PostHog what the key can do and reports whether it fits the requested role. For a key made for the agent role it prints this (example label and project):
$ node scripts/check-key.mjs agent
label: catalog-agent
projects: 12345
scopes: data_catalog:read, data_catalog:write, query:read, insight:read
Fits the agent role.
For a key that can approve, the same command prints too much: data_catalog_approval:write (this key can approve; the agent key should not) and ends with Does not fit the agent role.
We have not split the keys yet
Our whole build ran on one key: the owner's, with every scope. check-key.mjs agent rejects that key for the reason above. The second key, for the agent and without the approval scope, does not exist yet. Until it does, the rule "the agent never approves" rests on the agent's instructions (step 7), not on the key.
What went wrong for us
- Claude Code first told the owner that it had no write access. It had looked only at Claude's built-in PostHog connection, which logs in as the user; when we checked it on 12 October, that connection had no write scope. The owner keeps write keys in the projects'
.envfiles, and Claude Code had not asked PostHog what they could do. - Those keys, made in February, could write dashboards and insights but had no catalog scopes. A key keeps the scopes chosen when it was made; new scopes do not appear on an old key by themselves. The owner made a new key for the catalog.
- In Claude Code and Codex, the PostHog MCP server shows one tool,
exec, and the agent reaches every other tool through it. A missingdata-catalog-*name in the agent's tool list does not mean a missing scope.
Step 3: One internal-accounts list
Every number must leave out your own QA bots, test accounts and staff accounts. Our weekly report carried that list inside four scripts. Its method file warns that a new test account leaks into the numbers until someone adds it to all four. Writing the list into every catalog metric would create a fifth copy, and the catalog and report could drift apart without anyone noticing.
The view
Keep the list in one saved view in PostHog, a saved query that other queries read like a table. scripts/push-catalog.mjs pushes the definition in views/catalog_accounts.sql as catalog_accounts. Every metric that counts customers or accounts joins it and keeps is_internal = 0; the visitors metric counts people on the website and does not need it. The view has one row for every account PostHog knows, whether it is an account group or an account named on a Stripe customer. For accounts that have a Stripe customer, the row includes the owner's email, and every row has an is_internal flag.
The template ties events and Stripe together through one account key that sits in two places: the account group with index 0 on events ($group_0, read from the groups table with index = 0) and account_id in the Stripe customer's metadata. Check both in your project before you write any metric. If your account key sits elsewhere, change those lines in views/catalog_accounts.sql and the same expression in every metric. Otherwise every join to catalog_accounts finds nothing: the event metrics read 0 every week and the Stripe metrics come back empty.
(
-- 1. account-name rules: QA bots, demo and synthetic test accounts
startsWith(a.account, 'qa-')
OR startsWith(a.account, 'test-')
OR endsWith(a.account, '-demo')
-- 2. known test accounts of team members on outside email domains
OR a.account IN ('acme-sandbox', 'brightdesk-internal')
-- 3. owner email rules (known only for accounts that have a Stripe customer)
OR coalesce(o.internal_owner, 0) = 1
) AS is_internal
Have Claude Code build the list from the filters your report scripts already carry. Claude Code copies every rule onto its own line with a comment that says why, so a new test account takes one new line in this file.
Create the view in PostHog now, before you compare any metric. The metric queries and the name checks read catalog_accounts from PostHog, so until it exists compare.mjs stops with an error on every metric that joins it. Do not run push-catalog.mjs --apply for this: it would also propose every metric in catalog.json before you have compared them. Ask Claude Code to create catalog_accounts from views/catalog_accounts.sql with the PostHog MCP server's view-create command, after checking that no view in your project already has that name (see the cautions below). We did the same: our view went into PostHog before the first push of the catalog.
Two cautions
The PostHog MCP server's command for creating a view, view-create, overwrites an existing view with the same name. Our project already had six saved views, five of them made by other people, so give your view a name nobody uses. scripts/push-catalog.mjs refuses to touch a view another person created.
A metric of accounts that skips the internal-accounts list must say so in its SQL. In our own catalog, zapier_trigger_subscribers copies the report's Zapier rule, which keeps internal accounts, so its SQL says No internal filter on purpose. scripts/selftest.mjs fails such a metric when it has neither the filter nor that line, and it caught ours the first time it ran. The template's version of this metric keeps the filter.
Step 4: Write each metric
PostHog accepts two kinds of metric definition. You can write a query that PostHog runs with the same result every time, or numbered steps in plain text that the agent follows. Prefer a query; we wrote 7 queries and 2 step definitions.
Claude Code wrote all of our metric files from the report's rules; give it the six rules below. selftest.mjs and push-catalog.mjs see a metric only once catalog.json lists it, so for every file add an entry with its name, description and definition (kind and file; the format is in step 6). Delete the entries of Brightdesk's metrics you do not use, or push-catalog.mjs --apply proposes them in your project. Then run node scripts/selftest.mjs. It works offline and checks each name, that each description has at most 300 characters and names its period (week, month), that each definition file exists, the Monday week and the internal-accounts filter. Until step 7 replaces the golden questions, it also flags every question that still expects a deleted Brightdesk metric.
Six rules for every query
- Set every week from Monday 00:00 UTC to the next Monday, and write it in the SQL as
toStartOfWeek(toTimeZone(timestamp, 'UTC'), 1). Our PostHog project started weeks on Sunday because nobody had set it, and a Trends query follows the project's week. Changing the setting would have changed everybody's charts, so we put the week in the SQL. - Use the last 52 complete weeks as the window. A SQL metric keeps its window in the SQL, and PostHog rejects the agent's date overrides. We started with 12 weeks and widened the window after reading PostHog's description of the run command: with 12 weeks, a question about a week in spring would have had no answer.
- For counts that can be zero, such as new connections, return a row for every week. Our first version omitted a week without a new connection. An agent reads a missing row as missing data, not as zero.
- Count a connection once, in the week of the account's first-ever connection event. A reconnect sends the event again, so counting every event would overstate new connections.
- A synced table holds today's state, so read seats at purchase from the subscription's first invoice (
billing_reason = 'subscription_create'). Our first seat metric read the seat count on the Stripe subscription, which already included later upgrades and counted a few seats a week too many. It also counted one seat from a partner subscription with no account id; the report leaves such seats out, and now the metric does too. - For counts over a year of events, use
uniqinstead ofcount(DISTINCT). Our visitors metric usedcount(DISTINCT)over 52 weeks of pageviews and hit PostHog's query time limit when the catalog ran it. Theuniqversion returned the same weekly numbers in 8 seconds.
The template's metrics/hubspot_new_connections.sql shows the shape of a weekly metric.
SELECT
w.week_start AS week_start,
coalesce(c.hubspot_new_connections, 0) AS hubspot_new_connections
FROM (
SELECT toStartOfWeek(toTimeZone(now(), 'UTC'), 1) - toIntervalWeek(number + 1) AS week_start
FROM numbers(52)
) AS w
LEFT JOIN (
SELECT
toStartOfWeek(toTimeZone(x.first_connected, 'UTC'), 1) AS week_start,
count() AS hubspot_new_connections
FROM (
SELECT lower($group_0) AS account, min(timestamp) AS first_connected
FROM events
WHERE event = 'hubspot_connected'
GROUP BY account
) AS x
JOIN catalog_accounts AS ca ON ca.account = x.account
WHERE ca.is_internal = 0
AND x.first_connected >= toStartOfWeek(toTimeZone(now(), 'UTC'), 1) - INTERVAL 52 WEEK
AND x.first_connected < toStartOfWeek(toTimeZone(now(), 'UTC'), 1)
GROUP BY week_start
) AS c ON c.week_start = w.week_start
ORDER BY week_start
When the data is not in PostHog
Our signups and the first connections of our product's main channel live only in our app database, which is not synced into PostHog. We tried the nearest PostHog events and applied the same internal-account exclusions, but the signup event ran a few percent below the report and the connection event about a sixth below. We therefore wrote both as step definitions that say where the number lives and what to say when the week has no row yet. The signups steps let the agent count the signup event only when the person accepts an estimate, and the result must be labelled "estimate from PostHog events". The connections steps forbid the event even as an estimate, because it runs well below the report. The template's metrics/mailbox_connections.md shows the format.
This number is computed outside PostHog, from the app database. Do not count it from PostHog events.
1. Find the week the question is about. Weeks run from Monday 00:00 UTC to the next Monday 00:00 UTC.
2. Take the number from the weekly KPI report: sheet "Weekly report", column "Mailboxes connected", the row for that week. The report counts an account once, in the week of its first-ever mailbox connection. Reconnects do not count.
3. If the week has no row yet, say so and give no number.
4. Do not use the PostHog event `mailbox_connected` for this number, not even as an estimate. It does not fire for every way of connecting and runs well below the report.
Step 5: Compare with the report before anyone approves
Before a metric goes to PostHog, compare it with the report for three recent weeks. Check the totals first, then check the names behind every count of customers or accounts.
Totals
Put three recent weeks of the report into evals/report-weeks.csv under the header week_start,metric,value, one line per week and metric, for example 2026-09-28,new_seats,96. week_start is the Monday that starts the week. metric is the metric's file name without .sql; if no line matches it, the script stops with no rows for new_seats in evals/report-weeks.csv. Run scripts/compare.mjs totals to print the difference week by week. Give it a file to test a draft, or a metric's name to test what is stored in PostHog. A run that matches prints this (made-up numbers from the template's CSV):
$ node scripts/compare.mjs totals metrics/new_seats.sql
week_start catalog report difference
2026-09-28 96 96 same
2026-10-05 88 88 same
2026-10-12 101 101 same
All weeks match.
Names
Run compare.mjs names to list the accounts behind the number for one week and set them against the report's own list for that week. The template ships one such query, checks/new_paying_customers_names.sql. For every other count of customers or accounts, write a new file in checks/ with the same filters as the metric. It must return two columns: the week start, named week_start, and the account name. Save the report's list for the same week as a text file with one account name per line. Check the names even when the totals match, because two errors can cancel each other out.
That happened to us. Our first version of new paying customers counted each account login once. Two logins on the report's lists had been registered again by new users, so the rule dropped two accounts the report counts, one in each of two weeks. In one of those weeks the total still matched the report exactly, because the same week also had one account the report did not list. The missing account and the extra one cancelled each other out. For such a week the name check prints this (made-up names and numbers):
$ node scripts/compare.mjs names checks/new_paying_customers_names.sql 2026-09-28 report-list.txt
catalog 39, team list 39
only in the catalog: harbor-tiles
only in the team list: northfield-co
We changed the rule to one Stripe customer, one account. Both dropped accounts came back. The extra account stayed, and the next section explains it.
What we found
Of our 7 computed metrics, 4 matched the report exactly in all three weeks. The new Zapier connections metric counts differently from the report on purpose (step 1). The other 2 differed slightly. Website visitors differed by two people in two of the weeks. The report takes that number from PostHog's Web analytics page, which puts a visitor in the week when the session started. The metric instead counts the visitor by the time of the pageview. New paying customers had one extra account in one week. Its first paid invoice fell in that week after months of unpaid ones, so by the report's written rule (first-ever paid invoice) it is new. Why the report leaves it out we could not see without the app database, so the report's owner decides. Write down every difference and what you know about it before the owner approves the metric. Keep these causes in the build log rather than the catalog; step 7 explains why.
Check the comparison query as well as its result. PostHog's query API returns at most 100 rows unless the query sets its own LIMIT. Our first name check printed catalog 0 because the list of names ran past 100 rows and the requested week had been cut off. The script now filters the week inside the query.
Step 6: Push to PostHog as proposed
catalog.json lists the view and metrics that go into the catalog. It also lists the trust marks and joins.
{
"name": "new_seats",
"display_name": "New seats (weekly)",
"unit": "seats",
"description": "Seats bought on each account's first subscription, counted in the week the subscription was created, Monday to Monday, UTC. Later upgrades and subscriptions that started as a free trial are not included. Test and staff accounts are excluded.",
"definition": { "kind": "HogQLQuery", "file": "metrics/new_seats.sql" },
"confidence": 0.9,
"reasoning": "The synced Stripe subscription row holds today's quantity, after upgrades, so seats come from the subscription's first invoice (billing_reason = subscription_create). Matches the weekly report for the weeks checked."
}
name works as an address in the API: keep it lowercase, without version numbers, and never rename it. Renaming or deleting a metric frees its name and breaks every saved reference to it. The description is what the agent matches against a question, so follow PostHog's instruction to write it "the way a teammate would ask the question." Say what is counted, for which period and what is left out, in under 300 characters. The query does not go into the description.
The push
Run node scripts/push-catalog.mjs to print the plan. Add --apply to write through the PostHog MCP server in the same way an agent does. Everything lands as "proposed". Run the script again; when no view or metric line says new or changed, PostHog holds exactly what the files say for them. Trust marks and joins go out only once: if PostHog already has one for that table or pair, the line says exists, and an edited note in catalog.json is not sent. A first dry run of the template, before its joins are measured, prints lines like these (shortened; there is no real Brightdesk project behind them):
$ node scripts/push-catalog.mjs
view catalog_accounts: new
metric new_paying_customers: new
metric new_seats: new
mark stripe.invoice: new, certified
...
join stripe.invoice -> stripe.customer: skipped: measure the match rate first (scripts/match-rate.mjs)
...
Dry run: nothing was written. Run with --apply to push.
In PostHog, a new display name or owner keeps a metric approved. scripts/push-catalog.mjs does not: it writes with data-catalog-metric-create, which always lands a metric as "proposed". So if you change an approved metric's description, display name, unit, reasoning or definition file, the plan prints changed (approved -> proposed) and the owner reviews it again. Watch line endings when you write metric files on Windows. Our files reached PostHog with Windows line endings inside the stored SQL. scripts/push-catalog.mjs now sends plain line endings and compares definitions character by character.
Trust marks
Put a trust mark on a warehouse table or view. Use certified when the agent should prefer the source and deprecated when the agent should avoid it. The agent reads the note beside the mark, so write a note it can act on. This is the template's mark on the Stripe subscriptions table:
{ "table_name": "stripe.subscription", "proposed_status": "certified",
"notes": "quantity is today's seat count, after any upgrade, and status is today's status. For seats at purchase read the subscription's first invoice in stripe.invoice." }
Each Stripe table appears twice in PostHog, such as stripe.invoice and stripe_invoice. The second is an alias of the first, so put the trust mark on the first.
Joins
Propose each join with measured evidence. scripts/match-rate.mjs measures how many rows in one table find their partner in the other and prints the evidence block. Paste it into the join in catalog.json (the row counts here are made up):
$ node scripts/match-rate.mjs stripe.invoice customer_id stripe.customer id
{ "source_table_name": "stripe.invoice", "source_table_key": "customer_id",
"joining_table_name": "stripe.customer", "joining_table_key": "id", "field_name": "customer",
"confidence": 0.95,
"reasoning": "Paid-customer metrics go from an invoice to its customer to find the account.",
"evidence": { "match_rate": 0.9991, "rows_checked": 48210, "rows_with_key": 48210, "checked_at": "2026-10-20" } }
In the template every evidence block says fill in, because only your own data can fill it. scripts/selftest.mjs lists those joins, and scripts/push-catalog.mjs skips each of them until the placeholder is replaced. The template's third join, from stripe.customer to catalog_accounts, keys on the account expression from step 3, so pass that expression in quotes: node scripts/match-rate.mjs stripe.customer "lower(JSONExtractString(metadata, 'account_id'))" catalog_accounts account. If your account key sits elsewhere, put your own expression here and in the join's source_table_key. Our three joins matched 99.9%, 99.7% and 100% of the rows that carry the key. Of our Stripe customers, 94% carry an account id, and the join can match only those. Check before rejecting a join, because rejection is permanent and PostHog never proposes that pair again.
Step 7: Rules for the agent and golden questions
agent/NUMBERS.md is the agent's standing instruction for any question about a number. CLAUDE.md and AGENTS.md point Claude Code and Codex to it when they start in the template folder. An agent that starts anywhere else needs the file in its own instructions; scripts/run-golden.mjs passes it to each Claude Code session with --append-system-prompt. The agent reaches every PostHog tool through exec, so the file names each call:
1. Run `call metric-list {}` and read the names and descriptions. Pick the metric whose description
answers the question. Read its full definition with `call metric-describe {"name": "<name>"}` when two
metrics look alike.
2. Run the metric with `call data-catalog-metric-run {"name": "<name>"}`. Do not write your own SQL for a
number the catalog has.
3. Answer from the run result. Weeks run Monday 00:00 UTC to the next Monday; `week_start` is the Monday.
Say which week or weeks the number covers.
4. Say how much the number can be trusted, in one line:
- status `approved` and `is_drifted` false: "approved definition";
- status `proposed`: "proposed definition, not yet approved by its owner";
- drifted: "the definition changed since approval; ask the owner before using it".
Add the `posthog_url` from the run so the person can open the exact query.
For a step definition, the agent follows the steps. When the catalog has no metric, the agent says so first, labels anything it computes "not from the catalog", and never guesses a definition for money numbers such as MRR. Six rules hold for every answer:
- Never approve, certify, deprecate, accept, reject or delete anything in the catalog. People do that.
- Never use the project's default week. It starts on Sunday; our weeks start on Monday.
- Never count customers or accounts without removing internal ones. The `catalog_accounts` view holds them (`is_internal = 1`).
- Never present an estimate or your own query as a catalog number.
- Never answer with a number you found in a description, reasoning or notes field. Only a run result is an answer.
- Never say you read a source you did not open in this conversation (a sheet, a report, a database).
In our golden-question run we asked "What was our MRR last week?", a number our catalog does not have. This is the agent's whole answer:
The catalog has no definition for MRR, so I can't give you a number. None of the 9 metrics covers recurring revenue. The closest are `new_seats` and `new_paying_customers`, and both count new activity in a week. Neither is a revenue total.
I'm not going to guess a definition for a money number. Someone has to say what counts as MRR: which plans, whether annual plans are divided by 12, how coupons, $0 invoices and trials are treated, and which billing system is the source. Once that's settled, I can write the query, compare it against the number the team already uses, and propose it as a new metric. It would land as `proposed` for the owner to approve.
Who owns the MRR definition?
Golden questions
Write 10 to 12 questions in your team's own words. PostHog calls these golden questions. Include at least one trap involving two metrics with similar names. Also include one number whose data sits outside PostHog and one number the catalog omits on purpose. In our list, the missing number is MRR. The template's evals/golden-questions.json has 12 questions with Brightdesk's numbers; replace every number with your own.
Give each question the number the team already trusts for that week, taken from the report rather than the catalog. The exception is a difference that step 5 explained and the owner accepted: there, use the metric's number and keep the report's number and the cause beside it.
{ "id": "q04", "question": "How many new Zapier connections did we get in the week of 12 October 2026?",
"expect_numbers": ["3"], "expect_metric": "zapier_new_connections",
"trap": "zapier_trigger_subscribers counts people, repeats included, and gives a bigger number" },
{ "id": "q03", "question": "How many people visited our website in the week starting 28 September 2026?",
"expect_numbers": ["12,478", "12478"], "expect_metric": "website_visitors",
"report": "12480",
"why": "The report takes visitors from the Web analytics page, which puts a visitor in the week the session started; the metric uses the pageview time. Owner accepted the metric's number." }
Two of our 12 questions use the metric's number this way: website visitors, two people apart, and the three-week total of new paying customers, one account apart. Our owner has not ruled on either difference yet. With the report's numbers both questions would fail, and the failure would say nothing about the agent: step 5 already tested the metric.
A question about a number outside PostHog, or about the number the catalog leaves out, has no number to expect. It passes when the agent ran the metric named in expect_metric (none for MRR), the answer contains one of the phrases in expect_text_any, and no number from forbid_numbers appears. Put that week's count of the PostHog event into forbid_numbers.
{ "id": "q09", "question": "How many accounts connected a mailbox in the week of 5 October 2026?",
"expect_numbers": [], "expect_metric": "mailbox_connections",
"expect_text_any": ["report", "app database", "outside PostHog"],
"forbid_numbers": [] },
{ "id": "q10", "question": "What was our MRR last week?",
"expect_numbers": [], "expect_metric": null,
"expect_text_any": ["no definition", "not in the catalog", "no metric", "doesn't have", "does not have", "no governed", "no catalog"] }
Run node scripts/run-golden.mjs. The script asks each question in a fresh headless Claude Code session that sees only the PostHog MCP server and the rules. It checks that the answer contains the expected number, that the agent ran the expected metric through the catalog, and that no forbidden number appears. Our full Sonnet run asked 12 questions and cost $3.65. Each question took about 40 seconds with four sessions running at once. Before approval, every answer that uses a metric says "proposed definition"; the script does not check that, so read the answers. Run the questions again after the owner approves.
What the first run found
The first run passed 10 of 12 questions. Both failures came from our catalog. In its first version, the reasoning field of both step definitions carried the report's numbers from our comparison. For the signups question, the agent quoted one of them as "an unverified figure" from the metric's note. For the second question, the agent gave the number and claimed it had read it from the report's sheet, although the agent could not open that sheet.
The fix required two changes. We removed numbers from the description and reasoning fields, while keeping the evidence in the build log. scripts/selftest.mjs now checks those fields. We also added the last two "Never" rules above. A rerun of the two failed questions passed, and the full rerun scored 12 of 12.
The script was wrong about one answer in that full run. On the second of those two questions the agent again gave the report's number and wrote that it had opened the report's sheet, although its only tool was the PostHog MCP server. The question forbade only the PostHog event's number, and the answer mentioned the report, so the script passed it. When a number lives outside PostHog, put the report's own number into forbid_numbers too: the agent cannot have read it.
PostHog's launch post defines success as agents correctly answering at least 80% of a fixed set of golden questions.
Step 8: Approve, then look once a week
We have not done this step yet. Our 9 metrics, 4 trust marks and 3 joins still wait for the owner, and the agent's separate key from step 2 does not exist yet. What follows comes from PostHog's docs and from our scripts, not from an approval we made.
Each number's owner approves its metric. Approving in a conversation with an agent needs a key with the approval scope from step 2; once the agent has its own key without that scope, it cannot approve on its own. We have not opened the approval screen on the Data Catalog page yet, so we cannot say what it asks for.
Two ways to approve
- The owner opens the Data Catalog page with Cmd or Ctrl+K, then selects "Data Catalog". The owner reviews what agents proposed and approves metrics by hand. The owner also certifies tables and accepts joins there.
- The owner can approve in a conversation with the agent. Run that conversation on the owner's key. The template's
CLAUDE.mdandAGENTS.mdtell the agent never to approve, so do not hold this conversation in the template folder. Make an empty folder, copy the template's.mcp.jsoninto it, set the owner's key in the terminal as in step 2 and runclaudethere. The agent prepares the approval, then PostHog returns a signed hash and a message. The owner types the word "confirm" to continue. The second call checks the signed hash and the typed word before it completes the approval.
A metric made from a saved insight drifts when someone edits that insight, and PostHog marks its results as not canonical until a person takes a new snapshot; metrics written in SQL, like ours, cannot drift.
Once a week
Once a week, run node scripts/review-queue.mjs. The script lists metrics waiting for approval, drifted metrics, trust marks and joins waiting for review, and approved metrics that nobody has run for 30 days. PostHog recommends the same routine for teams. Agents propose metrics as they work, a data owner reviews the queue once a week, and each review takes minutes.
Our run on 20 October listed all 9 metrics, 4 trust marks and 3 joins as waiting, and nothing drifted or unused. A later week, after a first round of approvals, would look like this (an example with the template's names):
$ node scripts/review-queue.mjs
Metrics waiting for approval: 2
new_seats (New seats (weekly)), owner ana@brightdesk.example, confidence 0.9
signups (Signups (weekly)), owner ana@brightdesk.example, confidence 0.8
Drifted metrics: the source insight changed, results are not canonical: 0
Trust marks waiting: 1
stripe.subscription -> certified
Joins waiting: 0
Approved but not run for 30 days: 1
hubspot_new_connections, last run 2026-08-31
Review and approve here: https://us.posthog.com/project/12345/data-catalog
When the weekly report changes because of a new test account or a new rule, change the file, compare the metric's result with the report, and push the changed metric. The metric returns to the "proposed" state, so the owner sees the change in the next weekly review. Run the golden questions again after every round of approvals.
What the catalog does not do
- The catalog does not check the data. A trust mark records that a person vouches for a table; it does not test the rows behind it.
- The catalog does not compute numbers whose data is outside PostHog. A step definition sends the agent to those numbers, but the agent still cannot open your spreadsheet unless you connect it.
- A metric written in SQL keeps its window in the SQL. The agent cannot reshape that window. Typed queries, such as a trend or a funnel, accept other dates at run time, but they follow the project's week.
Plan and price
PostHog's docs and its pricing page do not tie the catalog to a plan, and the pricing page has no line for it. PostHog has two plans, Free and pay-as-you-go. We built ours on the pay-as-you-go plan and have not tried it on Free.
What you pay for is the synced data. The pricing page lists it under "Sources" and bills it per synced row. The first 1,000,000 rows each month are free. After that, the price is $0.000015 a row and falls with volume. Historical rows from a new source are free for its first 7 days.
Without PostHog
Use the dbt Semantic Layer if you have your own data analysts. It keeps definitions in a dbt project and has its own MCP server. Cube has an open-source core and gives agents access over SQL, REST, GraphQL and an MCP server. You can also skip a service and keep the same definitions as plain files that the agent reads before any number question. The template works this way too. Its agent/NUMBERS.md and metrics/ files read fine without PostHog, but nothing then enforces approval.
Launch checklist
The template's docs/checklist.md has every step with who does it. The launch is done when all of these hold:
- Every number of the report has a line that names its event or table in PostHog, or the system outside PostHog that holds it.
check-key.mjs builderpasses on the key the agent and the scripts use, andcheck-key.mjs ownerpasses on the owner's key.- Every exclusion from the report's scripts is in
catalog_accounts, and no metric keeps its own copy. selftest.mjspasses and lists no join with placeholder evidence.- Totals match the report for three weeks and names match for one week, or each difference has a written cause.
- A second run of
push-catalog.mjshas no line that saysneworchanged. - At least ten golden questions, with one trap, one number from outside PostHog and one the catalog does not have. No example number from the template is left.
review-queue.mjsshows no metric waiting, and at least 80% of the golden questions pass after approval.review-queue.mjsruns on a fixed day every week, and a named person reads it.