Why AI Gets Your Metrics Wrong (and How Definitions Fix It)

AI can write a confident answer about revenue, users, or conversions and still be wrong, because it guessed what you meant. Here is where those guesses happen in databases, marketing tools, spreadsheets, and billing data, and the definition that fixes each one.

  • By
  • Published
  • Updated
  • 11 min read

Better Analyst publishes this blog and makes the analysis tool mentioned below. The failures and fixes apply to any AI tool you use with business data.

On this page
  1. Why AI gets metrics wrong
  2. The five fixes
  3. SQL databases
  4. GA4, Search Console, and ads
  5. Spreadsheets and CSVs
  6. Billing and CRM data
  7. A five-metric checklist
  8. How Better Analyst handles it
  9. FAQ

The short answer

AI usually gets business metrics wrong because it has to guess a definition: what “revenue” includes, what one row in a table represents, how two tables join, or which of two similar fields to use. The arithmetic can be perfect and the answer still wrong, because it answered a slightly different question.

The fix is to write the definition down where the AI will use it: a published definition, a stated grain, an approved relationship, a reviewed query, or a clarifying question before it runs. In Better Analyst, SQL databases can hold these as published Data Context definitions. For GA4, spreadsheets, and billing exports, state the definition in your question or your AI Agent’s instructions, then check the code shown with the answer.

See how Data Context works

Why does AI get business metrics wrong?

A general AI assistant sees your data the way a new hire does on day one: column names, a few sample rows, and whatever you typed. It does not know that your finance team reports revenue net of refunds, that amount is stored in cents, or that one of your two customer tables still contains test accounts. So it fills the gaps with reasonable assumptions.

Reasonable assumptions are the problem. A wild error gets caught. A number that is 8% too high because refunds were included looks plausible, reads well, and ends up in a client report. That is why these mistakes are described as answers that sound right but are quietly wrong.

Four terms explain most of what follows:

  • Metric definition: one sentence that says what a number counts, what it excludes, and its unit. “Net revenue is paid invoice totals in USD, minus refunds, excluding test accounts.”
  • Grain: what one row represents, such as one order, one order line, or one session.
  • Relationship: how two tables join, and how many rows on each side match (one-to-one, one-to-many).
  • Fan-out: what happens when a join repeats rows from one table, so sums come out too high.

What fixes a wrong metric?

Nearly every failure below maps to one of five fixes:

  1. Published definition. Write the metric once and make it the default. Everyone who asks about “net revenue” gets the same calculation.
  2. Dataset grain. Record what one row represents, so counts and averages start from the right unit.
  3. Approved relationship. Record which fields join two tables and the cardinality, so joins do not double-count.
  4. Reviewed query. When a query gets a tricky question right, keep it as an example for the next similar question.
  5. Clarifying question. When a request could mean two things, the AI should ask, or at least say which reading it used.

How you apply them depends on where the data lives. A SQL database can hold definitions next to the tables. A GA4 property or a CSV usually cannot, so the definition goes in the question.

Where do SQL database answers go wrong?

Amounts in cents in one table and dollars in another

What goes wrong: A payments.amount column stores cents while invoices.total stores dollars. A revenue question that touches both is off by a factor of 100 on one side, and nothing in the column names says so.

The fix (published definition): Describe each amount field’s unit, then publish a Revenue metric that converts cents to dollars once. Every later question uses that calculation.

Two customers tables, only one without test accounts

What goes wrong: Your app writes to customers, and a cleaned crm_customers table excludes internal and test accounts. Ask “how many customers do we have?” and the AI may count the larger, wrong table.

The fix (published definition): Describe which table is the source of truth for customers. If the raw table should never be used, exclude it with data access rules so it is not an option at all.

A join that double-counts revenue

What goes wrong: To show revenue by product, the query joins orders to order_items and then sums orders.total. An order with three items now counts its revenue three times. This is a fan-out.

The fix (approved relationship): Record that many order items join to one order (order_items to orders is many-to-one) and that order_items has one row per order line. With the grain and relationship known, revenue is summed at the right level.

“Active user” means different things to Product and Finance

What goes wrong: Product counts anyone who logged in during the last 30 days. Finance counts paying accounts with activity. Both teams ask the same question and get different numbers, and both are right by their own definition.

The fix (clarifying question): Publish two named metrics, such as Active Users (30-day login) and Active Paying Accounts, instead of one ambiguous one. When someone asks for “active users,” the AI should ask which one, or say which it used.

The fix today: on connected SQL databases, Better Analyst’s Data Context holds these as published definitions: field descriptions and units, dataset grain, relationships with cardinality, and metrics. When a query gets a hard question right, endorse it with SQL Feedback and it is reused as an example for similar questions.

Where do GA4, Search Console, and ad platform numbers go wrong?

Sessions vs users

What goes wrong: “How many visitors did we get?” can mean sessions or users, and GA4 reports more than one user count, including active users, total users, and new users. One person on three visits is one user and three sessions.

The fix (state it in the question): Say which metric you mean, for example “active users by week,” and keep that wording in your recurring questions.

Conversions vs key events

What goes wrong: GA4 now calls its own conversion actions key events, while “conversions” refers to the actions counted for Google Ads. The two lists can differ, so “conversions last month” has two plausible answers.

The fix (state it in the question): Name the event, such as purchase or generate_lead, and say whether you want GA4 key events or Google Ads conversions.

Attribution windows that differ between GA4 and Ads

What goes wrong: Google Ads typically reports a conversion on the date of the ad click, while GA4 reports it on the date the event happened, and the two can use different attribution models and lookback windows. Comparing their totals for the same week rarely matches.

The fix (state it in the question): Pick one system as the source of truth for each question and say so: “use GA4 key events, data-driven attribution, by event date.”

Brand vs non-brand clicks in Search Console

What goes wrong: Total clicks mix people searching for your company name with people finding you for a topic. Growth in brand searches can hide a drop in non-brand traffic. Search Console also withholds some rare queries for privacy, so query-level rows do not add up to the site total.

The fix (state it in the question): List your brand terms and common misspellings, ask for brand and non-brand clicks separately, and ask the AI to report the unattributed remainder rather than drop it.

The fix today: Better Analyst connects to Google Analytics and Search Console, but these sources do not have Data Context definitions yet. Put your definitions in the question or in an AI Agent’s analysis instructions, and check the query and code shown with the answer. For ad platforms, the same rules apply to the reports you export.

Where do spreadsheet and CSV answers go wrong?

A totals row counted as data

What goes wrong: Many exports end with a Total or Grand Total row. Sum the column and the total is counted twice, so every figure is double what it should be.

The fix (state it in the question): Tell the AI to exclude total and subtotal rows, then check the row count in the code against the number of records you expect.

Mixed date formats in one column

What goes wrong: Is 03/04/2026 March 4 or April 3? Files merged from different systems often mix formats, and some dates are silently read the wrong way round or not at all.

The fix (state it in the question): State the format (“dates are DD/MM/YYYY”) and ask for the earliest and latest parsed dates as a quick sanity check.

Currency stored as text

What goes wrong: Values like “$1,200.00” or “(500)” are text, not numbers. Depending on how they are read, they are skipped, misread, or turned into blanks, and totals come out low.

The fix (state it in the question): Ask the AI to convert currency text to numbers, treat parentheses as negatives, and list any rows it could not convert.

The fix today: spreadsheets and uploaded files do not carry Data Context definitions. State the rules in your question or AI Agent instructions, and read the cleaning steps in the code before trusting the totals.

Where do billing and CRM answers go wrong?

MRR vs collected revenue

What goes wrong: In billing data such as Stripe, monthly recurring revenue (MRR) spreads each subscription evenly across months. Collected revenue is cash received. A customer who prepays a year adds a little to MRR every month but a large amount to collected revenue in one month.

The fix (published definition): Name which one you want. If your billing data is in a SQL warehouse, publish MRR and collected revenue as separate metrics.

Refunds and free trials

What goes wrong: Is a trial a customer? Is a refund subtracted in the month it happened or the month of the original charge? Different answers move churn, revenue, and customer counts.

The fix (published definition): Write the rule into the metric definition: “customers excludes active trials; refunds reduce revenue in the month refunded.”

One deal credited to two reps

What goes wrong: In CRM data such as HubSpot, a deal can change owners, or a report can join deals to an owner history table. Either way, one closed deal can be credited to two reps and counted twice in team totals.

The fix (approved relationship): Decide who gets credit, such as the owner at close, and make sure each deal counts once in team totals.

The fix today: if your billing or CRM data is synced into a SQL database or warehouse such as BigQuery, Snowflake, or PostgreSQL, publish these rules in Data Context. For exports, state them in your question or AI Agent instructions and check the code.

Checklist: pick five metrics from your last report

You do not need a data team to start. Take the last report you sent to a client or your leadership team, pick the five numbers people look at first, and work through these steps for each.

  1. Step 1Write the definition in one sentenceSay what it counts, what it excludes (refunds, test accounts, trials), and the unit.
  2. Step 2Name the source of truthOne table, report, or file per metric. If two exist, say which one wins.
  3. Step 3State the grainWhat does one row represent? One order, one order line, one session, one customer per day?
  4. Step 4Check the joinsIf the metric combines tables, note which side has many rows per key so nothing double-counts.
  5. Step 5Ask, then compareAsk AI for the metric and compare its number with your last report. Read the SQL or code before you trust a difference.
  6. Step 6Save what workedPublish the definition for SQL data, endorse the query that got it right, or add the definition to your AI Agent instructions for other sources.

How does Better Analyst handle metric definitions?

  • Data Context for SQL databases. Publish dataset grain and descriptions, field aliases and descriptions, relationships with cardinality, typed metrics, and workspace instructions. Owners and admins publish from a draft, and revision history lets you restore an earlier version.
  • Drafts from your schema and documents. The AI can draft definitions from your schema and an uploaded data dictionary, for you to review.
  • Shows its work. Every answer shows the SQL or code behind it, and which source each number came from when an answer combines sources.
  • Asks when it matters. When a request could mean more than one thing, it asks a clarifying question or states the assumption it made.
  • Data access rules. Data Controls limit the tables, columns, and rows the AI can query on a database connector, and definitions are withheld if those settings change after publishing.

Google Analytics, Search Console, and spreadsheets are connected and analyzed, but they do not have Data Context definitions yet. Comparing options? See Better Analyst vs ChatGPT and Claude for data analysis.

Start from your definitions, not a guess

Connect a database, publish the metrics your team already agrees on, and see the SQL behind every answer.

Try Better Analyst freeBook a demo

No credit card required. Free plan included.

Frequently asked questions

Why does AI give different answers for the same metric?
Usually because the definition was never written down. If “revenue” can mean gross or net, or “active user” can mean a login or a payment, the AI picks a reasonable reading, and two reasonable readings give two different numbers. Writing the definition once, where the AI will use it, removes the guess.
What is a semantic layer?
A semantic layer is a shared set of business definitions, such as metrics, what each table means, and approved joins, that sits between raw data and the people or tools asking questions. Better Analyst’s Data Context is a lightweight semantic layer for connected SQL databases, built for teams without a data team.
What is a fan-out join?
A fan-out happens when you join a table to another table that has several matching rows per key, so values from the first table repeat. Summing order revenue after joining orders to order lines or shipments is the classic example: each order’s revenue is counted once per match.
What does grain mean in a dataset?
Grain is what one row represents: one order, one order line, one session, or one customer per day. Counting rows or averaging values only gives the right answer when you know the grain.
Can I trust AI analysis for business reporting?
Treat it as a strong starting point you can check. Use a tool that shows the SQL or code behind each number, write down definitions for the metrics that matter, and compare key numbers with a report you already trust before you share them.
Do I need a data team to define metrics for AI?
No. Start with the five metrics you report most often and write one sentence for each. In Better Analyst, workspace owners and admins can publish definitions for SQL data in Data Context without writing modeling code, and the AI can draft field descriptions from your schema and uploaded documents for you to review.