Measuring an AI Feature: Quality, Cost, Trust
A successful AI feature is not only about how many people use it. This article proposes measuring three layers, quality, cost and trust, with a one-page board of nine metrics, a cost-per-successful-outcome formula that counts reviewer effort, a way to read acceptance rate without being fooled, and the usual traps.
For product people, team leads and programmers bringing AI into a product. Every technical term gets a short explanation.
The three-layer split and the metrics come from practitioners' guides (blogs and tool documentation), which are secondary sources. All the thresholds and figures in this article are illustrations, not standards. The shop "Hong" example is rebuilt.
Quick summary#
- Usage counts are not success. Measure the outcome of the work.
- Measure three layers at once: quality, cost, trust. A single layer creates blind spots.
- The right cost is the cost per successful outcome, reviewer effort included.
- A high acceptance rate can mean over-trust. Set it beside the pass rate from evals.
- Use a one-page board and decide in advance what happens when a metric worsens.
Hard words, plain meaning#
| Term | Plain meaning |
|---|---|
| Evals | Tests measuring the quality of an AI feature (see the dedicated post). |
| Acceptance rate | The share of AI output users take and use as is. |
| Edit rate | The share of output users change before using. |
| Override rate | The share of output users discard and do themselves. |
| p95 latency | The time under which 95% of requests finish. It shows the slowest cases, which users actually remember. |
| Token | The unit of text a model processes and bills by. |
| Cost per successful outcome | Total cost divided by the number of times the AI really helped. |
1. Why not measure one number#
It is easy to start with what is easy to count: number of calls, number of users. But an AI feature can be used heavily and still be bad: answering wrongly, costing a lot, or being quietly ignored by users.
Practitioners' guides (which I read via summaries) agree on two points: counting calls is shallow, what matters is work completed; and metrics should be grouped in layers, because focusing on one creates blind spots. A common example: a model that is 95% right is still useless if users do not trust it.
So I split three layers:
| Layer | Question | Data source |
|---|---|---|
| Quality | Is the output right and useful? | Error analysis, evals |
| Cost | What does each successful outcome cost, and is it fast? | Invoices, logs, timing |
| Trust | Do users trust and use it? | User behavior, feedback |
This post connects to Writing evals for an AI feature (the quality layer) and Do not measure productivity by lines of code (why activity metrics mislead).
2. The quality layer#
Quality is not a single "accuracy" number. Draw it from error analysis: group failures into categories and count them, as in the post on evals.
Suggested metrics:
- Pass rate per failure category. For example "misread the intent", "invented information", "tone too stiff". One number per category.
- Count of severe failures. An invented price is very different from a tone slip. Count failures with real consequences separately.
- Overall pass rate, but not as the only number.
Report an "all pass" figure, then drill down into each category. The evals guide also advises grading pass or fail rather than on a scale.
How to track: rerun the evals whenever you change the model, the prompt or the data. If you change the model without rerunning, you do not know how quality moved.
3. The cost layer#
3.1. Cost per call and per successful outcome#
Cost per call is the easy figure. But a feature may call the model cheaply while needing lots of human fixing, or need many calls to get one usable result. So the cost worth comparing is:
Cost per successful outcome = (model cost + infrastructure + effort of people who review or fix) ÷ number of successful outcomes
An illustration (rebuilt numbers): a customer assistant for the shop "Hong", one month.
| Item | Illustrative value |
|---|---|
| Customer questions | 10,000 |
| Model and infrastructure cost | 20 million dong |
| Answers usable as is | 6,500 |
| Answers needing staff fixes (3 minutes each, 60 thousand dong an hour) | 2,500 answers, that is 7,500 minutes, 125 hours, 7.5 million dong |
| Cases handed over entirely to staff | 1,000 |
| Total cost | 20 + 7.5 = 27.5 million dong |
| Successful outcomes (used as is or quickly fixed) | 6,500 + 2,500 = 9,000 |
| Cost per successful outcome | about 3,060 dong |
Looking only at model cost, 20 million over 10,000 calls is 2,000 dong a call, and you would miss the staff effort and the 1,000 failed cases. That is why "cost per successful outcome" reveals more.
3.2. Latency#
Measure latency at p95 (and p99), not just the average. The average hides the slowest cases, which is when users leave. For a chat feature, time to the first word strongly shapes the feeling of speed.
4. The trust layer#
This is the layer most often skipped. Look at user behavior:
| Metric | Meaning |
|---|---|
| Acceptance rate | Users take and use it as is |
| Edit rate | Users change it, then use it |
| Override rate | Users discard it and do it themselves |
| Error reports | Users actively say "this is wrong" |
| Cases handed to a person, and how they are handled | Whether the right cases get handed over |
| Reopen or complaint rate | Work that was "done" but came back |
Watch the trend, not only the absolute level. Some guides note that a sudden jump in the edit rate signals a quality drop, not personalization.
Context: the Stack Overflow 2025 survey (more than 49,000 respondents) found 46% of users do not trust AI's accuracy and only 3% trust it highly. Your product's users may feel similarly. Trust has to be earned, and has to be measured.
5. The trap: a high acceptance rate can be over-trust#
This is the point I find most important in this article. If users accept 90% of outputs but evals show only 75% are actually right, the 15-point gap is users over-trusting, not a good feature.
How to read it: set the acceptance rate beside the pass rate from evals.
| Situation | May mean |
|---|---|
| High acceptance, high pass | A good feature, trusted appropriately |
| High acceptance, low pass | Over-trust, high risk |
| Low acceptance, high pass | Users do not yet trust it, possibly due to interface or habit |
| Low acceptance, low pass | The feature falls short, so low use is right |
I have no data on what these rates typically are. This is a way of reading the metrics, not a measurement result.
6. A one-page board#
One board of nine metrics, three layers, recorded weekly.
| Layer | Metric | Warn when |
|---|---|---|
| Quality | Pass rate per failure category | A category drops, or a new severe failure appears |
| Quality | Count of severe failures | Any failure with real consequences |
| Cost | Cost per successful outcome | Rises as usage rises |
| Cost | p95 latency | Gets longer, or the first word is slow |
| Cost | Reviewer or fixer effort (hours) | Grows faster than outcomes |
| Trust | Acceptance rate versus pass rate | Acceptance far above pass |
| Trust | Edit and override rates | A sudden jump |
| Trust | Times users report an error | Rises |
| Trust | Cases handed to a person | Too few (missed) or too many |
Pair them: every speed or cost metric goes with a quality metric. Lower cost with a falling pass rate is not progress.
7. Decide in advance when a metric worsens#
A board without action rules is decoration. An illustration, with thresholds rebuilt:
| If | Then |
|---|---|
| A severe failure (like an invented price) | Switch that part off for now, fix, then turn back on |
| A category's pass rate drops below the set level | Stop rolling out further, review the prompt |
| Acceptance far above pass | Add a confirmation step, show sources |
| Cost per outcome exceeds the budget | Review the model, the way of calling, or the scope |
| p95 latency is long | Shorten the output or change the way of calling |
Specific thresholds depend on the feature's risk. A feature answering about money needs stricter thresholds than one suggesting titles.
8. Common traps#
- Measuring only usage. Heavy use does not mean good.
- Only thumbs up or down. Few people press it and it skews to the extremes. Combine with behavior (edits, overrides).
- Forgetting reviewer effort in cost. Model cost may be cheap while total cost is not.
- Averages hide the tail. Look at p95 and the worst cases.
- Changing the model without rerunning evals.
- Turning a metric into a target (Goodhart's law): if you reward acceptance rate, people will make it look good without improving quality.
9. A balanced view#
- Secondary sources. The three-layer split comes from practitioners' guides (blogs, tracking-tool documentation), not independent research.
- No standard for thresholds. Every number here is an illustration. I do not know what a "normal" acceptance or edit rate is for each kind of feature.
- Measuring takes effort. For a small experiment, a trimmed one-page board (quality, cost, one trust metric) is enough.
- Metrics do not replace listening to users. Read a few real exchanges each week.
- The shop "Hong" example is rebuilt, with figures I made up to illustrate the formula.
10. Conclusion#
A good AI feature is not the one called most, but the one that gives right results, at a reasonable cost, and is trusted appropriately. Measure three layers at once, compute cost per successful outcome, and read the acceptance rate beside the pass rate.
A small step today: pick an AI feature you run, compute its cost per successful outcome (remember to add the effort of fixing), and compare it with the figure you normally use.
References#
- Antoine Buteau: AI product metrics, economics and support burden (secondary, a practitioner's guide)
- PostHog: Product metrics to track for LLM apps (secondary)
- Hamel Husain and Shreya Shankar: LLM Evals, Everything You Need to Know
- Stack Overflow Developer Survey 2025, AI section
- Related posts in this blog: Writing evals for an AI feature, Do not measure productivity by lines of code, Vibe coding a prototype in one day
Frequently asked questions
- Where should I start measuring an AI feature?
- With three layers: quality (is the output right, by failure category), cost (what each successful outcome costs, reviewer effort included, and latency at p95) and trust (how much users accept, edit or override). Measuring one layer alone leaves blind spots: a model that is 95% right is still useless if users do not trust it.
- How is cost per successful outcome calculated?
- Take the total cost (model, infrastructure and the effort of people who review or fix) and divide by the number of successful outcomes, not by the number of calls. This exposes features that call the model cheaply but need lots of human fixing, or need many calls to get one usable result.
- Does a high acceptance rate mean the feature is good?
- Not necessarily. A high acceptance rate may come from over-trust, with users accepting wrong output too. Set it beside the pass rate from evals: if users accept more than is actually right, that signals over-trust. Watch the trend, not only the absolute level.
- Which thresholds are acceptable?
- There is no shared standard. The thresholds seen in practitioner guides (such as an edit rate below 0.3) are only illustrations. Set thresholds by the feature's risk, and decide in advance what happens when one is crossed.