---
title: "Measuring an AI Feature: Quality, Cost, Trust"
url: https://www.thinnd.site/en/product-thinking/measuring-an-ai-feature-quality-cost-trust/
language: en
author: "Nhu Duy Thin (Thinnd)"
published: 2026-10-29
updated: 2026-10-29
section: "Product thinking"
tags: ["AI product", "metrics", "evals", "cost", "trust", "measurement"]
summary: "An AI feature should be measured in three layers at once. Quality: the pass rate per failure category, drawn from error analysis and evals. Cost: cost per successful outcome, including reviewer effort, and latency at p95 rather than the average. Trust: users' acceptance, edit and override rates, the number of times they report an error and the cases handed to a person. Measuring one layer alone creates blind spots. The thresholds in this article are only illustrations, and there is no shared standard."
license: "Cite with the author name and the canonical URL above."
---

# Measuring an AI Feature: Quality, Cost, Trust

> For product people, team leads and programmers bringing AI into a product. Every technical term gets a short explanation.
>
> The three-layer split and the metrics come from **practitioners' guides** (blogs and tool documentation), which are secondary sources. **All the thresholds and figures in this article are illustrations**, not standards. The shop "Hong" example is rebuilt.

---

## Quick summary

- Usage counts are **not** success. Measure **the outcome of the work**.
- Measure **three layers at once**: **quality**, **cost**, **trust**. A single layer creates blind spots.
- **The right cost** is the cost per **successful outcome**, reviewer effort included.
- **A high acceptance rate** can mean over-trust. Set it beside the pass rate from evals.
- Use **a one-page board** and **decide in advance** what happens when a metric worsens.

---

## Hard words, plain meaning

| Term | Plain meaning |
|---|---|
| **Evals** | Tests measuring the quality of an AI feature (see the dedicated post). |
| **Acceptance rate** | The share of AI output users take and use as is. |
| **Edit rate** | The share of output users change before using. |
| **Override rate** | The share of output users discard and do themselves. |
| **p95 latency** | The time under which 95% of requests finish. It shows the slowest cases, which users actually remember. |
| **Token** | The unit of text a model processes and bills by. |
| **Cost per successful outcome** | Total cost divided by the number of times the AI really helped. |

---

## 1. Why not measure one number

It is easy to start with what is easy to count: number of calls, number of users. But an AI feature can be used heavily and still be **bad**: answering wrongly, costing a lot, or being quietly ignored by users.

Practitioners' guides (which I read via summaries) agree on two points: **counting calls is shallow**, what matters is **work completed**; and metrics should be **grouped in layers**, because focusing on one creates blind spots. A common example: **a model that is 95% right is still useless if users do not trust it**.

So I split three layers:

| Layer | Question | Data source |
|---|---|---|
| **Quality** | Is the output right and useful? | Error analysis, evals |
| **Cost** | What does each successful outcome cost, and is it fast? | Invoices, logs, timing |
| **Trust** | Do users trust and use it? | User behavior, feedback |

This post connects to [Writing evals for an AI feature](https://www.thinnd.site/en/engineering/writing-evals-for-ai-features-thirty-real-cases/) (the quality layer) and [Do not measure productivity by lines of code](https://www.thinnd.site/en/product-thinking/dont-measure-productivity-by-lines-of-code/) (why activity metrics mislead).

---

## 2. The quality layer

Quality is not a single "accuracy" number. Draw it from **error analysis**: group failures into categories and count them, as in the post on evals.

**Suggested metrics:**

- **Pass rate per failure category.** For example "misread the intent", "invented information", "tone too stiff". One number per category.
- **Count of severe failures.** An invented price is very different from a tone slip. Count failures with real consequences separately.
- **Overall pass rate**, but **not** as the only number.

**Report an "all pass" figure**, then drill down into each category. The evals guide also advises grading pass or fail rather than on a scale.

**How to track:** rerun the evals **whenever you change the model, the prompt or the data**. If you change the model without rerunning, you do not know how quality moved.

---

## 3. The cost layer

### 3.1. Cost per call and per successful outcome

Cost per call is the easy figure. But a feature may call the model cheaply while **needing lots of human fixing**, or need many calls to get one usable result. So the cost worth comparing is:

**Cost per successful outcome** = (model cost + infrastructure + effort of people who review or fix) ÷ number of successful outcomes

**An illustration (rebuilt numbers): a customer assistant for the shop "Hong", one month.**

| Item | Illustrative value |
|---|---|
| Customer questions | 10,000 |
| Model and infrastructure cost | 20 million dong |
| Answers usable as is | 6,500 |
| Answers needing staff fixes (3 minutes each, 60 thousand dong an hour) | 2,500 answers, that is 7,500 minutes, 125 hours, 7.5 million dong |
| Cases handed over entirely to staff | 1,000 |
| **Total cost** | 20 + 7.5 = **27.5 million dong** |
| **Successful outcomes** (used as is or quickly fixed) | 6,500 + 2,500 = **9,000** |
| **Cost per successful outcome** | about **3,060 dong** |

Looking only at model cost, 20 million over 10,000 calls is 2,000 dong a call, and you would **miss the staff effort** and the 1,000 failed cases. That is why "cost per successful outcome" reveals more.

### 3.2. Latency

Measure latency at **p95** (and p99), not just the average. The average hides the slowest cases, which is when users leave. For a chat feature, **time to the first word** strongly shapes the feeling of speed.

---

## 4. The trust layer

This is the layer most often skipped. Look at user behavior:

| Metric | Meaning |
|---|---|
| **Acceptance rate** | Users take and use it as is |
| **Edit rate** | Users change it, then use it |
| **Override rate** | Users discard it and do it themselves |
| **Error reports** | Users actively say "this is wrong" |
| **Cases handed to a person, and how they are handled** | Whether the right cases get handed over |
| **Reopen or complaint rate** | Work that was "done" but came back |

**Watch the trend, not only the absolute level.** Some guides note that a sudden jump in the edit rate signals a quality drop, not personalization.

**Context:** the Stack Overflow 2025 survey (more than 49,000 respondents) found 46% of users do not trust AI's accuracy and only 3% trust it highly. Your product's users may feel similarly. Trust has to be **earned**, and has to be **measured**.

---

## 5. The trap: a high acceptance rate can be over-trust

This is the point I find most important in this article. If users accept 90% of outputs but **evals show only 75% are actually right**, the 15-point gap is **users over-trusting**, not a good feature.

**How to read it:** set the **acceptance rate** beside the **pass rate from evals**.

| Situation | May mean |
|---|---|
| High acceptance, high pass | A good feature, trusted appropriately |
| **High acceptance, low pass** | **Over-trust, high risk** |
| Low acceptance, high pass | Users do not yet trust it, possibly due to interface or habit |
| Low acceptance, low pass | The feature falls short, so low use is right |

I have no data on what these rates typically are. This is a way of **reading** the metrics, not a measurement result.

---

## 6. A one-page board

One board of nine metrics, three layers, recorded weekly.

| Layer | Metric | Warn when |
|---|---|---|
| Quality | Pass rate per failure category | A category drops, or a new severe failure appears |
| Quality | Count of severe failures | Any failure with real consequences |
| Cost | Cost per successful outcome | Rises as usage rises |
| Cost | p95 latency | Gets longer, or the first word is slow |
| Cost | Reviewer or fixer effort (hours) | Grows faster than outcomes |
| Trust | Acceptance rate versus pass rate | Acceptance far above pass |
| Trust | Edit and override rates | A sudden jump |
| Trust | Times users report an error | Rises |
| Trust | Cases handed to a person | Too few (missed) or too many |

**Pair them:** every speed or cost metric goes with a quality metric. Lower cost with a falling pass rate is not progress.

---

## 7. Decide in advance when a metric worsens

A board without action rules is decoration. *An illustration, with thresholds rebuilt:*

| If | Then |
|---|---|
| A severe failure (like an invented price) | **Switch that part off for now**, fix, then turn back on |
| A category's pass rate drops below the set level | Stop rolling out further, review the prompt |
| Acceptance far above pass | Add a confirmation step, show sources |
| Cost per outcome exceeds the budget | Review the model, the way of calling, or the scope |
| p95 latency is long | Shorten the output or change the way of calling |

Specific thresholds **depend on the feature's risk**. A feature answering about money needs stricter thresholds than one suggesting titles.

---

## 8. Common traps

- **Measuring only usage.** Heavy use does not mean good.
- **Only thumbs up or down.** Few people press it and it skews to the extremes. Combine with behavior (edits, overrides).
- **Forgetting reviewer effort in cost.** Model cost may be cheap while total cost is not.
- **Averages hide the tail.** Look at p95 and the worst cases.
- **Changing the model without rerunning evals.**
- **Turning a metric into a target** (Goodhart's law): if you reward acceptance rate, people will make it look good without improving quality.

---

## 9. A balanced view

- **Secondary sources.** The three-layer split comes from practitioners' guides (blogs, tracking-tool documentation), not independent research.
- **No standard for thresholds.** Every number here is an illustration. I do not know what a "normal" acceptance or edit rate is for each kind of feature.
- **Measuring takes effort.** For a small experiment, a trimmed one-page board (quality, cost, one trust metric) is enough.
- **Metrics do not replace listening to users.** Read a few real exchanges each week.
- **The shop "Hong" example is rebuilt**, with figures I made up to illustrate the formula.

---

## 10. Conclusion

A good AI feature is not the one called most, but the one that **gives right results, at a reasonable cost, and is trusted appropriately**. Measure three layers at once, compute cost per successful outcome, and read the acceptance rate **beside** the pass rate.

A small step today: pick an AI feature you run, compute its **cost per successful outcome** (remember to add the effort of fixing), and compare it with the figure you normally use.

---

## References

- [Antoine Buteau: AI product metrics, economics and support burden](https://www.antoinebuteau.com/ai-product-metrics-economics-and-support-burden/) (secondary, a practitioner's guide)
- [PostHog: Product metrics to track for LLM apps](https://posthog.com/product-engineers/llm-product-metrics) (secondary)
- [Hamel Husain and Shreya Shankar: LLM Evals, Everything You Need to Know](https://hamel.dev/blog/posts/evals-faq/)
- [Stack Overflow Developer Survey 2025, AI section](https://survey.stackoverflow.co/2025/ai)
- Related posts in this blog: [Writing evals for an AI feature](https://www.thinnd.site/en/engineering/writing-evals-for-ai-features-thirty-real-cases/), [Do not measure productivity by lines of code](https://www.thinnd.site/en/product-thinking/dont-measure-productivity-by-lines-of-code/), [Vibe coding a prototype in one day](https://www.thinnd.site/en/product-thinking/vibe-coding-one-day-prototype/)