---
title: "Writing Evals for an AI Feature: Start from Thirty Real Cases"
url: https://www.thinnd.site/en/engineering/writing-evals-for-ai-features-thirty-real-cases/
language: en
author: "Nhu Duy Thin (Thinnd)"
published: 2026-10-29
updated: 2026-10-29
section: "Engineering"
tags: ["evals", "AI testing", "LLM", "quality", "error analysis", "AI product"]
summary: "To measure an AI feature, start with error analysis on real data rather than building infrastructure: read at least about thirty cases by hand, note what went wrong in each, group the notes into failure categories and count them. Only then write evals for the failures you actually see, preferring ordinary code checks where a rule can be stated, and using a model as a judge only when judgment is needed, then compare the judge with human labels. Grade pass or fail rather than on a scale."
license: "Cite with the author name and the canonical URL above."
---

# Writing Evals for an AI Feature: Start from Thirty Real Cases

> For people building features on language models, including beginners. Every technical term gets a short explanation.
>
> The method and the numbers (30, 100, 100 to 200 samples) come from Hamel Husain and Shreya Shankar's guide (I read the first part of the page directly but not all of it; roughly the last third is unread). It is practitioners' guidance, **not a standard**. The shop "Hong" and the figures in the table are **a rebuilt illustration**.

---

## Quick summary

- Measuring an AI feature **starts by reading real data**, not by building infrastructure.
- Read about **30 cases by hand**, note what went wrong, then **group the notes into failure categories and count them**.
- Write evals for **the failures you actually see**, not the ones you imagine.
- Use **ordinary code checks** where there is a clear rule. Use a **model as a judge** only when judgment is needed, and **compare it with human graders**.
- Grade **pass or fail**, not on a scale.
- It can be done in **three afternoons**.

---

## Hard words, plain meaning

| Term | Plain meaning |
|---|---|
| **Evals** | Tests that measure the quality of an AI feature. |
| **Trace** | One full run: the user's input, the steps the system took, the output. |
| **Error analysis** | Reading cases, noting what went wrong, grouping into categories. |
| **Open coding** | Reading each case and writing free notes on what went wrong, before grouping. |
| **Axial coding** | Grouping the notes into failure categories and counting each. |
| **LLM judge** | Using another model to grade pass or fail. |
| **Binary** | Only two outcomes: pass or fail. |

---

## 1. Why it is needed, and why it is hard

Ordinary software usually has a right answer: a function adding two numbers returns the sum. AI features differ: for the same question a model can answer many ways, and "good" is a judgment. Many teams therefore skip measuring and only learn about failures when customers complain.

Husain and Shankar's guide says plainly: **start with error analysis, not infrastructure**. Meaning that before buying tools, building dashboards or writing judges, **sit down and read real outputs**. It sounds dull, but it answers the most important question: *where is my feature breaking?*

This post links to [Why you should not trust AI blindly](https://www.thinnd.site/en/engineering/why-not-to-trust-ai-blindly/): a model can be fluent and wrong, so you need a way to measure beyond feelings. And to [OWASP Top 10 for AI apps](https://www.thinnd.site/en/engineering/owasp-top-10-for-ai-apps-everyday-examples/): attacking your own app is also a kind of test.

---

## 2. Step one: have thirty cases to read

According to the guide, start with about **100 diverse cases**, and **annotate at least the first 30** by hand. I use 30 as a practical starting point, small enough for one afternoon.

**If you already have real users:** take real conversations or requests, pick a diverse set (easy, hard, odd), and remove personal information.

**If you have no real data yet**, the guide suggests **structured** synthetic data, not just asking a model to "write test questions":

1. Pick a few **dimensions** (for example issue type, customer mood, complexity).
2. Write about **20 value combinations** by hand (one value per dimension).
3. Ask a model to turn each combination into a natural question, in a separate step.
4. Run the questions through the **real system** and read the results.

Synthetic data cannot tell you **which failures are common in reality**. Compare with real data when you have it.

---

## 3. Step two: read and annotate, one case at a time

For each case, someone with domain knowledge (not just anyone) writes a **free note** on what went wrong, focusing on the **first failure** they see. This is open coding.

Some advice from the guide:

- **Do not write criteria before reading.** Criteria shift as you read, so draft them afterward.
- **Record problems that are not the model's fault**, such as a missing feature or a broken link.
- **Keep reading until new kinds of failure stop appearing.** If after 100 cases you are still learning, read more.
- **Do not hand it to people without domain knowledge.** The guide lists this among common mistakes.

**A logging sheet (my suggestion):**

| # | Input | System output | Pass? | First failure note |
|---|---|---|---|---|
| 1 | "I want to change my shirt size" | Gives the refund policy | No | Misread the intent, answered something else |
| 2 | "When will my order arrive?" | Gives the right delivery date | Pass | |

---

## 4. Step three: group and count

From the notes, group similar failures into **categories** and **count each**. The guide calls this axial coding and treats it as **the most important step**, because it tells you what to fix first and which failures deserve an eval.

**An illustration (rebuilt numbers): the shop "Hong" has a customer assistant.** After reading 30 cases:

| Failure category | Cases | Suggested handling |
|---|---|---|
| Misread the intent, answered something else | 9 | Fix the prompt, add examples |
| Invented information (delivery date, price) | 6 | Force it to come from the order system, check with code |
| Tone too stiff, no apology when the customer is upset | 5 | Could use a judge, needs human comparison |
| Right answer but too long | 4 | Check length with code |
| Did not hand over to a person when needed | 3 | A clear rule, check with code |
| Other | 3 | Keep watching |

This table alone gives you **the first three things** to fix and measure. That is why you start by reading data.

---

## 5. Step four: write evals for real failures

For each failure category, choose a kind of check:

| Kind | Use when | Example |
|---|---|---|
| **Ordinary code check** | A rule can be stated | Output is valid JSON, has all fields, is under 120 words, delivery date matches the system |
| **LLM judge** | The failure needs human-style judgment | "Does the assistant apologize when the customer sounds upset" |

According to the guide: **before building a judge, see whether a code rule can do it**, because judges carry real maintenance cost. Each judge handles **one kind of failure**; do not ask one judge to catch many at once.

**Use pass or fail.** The guide gives its reasoning: scales (such as 1 to 5) have inconsistent gaps between levels, need larger samples and invite middle scores. To track gradual progress, split into several small binary checks, for example "four of the five required points are present".

---

## 6. Step five: do not trust the judge before checking it

If you use a model as a judge, check **the judge** like anything else. According to the guide:

- Label about **100 to 200 samples per failure type**, at least roughly 50 passing and 50 failing.
- Split the data: a small part as examples in the prompt, the rest divided between a development set and a held-out test set.
- Measure the **rate of catching real failures** (how many real failures the judge catches) and the **rate of correctly passing good samples**.
- **Look at the cases where the judge and humans disagree** before tuning the judge's prompt.
- If scores on the test set are much worse than on the development set, you may have fitted the judge too tightly to one set.

This part takes effort, so do not rush. I have read only the first part of the guide and have not re-verified the numbers, so treat them as **a practitioner's starting point**.

---

## 7. A three-afternoon plan

| Afternoon | Work | Output |
|---|---|---|
| **1** | Collect 30 cases (real, or structured synthetic). Run them through the real system | A 30-row sheet with inputs and outputs |
| **2** | Read each case, grade pass or fail, note the first failure | The notes sheet |
| **3** | Group, count, pick the three largest categories, write three checks (code first) | A failure-category table and three working checks |

After three afternoons you have: a list of failures by how common they are, three automatic checks, and a habit: **after each significant change, spend about half an hour reading 20 to 50 results**, which the guide suggests as a minimum.

---

## 8. Common mistakes

According to the guide:

- **Skipping error analysis** and using generic metrics like "helpfulness".
- **Writing evals for imagined failures** instead of observed ones.
- **One judge catching many failures** at once.
- **Trusting a judge without comparing** with humans.
- **Chasing 100% pass.** The guide suggests a pass rate around 70% may signal an appropriately hard eval.
- **Giving error analysis to people without domain knowledge.**

---

## 9. A balanced view

- **This is practitioners' guidance.** The numbers 30, 100 and 100 to 200 are starting points, not widely verified standards.
- **I have not read the whole guide.** Roughly the last third of the page is unread, so this article is not exhaustive.
- **Evals take effort.** For a throwaway prototype you may not need them yet. Do them when the feature has real users or meaningful risk.
- **Automatic checks do not replace reading.** The guide stresses looking at real data. Automation only keeps what you have learned from breaking again.
- **The shop "Hong" is an illustration**, with figures I built to make it easy to picture.

---

## 10. Conclusion

Measuring an AI feature does not start with tools. It starts with **an afternoon spent reading thirty real cases** and a pen. From that you learn where the feature breaks, and then write evals in the right places.

A small step today: take the last thirty outputs of the AI feature you are building, read each one, and write a line about what went wrong in each. Then count.

---

## References

- [Hamel Husain and Shreya Shankar: LLM Evals, Everything You Need to Know](https://hamel.dev/blog/posts/evals-faq/)
- Related posts in this blog: [Why you should not trust AI blindly](https://www.thinnd.site/en/engineering/why-not-to-trust-ai-blindly/), [OWASP Top 10 for AI apps](https://www.thinnd.site/en/engineering/owasp-top-10-for-ai-apps-everyday-examples/), [Reviewing AI-written code](https://www.thinnd.site/en/engineering/reviewing-ai-written-code/)