---
title: "Testing AI-Written Code: Avoid Tests That Are Always Green"
url: https://www.thinnd.site/en/engineering/testing-ai-written-code-avoid-always-green-tests/
language: en
author: "Nhu Duy Thin (Thinnd)"
published: 2026-10-29
updated: 2026-10-29
section: "Engineering"
tags: ["testing", "tests", "AI", "mutation testing", "property-based testing", "quality"]
summary: "When AI writes both the code and the tests, a test easily matches the code it just wrote rather than the requirement, so it stays green without catching bugs. High coverage is not enough either: a paper on mutation-guided test generation notes a test suite with 100% coverage but only about a 4% mutation score. Seven ways to keep tests valuable: write tests from the spec rather than the code, see the test fail before fixing, break the code yourself to see whether tests turn red, use mutation testing, property-based testing, boundary tables and one real run."
license: "Cite with the author name and the canonical URL above."
---

# Testing AI-Written Code: Avoid Tests That Are Always Green

> For people who write or review code with AI involved, including beginners. Every technical term gets a short explanation.
>
> The code example and its numbers are **rebuilt** to be easy to picture. The "100% coverage, 4% mutation score" figure comes from a summary of a paper (I have not read the full text). The tools named are only examples; I did not run them. The practices are **my suggestions**.

---

## Quick summary

- When **AI writes both the code and the tests**, a test easily confirms the code, not the requirement.
- **A green test is not enough.** High coverage is not enough either: a suite can cover 100% and catch very few bugs.
- The right question is not "is the test green" but **"if the code were wrong, would the test turn red?"**
- **Seven ways**: write from the spec, see red first, break the code yourself, mutation testing, property-based, boundary tables, a real run.
- Start with the cheapest: **break the code by hand** and watch the tests.

---

## Hard words, plain meaning

| Term | Plain meaning |
|---|---|
| **Test** | Code that runs other code and compares the result with what is expected. |
| **Coverage** | The share of code lines the tests run through. It does not say whether the tests check correctly. |
| **Mutation testing** | Deliberately breaking the code a little to see whether the tests turn red. |
| **Mutation score** | The share of broken versions that the tests catch. |
| **Property-based testing** | Instead of a few examples, describe a rule that always holds and let a tool try many inputs. |
| **Boundary value** | A value sitting right on the line of a rule, like exactly 500 thousand. |
| **An always-green test** | A test that never fails even when the code is wrong. |

---

## 1. The problem: the person who writes the tests also wrote the code

When one person writes the code and then the test, that person usually **carries the same understanding** into both. If they misread the requirement, the code is wrong and the test is wrong along with it, so it is green. With AI this is even more likely: AI writes the tests **after the code exists** and can see it, so tests are often built to match what the code does.

That is why the [15-point checklist for reviewing AI-written code](https://www.thinnd.site/en/engineering/reviewing-ai-written-code/) has two points on tests: do the tests check what matters, and for tests the AI wrote, break the code and see whether they turn red.

And this is not just a feeling. The paper on mutation-guided test generation with language models (MutGen, 2025) argues that **coverage is a weak indicator of fault detection**, and mentions a suite with **100% coverage but only about a 4% mutation score**. I read the summary, so treat the figure as an illustration of the problem, not a general statistic.

---

## 2. An example: the 500-thousand boundary

*A rebuilt example.* The requirement: **orders of 500 thousand or more get 10% off.**

The AI writes:

```js
function discount(total) {
  return total > 500000 ? total * 0.9 : total;
}

test("a large order gets 10% off", () => {
  expect(discount(600000)).toBe(540000);
});

test("a small order gets no discount", () => {
  expect(discount(100000)).toBe(100000);
});
```

Both tests are green, coverage 100%. But the code is wrong: the requirement says "500 thousand **or more**", yet the code uses `>`, so an order of **exactly 500,000** gets no discount.

**Break it by hand.** Change `>` to `>=`, which is to say fix the code to match the requirement. Run the tests again: **still green**. So the two tests cannot tell wrong code from right code at the boundary. This is called a "surviving mutant".

**Add a test from the spec:**

```js
test("exactly 500,000 gets the discount", () => {
  expect(discount(500000)).toBe(450000);
});
```

This one is **red** on the current code and green after the fix. It is a valuable test: it **fails when the code is wrong**.

---

## 3. Seven ways to keep tests valuable

### Way 1: write tests from the spec, not from the code

Look at the **requirement**, not the code, and write examples. The post [Spec-driven development](https://www.thinnd.site/en/engineering/spec-driven-development-write-the-spec-first/) suggests first writing three to six acceptable examples, odd cases included. Those are exactly the tests you need.

If you must have AI write the tests, give it **the spec and the examples**, not only the code.

### Way 2: see the test fail first

Write the test **before** fixing, run it and **see it fail for the right reason**, then fix it to green. A test you have never seen red is one you do not know can catch anything.

Anthropic's practice guide uses exactly this idea in a sample request for an agent: "write a failing test that reproduces it, then fix it".

### Way 3: break the code yourself

The cheapest way, no tools needed. After the tests are green, **deliberately break the code a little** and rerun:

- Change `>` to `>=`, or the reverse.
- Change `0.9` to `0.8`.
- Remove a condition, or change `&&` to `||`.
- Return a fixed value.

If the tests stay green, you just found a place the tests do not check. Then undo the change.

### Way 4: automatic mutation testing

Do what way 3 does, systematically: a tool creates many small broken versions and reports which ones **survive** (the tests did not catch them). There are tools per language, for example Stryker (JavaScript), mutmut (Python), PIT (Java). I name them so you can look into them, not because I ran them for this post.

Note: mutation testing is **slow** and can report many surviving versions that do not matter. Use it on the important core, not necessarily the whole project.

### Way 5: property-based testing

Instead of a few examples, you describe **a rule that always holds**, and a tool tries many random inputs. For the discount function:

- The result is **never larger** than the original total.
- For every total **of 500,000 or more**, the result is exactly 90%.
- For every total **below 500,000**, the result equals the total.

Tools like Hypothesis (Python) or fast-check (JavaScript) do this. They often find boundary values you did not think of.

### Way 6: a boundary table

For each rule with a threshold, write **three values**: just below, exactly at, just above.

| Input | Expected |
|---|---|
| 499,999 | 499,999 (no discount) |
| 500,000 | 450,000 (discount) |
| 500,001 | 450,000.9 (discount) |

Plus odd cases: 0, negative, very large, wrong type, empty.

### Way 7: one real run

A unit test does not replace **opening it and trying it**. The post [Three hours with an AI agent](https://www.thinnd.site/en/field-notes/agent-field-notes-real-project/) has bugs that logic checks missed because they are things the eye sees. Run the app, click through the main path, and note what you see.

---

## 4. The smell of a weak test

When reviewing AI-written tests, look for these signs:

| Sign | Why it is weak |
|---|---|
| **No check** (no `expect` or `assert`) | Only runs through the line, compares no result |
| **Checking the obvious** (the result is not empty, has the right type) | Almost always true |
| **Comparing a result with itself** | Always true, detects nothing |
| **Mocking everything** then checking the mock was called | Tests the mock, not behavior |
| **Snapshotting the current output as the answer** | Every existing bug becomes "correct" |
| **Catching and swallowing errors** then reporting green | Hides failure |
| **Trying one pretty value** (like 600,000) | Misses boundaries and odd cases |
| **Expected values computed with the same formula as the code** | The same mistake passes together |

Every test should be able to answer: **"What kind of wrong code would make this go red?"**

---

## 5. A six-question checklist for reviewing AI-written tests

1. Was the test written **from the spec** or from the code?
2. Have I **seen it fail** at least once?
3. If I **break the code a little**, does a test go red? (Try it at least once at an important spot.)
4. Is there a **boundary table** for every threshold?
5. Are there **real checks** (not just running through) and **no needless mocks**?
6. Have I **run the main path of the app for real**?

---

## 6. How to ask AI to write tests

*My suggestion.* Instead of "write tests for this function", try:

> "Here is the spec and three acceptable examples (the examples above). Write tests from the spec, **not from the existing code**. Include the value just below, exactly at and just above each threshold. Then **deliberately break the code in three ways** and tell me which tests go red. Show me the command you ran and its output."

The last request is also a way for the agent to **check itself**, as in the post [Seven mistakes when writing requests for an agent](https://www.thinnd.site/en/experiences/seven-mistakes-writing-requests-for-agents/).

---

## 7. A balanced view

- **The figures in this post are only a fragment.** I have not read the MutGen paper in full, and I have no measurements of my own for AI-written code in this project.
- **Mutation testing takes time.** Use it selectively, and do not chase a 100% score.
- **Tests are not the only thing.** Code review, watching bugs after release and a real run are all needed.
- **Good tests also cost maintenance.** Too many detailed tests can slow down changing the code.
- **The 500-thousand example is rebuilt**, but the kind of bug (wrong at the boundary) is very real and common.

---

## 8. Conclusion

A green test only says that **the test and the code agree with each other**, not that both are right. When AI writes both, that agreement can easily come from the same misunderstanding. The question worth asking is **"if the code were wrong, would the test go red?"**, and the cheapest way to find out is to break the code and look.

A small step today: pick one important function with AI-written tests, change one comparison sign or number in it, and rerun the tests. If they stay green, you have just found the first test that needs writing.

---

## References

- [On Mutation-Guided Unit Test Generation (MutGen), arXiv:2506.02954](https://arxiv.org/html/2506.02954v2)
- [Anthropic: Best practices for Claude Code](https://code.claude.com/docs/en/best-practices)
- Related posts in this blog: [Reviewing AI-written code](https://www.thinnd.site/en/engineering/reviewing-ai-written-code/), [Spec-driven development](https://www.thinnd.site/en/engineering/spec-driven-development-write-the-spec-first/), [Seven mistakes when writing requests for an agent](https://www.thinnd.site/en/experiences/seven-mistakes-writing-requests-for-agents/), [Writing evals for an AI feature](https://www.thinnd.site/en/engineering/writing-evals-for-ai-features-thirty-real-cases/)