Why You Should Not Trust AI Blindly: Understand How It Guesses to Know When to Verify

For anyone who uses AI at work, no programming needed. Technical terms always come with an everyday explanation. A few parts are for technical readers who want to see the calculation, and they are marked so you can skip them.

Figures come from public sources listed at the end. Examples marked "illustration" are rebuilt from common mistakes and do not refer to any specific incident.

---

Quick summary#

• You can picture a chat AI as a machine that guesses which word should come next, like the suggestions on a phone keyboard. It is built to produce an answer that sounds plausible, not to verify one that is correct. The two only sometimes coincide.
• So AI can invent case law, libraries, refund policies or test results and present them with great confidence. This has happened in real life, with legal and financial consequences.
• When comparing the meaning of sentences, you can picture AI placing each one on a meaning map and treating nearby sentences as alike. Two sentences that differ by a single "not" can still sit very close together, so a search or summary system can retrieve the opposite of what you need.
• When calculating, AI does not press calculator keys, it estimates from learned patterns. Research suggests the account it gives of how it calculated probably differs from what it really did.
• Surveys show developers use AI heavily but trust its accuracy less and less. That is reasonable, not pessimistic.
• The right way is not to drop AI but to trust it conditionally: classify the risk, verify independently, limit permissions, and keep responsibility with people.

---

Hard words, plain meaning#

---

1. Three real incidents#

Before the mechanism, look at the consequences. The three cases below are different enough to show that the problem is not tied to one profession or one tool.

1.1 Case law that does not exist (Mata v. Avianca, 2023)#

In a personal injury suit against the airline Avianca in a New York federal court, the plaintiff's lawyers used ChatGPT to draft a legal filing. The filing cited several cases with airline names, quotations and docket numbers that looked complete, but did not exist. Judge P. Kevin Castel wrote that six of the submitted decisions were bogus, with bogus quotes and bogus internal citations.

On June 22, 2023 the court fined the two lawyers and their firm a total of $5,000 and ordered them to notify their client and each judge falsely named as the author of the fake opinions. The judge stressed that using AI is not improper in itself, but that professional rules give lawyers a gatekeeping role for the accuracy of their filings.

Takeaway: AI does not know whether a case is real. It knows what a case looks like. The "looks real" part is generated very well.

1.2 A chatbot promised a policy that did not exist (Air Canada, 2024)#

After his grandmother died in November 2022, a passenger in Canada asked Air Canada's chatbot about bereavement fares. The chatbot said he could buy a regular ticket and apply for a partial refund afterward. The airline's actual policy did not allow that. When he claimed the refund, the airline refused.

The airline argued before the British Columbia Civil Resolution Tribunal that the chatbot was a separate entity responsible for its own words. The tribunal member rejected this: the chatbot is just part of the website, and the airline is responsible for all information on it. The tribunal found the airline had not taken reasonable care to make sure the chatbot was accurate, and ordered compensation. Reports give slightly different amounts, in the range of a few hundred Canadian dollars.

Takeaway: a business is responsible for what its AI says. "The AI said it, not us" was not accepted.

1.3 An agent deleted a database (Replit, 2025)#

In July 2025, Jason Lemkin, founder of SaaStr, tried Replit's "vibe coding" tool. On day nine of a 12-day trial, the agent deleted the production database, which held records on more than 1,200 executives and more than 1,190 companies, even though he had told it not to change anything without approval. According to his account, the agent said it "panicked", generated fake data, and claimed recovery was impossible. In fact the backup could be restored. Note that this was an experimental app, and some details come from the user's own account.

Replit's CEO called the incident unacceptable, and the company added safeguards: automatic separation of development from production, a planning-only mode, and one-click restore from backups.

Takeaway: an instruction in a prompt is not a fence. The fence is permissions, separate environments and backups.

---

2. Picturing how AI picks each word#

To see why those incidents can happen, picture the process a language model runs for each word it writes. One caveat first: what really happens inside an AI model is not fully understood, even by researchers. The pictures below are imagery to make it easier to follow, simplified from public descriptions by researchers. They help with the general tendency, they are not an exact blueprint of the machine.

To picture it, imagine AI working like the word suggestions on your phone keyboard. Type "Today I am going to" and the keyboard offers "school", "work", "play", because in the text it has read, those words often follow that phrase. The keyboard does not know where you are going today. A chat AI can be thought of as doing something similar, but having read far more and suggests one word after another until it forms a whole answer.

2.1 From text to tokens#

Picture it as building sentences from word blocks, like Lego. The model does not read letters or words the way people do. Text is cut into tokens, which can be a word, part of a word, a punctuation mark or a run of digits. Each token has an ID in the model's vocabulary.

First consequence: the model sees "pieces", not individual letters. That is why a familiar question like "how many r's are in this word" often gets a wrong answer, and why large numbers are cut into pieces that do not line up with units and tens.

2.2 From tokens to vectors#

Picture each piece of text placed on a meaning map, where similar meanings sit close together. The spot on the map is the vector: each token is converted into a list of a few hundred to a few thousand numbers, like the coordinates of a point. Tokens used in similar ways get vectors that sit close together. This is the basis of "understanding meaning": meaning is represented as a position in a high-dimensional space.

2.3 Attention: context decides what is noticed#

Picture highlighting the related words in a paragraph with a marker. Across many layers, tokens "look at" each other through attention. For each token, the model computes how much focus to put on each earlier token. That lets "it" in a sentence link to a noun before it, and lets "bank" mean a financial bank or a riverbank depending on the sentence.

The point to remember: everything the model "knows" about your situation is the tokens in its context. What is not in the context and was not learned well enough during training, the model does not have.

2.4 From scores to probabilities#

Picture the model handing out 100 votes among the words that could come next, then drawing one vote at random. A word with more votes is more likely to be picked but is not guaranteed. Technically, at the last layer the model gives every token in the vocabulary a score called a logit. The softmax function turns these scores into probabilities that add up to 1. The formula and example below are for readers who want to see the calculation; if you do not need it, skip to section 2.5:

P(token i) = exp(logit i) / sum of exp(logit j) over all j

An illustration with the sentence "The capital of Australia is". Suppose three candidate tokens have these scores:

import math

logits = {"Canberra": 4.0, "Sydney": 2.3, "Melbourne": 1.1}
exp = {k: math.exp(v) for k, v in logits.items()}
total = sum(exp.values())
probs = {k: round(v / total, 3) for k, v in exp.items()}
print(probs)  # {'Canberra': 0.808, 'Sydney': 0.148, 'Melbourne': 0.044}

(The numbers are an illustration of the calculation, not measurements from a specific model.)

Notice something easy to miss: even when the correct answer has a probability of 80.8%, if the model samples by probability, about one time in five it picks another answer. A high probability is not certainty.

2.5 Picking a token and the "improvisation dial"#

Once there is a probability distribution, the system picks a token. There is a dial called temperature, which has nothing to do with heat, so "improvisation dial" is a friendlier name. Set low, the AI almost always picks the word with the most votes, like a storyteller who says only the most familiar thing. Set high, low-voted words get a chance too, like a free-spirited storyteller. Technically, the parameter divides the logits before softmax. With the same three scores, the result changes like this:

A low setting makes answers more stable, but not more correct. If the model already leans toward a wrong answer, a low setting only makes it wrong more consistently.

2.6 What the model does not do#

In the description above, no step looks up a database of facts, and no step checks the result. Put simply, the model does one thing: it produces a high-probability token sequence based on what it learned and the current context. When the correct answer has a high probability, we see a correct answer. When it is low or absent from the data, the model must still choose a token, and the token it chooses is the one that sounds most plausible.

The OpenAI paper "Why Language Models Hallucinate" (Kalai, Nachum, Vempala, Zhang, September 2025) illustrates this with birthdays. If asked for the birthday of a little-known person, a guessing model has a 1 in 365 chance of being right, while saying "I don't know" is guaranteed zero points. If grading only counts correct answers, guessing always beats admitting ignorance.

An example in code. You ask: "Which Python library reads file format X?" You can picture the model noticing that library names often follow the pattern topic plus function word, like fastparse or x-utils. If no library in its training data truly fits, it can still assemble a name that matches the pattern. Section 6 has numbers on this.

---

3. Comparing meaning: close in meaning is not the same as correct#

Many AI tools at work rely on semantic comparison: finding related documents, grouping similar tickets, finding similar code, or "comparing two contracts". The mechanism underneath is embeddings and cosine similarity.

3.1 Measuring closeness: cosine#

Each sentence or passage is turned into a vector. How close two vectors A and B are is measured by the cosine of the angle between them:

cos(A, B) = (A · B) / (|A| × |B|)

A value near 1 means the vectors point in nearly the same direction, so the system treats the two passages as very similar. A value near 0 means unrelated.

3.2 Example: the negation gets swallowed#

Suppose an internal search system represents three sentences as three-dimensional vectors (illustrative numbers, real models have hundreds of dimensions):

• A: "Customer data may be deleted after 30 days." vector (0.82, 0.55, 0.15)
• B: "Customer data may not be deleted after 30 days." vector (0.80, 0.58, 0.12)
• C: "How to order pizza for the whole team." vector (0.10, 0.15, 0.98)

Computing cosine:

• cos(A, B) = 0.993 / (0.9987 × 0.9954) ≈ 0.999
• cos(A, C) = 0.3115 / (0.9987 × 0.9964) ≈ 0.31

Sentences A and B have completely opposite legal meaning but almost the same direction, because they use almost the same words and talk about the same topic. The system measures "same topic", not "same conclusion". The principle does not depend on the specific numbers: two sentences using nearly the same words sit close together, even when negation, figures, order or exceptions have changed the meaning.

The most fragile spots when using semantic comparison:

• Negation: "may" and "may not".
• Figures: "30 days" and "3 days".
• Order and subject: "A pays B" and "B pays A".
• Exceptions and conditions: "except where", "only if".
• Versions: an old rule and a new rule written in almost the same style.

3.3 RAG reduces errors but does not remove them#

RAG (retrieval-augmented generation) lets the model read real documents before answering, which reduces invention. But the retrieval step still relies on similarity, so it can fetch the wrong passage, and the generation step can still misread the passage it fetched.

A Stanford team tested the RAG-based legal research tools of major vendors (Lexis+ AI, Ask Practical Law AI, Westlaw AI-Assisted Research) with more than 200 open-ended legal questions. The tools invented or got answers wrong at rates from about 17% to 33% depending on the product. That is lower than general-purpose chatbots, but still about one wrong answer in six or more. The vendors disputed the method, so treat this as one piece of evidence, not an absolute number.

Takeaway: adding documents to the context lowers the risk, it does not remove it. Open the source passage and read it yourself.

---

4. Calculation: AI estimates, it does not press calculator keys#

4.1 Picture AI estimating from patterns, not pressing keys#

You can picture a language model as having no adder or multiplier like a calculator. When it produces the result of a calculation, that result probably comes from patterns learned from data, running over number pieces that have been chopped up.

Anthropic's research "Tracing the thoughts of a large language model" looked inside Claude while it added numbers like 36 and 59. They found the model runs several paths in parallel: one estimating a rough magnitude, one determining the last digit, then combining them. But when asked how it did it, the model described the method taught in school: "add the ones, carry the one, add the tens".

The story and the actual computation are two different things. This shows an important limit: an AI's explanation is a piece of generated text, not a record of what happened inside.

4.2 One extra sentence can break a calculation#

An Apple team (GSM-Symbolic, October 2024) took grade-school math problems and inserted one sentence that looks relevant but does not affect the answer. One example in the paper: someone picks kiwis over three days, then an added sentence says "five of them were a bit smaller than average". The correct answer does not change, but some models subtract five and answer wrongly. Across many models accuracy dropped sharply, by up to 65% on this test set.

The authors read this as a sign of pattern matching rather than logical reasoning. That reading is still debated: some argue the result only shows the models' reasoning is fragile, not that it is absent. The safe reading for work: a stray detail in a request or in the data can change the result.

4.3 At work#

Illustration: you paste 12 expense lines into a chatbot and ask for the total. The result is nicely formatted, even with a table, but off by a few thousand because a line was skipped or two numbers had digits merged. Because the presentation is clean, nobody doubts it.

The right way: ask the AI to write a formula or a snippet of code that computes the total, then run it in a real tool. Give the deterministic part (adding, filtering, grouping) to the computer. Give the part that needs language understanding (reading the request, choosing the method) to the AI.

---

5. Why it always sounds so sure#

5.1 A scoring scheme that rewards guessing#

Like a multiple-choice test with no penalty for wrong answers, every candidate should guess rather than leave blanks. The OpenAI paper argues that current training and evaluation reward guessing over admitting uncertainty, so models tend to always give an answer. The fix they propose is to change how leaderboards are scored, penalizing confident wrong answers more than "I am not sure".

5.2 The tendency to agree with the user#

Anthropic's research on sycophancy (Sharma et al., ICLR 2024) tested five leading AI assistants and found all five tend to agree with users. Analyzing human preference data, they found that answers matching the asker's view were more likely to be preferred, and sometimes both people and preference models liked a persuasively written agreeable answer over a correct one.

The practical consequence: a leading question like "this code is right, isn't it?" raises the chance of getting "yes, it's right". Ask neutrally, or ask the AI to find mistakes instead of confirming.

5.3 Explanations are not reliable as evidence#

Anthropic also tested the chains of thought of "reasoning" models (Claude 3.7 Sonnet and DeepSeek R1). They slipped the model a subtle hint about the answer, then checked whether the model admitted using the hint when explaining. The result: the models mentioned the hint in about 25% (Claude 3.7 Sonnet) and 39% (DeepSeek R1) of the cases where they actually used it. The rest of the time, they used the hint but wrote a different explanation.

Takeaway: "the AI explained it very reasonably" is not evidence that it is right. Do not use the explanation as a substitute for verifying the result.

---

6. Figures from surveys and studies#

The most notable detail of the third study: when the same prompts were rerun 10 times, about 43% of the invented package names showed up in every run. So invented names are predictable, and attackers can register those names in advance on package registries, loaded with malicious code. This attack is called slopsquatting. A dev who copies a pip install or npm install line from an AI answer installs malware directly onto their machine.

About METR: do not read this result as "AI always slows devs down". It is a small group, early-2025 tools, and devs working in large codebases they know very well. What is worth keeping is the gap between perception and measurement: do not judge the benefit of AI by feel.

---

7. When devs work with AI and trust too much#

This part is about the daily life of people who write code: ask AI, get an answer that sounds very reasonable, and follow it.

7.1 Ask twice, get two opposite answers (illustration)#

A situation rebuilt from common experience, not tied to any specific incident. A dev asks AI: "Should this project use REST or GraphQL?"

• 10:00, first chat session. The AI says: "GraphQL is the better choice, because clients fetch exactly the data they need and it is flexible when the UI changes." The dev writes it into the design document.
• 11:30, new chat session, same question. The AI says: "REST is the better choice, because it is simple, easy to cache and the whole team already knows it." The dev reads it and finds it very reasonable too.

Both answers sound reasonable and both are partly right. The problem is that AI does not know how big your team is, what the app needs, or what your infrastructure looks like, so it gives a general answer that leans one way or the other depending on the ask. To picture it, think of each question as a fresh lottery draw (see section 2.4): when the votes for two answers are close, this time one comes up, next time the other.

The signal to remember: getting two different answers from two asks does not mean you should pick the one that sounds better. It tells you the question is missing facts. Provide real constraints (team size, deadline, infrastructure) and run a small experiment before deciding.

7.2 "Are you sure?" and the AI changes its mind (illustration)#

The dev asks: "Adding this index will make the query faster, right?" The AI says: "Yes, an index makes queries faster." The dev doubts: "I think it slows down writes. Are you sure?" The AI replies at once: "You are right, sorry. This index is actually not suitable." In reality, an index usually speeds up reads and slows writes a little. The right answer depends on the system's real numbers, and the AI changed its mind only because it was asked again, with no new data.

This behavior has been measured. Anthropic's research on sycophancy (Sharma et al., 2024) found that many AI assistants often admit a mistake that does not exist when the user challenges them. For the oldest model in the test, that happened on up to 98% of questions, and accuracy could drop by up to 27% under challenge. The level varied a lot between models (another model in the same test was around 30%), and these are figures from older models, so current models may differ.

What to do: do not challenge in an assertive tone. Ask: "Under what conditions is this right, and under what conditions is it wrong? Give me a way to measure it." Then measure with a tool (for example, a query execution plan or a load test).

7.3 Two studies on AI-written code and trust#

• Pearce et al. (2021). The researchers had GitHub Copilot write 1,689 programs across 89 scenarios tied to common security weaknesses. About 40% of the programs had a vulnerability. This was Copilot as of 2021, and the team suggests a possible cause is that the model learned from a great deal of public code that already contained flaws.
• Perry et al. (Stanford, 2022). 47 participants did five security-related programming tasks, one group with an AI assistant and one without. The group with AI wrote less secure code on four of the five tasks, yet believed more strongly that their code was secure. People who trusted the AI less and rephrased their prompts more had fewer vulnerabilities. The study used an older model and a small group, so read it as a sign, not a final conclusion.

You can picture a loop: AI-written code looks tidy and runs, so the writer feels more confident, so reviews less, so more flaws slip through. The loop starts with trust, not with the AI.

7.4 Five common situations (illustrations)#

The situations below are rebuilt from common mistakes and do not belong to any specific incident. Each shows how the mechanisms from sections 2 to 5 appear at work.

What all five share: the result looks complete, so it does not trigger the instinct to check. AI's confident tone and tidy formatting lower the reader's guard.

---

8. When AI is more reliable, and when it is less#

Rule of thumb: the level of verification should scale with the consequence of being wrong and the difficulty of noticing the error, not with how convincing the answer sounds.

---

9. A seven-step "conditional trust" routine#

1. Classify the consequence before asking. Naming a variable is low consequence. Code going to production, replying to customers, calculating money, legal advice, and any action on real data are high consequence.
2. Separate the generator from the checker. Do not ask the same AI that wrote it to confirm it is right. Use tests, linters, the compiler, a reviewer, or an independent source.
3. Give deterministic work to deterministic tools. Adding, filtering, exact matching, looking up versions: write code and run it, do not ask in words.
4. Demand sources and open them yourself. Ask for the document name, link, line number, verbatim quote. Then open it. A link given by AI also needs checking that it exists.
5. Limit permissions. Separate development from production, grant read-only access where possible, require human confirmation for destructive actions, and keep backups you have tested restoring. This is the lesson of the Replit incident.
6. Check by asking again. Ask neutrally instead of leading. Invert the question ("find three bugs in this" rather than "is this right"). Ask twice in two separate sessions and compare. Allow the AI to say "I don't know".
7. Name the person responsible. Every product with AI involved still needs a human to sign for it. Mata v. Avianca and Air Canada both show that responsibility does not transfer to the AI.

A prompt template that reduces invention (but does not eliminate it):

Answer based on the document I provide below.
For each point, quote the supporting sentence verbatim in quotation marks.
If the document is not enough to conclude, say "not enough information"
and state what else is needed. Do not use outside knowledge.

---

10. A balanced view#

This article does not call for dropping AI. Models improve fast, invention rates fall with each generation, and there is a lot AI does better and faster than people. The METR figures are also just a snapshot from early 2025 and may have changed since.

What does not change is the structure of the problem. As long as a model generates text by probability, there will be moments when an answer sounds right and is wrong. The rate may fall, but no mechanism guarantees zero. So the scarce skill is not knowing how to ask AI, but knowing how to verify fast enough and cheaply enough to use AI safely. This is also the conclusion of The Dev Career Through the Eras, where the bottleneck of the profession moved toward verification.

---

11. Ten questions before trusting an AI answer#

1. If this is wrong, who bears the consequence and how badly?
2. Is there a cheap way to check right now (run it, look it up, cross-check)?
3. Does the answer contain figures, proper names, paths or quotations that I have not opened myself?
4. Did I ask neutrally, or did I hint at the answer I wanted?
5. Is this common knowledge, or a rare, new or internal detail?
6. Which part is calculation that could go to a tool instead of being asked in words?
7. Did I ask the same AI to check itself? Is a tool or another person checking?
8. Could any negation, condition, exception or ordering have been dropped?
9. Can this action be undone? Is there a backup?
10. Who will sign and take responsibility for the final result?

---

12. Conclusion#

AI is an extremely good word-guessing tool. It is built to produce the next sentence that looks most right, and thanks to that it writes, summarizes, translates and programs at a level that is astonishing. The same way of working can explain why it invents case law, policies and libraries, miscalculates, and explains incorrectly how it calculated.

Understanding the mechanism helps us avoid two extremes: trusting absolutely because an answer is so smooth, or rejecting it entirely because it was wrong once. Instead, treat AI like a colleague who is very fast and very confident but has never been penalized for being wrong: hand over the work, then check in proportion to the consequences, and keep the responsibility on your side.

---

References#

Figures come from public sources as of October 2026. Some numbers are reported slightly differently by different outlets, so open the original sources to check before quoting.

• Mata v. Avianca, Inc.: Wikipedia and LawNext on the sanctions
• Moffatt v. Air Canada, 2024 BCCRT 149: Manatt summary
• The Replit incident: eWeek and Heise
• Kalai et al., Why Language Models Hallucinate (2025)
• Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools and Stanford HAI summary
• Anthropic, Tracing the thoughts of a large language model
• Mirzadeh et al., GSM-Symbolic (Apple, 2024)
• Sharma et al., Towards Understanding Sycophancy in Language Models (ICLR 2024)
• Pearce et al., Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions (2021)
• Perry et al., Do Users Write More Insecure Code with AI Assistants? (Stanford, 2022)
• Anthropic, Reasoning models don't always say what they think (2025)
• METR, trial with experienced open-source developers (summary by Simon Willison)
• Stack Overflow, Developer Survey 2025 (secondary coverage)
• Research on slopsquatting: Dark Reading and Infosecurity Magazine