Dev Chroniclesby Thinnd
Reflections

Who Is Accountable When AI Is Wrong?

When an AI tool answers wrongly or breaks something, who carries it? This reflection goes through two real cases (Air Canada and Mata v. Avianca), draws a simple map of responsibility, and suggests five things a team can do now so that nobody blames the machine.

Nhu Duy Thin (Thinnd)7 min read
  • #accountability
  • #AI
  • #professional ethics
  • #risk management
  • #reflection

Slide deck

Click the slides, then use the arrow keys to move. Press L to change language and F for full screen.

Open full screen

For programmers, product people and team leads bringing AI into their work. Hard terms get a short explanation.

The author is not a lawyer and this is not legal advice. The two cases are retold from public sources (a law firm's summary, news, an open encyclopedia), not from the original rulings. The "responsibility map" and the five things at the end are my opinion, not rules.

Quick summary#

  • Two real cases show that "the AI did it, not me" helps nobody.
  • The person who uses AI output and the party who puts AI in front of customers both have something to check.
  • Responsibility does not find an owner on its own. Put a name on it before an incident.
  • Five things a team can do now: one owner, checkpoints where risk is high, logs, a path to a person, blameless write-ups.

Hard words, plain meaning#

Term Plain meaning
Chatbot A program that chats automatically on a website or app.
Case law Earlier court decisions people cite to support an argument.
Civil resolution tribunal A body for small disputes, usually quicker and simpler than a regular court.
Log A record of what the AI received, answered and when.
Handover When the AI is unsure or the matter is important, passing it to a person.
Blameless Writing up an incident to find where the system had gaps, not to find someone to punish.

1. Case one: an airline's chatbot got it wrong#

According to a summary by the law firm Manatt, a passenger named Moffatt asked the chatbot on Air Canada's site about a bereavement fare. The chatbot said he could buy a regular ticket and apply for a refund of the difference within 90 days. But the airline's real policy did not allow refunds after travel. When he applied, the airline refused.

The airline argued, per that summary, that the chatbot was a separate legal entity and the airline was not responsible for its words. The civil resolution tribunal in British Columbia (Moffatt v. Air Canada, 2024 BCCRT 149) rejected the argument, finding that the airline owed a duty of care and had made a negligent misrepresentation. The airline had to refund the difference, a figure given as 650.88 Canadian dollars.

What is worth thinking about: the sum is small, but the principle is not. A customer has no reason to know which page of a website is "right" and which is "the chatbot talking nonsense".

2. Case two: lawyers filed case law an AI invented#

In Mata v. Avianca (a federal court in New York, 2023), the plaintiff's lawyers filed a brief citing several cases. When the other side could not find them, it turned out ChatGPT had invented them and the lawyers had not checked. On 22 June 2023 the judge fined them 5,000 dollars in total and remarked that they had "abandoned their responsibilities" (per Wikipedia, Bloomberg Law and Above the Law).

What I take from this case: a tool does not sign, the user does. The duty to verify does not disappear because a draft was written by a machine.

3. Two cases, one pattern#

Air Canada Mata v. Avianca
Who put the AI forward A company, to customers Lawyers, to a court
How the AI was wrong Misstated a policy Invented case law
Argument rejected "The chatbot is a separate entity" "I trusted the tool"
What was missing Nobody verified the chatbot's words Nobody checked the cases

Both lacked a named person holding the duty to verify at the last step.

4. A responsibility map (my opinion)#

When AI is wrong there are usually four parties, each with a part:

Party A reasonable share
The end user or employee using AI Check results before relying on or sending them, within their role
The team that builds and ships the AI Design the checks, limit the scope, provide a path to a person, measure quality
The organization and managers Name an owner, give time and tools to check, do not reward speed at all costs
The model provider Contract terms and documentation of limits (every contract differs, read yours)

I do not know what the "correct" share is before the law in any given place. This table is only a way of dividing work, so a team does not have everyone assuming someone else has it.

5. Five things a team can do now#

  1. Name one owner for each AI feature. A name, not "the team". That person knows what the feature does, how it is measured and how to switch it off.
  2. Put human checkpoints where risk is high. Money, health, legal matters, personal data, anything irreversible. Low-risk places (suggesting titles) can be looser. See Reviewing AI-written code and OWASP Top 10 for AI apps for checking and limiting permissions.
  3. Keep logs so you can trace what happened. When a complaint arrives you need to know what the AI said, to whom, when and from what source.
  4. Offer a path to a person, and tell customers. When the AI is unsure, or a customer says "I want a person", there must be a way. Say clearly when someone is talking to a machine, and do not let the machine's words sound like a commitment when they are not.
  5. Write up incidents without blaming individuals. Ask "how did the system let this happen", not "who did it". The stories of GitLab 2017 and Amazon S3 2017 are examples of telling an incident with a focus on mechanism. Why not to trust AI blindly covers why verification is needed.

6. Common traps#

  • "The AI did it, not me." The two cases above show this argument is weak.
  • Rubber-stamp review. The checker clicks approve hundreds of times a day without reading. A name on paper is not real checking.
  • Blaming the individual at the end of the chain. The person who pressed the last button is often just the last link in a system without controls.
  • Pushing everything to the vendor. A contract may limit their liability. Do not assume, read it.
  • Logging without protecting. Logs contain what customers said and must be kept as carefully as any personal data.
  • Naming an owner but not giving them the power to switch the feature off.

7. A balanced view#

  • Only two cases, through secondary sources. I read about them through a summary, news and an open encyclopedia, not the original rulings. Details may be missing.
  • Law differs by place. A decision of a tribunal in British Columbia does not automatically apply elsewhere.
  • Checking everything is unrealistic. That is why the article speaks of high risk. Where risk is low, little checking is fine.
  • The responsibility map is a way of dividing work, not of dividing fault before the law.
  • I may be looking at a skewed sample, because both cases are ones the press noticed. Many times AI is wrong and nobody sues.

8. Conclusion#

An AI tool can draft quickly, but it does not sign and it is not fined. Responsibility still falls on the people and the organization that put the result forward. What to do is name responsibility before an incident, put checks where mistakes are costly, and learn from incidents without hunting for someone to blame.

A small step today: pick an AI feature you run and ask "if it tells a customer something wrong tomorrow morning, who finds out first, and who decides to switch it off?" If you cannot answer yet, that is the first job.

References#

Frequently asked questions

If a company's chatbot says something wrong, is the company responsible?
In Moffatt v. Air Canada (2024), a tribunal in British Columbia rejected the argument that the chatbot was a separate entity and required the airline to compensate the customer. It is one case in one place, and law differs from place to place, but it shows that 'the chatbot said it, not us' is a weak argument.
If I use AI to draft a document and submit it, who is responsible?
The person who signs and submits. In Mata v. Avianca (2023), lawyers were fined 5,000 dollars for filing a brief with case law ChatGPT invented, unchecked. The judge said they had abandoned their responsibilities. A tool can help draft, but verification still belongs to whoever uses it.
Where should a small team start?
With five things: name one owner for each AI feature, put human checkpoints where risk is high, keep logs so you can trace what happened, offer a path to a person, and write up incidents without blaming individuals.
Is this article legal advice?
No. The author is not a lawyer. The article retells two cases from public sources and suggests ways of working. For a real legal situation, ask a lawyer where you operate.