---
title: "The Day an Engineer Deleted the Wrong Data: The GitLab 2017 Story"
url: https://www.thinnd.site/en/stories/the-day-an-engineer-deleted-the-wrong-data/
language: en
author: "Nhu Duy Thin (Thinnd)"
published: 2026-10-29
updated: 2026-10-29
section: "Stories"
tags: ["incident", "backups", "deleted by mistake", "GitLab", "postmortem", "operations"]
summary: "On 31 January 2017, while rebuilding the secondary of GitLab.com's database, an engineer ran a command that wipes the data directory on the primary instead of the secondary and lost about 300 GB before stopping it. When they looked for backups, none of the layers was usable, so GitLab restored from a snapshot about six hours old, took about 18 hours, and lost data for at least 5,000 projects, 5,000 comments and about 700 accounts. GitLab's main lesson: a backup you have never restored is not yet a backup, and the mistyped command was only the last link in a chain of process failures."
license: "Cite with the author name and the canonical URL above."
---

# The Day an Engineer Deleted the Wrong Data: The GitLab 2017 Story

> This is a **true story, but not my own**. I retell it from the postmortem GitLab published on 10 February 2017 (linked at the end). I do not name the engineer: GitLab's first working notes did, and the company said it would redact names in later incidents.
>
> I do not know what that person thought or felt, so the article does not describe it as fact. Where something is my own inference or suggestion, I say so.

---

## Quick summary

- On 31 January 2017, while rebuilding the secondary of GitLab.com's database, an engineer ran a delete command on the **primary** instead of the secondary. About **300 GB** was removed before it was stopped, a second or two later.
- Then **the backup layers failed or proved unusable, one by one**. The only recoverable copy was a snapshot the same engineer happened to take by hand about 6 hours earlier.
- The service was **down about 18 hours**. GitLab estimated at least **5,000 projects, 5,000 comments and about 700 new accounts** lost changes. Git repositories and wikis were not affected.
- The mistyped command was only the **last link**. Before it came a load spike, an undocumented manual procedure and backups failing without anyone knowing.
- GitLab was **open about everything**: a shared document, a live stream of the recovery and a long postmortem. That is worth learning from as much as the technical lessons.

---

## Hard words, plain meaning

| Term | Plain meaning |
|---|---|
| **Primary** | The database machine in live use, receiving every user change. |
| **Secondary** | A machine that copies the primary in real time, to take over if it fails. |
| **Replication** | The secondary continuously receives change records from the primary to stay identical. |
| **Backup** | A copy of data for recovering after an incident. Not the same as replication. |
| **Snapshot** | A capture of a disk's state at one moment. |
| **Disaster recovery** | Getting the service back after serious data loss. |
| **Postmortem** | A written account of an incident: what happened, why, what will be fixed. |
| **Blameless** | An analysis style that looks at systems and processes, not at someone to blame. |

---

## 1. An ordinary day that starts going wrong

All times below are UTC on 31 January 2017.

**About 17:20.** An engineer takes a snapshot of the production database disk and loads it into a test environment, to load-test a connection balancer. The system normally did this automatically every day at 01:00. This manual snapshot would end up saving the night, though nobody knew it yet.

**About 19:00.** Database load spikes. The team suspects spam, and many people cannot post comments. Part of the load comes from a background process removing data of a GitLab employee, whose account had been mistakenly flagged for abuse and scheduled for removal. According to GitLab, the abuse reporting system made it too easy to overlook details.

**About 23:00.** The secondary slows and stops replicating: it needs change logs the primary has already cleaned up, and GitLab.com did not archive them for reuse at the time. The only way is to **rebuild the secondary from scratch by hand**.

The team starts. The rebuild tool hangs silently with no message. They adjust settings, then PostgreSQL refuses to restart, and they adjust more. Anyone who has done operations knows this kind of night: one small job drags in many small jobs, and each step looks reasonable.

---

## 2. About 23:30: one command, on the wrong machine

To rebuild replication you must wipe the data directory of the **secondary** and load it again. According to GitLab's postmortem, the engineer ran that wipe on the **primary**.

They stopped it after about a second or two. But **about 300 GB** of data was already gone.

I have no way of knowing what went through that person's head, and I will not guess. What can be said with confidence is that after the incident GitLab added one very plain item to its to-do list: **make the shell prompt show clearly which machine and environment you are on**. That item alone shows they saw telling the two machines apart as not clear enough at the time.

---

## 3. Looking for a backup: five layers, none usable

Right after, they went looking for backups. This part makes the story a lesson:

| Layer of protection | What happened |
|---|---|
| **Daily backup (pg_dump)** | Ran with the wrong tool version against the real database, so it failed. The cloud storage was empty. Failure notices went by email but were rejected for lack of email authentication setup, so **nobody knew**. |
| **Cloud disk snapshots** | Enabled for file servers but **not** for database servers, on the assumption the other layers were enough. |
| **LVM snapshots** | Taken every 24 hours, built to copy data to staging, **not for disaster recovery**. Only one manual one about 6 hours old. |
| **Replication to the secondary** | Built for **failover**, not disaster recovery. And the secondary had just been wiped to rebuild it. |
| **Copy in the staging environment** | Held the data, but on slow disks in another region. |

Two things stand out. **First**, each layer failed for a different reason, and each reason sounds reasonable. **Second**, according to GitLab, backups had **no owner**, so nobody regularly checked whether they could be restored.

---

## 4. Recovery: 18 hours of copying

The team chose the manual snapshot from 6 hours earlier, because it lost less data than the 24-hour option.

- The data sat on slow staging disks, and copying it back to production took **about 18 hours**. The disks were throttled at roughly 60 Mbps, and the disks were the bottleneck.
- The database was restored to its state at **17:20 UTC** on 31 January.
- **17:00 UTC on 1 February**: GitLab.com ran again, temporarily without webhooks. Webhooks were rebuilt from a separate extract.
- The database's ID counters were **increased by 100,000** to avoid reusing IDs issued before the outage.
- Final checks finished around 18:00 UTC that day.

**The damage.** GitLab estimated at least **5,000 projects, 5,000 comments and about 700 new accounts** lost changes made after about 17:20. (The postmortem gives the end of the window as about 23:30 in the damage section and 00:00 in the introduction, so do not treat that boundary as exact.) Projects created after 17:20 were restored from their Git repositories only, without issues or merge requests. Git repositories and wikis were unaffected because they are stored separately, and self-managed customers were not affected either.

---

## 5. How GitLab handled it in public

This is the part I found most worth learning, beyond the technical side.

- The team kept **a shared public document** logging each step.
- They **streamed the recovery live on YouTube**, peaking at about 5,000 viewers, and posted updates on Twitter.
- They published **a long postmortem** just ten days later, with a list of about **15 follow-up tasks**.
- They acknowledged that the early document named the engineer, and **promised to redact names in later incidents**.

This approach is often called **blameless**: treat one person's mistake as a sign the system allowed it, and ask "how do we make sure nobody can slip like this again" instead of "who slipped".

---

## 6. The mistyped command was only the last link

Unpacked from the postmortem, one wrong command sits behind a whole chain:

1. A load spike from spam and a mistakenly running data-removal job.
2. The secondary stopped replicating because the system did not archive its change logs.
3. A manual, poorly documented rebuild procedure, with a tool that hangs silently.
4. The daily backup failing quietly, its notices blocked.
5. Cloud snapshots not enabled for the database.
6. Nobody owning backups and test restores.
7. Two easily confused machines, with a prompt that did not tell them apart.

In that chain, **mistyping** is perhaps the hardest link to prevent at the personal level and the easiest to prevent at the system level.

---

## 7. What GitLab fixed

GitLab's public action list included items such as:

- **Clearer machine and environment in the shell prompt.**
- **Backup monitoring** with a public dashboard.
- **Hourly LVM snapshots** of production databases (GitLab reported this as done).
- **Cloud disk snapshots** for database servers.
- **Automated test restores of backups.**
- **Point-in-time recovery** through continuous log archiving.
- **Documentation and runbooks** for rebuilding replication.
- **Assigning an owner for data durability.**

Notably, GitLab said it was focusing on **recoverability** and **clear identification of machines**, rather than only trying to block dangerous commands. It treated a mistyped command as something that will happen again, and wanted its consequences to be small.

---

## 8. What I take from it: the minute before you type a delete command

*This part is my own suggestion, not GitLab's conclusion.*

1. **Which machine are you on?** Read the host name on the prompt, do not assume. If two machines look alike, color the prompt for the real one.
2. **Which directory are you in?** Print the current path before deleting.
3. **Run a list-only version first.** See what it would delete before deleting for real.
4. **Does the backup really exist?** Not "probably", but you have restored it once and seen the data come back.
5. **A second person looks at the screen.** For production data, one more pair of eyes is cheap.
6. **Do not do it when exhausted or mid-patch.** Stop, rest, then act. I infer this from general experience; GitLab's postmortem does not say that engineer was tired.
7. **Have a plan for the first minute after a mistake.** Stop the command, tell someone at once, do not try to fix it quietly. Stopping the command after a second or two here saved the rest.

---

## 9. When AI can run commands

This story happened in 2017, before AI agents that run commands for people. But it hints at something: **every AI agent with delete permission is a command-typer who never tires and also never feels that "something is off"**. The post [Why you should not trust AI blindly](https://www.thinnd.site/en/engineering/why-not-to-trust-ai-blindly/) mentions another public incident, and [What is MCP](https://www.thinnd.site/en/engineering/what-is-mcp-power-socket/) covers fuses for AI: least privilege, start read-only, a person approving sensitive work. Those principles are GitLab's lessons applied to a new kind of command-typer.

---

## 10. A balanced view

- **This rests on one main source.** I rely mostly on GitLab's own postmortem. Some details differ between its sections (for example the end of the data-loss window).
- **I do not know what the engineer felt.** The article does not describe feelings as fact.
- **The incident is nearly ten years old.** Tools have changed, but the causes (backups never restored, no owner, confusing machines) still feel very familiar.
- **"Blameless" does not mean no accountability.** It moves responsibility from one person to the system and the people who design it.

---

## 11. Conclusion

One engineer mistyped one command. But what made the story painful is that all five layers of backups were unusable exactly when needed. The lesson sits in two places: **make the wrong command harder to type**, and **make its consequences small when it happens anyway**.

A small step today: pick a backup you currently trust and **test restoring it this week**. If you have never tried, you do not have a backup, you have hope.

If you have a story of deleting data by mistake yourself, write it down by answering seven questions: when it happened, what you did just before, what was lost, how you noticed, how you recovered, what changed afterward, and what you want others to take from it.

---

## References

- [Postmortem of database outage of January 31 (GitLab, 10 February 2017)](https://about.gitlab.com/blog/postmortem-of-database-outage-of-january-31/)
- [Postmortem of GitLab Database Outage of January 31 (Linux.com)](https://www.linux.com/?p=474688)
- Related posts in this blog: [Why you should not trust AI blindly](https://www.thinnd.site/en/engineering/why-not-to-trust-ai-blindly/), [What is MCP? Explained with a power socket](https://www.thinnd.site/en/engineering/what-is-mcp-power-socket/), [Three hours with an AI agent on a real project](https://www.thinnd.site/en/field-notes/agent-field-notes-real-project/)