The Day an Engineer Deleted the Wrong Data: The GitLab 2017 Story

This is a true story, but not my own. I retell it from the postmortem GitLab published on 10 February 2017 (linked at the end). I do not name the engineer: GitLab's first working notes did, and the company said it would redact names in later incidents.

I do not know what that person thought or felt, so the article does not describe it as fact. Where something is my own inference or suggestion, I say so.

---

Quick summary#

• On 31 January 2017, while rebuilding the secondary of GitLab.com's database, an engineer ran a delete command on the primary instead of the secondary. About 300 GB was removed before it was stopped, a second or two later.
• Then the backup layers failed or proved unusable, one by one. The only recoverable copy was a snapshot the same engineer happened to take by hand about 6 hours earlier.
• The service was down about 18 hours. GitLab estimated at least 5,000 projects, 5,000 comments and about 700 new accounts lost changes. Git repositories and wikis were not affected.
• The mistyped command was only the last link. Before it came a load spike, an undocumented manual procedure and backups failing without anyone knowing.
• GitLab was open about everything: a shared document, a live stream of the recovery and a long postmortem. That is worth learning from as much as the technical lessons.

---

Hard words, plain meaning#

---

1. An ordinary day that starts going wrong#

All times below are UTC on 31 January 2017.

About 17:20. An engineer takes a snapshot of the production database disk and loads it into a test environment, to load-test a connection balancer. The system normally did this automatically every day at 01:00. This manual snapshot would end up saving the night, though nobody knew it yet.

About 19:00. Database load spikes. The team suspects spam, and many people cannot post comments. Part of the load comes from a background process removing data of a GitLab employee, whose account had been mistakenly flagged for abuse and scheduled for removal. According to GitLab, the abuse reporting system made it too easy to overlook details.

About 23:00. The secondary slows and stops replicating: it needs change logs the primary has already cleaned up, and GitLab.com did not archive them for reuse at the time. The only way is to rebuild the secondary from scratch by hand.

The team starts. The rebuild tool hangs silently with no message. They adjust settings, then PostgreSQL refuses to restart, and they adjust more. Anyone who has done operations knows this kind of night: one small job drags in many small jobs, and each step looks reasonable.

---

2. About 23:30: one command, on the wrong machine#

To rebuild replication you must wipe the data directory of the secondary and load it again. According to GitLab's postmortem, the engineer ran that wipe on the primary.

They stopped it after about a second or two. But about 300 GB of data was already gone.

I have no way of knowing what went through that person's head, and I will not guess. What can be said with confidence is that after the incident GitLab added one very plain item to its to-do list: make the shell prompt show clearly which machine and environment you are on. That item alone shows they saw telling the two machines apart as not clear enough at the time.

---

3. Looking for a backup: five layers, none usable#

Right after, they went looking for backups. This part makes the story a lesson:

Two things stand out. First, each layer failed for a different reason, and each reason sounds reasonable. Second, according to GitLab, backups had no owner, so nobody regularly checked whether they could be restored.

---

4. Recovery: 18 hours of copying#

The team chose the manual snapshot from 6 hours earlier, because it lost less data than the 24-hour option.

• The data sat on slow staging disks, and copying it back to production took about 18 hours. The disks were throttled at roughly 60 Mbps, and the disks were the bottleneck.
• The database was restored to its state at 17:20 UTC on 31 January.
• 17:00 UTC on 1 February: GitLab.com ran again, temporarily without webhooks. Webhooks were rebuilt from a separate extract.
• The database's ID counters were increased by 100,000 to avoid reusing IDs issued before the outage.
• Final checks finished around 18:00 UTC that day.

The damage. GitLab estimated at least 5,000 projects, 5,000 comments and about 700 new accounts lost changes made after about 17:20. (The postmortem gives the end of the window as about 23:30 in the damage section and 00:00 in the introduction, so do not treat that boundary as exact.) Projects created after 17:20 were restored from their Git repositories only, without issues or merge requests. Git repositories and wikis were unaffected because they are stored separately, and self-managed customers were not affected either.

---

5. How GitLab handled it in public#

This is the part I found most worth learning, beyond the technical side.

• The team kept a shared public document logging each step.
• They streamed the recovery live on YouTube, peaking at about 5,000 viewers, and posted updates on Twitter.
• They published a long postmortem just ten days later, with a list of about 15 follow-up tasks.
• They acknowledged that the early document named the engineer, and promised to redact names in later incidents.

This approach is often called blameless: treat one person's mistake as a sign the system allowed it, and ask "how do we make sure nobody can slip like this again" instead of "who slipped".

---

6. The mistyped command was only the last link#

Unpacked from the postmortem, one wrong command sits behind a whole chain:

1. A load spike from spam and a mistakenly running data-removal job.
2. The secondary stopped replicating because the system did not archive its change logs.
3. A manual, poorly documented rebuild procedure, with a tool that hangs silently.
4. The daily backup failing quietly, its notices blocked.
5. Cloud snapshots not enabled for the database.
6. Nobody owning backups and test restores.
7. Two easily confused machines, with a prompt that did not tell them apart.

In that chain, mistyping is perhaps the hardest link to prevent at the personal level and the easiest to prevent at the system level.

---

7. What GitLab fixed#

GitLab's public action list included items such as:

• Clearer machine and environment in the shell prompt.
• Backup monitoring with a public dashboard.
• Hourly LVM snapshots of production databases (GitLab reported this as done).
• Cloud disk snapshots for database servers.
• Automated test restores of backups.
• Point-in-time recovery through continuous log archiving.
• Documentation and runbooks for rebuilding replication.
• Assigning an owner for data durability.

Notably, GitLab said it was focusing on recoverability and clear identification of machines, rather than only trying to block dangerous commands. It treated a mistyped command as something that will happen again, and wanted its consequences to be small.

---

8. What I take from it: the minute before you type a delete command#

This part is my own suggestion, not GitLab's conclusion.

1. Which machine are you on? Read the host name on the prompt, do not assume. If two machines look alike, color the prompt for the real one.
2. Which directory are you in? Print the current path before deleting.
3. Run a list-only version first. See what it would delete before deleting for real.
4. Does the backup really exist? Not "probably", but you have restored it once and seen the data come back.
5. A second person looks at the screen. For production data, one more pair of eyes is cheap.
6. Do not do it when exhausted or mid-patch. Stop, rest, then act. I infer this from general experience; GitLab's postmortem does not say that engineer was tired.
7. Have a plan for the first minute after a mistake. Stop the command, tell someone at once, do not try to fix it quietly. Stopping the command after a second or two here saved the rest.

---

9. When AI can run commands#

This story happened in 2017, before AI agents that run commands for people. But it hints at something: every AI agent with delete permission is a command-typer who never tires and also never feels that "something is off". The post Why you should not trust AI blindly mentions another public incident, and What is MCP covers fuses for AI: least privilege, start read-only, a person approving sensitive work. Those principles are GitLab's lessons applied to a new kind of command-typer.

---

10. A balanced view#

• This rests on one main source. I rely mostly on GitLab's own postmortem. Some details differ between its sections (for example the end of the data-loss window).
• I do not know what the engineer felt. The article does not describe feelings as fact.
• The incident is nearly ten years old. Tools have changed, but the causes (backups never restored, no owner, confusing machines) still feel very familiar.
• "Blameless" does not mean no accountability. It moves responsibility from one person to the system and the people who design it.

---

11. Conclusion#

One engineer mistyped one command. But what made the story painful is that all five layers of backups were unusable exactly when needed. The lesson sits in two places: make the wrong command harder to type, and make its consequences small when it happens anyway.

A small step today: pick a backup you currently trust and test restoring it this week. If you have never tried, you do not have a backup, you have hope.

If you have a story of deleting data by mistake yourself, write it down by answering seven questions: when it happened, what you did just before, what was lost, how you noticed, how you recovered, what changed afterward, and what you want others to take from it.

---

References#

• Postmortem of database outage of January 31 (GitLab, 10 February 2017)
• Postmortem of GitLab Database Outage of January 31 (Linux.com)
• Related posts in this blog: Why you should not trust AI blindly, What is MCP? Explained with a power socket, Three hours with an AI agent on a real project