One Wrong Command, Four Hours: The Amazon S3 Story of 2017
On 28 February 2017, a command with one mistyped input removed more servers than intended from Amazon's S3 storage service and slowed or stopped many internet services for about four hours. This retelling follows AWS's own post-event summary and draws out the safeguards a tool with the power to remove should have.
This is a true story, but not my own. I retell it from the post-event summary AWS itself published (linked at the end, which I read directly). The summary does not name the person who ran the command, so I do not name them and do not guess what they thought.
This post differs from the GitLab 2017 one in one way: it is not a story about broken backups but about a tool that lets one mistyped input remove far too much. Times in the post are Pacific time (PST), as in the summary.
Quick summary#
- On 28 February 2017, at 9:37 AM, an authorized member of the S3 team ran a command to remove a small number of servers from a billing-related subsystem. One mistyped input caused more servers than intended to be removed.
- The removed servers supported two of S3's core systems: the index and placement. Both had to be fully restarted.
- S3 in US-EAST-1 returned to normal only at 1:54 PM, about four hours seventeen minutes later.
- Worth noting: the command did what it was built to do; nothing stopped it from removing too much. Even AWS's own service health dashboard depended on S3.
- AWS changed the tool to remove more slowly and never remove below a minimum.
Hard words, plain meaning#
| Term | Plain meaning |
|---|---|
| Amazon S3 | AWS's cloud file storage service, used by a great many websites and apps. |
| Region | A cluster of data centers in one area. US-EAST-1 is a large AWS region. |
| Subsystem | A part inside a service that does one job. |
| Index subsystem | Holds information about each stored object: where it is. |
| Placement subsystem | Picks where new objects are stored. Depends on the index. |
| Blast radius | How much surrounding stuff one failure can break. |
| Playbook | Prepared steps for an operational task. |
1. An ordinary operational task#
That morning the S3 team was debugging a slow billing process in US-EAST-1. An authorized team member followed an existing playbook: remove a small number of servers from a billing-related subsystem to see whether it would help.
There was nothing odd about it. Removing servers from a subsystem is everyday work for an operator, and there was a playbook for it. That is an important point: this was an allowed action, following a procedure, by a person with the right to do it.
2. 9:37 AM: one mistyped input#
According to the summary, the command ran at 9:37, and one of its inputs was entered incorrectly. The result: more servers than intended were removed.
The summary does not say which input, how it was wrong or why. I do not guess. What can be said with confidence is that the tool accepted the value and carried it out, with no checkpoint asking "how many will this remove, and is that safe".
The removed servers belonged not to the billing system but to two core systems:
- The index: holds the metadata and location information for every object in the region. Needed for get, list, put and delete requests.
- Placement: allocates storage for new objects on put requests. Depends on the index working properly.
Removing that much capacity meant both had to be fully restarted.
3. The restart: something not done for years#
This is the detail I find most worth thinking about in the summary. AWS wrote that it had not completely restarted these systems in its larger regions for many years. In that time S3 had grown enormously, and the safety checks that validate metadata integrity, needed for a safe restart, made it take longer than expected.
In other words: the ability to recover had never been tested at today's scale. It echoes GitLab's lesson: a backup never test-restored is unknown, and a restart procedure never run at real size does not have a known duration.
4. The timeline#
All times Pacific, 28 February 2017, region US-EAST-1.
| Time | What happened |
|---|---|
| 9:37 | The command runs, more servers than intended are removed |
| 11:37 | The service health dashboard can update individual service status again |
| 12:26 | The index subsystem has enough capacity to serve get, list and delete requests |
| 1:18 PM | The index subsystem is fully recovered |
| 1:54 PM | The placement subsystem finishes recovering, S3 operates normally |
From 9:37 to 1:54 PM is about 4 hours 17 minutes. The summary does not give the durations of each restart on its own.
5. Who was affected, and an ironic detail about the dashboard#
According to the summary, affected services included the S3 console, launching new virtual machines (EC2), EBS volumes needing data from snapshots stored in S3, and AWS Lambda. Some services built up a backlog during the disruption and needed extra time to recover.
The ironic detail: the admin console of AWS's service health dashboard also depended on S3. Until 11:37 it could not update individual service status, so AWS had to use its Twitter account and banner text to communicate.
A general lesson (my inference): the tool that reports incidents should not live on the very thing that is broken.
6. What turned one wrong command into a big incident#
From the summary, I see five layers that together turned one mistyped input into four hours of disruption. This reading is my own.
| Layer | What the summary shows |
|---|---|
| A tool that allowed removing too much | One mistyped input removed more than intended, with no minimum to stop it. AWS later added this safeguard |
| Removing too fast | AWS later changed the tool to remove more slowly |
| A large blast radius | Two core systems of a whole region, with many other services depending on them. AWS later partitioned into independent cells |
| Recovery untested at real scale | The restart was slower than expected because it had not been fully done in years |
| A reporting tool depending on the broken thing | The dashboard used S3. AWS later changed it to run across multiple regions |
The common thread: each layer is reasonable on its own, and only when they meet do they become an incident.
7. What AWS fixed#
According to the summary:
- Remove more slowly. The capacity-removal tool was changed to remove at a lower pace.
- A minimum. A safeguard was added to block any removal that would push a subsystem below its minimum required capacity level.
- Audit other tools. AWS is checking its other operational tools for similar safety checks.
- Reduce blast radius and shorten recovery. Partition subsystems into independent cells. Partitioning of the index subsystem, planned for later in the year, was moved up to start immediately.
- A dashboard running across multiple regions.
Overall, AWS did not focus on forbidding anyone from mistyping, but on making a mistyped input unable to cause great harm.
8. Compared with the GitLab story#
The two share the "one wrong command" motif, but the lessons sit in different places:
| GitLab (Jan 2017) | Amazon S3 (Feb 2017) | |
|---|---|---|
| The wrong command | Wiped the data directory on the primary instead of the secondary | One mistyped input removed more servers than intended |
| The main weakness | Backups unusable, data lost | A tool with no minimum, a large blast radius, a slow restart |
| Public handling | A shared document, a live stream, a long postmortem | A post-event summary and tool changes |
| The lesson | Recovery must be tested | A tool with power to delete needs safeguards and a minimum |
Together: make the wrong command harder to type, and make its consequence smaller.
9. What I take from it: questions for a tool that can remove#
This part is my suggestion, not AWS's conclusion. If you write or use a tool that can remove, delete or shut down, ask:
- Is there a minimum? Does the tool block any action that would push the system below a safe level?
- Is there a rate limit? Removing slowly gives the operator time to notice a mistake.
- Is there a preview? A list-only mode: "this will remove N machines" before removing.
- What is the blast radius? At most how much can one wrong command break?
- Has recovery been tested at real scale? When did you last restart the whole system?
- Does the incident-reporting tool live independently?
- Can a second person see the dangerous action?
10. When AI can run tools#
This story happened in 2017, but the motif is very modern: a tool with the power to remove takes a wrong input and nothing stops it. With an AI agent, the wrong input may come from a steered prompt, and the agent does not "feel something is off".
That is excessive agency (LLM06) in the OWASP Top 10 for AI apps: excess functionality, excess permissions, excess autonomy. And the mitigations there match AWS's lessons: limit functions, least privilege, require approval for high-impact actions. The post What is MCP talks about fuses, and S3's minimum is exactly a fuse.
11. A balanced view#
- This rests on one main source. I rely on AWS's own summary, which is written from AWS's point of view.
- The summary omits many details. No name, no specific mistyped value, no separate durations for each restart, so I do not guess.
- The comparison and the "five layers" table are my reading.
- The incident is nearly ten years old. The systems have changed a lot, but the causes (no minimum, a large blast radius, untested recovery, a tool depending on the broken thing) still feel very familiar.
- I do not know how the person who ran the command felt, so the post does not describe it.
12. Conclusion#
One mistyped input, four hours, and part of the internet slowed down. But what AWS chose to fix was not "tell operators not to mistype". They fixed the tool so a wrong command does not remove too much, and the system so the consequence is smaller. That is the way of thinking most worth learning.
A small step today: find a tool you use that can remove or delete, and ask whether it has a minimum and a preview mode. If not, add it to your to-do list.
References#
Frequently asked questions
- What happened to Amazon S3 on 28 February 2017?
- According to AWS's post-event summary, at 9:37 AM Pacific time an authorized member of the S3 team, debugging a slow billing process, ran a command to remove a small number of servers from a subsystem. A mistyped input caused more servers to be removed. S3 in US-EAST-1 returned to normal only at 1:54 PM, and many other services that depend on S3 were affected.
- Why did recovery take four hours?
- The removed servers supported two subsystems: the index (which manages information on every object in S3) and placement (which allocates storage for new objects). Removing that much capacity required both to be fully restarted. AWS said it had not completely restarted these systems in its larger regions for many years, and that S3's growth plus the safety checks that validate metadata integrity made the restart slower than expected.
- What did AWS fix afterward?
- The capacity-removal tool was changed to remove capacity more slowly. A safeguard now blocks any removal that would push a subsystem below its minimum required capacity. AWS is auditing its other operational tools for similar safety checks, partitioning subsystems into independent cells to reduce blast radius, and changed its service health dashboard to run across multiple regions.
- Was data lost in this incident?
- What I read in AWS's summary describes the service disruption and the recovery, and does not state loss of customer data. I do not claim anything beyond what the summary says.