--- title: "One Wrong Command, Four Hours: The Amazon S3 Story of 2017" url: https://www.thinnd.site/en/stories/one-wrong-command-four-hours-amazon-s3-2017/ language: en author: "Nhu Duy Thin (Thinnd)" published: 2026-10-29 updated: 2026-10-29 section: "Stories" tags: ["incident", "Amazon S3", "AWS", "postmortem", "operations tooling", "blast radius"] summary: "On 28 February 2017, while debugging a slow billing process in the US-EAST-1 region, an authorized member of the S3 team ran a command to remove a small number of servers from a subsystem, but a mistyped input removed more servers than intended, including S3's index and placement subsystems. Those two had to be fully restarted, something AWS said it had not done completely in its larger regions for many years, so S3 returned to normal only at 1:54 PM Pacific time, about four hours seventeen minutes after it began. AWS changed the tool to remove capacity more slowly and to block any removal that would drop a subsystem below its minimum required capacity." license: "Cite with the author name and the canonical URL above." --- # One Wrong Command, Four Hours: The Amazon S3 Story of 2017 > This is a **true story, but not my own**. I retell it from the **post-event summary AWS itself published** (linked at the end, which I read directly). The summary does not name the person who ran the command, so I do not name them and do not guess what they thought. > > This post differs from the [GitLab 2017](https://www.thinnd.site/en/stories/the-day-an-engineer-deleted-the-wrong-data/) one in one way: it is not a story about broken backups but about **a tool that lets one mistyped input remove far too much**. Times in the post are Pacific time (PST), as in the summary. --- ## Quick summary - On **28 February 2017**, at **9:37 AM**, an authorized member of the S3 team ran a command to remove **a small number of servers** from a billing-related subsystem. **One mistyped input** caused **more servers than intended** to be removed. - The removed servers supported two of S3's core systems: **the index** and **placement**. Both had to be **fully restarted**. - S3 in US-EAST-1 **returned to normal only at 1:54 PM**, about **four hours seventeen minutes** later. - Worth noting: **the command did what it was built to do**; nothing stopped it from removing too much. Even **AWS's own service health dashboard** depended on S3. - AWS changed the tool to **remove more slowly** and **never remove below a minimum**. --- ## Hard words, plain meaning | Term | Plain meaning | |---|---| | **Amazon S3** | AWS's cloud file storage service, used by a great many websites and apps. | | **Region** | A cluster of data centers in one area. US-EAST-1 is a large AWS region. | | **Subsystem** | A part inside a service that does one job. | | **Index subsystem** | Holds information about each stored object: where it is. | | **Placement subsystem** | Picks where new objects are stored. Depends on the index. | | **Blast radius** | How much surrounding stuff one failure can break. | | **Playbook** | Prepared steps for an operational task. | --- ## 1. An ordinary operational task That morning the S3 team was **debugging a slow billing process** in US-EAST-1. An authorized team member followed **an existing playbook**: remove a small number of servers from a billing-related subsystem to see whether it would help. There was nothing odd about it. Removing servers from a subsystem is everyday work for an operator, and there was a playbook for it. That is an important point: **this was an allowed action, following a procedure, by a person with the right to do it**. --- ## 2. 9:37 AM: one mistyped input According to the summary, the command ran at **9:37**, and **one of its inputs was entered incorrectly**. The result: **more servers than intended were removed**. The summary does not say which input, how it was wrong or why. I do not guess. What can be said with confidence is that **the tool accepted the value and carried it out**, with no checkpoint asking "how many will this remove, and is that safe". The removed servers **belonged not to the billing system but to two core systems**: - **The index:** holds the metadata and location information for every object in the region. Needed for get, list, put and delete requests. - **Placement:** allocates storage for new objects on put requests. Depends on the index working properly. Removing that much capacity meant **both had to be fully restarted**. --- ## 3. The restart: something not done for years This is the detail I find most worth thinking about in the summary. AWS wrote that it **had not completely restarted** these systems in its larger regions **for many years**. In that time S3 had **grown enormously**, and the **safety checks that validate metadata integrity**, needed for a safe restart, made it **take longer than expected**. In other words: **the ability to recover had never been tested at today's scale.** It echoes GitLab's lesson: a backup never test-restored is unknown, and a restart procedure never run at real size does not have a known duration. --- ## 4. The timeline All times Pacific, 28 February 2017, region US-EAST-1. | Time | What happened | |---|---| | **9:37** | The command runs, more servers than intended are removed | | **11:37** | The service health dashboard can update individual service status again | | **12:26** | The index subsystem has enough capacity to serve get, list and delete requests | | **1:18 PM** | The index subsystem is fully recovered | | **1:54 PM** | The placement subsystem finishes recovering, S3 operates normally | From 9:37 to 1:54 PM is **about 4 hours 17 minutes**. The summary does not give the durations of each restart on its own. --- ## 5. Who was affected, and an ironic detail about the dashboard According to the summary, affected services included the S3 console, **launching new virtual machines (EC2)**, **EBS volumes** needing data from snapshots stored in S3, and **AWS Lambda**. Some services built up a backlog during the disruption and needed extra time to recover. **The ironic detail:** the admin console of AWS's **service health dashboard** also **depended on S3**. Until 11:37 it could not update individual service status, so AWS had to use **its Twitter account and banner text** to communicate. A general lesson (my inference): **the tool that reports incidents should not live on the very thing that is broken.** --- ## 6. What turned one wrong command into a big incident From the summary, I see five layers that together turned one mistyped input into four hours of disruption. *This reading is my own.* | Layer | What the summary shows | |---|---| | **A tool that allowed removing too much** | One mistyped input removed more than intended, with no minimum to stop it. AWS later added this safeguard | | **Removing too fast** | AWS later changed the tool to remove more slowly | | **A large blast radius** | Two core systems of a whole region, with many other services depending on them. AWS later partitioned into independent cells | | **Recovery untested at real scale** | The restart was slower than expected because it had not been fully done in years | | **A reporting tool depending on the broken thing** | The dashboard used S3. AWS later changed it to run across multiple regions | The common thread: **each layer is reasonable on its own**, and only when they meet do they become an incident. --- ## 7. What AWS fixed According to the summary: - **Remove more slowly.** The capacity-removal tool was changed to remove at a lower pace. - **A minimum.** A safeguard was added to block any removal that would push a subsystem below its **minimum required capacity level**. - **Audit other tools.** AWS is checking its other operational tools for similar safety checks. - **Reduce blast radius and shorten recovery.** Partition subsystems into independent cells. Partitioning of the index subsystem, planned for later in the year, was moved up to start immediately. - **A dashboard running across multiple regions.** Overall, AWS **did not focus on forbidding anyone from mistyping**, but on **making a mistyped input unable to cause great harm**. --- ## 8. Compared with the GitLab story The two share the "one wrong command" motif, but the lessons sit in different places: | | GitLab (Jan 2017) | Amazon S3 (Feb 2017) | |---|---|---| | The wrong command | Wiped the data directory on the primary instead of the secondary | One mistyped input removed more servers than intended | | The main weakness | Backups unusable, data lost | A tool with no minimum, a large blast radius, a slow restart | | Public handling | A shared document, a live stream, a long postmortem | A post-event summary and tool changes | | The lesson | Recovery must be tested | A tool with power to delete needs safeguards and a minimum | Together: **make the wrong command harder to type, and make its consequence smaller.** --- ## 9. What I take from it: questions for a tool that can remove *This part is my suggestion, not AWS's conclusion.* If you write or use a tool that can **remove, delete or shut down**, ask: 1. **Is there a minimum?** Does the tool block any action that would push the system below a safe level? 2. **Is there a rate limit?** Removing slowly gives the operator time to notice a mistake. 3. **Is there a preview?** A list-only mode: "this will remove N machines" before removing. 4. **What is the blast radius?** At most how much can one wrong command break? 5. **Has recovery been tested at real scale?** When did you last restart the whole system? 6. **Does the incident-reporting tool live independently?** 7. **Can a second person see the dangerous action?** --- ## 10. When AI can run tools This story happened in 2017, but the motif is very modern: **a tool with the power to remove takes a wrong input and nothing stops it.** With an AI agent, the wrong input may come from a steered prompt, and the agent does not "feel something is off". That is **excessive agency (LLM06)** in the [OWASP Top 10 for AI apps](https://www.thinnd.site/en/engineering/owasp-top-10-for-ai-apps-everyday-examples/): excess functionality, excess permissions, excess autonomy. And the mitigations there match AWS's lessons: limit functions, least privilege, require approval for high-impact actions. The post [What is MCP](https://www.thinnd.site/en/engineering/what-is-mcp-power-socket/) talks about fuses, and S3's minimum is exactly a fuse. --- ## 11. A balanced view - **This rests on one main source.** I rely on AWS's own summary, which is written from AWS's point of view. - **The summary omits many details.** No name, no specific mistyped value, no separate durations for each restart, so I do not guess. - **The comparison and the "five layers" table are my reading.** - **The incident is nearly ten years old.** The systems have changed a lot, but the causes (no minimum, a large blast radius, untested recovery, a tool depending on the broken thing) still feel very familiar. - **I do not know how the person who ran the command felt**, so the post does not describe it. --- ## 12. Conclusion One mistyped input, four hours, and part of the internet slowed down. But what AWS chose to fix was not "tell operators not to mistype". They fixed **the tool** so a wrong command does not remove too much, and **the system** so the consequence is smaller. That is the way of thinking most worth learning. A small step today: find a tool you use that can remove or delete, and ask whether it has a **minimum** and a **preview mode**. If not, add it to your to-do list. --- ## References - [AWS: Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region](https://aws.amazon.com/message/41926/) - Related posts in this blog: [The day an engineer deleted the wrong data: the GitLab 2017 story](https://www.thinnd.site/en/stories/the-day-an-engineer-deleted-the-wrong-data/), [OWASP Top 10 for AI apps](https://www.thinnd.site/en/engineering/owasp-top-10-for-ai-apps-everyday-examples/), [What is MCP? Explained with a power socket](https://www.thinnd.site/en/engineering/what-is-mcp-power-socket/)