Amazon S3 Outage
Amazon (via Sam 北島-Kimbrel, Hacker News):
The Amazon Simple Storage Service (S3) team was debugging an issue causing the S3 billing system to progress more slowly than expected. At 9:37AM PST, an authorized S3 team member using an established playbook executed a command which was intended to remove a small number of servers for one of the S3 subsystems that is used by the S3 billing process. Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended. The servers that were inadvertently removed supported two other S3 subsystems. One of these subsystems, the index subsystem, manages the metadata and location information of all S3 objects in the region. This subsystem is necessary to serve all GET, LIST, PUT, and DELETE requests. The second subsystem, the placement subsystem, manages allocation of new storage and requires the index subsystem to be functioning properly to correctly operate. The placement subsystem is used during PUT requests to allocate storage for new objects. Removing a significant portion of the capacity caused each of these systems to require a full restart. While these subsystems were being restarted, S3 was unable to service requests.
[…]
While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years. S3 has experienced massive growth over the last several years and the process of restarting these services and running the necessary safety checks to validate the integrity of the metadata took longer than expected.
[…]
From the beginning of this event until 11:37AM PST, we were unable to update the individual services’ status on the AWS Service Health Dashboard (SHD) because of a dependency the SHD administration console has on Amazon S3.
Amazon Web Services (Hacker News):
The dashboard not changing color is related to S3 issue. See the banner at the top of the dashboard for updates.
Jim Dowling (via Hacker News):
Aside from the outage, there are many limitations of working with S3 that make it a less than ideal long term storage technology, and most of its problems relate to S3 object replication and metadata. S3 is a an eventually consistent key-value store for objects. However, eventual consistency tells us nothing about what guarantees S3 provides.
[…]
Netflix does not trust the metadata provided by S3. They have replaced it with their own metadata service, s3mper, which is essentially an eventually consistent key-value store that stores a copy of the metadata in S3 [s3mper]. Netflix rewrote their applications to account for s3mper. In the diagram below, you can see that application programming becomes more complex. Creating an object in S3 becomes a write to DynamoDB and a create operation in S3. This is not done transactionally. All S3 read/list operations need to be re-written to query DynamoDB and S3 and compare the results.
For me, the S3 outage brought down part of my FastSpring store, and a bunch of serial number reminder e-mails and crash reports didn’t go out because Amazon SES kept failing. My server code had assumed that sending e-mails would always succeed. In fact, it relied on sending e-mails to myself in order to report errors with the site and store. I’ve since added SparkPost and FastMail as backup SMTP providers.
I also plan to store e-mails in a database until they’ve been successfully sent. This seemed like it would be really easy, but I ran into a weird issue with my database layer not saving, and I haven’t had time yet to track that down.
3 Comments RSS · Twitter
Rather than storing your emails in a database, it might be simpler to setup a MTA like Postfix on your server, and have it relay all email to SES / SparkPost / FastMail. Then your application can connect to the MTA running locally on your server (via SMTP) and send email. If your external email provider goes down, then your local MTA will queue the email and keep trying until it’s back up.
See this for more:
https://sendgrid.com/docs/Integrate/Mail_Servers/postfix.html
@Jeff That is a good point. I realize I would kind of be reinventing the wheel, but it seemed like the database way would be simpler for my purposes and have fewer dependencies.
@Jeff The database problem was fixed by changing the default storage engine from InnoDB to MyISAM. I’m not sure why this was an issue—maybe because I’m using an old version of SQLObject.
