This interview with an AWS leader isn’t aging well, from CBS Sunday morning:

Pogue asked, "I don't mean to give anyone ideas, but let's say I figured out that one of these unmarked buildings was an AWS data center, and I blew it up. Are you saying that it's so backed up and redundant that you probably wouldn't notice?" Wood replied, "Yeah, you wouldn't notice. I mean, we might be a bit upset, but you wouldn't notice!"

https://www.cbsnews.com/news/cloud-computing-loudoun-county-...

I think this is due to the data residency requirements in UAE. I'm working with a client in the health space and the government requirements requires me to store data only in UAE! Tried with AWS but they were not allowing any new instances and I had to go with Azure.

When doing Disaster Recovery (DR) planning as a SRE in the New York City area, I sometimes use the phrase "Hurricane Sandy 2" to describe an event so big that it knocks out the power/compute/etc for an entire region.

I may just start using AWS Bahrain 2 as an additional example.

11.3 Force Majeure. Except for payment obligations, neither party nor any of their affiliates will be liable for any delay or failure to perform any obligation under this Agreement where the delay or failure results from any cause beyond its reasonable control, including acts of God, labor disputes or other industrial disturbances, electrical or power outages, utilities or other telecommunications failures, earthquake, storms or other elements of nature, blockages, embargoes, riots, acts or orders of government, acts of terrorism, or war.

The data center layout should look something like this:

    me (Middle East)
    ├── me-south-1 (Bahrain) DOWN since 2026-04
    │   ├── mes1-az1 DOWN
    │   │   └── mes1-mct1-az1 (Oman, Muscat) ???
    │   ├── mes1-az2 DOWN since 2026-03-01
    │   └── mes1-az3 DOWN
    ├── me-central-1 (United Arab Emirates)
    │   ├── mec1-az1
    │   ├── mec1-az2 DOWN since 2026-03-01
    │   └── mec1-az3         DOWN since 2026-03-01
    └── il-central-1 (Israel, Tel Aviv)
        ├── ilc1-az1
        ├── ilc1-az2
        └── ilc1-az3

Not sure about the Muscat local zone, whole me-south-1 region has been reported down despite Muscat still being operational.

If someone had told me a year ago that a whole AWS region could go down I'd called them crazy, but now me is close to exactly that happening.

See also previous discussion: https://news.ycombinator.com/item?id=49033240

They guaranteed 11 9's durability, didn't they?

e: Yep

https://aws.amazon.com/s3/storage-classes/

I think people are overthinking this. I'm the first one to criticize AWS (I run a competitor, carolinacloud.io) but I don't think we should hold it against them when they lost data due to their hard drives being physically bombed.

Ex AWS. AWS had one zone specific products. If those were in the affected AZ that data is lost. AWS has multiple multi-AZ products. S3 and EFS are good examples. If you loose an AZ for an extended periodic of time, the dataplane of those services will migrate shards around until you get back to your durability goals.

No disaster recovery plan? No offsite backups? Someone failed to applied the most basic principles that have existed for decades.

Since roughly zero of the articles I've seen link back to the source of their quotes, here it is: <https://status.aws.amazon.com/#multipleservices-me-south-1_1...>

Does this mean that even with 3 availability zones for Amazon S3 storage, that some data is lost?

I wonder if this will cause a mini cyber insurance crisis. I don't think any of those data loss plans have been tested at scale.

Well… they have a good excuse.

My personal bet is there's an 80% chance this is caused by some internal bootstrapping problem that they've messed up. AIUI all the main cloud vendors are in trouble here. The automation project I work on is expressly designed to help folks solve this DR/bootstrapping problem. Soo many people get this wrong. Of course missiles don't help things, but I'd bet AWS is primarily to blame here. I'd love an actual technical report of why they can't recover things.

I wonder if they'll start adding an underground bunker to new data centers so you can put an S3 replica there?

   me (Middle East)
   ├── me-south-1 (Bahrain)                                DOWN since 2026-04
   │   ├── mes1-az1                            me (Middle East)

├── me-south-1 (Bahrain) DOWN since 2026-04 │ ├── mes1-az1 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3 DOWN │ │ └── mes1-mct1-az1 (Oman, Muscat) │ ├── mes1-az2 DOWN since 2026-03-01 │ └── mes1-az3 DOWN ├── me-central-1 (United Arab Emirates) │ ├── mec1-az1 │ ├── mec1-az2 DOWN since 2026-03-01 │ └── mec1-az3 DOWN since 2026-03-01 └── il-central-1 (Israel, Tel Aviv) ├── ilc1-az1 ├── ilc1-az2 └── ilc1-az3

They say "some" data, i wonder what percentage that really is. I haven't seen pictures but I find it hard to imagine all of me-south-1 was completely leveled to the point where's there's just nothing left. On the other hand, if you have 100 rows of racks and then randomly take out a contiguous 10% across both rows and columns it may be functionally equivalent to taking out everything.

They can recover most of it? I assumed that those AZs had been offline for so long because they were like gone gone.

Ahhh the downside of “data sovereignty” rules and the de-globalization meme strikes again. I’d bet a bajillion dollars the reason this happened is due to government thinking it’s a good idea to make it illegal to store data outside their country.

Hey EU, take note of this next time you create silly data residency requirements that don’t allow data to travel outside your region. Encryption is an easy solution to multi-region residency…as long as the European Commission doesn’t stupidly keep trying to make encryption illegal too!

Hint: Russia absolutely knows where your data centers are.

I suppose this is why Microsoft and Google want to build data centers in the Netherlands. Its not cheap but it is safe.

"..even if something happens only once in a billion requests, that means it happens multiple times per day within S3."

but one in a trillion...

Seems like their multi-AZ redundancy didn't account for missile strikes. Good reminder to always have your own backups.

Guess those multi-AZ promises have an asterisk when actual missiles are involved. Makes you double-check your own off-site backups.

Hey but that's exactly as per design. It is the customer's responsibility to store stuff elsewhere as DR backup, not AWS.

There are many regulations over there which prevent data from leaving the country.

Sometimes those rules can have serious consequences.

I wonder what kind of "some data" is: military installations in Arab states and Israel?

What a nightmare scenario to tabletop. How do you even begin to recover from something like this?

Stuff hosted in overpriced cloud has actually no backups ? Unbelievable!

If all your backups are with AWS you don't have backups at all.

S3 has 11 9s of durability

(excluding the geopolitical incidents)

Backup both locally and everywhere no matter what.

What AZs were affected?

I don’t think I’ve seen them say?

Everything not saved will be lost.

Pretty soon we won’t need backups at all, just have AI regenerate all the data.

We collectively gave this company trillions over the years and they still can’t get it right.

Uh.

Uh-oh.

It's not clear from their messaging if multiple availability zones were severely damaged, or if the damage to one availability zone was simply more than they planned for. If it's the latter, that's a big uh-oh.

The wording certainly seems very careful:

"The damage to our infrastructure spanned multiple availability zones and exceeded what our regional and multi-AZ services are designed to withstand"

us-east-1 is finally looking pretty stable for once.

50%+ of companies that lose all of their data go out of business in 6 months.

DR/BCP costs are readily justified by doing a Business Impact Analysis (BIA).. budget up to some fraction of risk cost * risk probability.

And a friendly reminder that replication isn't a tested data backup.

undefined

I'm sorry but your data is in another castle.

It was always a bad bet for billionaires like Bezos to become Trump enablers. You weren't buying a seat at the table, or the privilege of being left alone, you were just signing yourself up to be force-fed shit sandwiches over and over (And the shit-to-bread ratio gets worse as time goes on)

You should have used your considerable resources to fight. If only billionaires would oppose aspiring tyrants with the same zeal with which they oppose even minor tax increases.

[dead]

[dead]

[dead]

[flagged]

[flagged]

So many AWS simps in this thread that don't actually understand how AWS scrimped out and fucked their customers. You can't all suck up to AWS at the same time, you need to serialize.

This is the flipside of data residency requirements that countries are now starting to require. If the EU wants to keep data in the EU, then great, but when the war comes and energy and infrastructure are hit, people would have wished for backups in North America, Asia, and the middle east.