Hacker news

  • Top
  • New
  • Past
  • Ask
  • Show
  • Jobs

Microsoft blames monster email outage on software defect introduced in July fix (https://www.theregister.com)

446 points by Bender 4 days ago | 355 comments | View on ycombinator

mike_hearn 4 days ago |

No big surprises here. Big outages at big firms seem to always look like this.

The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.

This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.

It's the sort of thing that requires a bit of preparation because naive ways of doing it like just randomly dropping 90% of requests will yield a broken service for everyone. Ideally your LB can drop traffic by modulus of the session cookie or similar, so users are let back in atomically.

Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.

sinuhe69 3 days ago |

No one has asked the obvious question: how much AI contributed to the epic failure by Microsoft. Was the patch programmed and tested by AI? Not because they are inherently bad but it begs the question of how much do the remaining humans at MS understand the patch? And did using AI contribute to the atrophy of knowledge and skills, especially deep, specific skills such as network failure inspection, load balancing and fail over recovery, that led to a slower response and recovery time?

Without a functioning intuition, discovering the causes to initiate recovery effort in a high-stress, unfamiliar situation with catastrophic dimensions could be significantly hampered.

They are valid questions and of particular interest in the age of AI. Productive, highly specialized skills could degrade faster than one imagine once we start rely too much on external helps.

andix 3 days ago |

There is one thing I don't get with this issue. The problem was basically that their system couldn't handle the normal load. Why couldn't they just reduce the load at some point just by dropping/limiting connections?

For example just accept connections from specific (random) IP ranges. They could even cycle through different ranges every 5 minutes to give clients a chance to sync at least once every hour.

The reduced load could be used to gradually bring up servers again without failing, add additional capacity, and gradually restoring access. Completely without fixing the bug at all. The system managed to handle usual load even with the bug, it just hit some critical point that made it collapse. Adding more resources would also fix that.

mindwok 4 days ago |

The biggest surprise here is that they're using Windows for critical network infrastructure. I'm surprised a Windows process has any role here outside the Exchange servers themselves.

nashashmi 4 days ago |

toss1 4 days ago |

>> "the processes that led to an update with a latent defect reaching production deserve a long, hard look. "

Indeed they do deserve a long hard look.

Toss1's conjecture:

We are now seeing the beginning of an era of latent technical debt brought about by too rapidly firing developers and switching to AI code generation (nevermind vibe-coding). Even well-prompted and well-monitored by top expert coders and architects, there is latent technical debt, and it is more diffuse, harder to find, and harder to fix, and will start showing up in outages like this.

(It first seemed the ATC outage was another example, but it now seems it was merely the usual management incompetence if not an outright manipulative ploy to 'justify' more spending for a single-bid contract [0])

[0] https://ajsignalnotnoise.substack.com/p/faa-air-traffic-cont...

ncr100 4 days ago |

They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.

Is this commonly acceptable? I'd imagine it's a costly trade-off?

malkia 3 days ago |

Lock contention, especially cache line contention in large NUMA systems. If hardware was upgraded to NUMA machines, or VM's were not cordoned each to sit in each it's own NUMA node (CPU, Caches & Memory) then something like this would happen.

We saw with our own tools back when an artist were given a low-core "server" like machine with many CPUs (good we thought), and 2 NUMA nodes, and suddely the level editor for our game was running much slower. We could blame std::shared_ptr<> all day long, or similar but it's there and it was real.

Tried the 4kb interleaved paging, but then everything was equally slow. At the end we simply blocked the App to run only on one NUMA node (still using memory from the other one), and things were back to "normal". "Normal", because promises were not kept - we could not use all CPUs.

colonCapitalDee 4 days ago |

Seems like a pretty standard big company severe outage to me. These things happen!

hmokiguess 4 days ago |

> The engineers who worked feverishly to restore services deserve commendation and will hopefully never have to buy their own beverages in Redmond again (or at least escape the next round of firings).

What does this mean? I didn't understand

k0morebi 4 days ago |

Not a network engineer so genuinely curious how these things are handled at the epic scale of 365, azure, aws and so on.

Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.

d_g_m 3 days ago |

> "Our load testing tended to focus on weaker hardware with fewer cores....The contention issue is worsened on machines with more cores and newer hardware."

Does anyone know why the situation got worse on better machines? Normally it is the other way around when it comes to test coverage of hardware specs

syngrog66 4 days ago |

broadstrokes: bitten by concurrency, contention, thundering herd, and a proximate cause which was not in fact the root cause and the root cause had been lurking latent for several months so hard to undo surgically fast. combined with insufficient load/release testing and insufficient mitigation in prod.

smallstepforman 3 days ago |

Monoculture is still a problem, from biology to markets to computers. People look at me puzzingly when they hear I run Haiku, that I write my own rendering engine, that I’m building a small country estate.

wduquette 4 days ago |

So maybe we shouldn’t be putting all our eggs in one basket?

kozikow 3 days ago |

Of course it's easy to say "haha AI slop".

Stuff like this also happened prior to AI. One of reasons it happens more frequently now is just now timelines of product development are compressed. So let's say we get 3x outages, but at 10x the speed of new features.

To me it sounds like Outlook team do not have proper gradual rollout for service of this size. Like rollout the new version to 10% servers, let it soak, compare metrics before rolling up higher. Most of "bigtechs" have something like this nowadays.

nelaggy 4 days ago |

what role did ai play in all of this?

can we blame it for the bug? can we credit it for the fix?

might be the most important part of the full report

cyberax 4 days ago |

So apparently this is the report: https://www.365fanatic.com/wp-content/uploads/2026/11/PostIn... I propose a "latent bug" drinking game.

So to recap:

1. They did not have proper cell-based architecture to partition their system into failure domains ("In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure.").

2. They did not have proper monitoring to see the encroaching collapse.

3. Stuff just keeps falling off: "Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day"

sandworm101 4 days ago |

>> "an update that returned the code path to the previously known good state in Windows."

I know I shouldn't be surprised, but I still struggle to imagine such high-capacity servers running with windows. Part of me wonders why any important server would be running windows. And I do chuckle at the "update" that returned windows to a previous state. Never admit a rollback. Never admit a previous update was less than perfect. Hold the line MS. Hold that line.

feverzsj 4 days ago |

Poor Microsoft can't afford more than one low-end testing environment.

devy 4 days ago |

Interesting that it's their EVP of Copilot, Agents, and Platform at Microsoft, building the next generation of AI-powered tools for work who spoke up for he post-mortem instead of their CTO.

Does that mean Microsoft's internal code are 100% outsourced to agents / AI and human review just won't be able to catch latent bugs like this?

ZiiS 4 days ago |

- we added a bug (yes, everyone does).

- our testing didn't cover it (again, this is expected occasionally).

- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).

- it took 15 hours to deploy the fix (not good, but at this scale maybe).

Feels like they are completely missing the point of where the problem is and how to fix it?

harrouet 3 days ago |

All we need to know is if an AI was involved in this fiasco.

tenderfault 4 days ago |

This happens when you are trying to do weird things with emails rather than just deliver them to destination.

Also this is karma because Microsoft blocked in ignorance entire datacenter networks making SMBs unable to deliver messages to their networks.

Ta-ta!

pluc 4 days ago |

Nothing more cuts and more AI won't fix!

vessenes 4 days ago |

Pretty interesting that testing was aimed the worry: low spec boxes, and missed a different problem: concurrency contention for high spec boxes. Gotta be covered by someone's rule of testing, I imagine. Also, woof.

SillyUsername 3 days ago |

Well it was about time, long suffering customers see outages like this caused by these bugs regularly.

Perhaps Microsoft will learn from this now they see the direct financial cost themselves of sloppy code.

chriscjcj 3 days ago |

Perhaps it should now be called "Microsoft 364 services."

hn_acc1 3 days ago |

There has long been this sci-fi / tease of a concept called "an unrestartable system". Let's hope they didn't build one of those :-)

dogscatstrees 4 days ago |

> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

Looks at my Windows Task Manager...

undefined 4 days ago |

undefined

spprashant 4 days ago |

These kinds of incidents should inject healthy dose of techno-skepticism in the larger conversation. The world is teetering on the edge of turning over all our processes to LLMs and retiring human labor for good.

But at the end of the day these are servers strung together sending packets over copper wires. They are extremely reliant till they are not.

When crisis strikes you can only hope there is a squad of well trained humans who can step in to untangle these messes. But as more of the labor force is stripped out of responsibilities, the longer these outages will take to resolve. Imagine being jolted out of your year-long UBI-funded vacation to login and resolve a multi-continent server outage that you only have a vague memory of.

zahlman 3 days ago |

I wonder how long until the Kevin Fang video?

jdlyga 3 days ago |

I thought we solved coding

undefined 3 days ago |

undefined

ChrisArchitect 4 days ago |

This MS rando on X just leaking his 'abridged' version instead of letting the official account post the report/blog later or whatever? Gotta be 'first' right bro, and now the source of every rabid tech blog for the rest of the day. bah

CrzyLngPwd 4 days ago |

Yeah, but was the defect written by a clanker?

AraceliHarker 3 days ago |

[dead]

pojzon 3 days ago |

Tl;dr they did a roll-forward-rollback