644 points by DeepLogin 5 days ago | 282 comments | View on ycombinator
amluto 5 days ago |
N19PEDL2 5 days ago |
bketelsen 5 days ago |
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
Havoc 4 days ago |
* 13 guests complaining about irritation of airways (+2 later)
* 4 taken to hospital, 1 described as medium injury, rest discharged
* Dual failure - dosing system had a defect (dosing too high) and the email of the monitoring failed (M365). Article sounds like it's two distinct systems.
Overall the German text sounds a lot less dramatic than the hn title.
discordance 5 days ago |
Anything that could result in that sort of harm should have a more resilient and redundant architecture.
k310 5 days ago |
> 15 guests injured at Phantasialand
> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
MallocVoidstar 5 days ago |
> During the incident, some of the gas monitors installed within the Yenkin-Majestic facility detected the flammable solvent release approximately one minute after the release began and sent an email to an offsite employee, but the monitors did not sound audible alarms to warn employees of the hazard.
1 dead, 8 non-fatally injured.
labrador 5 days ago |
spaldingcactus 5 days ago |
dekhn 5 days ago |
Animats 4 days ago |
This is usually a problem for a hotel/motel too small to have a control room, but big enough to have escalators, elevators, HVAC, pools, laundry equipment, fire suppression, water heaters, and other gear that needs to be monitored. What usually happens is that some manager's office or security guard station has a bunch of unrelated alarm panels. That's where medium-sized problems get lost.
The next time you walk by a fire alarm panel in an office building, look to see if it's displaying any faults.
[1] https://pacificblueengineering.com/advanced-control-system-a...
0xbadcafebee 4 days ago |
It should be illegal to build a safety system that depends on someone reading an e-mail. But there's no regulation saying "no safety system should depend solely on someone reading an e-mail." There's no generic regulation covering software used for safety. A few select industries and specific cases have limited regulations; but mostly it's the wild west. So we end up with these tragic, preventable accidents.
Governments need to pass regulation about how software can be used in safety-critical applications. At minimum, software should pass a systems engineering safety review, use standard approved methods, and pass independent lab testing. Every other safety-critical device has to follow these requirements. Software should not get a free pass. You can see now what happens when they do.
This event is just one story. There are thousands more like this, buried in the news, or never reported at all. Nobody notices because software is everywhere. You see the symptom but not the cause. CrowdStrike's shitty software ended up forcing airlines to cancel 7,000 flights, causing $500M in damage. Two days ago, shitty software/systems (related to AI) grounded all flights in the USA. In the UK, the BPO Horizon Scandal caused deaths of dozens and ruined hundreds of lives. Regulation can put processes in place to prevent these things. Without it, the problems will continue; with AI involved, it's going to get worse.
People will continue to be harmed by badly designed, badly tested, unregulated software until we do something. Next time it might be your kids poisoned at a water park; by then it'll be too late to act.
verst 4 days ago |
Actual headline: "15 Gäste im Phantasialand verletzt" (15 guests injured in Phantasialand)
And the browser tab shows the following summary title "Chlor: Phantasialand bekam wegen Outlook nichts mit" (Chlorine: Phantasialand didn't notice anything due to Outlook)
gamblor956 5 days ago |
It appears that the hotel is trying to duck liability by latching onto the outage as the root cause even though this happened before the outage.
undefined 4 days ago |
ChrisRR 4 days ago |
The indication needs to be proportionate to the risk, and so if there is a high risk of harm, then the monitoring device needs a auditory and/or visual alarm as well as the email notification
mentalgear 5 days ago |
jasonm23 4 days ago |
sharts 4 days ago |
WalterBright 4 days ago |
ctrlaltdileep 5 days ago |
metalman 4 days ago |
rambojohnson 5 days ago |
Arech 4 days ago |
They are dangerous idiots, and no wonder that's a gross incompetence. Even more so, since they stupidly blamed MSFT instead of themselves.
I wonder what else could an audit uncover.
thr0waway001 5 days ago |
anjel 5 days ago |
lazzurs 5 days ago |
This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.
johnnyApplePRNG 5 days ago |
tahham 4 days ago |
zkmon 5 days ago |
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.
dlcarrier 5 days ago |
dangoodmanUT 4 days ago |
tchalla 5 days ago |
ErroneousBosh 5 days ago |
JoeAltmaier 4 days ago |
Story: early days of Windows, they invented a power-save mode. We updated our furnace controller PC with the lastest. The customer complained - it would stop regulating the furnace in the middle of the night shift, just sit there without moving the electrodes for hours.
Upon investigation we discovered (of course) that it went into power-save mode. That cost the customer, but more interestingly, considering a 30MW furnace ran for 8 hours without producing a result meant that it would have been better to have never invented the 'power saving' ecological feature that year - the savings for the nation was burned up in that 8 hours.
leni536 5 days ago |
lwansbrough 5 days ago |
teekert 4 days ago |
LeonB 4 days ago |
mococa 5 days ago |
rpigab 4 days ago |
They should have use a highly available Cloud-based Kubernetes-cluster on Talos Linux!
undefined 4 days ago |
sylware 4 days ago |
Who is in charge of all that???
creatonez 5 days ago |
tjwebbnorfolk 4 days ago |
pidgeon_lover 4 days ago |
jon128 5 days ago |
__m 4 days ago |
londons_explore 5 days ago |
joshu 4 days ago |
mmaunder 4 days ago |
crest 3 days ago |
Edit: the current German headline is far better, and the first paragraph puts the responsibility where it belongs.
MadrasThorn 5 days ago |
yukIttEft 5 days ago |
userbinator 5 days ago |
rawgabbit 5 days ago |
Use of Microsoft products and services confer no warranties. Written explicit warranties are for entertainment purposes only. Use at your risk. Use of Microsoft products are dangerous to your mental and emotional health. If you experience thoughts of deep frustration, call 1-800-MSCares. All inquiries will be promptly processed in the order they will be received or when the sole contractor can get to it.
kittikitti 5 days ago |
bendergarcia 5 days ago |
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.
refresh89347 5 days ago |
lifestyleguru 5 days ago |
These systems can easily fail, and they tend to fail in a way that causes them to measure low or zero chlorine and thus overdispense chlorine. More below on why.
First, this means that an operator must always ensure safety, both against excessive chlorine and against insufficient chlorine. In a commercial context, that means that a real human (or maybe a totally different sensor design, e.g. a colorimetric sensor) need to measure the pool chlorine level, preferably with a high quality titration (drop test) every day before opening the pool. They should probably record the result in pen on water resistant paper, but this part is icing on the cake.
Second, hoping that an email gets noticed is ridiculous. If you do not have positive confirmation that the pool is safe to open, don’t open it.
Third, the systems should be sized so that they cannot overdose quickly. The maximum dispensing rate should be such that hours of continuous operation would be required to reach an uncomfortable level, let alone a dangerous level. (Hot tubs are sort of an exception here.)
Fourth, everyone should be using a reasonable amount of CYA in their pool. Even at a lowish level of 30ppm CYA it takes a lot of chlorine to become dangerous unless the operator has also lost control of pH. I realize that some codes are confused about this.
Fifth, chlorine is smelly. Someone should have noticed, even just casually. Admittedly, lots of swimmers may be conditioned to stinky pools, and checking for safety isn’t their job. But the staff should notice.
Sixth, the systems I’ve seen beep when there’s an alert. It’s easy to add flashing lights, too. (In fact, the system I’m familiar with automatically shuts down if it feeds for more than X minutes without reaching the setpoint, and you cannot turn this feature off. You can, however, defeat it by switching the power to the whole controller.)
—- sensors —-
Actual free chlorine sensors intended for pools use an amperometric probe immersed in iodide solution or gel, separated from the pool water by a fancy membrane. Free chlorine oxidizes iodide to iodine, and the sensor measures the iodine concentration. If the membrane fails (or general wear and tear or excessive pressure swings) cause the iodide to be lost, then the chlorine can’t react and the sensor reads zero (or low if there’s tiny bit of iodide left).
Interestingly, these sensors are rather expensive devices that appear to mostly come from France and Germany. The Chinese vendors seem to be rebranding European parts (!), and the major US vendor is very clearly reselling a French company’s product.
There are some vendors who sell a cheap (or absurdly marked up) potentiometric ORP-based closed-loop system. In my considered opinion, these are worse than useless.