144 points by surprisetalk 3 days ago | 115 comments | View on ycombinator
proxysna 3 days ago |
hdgvhicv 3 days ago |
This attitude of modern tech claiming 98% is good just doesn’t work in the old tech acceptance. We had individual components fail all the time. We’re still looking at a 230ms outage to a branch office last week caused by a power failure combined with a badly plumbed power distribution.
Modern software people don’t consider 230ms to be an outage. Glad they don’t work in electricity.
(The failure we had was only on the services we guarentee at 99.1%, our lowest sla. After that there’s 99.95 and 99.999.
(In reality we reach five nines year after year on even the lowest levels, but there are major concerns like “large bomb in data centre” which could cause some of our less critical units to drop way more than 5 minutes a year.
teraflop 3 days ago |
Also, 0.1% downtime in the form of a 45-minute outage per month is very different from 0.1% of requests failing in brief bursts. You often see downtime reported as "increased error rates" which is so vague as to be meaningless.
Google's "windowed user-uptime" attempts to deal with this a bit better, by exposing different views of the data instead of trying to condense uptime into a single number: https://www.usenix.org/system/files/nsdi20-paper-hauer.pdf
hx8 3 days ago |
> GitHub Actions: 12 hours affected in the last 30 days (98.31% uptime).
This is trying to shine the most favorable possible light onto a deteriorating situation. It doesn't take away from the fact that most businesses have measurable missed revenue in downtime. Customers that shop somewhere else, ads that were never severed, leads that grew a little colder. 12 hours of downed GitHub results in millions of dollars of lost developer productivity that was externalized by Microsoft to other companies.
We shouldn't be trying to spin downtime as "just a few hours a month." Those hours cost real dollars.
blakeashleyjr about 9 hours ago |
runjake 3 days ago |
PaulKeeble 3 days ago |
One thing I have noted over time is a lot of these AWS, Azure et el downtimes is they occur in the middle of everyones day, millions of people are impacted by them. Same with github its getting in the way of work. Whereas when we hosted services on our own equipment the downtime was usually out of main hours. The percentages are in many ways the wrong measure of downtime because hours aren't equal in impact to businesses.
procflora 3 days ago |
In the electric utility world we have a few IEEE standardized metrics (with appropriately IEEE'd acronyms) for tracking service reliability that I like much better and always wish for when I'm looking at a status page. Pie in the sky stuff for sure, nobody wants to do this analysis and publish the results without a regulator telling them have to, but c'est la vie.
SAIDI - System Average Interruption Duration Index. How many minutes an average customer experienced service interruption in a year. This is the big one I'd want to see on your service status page IMHO. For the power grid, we consider any outage longer than five minutes to be an interruption ("non-momentary outage").
SAIFI - System Average Interruption Frequency Index. How many total periods of interruption occurred for the average customer in a year.
CAIDI - Customer Average Interruption Duration Index. How long it takes service to be restored for the average customer when there is an interruption.
For the US, here is what these numbers look like: https://www.eia.gov/electricity/annual/html/epa_11_03.html. If you're outside the US look up yours and have a good laugh at us. :)
For the "right now" aspect you have probably visited your utility's outage map, but here I would say we do much better than most utilities. The level of detail on the investigation and resolution is often more detailed, and we usually know better than to bother providing much in the way of a concrete estimate for restoration time of a current outage (though this is getting better in the utility space).
jakevoytko 3 days ago |
If you do something 100 times a day against a four-nines service, you can reasonably expect that everything will succeed.
If you do something 10,000 times a day against a two-nines service, you can expect to hit a substantial number of errors during that day, or even have long periods where your work cannot happen at all.
People aren't frustrated with Github because Github has 98% uptime or whatever the specific number is. They're frustrated because it regularly interferes with their ability to work. The 98% number is just a concise way to say it.
dmurray 3 days ago |
Three nines reliability is great for most purposes. 8 hours downtime a year.
If your system produces money at a constant rate, it captures 99.9% of the available money. Even two nines or one nine might be pretty good on that basis, when the alternative is spending 2x or 10x as much - let's build another unreliable system with that money that captures some other independent market opportunity.
Poor reliability is a problem where you need to chain many systems together, or where the cost of a single failure is very large compared to a success. Or - as happens commonly because of load - if your periods of unreliability are correlated with periods of maximum opportunity, like an e-commerce site failing on Black Friday or a trading system failing when the market is most busy. But if you don't have one of those cases, evaluate whether investing in reliability is actually worth it to you.
GitHub is an example where two nines of reliability ought to be OK. The argument against it is that it's bad marketing to have an unreliable service, especially one aimed at software engineers. And if GitHub is largely a marketing play by Microsoft anyway (do they really make back its cost in enterprise subscriptions?) then marketing considerations need to drive its reliability.
jerf 3 days ago |
$ python3
Python 3.12.3 (main, Aug 31 2026, 10:18:26) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import math
>>> def nines(num):
... return -math.log10(1-num)
...
>>> nines(.9)
1.0
>>> nines(.99)
1.9999999999999996
>>> nines(.999)
2.9999999999999996
(Modulo floating point issues of course.)Which then smoothly covers the entire space:
>>> nines(.9321)
1.1681302257194985
>>> nines(.2)
0.09691001300805639
But good luck getting that standardized.lukevmorris 3 days ago |
There are projects like The Missing GitHub Status Page [2] that attempt a more accurate reporting, but taking 98.31% for granted, I think the more impactful framing would be _a floor_ of 1.5 business days lost every month. At 22 business days per month, that can feel more like 7+% downtime.
sandeepkd 3 days ago |
1. These are core to the (service level agreements) SLA's for the enterprise companies when they evaluate and sign the contract (must for government RFPs)
2. The contracts generally have provisions for payback or penalties for missed SLAs
3. When your product depends on any of the products directly then the availability of your product has to take into account the availability of the dependent services. Eg, I can easily sell a SLA of two nines for my service if I am building it on a platform which has SLA of three nines, other way around is always questionable (though possible)
4. All of this is what used to happen may be up until 5 years ago. As of today even with good number of outages, company like AWS has not updated their availability to reflect those outages. Ideally all the companies built on top of AWS infrastructure would have to update their availabilities in a cascading manner but some how everyone decided to skip the beat
5. At this point, calculation of the availability itself has become so opaque that no one understands it anymore so no one questions it either
pluc 3 days ago |
Point the AI at your logs, tell it to fix things, rinse and repeat. It's faster than debugging, and fast is good.
micromacrofoot 3 days ago |
These companies are happy that you don't know the difference between 99%, 99.9%, and 99.99% and that you think they all sound pretty good.
wavemode 3 days ago |
Mojah 3 days ago |
“It’s just a few minutes of downtime, nobody noticed, best not draw attention to ourselves by placing something on our status page”
Status pages should be tied into (external/simulated) monitoring to update in near real-time, without hiding incidents. Transparency is the only way. By reporting the _real_ hours of downtime, including the small stuff you think people didn’t notice.
jdauriemma 3 days ago |
cdkmoose 3 days ago |
One vendor in particular we deal with has a powerful feature which we use to a large extent. Unfortunately, that particular feature is all too often not working. The servers are up and the rest of the platform is working, but we need that feature, so if it's down, it doesn't help much that the rest of the platform is up.
legulere 3 days ago |
0xbadcafebee 3 days ago |
Which 12 hours? One block of 12 hours? Twelve blocks of 1 hours? Important hours or unimportant ones? Which hours are important to whom?
The audience for the status page is everybody, and everybody will want things their way. There is no pleasing everybody. That's why there's color. If you see a lot of red, that's bad. If there's not that much red, it's not that bad.
iLoveOncall 3 days ago |
The suggested format is equally unhelpful.
You can get 12 hours of downtime by being down once for 12 hours, or 144 times for 5 minutes. The user experience is VERY different in those two cases.
Ultimately the graphs are the most useful format.
wang_li 3 days ago |
Which is to say that the significance of downtime depends on the user. Talking about nines only makes sense internally when you are evaluating your infrastructure and operations. It doesn't tell you squat about impact to your customer.
stairlane 3 days ago |
For example we had a 6 9 (99.9999%) requirement from a customer for any given 3-6 month period. If we violated that, we owed them their money back (baring the outage wasn’t caused by us - I.e our cloud provider shit the bed).
That’s something like 7.5 seconds. For a contract over $1.5M. Am I the only one who thinks that’s outrageous expectations?
EDIT: The web app was for generating SBOMs of static assets.
denysvitali 3 days ago |
Keep the percentages, and regardless of that - GitHub fix your uptime
lucfranken 3 days ago |
Some measure quite detailled but some just don't summarize the downtime from all providers up and below their own platforms.
onion2k 3 days ago |
Downtime really matters if it's at a time you need something to be up, and Github is big enough to have users for that to be all the time. That moves the conversation from 'It's down for a few hours a month' to 'Github is failing a significant number of it's users'.
yieldcrv 3 days ago |
_russross 3 days ago |
mikewarot 3 days ago |
port3000 3 days ago |
karmakaze 3 days ago |
Similar for LLM measures from an ideal 1.0 mark.
Apocryphon 3 days ago |
swiftcoder 3 days ago |
charcircuit 3 days ago |
doublerabbit 3 days ago |
root@vixen:/fountain/crystals #
*** FINAL System shutdown message from dblrabbit@ ***
System going down IMMEDIATELY
System shutdown time has arrived
root@vixen:/fountain/crystals # uptime
3:05PM up 1931 days, 18:13, 0 users, load averages: 1.01, 1.03, 1.41
root@cookie:/srv/users/dblrabbit # uptime
3:07PM up 1931 days, 16:59, 1 user, load averages: 1.76, 1.17, 1.06
root@cookie:/srv/users/dblrabbit # poweroff
Shutdown NOW!
poweroff: [pid 47177]tyho 3 days ago |
stevenklein 3 days ago |
Madmallard 3 days ago |
feurio 3 days ago |
100%!
labithiotis 3 days ago |
yuye 3 days ago |
Also, is anyone else getting the bitter taste of AI writing from this page?
I am now _required_ to consult status page of github, circleci or MS services etc because i need to know why a build is not passing, why i cannot open a repo, why is my work stalling.
Percentages matter, it is just so much more obvious why they matter when it comes down to important pieces of the internet like github. And i highly doubt the number of 12 hours in the last month. MS has been downplaying the issues they have with GH performance for a while now and i don't think it is time to start to believe them yet. Maintaining these pieces of infrastructure is responsibility and a burden.
Overall i would be careful with "nonlinear significance of numbers near 100%" we are talking gh being well into the 90's this year and one number that infra people are also often being reminded about is that "1% is 3.5 days".
Things are tough for gh people and i feel for them but they are not a startup or a underdog of some sort to receive sympathy in that case.