Parent poster most likely works at Amazon. When the rate of purchases falls below a threshold, many engineers across the company get paged to find the issue. As you can imagine a news event would decrease activity on Amazon for a while.
So OnCall is mandatory for Amazon engineers and you will literally get paged just because some stat drops, not that it goes to 0? And there's no human in the loop for a basic sanity check? Sounds awful
> And there's no human in the loop for a basic sanity check?
I mean, I'm not going to go to bat for Amazon's worker practices, but... how would you get a human in the loop for a basic sanity check without someone getting paged?
Like, you have to alert someone so that they can do a sanity check.
You'd have a separate 24/7 ops teams instead of every single team being first-responders for their own things. This team would be watching the dashboards and alarms with some first-level investigation/triage playbooks so that if all that's needed is "press this button to reboot the server" or "it's a holiday, or the national team just won the world cup, this is a drop in traffic cause everyone is outside" then nobody else has to get paged.
I don't know which model Amazon uses myself, but it sounds more like the "everyone is their own first responder" model which surprises me a little, but each model has its pros and cons. The justification I'm most familiar with is "if there's a separate ops team you're going to be complacent and throw shit over the wall without caring about quality" which is a somewhat lazy excuse for not actually getting your engineers to care about quality with anything but a hammer to beat them on the head with, but ....
There are both models at Amazon. Individual teams have their own on-call; if for example an RDS instance is continually failing to boot, someone on-call (edit: from the RDS team) will get paged to look at it.
I believe order drops are looked at by dedicated reliability team, because it's such a significant problem. But they don't watch graphs because... why would they? They set up the machines to page them when metrics aren't within an expected range. Which is what happened.
The ops teams I've worked with watched graphs because (a) sometimes you see interesting changes or trends before they turn into actionable alerts and people can look into them/figure them out before shit's on fire, and (b) it makes it look more like you're doing something when everything is smooth sailing and not actively on fire, solving the "wait why are we paying for this if it doesn't take us long to respond to incidents anyway" naive-meddlesome-exec problem.
That someone was me when I used to be on call as a Fab engineer. I took responsibility and my team relied on my alertness when shit hit the fan a couple of times during my career. Yes, someone has to do it. The negativity and cynicism on HN astounds me.
I find the grandstanding and They-ification nauseating and possibly mastubatory.
It is possible to have someone who can say "oh there's a world shaking event in Japan" before paging engineers.
In fact, wouldn't this be an obvious efficiency improvement at world-wide scale?
It is also possible someone believes that is possible without them being a "negative cynic".
In fact, it's unclear why having this idea for an efficiency improvement would make one a "negative cynic"
I suggest coming with curiosity and asking why people believe something rather than working towards explaining what it says about people if they actually believe exactly what your mindreading says they do.
"Some stat" here translates to millions, maybe tens of millions of dollars every minute at Amazon scale.
If you were running a business and suddenly started losing a million dollars a minute, wouldn't you like to have someone investigate why that's happening?
