This is a combination of posters not understanding what the technology is used for and the tech guy exaggerating the urgency of the situation.
Remote cardiac monitoring isn't for people who are imminently going to die of catastrophic heart attacks or who risk dropping into a fatal arrhythmia (wonky heart rhythm). In fact, there isn't even solid evidence that it saves lives. What it is used for is investigating patients who have vague symptoms which might be related to transient changes in their heart beat.
These are people who come to their doctor at the age of 54 and say "I felt my heart beating really quickly and felt kind of faint" or "I felt kind of dizzy and then passed out. It's happened twice in the last month." The more traditional way to investigate these patients are with what's called a Holter monitor, as alluded to by a previous poster. These are little belt packs you carry around while wired-up and that record your ECG for 24-48 hours at a time. The main weakness? You said it: the device only has a 24-48 hour windows to capture the weird, often rare, rhythm.
There are different ways remote cardiac monitoring systems report their results and I'm not sure which this particular company was using, but it doesn't really matter. Some of them only report weird stretches, others only report events when the patient says they're feeling symptom X, others are reporting continuously.
Take-home message: this is not life-or-death data. When doctors (at least those who are allowed to keep practicing) think a patient needs critical cardiac monitoring, they admit them to hospital.
This was a tech guy, looking to jump the queue by trying to raise a red flag because his servers were being used for--OMFG!--cardiac monitoring. A lot of my doctor buddies use similar strategies when they're caught speeding by the police. "I just got called in to the hospital!"
From the website I think this analysis is correct. Holters store the 24 hour data on the machine as well. The 12 leads are not continuous but set pieces which also store locally. If they were providing real-time monitoring with alerts then this would be more serious. This seems to be a backup system, ironically itself without backups.
b) Good to know from responses that amazon's paid support is crap. They'll officially tell you "were on it".
c) People are bashing this person for relying on amazon not failing all at once. I would be surprised if they hosted on amazon relying on each individual node being up 99.98% of the time. That is crazy. You build on cloud computing knowing that each one node can fail at any moment, but you build it to fall over to another node. From experience, sometimes nodes just... die... or get really unstable.
d) People bashing the dude vs offering future advice. I'm sure if his company is in deep shit, he's well aware of it.
This reminds me of doing tech support back during the dot bomb. Everyday we'd have day traders phone in complaining that their internet was down and they were losing thousands of dollars. Then I'd advise them to switch to their backup connection, then they'd say they didn't have one.
Finally, I'd ask why if they were running a business that could generate / lose thousands of dollars in a few minutes why they had chosen a residential internet service instead of opting for the business package and why they did not have a backup connection as neither the business packages or residential packages came with a SLA that was suitable for the type of risk inherent in their business.
If their data is so important why are they hosting it on the absolute cheapest and shittiest servers they can find without a backup solution in place? If they read the TOS for EC2 it probably says not to run critical infrastructure on it.
Yes, it costs more than EC2, but there's a reason for that. Wait until this company finds out that if their values exceed what MySQL expects it will silently truncate it.
We are a monitoring company and are monitoring hundreds of cardiac patients at home.
If this is a serious post and there really is a service that monitors cardiac patients from home, first you would have to deal with the unreliability of in-home broadband connections. What if a router needs to be reset, or the DSL goes out? Even if you ignore that, you would expect that any server setup will go down at some time. It has happened to Google, Facebook, Twitter, and now AWS and PlayStation Network. I remember when Rackspace went down a few years back (a truck crashed into the transformer that powered the datacenter - http://www.datacenterknowledge.com/archives/2007/11/13/truck...). All of Armenia was knocked offline when a woman cut a fiber optic line while digging for scrap metal. I'm not sure I'd trust the internet with any kind of life or death monitoring.
I could be way off base here since I only know what I saw on the site, but it doesn't look like this is life-or-death monitoring. It looks like its long term monitoring for later review by a doctor. It doesn't seem at all likely that they have someone monitoring the incoming data in realtime for heart attacks, that would be far better suited by hooking up the system to dial 911.
I think some IT person was freaking out that they had downtime and slightly exaggerated the life endangering part, they likely just lost information that may have been used in a future diagnostic manner.
If that is the case, they should have a local copy of that data at the patient's home, which syncs with the remote server periodically. This would protect against internet connection interruptions. In the event that the remote server is unreachable, it should failover to an alternate server in a different location that is reasonably in sync with the failed server.
Apparently it only has a 24-48 hour limit which lowers the likelihood of finding the super rare event that they are looking for. Still they should have a local cache of a few days that they can sync when amazon gets back up at the very least. Typically though you would want to have a separate datacenter to handle the emergency situations like this though and a smaller local cache for local Internet outages.
I don't think it's so much about trust as alternatives and best practices. Ideally you should not trust any system when running mission critical apps and always have multiple levels of backups and monitoring including humans on call.
Interesting quote from a company providing such services:
"The Remote Health Monitoring System is the IT backbone that supports HealthFrontier’s entire portfolio of solutions, including the ecg@home™ and the microtel™/ecgAnywhere™. It captures data transmitted by the patient through USB, Bluetooth, or trans-telephonically. The RHMS™ then stores the information in a database, which can be accessed by the patient’s doctor through a web-based interface. Benefits include:
Cloud computing: The RHMS is hosted on a network of servers across the US, which increases reliability and eliminates the need for a large capital investment in on-site hardware and maintenance."
How many other companies provide home ECG monitoring which report over the web? I'm looking but having a hard time finding more. I will update this post as I do.
[UPDATE]
Other Home-based ECG monitoring services:
My uncle owns a few elder care franchises and had me look into monitoring services as a funemployment project. So I've spent a bit more time with this than I'd like to admit...
The company I was most impressed with was http://www.halomonitoring.com . The device seems to be designed well and the monitoring services (online portal for caregivers or family members) are impressive, comparatively.
There's kind of a sea change happening with home health monitoring -- from the "I've fallen and can't get up" devices to more robust and interactive solutions. GE recently bought a company that provides institutional monitoring equipment and services.
This is very popular. There are several products that take ECG samples at timed intervals and submit them in several different ways.
In addition these devices can be used to take 12-lead EKG measurements in transit (e.g. inside the ambulance) and send them to the doctor for analysis before the patient arrives.
I'm very surprised that they're hosting these services on EC2. The company I worked for recently finished building a huge highly-redundant data center specifically designed for hosting this kind of medically-sensitive information. I'm willing to bet it was a bit more fortified than even Amazon's.
Hmm... That would rather seem to be the monitoring company's own fault. Admittedly I haven't used ec2 for about 3 years, but I seem to recall there being an explicit disclaimer about how it wasn't supposed to be used for this kind of uptime-critical application. I want to say it even specifically mentioned medical stuff.
EC2 is fine for part of the servers. I'd honestly say you should have multiple locations of servers, managed via different means (so the single points of failure are not there).
Agree. I'd avoid cloud for half as critical apps where I cannot CLEARLY picture where my servers are in a physical sense and how I can physically get a hold of them if shit hits the fan.
How do you physically get a hold of your servers if the gas main under them blows up or a tornado sucks them up and drops them a hundred miles away?
Servers are not physical things that you can just go get when there are problems. They are imaginary devices that can suddenly vaporize, leaving no trace. Plan your deployment accordingly, and you won't need to be writing emails saying that people are dying because your data center blew up. Your systems will silently switch over to some other data center and you'll file a claim with your insurance company to have the servers replaced on Monday morning.
In other words, "the cloud" had nothing to do with this problem. It could happen if the server was physically attached to you or in a abandoned missle silo with an army of armed guards.
Elastic Load Balancer, anyone? I have no sympathy for this company. They deserve to get sued for this.
Yes, for something life threatening like this, EC2 is a bad idea, but they didn't even bother to take advantage of the geographic redundancy, much less something so basic as having backup AMIs ready in another AZ in your current region.
Of course, a whole lot of companies are learning this now.
ELB doesn't work across regions, the failure at EC2 affected multiple Availability zones in the Virginia Region. ELB might have helped, but it's likely that some people had all their "availabiliy" zones knocked out.
In fact, it's looking like all the automatic-failovers from people hosting on those data centers compounded the problem when everyone's scripts tried to recover at the same time – what was referred to in a previous article as a "bank run".
But yeah, if you're doing heart monitoring, you need to have your servers replicated across a wide geographic area.
Unfortunately, this last AWS problem affected multiple AZs in the us-east region. The OP very well may have had an alternate AZ failover plan, but like Quora, Sencha, Reddit, FourSquare and Heroku, they probably kept it region specific.
As for backing up to multiple regions, I can imagine them thinking that sending everything over the public internet as being a bad idea. However, not having a multi-region/multi-provider failover plan was a worse one.
This is where you wish you had a data center address and a cabinet number where you could pop in, grab your servers and move the heck out to another DC.
I wonder if this will get more companies to maintain a non-cloud, vanilla setup on some dedicated or collocated boxes.
May be we should have an annual day to bring down our primary servers and see how the backups do. The idea shouldn't be to confirm if everything works or not. It should be to determine what did work and did not work.
You know, that gets me thinking- wouldn't it be pretty damn slick if Amazon offered a way to plug your own hardware into the greater EC2 cluster remotely, so that if Big EC2 goes down, you still have copies of your machines running locally?
Maybe you could even timeshare your machines out for some other instances for a discount on your service, similar to how you can send solar power back up the power grid for an electricity credit. Then ec2 really WOULD be a cloud!
You could always add min spec requirements for machines to join, increase redundancy of instances ec2-wide, add quality tiers that control what machines your instances go on, etc.
Amazon could even simply benchmark your server and network connection, and compensate you as a function of that, which would drastically reduce the incentive to hook up old crappy hardware to the cloud.
It's really the idea of being able to run copies of your OWN instances locally that intrigues me though.
Just rather than supplying the hardware you have to rent it from Amazon.
Would be interesting to know how these instances were affected by the outage. Since the problem was with the EBS infrastructure not the actual compute hosts I'm guessing this wouldn't have helped any
Couldn't you already do that by just setting up a Eucalyptus cluster? From Wikipedia entry:
"Eucalyptus is a software platform for the implementation of private cloud computing on computer clusters. There is an enterprise edition and an open-source edition. Currently, it exports a user-facing interface that is compatible with the Amazon EC2 and S3 services but the platform is modularized so that it can support a set of different interfaces simultaneously. "
Could someone who knows better comment on the feasibility of this? It sounds like a really good idea, but I'd imagine it's a bit more difficult than it sounds.
Sure, I'll comment on the feasibility of this: it's a ridiculous idea.
One of the features that allows AWS to scale is that it runs on a unified and consistant hardware platform. Take a look at the recent Google Data Center video or the Facebook Open Server initiative - companies try where possible to run identical hardware for all of their needs as it removes inconsistances in behavior and performance between arbitrarily different pieces of equipment.
So the idea of giving Amazon your own hardware to host is terrible.
The other part of his idea sounds like mirroring servers so that if an EC2 instance goes down, some bare-metal hardware running elsewhere could kick in. That's totally feasible, but no need to do it in the same DC under Amazon's control. Just set up your own machines (bare-metal or virtual) and keep them in sync with your EC2 instance. In the event of EC2 going down you can just change your DNS over to point your fall-back IP or do some clever off-site IP load-balancing tricks (assuming the load balancer hasn't gone down too)
In fact this would have been a good strategy for the ECG monitoring folks if they needed to keep service up even during a prolonged EC2 outage.
Haha yea, the 'idea' was kind of a mix between totally crazy and realizable. Mostly just fun to speculate about (and bemusedly wonder if Amazon might be investigating)
Yes, they do (Cloud Connect), but they have to be in the same data center. I've just gone through some sales / tech meetings with a Rackspace rep about the setups and while Cloud Connect is definitely cool, it is not well documented at this point and I couldn't get all the info I needed. I'm sure we'll all hear more about it throughout the year.
I'd like to see more back story first. Medicine is about risk management. There are places where you need 1/100,000 reliability, and there are places where you need 1/1000, and there are even places where 1/10 is good enough. There are primary systems, and there are auxiliary systems.
Without any background, it is really tough to tell where this fits. You can't blame the people for making the wrong decision until you know what the risk profile for the failure is.
An event monitor isn't a life-critical system. Perhaps it messes up a study, and ticks off some cardiologists, but it's not as if somebody's implantable defibrillator is depending on EC2 being online.
Later in the thread they say two instances are up but the third has most of the patients. If any of their patients die, they should be convicted of negligent manslaughter, not Amazon. Amazon explicitly warns that single instances can be lost. Any EC2 system needs redundancy, let alone a patient critical one!
I recently had the "good" fortune of being at the receiving end of a sales pitch from one cloud hosting service (not Amazon). The pitch centered more around flexibility (ability to add CPUs, RAM, storage, network nodes) rather than reliability issues as the one EC2 is facing right now. All questions about reliability were brushed aside with blanket statements like "it's in the cloud" as if that was an end-all-be-all. After much prodding the sales rep admitted that the hosting was all in one location (low lying hurricane prone area) and if we desired redundancy we would have to mirror our apps at a different geographical location. As with any other product, don't believe the sales pitch. If it sounds too good to be true, it probably is. Always use redundancy specially for critical apps.
I wish they offered some more detail on how important/life-threatening this is. Are we talking monitoring as in "detect signs of impending/occurring cardiac event and notify emergency services"? Or is this monitoring in a longitudinal sense, assisting physicians to look at trends in their patients? If it is the later case, is data stored on the device until it's uploaded, or is it simply lost if the connection fails?
Still, not really forgivable to be 100% AWS, but it's a very different story if this is primarily a recording device vs. a monitoring device.
Well, first of all if they had used more geographical regions (I don't know what the official amazon buzzword), they shouldn't have real bad problems. I don't know about you, but I believe mission critical systems should be hosted in at least 2 physically different data centers. When it's life critical, this should be a no brainer.
Second of all, according to Amazon PR speak this failure should not be possible. Every person in IT worth their salt should know PR speak is not to be trusted. Being reliant on AWS for your average business app is completely okay though. However, hosting apps which might endanger peoples lives on black box technology over which you have little influence is moronic imo.
Last but not least I seriously question the privacy implications of this. Hosting parts of your medical app in the cloud is fine and I don't know the details of this app, but I wouldn't be happy to know that my medical history is stored by some third party I don't entirely trust with such sensitive data.
1) Amazon's outages took down multiple "availability zones"
2) Amazon is HIPAA certified, which means they adhere to the industry standard for storing private medical data. In this sense, they are as good as any other company that such data is being passed around.
This is too incredible to be true. But if it is true then I would expect the company in question to be out of business quite soon. A company that put's a life critical system on AWS shows a total disregard for their patients well being.
I've seen things much worse than this with medical records and data.
Up to a few months ago we had a ASP.NET shared hosting customer that was doing some kind of data relay web service for medical imaging. No encryption. Patient data in full view on the server. No redundancy.
Apparently it was used for outsourcing imagery review or something. If it didn't work doctors would have to drive in from home which slowed down the diagnostic process.
"Mission critical" on a $30 a month shared hosting plan. Very much not HIPPA compliant.
I'm actually thinking of that. You deploy to the service that take care of deployment, backups and transfer in down times. It would make up of it another kind of hosting company, though.
Yes, it would essentially be a hosting company with multiple providers. That in it self is a single point of failure, but at least it would give some insurance against outage of the more physical kind.
Just wow! How difficult can it be to have a backup dedi server with the DB mirrored running at some other provider to quickly switch to, just in case? Even many simple web sites have a config like that. Clouds have been around for decades, they are still new and may fail.
This is just supplementary monitoring, not a true life-or-death situation. Everyone (the company involved included) is welcome to stop blowing it out of proportion any time.
Triply redundant hosting providers is really setting the bar high. Lot of 20/20 hindsight being put into play the last few days!
In my experience, small companies typically have one hosting provider, off-site backups, and the probable but untested ability to bring up a fresh installation in a few hours. That's both common in the real world and good enough for most situations. It would get you through an outage like this.
Are you referring to the link in the title? It is true that the company was hosting cardiac patient monitoring on EC2. This was not life threatening infrastructure, but that is not implied by the HN title. How is it linkbait?
The poster on Amazon's forum misrepresented the severity of the infrastructure and tries to backpedal later on in the thread. This is an example of someone using EC2 and fucking up, firstly for hosting production level infrastructure without backups, and secondly for misrepresenting the importance of the infrastructure in an attempt to get more technical support, making them look liable for some sort of medical malpractice.
The poster probably panicked when they realised their first mistake, and thought they were showing that they were trying to get back in control. I don't think the poster was totally stupid, maybe she/he didn't understand the risks of EC2, or maybe it wasn't even their decision to have no backups --- in which case they should have pressed harder at meetings or refused to continue to work on the project. And then they panicked. We can all learn something from this.
Epic fail for the company which is hosting on their kit. Whoever threw it on there is a total dick. There is nothing wrong with EC2 here. It's just not the right tool for the job.
If they are monitoring cardiac patients, they should, at least for non-critical people, have the instrumentation cache the data and send when available over leased lines.
If it is critical, it should be PROPER HARDWARE attached to the POTS network (like we have in the UK!) with multiple redundant networks over several distinct carriers and multiple monitoring stations.
It sounds like their product is a) crap and b) dangerous if used for critical care.
He's saying that any computer system hosting the data would have downtime. Your own custom data center can blow up just as easily as Amazon's.
Now, if zero downtime is the requirement, then you just need to have a lot of data centers and well-tested failover procedures. It's all very simple, and EC2 can easily be one of those redundant data centers.
I wonder if they have the device connected through wifi to the patient's internet connection. I'm sure EC2 is not their only possible point of failure.
Yeah, I don't know how much I care that Amazon isn't delivering three nines when the local cable monopoly at dad's place is lucky to reach three eights.
If this story is true (and I have some doubts), you'd think this would have been a no-brainer to not try to keep critical data like this in the cloud.
But even for non-critical medical data, non-tech savvy medical folks (and this includes some physicians, although in general physicians are more tech-savvy than most managed care decision makers) are going to get into increasingly serious trouble if they don't think through these kinds of things very carefully.
Driven by electronic health care mandates and incentives, companies are pushing portable devices for medical data gathering (portable as in tablet, ie without local data store), and some groups are going to be knee-jerk tempted into buying into this without thinking through the ramifications of not owning your own data -- and by that, I don't mean vague marketing-speak hand-waving that "you own your data" through the simple mechanism of password-protected access to your patient's data living on someone's server farm.
By actually owning your patient's data, I mean strongly considering restricting your local practice to use of on-site media.
If you don't, you'd better have complete confidence that putting your patient's data on the web reflects a reasonable standard of care/privacy, that retention policies don't bite you in six months when your cloud provider goes out of business (or changes hands or farms out data storage or backup to some "economical" third-party), that you're going to have access to your data when you need it, that your cloud provider is going to audit access and changes to your patient's data and report the results of that audit to you accurately and regularly, that legal requests for data are not acted upon without you or your patient's consent, that backups are subjected to the same levels of security, etc. etc.
Electronic medical records are a good idea. It can be done right, as systems such as the Veteran's Administration's VISTA system have demonstrated. However, there is a fine line to walk here. Implementing EHR for the sake of government incentive, or for the convenience of insurance companies, or even the convenience of walking into an exam room carrying a light-weight tablet (which have their own disadvantages for data gathering) rather than a laptop with a hard-drive isn't going to cut it when you put your patients and your practice at risk of harm.
who was the IT professional that decided to rely soley on the AWS cloud without any kind of "just in case" solution....with people's lives at stake? lol. I think I'm gonna subscribe to the hoax theory
I hope no one dies because of this. Yet, the sad part is, it'll take someones death to get serious attention and help prevent this stupidity in the future.
Of course, then things will swing wildly in the other direction with massive legislation and regulation thats designed to get a lawmaker reelected more than it is to save lives.
Remote cardiac monitoring isn't for people who are imminently going to die of catastrophic heart attacks or who risk dropping into a fatal arrhythmia (wonky heart rhythm). In fact, there isn't even solid evidence that it saves lives. What it is used for is investigating patients who have vague symptoms which might be related to transient changes in their heart beat.
These are people who come to their doctor at the age of 54 and say "I felt my heart beating really quickly and felt kind of faint" or "I felt kind of dizzy and then passed out. It's happened twice in the last month." The more traditional way to investigate these patients are with what's called a Holter monitor, as alluded to by a previous poster. These are little belt packs you carry around while wired-up and that record your ECG for 24-48 hours at a time. The main weakness? You said it: the device only has a 24-48 hour windows to capture the weird, often rare, rhythm.
There are different ways remote cardiac monitoring systems report their results and I'm not sure which this particular company was using, but it doesn't really matter. Some of them only report weird stretches, others only report events when the patient says they're feeling symptom X, others are reporting continuously.
Take-home message: this is not life-or-death data. When doctors (at least those who are allowed to keep practicing) think a patient needs critical cardiac monitoring, they admit them to hospital.
This was a tech guy, looking to jump the queue by trying to raise a red flag because his servers were being used for--OMFG!--cardiac monitoring. A lot of my doctor buddies use similar strategies when they're caught speeding by the police. "I just got called in to the hospital!"
…to see a patient with a really nasty rash.