Reading this response it seems that crypto-mining is not allowed on digital ocean as they have checks against it. The TOS doesn't say so explicitly but does note that:
>violation of any of these Terms of Service or any law, or if you misuse system resources, such as, by employing programs that consume excessive network capacity, CPU cycles, or disk IO
By my reading that seems to mean that you're not allowed to use your VMs to their full capacity due to them being over-provisioned. This is in contrast to AWS who are more explicit on which instances (T instances) are over-provisioned and exactly how they're throttled.
If you want to do cryptocurrency mining on DO that is actually okay with us. Some of the other respondents are correct the behavior we were looking for was really around fraudulent accounts being created and performing cryptocurrency mining. This is why the trigger that flagged this account was using payment history as a key factor in the triggering.
The thing that has me scratching my head is how this chain of events unfolded.
I get that your fraud algorithm flagged it because of lack of established payment. how is this possible if what the tweet referred to as "locking us out of all of our backups and work"? surely an account history of any significance would have an established payment record. From their tweets they mention that they had 5 droplets and some storage of a not insignificant number of records (~500k) and that a script is required to be run every 2-3 months to process some data and that script spins up 10 droplets during that time. seems like it will take 13 hours to process the data based on row count and per record time.[0] I am struggling to see how they didn't have payment history. can you elaborate?
In addition another thing I'd think would help assuage fears of a complete lockout is some process where you can request and download the db or a snapshot of the virtual machine.
Your post-mortem implies this is not allowed at all.
Not sure why you were downvoted, I had the same impression, after reading:
...an automated service that monitors for cryptocurrency mining activity (Droplet CPU loads and Droplet create behaviors). These signals, coupled with a number of account-level signals (including payment history and current run rate compared to total payments) are used to determine if automated action is warranted to minimize the impact of potential fraudulent high-cpu-loads on other customers
This sounds like they don't permit extended high CPU loads due to the impact it can have on other customers.
Cryptojacking is a well-known, major problem for cloud compute providers. Catching and squashing new exploits that allow people to create a fresh account, run up compute bills and then abandon the account without paying is very important.
My guess would be that this is such a well-known problem (within the field of cloud compute at least) that they just didn't think they had to state that normal crypto mining by paying customers is completely fine.
Depending on the cryptocurrency's proof-of-work algorithm and new-ness, it can be profitable to mine in the cloud. I've done it briefly in the past. But generally it's not profitable.
In every cryptocurrency (the popular and functional ones anyway), there's a set global rate of mining rewards. All miners compete for a slice of that reward, so as more people mine, each individual miner gets less reward. (This causes an equilibrium to be reached where more people mine until it's no longer profitable for more people to start mining. If mining becomes unprofitable, some miners will drop out, and the remaining miners will each make a little more.) If masses of people realize that cloud mining for a particular cryptocurrency is profitable, then what generally happens is that lots of people pounce on cloud providers to mine, it becomes barely profitable, and then people operating their own hardware that's cheaper than cloud providers come in and push the mining rewards down to where it's no longer profitable for people to cloud mine.
Because cloud mining is never profitable in the long run, most cloud mining that happens is fraudulent activity using stolen cloud accounts or payment info. (If you're not paying for it, then making any amount of money from it is profitable.)
It depends on what crypto is going to be mined an how many accounts can be stolen given the fact that there is already plethora of bots that look all over GitHub for accidentally committed credentials. Heck, just a year ago people did scans for outdated WordPress installations to inject, among other things, some JavaScript (!) Monero miners [0]…
No. Cryptomining represents an arbitrage opportunity such that the spot price of the instances should be adjusted. In the long run it should not be profitable.
No, that quite clearly states that they treat high CPU loads as suspect on accounts without an established good payment history or if it significantly deviates from previous usage patterns.
What would "fraud" mean in this context? Are they talking about customers who don't pay their bill to DO? (If so, seems like the account should just be temporarily suspended until the bill is paid.) Or are they talking about fraud to other parties, like phishing sites? (If so, I don't see the connection to crypto mining.)
My understanding is that they're trying to prevent users from creating new accounts, running 100%CPU until it's time to pay the bill and then just not paying, moving on to another new account.
edit: from elsewhere ITT it seems they're doing this with stolen credit cards.
"Cryptocurrency mining mitigation detects suspicious behavior, including very high CPU utilization on an account with no payment history, which results in an account lock"
Lots might have been done wrong here, but it sounds like they had an account with trial or promotional credits - I can see how this could easily be abused.
Completely. At the same time, those promotional credits are going to be used by guys like me who will have to decide if their services are worth having to spend an extended period of time explaining why "we're not recommending AWS/Azure/Goober"
An account shutdown, or enough complaints from verifiable sources, and I'm not going to the trouble. Not to pull out an old trope "Nobody got fired for picking IBM". But that's the case: pulls a move like this and the entirety of the customer, who likely came in with AWS in mind (in some cases, was advised against it and insisted on it) is going to shrug their shoulders. Pick a provider that the customer hasn't heard of and I'm going to get a phone call that goes something like "You're the one that said we should use that basement-operation!" with raised voices. Heck, the last time there was an Azure outage, we didn't hear from most of our customers. It was so impacting that even customers well outside of software development/technology read news articles and connected the dots. I had one customer tell me he thought it was just their corporate internet connection; they assumed it was working[1].
[0] Plus, too lazy to put in the research; sorry.
[1] They were a customer who insisted on doing the app monitoring, themselves -- that guy was getting the alerts and similarly assumed it was the network since that happened regularly with another application they developed -- the monitoring server was on-prem.
Sounds like it's designed to counter stolen credit cards.
An attacker might load a stolen credit card number into an account and only use enough resources to generate a few dollars worth of billing. The owner of the credit card might not notice the small charge.
Then after a few months of low billing (to bypass a previous heuristic), they ramp up the utilization, mine a bunch of coins then the holder gets a massive bill.
The holder does a charge back and DO is left holding the bill.
It's also designed to keep everyone relatively happy in a shared-CPU environment. "Standard" droplets share CPU with others on the same node, so one droplet pegging the CPU 24/7 can be problematic.
AWS doesn't have this problem because either your instance is allowed to use all the CPU that's allicated to it, or else (t2 & t3) the platform will automatically limit your CPU usage. You don't have to care about how your usage affects other people. It's one more thing that AWS abstracts away. DO's abstraction, on the other hand, is rather leaky in this area. That's a problem in and of itself, in addition to the matter of credit card fraud that every company has to deal with.
I don't think that it isn't allowed but that they have seen that fraudulently acquired resources are generally used for crypto-mining so they felt that this was a good signal to look for.
Right. Reading between the lines a bit, it's not the activity itself that DO is worried about, but a pattern of usage that suggests that the account may have been created fraudulently or compromised.
This is correct. This was the primary thing we were attempting to solve for in this case and the bug in the algorithm started the chain of events documented in the postmortem.
Cryptocurrency mining is a favourite of fraudsters. Get a fake or stolen credit card number / identity, sign up for an account on a cloud provider, and spin up instances to do your mining. Depending on how quickly the provider reacts to the pattern of behaviour, or identifies the account as fraudulent etc, you can have earned a reasonable pay-off.
If the provider isn't paying attention and/or doesn't have good fraud detection in place, it may be a few months before your account and resources are terminated (assuming the payments eventually bounce, and the provider gives you a chance to fix it)
I believe the purpose of these checks is for when people's outdated wordpress install or whatever gets compromised by script kiddies. Generally, the scripts install crypto miners to mine for the hacker until your account gets shut down, running you up a huge bill in the process.
I recently logged into my long-dormant DO account to kick the tires on their now-GA managed Kubernetes service and to contribute some CPU cycles to a distributed computing project I'm interested in (not cryptocurrency, I swear).
I first requested and was approved for a droplet limit increase (to 25). I started 20 nodes, deployed my very CPU intensive workload, and 12 hours later my account was locked and my nodes all went NotReady.
I immediately replied to the abuse ticket explaining my usage. 4 days later, my account was unlocked and received the "allow high CPU" flag... but they billed me for the nodes as if they had been running that whole time. I asked for a credit (5 days ago) and they haven't replied yet. Probably a little busy over there right now.
So... I'm not too thrilled with DO. I get that cryptocurrency ruined everything, but this has been a frustrating experience and I'm glad I was only using it for a pet project.
Hey xnxn - Thanks for raising this issue. I'm Zach, Director of Support at DO. Please send me an email with your account detalis, first name at, and I'll take care of this.
Hey Sneak - I totally agree* :) We've already started efforts between Support and Security leadership to leverage the 24/7 structure of Support. Our goal is that no one will ever need to use social as an escalation path, and our new Support Engineers who are joining in mid-June and early July will be part of this a reality for our customers.
Fyi, I'm planning to migrate off of DO due to the crummy payment options. With every other recurring service I can have a PayPal subscription, DO doesn't support PayPal properly though.
Also, your billing system is incredibly spammy (1 to 2 emails a day at times). The bills I get are always for partial months despite the VMs on the account running for a full month.
Hey metildaa - Zach here from DO support. It is accurate that that do not currently support PayPal subscriptions, this certainly isn't the overall experience that you should be having. Can you shoot me an email (first name at) and I'll look into this for you?
That's my point. Their support got so bad that they had to be called out on in publicly before they did anything. That's hardly what you want to hear from a company like DO.
> That's my point. Their support got so bad that they had to be called out on in publicly before they did anything. That's hardly what you want to hear from a company like DO.
That's what you get with most providers these days. Ever tried reaching Google or Amazon?
Actually I have complained to Amazon... and they have always responded within a couple days and in one case where I complained about Prime Video, they called me on a Sunday, which shocked me because everything is otherwise closed in Austria, except gas stations, restaurants and hospitals.
Google on the other hand, I make complaints or suggestions once every couple months and 99% of the time I don't even get a boilerplate response.
I don't believe that Google or Amazon have ever done anything like what DO did before. However, if you notice their support guy Zach was responding to a completely different incident that is not related to the one being addressed with their post-mortem.
If you pay for enterprise support with Amazon you can open a case with a less than 15 minute SLA [1]. With business that goes down to an hour SLA. With both plans you can create chats or calls that typically get answered within a matter of minute with and assigned to an engineer with a background in the service.
>violation of any of these Terms of Service or any law, or if you misuse system resources, such as, by employing programs that consume excessive network capacity, CPU cycles, or disk IO
By my reading that seems to mean that you're not allowed to use your VMs to their full capacity due to them being over-provisioned. This is in contrast to AWS who are more explicit on which instances (T instances) are over-provisioned and exactly how they're throttled.