I'm going to try to suggest a couple of solutions regarding problems encountered in the article that people might find to be useful to know.
First, some pointers (non-exhaustive).
For workstations:
- Antivirus
- Firewall
- Disk encryption
- Password complexity requirements
- Updates
For servers:
- Key-based authentication
- Updates
- Firewall rules on host: explicit ingress/egress
- Run services as non-privileged users in a chroot jail
Firewall rules for network: explicit ingress/egress
Yes, that's two firewalls for your hosting infrastructure. Attackers would have to escalate privileges from a service user to a root user in order to modify the firewall rules, at which point they'll have access to the other components of your internal network. To get outbound connectivity, they'd have to gain root access to the external firewall as well. Keep in mind they can still upload data if they can execute arbitrary commands as the web service user.
Now, onto more specific items mentioned in the article.
>realizing only later that the e-mail address was a couple of letters off.
Tell your employees to only accept cryptographically signed email from you. S/MIME is simple to implement, and works across multiple platforms and clients. There is even support for it in iOS, Android, and Mail.app!
You can do a Google search on how to get these certificates installed in your mail and mobile clients of choice.
>clicked on the attachment
Use Google Apps. I'm not sure what the success rate of Gmail's 0-day detection is with attachments, but I'd be willing to bet that they're better if not as equally effective as the average desktop antivirus email scanner.
>Microsoft operating system
The article doesn't say if they used any sort of antivirus or firewall protection on their workstations. If you absolutely have to have Windows somewhere in your business, you need to have a decent antivirus and firewall solution running. Also, make sure all of the latest updates are applied in a timely manner. IIRC, you can set Domain Policies from Active Directory to do this sort of thing for you.
>automatically uploading more tools the hackers could use to control the network remotely.
This could have been addressed in a couple of ways. (non-exhaustive list)
1. If the malware was listening for incoming connections, explicit ingress (aka no port forwarding) with a NAT (most residential routers) would have made it more difficult, as another layer must be evaded to connect to the internal machines.
2. If the malware was using a connect-back shell, explicit egress (aka outbound) traffic would make it difficult to get a shell on a compromised machine. As soon as you realize you're under attack, you should go into hardcore lockdown paranoid mode, and only allow network ranges that are absolutely necessary for business operation.
>Then the company’s e-mail servers began shutting down, sometimes two or three times a week, slowing e-mail traffic, the main way the company provides customer service
Hosting an email service is a ridiculously difficult task security-wise. Postfix and Sendmail both have a history of security vulnerabilities, and the only way to properly host an email server is to harden a *nix system to the extreme, and always be paranoid. But that's hard and most people aren't in the business of hosting email services. That said, I'm pretty sure Google Apps has a great cost/benefit factor when you look at ease of implementation and security, because the Gmail team is more likely better at server hardening than you are. Oh, and make sure you backup your email and support PIN, in case anything goes awry.
>Similar problems began plaguing the web servers -- a bigger problem since web sales of CYBERsitter supply more than half of Solid Oak’s revenue.
I would have to recommend putting CloudFlare in front of your servers, as (through their blog posts about several incidents) they have demonstrated an ability to successfully defend against a large number of attacks. This also has the benefit of providing a proxy, masking your backend server IPs, which I'll talk about more later.
>Similar problems began plaguing the web servers
>figure out how the hackers might be behind it.
>But the agency shed almost no light on the situation, he says, and he was never told if the material was useful.
I'm surprised they didn't offer any advice on how to mitigate attacks such as these. At the very least, some pointers similar to the ones I'm writing now would probably gain some sort of ground.
>alternating between four different cell phones from three different carriers.
This seems a little extreme, if not excessive. I've used burners (temporary pre-paid cellphones, disposed of after a duration of use) before when I attended DEFCON (hackers like to do sketchy things with RF) but never more than one at a time. I'm sure you could probably use something like Google Voice with multiple Google Accounts or Twilio for temporary numbers while not exposing your actual number.
>constantly had to reboot servers
>couldn’t trace the source of the network problems
If you're going to have to do it more than twice, it's probably best to automate it. If reboots are required due to a crash, you should probably identify the cause of the crash (go through logs, etc.) and attempt to rectify it. If you seem to be way in over your head, bringing in an expert should definitely be on your mind.
>to find that his commercial-grade SonicWALL firewall had failed
>He spent a good part of the next day on the phone with the manufacturer, who was stumped.
If you have any sort of hosting infrastructure, and your expertise isn't network administration or security, I'd recommend getting (full time, or contracting) a certified network administrator to help you get your networking shit together. If I were to pick equipment, I'd probably go with the big dogs, Cisco or Juniper.
>He began writing his own software to monitor the connections his computers were making to outside networks, looking for tell-tale signs of the hackers at work.
Contrary to popular (or at least some peoples') beliefs, it is possible to harden a Windows server, you just have to really know your shit and be able to tolerate all the graphical clicky stuff.
>all eight passwords
If you're running a business, chances are you have more than 8 accounts.
1. Use unique passwords for each service/account.
2. Store them encrypted, in a password manager.
3. Use keys/certificates whenever possible.
4. Use two-factor authentication whenever possible.
>The folder was gone two days later, he says, and in its place were several pieces of software he didn’t recognize.
You generally don't want to leave a compromised server in production rotation. Take it off the network and perform forensics to determine how it got compromised, and how to prevent that in the future. Make sure to wipe the server and reconfigure it before placing it back on the network.
>Net losses averaged $58,000 a month
(aside) I'd fix the CRAP out of your security problems for that much a month.
>A hacker could certainly edit the script and break it so it wouldn’t work
This means they have access to your web servers, and/or to your deployment process/service. One of the ways you can mitigate direct access to your web boxes is to use some sort of load balancer (HAProxy) or web application firewall (CloudFlare). Then only allow connections to your web boxes from the proxy, as well as a specified IP address to allow for maintenance.
>That would be a great way to do it without calling attention to the fact that they were in the system.
If you choose to use git as a tool to deploy your code, you can continuously monitor the codebase for modifications.
I'm quite tired and I think I'm rambling at this point. I hope this is clear enough for some people to get something out of. I'll probably come back later and make some edits for clarity/correctness.
First, some pointers (non-exhaustive).
For workstations: - Antivirus - Firewall - Disk encryption - Password complexity requirements - Updates
For servers: - Key-based authentication - Updates - Firewall rules on host: explicit ingress/egress - Run services as non-privileged users in a chroot jail
Firewall rules for network: explicit ingress/egress
Yes, that's two firewalls for your hosting infrastructure. Attackers would have to escalate privileges from a service user to a root user in order to modify the firewall rules, at which point they'll have access to the other components of your internal network. To get outbound connectivity, they'd have to gain root access to the external firewall as well. Keep in mind they can still upload data if they can execute arbitrary commands as the web service user.
Now, onto more specific items mentioned in the article.
Tell your employees to only accept cryptographically signed email from you. S/MIME is simple to implement, and works across multiple platforms and clients. There is even support for it in iOS, Android, and Mail.app!http://en.wikipedia.org/wiki/S/MIME
You can get certificates here: http://www.symantec.com/verisign/digital-id
You can even get free ones if you're on a budget! (although they do say for personal use, implement at your own risk if you're a business!) http://www.comodo.com/home/email-security/free-email-certifi...
You can do a Google search on how to get these certificates installed in your mail and mobile clients of choice.
Use Google Apps. I'm not sure what the success rate of Gmail's 0-day detection is with attachments, but I'd be willing to bet that they're better if not as equally effective as the average desktop antivirus email scanner. The article doesn't say if they used any sort of antivirus or firewall protection on their workstations. If you absolutely have to have Windows somewhere in your business, you need to have a decent antivirus and firewall solution running. Also, make sure all of the latest updates are applied in a timely manner. IIRC, you can set Domain Policies from Active Directory to do this sort of thing for you. This could have been addressed in a couple of ways. (non-exhaustive list)1. If the malware was listening for incoming connections, explicit ingress (aka no port forwarding) with a NAT (most residential routers) would have made it more difficult, as another layer must be evaded to connect to the internal machines.
2. If the malware was using a connect-back shell, explicit egress (aka outbound) traffic would make it difficult to get a shell on a compromised machine. As soon as you realize you're under attack, you should go into hardcore lockdown paranoid mode, and only allow network ranges that are absolutely necessary for business operation.
Hosting an email service is a ridiculously difficult task security-wise. Postfix and Sendmail both have a history of security vulnerabilities, and the only way to properly host an email server is to harden a *nix system to the extreme, and always be paranoid. But that's hard and most people aren't in the business of hosting email services. That said, I'm pretty sure Google Apps has a great cost/benefit factor when you look at ease of implementation and security, because the Gmail team is more likely better at server hardening than you are. Oh, and make sure you backup your email and support PIN, in case anything goes awry. I would have to recommend putting CloudFlare in front of your servers, as (through their blog posts about several incidents) they have demonstrated an ability to successfully defend against a large number of attacks. This also has the benefit of providing a proxy, masking your backend server IPs, which I'll talk about more later. Bro, http://en.wikipedia.org/wiki/Host-based_intrusion_detection_... I'm surprised they didn't offer any advice on how to mitigate attacks such as these. At the very least, some pointers similar to the ones I'm writing now would probably gain some sort of ground. This seems a little extreme, if not excessive. I've used burners (temporary pre-paid cellphones, disposed of after a duration of use) before when I attended DEFCON (hackers like to do sketchy things with RF) but never more than one at a time. I'm sure you could probably use something like Google Voice with multiple Google Accounts or Twilio for temporary numbers while not exposing your actual number. If you're going to have to do it more than twice, it's probably best to automate it. If reboots are required due to a crash, you should probably identify the cause of the crash (go through logs, etc.) and attempt to rectify it. If you seem to be way in over your head, bringing in an expert should definitely be on your mind. If you have any sort of hosting infrastructure, and your expertise isn't network administration or security, I'd recommend getting (full time, or contracting) a certified network administrator to help you get your networking shit together. If I were to pick equipment, I'd probably go with the big dogs, Cisco or Juniper. Bro, http://en.wikipedia.org/wiki/Network_intrusion_detection_sys... Snort: http://en.wikipedia.org/wiki/Snort_(software) Contrary to popular (or at least some peoples') beliefs, it is possible to harden a Windows server, you just have to really know your shit and be able to tolerate all the graphical clicky stuff. If you're running a business, chances are you have more than 8 accounts.1. Use unique passwords for each service/account. 2. Store them encrypted, in a password manager. 3. Use keys/certificates whenever possible. 4. Use two-factor authentication whenever possible.
You generally don't want to leave a compromised server in production rotation. Take it off the network and perform forensics to determine how it got compromised, and how to prevent that in the future. Make sure to wipe the server and reconfigure it before placing it back on the network. (aside) I'd fix the CRAP out of your security problems for that much a month. This means they have access to your web servers, and/or to your deployment process/service. One of the ways you can mitigate direct access to your web boxes is to use some sort of load balancer (HAProxy) or web application firewall (CloudFlare). Then only allow connections to your web boxes from the proxy, as well as a specified IP address to allow for maintenance. If you choose to use git as a tool to deploy your code, you can continuously monitor the codebase for modifications.I'm quite tired and I think I'm rambling at this point. I hope this is clear enough for some people to get something out of. I'll probably come back later and make some edits for clarity/correctness.