Do you have a very small number of servers? Requiring manual steps to boot is not very scalable--if there is an event that requires all your servers to reboot (power outage, mandatory upgrade, etc) you would be in quite a tough spot.

I much prefer having to repeatedly enter the password dozens of times than risking data leaking out - or worse, data being tampered with.

Shouldn't all your servers have different passwords? How do you manage the passwords in case you're not available when the server reboots?

Have two people know all the passwords. Let each of them encrypt this list of passwords with a password only they individually know. Sync passwords each time someone changes something.

So "yes" – what would you do if you had, say, several hundred (or several thousand) servers?

If I have that many servers then I can assume I have a large budget for security, right?

I would make a password entering automation system. Ensure that that system is dead-simple and secured to death. It must run no other services, firewalled to death even from the intranet, physically secured in a cage, and must be off most of the time. It is only to be turned on when booting a system, and turning it on not only requires a password but also physically walking to the cage, opening it with a physical key, and pressing the "on" button. All target servers must be configured in such a way that they can obtain network access before mounting the encrypted part.

But I'm not a security expert. Maybe I've overlooked something.

Wait, this sounds way harder than the previous solution and entering the password over a remote IPMI console.

It's harder to set up. But when the password entering automation system exists, you just grab the key, go to that system, unlock & boot it, then tell it to enter the passwords for your 1000 servers.

With that many servers you probably have to reboot a few of them pretty often.. You'd have to live in the datacentre :)

I'm not sure you can assume that if you're, say, Instagram (pre-big-round/acquisition). You may well have several hundred, even thousands, of AWS instances with a staff in single digits.

Properly hard problem, as you've pointed out.

How does this help? The key is still stored in memory which I assume the hypervisor has access to.

Correct, but hacking into the hypervisor is harder than hacking an administration interface. In the end the only secure server is one in a vault at the bottom of the ocean, but there are ways to prevent certain attack vectors.

>In the end the only secure server is one in a vault at the bottom of the ocean[...]

I strongly doubt that:

https://en.wikipedia.org/wiki/Glomar_Explorer

Hetzner provide a lot of physical machines too, I believe this is what the other poster was talking about.

Physical machines don't prevent keys from leaking out. A physical attacker can analyze power usage usage patterns to extract the encryption key. :)

I don't think you have that sidechannel with AES-NI. Besides, as a physical attacker cold boot attack would be much easier Or if the server has any interfaces with DMA, like PCI or something, that's even easier.

Wouldn't that require physical access?

Ah, I missed the physical machines. No hypervisor to crack, just out of band management cards. :)

Well, maybe. Do you have servers with a secure boot environment?

Also, if the attackers are able to attack your servers in a similar fashion (gaining access without reboot) -- they have full access to the already decrypted disks (well virtual devices).

As for other comments on automating booting -- as eg cryptsetup allows entering the encryption key via ssh -- automating that shouldn't be too hard.

I still think that it mostly defends against someone stealing/cloning the physical disks though.

Even without secure boot environment, if you're paranoid enough you can throw away and reconstruct the boot environment on every unplanned boot.

Encrypted disks are not meant to protect against attackers that can gain access to the server without rebooting. But all your OS-level security mechanisms (user separation, AppArmor, file permissions, firewalls) become useless if the hosting provider's admin interface has an exploit and the attacker can reboot the machine into single user mode. In other words, the Linode scenario. That's what encrypted disks protect against.

My point was that getting back to a known good boot scenario isn't all that easy if you don't have physical access?

I suppose the sanest middle ground is to wait for the dust of an event like this to settle (or have someone investigate, in case of "just" an unexpected reboot) -- then wipe the server(s) in question, reinstall (hoping the install media is ok) -- and importing the disks/backup.

I still think it is little gain for a lot of work.

Basically if your server software is secure, you've avoided data compromise due to a reboot into single user mode or similar -- but you have to assume (for the sake of guaranteeing the safety of the encrypted data) -- that attackers have a copy of the data at rest (which they can get if the can boot into a recovery image etc.

So now you have to make sure that you can boot a safe environment in which to import the data to, without leaking your key(s).

All the while, it is rather likely there are a few vulnerabilities in your software stack that all this never did mitigate.

I'm not saying don't do it -- I'm in favour of doing it for the (relative) peace of mind that no that will get lost/out as the servers are recycled -- I'm not sure it makes much sense to avoid hackers -- the online threat seems to dominate for most cases I can think of.

What if attacker infects unencrypted part used for booting? Do you have protection against that?

It's relatively easy on Linux, where /boot is usually not encrypted.

Agreed, but typically /boot isn't automounted when it's a separate partition, which means they would need a root-access compromise already.

Hetzner allows to netboot any server via admin panel. After booting into recovery os, you can modify unencrypted parts as you wish.

They can replace the booting kernel. Unless you use SecureBoot, TPM or some such thing, there is no way to protect against that (assuming that the attacker has access to the shut-down system at one point and you boot it later).

The trick is putting all important data in the encrypted part. The unencrypted part must be reconstructable in an automated manner. If you are paranoid and you assume that all unplanned reboots are attacks, then you can reconstruct the unencrypted part every time. Tools like Chef make this relatively easy.

The kernel lives on /boot, and could be subverted to lie about its uptime to convince you that there was no unplanned reboot.

This is the same methodology I use, and I think about these attacks a lot. There is no good/cheap way to verify a remote execution environment right now with commodity hardware. :/

That's not possible. Even if you fool the kernel, you still can't mount the encrypted disk. The fact that the encrypted disk is not mounted is proof that there has been an unplanned reboot.

Wow, do you actually reconstruct /boot on every unplanned reboot, yourself?

I don't know. Ever since we set up encrypted disks we haven't had an unplanned reboot. I was simply pointing out that this is a possibility that we've considered.

What types of hacks does that protect you from exactly, considering you don't have physical access and can't even see the hardware?

It protects against hackers that compromise the provider's administration interface, and then use that interface to reboot the server to single user mode or a rescue system. Upon reboot, the attacker will encounter an unusable blob as filesystem because he doesn't have the encryption key.

It also protects against hosting provider employees that detach the hard disk and attach it onto another machine to copy data off it.

If the server is virtualized, then in theory it's still possible that the attacker hacks into the hypervisor, and then through modifications in the hypervisor's RAM gains access to the guest kernel, and thus the guest system. However the skill that is required for this is an a whole new level compared to "regular" hacks.

If the attacker has physical access then it's still possible that he uses electromagnetic emissions from the hardware or power supply patterns to obtain information about it to hack the system. But again, this requires skills on a whole new level.

Very good point about single user/rescue, thanks.

What kind of effect does this have on your I/O performance?

Minor. It costs more CPU because of encryption/decryption. But our servers have so much CPU power, and our workload is mostly dependent on disk I/O throughput, so enabling encryption is almost free.

AES-NI features in modern CPUs reduce it even further.