Thoughts on Security

Cloud services, where an organization remotely uses a third party company's servers to host the organization's websites, databases, or email, are very popular. It can save a lot of time and money to outsource these functions, but security is a frequent concern, since, as Joanna Rutkowska put it, it means:

We trust that the cloud provider is 1) non-malicious and ethical, and that they won't really read the memory of the virtual machine on which the previously mentioned cloud-service is running (and won't make it available to a local government officials, e.g. in China), and 2) that they secured their infrastructure properly (e.g. it wouldn't be easy for one customer to “escape” from a VM and read all the memory of the VMs belonging to other customers).

At some point in the future, perhaps there will be hardware support built-in to processors that will allow you to execute code on them in an encrypted memory section that the cloud provider won't be able to see inside, but there are so many ways that can go wrong it may never happen (no matter how you design it, the cloud provider might be able to extract any crypto keys from the chip hardware, reverse engineer it, and emulate its functionality in software and you'd never know).

On the other hand, if you simply want to use the cloud providers as a data store, you can do that right now, but it is not simple to do in a way that is available and flexible for live applications, keeps information confidential from the cloud provider and prevents the cloud provider from silently corrupting data to break your security model. The good news is that by layering existing technologies on top of each other, we can achieve those goals.

The basic idea is to use the untrusted system as a remote block storage device using NBD (Network Block Device), then layer an encrypted mapped block device on top of that using LUKS (Linux Unified Key Setup) to keep the cloud provider from seeing the data contents, then layer a filesystem that performs both metadata and data checksumming on top of that (Btrfs, which uses CRC32). Since the data is encrypted with a key the cloud provider doesn't have, any modifications to the data will corrupt the corresponding decrypted block in unpredictable ways; and with a probability of 4 billion to 1 (99.99999998%) the checksum will fail, causing the filesystem to detect and report an error before taking any unsafe action with the corrupted data, which is probably good enough for most purposes. The cloud provider won't be able to see your data or corrupt it; the worst they can do is to make it unavailable to you.

So how do we make this happen in practice?

1. On the untrusted cloud server, use your preferred firewall configuration, whether using the cloud firewall profile e.g. on AWS, or a local firewall solution like UFW or iptables on the server itself, to block all connections except those from your IP's. For speed, the protocol we'll be using, nbd, is not authenticated or encrypted itself, even though the data flowing over it is. Locking down the IP's allowed to connect means that it won't be easy for the rest of the internet to corrupt your data. Alternatively, you could tunnel nbd over ipsec or another tunnel.
2. Install NBD server on the untrusted server.

apt-get install nbd-server

3. Create a disk to serve.

You can either use an existing partition, or create a file to use as a disk:

dd if=/dev/zero of=/opt/encdrive bs=1M count=10240

4. Configure NBD to serve the drive

For this, you can either try using nbdkit if you have it:

nbdkit file file=/opt/encdrive

If that doesn't work for you (it didn't always for me), you can also configure NBD manually:

chown nbd:nbd /opt/encdrive
cat << EOF > /etc/nbd-server/config 
[generic]
	user = nbd
	group = nbd
	includedir = /etc/nbd-server/conf.d
	listenaddr = 192.168.56.101
	port = 1043
	authfile = /etc/nbd-server/allow
# What follows are export definitions.
[bob]
	exportname = /var/bobsdisk
EOF
nbd-server -C /etc/nbd-server/config

That's all you need to do on the cloud system.
Next, install the prerequisites on your system; the trusted system:

apt-get install nbd-client btrfs-tools
modprobe -v nbd

Connect to the remote NBD server to create a local block device that mirrors the remote drive:

nbd-client 192.168.1.11 1043 /dev/nbd0 -N bob

Set up an encrypted mapped device on top of the remote block device using luksFormat:

cryptsetup luksFormat /dev/nbd0

Connect to the encrypted mapped device you just set up:

cryptsetup luksOpen /dev/nbd0 backup_gateway

Format the newly created encrypted block device with Btrfs

mkfs.btrfs /dev/mapper/backup_gateway

Create a directory to mount to, and mount the new filesystem:

mkdir /mnt/encdrive
mount -t btrfs /dev/mapper/backup_gateway /mnt/mydrive

Now you have an encrypted, authenticated, yet live remote filesystem in the cloud. When you want to disconnect, simply run:

umount /mnt/mydrive
cryptsetup luksClose /dev/mapper/backup_gateway
nbd-client -d /dev/nbd0

To unmount, turn off the encrypted mapping, and detach the networked block device. When you want to re-connect, run:

nbd-client 192.168.1.11 1043 /dev/nbd0 -N bob
cryptsetup luksOpen /dev/nbd0 backup_gateway
mount /dev/mapper/backup_gateway /mnt/mydrive

And you'll be back in business.

One of the other advantages you get from this arrangement is all of the flexibility, power, and, if you want it, redundancy of Btrfs. You could establish multiple encrypted nbd devices to many cloud servers in many different countries run by different providers, and add them all as volumes to your Btrfs drive in a RAID configuration so that if any evil cloud provider tried to to mess with your data, any modifications would not only be detected by Btrfs, but the errors would also be automatically fixed and your applications would continue to run seamlessly. You can even easily add volumes to the Btrfs filesystem while it is mounted to increase the available size of the filesystem on the fly.