Securing a virtualisation server

I had to install a virtualisation server recently that would host a couple of mail servers and provide a temporary disk-to-disk backup facility for the VMs. I thought that I should probably harden it and put some type of security around it as well; lock down the extraneous services, put in a host-level firewall, configure a rootkit detector and some of that file integrity checking stuff and plug it into the monitoring and backup systems. Easier said than done.

I started off by drawing a picture of the host and VMs and labelling what services would need to be provided from where, to where. I added the services that the host and VMs would need to consume, such as DNS, LDAPS and NTP and overlayed that information. I was running the VMs in a virtual DMZ, so that added some complexity, but ten minutes later I had a fairly comprehensive diagram that would form the basis of my firewall ruleset. I’m truly paranoid when it comes to firewalls, so I like to block all outgoing traffic by default as well as incoming and only permit traffic that I know and want.

Next up was hardening the virtualisation server itself. I had chosen a Linux operating system for both stability and security reasons and so set about with the normal tasks of hardening a UNIX type OS – strip down the kernel to bare essentials, lock the BIOS and password protect boot-time modifications to the boot-loader, chroot jail all essential daemons, remove all unnecessary services and user accounts and tighten up all file access controls. In the past, I’ve played with full disk encrypted file systems but found that the performance hit on high disk IO transaction volume systems could be quite significant, so I compromised with a dedicated partition for the local home directories and the temporary backup files and encrypted that. I also encrypted the /tmp and swap partitions and configured quotas for maximum permitted disk usage by users.

Maybe it’s my background, but I prefer to use a command line to administer servers – green text on a black background is soothing in more ways than one. Pretty pictures and GUIs confuse me at times and seem to be designed for people who probably shouldn’t be administering boxes in the first place. I digress. SSH, the secure remote shell protocol of champions, is probably the most widely used remote access protocol for UNIX-like operating systems. It is also one of the most frequently attacked, with basic dictionary attacks being one of the ‘script kiddies’ favourites. As a general rule of thumb, if you don’t allow root login by SSH and user’s passwords are suitably complex, then dictionary attacks shouldn’t pose a significant threat, however they do get annoying and fill up your log files. If you absolutely must expose your SSH daemon to the internet or other public network, there are a couple of approaches to reduce the impact of this: firstly, use the public-key cryptography capabilities of SSH to authenticate users instead of passwords and secondly, you can use packages such as sshguard to detect dictionary attacks against SSH and temporarily blacklist the source address.

These steps prepared a system that was hardened and reasonably secured right now. One of the challenges was to keep it that way and notice if anything went awry. As per all the standard advice, security fixes need to be applied as early as possible. Most operating systems have some form of mechanism for this and the Gentoo Linux distribution that I was using is no exception. Gentoo has a fantastic tool called glsa-check which checks and applies only those patches for software that you have installed (unlike some OSes); I added a cron job to check and apply security patches every day.

Monitoring is an important part of maintaining a system’s stability and overall health. Any system can produce a huge volume of logs and manually inspecting them is incredibly time-consuming and risks overlooking important entries through sheer boredom. Manually inspecting the logs and other quotients, such as CPU utilisation, backup completion status and free disk space on multiple systems is a punishment that should be reserved for only the very worst class of IT offenders, such as <flamebait> Darl McBride </flamebait>. The far better solution is using a monitoring system, which checks your logs, how much disk space you have running, monitors whether all your services are running and so forth. Most importantly of all, these systems report by exception only, so they don’t waste your time telling you that things are all OK and most allow you to script automated responses to particular scenarios, such as restarting a failed service, raising a job ticket for the fault and SMSing the appropriate engineer. They also help with watching trends for capacity planning and SLA (Service Level Agreement) compliance. My personal favourite solution is Nagios, so I installed the client agent (called an NRPE addon in Nagios) and configured the various services and alerts I wanted on my Nagios server.

If your system does become compromised, then you really want to know what has been done, to what and whether a rootkit has been installed to facilitate the baddies using you box for other nefarious tasks. Enter logsentry, rkhunter and AIDE (Advanced Intrusion Detection Environment). One of the first things that an attacker will do if they compromise your box is to edit the logs to remove evidence of their attack and entry. Logsentry is a tool that scans your logs and detects deletion and anomalies. Rkhunter detects Linux rootkits and can email daily reports and AIDE is a HIDS (Host-based Intrusion Detection System) that uses a database of file digests to detect changes to the filesystem.

Getting back to the virtualisation bit. I’d reasonably secured the host and network traffic, but now needed to consider what needed to be done in relation to the VMs. They were going to be mail servers. Mail servers tend to have a few common attack vectors; vulnerabilities in the mail server code itself, spammers using misconfigured open relays and so-called ‘whale bombing’ a type of DoS (Denial of Service) attack. I was going to use qmail for the mail servers, which is well known for its security (no known vulnerabilities since 1996 – an impressive record for any software!) so I wasn’t so worried about code flaws or open relays (qmail doesn’t relay by default – you have to open it up deliberately). I was mainly concerned about ‘whale bombing’ and other DoS/DDoS attacks. ‘Whale bombing’ (I love the name!) refers to the practise of sending ridiculously large files to a particular mail server in an attempt to deplete resources at the targeted server. The file that was frequently used was a video of a dead, beached whale that was blown up with dynamite and resulted in 30 seconds of blubber raining down over the foreshore – hence the reference to whales.

Another common mail server DoS/DDoS variant is sending ‘zip bombs’ where a single character was repeated a stupidly large number of times and the resultant file then compressed; the compressed file is very small, but the mail server’s anti-spam and anti-virus scanners are often overwhelmed once the file is uncompressed. There is no easy preventative solution to these attacks, but they can be detected and mitigated quite easily through a couple of approaches: rate limiting, disc spooling and the use of RBL ([email] Routing BlackLists),  SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail). Rate limiting incoming traffic dynamically is typically applied at the perimeter (in this case the virtualisation server), or in combination with the perimeter to decrease the volume of mail traffic going to a particular mail server as the incoming traffic increases. This works especially well when combined with multiple backend mail servers, when other load balancing techniques can be used to reduce the impact of such attacks. On the mail servers themselves, spooling incoming mail to disc rather than to memory may be slower, but avoids memory depletion and allows a more orderly handling of large mail volumes. Similarly the pre-processing of incoming mail according to RBL, SPF & DKIM rules can significantly cut down SPAM and forged DoS and DDoS attacks.

These steps are not perfect by a long way, but are enough to keep most attackers busy for long enough for us to detect and respond to the attack, which is the main purpose of most security measures.

Next week, I’ll explain DKIM and how to open an oyster with a bus ticket...............