Computer crime is rampant and quite profitable. Websites have long been defaced for fun or viruses and worms released to cause disruptions.  Now there’s an incentive for criminals to avoid detection, to add your machines to a botnet for hire, or hide spam and ad links in your Wordpress blog.

This is why it’s super critical to us as at AnyHosting to focus on security, and follow best practices like secure backups, regular audits, intrusion detection, automatic updates, etc.

Our original goal was to provide websites for multiple users on one host, keeping them compartmentalized from each other, and to restrict break-ins. We chose to do reverse proxying using Apache, with separate Apache instances for each user, using mod_chroot.

Pros:

• user cannot see or read/write other users files, even on the same shared server, or use any system resources not explicitly provided in the chroot.
• mod_chroot is way less work than building a full chroot jail, and is potentially safer since you don’t have to provide system libraries or binaries, reducing your attack surface

Cons:

• multiple Apache instances consume more memory, and need special configuration and startup scripts
• it is possible to break out of a chroot. It is especially easy if the attacker can somehow get elevated to root (e.g. local security exploit)

Further investigation:

• apache-mpm-itk – This runs Apache child processes as a separate user per vhost. Cannot be combined with mod_chroot, unfortunately, and has some serious drawbacks (see the section on Quirks and Warnings).

• solaris containers, freebsd jail, linux vserver – These are real jails; less overhead than Virtual Machines, but provide more protection than a chroot. These tend to be significantly more complex to set up than something like mod_chroot, but are by design a lot harder to escape than a chroot.

There are some other interesting security-related modules such as mod_evasive that I am checking into further, which may mitigate some of the “cons” above.

Hopefully this will be useful to someone, however security is a process, not a product, and you should not rely solely on something like mod_chroot, but it can be a useful tool as part of an overall strategy that includes monitoring, intrusion detection, and regular security updates.

Ubuntu 8.10 Server

root@theo:~# apt-get install apache2 libapache2-mod-chroot

For each domain, create a virtual host config like /etc/apache2/sites-available/example.com:

<VirtualHost *:80>
ServerAdmin webmaster@example.com
ServerName example.com
ServerAlias www.example.com

ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

<Proxy *>
  Allow from all
</Proxy>

ErrorLog /var/www/example.com/logs/error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/www/example.com/logs/access.log combined

</VirtualHost>

This will be used by the Apache proxy, which is the normal system apache2 running as the default “www-data” user. This proxy handles name-based virtual hosts, and proxies the requests to a second process, running at localhost on port 8080.

Note that the logs are configured here and not in the user’s Apache process, for two reasons:

1. keep logs pristine in the event of a break-in on a user site (for example via a buggy or malicious PHP script)
2. single system-wide log file analysis process instead of per-user

Next, create a user account for each domain:

root@theo:~# mkdir /var/www/example.com
root@theo:~# cd /var/www/example.com/
root@theo:/var/www/example.com# useradd -d /var/www/example.com/htdocs example.com
root@theo:/var/www/example.com# mkdir htdocs logs conf
root@theo:/var/www/example.com# chown www-data:www-data logs
root@theo:/var/www/example.com# chown example.com:example.com htdocs/

Create the following in /var/www/example.com/conf/apache2.conf:

ServerRoot "/var/www/example.com/"
RequestHeader set Host example.com
LockFile /var/www/example.com/conf/accept.lock
PidFile /conf/apache2.pid
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
User example.com
Group example.com
AccessFileName .htaccess
<Files ~ "^\.ht">
  Order allow,deny
  Deny from all
</Files>

DefaultType text/plain
HostnameLookups Off
ErrorLog /var/www/example.com/conf/error.log
LogLevel warn
LoadFile /lib/libnss_dns.so.2
LoadModule chroot_module /usr/lib/apache2/modules/mod_chroot.so
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Listen 8080
DocumentRoot /htdocs
ChrootDir /var/www/example.com/

Start up the user Apache process:

root@theo:~# apache2 -f /var/www/example.com/conf/apache2.conf -k start

Everything should now work, but you may notice some problems with PHP or CGI scripts. Generally this means that you will need to make parts of the system available in the chroot; however making copies takes up a lot of space and can quickly get out of date, which is a security risk.

One workaround is to provide needed directories using the “mount -o bind” option, which will remount an existing, mounted part of the file hierarchy somewhere else. For example, this will provide all of “/usr” inside the chroot:

root@theo:~# mkdir /usr /var/www/example.com/usr
root@theo:~# mount -o bind,ro /usr /var/www/example.com/usr

PHP sessions require /tmp to exist inside the chroot; there is no benefit to sharing this one, an empty directory is fine.

That’s pretty much it! Testing, getting startup right etc. is left as an exercise for the reader, since we’re venturing a bit out of the standard Ubuntu Apache setup. Entries in /etc/fstab and creating an init script to handle user Apache processes is probably the best way to go.

In part 3 we’ll cover take a more general look at the pros and cons to this setup, as well as possible future directions.

EDIT 2009-Oct-03 2:25 PM Pacific – create dir before cding into it; add read-only (ro) option to usr mount; override “host” header in user Apache process

EDIT 2009-Oct-05 4:41 PM Pacific – load resolver library and install caching proxy dns server, needed by popular wordpress anti-spam plugin Akismet

At AnyHosting we want to have multiple users on one shared server, but protect them from reading or writing each other’s files (whether accidental or intentional) and also limit damage caused by a break-in via a poorly configured PHP script or CGI, or even an Apache exploit.

The solution here is to have each customer run their own Apache process, and do name-based virtual hosting and forwarding via a reverse proxy. The proxy is currently Apache but we are also considering more scalable alternatives such as nginx. An important thing to note here is that the proxy could be on a separate machine and also combine load-balancing into the mix, so it provides a lot of flexibility.

The classic way to chroot Apache (or any server process) is to copy the server process and all of it’s dependencies into the chroot jail, which makes keeping the files up to date not just a depressing chore but also a serious security hazard. Also it’s a ton of work to get right, especially if you want to support server-side scripting like PHP, since it depends on lots of system files and libraries.

Enter mod_chroot. This Apache module runs in the user’s process, and does the chroot system call after opening all the files it needs. The only caveats I’ve found (besides the ones on the mod_chroot caveats page) are some files that the Ubuntu PHP install assumes it can reach (such as the MySQL server socket, timezone info, and random number generator). I will cover this in more detail in part 2.

I have heard and read a lot about App Engine, so I knew roughly what to expect, but I am still impressed with it. It is a very simple model, it’s basically CGI with a 10-second limit. Only the Python programming language is supported right now (although they plan to add more), and the Django web framework is pre-installed. There is a nice little SDK for running the environment locally, which I just noticed is open-source as well (Apache license).

The really incredible thing about this is that it runs on and takes advantage of Google’s massive server infrastructure. In-memory or persistent storage is super fast and easy to use, and no need to worry about redundancy of individual servers (this is probably why they use the CGI+shared storage model, way simpler to distribute applications on-demand).

Today the roadmap was updated to include a few very cool features coming later this year:

• Support for running scheduled tasks
• Task queues for performing background processing
• Ability to receive and process incoming email
• Support for sending and receiving XMPP (Jabber) messages

This environment being so easy to use and the cost being low due, which is likely because the price of hosting so marginal to Google (I imagine that they are effectively outsourcing spare capacity) plus these new features pretty much replace the need for a traditional shared or dedicated server.

They haven’t yet started charging for the service, but proposed pricing is available, and they plan to start charging this year. The price is quite low considering the feature set, is pay-per-use, and is comparable with the popular cloud computing service Amazon Web Services (AWS).

The difference between this and something like AWS is that while it is much easier to get from start to finish on Google App Engine, one must (likely) re-write your application in Python, using Google’s libraries. You’ve got less flexibility than a shared PHP host, for example; you can’t easily take your code elsewhere. AWS is on the other end of the spectrum, more like dedicated servers where you can install anything you want: Linux or Windows, PHP or .Net, etc.

In any case I highly recommend checking out Google App Engine, especially if you’re doing any new development. If you’re looking to move your existing servers to the cloud, then I think Amazon Web Services still has the edge here.

Note that you can’t host secure (SSL) sites via any kind of name-based virtual hosting, because SSL happens before HTTP.  However, you can do IP-based virtual hosting, meaning that you need an IP address for each secure site.

A simple approach is to set up Apache to proxy requests from your primary server to the actual client server. IBM Developerworks has an excellent article on this, complete with an image that underscores the compartmentalization aspect I was hitting on before:

In this illustration, the “backend servers” can be on separate hosts, virtual hosts, or the same host (as long as each web server instance uses a different HTTP port).

There are many aspects to security; what I’m talking about today is compartmentalization -  that is, either physically or virtually separating customers from each other (into separate “compartments”), so that a break-in to one service does not result in a break in to all of them.

When you’re doing managed hosting of separate physical servers (or even virtual private servers), there’s generally a lot less to worry about – a malicious user getting access to one customer’s server does not generally lead to a break-in of all your customer’s servers, unless you’ve been lax with password or network security (e.g. not using encryption inside your network, using the same passwords everywhere, etc.).

However, virtual hosting presents much more of a challenge to secure properly, as the whole point is to use the same server instance for multiple sites. The primary problem is that most servers operating systems that use Apache (the most popular web server on the net right now) run all websites as the same user, which means that if you allow dynamic scripts to be run (CGI, PHP, etc.) then all users have read/write access to all files on that server.

The halfway point between VPS and virtual hosting is operating system-level virtualization. You’ve got different choices depending on OS: Solaris Containers, FreeBSD Jails, and Linux VServers  are all popular choices. These each give the customer a private filesystem and process list, as well as allowing the web host to limit resources like CPU, memory and I/O.

A solution of last resort is the Unix command “chroot“, which at least separates the filesystem from separate users, but does not allow the host to control resources and care must be taken to prevent users from breaking out. Tools like jailkit can be helpful, if you decide that you need to go this route.

The safest and easiest way to host separate sites is of course separate physical servers for each client; it’s also of course the most expensive. The cost/effort ratio continues in the same direction as you look at VPS, OS-level virtualization, chroot and finally no separation (this is only applicable if you either manage everything yourself and do not allow user uploads, or don’t allow dynamic scripting of any kind – however, you still run the risk of a break-in to one site affecting all of your customers).

(EDIT: fixed jailkit link)