AnyHosting

I have chosen Nagios to keep track of the anyhosting.com network. There are many alternatives (some I have explored and some not yet), what I like about Nagios:

• I’ve been using it for a long time; familiarity
• very simple/powerful plugin system
• tons of users, so lots of examples and plugins already available

Nagios version 3 is provided in the Ubuntu repositories, and is quite simple to install:

root@admin:~# apt-get install nagios3

The default config comes set up to monitor a set of services on localhost; I don’t really like the default Ubuntu/Debian setup of having one config file per host/service/etc, so on the master I’ve replaced the config file structure:

root@admin:/etc/nagios3/conf.d# cd /etc/nagios3/conf.d/
root@admin:/etc/nagios3/conf.d# ls
contacts.cfg  extinfo.cfg  groups.cfg  hosts.cfg  services.cfg  timeperiods.cfg

groups.conf contains the set of server types that I care about:

# A list of your web servers
define hostgroup {
hostgroup_name  http-servers
alias           HTTP servers
members         localhost
}

# A list of your mysql servers
define hostgroup {
hostgroup_name  mysql-servers
alias           MySQL servers
}

# A list of your VHosts
define hostgroup {
hostgroup_name  http-vhosts
alias           Virtual Host HTTP servers
}

Note that the “http-servers” can define “members” (localhost in this case), however in general I do not add members in this file but instead in the hosts.cfg:

define host {
host_name   anyhosting1
address     1.2.3.4
use         generic-host
hostgroups  http-servers
}

define host {
host_name   example.com
address     1.2.3.4
use         generic-host
hostgroups  http-vhosts
}

Note the “hostgroups” line; anyhosting1 is the physical server (this monitor is really checking on the reverse proxy), and example.com is a vhost (which is really proxying to a user running Apache for the “example.com” domain). These two checks make sure that the whole system is working and proxying correctly.

Finally, services.cfg brings it all together by defining which groups should run which services:

# check that web services are running
define service {
hostgroup_name                  http-servers
service_description             HTTP
check_command                   check_http
use                             generic-service
notification_interval           0 ; set > 0 if you want to be renotified
}
define service {
hostgroup_name                  http-vhosts
service_description             Virtual Host HTTP
check_command                   check_httpname
use                             generic-service
notification_interval           0 ; set > 0 if you want to be renotified
}

The Ubuntu nagios-plugins package (which by default is installed along with the nagios3 package) contains plugins that can intelligently check MySQL databases, disk space, load average, etc. By default these only work on the local machine, but these can be made to run on remote machines by installing the nagios-nrpe-server package. I will cover this further in a future blog post.

In this third and final installment, we’ll look at alternatives and future directions for shared web hosting on Ubuntu Server (go back in time to part 1 and part 2 in case you missed them).

Our original goal was to provide websites for multiple users on one host, keeping them compartmentalized from each other, and to restrict break-ins. We chose to do reverse proxying using Apache, with separate Apache instances for each user, using mod_chroot.

Pros:

• user cannot see or read/write other users files, even on the same shared server, or use any system resources not explicitly provided in the chroot.
• mod_chroot is way less work than building a full chroot jail, and is potentially safer since you don’t have to provide system libraries or binaries, reducing your attack surface

Cons:

• multiple Apache instances consume more memory, and need special configuration and startup scripts
• it is possible to break out of a chroot. It is especially easy if the attacker can somehow get elevated to root (e.g. local security exploit)

Further investigation:

• apache-mpm-itk – This runs Apache child processes as a separate user per vhost. Cannot be combined with mod_chroot, unfortunately, and has some serious drawbacks (see the section on Quirks and Warnings).

• solaris containers, freebsd jail, linux vserver – These are real jails; less overhead than Virtual Machines, but provide more protection than a chroot. These tend to be significantly more complex to set up than something like mod_chroot, but are by design a lot harder to escape than a chroot.

There are some other interesting security-related modules such as mod_evasive that I am checking into further, which may mitigate some of the “cons” above.

Hopefully this will be useful to someone, however security is a process, not a product, and you should not rely solely on something like mod_chroot, but it can be a useful tool as part of an overall strategy that includes monitoring, intrusion detection, and regular security updates.

Continuing from part 1, here are the nuts and bolts:

Ubuntu 8.10 Server

root@theo:~# apt-get install apache2 libapache2-mod-chroot

For each domain, create a virtual host config like /etc/apache2/sites-available/example.com:

<VirtualHost *:80>
ServerAdmin webmaster@example.com
ServerName example.com
ServerAlias www.example.com

ProxyPass / http://localhost:8080/
ProxyPassReverse / http://localhost:8080/

<Proxy *>
  Allow from all
</Proxy>

ErrorLog /var/www/example.com/logs/error.log

# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
LogLevel warn

CustomLog /var/www/example.com/logs/access.log combined

</VirtualHost>

This will be used by the Apache proxy, which is the normal system apache2 running as the default “www-data” user. This proxy handles name-based virtual hosts, and proxies the requests to a second process, running at localhost on port 8080.

Note that the logs are configured here and not in the user’s Apache process, for two reasons:

1. keep logs pristine in the event of a break-in on a user site (for example via a buggy or malicious PHP script)
2. single system-wide log file analysis process instead of per-user

Next, create a user account for each domain:

root@theo:~# mkdir /var/www/example.com
root@theo:~# cd /var/www/example.com/
root@theo:/var/www/example.com# useradd -d /var/www/example.com/htdocs example.com
root@theo:/var/www/example.com# mkdir htdocs logs conf
root@theo:/var/www/example.com# chown www-data:www-data logs
root@theo:/var/www/example.com# chown example.com:example.com htdocs/

Create the following in /var/www/example.com/conf/apache2.conf:

ServerRoot "/var/www/example.com/"
RequestHeader set Host example.com
LockFile /var/www/example.com/conf/accept.lock
PidFile /conf/apache2.pid
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 15
User example.com
Group example.com
AccessFileName .htaccess
<Files ~ "^\.ht">
  Order allow,deny
  Deny from all
</Files>

DefaultType text/plain
HostnameLookups Off
ErrorLog /var/www/example.com/conf/error.log
LogLevel warn
LoadFile /lib/libnss_dns.so.2
LoadModule chroot_module /usr/lib/apache2/modules/mod_chroot.so
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf
Listen 8080
DocumentRoot /htdocs
ChrootDir /var/www/example.com/

Start up the user Apache process:

root@theo:~# apache2 -f /var/www/example.com/conf/apache2.conf -k start

Everything should now work, but you may notice some problems with PHP or CGI scripts. Generally this means that you will need to make parts of the system available in the chroot; however making copies takes up a lot of space and can quickly get out of date, which is a security risk.

One workaround is to provide needed directories using the “mount -o bind” option, which will remount an existing, mounted part of the file hierarchy somewhere else. For example, this will provide all of “/usr” inside the chroot:

root@theo:~# mkdir /usr /var/www/example.com/usr
root@theo:~# mount -o bind,ro /usr /var/www/example.com/usr

PHP sessions require /tmp to exist inside the chroot; there is no benefit to sharing this one, an empty directory is fine.

That’s pretty much it! Testing, getting startup right etc. is left as an exercise for the reader, since we’re venturing a bit out of the standard Ubuntu Apache setup. Entries in /etc/fstab and creating an init script to handle user Apache processes is probably the best way to go.

In part 3 we’ll cover take a more general look at the pros and cons to this setup, as well as possible future directions.

EDIT 2009-Oct-03 2:25 PM Pacific – create dir before cding into it; add read-only (ro) option to usr mount; override “host” header in user Apache process

EDIT 2009-Oct-05 4:41 PM Pacific – load resolver library and install caching proxy dns server, needed by popular wordpress anti-spam plugin Akismet

We’ve gone over securing a shared hosting setup briefly in the past, but I have made some improvements and simplifications that I’d like to share. This is part 1 of 3, an introduction to secure shared web hosting on Ubuntu Server.

At AnyHosting we want to have multiple users on one shared server, but protect them from reading or writing each other’s files (whether accidental or intentional) and also limit damage caused by a break-in via a poorly configured PHP script or CGI, or even an Apache exploit.

The solution here is to have each customer run their own Apache process, and do name-based virtual hosting and forwarding via a reverse proxy. The proxy is currently Apache but we are also considering more scalable alternatives such as nginx. An important thing to note here is that the proxy could be on a separate machine and also combine load-balancing into the mix, so it provides a lot of flexibility.

The classic way to chroot Apache (or any server process) is to copy the server process and all of it’s dependencies into the chroot jail, which makes keeping the files up to date not just a depressing chore but also a serious security hazard. Also it’s a ton of work to get right, especially if you want to support server-side scripting like PHP, since it depends on lots of system files and libraries.

Enter mod_chroot. This Apache module runs in the user’s process, and does the chroot system call after opening all the files it needs. The only caveats I’ve found (besides the ones on the mod_chroot caveats page) are some files that the Ubuntu PHP install assumes it can reach (such as the MySQL server socket, timezone info, and random number generator). I will cover this in more detail in part 2.

I’ve just pushed a new front page to anyhosting.com with more details.

If you’re looking for simple, secure and affordable web hosting, check it out!

I’m also working on a series of blog posts and articles about the new setup (reverse proxy, mod_chroot, on ubuntu server).

I’ve been shopping around for virtual web hosting control panels. At this point, I am interested only in providing end-users things like file management, basic database administration, pre-defined package mangement.

They seem to fall into three broad categories:

1. expensive proprietary packages
2. free open-source packages
3. custom software developed by hosting companies

The leader in #1 seems to be cPanel. I am not at all impressed with their demo, it looks very cluttered to me and I really want something simple, easy and aesthetically pleasing.

I have checked out all of the free/open-source web hosting control panels that Wikipedia lists, and I am also very displeased with the UI, and the code doesn’t look very easy to jump into to me. There seems to be a lot of custom code (I’d be happiest with something based on Django, but the PHP ones could at least use Cake). This makes me a little worried on the security aspect (XSS, SQL injection, etc.). Most of these code bases seem to be very old and not necessarily very active.

So is #3 the way to go? I’ve seen and heard about lots of great hosting control panels that only exist behind close doors, is this the big differentiator for modern hosting companies?

Vendor lock-in is a serious concern for any business, and is a pretty tough problem in web hosting specifically.

All of the major control panels like cPanel, Plesk, etc. are licensed at substantial fees, which cut into what you as a web host must charge each user. There are open-source alternatives such as ISPConfig and  GNUPanel, but this of course means that you will be taking on a lot more of the support burden, although you are of course free to make and keep any customizations or enhancements that you like, unlike cPanel or Plesk.

Besides licensing fees, the other snare to recognize is switching costs. As the LAMP (Linux, Apache, MySQL, PHP) stack has grown in popularity, it has become quite easy to move from one web host to another. Don’t like DreamHost? Move to Linnode. Don’t like them? Try RimuHosting.

However with the slick new cloud computing services like Amazon’s EC2 and Google’s App Engine, there is a ton of opportunity and also some things to watch out for:

• EC2 can run a regular Linux VM, but the management tools and other services like S3 (storage), queueing, billing etc. will not work out-of-the-box elsewhere
• App Engine lets you use the free and open-source Python programming language and the popular Django web framework, but you must use Google’s storage service. It can be used in a very SQL-like way, or hidden behind Django’s ORM however

I am going to continue using both Amazon and Google’s services, but I am being very careful about putting all of my eggs in one basket. There are some impressive updates in the pipeline, but you might want to think twice about letting any one company collect the tolls on your users.

You can read more about vendor lock-in and switching costs at Wikipedia.

I’ve been trying out the cloud computing service Google App Engine for a simple dynamic site. I’ll publish more details on this as it gets further along.

I have heard and read a lot about App Engine, so I knew roughly what to expect, but I am still impressed with it. It is a very simple model, it’s basically CGI with a 10-second limit. Only the Python programming language is supported right now (although they plan to add more), and the Django web framework is pre-installed. There is a nice little SDK for running the environment locally, which I just noticed is open-source as well (Apache license).

The really incredible thing about this is that it runs on and takes advantage of Google’s massive server infrastructure. In-memory or persistent storage is super fast and easy to use, and no need to worry about redundancy of individual servers (this is probably why they use the CGI+shared storage model, way simpler to distribute applications on-demand).

Today the roadmap was updated to include a few very cool features coming later this year:

• Support for running scheduled tasks
• Task queues for performing background processing
• Ability to receive and process incoming email
• Support for sending and receiving XMPP (Jabber) messages

This environment being so easy to use and the cost being low due, which is likely because the price of hosting so marginal to Google (I imagine that they are effectively outsourcing spare capacity) plus these new features pretty much replace the need for a traditional shared or dedicated server.

They haven’t yet started charging for the service, but proposed pricing is available, and they plan to start charging this year. The price is quite low considering the feature set, is pay-per-use, and is comparable with the popular cloud computing service Amazon Web Services (AWS).

The difference between this and something like AWS is that while it is much easier to get from start to finish on Google App Engine, one must (likely) re-write your application in Python, using Google’s libraries. You’ve got less flexibility than a shared PHP host, for example; you can’t easily take your code elsewhere. AWS is on the other end of the spectrum, more like dedicated servers where you can install anything you want: Linux or Windows, PHP or .Net, etc.

In any case I highly recommend checking out Google App Engine, especially if you’re doing any new development. If you’re looking to move your existing servers to the cloud, then I think Amazon Web Services still has the edge here.

I have been following the excellent Data Center Knowledge blog lately, they have a good write-up on Cisco getting into the server business.

However it’s pretty unclear to me what “unified computing” means exactly, it seems like they are talking about some kind of virtualization approach, the end result being that you can buy all your server and networking gear from Cisco and have your own private cloud.

Hard to tell though, the NY times article is pretty fluffy (“New tools developed by VMWare, the market leader, make it possible to shuffle business applications around a data center just by pointing a computer mouse at an icon on the screen.”) and the Cisco blog posts are fairly grandiose (“If not us, who? If not now, when?”).

I guess I’ll just have to wait until they actually have something to sell. I would rather have a step towards commoditization and not away from it. If Cisco is going to sell solid-state servers with the same kind of support and reliability as their networking gear, then that sounds great. It so far does not sound like that is what this is.

One of the major concerns I have with running my own CMS like Joomla or Drupal (or blogging software like Wordpress for that matter) is keeping it up to date. These kinds of tasks are often seen as annoying busywork, and nobody wants to break a site that is working already, but having your site broken into and/or defaced via the CMS is a definite possibility.

This has happened to Wordpress quite a bit, which is one reason I advise people to use a hosted service if at all possible. The makers of Wordpress do their hosting, and you can find support on lots of low-cost web hosts.

Any recommendations for sites providing CMS hosting (including applying security updates)?