AnyHosting

In one of my earlier posts about web hosting with Ubuntu Server, I left the startup script for the user-owned Apache instances as an exercise for the reader; the gist of it is to create a script in /etc/init.d/ (based on the “skeleton” file in that directory). The meat of it is something like:

This iterates over the directories in /var/www and runs Apache for each one, each with it’s own config file.

One interesting thing you can do here is make the user’s Apache process “nice” to fellow processes:

“nice” changes priority for tasks waiting on CPU, while “ionice” changes the priority for tasks waiting on IO (for example disk reads and writes).

This can keep one busy or misbehaving server from interrupting other sites or important background tasks; alternatively, you could set one site to be highest-priority.

I have not found much on this subject since my last post, so instead of continuing to wait for someone else to do it I have started working on a new open-source web hosting control panel (alternative to Plesk, cPanel, etc).

All the work is in my local repo, but I will be pushing to github as milestones are hit.

I want to get the basics right, and not worry right now about competing feature-for-feature with the big guys:

• basic file management
• configure apache, stop/start server
• allow (and enthusiastically support!) plugins. I am using django/python, should be no problem.

That’s for the user-facing side, the backend takes care of scaling across multiple hosts (on-demand scaling), billing, activating/deactivating accounts, you get the idea.

What do you want to see in a free, open-source web hosting control panel? Leave a comment or feel free to email me – rhelmer@anyhosting.com – Thanks!

I have chosen Nagios to keep track of the anyhosting.com network. There are many alternatives (some I have explored and some not yet), what I like about Nagios:

• I’ve been using it for a long time; familiarity
• very simple/powerful plugin system
• tons of users, so lots of examples and plugins already available

Nagios version 3 is provided in the Ubuntu repositories, and is quite simple to install:

The default config comes set up to monitor a set of services on localhost; I don’t really like the default Ubuntu/Debian setup of having one config file per host/service/etc, so on the master I’ve replaced the config file structure:

groups.conf contains the set of server types that I care about:

Note that the “http-servers” can define “members” (localhost in this case), however in general I do not add members in this file but instead in the hosts.cfg:

Note the “hostgroups” line; anyhosting1 is the physical server (this monitor is really checking on the reverse proxy), and example.com is a vhost (which is really proxying to a user running Apache for the “example.com” domain). These two checks make sure that the whole system is working and proxying correctly.

Finally, services.cfg brings it all together by defining which groups should run which services:

The Ubuntu nagios-plugins package (which by default is installed along with the nagios3 package) contains plugins that can intelligently check MySQL databases, disk space, load average, etc. By default these only work on the local machine, but these can be made to run on remote machines by installing the nagios-nrpe-server package. I will cover this further in a future blog post.

Apparently it is “National Cybersecurity Awareness Month” in the US.

Computer crime is rampant and quite profitable. Websites have long been defaced for fun or viruses and worms released to cause disruptions.  Now there’s an incentive for criminals to avoid detection, to add your machines to a botnet for hire, or hide spam and ad links in your WordPress blog.

This is why it’s super critical to us as at AnyHosting to focus on security, and follow best practices like secure backups, regular audits, intrusion detection, automatic updates, etc.

Just wanted to point out another excellent post from the Blog O’ Matty on centralized logging with syslog-ng.

I actually helped to set up real-time web analysis with syslog-ng (using TCP) and a slightly hacked webalizer (it was ignoring multiple hits happening on the same second) from a FreeBSD/Apache web farm ~10 years ago, and have been looking into it again for my current logging needs.

His blog has consistently awesome posts (if you’re interested in systems administration), and as your doctor I highly suggest that you subscribe.

In this third and final installment, we’ll look at alternatives and future directions for shared web hosting on Ubuntu Server (go back in time to part 1 and part 2 in case you missed them).

Our original goal was to provide websites for multiple users on one host, keeping them compartmentalized from each other, and to restrict break-ins. We chose to do reverse proxying using Apache, with separate Apache instances for each user, using mod_chroot.

Pros:

• user cannot see or read/write other users files, even on the same shared server, or use any system resources not explicitly provided in the chroot.
• mod_chroot is way less work than building a full chroot jail, and is potentially safer since you don’t have to provide system libraries or binaries, reducing your attack surface

Cons:

• multiple Apache instances consume more memory, and need special configuration and startup scripts
• it is possible to break out of a chroot. It is especially easy if the attacker can somehow get elevated to root (e.g. local security exploit)

Further investigation:

• apache-mpm-itk – This runs Apache child processes as a separate user per vhost. Cannot be combined with mod_chroot, unfortunately, and has some serious drawbacks (see the section on Quirks and Warnings).

• solaris containers, freebsd jail, linux vserver – These are real jails; less overhead than Virtual Machines, but provide more protection than a chroot. These tend to be significantly more complex to set up than something like mod_chroot, but are by design a lot harder to escape than a chroot.

There are some other interesting security-related modules such as mod_evasive that I am checking into further, which may mitigate some of the “cons” above.

Hopefully this will be useful to someone, however security is a process, not a product, and you should not rely solely on something like mod_chroot, but it can be a useful tool as part of an overall strategy that includes monitoring, intrusion detection, and regular security updates.

Continuing from part 1, here are the nuts and bolts:

Ubuntu 8.10 Server

For each domain, create a virtual host config like /etc/apache2/sites-available/example.com:

This will be used by the Apache proxy, which is the normal system apache2 running as the default “www-data” user. This proxy handles name-based virtual hosts, and proxies the requests to a second process, running at localhost on port 8080.

Note that the logs are configured here and not in the user’s Apache process, for two reasons:

1. keep logs pristine in the event of a break-in on a user site (for example via a buggy or malicious PHP script)
2. single system-wide log file analysis process instead of per-user

Next, create a user account for each domain:

Create the following in /var/www/example.com/conf/apache2.conf:

Start up the user Apache process:

Everything should now work, but you may notice some problems with PHP or CGI scripts. Generally this means that you will need to make parts of the system available in the chroot; however making copies takes up a lot of space and can quickly get out of date, which is a security risk.

One workaround is to provide needed directories using the “mount -o bind” option, which will remount an existing, mounted part of the file hierarchy somewhere else. For example, this will provide all of “/usr” inside the chroot:

PHP sessions require /tmp to exist inside the chroot; there is no benefit to sharing this one, an empty directory is fine.

That’s pretty much it! Testing, getting startup right etc. is left as an exercise for the reader, since we’re venturing a bit out of the standard Ubuntu Apache setup. Entries in /etc/fstab and creating an init script to handle user Apache processes is probably the best way to go.

In part 3 we’ll cover take a more general look at the pros and cons to this setup, as well as possible future directions.

UPDATE link to secure shared hosting on ubuntu server part 3

EDIT 2009-Oct-03 2:25 PM Pacific – create dir before cding into it; add read-only (ro) option to usr mount; override “host” header in user Apache process

EDIT 2009-Oct-05 4:41 PM Pacific – load resolver library and install caching proxy dns server, needed by popular wordpress anti-spam plugin Akismet

We’ve gone over securing a shared hosting setup briefly in the past, but I have made some improvements and simplifications that I’d like to share. This is part 1 of 3, an introduction to secure shared web hosting on Ubuntu Server.

At AnyHosting we want to have multiple users on one shared server, but protect them from reading or writing each other’s files (whether accidental or intentional) and also limit damage caused by a break-in via a poorly configured PHP script or CGI, or even an Apache exploit.

The solution here is to have each customer run their own Apache process, and do name-based virtual hosting and forwarding via a reverse proxy. The proxy is currently Apache but we are also considering more scalable alternatives such as nginx. An important thing to note here is that the proxy could be on a separate machine and also combine load-balancing into the mix, so it provides a lot of flexibility.

The classic way to chroot Apache (or any server process) is to copy the server process and all of it’s dependencies into the chroot jail, which makes keeping the files up to date not just a depressing chore but also a serious security hazard. Also it’s a ton of work to get right, especially if you want to support server-side scripting like PHP, since it depends on lots of system files and libraries.

Enter mod_chroot. This Apache module runs in the user’s process, and does the chroot system call after opening all the files it needs. The only caveats I’ve found (besides the ones on the mod_chroot caveats page) are some files that the Ubuntu PHP install assumes it can reach (such as the MySQL server socket, timezone info, and random number generator). I will cover this in more detail in part 2.

UPDATE link to secure shared hosting on ubuntu server part 2

I’ve just pushed a new front page to anyhosting.com with more details.

If you’re looking for simple, secure and affordable web hosting, check it out!

I’m also working on a series of blog posts and articles about the new setup (reverse proxy, mod_chroot, on ubuntu server).

I’ve been shopping around for virtual web hosting control panels. At this point, I am interested only in providing end-users things like file management, basic database administration, pre-defined package mangement.

They seem to fall into three broad categories:

1. expensive proprietary packages
2. free open-source packages
3. custom software developed by hosting companies

The leader in #1 seems to be cPanel. I am not at all impressed with their demo, it looks very cluttered to me and I really want something simple, easy and aesthetically pleasing.

I have checked out all of the free/open-source web hosting control panels that Wikipedia lists, and I am also very displeased with the UI, and the code doesn’t look very easy to jump into to me. There seems to be a lot of custom code (I’d be happiest with something based on Django, but the PHP ones could at least use Cake). This makes me a little worried on the security aspect (XSS, SQL injection, etc.). Most of these code bases seem to be very old and not necessarily very active.

So is #3 the way to go? I’ve seen and heard about lots of great hosting control panels that only exist behind close doors, is this the big differentiator for modern hosting companies?