Archive for the ‘web services’ Category

new open-source web hosting control panel

Thursday, March 18th, 2010

I have not found much on this subject since my last post, so instead of continuing to wait for someone else to do it I have started working on a new open-source web hosting control panel (alternative to Plesk, cPanel, etc).

All the work is in my local repo, but I will be pushing to github as milestones are hit.

I want to get the basics right, and not worry right now about competing feature-for-feature with the big guys:

  • basic file management
  • configure apache, stop/start server
  • allow (and enthusiastically support!) plugins. I am using django/python, should be no problem.

That’s for the user-facing side, the backend takes care of scaling across multiple hosts (on-demand scaling), billing, activating/deactivating accounts, you get the idea.

What do you want to see in a free, open-source web hosting control panel? Leave a comment or feel free to email me – – Thanks!

Google App Engine becoming more useful

Sunday, February 8th, 2009

I’ve been trying out the cloud computing service Google App Engine for a simple dynamic site. I’ll publish more details on this as it gets further along.

I have heard and read a lot about App Engine, so I knew roughly what to expect, but I am still impressed with it. It is a very simple model, it’s basically CGI with a 10-second limit. Only the Python programming language is supported right now (although they plan to add more), and the Django web framework is pre-installed. There is a nice little SDK for running the environment locally, which I just noticed is open-source as well (Apache license).

The really incredible thing about this is that it runs on and takes advantage of Google’s massive server infrastructure. In-memory or persistent storage is super fast and easy to use, and no need to worry about redundancy of individual servers (this is probably why they use the CGI+shared storage model, way simpler to distribute applications on-demand).

Today the roadmap was updated to include a few very cool features coming later this year:

  • Support for running scheduled tasks
  • Task queues for performing background processing
  • Ability to receive and process incoming email
  • Support for sending and receiving XMPP (Jabber) messages

This environment being so easy to use and the cost being low due, which is likely because the price of hosting so marginal to Google (I imagine that they are effectively outsourcing spare capacity) plus these new features pretty much replace the need for a traditional shared or dedicated server.

They haven’t yet started charging for the service, but proposed pricing is available, and they plan to start charging this year. The price is quite low considering the feature set, is pay-per-use, and is comparable with the popular cloud computing service Amazon Web Services (AWS).

The difference between this and something like AWS is that while it is much easier to get from start to finish on Google App Engine, one must (likely) re-write your application in Python, using Google’s libraries. You’ve got less flexibility than a shared PHP host, for example; you can’t easily take your code elsewhere. AWS is on the other end of the spectrum, more like dedicated servers where you can install anything you want: Linux or Windows, PHP or .Net, etc.

In any case I highly recommend checking out Google App Engine, especially if you’re doing any new development. If you’re looking to move your existing servers to the cloud, then I think Amazon Web Services still has the edge here.

cloud computing reliability will not matter

Saturday, February 23rd, 2008

All the buzz about “cloud computing” is great, but isn’t it just a rehash of “dumb terminal”, “thin client” computing, that lost out big against the PC? Yes it is, but not for long; the browser does not need to be the modern equivalent of the terminal, chained to the call/response of HTTP requests in order to provide applications.

I wrote about this a while back, but I think it bears repeating.. HTML 5 includes support for “offline” applications, including client-side storage, which means that that in current and upcoming versions of Firefox, Safari, and Opera will support running web applications locally on the user’s computer, without needing to be in constant communication with the server.

Instead of asking your users to install your application in the traditional sense, visiting the website that hosts your application will cause the client to download and store everything needed to operate on the client side. The application can detect whether or not the computer is online, and attempt to connect to needed real-time, syncing, and other web services as needed, and only interrupt the user if absolutely necessary.

This means that the questionable reliability of having all of your applications hosted “in the cloud” is greatly mitigated, and impact on the end user is quite minimal. Even if your entire site is down, there’s no reason for that to interrupt the user of your snazzy application; in fact, with cross-site AJAX support, the user can continue to fetch and transmit data with other websites (I’m thinking a real-time price comparison site, or something like that, which today would be implemented completely server-side and just fall over in this scenario), so it may be totally acceptable for your site to receive the queued up responses from clients when it comes back up, depending on what your application does of course.

For IE support, you could use something like Google Gears or Adobe Flash’s offline capabilities, until Microsoft catches up to the rest of the world. This is the biggest pain point of the brave new offline world right now, however it’s a very real concern as Microsoft IE still has around 70% of the global web browser market share.. If this is something you need, check out Dojo’s storage classes as a high-level library to abstract away these details for you; if you’re doing a serious AJAX site nowadays you really should be using or at least intimiately familiar with the great toolkits like Dojo, Mochikit, JQuery, etc. There’s no need for handling each browser/version case by hand nowadays, unless you have a really good reason.

doing one thing well

Tuesday, February 19th, 2008

There’s a ton of business advice out there, and I tend to get one idea stuck in my head – do one thing, and do it well. There’s a similar idea behind the concept of the Unix shell and commands, although arguably fragmentation has muddied the waters quite a bit.

The tough part seems to be balancing what you think you do best against missed opportunities. Amazon saw the potential to lease out it’s datacenter, which is a huge shift toward developer-facing web services, from direct customer-facing web sites.

I don’t know yet whether this is a good or bad move for them, but it certainly takes guts. I wonder what kind of internal decision-making route this took, to get from someone’s idea to final implementation; it’s surprising to see this from a company already seeing huge success in it’s core business.

I guess I can see parallels in Google and Microsoft, who were able to parlay dominance in one area to success in others. Still, they tend to only actually be wildly successful in one area, and use that to support ventures in the others.

utility computing

Tuesday, January 22nd, 2008

Not sure how I missed 3tera before, they look like an interesting entrant to the “utility computing” market, which everyone is excited about again since Amazon’s got into it.

I haven’t really done my due diligence on this company yet, but I checked out the slick demo  (bonus points that it shows their app running in Firefox, takes the worry of cross-browser compatbility off the table) and have been reading back issues of the blog, and it sounds like a pretty solid offering.

If you don’t have time to watch the demo, in a nutshell it looks like you can architect your own virtual datacenter using their browser-based drag’n'drop schematic editor, including things like load balancers, and they automatically instantiate everything for you.

The Dynamic Appliance idea sounds pretty cool, as the idea of being able to seamlessly tap into more supply when demand runs high (and not pay for supply when demand runs low) seems to be the whole point of the utility computing thing.

I worry a little about what happens to my servers if this company goes away, or if they’re bought and put into maintenance mode, etc. but this is a pretty normal worry for any company. I think open-sourcing more of their stuff would assuage this a bit since I wouldn’t necessarily have to reconfigure everything to move to a competitor, but I can certainly understand their business reasons for not doing this.

However, like I said I haven’t looked at them in-depth yet, so take these worries with a grain of salt. Hosting providers of all flavors are susceptible to this kind of thing, as they often have their own home-grown or customized administration software, so it’s generally a pain to move between different providers.

I think that to be a true commodity, switching between providers has to be no-brainer as it is for services like telephone and electricity: you get the same dial-tone and same voltage, just at a different pricing model. I think this issue will be forced if the utility computing idea really catches fire this time.

web hosting with ubuntu server

Monday, January 14th, 2008

I’m thinking about doing series of how-to style articles on more technical subjects, here are some thoughts on a starting point: how to install and use Ubuntu Server, with AnyHosting as a case study.

AnyHosting currently uses Ubuntu Server (LTS) on a virtual host. Ubuntu is very easy to install and use as a desktop, but if you haven’t done administration purely from the command line then Server can be a bit daunting. There are excellent starter guides and forums on the Ubuntu website.


The following external services are provided (description followed by Ubuntu package name) :

  • web server – apache2
  • SMTP(+SSL) email server – postfix
  • IMAP/POP(+SSL) email server – courier
  • FTP – proftpd

Additionally, there are some internal services running, which are not visible from the internet (blocked by the “iptables” firewall):

  • Database – mysql
  • Monitoring/auto-recovery service – monit
  • Automatic installation of security updates – cron-apt
  • Log monitoring and reporting – logwatch logcheck
  • Append-only network backups – rsync


All unused ports are blocked. Any connection attempts are logged and reported. FTP and email services authenticate against the database, so clients do not need system or shell accounts.

For shared hosting, Apache is configured to proxy to chrooted installs which users have access to (as discussed previously). This is not as secure as having a real separate VM or better yet a separate machine for each web hosting client, but therein lies the dilemma of low-cost shared hosting versus moderate-to-high priced dedicated hosting.

The primary goal is to protect legitimate users from eachother; protecting the system from unauthorized intrusion (and detecting such intrustion) probably deserves it’s own series of articles, however the last few services listed in the “Services” section above should give some clues.

UPDATE link to secure shared hosting on ubuntu server part 1

Article on reliable, scalable web services posted

Thursday, September 6th, 2007

I’ve just put up a new article in AnyHosting’s article section – “Reliable, scalable, and growable web services“.

This touches on a few of my previous posts,  just trying to distill it down for anyone out there searching for some basic information.

Thin client or fat client – Why choose?

Friday, July 13th, 2007

There have been decades-long arguments over which is better, the thin client or fat client approach the network computing. Many maintain that the Web is just another turn of the wheel back to the bad old days of computer terminals which are useless without a connection to the Big Computer in the Back; what good is a web browser without Google to search, or Flickr to see your photos?

Some of the real wins you get with a browser is that you can make it share the workload, through rendering and client-side scripting; the upcoming Firefox 3 supporting offline mode for applications, and the Google Gears plugin that does this right now for IE and Firefox are also great steps in the right direction.

The big misunderstanding here about both client-side scripting and offline mode are that it’s not about being on the plane and wanting to browse your photo collection (that’s just a side-effect), it’s about:

  • seamlessness – if what the user is doing does not require the network, they shouldn’t be interrupted if their local network, the remote server, or anywhere in between is having problems
  • responsiveness – local storage is generally faster than network; it’s also necessary to be local for interactivity and any kind of scripted animation (think gaming).
  • privacy – some data you don’t want going across the network, period.

This is why I don’t really understand questions like “does offline mode still matter“  (or more harshly-worded criticisms of the idea) when you take the above into account. It’s about taking the best of the thin client world and the fat client world, and making applications that are always there for the person using them, and always responsive and fast, regardless of connection/network/server status.

The next big push has got to be blurring the line between web application and installed application, as well as the difference between online and offline. Using online office apps or webmail nowadays gives you an equivalent experience to running application, so what’s with the browser controls?

I’d like to be able to take my web applications out of the browser and manage them locally, and I’m not alone. Making apps more desktop-like seems to be the right way to get there.

business automation

Sunday, June 17th, 2007

One key to managing a small business that you do not spend full-time on is to spend the little time you have automating the processes that you know you will need, such as:

  • billing
  • account management (create/remove/update/delete)
  • monitoring
  • backups

It can be tough to know when you should do a particular task (say, moving records from one billing system to another) by hand, or take the time to automate it. My advice would be to do something by hand the first few times, and if you expect to have to do it again you’ll have the hands-on knowledge you’ll need to automate.

When I have a persistent task that I’ve had to do a few times, I first look for a system to suit my needs, before trying to write it on my own. Since I’ve done a bit of programming in the past, my priority is usually:

  1. find an external service that can do this for me
  2. find a suitable, well-supported, open-source application
  3. create a custom application, using as many open-source modules as possible

#1 is a fairly new addition there, for me :) I prefer not to used closed-source software with my own business, because it means I can directly modify (or hire someone to directly modify) the processes that run my business. I’m less concerned with software and processes that other businesses use, as long as I can easily integrate and replace their services as necessary.

Google Apps is one example of #1 for email/calendaring services. On #2, for instance managing server rollout, I’d look to things like kickstart and cfengine (or maybe puppet) to keep my servers in sync and up to date.

Account creation and possibly billing can be an example of #3, depending on how your business works, exactly how you configure customer’s accounts, etc. For billing, some find that a spreadsheet or other “offline” software is enough; personally I need a system that reminds me.

The most important things, to me, when writing custom software are:

  1. don’t do it if you don’t have to
  2. do the simplest thing that could possibly work

Both of these will save a lot of time, energy and money down the road, both your own and in contractor fees.

A simple, custom system is easier for someone to get started with, understand, fix and modify than a complicated, custom system; a system that other people actually know and use will ideally save you more time still, and make people who already understand the implementation easier to find.

is customer service relevant in web hosting?

Friday, June 1st, 2007

After finishing my post on customer focus, I found this blog entry by fellow planateer Isabel Wang. I wish I had found it before I wrote my previous post, as it’s a very insightful piece and I would’ve hit more on the customer focus versus “traditional customer service” aspect. Here’s an excerpt:

Which is why I don’t buy it when folks in web hosting say that they’re not worried about Amazon or Microsoft or Google – because they have “better customer service”. It’s also why I disagree with this Texan’s view that in order to build a better hosting company, you’ve got to make sure your employees understand the importance of customer service.

In my mind, there are two types of “customer service”. It’s great if you pick up the phone before the second ring, answer how-to questions with super human patience and respond to billing disputes with courtesy and generosity. But sooner rather than later, the need for this kind of generic support will be eliminated through innovation and automation.

On the other hand, there will be increasing demand for Rackspace’s kind of support, for helping the Slideshares and YouTubes of the world (aka the successors to today’s shared hosting resellers) plan and manage and scale their infrastructure. In fact, support might be the wrong word for their kind of requirement. Expert advice would be a better way to describe it. In depth knowledge from someone who understands your technology infrastructure – AND your business – at least as well as you do.

I totally agree, except that I think there is a further distinction – your customers (no matter who they are) should never need to contact your company for support, in the general. It’s not a user experience anyone wants, in almost all circumstances.

But no matter how redundant and automated and innovative your setup is, there are always bugs (or perceived bugs), network outages, and the like. Having appropriate communications channels with your customer is absolutely key, and one of those channels should always be a real human being (and they should have a “real” job as well, because they should almost never get calls :) ).

Hooking customers up with a knowledgeable person who has the authority to solve their problem is expensive, and should be treated as such. Their experience should be quick and easy if they are contacting you directly, because they have already run into something that is not working as they expect, and they are already frustrated.

In the end, it’s really all about the user experience. Everything should work seamlessly and easily as much as possible, and the last resort should  be getting a human to manually intervene for you. Customers really don’t want to be *forced* to interact with you (nothing personal!), so put effort into making it so they don’t have to.