At my current work I have this situation: every time we install some new software (9 out of 10 times the app is written in PHP), it is installed in its own server, consisting of a Debian OS, an Apache server and a PostgreSQL database. Because of this strategy, we currently have about 10 servers (note: when I say server, I say "regular PC without a screen", and not "we spent enough in this server to feed a small country"), most of them sharing the same software requirements.
This situation lead me to think: shouldn't we create a cluster, instead of adding individual servers for each app? I can attest that, while some servers need better hardware (RAM, mostly), some other ones have been idle for no less than three weeks, so with a cluster we could use those resources better. At the same time, I'm not sure the results are worth the time and effort needed to put together and mantain the aforementioned cluster (I always wanted to use the word "aforementioned").
After all this intro, the main question is: Would you suggest, in this situation, creating a cluster? Or would you keep the servers as they are now?
A cluster would add redundancy and improve scalability. However, it introduces complexity. You'll need some form of load balancing (and round-robin DNS is not suitable). You'll also need to deal with session sharing between servers. Your apps might write this to a database, which would work, if a little slow, but many are not designed to work in a clustered environment. You could use sticky session load-balancing, or alter the application yourself to deal with cluster-wide sessions.
You could take a look at virtualisation - replace your numerous boxes with one or two high servers and use VMWare, Virtual Server etc to allocate appropriate resources to each virtual machine.
Everyone else's opinions all have their own merits. I would argue that if all (or majority) of your apps are LAMP stack-ish, I'd say cluster the database portion (MySQL/PostGreSQL/etc.) either active-active/failover however you want.
In terms of application performance, you really need more metrics to prove a case for any kind of clustering of sorts. To add the complexity without regard to actual measurements let alone the possible manageability nightmare, make sure you have some proof.
From a logical standpoint, if you're just launching new sites/apps and databases are always a part of each app, I'd argue that clustering the databases together would be beneficial from a management standpoint and ultimately would allow you to virtualize the webservers (lowering future costs). Keeping both layers independent will help you in the following ways:
Data above all else the most important thing in IT. Lose info; lose customers which leads to losing money. A cluster could improve performance/availability/redundancy/manageability if done correctly (and tested vigorously).
Virtualizing your webservers allows for availability and manageability. If a virtual server goes down, the webserver could migrate to another viable virtual host and the database cluster could keep humming along. Plus, less to purchase for each new web app that needs to be installed. Wouldn't it be a lot cheaper to add more RAM to a Virtual Host rather than buying a whole new server?
So to answer your questions directly..
Yes, to some extent. Unless you're hosting one big webapp and performance is a major issue (like.. facebook?), clustering the webservers and maybe including some caching servers (memcached) for performance would help out definitely. But in your case, it sounds like you have lots of little or simpler sites/webapps with databases for each app. If load/performance isn't a problem here, virtualization will help with the uptime/availability.
From an uptime/redundant/management standpoint, I would definitely cluster the database servers in a solid group of servers. Keep the data safe, and lower your odds of downtime/disaste. That's much more important than a website as a site is simply just a bunch of files being served with apache, nginx (pick your webserver). Having a backup of a site isn't much in terms of storage (usually) and restoring a site should be fairly simple and straightfoward. Or you could get really creative and when all your sites are running in VMs, backup the VMs for any change on the webserver, back them up and if the running/production image becomes broken or difficult to fix, simply copy the good copy from archive. The database cluster should still be running and once the new VM is up and running, it should be pointing to the db cluster. Just an idea. I bet other people have better ideas/techniques, but whatever. It's just a thought.
If you're jumping to the conclusion that you need better hardware, you better figure that out first before building a cluster around bad parts. If you're having numerous, similar problems crop up on a consistent basis, a cluster won't really fix anything. Yes a cluster will adjust to the issues it was designed to but get to the root problem first. Why build on a bad foundation? But if your point is not having enough hardware, that's a different problem which can be out of your hands. I'd look into schmoozing whoever does the budget heavily.
Aside from your bubbling vocabulary, it's good to see that you're at least realizing the weight of the quest that lays before you should you choose to go the cluster route. I'm going to take a wild stab and say I think you're looking for the reason "to" go ahead with the cluster and while many people here may agree with you, only you know your technical strengths and weaknesses. 10 webservers isn't too bad but before too long when you get more and more webservers to host, between managing the servers, backing the up, dealing with uptime/availability and redundancy you're going to have to do something. If you don't do something now, that might be fine, but I would start investigating all your options for a date in the future. Who knows? Six months from now you'll add 10 more apps and your perspective changes drastically.
From my experience before doing any such adventure, do research, investigate your current assets, build a strategy/plan and fill in all the gaps in that plan. In fact here's a great document should you start researching clustering and such: http://www.scribd.com/doc/4069180/Caching-Performance-Lessons-from-Facebook. Granted that something always goes wrong in a plan, the more time you have to review your plan and revise you'll catch possible pitfalls.
If you aren't seeing a high load on any of the individual servers and you already have procedures in place for managing each of these servers, I think your best bet is to look at virtualization (I've been asking a lot of questions about it and experimenting with ESXi...nice so far!)
That way you can keep your procedures for maintenance, simplify your hardware cost, simplify some backup (although you'll need plenty of storage...on the server and for backups) chores, and make it easier to roll out new or experimental setups in the long run.
Sounds like you could probably roll the money you're putting into servers and electricity to use those servers into purchasing a couple servers for VMWare and support for their redundancy and management tools, like VMotion.
Tying all the sites into one means adding complexity to your configuration and probably some interdependencies in your web server. It can be done, but it may make things easier to break during upgrades or alterations; virtualizing it adds a layer that will slow things down a little and give you a single point of failure (if you don't invest in two+ servers and VMWare's redundancy solutions), but you gain a lot by still having separate machines for management and configuration.
Testing the configuration shouldn't be too difficult, since you can get ESXi for free as long as you have a server on the compatibility list on which to install it, then run the converter (available now for Linux and Windows, free) to create the virtual machines then power down the physical server and reconfigure the virtual one's network settings to take over the physical server's role. Install some monitoring software (like from Veeam, they have some tools that are free also for small business needs, registration is kind of a pain) and start seeing what kind of a load you put on the virtual server host. You may be surprised.
and if what you have working is basically the same thing on all servers, you can provide the same service off a single server with apache virtual hosts and multiple databases.
Another solution might be to set up a server with OpenVZ and set up a VPS per application