Forgive me if I use any of these terms incorrectly.
I am wondering if there is any kind of software that would allow my two "join" two computers together such that a cluster unaware application could utilize their combined computing resources? By "cluster unaware" I mean an application that isn't designed to share work across multiple services. My understanding is that clustering is enabled by the specific application by it's architecture, such that messaging with multiple instances of the application coordinate the sharing of work. Instead I am looking for something that enables clustering at the OS or virtualization level, so that any application could essentially be clustered.
Failing that, I am also wondering about the following scenario: We have 3 different applications we will call A, B, and C. We have 2 single core computers. At any given time lets say that any combination of those applications will be CPU intensive. In cases where only 2 of those apps are very active, have one of them moved over to a different server. In a nutshell, some sort of dynamic automatic shuffling of the application's load.
I have heard of virtual machines that can be migrated across physical machines while live, but I am wondering if this can be done automatically in response to an application's or VM's CPU activity?
You can virtualize the whole OS image and run your apps in these different virtualized OSes and move those OSes around to suite your loads. This is the ESX / Virtualization model. It can probably be traced back to IBM mainframe OSes where they virtualized the machine instead of making an operating system that could run several programs at once.
Hardware to hardware migrations are not super fast and you're going to burn lots of overhead on things like OS memory on each virtual instance.
Another option is to use something that presents a single OS image to the client programs but actually runs on lots of computers at once. An example of this model is mosix. Mosix extends linux such that a running program may under some situations be moved from one piece of hardware to another. The program itself doesn't realize this (just as it typically doesn't care if it moves from one CPU to another in a normal computer with 2 CPUs). There will be some overhead with this but it may be worth looking into.
If you install every app on a different VM, then yes, it can be live migrated using any given enterprise level VM technology
clustering for non clusterable software is possible if you're looking for failover clusters. Either use a VM and if the host dies, migrate the VM away to another host, or use software like XOsoft WANsync, NeverFail, DoubleTake etc.
resource sharing is a bit trickier, and the software generally has to be cluster aware, more so - it should be able to work in load balancing clusters, not just failover.
Since MOSIX seems to be associated with MPI, it seems appropriate to mention there was a competing technology called PVM (Parallel Virtual Machine). It may be useful to explore if there is any MOSIX-like clustering OS designed based on PVM as well. I had used PVM to run C++ Tasks in parallel using multiple Linux servers as a single clustered HW. For example, running SQL Queries on humungous RDBMS rows could be partitioned across.
Besides providing a message passing interface, PVM makes network communication transparent, and supports flexible parallel computing in heterogeneous cluster systems.
In case you are using Myrinet switches HW, an optimized version of PVM can be used; see http://www.huaxiaspace.net/research/wed930.pdf.
I have seen some slides implying Mosix can be used with either MPI or PVM technology, but not sure.