It's a dual-core machine and the load is around 24 but CPU usage is bouncing around from 60-80% and never reaching 100% so how can there be that many processes waiting? I/O is pretty much nill since everything is being read from memory databases and the network is only using around 7mbit from a gigabit connection. Would adding additional cores reduce the load or is the CPU not the issue since it's not even topping out?
No.
Try running nothing but Seti @ Home (or whatever its current incarnation is), and you'll see your server load pegged at 1.0, although your CPU would be pegged at 100% usage.
From the UPTIME(1) man page:
For example, we have a server that does nothing but run Spamassassin with two Xeon CPUs (for a total of 4 virtual CPUs) It sometimes reaches 100% CPU utilization, but more often it uses a lot of disk IO by reading and writing to the Bayes database. As such, the system load is often very high at around 20-40 because there are many processes that are waiting a little while for the disk.
Load average is the number of processes in runnable state: press "i" in top to limit it to these. This includes processes which are (R) running and (D) waiting for disk. Is it really not doing any IO? Has it not gone into swap?
Are a non-trivial number of these processes heavy memory users? Because you may be seeing the effects of heavy cache misses.
Memory is slow, so the CPU has a (or more often several levels of) cache. Now if process A is on the CPU and has loaded most of the cache with it's own memory, then process B comes along and wants to fill the cache with its memory there are a series of waits while that happen causing reduced CPU. Then, sooner or later Process A comes along and you get to go through it all again...