My CentOS server's munin graph shows high swap, but the amount of apps memory seems low. Does this still mean I need to upgrade my memory? What is causing all that swap? Is it because of the committed memory? What does it mean when swap_cache is high?
What is the best determining factor when I need to upgrade the memory?
You need to upgrade memory if you are suffering from low performance and can determine that the cause is the system's working set exceeding the amount of physical RAM the system has. None of the parameters you are measuring are particularly useful for that purpose. What you want to look at is paging I/O, load average, and so on.
Check the value called swappines (cat /proc/sys/vm/swappiness). By default, it is set to 60 which is quite generous. This means swapping will happen even if there are enough physical memory available.
You can try different values before setting one. Try 25 or even 10.