I have a small cluster of servers balancing a Java web app. Currently I have 3 memcached servers caching data and all web apps shares all 3 memcached instances.
I often get strange slowdowns and timeouts to some of the memcacheds and I wondering if there is a good way of analyzing the performance.
I am wondering whether my iptables rules (or some other system limitation) are blocking/slowing connections. I am considering reconfiguring the web apps so that they only query the memcached process on their own localhost.
These things depend on various factors. Obviously, querying a remote memcache is slower than querying a local memcache. So if you have enough memory to keep a reasonable amount of data in the cache, better keep a copy on each. Make sure the memcaches don't use the swap memory.
And think about when it is cheaper to wait for a (potentially slow) remote memcache or just recompute the data.
You can find some cacti templates for memcached here http://dealnews.com/developers/cacti/memcached.html. One thing you will want to look at is the number of connections allowed by memcached, by default set to 1024. You can see the relevant section of the documentation here http://code.google.com/p/memcached/wiki/NewConfiguringServer#Connection_Limit. If your client isn't properly closing connections this can cause symptoms like slowdowns.
Moving to local only memcached you will probably run into issues with cacheing, because everyone knows there are only two hard problems in CS, cache invalidation and naming things.