danakim's questions

Currently I am running a production environment with 4 dedicated memcached servers, each of them having 48Gb of RAM (42 dedicated to memcache). Right now they are doing fine, but traffic and content are growing and will surely be growing next year too.

What are your thoughts on strategies for scaling memcached further? How have you done until now:

Do you add more RAM to the boxes until their full capacity - effectively doubling the cache pool on the same number of boxes? Or do you scale horizontally by adding more of the same boxes, with the same amount of RAM.

The current boxes can surely handle more RAM as their CPU load is quite low, the only bottleneck being memory, but I wonder if it wouldn't be a better strategy to distribute the cache, making things more redundant and minimizing the impact on the cache of losing one box (losing 48Gb of cache versus losing 96Gb). How would you (or have you) handle this decision.

I am having a really odd problem and I am lost a bit as I do not know what to do any more.

We are running in production 4 dedicated memcached boxes. All boxes have 48Gb of RAM, they are running memcached and nothing else and the daemon's memory limit is set to 42Gb.

The problem is that no matter the amount of traffic and gets/sets the boxes receive the cache will fill up on all 4 of them to about 38Gb but then the amount of free RAM available to the operating system will start slowly dropping over the course of several days until the boxes will start swapping, filling up the swap and thrashing! Now this is really strange as there is nothing else running on the boxes that could fill up the rest of the RAM and memcached is eating up 38Gb and not growing (at least that is what the graphs and the stats show).

I have tried setting the swappiness to 0 but it didn't help. I have tried lowering the cache limit even more but I get the same behavior.

I am running Centos 5.6, 2.6.18-238, memcached 1.4.4 and libevent-1.4.13-1.

Have any of you ran into a similar problem before? Could memcached possibly be leaking memory and not showing up in the graphs or the usual Linux tools?

Thanks! Dan

I was recently asked to modify the GECOS field in the passwd file for a certain user so that it will contain his name with his original accented characters. My first was reaction was "sure, why not?" but then I started getting paranoid that there will surely be a *nix utility that will fail to parse the passwd file because of UTF-8 characters.

So my question is, have you ever ran into any problems using UTF-8 in the passwd file? Or do you know of any?

My locale is set to UTF-8 on all boxes. ( $LANG=en_US.UTF-8 )

Thanks!