Home / server / Questions / 554859
In Process
gansbrest
Asked: 2013-11-14 14:09:07 +0800 CST

Why redis instance is dropping keys

  • 772

Here is my Redis graph from graphite for the past 72 hours displaying current_keys.

enter image description here

as you may notice one instance ( green ) is dropping keys regularly, while other one is ok.. Both of them were configured with Chef and running same Redis version ( redis_version:2.6.5 ) and same config.

Any ideas what could cause keys to drop? Both instances are behind nutcracker/twemproxy ( redis proxy from Twitter https://github.com/twitter/twemproxy ) so should get equal amount of keys..

redis

1 Answers

  1. I’d start with checking for what happens around 1 am, each day; possible causes…

    • Chef-client is running with an --interval and causes a reset (check /var/log/chef/*)
    • Bad permissions on the jetpack host and logrotate (or other automated tasks) fail (check /var/log/syslog)
    • Leftover crontabs on host jetpack (because Chef is not transactional and does not delete artifacts of removed provider calls; check /etc/cron.d/)

    The only way I’ve been able to fix the --interval behavior of chef-client is to restart a host. I’ve then changed all chef commands to include the --once argument.

    • 1