I've setup Graphite and statsd and both are running well. I'm using the example-client.py from graphite/examples to measure load values and it's OK.
I started doing tests with statsd and at first it seemed ok because it generated some graphs but now it doesn't look quite well.
First, this is my storage-schema.conf
:
priority = 100
pattern = .*
retentions = 1m:395d
I'm using this command to send data to statsd:
echo 'ssh.invalid_users:1|c'| nc -w 1 -u localhost 8126
it executes, I click Update Graph in the Graphite web interface, it generates a line, hit again Update and the line disappears. Like this1 and this2
If I execute the previous command 5 times, the graph line will reach 2 and it will actually save it.
Again running the same command two times, graph line reaches 2 and disappears.
I can't find what I have misconfigured.
The intended use is this:
tail -n 0 -f /var/log/auth.log|grep --line-buffered "Invalid user" | while read line; do echo "ssh.invalid_users:1|c" | nc -w 1 -u localhost 8126; done
EDIT:
On a fresh system I reinstalled using the latest versions of graphite, carbon, nodejs, statsd and it's acting the same.
While tail-ing /opt/graphite/storage/log/carbon-cache/carbon-cache-a/query.log I get:
cache query for "stats_counts.ssh.invalid_users" returned 0 values
cache query for "stats.ssh.invalid_users" returned 0 values
whenever I press update in webapp. I noticed that it will randomly say returned 1 values
when drawing the lines, but will revert to returned 0 values
and the lines disappear.
The problem is the storage-schema retention:
retentions = 1m:395d
- which is taken from graphite wiki http://graphite.wikidot.com/installationI had to use
retentions = 10:2160,60:10080,600:262974
or something similar. This takes in consideration values saved every 10 seconds.Also, although I restarted graphite after changing storage-schema.conf, I had to use a different metric name because the previous would retain the same behavior/retention (and I can reproduce this).
So instead of
echo 'ssh.invalid_users:1|c'
, I had to useecho 'ssh.invalid_userstest2:1|c'
.