I'm trying to figure out something that's counter-intuitive for me in Graphite.
I am collecting (for now) test data (with statsd). In Graphite's dashboard, I'm seeing lower peaks for the same set of data for a longer time series.
My retention is set to:
retentions = 10s:15m,60s:1d,5m:7d,1h:4w,1d:5y
Why wouldn't the second chart showing 30 minutes not have peaks at least as high as the highest peaks in the 15 minute chart? How can I get this chart to accurately reflect these stats over time?
The peaks are lower because graphite uses averages to downsample metrics when thresholds are crossed. You cross the 15m threshold so the resolution went from 10s to 60s. E.g., suppose you have this data:
Viewing time T+50s on a 15m graph will give you 100. Viewing time T+50s on a 30m graph will give you 33.3 (200/6).
Since 0.9.9 you can change the aggregation method. For your case in
storage-aggregation.conf
you'd useaggregationMethod=max
. See http://graphite.readthedocs.org/en/latest/config-carbon.html#storage-aggregation-conf