Questions[statsd](server)

I'm trying to capture UDP traffic which is being sent as StatsD metrics (on port 8125).

The most simple approach I tried is to run nc -ul 8125 which prints the metrics to the terminal...but they are not newline separated. Additionally, piping the output to grep doesn't display anything.

The next approach I tried was socat - udp-listen:8125 which gives similar results to netcat.

However, when I pass in the -v flat and pipe stderr to stdout, (socat -v - udp-listen:8125 2> &1 | grep ...) then I get vaguely decent results but there's lots of additional unwanted output.

I also tried tshark, tshark -l -f "port 8125" -i ln0 -E separator=, -Tfields -e data but this prints the output as hexadecimal...not ascii. I can pass in the -x flag, which prints both the hex and the ascii side by side. Not really what I want, and not very grep friendly. I've also tried a bunch of other tshark options, but nothing was able to simply print the UDP metrics.

I have seen some examples where somebody doing something similar is pipling output like | while read output; do echo $output; done; However, I've not had any success with that approach and I'm not really a unix guy so I'm struggling to come up with a solution.

After all, it doesn't seem like such a uncommon task, and nc -ul 8125 is almost there....

To give more context...the code doing the sending looks like this:

var udpClient = new UdpClient();
udpClient.Connect("127.1", 9999);
while (true)
{
    var bytes = Encoding.UTF8.GetBytes("hi");
    udpClient.Send(bytes, bytes.Length);
    Thread.Sleep(1000);
}

the desired output would look like

hi
hi
hi
...

whereas nc -ul 9999 outputs hihihihihi....

My first post so be gentle please! So I have an installation of graphite up and running on Ubuntu 16.04 server. (version 0.10)

I am sending in metrics and everything works fine, but I am sending in multiple timestamps at once. As in I am sending in data once per minute with per 10s timestamps. Every minute new data comes in. It can happen, that a key with a corresponding timestamp (same as before) is resent a minute later. So the new value is being saved and last one lost, because of the way graphite works. So I put a carbon-aggregator in front of carbon-cache thinking it will aggregate the values per timestamp. It does not. It aggregates all values received in a time interval ignoring the timestamps.

I found that statsd has the same functionality. What I want is for the aggregator to aggregate values per timestamp. As in keep data for lets say 1 minute and see if a datapoint for the same key and timestamp is received. If so sum them. If no such data came, forward the original data to carbon-cache.

Is there a way to do this with graphite or statsd, or do I have to write my own little buffer in front of carbon-cache (instead of carbon-aggregator), that does this? It could be a linked list of self made structures in C resulting in about 150 lines of code, but would take precious time.

Someone has definitely had the same problem, but has anyone found a solution?

Bump: has noone really encountered this issue?

Thanks in advance!

My first inclination here would be something built into SAR, or the sysstat package in general. If that is indeed the case however, I can't seem to find this solution.

What I would like to see, and absolutely preferably through sysstat if possible, is a historical log of process usage (memory, CPU, etc.) in much the same format as SAR logs (if not already directly available through SAR files somehow). I know monitoring software is available, but I'm more-so looking for a mostly non-intrusive package, and actually something that could be parsed relatively intuitively via statsd/collectd.

I was trying to mimic this logstash.net/docs/1.1.0/tutorials/metrics-from-logs

I have following setup

nginx(app server) ==sends the increment==>Etsy statsD=====>Graphite

This setup is working fine since graph is being plotted just fine but not correctly. According to above configuration, I understand the graph step value should be integer but I am getting floating points numbers in Y axis(hits) as per the attached graph and I am completely baffled as to how do i interpret this graph.

Logstash agent configuration

input {
  file {
    type => nginx_web
    path => ["/var/log/nginx/access.log"]
  }

filter {
 grok {
   type => nginx_web
   pattern => "%{IP:ClientIP} (?:%{HOST:ClientHost}|-) (?:%{USER:ClientUser}|-) \[%{GREEDYDATA:TimeStamp}\] \"(?:%{WORD:Verb} %{URIPATHPARAM:Request} HTTP/%{NUMBER:HTTPversion}|%{DATA:UnparsedRQ})\" %{NUMBER:Response} (?:%{NUMBER:Bytes}|0) (?:%{QUOTEDSTRING:HTTPReferrer|\"-\"}) %{QUOTEDSTRING:HTTPUserAgent}"
 }

output {

 statsd {
        type => "nginx_web"
        host => "X.X.X.X"
        increment => [
                        "nginx.response.%{Response}"
                     ]
    }
}

I am using following URL API to generate this graph

render/?width=600&height=320&hideLegend=1&from=-60minutes&until=-0minutes&target=stats.logstash.*.nginx_web.response.*

Can somebody point me in right direction as to where to go to remedy my issue ?