Questions[telegraf](server)

To monitor a server we have imported a dashboard for Grafana. This uses Telegraf and InfluxDB as collector and database.

When certain graphs needed to be adjusted, I noticed that in the network speed queries, the bytes received (bytes_recv) are multiplied by 8, but the units on the graph itself are displayed in bits.

SELECT non_negative_derivative(mean(bytes_recv),1s)*8 as "in" FROM "net" WHERE host =~ /$server/ AND interface =~ /$netif/ AND $timeFilter GROUP BY time($interval), * fill(none)

According to my understanding is: 8Bit = 1Byte. According to this, if the data is available as bytes, it would have to be divided by 8 to get bits, if the graph should display bits.

Furthermore (if I am not mistaken): Megabit = Mb Mebibit = Mib Megabyte = MB Mebibyte = MiB

Refering to this, the unit on the graph seems to be megabit right? Do I understand something fundamentally wrong or was the "*" simply confused with the "/"?

Screenshot of the Grafana Dashboard

Please don't bother reading this question. syslog-ng is already set up to send RFC5424 messages with octet-counting framing by default. I was confused by the behaviour of another component. This question is invalid.

I have a syslog-ng OSE config (v3.31.2):

@version: 3.29
@include "scl.conf"
    
source s_network {
    udp(ip(0.0.0.0) port(514));
};
    
destination d_network_telegraf {
    syslog("telegraf" port(601) transport(tcp));
};
    
log {
    source(s_network);
    destination(d_network_telegraf);
};

The intention is to forward RFC3164 formatted syslog messages received on UDP port 514, and forward them as RFC5424 formatted messages to telegraf on TCP port 601.

With this configuration, syslog-ng appears to be emitting the forwarded messages as RFC5424 with non-transparent (octet-stuffed) framing (message starts with an ASCII < character). Unfortunately telegraf expects to receive messages with octet-counting framing (message starts with a digit). RFC6587 covers these.

Although it is possible to configure telegraf to expect non-transparent framing, this doesn't seem to be working correctly. Before I dig into that, I'd like to try the alternative, which is to configure syslog-ng to output RFC5424 messages with octet-counting framing. This is apparently the recommended framing type anyway.

However I am not able to find anything in the syslog-ng docs about this. In almost all cases, the documentation discusses the use of the syslog driver for input, not output, and there's almost no mention of octet-counting at all.

Is it possible to configure syslog-ng in this manner?

EDIT

This is the exact config I am using:

@version: 3.29
@include "scl.conf"

options {
    create-dirs(yes);
    flush-lines(1);
    time-reopen(60);
};

source s_local {
    system();
    internal();
};

source s_network {
    udp(ip(0.0.0.0) port(514));
};

destination d_local {
    file("/data/syslog-ng/var/log/messages");
    file("/data/syslog-ng/var/log/messages-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
};

destination d_network_files {
    file("/data/syslog-ng/var/log/messages-network");
    file("/data/syslog-ng/var/log/messages-network-kv.log" template("$ISODATE $HOST $(format-welf --scope all-nv-pairs)\n") frac-digits(3));
};

destination d_network_telegraf {
    syslog("telegraf" port(601) transport(tcp));
};

log {
    source(s_network);
    destination(d_network_telegraf);
    destination(d_network_files);
};

Here is a message sent by logger -i -d --server localhost this is a test to syslog-ng via UDP port 514:

0000   02 42 ac 15 00 09 02 42 69 3a f6 78 08 00 45 00   .B.....Bi:.x..E.
0010   00 a6 79 e7 40 00 40 11 68 2b ac 15 00 01 ac 15   ..y.@[email protected]+......
0020   00 09 b4 bd 02 02 00 92 58 d8 3c 31 33 3e 31 20   ........X.<13>1 
0030   32 30 32 31 2d 31 31 2d 31 38 54 31 32 3a 34 31   2021-11-18T12:41
0040   3a 35 39 2e 39 34 33 39 36 35 2b 31 33 3a 30 30   :59.943965+13:00
0050   20 6b 6f 72 69 6d 61 6b 6f 20 64 61 76 69 64 20    korimako david 
0060   36 38 38 31 34 30 20 2d 20 5b 74 69 6d 65 51 75   688140 - [timeQu
0070   61 6c 69 74 79 20 74 7a 4b 6e 6f 77 6e 3d 22 31   ality tzKnown="1
0080   22 20 69 73 53 79 6e 63 65 64 3d 22 31 22 20 73   " isSynced="1" s
0090   79 6e 63 41 63 63 75 72 61 63 79 3d 22 37 39 37   yncAccuracy="797
00a0   30 30 30 22 5d 20 74 68 69 73 20 69 73 20 61 20   000"] this is a 
00b0   74 65 73 74                                       test

And here is a captured message between syslog-ng and telegraf (TCP port 601):

0000   02 42 ac 15 00 07 02 42 ac 15 00 09 08 00 45 00   .B.....B......E.
0010   00 ec ba 61 40 00 40 06 27 70 ac 15 00 09 ac 15   ...a@.@.'p......
0020   00 07 a7 ab 02 59 be 1c 32 98 d1 f9 90 93 80 18   .....Y..2.......
0030   01 f6 59 19 00 00 01 01 08 0a 0c 3e 2a 2d 7a e2   ..Y........>*-z.
0040   58 de 3c 31 33 3e 31 20 32 30 32 31 2d 31 31 2d   X.<13>1 2021-11-
0050   31 37 54 32 33 3a 34 31 3a 35 39 2b 30 30 3a 30   17T23:41:59+00:0
0060   30 20 31 37 32 2e 32 31 2e 30 2e 31 20 31 20 2d   0 172.21.0.1 1 -
0070   20 2d 20 2d 20 32 30 32 31 2d 31 31 2d 31 38 54    - - 2021-11-18T
0080   31 32 3a 34 31 3a 35 39 2e 39 34 33 39 36 35 2b   12:41:59.943965+
0090   31 33 3a 30 30 20 6b 6f 72 69 6d 61 6b 6f 20 64   13:00 korimako d
00a0   61 76 69 64 20 36 38 38 31 34 30 20 2d 20 5b 74   avid 688140 - [t
00b0   69 6d 65 51 75 61 6c 69 74 79 20 74 7a 4b 6e 6f   imeQuality tzKno
00c0   77 6e 3d 22 31 22 20 69 73 53 79 6e 63 65 64 3d   wn="1" isSynced=
00d0   22 31 22 20 73 79 6e 63 41 63 63 75 72 61 63 79   "1" syncAccuracy
00e0   3d 22 37 39 37 30 30 30 22 5d 20 74 68 69 73 20   ="797000"] this 
00f0   69 73 20 61 20 74 65 73 74 0a                     is a test.

You can see that the message, which starts at byte 66 (0x42), begins with a < character (<13>1 2021...). According to RFC6587, this is interpreted not as octet-counting framing but as the alternative: non-transparent framing.

syslog-ng's full log for this event is:

[2021-11-17T23:41:59.944928] Incoming log entry; line='<13>1 2021-11-18T12:41:59.943965+13:00 korimako david 688140 - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="797000"] this is a test'
[2021-11-17T23:41:59.987236] Outgoing message; message='2021-11-17T23:41:59.944+00:00 172.21.0.1 HOST=172.21.0.1 HOST_FROM=172.21.0.1 LEGACY_MSGHDR="1 " MESSAGE="2021-11-18T12:41:59.943965+13:00 korimako david 688140 - [timeQuality tzKnown=\"1\" isSynced=\"1\" syncAccuracy=\"797000\"] this is a test" PROGRAM=1 SOURCE=s_network\x0a'
[2021-11-17T23:41:59.987403] Outgoing message; message='Nov 17 23:41:59 172.21.0.1 1 2021-11-18T12:41:59.943965+13:00 korimako david 688140 - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="797000"] this is a test\x0a'
[2021-11-17T23:42:31.994550] Syslog connection established; fd='12', server='AF_INET(172.21.0.7:601)', local='AF_INET(0.0.0.0:0)'
[2021-11-17T23:42:31.994946] Outgoing message; message='<13>1 2021-11-17T23:41:59+00:00 172.21.0.1 1 - - - 2021-11-18T12:41:59.943965+13:00 korimako david 688140 - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="797000"] this is a test\x0a'
[2021-11-17T23:42:36.996059] EOF occurred while idle; fd='12'
[2021-11-17T23:42:36.996187] Syslog connection closed; fd='12', server='AF_INET(172.21.0.7:601)', time_reopen='60'
[2021-11-17T23:43:36.996635] Syslog connection established; fd='12', server='AF_INET(172.21.0.7:601)', local='AF_INET(0.0.0.0:0)'

UPDATE 1: I've found the missing octet-counting value at the start of the message - it's in a preceding TCP frame. I think this means that the output from syslog-ng is using octet-counting framing after all. I'm now actively debugging the telegraf source because I think it's receiving the message but erroneously closing the connection afterwards. I'll update this as I discover more.

UPDATE 2: I've determined that the telegraf syslog plugin has either a bug or a mistake in its documentation, and is disconnecting any incoming TCP connections that are idle for more than 5 seconds. It is not a parsing issue merely a timeout.

Therefore there is no issue with syslog-ng and this entire question is null and void.

This is a follow-on question from this previous question, created because I found out more information and it's cleaner to pose this as a new question.

I'm using syslog-ng OSE v3.31.2 to receive RFC3164 syslog messages over UDP port 514 from a bunch of clients and write them to both a file, and forward them to telegraf via non-TLS RFC5424 TCP port 601 for insertion into an InfluxDB database.

My syslog-ng config is:

@version: 3.29
@include "scl.conf"

options {
    flush-lines(1);
};
    
source s_network {
    udp(ip(0.0.0.0) port(514));
};

destination d_file {
    file("/var/log/messages");
};
    
destination d_telegraf {
    syslog("telegraf" port(601) transport(tcp));
};
    
log {
    source(s_network);
    destination(d_telegraf);
    destination(d_file);
};

The relevant part of my telegraf config looks like this:

[global_tags]

[agent]
  interval = "100ms"
  round_interval = true
  metric_buffer_limit = 10000
  flush_buffer_when_full = true
  collection_jitter = "0s"
  flush_interval = "100ms"
  flush_jitter = "0s"
  debug = true
  quiet = false

[[outputs.influxdb]]
  urls = ["http://influxdb:8086"]
  database = "logs_db"

[[inputs.syslog]]
  server = "tcp://telegraf:601"

Essentially syslog-ng is set up to forward syslog entries over a TCP connection to telegraf.

The problem is that I'm seeing syslog-ng suffer frequent TCP disconnections from telegraf. These show up in the syslog-ng log as:

[2021-11-17T02:55:32.662972] EOF occurred while idle; fd='12'
[2021-11-17T02:55:32.663102] Syslog connection closed; fd='12', server='AF_INET(192.168.0.6:601)', time_reopen='60'
[2021-11-17T02:56:32.719139] Syslog connection established; fd='12', server='AF_INET(192.168.0.6:601)', local='AF_INET(0.0.0.0:0)'

This disconnection is usually triggered when I send a log to syslog-ng with:

logger -i -d --server localhost test

But if I just leave it all idle I'll also get:

[2021-11-17T02:57:05.392356] EOF on control channel, closing connection;

In these cases, 192.168.0.6 is the telegraf server.

Although I can set the option time-reopen(1) to speed up the reconnection, I'd prefer to find the root cause and prevent the disconnection in the first place.

Is it possible that there is an incompatiblity between syslog-ng and telegraf, that is causing this EOF and an unclean disconnection?

All of this is running within a docker-compose stack on a single host.

EDIT: I've started looking into RFC5424 and RFC6587. Using Wireshark to sniff packets out of syslog-ng, destined for telegraf, I've determined that these are using octet-stuffing (aka non-transparent framing), rather than octet-counting, which telegraf expects by default. The payload of each syslog message to telegraf begins with a "<" character rather than an integer.

I hypothesise that telegraf is accepting these messages but getting stuck parsing them, and therefore closing the connection. The first FIN to close the connection comes from telegraf.

Unfortunately when I set telegraf to accept non-transparent framing it rejects the entire entry and I haven't worked out why yet.

I also haven't yet figured out how to configure syslog-ng to output messages with octet-counting framing.

But at least the EOF message and disconnection has stopped happening. But I'm not sure that means much if telegraf is rejecting all messages outright.

Since my last reboot, I am seeing the following every 1-2 minutes:

Aug 02 13:53:00 monitor systemd[1]: influxdb.service: start operation timed out. Terminating.
Aug 02 13:53:00 monitor systemd[1]: influxdb.service: Failed with result 'timeout'.
Aug 02 13:53:00 monitor systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.
Aug 02 13:53:00 monitor systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 4.
Aug 02 13:53:00 monitor systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Aug 02 13:53:00 monitor systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Aug 02 13:53:00 monitor influxd-systemd-start.sh[3539]: Merging with configuration at: /etc/influxdb/influxdb.conf

on 29/07/2021 influx was updated from 1.8.6-1 to 1.8.7-1. The OS is Ubuntu 20.04 server. The first reboot after this is when the issues started.
Initially there was a permissions issue with /usr/lib/influxdb/scripts/influxd-systemd-start.sh, which prevented it starting. I changed the perms to 0755 and it started, but keeps restarting. It seems that it is accepting connections and data between the restarts, as telegraf is still populating the database, and Grafana is able to display the stats, so long as it doesn't coincide with the restart.

I am also seeing the message

influxd-systemd-start.sh[12171]: [tcp] 2021/08/02 14:21:40 tcp.Mux: Listener at 127.0.0.1:8088 failed failed to accept a connection, closing all listeners

It is listening on those ports

root@monitor$ ss -ilpn | grep influx
tcp     LISTEN   0        4096                                        127.0.0.1:8088                                              0.0.0.0:*                      users:(("influxd",pid=15115,fd=3))
tcp     LISTEN   0        4096                                                *:8086                                                    *:*                      users:(("influxd",pid=15115,fd=32))

As far as I am aware there have been no config changed. There is no firewall rules active.

Anybody have any idea why it started misbehaving?

I have the following telegraf configurations:

[[inputs.http_response]]
name_override="test"
address="https://localhost:12345/test"
response_timeout="5s"
method="GET"
follow_redirects=false
insecure_skip_verify = true
response_string_match="test is up"
 [inputs.http_response.tags]
    system="system"
    component="component"
[[processors.enum]]
 [[processors.enum.mapping]]
    field = "result_type"
    dest = "value"
   [processors.enum.mapping.value_mappings]
     success = 1

And i get the following response:

test,component=component,system=system result_type="success" 1611588662000000000

I was anticipating

test,component=component,system=system result_type="success",value=1i 1611588662000000000

What am i doing incorrectly here? I'm using version 1.9.2 and it shows it has the plugin.