Home / server / Questions / 1054138
In Process
olivierg
Asked: 2021-02-19 06:52:00 +0800 CST

huge difference in RX bytes vs TX bytes (ifconfig) on Syslog-NG server

  • 772

we have a syslog-ng server acting as an intermediary logger.

the server receives data from all devices on the network (from many different sources, on both UDP & TCP ports), filters them a bit, and forwards the data to a SIEM (splunk).

when looking at ifconfig stats, i can see this :

[root@xxxxxxxx ~]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:50:56:00:00:00
          inet addr:1.1.1.1    Bcast:1.1.1.1    Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:60451021996 errors:0 dropped:0 overruns:0 frame:0
          TX packets:158501574 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:5000
          RX bytes:12508573129969 (11.3 TiB)  TX bytes:220148785267 (205.0 GiB)

this part is quite shocking to me :

          RX bytes:12508573129969 (11.3 TiB)  TX bytes:220148785267 (205.0 GiB)

obviously, the server received around 11 Terrabytes of data since the last boot (15 days uptime) but "only" transferred a little part of it ? (250 Gbs).

is there a way for me to check how is this possible ? i know there's a bunch of UDP dropped messages, but is it taken in consideration ? (dropped is displayed 0).

i expect a little less TX (due to the fact that we filter some messages), but not that much (10% max).

For information, Netstat on UDP shows :

Udp:
 10903564401 packets received
 8401685 packets to unknown port received.
 49356622070 packet receive errors
 43665773 packets sent
 RcvbufErrors: 34287641

any tips on how to investigate this ? thanks !

linux networking udp tcp ifconfig

2 Answers

  1. To be clear, RX and TX are Receive and Transmit (not transfer).
    Ie, traffic sent to this host (RX on its network interface) and sent from this host (TX on its network interface), respectively.

          RX packets:60451021996 errors:0 dropped:0 overruns:0 frame:0
      TX packets:158501574 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:5000
      RX bytes:12508573129969 (11.3 TiB)  TX bytes:220148785267 (205.0 GiB)

    In your example above, that network interface has received much more data than it has transmitted.

    If the main traffic happening here is other hosts sending logs to it, it sounds expected that it receives more than it transmits.

    • 2

  2. Layer 3 counters do not prove what is happening at layer 7, the application. Possibly you have far more non logging traffic, like IP storage or software update. Or are filtering logs more aggressively than you thought.

    Do packet capture on the log server for a few hours to get a sample. Find the ratio of protocols, such as with Wireshark's Statistics > Protocol Hierarchy. What percent is syslog?

    Consider temporarily saving unfiltered log entries to do statistics on those. Would require a considerable sized fast storage. Log search engines like Graylog can ingest as many messages as you can send, at the cost of maintaining an Elasticsearch cluster.

    Compare the raw data going through syslog to the SIEM. Get the real ratio of syslog data ingested to stored in the SIEM. Count up the number of unique host names or IP addresses and compare to your inventory.

    If any hosts also store log messages locally, pick a few and query those files. Look for messages that should get to the SIEM, and confirm they do. (Local logs are not ideal when central logging exists, but might as well examine them.)

    • 0