Home / server / Questions / 1059355
Accepted
M.Ch.
Asked: 2021-04-05 11:08:03 +0800 CST

Piping nc to sed and evaluate datetime for each row

  • 772

Could someone provide any idea, how could i pipe data from nc to sed (or awk, or anything) and evaluate and write to file a timestamp after each row? What i have:

# cat /etc/systemd/system/ncserver.service
[Unit]
Description=netcat listener
After=network.target

[Service]
Restart=always
RestartSec=1
ExecStart=/usr/bin/bash -c '/usr/bin/nc -n -v -l -k -p 1313 >> /opt/data/$(hostname)-$(date -u +%%y%%m%%d-%%H).txt'

[Install]
WantedBy=multi-user.target

simple listener - receives text via raw tcp, sends into the file (restarts by cron once a day to generate new file)

I want to append current timestamp to each row of text coming (at the beginning or the end of the row, doesnt matter). Tried sed, awk, but the problem is that i cannot evaluate date for each row - it evaluates only once, when service started. Looks like this

109582 ?        Ss     0:00 /usr/bin/bash -c /usr/bin/nc -n -v -l -k -p 1313 | /usr/bin/sed 's/^/'$(date +%s)
 109583 ?        S      0:00 /usr/bin/nc -n -v -l -k -p 1313
 109584 ?        S      0:00 /usr/bin/sed s/^/1617555467\;/g

Pipe may be not the answer at all, but i'm kinda limited by bash. Any ideas?..

bash text pipe sed nc

2 Answers

  1. I want to append current timestamp to each row of text coming (at the beginning or the end of the row, doesnt matter).

    This sounds very like what syslog does. You might be able to replace nc with something like pysyslog and have it capture the text on the socket and append the date.

    • 0
  2. Best Answer

    sed does not have the ability to run a command again and again each time you run a script (well, GNU sed has the /e flag which does this, but it's not generally portable) so you should probably switch to a tool which does this.

    ExecStart=/usr/bin/bash -c '/usr/bin/nc -n -v -l -k -p 1313 | awk "{ system(\"date +%s | tr \\047\\\\n\\047 \\047\\\\t\\047\") }1"
 >> /opt/data/$(hostname)-$(date -u +%%y%%m%%d-%%H).txt'

    The quoting there is unsavory enough that you might want to save the command line in a separate script, and just put the path to that as the argument to ExecStart. For example, if you saved this in /usr/local/bin/dump1313

    #!/bin/sh
/usr/bin/nc -n -v -l -k -p 1313 |
awk '{ system("date +%s | tr \047\\n\047 \047\\t\047") }1' >> /opt/data/$(hostname)-$(date -u +%%y%%m%%d-%%H).txt

    and chmod a+x this file, you can then simply put

    ExecStart=/usr/local/bin/dump1313

    (There are no Bash-specific commands here, so I put #!/bin/sh instead of #!/usr/bin/bash or what have you.)

    Running tr to get rid of the line terminator from date is still rather a wart. If you want to make this self-contained and resource-efficient, perhaps create a simple C wrapper which adds a time stamp to each line it reads and prints ... or Python if C is too challenging:

    import sys
from datetime import datetime

for line in sys.stdin:
    print('{0}\t{1}'.format(int(datetime.utcnow().timestamp()), line), end='')
    • 0