Home / server / Questions / 930052
In Process
xendi
Asked: 2018-09-09 08:00:22 +0800 CST

sed removing line breaks tail

  • 772

In my Python script I needed a fast/efficient way to set a max filesize on a file I'm constantly writing to. Rather than bring the whole thing into py's RAM, I ran this shell command:

sed -i '1d' file.csv

I monitor the filesize periodically and run the command as needed. Problem is that now if I tail -f file.csv, tail stops tailing the file as soon as sed removes a line from it. Any solution?

linux

2 Answers

  1. The basic problem is sed creates a new file. This is shown by the extract below:

    $ strace -fe trace=file,read,write,close sed -i '1d'  /tmp/x.csv 
....
openat(AT_FDCWD, "/tmp/x.csv", O_RDONLY) = 3
...
openat(AT_FDCWD, "/tmp/sed1nFxY1", O_RDWR|O_CREAT|O_EXCL, 0600) = 4
write(5, NULL, 0)                       = 0
read(3, "1,1,1,1\n2,2,2,2\n3,3,3,3\n", 4096) = 24
write(4, "2,2,2,2\n", 8)                = 8
write(4, "3,3,3,3\n", 8)                = 8
read(3, "", 4096)                       = 0
close(3)                                = 0
close(4)                                = 0
rename("/tmp/sed1nFxY1", "/tmp/x.csv")  = 0
close(1)                                = 0
close(2)                                = 0

    There isn't, as far as I know, an easy way to truncate the start of a file.

    I'd suggest getting your python application to listen on a unix/tcp socket, and once accepted put that stream into the logging framework and then use nc ... | tail to watch output.

    • 1

  2. Michael Hampton suggests the obvious tail -F, but I personally mostly use less nowadays:

    less --follow-name +F file.csv

    There being no shorthand alternative to --follow-name, I have to use an alias:

    alias less='less --follow-name -XFR'
    • 1