Home / ubuntu / Questions / 562344
Accepted
Roboman1723
Asked: 2014-12-17 20:13:44 +0800 CST

What does grep line buffering do?

  • 772

Here's my command that I'm using in a script to grep real-time data. It doesn't seem to pull real-time data correctly as it just misses some lines.

tail -f <file> | fgrep "string" | sed 's/stuff//g' >> output.txt

What would the following command do? What is "line buffering"?

tail -f <file> | fgrep --line-buffered "string" | sed 's/stuff//g' >> output.txt
grep

1 Answers

  1. Best Answer

    When using non-interactively, most standard commands, include grep, buffer the output, meaning it does not write data immediately to stdout. It collects large amount of data (depend on OS, in Linux, often 4096 bytes) before writing.

    In your command, grep's output is piped to stdin of sed command, so grep buffer its output.

    So, --line-buffered option causing grep using line buffer, meaning writing output each time it saw a newline, instead of waiting to reach 4096 bytes by default. But in this case, you don't need grep at all, just use tail + sed:

    tail -f <file> | sed '/string/s/stuff//g' >> output.txt

    With command that does not have option to modify buffer, you can use GNU coreutils stdbuf

    tail -f <file> | stdbuf -oL fgrep "string" | sed 's/stuff//g' >> output.txt

    to turn on line buffering or using -o0 to disable buffer.

    Note

    • 61