Home / server / Questions / 30905
Accepted
Alex
Asked: 2009-06-25 03:13:36 +0800 CST

Changing files on the fly in Linux (writing to input file on a pipe)

  • 772

How to change a file on the fly along a pipe?

I'm probably looking for a way to buffer a file at the beginning of a pipe, which in contrast to this:

    cat foo.txt | grep bar > foo.txt

... would preserve the input data from destruction by the pipe itself. Is there such a buffer in stock?

linux pipe

6 Answers

  1. Try using sponge from moreutils like this:

    sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd

    It collects the whole input before writing to it's output.

    • 13
  2. Best Answer

    I would guess sed still might create the temp file, but the following might do what you want? (Using strace on this might show you if sed creates a temp file or not). 

    sed -i '/bar/!d' foo.txt

    The exclamation inverts the match, d is for delete, so this removes all lines that don't have bar in them.

    • 6

  3. Use >> to preserve the contents.

    cat foo.txt | grep bar >> foo.txt

    Now that will append to the file.

    AFAIK, there is no direct way to prepend data to a file in shell. If you want to prepend you may need to use a temporary file in between.

    • 1

  4. Depending on how complex your command line is, you may get mileage out of

    cat foo.txt | grep bar | tee -a foo.txt
    • 1

  5. The best way to do this is to remember how files work in unix -- the file exists as long as there is a link to the file (directory or process opening the file).

    So, open the file, delete that directory entry, then run your process that writes to a new directory entry with the same name but linked to a different inode.

    Lastly, the nice thing about this is that it works for any sort of pipeline operation, not just things that can do the temporary file buffer hack.

    { rm file ; sed -e s/this/that/g > file ; } < file

    A word of warning -- this may cause disaster if you must have your rewrite be an atomic operation. unix doesn't offer clean ways of locking files, especially not at shell level. This was an issue in the dark ages if you were doing something like editing a password file on a busy system that was running something like NIS. Any time more than one process is reading / writing to a file, be very careful if your system is busy or important.

    The only operations that are 100% certain to be atomic are directory entry manipulations -- rm / ln / mv (on the same filesystem).

    So now things get longer and uglier..

    The math part of this requires a posix shell or similarly extended bourne shell

    c=0
    while
      f=$SECONDS
      test -f file.$c.tmp
    do
      c=$(($c+1))
    done

  grep stuff < file > file.$c.tmp && mv file.$c.tmp file
    • 0

  6. Ooops, UUOC, useless use of cat :) you don't need cat there :

    grep something somefile > someotherfile

    As sekenre said, there is sponge. Or maybe you may want to try perl inline edition ( -i switch ) :

    • print only matching output:

      perl -ni -e 'm/foo/g and print ' bar.txt

    • replace "foo" with "bar" in the file :

      perl -pi -e 's/foo/bar/g' bar.txt

    • 0