Home / ubuntu / Questions / 528658
Accepted
whitenoisedb
Asked: 2014-09-26 21:26:25 +0800 CST

Unique sorting: Redirect output to the same file

  • 772

Is there any short way to save the pipe output to the same file that it's processed. For example, this is what I am actually doing

$ cat filename | sort | uniq > result
$ rm -f filename
$ mv result filename

I was wondering if there was a way to do it in just one line (not appending those commands using &&)

This is not the way, but to get an idea

$ cat filename | sort | uniq > filename
command-line

5 Answers

  1. Best Answer

    You can use sponge from moreutils package:

    LC_ALL=C sort -u filename | sponge filename

    You also don't need pipe to uniq, since when sort has -u option to unique lines when sorting.

    Note that on GNU system with UTF-8 locales, sort -u or sort | uniq didn't give you unique lines, but the first from sequence of lines which sort the same in current locale.

    $ printf '%b\n' '\U2460' '\U2461' | LC_ALL=en_US.utf8 sort | LC_ALL=en_US.utf8 uniq
①

    gave you only ①. Changing locale to C force the sorting order based on the byte values:

    $ export LC_ALL=C
$ printf '%b\n' '\U2460' '\U2461' | LC_ALL=C sort | LC_ALL=C uniq
①
②
    • 18

  2. You don't need any extra command like cat and uniq and also without using rm command and mv command to removing and renaming the filename. just use simple command.

    sort -u filename -o filename



     -u, --unique
        with -c, check for strict ordering; without -c, output only  the
        first of an equal run

 -o, --output=FILE
        write result to FILE instead of standard output

    How it work?

    sort command sorts your filename and with -u option, removes duplicate lines from it. then with -o option writes the output to the same file with in place method.

    • 15

  3. Your suggested example (below) doesn't work because you'd actually be reading from and writing to the same file simultaneously.

    $ cat filename | sort | uniq > filename

    The idea with a pipe or redirect is that the command on the left and right hand side of each pipe or redirect run simultaneously, in parallel. The command on the right processes information as it is handed over to it from the command on the left, while the command on the left is still running.

    In order for your scenario to work, the command that reads from the file would need to finish before the command that writes to the file begins. In order for this to work you would need to redirect the output into a temporary location first, then once that's finished, send it from the temporary location back into the file.

    A better way to do this is basically like in your former example, where you redirect to a temporary file then rename that file back to the original (except that you don't need to delete the file first, because moving deletes any existing target).

    $ cat filename | sort | uniq > result
$ mv -f result filename

    You could also save it into a string variable, except that only works when data is small enough to all fit in memory at once.

    • 4

  4. You can use the tee command:

    sort -u filename | tee filename > /dev/null

    The tee command reads from standard input and writes to standard output and files.

    • 2

  5. You can use Vim in Ex mode:

    ex -sc 'sort u|x' filename

    1. sort u sort unique

    2. x write if changes have been made (they have) and quit

    • 0