Home / ubuntu / Questions / 1355730
Accepted
Cas
Asked: 2021-08-04 05:35:08 +0800 CST

How to only remove duplicate lines if they're immediately after each other in the file

  • 772

Lets say I have the following file:

$ cat test.txt
a
-----
b
-----
-----
c
-----
-----
-----
d
-----
e
-----
-----

Now I want to remove all the -----, but only if they're repeating after each other. So the result should look like this:

a
-----
b
-----
c
-----
d
-----
e
-----

I tried grep -Pvz -- "-----\n-----", but that didn't work.

command-line

2 Answers

  1. Best Answer

    That's exactly what the uniq command is made for:

    NAME
       uniq - report or omit repeated lines

SYNOPSIS
       uniq [OPTION]... [INPUT [OUTPUT]]

DESCRIPTION
       Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT
       (or standard output).

       With no options, matching lines are merged to the first occurrence.

    So

    $ uniq test.txt 
a
-----
b
-----
c
-----
d
-----
e
-----

    Alternatively, you can use this sed one-liner 69. Delete duplicate, consecutive lines from a file (emulates "uniq") from Sed One-Liners Explained, Part III: Selective Deletion of Certain Lines and Special Applications

    sed '$!N; /^\(.*\)\n\1$/!P; D' test.txt

    which might be preferred if you want to edit test.txt in place (by adding the -i or --in-place option).

    • 10

  2. just do uniq filename.txt As the name states only unique lines are left and the repetitions are merged

    • 0