Home / ubuntu / Questions / 1269817
Accepted
user2413
Asked: 2020-08-25 12:02:32 +0800 CST

Replace two consecutive CR with one

  • 772

cat -e file.txt gives:

{"yellow":"mango"}^M$
^M$
{"yellow":"banana"}^M$
^M$
{"yellow":"blabla"}^M$
^M$

and I would like to just have:

{""yellow":"mango"}^M$
{"yellow":"banana"}^M$
{"yellow":"blabla"}^M$

in place for all files with txt extention in folder. So I tried:

find . -type f -name "*.txt" -print0 | xargs -0 sed -i "s/^M$^M$/^M$/g"

to no avail. Does anyone have a better idea?

head -n 3 file.txt | od -bc

yields:

0000000 173 042 171 145 154 154 157 167 042 072 042 155 141 156 147 157
          {   "   y   e   l   l   o   w   "   :   "   m   a   n   g   o
0000020 042 175 015 012 015 012 173 042 142 141 142 141 142 042 072 042
          "   }  \r  \n  \r  \n   {   "   b   a   b   a   b   "   :   "
0000040 155 141 156 147 157 042 175 015 012
          m   a   n   g   o   "   }  \r  \n
0000051

this:

awk 1 RS='\r\n' ORS= < file.txt

removes the new lines completely (so it's not good: I want to keep one of the successive two on each line, but it does something).

command-line text-processing

5 Answers

  1. Best Answer

    You can use sed -z 's/\r\n\r\n/\r\n/g'.

    Normally sed only works on one line at a time. By using the -z option, sed will work on lines, which are seperated by 0 bytes, which normally don't exist in a text file, so the whole file will be treated as one line and newlines can be replaced.

    (found on stackoverflow and added explanation)

    • 13

  2. You can also delete lines that contain only the carriage return.

    • With GNU Sed:

      sed '/^\r$/d' file

    • For a minimal but POSIX compliant machine (here we need to generate the carriage return with Printf):

      sed "/^$(printf "\r")$/d" file

    ^ matches the start-of-line and the last $, the end-of-line (\n).

    For example:

    $ cat -e file
AB^M$
^M$
CB^M$
^M$
$ sed '/^\r$/d' file|cat -e
AB^M$
CB^M$
    • 12

  3. If it's okay to remove all blank lines, you can do:

    perl -wlne '/\S/ and print' old_file > new_file

    And if you prefer to overwrite your file(s), you can use the -i (in-place) switch:

    perl -wlni.bak -e '/\S/ and print' file1 file2 file3 ...

    The above line will copy the original files as *.bak files. If you don't care about having backups, then you can just leave out the .bak part, like this:

    perl -wlni -e '/\S/ and print' file1 file2 file3 ...

    (You can even use wildcards, so instead of file1 file2 file3 ... you can write file* .)

    The advantage of this approach is that it makes changes to your files all at once (instead of having to run it once for each file).

    But remember: This will only keep lines that contain at least one non-whitespace character. So if a line consists only of five spaces, a tab, a carriage return, and a line-feed character, it won't be kept.

    • 2

  4. I think you could use awk's Record Separator and Output Record Separator to achieve the goal, which should be more efficient on very large files than sed -z ....

    awk '
  BEGIN {
    RS = "\r\n\r\n"
    ORS = "\r\n"
  }

  {
    print $0
  }

' inputFile > outputFile
    • 2

  5. Using Raku (the language formerly known as Perl6)

    ~$ raku -ne '.put if /\S/ ;' test_blank.txt
{"yellow":"mango"}
{"yellow":"banana"}
{"yellow":"blabla"}

    The example above only prints lines that contain non-whitespace characters (\S matches a single character that is not whitespace). A very readable version below:

    ~$ raku -ne '.put if .chars;' test_blank.txt
{"yellow":"mango"}
{"yellow":"banana"}
{"yellow":"blabla"}

    HTH.

    https://raku.org
    https://rakudo.org/downloads

    • 1