Home / server / Questions / 440805
Accepted
Mike B
Asked: 2012-10-22 18:21:59 +0800 CST

How can I easily convert HTML special entities from a standard input stream in Linux?

  • 772

CentOS

Is there an easy way to convert HTML special entities from a data stream? I'm passing data to a bash script and sometimes that data includes special entities. For example:

"test" & test $test ! test @ # $ % ^ & *

I'm not sure why some characters show up fine and other don't but unfortunately, I don't have control over the data coming in.

I'm thinking I might be able to use SED here but that seems like it would be cumbersome and possibly prone to false positives. Is there a Linux command I could pipe to that specializes in decoding this type of data?

linux

6 Answers

  1. Perl is (as always) your friend. I think this will do it:

    perl -n -mHTML::Entities -e ' ; print HTML::Entities::decode_entities($_) ;'

    E.g.:

    echo '"test" & test $test ! test @ # $ % ^ & *' |perl -n -mHTML::Entities -e ' ; print HTML::Entities::decode_entities($_) ;'

    With output:

    someguy@somehost ~]$ echo '"test" & test $test ! test @ # $ % ^ & *' |perl -n -mHTML::Entities -e ' ; print HTML::Entities::decode_entities($_) ;'
"test" & test $test ! test @ # $ % ^ & *
    • 14
  2. Best Answer

    PHP is well suited to this. This example requires PHP 5:

    cat file.html | php -R 'echo html_entity_decode($argn);'
    • 10

  3. recode seems available on default packages repositories of main GNU/Linux distributions. E.g. to decode HTML entities into UTF-8 :

    …|recode html..utf8
    • 10

  4. With Python 3:

    python3 -c 'import html,sys; print(html.unescape(sys.stdin.read()), end="")' < file.html
    • 5

  5. Takes text file from stdin:

    #!/bin/bash
#
while read lin; do
  newl=${lin//&gt;/>}
  newl=${newl//&lt;/<}
  newl=${newl//&amp;/<}
  # ...other entites
  echo "$newl"
done

    It probably needs bash >= version 4

    • 0

  6. I use this script. Save it as html2utf.py, and use it ala echo $some_html | html2utf.py.

    #!/usr/bin/env python3
"""
An alternative for `perl -Mopen=locale -MHTML::Entities -pe '$_ = decode_entities($_)'` (which you can use by `cpanm HTML::Entities`) and `recode html..`.
"""

import fileinput
import html

for line in fileinput.input():
    print(html.unescape(line.rstrip('\n')))
    • 0