Home / ubuntu / Questions / 1473060
Accepted
Ray Butterworth
Asked: 2023-06-18 11:27:07 +0800 CST

Is there a command-line method for converting UTF-8 values into Unicode values?

  • 772

$ od -t x1 <inputfile produces … 0a e2 8c a5 0a ….

The 0a are linefeeds, and e2 8c a5 are the UTF-8 representation of a Unicode character.

For this simple case, I can do it by hand:

[1110]0010  [10]001100  [10]100101  =  10  0011  0010  0101 = 2 3 2 5

What shell command line can convert e2 8c a5 or e28ca5 into 2325?

(For completeness, converting the other way would be good to know too.)

command-line

3 Answers

  1. Best Answer

    Try this command line to get what you want, '2325',

    $ <<<'e28ca5' xxd -r -p | iconv -t unicode | hexdump
0000000 feff 2325                              
0000004

    See also the output from the first part of the command line

    <<<'e28ca5' xxd -r -p

    (there is no line feed, so the prompt comes directly after its output),

    sudodus@c30 ~ $ <<<'e28ca5' xxd -r -p
⌥sudodus@c30 ~ $

    As you can see, I get that special character ⌥ in the gnome-terminal window.

    • 2

  2. Using the Perl Encode module, you could

    1. pack the character string back into a sequence of bytes1
    2. decode the byte sequence as a UTF-8 character
    3. encode the result as UTF-16be
    4. unpack it to get the hexadecimal code point

    So

    $ printf '%s' 'e28ca5' | perl -MEncode=encode,decode -nE '
    say unpack("H*", encode("UTF-16be", decode("UTF-8", pack("H*",$_))))
'
2325

    1. this step only required because you've unpacked it with od - if you start with the character itself or byte sequence you just need the decode-encode:

       $ printf '⌥' | perl -MEncode=encode,decode -nE '
     say unpack("H*", encode("UTF-16be", decode("UTF-8", $_)))
 '
 2325

      or

       $ printf '\xe2\x8c\xa5' | perl -MEncode=encode,decode -nE '
     say unpack("H*", encode("UTF-16be", decode("UTF-8", $_)))
 '
 2325
    • 1

  3. Try the following:

    iconv -f utf8 -t ucs2 <inputfile | hexdump -v -e '/2 "%04x "'

    For inputfile containing three bytes with values e2 8c a5 it outputs 2325.

    • 0