Home / server / Questions / 54911
Accepted
Johan
Asked: 2009-08-18 01:45:54 +0800 CST

Debian, How to convert filesystem from ISO-8859-1 into UTF-8?

  • 772

I have a old pc that is running Debian stable, that is in need of a upgrade. The problem is that it is using latin1 (ISO-8859-1) for everything, and since the rest of the world has moved to UTF-8 I plan to convert this computer as well.

And for this question I will focus in on the files that are served with Samba, and some has some latin1 characters in the filenames (like åäö).

Now my plan is to move all data of this old computer onto and a brand new one that is running Debian stable (but with UTF-8).

Does anybody have a good idea?

Note: later I plan to use iconv to convert the content of some files with something like this:

iconv --from-code=ISO-8859-1 --to-code=UTF-8 iso.txt > utf.txt

However I don't know of a good way to convert the filesystem it self.

Note: Normally I usally just scp from one computer to the next, but then I end up with latin1 characters in the utf-8 filesystem.

Update: Did a small test round with a hand full of files (with funny chars) in the filenames, and that seemed like it could work.

convmv -r -f ISO-8859-1 -t UTF-8  *

Note: -r = recursive; -f/-t = from/to.

So it was only to execute with the --notest

convmv -r -f ISO-8859-1 -t UTF-8 --notest *

Nothing more to it.

debian

5 Answers

  1. Best Answer

    Use convmv:

    Package: convmv
Priority: optional
Section: utils
Installed-Size: 88
Maintainer: Raphael Zimmerer <[email protected]>
Architecture: all
Version: 1.12-1
Depends: perl
Filename: pool/main/c/convmv/convmv_1.12-1_all.deb
Size: 20052
MD5sum: dcc45d5b8517026f588d769d81d67768
SHA1: 55da9650cfee5c64d8a4fdf278aaf9401a5e5dec
SHA256: 0a8b0165a78dc42f7dc665a14d21c22ce0433d115fe537be2af74682d3b82a5f
Description: filename encoding conversion tool
 convmv can convert a single filename, a directory tree or all files
 on a filesystem to a different encoding. It only converts the
 encoding of filenames, not files contents. A special feature of
 convmv is that it also takes care of symlinks: the encoding of the
 symlink's target will be converted if the symlink itself is being
 converted.
 .
 It is also possible to convert directories to UTF-8 which are already
 partially UTF-8 encoded.
 .
 Keywords: rename, move
Tag: devel::i18n, implemented-in::perl, interface::commandline, role::program, scope::utility, works-with::file

    apt-get install convmv

    :-)

    • 14

  2. Note for readers who come across this page before they transferred the files:

    you can do the transfer with a recent rsync and the --iconv option:

    rsync -va --iconv=utf8,iso88591 /source/latin1/ /destination/utf8

    (yes, the ordering of the iconv charsets is not intuitive!)

    For when the transfer has already been done, indeed convmv is the solution.

    • 6

  3. There is an Unicode Char Encoding Converter that supports character encoding conversion between Unicode(UTF-8/UTF-16/UTF-7/UTF-32) and non Unicode(Ansi,Chinese simplified GBK, Chinese traditional BIG5, Japanese SHIFT-JIS, Japanese EUC-JP, Korean euc-kr characters set encoding etc).

    • 0

  4. Execute this in the current directory:

     for f in `ls`; do
 iconv --from-code=iso-8859-1 --to-code=utf-8 $f -o $f;
 done
    • 0

  5. I would test first if the file isn't in the target codification, otherwise you can mess up your files, I guess. (the example below to zsh) you can adapt the code and put it inside a "for" loop.

    codification="`file -bi "$1" | awk -F"=" '{print $NF}'`"

[[ "$codification" != "utf-8" ]]  && iconv -f ISO8859-1 -t UTF-8 "$1" > $outfile
    • 0