Home / ubuntu / Questions / 1169038
In Process
abbasi_ahsan
Asked: 2019-08-28 22:19:20 +0800 CST

sort is very slow for big files

  • 772

I want to sort file (with removing duplicates) which contains wordlist with a size of almost 25GB. I am using sort command in Ubuntu terminal but it takes hours to output sorted file, the command is:

sort -u input.txt>output.txt

Is there some alternative and efficient way to do the same?

performance sort text-processing file-sorting

2 Answers

  1. From my personal experience: if you want unique lines make sure you also use 

    LC_ALL=C sort -u

    I see speed improvements by a factor 10 but it probably depends on the characters in the file (I often have to use it for translations so I have accented characters in the file).

    • 4

  2. A really incredible tool is xsv (https://github.com/BurntSushi/xsv). Running it for a 3173959 rows file

    xsv sort input.txt >output.csv

    I have the output in 3 seconds

    • 0