Comments on: Sorting large files faster with a shell script

By: Daniel

Daniel — Thu, 17 Jun 2010 20:03:19 +0000

ops, that sort line should be

sort $file (remove the -o $file)

By: Daniel

Daniel — Thu, 17 Jun 2010 20:02:02 +0000

This script runs into problems if the filesize is not large enough be split as requested.

I added the following lines to my copy

if [ $size -lt 10485760 ]
then
sort -o $file $file
exit
fi

While the above script works great on large files, I just wanted to just take the easy way out with

for FILE in $FILES; do ; distsort.sh $FILE > $FILE.new
done

By: Joel

Joel — Mon, 27 Apr 2009 23:57:28 +0000

Thank you! I just applied this to ~80 million lines of data (700MB in bz2), sorted in 40 minutes on 16 machines.

By: admin

admin — Tue, 22 Jan 2008 02:56:29 +0000

Good points guys. I think if you have multiple processors, the -C option makes sense. However, since compression is also CPU intensive, tests would need to be done to see whether it makes more sense to use -C or to just fire up an additional local sort process.

By: Yves Junqueira

Yves Junqueira — Tue, 22 Jan 2008 01:59:09 +0000

@Brandon,

Good point. Even if ‘sort’ is CPU-bound and SSH would be competing for processor cycles with the local ‘sort’ process, that may not be a problem in the end, since the local sort probably ends much faster than the others, so there is free local cpu power to spare.

And if you have SMP, ssh -C makes even more sense.

By: Brandon

Brandon — Sat, 19 Jan 2008 14:43:29 +0000

You may get even better performance if you transmit your data compressed (use the -C option to ssh).

By: Pages tagged "real"

Pages tagged "real" — Sat, 19 Jan 2008 04:16:50 +0000

[...] Sorting large files faster with a shell script. xmiszBubblez bookmarked on 01/18/08, saved by 1 others [...]