|Julien Vehent on 4 Apr 2011 11:37:41 -0700|
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
|Re: [PLUG] gnu parallel and tar|
On Mon, 4 Apr 2011 13:21:59 -0400, Austin Murphy wrote:
I've had a good experience with lbzip2, a multi-threaded implementation of bzip. It is compatible with regular bzip2. In other words, you can use lbzip2 to compress and bzip2 to decompress or vice versa. http://freshmeat.net/projects/lbzip2 It's also available packaged from Rpmforge. Austin
Cool ! I like that. It's in Debian also. Just gave it a quick shot and the results are interesting: Initial file: $ ls -s jmeter-server-node1.log --block-size=1 689274880 jmeter-server-node1.log === with bzip2 ==== $ time bzip2 -z -9 jmeter-server-node1.log real 8m33.220s user 8m31.444s sys 0m0.880s $ ls -s jmeter-server-node1.log.bz2 --block-size=1 1589248 jmeter-server-node1.log.bz2 $ time bunzip2 jmeter-server-node1.log.bz2 real 0m35.801s user 0m33.662s sys 0m0.964s === with lbzip2 ==== $ time lbzip2 -n 4 -z -9 -S jmeter-server-node1.log real 5m37.425s user 20m57.227s sys 0m5.016s $ ls -s jmeter-server-node1.log.bz2 --block-size=1 1601536 jmeter-server-node1.log.bz2 $ time lbzip2 -n 4 -d jmeter-server-node1.log.bz2 real 0m20.370s user 1m15.697s sys 0m1.316sCompression is of the same level, but I'm surprised to see that while lbzip2 is 65% faster, it also uses 250% more user time than bzip2. The efficiency per-core is a lot lower, but I'm happy to be using all my cores.
Julien ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug