Julien Vehent on 4 Apr 2011 11:37:41 -0700

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] gnu parallel and tar

On Mon, 4 Apr 2011 13:21:59 -0400, Austin Murphy wrote:
I've had a good experience with lbzip2, a multi-threaded
implementation of bzip.

It is compatible with regular bzip2.  In other words, you can use
lbzip2 to compress and bzip2 to decompress or vice versa.

It's also available packaged from Rpmforge.


Cool ! I like that. It's in Debian also.

Just gave it a quick shot and the results are interesting:

Initial file:
$ ls -s jmeter-server-node1.log --block-size=1
689274880 jmeter-server-node1.log

=== with bzip2 ====
$ time bzip2 -z -9 jmeter-server-node1.log

real	8m33.220s
user	8m31.444s
sys	0m0.880s

$ ls -s jmeter-server-node1.log.bz2 --block-size=1
1589248 jmeter-server-node1.log.bz2

$ time bunzip2 jmeter-server-node1.log.bz2

real	0m35.801s
user	0m33.662s
sys	0m0.964s

=== with lbzip2 ====
$ time lbzip2 -n 4 -z -9 -S jmeter-server-node1.log

real	5m37.425s
user	20m57.227s
sys	0m5.016s

$ ls -s jmeter-server-node1.log.bz2 --block-size=1
1601536 jmeter-server-node1.log.bz2

$ time lbzip2 -n 4 -d jmeter-server-node1.log.bz2

real	0m20.370s
user	1m15.697s
sys	0m1.316s

Compression is of the same level, but I'm surprised to see that while lbzip2 is 65% faster, it also uses 250% more user time than bzip2. The efficiency per-core is a lot lower, but I'm happy to be using all my cores.


Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug