Re: [PLUG] gnu parallel and tar

[Julien Vehent <julien@linuxwall.info>]

On Mon, 4 Apr 2011 13:21:59 -0400, Austin Murphy wrote:
> I've had a good experience with lbzip2, a multi-threaded
> implementation of bzip.
>
> It is compatible with regular bzip2.  In other words, you can use
> lbzip2 to compress and bzip2 to decompress or vice versa.
>
> http://freshmeat.net/projects/lbzip2
> It's also available packaged from Rpmforge.
>
> Austin

Cool ! I like that. It's in Debian also.

Just gave it a quick shot and the results are interesting:

Initial file:
$ ls -s jmeter-server-node1.log --block-size=1
689274880 jmeter-server-node1.log


=== with bzip2 ====
$ time bzip2 -z -9 jmeter-server-node1.log

real	8m33.220s
user	8m31.444s
sys	0m0.880s

$ ls -s jmeter-server-node1.log.bz2 --block-size=1
1589248 jmeter-server-node1.log.bz2

$ time bunzip2 jmeter-server-node1.log.bz2

real	0m35.801s
user	0m33.662s
sys	0m0.964s


=== with lbzip2 ====
$ time lbzip2 -n 4 -z -9 -S jmeter-server-node1.log

real	5m37.425s
user	20m57.227s
sys	0m5.016s

$ ls -s jmeter-server-node1.log.bz2 --block-size=1
1601536 jmeter-server-node1.log.bz2

$ time lbzip2 -n 4 -d jmeter-server-node1.log.bz2

real	0m20.370s
user	1m15.697s
sys	0m1.316s


Compression is of the same level, but I'm surprised to see that while 
lbzip2 is 65% faster, it also uses 250% more user time than bzip2. The 
efficiency per-core is a lot lower, but I'm happy to be using all my 
cores.


Julien


___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug