Re: [PLUG] gnu parallel and tar

[Austin Murphy <austin.murphy@gmail.com>]

On Mon, Apr 4, 2011 at 2:37 PM, Julien Vehent <julien@linuxwall.info> wrote:
> On Mon, 4 Apr 2011 13:21:59 -0400, Austin Murphy wrote:
>>
>> I've had a good experience with lbzip2, a multi-threaded
>> implementation of bzip.
...
> Initial file:
> $ ls -s jmeter-server-node1.log --block-size=1
> 689274880 jmeter-server-node1.log
>
>
> === with bzip2 ====
> $ time bzip2 -z -9 jmeter-server-node1.log
>
> real    8m33.220s
> user    8m31.444s
> sys     0m0.880s
>
> $ ls -s jmeter-server-node1.log.bz2 --block-size=1
> 1589248 jmeter-server-node1.log.bz2
...
> === with lbzip2 ====
> $ time lbzip2 -n 4 -z -9 -S jmeter-server-node1.log
>
> real    5m37.425s
> user    20m57.227s
> sys     0m5.016s
>
> $ ls -s jmeter-server-node1.log.bz2 --block-size=1
> 1601536 jmeter-server-node1.log.bz2
...
> Compression is of the same level, but I'm surprised to see that while lbzip2
> is 65% faster, it also uses 250% more user time than bzip2. The efficiency
> per-core is a lot lower, but I'm happy to be using all my cores.


My understanding is that bzip2 is highly optimized to avoid cache
misses.  If you have too many threads running at once you might be
blowing out a shared cache.   You might try running with -n 2 or
letting it decide how many threads to run.

Austin
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug