Jason Harlow on 1 Apr 2011 19:07:52 -0700


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] gnu parallel and tar


Not sure about the tar bit, but I've used pigz ( http://www.zlib.net/pigz/ ) for the compression piece on many-core servers for fairly significant performance improvements.


On Fri, Apr 1, 2011 at 9:26 PM, Julien Vehent <julien@linuxwall.info> wrote:
On Fri, 1 Apr 2011 19:39:20 -0400, K.S. Bhaskar wrote:
Switch from bz2 to gzip and you may get home before Monday morning!
bzip2 is terribly slow for large byte streams - I once found it faster
move data uncompressed than to use bz2 (gzip was faster yet).  bzip2
is great if you have to download a disk image over a 110 Baud modem.

Sorry, I don't know about parallelizing tar.


well. I compressed 2GB of log files into a 10MB archive, and be able to send it by email, so bzip2 was worth it in the end, even on one core.

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug