brent timothy saner on 10 Mar 2016 18:11:22 -0800


[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Command line tools faster than Hadoop cluster


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 03/10/2016 09:02 PM, bergman@merctech.com wrote:
> In the message dated: Thu, 10 Mar 2016 18:42:23 -0500,
> The pithy ruminations from Eric Lucas on 
> <[PLUG] Command line tools faster than Hadoop cluster> were:
> => I just throw this out for your perusal - I stumbled across it this morning
> => and found it interesting.
> 
> Thanks for sending that around.
> 
> I found it interesting, and was thinking of showing it some of the
> data analysts here at $WORK ...and then the author lost [almost] all
> credibility at only the 2nd example due to the useless use of cat[1]
> when discussing optimization:
> 	cat *.png | grep "Results"
> 
> That pipe is not in anyway "parallel"...if it was, then 
> 	cat *.txt | cat | cat | cat | grep "Results"
> would be even faster!


Not quite UUoC as it's *concatenating* (i.e., exactly what cat was
originally designed to do) all files matching the glob into one single
pipestream, which grep (albeit grep operates sequentially) operates on.
Which, for the record, is technically going to be faster than operating
grep on the globbed file list itself.

Sorry, that was severely bothering me.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBCgAGBQJW4ik7AAoJEIwATC+TSB9rRXAQAL+3tkeymAyd3R+y/Oh4Btff
g9A4pkAQbnBEWisZ9AwGwFGpUZoW4d3YVna55yV6S703B2DQOhrmIu+k9ud0JzFb
lJ/oyoquvem7jZ2vHqDFT4mW5AsNo7QONaOBHWu/M4HS+KUoYlm7dGB8qRGVvGCl
NlPcdCStvVJUrU3DZyxTosz1dcyvkF6ohgvF7cExDh1548yoAPVL2vmYGab+0CTa
l5Y3T1cjrCi2JYvwe9YIEBPdEbofz7qDkG4091wvGoFGBd/XTAJMa14UaS3U52a+
9AaN95aK+Wzh74NHsqyXGfPJGjk8Y77Up7NHe6ra/Hy26h1Em70oaRGqh/K/MZyI
K5DffZUxrstk9EpkF/qJCC9ejI6iT7BtXnXKRA5PSElqkNZwGvgxDITIZDtGLJcg
oJDGyQle5Xe6VHcoed8eHvltVF+bXyaq73gjN7r4D7jWKgS45IegpiGTNWuoNOsn
6jQRoWD4vkRY4irVC1hC2KxevgwCpVGvSc1jKqXqWejxJD6cy63sHUX/VnaLc8oN
FVW2W7AyXiXV+edbFvboQ09TeEoItuFUouttSr2Q7HCt2SfR2tBORlxxsAXOqHgi
oK4/gXJvRHngPiay7255WsE8tAH7d99D+HcpYfLwx23zsAHJYjU3/ipluCjx/KKj
ayicgNPveGTAYNtAS6D7
=tOaX
-----END PGP SIGNATURE-----
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug