| brent timothy saner on 10 Mar 2016 18:11:22 -0800 |
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
| Re: [PLUG] Command line tools faster than Hadoop cluster |
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 03/10/2016 09:02 PM, bergman@merctech.com wrote: > In the message dated: Thu, 10 Mar 2016 18:42:23 -0500, > The pithy ruminations from Eric Lucas on > <[PLUG] Command line tools faster than Hadoop cluster> were: > => I just throw this out for your perusal - I stumbled across it this morning > => and found it interesting. > > Thanks for sending that around. > > I found it interesting, and was thinking of showing it some of the > data analysts here at $WORK ...and then the author lost [almost] all > credibility at only the 2nd example due to the useless use of cat[1] > when discussing optimization: > cat *.png | grep "Results" > > That pipe is not in anyway "parallel"...if it was, then > cat *.txt | cat | cat | cat | grep "Results" > would be even faster! Not quite UUoC as it's *concatenating* (i.e., exactly what cat was originally designed to do) all files matching the glob into one single pipestream, which grep (albeit grep operates sequentially) operates on. Which, for the record, is technically going to be faster than operating grep on the globbed file list itself. Sorry, that was severely bothering me. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCgAGBQJW4ik7AAoJEIwATC+TSB9rRXAQAL+3tkeymAyd3R+y/Oh4Btff g9A4pkAQbnBEWisZ9AwGwFGpUZoW4d3YVna55yV6S703B2DQOhrmIu+k9ud0JzFb lJ/oyoquvem7jZ2vHqDFT4mW5AsNo7QONaOBHWu/M4HS+KUoYlm7dGB8qRGVvGCl NlPcdCStvVJUrU3DZyxTosz1dcyvkF6ohgvF7cExDh1548yoAPVL2vmYGab+0CTa l5Y3T1cjrCi2JYvwe9YIEBPdEbofz7qDkG4091wvGoFGBd/XTAJMa14UaS3U52a+ 9AaN95aK+Wzh74NHsqyXGfPJGjk8Y77Up7NHe6ra/Hy26h1Em70oaRGqh/K/MZyI K5DffZUxrstk9EpkF/qJCC9ejI6iT7BtXnXKRA5PSElqkNZwGvgxDITIZDtGLJcg oJDGyQle5Xe6VHcoed8eHvltVF+bXyaq73gjN7r4D7jWKgS45IegpiGTNWuoNOsn 6jQRoWD4vkRY4irVC1hC2KxevgwCpVGvSc1jKqXqWejxJD6cy63sHUX/VnaLc8oN FVW2W7AyXiXV+edbFvboQ09TeEoItuFUouttSr2Q7HCt2SfR2tBORlxxsAXOqHgi oK4/gXJvRHngPiay7255WsE8tAH7d99D+HcpYfLwx23zsAHJYjU3/ipluCjx/KKj ayicgNPveGTAYNtAS6D7 =tOaX -----END PGP SIGNATURE----- ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug