Eric on 18 Sep 2007 11:53:44 -0000 |
In case anybody else is interested.... http://www.infoq.com/articles/hadoop-interview Eric Toby DiPasquale wrote: > On Wed, Jun 06, 2007 at 03:52:32PM -0400, Evan Weaver wrote: >> Toby, >> >> Can you give a quick overview of why I would want to use Hadoop? I >> didn't really follow you last night, and figured we might as well have >> the discussion on-list. > > This list is not really the place for this discussion, but here goes: > > For you guys, using Hadoop would allow you to put your data in HDFS and > then MapReduce out whatever else you wanted (specifically, the indeces you > are using Memcached for). Those could even still be loaded into Memcached, > although you'd probably have to write some kind of adapter to get > Memcached to talk to HDFS. > > The thing I was really trying to get at is that Hadoop supports your > current style of infrastructure but also gives you more flexibility if you > needed to change things. E.g. if you needed to tear up some of your data > in a new way and export a new set of columns, that's one MapReduce job > away from being reality. Also, it sounded like you guys weren't doing very > frequent inserts and, more importantly, that random access was not the > primary focus of your system, so Hadoop is a good fit for that style of > computation. > > Hadoop itself is based on the core infrastructure at Google, as reflected > by these papers: > > http://labs.google.com/papers/gfs-sosp2003.pdf > http://labs.google.com/papers/mapreduce.html > > In any case, I was just saying that you guys should take a look at it > because it sounded to me like you were recreating a bunch of pieces of it > when its already in existence. I know its written in poopy ol' Java, but > despite that its still worth a look, IMHO. Of course, I may also have > misunderstood what you were saying, so I might be wrong about it. I guess > let me know what you think either way. > > For more on Hadoop, I'd start off here: > > http://wiki.apache.org/lucene-hadoop/ProjectDescription > -- # Eric Lucas # # "Oh, I have slipped the surly bond of earth # And danced the skies on laughter-silvered wings... # -- John Gillespie Magee Jr _______________________________________________ To unsubscribe or change your settings, visit: http://lists.phillyonrails.org/mailman/listinfo/talk
|
|