Rich Freeman on 5 Jul 2013 04:54:27 -0700

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] multiprocessing on Python

On Thu, Jul 4, 2013 at 3:37 PM, Christopher Ochsenreither
<> wrote:
> Someone at the PLUG central meeting was asking about distributing a Python
> process over multiple cores. I thought this article might be of interest:

I was messing around with those modules when I was trying to move my
git repository analyzer map/reduce solution out of hadoop.  I
struggled to get them to work properly and saturate my CPUs, and ended
up just using GNU parallel instead.

For cases where you are just processing a fixed input that can be
split and processed in any order (like map/reduce) GNU parallel works
very well.  Since i was running multiple map/reduce iterations it also
allowed me to run map and reduce states in parallel by pipe (the sort
step obviously limited the ability to completely pipe all the stages

However, if your parallelism needs are limited to a subset of your
program's functionality those python modules probably are the best
place to start...

Philadelphia Linux Users Group         --
Announcements -
General Discussion  --