Re: [PLUG] multiprocessing on Python

[Rich Freeman <r-plug@thefreemanclan.net>]

On Thu, Jul 4, 2013 at 3:37 PM, Christopher Ochsenreither
<c.g.ochsenreither@gmail.com> wrote:
> Someone at the PLUG central meeting was asking about distributing a Python
> process over multiple cores. I thought this article might be of interest:

I was messing around with those modules when I was trying to move my
git repository analyzer map/reduce solution out of hadoop.  I
struggled to get them to work properly and saturate my CPUs, and ended
up just using GNU parallel instead.

For cases where you are just processing a fixed input that can be
split and processed in any order (like map/reduce) GNU parallel works
very well.  Since i was running multiple map/reduce iterations it also
allowed me to run map and reduce states in parallel by pipe (the sort
step obviously limited the ability to completely pipe all the stages
effectively).

However, if your parallelism needs are limited to a subset of your
program's functionality those python modules probably are the best
place to start...

Rich
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug