Re: [PLUG] GNUMake

On Fri, Nov 21, 2014 at 8:58 AM, Gavin W. Burris <bug@wharton.upenn.edu> wrote:

Hi, Keith.

What you are describing is something referred to as batch queuing. One
way to do this is with a job queue and resource manager that distributes
jobs (shell scripts) across a pool of hosts. You may want to look into
installing Son of Grid Engine: https://arc.liv.ac.uk/trac/SGE

The default install is pretty easy. The tricky part can come when
adjusting job limits and priorities based on many projects, which you
didn't mention. That said, it may look like overkill, but it is
impossible to keep track of everything without something like SGE.

If you don't care about errors, or this is a one-time thing, maybe just
using parallel SSH to run commands is enough.
https://code.google.com/p/parallel-ssh/

Cheers.

On 08:33AM Fri 11/21/14 -0500, Keith Brown wrote:
> I was wondering if anyone used a hook with a Makefile to distribute
> the load across multiple nodes. I have few hundred awk/sed scripts
> which I use with make; there are many processes (1000+) which can run
> in parallel so I want to use other desktops to share the load. has
> anyone done this before
> ___________________________________________________________________________
> Philadelphia Linux Users Group -- http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

--
Gavin W. Burris
Senior Project Leader for Research Computing
The Wharton School
University of Pennsylvania
___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug