Re: [PLUG] GNUMake

["Gavin W. Burris" <bug@wharton.upenn.edu>]

Thumbs up for SLURM.  I've used this on Stampede.

On 06:55AM Fri 11/21/14 -0800, Doug Stewart wrote:
> If you're looking for distributed processing, there's also HTCondor: http://research.cs.wisc.edu/htcondor/description.html
> 
> 
> TORQUE/Maui: http://old.adaptivecomputing.com/docs/217
> 
> 
> 
> 
> 
> SLURM: http://slurm.schedmd.com/
> 
> 
> 
> 
> 
> Openlava: http://www.openlava.org/
> 
> 
> 
> 
> 
> 
> —
> Doug
> 
> On Fri, Nov 21, 2014 at 8:58 AM, Gavin W. Burris <bug@wharton.upenn.edu>
> wrote:
> 
> > Hi, Keith.
> > What you are describing is something referred to as batch queuing.  One
> > way to do this is with a job queue and resource manager that distributes
> > jobs (shell scripts) across a pool of hosts.  You may want to look into
> > installing Son of Grid Engine: https://arc.liv.ac.uk/trac/SGE
> > The default install is pretty easy.  The tricky part can come when
> > adjusting job limits and priorities based on many projects, which you
> > didn't mention.  That said, it may look like overkill, but it is
> > impossible to keep track of everything without something like SGE.
> > If you don't care about errors, or this is a one-time thing, maybe just
> > using parallel SSH to run commands is enough.
> > https://code.google.com/p/parallel-ssh/
> > Cheers.
> > On 08:33AM Fri 11/21/14 -0500, Keith Brown wrote:
> >> I was wondering if anyone used a hook with a Makefile to distribute
> >> the load across multiple nodes. I have few hundred awk/sed scripts
> >> which I use with make; there are many processes (1000+) which can run
> >> in parallel so I want to use other desktops to share the load. has
> >> anyone done this before
> >> ___________________________________________________________________________
> >> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> >> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> >> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug
> > -- 
> > Gavin W. Burris
> > Senior Project Leader for Research Computing
> > The Wharton School
> > University of Pennsylvania
> > ___________________________________________________________________________
> > Philadelphia Linux Users Group         --        http://www.phillylinux.org
> > Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug


-- 
Gavin W. Burris
Senior Project Leader for Research Computing
The Wharton School
University of Pennsylvania
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug