Re: [PLUG] >32K concurrent processes

[Walt Mankowski <waltman@pobox.com>]

If you get to about 32,000 process and then separately fork 500
processes that do nothing but sleep, does the same thing happen? That
would let you know it's the kernel at fault and not a bug in your
system.

Walt

On Mon, Apr 18, 2016 at 03:46:41PM -0400, K.S. Bhaskar wrote:
> Walt, I can (easily, and with a bit of patience) get past 32,000 processes
> concurrently accessing the database (and yes, at that point the 16 CPUs are
> spending about half their time in system mode, presumably doing scheduling,
> context switching, handling interrupts etc.). Somewhere around 32,500 GT.M
> processes - which would be around 32,768 total processes, that's my smoking
> gun, along with the fact that I get no more processes messages - the system
> basically becomes unresponsive. With the fans running full tilt at that
> point, I can feel the heat the system is putting out, but when it gets
> there, the only way to get it back to sanity is the power switch.
> 
> -- Bhaskar
> 
> 
> On Mon, Apr 18, 2016 at 3:32 PM, Walt Mankowski <waltman@pobox.com> wrote:
> 
> > Even with an O(1) scheduler I (and it seems many of the other posters
> > in this thread) would be concerned about performance and throughput
> > with that many processes. It seems to me that it's possible your
> > kernel could be spending a significant amount of time scheduling
> > processes.
> >
> > While you're trying to figure out how to have >32k processes, another
> > test I'd suggest running would be to crank things up to, say, 31500
> > processes and see what your performance is like then. Of course, given
> > your other comments I'm guessing you may have already done that, but
> > if not it seems like a good idea.
> >
> > Walt
> >
> > On Mon, Apr 18, 2016 at 02:51:24PM -0400, K.S. Bhaskar wrote:
> > > Rich --
> > >
> > > Yes, in a high volume production environment there are many context
> > > switches - at least tens of thousands per second, perhaps hundreds of
> > > thousands per second (I don't have the numbers from a benchmark at my
> > > finger tips, so I can't comment on actual numbers).
> > >
> > > Regards
> > > -- Bhaskar
> > >
> > >
> > > On Mon, Apr 18, 2016 at 12:43 PM, Rich Freeman <
> > r-plug@thefreemanclan.net>
> > > wrote:
> > >
> > > > On Mon, Apr 18, 2016 at 11:49 AM, K.S. Bhaskar <bhaskar@bhaskars.com>
> > > > wrote:
> > > > > Thanks for the suggestions, Gavin, but batching the load won't work
> > in
> > > > this
> > > > > case. We're trying to run a workload that simulates a large number of
> > > > > concurrent users (as you might find at a large financial or
> > healthcare
> > > > > institution) all of whom expect the system to respond immediately
> > when
> > > > they
> > > > > ask it to do something. I intend to play with the scheduler.
> > > > >
> > > >
> > > > My understanding is that linux uses an O(1) scheduler so it shouldn't
> > > > bog down in actual task switching.  Now, of course that doesn't change
> > > > the fact that your CPUs are split 32k ways.  You probably could tune
> > > > how much time each process gets in a slice, or how they're prioritized
> > > > when there is contention.
> > > >
> > > > I'm not actually sure that 2 processes switching back and forth 10k
> > > > times in a second is any better than 10k processes switching back and
> > > > forth once each in a second.  Obviously with more processes each gets
> > > > less time, but the overhead of task switching itself may not change by
> > > > much.
> > > >
> > > > --
> > > > Rich
> > > >
> > ___________________________________________________________________________
> > > > Philadelphia Linux Users Group         --
> > > > http://www.phillylinux.org
> > > > Announcements -
> > > > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > > General Discussion  --
> > > > http://lists.phillylinux.org/mailman/listinfo/plug
> > > >
> >
> > >
> > ___________________________________________________________________________
> > > Philadelphia Linux Users Group         --
> > http://www.phillylinux.org
> > > Announcements -
> > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > General Discussion  --
> > http://lists.phillylinux.org/mailman/listinfo/plug
> >
> >
> > ___________________________________________________________________________
> > Philadelphia Linux Users Group         --
> > http://www.phillylinux.org
> > Announcements -
> > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > General Discussion  --
> > http://lists.phillylinux.org/mailman/listinfo/plug
> >
> >

> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

Attachment: signature.asc
Description: PGP signature

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug