Re: [PLUG] >32K concurrent processes

You may have to do some kernel hacking... possibly changing the process scheduler and the preemption model as well.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
Keith C. Perry, MS E.E.

Owner, DAO Technologies LLC

(O) +1.215.525.4165 x2033

(M) +1.215.432.5167

www.daotechnologies.com

From: "K.S. Bhaskar" <bhaskar@bhaskars.com>
To: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org>
Sent: Monday, April 18, 2016 3:46:41 PM
Subject: Re: [PLUG] >32K concurrent processes

Walt, I can (easily, and with a bit of patience) get past 32,000 processes concurrently accessing the database (and yes, at that point the 16 CPUs are spending about half their time in system mode, presumably doing scheduling, context switching, handling interrupts etc.). Somewhere around 32,500 GT.M processes - which would be around 32,768 total processes, that's my smoking gun, along with the fact that I get no more processes messages - the system basically becomes unresponsive. With the fans running full tilt at that point, I can feel the heat the system is putting out, but when it gets there, the only way to get it back to sanity is the power switch.

-- Bhaskar

On Mon, Apr 18, 2016 at 3:32 PM, Walt Mankowski <waltman@pobox.com> wrote:

Even with an O(1) scheduler I (and it seems many of the other posters
in this thread) would be concerned about performance and throughput
with that many processes. It seems to me that it's possible your
kernel could be spending a significant amount of time scheduling
processes.

While you're trying to figure out how to have >32k processes, another
test I'd suggest running would be to crank things up to, say, 31500
processes and see what your performance is like then. Of course, given
your other comments I'm guessing you may have already done that, but
if not it seems like a good idea.

Walt

On Mon, Apr 18, 2016 at 02:51:24PM -0400, K.S. Bhaskar wrote:
> Rich --
>
> Yes, in a high volume production environment there are many context
> switches - at least tens of thousands per second, perhaps hundreds of
> thousands per second (I don't have the numbers from a benchmark at my
> finger tips, so I can't comment on actual numbers).
>
> Regards
> -- Bhaskar
>
>
> On Mon, Apr 18, 2016 at 12:43 PM, Rich Freeman <r-plug@thefreemanclan.net>
> wrote:
>
> > On Mon, Apr 18, 2016 at 11:49 AM, K.S. Bhaskar <bhaskar@bhaskars.com>
> > wrote:
> > > Thanks for the suggestions, Gavin, but batching the load won't work in
> > this
> > > case. We're trying to run a workload that simulates a large number of
> > > concurrent users (as you might find at a large financial or healthcare
> > > institution) all of whom expect the system to respond immediately when
> > they
> > > ask it to do something. I intend to play with the scheduler.
> > >
> >
> > My understanding is that linux uses an O(1) scheduler so it shouldn't
> > bog down in actual task switching. Now, of course that doesn't change
> > the fact that your CPUs are split 32k ways. You probably could tune
> > how much time each process gets in a slice, or how they're prioritized
> > when there is contention.
> >
> > I'm not actually sure that 2 processes switching back and forth 10k
> > times in a second is any better than 10k processes switching back and
> > forth once each in a second. Obviously with more processes each gets
> > less time, but the overhead of task switching itself may not change by
> > much.
> >
> > --
> > Rich
> > ___________________________________________________________________________
> > Philadelphia Linux Users Group --
> > http://www.phillylinux.org
> > Announcements -
> > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > General Discussion --
> > http://lists.phillylinux.org/mailman/listinfo/plug
> >

> ___________________________________________________________________________
> Philadelphia Linux Users Group -- http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug