Re: [PLUG] >32K concurrent processes

[Walt Mankowski <waltman@pobox.com>]

Even with an O(1) scheduler I (and it seems many of the other posters
in this thread) would be concerned about performance and throughput
with that many processes. It seems to me that it's possible your
kernel could be spending a significant amount of time scheduling
processes.

While you're trying to figure out how to have >32k processes, another
test I'd suggest running would be to crank things up to, say, 31500
processes and see what your performance is like then. Of course, given
your other comments I'm guessing you may have already done that, but
if not it seems like a good idea.

Walt

On Mon, Apr 18, 2016 at 02:51:24PM -0400, K.S. Bhaskar wrote:
> Rich --
> 
> Yes, in a high volume production environment there are many context
> switches - at least tens of thousands per second, perhaps hundreds of
> thousands per second (I don't have the numbers from a benchmark at my
> finger tips, so I can't comment on actual numbers).
> 
> Regards
> -- Bhaskar
> 
> 
> On Mon, Apr 18, 2016 at 12:43 PM, Rich Freeman <r-plug@thefreemanclan.net>
> wrote:
> 
> > On Mon, Apr 18, 2016 at 11:49 AM, K.S. Bhaskar <bhaskar@bhaskars.com>
> > wrote:
> > > Thanks for the suggestions, Gavin, but batching the load won't work in
> > this
> > > case. We're trying to run a workload that simulates a large number of
> > > concurrent users (as you might find at a large financial or healthcare
> > > institution) all of whom expect the system to respond immediately when
> > they
> > > ask it to do something. I intend to play with the scheduler.
> > >
> >
> > My understanding is that linux uses an O(1) scheduler so it shouldn't
> > bog down in actual task switching.  Now, of course that doesn't change
> > the fact that your CPUs are split 32k ways.  You probably could tune
> > how much time each process gets in a slice, or how they're prioritized
> > when there is contention.
> >
> > I'm not actually sure that 2 processes switching back and forth 10k
> > times in a second is any better than 10k processes switching back and
> > forth once each in a second.  Obviously with more processes each gets
> > less time, but the overhead of task switching itself may not change by
> > much.
> >
> > --
> > Rich
> > ___________________________________________________________________________
> > Philadelphia Linux Users Group         --
> > http://www.phillylinux.org
> > Announcements -
> > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > General Discussion  --
> > http://lists.phillylinux.org/mailman/listinfo/plug
> >

> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

Attachment: signature.asc
Description: PGP signature

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug