Re: [PLUG] >32K concurrent processes

On Mon, Apr 18, 2016 at 4:17 PM, Keith C. Perry <kperry@daotechnologies.com> wrote:

Ok, what is /proc/sys/kernel/threads-max showing?

Also, you might need to bump the limits up in the /etc/security/limits.conf for the user you are running the tests on.

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
Keith C. Perry, MS E.E.
Owner, DAO Technologies LLC
(O) +1.215.525.4165 x2033
(M) +1.215.432.5167
www.daotechnologies.com

From: "K.S. Bhaskar" <bhaskar@bhaskars.com>
To: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org>
Sent: Monday, April 18, 2016 4:14:28 PM

Subject: Re: [PLUG] >32K concurrent processes

Yes Keith, /proc/sys/kernel/pid_max shows 1048576.

Regards
-- Bhaskar

On Mon, Apr 18, 2016 at 4:07 PM, Keith C. Perry <kperry@daotechnologies.com> wrote:
Bhaskar,

Is /proc/sys/kernel/pid_max showing 1048576 ?

~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
Keith C. Perry, MS E.E.
Owner, DAO Technologies LLC
(O) +1.215.525.4165 x2033
(M) +1.215.432.5167
www.daotechnologies.com

From: "K.S. Bhaskar" <bhaskar@bhaskars.com>
To: "Philadelphia Linux User's Group Discussion List" <plug@lists.phillylinux.org>
Sent: Monday, April 18, 2016 4:01:25 PM
Subject: Re: [PLUG] >32K concurrent processes

Walt, I already have fs.file-max=3290410 and kernel.pid_max=1048576. Port configuration is not meaningful - the processes don't do any network activity.

Regards
-- Bhaskar

On Mon, Apr 18, 2016 at 3:38 PM, Walt Mankowski <waltman@pobox.com> wrote:
Aha.

I found this page linked off of a Stack Overflow thread. It's 5 years
old so it might be a bit dated, but it seems to address many of the
questions you originally raised.

http://web.archive.org/web/20111209081734/http://research.cs.wisc.edu/condor/condorg/linux_scalability.html

Walt

On Mon, Apr 18, 2016 at 03:31:13PM -0400, K.S. Bhaskar wrote:
> 32K+ processes concurrently accessing the database is exactly what I am
> trying to accomplish. On the machine I have, I cannot have those processes
> do any real application work (so an update by each process once every 10
> seconds is all I'm attempting). On a production machine, those processes
> will be doing real work resulting in between 5 and 10 million database
> accesses per second (and yes, this throughput is demonstrated and proven
> with real application code but slightly less than 32K processes)
>
> So I'm not trying to prove how much work I can push through the database -
> I know that already. I'm just trying to figure out what Linux parameters I
> need to tweak to push the number of processes above 32K).
>
> Regards
> -- Bhaskar
>
>
> On Mon, Apr 18, 2016 at 3:14 PM, Gavin W. Burris <bug@wharton.upenn.edu>
> wrote:
>
> > So, are you running 32k clients to hammer the database??? What are you
> > trying to accomplish? Cheers.
> >
> > On Mon 04/18/16 02:48PM EDT, K.S. Bhaskar wrote:
> > > Thanks Gavin. With GT.M real-time database replication, there is no
> > single
> > > point of failure. Furthermore, there are hooks to create and deploy
> > > applications that remain available not just in the face of unplanned
> > events
> > > (like system crashes) but even planned events (such as application
> > > upgrades, even many upgrades that involve schema changes). It is a proven
> > > architecture which first went into daily live production in 1999.
> > >
> > > Regards
> > > -- Bhaskar
> > >
> > >
> > > On Mon, Apr 18, 2016 at 12:36 PM, Gavin W. Burris <bug@wharton.upenn.edu
> > >
> > > wrote:
> > >
> > > > Hi, Bhaskar.
> > > >
> > > > AH, OK. I should have asked, "What are you trying to accomplish?"
> > Don't
> > > > run everything on one box! Scale horizontally, with at least two
> > > > user-facing nodes. You want to engineer in redundancy from
> > square-one. If
> > > > you don't, there will be no ability to sanely handle critical
> > > > patching/updates, or deal with scaling up.
> > > >
> > > > With Grid Engine, that would be two master hosts, and at least two
> > compute
> > > > nodes actually running the procs, all with an nfs shared cell
> > directory;
> > > > The secondary master is called the shadow master in Grid Engine-speak.
> > > > Grid Engine would be a good solution if you need to run some existing
> > > > command-line or batch code.
> > > >
> > > > If this is for web, strongly consider having a redundant API endpoint
> > to
> > > > run functions. A good way to do this would be with Docker and Swarm.
> > > > Docker is a completely different approach, but one that is correct for
> > > > scaling web applications.
> > > >
> > > > Cheers.
> > > >
> > > > On Mon 04/18/16 11:49AM EDT, K.S. Bhaskar wrote:
> > > > > Thanks for the suggestions, Gavin, but batching the load won't work
> > in
> > > > this
> > > > > case. We're trying to run a workload that simulates a large number of
> > > > > concurrent users (as you might find at a large financial or
> > healthcare
> > > > > institution) all of whom expect the system to respond immediately
> > when
> > > > they
> > > > > ask it to do something. I intend to play with the scheduler.
> > > > >
> > > > > Regards
> > > > > -- Bhaskar
> > > > >
> > > > >
> > > > > On Mon, Apr 18, 2016 at 9:13 AM, Gavin W. Burris <
> > bug@wharton.upenn.edu>
> > > > > wrote:
> > > > >
> > > > > > Good morning, Bhaskar.
> > > > > >
> > > > > > Have you considered using /dev/shm aka tmpfs for shared memory on
> > > > Linux?
> > > > > > Maybe stage all required files there and make sure you are
> > read-only
> > > > where
> > > > > > possible.
> > > > > >
> > > > > > With so many processes, your system is just constantly changing
> > > > threads.
> > > > > > Assuming you are not oversubscribing RAM (32GB / 32k is less than
> > 1MB
> > > > per),
> > > > > > you will want to tune the kernel scheduler.
> > > > > >
> > > > > >
> > > >
> > https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Performance_Tuning_Guide/sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-CPU-Configuration_suggestions.html#sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Configuration_suggestions-Tuning_scheduling_policy
> > > > > >
> > > > > > This very much sounds like an HPC problem (high-performance
> > > > computing), so
> > > > > > my initial reaction is why not use a resource manager tuned for
> > > > > > high-throughput? Take a look at Open Grid Scheduler (
> > > > > > http://gridscheduler.sourceforge.net/), an open source version of
> > Grid
> > > > > > Engine. This will give you a layer of control, a job queue, where
> > you
> > > > > > could then do a task array. Maybe you could launch 1000 jobs that
> > > > iterate
> > > > > > 320 times? The job queue could then be tuned to not overload the
> > > > system
> > > > > > and keep the system maximally / optimally utilized, aka don't run
> > > > > > everything at once but place it in a queue that runs through what
> > you
> > > > need
> > > > > > as resources are available. I would strongly consider using Grid
> > > > Engine,
> > > > > > expecially given your statement that the procs "do a teeny bit of
> > > > activity
> > > > > > every 10 seconds."
> > > > > >
> > > > > > Cheers.
> > > > > >
> > > > > > On Sun 04/17/16 11:12AM EDT, K.S. Bhaskar wrote:
> > > > > > > Thanks for the links Rohit. I'll check them out. The storage is
> > SSD,
> > > > the
> > > > > > > processes do minimal IO - I'm just trying to establish the
> > ability to
> > > > > > have
> > > > > > > a file open by more than 32K processes, and I'm clearly running
> > into
> > > > a
> > > > > > > system limit. This is a development machine (16 cores, 32GB RAM
> > - the
> > > > > > > production machine has something like 64 cores and 512GB RAM),
> > but I
> > > > > > can't
> > > > > > > get you access to poke around because it is inside a corporate
> > > > network.
> > > > > > >
> > > > > > > However, as the software is all open source, I can easily help
> > you
> > > > get
> > > > > > set
> > > > > > > up to poke around using your own system, if you want. Please let
> > me
> > > > know.
> > > > > > >
> > > > > > > Regards
> > > > > > > -- Bhaskar
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Apr 17, 2016 at 10:54 AM, Rohit Mehta <ro@paper-mill.com
> > >
> > > > wrote:
> > > > > > >
> > > > > > > > Some kernel parameters to research (which may not be right for
> > your
> > > > > > > > application)
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > https://www.debian-administration.org/article/656/Installing_Oracle11_and_Oracle12_on_Debian_Wheezy_Squeeze
> > > > > > > > and /etc/security.conf changes
> > > > > > > >
> > > > > >
> > > >
> > http://stackoverflow.com/questions/9361816/maximum-number-of-processes-in-linux
> > > > > > > >
> > > > > > > > Do these process do a lot of IO? Is your storage rotational
> > media
> > > > or
> > > > > > > > SSD? Can your application run off ramdisk storage? Have you
> > tried
> > > > > > > > enabling hyperthreading?
> > > > > > > >
> > > > > > > > Do you have the ability to test application loads
> > non-production
> > > > > > system?
> > > > > > > > If so i'd be interesting in helping you poke around. It might
> > be
> > > > an
> > > > > > > > education for me.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Apr 17, 2016 at 10:42 AM, Rohit Mehta <
> > ro@paper-mill.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Back many years ago, I installed Oracle on my Debian
> > workstation
> > > > for
> > > > > > fun,
> > > > > > > >> and I remember the guide had a lot of tweaks. "ulimit" is one
> > > > that I
> > > > > > can
> > > > > > > >> think of, but I don't remember them all. I'm poking around
> > the
> > > > > > internet to
> > > > > > > >> see if I can find the oracle guide (although it might not be
> > > > relevant
> > > > > > on
> > > > > > > >> newer kernels)
> > > > > > > >>
> > > > > > > >> On Sun, Apr 17, 2016 at 10:27 AM, K.S. Bhaskar <
> > > > bhaskar@bhaskars.com>
> > > > > > > >> wrote:
> > > > > > > >>
> > > > > > > >>> Thanks Steve, but in this case we have a customer need to
> > crank
> > > > up
> > > > > > the
> > > > > > > >>> number of processes on Linux.
> > > > > > > >>>
> > > > > > > >>> Regards
> > > > > > > >>> -- Bhaskar
> > > > > > > >>>
> > > > > > > >>> On Sat, Apr 16, 2016 at 4:09 PM, Steve Litt <
> > > > > > slitt@troubleshooters.com>
> > > > > > > >>> wrote:
> > > > > > > >>>
> > > > > > > >>>> On Fri, 15 Apr 2016 17:40:09 -0400
> > > > > > > >>>> "K.S. Bhaskar" <bhaskar@bhaskars.com> wrote:
> > > > > > > >>>>
> > > > > > > >>>> > I am trying to crank up more than 32K concurrent processes
> > > > (the
> > > > > > > >>>> > processes themselves hang and do a teeny bit of activity
> > > > every 10
> > > > > > > >>>> > seconds). But the OS (64-bit Debian 8 - Jessie) stubbornly
> > > > > > refuses to
> > > > > > > >>>> > crank up beyond 32K-ish processes. pid_max is set to a
> > very
> > > > large
> > > > > > > >>>> > number (1M), so that's not it. Any suggestions on what
> > limits
> > > > to
> > > > > > look
> > > > > > > >>>> > for appreciated. Thank you very much.
> > > > > > > >>>>
> > > > > > > >>>> This is old information, but back in the day people who
> > wanted
> > > > lots
> > > > > > and
> > > > > > > >>>> lots of processes used one of the BSDs to host that server.
> > > > > > > >>>>
> > > > > > > >>>> SteveT
> > > > > > > >>>>
> > > > > > > >>>> Steve Litt
> > > > > > > >>>> April 2016 featured book: Rapid Learning for the 21st
> > Century
> > > > > > > >>>> http://www.troubleshooters.com/rl21
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > >
> > > >
> > ___________________________________________________________________________
> > > > > > > >>>> Philadelphia Linux Users Group --
> > > > > > > >>>> http://www.phillylinux.org
> > > > > > > >>>> Announcements -
> > > > > > > >>>> http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > > > > > >>>> General Discussion --
> > > > > > > >>>> http://lists.phillylinux.org/mailman/listinfo/plug
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > >
> > > >
> > ___________________________________________________________________________
> > > > > > > >>> Philadelphia Linux Users Group --
> > > > > > > >>> http://www.phillylinux.org
> > > > > > > >>> Announcements -
> > > > > > > >>> http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > > > > > >>> General Discussion --
> > > > > > > >>> http://lists.phillylinux.org/mailman/listinfo/plug
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > > >
> > > > > >
> > > >
> > ___________________________________________________________________________
> > > > > > > > Philadelphia Linux Users Group --
> > > > > > > > http://www.phillylinux.org
> > > > > > > > Announcements -
> > > > > > > > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > > > > > > General Discussion --
> > > > > > > > http://lists.phillylinux.org/mailman/listinfo/plug
> > > > > > > >
> > > > > > > >
> > > > > >
> > > > > > >
> > > > > >
> > > >
> > ___________________________________________________________________________
> > > > > > > Philadelphia Linux Users Group --
> > > > > > http://www.phillylinux.org
> > > > > > > Announcements -
> > > > > > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > > > > > General Discussion --
> > > > > > http://lists.phillylinux.org/mailman/listinfo/plug
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Gavin W. Burris
> > > > > > Senior Project Leader for Research Computing
> > > > > > The Wharton School
> > > > > > University of Pennsylvania
> > > > > > Search our documentation:
> > http://research-it.wharton.upenn.edu/about/
> > > > > > Subscribe to the Newsletter:
> > http://whr.tn/ResearchNewsletterSubscribe
> > > > > >
> > > >
> > ___________________________________________________________________________
> > > > > > Philadelphia Linux Users Group --
> > > > > > http://www.phillylinux.org
> > > > > > Announcements -
> > > > > > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > > > > General Discussion --
> > > > > > http://lists.phillylinux.org/mailman/listinfo/plug
> > > > > >
> > > >
> > > > >
> > > >
> > ___________________________________________________________________________
> > > > > Philadelphia Linux Users Group --
> > > > http://www.phillylinux.org
> > > > > Announcements -
> > > > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > > > General Discussion --
> > > > http://lists.phillylinux.org/mailman/listinfo/plug
> > > >
> > > >
> > > > --
> > > > Gavin W. Burris
> > > > Senior Project Leader for Research Computing
> > > > The Wharton School
> > > > University of Pennsylvania
> > > > Search our documentation: http://research-it.wharton.upenn.edu/about/
> > > > Subscribe to the Newsletter: http://whr.tn/ResearchNewsletterSubscribe
> > > >
> > ___________________________________________________________________________
> > > > Philadelphia Linux Users Group --
> > > > http://www.phillylinux.org
> > > > Announcements -
> > > > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > > General Discussion --
> > > > http://lists.phillylinux.org/mailman/listinfo/plug
> > > >
> >
> > >
> > ___________________________________________________________________________
> > > Philadelphia Linux Users Group --
> > http://www.phillylinux.org
> > > Announcements -
> > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > > General Discussion --
> > http://lists.phillylinux.org/mailman/listinfo/plug
> >
> >
> > --
> > Gavin W. Burris
> > Senior Project Leader for Research Computing
> > The Wharton School
> > University of Pennsylvania
> > Search our documentation: http://research-it.wharton.upenn.edu/about/
> > Subscribe to the Newsletter: http://whr.tn/ResearchNewsletterSubscribe
> > ___________________________________________________________________________
> > Philadelphia Linux Users Group --
> > http://www.phillylinux.org
> > Announcements -
> > http://lists.phillylinux.org/mailman/listinfo/plug-announce
> > General Discussion --
> > http://lists.phillylinux.org/mailman/listinfo/plug
> >

> ___________________________________________________________________________
> Philadelphia Linux Users Group -- http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group -- http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug