Re: [PLUG] SOP's

[Bill Jonas <bill@billjonas.com>]

On Fri, Jan 11, 2002 at 11:07:19AM -0800, JP Toto wrote:
> I understand that this is kind of firm specific at
> times but Im looking for info or a template even, to
> flesh out some standard operating procedures for my
> company.

Well, a client of a former employer of mine did it this way.

All new code releases to production were to be done by 10:00am (or
11:00am, I forget).  This was so it would be done before the biggest
part of the day.  They had multiple web servers situated behind a Local
Director (Cisco gear, I believe), and when they prepared to do a
release, they would configure the LD not to point at one of the web
servers.  (When they got more than two or three web servers, they might
have increased the number.)  Then they would update from CVS, restart
Apache, and play with it internally a bit, giving it a cursory
inspection in the actual production environment (the release already
having been through QA).  At this point, they could still bail if
something was wrong.  Assuming it looked fine, they would reconfigure
the LD to point to the server with the new code and away from the one(s)
with the old code.  At this point, they would keep a close eye on the
new code before updating the other server(s) and putting them back
online.  If there was a problem that only showed up under load, it was
trivial to fall back to the old codebase.  This was only done twice a
week, on Tuesdays and Thursdays (a couple of the slowest days).  Updates
which were strictly content releases happened every day that there was
new content, which I think was pretty much every day.  (The procedure
was the same, I think; the whole site was in CVS.)

Changes to the database were a little trickier.  I think there wasn't a
set procedure, but good sense was used.  If a change to the DB happened
which had very little chance of affecting something else, such as adding
a brand-new table or column, it would be done during a non-busy time of
the day.  If it was something more extensive, it was generally done on
the evenings or on the weekend.  In all cases, the changes were made
before the code which needed them was put in production. ;-)

There were four (sets of) servers.  Dev, where all the developers
worked.  We all had our own instances of Apache running on different
ports.  Alpha, which the QA guy used to take a preliminary look at new
code.  I believe there were about three instances of Apache set up on
that one, all owned by the QA person.  Alpha and dev shared a dev
database.  Accessible from outside the firewall was beta, intended to be
as much like prod as possible (only one Apache daemon on it, used the
production DB, etc.), and, of course, prod, which they had set up as a
cluster.

This worked well for them.  I won't say it worked perfectly; it didn't.
But we learned from our mistakes as we went along and got better, and we
very seldom made the same mistake twice. :-) YMMV.

-- 
Bill Jonas    *    bill@billjonas.com    *    http://www.billjonas.com/

Developer/SysAdmin for hire!   See http://www.billjonas.com/resume.html

Attachment: pgpLOMH8ARg9k.pgp
Description: PGP signature