Re: [PLUG] dev vs production environments

[Kevin McAllister <kevin@mcallister.ws>]

Hi JP

This is a thing my company has struggled with for… a long time.  I think test should match production as closely as possible for many reasons.

1. Being able to detect as many bugs as possible before release.
2. Being able to test your rollout/release procedure (because everyone knows that never causes a problem at 2 AM)
3. Attempting to eliminate the "Well … it works on my machine" response to the bug report.
4. Being able to quickly reproduce the bugs your customers do find without impacting production and create and deploy a hotfix.  (due to 2 and 3)

Another item I find crucial and we are working toward.  Rapid creation of test/production environments.  We've found that no matter what you do your test machine will encounter an issue and the developer/sysadmin will fix it on the test machine (Oh yeah we need to yum install unobtainiumd-devel to make this work) and then not note that in the relevant rollout instructions, not due to malice but due to he wasn't working on installing unobtainiumd or testing the code that uses it, rather he was testing something else and the software wouldn't start without that package and he just forgot.  When you can rapidly redeploy the lab/test (push a button get some coffee and then come test again) you can catch that dependency when you ask the question, "Why the hell is nothing working after the rebuild?"  Plus if you lose a production machine, or need to add new production machines you can know the machine you install will have all the right stuff™

We've got a ways to go here but the way we are combatting that is through embracing Chef (http://www.opscode.com/chef/).  That way when you need to build a new machine you bootstrap it, and run chef. Then use it, a misconfigured box is not something you go repair, it's a bug, you fix the code and redeploy.

If you're interested in that checkout Mat Schaffer's screencasts about chef: http://devops.mashion.net/

We started small, by automating our asterisk box installs using chef-solo, and are now moving toward using chef-server to help manage our many servers.

Oh and I've found that for doing chef development, virtualization is crucial so you can iterate over and over, and then return to a base install snapshot and test again.

Thanks to Kyle Burton for introducing me to it a few years back at a PLUG talk 
https://github.com/kyleburton/presenting-chef-solo

- Kevin

On Jan 25, 2012, at 5:22 PM, JP Vossen wrote:

> Arguably at least semi-OT, but it's all about Linux & Java & stuff, so...  (Warning, I'm *not* a Java fan.)
> 
> I'm having a problem at work convincing some developers that the dev and testing environments should match production as close as possible. Since this is all Linux stuff, and we have machines and virtualization, this seems like it should be a no-brainer to me.  In fact, this is so blindingly obvious that I'm having trouble making a stronger argument for this than "because it's so blindingly obvious."
> 
> Also, much of the dev work is Java, which is "write once, run anywhere," right?  Ummm.  No.  I'll agree with "write once, debug everywhere" but that's about it.  *Running* a Java app in a production-quality way is a *tad* different on Linux and (the horror) Windows...  Even if the "Java code" part works, all the rest of the surrounding infrastructure and environment (init script, ulimit, user perms, SELinux, ports < 1024 (like 443)) has to work and be in sync too.  That isn't gonna happen on Windows.  (Hell, it's not even happening on Linux, which is part of my point.)  And don't get me started on the JBoss part of it...
> 
> So...  Am I wrong?  Or is this as blindingly obvious as I think it is? And if so, can anyone make a better argument, or point me at some "best practices" and/or horror stories I can beat people with?  (I'm Googling badly as I can't find anything good.  OK, besides just about everything at thedailywtf.com :)
> 
> FWIW, we're trying to be "Agile" though we just started and have a *long* way to go...
> 
> Thanks,
> JP
> ----------------------------|:::======|-------------------------------
> JP Vossen, CISSP            |:::======|      http://bashcookbook.com/
> My Account, My Opinions     |=========|      http://www.jpsdomain.org/
> ----------------------------|=========|-------------------------------
> "Microsoft Tax" = the additional hardware & yearly fees for the add-on
> software required to protect Windows from its own poorly designed and
> implemented self, while the overhead incidentally flattens Moore's Law.
> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug