JP Vossen on 27 Jun 2016 13:15:37 -0700

[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]

Re: [PLUG] Ansible for updating in the field

Thanks Brent! It sounds like I'm not missing anything obvious, and I like your path forward.

More in-line.

On 06/25/2016 02:54 PM, brent timothy saner wrote:
On 06/25/2016 02:39 PM, JP Vossen wrote:
I have 500+ CentOS-5 and 6 servers to manage in the field and I need to
find the best Ansible way to maintain and update about 120 custom
scripts and configs, but also a few binary library files.

ansible's a great solution to this problem!


I have SSH connectivity as a non-root user who does have `sudo` access.

Yup, I do that all the time, that's why I (unclearly) mentioned I'm good to go there.

By my design, all my 120+ files can be arbitrarily overwritten, any
special snowflakes are handled via *.local files that are per-device and
only ever touched manually.  (The nodes are mostly cattle, not pets, but
there are some "special" ones.)

this can be handled by ansible logic. there's a couple different ways to
do this- via jinja2, host_variables, or the least-elegant "when: != 'somehostname'".

Yes, but that also requires that the tier1/tier2 guy who did the work tells me about it, so I don't accidentally nuke it later. That will not happen, so the policy is "only ever touch one-offs manually, on the device." Yes , that sucks, but there it is.

I might have another project that might help that later, but that's months out at least, so for now...

I already have a `` that can install missing RPMs, create
symlinks, fix permissions/owner, etc.  It is idempotent and designed to
be run any time just for the hell of it to make sure stuff is correct.
There were Good Reasons to start that way instead of Ansible from
scratch, but now I'm torn.  The tier1/tier2 folks working on these
devices will not have Ansible access, nor do they have strong Linux
skills, so having a local "just make it right" script is very handy.
OTOH, it duplicates the hell out of Ansible's domain.

who says you can't have both? granted, it means any updated
checks/tasks/etc. have to be now added to two places instead of one, etc.

That's why, double-entry is BAD.

alternatively you could just use the copy module to copy over the
relevant task/playbook/what have you, have ansible installed on the
local machine, and run it via local_action or ansible host as "localhost".

I could. As below I was thinking was a better fit though.

there are more elegant ways, however:

Oh that's awesome! I used to with with Stavros at $WORK-- and I've talked about Ansible with him a little, but never about pull that I recall. I do follow his blog.

I really want a push though, my question is just about the best way to organize it in Ansible idiom.

I can think of two basic ways to handle on-going updates via Ansible:

1) Use Ansible rsync to upload the tree, with a handler of the existing
`` script.  This is simple, but blunt force.  It's also probably
fast to get working since it's just Ansiblizing  what we do now.

i'd recommend this as an *intermediate* measure; meanwhile, work on
fully ansible-izing e'rythang. all the things. as much as you can.
because then you can both do it from a central jumpbox with sudo access
AND implement the "pull-mode" i share above.

Yeah, Ansible on the jump hosts was always the plan, I just left that out while trying (and probably failing) to be brief.

2) List every single file, where it goes, owner, perms, etc.  This is
tedious, but surgical and would certainly involve double-entry with the
`` script, until/unless I can do away with that.

i'd say this is the *recommended* way, but makes for some huge task
files/playbooks. ansible does support things like recursive copy, but i
personally do it file by file so i have a firm control over things like
ownership and permissions at the time of copying.

It feels like #2 is more "correct" but #1 is faster to develop and a
better fit with current practice, though I control practice and can
slowly change or evolve it as needed.
Am I missing any better ways?  Am I over-thinking this?

like i said above- do both! :) use one as the intermediary to the other.

I didn't think of that, but I like it! I'm pretty sure that's going to be the plan going forward.

(you can even keep your and make it a wrapper to do a
client-side ansible pull, if you'd like, to help T1/T2 with the
transition habits.)

Now THAT's an interesting idea!  I'd never had thought of that...

as a last note, i'd highly recommend custom-brewing a
db-and-script-interface-driven inventory system. has a nice

Another awesome blog I follow, though he's mostly down the IoT rabbit hole at the moment. I've had the CMDB discussion many times at $WORK, but I've not yet gotten traction for any solution. The problems are legion but mostly boil down to "big company...issues" If I thought I could get approval and budget for Tower I'd probably do it, but I know I can't. I will not profane the list (and I do mean profane) with some of the "solutions" that have been tossed around. You don't want to know. You really, REALLY don't.

introduction to that, but you can go hog wild. create virtual groups,
return variables based on the number of hosts in the run, etc. etc. etc.
(plus it's way easier to deal with 100+ rows in a sqlite3 DB than it is
for 100+ lines in an ansible hosts file, plus 100+ host_vars files, etc.)

Yes, even for the 16-18 devices I manage at home the inventory got out of control. I have a spreadsheet matrix I use as a source for a Perl script to write the inventory file. Everything else is YAML/JSON, why the HELL isn't this too?!? One, count them ONE object for each node, with attributes like freakin groups!!! Shesh...

feel free to grab me (r00t^2 , if you're in #plug) on IRC if you need
any further help. i was worried ansible would feel too shiny/"devops-y"
for me, but after the first week and converting our provisioning over to
it, i can safely say i highly recommend it to oldhats and shiny devopsy
guys alike.

Not only is Ansible Just Awesome all by itself, but it's actually exactly how our v1 system worked (upload a script, run it, capture the output, download it, present it). It's just 800 million times better than that system. Raw, manual SSH is the v2 "system". The v3 system is what we're talking about. :-) So after about a decade of v1, that's how I think anyway. It took me *months* to figure out why Ansible felt so good and fit how my mind works so well... Boy that was an "ah HA" moment.

Thanks again,
--  -------------------------------------------------------------------
JP Vossen, CISSP | |
Philadelphia Linux Users Group         --
Announcements -
General Discussion  --