Re: [PLUG] Ansible for updating in the field

[brent timothy saner <brent.saner@gmail.com>]

On 06/25/2016 02:39 PM, JP Vossen wrote:
> Goal:
> I have 500+ CentOS-5 and 6 servers to manage in the field and I need to
> find the best Ansible way to maintain and update about 120 custom
> scripts and configs, but also a few binary library files.

ansible's a great solution to this problem!

> 
> Background:
> I have SSH connectivity as a non-root user who does have `sudo` access.

https://docs.ansible.com/ansible/become.html

> By my design, all my 120+ files can be arbitrarily overwritten, any
> special snowflakes are handled via *.local files that are per-device and
> only ever touched manually.  (The nodes are mostly cattle, not pets, but
> there are some "special" ones.)

this can be handled by ansible logic. there's a couple different ways to
do this- via jinja2, host_variables, or the least-elegant "when:
ansible.host != 'somehostname'".

> I already have a `setup.sh` that can install missing RPMs, create
> symlinks, fix permissions/owner, etc.  It is idempotent and designed to
> be run any time just for the hell of it to make sure stuff is correct.
> There were Good Reasons to start that way instead of Ansible from
> scratch, but now I'm torn.  The tier1/tier2 folks working on these
> devices will not have Ansible access, nor do they have strong Linux
> skills, so having a local "just make it right" script is very handy.
> OTOH, it duplicates the hell out of Ansible's domain.
> 

who says you can't have both? granted, it means any updated
checks/tasks/etc. have to be now added to two places instead of one, etc.
alternatively you could just use the copy module to copy over the
relevant task/playbook/what have you, have ansible installed on the
local machine, and run it via local_action or ansible host as "localhost".

there are more elegant ways, however:
https://www.stavros.io/posts/automated-large-scale-deployments-ansibles-pull-mo/

i'd highly recommend that ^ plus ansible-izing your setup.sh.

(SNIP)

> Problem:
> I can think of two basic ways to handle on-going updates via Ansible:
> 
> 1) Use Ansible rsync to upload the tree, with a handler of the existing
> `setup.sh` script.  This is simple, but blunt force.  It's also probably
> fast to get working since it's just Ansiblizing  what we do now.

i'd recommend this as an *intermediate* measure; meanwhile, work on
fully ansible-izing e'rythang. all the things. as much as you can.
because then you can both do it from a central jumpbox with sudo access
AND implement the "pull-mode" i share above.

> 2) List every single file, where it goes, owner, perms, etc.  This is
> tedious, but surgical and would certainly involve double-entry with the
> `setup.sh` script, until/unless I can do away with that.
> 

i'd say this is the *recommended* way, but makes for some huge task
files/playbooks. ansible does support things like recursive copy, but i
personally do it file by file so i have a firm control over things like
ownership and permissions at the time of copying.

> It feels like #2 is more "correct" but #1 is faster to develop and a
> better fit with current practice, though I control practice and can
> slowly change or evolve it as needed.
> Am I missing any better ways?  Am I over-thinking this?

like i said above- do both! :) use one as the intermediary to the other.

(you can even keep your setup.sh and make it a wrapper to do a
client-side ansible pull, if you'd like, to help T1/T2 with the
transition habits.)


as a last note, i'd highly recommend custom-brewing a
db-and-script-interface-driven inventory system.
http://jpmens.net/2013/06/18/adapting-inventory-for-ansible/ has a nice
introduction to that, but you can go hog wild. create virtual groups,
return variables based on the number of hosts in the run, etc. etc. etc.
(plus it's way easier to deal with 100+ rows in a sqlite3 DB than it is
for 100+ lines in an ansible hosts file, plus 100+ host_vars files, etc.)

feel free to grab me (r00t^2 , if you're in #plug) on IRC if you need
any further help. i was worried ansible would feel too shiny/"devops-y"
for me, but after the first week and converting our provisioning over to
it, i can safely say i highly recommend it to oldhats and shiny devopsy
guys alike.

Attachment: signature.asc
Description: OpenPGP digital signature

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug