| JP Vossen on 27 Jun 2016 13:15:37 -0700 |
[Date Prev] [Date Next] [Thread Prev] [Thread Next] [Date Index] [Thread Index]
| Re: [PLUG] Ansible for updating in the field |
More in-line. On 06/25/2016 02:54 PM, brent timothy saner wrote:
On 06/25/2016 02:39 PM, JP Vossen wrote:Goal: I have 500+ CentOS-5 and 6 servers to manage in the field and I need to find the best Ansible way to maintain and update about 120 custom scripts and configs, but also a few binary library files.ansible's a great solution to this problem!
Yes!
Background: I have SSH connectivity as a non-root user who does have `sudo` access.https://docs.ansible.com/ansible/become.html
Yup, I do that all the time, that's why I (unclearly) mentioned I'm good to go there.
By my design, all my 120+ files can be arbitrarily overwritten, any special snowflakes are handled via *.local files that are per-device and only ever touched manually. (The nodes are mostly cattle, not pets, but there are some "special" ones.)this can be handled by ansible logic. there's a couple different ways to do this- via jinja2, host_variables, or the least-elegant "when: ansible.host != 'somehostname'".
Yes, but that also requires that the tier1/tier2 guy who did the work tells me about it, so I don't accidentally nuke it later. That will not happen, so the policy is "only ever touch one-offs manually, on the device." Yes , that sucks, but there it is.
I might have another project that might help that later, but that's months out at least, so for now...
I already have a `setup.sh` that can install missing RPMs, create symlinks, fix permissions/owner, etc. It is idempotent and designed to be run any time just for the hell of it to make sure stuff is correct. There were Good Reasons to start that way instead of Ansible from scratch, but now I'm torn. The tier1/tier2 folks working on these devices will not have Ansible access, nor do they have strong Linux skills, so having a local "just make it right" script is very handy. OTOH, it duplicates the hell out of Ansible's domain.who says you can't have both? granted, it means any updated checks/tasks/etc. have to be now added to two places instead of one, etc.
That's why, double-entry is BAD.
alternatively you could just use the copy module to copy over the relevant task/playbook/what have you, have ansible installed on the local machine, and run it via local_action or ansible host as "localhost".
I could. As below I was thinking http://docs.ansible.com/synchronize_module.html was a better fit though.
there are more elegant ways, however: https://www.stavros.io/posts/automated-large-scale-deployments-ansibles-pull-mo/
Oh that's awesome! I used to with with Stavros at $WORK-- and I've talked about Ansible with him a little, but never about pull that I recall. I do follow his blog.
I really want a push though, my question is just about the best way to organize it in Ansible idiom.
Problem: I can think of two basic ways to handle on-going updates via Ansible: 1) Use Ansible rsync to upload the tree, with a handler of the existing `setup.sh` script. This is simple, but blunt force. It's also probably fast to get working since it's just Ansiblizing what we do now.i'd recommend this as an *intermediate* measure; meanwhile, work on fully ansible-izing e'rythang. all the things. as much as you can. because then you can both do it from a central jumpbox with sudo access AND implement the "pull-mode" i share above.
Yeah, Ansible on the jump hosts was always the plan, I just left that out while trying (and probably failing) to be brief.
2) List every single file, where it goes, owner, perms, etc. This is tedious, but surgical and would certainly involve double-entry with the `setup.sh` script, until/unless I can do away with that.i'd say this is the *recommended* way, but makes for some huge task files/playbooks. ansible does support things like recursive copy, but i personally do it file by file so i have a firm control over things like ownership and permissions at the time of copying.It feels like #2 is more "correct" but #1 is faster to develop and a better fit with current practice, though I control practice and can slowly change or evolve it as needed. Am I missing any better ways? Am I over-thinking this?like i said above- do both! :) use one as the intermediary to the other.
I didn't think of that, but I like it! I'm pretty sure that's going to be the plan going forward.
(you can even keep your setup.sh and make it a wrapper to do a client-side ansible pull, if you'd like, to help T1/T2 with the transition habits.)
Now THAT's an interesting idea! I'd never had thought of that...
as a last note, i'd highly recommend custom-brewing a db-and-script-interface-driven inventory system. http://jpmens.net/2013/06/18/adapting-inventory-for-ansible/ has a nice
Another awesome blog I follow, though he's mostly down the IoT rabbit hole at the moment. I've had the CMDB discussion many times at $WORK, but I've not yet gotten traction for any solution. The problems are legion but mostly boil down to "big company...issues" If I thought I could get approval and budget for Tower I'd probably do it, but I know I can't. I will not profane the list (and I do mean profane) with some of the "solutions" that have been tossed around. You don't want to know. You really, REALLY don't.
introduction to that, but you can go hog wild. create virtual groups, return variables based on the number of hosts in the run, etc. etc. etc. (plus it's way easier to deal with 100+ rows in a sqlite3 DB than it is for 100+ lines in an ansible hosts file, plus 100+ host_vars files, etc.)
Yes, even for the 16-18 devices I manage at home the inventory got out of control. I have a spreadsheet matrix I use as a source for a Perl script to write the inventory file. Everything else is YAML/JSON, why the HELL isn't this too?!? One, count them ONE object for each node, with attributes like freakin groups!!! Shesh...
feel free to grab me (r00t^2 , if you're in #plug) on IRC if you need any further help. i was worried ansible would feel too shiny/"devops-y" for me, but after the first week and converting our provisioning over to it, i can safely say i highly recommend it to oldhats and shiny devopsy guys alike.
Not only is Ansible Just Awesome all by itself, but it's actually exactly how our v1 system worked (upload a script, run it, capture the output, download it, present it). It's just 800 million times better than that system. Raw, manual SSH is the v2 "system". The v3 system is what we're talking about. :-) So after about a decade of v1, that's how I think anyway. It took me *months* to figure out why Ansible felt so good and fit how my mind works so well... Boy that was an "ah HA" moment.
Thanks again, JP -- ------------------------------------------------------------------- JP Vossen, CISSP | http://www.jpsdomain.org/ | http://bashcookbook.com/ ___________________________________________________________________________ Philadelphia Linux Users Group -- http://www.phillylinux.org Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce General Discussion -- http://lists.phillylinux.org/mailman/listinfo/plug