Re: [PLUG] AWS Scripting using awscli and bash

["Alan D. Salewski" <salewski@att.net>]

On 2018-03-16 10:32:28, Ronald P Guilmet spake thus:
> You can access the API with JavaScript, Node.js, C/C++, Python, Java.
> That's just off the top of my head. I have the book JP is talking about.
> The author says he uses Bash to prototype. He says he wants to know if it
> works before he uses another language. There is a lot of bash to be
> written, so I'm not sure about the saving time part.

Yeah, it's probably not the the fastest route to any but the most trivial of
AWS tasks. But awscli does lend itself to exploration and ad hoc tasks in
the shell, and (as with shell scripting in general) with practice one's
whip-up-itude adeptness increases.

I find that terraform[0] and it's AWS "provider"[1] to be a great tool for
declaratively manipulating AWS resources (VPCs, EC2 instances, whatever);
for getting AWS resources provisioned quickly, reliably, and at whatever
level of complexity is called for, it's the best tool I've seen for the job.

There's something of a natural boundary between provisioning the network and
EC2 instances with terraform, and then handing off to ansible (or similar)
for host-specific configuration.

With that said, for experimentation and/or small tasks you can get remarkably
far with a combination of awscli, bash, and a Makefile. You'll end up with a
bunch of targets that look something like this:

------------------------------------->8--------------------------------------
    @.PHONY: describe-ec2-instances
    describe-ec2-instances: describe-key-pair describe-security-group
        aws --region $(AWS_REGION) \
            --output text \
            ec2 describe-instances \
            --query 'Reservations[*].Instances[]'\
        '.[[VpcId, InstanceId, InstanceType, State.Name, PublicIpAddress, NetworkInterfaces[*].PrivateIpAddresses[*].PrivateIpAddress][][]]'
-------------------------------------8<--------------------------------------

and this:

------------------------------------->8--------------------------------------
    # "terminates" the EC2 instances
    #
    # XXX: This will just terminate /all/ EC2 instances; it does not have any
    #      smarts to limit its activity. This will stop working as soon as
    #      you have non-throw-away instances in the default VPC.
    #
    @.PHONY: delete-ec2-instances
    delete-ec2-instances:
        aws --region $(AWS_REGION) \
            --output text \
            ec2 describe-instances \
            --query 'Reservations[*].Instances[*].[InstanceId]' \
        | ( \
            declare -a ALL_INSTANCE_IDS=() ;\
            while read -r t_ec2_instance_id; do \
                ALL_INSTANCE_IDS+=( "$${t_ec2_instance_id}" ) ;\
                printf '$@: Terminating EC2 instance: %s\n' "$${t_ec2_instance_id}" 1>&2 ;\
                aws --region $(AWS_REGION) ec2 terminate-instances --instance-ids "$${t_ec2_instance_id}" || exit 1 ;\
            done ;\
            \
            for t_ec2_instance_id in "$${ALL_INSTANCE_IDS[@]}"; do \
                printf '$@: Waiting for EC2 instance "%s" to terminate...' "$${t_ec2_instance_id}" 1>&2 ;\
                aws --region $(AWS_REGION) ec2 wait instance-terminated --instance-ids "$${t_ec2_instance_id}" || { printf '\n' 1>&2; exit 1; } ;\
                printf 'done.\n' 1>&2 ;\
            done ;\
          )
-------------------------------------8<--------------------------------------

I find that I have a better sense of what other tools and APIs are doing
after I've explored a given AWS service via awscli, and I find that it is
handy to have the 'awscli' invocations captured in this way for future
reference and possible reuse in "real" scripts. With so many commands and
just the raw number of available AWS services available, I like to have a
collection of the stuff that I've used directly in the past. The JMESPath,
especially, is useful to have around, as the time spent fiddling with it
initially pays off when I reach for it again.

-Al


[0] https://www.terraform.io/
    https://github.com/hashicorp/terraform

[1] https://www.terraform.io/docs/providers/aws/index.html

[2] For those not familiar with it, you can think of terraform as analogous
    to 'make' for your infrastructure; it understands the dependency
    relationships amongst your resources. Among other nice features, it has
    a 'plan' command that generates and shows you an execution plan (which
    can be saved for future application) so you can determine whether
    resources will be created, destroyed, modified, etc:

        $ terraform plan

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug