Evan Payne

Ansible for the Solo Developer - Part 1

Let me preempt this whole ansible article by saying that this process will work just as well for teams.  Its only that the one-man/woman-band developer needs to learn this now.  She hasn't got a shoulder to lean on, you know.

In my daily development routines, I would be lost without my own Ansible/Vagrant powered setup. It takes the guess work out of server compatability, removes the need for forcing MAMP into doing things it shouldn't, and allows me to know that my sites are safely backedup, recoverable, and version controlled. Yes, this is all doable without Ansible, even without Vagrant, but its just so much easier!

Here's what I have in mind for this article:

Part One:

  1. What is Ansible
  2. Why not Bash?
  3. Setting Things Up

Part Two:

  1. Git
  2. Wordpress Itself
  3. Themes and Plugins
  4. Command Lines

In this first part, we'll start from scratch and get all the way to a running virtual machine, learning about Vagrant and Ansible in the process. (Note: Sorry Windows users, this tutorial is based on the assumption that you're using a Mac. That said, its not too hard to tweak things to get it working, and the principles are the same.)

What is Ansible?

Ansible is a simple utility that runs a set of scripts that do things on a remote machine. I hope no one gets mad at me for oversimplifying, but that's the core of it.

Ansible's main command runs a playbook, which in turn runs through a series of tasks contained in roles.  All this is written in YAML, a very simple indentation-based language, and it is pretty easy to get started with.  Here's an example task.yml file from a role:

- npm: name=gulp global=yes

This uses the NPM Module, built into Ansible, to install Gulp globally. So, each task is made up of a Module and its options, and that's it!  String together a bunch of tasks and you have a playbook that you can run on a host or a series of hosts and the same ssh command will be executed on each host.

What this means for you as a developer is that you can write a playbook that does all the tedious, repetitive parts of setting up a new site automatically.  Just type a command, and a few minutes later, you're ready to start developing.

Why not Bash?

Scripting is nothing new, and you could certainly do the same thing in Bash or Ruby or even Node, and sometimes that's the right thing to do.  But where Ansible stands out is the core concept of idempotency.

idempotent - denoting an element of a set which is unchanged in value when multiplied or otherwise operated on by itself.

That definition doesn't fit...  Stupid dictionary.  No, in our case idempotentcy means that the task won't execute if it doesn't need to.  Ansible is very aware of the penchant scripts have for messing things up on run two or three or forty-two, and so its built in to avoid making unneccessary changes.

So, to be clear, you can and should be building your playbooks to be run multiple times without re-installing things, without re-downloading or overwriting previous things.

And Ansible makes that easy.

Setting Things Up

Ok, what we want to accomplish by the end is to have a working Wordpress site living on a Virtualbox install with important files under version control. We'll need to follow these steps:

  1. Install requirements
  2. Setup Vagrantfile
  3. Setup variables and provisioning playbook
  4. Scaffold our playbook's roles and tasks
  5. Wire everything up
  6. Test!

Install Requirements

Ok, a few tedious things to follow.  You'll need to install:

Bonus points: If you use Homebrew, you can install http://caskroom.io/ and install things like Vagrant and VirtualBox directly through the command line!  But, the normal way is fine too.

I would recommend using Pip to install Ansible, but that's just because it seemed the simplest for me on my Mac. Do it however you're comfortable.

Create a folder for your test site, it can be anywhere.  I prefer to make a folder called Code in my /Users/USERNAME/ folder, but it really doesn't matter.

Then setup the basic empty files and structure for the project.  This is the bare minimum to get our setup working:

your_box_name/
== ansible/
==== group_vars/
====== all.yml
==== host_vars/
====== default.yml
==== hosts.yml
==== playbook.yml
== public/
== Vagrantfile

Bear in mind that Vagrantfile is a file, not a folder.  Most people also prefer to leave all.yml, default.yml and hosts.yml without extensions for ease of readability, so feel free to name them simply all, default, hosts.

Setup Vagrantfile

Ok, so to start with, here's the Vagrantfile I recommend using:

    # -*- mode: ruby -*-
    # vi: set ft=ruby :

    # Specify Vagrant provider as Virtualbox
    ENV['VAGRANT_DEFAULT_PROVIDER'] = 'virtualbox'

    require 'yaml'

    ANSIBLE_PATH = 'ansible'

    ansible_hosts = File.join(ANSIBLE_PATH, 'hosts')

    Vagrant.require_version '>= 1.7.4'

    Vagrant.configure('2') do |config|
      config.vm.box = 'scotch/box'
      config.ssh.forward_agent = true

      if !Vagrant.has_plugin? 'vagrant-hostsupdater'
        puts 'vagrant-hostsupdater missing, please install the plugin:'
        puts 'vagrant plugin install vagrant-hostsupdater'
      end

      require 'ipaddr'
      hostsfile = "/etc/hosts"

      ip = File.read(hostsfile).scan(/192\.168\.42\.\d{1,3}/).sort_by! {|ip| ip.split('.').map{ |octet| octet.to_i} }.last or ip = "192.168.42.1"
      ips = [ip]
      ips << IPAddr.new(ips.last).succ.to_s
      latest_ip = ips.last

      hostname, *aliases = File.read(ansible_hosts).scan(/([\da-z\.-]+\.[a-z\.]{2,6})/).flatten

      config.vm.network :private_network, ip: latest_ip
      config.hostsupdater.remove_on_suspend = true
      config.vm.hostname = hostname
      config.hostsupdater.aliases = aliases

      config.vm.synced_folder ".", "/var/www", :mount_options => ["dmode=777", "fmode=666"]

      config.vm.provision :ansible do |ansible|
        ansible.playbook = File.join(ANSIBLE_PATH, "playbook.yml")
      end

      config.vm.provider 'virtualbox' do |vb|
        # Give VM access to all cpu cores on the host
        cpus = case RbConfig::CONFIG['host_os']
          when /darwin/ then `sysctl -n hw.ncpu`.to_i
          when /linux/ then `nproc`.to_i
          else 2
        end

        # Customize memory in MB
        vb.customize ['modifyvm', :id, '--memory', 1024]
        vb.customize ['modifyvm', :id, '--cpus', cpus]

        # Fix for slow external network connections
        vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']
        vb.customize ['modifyvm', :id, '--natdnsproxy1', 'on']
      end

    end

Well, that seems like an awful lot, but most of it is standard fare for any vagrant setup.  Let's break it down:

# -*- mode: ruby -*-
# vi: set ft=ruby :

# Specify Vagrant provider as Virtualbox
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'virtualbox'

require 'yaml'

ANSIBLE_PATH = 'ansible'

ansible_hosts = File.join(ANSIBLE_PATH, 'hosts')

Vagrant.require_version ' &gt;= 1.7.4'

Here we're merely setting up the basics.  Its a ruby environment, we'll be parsing a yaml file at some stage.  The ANSIBLE_PATH is set in relation to the directory we're in, where the Vagrantfile lives.  (Bear in mind, this is usually a level or two below your public directory and should not be seeing the light of day).

We're also letting it know that our ansible hosts file will be directly inside the ansible folder.  Oh, and we'll set a required version of Vagrant, just so things don't break with an older one.

Vagrant.configure('2') do |config|
  config.vm.box = 'scotch/box'
  config.ssh.forward_agent = true

This is the most important part, and its where we tell Vagrant what base box we want to use.  There are many to choose from, and to some extent it depends on what you're most comfortable with as a server, or what your actual live server will be running.

I tend to use Ubuntu 14.04 because its as basic as they come, and half the tutorials out there use it as a base.  You'll see however that in our code above, we're calling 'scotch/box'.  This is because the awesome team at Scotch.io has released their own base box image, Scotch Box, with a lot of things I use on a daily basis pre-installed.

Eventually, you might want to take the time to build your own image with Docker, and use that instead, but in the meantime, we can use this one with only a few tweaks to make it work for us.

  if !Vagrant.has_plugin? 'vagrant-hostsupdater'
    puts 'vagrant-hostsupdater missing, please install the plugin:'
    puts 'vagrant plugin install vagrant-hostsupdater'
  end

  require 'ipaddr'
  hostsfile = "/etc/hosts"

  ip = File.read(hostsfile).scan(/192\.168\.42\.\d{1,3}/).sort_by! {|ip| ip.split('.').map{ |octet| octet.to_i} }.last or ip = "192.168.42.1"
  ips = [ip]
  ips &lt;&lt; IPAddr.new(ips.last).succ.to_s
  latest_ip = ips.last

  hostname, *aliases = File.read(ansible_hosts).scan(/([\da-z\.-]+\.[a-z\.]{2,6})/).flatten

  config.vm.network :private_network, ip: latest_ip
  config.hostsupdater.remove_on_suspend = true
  config.vm.hostname = hostname
  config.hostsupdater.aliases = aliases

Right, a big chunk, but not too hard to read.  First, we have a block that checks to make sure the vagrant plugin hostupdater is installed, because we need that for the next section.

Then, we check our own machine's hostfile (Mac only for now) to find the latest ip address used.  This is in case you have a box running already, or work with your hostfile frequently.  We don't want overlap.

Next, we read through our Ansible hosts file and for each hostname it finds, we add a rule to our local hostsfile that points to our chosen IP address.

  config.vm.synced_folder ".", "/var/www", :mount_options =&gt; ["dmode=777", "fmode=666"]

I've never messed with this line, but I know that it syncs the folder we're in with the folder living inside the box at /var/www/, meaning anything inside our current folder is also inside the box.

To be extra clear, the sync is one way, the files live on our local machine, but the box is convinced they belong to it, so it can work on them at will.

  config.vm.provision :ansible do |ansible|
    ansible.playbook = File.join(ANSIBLE_PATH, "playbook.yml")
  end

This is where we integrate Ansible.  If these lines weren't present, we'd still have a running virtual machine when we finished, and we could still use the public folder to run php and the hosts in our ansible hosts file would point to that public folder.  But we'd have to manually install anything else we needed, including wordpress.  That's boring!  So instead we tell Vagrant here to run the ansible playbook file inside our ansible folder name playbook.yml, whenever the box is first loaded, or when we type vagrant provision.

  config.vm.provider 'virtualbox' do |vb|
    # Give VM access to all cpu cores on the host
    cpus = case RbConfig::CONFIG['host_os']
      when /darwin/ then `sysctl -n hw.ncpu`.to_i
      when /linux/ then `nproc`.to_i
      else 2
    end

    # Customize memory in MB
    vb.customize ['modifyvm', :id, '--memory', 1024]
    vb.customize ['modifyvm', :id, '--cpus', cpus]

    # Fix for slow external network connections
    vb.customize ['modifyvm', :id, '--natdnshostresolver1', 'on']
    vb.customize ['modifyvm', :id, '--natdnsproxy1', 'on']
  end

end

Ok, I really don't know what all this does, but it looks like it helps with memory management and keeping things moving fast, so I leave it in.

And we're ready!  Well, not quite.  Lets keep going and setup what we need to have our box ready to host wordpress sites as well as simple php ones.

Setup variables and provisioning playbook

The first Ansible YAML file we're going to write is the one that lives at ./ansible/group_vars/all and holds default variables for any site we install, and indeed for the box as a whole.  We'll start simply, by giving ourself easy access to the root folder (where the Vagrantfile lives), and to our default public directory. Later on, we can override the public directory on a per site basis.

./ansible/group_vars/all

---
www_root: /var/www
public_root: "{{ www_root }}/public"

Don't forget those three dashes at the top.  Without them, Ansible won't start parsing the file, and will light on fire instead... Well, not really, but leave it in to be safe.

You can see that we're using our first variable right after we declare it.  Ansible uses the format of {{variable}} to spit out variables.  This can be used in different ways, as we'll see later, but it means that we're actually setting our public_root variable to "/var/www/public".  The quotes are important to keep in place whenever you put a variable immediately after a colon.  So, var_name: {{ my_var }} would give you an error whilevar_name: "{{ my_var }}" would not.

We also need to tell the box what hosts to use (remember in our Vagrantfile we refered to an Ansible hosts file).  For now, we're just giving our box a generic dev url:

./ansible/hosts

[default]
mybox.dev

Great.  We aren't doing this yet, but if we wanted to override the variables in group_vars for a specific host, we would add those variables again into the host_vars folder in a file with the same name as the host the variables apply to. We'll see this in action in Part 2.

Next we'll create the playbook that will provision our server.  If we weren't using Scotch Box, there would be a lot more to do, but we still have some basic housekeeping.  Here goes:

./ansible/playbook.yml

---
- name: "Setup vBox"
  hosts: default
  sudo: yes
  remote_user: vagrant
  vars_files:
    - "group_vars/all"

  roles:
    - { role: common }
    - { role: public-test }

So, here we have the basics of a playbook.  The whole playbook begins with a single task, named "Setup vBox".  We're running this on any hosts under the [default] group in our hosts file, which is the same as saying the server that url points to, which is to say, our dev box.  If we wanted to provision, we would use another group here, and a real ip address instead.  But that's for another lesson.

We're working as sudo, but using the remote user 'vagrant' so that we don't always have to.  This makes sure that any files we're creating are owned by the vagrant user, which is better for security than root, I'd guess.

Then, we're bringing in the variables we set in our group_vars/all file.  Now we can use our www_root and public_root when we need them.

Finally, we get to the actual tasks, bundled into roles for ease of sorting.  If we really wanted to, we could just continue running tasks in the playbook, but roles help us keep organised, and offer some other benefits besides.

Again, don't forget the '---' on the first line so ansible knows to parse the file.

Let's move on and explore the two roles we're using in this playbook:

Scaffold our playbook's roles and tasks

In your ansible directory, create the following folders and files:

./ansible/
== roles/
==== common/
====== defaults/
======== main.yml
====== tasks/
======== main.yml
==== public-test/
====== tasks/
======== main.yml
====== templates/
======== index.j2

So, we're creating two roles, common and public-test.  Every role in ansible should contain at least a tasks folder with a main.yml inside, where the playbook will start to run.  You can call other tasks from inside, which we'll do later when setting wordpress up, but for now, you're really only creating four files, and they're pretty straighforward.

./ansible/roles/common/tasks/main.yml

---
- name: Validate Ansible version
  assert:
    that:
    - "{{ ansible_version is defined }}"
    - "{{ ansible_version.full | version_compare(minimum_ansible_version, '&gt;=') }}"
    msg: "Your Ansible version is too old. We require at least {{ minimum_ansible_version }}. Your version is {{ ansible_version.full | default('&lt; 1.6') }}"

- name: Update Apt
  apt: update_cache=yes

- name: Checking essentials
  apt: name="{{ item }}" state=present
    with_items:
    - python-software-properties
    - python-pycurl
    - build-essential
    - python-mysqldb
    - python-pip

- name: Install Bitbucket-CLI
  pip: name=bitbucket-cli

- name: Add SSH pub key
  authorized_key: user="vagrant" key="{{ lookup('file', '~/.ssh/id_rsa.pub') }}"

We're adding in five tasks, each named for ease of reading, and for following along once ansible is running.  We're making sure the right version of ansible is running, Updating Apt, Making sure essential things are installed, Installing a bitbucket command line utility, and passing our local ssh keys into the box.

I've left out installing gulp and bower because npm runs really slowly on the box, like five times as slow.  Its far better to do any builds on your local machine and command line.

The playbook will run through these tasks one at a time, and for each task, will return either a fail or success, and if success, whether it changed something or not.  The idea is, you should be able to trust that installing bitbucket-cli won't happen if its already installed.  Idempotency, remember?

There's one thing we forgot!  Did you notice the variables in our first task?  ansible_version is something available to any playbook, but we need to define minimum_ansible_version.  In order to keep things a little cleaner, we'll put into the role's defaults folder, like so:

./ansible/roles/common/defaults/main.yml

minimum_ansible_version: 1.9.2

Our three dashes aren't needed here because this isn't a task to be run.  Also in the common role's tasks, we used a bit of trickery to find out information from our local machine (the one running the playbook) {{ lookup('file', '~/.ssh/id_rsa.pub') }} will copy the contents of the file on our local machine at ~/.ssh/id_rsa.pub, wich will come in handy.  For more information, read about the authorized_key module.

Ok, next, we'll work on this file:

./ansible/roles/public-test/tasks/main.yml

This role will run a single task to create an index.php in our public folder, allowing us to verify that the install worked, and to use some variables from our current playbook.

---
- name: Create index.php file
  template: src="index.j2"
            dest="{{ public_root }}/index.php"

This will use the template Module from Ansible.  It uses the templating language of Jinja2 to replace any variables with content available to the playbook, and write the filled-in "src" template to the "dest" file.

In our case, we want to write the contents of the "src" file (which resolves to the current role's templates directory and the index.j2 file within), to our public root (set to /var/www/public from within our group_vars/all file).

./ansible/roles/public-test/templates/index.j2

<h1>{{inventory_hostname}}</h1>
<h2>{{ansible_all_ipv4_addresses[1]}}</h2>
<?php phpinfo(); ?>

inventory_hostname is another variable available to ansible playbooks, that takes the value from hosts (or what hosts we're limited to), in our case, "default".  ansible_all_ip4_addresses[1] spits out the second item in another available variable, which should match our vbox.  We'll also spit out phpinfo to make sure php is working ok.

And geez, we're done!  Let's fire things up and see what happens!

Vagrant Up

vagrant up

Go for it.  In the same folder as the Vagrantfile, run vagrant up. If its the first time you've run it, it will automatically provision the box directly aftwerwards. If you ever need to provision the box again (adding new tasks to setup or somesuch), you can do so by running vagrant provision, or vagrant up --provision.

Now sit, back, enjoy the stream of stuff, and when you have control of your command line back, we can see if it worked. In your browser, visit mybox.dev. You should see the box default, the ip address, and a phpinfo dump.

Success!! Well done for setting it all up. If you had trouble, ask away in the comments below.

Next time, in Part 2, we'll add new roles to integrate Git into our setup, and to setup Wordpress as well. Thanks for reading!