AegirVPS: Hybrid Puppetmaster/Git-cron deployment

Welcome to the first article in our new series on the hows and whys of our AegirVPS services. It is intended to be part tutorial, part request for comment, and all shameless plug for our new AegirVPS services. This time we'll focus on the deployment architecture, which will introduce some of the tools and techniques that we'll cover in more depth in upcoming articles. I feel this is worth highlighting early in this series because it provides almost total control over the VPS(s) to our clients, via Puppet manifests, that, by virtue of being deployed via Git, keep all config under version control. This is a significant step up from just granting root or sudo access.

While I figure that you're at least somewhat familiar with Puppet, Git and Linux systems in general, I'll highlight the features I feel are most relevant to how we're implementing our AegirVPS service, with a bunch of links you can follow if you're curious enough to want to read further.

Tools of the trade

Puppet

First and foremost, we maintain all of Koumbit's infrastructure using Puppet, a powerful configuration management software with it's own declarative domain-specific language (DSL). Using it essentially allows us to maintain our systems by describing the ways they ought to be configured in 'manifests'. We often bundle these into 'modules' to more easily share common settings between system, or even with other organizations (as most are open source).

Git

Since manifests and modules are nothing more than a series of text files, they can easily be version controlled, for which we prefer Git. In this way, along with etckeeper, we maintain a complete history of the configuration of our systems. This allows us (among other things) to easily rollback any change that might cause problems. We follow the recommended best-practices of managing these centrally on a Puppetmaster server, access to which very tightly controlled. As of this writing, I don't even have access.

Redmine

For project management, we use Redmine, and a plugin for automatic management of Git repos associated with projects via gitolite. This gives us browsable repositories tightly integrated with our project management tools, issue tracking system and wiki documentation, with access granted on a per-project/repo basis. These include not only our Puppet manifest and modules, but also almost all the rest of the work we do and code we produce.

The problem

These are all mature systems at Koumbit, that have been in daily use by our team for years. We've certainly been using them for as long as I've been here (about 2 years) to reliably communicate and collaborate in a distributed fashion between members, partners and clients.

Unfortunately, running a centralized Puppetmaster doesn't really lend itself well to the kind of access control we'd need in order to provide infrastructure clients with secure control over their own systems. Being both a client of and the service lead for AegirVPSs, I really wanted to be able both to manage my own systems via Puppet, but also extend that level of control to other clients. However, I would never want to introduce the potential for a client's possible mistake (including my own) to affect any systems outside their specific purview.

The solution

After exploring the idea of multiple Puppetmaster servers, or alternative mechanisms, such as using just Git and cron to run puppet locally, we hit upon the idea of a hybrid system. This was our initial diagram illustrating the idea:

While perhaps not very intuitive, this is meant to show the use of Koumbit's Puppetmaster (Kt) to deploy VMs, along with all the standard stuff: monitoring, security updates, alerts, etc. Then, using a new Puppet module (A), we would deploy AegirVPS service-wide settings (B), and clients-specific manifests (CS) from a client project's Git repo (AVPS), along with adding cronjobs (unlabeled object towards the left) to update said repos and submodules, that would then apply the manifests (B). At least, I think that's how it went... This was obviously more inspiration than a real design document.

In reality, that's pretty much exactly how it works. We create a project in our Redmine for a client's AegirVPS, and invite them to register an account if they don't already have one. Then we both create the VM and define its configuration via Puppet. This then deploys our standard maintenance and monitoring modules, along with the 'aegirvps' module. This last is a very small module that actually fits on a single page:

class aegirvps {
 
  # Generate an SSH key-pair so that root can identify to gitolite
  file {"/root/.ssh/":
    ensure => directory,
    before => Exec['aegirvps_id_rsa'],
  }
  exec {"aegirvps_id_rsa":
    command => "ssh-keygen -t rsa -N '' -C '${client_id}' -f /root/.ssh/aegirvps_id_rsa",
    path    => ["/usr/bin", "/usr/sbin"],
    creates => ["/root/.ssh/aegirvps_id_rsa", "/root/.ssh/aegirvps_id_rsa.pub"],
    before  => File[
      '/root/.ssh/config',
      '/root/.ssh/known_hosts'
    ],
  }
  file {"/root/.ssh/config":
    ensure => file,
    source => 'puppet:///modules/aegirvps/config',
  }
  file {"/root/.ssh/known_hosts":
    ensure => file,
    source => 'puppet:///modules/aegirvps/known_hosts',
  }
 
  # Deployment of client's specific puppet manifests & sub-modules via git from
  # their Redmine project repo
  $client_git_repo = "ssh://redmine/${client_id}.git"
  exec { "git clone --recursive ${client_git_repo} ${client_id}":
    cwd     => "/root/",
    creates => "/root/${client_id}",
    path    => ["/usr/bin", "/usr/sbin"],
    onlyif  => "git ls-remote ${client_git_repo}",
    before  => Cron['aegirvps-update', 'aegirvps-apply'];
  }
  # Deployment of the service-wide aegirvps-clients puppet module & sub-modules
  exec { "git clone --recursive git://git.koumbit.net/aegirvps-clients.git aegirvps-clients":
    cwd     => "/root/",
    creates => "/root/aegirvps-clients",
    path    => ["/usr/bin", "/usr/sbin"],
    before  => Cron['aegirvps-update', 'aegirvps-apply'];
  }
  # Cronjobs to update git repos and apply puppet manifest
  cron { "client-update":
    command => "cd /root/${client_id}/ && git pull > /dev/null && git submodule update --init --recursive > /dev/null",
    user    => root,
    minute  => [0, 30],
  } 
  cron { "aegirvps-update":
    command => "cd /root/aegirvps-clients/ && git pull > /dev/null && git submodule update --init --recursive > /dev/null",
    user    => root,
    minute  => [1, 31],
  }
  cron { "aegirvps-apply":
    command => "puppet apply /root/${client_id}/manifests/site.pp --modulepath=/root/${client_id}/modules:/root:/root/aegirvps-clients/modules > /dev/null",
    user    => root,
    minute  => [2, 32],
  } 
 
}

The first half of this code does nothing but generate an SSH keypair, and drop in a couple files to make it possible to connect to our Redmine-managed Git repos securely. Then we login to the server (manually) to retrieve the public key, and add it as an authorized key for the client's user in Redmine, thus granting them access to pull and push to the repo in their project, and allowing their new AegirVPS server to pull down changes too.

This Puppet module also clones our shared configuration module ( 'aegirvps-clients'), along with its required sub-modules: aegir, drush, apt and common. Finally, it puts in place three cronjobs that run every 30 minutes to:

1. update the shared configuration module;
2. update the client-specific configuration manifests; and
3. apply the client manifest (sites.pp).

From that point, any changes published to our shared modules will automatically get synced. Since Puppet's '--modulepath' switch will stop at the first relevant module it finds, clients can override modules just by including them in their project repo's 'modules/' directory. Furthermore, since it's all called from the client manifests to begin with, this mechanism isn't limited to AegirVPS configuration, but can essentially allow total control over your systems.

Additional tools

We've built a number of additional tools and projects for or around this infrastructure. Chief among them are the Aegir and Drush Puppet modules, that enable installation and configuration of these projects. As a side project, I've been building a Drush extension to integrate Vagrant for local testing and development: Drush Vagrant. It includes a plug-in architecture, called 'blueprints', and we've implemented on specifically for AegirVPSs. It enables you to build a local VM (in Vagrant) that is provisioned using the same manifests and modules as your AegirVPSs. This allows you to make changes in a sandbox, where you can test them thoroughly before pushing them to your project repo, after which it will be pull into your live environment.

Each of these probably deserves an article of their own, and that's pretty much the plan. So keep an eye out for these follow-ups in the coming weeks. Also, I'll be discussing these in some depth at my DrupalCom Munich session: Fearless development with Drush, Vagrant and Aegir.

Cheers,

  Christopher 'ergonlogic' Gervais -- AegirVPS Service Lead
          & the rest of the Koumbit AegirVPS team