Infrastructure Lab: Followup to Version Control

Now, after we’ve refined the host install tools part of the infrastructure, standing up a new server involves:

  1. creating a VM connected to the lab network and recording its MAC address
  2. provisioning the VM’s DHCP IP and DNS name from Bolt
  3. adding the VM’s MAC address to the array of CentOS 7 systems defined in Puppet for the Kickstart server
  4. booting the VM and waiting for the OS and Puppet agent installation to complete
  5. signing the new VM’s Puppet certificate
  6. applying the appopriate Puppet role profile classes to the new VM
  7. waiting for Puppet to apply the VM’s configuration

At my rate of typing and with slightly intelligent use of command history, it takes about 2.5 minutes to get through steps 1-3, including running r10k on the Puppet server and running puppet agent on the Kickstart server. Kickstart takes an additional 7 minutes to start installing packages, and reboots the first time after a total elapsed time of 20 minutes. Those installation times should be faster in most production environments: I haven’t determined if I’m being constrained by disk I/O, network speed, or what.

But now we want to migrate from the basic shared folder Git repository we set up some time back to something a bit more GitHub-like. We’ll set up code.blab.renf.ro as a Gitea server, and also set up an outgoing webhook on Gitea to trigger a run of r10k each time we push changes to any branch of the puppet-control repository. This will let us branch off from the production repository to do new role and profile development, and r10k will automatically create a new Puppet environment to match the new branch. We point a host under development to the new environment with the --environment parameter to puppet agent, and the new settings can be tested out without disrupting the production environment.

Roles and Profiles Structure

To start with, we’ll create a data/nodes/code.blab.renf.ro.yaml file containing:

---
classes:
  - role::gitserver

We’ll also create a site-modules/role/manifests/gitserver.pp file containing:

class role::gitserver {
  include 'profile::gitserver'
}

and a site-modules/profile/manifests/gitserver.pp file containing:

class profile::gitserver {
}

The profile::gitserver class will shortly include a Gitea component module. Since development and testing of this module will likely take some time, it’s best to make a snapshot of the code.blab.renf.ro VM before continuing.

The Gitea Component Module

I made a fork of the kogitoapp/gitea module to update it to current versions of its dependencies, and also to migrate it to the puppet/archive module instead of iwf/remote_file.

After updating its dependencies and converting its remote_file resources to equivalent archive resources, I added the new module and its dependencies to the puppet-control repository’s Puppetfile:

mod 'puppetlabs-inifile', '4.2.0'
mod 'gitea',
  git: 'https://github.com/mikerenfro/puppet-gitea.git',
  branch: 'master'

I edited the profile::gitserver class to include the new gitea class and the firewalld class (since we’ll need to open up port 3000 to acces Gitea’s web interface).

class profile::gitserver {
    class { 'gitea': }
    class { 'firewalld': }
}

In the Hiera data file data/nodes/code.blab.renf.ro.yaml, I added the following settings:

gitea::robots_txt: ''
gitea::repository_root: '/opt/gitrepos'

firewalld::ports:
  'Allow Gitea from the public zone':
    ensure: present
    port: 3000
    protocol: 'tcp'
    zone: public

The robots_txt setting is a null by default, but the current gitea module requires it to be a string. The setting doesn’t appear to be used anywhere, but the data types must match, and I assume this is a bug in the default module settings. The repository root defaults into the /var folder, but I prefer to keep them under /opt. I’d really prefer to keep them under the installation root /opt/gitea as /opt/gitea/repos, but the module resource ordering requires that the repository root folder exists before the installation root folder.

After deploying the changes with r10k on the Puppet server and puppet agent -t on the Gitea server, I was able to access the Gitea web interface, complete the initial configuration, and create an initial administrative account.

Upgrading Gitea

Since the gitea module is a few years old, it installs Gitea 1.8.0 by default. To upgrade Gitea to current version (as of the date of this post), the following changes were made to the Hiera data:

gitea::version: '1.11.5'
gitea::checksum: 'd8d43c13e71596c79b541e85e29defe065b4f70ac5155e6d0212bcfc669e1b9c'

where the checksum value was found by examining https://dl.gitea.io/gitea/1.11.5/gitea-1.11.5-linux-amd64.sha256 . After pushing out the changes with r10k and puppet agent, the Gitea server was running version 1.11.5.

Migrating the Puppet Server Repositories to Gitea

Creating SSH Keys

We’ll be accessing the repositories in the Gitea server over ssh. The root account will need to both push and pull content to the repositories, and the puppet account will need to only pull content. Following instructions similar to GitHub’s “Generating a new SSH key”, we’ll create an identity for root. The root ssh identity could have a passphrase, but this is optional.

Similarly, after switching to the puppet account using sudo -u puppet -s /bin/bash, create an identity with ssh-keygen. The puppet account’s identity should have no passphrase, since it will need to run autonomously through r10k.

Creating New Repositories

Before we create any new repositories, we add root’s public ssh key to the giteaadmin account on Gitea. Then, in Gitea’s web interface, we make the following repositories:

  • bolt
  • puppet
  • puppetserver
  • r10k
  • puppet-control

and add the puppet account’s public ssh key to each repository as a deploy key.

Migrating to New Git Remotes

cd
for repo in bolt puppet puppetserver r10k; do
  cd ${repo}
  git remote add gitea git@code.blab.renf.ro:giteaadmin/${repo}.git
  git push -u gitea master
  git remote remove origin
  git remote rename gitea origin
  git branch --set-upstream-to=origin/master master
  git push
  cd ..
done
cd puppet-control
git remote add gitea git@code.blab.renf.ro:giteaadmin/puppet-control.git
git push -u gitea production
git remote remove origin
git remote rename gitea origin
git branch --set-upstream-to=origin/production production
git push
cd ..

Configuring the puppet and puppetserver Clones to Use the New Repositories

In the clone of puppet at /etc/puppetlabs/puppet, add a new Gitea remote, pull from it, remove the old remote, and rename the new Gitea remote:

cd /etc/puppetlabs/puppet
git remote add gitea git@code.blab.renf.ro:giteaadmin/puppet.git
git pull gitea master
git remote remove origin
git remote rename gitea origin
git branch --set-upstream-to=origin/master master
git pull

Do the same for the puppetserver clone at /etc/puppetlabs/puppetserver

cd /etc/puppetlabs/puppetserver
git remote add gitea git@code.blab.renf.ro:giteaadmin/puppetserver.git
git pull gitea master
git remote remove origin
git remote rename gitea origin
git branch --set-upstream-to=origin/master master
git pull

Configuring r10k to Use the New Repository

In working copy of r10k repository under /root, change r10k.yaml to:

    ---
    cachedir: '/opt/puppetlabs/puppet/cache'
    sources:
      blab.renf.ro:
        basedir: '/etc/puppetlabs/code/environments'
        remote: 'git@code.blab.renf.ro:giteaadmin/puppet-control.git'

and push to Git server.

Add a new remote in the clone at /etc/puppetlabs/r10k, pull from it, remove the old remote, and rename the Gitea remote:

cd /etc/puppetlabs/r10k
git remote add gitea git@code.blab.renf.ro:giteaadmin/r10k.git
git pull gitea master
git remote remove origin
git remote rename gitea origin
git branch --set-upstream-to=origin/master master
git pull

Testing r10k as puppet

As root, run the r10k workflow again:

(cd /tmp ; sudo -u puppet /opt/puppetlabs/bin/puppetserver ruby /opt/puppetlabs/server/data/puppetserver/jruby-gems/bin/r10k deploy environment -p)

You should get one prompt about accepting the ssh key for code.blab.renf.ro, indicating that r10k is pulling from the Gitea server instead of the local /opt/gitrepos directory. Afterwards, a puppet agent -t run on any managed host should return its regular configuration.

Configuring a Webhook Listener for r10k on the Puppet Server

As stated previously, we want to use https://pypi.org/project/r10k-webhook/ to listen on port 8088, and any time we push a new revision to the puppet-control repository, we want Gitea to notify the Puppet server that there are new changes to pull.

We’ll use the Puppet server’s profile to install and configure the webhook listener. The given manual installation instructions are:

pip3 install r10k-webhook
systemctl daemon-reload
systemctl enable r10k-webhook
systemctl start r10k-webhook

with optional configuration through /etc/r10k_webhook/config.json. One of the configuration items is r10k_path with a default value of r10k. Since we’ve got a relatively involved r10k procedure including a call to puppetserver and its r10k, we probably need to create an r10k shell script and point the webhook at that instead.

So the puppet resources will be:

  • Create an r10k shell script
  • Create a config.json file for the webhook listener
  • Open firewall rules for port 8088 tcp
  • pip3 install r10k-webhook ; systemctl daemon-reload; systemctl enable r10k-webhook; systemctl start r10k-webhook

In theory, the four items could run in any order, but the last item should happen in its given order, since there’s nothing to start or enable until the install and daemon-reload have occurred. In practice, there are a few changes that need to be made to in the last step to work around assumptions about installation paths.

Creating an r10k Shell Script

In site-modules/profile/manifests/puppetserver.pp, add:

  file { '/usr/local/bin/r10k':
    content => '#!/bin/bash
/opt/puppetlabs/bin/puppetserver ruby \
    /opt/puppetlabs/server/data/puppetserver/jruby-gems/bin/r10k "$@"
',
    owner   => root,
    group   => root,
    mode    => '0755',
  }

Creating config.json

In site-modules/profile/manifests/puppetserver.pp, add:

file { '/etc/r10k_webhook':
  ensure => directory,
  owner  => root,
  group  => root,
  mode   => '0755',
}
file { '/etc/r10k_webhook/config.json':
  content => '{ "r10k_path": "/usr/local/bin/r10k" }',
  owner   => root,
  group   => root,
  mode    => '0644',
  require => File['/etc/r10k_webhook'],
}

Installing and Configuring r10k-webhook

In Puppetfile, add:

mod 'puppet-python', '4.1.1'
mod 'puppet-epel', '3.0.1'

In site-modules/profile/manifests/puppetserver.pp, add:

class { 'python':
  ensure => present,
  version => '3'
}
python::pip { 'r10k-webhook':
  ensure       => present,
  pkgname      => 'r10k-webhook',
  pip_provider => 'pip3',
  notify       => Exec['r10k-webhook-daemon-reload'],
  require      => [ File['/usr/local/bin/r10k'],
                    File['/etc/r10k_webhook/config.json'] ],
}

to ensure that Python 3 and the r10k-webhook content is installed. Since r10k-webhook will need to deploy into /etc/puppetlabs/code/environments.webhook, we’ll add:

file { '/etc/puppetlabs/code/environments.webhook':
    ensure => directory,
    owner  => puppet,
    group  => puppet,
    mode   => '0700',
}

to site-modules/profile/manifests/puppetserver.pp to make sure it can download the Git branches as needed. The default systemd unit file for r10k-webhook assumes that the r10k_daemon file is installed into /usr/bin, but since pip installs by default into /usr/local/, we’ll edit its ExecStart line to match the actual location by adding:

file_line { 'r10k-webhook-execstart':
    ensure => present,
    path   => '/usr/lib/systemd/system/r10k-webhook.service',
    line    => 'ExecStart=/usr/local/bin/r10k_daemon -c /etc/r10k_webhook/config.json',
    match   => '^ExecStart',
    require => Python::Pip['r10k-webhook'],
    notify  => Exec['r10k-webhook-daemon-reload'],
}

to site-modules/profile/manifests/puppetserver.pp.

We’ll finally make sure systemd picks up on the changes to the unit file and set the r10k-webhook service to run on boot by adding:

exec { 'r10k-webhook-daemon-reload':
    command     => '/usr/bin/systemctl daemon-reload',
    refreshonly => true,
    before      => Service['r10k-webhook'],
}
service { 'r10k-webhook':
    ensure => running,
    enable => true
}

to site-modules/profile/manifests/puppetserver.pp.

Setting Firewall Rules

In data/nodes/puppet.blab.renf.ro.yaml, add:

firewalld:ports:
  'Allow r10k-webhook from the public zone':
    ensure: present
    port: 8088
    protocol: 'tcp'
    zone: public

The Final Puppetserver Profile Class

class profile::puppetserver {
  class { 'firewalld': }

  class { 'python':
    ensure => present,
    version => '3'
  }
  python::pip { 'r10k-webhook':
    ensure       => present,
    pkgname      => 'r10k-webhook',
    pip_provider => 'pip3',
    notify       => Exec['r10k-webhook-daemon-reload'],
    require      => [ File['/usr/local/bin/r10k'],
                      File['/etc/r10k_webhook/config.json'] ],
  }
  file { '/etc/puppetlabs/code/environments.webhook':
    ensure => directory,
    owner  => puppet,
    group  => puppet,
    mode   => '0700',
  }

  file_line { 'r10k-webhook-execstart':
    ensure => present,
    path   => '/usr/lib/systemd/system/r10k-webhook.service',
    line    => 'ExecStart=/usr/local/bin/r10k_daemon -c /etc/r10k_webhook/config.json',
    match   => '^ExecStart',
    require => Python::Pip['r10k-webhook'],
    notify  => Exec['r10k-webhook-daemon-reload'],
  }
  file { '/usr/local/bin/r10k':
    content => '#!/bin/bash
/opt/puppetlabs/bin/puppetserver ruby \
    /opt/puppetlabs/server/data/puppetserver/jruby-gems/bin/r10k "$@"
',
    owner   => root,
    group   => root,
    mode    => '0755',
  }
  file { '/etc/r10k_webhook':
    ensure => directory,
    owner  => root,
    group  => root,
    mode   => '0755',
  }
  file { '/etc/r10k_webhook/config.json':
    content => '{ "r10k_path": "/usr/local/bin/r10k" }',
    owner   => root,
    group   => root,
    mode    => '0644',
    require => File['/etc/r10k_webhook'],
  }
  exec { 'r10k-webhook-daemon-reload':
    command     => '/usr/bin/systemctl daemon-reload',
    refreshonly => true,
    before      => Service['r10k-webhook'],
  }
  service { 'r10k-webhook':
    ensure => running,
    enable => true
  }
}

Fixes to puppetserver‘s auth.conf

r10k-webhook calls puppet generate types after each r10k run to better isolate environments. As part of the type generation process, r10k-webbook makes a REST call to the puppetserver process to clear each environment’s type cache. By default, puppetserver will refuse that REST call, since it doesn’t come from a known Puppet agent.

{
     match-request: {
         path: "/puppet-admin-api/v1/environment-cache"
         type: path
         method: delete
     }
     allow-unauthenticated: true
     sort-order: 500
     name: "environment-cache"
 },

Testing Deployments Using r10k-webhook

Once everything has been pushed to the Gitea repository and deployed through r10k and puppet agent -t, add a webhook into Gitea’s puppet-control repository with the following settings:

  • Target URL: http://puppet.blab.renf.ro:8088/api
  • HTTP Method: POST
  • POST Content Type: application/json
  • Trigger On: Push Events
  • Active: checked

Clicking the Test Delivery button will trigger the webbook on Gitea. You may see Delivery: Post http://puppet.blab.renf.ro:8088/api: read tcp 192.168.1.5:45488->192.168.1.2:8088: i/o timeout messages in the Response tab of the webhook, but /var/log/messages on the Puppet server should still contain lines like:

May 12 19:52:07 puppet r10k_daemon: INFO: Deploying branch production.

and

May 12 19:52:24 puppet r10k_daemon: INFO: r10k -> Using Puppetfile '/etc/puppetlabs/code/environments.webhook/production/Puppetfile'
May 12 19:52:25 puppet r10k_daemon: INFO: r10k -> Deploying environment /etc/puppetlabs/code/environments.webhook/production
May 12 19:52:25 puppet r10k_daemon: INFO: r10k -> Environment production is now at a4ba2835198f5ac7f576588735ab7d786b7ad403
May 12 19:52:27 puppet r10k_daemon: INFO: Generating types for environment 'production'.
May 12 19:52:29 puppet r10k_daemon: INFO: puppet -> #033[mNotice: Generating Puppet resource types.#033[0m
May 12 19:52:29 puppet r10k_daemon: INFO: puppet -> #033[mNotice: No files were generated because all inputs were up-to-date.#033[0m
May 12 19:52:29 puppet r10k_daemon: INFO: Flushing cache of environment production.

indicating that the r10k deploy succeeded.