Ton Kersten

Puppet environments

For my job I do a lot of Puppet and I thought it was about time to write some tips and tricks down.

First part of this post is about my environment setup. In my test setup I use a lot of environments. They are not at all useful, but that’s not the point. It’s my lab environment so things need to break once in a while. But with multiple environments Puppetlabs says that you should switch to directory environments (PuppetDoc) but some way or another I cannot get that to work in a good way with my PE version (3.4.3 (Puppet Enterprise 3.2.3)). So I started implementing dynamic environments, which is a simple way of specifying the directories for your environments.

Part of my puppet.conf looks like

[master]
    environment = production
    manifest    = $confdir/environments/$environment/manifests/site.pp
    manifestdir = $confdir/environments/$environment/manifests
    modulepath  = $confdir/environments/$environment/modules:/usr/share/puppet/modules
    templatedir = $confdir/environments/$environment/templates

So, my default environment is production and a client can specify another environment to be in. The command

puppet agent --environment=test

would place this node in the test environment. A simple module places a new puppet.conf file on the client stating this new environment. Couldn’t be more simple.

Well, that’s what you think. But what if you need to deploy 10.000+ hosts of which there are about a third in environment test and about a 1000 in environment development? It would take a lot of time to ssh into all these servers and run Puppet with the correct environment.

There has to be a way around that. And, of course, there is. In Puppet version 3 and up Hiera is integrated into Puppet and we already use that a lot. Why not integrate the environment in Hiera? Well, our hiera.yaml is now:

---
:hierarchy:
    - "%{environment}/hiera/%{::fqdn}"
    - "%{environment}/hiera/%{::hostname}"
    - "%{environment}/hiera/%{::domainname}"
    - "%{environment}/hiera/%{::systemtype}"
    - "%{environment}/hiera/%{::osfamily}"
    - "%{environment}/hiera/common"

:backends:
    - yaml

:yaml:
    :datadir:
        /etc/puppetlabs/puppet/environments

This challenges me with a chicken and egg problem. To get the environment I need to know the environment. But what if I make Hiera into an ENC and let this one deliver the environment? Can this be done? Yes, it can.

This is how I did it:

First create a part of the Hiera structure that’s not in the current environment, for example like this:

---
:hierarchy:
    - "hiera/%{::fqdn}"
    - "hiera/default"
    - "%{environment}/hiera/%{::fqdn}"
    - "%{environment}/hiera/%{::hostname}"
    - "%{environment}/hiera/%{::domainname}"
    - "%{environment}/hiera/%{::systemtype}"
    - "%{environment}/hiera/%{::osfamily}"
    - "%{environment}/hiera/common"

:backends:
    - yaml

:yaml:
    :datadir:
        /etc/puppetlabs/puppet/environments

And in the directory /etc/puppetlabs/puppet/environments/hiera I place a very small file, called default.yaml, which contains:

---
environment: 'production'

This makes sure that any node without a specific file, will get the production environment. This is the default for Puppet as well, so nothing changes for that.

To test this, run:

hiera environment ::fqdn=$(hostname -f)

This will give you something like environment: production. For every host in another environment as the production one, create a small file named the FQDN of the host with the contents stating the wanted environment.

(Watch for the :: in front of the fqdn. This means that the fqdn variable is a top scope variable, as all facter variables are.

Now integrate this into Puppet. First create a little script that executes the command above and returns the wanted output.

My script is called getenv and placed in /etc/puppetlabs/puppet/bin

#!/bin/bash

penv="$(/opt/puppet/bin/hiera                       \
            -c /etc/puppetlabs/puppet/hiera.yaml    \
            environment ::fqdn="${1}")"

echo "environment: ${penv}"

This returns a string like environment: production.

And last, but not least, place this settings in the [master] of your puppet.conf

node_terminus  = exec
external_nodes = /etc/puppetlabs/puppet/bin/getenv

It took some work to get things started, but a small shell thingy read the file with all 10.000+ hosts and required environments, that created all the Hiera files for all nodes that are not in the production environment.

Just one thing to do: When I have a lot of host-files in a single directory, this could become slow. I could place all definitions in a simple database, but things would get complicated again, and that’s not what I want. I also could split things up per letter, but I’m not sure yet if I really want that.

When I have resolved this, this entry will be continued.