Automating The ICCLab: Part One

***Note:*** There are updated installation instructions for the new release of Foreman 1.0.x [in a complementing blog post](http://www.cloudcomp.ch/2012/09/installing-foreman-1-0-1/).

## Introduction
In the world of data centres, automation is not a nice-to-have it’s essential. Sure you could attempt some manual deployment and configuration if you’ve 2 machines to administrate or knock together a custom script in Perl or python if you’ve 30 machines. But what happens when this grows to thousands? This is where the likes of [Puppet](http://puppetlabs.com/), [Chef](http://www.opscode.com/chef/) and [CFEngine](http://cfengine.com/) come into play. These are the tools for managing deployment of cloud-scale data centres.

One of our current requirements here in the [ICCLab](http://www.cloudcomp.ch) is to design and implement scalable infrastructure automation. To do this we have some basic and simple requirements:

1. Toolset must be easy to use, extensible and have a minimal learning curve,
2. Toolset must allow for the deployment of software and the lifecycle management of that software,
3. Toolset must allow the provisioning of physical machines,
4. Toolset must be fully integrated.

Based on these requirements we have selected [Puppet](http://puppetlabs.com/) as the core of this toolset. [Puppet](http://puppetlabs.com/) will satisfy the first two requirements. To satisfy the third and in doing so the fourth, [Foreman](http://theforeman.org) will be selected as the means to provision physical servers.

Foreman is integrated with Puppet. It runs a puppetmaster server which once a physical server is provisioned, will deploy specified [Puppet modules](http://forge.puppetlabs.com/). Foreman itself has a user web front-end that talks to a backend service known as a smart proxy. The smart proxy provides a service that provides DHCP, DNS, Puppet, Puppet CA, and TFTP functionality. As well as the user web front-end Foreman also [has a useful web API](http://theforeman.org/wiki/foreman/API).

## Getting Started
To get started with Puppet and Foreman, we’re setting things up on a VM ([you can download it if you like](http://www.cloudcomp.ch/wp-content/uploads/2012/06/ForemanV.ova), user:fman password:fman) for flexibility. You can of course deploy Foreman to a physical machine but choosing a VM is solely to ease this walkthrough. To do this you can follow the following steps.

***Assumption***: An [Ubuntu 11.04 Server OS](http://releases.ubuntu.com/11.04/) running on [VirtualBox](http://virtualbox.org). Currently [there are issues with 12.04](http://theforeman.org/issues/1481) and Foreman that are related to DB schema migration.

### Foreman Virtual Machine Setup
Assign 2 NICs: one NAT’ed the other on a host-only network. The NAT’ed adapter will act as the public interface through which access to the internet is supplied. The host-only adapter will act as the private internal network in concertation with VirtualBox’s virtual switch (default name `vboxnet0`). The virtual switch `vboxnet0` is set with an IP address of `192.168.56.1` and mask of `255.255.255.0`. DHCP is disabled.

Install a Linux variant, in this case Ubuntu 11.04 “Natty Narwhal”, on to the virtual machine. During the install process choose to install the `openssh-server` so you can `ssh` into the virtual machine.

**Tip**: Most of these commands will require root privileges, so to save typing `sudo $COMMAND` run in interactive `sudo` mode:

[gist id=2888371]

Once the machine has booted update the VM’s packages to the latest:

[gist id=2888373]

During the install, the NAT’ed adapter will be setup. In order to complete the setup you will have to setup the host-only adapter also. To do this edit `/etc/network/interfaces`. The `metric` parameter is important here. Below is an example of how you can setup the host-only adapter, named `eth1`.

[gist id=2888052]

We need to allow traffic to pass to and from (packet forwarding) the host-only and NAT’ed adapters (`eth1` and `eth0`). To do this IP forwarding needs to be enabled:

[gist id=2888334]

You will need to setup iptables rules. In order to persist these rules over reboots, setup the following script:

[gist id=2888339]

Save this as `/etc/init.d/fwd-traff.sh`, make it executable and then set it to start on boot:

[gist id=2888354]

You should now execute the script this script in the case you do not reboot the node at this point.

You should setup your domain settings. By default in this setup, nameserver and domain settings are automatically managed by `resolvconf`. As puppet, hence Foreman, relies on the fully qualified host name of nodes its installed on, you should, if not using other means, configure `resolvconf` so that it does not overwrite your domain and nameserver settings. To do this edit `/etc/resolvconf/resolv.conf.d/head` and place the following content (here we’re using the google DNS resolver but you might want to use something else):

[gist id=2987340]

Otherwise you can edit `/etc/dhcp/dhclient.conf` and set the following:

[gist id=2987954]

Time synchronisation is important, especially for puppet (time drift and certificates do not mix well! ;-)). To look after this install `ntp`

[gist id=2987348]

Finally setup a port forwarding rule on the NAT’ed adapter (`eth0`) to the SSH service running on `eth1` in VirtualBox so that as a convenience you can ssh to the Foreman node using your own client and not the VM console.

### Installing Foreman
The most appropriate means to deploy foreman is of course via puppet! First install the necessary `git`.

[gist id=2888377]

Now install [puppet from puppetlabs.com](http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#for-debian-and-ubuntu). This is done as we need the latest version, 2.7.x. In version 2.6.x, the default in Ubuntu 11.04, there is an Augeas bug that makes Foreman installation tricky.

[gist id=2888381]

You should also disable `puppetmaster` from `/etc/defaults/puppetmaster` as this will effect how apache executes.

[gist id=2988152]

Now, get Foreman. You should place this collection of puppet modules in the place where you store all you modules (e.g. `/etc/puppet/modules`). For this installation we’ve placed them in `/etc/puppet/modules/common`. Of course, you being a DevOps nut, you want to have all of these configurations under SCM (e.g. git, mercurial or subversion).

[gist id=2888384]

You will also need a DHCP, a DNS and the Concat module. The following modules do the job perfectly:

– [DHCP Module](https://github.com/GregSutcliffe/puppet-dhcp)

git clone git://github.com/GregSutcliffe/puppet-dhcp.git dhcp

– [DNS Module](https://github.com/GregSutcliffe/puppet-dns)

git clone git://github.com/GregSutcliffe/puppet-dns.git dns

– [Concat Module](https://github.com/GregSutcliffe/puppet-concat)

git clone git://github.com/GregSutcliffe/puppet-concat.git concat

***Note:*** As we’re installing onto Ubuntu, a Debian variant, there maybe some gotchas you need to read up on. See [README.debian](https://github.com/theforeman/foreman-installer/blob/master/README.debian).

You now have all the necessary Foreman modules. A quick directory listing should show the following:

[gist id=2888400]
***Foreman 0.4.2 only***: In the current Foreman installer there is a minor bug in `/etc/puppet/modules/common/tftp/manifests/install.pp`. To remedy this run the following `sed` script:

[gist id=2888404]

***Foreman 0.4.2 only***: You will also have to make the following addition in `/etc/puppet/modules/common/tftp/manifests/params.pp`:

[gist id=2888406]

Now, customise `/etc/puppet/modules/common/foreman_proxy/manifests/params.pp`. For this installation, change the following TFTP, DHCP and DNS variables as:

[gist id=2888416]

Also review the settings in `/etc/puppet/modules/common/foreman/manifests/params.pp`.

If you intend on running a DNS server for the subnet and wish to forward the DNS requests upstream to another DNS server then you will need to modify the DNS module file `/etc/puppet/modules/common/dns/templates/options.conf.erb` and insert the following `forwarders` clause to add the Google (8.8.8.*) and OpenDNS (208.67.222.*) upstream servers:

[gist id=2888418]

If you’re foreman does not have a domain name associated with it then you can quick supply one by adding a `domain` statement in `/etc/resolv.conf` (e.g. `domain mydomain.net`).
Also as you are using Foreman’s managed DNS server, you should make a new `nameserver` entry in `/etc/resolv.conf` that points to the DNS server’s IP address.

**Tip**: At this stage you can make a snapshot of the VM before installing Foreman so you can revert to it if your install does not work out.

Now it’s time to let puppet do the work! Execute the following:

[gist id=2888421]

Enable the foreman service:

[gist id=2888423]

Now start the Foreman server by executing:

[gist id=2888426]

Foreman will now be available on its URL serving from port 3000. Finally commit and optionally push your modified modules to your SCM repository. If you ever want to setup Foreman again on a fresh machine all you’ll now ever need to do is to install puppet, check out foreman from your repository and apply the relevant modules.

#### Possible Installation Issues

If you experience issues on executing this command and you want more information, simply add the flags `-v -d` to the `puppet` command.

1. You might see the error:

[gist id=2888441]

You can safely ignore this but if you want to remove the error it simply create the directory.

2. If you encounter an error telling you that “no fqdn” is available to `facter` then [see this useful article](http://stackoverflow.com/questions/7780322/puppet-facter-could-not-retrieve-fact-fqdn-how-to-fix-or-circumvent) on how to resolve the issue.

3. ***Foreman 0.4.2 only***: If you get the following error output:

[gist id=2888442]

or

[gist id=2988063]

then your issue lies with Foreman and a specific issue with Augeas attempting to add certain `sudoers` permissions for Foreman and persist them. To resolve this you will need to comment out the `augeas` entry in `/etc/puppet/modules/common/foreman_proxy/manifests/config.pp` and add the following entry:

[gist id=2888445]

The contents of the template (located at : `/etc/puppet/modules/common/foreman_proxy/templates/foreman.erb`) should be:

[gist id=2888449]

**Note**: With each change(s) reapply the foreman puppet command as above.

#### Customising Foreman
There are a number of customisations that you might want to make. Don’t edit the Foreman configuration files directly. Edit the `param.pp` files and then use `puppet apply`.

1. Enable Foreman front-end authentication:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
change:
`$authentication = false`
to:
`$authentication = true`

2. In a trusted environment and for troubleshooting, disable SSL:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
change:
`$ssl = true`
to:
`$ssl = false`

3. By default Foreman uses your fully qualified domain name. If you do not want this and say just the hostname then:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
change:
`$foreman_url = “http://${::fqdn}”`
to:
`$foreman_url = “http://${::hostname}”`

4. By default Foreman uses a sqlite3 backend. You can change this. See [the following Foreman article](http://theforeman.org/projects/foreman/wiki/Database_configuration).

If you apply such changes make sure to re-run the foreman puppet command (as above):

[gist id=2888421]

## Next up
Things are hotting up in the area of automation. Just recently EMC announced their new [open source provisioning framework, Razor](https://github.com/puppetlabs/Razor), built on Puppet. There is also [support added within Foreman](http://cloudprovisioning.wordpress.com/) for provisioning VMs via oVirt and Amazon EC2.

Big thanks goes out to [all those that supplied suggestions](https://groups.google.com/forum/?hl=en&fromgroups#!topic/foreman-users/-ASnNeC5SyE) to improve this guide!

In the next blog post we’ll look at configuring and provisioning bare metal hosts with Foreman. Stay tuned!