In [the previous post](http://www.cloudcomp.ch/2012/06/automating-the-icclab-part-two/) we looked at getting Foreman setup and installed. In this post we’ll run through how you can provision bare metal hosts using Foreman. The first thing that we’ll need to do is configuring Foreman for provisioning! Don’t forget [a VM in which all of this is setup is available](http://www.cloudcomp.ch/wp-content/uploads/2012/06/ForemanV.ova).

## Configuring Foreman to Provision Bare Metal

You will now need to configure Foreman. To do this you need to create the following pieces of information:

0. **Architectures**

Here, if they are not already populated, you need to enter in the OS hardware architectures that are required for your infrastructure fleet. Common architectures include ‘i386’ and ‘x86_64’.

1. **Smart Proxies**

Smart proxies are the entities that interact with the core infrastructural services like DNS, DHCP, TFTP etc. When setting up your Foreman instance you need to tell Foreman where its location is by creating a new Smart Proxy entry and supplying its URL.

2. **Domains**

Here you need to enter in the domain name system information. All that is needed is the top level domain name (e.g. mydomain.com) and an identifier (e.g. “the main TLD”). If your smart proxy has DNS enabled then ensure the DNS value is populated. This value specifies what smart proxy DNS service to use when adding new DNS entries.

Once the smart proxy and DNS entries are entered, go back to the Smart Proxies section and click on ‘Import Subnets’. This will populate Foreman with the required networking information needed to allocate IP addresses against a newly provisioned machine.

3. **Subnets**

If you carried out the auto-import of subnets when setting up your smart proxy you shouldn’t need to change this, unless you want to add more or modify subnets.

4. **Environments**

Environments specify a group of puppet modules that can be applied against a newly provisioned host. To import default ones, `cd /usr/share/foreman` and execute `rake puppet:import:puppet_classes[batch] RAILS_ENV=production`. See the [Foreman wiki page](http://theforeman.org/projects/foreman/wiki/External_Nodes) for further details. You can also import existing environments, along with their associated classes, by navigating to the ‘Puppet Classes’ section and clicking on ‘Import new puppet classes and environments’.

5. **Installation Media**

To install an operating system you need to specify the location of where the source files will come from. Foreman supplies you with some installation media locations to begin with and you can add more either remote or local ones. A suggested minor edit with the default Foreman installation media is to change the mirrors that are closest to you.

**Tip**: Setup a caching server to cache all your downloaded operating system files (e.g. apt-cacher NG for Debian based systems; [puppet module here](https://github.com/lelutin/puppet-apt-cacher-ng)), otherwise you can install from local media locations, say, distributed over HTTP.

6. **Operating Systems**

In the case that you want to deploy Operating Systems to hosts, you will need to define the operating systems that you support. To do this create an operating system entry. Once done go to the Provisioning Template section.

7. **Provisioning Template**

The provisioning templates are key in the TFTP and DHCP boot process of your new host. These specify, depending on the OS automation suite (e.g. preseed, kickstart etc.), how the OS should be installed on the new host, how disk partitions should be setup, what OS packages should be installed etc.

An **important thing to do** here is to create a ‘Default PXE Menu’. This will allow for PXE boots with a fall back of boot by local disk, should a host have an OS already installed.

You need to specify the template for the appropriate operating systems you are supporting. In the case of Ubuntu OS’s you will need to go to each ‘Preseed’ entry and ensure that it is associated to the Ubuntu Operating systems. Once done go back to the Operating Systems entries and ensure that they are associated with the correct templates.

**Note**: A modification to the Ubuntu Default disk partition needs to be made to that a user prompt is avoided. To do this navigate to ‘Partition Tables’, select the ‘Ubuntu default’ and then add the following as a replacement:

[gist id=2888536]

8. **Host Group**

You can think of a Host Group as a profile or policy that is applied to a new host or group of hosts that is/are provisioned. Here selected puppet modules can be enforced on associated hosts to the Host Group. An example Host Group would be, say, an OpenStack Compute (Nova) Host Group. Any host associated with this group would assume the role of providing virtual machines for an OpenStack installation.

With the above items created you are now ready to create a Foreman/puppet managed node! Click on ‘Hosts’ and ‘New Host’. Most parameters will be pre-populated. The only parameter you need to supply is the new host’s MAC address.

### Potential Issues

1. If you come across this DNS related issue when creating a node:

[gist id=2888540]

First check the permissions of your `/etc/bind/rndc.key`. The foreman process needs to have read permissions to this. This is best done by adding the `foreman-proxy` user to the `bind` group (Thanks to Andreas N for [the suggestion](https://groups.google.com/forum/?fromgroups#!topic/foreman-users/TzmrYhoFKFI)!). E.g. on Ubuntu:

[gist id=2888549]

Secondly, a work around, although not elegant, is to disable the dynamic DNS functionality. To do this:

1. Go to ‘Domains’ and clear the value for ‘Dns’ and then save.
2. Edit `/etc/puppet/modules/common/foreman_proxy/manifests/params.pp` and set `$dns = false`.

2. If when provisioning a new host and the PXE boot process complains that it cannot download the pre-configuration file (say in the case of Ubuntu) and the URL is malformed (e.g. http://foreman:3000.mydomain.net), then ensure that you are accessing the Foreman front-end via the fully qualified hostname (e.g. http://foreman.mydomain.net:3000) and not just the host name (e.g. http://foreman:3000).

3. If you come across the following issue when creating a node:

[gist id=2888551]

The first solution is to [apply the changes that are present in this diff](http://theforeman.org/projects/foreman/repository/revisions/a7ad2c1c5e128347fea66478614d8784b222c2d9) that fixes [Foreman issue 1541](http://theforeman.org/issues/1451).
The last resort work around is to install the 0.5 `nightly` release of Foreman. You can select what release of Foreman to install by editing `/etc/puppet/modules/common/foreman/manifests/params.pp` and changing `$package_source`. 0.5 appears to be relatively stable with many [people in the Foreman IRC room](http://theforeman.org/projects/foreman/wiki/Support) using it.

## Integrating Physical Nodes with Foreman & Puppet
For Foreman to use puppet, puppet needs to send reports to Foreman. To do this a script is registered with puppet. Thankfully this is looked after by the previous install steps above. If you want to check the script out, it can be [found at the Foreman repository](https://github.com/theforeman/puppet-foreman/blob/master/templates/foreman-report.rb.erb) or just look in `/usr/lib/ruby/1.8/puppet/reports/foreman.rb`. The other step that the install looks after is registering the report with puppet via the puppet configuration file (`/etc/puppet/puppet.conf`).

At this stage you should have Foreman up and running and acting as not just a puppet master but as an ENC (External Node Classifier). This is looked after (this setup is automated) by yet another simple script (`/etc/puppet/node.rb`) that is registered with puppet. This script supplies node information from Puppet to Foreman.

## Possible Issues on Provisioned Nodes

1. If you are seeing **”hostname was not match with the server certificate”** in `/var/log/syslog` on your provisioned host then [please see this page](http://docs.puppetlabs.com/guides/troubleshooting.html). Normally you need to adjust the `server` parameter in the `/etc/puppet/puppet.conf` file.
2. If you see **”Error 400 on SERVER: Could not find node…”** in `/var/log/syslog` then you will likely have to adjust the variable `url` in `/etc/puppet/node.rb` and `$foreman_url` in `/usr/lib/ruby/1.8/puppet/reports/foreman.rb`.
3. If you see:

[gist id=2888556]

then check out [this article on disabling factsync](http://serverfault.com/questions/66124/puppet-0-25-0-problems-with-access-control-how-can-i-fix-them).

## Next up
The next post in this series will look into deploying OpenStack using puppet in a distributed (multiple nodes scenario) and then followed up with integration with Foreman. Stay tuned!

***Note:*** There are updated installation instructions for the new release of Foreman 1.0.x [in a complementing blog post](http://www.cloudcomp.ch/2012/09/installing-foreman-1-0-1/).

## Introduction
In the world of data centres, automation is not a nice-to-have it’s essential. Sure you could attempt some manual deployment and configuration if you’ve 2 machines to administrate or knock together a custom script in Perl or python if you’ve 30 machines. But what happens when this grows to thousands? This is where the likes of [Puppet](http://puppetlabs.com/), [Chef](http://www.opscode.com/chef/) and [CFEngine](http://cfengine.com/) come into play. These are the tools for managing deployment of cloud-scale data centres.

One of our current requirements here in the [ICCLab](http://www.cloudcomp.ch) is to design and implement scalable infrastructure automation. To do this we have some basic and simple requirements:

1. Toolset must be easy to use, extensible and have a minimal learning curve,
2. Toolset must allow for the deployment of software and the lifecycle management of that software,
3. Toolset must allow the provisioning of physical machines,
4. Toolset must be fully integrated.

Based on these requirements we have selected [Puppet](http://puppetlabs.com/) as the core of this toolset. [Puppet](http://puppetlabs.com/) will satisfy the first two requirements. To satisfy the third and in doing so the fourth, [Foreman](http://theforeman.org) will be selected as the means to provision physical servers.

Foreman is integrated with Puppet. It runs a puppetmaster server which once a physical server is provisioned, will deploy specified [Puppet modules](http://forge.puppetlabs.com/). Foreman itself has a user web front-end that talks to a backend service known as a smart proxy. The smart proxy provides a service that provides DHCP, DNS, Puppet, Puppet CA, and TFTP functionality. As well as the user web front-end Foreman also [has a useful web API](http://theforeman.org/wiki/foreman/API).

## Getting Started
To get started with Puppet and Foreman, we’re setting things up on a VM ([you can download it if you like](http://www.cloudcomp.ch/wp-content/uploads/2012/06/ForemanV.ova), user:fman password:fman) for flexibility. You can of course deploy Foreman to a physical machine but choosing a VM is solely to ease this walkthrough. To do this you can follow the following steps.

***Assumption***: An [Ubuntu 11.04 Server OS](http://releases.ubuntu.com/11.04/) running on [VirtualBox](http://virtualbox.org). Currently [there are issues with 12.04](http://theforeman.org/issues/1481) and Foreman that are related to DB schema migration.

### Foreman Virtual Machine Setup
Assign 2 NICs: one NAT’ed the other on a host-only network. The NAT’ed adapter will act as the public interface through which access to the internet is supplied. The host-only adapter will act as the private internal network in concertation with VirtualBox’s virtual switch (default name `vboxnet0`). The virtual switch `vboxnet0` is set with an IP address of `192.168.56.1` and mask of `255.255.255.0`. DHCP is disabled.

Install a Linux variant, in this case Ubuntu 11.04 “Natty Narwhal”, on to the virtual machine. During the install process choose to install the `openssh-server` so you can `ssh` into the virtual machine.

**Tip**: Most of these commands will require root privileges, so to save typing `sudo $COMMAND` run in interactive `sudo` mode:

[gist id=2888371]

Once the machine has booted update the VM’s packages to the latest:

[gist id=2888373]

During the install, the NAT’ed adapter will be setup. In order to complete the setup you will have to setup the host-only adapter also. To do this edit `/etc/network/interfaces`. The `metric` parameter is important here. Below is an example of how you can setup the host-only adapter, named `eth1`.

[gist id=2888052]

We need to allow traffic to pass to and from (packet forwarding) the host-only and NAT’ed adapters (`eth1` and `eth0`). To do this IP forwarding needs to be enabled:

[gist id=2888334]

You will need to setup iptables rules. In order to persist these rules over reboots, setup the following script:

[gist id=2888339]

Save this as `/etc/init.d/fwd-traff.sh`, make it executable and then set it to start on boot:

[gist id=2888354]

You should now execute the script this script in the case you do not reboot the node at this point.

You should setup your domain settings. By default in this setup, nameserver and domain settings are automatically managed by `resolvconf`. As puppet, hence Foreman, relies on the fully qualified host name of nodes its installed on, you should, if not using other means, configure `resolvconf` so that it does not overwrite your domain and nameserver settings. To do this edit `/etc/resolvconf/resolv.conf.d/head` and place the following content (here we’re using the google DNS resolver but you might want to use something else):

[gist id=2987340]

Otherwise you can edit `/etc/dhcp/dhclient.conf` and set the following:

[gist id=2987954]

Time synchronisation is important, especially for puppet (time drift and certificates do not mix well! ;-)). To look after this install `ntp`

[gist id=2987348]

Finally setup a port forwarding rule on the NAT’ed adapter (`eth0`) to the SSH service running on `eth1` in VirtualBox so that as a convenience you can ssh to the Foreman node using your own client and not the VM console.

### Installing Foreman
The most appropriate means to deploy foreman is of course via puppet! First install the necessary `git`.

[gist id=2888377]

Now install [puppet from puppetlabs.com](http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#for-debian-and-ubuntu). This is done as we need the latest version, 2.7.x. In version 2.6.x, the default in Ubuntu 11.04, there is an Augeas bug that makes Foreman installation tricky.

[gist id=2888381]

You should also disable `puppetmaster` from `/etc/defaults/puppetmaster` as this will effect how apache executes.

[gist id=2988152]

Now, get Foreman. You should place this collection of puppet modules in the place where you store all you modules (e.g. `/etc/puppet/modules`). For this installation we’ve placed them in `/etc/puppet/modules/common`. Of course, you being a DevOps nut, you want to have all of these configurations under SCM (e.g. git, mercurial or subversion).

[gist id=2888384]

You will also need a DHCP, a DNS and the Concat module. The following modules do the job perfectly:

– [DHCP Module](https://github.com/GregSutcliffe/puppet-dhcp)

git clone git://github.com/GregSutcliffe/puppet-dhcp.git dhcp

– [DNS Module](https://github.com/GregSutcliffe/puppet-dns)

git clone git://github.com/GregSutcliffe/puppet-dns.git dns

– [Concat Module](https://github.com/GregSutcliffe/puppet-concat)

git clone git://github.com/GregSutcliffe/puppet-concat.git concat

***Note:*** As we’re installing onto Ubuntu, a Debian variant, there maybe some gotchas you need to read up on. See [README.debian](https://github.com/theforeman/foreman-installer/blob/master/README.debian).

You now have all the necessary Foreman modules. A quick directory listing should show the following:

[gist id=2888400]
***Foreman 0.4.2 only***: In the current Foreman installer there is a minor bug in `/etc/puppet/modules/common/tftp/manifests/install.pp`. To remedy this run the following `sed` script:

[gist id=2888404]

***Foreman 0.4.2 only***: You will also have to make the following addition in `/etc/puppet/modules/common/tftp/manifests/params.pp`:

[gist id=2888406]

Now, customise `/etc/puppet/modules/common/foreman_proxy/manifests/params.pp`. For this installation, change the following TFTP, DHCP and DNS variables as:

[gist id=2888416]

Also review the settings in `/etc/puppet/modules/common/foreman/manifests/params.pp`.

If you intend on running a DNS server for the subnet and wish to forward the DNS requests upstream to another DNS server then you will need to modify the DNS module file `/etc/puppet/modules/common/dns/templates/options.conf.erb` and insert the following `forwarders` clause to add the Google (8.8.8.*) and OpenDNS (208.67.222.*) upstream servers:

[gist id=2888418]

If you’re foreman does not have a domain name associated with it then you can quick supply one by adding a `domain` statement in `/etc/resolv.conf` (e.g. `domain mydomain.net`).
Also as you are using Foreman’s managed DNS server, you should make a new `nameserver` entry in `/etc/resolv.conf` that points to the DNS server’s IP address.

**Tip**: At this stage you can make a snapshot of the VM before installing Foreman so you can revert to it if your install does not work out.

Now it’s time to let puppet do the work! Execute the following:

[gist id=2888421]

Enable the foreman service:

[gist id=2888423]

Now start the Foreman server by executing:

[gist id=2888426]

Foreman will now be available on its URL serving from port 3000. Finally commit and optionally push your modified modules to your SCM repository. If you ever want to setup Foreman again on a fresh machine all you’ll now ever need to do is to install puppet, check out foreman from your repository and apply the relevant modules.

#### Possible Installation Issues

If you experience issues on executing this command and you want more information, simply add the flags `-v -d` to the `puppet` command.

1. You might see the error:

[gist id=2888441]

You can safely ignore this but if you want to remove the error it simply create the directory.

2. If you encounter an error telling you that “no fqdn” is available to `facter` then [see this useful article](http://stackoverflow.com/questions/7780322/puppet-facter-could-not-retrieve-fact-fqdn-how-to-fix-or-circumvent) on how to resolve the issue.

3. ***Foreman 0.4.2 only***: If you get the following error output:

[gist id=2888442]

or

[gist id=2988063]

then your issue lies with Foreman and a specific issue with Augeas attempting to add certain `sudoers` permissions for Foreman and persist them. To resolve this you will need to comment out the `augeas` entry in `/etc/puppet/modules/common/foreman_proxy/manifests/config.pp` and add the following entry:

[gist id=2888445]

The contents of the template (located at : `/etc/puppet/modules/common/foreman_proxy/templates/foreman.erb`) should be:

[gist id=2888449]

**Note**: With each change(s) reapply the foreman puppet command as above.

#### Customising Foreman
There are a number of customisations that you might want to make. Don’t edit the Foreman configuration files directly. Edit the `param.pp` files and then use `puppet apply`.

1. Enable Foreman front-end authentication:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
change:
`$authentication = false`
to:
`$authentication = true`

2. In a trusted environment and for troubleshooting, disable SSL:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
change:
`$ssl = true`
to:
`$ssl = false`

3. By default Foreman uses your fully qualified domain name. If you do not want this and say just the hostname then:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
change:
`$foreman_url = “http://${::fqdn}”`
to:
`$foreman_url = “http://${::hostname}”`

4. By default Foreman uses a sqlite3 backend. You can change this. See [the following Foreman article](http://theforeman.org/projects/foreman/wiki/Database_configuration).

If you apply such changes make sure to re-run the foreman puppet command (as above):

[gist id=2888421]

## Next up
Things are hotting up in the area of automation. Just recently EMC announced their new [open source provisioning framework, Razor](https://github.com/puppetlabs/Razor), built on Puppet. There is also [support added within Foreman](http://cloudprovisioning.wordpress.com/) for provisioning VMs via oVirt and Amazon EC2.

Big thanks goes out to [all those that supplied suggestions](https://groups.google.com/forum/?hl=en&fromgroups#!topic/foreman-users/-ASnNeC5SyE) to improve this guide!

In the next blog post we’ll look at configuring and provisioning bare metal hosts with Foreman. Stay tuned!