Tag: automation (page 2 of 3)

Automating OCCI Installations

As part of the work here in the ICCLab not only are we active in the [OCCI working group](http://www.occi-wg.org) but also contributing not only [contributing to its implementation on OpenStack](https://github.com/tmetsch/occi-os) but we also make available our work on automating the install of OpenStack. We recently made a contribution to the [puppetlab-nova project](https://github.com/puppetlabs/puppetlabs-nova). This [contribution allows](https://github.com/puppetlabs/puppetlabs-nova/pull/150) users of the nova module to specify the APIs to enable in nova, as well as enabling the OCCI if specified.

The contribution, [submitted as a pull request](https://github.com/puppetlabs/puppetlabs-nova/pull/150) can be used in the following fashion:

[gist id=3778884]

The `nova::api` class declared above enables all the usual OpenStack APIs as well as the OCCI interface. Where the OCCI API is enabled, puppet then will look after installing the necessary components.

From Bare Metal to Cloud

This is the presentation that was presented at the [EGI Technical Forum 2012 in Prague](http://tf2012.egi.eu/).

If you like, [download the slides as pdf](http://blog.zhaw.ch/icclab/files/2012/09/From-Bare-Metal-to-Cloud.pdf).

There is also a youtube video showing the various stages of bring bare metal machines to a state such that they have OpenStack installed and operational.

For those in attendance or those that are interested in how all of this is done, all information, HOWTOs, code, virtual machine images are available from this site.

The talk had an excellent attendance and there is great interest in using OpenStack within the EGI FedCloud environment, especially one where the installation is automated as with our work.

ICCLab EGI TF Audience


A Quick Performance Boost Tip for Foreman

It doesn’t make sense to continually download the same operating system packages when you can cache them along side your Foreman installation. Assuming you use a debian based OS, in order to do this and have a cached copy of all packages you use within your infrastructure simply install apt-cacher or apt-cacher-ng. Our preference is for apt-cacher-ng but we’ll show you how to install both.

# Installing apt-cacher-ng

[gist id=3720930]

You shouldn’t have to adjust the configuration of apt-cacher-ng for the basic functionality it offers, however if you need to adjust settings in `/etc/apt-cacher-ng/acng.conf`.

# Installing apt-cacher

[gist id=3718168]

Then set the contents of `apt-cacher-conf` to:

[gist id=3718171]

***Note:*** you might want to change the hostname of the ubuntu mirror and interface (eth1 is specific to previous articles on Foreman)

Reports are created every 24 hours, which can be accessed at `http://$FOREMAN:3241/report`. To force the creation run:

[gist id=3718175]

# Configuring Foreman
Once you have setup apt-cacher you can then create a new Foreman “Installation Media Source”. Simply supply a sensible name and importantly set the URL of that installation media source to `http://$FOREMAN:3241/ubuntu`, where $FOREMAN is either the IP or FQDN of your Foreman host.

Installing Foreman 1.0.1

Just recently the Foreman project released the latest version, 1.0.1. If you are following our [previous guide to install 0.4.2](http://www.cloudcomp.ch/2012/06/automating-the-icclab-part-one/) then you should also follow this.

# Installing & Configuring Foreman
You should setup your virtual machine exactly as we did in the previous guide, install puppet and checkout the foreman-installer modules from github. There is a small number of issues with the installer but we’ll easily walk you through them!

To check things out quickly, you can [download a VM (OVA) that has Foreman 1.0.1](http://www.cloudcomp.ch/wp-content/uploads/2012/09/Foreman1.01.ova) preconfigured. The username/password is `root` and `root`. This also includes puppet modules to deploy OpenStack compute and controller nodes.

> Side note: the puppetlabs repository has changed. Make sure to:
> `wget http://apt.puppetlabs.com/puppetlabs-release-stable.deb`

Ensure that the foreman-proxy is part of the bind group. If not add the user:

[gist id=3667157]

Configure your `foreman_proxy/manifests/params.pp as` before, ensuring to enable DHCP and DNS and for each of those setting the correct network settings (subnet etc)

Configure you `foreman/manifest/params.pp` as before. For us we disabled SSL. Very important here is that you set the `foreman_url` parameter to include the port number on which foreman listens (port 3000).

[gist id=3667152]

If it is not set then the scripts that tie puppet and foreman together will not work. This is a [known and reported issue](http://theforeman.org/issues/1855), which will be resolved.

Currently there is a bug in `foreman_proxy/manifests/proxydhcp.pp`. For now you need to manually set the DNS `nameserver` and TFTP `nextserver` parameters. This [bug has been reported](https://groups.google.com/d/topic/foreman-users/t1m8JeWVd7U/discussion) and will be resolved soon.

Finally you need to apply [this patch](https://github.com/theforeman/smart-proxy/commit/a402c71290f2d8205e60b876f2a40dfa9fefacda). Puppet in its most recent version changed the value of the return code from operations related to `puppetca`. This causes blocking issues with provisioning and deleting hosts with foreman. You can use this sed command if it suits you:

[gist id=3667151]

Once applied you should restart the foreman-proxy service

[gist id=3667147]

Note, if you start the foreman service and it halts with a stacktrace then you will have to reinitialise the database. This is a one-time operation.

[gist id=3667139]

Once these step have been complete, you can then configure Foreman itself (setting smart proxy, host groups etc).

When configuring these various aspects you should update the ‘Ubuntu default’ disk partition table configuration. Use the following to ensure a complete automatic install:

[gist id=3667133]

One of the issues that we’re dealing with currently is that rather than the puppetmaster’s hostname being placed in the relevant configuration files (e.g. `puppet.conf`), an IP address is inserted. This will not work as it will fail with SSL issues. The current work around is to create a ‘snippet’ in the Provisioning Templates section. With this snippet created then set the content of the config files in Provisioning Templates to (using `puppet.conf` as an examples):

[gist id=3667127]

Foreman, Puppet and OpenStack

# Foreman, Puppet and OpenStack

So all our work in the previous articles has been leading up to this one. In this article we’ll describe how you can deploy a full multi-node [OpenStack](http://www.openstack.org) cluster beginning from bare metal using [Foreman](http://www.theforeman.org) and [puppet](http://www.puppetlabs.com). Before continuing we should note what exactly ‘bare metal’ is in this context. Bare metal refers to physical server hardware that has not yet been provisioned with an operating system. When provisioning this bare metal, it is assumed that the underlying network has been setup (e.g. L2 configurations).

For the purposes of this article all setup will happen in a virtualised environment just as in the previous articles. It will also draw upon those previous articles.

The first requirement is to have a successfully running installation of Foreman. You should find all the information on how to do this in [the articles](http://www.cloudcomp.ch/2012/06/automating-the-icclab-part-one/) on [setting up Foreman](http://www.cloudcomp.ch/2012/06/automating-the-icclab-part-two/).

Once Foreman has been setup the next thing that you will need to do is to deploy the OpenStack puppet modules as described in [the article on puppet and OpenStack](http://www.cloudcomp.ch/2012/07/puppet-and-openstack-part-one/).

Note, that you must only execute up to and including `git clone https://github.com/puppetlabs/puppetlabs-openstack` in the “Select Puppet Modules” section. Once you have completed this then follow this procedure.

The OpenStack modules require that `storeconfigs` is enabled. To do this you need to edit the foreman configuration file, in our case the foreman-installer manifest located at `/etc/puppet/modules/common/foreman/params.pp`. In there you will need to set `storeconfigs => true`. Once this is done you will need to run the foreman-installer again.

For the purposes of this article we will create our own isolated puppet environment named `iaas`. To do this:

[gist id=3119051]

Then configure puppet so that it knows of this environment by editing `/etc/puppet/puppet.conf` and insert this definition:

[gist id=3119055]

Once you have defined the `iaas` environment there are two things to do:

1. Install all the necessary OpenStack puppet modules.

You will already have the puppet modules checked out if you have followed the [puppet and OpenStack article](http://www.cloudcomp.ch/2012/07/puppet-and-openstack-part-one/). Copy the OpenStack modules as follows:

[gist id=3119076]

Lastly in this step, you need to change a variable in the `Rakefile` located in the folder where the OpenStack modules are cloned into. To do this change `default_modulepath` to the following:

[gist id=3119084]

Now execute the rake command `rake modules:clone` as in the [puppet OpenStack article](http://www.cloudcomp.ch/2012/07/puppet-and-openstack-part-one/). All other modules that the OpenStack requires will now be placed in the `iaas` environment.

[gist id=3119059]

2. Import the new environment and all associated puppet classes into Foreman.

In the Foreman web interface import the new `iaas` environment and all its associated modules. Do this by navigating in the web interface “Other->Puppet Classes” and clicking on “Import new puppet classes and environments”. A screen will then show the new environment to be imported, along with its modules and await your confirmation.

Now that these steps are complete, you will want to create some class definitions in a module with which you can apply against new Foreman managed hosts. Below is an example structure of module followed by the contents that defines both the OpenStack all-in-one, controller and compute node roles.

***Module structure***:

[gist id=3119093]

***Module Content***:

* [`all_in_one.pp`](https://gist.github.com/3118975): This will setup an all in one installation of openstack.
* [`controller.pp`](https://gist.github.com/3118972): This will setup the controller on a specified host
* [`compute.pp`](https://gist.github.com/3118973): This will setup the compute services on a specified host
* [`params.pp`](https://gist.github.com/3118970): This is some *basic* common configurations for `controller.pp` and `compute.pp`

Note that as we’re setting this up in a virtualised environment, `libvirt_type` is set to `qemu`. If you were deploying to bare metal, this value would be `kvm` or another suitable hypervisor. Also to note, this file looks almost identical to the `site.pp` file we previous used.

With this module created, you will need to import it just as you did when importing the `iaas` environment. Doing this will simply update the current environment.

Now that you have imported everything, you can now provision a host with Foreman and when doing so select the `icclab::controller` resource class for the new host after ensuring the field ‘Environment’ is ‘iaas’. Once the new host boots over PXE, installs the selected OS (Ubuntu 12.04 in this case) and the post-install script executes, the OpenStack resource class will be applied to the host and you will now have a brand new and shiny OpenStack controller node.

To add compute nodes is almost the exact same process except rather than selecting `icclab::os_controller`, you will select `icclab::compute`. Simple, eh?! You can add as many compute nodes as you have hardware to this arrangement.

# Issues Encountered
Ok, so not everything is as simple. There was one major issue encountered while doing this work.

Puppet’s support of environments has a number of associated issues with it. This is manifested when a newly provisioned host has a module applied and shown by the following error message:

[gist id=3119096]

What this error is telling us is that the puppet resource type cannot be found on the puppet module path. This is because it lives in a location that puppet is unaware of, namely the `iaas` environment. You will also encounter the same issue if you use the `production` or `development` environments that Foreman setups up. There are more details about this issue on the puppetlabs’ tracker [under bug #4409](http://projects.puppetlabs.com/issues/4409). Thankfully there is a work around pointed out in the thread and it is the following:

[gist id=3119097]

Here we manually place the types and providers in a place where puppet will find them. Naturally this solution is not perfect however can be further automated by a `cron` job or executed when there are updates to the effected modules.

ICCLab to Present “From Bare-Metal to Cloud” at EGI Technical Forum 2012

ICCLabs and GWDG will present on the topic of “From Bare-Metal to Cloud” at EGI Technical Forum 2012 in Prague.

The ICCLab and GWDG had a shared, common problem, namely how to deploy infrastructural service technology (e.g. OpenStack, CloudStack etc.) with the least amount of user interaction (i.e. automated) during the deployment process, across a large number of servers. The solution to be presented allows for the easy deployment of operating systems on to bare-metal (physical servers) and the deployment and management of specified software packages upon those provisioned bare-metal systems. To accomplish the combination of Foreman and Puppet was chosen. For the work, it was assumed that the network architecture, partitioning etc. is already determined.

This presentation will detail what measures have been taken to automate the provisioning of OpenStack clusters at the two research labs. The presentation will describe the technology stack, discuss the individual technologies used and share the information with others. It will conclude with a demonstration of provisioning a multi-cluster OpenStack deployment upon virgin bare metal servers.

Puppet and OpenStack: Part Two

In our [last article](http://www.cloudcomp.ch/2012/07/puppet-and-openstack-part-one/ ) we setup OpenStack using puppet on one node. In this article we’ll look at setting up OpenStack in a clustered environment. For this we’ll use the following configuration of VMs:

* The hypervisor is [VirtualBox](http://www.virtualbox.org).

* 1 OpenStack **Controller** VM.

This will host all the main services of OpenStack and will not run any virtual machines. This will be running Ubuntu 12.04 and acting as the puppetmaster. It will require 3 adapters:

– `eth0`: Host-only. Assign this an IP that matches the VirtualBox virtual switch.
– `eth1`: Host-only. Leave this unassigned. OpenStack will look after this and place a linux kernel bridge upon in named `br100`.
– `eth2`: NAT. Setup this adapter as [shown in previous articles](http://www.cloudcomp.ch/2012/06/automating-the-icclab-part-one/).

* 2 OpenStack **Compute Node** VMs.

This will only act as a node that that provide virtual machines and storage volumes when requested by the OpenStack controller VM. This will be running Ubuntu 12.04 and acting as a puppet agent (slave). These VMs will require 2 adapters:

– `eth0`: Host-only. Assign this an IP that matches the VirtualBox virtual switch.
– `eth1`: Host-only. Leave this unassigned. OpenStack will look after this.

In both cases above, `eth0` and `eth1` can share the same or different networks. In our case we set them to different virtual networks (`vboxnet0` and `vboxnet1` respectively).

## OpenStack Controller Setup

Once the controller VM is up and running and you have configured `eth2`, the next task at hand is to configure puppet. You can use the same puppet configuration as was shown in [the article on creating an OpenStack all-in-one installation](http://www.cloudcomp.ch/2012/07/puppet-and-openstack-part-one/). The same hostname configuration is also the same, however as there are two other nodes to deal with in the setup you should place these in your `/etc/hosts` file (unless you’re using a setup with a DNS server). With your puppet and hostname resolution configured, install the puppet modules as shown in the previous article. Where things begin to differ is in the configuration of `/etc/puppet/manifests/site.pp`.

In the `site.pp` file we will be using 2 different `node` definitions. One for the controller node and the second for compute nodes. For the controller node we will explicitly set the node name to that of the fully qualified domain name of the controller VM. The definition is then:

[gist id=3029148]

In the second case we are going to set the node name to one that contains a regular expression so that any hostname that is certified by the puppetmaster and matches the regular expression can partake to provide virtual machine capabilities. The definition is then:

[gist id=3029150]

With your nodes defined in `site.pp` you will need to set some particular variables:

* Global:
* set `$public_interface` to `’eth0’`
* set `$private_interface` to `’eth1’`
* Controller-specific:
* set `$controller_node_address` to the IP address that you have set to `eth0`. In our case it’s `’’`
* set `floating_range` to `’’`. This will give you enough floating IP address in this test setup.
* Compute node-specific:
* set `libvirt_type` to `’qemu’`

Now either ensure the puppet agent is running on the controller node or run the puppet agent in the foreground. Once the puppet agent on the controller node contacts the puppetmaster it will install all the necessary services for an OpenStack controller.

## OpenStack Compute Node Setup
There is little here for you to do other than ensure that the puppet agent process is configured properly and can contact the puppetmaster. When each compute node first contacts the puppetmaster they will have issued a new certificate request and will wait until the puppetmaster signs these certificates. In order to sign these you can quickly do this by issuing the command:

[gist id=3029151]

Again as in the case of the all-in-one installation, the `nova-volumes` group is not setup so [follow the previous steps to setup is up](http://www.cloudcomp.ch/2012/07/puppet-and-openstack-part-one/). Once the puppet agent on each of the compute node contacts the puppetmaster it will install all the necessary services for an OpenStack compute node.

## Validating the Install
Once all services on all VMs have been installed you should be able to list that all nodes are operational. To validate this, execute the following:

[gist id=3029153]

You should see the following output:

[gist id=3029145]

You may want to run some VMs at this stage so you will need to import a VM image into glance. This is [detailed here](http://www.cloudcomp.ch/2012/07/puppet-and-openstack-part-one/).

## Next up
We’ve now successfully installed a clustered OpenStack using puppet. In the next article we’ll look at expanding this to combine both Foreman, puppet and OpenStack so that we can go from bare metal to a multi-node, clustered OpenStack installation. Stay tuned!

***Note***: If you want copies of the VMs used in this article ping [@dizz](http://www.twitter.com/dizz)

Puppet and OpenStack: Part One

In this guide we’ll explain how you can setup a simple OpenStack all in one installation using puppet. In this guide we’ll be using a virtual machine to simulate the hardware. We’ll use the same network configration as was used [in the article on Foreman](http://www.cloudcomp.ch/2012/06/automating-the-icclab-part-one/). We’ll also be using Ubuntu 12.04.

By default, nameserver and domain settings are automatically managed by `resolvconf` (this is due to the primary adapter being managed by DHCP). As puppet relies on the fully qualified host name of nodes its installed on, you should, if not using other means, configure `resolvconf` so that it does not overwrite your domain and nameserver settings. To do this edit `/etc/resolvconf/resolv.conf.d/head` and place the following content:

[gist id=3029170]

Of course feel free to use what ever other values you like to use. Performing this configuration will ensure that resolvconf always generates settings with your values.

If you are not using your own managed DNS server then you should place IP-host aliases in your `/etc/hosts` file. Here’s the relevant file entries used in this article:

[gist id=3029172]

Once you have the virtual machine installed, it might help you if you take a snapshot of the it at this stage so that you can roll-back to a fresh state.

## Install Puppet

Execute the following:

[gist id=3029174]

## Configure Puppet
In order to deploy OpenStack, we will be using puppet in both agent and master mode.

### Agent Configuration
Configure puppet agent. Edit `/etc/puppet/puppet.conf` so that it the `[agent]` section has the set values of:

[gist id=3029176]

### Master Configuration
Configure puppet master. Edit `/etc/puppet/puppet.conf` so that it the `[master]` section has the set values of:

[gist id=3029177]

***Note*** that this particular configuration will change when we integrate with Foreman in the article describing Foreman, puppet and OpenStack integration.

## Select Puppet Modules

We’ll use the [official Puppetlabs OpenStack modules](https://github.com/puppetlabs/puppetlabs-openstack). Install the prerequisites and checkout the OpenStack modules from Github:

[gist id=3029179]

Once done you’ll need to follow the setup instructions (they’re repeated here for completeness):

[gist id=3029180]

After executing these steps, the rake script will have placed other required puppet modules in `/etc/puppet/modules/`

## Assign the OpenStack Role
We now have to tell puppet that the current VM is to run as a complete OpenStack instance. To do this copy the example `site.pp` file to `/etc/puppet/manifests/` and then edit so that:

1. the node definition for the VM reads as `node /controller.cloudcomplab.ch/ {`
2. `libvirt_type` is set to `qemu`
3. if you want further logging information to help you, then set `$verbose` to `true`
4. you might want to specify a floating (i.e. static) IP range. For this setup you can add and set `floating_range` to `’’`

Once done, wait! This setup will run perfectly fine until you attempt to invoke the services of `nova-volume`. `nova-volume` is not fully setup as there is no LVM group (`nova-volumes`) setup. To set this up manually execute these steps as root:

1. `truncate -s 2052M /root/nova-vol-file`
2. Find the loop-back device associated with `nova-vol-fil`:

`losetup -f –show /root/nova-vol-file`

In this setup the value is `/dev/loop1`

3. Now, finally, create the LVM volume group:

`vgcreate nova-volumes /dev/loop1`

You may need to install the LVM tools: `apt-get -y install lvm2`

**Note** that this LVM mapping is reset on reboot.

Now that you have setup LVM and puppet to install OpenStack, do just that! Execute:

`puppet agent –no-daemonize –verbose`

You’ll see alot of output as puppet installs OpenStack but at the end of the process you will be able to access your OpenStack installation at ``

## Using OpenStack
Now that you’ve seen the shiny UI of OpenStack you will have noticed that there are no VM images to run. To get a VM image into OpenStack do the following:

1. Import the authentication credentials into your shell (puppet was kind enough to create these for you)

`source /root/openrc`

2. Download a VM image, CirrOS in this case:

`wget https://launchpad.net/cirros/trunk/0.3.0/+download/cirros-0.3.0-x86_64-disk.img`

3. Import the CirrOS image into glance:

`glance add name=”CirrOS 0.3 VM Image” disk_format=qcow2 container_format=ovf < cirros-0.3.0-x86_64-disk.img`

4. Go back to your web user interface and you will see that there is now a VM image to instantiate and execute.

By the way, the puppetlabs OpenStack github repository [has some decent documentation](https://github.com/puppetlabs/puppetlabs-openstack).

# Next up
We've now successfully installed an "all-in-one" OpenStack using puppet. In the next article we'll look at expanding this to a multi-node scenario. Stay tuned!

Automating The ICCLab: Part Two

In [the previous post](http://www.cloudcomp.ch/2012/06/automating-the-icclab-part-two/) we looked at getting Foreman setup and installed. In this post we’ll run through how you can provision bare metal hosts using Foreman. The first thing that we’ll need to do is configuring Foreman for provisioning! Don’t forget [a VM in which all of this is setup is available](http://www.cloudcomp.ch/wp-content/uploads/2012/06/ForemanV.ova).

## Configuring Foreman to Provision Bare Metal

You will now need to configure Foreman. To do this you need to create the following pieces of information:

0. **Architectures**

Here, if they are not already populated, you need to enter in the OS hardware architectures that are required for your infrastructure fleet. Common architectures include ‘i386’ and ‘x86_64’.

1. **Smart Proxies**

Smart proxies are the entities that interact with the core infrastructural services like DNS, DHCP, TFTP etc. When setting up your Foreman instance you need to tell Foreman where its location is by creating a new Smart Proxy entry and supplying its URL.

2. **Domains**

Here you need to enter in the domain name system information. All that is needed is the top level domain name (e.g. mydomain.com) and an identifier (e.g. “the main TLD”). If your smart proxy has DNS enabled then ensure the DNS value is populated. This value specifies what smart proxy DNS service to use when adding new DNS entries.

Once the smart proxy and DNS entries are entered, go back to the Smart Proxies section and click on ‘Import Subnets’. This will populate Foreman with the required networking information needed to allocate IP addresses against a newly provisioned machine.

3. **Subnets**

If you carried out the auto-import of subnets when setting up your smart proxy you shouldn’t need to change this, unless you want to add more or modify subnets.

4. **Environments**

Environments specify a group of puppet modules that can be applied against a newly provisioned host. To import default ones, `cd /usr/share/foreman` and execute `rake puppet:import:puppet_classes[batch] RAILS_ENV=production`. See the [Foreman wiki page](http://theforeman.org/projects/foreman/wiki/External_Nodes) for further details. You can also import existing environments, along with their associated classes, by navigating to the ‘Puppet Classes’ section and clicking on ‘Import new puppet classes and environments’.

5. **Installation Media**

To install an operating system you need to specify the location of where the source files will come from. Foreman supplies you with some installation media locations to begin with and you can add more either remote or local ones. A suggested minor edit with the default Foreman installation media is to change the mirrors that are closest to you.

**Tip**: Setup a caching server to cache all your downloaded operating system files (e.g. apt-cacher NG for Debian based systems; [puppet module here](https://github.com/lelutin/puppet-apt-cacher-ng)), otherwise you can install from local media locations, say, distributed over HTTP.

6. **Operating Systems**

In the case that you want to deploy Operating Systems to hosts, you will need to define the operating systems that you support. To do this create an operating system entry. Once done go to the Provisioning Template section.

7. **Provisioning Template**

The provisioning templates are key in the TFTP and DHCP boot process of your new host. These specify, depending on the OS automation suite (e.g. preseed, kickstart etc.), how the OS should be installed on the new host, how disk partitions should be setup, what OS packages should be installed etc.

An **important thing to do** here is to create a ‘Default PXE Menu’. This will allow for PXE boots with a fall back of boot by local disk, should a host have an OS already installed.

You need to specify the template for the appropriate operating systems you are supporting. In the case of Ubuntu OS’s you will need to go to each ‘Preseed’ entry and ensure that it is associated to the Ubuntu Operating systems. Once done go back to the Operating Systems entries and ensure that they are associated with the correct templates.

**Note**: A modification to the Ubuntu Default disk partition needs to be made to that a user prompt is avoided. To do this navigate to ‘Partition Tables’, select the ‘Ubuntu default’ and then add the following as a replacement:

[gist id=2888536]

8. **Host Group**

You can think of a Host Group as a profile or policy that is applied to a new host or group of hosts that is/are provisioned. Here selected puppet modules can be enforced on associated hosts to the Host Group. An example Host Group would be, say, an OpenStack Compute (Nova) Host Group. Any host associated with this group would assume the role of providing virtual machines for an OpenStack installation.

With the above items created you are now ready to create a Foreman/puppet managed node! Click on ‘Hosts’ and ‘New Host’. Most parameters will be pre-populated. The only parameter you need to supply is the new host’s MAC address.

### Potential Issues

1. If you come across this DNS related issue when creating a node:

[gist id=2888540]

First check the permissions of your `/etc/bind/rndc.key`. The foreman process needs to have read permissions to this. This is best done by adding the `foreman-proxy` user to the `bind` group (Thanks to Andreas N for [the suggestion](https://groups.google.com/forum/?fromgroups#!topic/foreman-users/TzmrYhoFKFI)!). E.g. on Ubuntu:

[gist id=2888549]

Secondly, a work around, although not elegant, is to disable the dynamic DNS functionality. To do this:

1. Go to ‘Domains’ and clear the value for ‘Dns’ and then save.
2. Edit `/etc/puppet/modules/common/foreman_proxy/manifests/params.pp` and set `$dns = false`.

2. If when provisioning a new host and the PXE boot process complains that it cannot download the pre-configuration file (say in the case of Ubuntu) and the URL is malformed (e.g. http://foreman:3000.mydomain.net), then ensure that you are accessing the Foreman front-end via the fully qualified hostname (e.g. http://foreman.mydomain.net:3000) and not just the host name (e.g. http://foreman:3000).

3. If you come across the following issue when creating a node:

[gist id=2888551]

The first solution is to [apply the changes that are present in this diff](http://theforeman.org/projects/foreman/repository/revisions/a7ad2c1c5e128347fea66478614d8784b222c2d9) that fixes [Foreman issue 1541](http://theforeman.org/issues/1451).
The last resort work around is to install the 0.5 `nightly` release of Foreman. You can select what release of Foreman to install by editing `/etc/puppet/modules/common/foreman/manifests/params.pp` and changing `$package_source`. 0.5 appears to be relatively stable with many [people in the Foreman IRC room](http://theforeman.org/projects/foreman/wiki/Support) using it.

## Integrating Physical Nodes with Foreman & Puppet
For Foreman to use puppet, puppet needs to send reports to Foreman. To do this a script is registered with puppet. Thankfully this is looked after by the previous install steps above. If you want to check the script out, it can be [found at the Foreman repository](https://github.com/theforeman/puppet-foreman/blob/master/templates/foreman-report.rb.erb) or just look in `/usr/lib/ruby/1.8/puppet/reports/foreman.rb`. The other step that the install looks after is registering the report with puppet via the puppet configuration file (`/etc/puppet/puppet.conf`).

At this stage you should have Foreman up and running and acting as not just a puppet master but as an ENC (External Node Classifier). This is looked after (this setup is automated) by yet another simple script (`/etc/puppet/node.rb`) that is registered with puppet. This script supplies node information from Puppet to Foreman.

## Possible Issues on Provisioned Nodes

1. If you are seeing **”hostname was not match with the server certificate”** in `/var/log/syslog` on your provisioned host then [please see this page](http://docs.puppetlabs.com/guides/troubleshooting.html). Normally you need to adjust the `server` parameter in the `/etc/puppet/puppet.conf` file.
2. If you see **”Error 400 on SERVER: Could not find node…”** in `/var/log/syslog` then you will likely have to adjust the variable `url` in `/etc/puppet/node.rb` and `$foreman_url` in `/usr/lib/ruby/1.8/puppet/reports/foreman.rb`.
3. If you see:

[gist id=2888556]

then check out [this article on disabling factsync](http://serverfault.com/questions/66124/puppet-0-25-0-problems-with-access-control-how-can-i-fix-them).

## Next up
The next post in this series will look into deploying OpenStack using puppet in a distributed (multiple nodes scenario) and then followed up with integration with Foreman. Stay tuned!

Automating The ICCLab: Part One

***Note:*** There are updated installation instructions for the new release of Foreman 1.0.x [in a complementing blog post](http://www.cloudcomp.ch/2012/09/installing-foreman-1-0-1/).

## Introduction
In the world of data centres, automation is not a nice-to-have it’s essential. Sure you could attempt some manual deployment and configuration if you’ve 2 machines to administrate or knock together a custom script in Perl or python if you’ve 30 machines. But what happens when this grows to thousands? This is where the likes of [Puppet](http://puppetlabs.com/), [Chef](http://www.opscode.com/chef/) and [CFEngine](http://cfengine.com/) come into play. These are the tools for managing deployment of cloud-scale data centres.

One of our current requirements here in the [ICCLab](http://www.cloudcomp.ch) is to design and implement scalable infrastructure automation. To do this we have some basic and simple requirements:

1. Toolset must be easy to use, extensible and have a minimal learning curve,
2. Toolset must allow for the deployment of software and the lifecycle management of that software,
3. Toolset must allow the provisioning of physical machines,
4. Toolset must be fully integrated.

Based on these requirements we have selected [Puppet](http://puppetlabs.com/) as the core of this toolset. [Puppet](http://puppetlabs.com/) will satisfy the first two requirements. To satisfy the third and in doing so the fourth, [Foreman](http://theforeman.org) will be selected as the means to provision physical servers.

Foreman is integrated with Puppet. It runs a puppetmaster server which once a physical server is provisioned, will deploy specified [Puppet modules](http://forge.puppetlabs.com/). Foreman itself has a user web front-end that talks to a backend service known as a smart proxy. The smart proxy provides a service that provides DHCP, DNS, Puppet, Puppet CA, and TFTP functionality. As well as the user web front-end Foreman also [has a useful web API](http://theforeman.org/wiki/foreman/API).

## Getting Started
To get started with Puppet and Foreman, we’re setting things up on a VM ([you can download it if you like](http://www.cloudcomp.ch/wp-content/uploads/2012/06/ForemanV.ova), user:fman password:fman) for flexibility. You can of course deploy Foreman to a physical machine but choosing a VM is solely to ease this walkthrough. To do this you can follow the following steps.

***Assumption***: An [Ubuntu 11.04 Server OS](http://releases.ubuntu.com/11.04/) running on [VirtualBox](http://virtualbox.org). Currently [there are issues with 12.04](http://theforeman.org/issues/1481) and Foreman that are related to DB schema migration.

### Foreman Virtual Machine Setup
Assign 2 NICs: one NAT’ed the other on a host-only network. The NAT’ed adapter will act as the public interface through which access to the internet is supplied. The host-only adapter will act as the private internal network in concertation with VirtualBox’s virtual switch (default name `vboxnet0`). The virtual switch `vboxnet0` is set with an IP address of `` and mask of ``. DHCP is disabled.

Install a Linux variant, in this case Ubuntu 11.04 “Natty Narwhal”, on to the virtual machine. During the install process choose to install the `openssh-server` so you can `ssh` into the virtual machine.

**Tip**: Most of these commands will require root privileges, so to save typing `sudo $COMMAND` run in interactive `sudo` mode:

[gist id=2888371]

Once the machine has booted update the VM’s packages to the latest:

[gist id=2888373]

During the install, the NAT’ed adapter will be setup. In order to complete the setup you will have to setup the host-only adapter also. To do this edit `/etc/network/interfaces`. The `metric` parameter is important here. Below is an example of how you can setup the host-only adapter, named `eth1`.

[gist id=2888052]

We need to allow traffic to pass to and from (packet forwarding) the host-only and NAT’ed adapters (`eth1` and `eth0`). To do this IP forwarding needs to be enabled:

[gist id=2888334]

You will need to setup iptables rules. In order to persist these rules over reboots, setup the following script:

[gist id=2888339]

Save this as `/etc/init.d/fwd-traff.sh`, make it executable and then set it to start on boot:

[gist id=2888354]

You should now execute the script this script in the case you do not reboot the node at this point.

You should setup your domain settings. By default in this setup, nameserver and domain settings are automatically managed by `resolvconf`. As puppet, hence Foreman, relies on the fully qualified host name of nodes its installed on, you should, if not using other means, configure `resolvconf` so that it does not overwrite your domain and nameserver settings. To do this edit `/etc/resolvconf/resolv.conf.d/head` and place the following content (here we’re using the google DNS resolver but you might want to use something else):

[gist id=2987340]

Otherwise you can edit `/etc/dhcp/dhclient.conf` and set the following:

[gist id=2987954]

Time synchronisation is important, especially for puppet (time drift and certificates do not mix well! ;-)). To look after this install `ntp`

[gist id=2987348]

Finally setup a port forwarding rule on the NAT’ed adapter (`eth0`) to the SSH service running on `eth1` in VirtualBox so that as a convenience you can ssh to the Foreman node using your own client and not the VM console.

### Installing Foreman
The most appropriate means to deploy foreman is of course via puppet! First install the necessary `git`.

[gist id=2888377]

Now install [puppet from puppetlabs.com](http://docs.puppetlabs.com/guides/puppetlabs_package_repositories.html#for-debian-and-ubuntu). This is done as we need the latest version, 2.7.x. In version 2.6.x, the default in Ubuntu 11.04, there is an Augeas bug that makes Foreman installation tricky.

[gist id=2888381]

You should also disable `puppetmaster` from `/etc/defaults/puppetmaster` as this will effect how apache executes.

[gist id=2988152]

Now, get Foreman. You should place this collection of puppet modules in the place where you store all you modules (e.g. `/etc/puppet/modules`). For this installation we’ve placed them in `/etc/puppet/modules/common`. Of course, you being a DevOps nut, you want to have all of these configurations under SCM (e.g. git, mercurial or subversion).

[gist id=2888384]

You will also need a DHCP, a DNS and the Concat module. The following modules do the job perfectly:

– [DHCP Module](https://github.com/GregSutcliffe/puppet-dhcp)

git clone git://github.com/GregSutcliffe/puppet-dhcp.git dhcp

– [DNS Module](https://github.com/GregSutcliffe/puppet-dns)

git clone git://github.com/GregSutcliffe/puppet-dns.git dns

– [Concat Module](https://github.com/GregSutcliffe/puppet-concat)

git clone git://github.com/GregSutcliffe/puppet-concat.git concat

***Note:*** As we’re installing onto Ubuntu, a Debian variant, there maybe some gotchas you need to read up on. See [README.debian](https://github.com/theforeman/foreman-installer/blob/master/README.debian).

You now have all the necessary Foreman modules. A quick directory listing should show the following:

[gist id=2888400]
***Foreman 0.4.2 only***: In the current Foreman installer there is a minor bug in `/etc/puppet/modules/common/tftp/manifests/install.pp`. To remedy this run the following `sed` script:

[gist id=2888404]

***Foreman 0.4.2 only***: You will also have to make the following addition in `/etc/puppet/modules/common/tftp/manifests/params.pp`:

[gist id=2888406]

Now, customise `/etc/puppet/modules/common/foreman_proxy/manifests/params.pp`. For this installation, change the following TFTP, DHCP and DNS variables as:

[gist id=2888416]

Also review the settings in `/etc/puppet/modules/common/foreman/manifests/params.pp`.

If you intend on running a DNS server for the subnet and wish to forward the DNS requests upstream to another DNS server then you will need to modify the DNS module file `/etc/puppet/modules/common/dns/templates/options.conf.erb` and insert the following `forwarders` clause to add the Google (8.8.8.*) and OpenDNS (208.67.222.*) upstream servers:

[gist id=2888418]

If you’re foreman does not have a domain name associated with it then you can quick supply one by adding a `domain` statement in `/etc/resolv.conf` (e.g. `domain mydomain.net`).
Also as you are using Foreman’s managed DNS server, you should make a new `nameserver` entry in `/etc/resolv.conf` that points to the DNS server’s IP address.

**Tip**: At this stage you can make a snapshot of the VM before installing Foreman so you can revert to it if your install does not work out.

Now it’s time to let puppet do the work! Execute the following:

[gist id=2888421]

Enable the foreman service:

[gist id=2888423]

Now start the Foreman server by executing:

[gist id=2888426]

Foreman will now be available on its URL serving from port 3000. Finally commit and optionally push your modified modules to your SCM repository. If you ever want to setup Foreman again on a fresh machine all you’ll now ever need to do is to install puppet, check out foreman from your repository and apply the relevant modules.

#### Possible Installation Issues

If you experience issues on executing this command and you want more information, simply add the flags `-v -d` to the `puppet` command.

1. You might see the error:

[gist id=2888441]

You can safely ignore this but if you want to remove the error it simply create the directory.

2. If you encounter an error telling you that “no fqdn” is available to `facter` then [see this useful article](http://stackoverflow.com/questions/7780322/puppet-facter-could-not-retrieve-fact-fqdn-how-to-fix-or-circumvent) on how to resolve the issue.

3. ***Foreman 0.4.2 only***: If you get the following error output:

[gist id=2888442]


[gist id=2988063]

then your issue lies with Foreman and a specific issue with Augeas attempting to add certain `sudoers` permissions for Foreman and persist them. To resolve this you will need to comment out the `augeas` entry in `/etc/puppet/modules/common/foreman_proxy/manifests/config.pp` and add the following entry:

[gist id=2888445]

The contents of the template (located at : `/etc/puppet/modules/common/foreman_proxy/templates/foreman.erb`) should be:

[gist id=2888449]

**Note**: With each change(s) reapply the foreman puppet command as above.

#### Customising Foreman
There are a number of customisations that you might want to make. Don’t edit the Foreman configuration files directly. Edit the `param.pp` files and then use `puppet apply`.

1. Enable Foreman front-end authentication:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
`$authentication = false`
`$authentication = true`

2. In a trusted environment and for troubleshooting, disable SSL:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
`$ssl = true`
`$ssl = false`

3. By default Foreman uses your fully qualified domain name. If you do not want this and say just the hostname then:

In `/etc/puppet/modules/common/foreman/manifests/params.pp`
`$foreman_url = “http://${::fqdn}”`
`$foreman_url = “http://${::hostname}”`

4. By default Foreman uses a sqlite3 backend. You can change this. See [the following Foreman article](http://theforeman.org/projects/foreman/wiki/Database_configuration).

If you apply such changes make sure to re-run the foreman puppet command (as above):

[gist id=2888421]

## Next up
Things are hotting up in the area of automation. Just recently EMC announced their new [open source provisioning framework, Razor](https://github.com/puppetlabs/Razor), built on Puppet. There is also [support added within Foreman](http://cloudprovisioning.wordpress.com/) for provisioning VMs via oVirt and Amazon EC2.

Big thanks goes out to [all those that supplied suggestions](https://groups.google.com/forum/?hl=en&fromgroups#!topic/foreman-users/-ASnNeC5SyE) to improve this guide!

In the next blog post we’ll look at configuring and provisioning bare metal hosts with Foreman. Stay tuned!

« Older posts Newer posts »