OpenStack on SmartOS

SmartOS is an open source type 1 hypervisor platform based on Illumos, a descendant of OpenSolaris, and developed by Joyent. SmartOS is a live operating system, meaning that can be booted via PXE, USB or an ISO image, and runs entirely from memory, leaving the full space on the local disk to be used for virtual machines. This type of architecture makes SmartOS very secure, easy to upgrade and recover. Given its performances and reliability, in the context of the Mobile Cloud Networking project, SmartOS has been chosen to support telco-grade workloads and provide carrier-grade performances.

SmartOS as Cloud OS

Cloud providers must be able to offer a single server to multiple users without them noticing that that they are the only user of that machine. This means that the underlying operating system must be able to provision and deprovision, i.e. create and destroy, virtual machines in a very fast seamless way; it should also allocate physical resources efficiently and fairly amongst the users and should be able to support multithreaded and multi-processor hardware. Lastly, the operating system must be highly reliable and, in case something doesn’t work as it should, it must provide a way to quickly determine what the cause is. A customer of the cloud provider will also expect the server to be fast, meaning that the observed latency should be minimal. The provided server should also give the flexibility to get extra power when needed, i.e. bursting and scaling – and be secure, meaning that neighboring users must not interfere with each other.

Thanks to the Illumos inheritance, SmartOS presents a set of features that address these needs and make it a perfect candidate as a truly Cloud OS:

• OS Virtualization. SmartOS offers both container-based virtualization, i.e. a lightweight solution combining resource controls and Solaris zones, and KVM virtual machines, a full, hardware-assisted virtualization solution for running a variety of guest OS’s, including Linux and Windows. Brendan Gregg of Joyent wrote a post comparing performances between OS virtualization techniques.
• ZFS and I/O throttling. ZFS combines file system and logical volume manager in a single feature. Key characteristics of ZFS are fast file system creation and data integrity guarantee. ZFS also includes storage pools, copy-on-write snapshot creation and snapshot cloning. Joyent further extended SmartOS adding disk I/O throttling. This feature, particularly interesting for a Cloud OS, overcomes a drawback in classic Solaris where a zone or application could effectively monopolize access to local storage, causing performance degradation for other applications or zones. With this new feature all zones/applications are ensured to get a reliable turn at reading/writing to disk.
• Network Virtualization. SmartOS makes use of Crossbow to provide a network virtualization layer. Crossbow is fully integrated with the virtual machine administration tool of SmartOS, i.e. vmadm, and allows each virtual machine can get up to 32 virtual network interfaces (VNICs). But with this ability to offer so many VNICs, how can we supply sufficient bandwidth? As SmartOS is a Solaris derivative it can leverage advanced networking features such as multipath IP (IPMP). Operating at a lower level, at the data link level, SmartOS has the possibility of levering data link multi-pathing (DLMP), which is close to trunk aggregation.
• Observability with DTrace. DTrace is a performance analysis tool included by default in different operating system, amongst them Illumos and Solaris and therefore SmartOS. DTrace, short for Dynamic Tracing, can instrument code by modifying a program after it has been loaded into memory. DTrace is not limited to use with user-space application, but can be used to inspect the OS kernel and device drivers. In SmartOS, DTrace can be used to analyze and troubleshoot issues across all zones in a server or within an entire datacenter.
• Resource control. Resource control is an essential part of the Container-based virtualization. In SmartOS there are two methods to control resource consumption: fair share scheduler and CPU capping. Fair share scheduler allows the administrator to set a minimum guaranteed share of CPU, to ensure that all zones get the a fair share of CPU when the system is busy. CPU capping sets a limit on the amount of CPU that a particular user will get. In addition to these two methods, Joyent added a CPU bursting feature that let administrators define a base level of CPU usage and an upper bound and also limit how much time a zone can burst.
• Security. Thanks to the Illumos and Solaris inheritance, SmartOS offers a high level of security. Zones are complete separate environments and activity in one zone will not affect neighbouring zones on the same server. Data security is also guaranteed through the use of zones and ZFS file systems.
• Reliability. SmartOS offers Fault Management (FMA) and Service Management Facility (SMF) that makes it more reliable. The Fault Management feature helps detect, report and diagnose any fault or defect that can occur on a SmartOS system. The Service Management Facility (SMF), another feature SmartOS inherits from Solaris, introduces dependencies between services – meaning that the system will ensure that all services a particular service depends on are up and running before starting it, parallel starting and automatic restart upon failure to allow fast booting time and service recovery, and delegation of services to non-root users to limit the privileges of a certain service. Complementing these is the ability of highly available load balancing with the virtual router redundancy protocol (VRRP). This is an additional feature that needs to be installed on SmartOS, yet it provides a means to implement hot-failover via virtual IP sharing. This is very similar to the combination of pacemaker and corosync. 

OpenStack on SmartOS

Given the set of features that makes SmartOS the ideal Cloud OS, it seems only logical to combine it with OpenStack to provide a reliable, high-performance cloud platform. This idea was already blueprinted within OpenStack and some preliminary work has been already carried out by Thijs, Andy and Hendrik.

The existing work has now been further extended and the code has been updated to the latest OpenStack release, Grizzly, and is available on github. At the moment, the nova-compute service is running on SmartOS, being able to instantiate virtual machines, both Container-based and KVM. The nova-network service is still a work-in-progress and further work needs to be carried out in order to make SmartOS fully Quantum compatible.

Further and interesting work include enabling the integration of OpenFlow controllers (e.g. Ryu, trema, floodlight). This coupled with IPMP and DLMP will make SmartOS truly a high performance virtualization platform. With high availability features of SmartOS interesting and valuable reliable compute services can be offered both with container and KVM virtualization techniques. Having all these capabilities are immensely useful however in order to truly manage this hypervisor platform, in-depth monitoring will be required and this is where DTrace will be leveraged as a configurable source of system metrics. These metrics can be supplied to the OpenStack Ceilometer monitoring system, for both performance and billing purposes. Whereas we’re currently focused on the compute and networking capabilities, SmartOS’s ZFS storage capabilities will be leveraged in both providing block-type storage services.