In a previous blog post, we showed how “bringing the code to the data” can highly improve computation performance through the active storage (also known as computational storage) concept. In our journey in investigating how to best make computation and storage ecosystems interact, in this blog post we analyze a somehow opposite approach of “bringing the data close to the code“. What the two approaches have in common is the possibility to exploit data locality moving away in both cases from the complete disaggregation of computation and storage.
The approach in focus for this blog post, is at the basis of the Alluxio project, which in short is a memory speed distributed storage system. Alluxio enables data analytics workloads to access various storage systems and accelerate data-intensive applications. It manages data in-memory and optionally on secondary storage tiers, such as cheaper SSDs and HDDs, for additional capacity. It achieves high read and write throughput unifying data access to multiple underlying storage systems reducing data duplication among computation workloads. Alluxio lies between computation frameworks or jobs, such as Apache Spark, Apache MapReduce, or Apache Flink, and various kinds of storage systems, such as Amazon S3, OpenStack Swift, GlusterFS, HDFS or Ceph. Data is available locally for repeated accesses to all users of the compute cluster regardless of the compute engine used avoiding redundant copies of data to be present in memory and driving down capacity requirements and thereby costs.
For more details on the components, the architecture and other features please visit the Alluxio homepage. In the rest of the blog post we will present our experience in integrating Alluxio on our Ceph cluster and use a Spark application to demonstrate the obtained performance improvement (the reference analysis and testing we aimed to reproduce can be found here).
The SI represents the computer science profession in Switzerland: professionals with a university degree or equivalent qualification, educational and research institutes, and companies that contribute to computer science. It networks its members and represents their interests in politics, education, business and research.
The 30th birthday of the Swiss Informatics Society will be holding a celebration event this Tuesday () at HES-SO in Fribourg. The programme is available here.
Including the inaugural Meeting of the Swiss AIS Chapter, from 8.45 AM to 10.45 AM there will be six parallel workshops: Green IT, Open Government Data, Cloud Computing in CH, Usability in der Praxis and Entertainment Computing.
The Cloud Computing in CH workshop is chaired by ICCLab and it will cover the following track theme:
Cloud Computing is transforming the IT industry, and this concerns a high-tech country like Switzerland in particular. The resulting potentials and risks need to be well understood in order to be able to fully leverage the technical as well as economical advantages. This workshop will provide an overview of current technological and economical trends with a particular focus on Switzerland and its currently released Federal Cloud Computing strategy.
The speakers will be: Christof Marti (ZHAW, moderator), Peter Kunszt (Systemsx), Markus Brunner (Swisscom), Sergio Maffioletti (University of Zurich), Michèal Higgins (CloudSigma) and Muharem Hrnjadovic (RackSpace).
Final agenda of this workshop will be posted soon.
On the 19th February the 2nd Swiss OpenStack User Group Meeting took place. One of the presentations was held on Ceilometer by Toni and Lucas from the ICCLab. They talked about the history, the current and future features, the architecture and the requirements of ceilometer and explained how to use and extend it. You can take a look at the presentation here:
We, the ICCLab, are proud to announce that the past Presidential Conference of the Swiss Informatics Society accepted our proposal (slides) for setting up a Special Interest Group in Cloud Computing.
The SIG is currently being formed. If you wish to participate and influence the future of Swiss Cloud Computing in this context please don’t hesitate to contact us. Any active participation is more than welcome.
The European commission recently released the outcomes of a number of cloud computing expert meetings in the form of a report. Members of the ICCLab have contributed to this report, “A Roadmap for Adanced Cloud Technologies under H2020“. It builds and follows on from previous reports (see previous summary article here). The report identifies 10 main technological topics for areas of research in cloud computing and categorises them into three topics: immediate, sustainability and game-changing. This report is especially useful for those wanting to understand where certain priorities lie for the EC’s H2020, the successor to the FP7 research and innovation programme.
Many people think it maybe an unnecessary burden to set up a monitoring system for their infrastructure. However this, when it comes to an OpenStack installation should be considered indispensable and required. Knowing which resources are used by which VMs (and tenants) is crucial for cloud computing providers as well for their customers from billing and usage perspectives.
Customers want to be sure they get what they pay for at any time whereas the cloud provider needs the information for his billing and rating system. Furthermore this information can be useful when it comes to dimension and scalability questions.
Requirements for OpenStack monitoring
For monitoring an OpenStack environment there are different requirements:
An OpenStack monitoring tool must be able to monitor not only physical machines but also virtual machines or network devices.
The information of the monitored resources must be assignable to its tenant.
The metered values must be collected and correlated automatically
The monitoring tool must be as generic as possible to ensure support of any device.
The monitoring tool must offer an API.
Architecture Monitoring Tool
There exist a lot of tools for network and server monitoring like Nagios, Zabbix and Munin. Most of them do not easily support OpenStack monitoring.
Zenoss is one of the few monitoring tools that supports an integration for OpenStack. It is possible to download and install an extension for OpenStack monitoring (https://github.com/zenoss/ZenPacks.zenoss.OpenStack). Unfortunately the latest version of this extension does only support the OpenStack API Version 1.1. The Folsom release ships with an OpenStack API version 2.0. The extension allows Zenoss to collect data only from a single tenant. That is not good enough because we need some more data to do rating and billing.
Another promising monitoring tool will be included in the upcoming OpenStack release Grizzly (March 2013) and is known as Ceilometer. It will be part of the OpenStack core. Ceilometer makes it easy to monitor VMs belonging to a tenant. What Ceilometer cannot offer at the moment is physical device monitoring.
After an evaluation we decided to extend Ceilometer to monitor the physical devices as well. With this extension Ceilometer will be able to monitor the whole OpenStack environment of the ICCLab and provide the data for further systems like the billing module.
The ICCLab presented at SwiNG SDCD 2012 on how you can easily provision bare-metal physical servers. This presentation, “From Bare-Metal to Cloud” was an updated version of the presentation that was made at the EGI Technical Forum in Prague. The slides can be viewed below or downloaded from here.
As well as presenting the ICCLab was part of a discussion panel on the role of Cloud Computing and academic research. On the whole, it was a very interesting and rewarding event.
Willy Müller from the Swiss Federal Government visited the ICCLab and provided invaluable insights into the Swiss federal Cloud Computing strategy. Amazing to see the actual advancement of the Swiss approach. Another poof of Switzerland’s excellent sense of innovations and forward looking approach way beyond the financial and clock-maker industry.
Die Bundesverwaltung möchte eine eigene eGovCloud Infrastruktur aufbauen und hat dazu eine Strategie in die Vernehmlassung gegeben. Willy Müller, wird im Rahmen dieses Kolloquiums wichtige Eckpunkte und Herausforderungen der Strategie vorstellen.