In some of our project work, we need to understand the energy consumption of our servers. One basic test we performed was to determine how server power consumption increases with load. In this blog post we document the results we obtained.
In some of our project work, we need to understand the energy consumption of our servers. One basic test we performed was to determine how server power consumption increases with load. In this blog post we document the results we obtained.
Click on one of the links below to jump to the relevant section of this page, or just scroll down if you want to peruse our outputs.
Open Access Preprints
This list may be out of date. Please refer to the arXiv search for all results first-authored by SPLab staff: SPLab preprints.
In addition to these preprints, please enjoy our ACM publications through the Author-Izer service.
Towards Quantifiable Boundaries for Elastic Horizontal Scaling of Microservices
Practical Tooling for Serverless Computing
Cloud Robotics: SLAM and Autonomous Exploration on PaaS
Selected Talks and Public Appearances
Books and Book Chapters
Journals and Magazines
Conference Publications
Standard Specifications
Theses, Student Papers
Lab Reports
COSBench is a tool developed by Intel for benchmarking cloud object storage services.
Here’s a brief video showing some functions of the web interface.
For more details, please refer to the COSBench user guide.
During his travel in USA, in Jun 2014, the researcher Antonio Cimmino (Zurich University of Applied Science, ICCLab) had the opportunity to visit the MIT SENSEable City Lab, in Cambridge Massachusetts USA, invited by the Director Dr. Carlo Ratti.
The SENSEable City Lab is radically transforming the way the cities are described and understood; alongside the tools they use to design them and impact on their physical structure. Studying these changes from a critical point of view and anticipating them is the goal of the SENSEable City Laboratory, a new strong research initiative at the Massachusetts Institute of Technology.
InIT Cloud Computing Lab (ICCLab), is deeply involved in the Future Internet Public Partnership Programme (FI-PPP) of the European Commission which includes in its objectives enabling infrastructures for the internet and use cases for smart city sectors.
It is therefore envisaged that in future the initiatives of the two Laboratories, might have some common activities and collaborations at same international events.
Many thanks to the staff of Carlo.
Many large enterprise applications rely heavily on the underlying information stored in databases. Most of these are used every day via the web browser by hundreds of users, or run calculations in a batch mode. These actions are database intensive. As long as each resource is on premise, communication speed to those databases is mostly not an issue because data center internal network latency and bandwidth is good enough.
Modern cloud offerings allow a new flexibility for companies to plan and scale their datacenters. The data- and compute services do not need to remain in the on premise datacenter, but can be spread in an internal private cloud or externally among one or many public cloud providers. This new concept challenges the traditional IT environment and provides an unprecedented flexibility that pushes traditional data and compute services to their limit.
The hybrid cloud concept involves the usage of internal resources as well as cloud-based resources in case of bursting, which happens when internal resources are overloaded. We conducted a study of the feasibility of using public cloud resources in different use cases for large enterprises, especially in cases where the application computing core has to remain on premises (for security reasons typically) and the database is moved to the cloud.
One reason to go to a public cloud is to lower operational effort. This can be done by letting the cloud provider operate the infrastructure. The enterprise just uses it as a service and take advantage of the cloud providers’ economics of scale. This would be mainly feasible for non-production applications, where performance and data security are not the main focus.
For short tests and new setups of small applications, the cloud can help to keep CAPEX low, as no hardware needs to be acquired. This saves money when resources are only needed for short time periods.
For performing benchmark tests on different set ups a simple java web application was created. This application wraps the TPC-C implementation of the OLTP-Benchmark and the Apache JMeter application. The tests executed were the TPC-C standard industry benchmark tests for database and custom tests with JMeter against a database structure found in one of SwissRe’s applications and running the most usual queries made by this application. The TPC-C test simulates the OLTP workload of an artificial wholesale supplier company. The test consists of five types of transactions: New order, Payment, Order-status, delivery and stock-level transaction. The most important type is the new order, which enters a complete order in a single database transaction. It was designed to simulate the variable workload found in productive OLTP environments.
Tests done in an enterprise environment showed that the internal database performance is still largely better than databases in the best known public cloud providers when the computing remains local. Going through the internet to issue DB requests simply affects the requests’ throughput by multiple order of magnitude and renders the usage of cloud databases impractical when the computing resources are not located on the same Cloud.
TPC-C Throughput comparison between local Oracle instance and remote MySQL at cloud providers with the application core on premises
As most large enterprises already have a sophisticated database and application server infrastructure, changing to a public cloud offering can be challenging and expensive.
Using the given hardware and adding an IaaS and possibly PaaS layer would provide the flexibility of a cloud with no drastic performance impact. The existing database offering – which is often Oracle or DB2 – could be extended with a cheap MySQL alternative. Nonetheless a high performance offering close to the application improves the overall performance of heavy database relying applications drastically.
Some cloud providers allow a billing concept where a customer only pays for the hour when an actual request is made. This allows running applications in the public cloud that are not very database intensive and not often used (as an example a Ski Event Registration Application). Through this model no internal hardware is required and the costs of the application infrastructure can be kept low.
In a hybrid cloud setup, development instances could also easily be deployed to a public cloud. The productive instance could be run in the private high-performance cloud.
For short bursts in the private cloud environment, it is currently not very reasonable to add additional compute power from an external provider when the working data is in the on premise environment, as the performance impact of the latency and throughput is huge.
We are changing our ceilometer backend from mysql to mongodb in one of our experimental Openstack installations. The reason for this change is that mongo seems to deliver better ceilometer performance than mysql; further ceilometer data structures are a more natural fit with mongo (and indeed, this is largely where they came from). This can be seen in a typical record below which is clearly hierarchical and contains so-called embedded documents (in mongo terminology):
"_id" : ObjectId("53bbe7ea926fc4597b42aafc"), "counter_name" : "instance", "resource_metadata" : { "status" : "active", "display_name" : "Test", "name" : "instance-00000001", "image" : { "id" : "bdfaab74-6542-4cbb-94f1-5306662208a7", "name" : "cirros-0.3.2-x86_64-uec" }, "host" : "7c261f3a33c099538d448be797e5ce0c0d8cf8bf9f75dd59ce04df86", } ...
Mongo natively provides support for queries of data structured in this fashion. More specifically, mongo enables data at different levels of the hierarchy to be queried – something which is difficult in SQL.
In python, this can be done simply as follows:
query = [{'field': 'timestamp', 'op': 'gt', 'value': date},
{'field': 'metadata.status', 'op': 'eq', 'value': 'active'}]
sample_list = ceilometer.samples.list(meter_name='instance', q=query)
The interesting point, which we did not understand clearly until now, is that ceilometer with a mongo backend supports exactly this type of query. Thus, the following command line query can obtain all instances that were active for a certain period of time:
ceilometer sample-list -m instance -q “timestamp>date; metadata.status=’active’”
Then ceilometer will return all the samples of instances active in this time range.
+--------------------------------------+-----------+-----------+--------------------+ | Resource ID | Name | Type | Timestamp | +--------------------------------------------------+----------+---------------------+ | 7535b9f6-01a6-410e-980d-338031e7a2c4 | instance | instance | 2014-07-09T09:30:05 | | 7535b9f6-01a6-410e-980d-338031e7a2c4 | instance | instance | 2014-07-09T09:20:05 | | 7535b9f6-01a6-410e-980d-338031e7a2c4 | instance | instance | 2014-07-09T09:10:04 | +--------------------------------------+-----------+----------+---------------------+
Querying ceilometer with a mysql database in this fashion results in an error.
Ongoing adoption of Cloud Computing has lead to an increase in the level of complexity of the existing services and inherently increasing risks for companies willing to migrate in order to benefit from this new paradigm. Unfortunately, one area that has not been properly engaged is Incident Management.
Cloud Incident Management is a new research direction which focuses on conducting forensic investigations, electronic discovery (eDiscovery), and other critical aspects of security that are inherent in a multi-tenant, highly virtualized environment, along with any standards that need to be followed.
An Incident is an event which occurs outside the standard operation plan and which can lead to a reduction or interruption of quality of service. Incidents, in Cloud Computing, can lead to service shortages at all infrastructure levels (IaaS, PaaS, SaaS).
Incident Management provides a solid approach to address SLA incidents by covering aspects pertaining to service runtime in cloud through monitoring and analysis of events that may not cause SLA breaches but may disrupt service execution, or by covering aspects related to security by correlating and analysing information coming from logs and generating adequate corrective responses.
More information soon to come…
We are working on upgrading some of our servers from Havana to Icehouse. One of the reasons for doing this is that the performance of Ceilometer on Havana is underwhelming. Part of the move involves moving to a mongodb backend (default in Mirantis Openstack 5.0) but we also want to keep our energy consumption data from our Havana system. Consequently, we have a data migration issue from Havana/mysql to Icehouse/mongo. Here, we describe how we did this migration. (Note that as this is just a small experimental system, we don’t have important user and account data to transfer).
We’ve given some impressions of IEEE Cloud over the last couple of days. The final day definitely had less energy with some no-shows for presentations and the audiences being generally small – getting to/from Alaska is not so easy so I guess some folks are already on their way home.
That said, there were a few interesting talks today. One talk from guys who are just down the road from us was given by Luca Gherandi from ETH on a PaaS platform dedicated to robotics applications called Rapyuta. This came out of an FP7 project and includes capabilities such as computer vision, mapping, motor control etc and addresses the split between capabilities on device and in-cloud. He focused on a particular approach to make it easier to configure.
Another talk by Byung Chul Tak arising from an IBM/facebook collaboration described some work they have done to facilitate easier migration of applications to the cloud. The problem they focus on relates to basic IP address issues – noting that it’s often problematic to simply change the IP addresses of the hosts that are running a complex application. In their AppCloak system, they make changes to system libraries to intercept any network calls and substitute the new (cloud) IP address with the old IP address. A nice trick, but it does look like an ugly patch which of course will lead to an even greater fragmentation in address space and overall complexity if it does see any widespread use.
Another interesting talk involving a Purdue/VMWare collaboration focused on understanding VM requirements in detail through log analysis. Specifically, they wanted to understand memory consumption of an application running in a VM through the rich set of log data generated within VMWare in order to predict how much memory a new instance would require. The managed to identify the most critical parameters which influence this and developed good predictors of memory consumption.
Overall, we were pleasantly surprised with the conference. While the conference was generally too long and surprisingly there seemed to be no social media dialogue around it, there were some interesting folks saying interesting things. From our reporting, it’s obvious that we found some of the industry talks a bit more interesting, but perhaps that’s our bias; also, it probably does not make it clear that the level of industrial engagement was generally not so high with a notable absence of the big players in this space.
Big Data was certainly a prevalent theme throughout the conference, with a lot of the talks and sessions focused on Map Reduce and large data management problems. For a cloud conference, there seemed to be very little talk of SDN and even technologies such as Openstack hardly featured. Generally, there was a strong bias towards the classical world of academic publishing and the science perspective rather than engineering; it was interesting to see that there was a big data hackathon and that they are quite open to the introduction of new types of sessions next year.
Would we go again next year? Alaska is a very long way from home – I’m not sure that it makes so much sense to go so far for a few days of conference attendance, but certainly east coast US could make sense and depending on how it’s organized, it could be an attractive proposition.
Recent Comments