Hunting down memory leaks

I assume you are confident using a shell, installing software and generally building software from source.

While writing C code for a networking library I did some simple stability tests by setting up a simple server replying to a client query, no magic at all just hard coded strings. Obviously in such situations you will a have an eye on a process monitor like htop.

So I just ran the server and three clients in a endless loop:

while true; do binary; done

server and three clients

A quick look in htop revealed that my code leaked memory, not much per loop (around 20-30 Bytes) but still: In code which shall be once shipped in productive environment this is fatal. You can easily recognize such problems by looking at the RES column, if this value increases without clear reason you have somewhere a memory leak.

htop

For this type of problem you take valgrind. But since valgrind warns that their support on Mac OS X being broken I had to switch over to Linux. For such cases I recommend the Fedora Security Spin which comes with a huge load of tools aimed at security, auditing, research, rescue and obviously developper.

But before analyzing you need to tweak a few things: If you compile third party libraries by yourself consider passing the compiler flags -g -O0 to make sure the compiler produces debug symbols and doesn’t optimize too much making it harder to find the leaks. So make sure you invoke the configure script as follows:

make clean
./configure CFLAGS="$CFLAGS -g -O0" CXXFLAGS="$CXXFLAGS -g -O0"
make all
sudo make install

And evidently your code has also to be built with debug symbols and optimizations turned off, I’ll show you how this looks like in my Makefile:

CC = clang
BASICOPTS = -g -O0
CFLAGS = $(BASICOPTS) -pedantic -Wall -std=c11

Using -pedantic -Wall allows the compiler to warn you about every little mistake you write, always a good idea. Personally I recommend clang since it produces better error output.

Then you are ready to go hunting memory leaks! Start valgrind as following:

valgrind --tool=memcheck --leak-check=full 
    --show-possibly-lost=no your-binary

Then after one I killed my server binary with ctrl-c and got a nice output:

==3955== 
==3955== HEAP SUMMARY:
==3955==     in use at exit: 58,693 bytes in 76 blocks
==3955==   total heap usage: 162 allocs, 86 frees, 104,671 bytes all
ocated
==3955== 
==3955== 25 bytes in 1 blocks are definitely lost in loss record 13 
of 71
==3955==    at 0x4A06409: malloc (in /usr/lib64/valgrind/vgpreload_m
emcheck-amd64-linux.so)
==3955==    by 0x4EB2717: zframe_strdup (zframe.c:246)
==3955==    by 0x4014B9: _recv_message (kt_server.c:119)
            [snip] 
==3955== 36 bytes in 1 blocks are definitely lost in loss record 15 
of 71
==3955==    at 0x4A08121: calloc (in /usr/lib64/valgrind/vgpreload_me
mcheck-amd64-linux.so)
==3955==    by 0x4EB2117: safe_malloc (czmq_prelude.h:445)
==3955==    by 0x4EB21BB: zframe_new (zframe.c:59)
==3955==    by 0x4EB231A: zframe_recv (zframe.c:115)
==3955==    by 0x4EB780D: zmsg_recv (zmsg.c:101)
==3955==    by 0x40144E: _recv_message (kt_server.c:115)
==3955== 
==3955== LEAK SUMMARY:
==3955==    definitely lost: 61 bytes in 2 blocks
==3955==    indirectly lost: 0 bytes in 0 blocks
==3955==      possibly lost: 1,256 bytes in 12 blocks
==3955==    still reachable: 57,376 bytes in 62 blocks
==3955==         suppressed: 0 bytes in 0 blocks
==3955== Reachable blocks (those to which a pointer was found) are n
ot shown.
==3955== To see them, rerun with: --leak-check=full --show-reachable
=yes
==3955== 
==3955== For counts of detected and suppressed errors, rerun with: -v
==3955== ERROR SUMMARY: 14 errors from 14 contexts (suppressed: 2 fro
m 2)
Killed

Which shows I’m leaking memory in my code in two places. Reading the stack-trace when we go up a few stack frames while skipping the internal calls of zeromq we see two matching lines: kt_server.c:115 and kt_server.c:119. Let’s first tackle the bigger leak at kt_server.c:119:

msg->msgData = zframe_strdup(zmsg_pop(m));

That for I have to look closely what my code does and consult the API reference of czmq:

//  Remove first frame from message, if any. Returns frame, or NULL.
// Caller now owns frame and must destroy it when finished with it.
CZMQ_EXPORT zframe_t *
    zmsg_pop (zmsg_t *self);

And that’s what I forgot: “Caller now owns frame and must destroy it when finished with it.” I simply popped memory but didn’t care about freeing it. After changing the line to

msg->msgData = zmsg_popstr (m);

I was left with the first memory leak at kt_server.c:115. My code calls _recv_message() and receives a struct with a char* to the received message. Then I simply added a new response message and called _send_message():

msg = _recv_message ();

char *response = malloc(sizeof(char) * 128);
memcpy (response, "Hello World", 12);
printf ("Request: %sn", msg->msgData);
msg->msgData = response;

_send_message (msg);

The problem here is also pretty obvious: char *msgData formerly pointing to the received message is newly pointing to my freshly malloc-ed memory without freeing the old memory. So adding free (msg->msgData); solved this leak. I’ll leave figuring out where this statement goes in as an exercise to the reader.

Also an easy mistake is to forget freeing memory in _send_message() since zframe_send() requires you to destroy the passed zframe_t.

int _send_message (message_t msg)
{
    zframe_t *frame_reply = zframe_new (msg->msgData,
        strlen(msg->msgData));
    zframe_send (&frame_reply, sock, ZFRAME_REUSE);
    zframe_destroy (&frame_reply);
    free (msg);
    return 0;
}

Did you see what’s wrong here? Let’s have a look at the struct message_t which I simplified down to the relevant:

typedef struct message_t {
    char *msgData;
} message_t;

Correct: I call free (msg); which indicates that msg is a struct on the heap (allocated by malloc()) but the member char *msgData is simply a pointer on an other memory which waits to be free’d too. Let’s assume I was tired and put it after the first free:

free (msg);
free (msg->msgData);

What would happen? Well yes, your code would probably just segfault. But why? msg points after the free() to a probably otherwise used memory or is even NULL. The manpage of free(3) clearly says that ” … the behavior is undefined if the memory area referred to by ptr has already been deallocated … ” so eventually I cannot access msgData anymore. Correct would be to deallocate the memory from inside out:

free (msg->msgData);
free (msg);

A final run of valgrind shows now: My code is memory leak free!

using valgrind in my VM

Toni Zehnder

Toni Zehnder is currently in the sixth semester of his Bachelor of Engineering studies at ZHAW.

After a first qualification as plant layout designer he decided to move on in the academic area  and now studies Information Technology at ZHAW.

As part of his course of studies he joined the ICCLab as research assistant.  His research is dedicated to cloud computing infrastructure monitoring.

 

Future Internet Assembly (FIA Athens 2014)

The 11th Future Internet Assembly  will take place from 18 to 20 March 2014 in Athens, Greece.

This year the focus will be on “Reshaping the Future Internet infrastructure for innovation”.

Open call has been launched to select the best  9 FIA working sessions for the conference, selecting the proposals that cover aspects related to:

  • The new Internet technological landscape based on network/cloud integration through Software Defined Networking (SDN),  Network Functions Virtualization (NFV), and innovative software and services that enable application innovation;
  • The contribution of  the EU National Research and Education Networks (NRENs) developments in the SDN/NFV domain;
  • The role of SDN/NFV in i) building Internet applications of major impact (e.g., social networks, open data, big data analysis, etc.) with virtual services capabilities; ii) enabling  the reduction in resources used (energy efficiency, reduction of raw-materials, etc….); iii) fostering the emergence of open platforms to create downstream markets for third party developers;
  • The EU Public-Private-Partnerships (PPPs) and how they are positioned relative to these developments and relevant requirements towards demonstration and test-beds in Europe/globally.

These sessions should:

  • Contain new and forward-looking ideas;
  • Feature a diverse array of presenters and experiences;
  • Introduce visionary ideas and practices which will inspire audience;
  • Promote cross cutting approaches and technologies to attract stakeholders and entrepreneurs;
  • Deliver best practices & creative approaches towards research and technological innovations;
  • Stimulate and provoke discussion.

The working session proposals can be submitted following the template available at https://osqa.eurescom.eu  by Tuesday, 15th October 2013.

 

CLEEN 2014 – Call for Papers

The Future Internet will consists in more flexible radio access networks which are less centralized than the network infrastructure we have today. Integration of flexible heterogeneous radio access networks in HetNets will allow further social diffusion of the mobile internet. Therefore the IEEE is exploring novel concepts to allow for flexibly centralised radio access networks using cloud-processing based on open IT platforms. The Second International Workshop on Cloud Technologies and Energy Efficiency in Mobile Communication Networks” (CLEEN) 2014 is scheduled for April 2014 in Istanbul, Turkey. The goal of the workshop is research and discussion of technologies which enable cloud-based radio access networks that allow for high quality networking in terms of energy efficiency and cost-effectiveness. Building the Future Internet as a cloud-based Internet requires new concepts for the design, operation, and
optimization of radio access networks and backhaul networks as well as a tight integration of networks in cloud-processing IT infrastructures. Therefore a call for papers is assigned.

Paper submission deadline is: October 15th 2013

Acceptance Notification: December 15th 2013
Camera-ready: January 10th 2013

Further Information about the CLEEN 2014 can be found here:

http://www.ict-ijoin.eu/cleen2014/

VTC Fall 2013: The future of Cloud Operating Systems

Thomas M. Bohnert explains his view on the future of Cloud Computing to the audience.

Thomas M. Bohnert explains his view on the future of Cloud Computing to the audience.

Telecommunication and IT industries must work much more closely together than they do it nowadays in order to successfully manage the Internet in the future – especially if we consider that the Internet has become mobile. The future challenges for the mobile internet lie in increasing numbers of mobile end users, rapidly growing data traffic and massive consumption of energy and network resources. Technologies like HetNet and “Small Cell” networks must be used in combination to efficiently manage the data traffic flows and energy consumption. The most promising approach for creating HetNets is “Software Defined Networking” (SDN) integrated into a Cloud environment. Cloud-based SDN provides the elasticity required to integrate heterogeneous network devices that form a HetNet. Since SDN makes network management better understandable to software developers, the disrupting technology has the potential to build a bridge between IT architects and Network managers. This could be the resumé of the “First International Workshop on Cloud Technologies and Energy Efficiency in Mobile Communication Networks” (CLEEN) which took place September 2 – in conjunction with the IEEE Vehicular Technology Conference (VTC).

Cloud-based SDN will be a key technology for facing the challenges of the future Internet, said Keynote Speaker Artur Hecker, researcher working at Chinese telecommunication provider Huawei. Though this is a positive message, Hecker admits that Cloud Computing has some difficulties in delivering the required levels of  availability, performance and security. Therefore Cloud Operating Systems must be made more secure and reliable than they are at the moment. A major technical challenge will be carrier grade availability (99.999% availability) for Cloud environments. TelCos need high degrees of availability when using Cloud software for delivering telecommunication services due to strong restrictions in their SLA requirements.

The ICCLab was also present with a presentation about the “Dependability Modeling Framework”. Konstantin Benz explained the framework which is used to test availability capabilities of Cloud environments. The audience was particularly interested in such methods to make Cloud Computing ready to be used as an essential part of the architecture of the “Future Internet”. Cloud-based SDN should follow strict carrier grade requirements.

The CLEEN workshop closed with a panel discussion about the future of Cloud Computing, Cloud-based SDN and its role in the “Future Internet”. Thomas M. Bohnert stated that Cloud Computing should be more than just a marketing buzz word: it should simplify delivery of telecommunication services rather than becoming a redesign of current telecommunication architectures – especially because access to the Internet has become a commodity in recent years. Cloud Computing can enhance telecommunication services by adding elasticity to (currently) inflexible network architectures, but network architects should be aware that it is not the goal of cloud-based telecommunication technology to simply mimic behaviour of conventional networks. An advantage could be the combination of cloud-based SDN with data collection applications which combines the Cloud Computing approach with Big Data and turns telecommunication services into a data-driven product. This could turn networking which is a commodity into a promising new product for the Future Internet.

 

Information day in Brussels on the upcoming Open Calls of FI PPP

On Wednesday afternoon, 25 September 2013, the Future Internet Public-Private Partnership programme, FI-PPP, will hold an information day in Brussels on the upcoming Open Calls organised by the six FI-PPP phase 2 projects.

All FI-PPP phase 2 projects will run Open Calls for new project partners to complement their work. This includes the five use case projects FI-CONTENT 2, FI-STAR, FIspace, FINESCE, and FITMAN as well as the capacity building and infrastructures project XIFI.

This Open Call Information Day aims to give potential proposers first-hand technical and organisational information on the Open Calls.

Representatives of all FI-PPP phase 2 projects will present their calls and answer questions at the Information Day. Anyone interested in submitting a proposal to any of the six Open Calls is invited to participate in the Information Day. Participation is free, but registration is required.

The registration deadline is Thursday, 12 September 2013. + + + The number of seats is limited to 130. Registrations are coming in fast, soplease register soon.

Further information and the registration form for the Open Call Information Day in Brussels on 25 September 2013

On behalf of CONCORD SA  of FI PPP Programme

 

VTC Fall 2013: can vehicles be intelligent?

VTC Fall 2013 Conference

VTC Fall 2013 Conference

Making vehicles intelligent was one of the major topics this Wednesday. Transportation is an influential factor to economic growth, but it faces many challenges in the areas of safety, mobility (traffic jams) and protection of the environment. Building intelligent traffic surveillance systems is a major goal of US Department of Transportation researchers James Pol and Walton Fehr.

Intelligent traffic surveillance systems heavily rely on in-vehicle information systems which are connected to each other. The information systems in cars can help to collect transportation data which is required in order to predict traffic jams, avoid car accidents and measure pollution. As long as vehicles are not equipped with information systems, data-driven analysis of traffic is not possible. James Pol refers to this issue as a “chicken-egg-problem”: we need intelligent vehicles in order to have a functioning traffic surveillance, but the individual driver might require an intelligent traffic surveillance in the first place in order to invest into (possibly expensive) in-vehicle information systems.

There is much research activity going on in both areas: intelligent cars with in-vehicle information systems are developed as well as networks that connect the different in-vehicle information systems.
Walton Fehr presented some of the ongoing US Department of Transportation “Research and Innovative Technology Administration” (RITA) projects that face the current challenges in building intelligent traffic and transportation systems. Connected Vehicle Technology is a project which has the goal to develop and deploy a communication platform to fully connect all vehicles in a transportation system. Vehicles (or more precisely: in-vehicle information systems) are building mobile ad-hoc networks (MANets) which allow vehicle drivers to collect and exchange data on the overall traffic as well as other vehicles. This technology offers opportunities to researchers which can develop applications for the connected vehicle MANets.

Future research directions will be:

  • Interoperability of different in-vehicle information systems,
  • Automation of traffic data collection and
  • Management of the traffic data.

For more information about the ongoing projects visit the following site:

http://www.its.dot.gov/