{"id":3826,"date":"2013-11-22T17:34:25","date_gmt":"2013-11-22T15:34:25","guid":{"rendered":"http:\/\/www.cloudcomp.ch\/?p=3826"},"modified":"2014-08-05T16:10:05","modified_gmt":"2014-08-05T14:10:05","slug":"infiniband-an-introduction-simple-ib-verbs-program-with-rdma-write","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/infiniband-an-introduction-simple-ib-verbs-program-with-rdma-write\/","title":{"rendered":"InfiniBand: An Introduction + Simple IB verbs program with RDMA Write"},"content":{"rendered":"

This blogpost aims to give you a short introduction to InfiniBand. At the end you should have a rough overview over the technology, much of its terminology and on how to program a very simple RDMA application with IB verbs.<\/p>\n

<\/p>\n

The first part explains the basic characteristics\/properties of the InfiniBand technology and the physical parts that a network consists of. The second part takes a closer look at the logical parts of the technology that are needed for communication. In the third and last part I\u2019ll explain the structure of a simple IB verbs application. IB verbs are abstract representations of functions. You can think of IB verbs as functions\/methods that have to be offered by an (IB)-API.<\/p>\n

You might wonder, why there is an article on InfiniBand on a cloud computing blog. The ICCLab is currently one of the members of the\u00a0FI-WARE<\/a>\u00a0open call project \u201cMiddleware for efficient and QoS\/Security-aware invocation of services and exchange of messages\u201d named\u00a0KIARA<\/a>. One of its features will be the support of InfiniBand in the transport layer.<\/p>\n

This whole blogpost is a big compilation from various sources that I\u2019ve found while researching InfiniBand over the last couple of weeks. The same goes for the IB verbs example program. All those sources that were extremely helpful to me can be found at the end of this blogpost. For those who\u2019ll want to dive deeper into the subject this should give you a good starting point. A big \u2018thank you\u2019 to all those writers of introductions, summaries and tutorials.<\/p>\n

Basics, Network and End Nodes<\/h2>\n

InfiniBand (IB) is a networking technology developed by the InfiniBand Trade Association in 1999. It is used for high-performance computing and in enterprise data centers.\u00a0Its features include high throughput, low\u00a0latency,\u00a0quality of service\u00a0and\u00a0failover.<\/p>\n

The smallest complete InfiniBand Architecture (IBA) unit is a subnet. A subnet consists of end nodes (e.g. servers), switches, copper or fibre links and a subnet manager. End nodes use so called Channel Adapters (CAs) to connect to links. There are Host Channel Adapters (HCAs) and Target Channel Adapters (TCAs). HCAs are accessible by user-applications, TCAs not. The subnet manager has an overview over and manages the whole subnet.<\/p>\n

\n
\"InfiniBand<\/a><\/dt>\n
InfiniBand Subnet<\/dd>\n<\/dl>\n

InfiniBand allows an application to communicate directly<\/i> with another application. This means that an application does not need to rely on the operating system to transfer messages.<\/p>\n

\n
\"InfiniBand<\/a><\/dt>\n
InfiniBand creates a channel directly connecting an application in its virtual address space to an application in another virtual address space<\/dd>\n<\/dl>\n

This was just a very basic and short overview of what InfiniBand is. The IB specification is 1500 pages long! The important points were to get a rough overview of how an IB network looks like, understand that the NICs are called Channel Adapters and that IB creates a channel between those CAs which allows applications to directly communicate with each other without involving the operating system.<\/p>\n

Communication<\/h2>\n

CAs communicate with each other using work queues<\/i>. There are three types of work queues: Send, Receive and Completion. Send and Receive Queues are always used as Queue Pairs <\/i>(QP). A particular QP in a CA is the destination or source of all messages. Each QP also has an associated port which is an abstraction of the connection of a CA to a link.<\/p>\n

\n
\"Queue<\/a><\/dt>\n
Queue Pairs (send\/receive) in the Channel Adapters<\/dd>\n<\/dl>\n

To send or receive messages, Work Requests<\/i> (WRs) are placed onto a QP. There are send work requests and receive work requests. When processing is completed, a Work Completion<\/i> (WC) entry is optionally placed onto a Completion Queue<\/i> (CQ) associated with the work queue.<\/p>\n

\u00a0\"infiniband_queuePair_completionQueue\"<\/a><\/p>\n

To define what address in memory to write to or read from, Scatter\/Gather Elements<\/i> (SGE) are used – and associated with a WR. An SGE is a pointer to a Memory Region<\/i> (MR) which the HCA can read from or write to. A memory region is a contiguous set of memory buffers that has been registered with an HCA. Registration of a MR causes the operating system to provide the HCA with the virtual-to-physical mapping of that region and pin the memory (prohibit swapping it out in virtual memory operations). Memory registration also creates objects called L_Key<\/i> and R_Key<\/i> which need to be used – for authentication – when accessing MRs. With the L_Key (local Key) one can access local MRs. The R_Key (remote Key) can be sent to peers so they can directly access a local MR (RDMA Write, RDMA Read). A MR in turn is part of a Protection Domain<\/em> (PD). PDs effectively glue QPs to memory regions and can be seen as a an aggregating entity. Both QPs and MRs must be defined in the context of a PD.<\/p>\n

\n
\"Relation<\/a><\/dt>\n
Relation of Work Requests, Scatter\/Gather Elements, Memory, Memory Regions and Protection Domain<\/dd>\n<\/dl>\n

By now you should be quite fed up with all those new abbreviations. But especially when programming with the ibverbs<\/i> library, it is more than helpful knowing these abbreviations. Therefore here a short recap and clearer overview of those InfiniBand concepts needed for communication.<\/p>\n\n\n\n\n\n\n\n\n\n\n
\n

Abbr.<\/b><\/p>\n<\/td>\n

Name<\/b><\/td>\nFunction<\/b><\/td>\n<\/tr>\n
\n

PD<\/p>\n<\/td>\n

\n

\u00a0Protection Domain<\/p>\n<\/td>\n

Glues queue pairs and memory regions<\/td>\n<\/tr>\n
\n

MR<\/p>\n<\/td>\n

\n

\u00a0Memory Region<\/p>\n<\/td>\n

Registered memory region that HCA can read from or write to. Contains R_Key and L_Key<\/td>\n<\/tr>\n
\n

QP<\/p>\n<\/td>\n

\n

\u00a0Queue Pair<\/p>\n<\/td>\n

Send \/ Receive work queue. Send or receive work requests are placed onto a queue pair<\/td>\n<\/tr>\n
\n

CQ<\/p>\n<\/td>\n

\n

\u00a0Completion Queue<\/p>\n<\/td>\n

Completion Queue. Completed work requests, so called work completions are placed onto a completion queue. Is associated with queue pair.<\/td>\n<\/tr>\n
\n

WR<\/p>\n<\/td>\n

\n

\u00a0Work Request<\/p>\n<\/td>\n

Either send or receive work request. Specifies action to be processed and will be put onto send or receive queue (QP). References scatter\/gather element<\/td>\n<\/tr>\n
\n

SGE<\/p>\n<\/td>\n

\n

\u00a0Scatter\/Gather Element<\/p>\n<\/td>\n

Defines address(es) in memory to read from or to write to. Must be given L_Key or R_Key to authenticate access to memory region<\/td>\n<\/tr>\n
\n

WC<\/p>\n<\/td>\n

\n

\u00a0Work Completion<\/p>\n<\/td>\n

After a work request has been completed the work completion delivers result<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Simple IB verbs RDMA program<\/h2>\n

The program – simply called rdma<\/i> – described in this section is mainly based on the source code of the \u2018ib_rdma_bw\u2019 application. This application is part of the perftest<\/i> package, available for various Linux distributions. The link to the source-code file can be found at the end of this blogpost. The code in the example program has been greatly simplified and stripped down. \u00a0Almost all the functions were renamed, some functions were put together and lots of code was just removed. Depending on the argument passed to the example you either are the server\/sender or the client\/receiver. At the moment the client connects to a server and then the server writes a string directly into a local buffer of the client which displays it. The source code of the example program can be downloaded at the end of this blogpost.<\/p>\n

First a simplified description of what happens in the program. Most points are identical for the server and the client.<\/p>\n

    \n
  1. Initialize InfiniBand Context (Structures needed for communication and memory)<\/strong>\n
      \n
    1. Get and open InfiniBand device. This will give you a \u2018context\u2019 which is used to create all the following structures<\/li>\n
    2. Allocate a Protection Domain<\/li>\n
    3. Register a Memory Region<\/li>\n
    4. Create a Send and a Receive Completion Queue<\/li>\n
    5. Create a Queue Pair<\/li>\n<\/ol>\n<\/li>\n
    6. Initialize the Queue Pair (change QP status to INIT)<\/strong><\/li>\n
    7. Exchange information to later be able to communicate with peer via IB<\/strong>.<\/strong> This is done via TCP in this example. Another possibility would be to use the RDMA Connection Manager which would need IPoIB enabled hosts. The following information is exchanged\n
        \n
      1. LID – Local Identifier, 16 bit addr. assigned to end nodes by subnet manager<\/li>\n
      2. QPN – Queue Pair Number, identifier assigned to QP by HCA<\/li>\n
      3. PSN \u2013 Packet Sequence Number, used by HCA to verify correct order of packages \/ detect package loss<\/li>\n
      4. R_Key<\/li>\n
      5. VADDR, address of memory region for peer to write into<\/li>\n<\/ol>\n<\/li>\n
      6. Change the QP status to Ready to Receive (RTR)<\/strong><\/li>\n
      7. * ONLY SERVER *<\/b> – Change the QP status to Ready to Send (RTS)<\/strong><\/li>\n
      8. Perform RDMA write<\/strong>\n
          \n
        1. Define memory region to read from with scatter\/gather element (SGE)<\/li>\n
        2. Use work request to define where to write to<\/li>\n
        3. RDMA write into buffer of client\/receiver<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

          The following diagram shows you the flow of the program. Function names are written in bold text and were arbitrarily chosen by me. Just below the function name is a short description of what the function does. The red text marks used IB_verbs.<\/p>\n

          \"ib_rdma_simple_app_flow_sender_receiver\"<\/a><\/p>\n

          The program is far from being finished. At the moment you cannot pass a buffer to it, choose an IB port number or define the size of the buffer. The client does also \u00a0not get notified when the RDMA write from the server has been completed (flow control). This additional functionality will be added in the next steps.<\/p>\n

          Source Code<\/h1>\n

          Example program \u2018rdma\u2019<\/strong><\/h2>\n