(Part 1/3 – Installation – Part 2/3 – troubleshooting)

This part of the tutorial describes how to setup a simple Ceph client using librados (for C++).

The only information that the client requires for the cephx authentication is

• Endpoint of the monitor node
• Keyring containing the pre-shared secret (we will use the admin keyring)

Install librados APIs

On Ubuntu, the library is available on the repositories

$ sudo apt-get install librados-dev

Create a client configuration file

This is the file from which librados will read the client configuration.

The content of the file is structured according to this template:

[global]
mon host= <IP address of one of the monitors>
keyring = <path/to/client.admin.keyring>

for example:

[global]
mon host = 192.168.252.10:6789
keyring = ./ceph.client.admin.keyring

The public endpoint of the monitor node can be retrieved with

$ ceph mon stat

The keyring file can be copied from the admin node. No change is needed to this file. The same information that is contained in the file can be retrieved with this command that will also list the client capabilities:

$ ceph auth get client.admin

Connect to the cluster

The following simple client will perform the following operations:

• Read the configuration file (ceph.conf) from the local directory
• Get an handle to the cluster and IO context on the “data” pool
• Create a new object
• Set an xattr
• Read the object and xattr back
• Print the list of pools
• Print the list of objects in the “data” pool
• Cleanup

1. #include <rados/librados.hpp>
2. #include <string>
3. #include <list>
5. int main(int argc, const char **argv)
6. {
7.   int ret = 0;
9.   /*
10.    * Errors are not checked to avoid pollution.
11.    * After each Ceph operation:
12.    * if (ret < 0) error_condition
13.    * else success
14.    */
16.   // Get cluster handle and connect to cluster
17.   std::string cluster_name(“ceph”);
18.   std::string user_name(“client.admin”);
19.   librados::Rados cluster;
20.   cluster.init2(user_name.c_str(), cluster_name.c_str(), 0);
21.   cluster.conf_read_file(“ceph.conf”);
22.   cluster.connect();
24.   // IO context
25.   librados::IoCtx io_ctx;
26.   std::string pool_name(“data”);
27.   cluster.ioctx_create(pool_name.c_str(), io_ctx);
29.   // Write an object synchronously
30.   librados::bufferlist bl;
31.   std::string objectId(“hw”);
32.   std::string objectContent(“Hello World!”);
33.   bl.append(objectContent);
34.   io_ctx.write(objectId, bl, objectContent.size(), 0);
36.   // Add an xattr to the object.
37.   librados::bufferlist lang_bl;
38.   lang_bl.append(“en_US”);
39.   io_ctx.setxattr(objectId, “lang”, lang_bl);
41.   // Read the object back asynchronously
42.   librados::bufferlist read_buf;
43.   int read_len = 4194304;
44.   //Create I/O Completion.
45.   librados::AioCompletion *read_completion =
46.                                              librados::Rados::aio_create_completion();
47.   //Send read request.
48.   io_ctx.aio_read(objectId, read_completion, &read_buf, read_len, 0);
50.   // Wait for the request to complete, and print content
51.   read_completion–>wait_for_complete();
52.   read_completion–>get_return_value();
53.   std::cout << “Object name: “ << objectId << “\n“
54.             << “Content: “ << read_buf.c_str() << std::endl;
56.   // Read the xattr.
57.   librados::bufferlist lang_res;
58.   io_ctx.getxattr(objectId, “lang”, lang_res);
59.   std::cout << “Object xattr: “ << lang_res.c_str() << std::endl;
62.   // Print the list of pools
63.   std::list<std::string> pools;
64.   cluster.pool_list(pools);
65.   std::cout << “List of pools from this cluster handle” << std::endl;
66.   for (auto pool_id : pools) {
67.     std::cout << “\t“ << pool_id << std::endl;
68.   }
70.   // Print the list of objects
71.   librados::ObjectIterator oit = io_ctx.objects_begin();
72.   librados::ObjectIterator oet = io_ctx.objects_end();
73.   std::cout << “List of objects from this pool” << std::endl;
74.   for (; oit != oet; oit++) {
75.     std::cout << “\t“ << oit–>first << std::endl;
76.   }
78.   // Remove the xattr
79.   io_ctx.rmxattr(objectId, “lang”);
81.   // Remove the object.
82.   io_ctx.remove(objectId);
84.   // Cleanup
85.   io_ctx.close();
86.   cluster.shutdown();
88.   return 0;
89. }

Find the pastebin here.

This example can be compiled and executed with

$ g++ client.cpp -lrados -o cephclient
$ ./cephclient

Operate with cluster data from the command line

To quickly verify if an object was written or to remove it, use the following commands (e.g., from the monitor node).

• List objects in pool data

  $ rados -p data ls

• Check the location of an object in pool data

  $ ceph osd map data <object name>

• Remove object from pool data

  $ rados rm <object name> --pool=data