{"id":4872,"date":"2014-05-02T11:30:59","date_gmt":"2014-05-02T09:30:59","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=4872"},"modified":"2014-05-02T13:50:33","modified_gmt":"2014-05-02T11:50:33","slug":"deploy-ceph-troubleshooting-part-23","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/deploy-ceph-troubleshooting-part-23\/","title":{"rendered":"Deploy Ceph and start using it: end to end tutorial \u2013 Troubleshooting (part 2\/3)"},"content":{"rendered":"

(Part 1\/3 – Installation<\/a>\u00a0– Part 3\/3 – librados client<\/a>)<\/p>\n

It is quite common that after the initial installation, the Ceph cluster reports health warnings. Before using the cluster for storage (e.g., allow clients to access it), a\u00a0HEALTH_OK<\/code> state should be reached:<\/p>\n

cluster-admin@ceph-mon0:~\/ceph-cluster$ ceph health\r\nHEALTH_OK<\/pre>\n
This part of the tutorial provides\u00a0some troubleshooting hints that I collected during the setup of my deployments. Other helpful resources are the Ceph IRC channel and mailing lists.<\/p>\n
Useful diagnostic commands<\/h1>\n
A collection of diagnostic commands to check the status of the cluster is listed here. Running these commands\u00a0is\u00a0how we can understand that the Ceph cluster is not properly configured.<\/p>\n
    \n
  1. Ceph status\n
    $ ceph status<\/pre>\n
    In this example, the disk for one OSD had been physically removed, so 2 out of 3 OSDs were in and up.<\/p>\n
    cluster-admin@ceph-mon0:~\/ceph-cluster$ ceph status\r\n    cluster 28f9315e-6c5b-4cdc-9b2e-362e9ecf3509\r\n     health HEALTH_OK\r\n     monmap e1: 1 mons at {ceph-mon0=192.168.0.1:6789\/0}, election epoch 1, quorum 0 ceph-mon0\r\n     osdmap e122: 3 osds: 2 up, 2 in\r\n      pgmap v4699: 192 pgs, 3 pools, 0 bytes data, 0 objects\r\n            87692 kB used, 1862 GB \/ 1862 GB avail\r\n                 192 active+clean<\/pre>\n<\/li>\n
  2. Ceph health\n
    $ ceph health\r\n$ ceph health detail<\/pre>\n<\/li>\n
  3. Pools and OSDs configuration and status\n
    $ ceph osd dump\r\n$ ceph osd dump --format=json-pretty<\/pre>\n
    the second version provides much more information, listing all the pools and OSDs and their configuration parameters<\/li>\n
  4. Tree of OSDs reflecting the CRUSH map\n
    $ ceph osd tree<\/pre>\n
    This is very useful to understand how the cluster is physically organized (e.g., which OSDs are running on which host).<\/li>\n
  5. Listing the pools in the cluster\n
    $ ceph osd lspools<\/pre>\n
    This is particularly useful to check clients operations (e.g., if new pools were created).<\/li>\n
  6. Check the CRUSH rules\n
    $ ceph osd crush dump --format=json-pretty<\/pre>\n<\/li>\n
  7. List the disks of one node from the admin node\n
    $ ceph-deploy disk list osd0<\/pre>\n<\/li>\n
  8. Check the logs.
    \nLog files in\u00a0\/var\/log\/ceph\/<\/code> will provide a lot of information for troubleshooting. Each node of the cluster will contain logs about the Ceph components that it runs, so you may need to SSH on different hosts to have a complete diagnosis.<\/li>\n<\/ol>\n
    Check your firewall and network configuration<\/h1>\n
    Every node of the Ceph cluster must be able to successfully run<\/span><\/span><\/span><\/p>\n
    <\/span>$ ceph status<\/pre>\n
    If this operation times out without giving any results, it is likely that the firewall (or network configuration) is not allowing the nodes to communicate.<\/p>\n
    Another symptom of this problem is that OSDs cannot be activated, i.e., the\u00a0ceph-deploy\u00a0osd\u00a0activate\u00a0<args><\/code>\u00a0command will timeout.<\/span><\/span><\/p>\n
    Ceph monitor default port is\u00a06789<\/tt>.\u00a0<\/span>Ceph OSDs and MDS try to get the first available ports starting at\u00a06800<\/tt>.<\/span><\/span><\/p>\n
    A typical Ceph cluster might need the following ports:<\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/p>\n
    <\/span>Mon:  6789\r\n<\/span>Mds:  6800\r\n<\/span>Osd1: 6801\r\n<\/span>Osd2: 6802\r\n<\/span>Osd3: 6803<\/pre>\n
    Depending on your security requirements, you may want to simply allow any traffic to and from the Ceph cluster nodes.<\/p>\n
    References:\u00a0http:\/\/comments.gmane.org\/gmane.comp.file-systems.ceph.devel\/2231<\/a><\/p>\n
    Try restarting first<\/h1>\n
    Without going for fine troubleshootings and log analysis, sometimes (especially after the first installation), I’ve noticed that a simple restart of the Ceph components has helped the transition from a\u00a0HEALTH_WARN<\/code> to a\u00a0HEALTH_OK<\/code> state.<\/p>\n
    If some of the OSDs are not in or not up, like in the case below<\/p>\n
        cluster 07d28faa-48ae-4356-a8e3-19d5b81e159e\r\n<\/span>     health HEALTH_WARN 192 pgs incomplete; 192 pgs stuck inactive; 192 pgs stuck unclean; 1\/2 in osds are down; clock skew detected on mon.1, mon.2\r\n<\/span>     monmap e3: 3 mons at {0=192.168.252.10:6789\/0,1=192.168.252.11:6789\/0,2=192.168.252.12:6789\/0}, election epoch 36, quorum 0,1,2 0,1,2\r\n<\/span>     osdmap e27: 6 osds: 1 up, 2 in\r\n<\/span>      pgmap v57: 192 pgs, 3 pools, 0 bytes data, 0 objects\r\n<\/span>            84456 kB used, 7865 MB \/ 7948 MB avail\r\n<\/span>                 192 incomplete<\/pre>\n
    try to start the OSD daemons with<\/p>\n
    # on osd0\r\n$ sudo \/etc\/init.d\/ceph -a start osd0<\/span><\/pre>\n
    If the OSDs are in, but PGs are in weird states, like in the example below<\/p>\n
    cluster 07d28faa-48ae-4356-a8e3-19d5b81e159e\r\n<\/span>     health HEALTH_WARN 192 pgs degraded; 192 pgs stuck unclean; clock skew detected on mon.1, mon.2\r\n<\/span>     monmap e3: 3 mons at {0=192.168.252.10:6789\/0,1=192.168.252.11:6789\/0,2=192.168.252.12:6789\/0}, election epoch 36, quorum 0,1,2 0,1,2\r\n<\/span>     osdmap e34: 6 osds: 6 up, 6 in\r\n<\/span>      pgmap v71: 192 pgs, 3 pools, 0 bytes data, 0 objects\r\n<\/span>            235 MB used, 23608 MB \/ 23844 MB avail\r\n<\/span>                 128 active+degraded\r\n<\/span>                  64 active+replay+degraded<\/pre>\n
    try to restart the monitor(s) with<\/p>\n
    # on mon0\r\n$ sudo \/etc\/init.d\/ceph -a restart mon0\r\n<\/span><\/pre>\n
    Unfortunately, a simple restart will be the solution in just a few rare cases. More troubleshooting will be required in the majority of the situations.<\/p>\n
    Unable to find keyring<\/h1>\n
    During the deployment of the monitor nodes (the ceph-deploy <mon> [<mon>] create-initial<\/code> step), Ceph may complain about missing keyrings:<\/p>\n
    <\/span>[ceph_deploy.gatherkeys][WARNIN] Unable to find\r\n<\/span>\/etc\/ceph\/ceph.client.admin.keyring on ['ceph-server']<\/pre>\n
    If this warning is reported\u00a0(even if the message is not an error), the Ceph cluster will probably not reach an healthy state.<\/p>\n
    The solution to this problem is to use exactly the same names for the hostnames (i.e., the output of\u00a0hostname -s<\/code>) and the Ceph node names.<\/p>\n
    This means that\u00a0the files<\/p>\n