{"id":5099,"date":"2014-06-04T17:45:24","date_gmt":"2014-06-04T15:45:24","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=5099"},"modified":"2014-06-10T11:57:39","modified_gmt":"2014-06-10T09:57:39","slug":"ceph-osd-down-and-out-of-the-cluster","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/","title":{"rendered":"Ceph: OSD “down” and “out” of the cluster – An obvious case"},"content":{"rendered":"

When setting up a cluster with ceph-deploy<\/a>, just after the\u00a0ceph-deploy osd activate<\/code>\u00a0phase and the distribution of keys, the OSDs should be both “up” and “in”\u00a0the cluster.<\/p>\n

One thing that is not mentioned in the quick-install documentation with ceph-deploy<\/a>\u00a0or the OSDs monitoring<\/a> or troubleshooting<\/a> page\u00a0(or at least I didn’t find it), is that, upon (re-)boot, mounting the storage volumes to the mount points that ceph-deploy prepares is up to the administrator (check this discussion<\/a> on the Ceph mailing list).<\/p>\n

So, after a reboot of my storage nodes, the Ceph cluster couldn’t reach a healthy state showing the following OSD tree:<\/p>\n

$ ceph osd tree\r\n# id weight type name up\/down reweight\r\n-1 3.64 root default\r\n    -2 1.82 host ceph-osd0\r\n        0 0.91 osd.0 down 0\r\n        1 0.91 osd.1 down 0\r\n    -3 1.82 host ceph-osd1\r\n        2 0.91 osd.2 down 0\r\n        3 0.91 osd.3 up 1\r\n<\/pre>\n
I wasn’t thinking about mounting the drives, as this process was hidden to me during the initial installation, but a simple mount<\/code> command would have immediately unveiled the mistery :D.<\/p>\n
So, the simple solution was to mount the devices:<\/p>\n
sudo mount \/dev\/sd<XY> \/var\/lib\/ceph\/osd\/ceph-<K>\/\r\n<\/pre>\n
and then to start the OSD daemons:<\/p>\n
sudo start ceph-osd id=<K>\r\n<\/pre>\n
For some other troubleshooting hints\u00a0for Ceph, you may look at this page<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"
When setting up a cluster with ceph-deploy, just after the\u00a0ceph-deploy osd activate\u00a0phase and the distribution of keys, the OSDs should be both “up” and “in”\u00a0the cluster. One thing that is not mentioned in the quick-install documentation with ceph-deploy\u00a0or the OSDs monitoring or troubleshooting page\u00a0(or at least I didn’t find it), is that, upon (re-)boot, mounting […]<\/p>\n","protected":false},"author":96,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[5,15],"tags":[],"features":[],"class_list":["post-5099","post","type-post","status-publish","format-standard","hentry","category-articles","category-howtos"],"yoast_head":"\nCeph: OSD "down" and "out" of the cluster - An obvious case - Service Engineering (ICCLab & SPLab)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Ceph: OSD "down" and "out" of the cluster - An obvious case\" \/>\n<meta property=\"og:description\" content=\"When setting up a cluster with ceph-deploy, just after the\u00a0ceph-deploy osd activate\u00a0phase and the distribution of keys, the OSDs should be both “up” and “in”\u00a0the cluster. One thing that is not mentioned in the quick-install documentation with ceph-deploy\u00a0or the OSDs monitoring or troubleshooting page\u00a0(or at least I didn’t find it), is that, upon (re-)boot, mounting […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Engineering (ICCLab & SPLab)\" \/>\n<meta property=\"article:published_time\" content=\"2014-06-04T15:45:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2014-06-10T09:57:39+00:00\" \/>\n<meta name=\"author\" content=\"piiv\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"piiv\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/\"},\"author\":{\"name\":\"piiv\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/b75ac4e936b5921c8a9de4fb84202703\"},\"headline\":\"Ceph: OSD “down” and “out” of the cluster – An obvious case\",\"datePublished\":\"2014-06-04T15:45:24+00:00\",\"dateModified\":\"2014-06-10T09:57:39+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/\"},\"wordCount\":175,\"commentCount\":3,\"articleSection\":[\"Articles\",\"HowTos\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/\",\"name\":\"Ceph: OSD \\\"down\\\" and \\\"out\\\" of the cluster - An obvious case - Service Engineering (ICCLab & SPLab)\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\"},\"datePublished\":\"2014-06-04T15:45:24+00:00\",\"dateModified\":\"2014-06-10T09:57:39+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/b75ac4e936b5921c8a9de4fb84202703\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/icclab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Ceph: OSD “down” and “out” of the cluster – An obvious case\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/\",\"name\":\"Service Engineering (ICCLab & SPLab)\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/b75ac4e936b5921c8a9de4fb84202703\",\"name\":\"piiv\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/c7b4c6df4485c24b56af1a1a92442259dfc735b8c0dcf8d3ddcb16f88deeb723?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c7b4c6df4485c24b56af1a1a92442259dfc735b8c0dcf8d3ddcb16f88deeb723?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c7b4c6df4485c24b56af1a1a92442259dfc735b8c0dcf8d3ddcb16f88deeb723?s=96&d=mm&r=g\",\"caption\":\"piiv\"},\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/author\/piiv\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Ceph: OSD \"down\" and \"out\" of the cluster - An obvious case - Service Engineering (ICCLab & SPLab)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/","og_locale":"en_US","og_type":"article","og_title":"Ceph: OSD \"down\" and \"out\" of the cluster - An obvious case","og_description":"When setting up a cluster with ceph-deploy, just after the\u00a0ceph-deploy osd activate\u00a0phase and the distribution of keys, the OSDs should be both “up” and “in”\u00a0the cluster. One thing that is not mentioned in the quick-install documentation with ceph-deploy\u00a0or the OSDs monitoring or troubleshooting page\u00a0(or at least I didn’t find it), is that, upon (re-)boot, mounting […]","og_url":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/","og_site_name":"Service Engineering (ICCLab & SPLab)","article_published_time":"2014-06-04T15:45:24+00:00","article_modified_time":"2014-06-10T09:57:39+00:00","author":"piiv","twitter_card":"summary_large_image","twitter_misc":{"Written by":"piiv","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/"},"author":{"name":"piiv","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/b75ac4e936b5921c8a9de4fb84202703"},"headline":"Ceph: OSD “down” and “out” of the cluster – An obvious case","datePublished":"2014-06-04T15:45:24+00:00","dateModified":"2014-06-10T09:57:39+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/"},"wordCount":175,"commentCount":3,"articleSection":["Articles","HowTos"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/","url":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/","name":"Ceph: OSD \"down\" and \"out\" of the cluster - An obvious case - Service Engineering (ICCLab & SPLab)","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#website"},"datePublished":"2014-06-04T15:45:24+00:00","dateModified":"2014-06-10T09:57:39+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/b75ac4e936b5921c8a9de4fb84202703"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/icclab\/ceph-osd-down-and-out-of-the-cluster\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/icclab\/"},{"@type":"ListItem","position":2,"name":"Ceph: OSD “down” and “out” of the cluster – An obvious case"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/icclab\/#website","url":"https:\/\/blog.zhaw.ch\/icclab\/","name":"Service Engineering (ICCLab & SPLab)","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/b75ac4e936b5921c8a9de4fb84202703","name":"piiv","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c7b4c6df4485c24b56af1a1a92442259dfc735b8c0dcf8d3ddcb16f88deeb723?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c7b4c6df4485c24b56af1a1a92442259dfc735b8c0dcf8d3ddcb16f88deeb723?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c7b4c6df4485c24b56af1a1a92442259dfc735b8c0dcf8d3ddcb16f88deeb723?s=96&d=mm&r=g","caption":"piiv"},"url":"https:\/\/blog.zhaw.ch\/icclab\/author\/piiv\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/5099","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/users\/96"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/comments?post=5099"}],"version-history":[{"count":4,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/5099\/revisions"}],"predecessor-version":[{"id":5114,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/5099\/revisions\/5114"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/media?parent=5099"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/categories?post=5099"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/tags?post=5099"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/features?post=5099"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}