{"id":244,"date":"2024-03-03T19:11:27","date_gmt":"2024-03-03T19:11:27","guid":{"rendered":"https:\/\/www.ogselfhosting.com\/?p=244"},"modified":"2024-03-17T11:53:01","modified_gmt":"2024-03-17T11:53:01","slug":"troubleshooting-my-offline-zpool","status":"publish","type":"post","link":"https:\/\/www.ogselfhosting.com\/index.php\/2024\/03\/03\/troubleshooting-my-offline-zpool\/","title":{"rendered":"Troubleshooting my offline Zpool"},"content":{"rendered":"\n

It’s a quiet Sunday, and I wasn’t planning on writing an article.<\/p>\n\n\n\n

There I was copying files and doing some maintenance, and my network drive was offline. I figured I must have done something dumb, so I logged into my server and checked. My 8 x 6TB iron wolf raid-z2 zfs array was offline. So much for a quiet day.<\/p>\n\n\n\n

Four of the eight disks were showing errors. And the ‘lsblk’ command could only find four of the eight disks:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Where have my drives gone?<\/p>\n\n\n\n

In fact, I was a little relieved – one drive error might be real, but I thought 4 is probably a glitch. Hopefully software, but I have to troubleshot to find out. Here’s what I did. Firstly, server reboot – that should fix software issues, if any. It almost worked too: The drives reappeared, and the raid away came back to life. But then it died a few minutes later during a scrub I initiated. Again, FOUR disks gave errors. It’s probably not the software.<\/p>\n\n\n\n

So I rebooted the server, logged into the IPMI interface and spammed the delete key a few times so I could check interrupt the reboot and enter the bios setup screen of my H12SSi-NT motherboard. I wanted to see what the motherboard could detect. The H12 motherboard has a pair of slim-SAS connectors, and I was using all of one of them:<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Both 8-port SATA connectors showed up, but I still wondered if the port I was using was somehow at fault (it’s a new motherboard… and wouldn’t make me smile if it was dead already). So I powered off, switched SAS port connectors and rebooted.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n\n\n\n

At power-up, however, the zpool array was still dead with four drives not showing.<\/p>\n\n\n\n

Believe it or not, I felt BETTER: the chances of both SAS ports faulting is…low. And if the SATA ports were both working properly then it’s probably NOT the motherboard: remember that I said four drives were dead? Well each pair of four-drives is powered by a separate power cable connected to the single power supply. Could this be a dodgy power connection?<\/p>\n\n\n\n

So I took the cover off and juggled the SATA power leads a little on each drive and on each power connector to the power supply. All the leads were all clicked-in-place, so I couldn’t easily see a problem. But I rebooted anyway as it’s an easy check. Wonder of wonders, on power-up, all eight drives reappeared and the zpool imported without issue.<\/p>\n\n\n\n

As I type, I am scrubbing the zpool…but I am also going to order a new SATA power cable as I can’t really expect a ‘cable-jiggle’ to be a good long-term solution.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

I also put my SAS connector back to the original port as the cabling was less stressful (I would have to re-route the cable to use that port permanently):<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

So the GOOD news is, I think it’s an inexpensive problem: a power lead. The BETTER news is that by systematically checking out the potential problems, I have a likely root-cause and a short-term fix (‘jiggling power leads’). I also have an executable plan for eliminating this (i.e. buy new (different?) power lead(s) for the drives).<\/p>\n\n\n\n

The takeaway? Check one thing at a time. \ud83d\ude42<\/p>\n\n\n\n

Enjoy your Sunday!<\/p>\n","protected":false},"excerpt":{"rendered":"

It’s a quiet Sunday, and I wasn’t planning on writing an article. There I was copying files and doing some maintenance, and my network drive was offline. I figured I must have done something dumb, so I logged into my server and checked. My 8 x 6TB iron wolf raid-z2 zfs array was offline. So […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[],"class_list":["post-244","post","type-post","status-publish","format-standard","hentry","category-zfs"],"_links":{"self":[{"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/posts\/244","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/comments?post=244"}],"version-history":[{"count":4,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/posts\/244\/revisions"}],"predecessor-version":[{"id":256,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/posts\/244\/revisions\/256"}],"wp:attachment":[{"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/media?parent=244"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/categories?post=244"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/tags?post=244"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}