{"id":80,"date":"2022-06-28T15:07:14","date_gmt":"2022-06-28T15:07:14","guid":{"rendered":"https:\/\/www.ogselfhosting.com\/?p=80"},"modified":"2022-07-09T12:06:49","modified_gmt":"2022-07-09T12:06:49","slug":"nextcloud-fail-over","status":"publish","type":"post","link":"https:\/\/www.ogselfhosting.com\/index.php\/2022\/06\/28\/nextcloud-fail-over\/","title":{"rendered":"Nextcloud Fail-over"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">I have operated a Nextcloud instance for several years.  It has completely replaced DropBox, OneDrive and even Google Drive for me.  However, my single-instances of Nextcloud have occasionally had downtime (power cuts, server issues and especially &#8216;administrator configuration fubars&#8217;).  I have experimented with a Nextcloud failover service to try to improve mmy uptime, and it&#8217;s now in &#8216;experimental operation&#8217;.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">At the present time, I now have TWO instances running on two different hardware platforms.  Both instances run in a virtual environment.  One, running on my new dual-EPYC server, is the primary instance intended to be in operation &#8216;all of the time&#8217;.  The other, on a purpose-built server based on consumer hardware, is a mirror of the primary instance but theoretically is always hot and able to come online at a moments notice.   If my primary server goes down, the backup takes over in about 1-3 seconds. <\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1912\" height=\"1039\" src=\"https:\/\/www.ogselfhosting.com\/wp-content\/uploads\/2022\/06\/Solo-Operating-1.png\" alt=\"\" class=\"wp-image-86\"\/><figcaption>Primary Nextcloud container running on server1 (top right), backup on server2 (top left)<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">I rely upon two key software packages to help me make this happen: (1) lxd, which I use to run all my containers and even some of my vm&#8217;s (I suspect Docker would work equally well); and (2) keepalived, which provides me with a &#8216;fake&#8217; IP I can assign to different servers depending on whether they are operational or not.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">I am going to run this service with just two instances (i.e. one fail-over server).  For now, both services are hosted in the same physical property and use the same power supply &#8211; so I do not have professional-grade redundancy (yet).  I may add a third instance to this setup and even try to place that in a different physical location which would considerably improve robustness against power loss, internet outages etc.  But that&#8217;s for the future &#8211; today I just finally have some limited albeit production-grade fail-over capability.  I shall see if this actually makes my reliably better (as intended), or if the additional complexity just brings new problems that make things worse or at least no-better.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1912\" height=\"1027\" src=\"https:\/\/www.ogselfhosting.com\/wp-content\/uploads\/2022\/06\/Solo-down-Skywalker-up.png\" alt=\"\" class=\"wp-image-85\"\/><figcaption>Server2 has kicked-in when I shutdown server 1.<\/figcaption><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A couple of additional details &#8211; I actually hot-backup both my Nextcloud server and a wordpress site I operate.  As you can also see from the above image, I also deliberately change the COLOR of my Nextcloud banners (from blue to an unsubtle RED) just to help me realize something is up if my EPYC server goes down since I don&#8217;t always pay attention to phone notifications.   I only perform a one-way sync, so any changes made to a backup instance will not be automatically regenerated on the primary server as\/when it comes back online after a failure.  This is deliberate, to reduce making the setup too complicated (which could otherwise not go unpunished!).  A pretty useful feature: my ENTIRE Nextcloud instance is hot-copied &#8211; links, apps, files, shares, sql daabase, ssl certs, user-settings, 2FA credentials etc.  Other than the color of the banner ( and a pop-up notification), the instances are &#8216;almost  identical&#8217;*.  Lxd provides me with this level of redundancy as it copies everything when you use the refresh mode.  Many other backup\/fail-over implemetations I have explored in the past do not provide the same level of easy redundency for a turn-key service.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">(<strong>*<\/strong>) Technically, the two instances can never be truly 100.0000000&#8230;% identical no matter how fast you mirror an instance.  In my case, there is a user-configurable difference between the primary server and the backup server at the time of the fail-over coming online.  I say user-cobfigurable because this is the time delay for copying the differences between server1 and server2.  I configure this via the scheduling of the &#8216;lxc copy &#8211;refresh&#8217; action.  On a fast network, this can be as little as a minte or two, or potentially even faster.  For my use-case, I accept the risk of losing a few minutes worth of changes, which is my maximum risk for the benefit of having a fail-over service.  Accordingly, I run my sync script &#8220;less frequently&#8221; and as of now, it&#8217;s a variable I am playing with vs running a copy &#8211;refresh script constantly.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If anyone has any interest in more details on how I configure my fail-over service, I&#8217;ll be happy to provide details.  Twitter: @OGSelfHosting<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I have operated a Nextcloud instance for several years. It has completely replaced DropBox, OneDrive and even Google Drive for me. However, my single-instances of Nextcloud have occasionally had downtime (power cuts, server issues and especially &#8216;administrator configuration fubars&#8217;). I have experimented with a Nextcloud failover service to try to improve mmy uptime, and it&#8217;s [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":56,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,4,11,10,9,1],"tags":[],"class_list":["post-80","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-100daysofhomelab","category-jus-blogging","category-keepalived","category-lxd","category-ubuntu","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/posts\/80","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/comments?post=80"}],"version-history":[{"count":8,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/posts\/80\/revisions"}],"predecessor-version":[{"id":130,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/posts\/80\/revisions\/130"}],"wp:attachment":[{"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/media?parent=80"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/categories?post=80"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ogselfhosting.com\/index.php\/wp-json\/wp\/v2\/tags?post=80"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}