Homelab - Part 3: Backups September 30, 2021 on Jonathan's Blog Tags: homelab, devops, self-hosting, backups

Contents

As mentioned in Part 2 of this series, backups were a priority of mine over the last few weeks. Although I was backing up some data when I wrote that article, not everything was being backed up properly and nothing was being backed up offsite.

That’s all changed now, and this post will explain the process I went through to get a reasonable backup strategy for my homelab in place.

My Existing Backup “Strategy” and the Goal

My existing backup strategy consisted of a cronjob running a simple script 8 times a day (every 3 hours). That script runs a Borg1 backup which includes most of the key directories on my homelab, including all the data and database directories. The script also prunes the backup repository so that I only have 7 daily backups (one for each weekday), 4 weekly backups (one for each week of the month) and 6 monthly backups.

This works well and gives a me a range of points to restore from if I make a mistake or something bad happens. However, the backup repository is local, meaning it sits on the same drive as the data originally came from, so in the case of hard drive failure, I’d lose everything.

The 3-2-1 Backup Strategy2 is widely regarded as being pretty air-tight and is used by a lot of the self-hosting and homelab community, not to mention in business. The 3-2-1 strategy has 3 parts:

So to quickly compare my old strategy to 3-2-1:

The goal is therefore to sort out some off-site location for my most important data and continue doing local backups, but change the destination to a different medium/device.

Exploring strange new worlds

One thing I wanted to be certain of was that my choice of backup software was sound. Therefore, before putting work into everything else I looked into some other incremental backup solutions.

Aside from Borg1 there are several backup solutions such as Duplicati4 and Restic5. Both initially appeared worth a shot but after a little research it became clear Duplicati wouldn’t meet my needs since I wanted to do local backups too.

Restic offers much the same as Borg, and even uses similar terminology and commands, making it quite easy to try out. I got everything setup and working nicely but unfortunately ran into some resilience issues with the backup process. My poor home broadband means that disconnections and timeouts are a regular occurrence and I need something that can handle this nicely.

I persevered with Restic for a little over a week, but after failing to find a solution to my problem I returned to Borg. Frequent cut outs happened with Borg too, which I was able to resolve with some SSH options6, but partial backups are stored with Borg so after restarting the backup, everything just continues from where it left off.

I found that there isn’t much to choose between Borg and Restic, but this little difference was quite important for me. Restic is worth a look and I will certainly revisit it in the future, but for now Restic is futile and Borg shall live on.

Where do off-site backups live?

The logistics of the off-site part was probably the biggest challenge of the whole backup overhaul. As much as possible I want to be in control of my data and that means not using big cloud services, hence I’m self-hosting everything. Therefore, using a cloud service as a backup destination is not ideal. My ideals coupled with the cost of storage in the cloud is probably why it took so long to get this sorted.

Here were the options I considered:

For my current usage, a 100GB BX108 storage box from Hetzer, at a little less than £3 a month, is the best choice. It was easy to setup and I can use SFTP/SSH to access the data on the box. There are no overcomplicated extras, its simple and that’s all it needs to be. Another solution might be the better for me in the future though as the quantity of data I’m backing up increases.

Ansible Roles and Automation

I have a few different machines that I’d like to start backing up including my personal laptop, desktop and homelab server. Therefore I would have to set up a backup on each of the machines and any time a change in my backup setup occurs I’d have to propagate that change to each machine. This caused me to revisit an old friend by the name of Ansible to try and automate the installation (and also the running) of my backup setup.

Restic was still being tested when I started playing with Ansible and I discovered a pretty decent role on Github that I was able to use as a starting point for my own backup role.

I wanted a few things from this role:

My initial task was to adapt a backup script I have been playing around with to a Jinja2 template so that Ansible could populate it with the details of whichever repository was configured. Then I added a Nextcloud backup script which I will detail more in the next section. Finally I added in a playbook, default configuration and inventory file and was able to start testing.

My playbook installs an SSH key and configuration for the remote Hetzner storage box. It then creates backup scripts for each configured repository and a separate script for the Nextcloud backup. Next a cronjob is added for each of the scripts at a configurable interval. I have been using this for a few days and everything has gone smoothly so far.

In addition to the basics, both Restic and Borg behaviour can be used and if Borg is used a set of init scripts are generated to create the initial repository if it does not already exist.

More detail on the configuration and usage of this Ansible role can be found on my Github9.

It is definitely not perfect and still a work in progress, but it will do for now and as I learn more about Ansible and my needs it will improve.

Nextcloud Specific Backups

Nextcloud is probably tied first for the most important thing to backup for me - photos being the joint most important. Until this point Nextcloud had been backed up as a folder like everything else, which isn’t good as it can lead to inconsistencies in files which are then part of my backups. Additionally the database was being backed up as a folder too, which again is far from the recommended way to do that.

In order to safely backup Nextcloud you must enable maintenance mode, then backup the config, data and theme folders (or more if you want) and finally dump the database10. Then you can disable maintenance mode and continue using Nextcloud.

To enable maintenance mode on my Nextcloud instance I must exec into the Kubernetes pod and run a command, something like

kubectl exec -n nextcloud $POD -- \
    su - -s '/bin/sh' www-data -c \
    "php -d memory_limit=-1 /var/www/html/occ maintenance:mode --on"

which runs the command on the remote pod as the www-data user with php memory limit disabled (this was necessary at time of writing).

Once this is done I can backup the data folders and then run another command on the postgres pod to dump the database

kubectl exec -n nextcloud $POD -- \
    /bin/bash -c "/opt/bitnami/postgresql/bin/pg_dump -Unextcloud nextcloud" \
    > nextcloud.bak.sql

I combined these commands into a Nextcloud backup script and piped the output of the database dump to Borg/Restic, as both support backups from stdin. Now my data from Nextcloud is backed up safely.

Next Steps

I will update this list as I tick off these items :)

Articles from blogs I read Generated by openring

Neurodivergence and accountability in free software

In November of last year, I wrote Richard Stallman’s political discourse on sex, which argues that Richard Stallman, the founder of and present-day voting member of the board of directors of the Free Software Foundation (FSF), endorses and advocates for a ha…

via Drew DeVault's blog September 25, 2024

Status update, September 2024

Hi! Once again, this status update will be rather short due to limited time bandwidth. I hope to be able to allocate a bit more time slots for my open-source projects next month. We’re getting closer to a new Sway release (fingers crossed), with lots of help f…

via emersion September 20, 2024