Jonnobrows Musings

I've been outside

Tue, 08 Apr 2025 00:00:00 +0000

So much has happened since my previous post, so here is the update.

Life

In my last post I mentioned that I was engaged and planning to buy a house.

Shortly after that post we started the house hunting process and after nearly 15 months we finally completed on our purchase. Then, just six weeks later, we got married!

We then spent a couple of weeks taking it slow on Honeymoon in Croatia.

The (nearly) a year since has been great too, we’ve travelled, made our house into a home, made memories with friends and family, and found our new normal after being in a state of flux for a few years.

I talked about wanting to get back into hiking, and while it may not be hiking, I have been walking more, and exploring the surrounding area.

The plan is to keep travelling and exploring, and we would also like to adopt/rescue a cat at some point this year 🐱. Our garden is the priority at home this year, and it’s been really nice to spend time outside in this unseasonably mild spring weather.

Work

Quite a bit has happened here too:

December 2022: I was promoted to Site Reliability Engineer
April 2024: The platform I was working on was deprecated and I was moved to a new team
September 2024: I moved to Sky Protect, and was promoted to Senior SRE

It’s been a big change working for Sky Protect, but it’s definitely interesting and a good opportunity for growth, in what is a relatively new team.

Homelab

My Homelab is really the reason my blog exists, so it gets it’s own section.

After I moved house my Homelab became a 24/7 feature again, it even got an upgrade, with more RAM and some hard-drives. More recently I’ve ditched K8s and focused on the few main services I actually use:

Paperless: Scans / “paper” documents
Immich: Photos - although I still rely heavily on Google Photos
Nextcloud: File sync

These are all exposed via Traefik reverse proxy and backed-up with Borgmatic to my Hetzner storage box. It works like a charm, is low maintenance and is simple enough that I don’t have to re-learn everything if I step away for a couple of months.

I’d like to have a “lab” again, but I don’t get all that much time to sink into it at the moment (see Life). For now, Home automation is scratching-the-itch. I have Home Assistant on a Raspberry Pi 5 (overkill), and have been enjoying tinkering when I can - all Wife Approved of course!

I make no promises about when I’ll blog next - I guess it will be when I have another big update to share.

Until next time…

A Year in Review

Sun, 23 Oct 2022 00:00:00 +0000

It has been 10 months since my last update, here is what I have been up to…

October 2021 - Joining Sky

Sky was, and continues to be, a fantastic first workplace. It provided a great space for me to learn and explore technologies with no real pressure to deliver results unless I felt ready.

The first few weeks were occupied by a technology Bootcamp where we learned about everything from basic Python programming through to AWS and configuration management in Puppet. I then joined a small team with some other graduates to build an internal tool for capturing and analysing the cost of incident investigations - this primarily used Golang and GCP.

January 2022 - Dendrite Helm Chart

Towards the end of December 2021 I started working on a Helm chart to deploy Dendrite¹, on Kubernetes. You can see the post where this was announced here: https://matrix.org/blog/2022/01/07/this-week-in-matrix-2022-01-07#dendrite-helm-chart-website

February 2022 - Joining the Software Defined Streaming Team

After completing the internal tool we were all assigned to teams within the Global Reliability department at Sky. I joined the Software Defined Streaming (SDS) Site Reliability Engineering (SRE) team - SDS SRE for short. The platform within the team is sizeable and split between on-premise vsphere clusters and GCP. We manage all of this using tools like Puppet, Jenkins and Terraform and my main responsibilities in the team are to improve, modernise and maintain the platform, whilst also supporting the Streaming Engineers and Testers that make up the rest of the Streaming team.

I also released another update for the Dendrite Helm Chart which added the ability to do a Polylith deployment.

May 2022 - Getting Engaged

My long-term partner, Izzy, and I got engaged after dating for five-and-a-half years! We have now set a date (4th May 2024) and are enjoying planning the ins and outs of our wedding day.

At some point we hope to start sharing updates via our wedding website, so save that if you want to stay in-the-know.

July/August 2022 - Going abroad for the first time in a while

Before COVID-19 a trip abroad was pretty much an annual thing for me and my family, and sometimes it was even more frequent since we love to travel and explore the world. My last trip abroad was towards the end of 2019.

We spent two weeks exploring the Loire Valley in France, drinking good wine and eating good food. It was certainly a much needed and enjoyable break from carrying out production releases!

September 2022 - Planing to buy a house

The next step for us to get on the property ladder and we are hoping to do that early in the new year. It’s early days at the moment but it has been fun to start nosing around on Rightmove and Zoopla.

October 2022 - What’s next?

For the homelab: Earlier this month I turned off my server for the first time since it went on in my first homelab post. My usage has decreased over the past few months and now the cost of electricity doesn’t really make it worthwhile. I have a couple of projects and training goals that will make use of it again soon, but for now I’m happy to turn it on to backup photos and files every once-in-a-while.

For Dendrite: The Dendrite chart repository was deprecated so has moved somewhere else and I haven’t followed. I felt it has reached a point where maintaining it is fairly straightforward and/or no longer required so I am leaving it up to those who actually use it keep it going if they want.

For Sky: I am looking to get promoted to Site Reliability Engineer in the coming months and take on some more responsibilities in my team. We have some exciting projects just around the corner that I am looking to lead on and/or be a large part of. The platform is changing and modernising at speed, so its definitely a good place to see a progression from the way it was 10 years ago to the shiny cloud-centric world we now live in.

For Personal Projects: I’m looking to start/join a project again soon to fill the space the Dendrite chart has left. This will probably tie in nicely with the Homelab again and be largely focused on K8s and Golang, so watch this space.

Beyond Tech: My focus outside of work is planning for my wedding, buying a house and generally looking forward to the future. I’m enjoying that as my main hobby and its certainly time consuming enough to be one. Getting back into hiking is also quite high up on my list since I live relatively close to some lovely national parks and can’t remember the last time I spent any real time there. In December I will also be seeing The Cure live for a second time which is awesome!

See https://github.com/matrix-org/dendrite. Dendrite is a Matrix Homeserver. ↩︎

Homelab - Part 5: Reset, Renovate, Rebuild

Thu, 13 Jan 2022 00:00:00 +0000

Checkout my homelab GitOps here: https://github.com/Jonnobrow/coffee-shop

Since my previous post not a great deal has changed in my homelab. I have been busy with work and had some other projects that have interested me more.

Here is a quick rundown of what I have added/changed since part 4.

Reset

I wiped out a bunch of services that I wasn’t using and now just have my core, daily use services:

Nextcloud¹: File storage and photo ingress for me and my partner
PhotoPrism²: Photo management and sharing
Paperless³: Document Manager
Mealie⁴: Recipe Manager and Meal Planner
Jellyfin⁵: Media System
I also have a bunch of utilities and background services

Renovate

I now self-host renovate and have a bot user dedicated to operating that.

The rationale behind self-hosting was to learn more about the bot rather than using the Github action in which I would only have had to do the repository specific config rather than the bot config. I also wanted to learn more about cron jobs in k8s.

Rebuild

I went back to the roots of why I started a homelab and made space for learning again.

A lot of my services were barely being used and had run their course in terms of teachable moments. They went in the bin and will hopefully get replaced with new things, some possiblities are:

Dendrite⁶: A matrix homeserver
Drone⁷: A CI/CD server
Gitea⁸: A git server
Vault⁹: A secrets management tool

Homelab - Part 4: From the ground up

Sat, 30 Oct 2021 00:00:00 +0000

For the last month I have been working on restructuring my Kubernetes cluster to make everything a bit more organised and sensible. During the process I also scripted the setup of the host using Ansible and made some changes to the way I do things.

If you’re just interested in seeing the GitOps repository, you can check that out here: https://github.com/Jonnobrow/coffee-shop.

Motivation for change

Before I dive in to the new structure of my cluster, the things I have added and the things I have changed, I want to explain why this was necessary (or why I felt it was anyway).

Three main things prompted the changes:

Messy Manifests: I mentioned this towards the end of my second post and although functionally everything was fine, having your manifests organised makes everything a little easier to maintain.
Backups: Everything important is backed up so I can mess around with my cluster knowing a restore is (a) possible, and (b) quite straightforward thanks to the work I did in part 3.
Learning and Automation: My initial cluster had been somewhat adapted from a friends cluster¹, so I hadn’t really learnt all that much about what I was doing. Also, I had left a lot of manual processes creep into my maintenance and I wanted to learn how to remove as much of these as possible - automate everything!

Some new tools

A couple of new tools were integrated into my workflow during this process. Some replaced tools I was already using while others simply improved my quality of life or were necessary for the automation I was trying to accomplish.

Mozilla SOPS²

The Github repository describes SOPS (Secrets OPerationS) as being a “simple and flexible tool for managing secrets”. It supports a wide range of file formats and can encrypt using a few different mechanisms depending on your needs.

For me it replaces Bitnami Sealed Secrets³, but also gives me a way to encrypt secrets that don’t like within my cluster, like secrets used in Ansible deployments.

Some examples of usage:

To Encrypt a file in place: sops -e -i /path/to/file
To Decrypt a file in place: sops -d -i /path/to/file
To edit an encrypted file with $EDITOR: sops /path/to/file

FluxCD provides a guide⁴ on using Mozilla SOPS, as does Ansible⁵.

Task⁶

Task is a task runner … that aims to be simpler and easier to use than … GNU Make.

Instead on the Makefile format, Task uses yaml, which I am much more familiar with. It also has a more natural feel to usage (for me) and I found the ability to use templates, include other task files and define dynamic variables really useful for the kinds of things I wanted to do with it.

I predominantly use Task for running small admin tasks like:

Enabling/Disabling Nextcloud Maintenance Mode
Running an Ansible playbook
Running a flux reconciliation

The documentation⁶ explains how to install and use Task and I recommend checking it out for a little bit more automation in all of your projects. For an example of multiple task files, see my GitOps repository.

pre-commit⁷

This tool managed pre-commit hooks for Git. These hooks mean that on every commit my code or manifests are run through a linter and any other tools I want.

So far I am only using this for a few checks for my manifests such as:

Checking for trailing whitespace and line endings
Checking for consistent yaml using yamllint⁸
Checking for unencrypted secrets⁹

However, in the future I can see me using this to run linters and tests before committing python code and much more.

A small configuration file is included in my GitOps repository and simply running pre-commit install-hooks will add the hooks to that repository, causing them to run on every git commit command, stopping the commit if a check fails.

Still Messy Manifests

My original plan was to just put the manifests in to a new directory structure, perhaps move around some services into more sensible namespaces and then be done. However, I am somewhat prone to diving deep into the rabbit hole¹⁰ and found myself looking at other GitOps Kubernetes Clusters on k8s-at-home/awesome-home-kubernetes. This quickly led to me excitedly creating a new branch on my own GitOps repository and the procrastination had officially begun.

A new cluster is born

I run a single node cluster, so by “a new cluster” what I really mean is “a new node”. In order to test the re-done manifests I decided it was best to create a separate environment because I wasn’t sure how long it would take and I didn’t want to be without some of my services for an unknown amount of time.

For ease, I decided to look into using k3s¹¹, rather than doing everything from scratch, and here begins the first descent into the rabbit hole.

Many of the aforementioned repositories contain a directory with deployments using either Ansible, Terraform or a combination of the two. Of course, I liked the idea of having a single command I could run that would install dependencies, configuration files and setup k3s - so I made it happen. It was a chance to learn some more about Ansible¹² and also make my life easier in the future should I decide to move to new hardware or rebuild my cluster again.

So after about two days work I had a new cluster up an running, but still hadn’t done anything that I originally set out to do, oh well.

Making use of Kustomize

Another thing that was common across most of the repositories linked in the awesome-home-kubernetes project was the use of Kustomize¹³ and the Flux Kustomize Controller¹⁴.

Kustomize is a powerful tool that can do some really cool stuff like generating secrets and config maps from the actual files, so you aren’t managing two copies of something. It can also apply patches, perform replacements and much more. When coupled with the Kustomize Controller for flux it allows me to have quite fine-grained control over my manifests and I had to have that.

These are the things I mainly use Kustomize for:

Decrypting secrets
Substituting cluster wide values in all my manifests
Applying the resulting manifests

Kustomizations also mean that I can specify exactly which manifests should be applied, rather than the alternative which seemed quite flakey to me. I can reconcile a whole kustomiztion in the knowledge that my volumes, secrets and configs will also get applied whereas before only the helm release itself would be updated.

Once again, this is all very cool but at this point I’m three days in an not a lot has changed in the way of removing messy manifests.

Setting up Tasks and SOPS

Of course, things must come in threes, so here we are with a third change before the real work even begins. As mentioned in the “Some new tools” section, I wanted to use SOPS to manage secrets and Task to run tasks. I set up both of these tools when creating the Ansible deployments so that I could easily keep secret variables and run playbooks. However I needed to add a couple more tasks and generate some extra keys before I could use everything with Kubernetes.

I wrote a couple of tasks that simply ran Flux commands I use all the time, so instead of running flux reconcile source git flux-system I could type task flux:sync which saves me quite a few key presses over a day of reconciliations.

Additionally, I followed the instructions on the flux website⁴ for using SOPS and generated a separate key that the cluster would use to decrypt secrets. I then added some tasks to my task files for generating those secrets again in the future, although hopefully I will never have to as that would be very bad! I also created a task to generate the secret in the cluster that flux uses to decrypt everything and that was that.

Finally getting somewhere

Now that there is new cluster, I have decided to use Kustomize and I’ve set up some new tools, I can finally start re-organising my services. I started by copying a structure from the repos I was using as inspiration.

/coffee-shop
├── cluster
│ ├── apps
│ ├── base
│ ├── core
│ └── crds
├── server # Notes live elsewhere, see above
└── Taskfile.yml # Contains tasks

Let’s break that down a bit:

cluster contains all the manifests for my K8S cluster
- apps contains the services I am running on the cluster and inside lives a directory for each namespace, which then holds a directory for each service. More on that later.
- base is the directory that flux uses as a source. It contains the cluster settings and secrets, the core, apps and crd kustomizations and the gotk manifests that contain everything required by Flux. I also put other sources for my services, such as helm repositories, here.
- core contains the namespaces themselves and the cert-manager manifests. It gets applied before anything else as the namespaces and cert-manager are crucial to the whole cluster.
- crds contains custom resource definitions used by some of the services. Although all of the services have an option to manage crds with the helm chart, by having them here I can ensure they exist and have finer control over their versions.
server contains the Ansible roles, playbooks and inventory files
Taskfile.yml contains tasks for the project
- There is also a .taskfiles directory with more task files that relate to specific things like nextcloud or ansible.

Okay, so now a nice new structure is in place, what does a typical service look like in that structure. Here is an example using Jellyfin:

/coffee-shop
└── cluster # Top Level
└── apps # Jellyfin is a service, so under apps
└── media # Jellyfin is a "Free Software Media System"
├── _pvc # Namespace level Persistent Volume Claims
└── jellyfin # Jellyfin get's its own directory
├── helm-release.yaml # The helm release itself
├── kustomization.yaml # A kustomization to say what should be included
└── config-pvc.yaml # Service level Persistent Volume Claims

By grouping resources into common namespaces I avoid having separate persistent volume claims and persistent volumes for each service. I have six services in the media namespace that all use the same volumes, but I only have one definition for each, not six like in my old setup.

The logical next step would be moving everything over, but of course the rabbit hole re-opened and I couldn’t help but jump in.

Switching to Traefik¹⁵

My cluster had been running fine using ingress-nginx¹⁶ for a few months, so I had no real reason to change to a different proxy. However, the main aim of all of this is to learn AND to follow trends, so naturally I took a look at something new that goes by the name Traefik. Traefik is “The Cloud Native Application Proxy” and provides a lot of nice features:

Routing and load balancing
Security: HTTPS, Let’s Encrypt Support and More
Configuration: Service Auto-Discovery and Middlewares
Observability: Built-in dashboard, real-time metrics

Further to this, Traefik seems to be the in-thing so with the aim of staying somewhat current with my homelab, I decided to move to it.

The process didn’t go off without a hitch though. For most services it was as adding some annotations for Traefik service discovery. Something like:

annotations:
 traefik.ingress.kubernetes.io/router.entrypoints: "websecure"
 traefik.ingress.kubernetes.io/router.middlewares: "list,of,middlewares"

However, for services like Nextcloud that required more configuration, I had to define middlewares (see my nextcloud middleware here) in order to replace inline nginx configuration snippets in my old setup. This took a bit of working out and quite a few commits to get sorted.

As for some of the benefits, I could now define middlewares as an extra layer of security. For example, I have a middleware that only allows traffic from cloudflare IP addresses, for my external traffic, and another than only allows RFC1918¹⁷ IP addresses for my local traffic. I could also then define multiple ingresses for some of my services, like those with a /api path so that only things on my local network could access the API, and only traffic through cloudflares proxy could access the rest¹⁸.

https://github.com/sseneca/sserver ↩︎
https://github.com/mozilla/sops ↩︎
https://github.com/bitnami-labs/sealed-secrets ↩︎
https://fluxcd.io/docs/guides/mozilla-sops/ ↩︎ ↩︎
https://docs.ansible.com/ansible/latest/collections/community/sops/docsite/guide.html ↩︎
https://taskfile.dev/#/ ↩︎ ↩︎
https://pre-commit.com/ ↩︎
https://yamllint.readthedocs.io/en/stable/index.html ↩︎
https://github.com/k8s-at-home/sops-pre-commit ↩︎
a complexly bizarre or difficult state or situation conceived of as a hole into which one falls or descends, especially one in which the pursuit of something leads to other questions, problems or pursuits. — Merriam-Webster ↩︎
https://k3s.io/ ↩︎
https://www.ansible.com/ ↩︎
https://kustomize.io/ ↩︎
https://fluxcd.io/docs/components/kustomize/ ↩︎
https://traefik.io/traefik/ ↩︎
https://kubernetes.github.io/ingress-nginx/ ↩︎
https://en.wikipedia.org/wiki/Private_network ↩︎
I appreciate that this might be possible with nginx but the simplicity and declarative approach with Traefik makes it much nicer for me. ↩︎

Keeping a Sourdough Starter Alive

Thu, 21 Oct 2021 00:00:00 +0000

A little about my starter

The Lost Boys: A few years ago I tried to create a couple of starters from flour, water and air. These were known as Peter Pain¹ Mk1 and Mk2. The fact I tried more than once says it all - making a sourdough starter from scratch isn’t easy. Both sadly never made anything more than a boule or two, all of which resembled discuses². So I resorted to plan B.

Peter Pain Mk3: Around October 2019 I purchased a small envelope of dried sourdough starter from John Kirkwood³ via his website profoodhomemade.com. I’ve fed it almost every week and have used it to bake some great bread (if I do say so myself) and a wide array of other things. It is still going strong and I highly recommend getting a well established starter if you want to take up baking sourdough.

Looking after a Sourdough Starter

I recently changed how I look after my starter and now use a technique called the scrapings method which I learnt from Bake with Jack⁴. I have included both methods below as they both have a use.

Before getting to the methods, there are a few things that are useful to know when looking after a starter:

How much does your jar weigh? This can come in handy when feeding the starter with either method. By knowing the weight I can put the jar on the scales, deduct the jar weight and know how much starter is there. It saves decanting starter to a new container to feed.
How often are you going to bake? If you bake more regularly you’ll have to feed more regularly and may want to keep your starter at room temperature to encourage growth. If you bake less frequently (like me) you’ll have to feed less and should keep your starter in the fridge to slow growth.
A liquid on top of your starter isn’t a problem. After 5 days or so, my starter usually starts to get a thin layer of liquid. Give it a sniff and it should smell sweet and yeasty, this is fine and can be stirred back in before discard/feeding.
What ratio is your starter? If you made it yourself you probably know this, but the ratio refers to the amount of your starter that is flour versus the amount that is water. My starter (and most others) are 1:1, meaning equal parts flour and water.
What flour should you use? I use white bread flour, as this was what my starter was made with. If you used a different flour to create your starter, feed it with that.

What is a Feed?

A feed simply means giving the starter new flour and water so it can continue to grow. The ratios may vary depending on your answers above, but for me (1:1 ratio) I feed as follows:

Mix 1 part starter with 2 parts filtered water (doesn’t have to be perfectly mixed)
Add 2 parts flour and mix until no dry flour is visible
Scrape down the sides of your container to stop starter drying onto the sides
Loosely cover and put in a safe place

This same formula is used for making a levain, which is then used to make bread.

(#1) The Scrapings Method

This method is less wasteful since you don’t really discard any starter. It also means you don’t have to find a load of recipes to use up your discard and is very easy to pull off.

A day or so before you wish to bake with the starter, feed the starter so that it is active. The first time you do this you will have to discard, but subsequently you should have the perfect amount for a feed in the jar.
On bake day, use the active starter straight from the jar to make your bread.
After using the starter for baking, simply scrape down the sides of the jar and put it in the fridge ready for next week.
When next week comes around you can start at step one again.

Example:

I need 200g of starter to make my bread.
I use 50g of starter plus 100g each of flour and water to make a starter on Friday morning.
On Saturday I use 200g of the starter to make my bread and make sure to scrape down the sides of the container so my starter doesn’t dry out.
I have 50g of starter in the fridge ready for next weeks feed.

(#2) The Discard Method

This method is great if you want to make lots of things with the discard (more on that later). It is also useful if you bake less frequently than once a week. I still use this method when I don’t bake in a given week and I need to recharge my starter, or if I don’t use the full amount of starter I made.

Weigh the jar with the starter in and deduct the jar weight. This is how much starter you have. From this calculate how much to discard.
- Example: Total weight = 500g and jar weight = 250g, then starter weight = 250g, so discard = 200g to have 50g for feed.
- Example 2: Total weight = 410g and jar weight = 250g, then starter weight = 160g, so discard = 110g to have 50g for feed.
“Discard” the amount you calculated.
- For me this means putting it into a bowl and into the fridge so I can use it for something else. You could bag and bin it though if you aren’t going to use it.
- You can also use this “discard” to make a levain for baking or if the discard is active, even use that directly to bake.
Perform a feed and put the starter back in the fridge.

Example (No Bake):

I’ve not baked in a week and don’t plan on baking soon.
I weigh my starter and calculate that I have 210g.
I pour 160g into a bowl to make a pancake batter in the morning.
I feed the remaining 50g to make 250g total and store in the fridge.

Example (Bake):

I’ve not baked in a while but plan on baking this weekend.
I weigh my starter and calculate that I need to discard 200g.
I use 50g of discard to make a levain on Friday night ready to bake on Saturday.
I use the other 150g to make a spring onion and sesame pancake.
I feed the 50g in the jar and store in the fridge.

Both methods have a place and I used both when it’s needed. I try to waste as little as possible by using the scrapings method predominantly, but something I don’t bake and need to feed to keep everything alive and that is where the discard method comes in handy.

Some on my favourite Sourdough recipes

(Active) Sourdough Loaf

Mix 250g active starter with 700g room temperature filtered water until a few small chunks of starter remain.
Add 1kg of white bread flour and mix to combine, making sure no dry flour remains.
Cover and leave for 45 minutes.
Sprinkle over 20g of fine sea salt and 50g of room temperature water, dimple this into the dough with your fingers, then mix to incorporate.
Perform 3 stretch and folds over the next 2 hours then allow to bulk ferment until doubled in size.
Tip the dough out, split and shape into a shape of your choosing
Cover and keep in fridge overnight.
Bake for 45 minutes in a 200c oven.

(Active) Sourdough Pizza

Recipe by: Joshua Weissman

Mix 700g all purpose flour, 14g fine sea salt, 98g active starter and 490g water.
Perform 3 stretch and folds and 30 minute intervals to build structure.
Refrigerate dough overnight
Shape dough balls (Recipe makes 4)
Proof dough balls for 5 hours
Refrigerate until ready, up to 1-2 days

(Active) Sourdough Focaccia

Recipe: Alexandra Cooks

(Discard) Sourdough Crepe

Combine 1 egg, 150g sourdough starter and a splash of milk in a bowl and whisk to form a thin batter.
Melt 2 Tbsp of butter in a large frying pan or crepe pan.
Add melted butter to batter and stir to combine.
Add a thin layer of batter to the pan and cook until edges start to pull away from the pan, then flip and cook on the other side for 30 seconds to a minute.
Keep crepes warm in the oven and serve with your favourite toppings.

(Discard) Sourdough Spring Onion Savoury Pancake

Heat a little oil in a small to medium non-stick frying pan over medium heat.
Pour the discard directly into the pan.
Add sliced spring onions and a generous sprinkle of sesame seeds to the top and cover for 3 minutes.
Flip and cook for another 2-3 minutes or until onions are dark brown and caramelised.
Serve with sweet chilli dipping sauce or a mixture of soy sauce, sriracha and sesame oil

pain: French for bread and pronounced pan ↩︎
discus: A disk, typically wooden, plastic or rubber with a metal rim, that is thrown for distance in athletic competitions. ↩︎
Retired professional chef turned YouTuber ↩︎
Baking YouTube Channel ↩︎

Homelab - Part 3: Backups

Thu, 30 Sep 2021 00:00:00 +0000

As mentioned in Part 2 of this series, backups were a priority of mine over the last few weeks. Although I was backing up some data when I wrote that article, not everything was being backed up properly and nothing was being backed up offsite.

That’s all changed now, and this post will explain the process I went through to get a reasonable backup strategy for my homelab in place.

My Existing Backup “Strategy” and the Goal

My existing backup strategy consisted of a cronjob running a simple script 8 times a day (every 3 hours). That script runs a Borg¹ backup which includes most of the key directories on my homelab, including all the data and database directories. The script also prunes the backup repository so that I only have 7 daily backups (one for each weekday), 4 weekly backups (one for each week of the month) and 6 monthly backups.

This works well and gives a me a range of points to restore from if I make a mistake or something bad happens. However, the backup repository is local, meaning it sits on the same drive as the data originally came from, so in the case of hard drive failure, I’d lose everything.

The 3-2-1 Backup Strategy² is widely regarded as being pretty air-tight and is used by a lot of the self-hosting and homelab community, not to mention in business. The 3-2-1 strategy has 3 parts:

3 copies of your data
2 different mediums/devices
1 copy off-site

So to quickly compare my old strategy to 3-2-1:

I have: 2 copies of my data and need 3
I have: 1 medium and need 2
I have: 0 copies off-site³ and need at least 1

The goal is therefore to sort out some off-site location for my most important data and continue doing local backups, but change the destination to a different medium/device.

Exploring strange new worlds

One thing I wanted to be certain of was that my choice of backup software was sound. Therefore, before putting work into everything else I looked into some other incremental backup solutions.

Aside from Borg¹ there are several backup solutions such as Duplicati⁴ and Restic⁵. Both initially appeared worth a shot but after a little research it became clear Duplicati wouldn’t meet my needs since I wanted to do local backups too.

Restic offers much the same as Borg, and even uses similar terminology and commands, making it quite easy to try out. I got everything setup and working nicely but unfortunately ran into some resilience issues with the backup process. My poor home broadband means that disconnections and timeouts are a regular occurrence and I need something that can handle this nicely.

I persevered with Restic for a little over a week, but after failing to find a solution to my problem I returned to Borg. Frequent cut outs happened with Borg too, which I was able to resolve with some SSH options⁶, but partial backups are stored with Borg so after restarting the backup, everything just continues from where it left off.

I found that there isn’t much to choose between Borg and Restic, but this little difference was quite important for me. Restic is worth a look and I will certainly revisit it in the future, but for now Restic is futile and Borg shall live on.

Where do off-site backups live?

The logistics of the off-site part was probably the biggest challenge of the whole backup overhaul. As much as possible I want to be in control of my data and that means not using big cloud services, hence I’m self-hosting everything. Therefore, using a cloud service as a backup destination is not ideal. My ideals coupled with the cost of storage in the cloud is probably why it took so long to get this sorted.

Here were the options I considered:

Amazon S3 (or similar) due to the relatively low cost of storage and my familiarity with S3 from an internship. Sadly individual requests have an associated cost in Amazon’s billing model and that means doing frequent backups, as I do, could cost a fortune in the long run. It is also difficult to predict costs, especially at this early stage in my homelab.
IDrive/Backblaze/Crashplan… since they seem to be at the top of every list entitled “Best Backup Service 2021”. These are very nice solutions and allow for huge amounts of data, but I don’t need more than 100GB at the moment for all of my most important files. Additionally I want a place I can host a Borg repository, but most of these backup solutions have some proprietary software or a lot of features I’ll never use and that is reflected in the price.
Dropbox/Google Drive (or similar) is another option that seems popular. Dropbox would let me keep 2TB of stuff in the cloud for £7.99 a month⁷, but the same issues that I have with those dedicated backup solutions still exist. I’m a DevOps engineer, I’m not afraid to get my hands dirty and I want something fairly basic that I can build on not something that provides everything imaginable.
Hetzner Storage Box is a much simpler solution with a wide range of price points and ways to access the box. Hetzner also seems to be highly thought of in the cloud hosting space. They offer everything from 100GB to 10TB and at reasonable prices.

For my current usage, a 100GB BX10⁸ storage box from Hetzer, at a little less than £3 a month, is the best choice. It was easy to setup and I can use SFTP/SSH to access the data on the box. There are no overcomplicated extras, its simple and that’s all it needs to be. Another solution might be the better for me in the future though as the quantity of data I’m backing up increases.

Ansible Roles and Automation

I have a few different machines that I’d like to start backing up including my personal laptop, desktop and homelab server. Therefore I would have to set up a backup on each of the machines and any time a change in my backup setup occurs I’d have to propagate that change to each machine. This caused me to revisit an old friend by the name of Ansible to try and automate the installation (and also the running) of my backup setup.

Restic was still being tested when I started playing with Ansible and I discovered a pretty decent role on Github that I was able to use as a starting point for my own backup role.

I wanted a few things from this role:

To be as platform agnostic as possible (especially Linux/MacOS)
To handle general backups but also more service specific backups
To be very configurable as my use on different machines varies

My initial task was to adapt a backup script I have been playing around with to a Jinja2 template so that Ansible could populate it with the details of whichever repository was configured. Then I added a Nextcloud backup script which I will detail more in the next section. Finally I added in a playbook, default configuration and inventory file and was able to start testing.

My playbook installs an SSH key and configuration for the remote Hetzner storage box. It then creates backup scripts for each configured repository and a separate script for the Nextcloud backup. Next a cronjob is added for each of the scripts at a configurable interval. I have been using this for a few days and everything has gone smoothly so far.

In addition to the basics, both Restic and Borg behaviour can be used and if Borg is used a set of init scripts are generated to create the initial repository if it does not already exist.

More detail on the configuration and usage of this Ansible role can be found on my Github⁹.

It is definitely not perfect and still a work in progress, but it will do for now and as I learn more about Ansible and my needs it will improve.

Nextcloud Specific Backups

Nextcloud is probably tied first for the most important thing to backup for me - photos being the joint most important. Until this point Nextcloud had been backed up as a folder like everything else, which isn’t good as it can lead to inconsistencies in files which are then part of my backups. Additionally the database was being backed up as a folder too, which again is far from the recommended way to do that.

In order to safely backup Nextcloud you must enable maintenance mode, then backup the config, data and theme folders (or more if you want) and finally dump the database¹⁰. Then you can disable maintenance mode and continue using Nextcloud.

To enable maintenance mode on my Nextcloud instance I must exec into the Kubernetes pod and run a command, something like

kubectl exec -n nextcloud $POD -- \
 su - -s '/bin/sh' www-data -c \
 "php -d memory_limit=-1 /var/www/html/occ maintenance:mode --on"

which runs the command on the remote pod as the www-data user with php memory limit disabled (this was necessary at time of writing).

Once this is done I can backup the data folders and then run another command on the postgres pod to dump the database

kubectl exec -n nextcloud $POD -- \
 /bin/bash -c "/opt/bitnami/postgresql/bin/pg_dump -Unextcloud nextcloud" \
 > nextcloud.bak.sql

I combined these commands into a Nextcloud backup script and piped the output of the database dump to Borg/Restic, as both support backups from stdin. Now my data from Nextcloud is backed up safely.

Next Steps

Make service specific scripts for Jellyfin and Photoprism
Tidy up the Ansible role
Add some resilience to the scripts
Add general purpose database dumping to the ansible role

I will update this list as I tick off these items :)

https://www.borgbackup.org/ ↩︎ ↩︎
A decent article about 3-2-1 ↩︎
Not quite true as most of my data is still in the cloud somewhere until I get this self-hosting stuff sorted. ↩︎
https://www.duplicati.com/ ↩︎
https://restic.net/ ↩︎
See this issue: https://github.com/borgbackup/borg/issues/636 ↩︎
https://www.dropbox.com/buy ↩︎
https://www.hetzner.com/storage/storage-box ↩︎
https://github.com/Jonnobrow/ansible-backup ↩︎
https://docs.nextcloud.com/server/latest/admin_manual/maintenance/backup.html ↩︎

Homelab - Part 2

Thu, 02 Sep 2021 00:00:00 +0000

Since my previous post I have spent a fair amount of time tinkering with my homelab. In part 1 I hadn’t even built a server yet and now I have a number of services running that my partner and I use on a daily basis. This post includes some of the ups and downs of that tinkering and details the services I am hosting and my plans for the future.

The Good, the Bad and the Ugly

The Good: Shortly after I released the first post in this series the components for my server arrived. I have built a few PCs before and enjoyed the build process, which, in the world of plastic constructions toys, I would equate to Duplo as opposed to Lego, although the stakes can be a little higher. The entire build took a little under two hours, and the system booted straight into the PVE live USB I had plugged in.

The Bad: In my eagerness to get started I neglected to plan as thoroughly as perhaps I should have. I sped through the guided installer for Proxmox¹ and set up my first virtual machine. Once Ubuntu 21.04² had installed, I SSH-ed into the VM and installed the required applications to get a Kubernetes cluster up and running. This may all sound as though it was going well, but in the setup I failed to set up static IP addressing for both the Proxmox host and the Kubernetes VM. This would later result in me having to basically start the installation from scratch, with a much more rigidly defined IP addressing strategy for my home network. The re-installation of Kubernetes went smoothly, as it had before, and I had a cluster up and running.

The Ugly: At this point in the story, my experience goes from limited to non-existent. I knew that I wanted to use GitOps³ where possible and had a pretty decent idea of what I wanted to host, but claiming I understood more than that would be straight up lying to you. I would like to stress the fact that this is exactly the reason why I was doing any of this in the first place - to learn. My lack of knowledge did mean that what follows isn’t the best and my homelab is still nowhere near perfect or even good.

I chose FluxCD (v2)⁴ as my GitOps³ provider and used a friends manifests as the starting point for my own. The problem here arose from me adding in stuff for my own particular use case. I added in files and created Persistent Volumes and Persistent Volume Claims that were not needed. I had to install a local-path provisioner⁵ in order for volumes to be automatically created if I hadn’t explicitly defined them. Neeedless to say, it got quite messy, and my manifests are still rather disorganised and my volume management is far from ideal.
Data storage and backups was the next issue. Until this point my solution for backups left a lot to be desired. I had multiple copies of the really important stuff but a lot of those copies hadn’t been updated in a while and had therefore grown out of sync with the original, defeating the point of a backup. All of my data lived (and sadly still lives) on a 2TB Portable HDD. This gets mounted to the VM on boot and uses BTRFS (wanted to learn a bit about this too) subvolumes for each main category of data. I now run borg-backup⁶ 4 times a day to backup my photos and any essential documents and data from my cluster. The ugliness comes from the fact my backups live on the same portable HDD and that my database backups just copy the files rather than using the intended backup functions for postgres. Additionally, I make no use of BTRFS.
As for how the data is served to my services, I use NFS. NFS is perfect for my usage as I can mount exports on multiple services and have everything play nicely. The issue here is permissions, and this is such an issue that I cannot yet explain what is going on and has affected so much that it warrants it own bullet. I’m sure a future post will address my solution to this. However, for now it remains an issue that prevents Jellyfin from updating metadata, prevents me from directly managing my photo library or prevents Nextcloud from backing up photos from my devices.

The End Result

So after all of this about the highs and lows of my experience, you may be wondering what the end result is like. I changed my plans for what to host a little during the process since I discovered certain things that I needed sooner and found that a few services would not be required if I used something else (*cough cough* nextcloud).

So here is a bit about what I’m hosting, my thoughts on the service as a whole and how the setup and maintenance has gone.

Nextcloud⁷ is probably the most crucial service I host (except services I regard as being system services such as a load balancer and reverse proxy). It is a self-hosted cloud with an abundance of features.

I rely on Nextcloud for: storage of my documents, backup and sync of my photos, my calendar, my todo-list, my contacts, my quick notes, my RSS feed aggregation, my bookmarks and probably some more things that I can't remember right now.
I am very pleased with Nextcloud, the community surrounding it and the ongoing development and would highly reccomend taking a look at it if you are thinking of removing a cloud provider, like Google, from your day to day. And as far as maintenance is concerned, after you have done the initial setup the day to day is minimal and larger updates are generally pain-free too.
PhotoPrism⁸ is an app for managing and viewing your photo collection. For years my photos have lived on Google Photos or on my portable HDD. This worked for me and my family, but when Google announced changes to the Google Photos storage policy⁹ I knew it was time for a change.

I use photoprism to manage my photo library. I can put photos into albums, manage their metadata, archive lower quality snaps, share albums with family and have full size images forever with not 15GB limit or crazy compression.
PhotoPrism is okay, it’s not perfect but is still in the early stages of life and many features that would make it a really good Google Photos replacement are in the roadmap or under development. These include multiple user accounts, feeds (a bit like a personal Instagram/PixelFed) and more. It’s quite easy to setup and maintain, indexing takes a while and so does the initial photo organisation, but once that is done its trivial to maintain your library as a user.
Paperless-ng¹⁰ is a self-hosted document manager with a nice Django¹¹ web interface. I spun it up to have a play and its here to stay. I can scan documents with my phone or a dedicated scanner and then add the PDFs to a directory on Nextcloud which gets consumed by paperless.

I can tag documents, give them a correspondent, document type and date and also inform paperless of the keywords or phrases in documents that relate to the preceding things and paperless with use OCR[^fn:11] to work it out automatically in the future. It has meant I don't need to keep paper copies of a lot of things and can take all my documents with me wherever I go - super useful.
As for the maintenance, it requires very little and I am yet to encounter any issues with paperless whatsoever.
Jellyfin¹² is a free-software media system and was one of the few services I self-hosted prior to this new setup. My library was already setup and its something I use a lot so was probably the second thing I setup after Nextcloud.

I use it for Movies, Shows, Music, Podcasts and Audiobooks, but Jellyfin has support for some other things as well. It can also grab metadata from many sources, handle subtitles, stream to most devices and platforms and handle multiple users.
This is unfortunately a service I am experiencing issues with. Since setting it up on my new server I have yet to get metadata fetching working which leaves my library ugly and uninformative (but functional). I have experimented with several different fixes but I believe the issue lies with the permissions error mentioned earlier.

Tackling the Ugliness

Backups are my next priority. My services, from a user perspective, are working nicely and I have started relying upon them. Therefore, backing up data is super important to keeping the self-hosting dream alive. I need to work out a solution to creating offsite backups and will likely be looking into this over the next few weeks and months - so watch this space.

Once backups are sorted I will likely start tidying my manifests and organising my services a little better. Then I will tackle the permissions issues I am experiencing and try to get Jellyfin back to where it was before. I also want to set up a recipe manager (with shopping list support) in the near future. My personal website is also still on Gitlab pages, and while I don’t think it is necessary to self-host this, it would be nice if redirected to this site rather than just being a dead end.

I also want to start creating Ansible roles for certain things I setup. For example, a role to setup whatever my backup solution is and a role to setup wireguard. These are things I need on 4+ different machines (a couple of servers and a couple of personal devices) and therefore the time and effort of creating an Ansible role will pay off quickly.

Thats all for now, but I will hopefully have some more useful and interesting posts to share in the near future.

https://www.proxmox.com/en/proxmox-ve ↩︎
https://releases.ubuntu.com/21.04/ ↩︎
GitOps: versioned CI/CD on top of declarative infrastructure. Stop scripting and start shipping. - Kelsey Hightower ↩︎ ↩︎
https://fluxcd.io/docs/ ↩︎
https://github.com/rancher/local-path-provisioner ↩︎
https://www.borgbackup.org/ ↩︎
https://nextcloud.com/ ↩︎
https://photoprism.app ↩︎
https://blog.google/products/photos/storage-changes/ ↩︎
https://github.com/jonaswinkler/paperless-ng ↩︎
https://www.djangoproject.com/ ↩︎
https://jellyfin.org/ ↩︎

Homelab - Part 1

Tue, 18 May 2021 00:00:00 +0000

Now that I have finished university I have nearly 5 months of unemployment ahead of me before I start my graduate role at Sky. Therefore I need a project to keep my entertained. I have a huge list of things to get done, some of which will end up immortalised as blog posts on here. However, if you are reading this you are probably here to hear about the beginnings of my Homelab.

What is a Homelab

r/homelab provides the following definition¹:

Homelab [hom-læb](n): a laboratory of (usually slightly outdated) awesome in the domicile

So essentially, it is some hardware that you set up a home in order to:

Experiment with “enterprise” kit in your home
Learn about network infrastructure
Self-host your own apps and services

For me it’s the predominantly the last one, with a little of number 2 in there as well. I will get to play with the enterprise stuff at work and my degree has taught me about the underlying infrastructure, therefore its mainly the benefits of self-hosting and the DevOps experience I get along the way that I am interested in.

So to quickly summarise, a homelab is some tech you have at home that you can play around with, learn about and have fun with all at the same time.

The benefits of self-hosting

You own your data. If it’s on your hardware, then you own it. You aren’t giving your memories to Google or relying on Spotify and Netflix to have the music and tv you want to watch.
You learn a lot. Setting up some of this stuff isn’t easy and isn’t for beginners (there is a lot that is though) and it can be a bit of a steep learning curve. However the community is great and will probably help you if you ask 😜
You can make it your own. This may seem obvious, it may not, but self-hosting gives you that all important freedom to do whatever you like and with that comes the ability to tweak pretty much everything to your liking. There is a big overlap between free-software² and self-hosting so if you are so inclined you are almost certainly able to contribute to or fork a project so that a feature you have always wanted is included. And failing that, ask nicely and someone might get to it eventually.

My current experience

So you might be able to tell that this isn’t exactly new to me. Although this is first time I am inclined to call it a Homelab, I have had some form of self-hosting going for a little while. Mainly on Raspberry Pis and as virtual machines on my desktop PC, so I have been quite limited in what I can do, but its a great place to start if you aren’t sure it’s for you, or if you can’t afford to build or buy a server.

My credentials in the Homelabbing space are:

Self-hosted Nextcloud³, Minecraft, Jellyfin⁴ (Just for me on a RPi)
Deployed K8s⁵ clusters the hard way⁶ and with various scripts
Work experience with cloud technologies, docker and networking
My dissertation was half DevOps
I have a computer science degree in which I took some advanced networking modules

I definitely do not know everything, far from it and I would still call myself a Homelab beginner. A lot of the things I have set up were relatively easy and my practical experience is limited, hence Homelab - Part 1 and not Homelab - Part x.

The sad state of affairs that is my home network

I wish I could tell you my home network was dreamy with gigabit fibre broadband, high quality switches and access points, ethernet everywhere and all the bells and whistles.

However, the real state of my network is dire. I am a student still (technically) when writing this and I live in a flat, which I rent, therefore my ability to install ethernet, access points and the rest is limited.

I also had the misfortune of falling in love with a flat that receives a maximum of 10Mbps download and 1Mbps upload speed.

My home network actual consists of a Sky Hub with a single 25m ethernet cable clipped to the skirting board all the way to the office. In there it goes in to a 5 port gigabit switch and provides the upstream for my PC, a range extender (because we have concrete walls too 😦), the printer and my Raspberry Pi.

I do plan to upgrade all this in the future but I don’t see the point in getting good network hardware when the highest speed it will see is 10Mbps.

Building a “Server”

From my research it seems that there are four main choices when it comes to getting in to homelabbing. These are listed below in order of cost (both initial and ongoing, descending):

The first is to buy enterprise grade kit second or third hand from somewhere like eBay and deal with the incessant whining of high rpm fans and the power bill that will kill your WAF⁷.
(My choice) Buy components, either new or used, and build a server PC yourself.
Buy some kit like Raspberry Pis and have a play since they are cheap and low power. One day you might even get to a crazy setup that dwarfs enterprise solutions⁸.
The last is that you just had some old kit lying around like an old PC and you install an operating system on it an you’re good to go. Unfortunately I am not in that position and all the kit I have is either in use or useless.

So I decided to use consumer parts and build my own server PC. Although for similar money I could have purchased a rack mountable, ex-enterprise server, this would be extremely overkill for my needs and in the long run probably cost me an arm and a leg whilst killing baby seals in the process.

The Parts

Part Type	Part Choice	Rationale
CPU	Ryzen 3600	65W TDP and 12 Threads
Memory	16GB DDR4 3200Mhz (KLEVV BOLT X)	16GB seems like the minimum these days
Storage	500GB WD Blue SSD (For now)	Boot Drive and VM/OS Storage
PSU	AeroCool Integrator 500 W, 80+ Bronze	Cheap and reasonably efficient
Motherboard	Gigabyte B450M DS3H	64GB RAM, Good expandability
Case	Antec VSK-3000-Elite	4 3.5" Bays, Not Ugly and MicroAtx

What services will I be deploying?

I can divide the software portion into a few different categories. These being the host operating system, main virtual machines and then the services I will use internally and the services I will use externally.

Host Operating System

For the host operating system I will be using Proxmox Virtualization Environment (PVE)⁹ which is AGPLv3¹⁰ and is a distribution of linux based on Debian. It is virtualization management system that is based on QEMU/KVM, with which I am familiar, and will therefore allow me to create virtual machines easily. It also provides some nice management features and supports clustering with other PVE nodes for redundancy - something I may explore in the future.

My reasoning for choosing a virtualization environment over a server OS like Ubuntu 20.4 LTS is that I hope to run several virtual machines in the future including TrueNAS Core for data storage, pfsense for routing and K8s.

Main Virtual Machines

For now there will be only a single virtual machine, however as I explained in my rationale for the host operating system, I plan to add more in the future.

The one virtual machine will basically use all of the system resources not consumed by Proxmox and will house a K8s cluster. I have toyed with splitting my cluster into a few separate virtual machines but given that they will all be on the same hardware, the redundancy and high-availability benefits are void - it’s also just me using it and if it goes down, I just have to deal with it.

K8s is a container orchestration tool that will allow me to neatly run all of my services with much less overhead than if I provisioned virtual machines for them all. It is also a great technology to learn in 2021.

Utilities and Services for Internal Use

High Priority:

NGINX Proxy¹¹: Access to services
Wireguard¹²: Remote access to home network
Prometheus¹³: Monitoring
Grafana¹⁴: Dashboard for metrics
Syncthing¹⁵: File synchronisation between devices

Nice to Have:

Transmission¹⁶: BitTorrent client
Sonarr¹⁷: TV Show Management
Radarr¹⁷: Movie Management
Lidarr¹⁷: Music Management
Jackett¹⁸: Query aggregator for *arr apps above
Organizr¹⁹: Dashboard for services

Services for External Use

High Priority:

Jellyfin⁴: Movies, TV, Music
Nextcloud³
Tiny Tiny RSS²⁰: RSS Aggregator
My Personal Website

Medium Priority:

Photos: Photoprism (Synced with Nextcloud)
Prosody IM²¹: XMPP
Paperless-ng: Document manager

Nice to Have:

Recipe Sage²²: Recipe management and food planner
Sourcehut²³: Software Development

What’s next?

Well, the first thing is to build the PC. The parts are on the way and as soon as they arrive I will get started. Then I need to setup Proxmox and Kubernetes. After that I will get started on moving over some of the services I already self-host and the things that I have labelled as High Priority. Once all that is done I will move on to the other services, and eventually I will start adding in more VMs for things like TrueNAS Core and pfsense, but thats a while away yet.

Subscribe to my RSS feed so you don’t miss Homelab - Part 2.

https://www.reddit.com/r/homelab/wiki/introduction ↩︎
Free as in freedom. Free Software gives everybody the rights to use, understand, adapt and share software. https://fsfe.org/ ↩︎
https://nextcloud.com/ ↩︎ ↩︎
https://jellyfin.org/ ↩︎ ↩︎
K8s == Kubernetes https://kubernetes.io/ ↩︎
https://github.com/kelseyhightower/kubernetes-the-hard-way ↩︎
Wife Acceptance Factor https://openhomelab.org/index.php/WAF or https://en.wikipedia.org/wiki/Wife_acceptance_factor ↩︎
https://www.reddit.com/r/homelab/comments/jqh6m8/raspberry_pi_k8s_cluster ↩︎
https://pve.proxmox.com/wiki/Main_Page ↩︎
https://www.gnu.org/licenses/agpl-3.0.en.html ↩︎
https://nginx.org/en/ ↩︎
https://www.wireguard.com/ ↩︎
https://prometheus.io/docs/introduction/overview/ ↩︎
https://grafana.com/ ↩︎
https://syncthing.net/ ↩︎
https://transmissionbt.com/ ↩︎
https://wiki.servarr.com/Main_Page ↩︎ ↩︎ ↩︎
https://github.com/Jackett/Jackett ↩︎
https://github.com/causefx/Organizr/ ↩︎
https://tt-rss.org/ ↩︎
https://prosody.im/ ↩︎
https://recipesage.com/#/welcome ↩︎
https://sr.ht/ ↩︎

A brief history of my dotfile management

Fri, 14 May 2021 00:00:00 +0000

tl;dr

Over the past 2-3 years I have managed my dotfiles in various different ways. This post covers some of the highlights of that journey, and some tips for those just starting out on their configuration journey.

What is a dotfile and why would you manage them?

For the uninitiated amongst you, a “dotfile” is a configuration file for a tool, program or service on your computer. They usually refer to the user specific configuration and often on a Unix based operating system like MacOS or Linux. The name comes from the .(dot) usually at the start of the filename which makes the file “hidden” in most operating systems.

For example ~/.config/nvim/init.lua is the “dotfile” for my NeoVim configuration since it lives in the hidden “.config” directory.

As to why you should manage them, well you don’t have to and if you are using a flashy distribution of Linux where everything is configured with graphical user interfaces then chances are you will have very few dotfiles to manage even if you want to. In my situation, I run a more customisable distribution of Linux called Arch Linux¹ so there are more than a handful of dotfiles to keep track of. Furthermore my configuration changes on a regular basis and I might want to revert certain parts so version control if a really nice thing to have.

So here are my top reasons for managing dotfiles:

You can revert things that don’t go to plan
You can easily move dotfiles between machines (I run at least two machines at any given time)
You can keep your configuration in text format and not have to deal with configuration GUIs

So you may now be thinking “so how do I manage my dotfiles?” and here are some approaches I have experience with that will hopefully answer that question.

Symlinking

Symlinking (short for symbolic linking) creates a relationship between the real file and a linked file, allowing the same file to appear multiple times in your filesystem. This can be really useful for dotfiles since you can keep all your configurations in one place, say ~/dotfiles and then symlink those files to their required locations.

For example:

# ZSH Configuration
ln -sf ~/dotfiles/.zshrc ~/.zshrc

Now when we update the file in ~/dotfiles the changes will propagate to the relevant places and your config will be updated.

There are even tools to manage links like this, the most notable is Gnu Stow².

My personal experience is limited, I never got into Stow² and only briefly used the symlink method. Since I use linux and almost every tool in my day to day has a config file that lives in its own unique location I ended up confusing myself with symlinks and therefore I quickly started looking for something else.

Now I am not saying don’t use Stow² or plain old symlinks, but it wasn’t for me when I tried it.

Another downside I noticed is that you either have to script the linking or do it manually, which again I didn’t get along with.

`.git` at `~`

My second management tool involved creating a git repository in my home directory (~/).

This is a fairly simple concept. You create a git repository, add most things to .gitignore and then add your dotfiles relative to the home directory.

Another similar approach involves creating a bare repository, again in somewhere like ~/dotfiles, and then changing the work tree for the git operations to be ~/ or $HOME. There is a pretty good blog post by Marcel Krčah that explains it better than I can.

I have taken both approaches, for similar periods of time and neither stuck.

The first approach meant everything below my home directory was inside a git repository and this was far from ideal and meant that maintaining the .gitignore file was more than half the battle here. Furthermore, the process of using this repo when setting up a new machine was a little confused.

The second approach was arguably better, however the work tree still covered all of the home directory and there were special aliases required for calling git. In my mind, git should be git and not my dotfile manager. Git can play a valuable part in managing dotfiles and it essential to each of the strategies in this post, but it just isn’t what a configuration management tool should be (in my mind) on its own.

chezmoi part 1

Chezmoi³ solves some problems with previous setups. It is a tool that on the surface appears to be a wrapper for git, and in some ways it is, but it can manage secrets, run scripts and conditionally apply configuration. These features pulled me in because the state of my laptop and PC were largely separate apart from the basic building blocks.

Chezmoi allowed me to create a simple setup (which I ran on my laptop) and then add to that to create the setup I used on my PC. Further to this, I was able to use my same Chezmoi repo to manage the dotfiles on my Windows VM when I was working at my summer internship.

Although possibly biased, there is a comparison between Chezmoi and other dotfile managers that you can find here.

The reason I left Chezmoi for a period was that some features got a little abstract. My least favourite part was when managing dotfiles that were edited externally (not with chezmoi edit <filename>) since these would have to be added to chezmoi again or you would risk losing them next time you applied your changes. As a user of Emacs, where I would edit my dotfiles daily and from within Emacs, this was a pain.

The Ansible Era

During a boredom driven spiral through YouTube I stumbled across a series of videos by Jeff Geerling which go through basically everything you need to know about Ansible⁴, especially for a hobbyist like me. I had heard of Ansible⁴ prior to this point but had never really taken the time to learn about it, and Jeff seemed like a reputable source so I started to watch.

During the series he mentions at least once about managing his personal Mac with an Ansible playbook. So I set about looking for ways to do that myself. This is when I found myself clicking through a repository by kespinola on Github ⁵. These seemed a little more my speed as they were targeted towards linux, so I used the layout of his repository and copied the bin directory for the bootstrapping script.

I switched to Ansible when I found Chezmoi⁶ to be limited when it came to system operations such as installing packages. Ansible provides this in an idempotent fashion meaning that it won’t try to install stuff that’s already installed, it won’t copy files if they exist or, more generally, it won’t do something it’s already done. This saved some pain in trying to script that behaviour.

I left Ansible when it became too much effort to update dotfiles. It was perfect for the initial setup of a new machine but fell apart in the day-to-day. If you have a large number of machines that you want to be configured exactly the same way then Ansible is for you - otherwise you are better off with one of the other options in this post.

chezmoi part 2 electric boogaloo

Chezmoi didn’t change massively from part 1 other than a few nice features⁷. I had seen the full spectrum (email me if you think I haven’t) and had to choose and Chezmoi seemed like the lesser of many evils. It doesn’t do the system stuff but otherwise is the best option as it is relatively simple and doesn’t rely on symbolic links. As for the scripting I have switched to using a Makefile to run some simple shell scripts that install dependencies, start services, create directories and more.

Chezmoi isn’t perfect, but neither are any of the other options in this article, for me anyway.

The Takeaway

I am by no means an expert in dotfile management and have certainly not used all the possible approaches. There is a much more comprehensive list of tools at https://dotfiles.github.io/ which I recommend looking at if you don’t see something in this post you like the look of.

For me, Chezmoi is king at the moment. It offers me the things I need for my current situation and the near future. Ansible is a close second but is overkill for my personal dotfiles and I don’t set up new machines from scratch often enough to warrant it. A bare git repository is a cool, minimalist approach but only works if you have a single machine with the same structure, or you will be managing a bunch of branches and it gets needlessly complicated.

My Dotfiles

This is an ongoing project of mine. I continuously modify the configurations and scripts as my needs change with the view of improving my quality of life and making stuff look nice.

You can check them out at https://github.com/jonnobrow/dotfiles.

https://www.archlinux.org/ ↩︎
https://www.gnu.org/software/stow/ ↩︎ ↩︎ ↩︎
https://www.chezmoi.io/ ↩︎
https://www.ansible.com/ ↩︎ ↩︎
https://github.com/kespinola/dotfiles ↩︎
https://www.chezmoi.io/ ↩︎
Checkout Version 2 Features: https://www.chezmoi.io/docs/changes/ ↩︎

Emacs Fatigue

Wed, 28 Oct 2020 00:00:00 +0000

As users of Emacs¹ and Arch linux² will be aware, you can’t simply perfect your setup. There is almost always something that needs tweaking, or something new a like-minded FOSS-head is working on that you want to try instead. This is all good fun until you reach the point where it all feels a bit excessive and overly specific to one tool and you enter a depressive funk about your setup.

This hasn’t happened to me before, but it’s happening to me now and the following is my journey to this point and some thoughts on the issue.

From `vim` to `VimWiki`

When I first switched from ~~Windows~~ to Linux I started using vim to take notes. They were simple and written in Markdown. It was very pure and frankly I do miss the simplicity of it. vim is very lightweight and this approach required little to no plugins to be effective.

However as my notes started to grow in quantity and the complexity of each note evolved (I wanted LaTeX maths and code snippets), I started to look for something a little bit more. Enter VimWiki ³ , a “Personal Wiki for Vim”.

VimWiki lets you organise your notes and thoughts into a more structure knowledge base and this worked for me. I could create different files for everything and navigate through a file structure I have defined in my markup. I could also maintain to-do lists and write diary entries.

Sadly, VimWiki uses a custom markup language, and although Markdown is supported, I always felt like it just wasn’t quite right. So once again I set out to find something a little bit more. It wasn’t hard to find Emacs, but I wasn’t ready for that yet so I kept on with VimWiki until the summer, when I had far too much time on my hands and took the plunge into Spacemacs ⁴.

Slowly getting more EVIL (The Spacemacs Era)

The transition from vim to Emacs wasn’t particularly hard for me. EVIL⁵ mode and some of the easy beginner friendly features of Spacemacs like layers made it a smooth transition. I was also willing to sink time into it because I had read about the supposed benefits of org-mode and wasn’t going to miss out on the opportunity.

Spacemacs had the potential to be so much more that vim but I didn’t want to overdo it and reach the stage where the quote “a great operating system, lacking only a decent editor” applied to me. I made myself somewhat of a promise that I would use it only for things I could do in something like VS-Code or Atom, that included:

Note taking in plain text
Fully featured IDE for most if not all language I used
A terminal and file-tree
Syntax Highlighting
Some other nice-to-haves

I pulled this off quite easily, Spacemacs lets you drop in a list of layers and, due to its relatively wide adoption, a lot of these things already existed in a layer. Perhaps the most important thing on the list was note taking since I am mainly a student and note-taking forms a dominant portion of my week.

`org-mode` - your life in plain text

First off, I highly recommend org-mode ⁶ , it is flexible, simple and versatile and offers something for everyone if you are willing to put in the work. You can read more about it on their shiny new website.

When I first started using org-mode my setup consisted of a relatively small portion of the what was possible. I took what I was comfortable with in VimWiki and replicated it in org-mode but I also started using things like TODO keywords, scheduling and deadlines which helped me to keep track of work I had to do for Uni.

Another feature I really liked was org-babel because it let me keep bits of code in my notes, and execute them, and if necessary, export them to a source file. This was a little out of my initial scope for what a text editor needed to be, but wasn’t too far off, so I let it happen.

When you use org-mode every day, as I did and still do, you discover new things you can do and if you have the same disposition as me you will start to explore these things when you get a little time. Next I want to discuss the broader topic of Emacs and how I started to stray from the idea of a text editor.

EMACS Makes Any Computer Slow

There is a reason Emacs is more demanding than vi and that is because it can do so much more, it can become the aforementioned operating system if you make it. A few features took my fancy such as Email (mu4e) and PDF Browsing (pdf-tools).

I had been using mutt and then neomutt for a little while before I switched to Emacs, so mail in the terminal was no stranger to me. When I switched the Emacs I stuck with using a dedicated tool for mail for a couple of months, then finally caved - the first instance of wanting Emacs to do more that I had originally set out. Along comes mu4e which does everything neomutt did, it even uses the same tools in the background the sync mail, but it handles accounts in a cleaner way and I can write emails in org-mode so they are prettier, plus I can have images in my emacs buffers so all those modern emails don’t look as bad.

I felt good about my switch to mu4e because I had streamlined my workflow, I had removed an entire application from my day-to-day and replaced it with a package for emacs. I wanted more of this feeling so I started to add other little bits and pieces into my day-to-day until Emacs was more than I had agreed for it to be, but that was okay, I was streamlining my workflow, I felt more productive, I had to learn fewer specific tools, I knew Emacs.

I found that, if anything, Emacs made my computer faster because I could switch buffers in Emacs quicker than I could switch applications, or start a new application in Arch.

Social Networking for Thoughts (`org-roam`)

We are now at October 2020 in my story and I am using Emacs for a lot of things, I refuse to browse the internet in Emacs and a few other things, but most things happen in Emacs. A friend, @sseneca, mentioned to me that something called org-roam ⁷ existed, they hadn’t had the chance to explore it yet so I was the guinea pig and installed in on my setup.

org-roam is an org-mode remake of the popular Roam Research tool, which employs the Zettelkasten ⁸ method of note taking. Basically each note forms a node in the network of your knowledge, you can then form links between these notes and relate ideas together. Building your knowledge in this way makes it very easy to jump into your notes and find everything you have on a topic.

Since it was the start of a new year at Uni, now was the time to adopt it, so I did, I created a basic structure for all of my notes, and then started from scratch, and repeated that process probably three to four times before I got it just right. Now I am happy with the way I take notes and I feel that I am not just taking notes for a university module, I am recording my knowledge on a topic and its not just my “uni notes”, its my notes forever.

I even found something that lets you view all of you notes in a big interactive graph called org-roam-server and had that open in my browser most of the time when I was taking notes so I could see how my thoughts were building up, and get a kick of saying “wow, look at all then stuff I know!”.

We are all DOOM-ed

DOOM Emacs⁹ was the next thing, Spacemacs started to feel sluggish and wasn’t handling the masses of org-mode files I was accruing. So one afternoon in early October I ditched Spacemacs and said hello to DOOM, which is faster, more barebones and still lets me use vi bindings and easily manage packages.

Moving my configurations across meant I was looking into my configuration files for the first time in a little while, so a few things started to bug me and I started to go down the rabbit hole, I removed a lot of stuff, then added it back again, and then I decided to commit a weekend to making the org-agenda more useful to me, so that I could manage projects and schedule my uni work for my final year.

The Agenda Rabbit Hole

This is the last step before it all went wrong. I found some very well maintained dotfiles by a few different people ¹⁰ ^,¹¹ ^,¹² ^,¹². They had all adopted the Get Things Done way of staying organised and had really perfected the way they managed that in org-mode. One feature of org-mode that I hadn’t really spent any time on was the org-agenda and that seemed to be a focus of the Get Things Done¹³ way of life.

Another thing that came up was the ability to log your time against heading in org-mode, which I fancied doing after logging time for an internship all summer and finding it useful in terms of predicting what I could do in a day.

So it began, I sat down on a Saturday to setup the following things:

A tagging system
A file structure for my org-mode files
New TODO keywords and associated task flows
Custom agenda views for different purposes
Time clocking

It took all weekend but the result was amazing, I could see all the things I had to do next in one neat little view, I could see all the projects I was working on, I could see the things I had scheduled for today, I could see my deadlines for the next 30 days, and so much more. It was … perfect?

So where did it all go wrong?

I wish I understood why I feel like I do, but I don’t, I can only guess and try to solve the problem as I always do. So my number one theory at the time of writing this is: “emacs fatigue” which to me means that I have done so much with emacs that whenever I can’t do something with emacs I feel like all that work trying to make it perfect has been for nothing. It also means that I’ve run out of the easy wins, there aren’t many more untapped sources of pristine dotfiles out there. I have started writing my own functions and learning elisp, and it means the return of my investment is diminishing.

I think the trigger for all of these feelings came from the realisation I had strayed from my goal of keeping Emacs as a text editor and nothing more. I immediately decided that switching back to vim was the only option, and going cold turkey was the only way. I quickly realised how much I would miss org-mode and scurried back to the comfort of emacs but now with the feeling that even though I loved emacs it would never be perfect and there will probably never be the perfect tool for all the things I do on a computer.

I am too invested to just ditch emacs but I also know I have gone too far, but now I don’t have the time of the energy to dig myself out of the emacs hole I have dug for myself. The best I can do is to remove the things that aren’t “supposed” to be in emacs and hope the fatigue passes and one day in the future I build up the courage to streamline even further, so emacs will be nothing more than a note-taking tool and maybe a code editor.

Closing Thoughts

I love computers, I love Linux and I love Emacs but I was exposed to too much fun and exciting stuff in a short period of time and now there isn’t anything else for me to do. I also feel like what I have done isn’t enough because I am a perfectionist, but I know I have really done too much.

Therefore my only closing thought is that people like me need to spend time actually enjoying a new thing before adding something else to the mix, the Agenda Rabbit hole might have just been one step too far or maybe it was even org-roam. I was just as happy before org-roam as I was after it, perhaps a little more organised and productive but the same level of happy, so I could have relished that longer before working org-roam into my day-to-day and maybe I wouldn’t have crashed as hard, if at all.

So, slow down and enjoy the things you make, or, like me, you will experience Emacs fatigue.

Updates

Kicking the Fatigue (2020-11-05)

I think I have now overcome the fatigue, I am a little too busy to write a whole new post so I am going to briefly explain what I did in an updates section here.

I removed most of the features I talked about as being out of scope for a text editor. I can use the client in a browser or maybe setup neomutt again as and when I feel the need.
I also automated my dotfiles in an Ansible playbook (it will save me time in the long run) so that I am not so involved in the process of a re-install. Blog post coming soon.
I took a complete break, a few days off from everything (this way probably most effective).

I quit! (2021-03-15)

Emacs is a thing of the past for me. It just got too complicated and tried to do too much. Back to simple tools that do one thing well!