Some years ago, I was leading a tech team in a non-profit organization, and I made a pretty severe mistake. In a nutshell, I had configured a Postgres Docker container, but the path of the persistent volume was misconfigured. As a consequence, all data (which I thought was safely stored in disk) was actually in Docker memory. One day, months after the initial setup, I made an innocent restart, and all of a sudden the data was gone. Of course, I didn’t have backups configured.

I never managed to recover that data (to this day I don’t think that’s even possible), and I have to admit that was one of the most stressful situations I had in months. In retrospect, it’s obvious I should have had backups from day one, but it’s not until you suffer the pain of not having them that you realize how important they are. Of course, I knew that could happen, and of course I knew I should have made backups, but for some reason you always think that will happen to someone else, and developing xyz is more urgent than having a proper backup system.

After a sincere apology with the team whose work I had thrown to the bin, I worked on a backup procedure, and it was alright.

Multiple network interfaces

When I started self-hosting some services, I always kept this story in the back of my mind. Not long after taking it seriously, I got a NAS and started using it for regular backups. In this post, I want to reflect some learnings and pieces of configuration that I found relevant while doing so.

My homelab server was initially connected via Wi-Fi, which I know, is terrible in terms of speed. It was not until I realized there are RJ45 sockets on the wall that I wired the server, getting a dramatic speed increase. Because I had two network interfaces (wired and wireless), I also had 2 IPs.

My NAS was configured to only accept connections from one of them, so sometimes the connection worked, and sometimes it didn’t. When I reviewed the logs, I realized some forbidden connections came from one of the two IPs, so I had to update the config and everything worked like a charm.

Out of space

I also started getting errors while storing the backups of a little Moodle instance I recently set up. The reason was that I had configured a maximum storage of 10GB for the shared folder, and over the weeks it got full. I followed this forum thread where the author explains how to configure custom scheduled tasks that allowed me to delete old backups, making room for the new ones.

Mounting NFS directory

Another thing I had to change is the way I mounted the NAS volume via NFS for the borg backups of my Nextcloud instance. Out of pure laziness, I had first mounted the directory with a mount command, but that volume disappears on a restart. Months later (this morning, to be precise), I replaced that with an entry in the /etc/fstab file, restarted the server, and verified the volume was properly mounted.

This was not particularly difficult, but I suspect every time you face a problem for the first time it looks bigger than it actually is. When I was first mounting the backups, I had enough understanding NFS, SSH tunneling, etc. and I just couldn’t bear with another moving piece.

This was my little adventure for this Wednesday morning, after a sleepless night due to hot weather. I’m glad I fixed this once and (hopefully) for all, but I wouldn’t say no to a straight 8-hour-sleep night either.

Happy coding!