personal-site/content/Backup Strategy.md

Title: My Backup Strategy
Date: 2021-04-08
Category: Writing
Summary: Details about the backup system for all of my data.
Image: backup1.svg
Wide: true
Tags: feed

[TOC]

Regularly backing up all the data I care about is very important to me. This article outlines my strategy to make sure I never lose essential data.

## Motivation

Backups should be as automatic as possible. This ensures laziness and forgetfulness won't interfere with the regularity.

All software used to create and store the backups should be free and open source so I'm not depending on the survival of a company.

Backups need to be tested to ensure they are correct and happening regularly.  Multiple copies of the backups should exist, including at least one offsite to protect against my building burning down.

Backups should also be incremental when possible (rather than mirror copies) so an accidental deletion isn't propagated into the backups, making the file irrecoverable.

## Strategy

The key is to have one central location that all your files, projects, and data are cloned to and then back that directory up to multiple locations.

I have one backup folder `/mnt/backup` on my media server at home that serves as the destination for all my backup sources. All scheduled automatic backups write to their own subfolder inside of it.

This backup folder is then synced to encrypted 2.5" 1 TB hard drives which I rotate between my bag, offsite, and my parents' house.

![[backup1.svg | a diagram of my setup. servers and computers on the left, pointing to my home server in the middle, pointing to external hard drives on the right]]

## Backup Sources

I use the tool `rdiff-backup` extensively because it allows me to take incremental backups locally or over SSH. It acts very similar to `rsync` and has no configuration.

### Email

I have every email since 2010 backed up continuously in case my email provider disappears.

I use `offlineimap` to sync my mail to the directory `~/email` on my media server as a Maildir. Since offlineimap is only a syncing tool, the emails need to be copied elsewhere to be backed up. I run `rdiff-backup` from a weekly cron job:

<span class="aside">(I'll explain what backup_check.txt does below)</span>

```
*/15 * * * * offlineimap > /var/log/offlineimap.log 2>&1
00 12 * * 1 date -Iseconds > /home/email/email/backup_check.txt

20 12 * * 1 rdiff-backup /home/email/email /mnt/backup/local/email/
40 12 * * 1 rdiff-backup --remove-older-than 12B --force /mnt/backup/local/email/
```

Here's my `.offlineimaprc` for reference:

```
[general]
accounts = main
[Account main]
localrepository = Local
remoterepository = Remote
[Repository Local]
type = Maildir
localfolders = ~/email
[Repository Remote]
type = IMAP
readonly = True
folderfilter = lambda foldername: foldername not in ['Trash', 'Spam', 'Drafts']
remotehost = example.com
remoteuser = mail@example.com
remotepass = supersecret
sslcacertfile = /etc/ssl/certs/ca-certificates.crt
```

### Notes

I use Standard Notes to take notes and wrote the tool [standardnotes-fs](https://github.com/tannercollin/standardnotes-fs) to mount my notes as a file system to view and edit them as plain text files.

I take weekly backups of the mounted file system on my media server with cron:

```
00 12 * * 1 date -Iseconds > /home/notes/notes/backup_check.txt
15 12 * * 1 rdiff-backup /home/notes/notes /mnt/backup/local/notes/
```

### Nextcloud

I self-host a Nextcloud instance to store all my personal documents (non-code projects, tax forms, spreadsheets, etc.). Since it's only a syncing software, the files need to be copied elsewhere to be backed up.

I take weekly backups of the Nextcloud data folder with cron:

```
00 12 * * 1 rdiff-backup /var/www/nextcloud/data/tanner/files /mnt/backup/local/nextcloud/
30 12 * * 1 rdiff-backup --remove-older-than 12B --force /mnt/backup/local/nextcloud/
```

### Gitea

I self-host a Gitea instance to store all my git repositories for code-based projects. My home folder is also a git repo so I can easily sync my config files and password database between servers and machines.

I take weekly backups of the Gitea data folder with cron:

```
00 12 * * 1 date -Iseconds > /home/gitea/gitea/data/backup_check.txt
10 12 * * 1 rdiff-backup --exclude **data/indexers --exclude **data/sessions /home/gitea/gitea/data /mnt/backup/local/gitea/
35 12 * * 1 rdiff-backup --remove-older-than 12B --force /mnt/backup/local/gitea/
```

### Telegram

Telegram Messenger is my main app for communication. My parents, most of my friends, and friend groups are on there so I don't want to lose those messages in case Telegram disappears or my account gets banned.

Telegram includes a data export feature, but it can't be automated. Instead I run the deprecated software [telegram-export](https://github.com/expectocode/telegram-export) hourly with cron:

<span class="aside">(Saves the messages to a sqlite db)</span>

```
0 * * * * bash -c 'timeout 50m /home/tanner/opt/telegram-export/env/bin/python -m telegram_export' > /var/log/telegramexport.log 2>&1
```

It likes to hang, so `timeout` kills it if it's still running after 50 minutes.  Hasn't corrupted the database yet.

### Phone

[Signal Messenger](https://play.google.com/store/apps/details?id=org.thoughtcrime.securesms&hl=en_CA&gl=US) automatically exports a copy of my text messages database, and [Aegis](https://play.google.com/store/apps/details?id=com.beemdevelopment.aegis&hl=en_CA&gl=US) allows me to export an encrypted JSON file of my two-factor authentication codes.

I mount my phone's internal storage as a file system on my desktop using [adbfs-rootless](https://github.com/spion/adbfs-rootless). I then rsync the files over to my media server:

```
$ ./adbfs ~/mntphone 
$ time rsync -Wav \
  --exclude '*cache' --exclude nobackup \
  --exclude '*thumb*' --exclude 'Telegram *' \
  --exclude 'collection.media' \
  --exclude 'org.thunderdog.challegram' \
  --exclude '.trashed-*' --exclude '.pending-*' \
  ~/mntphone/storage/emulated/0/ \
  localmediaserver:/mnt/backup/files/phone/
```

Unfortunately this is a manual process because I need to plug my phone in each time. Ideally it would happen automatically while I'm asleep and the phone is charging.

### Miscellaneous Files

The directory `/backup/files` is a repository for any kind of files I want to keep forever. My phone data, old archives, computer files, Minecraft worlds, files from previous jobs, and so on.

All the files will be included in the 1 TB hard drive backup rotations.

### Web Services

Web services that I run like [[t0 Services]] and [[QotNews]] are backed up daily, weekly, and monthly depending on how frequently the data changes.

I run `rdiff-backup` on the remote server with cron:

```
00 14 * * * date -Iseconds > /home/tanner/tbot/t0txt/data/backup_check.txt                                                  

04 14 * * * rdiff-backup /home/tanner/tbot/t0txt/data tbotbak@remotebackup::/mnt/backup/remote/tbotbak/daily/t0txt/
14 14 * * * rdiff-backup --remove-older-than 12B --force tbotbak@remotebackup::/mnt/backup/remote/tbotbak/daily/t0txt/

24 14 * * 1 rdiff-backup /home/tanner/tbot/t0txt/data tbotbak@remotebackup::/mnt/backup/remote/tbotbak/weekly/t0txt/
34 14 * * 1 rdiff-backup --remove-older-than 12B --force tbotbak@remotebackup::/mnt/backup/remote/tbotbak/weekly/t0txt/

44 14 1 * * rdiff-backup /home/tanner/tbot/t0txt/data tbotbak@remotebackup::/mnt/backup/remote/tbotbak/monthly/t0txt/
55 14 1 * * rdiff-backup --remove-older-than 12B --force tbotbak@remotebackup::/mnt/backup/remote/tbotbak/monthly/t0txt/
```

The `tbotbak` user has write access to the `/mnt/backup/remote/tbotbak` directory only. It has its own passwordless SSH key that's only permitted to run the `rdiff-backup --server` command for security.

### Protospace

I run a lot of services for [[Protospace]], my city's makerspace.

The member portal I wrote called [[Spaceport]] creates an archive I download daily:

```
40 10 * * * wget --content-disposition \
  --header="Authorization: secretkeygoeshere" \
  --directory-prefix /mnt/backup/remote/portalbak/ \
  --no-verbose --append-output=/var/log/portalbackup.log \
  https://api.my.protospace.ca/backup/
```

The website and [wiki](https://wiki.protospace.ca) that I sysadmin both get backed up weekly:

```
0 12 * * 1 mysqldump --all-databases > /var/www/dump.sql
15 12 * * 1 date -Iseconds > /var/www/backup_check.txt
20 12 * * 1 rdiff-backup /var/www pshostbak@remotebackup::/mnt/backup/remote/pshostbak/weekly/www/
```

The Protospace [Minecraft server](http://games.protospace.ca:8123/?worldname=world&mapname=flat&zoom=3&x=74&y=64&z=354) I run gets backed up daily:

```
00 15 * * * date -Iseconds > /home/tanner/minecraft/backup_check.txt
00 15 * * * rdiff-backup --exclude **CoreProtect --exclude **dynmap /home/tanner/minecraft psminebak@remotebackup::/mnt/backup/remote/psminebak/
30 15 * * * rdiff-backup --remove-older-than 12B --force psminebak@remotebackup::/mnt/backup/remote/psminebak/
```

I also back up our Google Drive with rclone:

```
45 12 * * 1  rclone copy -v protospace: /mnt/backup/files/protospace/google-drive/
```

## Backup Copies

My backup folder `/mnt/backup` now looks like this:

```
/mnt/backup/
├── files
│   ├── docs
│   ├── phone
│   ├── protospace
│   ├── telegram
│   ├── usbsticks
│   └── ... and so on
├── local
│   ├── email
│   ├── gitea
│   ├── nextcloud
│   └── notes
└── remote
    ├── portalbak
    ├── pshostbak
    ├── psminebak
    ├── tbotbak
    └── telebak
```

This directory tree is the master backup and I make a copy of the entire tree every Saturday to a hard drive.

The directory is copied over with the following script:

```text
#!/bin/bash

cryptsetup luksOpen /dev/sdf external
mount /dev/mapper/external /mnt/external

time rsync -av --delete /mnt/backup/local/ /mnt/external/backup/local/
time rsync -av --delete /mnt/backup/remote/ /mnt/external/backup/remote/
time rdiff-backup --force -v5 /mnt/backup/files/ /mnt/external/backup/files/

python3 /home/tanner/scripts/checkbackup.py

umount /mnt/external
cryptsetup luksClose external
```

I wrote a Python script `checkbackup.py` that goes through each backup and compares the timestamp in `backup_check.txt` files to the current time. This makes sure that the cron ran, backups were taken, and transferred over correctly.

## Rotating Hard Drives

I rotate through 2.5" 1 TB hard drives each Saturday when I do a backup. They are quite cheap at [$65 CAD](https://www.memoryexpress.com/Products/MX65194) each so I can have a bunch floating around.

I keep one connected to the server, one in my bag, one offsite, one at my mother's house, and one at my dad's house. Every Saturday I run the script above to take a copy and then swap the drive with the one in my bag. It then gets swapped when I visit my offsite location. Same for when I visit my parents<span class="aside"> (I go back home about twice per year)</span>. This means that all hard drives eventually get rotated through with new data and don't sit too long unpowered.

The drives are all encrypted with full-disk LUKS encryption using a password I'm unlikely to forget.

I run the check-summing `btrfs` file system on them in RAID-1 to protect against bitrot. This means I can only use 0.5 TB of storage for my backups, but the data is stored redundantly.

Here's how I set up new hard drives to do this:

```
$ sudo cryptsetup luksOpen /dev/sdf external
$ sudo mkfs.btrfs -f -m dup -d dup /dev/mapper/external
$ sudo mount /dev/mapper/external /mnt/external/
$ sudo mkdir /mnt/external/backup
$ sudo chown -R tanner:tanner /mnt/external/backup
$ sudo umount /mnt/external
$ sudo cryptsetup luksClose external
```

## Future Improvements

I'm working on a system to automatically back up all my home directories to my media server. I need this to grab Bash histories and code that's work-in-progress. I've been burned by not having this once when a server died.

I'd like to automate backing up my phone by connecting it to a Raspberry Pi when I go to sleep.

I need to get better at fully testing my backups by restoring them on a blank machine.