Add margin notes

eee -> ddd
fff -> eee

content/backup-strategy.md

@@ -0,0 +1,341 @@
+Title: My Backup Strategy
+Date: 2021-04-08
+Category: Writing
+Summary: Details about the backup system for my data.
+Wide: true
+
+[TOC]
+
+Regularly backing up all the data I care about is very important to me. This
+article outlines my strategy to make sure I never lose essential data.
+
+## Motivation
+
+Backups should be as automatic as possible. This ensures laziness and
+forgetfulness won't interfere with the regularity.
+
+All software used to create and store the backups should be free and open source
+so I'm not depending on the survival of a company.
+
+Backups need to be tested to ensure they are correct and happening regularly.
+Multiple copies of the backups should exist, including at least one offsite to
+protect against my building burning down.
+
+Backups should also be incremental when possible (rather than mirror copies) so
+an accidental deletion isn't propagated into the backups, making the file
+irrecoverable.
+
+## Strategy
+
+I have one backup folder `/mnt/backup` on my media server at home that serves as
+the destination for all my backup sources. All scheduled automatic backups write
+to their own subfolder inside of it.
+
+This backup folder is then synced to encrypted 2.5" 1 TB hard drives which I
+rotate between my bag, offsite, and my parent's house.
+
+## Backup Sources
+
+I use the tool `rdiff-backup` extensively because it allows me to take
+incremental backups locally or over SSH. It acts very similar to `rsync` and has
+no configuration.
+
+### Email
+
+I have every email since 2010 backed up continuously in case my email provider
+disappears.
+
+I use `offlineimap` to sync my mail to the directory `~/email` on my media
+server as a Maildir. Since offlineimap is only a syncing tool, the emails need
+to be copied elsewhere to be backed up. I run `rdiff-backup` from a weekly cron
+job:
+
+<span class="aside">I'll explain what backup_check.txt does below</span>
+
+```
+*/15 * * * * offlineimap > /var/log/offlineimap.log 2>&1
+00 12 * * 1 date -Iseconds > /home/email/email/backup_check.txt
+
+20 12 * * 1 rdiff-backup /home/email/email /mnt/backup/local/email/
+40 12 * * 1 rdiff-backup --remove-older-than 12B --force /mnt/backup/local/email/
+```
+
+Here's my `.offlineimaprc` for reference:
+
+```
+[general]
+accounts = main
+[Account main]
+localrepository = Local
+remoterepository = Remote
+[Repository Local]
+type = Maildir
+localfolders = ~/email
+[Repository Remote]
+type = IMAP
+readonly = True
+folderfilter = lambda foldername: foldername not in ['Trash', 'Spam', 'Drafts']
+remotehost = example.com
+remoteuser = mail@example.com
+remotepass = supersecret
+sslcacertfile = /etc/ssl/certs/ca-certificates.crt
+```
+
+### Notes
+
+I use Standard Notes to take notes and wrote the tool
+[standardnotes-fs](https://github.com/tannercollin/standardnotes-fs) to mount my
+notes as a file system to view and edit them as plain text files.
+
+I take weekly backups of the mounted file system on my media server with cron:
+
+```
+00 12 * * 1 date -Iseconds > /home/notes/notes/backup_check.txt
+15 12 * * 1 rdiff-backup /home/notes/notes /mnt/backup/local/notes/
+```
+
+### Nextcloud
+
+I self-host a Nextcloud instance to store all my personal documents (non-code
+projects, tax forms, spreadsheets, etc.). Since it's only a syncing software,
+the files need to be copied elsewhere to be backed up.
+
+I take weekly backups of the Nextcloud data folder with cron:
+
+```
+00 12 * * 1 rdiff-backup /var/www/nextcloud/data/tanner/files /mnt/backup/local/nextcloud/
+30 12 * * 1 rdiff-backup --remove-older-than 12B --force /mnt/backup/local/nextcloud/
+```
+
+### Gitea
+
+I self-host a Gitea instance to store all my git repositories for code-based
+projects. My home folder is also a git repo so I can easily sync my config files
+and password database between servers and machines.
+
+I take weekly backups of the Gitea data folder with cron:
+
+```
+00 12 * * 1 date -Iseconds > /home/gitea/gitea/data/backup_check.txt
+10 12 * * 1 rdiff-backup --exclude **data/indexers --exclude **data/sessions /home/gitea/gitea/data /mnt/backup/local/gitea/
+35 12 * * 1 rdiff-backup --remove-older-than 12B --force /mnt/backup/local/gitea/
+```
+
+### Telegram
+
+Telegram Messenger is my main app for communication. My parents, most of my
+friends, and friend groups are on there so I don't want to lose those messages
+in case Telegram disappears or my account gets banned.
+
+Telegram includes a data export feature, but it can't be automated. Instead I
+run the deprecated software
+[telegram-export](https://github.com/expectocode/telegram-export) hourly with
+cron:
+
+```
+0 * * * * bash -c 'timeout 50m /home/tanner/opt/telegram-export/env/bin/python -m telegram_export' > /var/log/telegramexport.log 2>&1
+```
+
+It likes to hang, so `timeout` kills it if it's still running after 50 minutes.
+Hasn't corrupted the database yet.
+
+### Phone
+
+[Signal
+Messenger](https://play.google.com/store/apps/details?id=org.thoughtcrime.securesms&hl=en_CA&gl=US)
+automatically exports a copy of my text messages database, and
+[Aegis](https://play.google.com/store/apps/details?id=com.beemdevelopment.aegis&hl=en_CA&gl=US)
+allows me to export an encrypted JSON file of my two-factor authentication
+codes.
+
+I mount my phone's internal storage as a file system on my desktop using
+[adbfs-rootless](https://github.com/spion/adbfs-rootless). I then rsync the
+files over to my media server:
+
+```
+$ ./adbfs ~/mntphone 
+$ time rsync -Wav \
+  --exclude '*cache' --exclude nobackup \
+  --exclude '*thumb*' --exclude 'Telegram *' \
+  --exclude 'collection.media' \
+  --exclude 'org.thunderdog.challegram' \
+  --exclude '.trashed-*' --exclude '.pending-*' \
+  ~/mntphone/storage/emulated/0/ \
+  localmediaserver:/mnt/backup/files/phone/
+```
+
+Unfortunately this is a manual process because I need to plug my phone in each
+time. Ideally it would happen automatically while I'm asleep and the phone is
+charging.
+
+### Miscellaneous Files
+
+The directory `/backup/files` is a repository for any kind of files I want to
+keep forever. My phone data, old archives, computer files, Minecraft worlds,
+files from previous jobs, and so on.
+
+All the files will be included in the 1 TB hard drive backup rotations.
+
+### Web Services
+
+Web services that I run like [txt.t0.vc](https://txt.t0.vc) and
+[QotNews](https://news.t0.vc) are backed up daily, weekly, and monthly depending
+on how frequently the data changes.
+
+I run `rdiff-backup` on the remote server with cron:
+
+```
+00 14 * * * date -Iseconds > /home/tanner/tbot/t0txt/data/backup_check.txt                                                  
+
+04 14 * * * rdiff-backup /home/tanner/tbot/t0txt/data tbotbak@remotebackup::/mnt/backup/remote/tbotbak/daily/t0txt/
+14 14 * * * rdiff-backup --remove-older-than 12B --force tbotbak@remotebackup::/mnt/backup/remote/tbotbak/daily/t0txt/
+
+24 14 * * 1 rdiff-backup /home/tanner/tbot/t0txt/data tbotbak@remotebackup::/mnt/backup/remote/tbotbak/weekly/t0txt/
+34 14 * * 1 rdiff-backup --remove-older-than 12B --force tbotbak@remotebackup::/mnt/backup/remote/tbotbak/weekly/t0txt/
+
+44 14 1 * * rdiff-backup /home/tanner/tbot/t0txt/data tbotbak@remotebackup::/mnt/backup/remote/tbotbak/monthly/t0txt/
+55 14 1 * * rdiff-backup --remove-older-than 12B --force tbotbak@remotebackup::/mnt/backup/remote/tbotbak/monthly/t0txt/
+```
+
+The user `tbotbak` has write access only to the `/mnt/backup/remote/tbotbak`
+directory. It has its own passwordless SSH key that's only permitted to run the
+`rdiff-backup --server` command for security.
+
+### Protospace
+
+I run a lot of services for [Protospace](https://protospace.ca/), my city's
+makerspace.
+
+The member portal I wrote called [Spaceport](https://my.protospace.ca/) creates
+an archive I download daily:
+
+```
+40 10 * * * wget --content-disposition \
+  --header="Authorization: secretkeygoeshere" \
+  --directory-prefix /mnt/backup/remote/portalbak/ \
+  --no-verbose --append-output=/var/log/portalbackup.log \
+  https://api.my.protospace.ca/backup/
+```
+
+The main website and [wiki](https://wiki.protospace.ca) that I sysadmin gets
+backed up weekly:
+
+```
+0 12 * * 1 mysqldump --all-databases > /var/www/dump.sql
+15 12 * * 1 date -Iseconds > /var/www/backup_check.txt
+20 12 * * 1 rdiff-backup /var/www pshostbak@remotebackup::/mnt/backup/remote/pshostbak/weekly/www/
+```
+
+The Protospace [Minecraft
+server](http://games.protospace.ca:8123/?worldname=world&mapname=flat&zoom=3&x=74&y=64&z=354)
+I run gets backed up daily:
+
+```
+00 15 * * * date -Iseconds > /home/tanner/minecraft/backup_check.txt
+00 15 * * * rdiff-backup --exclude **CoreProtect --exclude **dynmap /home/tanner/minecraft psminebak@remotebackup::/mnt/backup/remote/psminebak/
+30 15 * * * rdiff-backup --remove-older-than 12B --force psminebak@remotebackup::/mnt/backup/remote/psminebak/
+```
+
+I also back up our Google Drive with rclone:
+
+```
+45 12 * * 1  rclone copy -v protospace: /mnt/backup/files/protospace/google-drive/
+```
+
+## Backup Copies
+
+My backup folder `/mnt/backup` now looks like this:
+
+```
+/mnt/backup/
+├── files
+│   ├── docs
+│   ├── phone
+│   ├── protospace
+│   ├── telegram
+│   ├── usbsticks
+│   └── ... and so on
+├── local
+│   ├── email
+│   ├── gitea
+│   ├── nextcloud
+│   └── notes
+└── remote
+    ├── portalbak
+    ├── pshostbak
+    ├── psminebak
+    ├── tbotbak
+    └── telebak
+```
+
+This directory tree is the master backup and I make a copy of the entire tree
+every Saturday to a hard drive.
+
+The directory is copied over with the following script:
+
+```text
+#!/bin/bash
+
+cryptsetup luksOpen /dev/sdf external
+mount /dev/mapper/external /mnt/external
+
+time rsync -av --delete /mnt/backup/local/ /mnt/external/backup/local/
+time rsync -av --delete /mnt/backup/remote/ /mnt/external/backup/remote/
+time rdiff-backup --force -v5 /mnt/backup/files/ /mnt/external/backup/files/
+
+python3 /home/tanner/scripts/checkbackup.py
+
+umount /mnt/external
+cryptsetup luksClose external
+```
+
+I wrote a Python script `checkbackup.py` that goes through each backup and
+compares the timestamp in `backup_check.txt` files to the current time. This
+makes sure that the cron ran, backups were taken, and transferred over
+correctly.
+
+## Rotating Hard Drives
+
+I rotate through 2.5" 1 TB hard drives each Saturday when I do a backup. They
+are quite cheap at [$65 CAD](https://www.memoryexpress.com/Products/MX65194)
+each so I can have a bunch floating around.
+
+
+I keep one connected to the server, one in my bag, one offsite, one at my
+mother's house, and one at my dad's house. Every Saturday I run the script above
+to take a copy and then swap the drive with the one in my bag. It then gets
+<span class="aside">I go back home about twice per year</span>
+swapped when I visit my offsite location. Same for when I visit my parents. This
+means that all hard drives eventually get rotated through with new data and
+don't sit too long unpowered.
+
+The drives are all encrypted with full-disk LUKS encryption using a password I'm
+unlikely to forget.
+
+I run the check-summing `btrfs` file system on them in RAID-1 to protect against
+bitrot. This means I can only use 0.5 TB of storage for my backups, but the data
+is stored redundantly.
+
+Here's how I set up new hard drives to do this:
+
+```
+$ sudo cryptsetup luksOpen /dev/sdf external
+$ sudo mkfs.btrfs -f -m dup -d dup /dev/mapper/external
+$ sudo mount /dev/mapper/external /mnt/external/
+$ sudo mkdir /mnt/external/backup
+$ sudo chown -R tanner:tanner /mnt/external/backup
+$ sudo umount /mnt/external
+$ sudo cryptsetup luksClose external
+```
+
+## Future Improvements
+
+I'm working on a system to automatically back up all my home directories to my
+media server. I need this to grab Bash histories and code that's
+work-in-progress. I've been burned by not having this once when a server died.
+
+I'd like to automate backing up my phone by connecting it to a Raspberry Pi when
+I go to sleep.
+
+I need to get better at fully testing my backups by restoring them on a blank
+machine.

content/light-switch.md

@@ -27,4 +27,7 @@ by an AC-DC converter.
 This entire process happens quicker than half a second, so it feels instant.
 
 ![a rusted metalic hand]({static}/images/light-switch/light2.jpg)
+
+<span class="aside">Black stuff's liquid electrical tape</span>
+
 ![a rusted metalic hand]({static}/images/light-switch/light3.jpg)

content/linux-flavour.md

@@ -2,6 +2,7 @@ Title: Choosing a Linux Flavour
 Date: 2020-10-31
 Category: Writing
 Summary: A recommendation on which flavour of Linux to run.
+Wide: true
 
 [TOC]
 
@@ -14,6 +15,7 @@ I run Debian on my computers and servers.
 ## Linux Distributions
 
 When people refer to the "flavour of Linux" they are talking about a Linux
+<span class="aside">Interjection: it's technically called GNU/Linux</span>
 distribution (distro). It mostly describes what software is distributed in its
 software repository.
 

content/plant-waterer.md

@@ -8,7 +8,7 @@ It's also great for when I'm on vacation. The plant is a year old now and
 doesn't look as good as it used to (kinda like you). So this machine is like its
 life support.
 
-Update: this plant died long ago.
+<span class="aside">Update: this plant died long ago</span>
 
 ![the device and pump on a 2L pop bottle with a tube running to a flowerpot]({static}/images/plant-waterer/waterer1.jpg)
 
@@ -36,4 +36,6 @@ Another feature was the ability to run the pump backwards. This completely
 eliminated the siphoning problem from before. After pumping for a set duration,
 it would run backwards until the tube was cleared of water.
 
+<span class="aside">Also dead :(</span>
+
 ![the new version beside a big Min aralia plant]({static}/images/plant-waterer/waterer3.jpg)

themes/theme/templates/article.html

@@ -24,7 +24,25 @@
 
 {% endblock %}
 
+{% block info %}
+	Tanner Collin
+	<p class='theme-select'>
+		<a onClick="setTheme('light')">Light</a> / <a onClick="setTheme('dark')">Dark</a>
+	</p>
+{% endblock %}
+
 {% block content %}
+
+	{% if article.wide %}
+		<div class="content content-wide">
+	{% else %}
+		<div class="content">
+	{% endif %}
+
+		<div class="topbar">
+			{{ info() }}
+		</div>
+
 		<p><a href="/">← Return to Home</a></p>
 		<header>
 			<h1>{{ article.title }}</h1>
@@ -42,4 +60,5 @@
 		<article>
 			{{ article.content }}
 		</article>
+	</div>
 {% endblock %}

themes/theme/templates/base.html

@@ -49,42 +49,24 @@
 	{% else %}
 		<body>
 	{% endif %}
+
+	{%- macro info() -%}
+		{% block info %}{% endblock %}
+	{%- endmacro -%}
+
 	<div class="container">
 		<div class="sidebar">
-				<img src="/theme/me.jpg" class="me" alt="A picture of me smiling" />
+			{{ info() }}
-				<div class="info">
-					<p>
-						Tanner Collin
-					</p>
-					<p class="contact-icons">
-						<a href="mailto:site@tannercollin.com" rel="noreferrer noopener"><img alt="email icon" src="/theme/mail.svg" /></a>
-						<a href="https://t.me/tannercollin" target="_blank" rel="noreferrer noopener"><img alt="telegram logo" src="/theme/telegram.svg" /></a>
-						<a href="https://github.com/tannercollin" target="_blank" rel="noreferrer noopener"><img alt="github logo" src="/theme/github.svg" /></a>
-					</p>
-					<p class='theme-select'>
-						<a onClick="setTheme('light')">Light</a> / <a onClick="setTheme('dark')">Dark</a>
-					</p>
-				</div>
-			</div>
-			<div class="content">
-				<div class="topbar">
-					<img src="/theme/me.jpg" class="me" alt="A picture of me smiling" />
-					<div class="info">
-						Tanner Collin
-						<p class="contact-icons">
-							<a href="mailto:site@tannercollin.com" rel="noreferrer noopener"><img alt="email icon" src="/theme/mail.svg" /></a>
-							<a href="https://t.me/tannercollin" target="_blank" rel="noreferrer noopener"><img alt="telegram logo" src="/theme/telegram.svg" /></a>
-							<a href="https://github.com/tannercollin" target="_blank" rel="noreferrer noopener"><img alt="github logo" src="/theme/github.svg" /></a>
-						</p>
-						<p class='theme-select'>
-							<a onClick="setTheme('light')">Light</a> / <a onClick="setTheme('dark')">Dark</a>
-						</p>
-					</div>
 		</div>
+
 		{% block content %}
 		{% endblock %}
 	</div>
-		</div>
+
+	<p class="copyright">
+		© 2012–2021 Tanner Collin
+	</p>
+
 
 	<script>
 		function setTheme(theme) {

themes/theme/templates/index.html

@@ -7,7 +7,28 @@
 	<meta name="summary" content="The personal website of Tanner Collin." />
 {% endblock %}
 
+{% block info %}
+	<img src="/theme/me.jpg" class="me" alt="A picture of me smiling" />
+	<div class="info">
+		Tanner Collin
+		<p class="contact-icons">
+			<a href="mailto:site2@tannercollin.com" rel="noreferrer noopener"><img alt="email icon" src="/theme/mail.svg" /></a>
+			<a href="https://t.me/tannercollin" target="_blank" rel="noreferrer noopener"><img alt="telegram logo" src="/theme/telegram.svg" /></a>
+			<a href="https://github.com/tannercollin" target="_blank" rel="noreferrer noopener"><img alt="github logo" src="/theme/github.svg" /></a>
+		</p>
+		<p class='theme-select'>
+			<a onClick="setTheme('light')">Light</a> / <a onClick="setTheme('dark')">Dark</a>
+		</p>
+	</div>
+{% endblock %}
+
 {% block content %}
+	<div class="content">
+
+		<div class="topbar">
+			{{ info() }}
+		</div>
+
 	<p>
 		Hi, I'm Tanner! I do firmware and web development in Calgary.
 	</p>
@@ -15,7 +36,7 @@
 	<h2>Contact Info</h2>
 
 	<p>
-	Email: <a href="mailto:site@tannercollin.com">site@tannercollin.com</a> <br />
+		Email: <a href="mailto:site2@tannercollin.com">site2@tannercollin.com</a> <br />
 		Telegram: <a href="https://t.me/tannercollin" target="_blank" rel="noreferrer noopener">@tannercollin</a>
 	</p>
 

themes/theme/templates/style.css

@@ -15,11 +15,22 @@ a {
 pre {
 	font-size: 1rem;
 	padding: 1rem;
+	overflow-x: auto;
+}
+
+:not(pre)>code {
+	padding: 0 2px;
+	font-size: 0.9rem;
 }
 
 .container {
 	max-width: 56rem;
-	margin: 2rem auto 0 auto;
+	margin: 2rem auto 12rem auto;
+}
+
+.copyright {
+	font: 1rem/1.5 Apparatus SIL,serif;
+	text-align: center;
 }
 
 .theme-select {
@@ -124,6 +135,10 @@ pre {
 	max-width: 36rem;
 }
 
+.content-wide {
+	max-width: none;
+}
+
 .content p {
 	font: 1.2rem/1.5 Apparatus SIL,serif;
 }
@@ -141,6 +156,15 @@ pre {
 	height: auto;
 }
 
+.content .aside {
+	display: inline;
+	float: left;
+	position: relative;
+	width: 8rem;
+	margin-left: -9rem;
+	font-size: 1rem;
+}
+
 .toclink:not(:hover)::after {
 	visibility: hidden;
 }
@@ -169,7 +193,7 @@ pre {
 
 
 body {
-	background-color: #fff;
+	background-color: #eee;
 	color: #000;
 }
 
@@ -179,11 +203,15 @@ a {
 }
 
 pre {
-	background-color: #eee;
+	background-color: #ddd;
+}
+
+:not(pre)>code {
+	background-color: #ddd;
 }
 
 .toc {
-	background-color: #eee;
+	background-color: #ddd;
 }
 
 .content p.metadata {
@@ -192,18 +220,22 @@ pre {
 
 body.dark {
 	background-color: #000;
-	color: #fff;
+	color: #eee;
 }
 
 body.dark a {
-	color: #fff;
+	color: #eee;
-	border-bottom: 1px solid #fff;
+	border-bottom: 1px solid #eee;
 }
 
 body.dark pre {
 	background-color: #222;
 }
 
+body.dark :not(pre)>code {
+	background-color: #222;
+}
+
 body.dark .toc {
 	background-color: #222;
 }
@@ -222,7 +254,7 @@ body.dark .contact-icons img {
 
 @media (prefers-color-scheme: dark) {
 	body.light {
-		background-color: #fff;
+		background-color: #eee;
 		color: #000;
 	}
 
@@ -232,11 +264,15 @@ body.dark .contact-icons img {
 	}
 
 	body.light pre {
-		background-color: #eee;
+		background-color: #ddd;
+	}
+
+	body.light :not(pre)>code {
+		background-color: #ddd;
 	}
 
 	body.light .toc {
-		background-color: #eee;
+		background-color: #ddd;
 	}
 
 	body.light .content p.metadata {
@@ -253,18 +289,22 @@ body.dark .contact-icons img {
 
 	body {
 		background-color: #000;
-		color: #fff;
+		color: #eee;
 	}
 
 	a {
-		color: #fff;
+		color: #eee;
-		border-bottom: 1px solid #fff;
+		border-bottom: 1px solid #eee;
 	}
 
 	pre {
 		background-color: #222;
 	}
 
+	:not(pre)>code {
+		background-color: #222;
+	}
+
 	.toc {
 		background-color: #222;
 	}