My Wireguard Setup

Disclaimer

Someone has been submitting my recent posts to online tech news aggregators, where they are criticized for not being cutting edge or paradigm shifting enough. If you’ve been led to believe that this post awe and amaze you, complain to the person who submitted it, not me. This is just my personal blog where I write about stuff that I’m doing, mostly technology based. It will not change your life. That said…

Background

I’ve had a “home server” for close to ten years now. It’s a Linux-based desktop pc. It acts as a file server, media server, backup server and a place to try out different things. I guess it’s what is now popularly called a “home lab”. All that’s great when I’m at home on my home network. I can stream movies and music, get files, ssh into the server and do whatever I need to do.

But when I’m out and about, traveling, working (when we used to go out and do stuff like that), I’d also like to have that same access. That’s all simple enough. You go into your router settings, do some port forwarding to that box and then you can stream, ssh, ftp, vnc, whatever. I’ve certainly done just that often enough. But as I became more security conscious, this started to worry me more and more. Having all those ports open into my main machine made me nervous. Yeah, they are behind passwords, or hopefully keys. I locked down ssh pretty tightly, but still worried about it, and all those other services. When I was on Xfinity for home internet, their management app provided a security section which listed all the various attempts to access different ports on the network with their IPs and locations. It was shocking. It became something that was not just theoretical. People were (and are) actually trying to hack into my network. That’s when I shut everything down.

Enter Wireguard

I’d heard quite a bit about Wireguard and it sounded like what I needed. I came upon this tutorial which described exactly what I wanted to do and in pretty clear terms:

https://zach.bloomqu.ist/blog/2019/11/site-to-site-wireguard-vpn.html

This all went together really well. It took a bit of learning and messing things up and fixing them, but I eventually got it all working really nicely and doing exactly what I need. Here’s my current setup:

  • Main wireguard server hosted on an inexpensive VPS in the cloud.
    • ufw set up to block all traffic other than specific ports from specific wireguard clients.
    • rinetd to forward any needed ports to my home server. Currently, that’s just the port that my airsonic server is running on.
  • Main wireguard server hosted on an inexpensive VPS in the cloud.
    • ufw set up to block all traffic other than specific ports from specific wireguard clients.
    • rinetd to forward any needed ports to my home server. Currently, that’s just the port that my airsonic server is running on.
  • wireguard client running on my home server.
    • airsonic music streaming server running there.
  • wireguard clients running on a couple of laptops, my Android phone and tablet. Each client has it’s own private key and the public key of the server. The server has its own private key and the public keys of each client.

With this setup I can ssh into the VPS from anywhere in the world, provided I’m doing it from one of the configured clients. Once I’m into the VPS, I can then ssh into any one of the other clients that has an ssh server running. I could use rinetd to forward ssh on specific ports to specific clients. But for now, that use case is not that common. When the world gets back to normal and I’m out of the house more, that will be useful.

I’ve got my airsonic server running on a specific port of my home server, let’s say it’s 1234. rinetd is set up to forward port 1234 on the VPS to port 1234 on the home server. So I can access my music in the browser from any wireguard client, or I can use any one of many subsonic-compatible Android apps and have my music streaming to my phone or tablet no matter where I am.

This setup is pretty flexible, and I will be able to add other services to it just by opening up a port in ufw and forwarding it as needed using rinetd. Important thing to remember is that when I say “opening up a port in ufw” I mean a wireguard client accessible port. Nothing is open on the VPS except via wireguard. Nothing is open on my home server except via the VPS or local LAN.

Monitoring and Recovery

One downside to this setup is that to access my music for example, I’m relying on a chain of multiple links: wireguard on VPS, ufw, rinetd, wireguard on home server, airsonic. If any one of those doesn’t function just right, I’m listening to silence. This has happened a couple of times, especially when I first set things up and had some things not quite right. Actually, if ufw goes down, I’ll still be able to listen to my music, but my VPS will be open. So I wanted to get some monitoring in place. When things were down early on, I’d be making assumptions on which piece was broke and spending time trying to fix it, only to find out it was one of the other links. With correct monitoring, I can now tell exactly what is up and down.

Monitoring with Healthchecks

I’ve been a big fan of Healthchecks.io. You set up “checks” which provide you with a url to ping. If a check doesn’t get a ping within a specified time period, it notifies you via email, sms, or through more than twenty other integrated services. I’ve been using it to monitor my daily backups. If a backup doesn’t happen at a specified time, I know about it.

So I set up a cron job that runs a script every 10 minutes on my VPS, and a similar one on my home server. This script first checks the status of wireguard. If it’s up, it pings Healthchecks. It does the same for rinetd and ufw. My home server checks wireguard and airsonic. Each of these five services is set up as a separate check in Healthchecks so I can see the status of each of them separately. The cron job runs every 10 minutes, so I give it one extra minute leeway – if Healthchecks doesn’t get a new ping after 11 minutes, that service is marked as down.

Recovery

Eventually I realized that if a particular service was down, once I became aware of it, I’d just go to whatever machine and restart it, so why not just do that automatically. So I built that into each of my checks.

If, say, wireguard is down on the VPS, it will NOT send the ping to Healthchecks. So a minute or so later it will be flagged as being down. But in this case, the script will also automatically try to restart wireguard. The next time it runs (10 minutes later), hopefully it sees that wireguard is up and sends the ping.

Healthchecks also has a “grace period” configuration. Once it notices something is down, it will not alert you until that grace period is done. I set this to 10 minutes. This results in the following sequence if something goes down:

  1. Service X is up and Healthchecks gets pinged at 10:00 pm.
  2. Service X goes down at 10:05 pm.
  3. At 10:10 pm, the script sees that Service X is down and fails to ping Healthchecks.
  4. The script also attempts to restart Service X.
  5. At 10:11 pm Healthchecks has not had a ping in 11 minutes and marks Service X as down.
  6. At 10:20 pm, the script runs again. Service X is up so it pings Healthchecks, which marks Service X as up again.
  7. Alternately, the restart didn’t work and at 10:20 pm no ping is sent.
  8. In this alternate case, at 10:21 pm, Healthchecks emails and texts me about the fact that Service X is down.

A potential improvement to this is that after step 4, when Service X is restarted, I could verify that it’s now working and ping Healthchecks. immediately. This way, if the restart works, nothing is marked as down. But I’m going to run it as is for a while and see how this works out. So far, so good.

I’ve gone through and tested each on of these checks, turning the service off and leaving it off. Within 11 minutes it was marked as down and restarted. And shortly thereafter marked as back up. All automatically.

If this were some kind of public service or mission critical workflow, I could easily set up the pings for every minute or so. But the 10 minutes seems perfectly adequate for my purposes.

More Details?

This post is pretty high level. Most of what went into the wireguard setup is covered in the above link. If you want to set up something similar, I’d be happy to go into more detail on any specific points. Just let me know.

version 1.3

Me again, talking about this silly version program still.

Actually, there are some pretty cool updates over the past few point releases. They came fast and on the heels of each other. The idea was posed to use the Linux package manager – apt or pacman or whatever – to get data on a program instead of relying on a hard-coded list.

Background

After some back and forth I warmed up to the idea, but as a backup to the known program list, not as a replacement. My reasoning is that you might have multiple versions of foo installed. Maybe one was through the default package manager, one through some download-and-run-an-install-script method. They might get installed to different locations in your PATH. But when you call foo on the command line, you’ll only get one of them.

If you query the package manager, it’s going to tell you about the one that it knows, which may or may not be the default. But when you run foo -v on the command line, you will get the one that’s going to be actually run in most cases. So that should be the first place we look. If version doesn’t know about foo then it can turn to the package manager.

Details

I decided to tackle two of the major Linux package managers first – apt (used on Ubuntu and most other Debian derivatives) and pacman (used on Manjaro and other Arch derivatives).

On apt, you can find info about a package, say neovim, you’d type:

apt list neovim --installed

This will give you something like:

neovim/focal,now 0.4.3-3 amd64 [installed]

That 0.4.3-3 is the version number that we’re looking for. It took a bit of regex trickery, but I was able to parse that bit out of it.

On pacman you’d type pacman -Qi neovim and the result would look something like:

Name : neovim
Version : 0.4.4-1
Description : Fork of Vim aiming to improve user experience, plugins, and GUIs
Architecture : x86_64
URL : https://neovim.io
Licenses : custom:neovim
Groups : None
Provides : vim-plugin-runtime
Depends On : libtermkey libuv msgpack-c unibilium libvterm luajit libluv
Optional Deps : python-neovim: for Python 3 plugin support (see :help python)
xclip: for clipboard support on X11 (or xsel) (see :help clipboard) [installed]
xsel: for clipboard support on X11 (or xclip) (see :help clipboard) [installed]
wl-clipboard: for clipboard support on wayland (see :help clipboard)
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 20.45 MiB
Packager : Sven-Hendrik Haase svenstaro@gmail.com
Build Date : Wed 05 Aug 2020 04:16:43 AM EDT
Install Date : Fri 21 Aug 2020 07:37:52 AM EDT
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature

So we can use grep and/or sed to find the one line of that which starts with Version: and grab the 0.4.4-1 part of it.

I then did basically the same thing for dnf which is the package manager on Redhat, Fedora, and derivatives.

So the process is:

  1. Check to see if version already knows about the program. If so, just do what it already does.
  2. If now, check apt, pacman and dnf. First we can just check to see if each one of those exist and only run the one that does exist. It’s unlikely that many people will have more than one of those. If we find one of those, we do the parsing and spit out the version it tells us about.
  3. If those all fail, then we can just tell the user we couldn’t find any information on that command.

Can we do more?

There are all kinds of other package managers on both Linux and Mac. I started making a list of the different ways you can find and install software and came up with

  • snaps
  • flatpaks
  • pip
  • npm
  • homebrew / linuxbrew

There are others, but those all cover a huge amount of ground. And it turns out that most of them were able to be solved with the same general strategy:

  • Does this package manager exist?
  • Does it know about this program and what info does it have?
  • Parse out the version number from the info it returns.

So, now version supports all of those. It just looks at each one of them in turn until it finds one that give an answer.

This also has the added functionality of being able to return the version of more than just executable programs. Package managers know about various libraries and other assets that aren’t directly executable or don’t have any way of querying them directly for their version. But version can tell you about them. Want to know what version of libusb you have installed? Typing version libusb will tell you.

A personal perk of doing this project is that I was forced to really learn grep and sed. Two programs that ranged from confusing to very mysterious in my mind. Now I get them and really like them. I wrote something up about them too: https://www.bit-101.com/blog/2020/09/grep-and-sed-demystified/