The latest in VPN land
In a previous blog post I wrote about my current VPN setup and whether it might not be time to switch. I considered several options and tried almost all of them to get a feel for them.
wg-easy
This project advertises itself as the easiest way to run Wireguard. Indeed, it's simple to set up. You Start a Docker container and start connecting clients via the official Wireguard app. The setup is meant to be a hub and spokes architecture in which each client is a spoke that reaches out to the coordinating server (the hub). The container comes with a web interface to manage your clients and specify some settings.
I gave it a go and I quite liked it. It was just the right amount of simplicity and configurability that I was looking for. There was only one problem: at least one of my clients simply wouldn't connect to the rest. I tried various things, switches off firewalls but no matter what I tried, that one client just wouldn't accept connections.
Another potential issue is that the hub has to decrypt and reencrypt all traffic to pass it along to another client so it's not end-to-end encrypted. There are ways around that but that entails a level of complexity in the setup I wasn't prepared to deal with.
It is probably my fault for misconfiguring something but I was able to figure it out so I had to abandon this project, at least for now.
Netbird
This project seemed like the obvious drop-in replacement for Tailscale from the looks of things. It offers much the same feature-set as Tailscale does (at least in terms of features I'm interested in) with the biggest difference being that Netbird is fully open-source and offers a managed service but also allows self-hosting the complete stack. This involves an identity provider (Zitadel in this case) because, like Tailscale, you can only log in to the service via one of these.
The legwork to get up and running is almost completely done for you, You only have to download an installation script and it takes care of rendering a compose file, starting it up and configuring everything. The downside is, of course, that this makes it more tedious to change the standard configuration. I had to go through a few iterations of tearing it down and starting fresh to get everything set up how I liked it which, unfortunately, led to a run-in with the rate limiting of Let's Encrypt. How about that?
I do have to say, Netbird looks slick, has a good web UI and the configurability of the access management is better than Tailscale's. I was able to set up everything I wanted without trouble from the UI which is how I like it. I liked it so much that I decided to ho ahead and migrate to Netbird with all my clients. It wasn't difficult to do, I simply had to install the Netbird agent on all my devices, same as Tailscale. You notice a pattern here, I imagine.
And this is where things started to go south. It was all fun and games as long as I was only connecting my servers and laptop but as soon as my Android devices came into play, things got more rocky. The app is... not great. I mean, it does what is needed but it definitely looks like it could use some love and I'm apparently not the only one feeling that way. But the bigger problem was that I could not get all my devices to connect.
One of my Android devices simply would not connect to the network, no matter what I did. I tried different clients (there's a third-party one), playing with firewalls, reinstalling, everything I could think of but nothing. This really bummed me out, I thought I had found the perfect solution but what use is a VPN that won't connect all my devices?
Pangolin
This hadn't been on my radar originally but it was mentioned by people on Hybrid Cloud Show so I decided to have a go at it.
It took me a while to understand what this actually is because it's not a mesh VPN service like the others I was considering. As far as I understand this is more of a variant of Cloudflare Tunnels (which I've never used). Pangolin serves as public frontend to applications you host somewhere else. So you visit the URL of the service you want to access, are met with a Pangolin login page, authenticate to it and are then redirected to the service you want to visit. The connection between Pangolin and the backend service happens via Wireguard.
I thought this was nifty, it just has one issue: the vast majority of use cases I have don't involve web browsers but apps. And those apps don't expect to need a third-party login before being able to access the actual service (which usually needs a separate login) so there's something of a problem here.
It is possible to write rules and exceptions that let apps access the backend without having to authenticate but that seemed doomed to fail. Some of the stuff I run is actually documented but at the end of the day that would mean circumventing the thing that I actually wanted to implement: private access. Poking holes in the authentication process just seemed like a bad idea to me, also quite fiddly.
So, that's a no.
Headscale
And now we've come full circle. Well, almost. Headscale does what I want it to do and it seems to be reasonably simple to run and well-documented. That said, I struggled for an inordinate amount of time with the setup. I'm pretty sure I was being stupid or misunderstanding something basic in the whole process but I just couldn't figure out how to get Headscale to come up, listen on port 80/443, get a TLS cert and start doing its thing. I even asked for help on the fediverse but still couldn't figure it out.
That's when I abandoned this endeavour for a while to take a look at all the solutions I described above. After all of that came to nothing as well, I decided to have a look at a Docker-based setup. I threw in a reverse proxy and Headplane as a web UI for good measure and this time around it worked. I connected my clients, which is simple since the necessary software is already installed, I just needed to switch servers, and that was that, really. Just like that I had connectivity between my clients via a server of my own.
It's not all sun and rainbows, though. The Headplane UI is okay but not great. The way I set it up does not let me do a whole lot with it, it's mostly
used to look at my machines and their details. Headscale itself works fine but it lacks quite a few features that Tailscale has been pumping out recently.
For example grants in ACLs, serve with automatic TLS certs, tailnet lock and probably a bunch more. It's not a big deal, I'm happy with what I have,
the bigger issue is that because of this the Tailscale documentation does not necessarily apply anymore. I painfully learned this when writing ACLs, that
was way more hassle than I thought it would be.
Another things is that my clients frequently report issues of some kind after making the switch. Things like them not being able to connect to the configured DNS server (a node on my tailnet) or the coordination server. I don't know what's going on there because the connectivity doesn't seem to actually be impaired, everything works fine, it's still weird.
So am I happy? For the time being, yes. Will I stick with this solution long term? Not sure but I don't intend to switch in the near future at least.