A traditional VPN has one (or more) servers exposed to the Internet. Your work machine holds an encrypted tunnel with this VPN gateway, which will act as the hub of your internal infrastructure. Basically, it first exchanges some keys and then encrypts the traffic between the client and the server (OpenVPN uses TLS for both). This is the "hub-and-spoke" architecture.

Now, modernity asks for modern tooling. One of the Tailscale's selling points is that each participant can talk to each other directly with Wireguard. No intermediary or central servers involved. Only the control plane holds all the keys, and, most importantly, it's not hosted by you. The only requirement is to install Tailscale on both machines and adjust the access rules to allow the connection.

After reading the classic how Tailscale works blog post, you wonder how you lived without a mesh VPN. The promise of no single point of failure and fine-grained access to your infrastructure makes your eyes light up.

But then the enterprise life hits hard. You know that old Jenkins instance that the infra team is praying to sweet baby Jesus to keep running? What about the CRM that's held up by glue and duct tape. Are you able to install Tailscale on it? Of course not (this follows the Betteridge's law of headlines =)).

For a rapid migration, or for legacy infrastructure you can't touch, you install a component called subnet-router as a gateway to your internal networks. Clients will connect to this hub, and then forward them to your non-Tailscale infrastructure.

Hmmm…That walks and quacks like a hub-and-spoke VPN, doesn't it?

Remember how you were hoping for fine-grained access to your infra. Well, scratch that if you're going the subnet-router route. Either everyone can connect to the infrastructure behind it, or no one can. Like an old VPN.

Let's look at where Tailscale shines: peer-to-peer connectivity with no central server, and zero-trust access between devices.

Remember the requirement of each peer having to connect to Tailscale, right? For that to happen, each webserver or infra component you want to reach directly needs to join the network (Tailnet), and the method depends on how you run that workload. If you're running it on a VM or container, you start a daemon at boot, either by baking the Tailscale binary into an image or downloading it during startup. We were on Kubernetes, so it's natural to install the tailscale operator, and expose a pod to every connected device with a single annotation.

apiVersion: v1
kind: Service
metadata:
  name: jenkins
  annotations:
    tailscale.com/expose: "true"
spec:
  type: ClusterIP
  ports:
  - port: 8080
    protocol: TCP
    name: jenkins
  selector:
    app.kubernetes.io/name: jenkins

Now, you can reach this service by typing https://jenkins.dog-einstein.ts.net into the browser. The infra team doesn't need to manage certificates, firewalls, load balancers, reverse proxies, ACLs from the Jenkins UI. Just for clarity, you can still use your own domain, but you will need to manage all these hurdles yourself.

dog-einstein because Tailscale couples the same random animal with a scientist for its domain. At least, they didn't go with <adjective>-<animal>. At a previous job, the sandbox platform gave a teammate a box called dirty-rat. He could have changed it but proudly kept it as a badge of honor.

However, my main beef with this approach is the tight coupling between the VPN and the domain your users rely on day-to-day. Imagine how, in case you need to move away from Tailscale, switching domains will break documentation links, browser bookmarks, application configuration, integrations, etc. etc. Annoying for a home server, but this vendor lock-in is a dealbreaker for a company IMO.

Here comes the worst part. After making Tailscale the official VPN, either by hiding your infrastructure under subnet-router or by allowing devices to connect directly, people will start complaining that your systems are unbearably slow. Performance was consistently bad with the old VPN, but now it's unusable for some people. Wasn't it supposed to be faster because of Wireguard?

You eventually realize the issue lies with a sneaky protocol called DERP (Designated Encrypted Relay for Packets) that will haunt you for the next few months. DERP is used as a relay until Tailscale can punch some NAT holes to allow direct connectivity between peers (the DERP relay also acts as a STUN server). The traffic then can flow after both gateways have mapped their ports. If either side is behind hard NAT (CGNAT or an AWS NAT Gateway), the DERP relay stays in the path permanently.

Another classic blog post from Tailscale that goes deeper into this problem is how NAT traversal works. It's such a great read.

Technically, that's not a big deal. But Tailscale provides these relays as a best effort kind of thing, meaning "might be okay today, but awful tomorrow or might be okay for you, but not for your colleague in Portugal". To bypass this limitation, you can host your own DERP servers, but then the complexity goes through the roof. I already had to install a subnet router, and now a DERP server as well?! I wasn't supposed to install any central VPN component in the first place, remember?

To run away from DERP, Tailscale support asked us to put the Tailscale devices on a public network. If those are EC2 instances, it's the easy path. Simply put them in public subnets, and open some ports to the internet (with a furrowed brow from the security team). For Kubernetes, schedule the Tailscale pods on a public subnet and disable SNAT source-port randomization, which will expose the cluster to racing issues, like this one.

But even after adding those options, it was not 100% effective. Sometimes the direct connection never appeared, even when clients were on a normal non-CGNAT ISP and the remote devices were not behind hard NAT. Good luck debugging that when some angry employee was complaining that you'd blocked them (with reason!).

❯ tailscale ping jenkins
pong from jenkins (100.86.155.54) via DERP(ams) in 94ms
pong from jenkins (100.86.155.54) via DERP(ams) in 30ms
pong from jenkins (100.86.155.54) via DERP(ams) in 29ms
pong from jenkins (100.86.155.54) via 63.167.214.122:35970 in 18ms

Tailscale introduced peer relays quite recently, but we never needed them, since all the required infra was already on public subnets. It's a step in the right direction, but yet another component you need to understand and place somewhere without hard NAT.

Looking back, I wouldn't want to be in Tailscale's shoes because hacking through NAT in a reliable and scalable way is super complex and error-prone. NAT is a hack on top of IP, and these workarounds are hacks on top of NAT. It's hacks all the way down.

Not everything is bad, though. If you really commit to Tailscale, it replaces access logic scattered across domains, programming or markup languages, and config files with a single JSON interface. You just need to avoid DERP as much as possible and adapt your workloads to join the Tailnet directly.

Tailscale also gets interesting when stuff lives across heterogeneous networks. For example, hosting macOS machines as ephemeral CI runners in any cloud is super pricey. With Tailscale, we used the Github-hosted runners, added the tailscale/github-action, wired up a few ACL rules, and a machine running in Azure had access to our infra in AWS; and we paid by the minutes the machine ran, not by the day. It's also useful for AI agents running wherever, with Tailscale acting as the guardian to ensure they only access the infrastructure they need.

Another solid point for Tailscale is how predictable the pricing is. It's $18/month per person (a bit on the high end compared with the alternatives, but alas). Multiply it by the number of employees at your company, and you've got your monthly bill. Contrast that with AWS VPN (the usual AWS open-source stealaroo, in this case OpenVPN), where you need to do the math on how many minutes each server is online, how many minutes your clients stay connected, bandwidth costs, etc. etc. Don't even get me started on the VPNs that require you to buy some appliance first.

Finally, as companies become increasingly closed, Tailscale goes on the opposite path. The Linux and Android clients are open-source, and they support Headscale, a project building an open-source control plane for Tailscale clients. The short-term future looks bright if you choose them as your vendor and understand what they require from you.

In retrospect, aside from the macOS example, Tailscale was underwhelming because it aims to solve problems we didn't have.

The company was already using Teleport for fine-grained access to all computing, e.g. kubernetes pods, and CI runners (Teleport only requires licenses for tech, unlike a VPN, which needs licenses for everyone). In the SaaS golden age, most systems were already exposed directly through the vendor's own domain, not through our internal infra. The days of self-hosted Jira are long gone. For the internal web services, we also decided to not switch our custom domains to the Tailscale ones, not only because of vendor lock-in, but also because we didn't wanna break compatibility with existing systems, and the current "our own DNS records" + internal load balancer + custom auth/authz gateway combo wasn't hurting us enough to justify migrating off it. Even after a couple of years, the Tailscale setup consists of a simple subnet router with binary access, just like a 2003 VPN. All the added complexity for almost no benefit.

Whether you need Tailscale depends on your company's appetite to bet on whatever it offers, not on FOMO or to use shiny tools.