Disposable Cloud Environments With Vagrant and Tailscale

03 May 2023 • software

There's a lot in this blog post, so I'll summarize it first, and then tell you a horrible joke that got me a content warning from Slack (but it won't make sense unless you're a functional programming nerd).

  • Goal #1: I want to build out an ELK cluster and Do Science To it.
  • Goal #2: I want to not start from scratch or figure out how to undo things when I screw up.
  • Goal #3: I want to be able to keep working on it from different computers.
  • Goal #4: I want to work through Kubernetes the Hard Way and set up a cloud environment.

The solution to #1 is containerization. Run Docker Compose and set up an multi node ElasticSearch cluster.

The solution to #2 is virtualization. Create a virtual machine using Vagrant, install Docker on it, then do #1. Now I can take VM snapshots before config changes and rollback if I screwed up, and I can dispose of the boxes when I'm done.

The solution to #3 is to build out a homelab server (incredibly cheap at $391). Install Ubuntu, then #2.

The solution to #4 is to throw more memory at the homelab server, then install Kind after #2. Because of #2, I can now also mess with Terraform state and get away with it.

Pause to build and install everything…

Problem: I want to see Kibana from my laptop browser. Docker Compose forwards everything to localhost, and then the VM also requires networking magic to expose it to the host. I have a box containing a box, containing a box, and I don't want to have to port forward all the things.

Solution: Install Tailscale on the VM, exposing it as a host on the network (tailnet in Tailscale parlance).

Problem: Kubernetes is an orchestration layer, so now there are many boxes and portforwarding is impossible.

Solution: Set up Tailscale as a subnet router inside the VM, using Escaping the Nested Doll with Tailscale as a guide. Now I have infinite hosts on the network, and if I want a different configuration I can rollback to a base k8s state, or even set up them up side by side.

Now I'm going to tell you the horrible joke.

Imagine that we're defining processes as code, so a physical server is a container for processes: `

// http://devserver:80 from nginx -p 80 (the trivial case)
val devServer: Server = Server(Process("nginx", 80))

We can describe Docker and VirtualMachine the same way:

// docker run -p 80:80 nginx
val docker: Docker = Docker(Process("nginx", 80), PortForward(guest = 80, host = 80))

// config.vm.provision "shell", inline: "nginx -p 80"
// config.vm.network "nginx-port", guest: 80, host: 80
val standardVM: VirtualMachine = 
  VirtualMachine(Process("nginx", 80), PortForward(guest = 80, host = 80))

And we can build this up by putting Docker inside of a VM:

// config.vm.provision "shell", inline: "docker run -p 80:80 nginx"
// config.vm.network "nginx-port", guest: 80, host: 80
val vagrantDocker: VirtualMachine = 
  VirtualMachine(docker, PortForward(guest = 80, host = 80)))

And see how Kubernetes is a bit different:

// podIP: 10.244.0.6:80, 10.244.0.7:80 
val kubernetes: Kubernetes = Kubernetes(
  Set(
    Pod(Process("nginx", 80)),
    Pod(Process("nginx", 80))
  )
)

// Port mapping breaks down when we have multiple pods on a single VM :-(
val vagrantKubernetes: VirtualMachine = 
  VirtualMachine(kubernetes, ???)

From this, we can infer that Server, Docker and Pod are all Container types:

trait Container[T]
trait Server extends Container[Set[Process]]
trait Docker extends Container[Set[Process]] with Process
trait Pod extends Container[Set[Process]] with Process

And that VirtualMachine and Kubernetes are also instances of Container:

trait VirtualMachine extends Container[Server]
trait Kubernetes extends Container[Set[Pod]] extends Process

And Tailscale creates a Server from VirtualMachine:

val vagrantDocker = VirtualMachine(docker, portMapping))

// http://vagrant-docker:80 on the tailnet
val exposedNginxHost: Server = Tailscale(vagrantDocker)

But if VirtualMachine is a Container[Container[Set[Process]]] and Server is a Container[Set[Process]]:

val Tailscale: Container[Container[Set[Process]]] => Container[Set[Process]]

Tailscale is flatMap for the Containerization monad.

Still here? Let's dig into the details and I'll show all the setup steps.

Putting The Machine Together

Developing on my laptop has issues.

I installed elementary 5 on my laptop a while ago. It's based on Ubuntu 18.04, and I've started running into "version GLIBC_2.28' not found" errors more and more as it gets further behind. There's no way to upgrade Elementary 5 – the upgrade path is to reinstall the operating system from scratch. And… well, it has all kinds of cruft on it from various docker/k8s/cluster management tools all over it. It works fine as a laptop, but as a development environment it's not great. And trying to use Windows with WSL2 was even worse.

The easiest thing to do – obviously – is to take a week off work, put together a cheap headless machine as a homelab server, stick it in the basement, move everything to that box, and then connect the laptop remotely.

I went with the Ars Technica System Guide base specs, with a couple of changes: I added 64GB of memory, and I picked out an AMD Ryzen 5 5600X instead of the 5600G. (This was a mistake – the 5600X doesn't have an integrated GPU, leading to a frantic moment trying to figure out why the BIOS wouldn't come up on the HDMI port.) After upgrading the BIOS, staring at the manual for pins, and enabling virtualization by turning on SVM Mode, it was finally ready for a minimal Ubuntu install, using xrdp and Remmina to connect remotely.

I named it devserver.

Using Tailscale for "Server in the Basement"

The first thing to do was to install Tailscale on absolutely everything and enable every single feature, especially DNS.

Tailscale is good at the core use case, but does have some client based issues. For WSL, it won't recognize the tailscale client in the main Windows app, so you have to run tailscaled explicitly and distinguish it from the Windows host:

sudo nohup tailscaled &
sudo tailscale up --hostname windows-wsl

With the laptop, I also sometimes had to do tailscale down/up or --reset in order to get the mappings to resolve correctly.

There's a couple of things to be aware of when setting up Tailscale for a server. The first one is disabling key expiry for the server, since it's going to be hanging around for a while. The second is that tailscale provides its own ssh, which requires its own parameters:

sudo tailscale up --ssh --operator=$USER

Once SSH was up, it was time to futz with configuration files. I like to use Visual Studio Code with SSH remote development, which comes with a secret command line tool for connecting to any host on the tailnet:

code --folder-uri "vscode-remote://ssh-remote+devserver/home/wsargent/"

There are also some utilities that Tailscale provides for ad-hoc port forwarding. For example, I can run jekyll serve on devserver and it will start a server on port 4000 – I can see how that looks on my phone by using tailscale serve:

sudo tailscale serve tcp:4000 tcp://localhost:4000

And then I can go to http://devserver:4000 on my phone and see how the blog post looks from there. The services page on Tailscale shows a list of ports open on all the machines, so it's easy to see what services are active and how to get to them.

Adding Tailscale to Vagrant

Adding Tailscale to Vagrant is straightforward. Generate an authentication key, make it reusable, and save it into 1Password for provisioning. 1Password has a CLI that's very useful in managing secrets – this blog post is already too long, but there's an example repository that shows how to provision secrets.

I started off with Virtualbox, but have been experimenting with libvert. To use libvert, add vagrant-libvert plugin:

sudo apt install libvert-dev
sudo apt-get purge vagrant-libvirt
sudo apt-mark hold vagrant-libvirt
sudo apt-get update 
sudo apt-get install -y qemu libvirt-daemon-system ebtables libguestfs-tools vagrant ruby-fog-libvirt

Using vagrant-env plugin, you can then set up Tailscale on startup and shutdown:

Vagrant.configure("2") do |config|
  config.env.enable

  config.vm.box = ENV['VM_BOX']
  config.vm.hostname = ENV['VM_HOSTNAME']    
  config.vm.provider ENV['VM_ENGINE'] do |v| 
    v.name = ENV['VM_HOSTNAME']
    v.memory = ENV['VM_MEMORY']
    v.cpus = ENV['VM_CPUS']
  end

  config.vm.provision "tailscale-install", type: "shell" do |s|
    s.inline = "curl -fsSL https://tailscale.com/install.sh | sh"
  end

  config.vm.provision "tailscale-up", type: "shell" do |s|
    s.inline = "tailscale up --ssh --operator=vagrant --authkey #{ENV['TAILSCALE_AUTHKEY']}"
  end

  config.trigger.before :destroy do |trigger|
    trigger.run_remote = {inline: "tailscale logout"}
    trigger.on_error = :continue
  end

end

To start it, I run vagrant up. To stop it, I run vagrant halt. When I'm done experimenting with the environment, I destroy it with vagrant destroy, and it removes itself from the tailnet automatically.

Adding Docker to Vagrant

Now we have to add Docker to Vagrant:

Vagrant.configure("2") do |config|
  # ...

  config.vm.provision "docker-install", type: "shell" do |s|
    s.inline = <<-SCRIPT
curl -fsSL https://get.docker.com -o get-docker.sh &&
sudo sh get-docker.sh &&
sudo adduser vagrant docker
SCRIPT
  end

end

Now we can have some fun – we'll run two docker compose instances side by side without port conflicts. Checkout awesome-compose so it shows up on the /vagrant mount:

$ vagrant ssh
$ cd /vagrant/awesome-compose/nginx-golang
$ docker compose up

And then again, only this time we have vagrant-nginx-nodejs-redis as the hostname:

$ vagrant ssh
$ cd /vagrant/awesome-compose/nginx-nodejs-redis
$ docker compose up

Now we've got two nginx instances, both running on port 80 – but they just appear as different hosts.

$ curl vagrant-nginx-nodejs-redis
$ web1: Number of visits is: 1

and

$ curl vagrant-nginx-golang

          ##         .
    ## ## ##        ==
 ## ## ## ## ##    ===
/"""""""""""""""""\___/ ===
{                       /  ===-
\______ O           __/
 \    \         __/
  \____\_______/


Hello from Docker!

I also have a vagrant-docker box that I use for ad-hoc installations. From the laptop, I can install only the docker CLI and set DOCKER_HOST to point the box:

sudo apt install docker-ce-cli
export DOCKER_HOST=ssh://vagrant@vagrant-docker
ssh vagrant@vagrant-docker # or tailscale ssh vagrant@vagrant-docker
docker ps # will work after ssh succeeded!

And now I can run various services directly and hit them at http://vagrant-docker:3000.

Disposable Cloud Environments

The limitation of using Docker Compose is that you're still referencing the Vagrant box, and picking out a service by port. If you have a more complex environment, you'll probably have several databases, a key/store server, several microservices and so on. Really, you'd like to be able to spin up Kubernetes inside a Vagrant Box and access all the pods through Tailscale automatically.

Setting up Kubernetes itself is suprisingly simple in a Vagrantfile. For example, setting up Kind is as simple as a vagrant up:

Vagrant.configure("2") do |config|

  # ...install Docker

  config.vm.provision "kind", type: "shell" do |s|
    s.inline = <<-SCRIPT
curl -Lo ./kind "https://kind.sigs.k8s.io/dl/v0.18.0/kind-$(uname)-amd64"
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
SCRIPT
  end

  config.vm.provision "kubectl", type: "shell" do |s|
    s.inline = "sudo snap install kubectl --classic"
  end

  config.vm.provision "kubectl-completion", type: "shell" do |s|
    s.inline = 'echo "source <(kubectl completion bash)" >> ~/.bashrc'
  end

end

Setting up Tailscale in Kubernetes is… not so simple. This took some trial and error, and I leaned heavily on Escaping the Nested Doll with Tailscale when going through this.

I like to start fresh so I immediately wipe the cluster to clean everything:

$ kind delete cluster
$ kind create cluster

Tailscala's Kubernetes subnet routing section is a bit confusing and out of date, so I used the README.md in https://github.com/tailscale/tailscale/tree/main/docs/k8s which is slightly different.

First we need to set up with an auth key and write it as a k8s service account token secret and pass it through:

$ kubectl apply -f - <<EOF apiVersion: v1
kind: Secret
metadata:
  name: tailscale-auth
stringData:
  TS_AUTHKEY: <your-auth-key>
EOF

Then we checkout the github project and get to the Makefile:

$ git clone https://github.com/tailscale/tailscale
$ cd tailscale/docs/k8s

And execute make rbac:

$ sudo apt install make
$ export SA_NAME=tailscale
$ export TS_KUBE_SECRET=tailscale-auth
$ make rbac

Next, we want to set up a subnet router. We need pod and service IP addresses. We can set up an nginx instance from the k8s docs:

$ kubectl apply -f https://k8s.io/examples/application/deployment.yaml

From there, we can see the IP address of the pods:

$ kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE     IP           NODE                 NOMINATED NODE   READINESS GATES
nginx-deployment-85996f8dbd-6clrn   1/1     Running   0          5h34m   10.244.0.6   kind-control-plane   <none>           <none>
nginx-deployment-85996f8dbd-jwk4q   1/1     Running   0          5h34m   10.244.0.5   kind-control-plane   <none>           <none>

and the service:

$ kubectl get svc
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   5h35m

Then we set the TS_ROUTES and call make subnet-router:

$ SERVICE_CIDR=10.96.0.0/16
$ POD_CIDR=10.244.0.0/15
$ export TS_ROUTES=$SERVICE_CIDR,$POD_CIDR
$ make subnet-router
pod/subnet-router created

And finally I can see the subnet router defined in Tailscale:

$ tailscale ping subnet-router
pong from subnet-router (100.96.237.125) via DERP(sfo) in 26ms

Next, we need to go to the machines page and it will have a little alert next to it saying "Unapproved subnet routes!" Go to the "Edit routes settings" menu option and click Approve All.

And now the pods are accessible via IP address through Tailscale and I can see them from my Windows machine and my iPhone:

C:\Users\wsargent>curl 10.244.0.6
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Note that the pods do not show up in tailscale status, and they are not accessible by pod name – only IP address. There is a way to hook Tailscale into the k8s DNS, but I haven't dug into it. I'll abstract this into a Vagrantfile eventually, but this is a good place to stop.

Further Work

This approach works out of the box, but could use some optimization.

I need to set up a Docker registry to cache images, and an apt cache so that initializing Vagrant boxes doesn't go over the network. It might make sense to have them run on devserver specifically so they don't have to rely on specific vagrant instances being up (and I can't wipe them out by accident).

I could also make a Vagrant basebox with Tailscale and Docker and remove that bit from initialization.