k8s-infra

Infrastructure code for extending the eva01–04 k3s homelab cluster with AWS-hosted agent nodes. Nodes are built as AMIs with Packer, provisioned with Terraform, and bootstrap themselves into the existing cluster over Tailscale on first boot. Terraform state and all secrets live on HCP Terraform.

Architecture

packer/            Packer template -> "homelab-k3s-node-*" Ubuntu 24.04 amd64 AMIs
image/
  bootstrap.sh     OS-level provisioning baked into the AMI (packages only, no secrets)
terraform/
  main.tf          EC2 instances + security group
  variables.tf     Input variables (secrets supplied via HCP Terraform workspace vars)
  versions.tf      Providers + HCP Terraform cloud backend
  outputs.tf       Node IDs / private IPs
  node-cloud-init.yaml.tftpl   First-boot user data (Tailscale + k3s join)

How node bring-up works

1. make build-image produces an AMI containing common OS dependencies only. Cluster credentials are never baked into images.
2. On first boot, cloud-init runs in strict order:
   • installs Tailscale and joins the tailnet using a tagged, ephemeral auth key (tag:k3s), accepting subnet routes;
   • joins k3s as an agent with K3S_URL pointed at the control plane's Tailscale address and K3S_TOKEN from workspace vars.
3. The security group has no ingress rules. k3s traffic and SSH both ride the tailnet (outbound-established tunnel); SSH to nodes via their Tailscale IP / MagicDNS name. A ssh_cidr variable exists as break-glass public access, default off.

Prerequisites

• Terraform >= 1.9, authenticated to HCP Terraform (terraform login once)
• Packer >= 1.10 (amazon plugin auto-installed by packer init)
• AWS credentials in a profile (AWS_PROFILE, default homelab)
• Optional for linting: tflint, yamllint

Workflow

make build-image        # packer init + build -> AMI "homelab-k3s-node-*"
make lint               # fmt + terraform validate + tflint + packer validate
make plan               # remote plan against HCP Terraform workspace
terraform -chdir=terraform apply

The AMI is discovered automatically (owners = ["self"], name prefix homelab-k3s-node-*); set ami_id only to pin a specific build.

HCP Terraform workspace variables

Workspace: k8s-infra-aws-nodes. Secrets are marked Sensitive in the workspace UI and never committed anywhere.

Variable	Sensitive	Notes
vpc_id, subnet_id	no	Target networking
ssh_key_name	no	Existing EC2 key pair
cluster_endpoint	no	https://<tailscale-ip>:6443 of the control plane
k3s_token	yes	From /var/lib/rancher/k3s/server/node-token
tailscale_auth_key	yes	Tagged ephemeral key, short expiry
node_count, instance_type, aws_region, tailscale_tags, tailscale_accept_routes, ami_id, ssh_cidr	no	Have defaults

Server-side prerequisites (on eva01)

• k3s server must include its Tailscale IP in the API TLS SAN (--tls-san=<tailscale-ip>), otherwise agents fail the join with x509 errors.
• Create the Tailscale auth key as ephemeral + tagged (tag:k3s) with a short expiry; rotate rather than reuse long-lived keys.

Post-join checks

• kubectl get nodes shows the new node Ready.
• Assert Cilium came up with pod MTU 1280 on the new node (ip link show cilium_host) — tunnel MTU on this network is 1280 and agents have historically come up wrong after restarts. Do not trust Helm values here.

Security posture

• EBS root volumes encrypted; IMDSv2 required (http_tokens = "required").
• No public ingress by default; password SSH disabled via cloud-init.
• Secrets exist only as sensitive HCP Terraform workspace variables; state is encrypted at rest on HCP Terraform.