This repo contains a script to deploy a Talos Kubernets Cluster on Proxmox Virtual Enviroment using Terraform and a set of application using ArgoCD.
The VM are created with EUFI (q35 vm type) and Secure Boot for disk encryption (refer to https://www.talos.dev/v1.9/talos-guides/install/bare-metal-platforms/secureboot/#booting-talos-linux-in-secureboot-mode).
The Talos image needs to be generated with Image Factory choosing the options for secure boot and QEMU Agent support.
To use a different version from the one in this report the generated links needs to be inserted in the terraform.tfvarsfile.
Always use the latest version of the terraform providers if possible
Table of content:
The first thing to do is create a file named secrets.auto.tfvars and inside it define three variables:
proxmox_api_url = "https://proxmox_url:8006"
proxmox_api_token_id = "" # API user token create on proxmox (see below)
proxmox_api_token_secret = "" # API user secret
Refer to the ufficial provider documentatio for API user creation on Proxmox: https://registry.terraform.io/providers/bpg/proxmox/latest/docs
Ensure to properly configure .gitignore to excluse sensitive information. Refer to this example.
Populate terraform.tfvars with the required variable (refer to proxmox-var-declaration.tf and talos-var-declaration.tf for a complete list of declared variables).
Example of terraform.tfvars:
# VM creation - generic
starting_vm_id = 201
# VM creation - Control Plane
cp_total_count = 1 # Number of requied Control Plane VM/nodes
cp_proxmox_nodes = [ "pve" ] # Proxmox nodes where to provision Control Plane. Example ["pve", "pve02", "pve03"]
cp_ram = 2048 # Control Plane VM ram (in MB)
cp_core = 2 # Control Plane VM number of core
cp_vm_datastore = "local-zfs"
cp_disk_size = 15 # Control Plane VM disk size
cp_mac_address = ["bc:24:11:88:51:e1"] # Control Plane VM mac address (for DHCP reservation)
cp_network_bridge = "vmbr0"
# VM creation - worker
worker_total_count = 1 # Number of requied Worker VM/nodes
worker_proxmox_nodes = [ "pve" ] # Same as above for Control Plane
worker_ram = 2048 # Worker VM ram (in MB)
worker_core = 2 # Worker VM number of core
worker_vm_datastore = "local-zfs"
worker_disk_size = 15 # Worker VM disk size
worker_disk2_size = 50
worker_mac_address = ["bc:24:11:11:93:37"] # Worker VM mac address (for DHCP reservation)
worker_network_bridge = "vmbr0"
# Talos configuration
iso_download_url = "https://factory.talos.dev/image/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515/v1.9.3/metal-amd64-secureboot.iso"
installation_image = "factory.talos.dev/installer-secureboot/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.9.3"
cluster_name = "talos-proxmox-cluster"
control_planes_ip = ["192.168.100.201"]
worker_ip = ["192.168.100.204"]
vip = "192.168.100.200" # https://www.talos.dev/v1.9/talos-guides/network/vip/
cluster_endpoint = "https://192.168.100.200:6443" # This needes to be the same as the Talos "vip" for redundancy
Initialize the Terraform enviroment:
terraform init
This will download the required provider listed in the providers.tf file.
Dry-run the script execution:
terraform plan
Execute the script:
terraform apply
Note that it may be required to manually unmount the ISO on the first reboot, otherwise the VM may boot back in the ISO and the script will get stuck.
Download the talosconfig file generated by Terraform:
terraform output talosconfig > talosconfig
Check the cluster nodes:
talosctl get members
NODE NAMESPACE TYPE ID VERSION HOSTNAME MACHINE TYPE OS ADDRESSES
192.168.100.201 cluster Member talos-cp-01 2 talos-cp-01.greyroom.net controlplane Talos (v1.9.3) ["192.168.100.200","192.168.100.201"]
192.168.100.201 cluster Member talos-wrk-01 1 talos-wrk-01.greyroom.net worker Talos (v1.9.3) ["192.168.100.204"]
Make sure to exclude this file from git since it contains sensitive information and store it securely
Retrive kubectl config file and merge it with the default:
talosctl kubeconfig -n "one_of_the_cp_ip" --talosconfig=talosconfig
Test Kubernet API:
kubectl get nodes
NAME STATUS ROLES AGE VERSION
talos-cp-01 Ready control-plane 86m v1.32.0
talos-wrk-01 Ready <none> 86m v1.32.0
To avoid having to type --talosconfig=talosconfig on every command:
export TALOSCONFIG="/path/to/talosconfig"
Refer to the official documentation for detailed information
The Promox VM deployment is managed by the file "main.tf`.
First thing is to download the Talos ISO (for each node):
resource "proxmox_virtual_environment_download_file" "download-talos-iso" {
count = var.cp_total_count
content_type = "iso"
datastore_id = "local"
file_name = "metal-amd64-secureboot.iso"
node_name = var.cp_proxmox_nodes[count.index]
url = var.iso_download_url
}
Then the first VMs to be created are the Control Plane:
resource "proxmox_virtual_environment_vm" "cp-vm" {
depends_on = [
proxmox_virtual_environment_download_file.download-talos-iso # wait for the ISO download to be finished
]
count = var.cp_total_count # repeat this task for cp_total_count times
...
}
The Control Plane VM relevant settings are:
- operating system type: l26 (linux)
- bios type: uefi (ovmf)
- efi disk and tpm
- qemu agent
- resources: cpu, ram, disk, network
Refer to main.tf.
It then does basically the same thing but for the worker VMs but it also configure the secondo disk for storate.
Refer to the official documentation for detailed information
The Talos Cluster configuration is managed by the file talos.tf.
The first thing the script does is generating the cluster secrets:
resource "talos_machine_secrets" "this" {}
Which is the same thing as doing as the Talos documentaion suggests:
talosctl gen secrets -o secrets.yaml
Then it generates the client configuration (the one we later export using terraform output talosconfig):
data "talos_client_configuration" "this" {
client_configuration = talos_machine_secrets.this.client_configuration
cluster_name = var.cluster_name
nodes = var.control_planes_ip
endpoints = var.control_planes_ip
}
Now it proceed with generating the Control Plane generic configuration and applying some patches :
data "talos_machine_configuration" "cp" {
cluster_name = var.cluster_name
machine_type = "controlplane"
cluster_endpoint = var.cluster_endpoint
machine_secrets = talos_machine_secrets.this.machine_secrets
}
resource "talos_machine_configuration_apply" "control-planes" {
count = var.cp_total_count
client_configuration = talos_machine_secrets.this.client_configuration
machine_configuration_input = data.talos_machine_configuration.cp.machine_configuration
node = var.control_planes_ip[count.index]
config_patches = [
templatefile("${path.module}/talos-config/control-plane.yaml.tpl", {
vip = var.vip
installation_disk = "/dev/vda"
installation_image = var.installation_image
}),
]
}
It then does the same thing but for the Worker nodes.
The changes made by the configuration patches are:
- Changing the installation disk to /dev/vda since the VM are created with VirtIO SCSI Controller
- Enabling disk encryption
- Setting a Virtual (shared) IP, only for the Control Plane of course (refer to https://www.talos.dev/v1.9/talos-guides/network/vip/)
- Changing the installation image to the custom (and updated) one generated using Image Factory
If all this would have to be done manually it would look something like this:
talosctl gen config --with-secrets secrets.yaml talos-cluster https://vip:6443 --install-image factory.talos.dev/installer-secureboot/ce4c980550dd2ab1b17bbf2b08801c7eb59418eafe8f279833297925d67c7515:v1.9.3
# Need to manually create cp_patch.yaml and worker_patch.yaml file with the configuration patches
talosctl machineconfig patch controlplane.yaml --patch @cp_patch.yaml -o patched_cp.yaml
talosctl machineconfig patch worker.yaml --patch @worker_patch.yaml -o patched_worker.yaml
talosctl apply-config --insecure --nodes cp1_ip --file patched_cp.yaml
... # all the other Control Plane
talosctl apply-config --insecure --nodes worker1_ip --file patched_worker.yaml
... # all the other worker
At the end the script bootstrap the kubernetes cluster:
resource "talos_machine_bootstrap" "control-planes" {
depends_on = [
talos_machine_configuration_apply.control-planes
]
node = var.control_planes_ip[0]
client_configuration = talos_machine_secrets.this.client_configuration
}
Terraform will save as "output" the talosconfig and kubeconfig, to retrive them:
terraform ouput talosconfig > talosconfig
terraform ouput kubeconfig > kubeconfig