Abstract

Demonstrate a quick and dirty deployment with kubeadm.

Quick and Dirty control plane with kubeadm

For this demonstration we will be using kubeadm to deploy and configure a control plane node which is the Lenovo ThinkCentre M910q that was acquired for this purpose.

Q&D Control Plane Requirements

The remainder of this guide assumes the presence of an ArchLinux system on a locally-accessible network that accepts ssh connections from a non-root user with sudo access to the system in question, which should also have a functional installation of yay.

Prepare for kubeadm

In the interests of keeping things in relatively standard locations, all of the configuration files used by kubeadm will be kept in /etc/kubeadm, which we can create now.

Create the kubeadm configuration directory

ssh kcp01
yay -S containerd kubeadm kubelet
sudo su -
mkdir /etc/kubeadm
cd /etc/kubeadm
modprobe br_netfilter
sysctl net.ipv4.ip_forward=1

List of commands

The remainder of this guide assumes that the commands listed above have been successfully executed, so far you should have:

1. created an ssh connection

   ssh kcp01
   

2. installed kubelet and kubeadm with yay

   yay -S kubelet kubeadm
   

3. used sudo to become the root user

   sudo su -
   

4. enabled the kubelet service

   systemctl enable --now kubelet
   

   Tip

   It is safe to ignore any errors that the command above may throw, they will be resolved by kubeadm later.

5. created the kubeadm configuration directory

   mkdir -pv /etc/kubeadm
   

6. updated your working directory

   cd /etc/kubeadm
   

7. enable the br_netfilter kernel module

   modprobe br_netfilter
   

8. enable ip forwarding

   sysctl net.ipv4.ip_forward=1
   

Create an initial configuration

While it is possible to write a kubeadm initial configuration file from scratch, you are much better off letting kubeadm do that for you.

create default kubeadm init configuration

kubeadm config print init-defaults \
  --component-configs KubeProxyConfiguration,KubeletConfiguration > init.yaml

This should leave you with something similar to the output below:

[root@kcp01:/etc/kubeadm]{0} # ls -l
total 4
-rw-r--r-- 1 root root 1139 Jun 29 18:53 init.yaml

default init.yml

apiVersion: kubeadm.k8s.io/v1beta4
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: abcdef.0123456789abcdef
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 1.2.3.4
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  imagePullSerial: true
  name: node
  taints: null
timeouts:
  controlPlaneComponentHealthCheck: 4m0s
  discovery: 5m0s
  etcdAPICall: 2m0s
  kubeletHealthCheck: 4m0s
  kubernetesAPICall: 1m0s
  tlsBootstrap: 5m0s
  upgradeManifests: 5m0s
---
apiServer: {}
apiVersion: kubeadm.k8s.io/v1beta4
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
encryptionAlgorithm: RSA-2048
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.33.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
proxy: {}
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
bindAddressHardFail: false
clientConnection:
  acceptContentTypes: ""
  burst: 0
  contentType: ""
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  qps: 0
clusterCIDR: ""
configSyncPeriod: 0s
conntrack:
  maxPerCore: null
  min: null
  tcpBeLiberal: false
  tcpCloseWaitTimeout: null
  tcpEstablishedTimeout: null
  udpStreamTimeout: 0s
  udpTimeout: 0s
detectLocal:
  bridgeInterface: ""
  interfaceNamePrefix: ""
detectLocalMode: ""
enableProfiling: false
healthzBindAddress: ""
hostnameOverride: ""
iptables:
  localhostNodePorts: null
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: false
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s
kind: KubeProxyConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
metricsBindAddress: ""
mode: ""
nftables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
nodePortAddresses: null
oomScoreAdj: null
portRange: ""
showHiddenMetricsForVersion: ""
winkernel:
  enableDSR: false
  forwardHealthCheckVip: false
  networkName: ""
  rootHnsEndpointName: ""
  sourceVip: ""
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
crashLoopBackOff: {}
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

Most of the default values are acceptable, but a number of them will need to be updated.

Configure and initialize the control plane

The lines that must be updated have been highlighted below, their values are described in the following table.

number	attribute	value	description
5	token	$(kubeadm token create)	a basic authentication token for bootstrapping
12	advertiseAddress	172.17.0.100	an ipv4 address for the control plane node to listen on
18	name	kcp01	an arbitrary dns-compatible name that describes the control plane
34	clusterName	the-hard-way	an arbitrary dns-compatible name that describes the cluster
47	podSubnet	10.244.0.0/16	the default cidr block for the flannel networking plugin
101	metricsBindAddress	0.0.0.0:9090	the address of a metrics receiver to send metrics to
157	swapBehavior	LimitedSwap	enable experimental swap support on the kubelet

Configured kubeadm init.yaml

updated init.yaml

apiVersion: kubeadm.k8s.io/v1beta4
bootstrapTokens:
- groups:
  - system:bootstrappers:kubeadm:default-node-token
  token: 0fa83j.zskjs5g0h1lqz8mi
  ttl: 24h0m0s
  usages:
  - signing
  - authentication
kind: InitConfiguration
localAPIEndpoint:
  advertiseAddress: 172.17.0.100
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  imagePullSerial: true
  name: kcp01
  taints: null
timeouts:
  controlPlaneComponentHealthCheck: 4m0s
  discovery: 5m0s
  etcdAPICall: 2m0s
  kubeletHealthCheck: 4m0s
  kubernetesAPICall: 1m0s
  tlsBootstrap: 5m0s
  upgradeManifests: 5m0s
---
apiServer: {}
apiVersion: kubeadm.k8s.io/v1beta4
caCertificateValidityPeriod: 87600h0m0s
certificateValidityPeriod: 8760h0m0s
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
encryptionAlgorithm: RSA-2048
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: 1.33.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
proxy: {}
scheduler: {}
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
bindAddress: 0.0.0.0
bindAddressHardFail: false
clientConnection:
  acceptContentTypes: ""
  burst: 0
  contentType: ""
  kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
  qps: 0
clusterCIDR: ""
configSyncPeriod: 0s
conntrack:
  maxPerCore: null
  min: null
  tcpBeLiberal: false
  tcpCloseWaitTimeout: null
  tcpEstablishedTimeout: null
  udpStreamTimeout: 0s
  udpTimeout: 0s
detectLocal:
  bridgeInterface: ""
  interfaceNamePrefix: ""
detectLocalMode: ""
enableProfiling: false
healthzBindAddress: ""
hostnameOverride: ""
iptables:
  localhostNodePorts: null
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
ipvs:
  excludeCIDRs: null
  minSyncPeriod: 0s
  scheduler: ""
  strictARP: false
  syncPeriod: 0s
  tcpFinTimeout: 0s
  tcpTimeout: 0s
  udpTimeout: 0s
kind: KubeProxyConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
metricsBindAddress: "0.0.0.0:9090"
mode: ""
nftables:
  masqueradeAll: false
  masqueradeBit: null
  minSyncPeriod: 0s
  syncPeriod: 0s
nodePortAddresses: null
oomScoreAdj: null
portRange: ""
showHiddenMetricsForVersion: ""
winkernel:
  enableDSR: false
  forwardHealthCheckVip: false
  networkName: ""
  rootHnsEndpointName: ""
  sourceVip: ""
---
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
containerRuntimeEndpoint: ""
cpuManagerReconcilePeriod: 0s
crashLoopBackOff: {}
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMaximumGCAge: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
    text:
      infoBufferSize: "0"
  verbosity: 0
memorySwap:
  swapBehavior: LimitedSwap
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

The changes to lines 156-157 can be skipped if you disable any active swap[1] file systems on the control plane.

Initialize the control plane

Now that we’ve got an initial configuration that resembles the intended control plane configuration, we can go ahead and spin it up.

kubeadm init --config init.yaml

If things go well, your shell should produce something like the following output.

control plan init results

[init] Using Kubernetes version: v1.33.0
[preflight] Running pre-flight checks
        [WARNING Swap]: swap is supported for cgroup v2 only. \
          The kubelet must be properly configured to use swap. \
          Please refer to https://shorturl.at/BKSgm \
            or disable swap on the node
        [WARNING Service-Kubelet]: kubelet service is not enabled, \
          please run 'systemctl enable kubelet.service'
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of \
  your internet connection
[preflight] You can also perform this action beforehand using \
  'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names \
  [kubernetes kubernetes.default kubernetes.default.svc \
  kubernetes.default.svc.cluster.local kcp01] and IPs \
  [10.96.0.1 172.16.0.100]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names
  [localhost kcp01] and IPs [172.16.0.100 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names
  [localhost kcp01] and IPs [172.16.0.100 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "super-admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[etcd] Creating static Pod manifest
  for local etcd in "/etc/kubernetes/manifests"
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod
  manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[kubelet-start] Writing kubelet environment file
  with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration
  to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[wait-control-plane] Waiting for the kubelet to boot
  up the control plane as static Pods
  from directory "/etc/kubernetes/manifests"
[kubelet-check] Waiting for a healthy kubelet at
  http://127.0.0.1:10248/healthz. This can take up to 4m0s
[kubelet-check] The kubelet is healthy after 3.001790154s
[control-plane-check] Waiting for healthy control
  plane components. This can take up to 4m0s
[control-plane-check] Checking kube-apiserver
  at https://172.16.0.100:6443/livez
[control-plane-check] Checking
  kube-controller-manager at https://127.0.0.1:10257/healthz
[control-plane-check] Checking
  kube-scheduler at https://127.0.0.1:10259/livez
[control-plane-check]
  kube-controller-manager is healthy after 1.506311953s
[control-plane-check]
  kube-scheduler is healthy after 2.300444114s
[control-plane-check]
  kube-apiserver is healthy after 4.00284659s
[upload-config] Storing the configuration
  used in ConfigMap "kubeadm-config"
  in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config"
  in namespace kube-system with the configuration for
  the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node kcp01 as control-plane by
  adding the labels: [
    node-role.kubernetes.io/control-plane
    node.kubernetes.io/exclude-from-external-load-balancers
  ]
[mark-control-plane] Marking the node kcp01 as control-plane by
  adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: r51n8g.qj8psxn8wszh915u
[bootstrap-token] Configuring bootstrap tokens,
  cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow
  Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node
  Bootstrap tokens to post CSRs in order for nodes to
  get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover
  controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate
  rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap
  in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf"
  to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one \
  of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes \
  by running the following on each as root:

kubeadm join 172.16.0.100:6443 --token r51n8g.qj8psxn8wszh915u \
  --discovery-token-ca-cert-hash sha256:012345678

Check the initialized control plane

We can verify that this worked with the kubectl command, after we configure the new cluster’s authentication.

add authentication

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl get nodes

The last command should produce output similar to this.

list of active nodes

NAME     STATUS     ROLES           AGE   VERSION
kcp01    NotReady   control-plane   19m   v1.33.2

Note

The node status is not ready. This is because we still need to installed the flannel network layer, fortunately this is simple.

Install flannel networking layer

apply flannel network layer

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

flannel application result

namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created

Now, after a few moments you should be able to obtain a different result from the node list request.

Verify the control-plane is ready

get nodes again

kubectl get nodes

and we have a working control plane

NAME     STATUS   ROLES           AGE   VERSION
kcp01   Ready    control-plane   21m   v1.33.2

---

[1]

The Kubernetes reference manual has more detail about the necessity of these lines.

Automate the etcd deployment with Ansible