Setup a Kubernetes cluster with K3S, Traefik, CertManager and Kubernetes Dashboard

Let's setup step by step our own K3S cluster !

Saturday, April 20, 2024

Introduction
Install K3S
Install kubectx and kubens (optional)
Install Helm
Install CertManager
Setup with Let’s Encrypt
Install Traefik
Setup our own applications
Kubernetes dashboard
Conclusion

Introduction

Hello everyone ! In this blog, we will see how to setup a Kubernetes cluster with:

K3S: it is a lightweight Kubernetes distribution created by Rancher Labs. It is fully certified by the Cloud Native Computing Foundation (CNCF) and is designed for production workloads in unattended, resource-constrained, remote locations or inside IoT appliances. K3s is highly available, production-ready, and has a very small binary size and very low resource requirements.
Helm: it is a package manager for Kubernetes. It simplifies the deployment of applications and services onto Kubernetes clusters by automating the creation, packaging, configuration, and deployment. Helm packages are called charts, which contain all the resource definitions necessary to run an application, tool, or service inside a Kubernetes cluster.
Traefik: it is an open-source Edge Router and a modern HTTP reverse proxy and load balancer. It makes deploying microservices easy by automatically and dynamically integrating with your existing infrastructure components.
CertManager: it is a native Kubernetes certificate management controller. It creates TLS certificates for workloads in your Kubernetes cluster and renews the certificates before they expire. CertManager can obtain certificates from a variety of certificate authorities, including Let’s Encrypt.
Kubernetes Dashboard: it is a web-based user interface for Kubernetes clusters. It provides a graphical representation of various aspects of a Kubernetes cluster.

Install K3S

curl -sfL https://get.k3s.io | sh -
# If you want to access to the cluster from the outside, add this flag:
# curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--tls-san <public ip address or hostname>" sh -

# Check for Ready node, takes ~30 seconds
sudo k3s kubectl get node

Install the certs to your machine to connect to the cluster

mkdir -p $HOME/.kube
sudo cp /etc/rancher/k3s/k3s.yaml $HOME/.kube/config
sudo chmod 600 $HOME/.kube/config
sudo chown -R $(id -u):$(id -g) $HOME/.kube/
export KUBECONFIG=~/.kube/config

Setup Vim (optional)

syntax on

set tabstop=2 softtabstop=2 shiftwidth=2
set expandtab
set number ruler
set autoindent smartindent
syntax enable
filetype plugin indent on

Setup bashrc

To automatically setup the configuration:

export KUBECONFIG=~/.kube/config
alias k="kubectl"

source <(kubectl completion bash)
# source <(helm completion bash) # Uncomment this line if you want to use helm
complete -o default -F __start_kubectl k

export EDITOR="vim"

Install kubectx and kubens (optional)

Source: https://github.com/ahmetb/kubectx?tab=readme-ov-file#installation

kubectx is a tool to switch between contexts (clusters) on kubectl faster.
kubens is a tool to switch between Kubernetes namespaces (and configure them for kubectl) easily.

You can use Krew for the installation:

kubectl krew install ctx
kubectl krew install ns

Or we can use the easy way with apt package:

sudo apt install kubectx

Install Helm

Source: https://helm.sh/docs/intro/install/#from-script

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Install CertManager

Source: https://artifacthub.io/packages/helm/cert-manager/cert-manager#configuration

helm repo add jetstack https://charts.jetstack.io

helm repo update

helm install \
 cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set installCRDs=true

Then verify the instalation:

kubectl -n cert-manager get pod

Setup with Let’s Encrypt

Source: https://letsencrypt.org/docs/staging-environment/

Here is the configuration for staging environments:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
  namespace: default
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL> # replace with your email adress
    privateKeySecretRef:
      name: letsencrypt-staging
    solvers:
    - selector: {}
      http01:
        ingress:
          class: traefik

Here is the configuration for production environments:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
  namespace: default
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: <YOUR_EMAIL> # replace with your email adress
    privateKeySecretRef:
      name: letsencrypt-production
    solvers:
    - selector: {}
      http01:
        ingress:
          class: traefik

Then, do kubectl apply -f <file> to apply the configurations.

Note: here, we create ClusterIssuer resources. They are non-namespaced resources. If you want to limit the certificate creation to a certain namespace, you may create an Issuer.

Source: https://cert-manager.io/docs/concepts/issuer/#namespaces

To verify the installation, we can do:

kubectl get ClusterIssuer -A

kubectl describe clusterissuer letsencrypt-staging
kubectl describe clusterissuer letsencrypt-production

Install Traefik

Note: on K3S, Traefik is installed by default (source: https://docs.k3s.io/networking/networking-services?_highlight=traefik#traefik-ingress-controller). If traefik is not already installed, you can follow these steps:

# Source: https://doc.traefik.io/traefik/getting-started/install-traefik/#use-the-helm-chart
# helm repo add traefik https://traefik.github.io/charts
# helm repo update
# helm install traefik traefik/traefik

Then we can configure Traefik (here I will expose the dashboard on “traefik.alexandre-hublau.com”):

apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    globalArguments:
      - "--serversTransport.insecureSkipVerify=true"
    additionalArguments:
      - "--api"
      - "--api.dashboard=true"
      - "--api.insecure=true"
      - "--log.level=ERROR"
    ports:
      websecure:
        tls:
          enabled: true
      web:
        redirectTo:
          port: websecure
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-dashboard
  namespace: kube-system
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik-dashboard
spec:
  type: ClusterIP
  ports:
  - name: traefik
    port: 9000
    targetPort: traefik
    protocol: TCP
  selector:
    app.kubernetes.io/instance: traefik-kube-system
    app.kubernetes.io/name: traefik
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: kube-system
  annotations:
    spec.ingressClassName: traefik
    traefik.frontend.rule.type: PathPrefixStrip
    cert-manager.io/cluster-issuer: letsencrypt-production
spec:
  rules:
    - host: traefik.alexandre-hublau.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: traefik-dashboard
                port:
                  number: 9000
  tls:
  - hosts:
    - traefik.alexandre-hublau.com
    secretName: traefik-alexandre-hublau-com-tls

Setup our own applications

Here is a small example for deploying an application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: application
spec:
  selector:
    matchLabels:
      app: application
  strategy: {}
  template:
    metadata:
      labels:
        app: application
    spec:
      dnsPolicy: ClusterFirst
      containers:
      - image: nginx:1.25
        name: application
        resources: {}
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: application
  name: application
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: application
  type: ClusterIP
---
# apiVersion: traefik.containo.us/v1alpha1
# kind: Middleware
# metadata:
#   name: stripprefix
# spec:
#   stripPrefix:
#     prefixes:
#       - "/v1"
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: application
  annotations:
    spec.ingressClassName: traefik
    traefik.frontend.rule.type: PathPrefixStrip
    # Change "default" with the namespace name if needed
    # traefik.ingress.kubernetes.io/router.middlewares: default-stripprefix@kubernetescrd
    cert-manager.io/cluster-issuer: letsencrypt-production
spec:
  rules:
  - host: application.alexandre-hublau.com
    http:
      paths:
      - path: /v1
        pathType: Prefix
        backend:
          service:
            name: application
            port:
              number: 80
  tls:
  - hosts:
    - application.alexandre-hublau.com
    secretName: application-alexandre-hublau-com-tls

Kubernetes dashboard

Now, let’s install the Kubernetes Dashboard (source: https://github.com/kubernetes/dashboard?tab=readme-ov-file#installation):

helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard

Let’s setup the configurations:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kubernetes-dashboard
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kubernetes-dashboard
  annotations:
    spec.ingressClassName: traefik
    traefik.frontend.rule.type: PathPrefixStrip
    cert-manager.io/cluster-issuer: letsencrypt-production
spec:
  rules:
  - host: dashboard.alexandre-hublau.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard-kong-proxy
            port:
              number: 443
  tls:
  - hosts:
    - dashboard.alexandre-hublau.com
    secretName: dashboard-alexandre-hublau-com-tls

To connect to the dashboard API, you need a token. To generate one, you can execute the command:

kubectl -n kubernetes-dashboard create token admin-user

Conclusion

🎉 Tadaaa 🎉 ! We did it! We have a full working cluster! Setting up a Kubernetes cluster with K3S, Helm, Traefik, CertManager, Let’s Encrypt, and Kubernetes Dashboard offers a streamlined and efficient way to manage containerized applications.

I hope this post is useful for you. Don’t forget to check out my other blog posts where we cover various DevOps topics!

Happy clustering and may your deployments be smooth sailing! 😊🏗️