K3s Cluster Lab

A project for automating the deployment of a Kubernetes (K3s) cluster, complete with integrated storage (Rook-Ceph) and monitoring (Prometheus & Grafana) for a hands-on DevOps learning environment.

Table of Contents

• K3s Cluster Lab
  • Table of Contents
  • About the Project
  • Features
  • Prerequisites
  • Installation
  • Usage
  • Configuration
  • Troubleshooting
  • Installtion Sequence

About the Project

This project is intended for learning and experimentation. It is not recommended for production use.

Features

• Automated setup of a K3s cluster.
• Integrated storage solution using Rook-Ceph.
• Monitoring with Prometheus and Grafana.
• Lightweight and easily customizable for different environments.

Prerequisites

• Linux-based OS
• curl
• qemu-img
• xorriso
• awk
• yq
• Basic familiarity with command-line tools.

Installation

1. Clone the repository:

   git clone https://github.com/Andrej220/k3s-cluster-lab.git
   cd k3s-cluster-lab

2. Edit Configuration: Edit the config.yml to customize your VM settings (e.g., CPUs, memory, disk size).
3. Run Setup:

   make setup

Usage

• To start the VM:

  ./<project_name>_run

• Access the K3s cluster with kubectl commands:

  export KUBECONFIG=./k3s-cluster/config
  kubectl get nodes

Configuration

• project_name: Sets the project name. It affects the generated VM names and script filenames.
• cpus: Number of CPUs allocated to the VM.
• memory: Amount of RAM for the VM.
• disk_size: Size of the VM's disk.
• network: Network configuration settings.

Troubleshooting

• Problem: Unable to connect to Grafana. Solution: Ensure port forwarding is set up correctly using kubectl port-forward.
• Problem: Rook-Ceph isn't creating pools. Solution: Verify that the correct storage class is set up and the cluster has sufficient resources.

Installtion Sequence

sequenceDiagram
    autonumber
    participant Dev as Host machine
    participant VM as VM (QEMU)
    participant CI as cloud-init
    participant K3s as k3s server
    participant Helm as Helm CLI
    participant RC as Rook-Ceph
    participant Mon as kube-prometheus

    Dev->>Dev: make setup / run helper scripts
    Dev->>VM: Create VM via vm-runner (QEMU)
    VM->>CI: cloud-init boots and starts provisioning
    CI->>CI: OS prep (packages, sysctl, container runtime cfg)
    CI->>K3s: Install & start k3s server
    K3s-->>Dev: KUBECONFIG available
    Note over K3s: Cluster API is up

    Dev->>Helm: helm repo add/update
    Dev->>RC: helm install rook-ceph
    RC-->>RC: Reconcile & create CephCluster
    RC-->>K3s: Default StorageClass ready

    Dev->>Mon: helm install kube-prometheus stack
    Mon-->>K3s: Deploy Prometheus & Grafana
    Mon-->>Dev: Dashboards accessible

    alt Additional nodes
      participant Agent as k3s agent
      Dev->>Agent: Join with server URL + token
      Agent->>K3s: Register node
    end

Loading