Proxmox Virtual Environment

Proxmox VE is a complete, open-source enterprise virtualization management platform. It tightly integrates the KVM hypervisor and Linux Containers (LXC), software-defined storage, and networking functionality on a single platform. With the integrated web-based user interface, you can easily manage VMs and containers, high availability for clusters, or the integrated disaster recovery tools with ease.

In our CSNIS Laboratory, Proxmox serves as the foundational Tier-1 infrastructure allowing us to dynamically allocate resources for various research projects, student labs, and internal services without needing dedicated bare-metal hardware for every single task.

Note: We highly recommend using a dedicated physical server (Bare Metal) for installing Proxmox VE. Installing it inside another hypervisor (Nested Virtualization) is possible for testing but not recommended for production workloads.

Prerequisites

Before installing Proxmox VE, ensure your hardware meets the following minimum requirements:

  • CPU: 64bit (Intel EMT64 or AMD64) with Intel VT/AMD-V virtualization extensions.
  • RAM: Minimum 2 GB for OS and Proxmox services, plus designated memory for guests.
  • Storage: Hardware RAID with batteries protected write cache (BBU) or non-RAID with ZFS.
  • Network: At least one reliable NIC (Network Interface Card).

Installation Steps (Debian Bookworm)

If you are not using the official Proxmox ISO, you can install it on top of an existing Debian Bookworm instance. First, add the Proxmox repository to your APT sources: 

echo "deb [arch=amd64] http://download.proxmox.com/debian/pve bookworm pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
wget https://enterprise.proxmox.com/debian/proxmox-release-bookworm.gpg -O /etc/apt/trusted.gpg.d/proxmox-release-bookworm.gpg

Update your system and install the Proxmox VE package along with necessary networking tools:

apt update && apt full-upgrade -y
apt install proxmox-ve postfix open-iscsi chrony -y

Network Setup (Linux Bridge)

Proxmox uses a bridged networking model. A bridge is a logical switch. All virtual machines can share a single physical network cable, but they appear to the outside network as individual physical machines.

Configuring vmbr0

Edit your network interfaces file typically located at /etc/network/interfaces to create your primary bridge (vmbr0):

auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
        address 192.168.1.10/24
        gateway 192.168.1.1
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

Storage Management

Storage in Proxmox is highly flexible. You can use local storage like LVM (Logical Volume Manager), directory-based storage, or advanced filesystems like ZFS which provides software RAID capabilities directly integrated into the hypervisor. For CSNIS, we utilize ZFS pools for its snapshot and replication features.

Docker Engine

Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. By taking advantage of Docker’s methodologies for shipping, testing, and deploying code quickly, you can significantly reduce the delay between writing code and running it in production.

Installation on Debian

To get the latest stable version, we install Docker from their official APT repository rather than the default Debian repo.

1. Add GPG Key

sudo apt-get update
sudo apt-get install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

2. Setup Repository & Install

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Docker Compose

Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application's services. Then, with a single command, you create and start all the services from your configuration.

Tip: In modern Docker installations, docker-compose is integrated as a plugin. You use `docker compose` instead of the old standalone `docker-compose` command. 
version: '3.8'
services:
  web:
    image: nginx:alpine
    ports:
      - "80:80"
  db:
    image: postgres:15
    environment:
      POSTGRES_PASSWORD: example

Best Practices

  • Keep your images small: Use Alpine Linux base images when possible.
  • Do not run containers as root: Always define a USER in your Dockerfile.
  • Use volumes for persistent data: Containers are ephemeral; data inside them is lost on restart.

Kubernetes Architecture

Kubernetes coordinates a highly available cluster of computers that are connected to work as a single unit. The abstractions in Kubernetes allow you to deploy containerized applications to a cluster without tying them specifically to individual machines.

Setup with Kubeadm

Kubeadm is a tool built to provide kubeadm init and kubeadm join as best-practice "fast paths" for creating Kubernetes clusters.

# Initialize the control-plane node
sudo kubeadm init --pod-network-cidr=10.244.0.0/16

# Configure kubeconfig for your regular user
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

Installing CNI (Flannel)

You must deploy a Container Network Interface (CNI) based Pod network add-on so that your Pods can communicate with each other. Cluster DNS (CoreDNS) will not start up before a network is installed.

kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

Understanding VLANs

A Virtual Local Area Network (VLAN) is any broadcast domain that is partitioned and isolated in a computer network at the data link layer (OSI layer 2). VLANs allow network administrators to group hosts together even if the hosts are not directly connected to the same network switch.

Static Routing

Static routing is a form of routing that occurs when a router uses a manually-configured routing entry, rather than information from dynamic routing traffic. Unlike dynamic routing, static routes are fixed and do not change if the network is changed or reconfigured.

Firewall (UFW)

Uncomplicated Firewall (UFW) is a program for managing a netfilter firewall designed to be easy to use. It uses a command-line interface consisting of a small number of simple commands.

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable