Homelab-Ops: Sovereign Bare-Metal Cloud & GitOps Platform

Project Overview

#

Homelab-Ops v3.0 is an enterprise-grade sovereign bare-metal cloud and GitOps platform engineered to solve real-world infrastructure, networking, and data persistence constraints. Hosted on physical bare-metal hardware running Proxmox VE 8 in Mumbai, it delivers high-availability container orchestration, zero-trust edge ingress, continuous automated delivery, and automated multi-cloud disaster recovery—operating at $0 recurring monthly cloud costs.

Designed and operated with the discipline of a production platform engineering team, the entire platform is managed declaratively via Infrastructure as Code (Terraform & Ansible) and GitOps (Flux CD v2), backed by 16 formal Architecture Decision Records (ADRs) and comprehensive disaster recovery runbooks.

System Architecture

#

The platform architecture bridges on-premise bare-metal virtualization with edge ingress and multi-cloud resilience:

1. Bare-Metal Virtualization & Cluster Topology

#

Physical hypervisor running Proxmox VE 8 on bare metal, partitioning resources across K3s control plane, worker nodes, and auxiliary storage engines with strict memory fencing and SSD/HDD storage tiering.

2. Zero-Trust Global Network Topology

#

Inbound public traffic routes through Cloudflare’s global Anycast edge network and traverses residential Carrier-Grade NAT (CGNAT) via bidirectional encrypted QUIC tunnels (cloudflared). Administrative traffic is isolated through a Tailscale WireGuard mesh with ACLs and MFA.

3. Declarative GitOps Delivery & In-Repo Encryption

#

Flux CD v2 continuously synchronizes cluster state against the GitHub monorepo. All Kubernetes Secret resources are committed directly to Git encrypted with Mozilla SOPS and Age asymmetric cryptography, hydrated purely in-memory by the Flux kustomize controller.

Key Engineering Breakthroughs

#

Breakthrough 1: Zero Open Ports CGNAT Ingress

#

• Challenge: Residential ISP enforces strict Carrier-Grade NAT (CGNAT) and dynamic IP allocation, blocking inbound ports 80/443 and traditional dynamic DNS.
• Solution: Deployed Cloudflare Zero Trust dual-tunnel architecture running directly within Kubernetes. Dual cloudflared replicas establish outbound encrypted QUIC sessions to Cloudflare Anycast edge servers.
• Impact: 100% elimination of port forwarding, public IP exposure, and DDoS vulnerability, with sub-15ms edge latency across Mumbai and South Asia.

Breakthrough 2: Pure GitOps Delivery & 8-Minute Rebuild

#

• Challenge: Manual cluster configurations create drift, incomplete disaster recovery runbooks, and unrepeatable snowflake environments.
• Solution: Implemented 100% declarative GitOps using Flux CD v2. Cluster bootstrap is orchestrated via Terraform and Ansible, after which Flux takes ownership of all namespaces, CRDs, and workloads.
• Impact: Zero manual kubectl apply interventions in production. Complete cluster rebuild time reduced to under 8 minutes from clean hypervisor boot.

Breakthrough 3: Shift-Left In-Git Secrets (SOPS + Age)

#

• Challenge: Managing secrets securely without paying for expensive external SaaS secret managers (Vault enterprise, AWS Secrets Manager) while keeping configs version-controlled.
• Solution: Implemented Mozilla SOPS with Age 256-bit asymmetric key encryption. Private Age keys are injected once during cluster bootstrap; Flux decrypts manifests natively during reconciliation.
• Impact: Zero plaintext credentials in Git history; full auditability of configuration changes alongside code.

Breakthrough 4: Stateful HA with CloudNativePG Operator

#

• Challenge: Managing high-availability relational databases on Kubernetes without data corruption or split-brain during node reboots.
• Solution: Integrated the enterprise CloudNativePG (PostgreSQL) operator with automated primary/standby clustering, streaming replication, automatic failover, and continuous WAL archiving to cloud object storage.
• Impact: Zero-data-loss database durability with automated point-in-time recovery (PITR) capabilities.

Breakthrough 5: Multi-Cloud Out-of-Band Disaster Recovery

#

• Challenge: A single power or ISP failure at the on-premise location blindfolds observability and prevents disaster alerts.
• Solution: Architected out-of-band monitoring and cold-standby disaster recovery using Oracle Cloud Infrastructure (OCI Always Free) and Google Cloud Platform (GCP). OCI hosts Uptime Kuma synthetic probes and offsite S3-compatible backup replicas, completely outside the homelab blast radius.
• Impact: Independent 24/7 telemetry and automated alerts delivered to mobile even during complete local power outages.

FinOps & Hardware Economics

#

By architecting a sovereign bare-metal platform, homelab-ops provides enterprise cloud capabilities while completely avoiding recurring monthly cloud bills:

Hands-On Competencies Demonstrated

#

• Bare-Metal Virtualization: Proxmox VE 8 hypervisor administration, Linux bridge networking, memory ballooning, NUMA architecture, CPU pinning, and ZFS/storage tiering.
• Production Kubernetes Administration (CKA): K3s multi-node clustering, Flannel CNI, CoreDNS tuning, persistent storage volumes (Local-Path & Longhorn), and RBAC security policies.
• GitOps & Delivery Engineering: Flux CD v2 kustomization controllers, Helm releases, webhook notifications, dependency health checks, and reconciliation loops.
• Enterprise Security & Secrets: Zero Trust network access, Age encryption, SOPS secret decryption, network policies, and role-based access control.
• Disaster Recovery & SRE Practice: 16 Architecture Decision Records (ADRs), recovery point objective (RPO) < 15 min, recovery time objective (RTO) < 30 min, and automated offsite backups following the 3-2-1 rule.

Technical Stack

#

• Hypervisor & OS: Proxmox VE 8, Debian Linux 12, Ubuntu 22.04 LTS
• Container Orchestration: Kubernetes (K3s v1.30+)
• GitOps Continuous Delivery: Flux CD v2
• Infrastructure as Code: Terraform, Ansible
• Networking & Ingress: Cloudflare Zero Trust (cloudflared), Tailscale WireGuard Mesh, MetalLB
• Data Persistence & Operators: CloudNativePG (PostgreSQL HA Operator), Local-Path Provisioner
• Secrets Management: Mozilla SOPS, Age Cryptography
• Observability & Out-of-Band Monitoring: Prometheus, Grafana, Uptime Kuma (OCI-hosted)
• Multi-Cloud Integration: Oracle Cloud Infrastructure (OCI Always Free), Google Cloud Platform (GCP)

Project Documentation & Verification

#