Every Homelab I Built Died in a Cardboard Box

Not metaphorically.

There were several of them over the years, all built on Raspberry Pis, each one a little better than the last. None of them was ever killed by a technical problem. They were killed by moving house. The rack — and I am being generous with the word "rack" — gets unplugged on a Saturday, goes into a box, and comes out three weeks later on a different shelf with a different router in front of it. Whatever was running on it has been down the whole time, and half of it does not come back, because the reason it worked in the first place was a thing I did by hand eight months ago and did not write down.

My uptime was never measured in nines. It was measured in "until the next move."

Then the free time went away. I have small children, which turns out to be less a problem of how much time you have than of when — you get an unpredictable hour, not a planned evening. A homelab that demands attention at a moment of its own choosing loses that negotiation every single time. And I still wanted one. Not out of nostalgia: I work on infrastructure, and I do not entirely trust opinions about systems I have never had to keep alive myself.

So the machine had to leave my apartment.

The constraint I picked on purpose

It runs on a single Oracle Cloud instance in the Always Free tier. Four Ampere cores, 24 GB of RAM, 200 GB of disk, ARM, free forever, in a datacentre where nobody is going to unplug it to vacuum the living room.

Free is not primarily about the money. Free is a constraint that keeps me honest. There is no budget to spend my way out of a design problem — I cannot add a node, cannot rent a managed control plane, cannot make an availability issue disappear with a load balancer someone else operates. Every layer of this thing is a decision I had to make on purpose, and every one of them stays small enough to rip out and redo in an afternoon.

The tier does bite. Getting the instance at all took two weeks of losing a race against everybody else on the internet who noticed those cores are free; the provisioning job in my CI retries the Terraform apply up to 300 times, 30 seconds apart, rotating availability domain on each attempt, and only aborts early if the error is something other than Out of host capacity. That is post one of this series. It is a for loop with a sleep in it and I am not proud of it, but it is the toll.

Doing it properly, for once

The interesting decision was not where to put the box. It was refusing the sentence "it's just a homelab."

That sentence is how you end up with a snowflake: a machine that works, that nobody — including you — can rebuild, and that you therefore never dare touch. My Raspberry Pis were exactly that. Every one of them worked right up until the moment I needed to move it, and then the undocumented parts of it stayed behind.

So the rules this time were the ones I apply at work, with no discount for it being personal:

Everything that exists is in Terraform. The network, the instance, the DNS zone, the object storage. If it was clicked in a console, it does not count, and it will not survive.

Everything in the cluster comes from Git. Flux reconciles the whole cluster from a repository. I do not run kubectl apply against my own cluster; if I want a change, I push it, and if I want to know why something is the way it is, there is a commit. The cluster is a consequence of a repo, not a pet.

I understand every layer. k3s rather than a managed Kubernetes, not because managed is bad but because the whole point is the parts a managed offering hides from me. There is no layer in this stack that I would have to shrug at.

What a platform actually is

Here is the part that a tutorial cannot teach you, because a tutorial deploys one application.

A platform is the machinery that owns the concerns an application should never have to think about. TLS certificates that renew themselves. One shared edge that routes traffic to whoever asks for it. Logs and metrics that exist whether or not the app author remembered to instrument anything. Backups that run at two in the morning and keep a week. Secrets that live encrypted in Git and get decrypted only in the cluster. Authentication in front of the things that need it.

Get that right and an application deployed onto it is only an application: a container, a route, a hostname. It inherits security, observability, and durability by landing there. Get it wrong and every new service drags its own half-solution for all six concerns behind it, and you are not running a platform, you are running six copies of the same unfinished work.

That separation is the thing I actually wanted to build. Not "Kubernetes at home" — I have done Kubernetes at home, and what I had was a cluster with applications on it. This time each concern is a layer, deliberately chosen, deliberately isolated, and the applications on top of it are almost boring to deploy. That is the whole ambition, and the name of this series: a production environment in miniature.

One node, no high availability, and I will say so every time it matters. But every concern a real production has is present and wired up, and none of them depend on me remembering to do something.

Which points at a claim I intend to make properly in its own post: the size of the cluster is not where most of the value of this stack comes from. Declarative infrastructure, reconciliation from Git, an edge that terminates TLS for everyone, backups that run whether or not anyone is watching — those pay off at one node exactly as they pay off at fifty. What you lose by going small is capacity and redundancy, which is real but is one line item. Plenty of teams run a cluster ten times this size and have none of the rest. Small is not the compromised version of this; it is most of it.

The other reason this exists now

This platform was built almost entirely in Claude Code sessions. Twenty-two merged pull requests on the infrastructure repo, most of them written by an AI session I supervised — including a full security review of the Terraform and CI pipelines, and the entire Kubernetes monitoring stack, built end to end that way.

That is not a side note, it is load-bearing. An unpredictable hour a week does not build a platform by hand. It does build one if the tedious 80% — the YAML, the Helm values, the first draft of a design, the second opinion on a security posture — is delegated to something that works while I read the diff. The judgment stayed mine; every one of those PRs went through review, and several got reverted. But the throughput is not something I could have faked with willpower.

How that actually works, where it fails, and what I refuse to hand over, is its own pillar on this blog.

What this series is

I am going to document the whole thing, bottom up, one layer per post — in the order the layers depend on each other, because every explanation of Kubernetes ingress that starts at ingress leaves you wondering what is holding the floor up.

flowchart BT
    A["Compute — OCI, Terraform"] --> B["Runtime — k3s"]
    B --> C["GitOps — Flux"]
    C --> D["Ingress — one shared Envoy Gateway"]
    D --> E["TLS — cert-manager"]
    E --> F["Observability — VictoriaMetrics, Grafana"]
    F --> G["Backup — Velero"]
    G --> H["This blog"]

The stack, bottom up: compute, runtime, GitOps, ingress, TLS, observability, backup — and this blog sitting on top of all of it.

Eight posts. Compute and the fight for free capacity. k3s, and what you give up by running a single node on purpose. Flux, and why I would rather push to main than run kubectl apply from my laptop. One Gateway to rule them all — the case for a single shared edge instead of an ingress per application. cert-manager, and the two lines of annotation that make TLS stop being a chore. Observability that costs nothing and still tells me when the disk is filling. Backups, and the uncomfortable question of whether I have ever restored one. And finally this blog, the newest tenant on the cluster and the one I will be most annoyed to break.

Three rules I am holding myself to.

Only decisions I actually made, on a system that is actually running. No reference architectures, no diagrams of systems nobody has operated.

The mistakes stay in. I put my Terraform state on an S3-compatible backend whose use_lockfile option does not, in fact, lock anything against OCI buckets — an upstream bug — and spent the rest of that day finding out and moving to the native OCI backend, which locks for real. My first instance destroy left the boot volume behind, quietly eating the free storage quota, because preserve_boot_volume defaults to true. My original bootstrap ran over SSH from the CI runner with no timeout, so a k3s install that hung turned into a Terraform apply that hung for hours. Each of those is worth more to you than the version where I got it right the first time.

No production-ready theatre. When something is a compromise, I will tell you what it costs, so that if you inherit one of my decisions you also know which assumption came with it.

This is written for engineers who are comfortable with the shape of this stuff — you know what a container is, you have been near Kubernetes, you can read a Terraform file — but who have never had to assemble the entire chain themselves, and would like to see someone else's homework first.

The last post is about this blog: an Astro app on the same cluster, behind the same Gateway, deployed by the same Flux loop as everything else. Which means that if you are reading this, that chapter already shipped.

Let's start at the bottom.