Private AI Infrastructure

A design blueprint for running a large language model that a single operator owns end to end: the weights, the training data, the alignment rules and the hardware it runs on. No API provider in the loop, and no one else’s terms of service deciding what the model will and won’t do. The tradeoff is that the operator also owns every decision the model makes, so the interesting part isn’t the training run, it’s the constitution written before it.

Download the full blueprint (.docx)

Why not just call an API

Because the two workloads behind a language model don’t look alike. Training needs a large amount of compute for a short burst. Running the model day to day needs a moderate, steady amount of compute for as long as it’s in service. The blueprint treats those as separate problems: rent GPU clusters in the cloud for training, then export the finished weights and run them on hardware you own. Renting solves the burst; owning solves the steady state, and it’s the only way to keep the model itself off someone else’s servers.

Six layers, base model to deployment

The architecture is a pipeline. It starts from an open-weights base model, one of the Llama, Mistral, Mixtral or DeepSeek families, so the enormous cost of teaching a model language and reasoning from nothing is already paid by someone else. From there: a behavioral dataset defines what the model should know and how it should act, supervised fine tuning teaches it to follow that dataset using parameter-efficient methods rather than retraining the whole network, and alignment training shapes its preferences, either from ranked human feedback or from a written constitution the model critiques its own answers against. The result is exported as a plain set of weight files and deployed on local GPUs, at which point no API provider is involved and the operator controls runtime behavior directly.

The constitution is the product

Everything the model becomes traces back to four kinds of data: instruction examples that teach it to answer prompts correctly, domain knowledge relevant to what it’s for, comparisons that show it a good response next to a bad one, and a written set of constitutional rules, things like prioritizing factual accuracy, showing reasoning for hard problems, and admitting uncertainty instead of guessing. That last piece is the one that’s easy to underweight going in. The base model already supplies fluency and general reasoning; the dataset and constitution are the only place the operator’s judgment actually enters the system.

What it costs to train

Model size drives everything downstream. A 7B model is cheap to fine tune, tens to a couple hundred dollars of rented GPU time, but its reasoning is basic. A 70B model is strong enough to be useful and still fits on a handful of high-end GPUs, with fine tuning in the low thousands of dollars and alignment training running from roughly five thousand to forty thousand depending on how much comparison data it takes to shape the behavior. For most private deployments, 30B to 70B parameters is the sweet spot between capability and what a single operator can afford to train and then host. A full custom 70B-class system, dataset, fine tuning, alignment, evaluation and hardware included, lands somewhere between $30k and $120k.

Running it at home

Deployment hardware scales in clear steps. A $5,000 workstation with a single RTX 4090 runs 7B to 13B models for development. Around $15,000, two 4090s and a Threadripper handle 13B to 30B. A $30,000 build with four 4090s and liquid cooling reaches into 30B–70B territory. Past that it stops being a workstation: a $100,000 server with eight H100s handles 70B–200B, and a $250,000 cluster of four nodes, each with eight H100s, gets into the 200B–400B range on roughly 200Gb InfiniBand between nodes. Power draw scales right alongside it, a few kilowatts and a normal circuit for the workstation tier, industrial cooling and 240V infrastructure once you’re running a real cluster.

Private systems fine-tune an existing base model instead of training one from scratch, which is the only reason any of this is affordable. Frontier labs train from scratch on 10,000 to 25,000 GPUs at a time, tens to well over a hundred million dollars per run. A $250,000 cluster is 32 GPUs. That gap doesn’t close; the point isn’t to compete with frontier labs, it’s to own a smaller model outright instead of renting access to a bigger one.

Where the trend line points

Compute cost per dollar has historically doubled roughly every 1.5 to 2 years, which over five years works out to an 8–10x improvement. Read against today’s hardware tiers, that suggests a $20,000 budget reaching 100B–200B class models and a $100,000 budget reaching 500B–1T within about five years, the same money buying a materially bigger model over time. The roadmap to get there doesn’t change: pick a base model, write the constitution first, build the instruction and alignment datasets, train and evaluate in the cloud, then export and move the weights to hardware you control, and repeat the cycle as the constitution and datasets improve.

What it covers

Base model selection, dataset and constitutional design, parameter-efficient fine tuning, alignment training, and matching hardware to model size from a single workstation up through a small cluster. I’m happy to go deeper on any of it, just shoot me an email.