Why We Built Our Own LLM

In mid-2023, at the height of the GPT-4 wrapper gold rush, our startup did something that made several smart people question our judgment. We announced Nebula, our own large language model built for human conversation, and shipped it as a production API.

The prevailing wisdom was unambiguous. Foundation models were a game for companies with billion-dollar compute budgets, and startups were supposed to sit on top of the frontier, not compete with it. When I told people we were training our own model, the polite ones asked why. The less polite ones just told us we would regret it.

They were partly right and partly very wrong, and the difference between those two parts is the most useful thing I learned from the whole project. Now that some version of “should we train our own model?” lands on every AI engineering leader’s desk, whether it’s fine-tuning, distillation, or small models at the edge, it seems worth writing down what the decision actually looked like from inside.

We didn’t start from zero

The context matters. By 2023 we had spent five years at Symbl.ai on one problem, making machines understand human conversations rather than just transcribe them. That meant knowing who committed to what, what topics actually got discussed, which questions went unanswered, and what should happen next.

We had built that capability the pre-LLM way, as a real-time pipeline of purpose-built models, each doing one job on live speech. The work became a family of US patents on detecting conversational structure, extracting topics and action items, tracking concepts, and summarizing dialogue. It was also a long education in everything that makes conversation different from written text. Disfluency, interruption, multi-party crosstalk, ASR errors compounding into NLU errors, and meaning that assembles itself across turns rather than sentences.

So when large language models arrived and started collapsing whole categories of NLP pipelines into a single prompt, we faced the question every NLP company faced that year. Does our pipeline still matter? And behind it, a sharper one. If the answer is to use an LLM, whose LLM?

Why not just use GPT-4?

We used frontier models where they made sense. But for the core of our product, understanding conversations at scale for enterprises, the case for depending entirely on someone else’s general-purpose model kept failing on four grounds.

Distribution. General LLMs are trained overwhelmingly on written text, which is edited, linear, single-author prose. Real conversation looks nothing like that. It’s co-constructed, self-interrupting, elliptical, and by the time it reaches the model it has been filtered through speech recognition noise. We saw this daily in practice and later measured it. Models that clearly specialize for code and math treat dialogue mostly like noisy web text, so a model trained on conversation had genuine headroom.

Economics. Conversation workloads are token-heavy in a way chat products are not. An hour-long meeting is tens of thousands of tokens before you’ve asked a single question about it, and our customers processed conversations continuously. At frontier per-token prices, running everything through the biggest model didn’t work as unit economics, and the real-time use cases added latency budgets that big general models couldn’t reliably meet.

Control. Enterprises trust you with their most sensitive data when they trust you with their conversations. Sales calls, support calls, internal meetings. Where the model runs, how it gets updated, what its failure modes are, and whether its behavior can change under you without warning stop being philosophical concerns and become contract terms. Owning the model meant owning those answers.

Differentiation. If your product is a thin layer over someone else’s model, your roadmap depends on theirs. We had something most wrapper companies didn’t, years of conversational data expertise and task definitions no general model was optimized for, and the way to convert that into product advantage was to push it into the model layer.

None of these arguments has expired. What has changed is the menu of ways to act on them, which I’ll come back to.

What it actually took

Here is what surprised me, and what I tell everyone who asks. Training the model was not the hard part. The hard parts were on either side of it.

Before the model, data. The overwhelming majority of the effort was corpus work, curating conversational data that was clean, licensed, representative, and structured to teach a model what dialogue actually is. Good conversation data is rare in a way web text never is. High-quality, multi-party, real-world dialogue with reliable structure is one of the rarest resources in machine learning, and most of what exists is synthetic, scripted, or legally untouchable. Our years of domain work were the only reason this was possible at all.

After the model, evaluation. This was the hardest problem and, in hindsight, the most valuable thing we built. Standard benchmarks tell you very little about conversation understanding. A model can ace MMLU and still not be able to tell you who promised what on a sales call. Whether the model understood this conversation is a question you have to operationalize yourself, with task definitions, graded rubrics, adversarial transcripts, robustness checks against ASR noise, and regression suites that catch quality drift before customers do. We had to build our evals before we could trust our model, and the evals outlived the model itself.

Underneath everything, serving. Owning a model means owning its inference economics. Throughput, quantization, latency SLOs, capacity planning. That operational muscle is invisible from the outside and decisive in production.

What I got wrong

I underestimated the slope of the frontier.

When we started Nebula there was a real, measurable capability gap on conversational tasks between what we could build and what general models did out of the box. That gap was our justification, and it was genuine. What I misjudged was how fast frontier models would improve at everything, including our corner of the world. Capabilities we spent months engineering became emergent properties of the next model generation. A small team can absolutely build a model that beats the frontier on a narrow domain today. Keeping it ahead every quarter is a different sport with different economics.

If I had treated the model weights as the asset, this story would end in regret. The reason it doesn’t is that the durable assets turned out to be everything around the weights.

• The evals. Still the moat. Whoever can measure conversation understanding best can adopt any model and know quickly what it’s actually worth.
• The data flywheel. Curation pipelines, structure, licensing know-how. All of it model-agnostic, all of it compounding.
• The serving infrastructure. Every model, ours or anyone’s, rides on it.
• The organizational capability. A team that has trained, evaluated, and operated its own LLM reads the field differently. When a new frontier model drops, that team knows within a day what it means for the product and where it still falls short.

How I’d make the decision in 2026

The question I ask now is not build or buy. It’s which layers you actually need to control.

Almost nobody needs to own frontier-scale weights. Everybody needs to own their evaluation and their data, because that’s the layer where your domain understanding lives, and outsourcing it means flying blind inside someone else’s model.

Owning the model layer itself still makes sense in specific, recognizable situations. When latency is physics rather than preference, like real-time voice. When deployment constraints are contractual and data can’t leave a boundary. When your domain distribution is genuinely far from web text and you hold proprietary data for it. Or when unit economics at your scale make small specialized models decisively cheaper than frontier calls. It’s no accident that voice agents, which are about as latency-sensitive and conversation-shaped as workloads get, are where small purpose-built models are making a comeback.

Would I build Nebula again? Yes, and this time I’d know what I was actually buying. Not the weights, which were a wasting asset from the day training finished. What we bought was the capability. The evals, the data discipline, the serving muscle, and a team that understands language models from the inside. In an industry where the ground moves every quarter, that’s the part that compounds.