The Playbook
How to Build a Domain-Specific Sovereign Model — Pre-Training, SFT & Alignment
Starting from a frontier OSS base model like Kimi, LLaMA, or Mistral — here's the end-to-end journey covering pre-training, SFT dataset curation, preference tuning with GRPO/DPO, and LLM-as-a-judge evaluation.
Phase 01
Dataset Curation & Preparation
Your data is the moat — not the model.
Start with your proprietary knowledge: internal docs, domain ontologies, decision logs, and curated public data. The goal is a high-quality, deduplicated, domain-aligned corpus.
Source Aggregation
Collect internal docs, codebases, knowledge bases, regulatory filings, and curated public datasets relevant to your domain.
Cleaning & Dedup
Remove noise, near-duplicates, and low-quality samples using MinHash / SimHash pipelines. Normalize formatting across sources.
Domain Taxonomy
Build or integrate an existing ontology to tag and categorize data — this drives downstream alignment and evaluation.
Synthetic Augmentation
Generate additional training pairs (Q&A, instruction-response, chain-of-thought) using a teacher model to fill coverage gaps.
Quality Scoring
Assign quality scores using perplexity filters, human spot-checks, and automated toxicity / factuality classifiers.
Phase 02
Continued Pre-Training (CPT)
Teach the foundation model to speak your language.
Take a frontier OSS base model (e.g., Kimi, LLaMA, Mistral) and continue pre-training on your domain corpus. This injects deep domain knowledge without starting from scratch.
Base Model Selection
Pick a frontier OSS checkpoint (e.g., Kimi-VL, LLaMA-3, Mistral-Large) based on your modality needs, license, and parameter budget.
Learning Rate Warmup
Use a low learning rate (1e-5 range) with cosine decay to avoid catastrophic forgetting of general capabilities.
Data Mixing
Blend domain data with a fraction of general-purpose data (e.g., 70/30) to preserve broad reasoning while gaining specialization.
Checkpointing
Save intermediate checkpoints every N steps for rollback and evaluation — domain knowledge absorption is non-linear.
Phase 03
Supervised Fine-Tuning (SFT)
Shape behavior with curated instruction data.
Fine-tune the CPT checkpoint on high-quality instruction-response pairs specific to your use case. This is where the model learns your workflows, tone, and task patterns.
Instruction Dataset
Curate 5K–50K high-quality examples: task prompts paired with ideal completions, reviewed by domain experts.
LoRA / QLoRA
Use parameter-efficient fine-tuning (LoRA adapters) to reduce GPU requirements while preserving base model strength.
Multi-Task Formatting
Structure data across task types (summarization, extraction, reasoning, code) using a unified chat template.
Hyperparameter Sweep
Grid search over batch size, learning rate, and LoRA rank. Monitor loss curves and downstream task accuracy.
Phase 04
Alignment & RLHF / DPO
Align outputs to human preferences at scale.
Apply reinforcement learning from human feedback (RLHF) or Direct Preference Optimization (DPO) to align the model's outputs with your quality standards, safety policies, and brand voice.
Preference Data
Collect pairwise comparisons (chosen vs. rejected) from domain experts on model outputs across representative prompts.
Reward Modeling
Train a reward model on preference data, or skip it entirely with DPO which optimizes directly on preferences.
Policy Optimization
Run PPO (with reward model) or DPO to shift the model's distribution toward preferred outputs while staying close to the SFT checkpoint.
Safety Guardrails
Embed refusal behaviors, content policies, and domain-specific constraints during alignment.
Phase 05
Enterprise-Grade Evaluation
Measure what matters — not generic benchmarks.
Evaluate the model against your actual KPIs: domain accuracy, factual grounding, latency, cost, and safety. Generic leaderboard scores don't tell you if the model works for your business.
Domain Test Suites
Build eval sets from real production queries, edge cases, and adversarial inputs specific to your domain.
Automated Metrics
Track perplexity, BLEU/ROUGE (where applicable), factuality scores, and latency per token across deployment targets.
Human Eval
Run blind A/B tests with domain experts comparing your tuned model against the base model and leading alternatives.
Regression Testing
Ensure new capabilities don't degrade performance on previously passing test cases — track regressions across checkpoints.
Phase 06
Deployment & Iteration
Ship to your infra. Own every byte.
Deploy the final model to your S3 bucket, private cloud, or on-prem infrastructure. No vendor lock-in. Set up monitoring and a feedback loop for continuous improvement.
Model Export
Export in your preferred format (GGUF, SafeTensors, ONNX) optimized for your serving stack (vLLM, TGI, Triton).
Infrastructure
Deploy on your own GPUs, cloud VMs, or edge devices. We hand over the model — you own the runtime.
Monitoring
Instrument production inference with logging, drift detection, and quality sampling for continuous model health checks.
Feedback Loop
Pipe production feedback back into training data for the next iteration — sovereign models improve with use.
Ready to Build Your Sovereign Model?
We handle the heavy lifting — from dataset curation to deployment. Talk to us about your use case.