The default assumption in enterprise AI is that you should use the best frontier model available (GPT-4o, Claude Sonnet, Gemini Pro) and tune the prompt until it works. This is often the right call. But it’s not always the right call, and knowing when to deviate is worth real money.
Fine-tuning open-source models for specific tasks is now more accessible than it’s ever been. The question is no longer “can we do it” but “should we, and what does it actually cost?”
The Four Reasons to Fine-Tune
There are exactly four legitimate reasons to fine-tune rather than prompt:
1. Latency. Frontier models are large. GPT-4o has a significantly larger parameter count than, say, Llama 3 8B. A fine-tuned 8B model running on your own infrastructure can respond in under 100ms at P99. Frontier model APIs typically add 500–2000ms of API latency on top of your application logic.
2. Cost at scale. GPT-4o costs roughly $5–15 per million output tokens depending on your tier. A fine-tuned Llama 3 8B running on an A10G instance ($0.75/hr on Lambda Labs) can process millions of tokens per hour. At sufficient volume, the economics invert dramatically.
3. Data privacy. If you can’t send your data to a third-party API (HIPAA, GDPR, contractual NDAs, or internal policy), you need a self-hosted model. Fine-tuning lets you create a capable model without ever exposing sensitive data externally.
4. Quality on a narrow task. For highly specific tasks (classifying documents in a particular taxonomy, generating text in a proprietary format, extracting fields from structured documents), a fine-tuned small model can outperform a prompted large model. Not always, but often enough to check.
If none of these apply, don’t fine-tune. Prompting is easier to iterate, cheaper to start, and simpler to maintain.
Training Cost Breakdown
Fine-tuning cost has three components: compute, data preparation, and engineering time. The last one is consistently underestimated.
Compute Costs
For a full fine-tune of Llama 3 8B on a 10,000 example dataset:
| Method | Hardware | Duration | Cost |
|---|---|---|---|
| Full fine-tune | 8× A100 (80GB) | ~6 hours | ~$50–80 |
| QLoRA (4-bit) | 1× A100 (80GB) | ~8 hours | ~$12–15 |
| QLoRA (4-bit) | 1× A10G (24GB) | ~14 hours | ~$10–12 |
QLoRA (Quantized Low-Rank Adaptation) is the practical choice for most use cases. It achieves 85–95% of full fine-tune quality at a fraction of the cost and hardware requirement.
For Llama 3 70B, multiply compute costs by approximately 8–10x. At that scale, the economics need to be compelling before you commit.
Data Preparation
This is where the actual time goes. Assuming you’re starting from raw data (logs, documents, existing examples):
- Cleaning and formatting: 20–40 hours per 10,000 examples if automated with some human QA
- Quality review: At minimum, a human needs to verify a sample. Budget 1 hour per 100 examples for meaningful QA.
- Instruction formatting: Converting raw data to the instruct format the model expects (system/user/assistant turns)
Real data preparation cost for 10,000 high-quality examples: typically $3,000–$8,000 in engineering time. This often exceeds the training compute cost by 5–10x.
Inference Costs
This is where fine-tuning pays back:
Llama 3 8B (quantized, INT4) on A10G:
- Throughput: ~2,000 tokens/second per GPU
- Cost: $0.75/hr for the GPU
- Effective cost: $0.37 per million tokens
GPT-4o:
- Input: $2.50 per million tokens
- Output: $10 per million tokens
At 100 million tokens per month (a moderate enterprise workload), you’re looking at $375/month for self-hosted vs $1.25M+ for GPT-4o. Even with infrastructure overhead (reliability, team time), the self-hosted number is dramatically lower.
The crossover point is roughly 10 million output tokens per month before self-hosting becomes clearly cost-effective, accounting for engineering overhead.
Quality: Where Fine-Tuning Wins and Loses
Fine-tuned small models beat prompted large models in specific conditions:
Classification and extraction. A fine-tuned Llama 3 8B on a legal document classification task consistently beats GPT-4o prompted with a taxonomy description. The fine-tuned model has internalized the class distinctions; the prompted model is reasoning about them at inference time.
Format adherence. If your output must follow a rigid structure (JSON with specific field names, Markdown with a particular template, CSV with specific columns), fine-tuning is more reliable than prompting. Large models occasionally deviate from format instructions, especially in long outputs.
Domain terminology. In highly specialized domains (medical, legal, financial, scientific), fine-tuning on domain-specific text makes the model fluent in that vocabulary without needing extensive few-shot examples in every prompt.
Where frontier models maintain the advantage:
Complex multi-step reasoning, open-ended generation, tasks that require broad world knowledge, and anything requiring the model to handle novel situations it wasn’t trained on. Fine-tuning specializes; it doesn’t generalize.
Data Requirements
The minimum bar for useful fine-tuning results is approximately 500–1,000 high-quality examples. Below that, results are inconsistent and often worse than a well-prompted baseline.
“High quality” means:
- Examples that accurately represent the target task distribution
- No ambiguous or contradictory examples
- Correct outputs (human-verified, not LLM-generated without review)
- Diverse: not 500 examples of the same pattern with slightly different wording
More data is better, with diminishing returns above ~10,000 examples for most classification and extraction tasks. Generative tasks benefit from more data for longer, up to ~50,000 examples for stable quality.
The Recommendation Framework
Use this as a starting point:
Volume > 10M tokens/month? → evaluate self-hosting
AND (latency P99 < 200ms required? → strong fine-tune case)
AND (data privacy required? → must fine-tune or use on-prem API)
AND (narrow, well-defined task? → fine-tune likely beats prompting)
Otherwise:
Start with prompting (GPT-4o / Claude Sonnet)
Evaluate quality first, then optimize cost once quality is proven
The biggest mistake we see: teams investing in fine-tuning before they’ve validated that the task is well-defined and achievable. A well-prompted frontier model is the right tool for exploration. Fine-tuning is a production optimization, not a development tool.
Practical Notes
Use Axolotl for training. It’s the most maintained open-source fine-tuning framework, supports QLoRA natively, and has good documentation. Avoid raw HuggingFace training scripts unless you need unusual customization.
Evaluate on a held-out set before you declare victory. It’s surprisingly common to overfit on a small dataset and ship a model that performs worse on the distribution it’ll actually see in production.
Plan your model lifecycle. Fine-tuned models need re-training as your task distribution shifts. Build the training pipeline as a repeatable process, not a one-time event.
Start small. Before committing to 70B, test your task with 8B. If 8B achieves 90% of the quality you need, ship it and save the infrastructure cost.
The economics of fine-tuning have never been better. The discipline required to do it well has never been more important.