In short
Most companies do not need to train a model first. They should usually start with RAG: connect AI to approved company documents and retrieve relevant context. Fine-tuning is useful only when the model must learn a repeatable style or task pattern.
Why this matters: When to use RAG, fine-tuning, or custom workflows for company knowledge.
What to do next
- 1Clean and classify documents.
- 2Decide access rights.
- 3Build retrieval with citations.
- 4Evaluate answers before launch.
What to look at first
Company data projects should begin with data governance, not model choice.
- Clean and classify documents.
- Decide access rights.
- Build retrieval with citations.
- Evaluate answers before launch.
What the result should be
The result is an AI assistant that uses company knowledge without blindly exposing every document.
Start with the work that should genuinely disappear
'Training on company data' rarely means training a new language model. For current knowledge, retrieval is usually the right pattern: the model searches permitted sources at question time. Fine-tuning is more useful for stable output style or task behaviour. Start with the outcome, source quality, permissions and evaluation rather than the heaviest technical option.
- Decide whether knowledge, behaviour or prediction must improve.
- Remove duplicate, outdated and contradictory sources.
- Carry source permissions into retrieval and answers.
- Build realistic questions with expected sources and refusals.
A realistic delivery example
An internal knowledge assistant uses procedures, approved templates and technical documentation. Employees receive source passages only from files they may access. New versions are indexed automatically and superseded procedures leave the active set. No fine-tuning is required because currency and traceability matter more than a unique writing style.
Move from trial to dependable operation
Begin with one bounded workflow, representative examples and human review. Increase automation only after failures are visible and recoverable.
- Choose one audience and valuable task.
- Inventory sources, owners, sensitivity and freshness.
- Build retrieval with citations and access filters.
- Evaluate answer, source and safe refusal separately.
- Consider fine-tuning only when retrieval demonstrably cannot solve the problem.
Measure quality and business outcome
Record a baseline before launch. Model activity is not an outcome; the application should improve time, quality, revenue or risk.
- Correct answers supported by the correct source.
- Sensitive or unsupported questions safely refused.
- Time to find information before and after launch.
- Outdated sources, access leaks and user corrections.
Where human control must remain
Company data should not be sent to model providers without a lawful purpose and appropriate contracts. Remove personal information where possible, enforce document permissions before retrieval and protect system instructions and secrets. High-risk decisions still require qualified review.
Choose RAG, fine-tuning or rules by required behaviour
Use RAG when answers need current citable knowledge, fine-tuning for a stable format or repeated classification pattern, and deterministic code for hard decisions. Combine them only when evaluation shows a genuine gap. First create a test set of real questions, roles and expected sources. Measure accuracy, evidence, refusal and data leakage before user access.
- RAG for changing, citable knowledge.
- Fine-tuning for repeated format or behaviour.
- Rules for hard conditions and transactions.
- Access filtered before retrieval.
- Evaluation set by role, topic and risk scenario.
Written and reviewed by
Ingmar van Maurik
Founder, AI JOB TEAM
Builds practical AI, automation, and custom software systems for growing companies that need less tool sprawl and more ownership.
Editorial note
Written for decisions, not generic search traffic
AI JOB TEAM uses AI-assisted drafting for research structure and coverage checks. Ingmar van Maurik reviews the positioning, examples, and final recommendations so every article stays practical for growing companies.
Industry applications
See how this topic translates into a concrete workflow for a specific business type.
FAQ
Where should a growing company start?
Start with one workflow where volume, cost, or customer impact is already visible. That keeps scope small and learning fast.
When is this worth a deeper roadmap?
It is worth a roadmap when the topic touches multiple teams, systems, or recurring decisions.
Do we need to train our own LLM?
Almost never as a first step. Retrieval is faster, cheaper and easier to keep current.
Can existing permissions be respected?
Yes, when access is enforced before retrieval rather than only in the interface.
What happens when documents change?
Updated files are reprocessed; the language model itself does not need retraining.
Must company data be copied to a model provider?
Not necessarily. Choose architecture by purpose, contract, retention and security and send only minimal relevant context.
When is fine-tuning the wrong choice?
When the main issue is missing current knowledge or access control. Fine-tuning is not a document database.
Next step
Make the AI opportunity concrete
Use the AI Roadmap to choose use cases, data readiness, tooling, governance, and the first safe implementation step.
