Fine-tuning changes model behavior through training. It is not the default fix for weak retrieval, vague policy, bad tool design, or missing evals. For most Belle Realty and Deal Leverage workflows, prompt discipline, structured outputs, RAG, and routing produce more value with less operational burden.
Fine-tune only when a stable, high-volume task has a large, lawful, high-quality labeled dataset; prompting has plateaued; and the gain is measurable on a locked eval set. You then own dataset provenance, privacy, training version, safety regressions, deployment, and rollback.
Use the decision guide, dataset prep rules, and tuned-model evaluation before approving a training run. A smaller tuned model can be a strong cost move, but only after you prove it preserves the behavior that matters.
Operating standard
Make this practice operational, not aspirational. Assign one directly responsible owner and name the decision they can make without another meeting. Put the key measure, threshold, and review cadence in the owning team’s regular operating rhythm. A change to model, prompt, data scope, retrieval index, tool permission, or policy should be recorded with its expected impact and a rollback path. Preserve enough trace information to explain an individual bad outcome without exposing more tenant or deal data than necessary. Review a small sample of real runs with the people doing the work; dashboards reveal trends, but operators reveal missing context. When the rule is violated, capture the incident, contain impact, add an eval or control, and update this document if the standard itself was unclear. The point is repeatable judgment under real workload, not a one-time compliance exercise.
Training data is the product. Start with a written task schema, inclusion/exclusion rules, label guide, and quality bar. Remove secrets, unnecessary personal data, protected characteristics, and unauthorized tenant content. Keep provenance for every example: source, consent/legal basis, annotator, transformations, and version.
Deduplicate near-identical examples, balance common and rare risk cases, and split by source entity or time to prevent leakage. For lease work, do not place versions of the same template in both train and test. Include abstentions, escalation examples, malformed inputs, and adversarial messages—not only perfect completions.
Use double annotation and adjudication for high-impact labels. Validate JSON targets automatically. Version the dataset immutably and document known limitations. If labelers cannot agree on the desired answer, the model will not solve the ambiguity; fix the policy first.
Operating standard
Make this practice operational, not aspirational. Assign one directly responsible owner and name the decision they can make without another meeting. Put the key measure, threshold, and review cadence in the owning team’s regular operating rhythm. A change to model, prompt, data scope, retrieval index, tool permission, or policy should be recorded with its expected impact and a rollback path. Preserve enough trace information to explain an individual bad outcome without exposing more tenant or deal data than necessary. Review a small sample of real runs with the people doing the work; dashboards reveal trends, but operators reveal missing context. When the rule is violated, capture the incident, contain impact, add an eval or control, and update this document if the standard itself was unclear. The point is repeatable judgment under real workload, not a one-time compliance exercise.
Compare a tuned model against the current baseline on a locked, task-representative set. Report quality by critical slice, schema validity, citation/groundedness where applicable, safety behavior, latency, cost, and calibration. Do not accept a single average score or vendor training metric.
Test distribution shift: new property templates, scans, slang, incomplete records, prompts that ask for prohibited actions, and inputs outside the training taxonomy. Verify that the tuned model abstains or routes correctly instead of forcing a familiar-looking answer. Test for memorization and data leakage with canaries and held-out entities.
Launch in shadow mode, log disagreements with the baseline and staff, and keep an immediate fallback. Re-evaluate after policy changes and periodically as inputs drift. Fine-tuning is successful only if it improves the actual workflow enough to pay for ongoing data and governance work.
Operating standard
Make this practice operational, not aspirational. Assign one directly responsible owner and name the decision they can make without another meeting. Put the key measure, threshold, and review cadence in the owning team’s regular operating rhythm. A change to model, prompt, data scope, retrieval index, tool permission, or policy should be recorded with its expected impact and a rollback path. Preserve enough trace information to explain an individual bad outcome without exposing more tenant or deal data than necessary. Review a small sample of real runs with the people doing the work; dashboards reveal trends, but operators reveal missing context. When the rule is violated, capture the incident, contain impact, add an eval or control, and update this document if the standard itself was unclear. The point is repeatable judgment under real workload, not a one-time compliance exercise.
Fine-tune when the task is narrow, repeated, and stable: classify a well-defined OTB message taxonomy, extract a consistent internal document format, or produce a tightly controlled output style. Do not fine-tune to memorize current leases, replace retrieval, solve live factual questions, or bypass human approval.
Before training, prove a baseline using prompt, few-shot examples, retrieval, and a capable model. Identify the persistent error slice and estimate volume. If a prompt/routing change fixes it, do that. If a model repeatedly misses domain conventions despite clear examples and the workflow has thousands of runs, training may be justified.
Require data rights, privacy review, a locked eval set, a rollback route, and a cost model that includes labeling and maintenance. Fine-tuned models can encode stale policy or sensitive patterns. The decision should be an evidence-backed product investment, not an attempt to make a model “know the business.”
Operating standard
Make this practice operational, not aspirational. Assign one directly responsible owner and name the decision they can make without another meeting. Put the key measure, threshold, and review cadence in the owning team’s regular operating rhythm. A change to model, prompt, data scope, retrieval index, tool permission, or policy should be recorded with its expected impact and a rollback path. Preserve enough trace information to explain an individual bad outcome without exposing more tenant or deal data than necessary. Review a small sample of real runs with the people doing the work; dashboards reveal trends, but operators reveal missing context. When the rule is violated, capture the incident, contain impact, add an eval or control, and update this document if the standard itself was unclear. The point is repeatable judgment under real workload, not a one-time compliance exercise.