At Level 2, prompting stops being clever phrasing and becomes interface design. A prompt defines role boundaries, data contracts, tool permissions, output schema, escalation behavior, and attack resistance. If your prompt is doing business logic, move that logic into code. If it is hiding an ambiguous requirement, write an eval before adding more instructions.
Use reasoning patterns selectively. Tree-of-thoughts can explore alternatives, but it is expensive and often unnecessary. Reflexion can repair a draft, but only with a concrete rubric and source evidence. DSPy can systematically optimize modules, but only after you have a trustworthy metric. None of these replace retrieval, typed outputs, or authorization.
The practical baseline is short instructions, relevant evidence, strict schemas, named refusal conditions, and examples only for behavior the model consistently misses. Separate the internal reasoning you do not need from the observable decision record you must audit. In property workflows, the trace should show evidence and policy rule—not invented prose about hidden thought.
Adversarial hardening is part of prompt design. Tenant messages, lease PDFs, and data rooms are untrusted inputs. Treat every embedded instruction as content, never as authority.
Before adding complexity, run this design through a small representative eval and inspect the trace with the operator who will own failures. Make the boundary, escalation, and rollback visible in the product. That discipline will expose more useful work than another round of prompt cleverness.
In this module
prompting-techniques.md — the 10 techniques that actually move the needle, each with an eval you can run
Assume documents and messages can contain instructions designed to redirect the model: “Ignore your rules,” “export all tenants,” “approve this payment,” or subtler text inside a lease PDF. This is prompt injection. It is not solved by adding “do not be injected” to a system prompt.
Create a trust boundary. System instructions, application policy, tool schemas, and authenticated user permissions are authority. Retrieved lease text, tenant messages, web content, file names, and tool results are data. Label untrusted content in the prompt and instruct the model to extract facts from it, never execute its instructions. More importantly, make tools incapable of prohibited actions.
Use allowlisted tools, typed arguments, server-side authorization, row limits, and proposed writes with human approval. Strip or flag instruction-like text in retrieved chunks for reviewer visibility; do not silently alter source evidence. Separate retrieval from action: an untrusted message can lead to a draft or escalation, not a database write or an external send.
Build red-team cases: direct override attempts, encoded instructions, document metadata tricks, cross-tenant data requests, tool-argument injection, and social-engineering requests for policy exceptions. Measure tool-call policy violations and unsafe-action proposals, not only refusal wording. Add canaries—fake secrets or forbidden record IDs—that must never appear in output.
Prompt hardening is defense in depth. The strongest prompt cannot compensate for an MCP tool with excessive privileges.
DSPy is a framework style for expressing LLM work as modules with declared input/output signatures and optimizing demonstrations or instructions against a metric. The important idea is transferable even if you do not adopt the library: treat prompts as compiled components with tests, not handwritten incantations.
A Belle Realty extractor signature might accept clause_type and evidence_chunks and return normalized_value, quote, page, confidence, and status. Its metric checks exact value, citation support, valid schema, and abstention. You can then compare prompt variants, few-shot examples, model choices, and retrieval settings reproducibly. Optimization must use a development set; keep a locked set to detect overfitting.
Do not optimize a vague “quality” score. Metrics shape behavior. If you reward only exact extraction, the system may guess rather than abstain. Add penalties for unsupported values, wrong-lease citations, and false confidence. If human review matters, include its rate and correction rate in the objective.
DSPy is not a shortcut around data. It amplifies whatever your eval tells it to optimize. Start with a hand-built baseline and clear signature. Use it when manual prompt iteration has become unrepeatable, not before you understand the task.
Before adding complexity, run this design through a small representative eval and inspect the trace with the operator who will own failures. Make the boundary, escalation, and rollback visible in the product. That discipline will expose more useful work than another round of prompt cleverness.
Prompting techniques: what actually moves the needle
Most "prompt engineering" content online is folklore. This document is the opposite: a small set of prompting techniques that measurably change model behavior, each paired with an eval you can run to see the effect for yourself.
The rule for this document — and for how you should think about prompting in production — is:
A prompting technique that you cannot measure is a superstition.
If adding "you are a world-class expert" to your prompt doesn't change your eval score, it doesn't do anything. It just makes you feel better.
Adversarial self-check — moderate ROI on safety-critical work
Meta-cognitive uncertainty gating — low-moderate ROI, high value on HITL
Output-shape scaffolds — moderate ROI, overlaps with #1
"Power words" (neuro-symbolic, etc.) — low ROI in isolation, sometimes real inside a system
Everything below the fold has an eval. Run it against your own model + task before you commit to it in a prompt.
1. Structured output enforcement
What it is: Force the model's output through a JSON schema, either via tool-use, response_format json_schema, or Zod-validated re-prompt on failure.
Why it works: The model can no longer wander. Every field is either present or absent, every enum is either valid or invalid. Downstream code doesn't need to parse. Debugging is trivial.
Eval:
Task: Classify 100 support tickets by priority (P1, P2, P3).
A) Ask for prose "please respond with the priority": 100 responses.
B) Ask for JSON with a schema {priority: enum, reason: string}: 100 responses.
Measure:
- Parse-error rate (A: usually 3-15%. B: 0-1%.)
- Time-to-consume-in-downstream-code (A: needs regex/parsing. B: JSON.parse.)
- Consistency across identical inputs (B is dramatically more consistent.)
When it fails: When the schema is wrong for the task (e.g., you enum'd to 3 categories but the real world has 5). No prompting technique fixes a wrong schema.
2. Few-shot with worked examples
What it is: Include 2–5 input/output pairs in the prompt showing exactly the transformation you want.
Why it works: Models learn the mapping from examples much faster than from descriptions. Especially valuable for edge cases and format quirks.
Eval:
Task: Extract vendor names from 50 varied invoice PDFs (some have logos, some misspellings, some no vendor line).
A) Zero-shot ("Extract the vendor name")
B) Zero-shot with detailed rules
C) 3-shot with worked examples covering the edge cases
Measure exact-match accuracy against a hand-labeled ground truth.
Typical result: A ~65%, B ~78%, C ~92%.
Rules:
- Cover edge cases in your examples, not the easy path
- Use examples that were actually confusing before
- Keep examples in the same format as the real task
- 3 examples > 20 examples for most cases (diminishing returns after ~5)
3. Chain-of-thought (for hard reasoning only)
What it is: Ask the model to reason step-by-step before answering. Either explicit ("Let's think step by step") or via a scratchpad tool.
Why it works: Models allocate more compute per token when reasoning is visible. For arithmetic, multi-step logic, and complex classification, this dramatically improves accuracy.
Why it can hurt: For simple classification, forcing the model to reason introduces noise. The model rationalizes its way to worse answers.
Eval:
Task: 100 math word problems from GSM8K.
A) Direct answer only.
B) "Let's think step by step." then the answer.
Measure: correct-answer rate.
Typical: A 40-60%, B 75-92% (on capable models).
Counter-eval:
Task: 100 tickets, is this P1 or not?
A) Direct answer P1/not-P1.
B) "Reason step by step, then answer."
Sometimes A and B are equal. Sometimes A is BETTER because the model doesn't talk itself into upgrades.
Rule: CoT for arithmetic and multi-step logic. Don't reflexively apply it to simple classification.
4. Role and persona (targeted)
What it is: "You are a compliance lawyer reviewing this NDA."
Why it can work: The model has seen enough training data associated with specific roles that "compliance lawyer" pulls in relevant vocabulary, caution levels, and structure.
Why it's overused: "You are a world-class expert in X" is background noise the model has seen millions of times. It doesn't add real signal.
When it moves the needle:
- Specific role: "You are a paralegal preparing a redline for a partner's review"
- Combined with a task shape: "Return a redline as a numbered list of proposed changes"
- Where the role implies specific vocabulary: "You are an ophthalmologist"
When it's noise:
- "You are a world-class expert."
- "You are an award-winning writer."
- "You are the best in the world at this."
Eval:
Task: Draft 50 tenant maintenance responses.
A) No persona.
B) "You are a helpful assistant."
C) "You are a property manager with 15 years of experience at a class-B retail center."
Measure with a rubric (clarity, empathy, specificity) scored by a second LLM as judge.
Typical: A ≈ B, C moderately better. Deltas are real but small vs #1 and #2.
5. Discipline / domain prefixes
What it is: Prefixing the request with a domain framing like "This is a real-estate diligence question." or "Approach this as a legal compliance issue."
Why it works: Similar to persona, but framing the problem instead of the answerer. Sometimes even more effective because it changes how the model interprets ambiguous terms in the input.
Eval:
Task: 40 ambiguous questions like "What's the biggest risk here?"
A) No framing.
B) "Approach this as a legal question."
C) "Approach this as a financial question."
Measure how the answers cluster. B pulls answers toward liability, indemnification, notice deadlines. C pulls answers toward cash flow, covenants, DSCR.
Practical use: When your task has ambiguous domain terms (e.g., "risk", "value", "priority"), a domain prefix disambiguates cheaply.
6. Quality-forcing (critique-then-improve)
What it is: Have the model produce a first draft, then critique it, then produce a final version. Usually as one prompt with three sections.
Why it works: The model catches its own errors on the critique pass. Especially good for writing tasks.
Cost: Roughly 2× the tokens. Roughly 2× the latency. Sometimes worth it.
Eval:
Task: 30 client-facing emails responding to complaints.
A) Direct draft.
B) Draft → self-critique → revise (single prompt with three sections).
Measure with a rubric via LLM-as-judge:
- Clarity, empathy, specific-remedy, professionalism
Typical result: B moderately better. Bigger effect on longer outputs, smaller effect on short structured outputs where structured output enforcement is already active.
When to skip: Short structured outputs, classification tasks, or when latency budget is tight.
7. Adversarial self-check
What it is: After a draft, ask the model: "What's the strongest objection to this response? What would a skeptical reviewer flag?"
Why it works: Explicitly framing the check as adversarial pulls the model out of confirmation-bias mode.
Eval:
Task: 50 diligence findings that should have a confidence score.
A) Direct finding + confidence.
B) Finding → adversarial critique → revised finding + confidence.
Compare confidence calibration: for the 20% of findings you flagged as "high confidence,"
what percent are actually right when you check against ground truth?
Typical result: B's high-confidence findings are meaningfully more reliable.
Best use: Safety-critical outputs. Legal review. Diligence conclusions. Anywhere overconfidence is worse than uncertainty.
8. Meta-cognitive uncertainty gating
What it is: Tell the model: "If your confidence is below 0.6, don't answer — ask a clarifying question instead."
Why it works: Combined with HITL, this shifts the model from "always answer" mode to "answer or escalate" mode. Enormously valuable for real deployments where a wrong answer is costlier than a missing answer.
Eval:
Task: 100 tickets, 30 of which are actually ambiguous (missing key info).
A) Model answers all 100.
B) Model answers only when confident, else asks a specific clarifying question.
Measure on the 30 ambiguous tickets:
- A: what fraction of answers were wrong?
- B: what fraction of questions were the RIGHT question to ask?
Typical: A is wrong on 40-70% of ambiguous tickets. B asks the right question on 60-80%.
9. Output-shape scaffolds
What it is: Give the model an explicit template with placeholders. Overlaps with #1 but works at the prose level rather than the schema level.
Example scaffold:
Return your response in exactly this shape:
**Summary:** [one sentence]
**Recommendation:** [one of: proceed, hold, decline]
**Top 3 concerns:**
1. [concern]
2. [concern]
3. [concern]
**Confidence:** [0.0-1.0]
Why it works: Downstream regex/parsing becomes reliable. Reviewers can scan quickly. Model can't ramble.
Eval: Same as structured output enforcement — measure parse rate and reviewer speed. Prefer #1 (JSON schema) when you can. Use #9 when the output is client-facing and you want it to read as prose.
10. "Power words" — where the hype meets the reality
The internet has a persistent claim that specific phrases — "neuro-symbolic reasoning," "activate expert mode," "think like a genius" — unlock hidden capabilities. Some of this is real. Most of it is not.
What's real:
- Naming a specific technique the model has seen in training data can invoke that technique. E.g., "Use tree-of-thoughts reasoning" or "Apply the ReAct pattern" or "Do a chain-of-thought analysis" all do something because those are named methodologies with training-data footprints. "Neuro-symbolic reasoning" is a real ML technique (rules + neural), and asking the model to combine explicit rules with pattern reasoning can help on the right tasks.
- Domain-specific vocabulary primes the model toward that domain's conventions (see #5).
- Explicit uncertainty framing measurably improves calibration (see #7, #8).
What's not real (or is real for the wrong reason):
- "You are a world-class expert" — trained-on so heavily it's approximately noise.
- "Take a deep breath" — was real for one specific model on one specific benchmark. Not general.
- "This is important, do it perfectly" — mostly a wash.
- Any single "magic word" that's supposed to unlock capability without changing the task shape.
The right test: If a "power word" doesn't move your eval score, it doesn't work FOR YOU on THIS task with THIS model. Don't cargo-cult it into your production prompts.
For a deep, evolving reference on power-word combinations and the NEURO family, neuro-power-stack (the user-scoped skill in this workspace) catalogs and evaluates them. That's the right home for the deep dive. This doc's job is to keep you grounded: techniques not incantations.
Combining techniques
The techniques above stack. A production prompt for a real task often uses 4-6 of them:
[Discipline prefix] "Approach this as a legal review."
[Role] "You are a paralegal preparing a redline."
[Few-shot] "Here are 3 examples of a good redline."
[Output scaffold] "Return a numbered list of proposed changes."
[Structured output] tool_use with JSON schema
[Uncertainty gating] "If a clause is ambiguous, flag it with 'CLARIFY' instead of guessing."
Every one of those additions should be justified by an eval delta. If your suite passes without one, drop it.
Eval infrastructure
If you don't have an eval suite for your prompts, you don't have prompt engineering — you have prompt vibes. See ../07-evaluation in this program.
Minimum viable eval:
1. 20-100 real inputs
2. A ground-truth answer or a rubric
3. A script that runs the prompt against all inputs
4. A score (accuracy, rubric mean, calibration, whatever the task cares about)
With that in place, every prompting technique above becomes a hypothesis you can test in 5-10 minutes. Without it, you're a person on the internet trading power words.
Reflexion is an improvement loop: generate an answer, inspect failures, produce targeted feedback, then revise. It is useful when a response can be checked against explicit evidence or criteria. It is weak when the “reflection” is just another model saying its own writing is good.
For lease extraction, the first pass produces fields and source quotes. A verifier checks schema, quote containment, page accuracy, normalization, and conflict with addenda. It returns a repair list such as “late fee quote does not state amount” or “notice period is from base lease but addendum supersedes it.” The repair pass receives only the evidence and the defects. It may correct, abstain, or create a review task.
Limit reflexion to one repair cycle unless a deterministic condition calls for another source retrieval. Repeated self-revision adds cost and can turn an abstention into confident fiction. Keep generator and critic prompts separate; ideally use distinct models or at least contexts. Do not expose hidden chain-of-thought to staff. Store concise, inspectable failure tags instead.
Measure first-pass accuracy, repair uplift, false-repair rate, extra latency, and cost. A critic that improves style but harms citation precision is not useful. Reflexion earns its place when it fixes observable defects more often than it introduces new ones.
Tree of Thoughts: Use Search, Not Decorative Reasoning
Tree of Thoughts (ToT) asks a model to generate several candidate approaches, score them, and continue from the strongest branch. It can help when there are genuinely different plans: decide which diligence documents could resolve a discrepancy, or construct a constrained sequence of investigation steps. It is not useful for deterministic extraction, tenant routing, or simple RAG answers.
Implement it as bounded search. Define a state object, allowed next steps, scoring rubric, branch factor, depth, and stop rule. For a data-room conflict, a state might contain documents checked, unresolved claims, candidate evidence, and next retrieval query. Generate at most three branches, score against evidence coverage and cost, then execute the best one. Persist the state and trace.
Never let ToT manufacture facts by treating a plausible branch as evidence. Branches are hypotheses or plans; tool output is evidence. Score candidate plans with deterministic constraints first: document accessible, authorized, not already checked, likely to resolve the checklist row. Use a model only for relevance among valid choices.
Evaluate whether ToT beats a single plan on completeness, error rate, latency, and cost. If its benefits appear only in contrived examples, remove it. A builder should prefer a fixed checklist or retrieval query when that solves the problem. ToT is a search algorithm with token costs, not a badge of sophistication.