RAG Pipeline: Lease Knowledge That Can Prove Its Work
Retrieval-augmented generation (RAG) is the system behind answers grounded in your documents. It is not “put PDFs in a vector database.” A usable Belle Realty lease RAG system must ingest documents correctly, preserve versions and page citations, retrieve the right chunks, reject weak evidence, and show the answer’s support.
Start with leases because they are high-value, repetitive, and version-sensitive. A tenant or operator question should retrieve the current signed lease and governing addenda for that tenant—not a semantically similar lease from another property. Metadata filters and authorization are as important as embeddings.
The pipeline has six stages: ingest, normalize, chunk, embed, retrieve, rerank, then answer with citations. Store source version and extraction status at every stage. If you cannot reproduce an answer from the document version that existed at the time, you do not have an operational RAG system.
Use the build guide first. Contextual retrieval and hybrid search are upgrades after you establish a baseline and a retrieval eval set. Retrieval quality, not model eloquence, determines whether this is useful.
Before adding complexity, run this design through a small representative eval and inspect the trace with the operator who will own failures. Make the boundary, escalation, and rollback visible in the product. That discipline will expose more useful work than another round of prompt cleverness.
Create a document record with tenant, property, lease, document type, version, effective date, checksum, access scope, and extraction status. Extract page text; retain source PDF and page offsets. For scanned files, store OCR confidence per page. Do not embed a lease until its identity and scope are known. Addenda must link to the base lease and carry a precedence field.
Chunk with legal structure
Chunk by section heading, numbered clause, page, and paragraph—not fixed token windows alone. Aim for 250–500 tokens with 40–80 token overlap only where a clause crosses a boundary. Each chunk stores document_id, lease_id, page range, heading path, effective date, and text. A clause such as “Notice” should remain intact. A generic vector match for the word “notice” is not enough without the correct lease filter.
Retrieve, rerank, answer
Filter first by organization and caller authorization, then tenant/lease when the question is contextual. Retrieve 20 candidates with vector plus keyword search. Rerank the top candidates using a cross-encoder or small judge tuned for query–passage relevance, then pass the best 3–6 to the answer model. The answer contract: cite every factual claim, quote when stakes are high, state document version, and say “I can’t find that” when evidence is weak.
Guardrails and evaluation
Require a minimum retrieval score and a source from an active, signed document before answering policy questions. Never silently use a superseded lease. Build a test set of real questions with expected source sections, including misleading and unanswerable questions. Measure recall@k for the gold chunk, citation correctness, grounded answer rate, abstention correctness, latency, and cost. Diagnose retrieval before changing prompts. Most RAG failures are bad metadata, chunks, or filters—not a lack of clever instructions.
Contextual Retrieval: Make Chunks Understandable Alone
A chunk often loses the facts that make it meaningful. “Tenant shall provide notice within 30 days” is useless if the retrieved text omits the property, lease version, clause heading, and whether it concerns renewal or move-out. Contextual retrieval solves that by attaching compact, document-specific context before embedding and storing it with the chunk.
For Belle Realty, generate or deterministically build a context prefix such as: “Belle Realty | Oak Ridge Unit 204 | Lease executed 2026-02-01 | Addendum A | Section 18: Renewal and Non-Renewal.” Embed prefix + chunk; display the original chunk plus human-readable source metadata. This makes semantic retrieval distinguish a renewal notice from a maintenance notice without filling every answer prompt with repeated boilerplate.
Use deterministic metadata wherever possible. An LLM-generated context can enrich messy documents, but validate it and keep it separate from the source text. Do not let generated context introduce legal facts or replace citations. Record the context-generation version so you can reindex safely.
Context helps recall, not authorization. Still filter by organization, active lease, and caller scope before similarity search. It also does not fix overly broad chunks or poor OCR. Test it with queries that are ambiguous without headers: “How much notice?”, “Are pets allowed?”, “Who handles the filter?” Compare gold-chunk recall before and after.
The goal is not longer chunks. It is chunks that retain enough identity and purpose to be retrieved for the right reason.
Vector search understands paraphrases. Keyword search is excellent at exact tokens: unit numbers, clause names, “Section 14,” dollar amounts, vendor IDs, and unusual legal terms. Belle Realty needs both. A tenant asking “Can I have a dog?” should find a pet clause even if the lease says “domestic animals.” An operator asking for “$75 late fee” should find the exact amount.
Run lexical and vector retrieval over the same authorized corpus. Normalize each result score independently, then combine them with a weighted method such as reciprocal rank fusion. Start with equal influence; tune only against an eval set. Apply metadata filters before both searches, not after. If a request concerns a specific lease, a perfect semantic match in another tenant’s lease is still an invalid result.
Rerank the merged top 20 with a relevance model that sees query, passage, and selected metadata. Preserve the originating search signals for debugging: lexical rank, vector rank, final rank, filter set, and document version. This tells you whether failures come from vocabulary, embedding, or reranking.
Use hybrid search when exact identifiers and natural-language questions coexist. Do not use it as an excuse to dump 50 chunks into a model context. Retrieval should be ruthless: return a small, diverse, cited evidence set. Evaluate gold-chunk recall@5 and recall@10, plus answer groundedness. If keyword search consistently wins for clause lookup, keep it. Architecture should follow corpus behavior, not fashion.