Agents and RAG are downstream of data operations. If Belle Realty leases are duplicated, incorrectly scoped, stale, or missing page text, no model choice will produce trustworthy answers. The data pipeline owns ingestion, normalization, identity, versioning, enrichment, indexing, retries, and freshness.
Treat every source file and record as an event with a stable ID, checksum, source timestamp, owner, access scope, and processing state. Make work idempotent: a retry should not create duplicate lease chunks, embeddings, or diligence rows. Separate the raw source, normalized representation, derived artifacts, and user-facing answer index. This makes reprocessing safe when OCR, chunking, embeddings, or policies change.
Use queues for slow or bursty work, dead-letter failures that need inspection, and explicit backoff for provider limits. Reindexing must be selective and versioned. A new chunker should not force you to lose the prior index before the replacement is validated.
The guides describe proven ingestion patterns, reindex strategies, and the actual tradeoff between fresh information and compute spend. Your pipeline should make “what changed and what answered from it?” easy to answer.
Before adding complexity, run this design through a small representative eval and inspect the trace with the operator who will own failures. Make the boundary, escalation, and rollback visible in the product. That discipline will expose more useful work than another round of prompt cleverness.
Freshness is not universally good. A signed lease should be indexed immediately because the system may answer contractual questions from it. A static property brochure can refresh weekly. A live maintenance status may be queried from the operational database at answer time rather than embedded at all. Choose a freshness objective per data class.
Define freshness as source-change-to-serving latency, then set targets: access revocations near real time; lease uploads within minutes; financial statements after validation; broad reference content nightly. Measure actual lag and use it in answers when relevant: “Lease index updated at…” is better than pretending data is current.
Avoid embedding volatile facts that can be fetched safely from the source of truth. An embedding index is a discovery layer, not a transaction system. For frequently edited material, use event-driven partial updates, debounce bursts, and TTLs for caches. Reserve full rebuilds for representation changes, not ordinary edits.
Balance cost using priority queues: high-risk or user-requested documents first, bulk backfills later. Coalesce repeated changes, skip unchanged checksums, cache embeddings by exact text and model version, and backpressure uploads when providers are constrained. Never defer deletion or permission revocation to save money.
Freshness decisions are product promises. Document them, expose them to operators, and test the ugly case: a lease addendum uploaded after a tenant asks a question.
Use an event-driven pipeline for documents and operational records. When a lease is uploaded, write a source record first, calculate a checksum, assign organization/property/lease scope, and enqueue processing. Workers extract text, run OCR if needed, classify document type, normalize metadata, create chunks, embed, and mark the index version ready. Each stage writes a status and idempotency key.
Separate raw, normalized, and derived storage. Raw is the immutable original PDF or export. Normalized is page text, tables, metadata, and source offsets. Derived is chunks, embeddings, summaries, clause proposals, and search indexes. Never overwrite raw evidence with model output. Derived artifacts name the source checksum and pipeline version that created them.
Use a queue with bounded concurrency, retryable versus permanent errors, and a dead-letter queue. Network timeout: retry. Password-protected PDF, unreadable scan, missing ownership metadata: mark for review. Do not continually retry semantic failures. Emit events for each transition so the application can show users “uploaded,” “processing,” “needs review,” or “ready.”
For database records, use change-data capture or application outbox events rather than periodic full scans when possible. Debounce rapid edits and coalesce events by entity. Make deletes and access revocations high priority: remove or filter derived content immediately, then clean physical indexes. Ingestion is a security system as much as it is an ETL system.
Reindex when a source changes or a derived representation becomes obsolete: new lease version, corrected OCR, different chunking, new embedding model, metadata bug, access-scope change, or retrieval schema update. Keep source version, chunker version, embedding model, and index version on every artifact so the reason is explicit.
Prefer selective reindexing. A changed lease should rebuild only that document and invalidate its previous active chunks. A revised chunker may require a corpus-wide rebuild, but run it into a new index namespace rather than mutating the live one. Validate retrieval metrics, access filters, cost, and latency, then switch a feature flag or alias. Keep the old index long enough for rollback and trace reproducibility.
Use a manifest to enumerate intended documents and completion state. Compare expected versus produced chunks, embeddings, and active source versions. Rebuild failures go to a queue; do not quietly omit them. Deletions and permission changes require immediate serving-layer enforcement even if physical vector deletion is asynchronous.
Test reindexing with known difficult leases: addenda, scans, duplicate templates, superseded documents, and revoked tenant access. Monitor index freshness lag, failed jobs, chunk count drift, retrieval recall, and storage cost. Reindexing is a deploy; treat it with the same change control as application migrations.