TBW / DATA / RETRIEVAL
Data Integration and RAG
Engineer ingestion, parsing, permissions, retrieval, freshness and evaluation pipelines for enterprise knowledge systems.
Engineering the Knowledge Pipeline
This capability focuses on source-to-index operations and retrieval controls, not the user-facing assistant experience covered by RAG. Every answer depends on current, authorized and interpretable source material.
From Source to Evaluated Retrieval
Connect → Parse → Clean → Chunk → Enrich → Embed → Index → Retrieve → Evaluate. Connect under a source owner; parse formats into structured text; clean artifacts and duplicates; chunk along meaningful boundaries; enrich with provenance and permissions; embed where useful; index searchable fields; retrieve only permitted candidates; evaluate against representative questions. Keep source lineage through every transformation.
Connector Assessment Checklist
Potential sources include SharePoint, Confluence, file shares, databases, ticketing, email archives and APIs. For each, document the source owner, permission semantics, change detection, deletion behavior, rate limits, document version and available metadata. A connector that imports text but misses permission revocation is not production-ready for sensitive knowledge.
Connector assessment template — copy these fields for each source; record the evidence and owner rather than assuming connector support.
- Source and owner: system, corpus, accountable owner and point of contact.
- Permission model: user/group/tenant inheritance, query-time enforcement and negative test.
- Change detection: event, polling or snapshot method and observed lag.
- Deletion and revocation: propagation path, reconciliation check and failure alert.
- Rate limits: allowed throughput, retry/backoff and recovery behavior.
- Version: source identifier, modified time and retained prior version.
- Metadata: classification, lineage, content status and fields missing from the source.
Difficult Content Needs Format-Specific Tests
PDF reading order, scans and OCR errors can distort meaning. Tables need headers tied to cells; slides and diagrams may require descriptions and validation against originals. Code needs repository and version context; nested documents need parent-child lineage. Preserve the original reference so a reviewer can inspect ambiguous extraction before indexing.
Chunking Is a Tested Design Choice
Choose among recursive, semantic, section-aware, page-aware, table-aware and parent/child strategies based on document structure, semantics, query types, measured retrieval quality and model behavior. No single chunk size is correct for all corpora. Test whether a retrieved unit contains enough context to support a claim without mixing unrelated access scopes.
Metadata and Access-Control Inheritance
Each indexed unit may need source, owner, version, modified date, classification, tenant, allowed users/groups/roles and content status. Propagate ACL changes into the index and recheck authorization at query time; never rely on the model to discard an unauthorized passage. Test inherited permissions and cross-tenant boundaries with negative cases.
Embeddings, Index and Ranking
Evaluate embedding models on domain queries. Hybrid keyword and vector retrieval can capture exact identifiers alongside semantic matches; apply permission filters before ranking and rerank permitted candidates where useful. Index selection depends on workload, filter complexity, index type, query volume, latency, availability, operations, scale and the existing platform—not a universal database-size rule.
Freshness and Revocation
Use content hashes and change events for incremental indexing; track version metadata so re-embedding occurs when text or embedding models change. Propagate deletions and revoked access promptly according to source policy. Record checkpoints and support rollback to a known index version after a bad ingest. Monitor lag between source changes and searchable state.
Do Not Embed a Moving Transaction
Live inventory, ticket state, prices and transactional records often belong behind an authorized direct query, not in a stale vector snapshot. Use retrieval for stable explanatory material and fetch current facts from the system of record with appropriate permission checks and failure handling.
Four Independent Evaluation Lenses
Retrieval quality
For representative questions, inspect relevant source coverage, ranking and citation provenance.
Generation quality
Check whether responses faithfully use retrieved evidence and admit missing support.
Authorization
Probe forbidden documents, tenant separation and revoked access.
Operations
Track ingestion errors, source lag, index health, deletion propagation and rollback readiness.
Illustrative evaluation questions: “Which version of the policy applies to this request?” tests relevant-source retrieval and citation; “What does the source not establish?” tests grounded abstention; “Can a newly revoked user still retrieve it?” tests authorization; and “What happens if an updated source fails indexing?” tests operational recovery. Use representative, access-controlled cases from the organization rather than treating these examples as benchmark results.
Repeat evaluations after connector, parsing, chunking, embedding or access-policy changes.
Implementation and Deliverables
Inventory sources, prototype extraction and permissions with a representative sample, compare retrieval configurations, then stage ingestion and monitor freshness. Deliver a connector map, ingestion design, metadata and ACL models, retrieval configuration, evaluation set, freshness plan and runbook.
Related: RAG overview · Internal Knowledge Assistants · Model Selection
Frequently Asked Questions
Why not index every available document?
Ownership, permissions, extractability and freshness differ by source. Assess each connector and corpus before exposing its content to retrieval.
What is the right chunk size?
There is no universal size. Compare strategies against actual document structures and representative queries, including whether context survives splitting.
How is revoked access removed?
Propagate permission and deletion events to the index, recheck query-time access and test the interval until revoked material is no longer retrievable.
When should data be queried directly?
Use a scoped live query for changing transactional values such as inventory, prices or ticket status instead of relying on an old embedding.
How do you tell retrieval errors from answer errors?
Evaluate whether the right evidence was retrieved separately from whether the generated response used that evidence faithfully.
Assess a Source Corpus
Identify a knowledge source, its owner and access rules. We can examine extraction, permission propagation and a representative retrieval test set.