Build and maintain an AI agent knowledge base
The short answer
A support knowledge base is a managed copy of your approved sources. Track who owns each source, who may see it, which version is current and whether it synced. Update only what changed, remove deleted or restricted content, and be able to prove an answer used the current permitted version.
Create a source register
Before you import anything, list every source the agent might use: help center articles, website pages, Drive folders, uploaded files, approved custom answers and relevant support macros. For each one, record an owner, its audience, a review date and its priority when sources disagree. That last field is what resolves the Northstar case: if the help center outranks old PDFs, the agent knows which return window to trust, and the PDF’s owner gets a review task. Review imported macros before treating them as customer-facing facts, since they often contain internal instructions or exceptions that no longer apply.
Classify each source as public, restricted, approved internal instruction or unknown. Public content can be shared with anyone. Restricted facts, such as account-specific details or partner pricing, are only for people allowed to see them. Internal instructions tell the agent how to behave but should never be quoted to a customer. Anything with unknown visibility stays restricted until an owner decides. Just because a connector can read a file doesn’t mean every customer may see what’s in it, so store the permission information with the indexed content and apply it before retrieval, not after.
Parse meaning before splitting text
Parsing is the step that turns a page or file into clean text. Keep the structure that carries meaning: headings, lists, tables, product names and links. A table row separated from its column heading might just say “14”, with no clue whether that means days, dollars or items. Strip out noise such as navigation menus and footers, because a footer repeated on every page can crowd out the content that actually answers a question.
The parsed text is then split into chunks, the smaller passages that search returns to the agent. Choose chunk boundaries that keep a whole policy or procedure step together, so the agent never sees the rule without its exception. Then test real customer questions against the chunks and look at what comes back. No single chunk size works for every document. Record the parser and splitting version, so you can compare the results of a change and roll it back if answers get worse.
Attach the source ID, version, title, location and audience to every chunk. Those fields let your team explain where an answer came from, remove an outdated source cleanly and see what a permission change affects. If extraction produces empty or garbled text, mark the source as failed. A successful download doesn’t mean the source synced.
Treat each source change differently
Not every change needs the same work. An edited article needs new content in the index. A change to who can see an article needs new permissions even if no text changed. A deletion needs the old content gone everywhere it was copied, including caches. The table lists the common cases, and the last column says how to confirm each one actually worked.
| Change | Required action | Proof |
|---|---|---|
| New document | Parse, classify, index and mark ready | An intended question retrieves the new evidence |
| Content edit | Compare a content fingerprint and replace changed material | The old claim no longer appears for affected questions |
| Metadata-only edit | Update metadata without unnecessary embedding work | Title or audience changes are visible |
| Deletion or revocation | Remove or block chunks and invalidate affected caches | A known old fact cannot be retrieved |
| Failed sync | Keep status and freshness visible; apply the stale-content policy | An operator can identify which sources are unsafe to use |
Make sync an observable job
Detect changes using whatever the source supports: cursors, update timestamps, content hashes (a short fingerprint that changes whenever the text changes) or change events. Handle pagination properly and save the last checkpoint that completed. A page fetch that fails must not look like an empty folder, or the next sync may delete everything in it. Track each source through separate states, discovered, downloaded, parsed, indexed and ready, so you can see exactly where it stopped.
Decide in advance what happens when a sync only partly completes. One approach is to stage the new version and switch to it only after it passes validation. Another is to replace records in a carefully controlled order. Either way, make sure the agent never sees the old and new versions of a policy side by side. Monitor how long jobs wait in the queue and how fresh each source is, not just the total count of indexed files, because a large count can hide a source that hasn’t updated in weeks.
Control reindexing cost and migrations
Search over a knowledge base usually relies on embeddings: lists of numbers that represent the meaning of each chunk, stored in a vector database so similar questions find similar passages. Creating embeddings costs money and time, so embed only the content that changed rather than the whole collection on every run. Keep the embedding settings separate from the storage. Switching to a different embedding model usually means building a new index, because vectors from different models aren’t comparable.
Before switching to a replacement index, run the same set of test questions, permission filters and deletion cases against it and compare the results with the current one. Plan how you’ll switch over and how you’ll roll back, without losing the source register. A managed connector can save integration work, but you still own the rules for freshness, audience and conflicting sources.
Ask AI to explain this guide
Build decisions and checks
Use these decisions to turn the worked case into your own implementation. A check is complete only when you have evidence for the stated result.
Choose trusted sources and owners
Required. Your website, a Drive folder and an internal SOP (standard operating procedure) can give different answers to the same question.
Build. List each source with its business owner, audience, authority, language and review date. Decide which policy wins when sources conflict. Leave out drafts and obsolete material. Inventory help center, website, Drive and internal content separately, and classify each source as public, restricted, approved internal instruction or unknown. Treat unknown visibility as restricted until an owner resolves it.
Check. A conflicting policy creates a review item or a safe response, never an arbitrary choice.
Parse and split content without losing meaning
Required. A table row or an exception can lose its meaning once it's split away from its heading.
Build. Keep headings, tables, lists, links and source IDs intact. Strip out navigation noise. Test chunk boundaries and overlap (how content is split up for search) against real questions. Flag extraction failures, and use OCR only where you need it.
Check. Check for exact answers from tables, long policies, PDFs and pages with repeated navigation.
Sync additions, edits, deletions and permissions
Required. An index can stay out of date even when the sync job reports success.
Build. Track source IDs plus update markers or content hashes. Retry in batches that can resume. Detect content that has been deleted or had its access revoked. Update changed material and the chunks built from it. Confirm indexing has finished before you make a new version live.
Check. Edit and delete a source, then search for it. New facts appear and removed facts disappear within your freshness target. A source that is deleted or becomes restricted can't linger in old chunks or caches.
Choose storage and plan migrations
When it applies. Vector dimensions, metadata and embedding versions must match, and a migration can quietly mix incompatible indexes.
Build. Choose hosted or self-managed storage based on filtering, backups, deletion, latency and operating effort. PostgreSQL with pgvector and Qdrant are both options. Keep a content manifest and record the embedding version. Test a new index before you switch traffic to it.
Check. Restore an index from backup and compare retrieval results. Test filtered search and deletion.
Ask AI to apply these decisions
Templates and examples
Every file uses synthetic data. Adapt it to your own policies and test it somewhere safe before it touches customers.
- source-register.csvSpreadsheet worksheet (CSV)
- knowledge-change-test.mdTemplate (Markdown or text)
Keep it working
Launch is the start of the work, not the end. These are the recurring jobs this part of the agent needs, and who usually owns them.
| Cadence | Owner | Work |
|---|---|---|
| Weekly | Knowledge owner | Owners review stale and conflicting sources. |
| Weekly | Engineering | Spot-check newly ingested files and parsing failures. |
| Daily | Engineering | Monitor sync lag, failed files, expiring source credentials and indexing spend. |
| Monthly | Engineering | Review capacity and backup recovery. Re-evaluate retrieval whenever you change embeddings. |
Test your decision
A source was public yesterday and is restricted today. The text hasn’t changed. Can the sync skip it?
Show the expected decision
No. A permission change is a real change. Update the access controls and clear affected caches. Then test that an anonymous visitor can’t retrieve the restricted fact or get a paraphrase of it in an answer.
