Keeping Your AI Chatbot's Knowledge Base Fresh: Webhook-Driven Sync Without Downtime

The stale knowledge problem: chatbot trained on last month's docs answers with outdated info

A retrieval-augmented chatbot is only as current as the index it searches. Last month’s product docs, pricing page, or API reference can silently become the source of truth. A user asks, “What is the Pro plan price?” and the bot returns $10/user/month because the index still holds the February version. The March page says $12. The answer looks plausible, costs money, and erodes trust. The fix is not retraining a model; it is continuously syncing the knowledge base and invalidating the parts that changed.

Push vs pull sync strategies (webhook vs polling) — when each makes sense

There are two ways to know the source changed: ask repeatedly or be told.

Webhook (push) is the better default when your content source supports reliable event delivery. The source calls your ingestion service on every create, update, or delete. Latency is low, you skip no-op checks, and you do not burn API quota polling unchanged pages. The trade-offs are delivery retries, duplicate events, out-of-order events, and transient failures when your service is down.

Polling (pull) makes sense when the source has no webhooks, webhooks are unreliable, or you want to batch and deduplicate changes yourself. A cron job reads the source API every N minutes, compares a stored cursor, and imports diffs. The cost is lag and wasted calls. A common hybrid is: use the webhook as the trigger, but let a poller run every few hours to catch anything the webhook missed.

For a production chatbot, prefer webhook-driven sync with a robust fallback poller.

Implementing incremental updates: diff detection, chunk invalidation, re-embedding only changed sections

A full re-embedding of every document on every change does not scale. The pipeline should treat the vector store like a cache, not a snapshot.

1. Diff detection. Store a source_version for each document: a timestamp, etag, or content hash. When a webhook arrives, fetch the current document and compare its hash with the stored one. If unchanged, return early.
2. Chunk-level invalidation. Split each document into chunks with deterministic IDs such as doc_id:chunk_index:content_hash. When a section changes, identify which chunks changed, which moved, and which disappeared.
3. Re-embed only changed sections. Delete chunks whose hash is no longer present and upsert new ones. Unchanged chunks stay in the vector store untouched.
4. Idempotent upserts. Because chunk IDs include the content hash, the same payload processed twice produces the same database row. This eliminates duplicate vectors from duplicate webhooks.

Use chunk sizes around 512 tokens with 64–128 token overlap for dense text, and larger windows for sparse pages. For embedding, models such as OpenAI text-embedding-3-small or text-embedding-3-large work well; choose large when semantic precision is critical and small when cost and throughput matter more. Batch changed chunks before calling the embeddings endpoint.

Handling sync failures gracefully: idempotent upserts, dead-letter queue, admin alerts

Webhooks fail. The source may send a malformed payload, your vector store may throttle, or a page may be temporarily unreachable. Design the pipeline to fail visibly but recover cleanly.

• Idempotent upserts with deterministic IDs prevent half-applied updates from corrupting the index.
• Retry with backoff for transient errors such as rate limits or connection timeouts. Use a queue so retries do not block new events.
• Dead-letter queue (DLQ) for permanent failures: invalid JSON, unparseable documents, or missing required metadata. Store the raw payload, error reason, and timestamp.
• Per-document checkpoints record the last successfully synced version so a retry resumes from the right place.
• Admin alerts when DLQ depth grows or a document has not synced in longer than a configured threshold. Connect the alert to Slack, PagerDuty, or your existing on-call channel.
• Ordering guard for rapid edits: keep a last_modified field and reject updates older than the currently stored version, even if the webhook arrives late.

Worked example: Notion webhook → embedding pipeline → live chatbot, specific latency numbers

Consider a support chatbot whose knowledge base lives in Notion. The flow looks like this:

1. A writer edits the “Pricing” page. Notion sends a webhook POST to your ingest service.
2. The service fetches the page via the Notion API (GET /v1/pages/{page_id} and GET /v1/blocks/{block_id}/children), converts blocks to plain text, and computes a SHA-256 hash per top-level block.
3. It compares block hashes against the previous version stored in a relational table. Two paragraphs changed; the rest are identical.
4. The changed blocks are chunked into 512-token windows with 64-token overlap. New chunks get deterministic IDs: notion_page_id:chunk_index:sha256(content).
5. A single batched request sends the new chunks to OpenAI’s embeddings endpoint using text-embedding-3-small. A batch of 20–50 chunks typically returns in 150–400 ms in production deployments.
6. The service upserts vectors into pgvector (or Pinecone, Weaviate, Qdrant) by chunk ID using INSERT ... ON CONFLICT. Stale chunks from the old version are deleted.
7. The chatbot query pipeline retrieves the new chunks within seconds.

Representative latency budget

These timings are estimates from typical production deployments, not guarantees:

Step	Latency
Webhook delivery	1–3 s
Fetch page from Notion API	200–500 ms
Diff + chunking	50–150 ms
Embedding (batch)	150–400 ms
Vector upsert/delete	50–100 ms
Total end-to-end	~2–5 s

If the page is edited twice within seconds, the per-document ordering guard ensures the later edit wins, and idempotent chunk IDs prevent duplicates.

Testing sync: how to verify the chatbot answers with updated content

Writing the pipeline is half the work; proving it works is the other half.

• Golden-answer tests. After a sync, query the chatbot with a question whose answer changed. Assert that the response contains the new value, not the old one. Store expected phrases in version control.
• Retrieval-level tests. Check that the top-k retrieved chunks include the updated chunk IDs. This isolates sync failures from LLM hallucinations.
• Synthetic document CI. Maintain a dedicated test page. A CI job updates it, waits for the webhook and pipeline, then queries the chatbot. If the answer does not match the new content within a timeout, the build fails.
• Shadow comparisons. Run staging and production indexes side by side. For a sample query set, compare answers before and after a sync. Flag regressions where the new answer loses accuracy.
• Observability. Log every sync with document ID, version, chunks changed, and chunk IDs used at query time. When a user reports a wrong answer, trace exactly which index entries contributed.

Production readiness checklist

1. Store a content hash or source timestamp for every document to detect actual changes.
2. Use deterministic chunk IDs so upserts are idempotent across duplicate webhooks.
3. Re-embed only changed chunks; never re-index the whole corpus on a single edit.
4. Implement retry with backoff and a dead-letter queue for unrecoverable failures.
5. Record per-document sync checkpoints so retries resume from the correct version.
6. Add a guard against out-of-order edits using a last_modified timestamp.
7. Set alerts on DLQ depth and documents that have not synced within your SLA.
8. Test sync with golden-answer queries and a synthetic document in CI.
9. Log chunk IDs at retrieval time to make wrong answers debuggable.

Simple Agent handles webhook ingestion, diff-aware chunking, idempotent upserts, and failure recovery automatically.