How to Audit Website Agent Answers Against Their Sources
How to Audit Website Agent Answers Against Their Sources
A website agent can sound confident and still misread a policy, overlook a key condition, or answer from a page that no longer reflects your business. Before embedding it widely or giving it more responsibility, review its answers against the material it uses.
A practical website agent answer audit helps you see where answers are supported, where citations are missing, and where your content needs attention. It also helps you decide when the agent should say it does not have enough information.
1. Build a review set from real visitor questions
Start with questions people actually ask. Useful sources include support conversations, sales calls, contact forms, search queries, and questions your team hears repeatedly. Remove personal or sensitive details before using examples.
Keep the first set small enough to review carefully. Include a mix of questions:
- Clear-answer questions: The site directly explains a price, process, eligibility rule, service area, or product detail.
- Conditional questions: The answer depends on a date, location, customer type, plan, or other qualification.
- Unanswerable questions: The site does not contain the answer, or the answer depends on information the agent cannot access.
- Ambiguous questions: A visitor could mean more than one thing, or the wording leaves out a detail needed to answer reliably.
For each question, note what a dependable answer should include and which page or pages contain the relevant information. For unanswerable questions, write down what the agent should do instead, such as ask a clarifying question or explain that the site does not provide that information.
This review set gives you a consistent baseline. Without one, it is easy to remember only the answers that looked good and miss the cases that matter to visitors.
2. Compare each response with the source
Run each question through the agent configured with your site’s content. Review the response claim by claim. A single paragraph may contain several factual statements, and each one may need different source material.
For every claim, ask:
- Does the source page state this directly?
- Does the answer preserve important conditions, limits, dates, and exceptions?
- Did the agent combine details from multiple pages correctly?
- Is the information current, and does another page contradict it?
- If retrieved source material is available, does the answer cite the relevant source?
This is AI agent source verification: checking whether the response follows the source material available to the configured agent. A citation is useful only if it points to a source that supports the claim. Open the cited page and confirm the wording; do not treat the presence of a link as proof by itself.
When available, chunk-level citations can make the review more precise by connecting a response to a specific passage. If the agent has no relevant retrieved source material, a citation cannot make an unsupported answer reliable. Record that gap rather than inferring a source.
To review your own sources, use the agent configured with those sources. See the Simple Agent plans.
3. Keep findings in separate categories
Separate categories make it easier to choose the right fix. A missing citation is different from an unsupported claim, and both differ from a source that conflicts with another source.
Use a simple review log with the question, answer, claim, relevant page, finding, and next action. Sort findings into categories such as:
- Unsupported claims: The response states a fact that no relevant source supports, or extends beyond what the source says.
- Missing citations: Relevant retrieved source material was available, but the answer did not cite it where citations are supported.
- Conflicting pages: Two or more pages give different answers, such as different prices, requirements, dates, or service conditions.
- Unanswered questions: The agent does not answer, gives an incomplete answer, or should have declined because the source material does not contain the needed information.
You may also record correct answers. These provide examples of what the agent handles well and help you tell whether a later source change improved one area while affecting another.
Do not merge all issues into a general “bad answer” label. The category points toward the likely remedy: correct the claim, improve source retrieval or citation behavior, reconcile pages, or clarify the agent’s scope.
4. Revise the content or narrow the scope
When an answer is unreliable, fix the source material or change what the agent is expected to answer. First determine whether the needed information exists and whether it is written clearly enough for a visitor to find.
If the answer is missing from the site, add it to the right page or decide that the agent should not answer that question. If two pages disagree, establish the current policy and update the outdated page. If the source is correct but easy to misread, make the condition or exception explicit near the claim it qualifies.
For questions that require private account data, current inventory, professional judgment, or other information the configured agent cannot access, narrow the scope. The agent can explain what the site does cover and direct visitors to an appropriate next step, if that information is available. Do not ask it to fill gaps with plausible-sounding guesses.
After a revision, rerun the affected questions and nearby questions that could rely on the same source. Confirm both that the corrected answer is supported and that the agent handles the relevant boundary case. For example, if you clarify a location rule, check questions about locations that qualify and those that do not.
For Simple Agent, predictable plan allowances make it possible to plan around the included usage for a chosen plan. Review the pricing details when deciding how you want to run ongoing checks.
5. Embed the widget and repeat the audit
Once the configured agent’s answers are ready for a live check, add its script-tag widget to your site following the setup instructions in the Simple Agent documentation. Then try the review questions on the live site. This checks the visitor-facing experience as well as the answer content: the widget loads, the response is readable, citations are usable when present, and the agent handles questions it cannot answer appropriately.
Keep the first live review focused on your chosen questions. Record what you tested and any difference between the configured-agent review and the live-site response. If a question behaves differently, investigate that specific case before expanding the test set.
Repeat the audit whenever you make a meaningful change to source pages, policies, products, or the agent’s scope. Revisit conflicting pages after updates, and keep questions that previously exposed unsupported claims or missing citations in the review set. This turns a one-time website agent quality check into a practical content review process.
For an audit of your own material, configure the agent with your sources, then run your review questions against that agent before and after embedding the widget. See the Simple Agent plans.
Ready to build your AI agent?
Guided setup in 90 seconds. One flat price, published monthly volume.
Create my agent