Build a Test Matrix for Your Website Agent’s Answers
Build a Test Matrix for Your Website Agent’s Answers
A website agent can sound confident while missing a key detail, citing the wrong page, or answering a question your business has not prepared it to handle. A test matrix gives you a practical way to check AI agent answers before visitors rely on them. It records what you asked, what the answer should include, which source supports it, and what you need to fix if the response falls short.
You do not need a huge test set to get started. Focus on the topics your agent is meant to handle, use questions real visitors ask, and review answers against the information you have actually provided.
1. Choose a narrow set of topics
Start by writing down the topics the agent is responsible for. A focused scope makes website chatbot quality checks easier: you can judge whether an answer is useful, and identify questions the agent should hand off or decline.
For example, a small online shop might include:
- Shipping options and delivery areas
- Returns and exchanges
- Product sizing and care
- How to contact support
Keep the list tied to pages or documents you can provide as source material. If you do not have a reliable source for a topic, decide whether to write one, keep that topic out of scope, or direct visitors to a human.
Also list subjects the agent should not answer. These may include account-specific questions it cannot verify, exceptions that require staff approval, or advice outside your business’s expertise. Clear boundaries help you distinguish an answer issue from an unsupported question during review.
2. Turn visitor questions into a test matrix
Use questions from support emails, contact forms, sales calls, and site searches. Keep the wording close to what a visitor might type. Include short questions, detailed questions, and common variations.
For each question, record:
| Test question | Expected facts | Source page | Boundary or notes |
|---|---|---|---|
| “Can I return a sale item?” | State the sale-item rule and any time limit | Returns policy | Do not promise exceptions |
| “How long does delivery take to Bristol?” | Give the relevant estimate and conditions | Shipping page | Clarify whether this is an estimate |
| “Will this fit a 14-inch laptop?” | Answer only if the product dimensions support it | Product page | Ask which model if needed |
| “Can you update my order address?” | Explain the available support route | Contact or order-help page | The agent cannot claim to change an order unless that action is available |
Expected facts should be specific enough to assess. “Answer correctly” is too vague. Write down the detail that matters, such as a return window, a geographic limit, or a dimension. Note important conditions too: an answer may be technically true but misleading if it leaves out an exception.
Track the source page you expect to support each fact. That makes source citation review concrete: you can check whether a citation points to material that actually backs the relevant claim.
3. Check accuracy, useful detail, and citations
Run each question through the agent configured with the sources you want to evaluate. Then compare the response with your expected facts and source pages. To check AI agent answers consistently, use the same questions after each source revision.
Review each response for three things:
- Accuracy: Are the stated facts supported by the source? Did the answer add a price, promise, deadline, or exception that the source does not establish?
- Useful detail: Does it address the visitor’s question directly? Is a key condition missing? Would a reasonable visitor know what to do next?
- Citations: When retrieved source material is available, does each citation support the nearby claim? Open the cited page and verify the relevant passage rather than judging by the page title alone.
A citation is useful only when it helps a reader check the answer. A link to a broad page may be less helpful than a citation to the specific section that states the return period or delivery condition. If the agent gives no citation, first check whether it retrieved source material for that response. Do not assume every answer will have a citation when no supporting source was retrieved.
Keep notes brief and observable. For example: “Correct return window; omitted that personalized items are excluded; citation points to the returns section.” This is easier to act on than a general score like “weak.”
4. Label failures so you know what to fix
A consistent label turns a failed test into a next step. Use one primary category for each problem:
- Missing source: The needed fact is absent from the material available to the agent.
- Unclear wording: The source contains the fact, but its wording, structure, or conditions make it hard to interpret.
- Unsupported question: The question falls outside the agent’s intended scope or requires information it cannot access.
- Answer issue: The source is adequate, but the response is inaccurate, incomplete, confusing, or poorly cited.
These labels prevent you from rewriting a good source when the actual issue is an out-of-scope question. They also help you spot patterns. Several failures about delivery regions may point to a missing source detail; repeated confusion over an exception may mean the policy needs clearer wording.
Choose a fix that matches the label. Add verified facts to the source when information is missing. Rewrite ambiguous passages and state conditions plainly when wording is unclear. For unsupported questions, make the boundary or support route clear. For an answer issue, retest the same question and related variations after checking the available source content.
Do not fill gaps by guessing. If you cannot verify a policy or product fact, get it confirmed by the person responsible before adding it to the source material.
5. Retest revised sources before embedding
After updating source content, rerun the questions that failed. Include a few nearby variations so you can see whether the change helps beyond one exact phrasing. Keep the original expected facts and source pages alongside the new results.
To review your own answers, use your configured agent with the sources you intend it to use. Simple Agent provides chunk-level source citations when retrieved source material is available, which you can inspect as part of the review. See the Simple Agent plans.
When your checks look right, review the setup documentation for the next steps in preparing the agent for your site. If a question still fails, keep it out of scope or revise the source again before embedding the widget.
Start by testing Simple Agent, then prepare a short matrix from real visitor questions and verify the answers against your configured sources before adding the widget to your website.
Ready to build your AI agent?
Guided setup in 90 seconds. One flat price, published monthly volume.
Create my agent