AI internal knowledge assistants: the questions a demo cannot answer

A teammate asks which version of the onboarding checklist applies to a new customer. The answer exists somewhere, but the shared drive contains three copies, an old proposal and a meeting note that contradicts them. An AI internal knowledge assistant could make that search easier. It could also produce a confident answer from the wrong document.

For a small US business, the useful buying question is whether the assistant can return an answer that the right person can verify. A search box that writes fluent paragraphs is only the visible part of that system. Document ownership, permissions, version control and exception handling determine whether people can rely on it during everyday work.

It does not claim a particular vendor will pass these tests without configuration.

Define one recurring question family

Start with a narrow collection of questions: how an account manager prepares a handover, where a project coordinator finds the current delivery checklist, or which internal form starts a purchase request. Avoid launching with the promise that the assistant knows everything about the business.

Collect actual questions from the people who perform that work. Ask what they do after receiving an answer. A link to a procedure may be enough for one task; another may require an effective date, an exception rule and the person who can authorize a departure.

Use those observations to define a finished answer. For example, an onboarding response should identify the current procedure, explain the relevant step and provide the source section. If two approved documents disagree, it should surface that conflict rather than silently choose one.

Treat access as a separate requirement

An employee may be allowed to read the onboarding handbook but not a customer's contract. Both documents might contain the same customer name. Retrieval must respect the difference before either document becomes answer material.

Microsoft's Azure AI Search security-filter documentation describes a pattern that includes or excludes results using principal identifiers. It explicitly distinguishes that filtering pattern from authentication and authorization. That distinction matters when evaluating a custom assistant: a field containing a user ID does not, by itself, establish the user's identity.

Test the same question with two accounts that have different document access. A good answer for one account can be a data exposure for the other.

Also test what happens after access is revoked. Ask how quickly permission changes reach the search index and whether previously generated copies remain accessible. Have the implementation team document the actual behavior rather than assume the source system handles every downstream copy.

Build a source register before indexing

Create a small register of the material the pilot can use. It should identify the owner, current version, audience and review date. This is an operating document, not a one-time migration checklist.

A review date does not prove that a document is correct. It gives someone a visible reason to inspect it. If every file has the same automatically generated date, the register is unlikely to help resolve an actual disagreement.

Make citations useful at the moment of work

A citation should let the reader check the claim without another search. Prefer a document title, a section or page reference, and a working link the reader is authorized to open. Show when the source was retrieved if freshness affects the decision.

Check that the cited passage supports the specific answer. A link to a long policy document can look reassuring while containing no evidence for the recommendation. Include questions where the source mentions a related concept but does not establish the requested fact.

Our business RAG guide explains the retrieval layer in more detail. The practical test remains straightforward: can a colleague follow the evidence and reach the same conclusion?

Use an acceptance set with difficult cases

Prepare a test set before tuning the system. Include common questions, ambiguous requests, missing answers, outdated documents and access restrictions. Keep a separate set of questions for the final check so the team does not optimize only for the examples used during development.

For each question, record the expected evidence and acceptable behavior. Some answers should be a request for clarification. Others should be a refusal to reveal restricted information. These outcomes can be useful even though they do not resemble the polished answer shown in a demo.

Measure how long a user takes to finish the underlying task, including source checking. Record unsupported statements and permission failures separately from writing quality. A readable answer with a serious access failure should not disappear inside an average satisfaction score.

Assign the maintenance work

Someone must remove obsolete sources, investigate failed answers and handle permission changes. Name that person and define how a teammate reports a problem. Preserve enough diagnostic information to reproduce the issue without unnecessarily copying sensitive content into logs.

Add a visible fallback: open the source library, contact the process owner or submit a question for review. When the assistant cannot answer, users should still know where to go.

For a pilot, review failures regularly and group them by cause. Missing content requires a different fix from poor retrieval. A wrong permission filter requires a different response from an unclear document title. Treating every failure as a prompt problem wastes time.

Decide whether the pilot earns a wider rollout

Expand only after the original question family works reliably for the intended users. Add a second source collection with its own owner and access tests. Re-run the earlier checks so the expansion does not weaken the original boundaries.

VUZURAN builds custom AI tools and connected business workflows. If your team needs an assistant across several systems, a project discussion can begin with the source register, access model and acceptance set. A native search feature may be sufficient when the work stays inside one well-managed platform.