The fastest way to make an internal AI assistant unsafe is also the fastest way to make its demonstration impressive: connect it to a broad document estate and ask it to answer anything. The demonstration may produce fluent answers within hours. The organisation still cannot tell which material was approved, which user was entitled to see it, whether the answer is supported or who is responsible when it is wrong.

A useful assistant is a controlled knowledge service, not a general chat window. It serves a named group, completes a bounded information task and answers from sources with owners, versions and permissions. This is as relevant to a mid-market manufacturer or professional-services group as it is to a smaller business, even though the indexed URL of this article retains the original “SME” wording.

Start with a decision about work

Choose one recurring moment where a person needs to find, compare or apply approved information. Examples include locating the current operating procedure, preparing a draft answer from controlled product material, identifying the owner of an exception or summarising a defined case file for an authorised colleague. The output should be easy for the user to inspect before it changes a record or affects another person.

“Answer questions about the company” is not a use case. It hides the audience, consequence, source boundary and definition of a good answer. A better statement is: “Help the service team find the current approved procedure for one request category, show the supporting passages and route unresolved cases to the process owner.”

The Government Digital Service: Artificial Intelligence Playbook for the UK Government advises teams to begin with the problem, choose the right tool and retain meaningful human control at the right stages. That discipline prevents an assistant becoming an expensive answer to a problem that search, clearer ownership or conventional workflow automation would solve better.

Grounding begins with an approved knowledge boundary

Retrieval-augmented generation can search selected material, provide relevant passages to a model and ask it to compose an answer. It can improve currency and traceability, but retrieval does not make the source true. If a repository contains obsolete policies, duplicated files and contradictory instructions, the assistant can return the wrong document very efficiently.

Prepare the knowledge domain before tuning prompts:

Canonical source: identify where the authoritative version lives and remove convenience copies from the retrieval set.
Owner and status: record who approves the material, whether it is draft or effective, and when it must be reviewed.
Structure: preserve headings, tables and document relationships so a retrieved passage retains its meaning.
Scope: include only the material required for the first workflow, not every drive, mailbox and collaboration space.
Conflict: resolve competing instructions or expose the conflict to the user; do not let the model choose silently.

Minimum metadata usually includes title, canonical location, owner, version, effective date, review date, business domain and access group. The quality of this layer often matters more than minor differences between foundation models.

Make access inheritance non-negotiable

An assistant should not create a new path around existing permissions. It should authenticate the user, carry their role into retrieval and return only passages they could access through the source system. Search indexes, caches, logs and evaluation datasets all need the same scrutiny; a secure source can still leak through an over-broad index or an unnecessarily detailed log.

The Information Commissioner’s Office: security and data minimisation in AI requires organisations processing personal data to assess security in the context of the particular processing and to limit data to what is necessary. “The model needs context” is not a lawful purpose and is not a reason to ingest a complete employee or customer record.

Separate three kinds of authority:

Read: which sources and records may this user query?
Draft: which outputs may the assistant prepare without changing a system of record?
Act: which actions, if any, may be executed after explicit confirmation by an authorised person?

A first release should normally remain read-only. Tool access, write-back and external communication add different failure modes: prompt injection, inappropriate action, duplicate execution and a much larger recovery problem. Treat each tool as its own permission boundary rather than granting the assistant a broad service account.

Design the answer for inspection

A strong interface does not merely display prose. It shows the source title, relevant passage, version or effective date and a direct route to the canonical document. It distinguishes a supported answer from a model inference and says when the evidence is incomplete. Users need a visible escalation path, not a confident guess wrapped in a disclaimer.

Instruct the assistant to stay within retrieved evidence, but do not rely on the instruction as the control. Validate source identifiers, constrain tools, filter unauthorised material and keep consequential action behind a policy gate. The NIST: Generative AI Profile identifies risks specific to generative systems and sets out actions that organisations can adapt to their own context. It is a useful prompt to test the whole service, not only the model response.

Build an evaluation set before the pilot

Collect representative questions from intended users while the workflow is being mapped. Include straightforward retrieval, ambiguous wording, stale terminology, conflicting documents, missing information, unauthorised requests and questions that should be refused or escalated. Preserve the expected source and acceptable response characteristics for each case.

Measure the components separately:

Retrieval: did the service find the correct, current and authorised passage?
Support: is every material statement grounded in the displayed evidence?
Task quality: can the user complete the defined job accurately and with less avoidable searching?
Boundary behaviour: does the assistant refuse, qualify or escalate when evidence or permission is absent?
Operations: how much review, correction and exception handling does the service create?

The NIST AI Resource Center: AI RMF Core says evaluation should use documented methods, conditions similar to deployment and continuing measurement in production. Its Measure function also recognises uncertainty and the limits of generalising beyond tested conditions. A single aggregate accuracy score cannot show whether permission failures or unsupported answers are hiding inside an acceptable average.

Set release thresholds before reviewing results. Otherwise a persuasive demonstration can cause the team to reinterpret weak evidence as success. Keep a fixed regression set and a changing set drawn from real, sanitised usage. Rerun both after material changes to the model, prompts, retrieval logic, source corpus or access design.

Treat governance as part of the product

The NIST: AI Risk Management Framework organises work through Govern, Map, Measure and Manage. For an internal assistant, Govern establishes ownership and appetite; Map defines the users, purpose and likely harms; Measure tests behaviour; Manage determines whether to proceed, narrow or stop. The sequence is more useful than a checklist completed after launch.

Where personal data is involved, the Information Commissioner’s Office: accountability and governance implications of AI makes clear that senior management cannot delegate accountability to technical teams. A data protection impact assessment may be required and should remain live as the purpose, data, users or risks change.

Name an operational owner for the assistant and separate owners for source quality, access approval, technical service, evaluation and incidents where appropriate. Define who may approve a new source, change a prompt, add a tool, accept a failed threshold or suspend the service. ISO: ISO/IEC 42001 AI management systems provides a broader management-system structure for defined responsibility, maintained controls and continual improvement. An organisation need not pursue certification to use that discipline.

Release one narrow service, then earn expansion

A credible first release serves one team, one knowledge domain and one read-only task for a limited period. It uses existing identity, preserves source permissions, cites evidence, has a versioned evaluation set and gives users a clear route to challenge an answer. Monitor unsupported claims, failed retrieval, access incidents, unresolved questions, user corrections, latency, cost and the effort required to keep sources current.

Expansion should follow evidence. A second domain needs its own owners and tests. Write access needs transaction controls, confirmation and recovery. External communication needs a review of audience, records and regulatory consequence. The fact that the first workflow behaved well does not prove that the next one will.

The right answer is to stop when authoritative sources cannot be identified, permissions cannot be preserved, representative testing is impossible, residual data-protection risk remains unacceptably high or nobody will own the service after the pilot. An assistant cannot repair an ownerless policy estate or turn contradictory records into organisational truth. In those conditions, fix the knowledge and workflow first.

Further reading: Government Digital Service: Artificial Intelligence Playbook for the UK Government; NIST: AI Risk Management Framework; NIST: Generative AI Profile; Information Commissioner’s Office: guidance on AI and data protection; ISO: ISO/IEC 42001 AI management systems.

See how a controlled AI Sprint can prove one internal assistant workflow.