Start with a release decision, then design the test
The useful question is specific: can this system handle this task for this group of users, with these permissions, under these operating conditions? “The AI works” is too broad to accept or reject. A pilot should narrow uncertainty around a release decision, such as letting a support team draft replies or allowing an assistant to prepare orders for approval.
The checklist below is Kooya’s recommended operating approach, not a certification standard or a report of client results. NIST describes its AI Risk Management Framework as voluntary and applicable across design, development, use, and evaluation. That supports treating evaluation as an ongoing practice rather than a single launch demonstration.[1]
- Name one workflow owner who can accept or reject the release.
- Write the in-scope users, channels, data sources, and permitted actions.
- State what the pilot will not do and who handles those requests.
- Record the current manual process so the comparison includes review and correction work.
Build a test set from the work you want to change
Choose examples with the operators who receive the work. Include routine requests, incomplete inputs, ambiguous instructions, exceptions, and situations where the correct response is to ask for clarification or hand over. Use synthetic or appropriately approved and minimized records; an evaluation set should not become an uncontrolled copy of customer data.
For a Philippine customer operation, test the languages and phrasing actually present in its channels. That may include English, Filipino, and mixed-language messages, informal spelling, abbreviated addresses, or a customer changing details halfway through a conversation. Do not assume that passing English examples proves performance in every language or locality.
Separate the cases used to improve the system from a held-out set used to judge readiness. Keep those acceptance examples stable for a comparison, then add newly discovered failures as regression cases. Repeat high-impact cases to check whether the same input produces inconsistent actions.
- For each case, record the input, user role, available evidence, and expected outcome.
- List allowed actions and actions that must never occur.
- Include an expected escalation destination and the information it needs.
- Tag cases by language, channel, task, and consequence of failure.
Check the answer and the completed action independently
Microsoft’s agent evaluation guidance distinguishes end-to-end results from the steps used to reach them. It includes checks for task completion, tool selection, tool inputs, and tool-call success. These are separate questions: a system can choose the right tool, pass the wrong customer identifier, and still produce a reassuring final message.[2]
An order assistant that says “updated” has not passed until the expected record has changed and no other record has changed. Test this in a sandbox with controlled fixtures. Inspect the stored result, the audit event, and any notification or downstream task, rather than taking the assistant’s wording as proof.
For a knowledge assistant, inspect whether the retrieved evidence supports the answer and whether it answers the actual request. Microsoft documents groundedness and relevance as different evaluation dimensions. A well-supported answer can still miss the question; a useful-sounding answer can still introduce unsupported facts.[3]
Use one concrete acceptance case to expose hidden gaps
Consider a fictional order-support pilot. A customer writes, “Pa-change po delivery to Friday, same address,” but the test account has two open orders. The accepted behavior is to identify the ambiguity, ask which order they mean, and make no change yet. If the assistant silently picks an order, the case fails even if the eventual confirmation is polite and grammatical.
Continue the same case: after the customer identifies the order, the assistant checks eligibility, presents the proposed change, obtains the required approval, and calls the update tool. Simulate a timeout after the service records the change. The assistant must reconcile the outcome before retrying, so a network problem does not create a duplicate action.
This is an illustrative test design, not evidence about any existing Kooya deployment. Adapt the conditions and permitted actions to your own workflow. Include the final handoff too: the person receiving an exception should see the relevant record, the user’s request, attempted steps, and what remains unresolved.
Set separate gates for critical failures and ordinary quality
Avoid reducing the pilot to one headline accuracy number. Decide in advance which observed failures block release: unauthorized access, action on the wrong record, bypassed approval, or loss of the ability to stop the workflow are examples to assess with the accountable owner. A strong average must not hide one of these events.
For lower-consequence quality issues, agree on measures that fit the task: correct completion, unsupported claims, clarification quality, reviewer edits, and time to a usable outcome. Report results by important segments as well as overall. A small language segment can disappear inside a large average.
Passing a finite set of tests is evidence about those tests, not proof of zero future failures. Record sample coverage, known gaps, and uncertainty. Choose thresholds before the acceptance run and document why the remaining risk is acceptable for the proposed release scope.
- Critical controls: an observed breach blocks expansion until investigated, fixed, and retested.
- Task quality: compare correct outcomes with a written rubric and human-reviewed examples.
- Operational usefulness: count correction effort and escalation load alongside apparent time savings.
- Service behavior: test unavailable tools, missing evidence, timeouts, and safe recovery.
Count the cost of the whole workflow
A pilot can look inexpensive when the estimate counts only model usage. Track the complete path: retrieval, tool calls, retries, review time, exception handling, and operational support. Compare cost per usable completed task with the current process, using observed pilot figures instead of a promised percentage saving.
Measure elapsed time from request to a useful outcome. A fast first response can conceal a long approval queue. If the pilot moves work from frontline staff to an already overloaded supervisor, that tradeoff belongs in the release decision.
Ask the supplier to preserve the configuration used for each evaluation: model version, prompts, knowledge snapshot, tool definitions, and application version. Without that record, a later quality change is harder to reproduce or explain.
The build-or-buy guide uses the same complete-cost view to compare purchased services with custom engineering.
Make the release a controlled operating change
Begin with the smallest scope that the evidence supports. A draft-only release may be appropriate before allowing actions, while a supervised action flow may be appropriate before broader autonomy. The sequence depends on the consequences and reversibility of the task, not on a universal maturity ladder.
Name the person who watches failures, the person who can disable actions, and the team that receives exceptions. Define the events that trigger a pause or rollback. Confirm that the manual path still works when the AI service is unavailable.
Rerun the relevant acceptance cases when a model, prompt, document collection, policy, or integration changes. Retain failures found in operation and review whether the test set still reflects the users now being served. Evaluation earns its place when it changes a release decision or catches a regression.
The acceptance pack to request before signing off
Ask for a small, inspectable set of evidence rather than a presentation full of selected successes. A buyer should be able to trace an important claim from the acceptance criterion to the test input, observed behavior, reviewer judgment, and final release decision.
The Kooya view: the deliverable is a workflow your team can operate with justified confidence. A successful pilot makes its limits visible, demonstrates a useful outcome, and leaves people able to intervene. If those conditions are missing, keep the scope narrow and resolve the gap before expanding.
- A versioned test set, scoring rubric, and results split by the agreed task and language segments.
- Evidence for permissions, approvals, record changes, failures, and human handoffs.
- Observed operating cost and completion time, including review and correction.
- Named ownership, monitoring, stop controls, rollback steps, and known limitations.
- A signed decision covering the exact release scope and the conditions for reassessment.