An AI-generated response appears in seconds. The demonstration looks impressive. Then a reviewer spends several minutes checking facts, removing unsupported statements, and finding the source material the draft should have used.
The system may still be useful. The demonstration simply has not established where the value is.
Before introducing AI into a workflow, we would measure how the task is completed today and decide what an improvement would mean across the whole process.
Include the work people do around the output
Consider an internal team preparing responses to supplier questions. Drafting is one step. Staff also locate information, check its relevance, obtain approval, and record what was sent.
A useful baseline follows a representative set of requests through those steps. Capture elapsed time, active effort, common corrections, and situations that require escalation. Avoid treating every request as equivalent; a routine question and an unusual contractual question may need different handling.
This is not a demand for a perfect measurement programme before experimenting. A small, explicit baseline is more informative than comparing a polished demo with an informal impression that the current process is slow.
Define the improvement and its limits
The desired result might be less time locating source information, fewer omissions in routine responses, or faster preparation of a reviewable draft. State which of those you intend to test.
Then define what would make the experiment unacceptable. A shorter preparation time does not justify exposing information to the wrong user or creating answers reviewers cannot reliably assess. NIST's AI Risk Management Framework provides a broader framework for considering AI risk in context; the product team still has to translate its concerns into the specific workflow.
For this example, that could mean keeping the experiment to a defined set of internal sources and requiring approval before any response is sent. The limits should be reflected in the implementation and the evaluation, not just in an introductory slide.
Compare complete attempts at the task
Use representative requests and retain difficult examples. Count the effort to review and correct the result. Record cases where the system should ask for clarification or leave the task to a person.
Look for where the assistance helps and where it adds work. A useful conclusion may be that retrieval is valuable while drafting contributes little, or that one narrow request category benefits while another should remain unchanged.
That result is more actionable than a general claim that the organisation needs AI. It identifies a product improvement with a boundary and a reason to invest.
The first question is not which model to buy. It is which part of the customer's or employee's task you intend to improve, and how you will know whether the improvement survived contact with the rest of the workflow.