Test the experience, not just the code
Test one complete customer task, record where people need help, and decide what to fix before launch.
The idea
A passing build does not prove a useful product. Test functional behavior, usability, accessibility, data boundaries, and recovery from failure. Prioritize by how much harm a failure would cause.
Put it into practice
Walk through the core journey on phone and desktop. Try empty inputs, long content, keyboard navigation, slow connections, and failed requests. If there are accounts, test that one user cannot access another user’s data.
A summary tool needs to handle blank text, repeated submissions, an unavailable AI service, and a mistaken result. The user must be able to correct the output and keep their work.
Use AI thoughtfully
Ask AI for a risk-based test matrix. Convert the highest-risk cases into checks you can actually run. Treat generated test coverage as a proposal, not evidence of correctness.
Your checkpoint
Can the user complete the core job and recover safely from the most likely failures?
A prototype test plan you can use before launch
Test one complete customer task, define what success would look like before the session, and record what the person actually does. A clickable demo can reveal confusing instructions; it cannot establish production reliability, security, willingness to pay, or demand. Keep those questions separate.
By Ashley Kays · Published · Updated
Download the editable test plan — a free Markdown worksheet. No signup required.
1. Write the decision, not a wish list
Start with a sentence such as: “Before we invite clients, we need to know whether a designer can turn scattered feedback into an accurate revision list without our explanation.” Specify the prototype version, the starting screen and the information available. If the question is whether the problem matters at all, return to the validation lesson first. Usability success is not evidence that anyone needs the product.
Choose a small first learning round that you can observe and review carefully. Do not treat its size as a statistical guarantee. Include people who plausibly do this work, including relevant access needs, rather than only colleagues who already know the interface.
2. Give the person a goal, not directions through the UI
For the fictional client-feedback tool used throughout this course, prepare invented client notes: “Make the logo smaller,” “Keep the original headline,” and “Please use the blue version.” Do not upload confidential client files to an unfinished prototype.
“Your client has sent these comments in three places. Prepare the revision list you would use for your next design pass. Before you finish, make sure it reflects the client’s requests and save a copy you could share.”
Do not say “click Summarize, edit the second bullet, then press Export.” That tests whether someone can follow your instructions, not whether the experience is understandable. For this example, the observer checks whether the saved list contains the three requests accurately. Agree on that criterion before watching anyone.
3. Observe the attempt and label assistance
Explain the prototype’s limits and that you are evaluating the design, not the participant. Agree on any recording beforehand; written notes may be enough. Give one task at a time. If someone gets stuck, ask what they expected to happen before showing the answer. If you intervene, record where and why. A completed task after coaching is an assisted completion, not an independent success.
Use separate fields for observation, interpretation and next question. “Opened Export before reviewing the list” is an observation. “The review step may be unclear” is an interpretation. “Would a review reminder help without blocking experienced users?” is a question for the next version.
4. Test recovery without creating real harm
After the main task, use a controlled test environment to check one failure state. For this tool, simulate an unavailable summarization service. Does the screen explain the failure? Is the original input preserved? Can the person retry or leave with a copy? Do not deliberately interrupt a real customer’s work or send real messages or payments as a test.
Make a separate engineering checklist for keyboard access, narrow screens, empty inputs, duplicate submissions and data boundaries. A prototype session does not replace accessibility evaluation, security review or automated checks. If the product handles accounts or sensitive data, obtain the appropriate technical review before widening access.
5. Decide what to fix, retest or stop
Write a stopping rule before the round. For this illustrative tool: any observed loss of input, accidental disclosure or incorrect export that cannot be corrected blocks a wider pilot until repaired and checked. A confusing label might instead lead to one design change and a repeat of the same task with a new participant. This is a proposed rule for this example, not an industry benchmark.
Record the number of attempts and separate independent completion, assisted completion and non-completion. Include exclusions and why a session ended. Do not turn a handful of attempts into a market-wide success rate. Preserve contrary evidence: someone who prefers their current process may be telling you the product solves the wrong problem.
Worked decision example: keep the finding smaller than the claim
Hypothetical only: suppose a participant completes the list, but overlooks that an AI-generated item contradicts the original note. The finding is not “the product works” or “AI summaries are unreliable.” It is that this version allowed an unreviewed error through this task. A next change could place the source note beside each generated item and require an explicit review before export. Repeat the task and inspect whether correction becomes possible without coaching. These are proposed actions, not reported results.
Your one-page evidence record
- Decision and prototype version.
- Participant’s relevant work context, without unnecessary identifiers.
- Task prompt and prewritten completion criterion.
- Observed actions, exact error and any help given.
- Outcome: independent, assisted, not completed or not tested.
- Interpretation, alternative explanation and unresolved risk.
- One change, a retest task and the condition for proceeding.
Use the core-flow design lesson to revise the confusing step, or the free resource library to connect this test to your product brief. Keep the original notes: AI can help organize anonymized observations, but must not fill missing results or invent participant quotes.
Method reference: GOV.UK Service Manual: Using moderated usability testing (published February 21, 2017; updated October 3, 2017; checked October 24, 2026). It recommends observing realistic tasks with neutral instructions and actual or likely users. The feedback-tool scenario, worksheet and proposed decision rules above are original teaching examples, not government standards or completed research.
Bring a small prototype and one question
The Wealth Experiment’s free December 7 and 14, 2026 founding cohort provides time to build and refine a small first version. A request does not reserve a seat; Ashley confirms places separately. Software and hosting costs are separate.
Review the current workshop details and request a place ↗Write your answer to the checkpoint. Notes stay in this browser.