Collecting User Feedback for AI Applications
Feedback can and should be collected throughout the user journey, and users should have a way to report errors whenever the need arises.
The Timing Question Behind Feedback
Suppose you are building an AI application and you already know that user feedback can improve it. The next question is practical: at what moment should you ask for that feedback? Ask too early, and people who have not yet seen the product work may be confused by the question. Ask too late, and you may lose the chance to fix a problem that a user noticed and then quietly abandoned. The answer depends on a principle that applies at every stage of the product, which this article works through step by step.
What do you think happens?
Before reading on: should a feedback option appear only at one convenient checkpoint chosen by the design team, or should it stay available whenever a user needs it?
Reveal answer
Whenever the user needs it, throughout the product
The source states that feedback is not a single event at one point in the product. It can be collected throughout the user journey, and the option should exist whenever the need arises, not only at a checkpoint the designer chose in advance.
Stages of a User Journey
A useful way to reason about feedback is to think of the user journey as a sequence of stages. The earliest stage is sign-up, when a person meets the application for the first time. After that comes calibration, where the application may need information from the user before it works well for them. Then comes ongoing use, which lasts for as long as the person keeps using the product. Feedback can be relevant at each of these stages.
The arrow from ongoing use to an error report is not a single step that happens once. A user might move between ongoing use and error reporting many times. That is why the feedback option has to remain available for the life of the product rather than disappearing after setup.
The need for feedback is driven by the user's situation, not by the schedule of the development team. Errors and surprises tend to appear when people use a product in their own circumstances. A feedback channel that exists only during a special survey window misses many of the moments that matter most.
Why the Feedback Option Must Stay Out of the Way
Once you accept that feedback should be available throughout the journey, a second constraint appears. The feedback option must be nonintrusive, meaning it should not interfere with the user workflow. The workflow is the sequence of things a person is trying to accomplish with the product. If a feedback prompt interrupts that sequence, the person has to stop, respond, and then try to recover their place. That is a cost the user pays for the sake of the developers.
These two requirements pull in slightly different directions. The first says feedback should be easy to give whenever it is needed. The second says offering feedback must not get in the way of the task the user came to do. A well-designed option satisfies both by staying reachable without demanding attention, so the user can respond when they choose to rather than being pulled away from their work.
Calibration at Sign-Up
The first stage of the journey is where feedback can do a particular kind of work. When a user has just signed up, feedback can help calibrate the application for that user. Calibration means adjusting the application so that it fits the person it is serving. Instead of guessing how a new user will behave or what they will need, the application asks for a small amount of information and uses it to set itself up.
Several everyday examples show the idea. A face recognition app first has to scan the user's face before it can work at all. A voice assistant might ask the user to read a sentence aloud so that it can recognize their voice for wake words, which are the words that activate the assistant. A language learning app might ask a few questions to estimate the user's skill level so that its lessons start at a sensible point. In each case, the calibration step is what makes later behavior possible or useful.
Following a Calibration Choice
Deciding Whether a Language App Calibration Is Optional
A language learning app asks new users a few questions to estimate their skill level so that lessons start at a sensible point. Should this calibration be required or optional, and what happens to a user who skips it?
- 1Ask whether the app can work without calibration
The app can still present lessons without a skill estimate, even if the first lessons may not start at the best point. So the app does not depend on calibration to function at all.
- 2Classify the calibration
Because the app can work without it, calibration falls under the optional case. The app should offer it without forcing it.
- 3Trace the declining path
A user who declines is given a neutral option rather than being blocked. The application keeps that neutral behavior until it can refine itself over time.
- 4Trace the accepting path
A user who answers the questions gets lessons that start at an estimated skill level, so the application is tailored from the first session.
Calibration is optional in this app. It is offered without being forced, a user who declines receives a neutral starting point, and the app refines its behavior over time.
Mistakes in Feedback Timing
Common mistake Treating feedback as a single event at one checkpoint
What it looks likeA feedback survey appears only during a special window after launch, and a user who hits an error in week three has nowhere to report it.
Why it failsThe need for feedback is driven by the user's situation. Errors appear when people use the product in their own circumstances, so a single window misses many important moments.
Do this insteadKeep the feedback option available throughout the user journey, for the life of the product.
Common mistake Making the feedback prompt block the user's task
What it looks likeA modal dialog opens in the middle of a multi-step task and must be dismissed before the user can continue.
Why it failsThe user has to stop, respond, and then recover their place. That cost is paid for the developers' benefit, not the user's.
Do this insteadKeep the option reachable without demanding attention, so the user responds when they choose to.
Common mistake Forcing optional calibration
What it looks likeA language app blocks access to lessons until the user answers every skill question.
Why it failsCalibration is required only when the application cannot work without it. Forcing an optional step removes the user's choice.
Do this insteadOffer calibration without forcing it, and fall back to a neutral option if the user declines.
Common mistake Assuming calibration is always required
What it looks likeA designer makes every sign-up step mandatory because calibration seems useful.
Why it failsThe source limits required calibration to cases where the application cannot work without it. Other calibration can be optional.
Checking Your Reasoning on Calibration
What do you think happens?
A voice assistant asks the user to read a sentence aloud so it can recognize their voice for wake words. Should this calibration step be required or optional?
Reveal answer
Required, because the assistant cannot work without it
Recognizing the user's voice for wake words is what the assistant depends on to activate. Since the assistant cannot work without that calibration, the step falls under the required case.
A note-taking app wants to ask new users three questions about how they plan to use it, so that the home screen can be arranged for them. The app works fine with a default layout if the user skips the questions. Decide whether the calibration should be required or optional, describe what the user sees on each path, and explain where an error-report option should remain available.
Hints
- Start by asking whether the app can work without the calibration.
- Trace both the accepting path and the declining path to the end state.
- Remember that the feedback option should stay reachable after sign-up, not only during setup.
Key Takeaways
- Feedback can and should be collected throughout the user journey, from sign-up through ongoing use, and users should be able to report errors whenever the need arises.
- The feedback option must be nonintrusive so that it does not interfere with the user workflow; a reachable option lets the user respond when they choose rather than being pulled away from their task.
- Calibration at sign-up tailors the application to a new user by gathering a small amount of information and using it to set the application up.
- Calibration is required only when the application cannot work without it; otherwise it should be offered without being forced, with a neutral fallback until the application refines its behavior over time.