It’s 9 AM on a Monday when you get a Slack ping from your PM: “can we get feedback on the new onboarding flow by Friday?"

“By Friday?” you wonder, thinking about the forty interviews you need to see some real patterns. But you only have bandwidth for six.
So you slash it down to six interviews and pray they’re “representative”. Or you send a survey and get a barrage of “It’s fine”s that remain unsubstantiated. Or worse, the study never happens, and the team ships the feature on a hunch.
This is where AI-moderated interviews become the elves to your shoemaker: an AI runs the session, real participants show up when it suits them, and a study that usually takes three weeks of calendar tennis happens overnight.
It sounds incredible. It also sets a few alarm bells ringing in every researcher’s mind: is the data any good? Is the AI coming for my job? (Short answers: sometimes, and no.)
The long answer depends on one distinction that most vendor pages skip. But first, the basics…
What is an AI-moderated interview?
An AI-moderated interview is a live interview with a real participant, run by an AI with a synthetic voice.
Here’s how it works: you hand the AI a discussion guide. When the participant joins, it asks questions, listens to the answers, and decides whether to follow up or move on. Nobody from your team needs to be on the call.
Keep in mind that it’s not:
- A synthetic user: Synthetic users are AI personas answering instead of real people. Here, the participant is a real human, and only the moderator is AI.
- A survey with a voice: When someone says onboarding was confusing, a survey moves on to the next question. An AI moderator can ask, “What made it confusing?”
Most AI moderators combine three things: speech-to-text to hear the participant, a large language model (LLM) to decide what to say next, and text-to-speech to say it. That setup matters more than you think (and we’ll revisit this).
Structured vs. semi-structured interviews
If there’s one key takeaway from this post, it’s this: modern AI moderators run structured interviews, not semi-structured ones.
A structured interview sticks to the script. When an answer is short or vague, the interviewer probes. Then they move on.
A semi-structured interview is what most researchers mean by “discovery”: you have a guide, but go off-script when something interesting comes up. You skip a question that’s clearly irrelevant to the participant. And with eight minutes left on the clock, you decide whether to dig deeper or cover the last topic.
Nielsen Norman Group (NN/G) makes a point that’s easy to miss: AI moderators don’t do semi-structured interviews well, and this gap might not close soon. Language models perform best on tightly bounded tasks, like “if the participant mentions X, ask Y.” Give an AI moderator more freedom and it likely gets worse, not better. And you can't pre-load context for topics you didn't know would come up.
So you need to ask, “Is this study structured?” instead of “Is AI moderation good?”
What happened when NN/G put researchers in the hot seat
NN/G ran 10 research leaders and ResearchOps folks through AI-moderated interviews on two commercial tools, with the same guide. Here’s what they found:
What was good
People felt heard. The AI’s summaries were the most-praised part of the experience. Reflecting back what someone said made participants feel understood. One participant switched the whole interview to Spanish mid-session and the AI followed suit, without a hitch.
What was awkward
Only 3 out of 10 said the conversation felt natural. The reasons were specific:
- Turn-taking: While one tool interrupted people mid-thought, the other left long silences, sometimes forgetting to ask the next question.
- Summarize, then sprint: The AI summarized answers but never paused to let the participants confirm. A human moderator would check, “Did I get that right?”
- Over-the-top praise: Mundane answers got replies like “Fascinating”, “That’s so interesting”, and “Wow!”, which made participants roll their eyes.
- Nothing to look at: An animated orb doesn’t nod, and without facial cues, long questions were harder to follow.
- Abrupt starts: Sessions skipped the usual intro about recording and confidentiality. A couple of participants held back details, like where they work.
- No clock: The same script produced interviews from 13 to 56 minutes. Neither tool let researchers set a target length.
A caveat worth mentioning: these participants were expert interviewers, so they were more likely to spot the rough edges. Your customers might be more forgiving. But all the problems above are real, and most of them land on data quality.
Why AI interviews feel awkward (and why you shouldn’t blame your discussion guide)
Interruptions and dead air may sound like small UX gripes, but they decide whether a participant finishes their thought, and that decides what ends up in your transcript.
Remember the setup we covered earlier? Every voice AI moderator runs three steps in a loop:
- Speech-to-text turns what the participant says into text.
- A language model reads it and decides what to say next.
- Text-to-speech says it out loud.
There’s a fourth piece atop these three that does the hard part: deciding when the participant has finished talking. It has one signal to go on: silence.
But humans don’t work that way. In a real interview, you see someone look up while they think. You hear the breath before a second sentence, so you nod, say “mm-hmm”, and wait. The AI sees none of that, so it has to guess from the length of a pause. Every setting here is a trade-off:
- Too eager, and it jumps in when someone pauses to think. That’s the interrupting pattern noted by NN/G. So participants either rush, or give up on the second half of their answer.
- Too cautious, and it waits several seconds after every answer, then adds the time it takes to generate a reply. That’s the dead air, and participants fill it, wander off, or start multitasking.
Neither problem shows up in a demo with a fluent salesperson answering crisply. So when you evaluate a tool, test it the way real participants talk:
- Pause for three seconds mid-sentence. Does it cut you off?
- Say, “Hmm, let me think about that.” Does it wait?
- Talk over it while it’s asking a question. Does it stop and listen?
- Give a one-word answer. Does it probe, or move on?
Where does AI moderation earn its place?
The rule of thumb: use an AI moderator when you already know what to ask.
Where AI moderation falls short
- Early discovery in a space you don't understand yet. You can't configure probes for threads you can't predict.
- Deep domain expertise. If participants talk in jargon and acronyms, the AI won't know which answer deserves a follow-up.
- Studies where every participant takes a different path. Having the same structure for everyone means irrelevant questions for some.
- Watching someone work on their own machine. Show-and-tell needs a moderator who reacts to what they see.
- Sensitive topics and senior B2B conversations. Rapport still does the heavy lifting.
The teams getting this right choose both: AI interviews for breadth (spotting the patterns and open questions), and a researcher for depth (conversations where nuance matters).
What a hybrid study looks like (start to finish)
Note: The team and numbers below are illustrative.
The setup: A B2B SaaS team sees 40% of new accounts never invite a teammate. The PM wants to know why before planning the next sprint. The readout is in two weeks. The research team is two people.
First 2 days: Write the guide and pilot it
One researcher writes a structured guide: eight questions about the first week of use, with specific probes on the invite step. Three internal pilots catch an interruption problem and a question that runs long. Both get fixed.
Days 3–5: Run 60 AI-moderated interviews
The study link goes to new accounts by email and in-app. Sessions run around the clock, across three time zones. Nobody on the team moderates a single one.
Days 6–7: Analyze
The team codes all 60 transcripts into themes, where four emerge clearly:
- Theme 1: Admins want to finish setup before inviting anyone (most common).
- Theme 2: People didn't notice the invite button.
- Theme 3: Teammates already use a competing tool.
- Theme 4: A small group mentions their security team must approve new vendors first.
Themes 1 and 2 are clear enough to act on now. Theme 4 is the surprise, coming up in only six interviews, and the AI never asked for a follow-up, because the guide didn't instruct so.
Days 8–11: Conduct the six interviews with a human researcher
Recruit people from that group and run semi-structured interviews using questions like: who approves? What do they ask for? How long does it take? This is exactly the open-ended, judgment-heavy work AI moderators can't do.
Day 12: The readout
The PM gets two quick fixes backed by 60 voices, plus a deeper finding about procurement that changes the onboarding roadmap. Every theme links back to the quotes behind it.
The team ran 66 interviews in two weeks, and moderated six of them. The AI did not replace the researchers. It did the volume so they could spend their hours on the one question that needed a human.
How to evaluate an AI moderation tool?
The interview itself
Participants and data
What happens after
If you can, ask for a session recording from a real customer study (not the demo account). How it handles a real participant tells you more than any feature list could.
How to set up an AI-moderated study that feels human-like
Here’s a checklist to run before your first study:
Before launch
- Confirm if the study is structured: If you can't write the follow-ups in advance, it's a human study.
- Write the intro yourself: Who the AI is, who it's working for, how the recording is used, how long it takes, that answers are confidential. This is the fix for participants holding back.
- Tell it exactly when to probe: "If they don't mention pricing, ask what they'd expect to pay." Specific triggers beat "probe more." Some tools let you set probing per question.
- Cut the guide shorter than feels right: The AI won't trim it for you mid-session, which is how you get a 56-minute interview.
- Mix question types: Ratings and multiple choice keep data comparable. Save open-ended probing for the questions where you need the why.
- Take it yourself, twice: Once answering briefly, once rambling. Listen for interruptions, dead air and gushing praise.
- Pilot with 3–5 real participants before opening it to the larger audience.
After the first batch
- Read transcripts, not just summaries: The AI summarizes without checking if it understood. Spot-check that its summaries match what people actually said.
- Look for thin or suspiciously polished answers: Recordings and follow-ups make low-effort or AI-written responses easier to catch.
- Flag what surprised you: Those threads are your list for human-led follow-up interviews.
A sample discussion guide for your AI moderator
Here’s a complete example you can copy and adapt.
The study: First-week onboarding feedback for a project management app
Target length: 15 minutes
1. The intro script (read word for word)
“Hi, thanks for joining. I’m [AI bot name], an AI interviewer working on behalf of the research team at [Company]. I’d like to ask you about your first week using [Product]. It takes just 15 mins. The conversation is recorded so the research team can review it. Your answers are confidential and won’t be linked to your account or shared outside the team. There are no right or wrong answers. Honest feedback, including criticism, is the most useful thing you can give us.
Please take your time with each answer, and when you're ready for the next question, just say ‘next.’ Ready to start?”
This does three jobs: it says who the AI represents, covers privacy before the first question, and gives the participant a way to control the pace.
2. The questions
3. The close (read word for word)
“That's everything. Thank you, this is genuinely helpful. The research team reviews every conversation. [Incentive details, if any.] Have a good rest of your day.”
4. Instructions for the moderator
- Acknowledge answers briefly and naturally. Do not call answers "great," "fascinating" or "insightful."
- When you summarise, keep it to one sentence and ask, "Did I get that right?"
- Ask one question at a time. Do not combine two questions in one turn.
- If the participant is past 15 minutes, skip remaining probes and go straight to questions 6 and 7.
Not every tool supports every instruction here. Settings for summaries, tone and timing vary a lot. That's exactly why they're on the evaluation checklist above.
What makes this guide AI-friendly
- Every probe has a trigger: "If they mention X, ask Y" is something an AI can follow. "Explore their frustrations" is not.
- It's short: Seven questions, two of them ratings. The AI won't trim a long guide for you.
- Ratings are anchors: They give you numbers to compare across 60 sessions, while the open questions carry the why.
- It names the question to keep if time runs out: That's the judgment call a human would make in the moment.
Can you trust the data?
This is the question every researcher asks first, and it deserves a straight answer: AI moderation adds new ways for data to go wrong. It also leaves more evidence to catch them than a survey does.
There are four risks worth knowing about:
1. Participants using AI to answer
Research Ops teams report participants using AI to get past screeners, and even during live sessions. In the 2025 State of ReOps survey, 4 of 21 specialists said they'd seen it. An async session with no human watching is an easy target.
What helps: Recruit from your own customers or a vetted panel. Use follow-up probes: asking for a specific example is harder to fake than a polished first answer. Review recordings for answers that sound read aloud.
2. The moderator nudging answers to positive
NN/G's participants noticed the AI praising mundane answers. Nobody has measured whether that pushes participants toward more positive feedback. It's a fair worry, though: people tend to match the tone of whoever is interviewing them.
What helps: Tell the moderator to stay neutral (see the guide above). Treat strongly positive results with the same scepticism you'd apply to a leading question.
3. Summaries that miss the point
The AI summarizes an answer and moves on, without checking if it understood. If the summary is wrong, nobody corrects it and if you rely solely on AI summaries, the error carries straight into your readout.
What helps: Analyze full transcripts, not the tool's session summaries. Spot-check a handful against the recordings.
4. “Consistent” isn’t always consistent
Consistency is a big part of the pitch: the same guide, run the same way every time. NN/G saw the same script produce warm, chatty sessions and cold, brief ones. Some of that may have been product updates during the study, but it's worth checking.
What helps: Skim a sample of sessions from early, middle and late in the study before you trust comparisons across them.
200 interviews = 200 transcripts; something nobody mentions on their demo
AI moderation moves the bottleneck: yesterday, you were short on moderator hours. Tomorrow, you have 200 transcripts in a folder, a readout on Friday, and a PM asking “so what did people say?”
The stakes on analysis just shot through the roof: nobody from your team sat in on the sessions, and the transcript is the only record of what happened. Every theme in your readout needs to point back to the exact moment a participant said it. Otherwise you end up scrubbing through every recording when a stakeholder asks, “Who said that?”
This is what we built Looppanel for: upload your transcripts and recordings, from an AI moderator or a Zoom call, and Looppanel codes them into themes you can edit. Every tag traces back to the source quote and timestamp. You stay in control of the analysis: rename a theme, merge two, add your own tags, and the AI follows your lead.
To recap…
- AI-moderated interviews use real participants and an AI moderator. They are not synthetic users.
- They run structured interviews well and semi-structured discovery poorly. Pick studies accordingly.
- Expect rough edges: interruptions or silences, gushing praise, no sense of time. Most can be reduced with a good intro, specific probing rules and a short guide.
- Use them for breadth. Send a human in for depth.
- The interview getting cheaper makes analysis matter more, not less. Traceability is what lets you trust 200 transcripts you didn't sit in on.
Your peers also ask
Will AI moderators replace UX researchers?
No. They take over some structured sessions. Researchers still decide what to study, write the guide, judge the evidence and turn it into decisions. NN/G concludes they are a complement, not a replacement.
How is this different from an unmoderated test?
An unmoderated test follows a fixed script. An AI moderator can ask follow-up questions based on what each participant says.
Are participants honest with an AI?
It depends on the setup. Some vendors report more candor on sensitive topics. NN/G saw participants hold back when the session started without a proper intro on privacy.
Can AI moderators run interviews in other languages?
Many can. Several tools support 40+ languages, and NN/G saw one switch to Spanish mid-interview on request. Check your tool; some are English-only for now.
What’s the difference between AI-moderated interviews and synthetic users?
AI-moderated interviews talk to real people; only the moderator is AI. Synthetic users are AI personas that answer in place of real people. Researchers broadly trust the first for evaluative studies and stay skeptical of the second for anything beyond early hypotheses.
Do participants know they’re talking to an AI?
They should. Say so in the consent form and again in the session intro, along with who the AI represents and how recordings are used. NN/G found participants held back when sessions skipped this.
How long should an AI-moderated interview be?
Shorter than a human-led one. Aim for 10–20 minutes and fewer questions than feels natural. AI moderators don’t manage time well: in NN/G’s study, the same script ran anywhere from 13 to 56 minutes.
How many participants do I need?
There’s no magic number. Because extra sessions cost little moderator time, teams often run more than they would with human moderators. Think in segments: keep going until new interviews in each segment stop surfacing new themes. Pilot with 3–5 people first.
How much do AI-moderated interviews cost?
Pricing models vary: per interview, per study, per seat, or credits. Add participant incentives and recruitment on top. Compare tools at your real volume, since costs look very different at 20 sessions than at 200.
Can AI moderators run usability tests?
Some can. Tools with screen-sharing can watch a participant on a website or prototype and ask about where they got stuck. They’re less suited to watching someone work through a complex workflow on their own machine.
Can I use AI-moderated interviews for B2B research?
Yes, for structured studies like product feedback, onboarding or win/loss interviews with existing customers. Keep humans in conversations with senior buyers, or anywhere participants use heavy industry jargon.
How do I analyze AI-moderated interview data?
The same way you’d analyze any interview data: code the full transcripts into themes and trace every theme back to the quotes behind it. Don’t rely only on the tool’s session summaries, since the AI never checks its summaries with participants.




