I still pilot every single study
People frequently ask about piloting user research sessions. How, when, with whom, for what purpose, and more. Here is some context to help explain the concept, importance, and mechanics:
Think of piloting as a dress rehearsal of sorts. You want to pilot ALL user research sessions, and everything associated with them, to test for question clarity, flow, stimuli, length of time, and technologies as well as to get more comfortable with the actual content, the “moderator” role, and your improv skills.
I still pilot every single study. I even pilot my unmoderated studies. EVERY SINGLE TIME.
Step by step:
Finalize your discussion guide to the best of your ability.
Recruit pilot participants who resemble your target recruiting criteria
A day or two (or more) before your actual sessions begin, run through your entire session plan (detailed in your completed discussion guide) with your pilot participants while simulating the test environment to the absolute best of your abilities. Include all activities, associated links, tools, prototypes, and technologies.
If you have notetakers and or a note-taking strategy, pilot this too. Make sure everyone is using the same approach, it’s crystal clear, and all session participants are labeled in the same way.
Pilot whatever process you’re using to field questions from your stakeholders/observers in real-time (e.g. a dedicated Slack channel)Record your pilot session and watch it afterward. Evaluate whether you introduced bias and if so, whether efforts were taken to mitigate it. Evaluate whether the session started and ended on time, if there were any issues, and what worked well and what didn’t.
Solicit feedback from your observers/stakeholders about what worked well for them and what could be improved. Consider whether the questions are interpreted as intended. if you fit them into the allotted session time, or if they need to be reduced or reframed. Can anything be omitted or clarified? Are the permissions to the prototypes set correctly?
Update your discussion guide and any related materials to address your learnings.
Send a recap to your stakeholders with updates on what you learned, what was revised, and why.
The UX of working with you, as a researcher, can always be improved and should always be considered. MODEL THE WAY!
Piloting allows you to assess how to improve your session including your own skills/approach, the experience of the observers/stakeholders commissioning the work to ensure it will yield the types of actionable responses to meet the goals, and your future session participants' experience ALL AT ONCE.
This week our Ask Like A Pro All-In students are piloting their discussion guides and EVERYTHING associated with their sessions in preparation to facilitate their live sessions. I am SUPER excited for them and thrilled to see how far they have come in the four short weeks since our cohort started!