Moderator presence boosts user preference ratings in usability tests
Category: User-Centred Design · Effect: Moderate effect · Year: 2015
The presence of a moderator during usability testing can lead users to rate the system more favorably, even if it doesn't significantly alter the number of usability issues found.
Design Takeaway
When conducting formative usability testing, consider using a moderator to facilitate clarification, but be mindful that their presence may positively bias user feedback on system preference.
Why It Matters
Understanding how testing conditions influence user perception is crucial for accurately interpreting usability data. This insight suggests that the social dynamic of testing can impact subjective feedback, requiring designers to consider this when evaluating user sentiment.
Key Finding
Having a moderator present during a usability test makes users like the system more, but doesn't necessarily change how many problems they find. Thinking aloud doesn't change problem discovery but makes users slower.
Key Findings
- Relaxed 'think aloud' protocol did not significantly affect the number of usability problems or subjective ratings, but it did slow down user performance.
- The presence of a moderator led to significantly higher subjective preference ratings for the system compared to participants testing alone.
Research Evidence
Aim: To investigate the effects of a moderator's presence and a relaxed 'think aloud' protocol on user performance and subjective ratings during usability testing.
Method: Experimental study with literature review
Procedure: An experiment was conducted where participants evaluated a system under different conditions: with a moderator present versus alone, and using a relaxed 'think aloud' protocol versus silent performance. Data on usability problems and subjective ratings were collected.
Context: Usability testing of digital systems
Design Principle
The social context of user research can influence observed outcomes, particularly subjective user perceptions.
How to Apply
When planning usability tests, decide whether the goal is to capture raw, uninfluenced user opinion or to gather rich, contextual feedback that might be influenced by a moderator's presence.
Limitations
The study did not find a significant effect of 'thinking aloud' on problem discovery, and the specific system tested may influence generalizability.
Student Guide (IB Design Technology)
Simple Explanation: When someone is watching you use a product, you might say nicer things about it, even if you still find the same number of problems.
Why This Matters: This helps you understand how the way you run your tests can affect what users tell you, making your research more accurate.
Critical Thinking: How might the cultural background of the moderator or the user influence the perceived effect of moderator presence?
IA-Ready Paragraph: The presence of a moderator during usability testing has been shown to positively influence users' subjective ratings of a system (Riihiaho, 2015). While this can be beneficial for gathering immediate clarifying insights, designers must be aware that this social dynamic may inflate positive sentiment, potentially masking genuine usability concerns.
Project Tips
- When designing your usability test, clearly define the role of any observer or moderator.
- Consider running parallel tests with and without a moderator to compare subjective feedback.
How to Use in IA
- Reference this study when justifying your choice of usability testing methodology, especially regarding moderator presence or 'think aloud' protocols.
Examiner Tips
- Demonstrate an awareness of how testing conditions can bias results.
- Justify your methodological choices with reference to established research.
Independent Variable: ["Moderator presence (present vs. absent)","Thinking aloud protocol (relaxed vs. silent)"]
Dependent Variable: ["Number of usability problems encountered","Subjective user ratings of the system"]
Controlled Variables: ["System being tested","Task instructions","User demographics (potentially)"]
Strengths
- Combines literature review with experimental data.
- Addresses under-researched aspects of usability testing methodology.
Critical Questions
- What are the ethical implications of potentially biasing user feedback?
- How can the positive aspects of moderator presence (e.g., clarification) be leveraged without compromising the objectivity of subjective ratings?
Extended Essay Application
- Investigate the impact of different observer roles (e.g., neutral observer, facilitator, interviewer) on user performance and feedback in a design project.
Source
Experiences with usability testing: Effects of thinking aloud and moderator presence · Aaltodoc (Aalto University) · 2015