What are attention checks?
- They are a simple way to determine who is paying attention to your study instructions (Oppenheimer et al, 2009)
Here's an example:
But the jury’s out...
- Qualtrics’ research (Vanette, 2017): If you exclude participants only because of failed attention checks, that could harm data quality by introducing demographic and response biases into the sample.
- They could also change participants' behaviour in a study (Hauser & Schwarz, 2015).
Prolific’s Policy
- We strongly believe in the principle of ethical and fair rewards!
- So, any attention checks must comply with the following guidelines:
What is a fair attention check?
- It should check whether a participant has paid attention to the question, not so much to the instructions above it.
- You should only use the check if, without it, the task couldn’t be completed properly
A fair attention check:
The colour test is simple, when asked for your favourite colour you must enter the word puce in the text box below.
Based on the text you read above, what colour have you been asked to enter?
An Unfair Check:
The colour test is simple, when asked for your favourite colour you must enter the word puce in the text box below.
What is your favourite colour?
This does not comply with our guidelines, because it tests whether a participant has paid attention to the instructions above the question, not the question itself
Additional Requirements
- They cannot:
- Be in repeated, unchanging text
- Use an intentionally small font
- Rely on memory recall
- Participants must be explicitly instructed to complete a task in a certain way (e.g. "click 'Strongly disagree' for this question"), rather than leaving room for mis-interpretations (e.g. "Prolific is a clothing brand. Do you agree?").
- If you're unsure whether your attention check is fair, ask us!
Can I use attention checks to detect low quality submissions?
How many should I use?
- We recommend that you have at least one in any study.
How many failures is a good basis for rejection?
- You can use multiple checks to reliably detect submissions of low quality.
- If a participant fails more than one fair attention check, they can justifiably be rejected.
What if they only fail one attention check?
- A single failed attention check can only be used as a basis for rejection if your study is very short (e.g. under 5 mins).
Payments and attentions checks
- You can make payments contingent on attention checks, if they‘re in line with our guidance.
- Please be considerate rejecting submissions. Always keep in mind that participants are real people who are helping you get the data you need!
If you want to find out more, read our blog:
- Our thoughts on failing attention checks
- Deciding upon who to accept/reject
- Improving your data quality
References
Hauser, D. J., & Schwarz, N. (2015). It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks. SAGE Open.
Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872.
Vannette, D. (2017). Using attention checks in your surveys may harm data quality. Retrieved from https://www.qualtrics.com/blog/using-attention-checks-in-your-surveys-may-harm-data-quality/