Using attention checks as a measure of data quality

What are attention checks?

  • They are a simple way to determine who is paying attention to your study instructions (Oppenheimer et al, 2009)

Here's an example:

But the jury’s out...


Prolific’s Policy

  • We strongly believe in the principle of ethical and fair rewards!
  • So, any attention checks must comply with the following guidelines:

What is a fair attention check?

  • It should check whether a participant has paid attention to the question, not so much to the instructions above it
  • You should only use the check if, without it, the task couldn’t be completed properly

A fair attention check:

The colour test is simple, when asked for your favourite colour you must enter the word puce in the text box below.

Based on the text you read above, what colour have you been asked to enter?

Screen_Shot_2018-12-20_at_12.10.22.png

An Unfair Check:

The colour test is simple, when asked for your favourite colour you must enter the word puce in the text box below.

What is your favourite colour?

 Screen_Shot_2018-12-20_at_12.10.22.png

This does not comply with our guidelines, because it tests whether a participant has paid attention to the instructions above the question, not the question itself

Additional Requirements

  • They cannot: 
    • Be in repeated, unchanging text
    • Use an intentionally small font
    • Rely on memory recall
  • Participants must be explicitly instructed to complete a task in a certain way (e.g. "click 'Strongly disagree' for this question"), rather than leaving room for mis-interpretations (e.g. "Prolific is a clothing brand. Do you agree?").
  • If you're unsure whether your attention check is fair, ask us!

Can I use attention checks to detect low quality submissions?

How many should I use?

  • We recommend that you have at least one in any study.

How many failures is a good basis for rejection?

  • You can use multiple checks to reliably detect submissions of low quality. 
  • If a participant fails more than one fair attention check, they can justifiably be rejected.

What if they only fail one attention check?

  • A single failed attention check can only be used as a basis for rejection if your study is very short (e.g. under 5 mins).

Payments and attentions checks

  • You can make payments contingent on attention checks, if they‘re in line with our guidance. 
  • Please be considerate rejecting submissions. Always keep in mind that participants are real people who are helping you get the data you need!

If you want to find out more, read our blog:

 

References

Hauser, D. J., & Schwarz, N. (2015). It’s a Trap! Instructional Manipulation Checks Prompt Systematic Thinking on “Tricky” Tasks. SAGE Open.

Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867-872.

Vannette, D. (2017). Using attention checks in your surveys may harm data quality. Retrieved from https://www.qualtrics.com/blog/using-attention-checks-in-your-surveys-may-harm-data-quality/




Was this article helpful?
59 out of 63 found this helpful
powered by Typeform