Reviewing submissions: How do I decide who to accept/reject?

After you have checked your participants’ submissions, you should pay them as soon as possible. Ideally, you would reward them within 24-48 hours after they have completed your study.

You have a maximum of 21 days to review submissions. If you haven't reviewed submissions after 21 days, our system will automatically approve all unreviewed submissions.

On Prolific, you have the option to reject low-quality submissions. However, before you reject submissions, please think carefully whether the rejection is justified. Nobody likes to get rejected, so please try to keep rejections to a minimum. Participants are penalised for rejections, and if they receive too many, they will be removed from the pool. If you don't feel like a participant has been negligent, then please avoid rejecting. Instead, participants are able to voluntarily return submissions and will often do so upon request if it's demonstrated that a mistake was made.

General rule of thumb when reviewing submissions: If a participant completed the survey in time and submitted a completion code, then they should be compensated.

Valid reasons for rejections are, for example:

  • The participant completed your study exceptionally fast.
    • By "exceptionally fast," we mean participants who are statistical outliers (3 standard deviations below the mean).
  • The participant skipped crucial questions.
    • Crucial questions are those that are inevitable to answering your research question.
    • Demographic questions, for example, do not necessarily fall under "crucial questions" because participants have the right to withhold such sensitive information (according to guidelines by the British Psychological Society).
  • The participant failed one or multiple fair attention checks (please refer to our attention check policy).
  • The participant did not sufficiently engage in a task where the required level of engagement was clearly specified.
    • e.g. the participant was asked to write a paragraph but only put a few words or gibberish. 

Below are invalid reasons for rejection:

  • A participant completed your study "too fast"Participants completion time varies around an average. They have variable reading speeds and variable knowledge of keyboard shortcuts and other browser tools. Please use more reliable measures to check for attention, rather than completion time. Completion time cannot be used as a reason for rejection unless they are a statistical outlier below the mean.
  • A participant failed comprehension checks at the beginning of your survey. In these cases participants should be asked to return their submissions instead. If they do not return the submission voluntarily within 5 days then our researcher support team can do it for you.
  • I can't use the data because:
    - The participant completed the study but failed an internal measure so I can't use their data. e.g. the participant's variance on a measure was not above an arbitrary threshold. Unfortunately, in most cases, if you use internal measures as a quality control that are not explicitly explained to participants, then you'll still need to pay for their response even if you can't use their data. Please get in touch if you need clarity on your study.
    - There was a technical error in my study that caused the data not to be saved but wasn't the participant's fault. Unfortunately, technical errors sometimes occur, but if it was not the participant's fault, then this is not a valid reason for rejection. Please get in touch if this happens and we'll help as best as we can.
  • NOCODE, or the submission of an incorrect completion code, is not a valid reason for rejection by itself - read more about why this may occur here.

Please be respectful to participants and try to resolve issues with participants as best as you can - remember that these are people that have taken their time to participate in your research :) 

Please note that Prolific may overturn rejections in certain occasions.

You may also be interested in:

Was this article helpful?
112 out of 136 found this helpful
powered by Typeform