Representative Samples FAQ

Prolific Team

13 February 2020 14:12

This article provides more detailed answers to some common questions about representative samples.

Who is eligible to be in a representative sample of the United Kingdom or United States?

In order for a participant to be eligible for a representative sample, there must be space remaining in their matching age, sex, or ethnicity subgroup. Additionally, they must be a current resident of the country being sampled and they must be fluent in the language of that country.

Where do we get our census data from?

We use census data from the US Census Bureau and the UK Office of National Statistics.

The age by sex by ethnic group proportions for the UK can be calculated from 2011 Census data found here.
The age by sex by ethnic group proportions for the US can be calculated from US Census Bureau population group estimates from 2015, found here.

How do you create the demographic subgroups used for representative samples?

Participant eligibility for representative samples is calculated on the basis of prescreening answers. We use three prescreeners: 'Date of Birth', 'Sex’, and ‘Ethnicity (Simplified)’.

Starting with the youngest allowable participation age on Prolific, we stratify age using five 9-year brackets: 18-27, 28-37, 38-47, 48-57, and 58+.
‘Sex’ is stratified into male and female.
‘Ethnicity (Simplified)’ is stratified into the five categories recommended by the UK Office of National Statistics: White, Mixed, Asian, Black and Other.

What type of allocation algorithm do you use?

Cross stratifying on age (5 brackets), ethnicity (5 groups) and sex (2 groups) results in 50 subgroups: one for every combination of answers. Using census data, we can calculate the proportion of each subgroup in the national population. In order to ensure minimal representation, we first allocate 1 space per subgroup. We then allocate the remaining spaces in the sample proportionally, according to the national population.

Where we need to round subgroup sizes (i.e proportionate allocation indicates a subgroup should have 2.6 participants) we try to round to the closest whole number, while balancing round-ups and round-downs to ensure the total sample size remains exactly what was asked for.

How are participants recruited to my representative sample?

Participants take part in a representative sample study in exactly the same way as a normal Prolific study. In other words, a representative sample is collected on a (mostly) first come, first serve basis: though we do have processes in place to ensure fair distribution of studies across the participant pool.

How do you ensure a representative sample reaches my intended sample size?

If a representative sample study is still awaiting submissions 48 hours after you have launched it on Prolific, we loosen the eligibility requirements of each subgroup a little, in order to strike a balance between timely data collection and sample representativeness. We do this by removing age bracket restrictions from all unfilled places in the sample

As an example, if you have 2 unfilled places for 28-37 year old Asian Women in your study 48 hours after publication. These places will be opened up to all Asian Women in the country, regardless of age.

In practice, we find most representative samples are at least 90% full after 48 hours, and fill completely after loosening. This normally results in sample accuracy of 95% or more.

How long does it take to collect a representative sample?

We expect a representative sample study to complete within 2 to 4 days.

How is sample accuracy calculated?

We calculate final representative sample accuracy by summing the total number of subgroup requirements met by participants in the final sample, and dividing it by the total number of requirements that could have possibly been met.

For example: You have a final sample of 1000 participants.

The total possible number of requirements your sample could meet is 3000. 3 (age, ethnicity, sex) x 1000 (the total sample size)
900 places have been filled by participants that met their subgroup’s requirements for age, ethnicity and sex: 900 x 3 = 2700 requirements met.
80 participants were collected after one round of reallocation: These participants met the requirements for ethnicity and sex: 80 x 2 = 160 requirements met.
20 participants were collected after two rounds of reallocation: These participants met the requirement for sex only: 20 x 1 = 20 requirements met.
2700 + 160 + 20 = 2880
2880 / 3000 = 96% sample accuracy.

Can I add more participants to a representative sample study after publishing?

It is not currently possible to increase places on a representative sample study after it has been published, due to complications around pricing and sample stratification. If you wish to recruit more participants for a representative sample study, we recommend that you run a duplicate study, and exclude any existing participants using "Previous Studies" screening.

For more help with excluding previous participants from a representative sample study, please contact our support team via info@prolific.co.

Please note that representative sample accuracy is only calculated internally for the time being. But we will display the sample accuracy on the user interface soon.