Pekka Räsänen, University of Turku
Atte Oksanen, Vili Lehdonvirta, Grant Blank
There are growing difficulties with traditional recruitment methods for survey researchers. One of the reasons behind this is the use of the new communication technology. In second decade of the 21st century, the internet penetration rates had exceeded 80 and 90 percent in all industrialized countries. Social scientists are increasingly using digital media as an alternative means of recruiting respondents to surveys.
In a typical online survey, respondents are recruited by publishing a link to the survey in an online platform where it is likely to catch the attention of members of the target population, such as a relevant mailing list, social media group, or web page. This effect can be further boosted by paying for attention. Popular social media platforms such as Facebook allow target marketing based on sociodemographic variables and interests. Another option is to use dedicated online survey panels. Providers such as Social Survey International (SSI) and Amazon Mechanical Turk offer access to a dedicated ‘crowd’of respondents who have signed up to their system. These ‘semi-professional’respondents can participate in multiple surveys over time and are compensated for each survey with money, discount vouchers or similar.
This methodology paper examines recent changes in the modes of survey data collection and its consequences for nonresponse and measurement errors. We compare the findings from four types of survey data that were collected for the Hate Communities: A Cross-National Comparison Project.
The first sample consists of the respondents of YouNet 2013 YLE Finland Survey who were recruited by providing a link to the survey in a news article on threatening and hateful material online on May 2013. There were in total 493 respondents and 117 of the respondents were aged 16 to 24. This sample represents attention-based recruitment.
The second sample consists of respondents of the same survey who were recruited using three Facebook advertising campaigns targeted at Finnish users aged 15 to 30. The campaigns were launched in between April and May 2013. We used four images and four short marketing texts in 15 different combinations to attract users to fill out the YouNet2013 Survey. The possibility of winning a movie ticket prize was mentioned in the marketing text. After the banner click, an introductory text provided key details about the survey. The campaigns generated 1,337 survey responses in all, with 779 of the respondents aged 16 to 24. This sample represents advertising-based attention-based recruitment with a small element of compensation (prize draw), but the respondents are not ‘semi-professional’survey takers.
The third sample consists of respondents to the same survey (n=555) who were recruited from a demographically balanced panel of Finnish respondents who had voluntarily agreed to participate in research surveys. The panel aw administered by Survey Sample International (SSI) in May’June 2013. E-mail invitations were sent to a sample of panel members stratified to mirror the Finnish population aged 15 to 30 on age, gender and geographic region. Sample quota was estimated to be nationally representative in terms of age and gender, with 348 of the respondents were aged 16 to 24. This sample represents purely compensation-based surveying, and the respondents are ‘semi-professionals’who are frequently called upon to complete surveys.
Finally, our fourth sample comes from Statistics Finland’s Use of Information and Communications Technology by Individuals Survey, which was fielded to 16 to 74-year-olds in April’May 2013. This survey serves as the source for official statistics for Finland, and is methodologically rigorous. Respondents aged 16 to 30 years were selected for further analysis. The survey includes two items designed originally for the YouNet 2013 survey. This sample represents the ground truth against which the other samples are compared.
Our findings show that the attention-based samples (news site and FB) differ considerably from the compensation-based sample (SSI) and from the ground truth data provided by Statistics Finland. The compensation-based data is also different from the ground truth on a prevalence of response variables. The findings suggest that it is difficult to collect nationally-representative samples by using online platforms. In particular, the distributions of socio-demographic background variables deviate from one survey sample to another. However, this is not a problem in many types of Internet research, because typically the populations are not well-known prior to the research. Self-selecting attention-based surveys may be more useful for surveying attitudinal and psychological characteristics than socio-demographic characteristics.
We discuss theoretical and practical implications for online survey research. We argue that survey research should increasingly make use of technology, since there are many interesting and important phenomena that predominately take place in online platforms.