Skip to main content Accessibility help

Missing the Target? Using Surveys to Validate Social Media Ad Targeting

Published online by Cambridge University Press:  29 March 2019

Michael W. Sances
University of Memphis, Memphis, TN
E-mail address:


Facebook ads are increasingly used by political scientists as a method of survey recruitment. A key advantage is said to be the ability to recruit targeted audiences defined by demographics, political beliefs, location, and numerous other attributes. The same feature has been decried by non-researchers concerned about potential racial discrimination and foreign influence in elections. The extent to which these ads actually reach their targets, however, is unknown. Using a series of six surveys and 20 targeted ads, I show these ads regularly fail to reach their targets. The success rate ranges from 23 to 99 percent, and ads targeted toward groups defined by self-reported data and broader geographic locations are generally more successful.

Research Notes
Copyright © The European Political Science Association 2019 

1. Introduction

Advertisements placed on social media websites are increasingly used by political scientists as a means of survey recruitment. In addition to the low cost, relative to more traditional methods, a key advantage is the power to target advertisements toward particular geographic, demographic, and political groups. The ability to target surveys toward specific subpopulations potentially allows researchers to focus on specific groups, such as voters in particular geographic locations or with particular demographic or political characteristics, who may be key to testing empirical claims.

Several recent papers in political science have taken advantage of such targeted ads. For instance, Samuels and Zucco (Reference Samuels and Zucco2013; Reference Samuels and Zucco2014) show Facebook can be used to recruit large samples of Brazilian respondents, Hirano et al. (Reference Hirano, Lenz, Pinkovskiy and Snyder2015) use Facebook ads to recruit likely primary voters in particular state elections, and Sances (Reference Sances2018) uses these ads to recruit local voters in particular cities. Beyond geography, Zhang et al. (Reference Zhang, Mildenberger, Howe, Marlon, Rosenthal and Leiserowitz2018) show that demographic ad targets can be used to implement a form of quota-sampling, allowing researchers to obtain a sample that is more nationally representative than low-cost alternatives such as Mechanical Turk (Berinsky et al. Reference Berinsky, Huber and Lenz2012; see also Boas et al. Reference Boas, Christenson and Glick2018). And for targeting users by political beliefs, Jäger (Reference Jäger2017) uses Facebook to recruit party activists in the US, Germany, and Thailand.

The same technology that political scientists have celebrated for research purposes is also decried by members of the broader public. Targeted campaign appeals have been at the center of alleged foreign interference in the 2016 election, during which several “Russian-linked Facebook ads” were reportedly “highly sophisticated in their targeting of key demographic groups in areas of (Michigan and Wisconsin) that turned out to be pivotal” (Raju et al. Reference Raju, Byers and Bash2017; see also Rosenberg et al. Reference Rosenberg, Confessore and Cadwalladr2018). Outside of campaigns, Facebook's “ability to slice and dice the habits of its hundreds of millions of users” may have enabled some marketers to engage in illegal discriminatory practices (Maheshwari and Isaac Reference Maheshwari and Isaac2016). In response to the controversy surrounding such discriminatory ads, Facebook announced it would no longer allow those advertising housing, credit, or employment to target users by race or ethnicity (targeting by race/ethnicity for academic purposes is still permitted).

Despite the alleged benefits and risks of this technology, it is currently unclear just how well it actually works. In this note, I validate these ad targets using direct survey measures. Specifically, I compare the purported characteristics of targeted groups to the self-reported characteristics of survey respondents recruited via ads targeting these groups. Using a series of six studies and a total of 20 targeted ads, I show that targeted ads often do not reach their intended targets. The success rate—the percentage of respondents who report being in the group toward which the ad was targeted—ranges from a low of 24 percent (for ads targeted toward African Americans) to a high of 99.8 percent (for ads targeted toward those over 25). Higher success rates do not appear to be driven by whether the ads target demographics, political attitudes, or geography. Rather, the most successful targets appear to be those defined by characteristics that Facebook users self-report on their user profiles, with lower success rates for targets defined by statistical models and third-party data sources.

2. Data and measurement

The six studies were conducted between October 2016 and February 2018. Table 1 provides a summary. The first column assigns each study an arbitrary number for reference. The second column shows the dates the study was fielded. The third column lists the targets that did not vary within each study. For instance, all ads associated with Study II were targeted toward residents of Memphis aged 18 and older. The fourth column lists any targets that varied within each study. For instance, one ad in Study II targeted residents of Memphis who were classified by Facebook as White; etc. The remaining columns list the number of ad clicks, cost per click, and total sample recruited.

Table 1. Studies used to validate Facebook ad targeting.

“Common Targets” lists the ad targets that did not vary within each study (e.g., all ads associated with Study II were targeted toward residents of Memphis aged 18 and older). “Varying Targets” lists any targets that varied within each study (e.g., one ad in Study II targeted residents of Memphis who were classified by Facebook as White).

Each study was associated with a different Qualtrics survey, and each ad within a study used the same base URL. To record which respondents were recruited by which ads, the base link was appended with an identifier recorded by Qualtrics in an embedded data field. For Studies I–IV, the ad recruitment text was modeled after recruitment ads used in Berinsky et al. (Reference Berinsky, Huber and Lenz2012), and respondents were incentivized to participate with the chance to win an gift card. For Studies V and VI, the text was modeled after Jäger 2017. No incentives were given in Study V, and in Study VI the promise of incentives was randomized such that the ad text in brackets was shown only to those residing in 20 of the 40 targeted zip codes.

Readers unfamiliar with the technical details of specifying ad targets may consult the papers cited in the introduction (see especially the appendix to Jäger Reference Jäger2017). However, it is important to discuss the details regarding certain targeted characteristics. Regarding demographics, Facebook uses the phrase “multicultural affinity” instead of “race” or “ethnicity.” When introducing this category in 2015, they described it as “the quality of people who are interested in and likely to respond well to multicultural content.” According to the marketing firm Performics, “The targeting is based on affinity, not ethnicity” (Allen Reference Allen2015), and in 2017 Facebook moved multicultural affinity from the “demographics” target cluster to the “behaviors” cluster in the ad interface (personal communication with Facebook Support, 9/15/2017). However, the subcategories (as of 17 January 2018) for affinity are racial and ethnic groups: “African American (US)”; “Asian American (US)”; “Hispanic (US—All)”; “Hispanic (US—Bilingual)”; “Hispanic (US—English Dominant)”; “Hispanic (US—Spanish Dominant)”.Footnote 1 Additionally, Facebook in late 2017 removed the ability of advertisers to exclude certain affinity groups from being targeted, explicitly citing concerns that racial and ethnic groups could be excluded via the affinity targets (Guynn Reference Guynn2017).

When targeting by location in my studies, I always targeted those residing in a particular area, setting a radius of zero miles so as to recruit only city residents and not those living in a broader metropolitan area. City-recruited respondents affirmed they lived in a given city on the Qualtrics consent screen. This may artificially boost the success with which these ads appear to target city residents; however, only 6 percent of those who clicked through the consent screen failed to consent, suggesting that not many users outside the city were recruited into Qualtrics via the ad.

The 1,244 zip codes targeted in Study I were chosen for the purpose of reaching downstate Illinois residents living outside of large cities. The 12 zip codes in Study II were chosen for the purpose of reaching African American voters, after the multicultural affinity targeting was found to perform poorly; these are the 12 zip codes with at least 75 percent African American population according to the US Census. The 40 zip codes in Study VI come from the Miami, FL area, and were chosen for having the highest share of enrollment in marketplace insurance plans under the Affordable Care Act. Zip codes were targeted collectively in Studies I and II, such that in Study I there was one ad for all 1,244 zip codes and in Study II there was one ad for all 12 zip codes. In Study VI, there was one unique ad for each of 40 zip codes.

3. Results

Table 1 shows the results, sorted by the type of ad target category: demographics, political behavior, and geography. The first column lists the ad target, and the second column gives the study number for referring back to Table 2. For each target, the survey included a question with response options that closely matched the target categories advertisers may select on Facebook. The third column of Table 1 gives the number of responses given to the survey item used to validate a given target. The fourth column gives the percentage of these responses that match the ad target (total recruited matching the target/total recruited by the ad).

Table 2. Validation results.

For each target-study combination, this table gives the number of Facebook users recruited into the survey via the ad target, and the match rate, or the number of Facebook users recruited whose self-reported characteristics match the target, divided by the number recruited.

Starting with demographics, ads that target users based on age are highly successful. Of the 78 respondents recruited by the 18–24 target, 91 percent report an age that was between 18 and 24. Similarly, of the 551 recruited by the 25+ target, 99.8 percent report an age of 25 or above. However, education and race show much lower match rates. Just 68 percent of the 283 recruited with the “Less than BA” target report an education level lower than a BA, while 74 percent of the 350 “BA or Higher” recruits report having at least a BA. Of the 176 respondents recruited using the “White” target, 93 percent identify as White, but only 23 percent of the 547 “Black” recruits identify as Black.Footnote 2 It is intuitive that targets based on what Facebook calls “cultural affinity” would not successfully recruit by race; on the other hand, much public attention has been devoted to the possibility of marketers using cultural targets to discriminate based on race. To my knowledge, the current analysis is the only evidence as to whether such discrimination is possible.

Targeting users by political ideology is generally successful (see also Bond and Messing Reference Bond and Messing2015). Of the 34 recruited by the ad targeting “Very Liberal” voters, 97 percent are indeed at a 1 or a 2 on a 7-point conservatism scale. Of the 34 recruited by the “Liberal” target, 80 percent are at a 3 or lower on the same scale. Partisanship is much less reliable. Of the 493 respondents recruited by an ad targeting Democratic voters, just 59 percent identify as a Democrat or a Democrat-leaning independent. And out of the 44 respondents recruited with an ad targeted toward those having donated to liberal political causes, just 39 percent report donating to liberal causes in the past two years.

The seven ads targeting users by city of residence are generally successful. Excluding those recruited by ads targeting Atlanta residents, on average 97 percent of those recruited by a given city target report living in zip codes contained in that city. In Atlanta, the match rate is 82 percent.Footnote 3 Zip code targeting gives mixed results. In Study 1, the same ad was targeted toward users reportedly residing in any of 1,244 zip codes in Illinois. Out of 1,035 respondents recruited via this ad, 97 percent reported a zip code in the set of 1,244. More focused recruitment by zip code is less successful. In Study II, 12 zip codes in Memphis were targeted. Of the 243 respondents recruited using these targets, only 44 percent reported living in these zip codes. In Study VI, 78 percent of respondents’ zip codes matched their ad targets' codes.

4. Limitations

Poor match rates may result from careless or dishonest respondents, and this risk may be greater when respondents are incentivized. However, inattention and dishonesty should be problematic for all items, yet we see that some items (such as age) have significantly higher match rates than others. Note also that incentives were not used on Study V, which saw match rates as low as 39 percent.

To more directly assess these issues, Study VI included a Screener question (Berinsky et al. Reference Berinsky, Margolis and Sances2014). Respondents were asked their favorite color, then informed the question was in fact an attention check, and that to demonstrate attention they should select both “red” and “green” as their answer. Twenty out of 51 respondents, or 39 percent, passed the attention check. This is much lower than the 70 percent passage rate observed on a conventional online panel (Berinsky et al. Reference Berinsky, Margolis and Sances2014). I examine the relationship between attention, match rates, and incentives in the Appendix. To summarize, while the match rate is higher among passers, the difference is imprecisely estimated (p = 0.083); neither attention nor match rates are significantly different when incentives are used.

Poor geographic match rates could also result from disparities in access to Facebook. If certain areas are low-access, ads targeted to those areas may have less success. In the Appendix, I examine the correlation between match rates and Facebook access (number of users divided by population) in Study VI. I find discrepant access does not explain discrepant match rates.

5. Discussion and conclusion

Social media websites such as Facebook collect and store voluminous amounts of data about their users, data that may present a boon to social researchers and a nuisance to the public. However, the extent to which either of these possibilities is realized depends on the ability of researchers and marketers to use this data to reach particular groups. Using a series of surveys linked to targeted ad campaigns on the Facebook platform, I have shown the ability varies greatly depending on the target category. While some ads reach their target groups 99 percent of the time, others reach their targets only 23 percent of the time.

What explains the relative success or failure of different targets to reach their intended audience? Examining the demographics and political behavior targets in Table 1, age and ideology have match rates in the 80s and 90s, while education, race, party, and political donating have match rates as low as the 20s and only as high as the 70s. Looking further, the more accurate categories tend to have different sources than the less accurate categories. Age, for instance, is (presumably) observed directly by Facebook using the date of birth reported on an individual's profile page. The “very liberal” target, moreover, is described by Facebook on the advertiser interface as matching “People in the USA who have a very liberal political affiliation,” while the target for college graduates is described as “People who indicated their highest education level as College grad.”

Facebook's descriptions of the less successful categories are more nuanced. The description for the “Democratic Party (United States)” category is “People who have expressed an interest in or like pages related to Democratic Party (United States)”; the description for the African American multicultural affinity category is, “People who live in the United States whose activity on Facebook aligns with African American multicultural affinity”; and the description for those matching ‘Donate to liberal political causes’ is “People who are interested in donating to liberal political causes. / Source: Partner Category based on information provided by Epsilon: Multi-sourced.” Assignment to these categories is apparently less a function of self-reporting by users, and more a function of Facebook predicting categories based on user activity and third-party data.Footnote 4

Geographic targeting is generally successful, provided the targeting is not too granular. At the level of cities and large collections of zip codes, match rates tend to be in the 90s, as high as 97 percent for the 1,244 zip codes targeted in Study I. At a finer level of detail, targeting fails: while Study II attempted to recruit respondents from 12 particular zip codes in a given city, the success rate was just 52 percent. This variation in success may be because broad geographic categories such as city are directly reported to Facebook by users via their profiles, while more granular location data such as zip code may be obtained via IP geolocation.Footnote 5

Those interested in using Facebook for survey recruitment can adopt a validation strategy from the literature on online panels, which have faced similar issues involving “false qualifying” (Baker et al. Reference Baker, Blumberg, Brick, Couper, Courtright, Dennis, Dillman, Frankel, Garland, Groves, Kennedy, Krosnick, Lavrakas, Lee, Link, Piekarski, Rao, Thomas and Zahs2010, pp. 744–745). As suggested by Phillips (Reference Phillips2016) and implemented by Craig et al. (Reference Craig, Hays, Pickard, Cella, Revicki and Reeve2013), researchers can simply compare survey responses to profile characteristics, or in the case of Facebook ads, ad target characteristics. I include instructions for implementing this strategy in the Appendix.

Facebook does indeed hold promise as a way for researchers to target specific groups. However, researchers should exercise caution, and should always validate their recruitment strategies using direct survey measures when possible. Ads targeted by age, ideology, and broader geographic areas such as city are far more likely to reach their intended audience than those targeted by race, partisanship, and sub-municipal areas.

At the same time, concerns about marketers using targets to discriminate by race, or for political actors to influence key demographics in small geographic areas, may be overstated. Since the 2016 election, there has been much public concern about foreign actors using social media to target groups susceptible to persuasion and mobilization. As one Washington Post columnist summarizes, “imagine being able to target this message with minute precision: say, telling black voters in swing counties that Hillary Clinton was an incorrigible racist, or enraging white, male gun lovers with her supposed plans to roll back the Second Amendment. Imagine how quickly such misinformation could spread and metastasize” (Emba Reference Emba2017). Of course, whether such messages could actually change anyone's mind is a classic question in social science. Yet, as this note has shown, it is a debate that may not be necessary here, given the limited power of ad targeting.

Supplementary Material

The supplementary material for this article can be found at


I thank Eric Groenendyk, Baobao Zhang, and participants at the 2017 APSA conference for helpful comments. This work was supported in part by grants from the University of Memphis FedEx Institute of Technology, the City of Memphis, and the University of Memphis Faculty Research Grant Fund. All interpretations and conclusions are my own.


1 There is no multicultural affinity category for White or Caucasian. Thus, the “White' target in my study was created using a target that excluded all of these affinity groups. Similarly, there is no category for less than college degree, so I designed this target to exclude those in any of the following educational categories: “College grad”; ‘Doctorate degree”; “In grad school”; “Master's degree”; “Some grad school.” Other possible targets include high school graduate and some high school (see\#education_and_workplace).

2 Of the remaining 77 percent, 70 percent identified as White.

3 To validate city of residence, self-reported zip codes were compared to a list of zip codes known to be contained within the limits of a particular city. I generated these lists of contained zip codes using GIS software.

4 Success may also depend on the accuracy of the algorithm Facebook uses to translate online activity into demographic categories. For instance, the particularly low success of the African American target may be due to the relatively low proportion of African Americans in the Facebook user base, which could lead to a worse-performing algorithm. The varying success rates of categories based on likes is not entirely inconsistent with the results of Bond and Messing (2015), who construct a measure of political ideology based on page likes. While the resulting measure is highly successful at discriminating liberals from conservatives, it is less successful at discriminating weak ideologues from strong ideologues. At the elite level, Bond and Messing find the within-party correlations between like-based ideology and DW-NOMINATE scores are 0.47 for Democrats and 0.42 for Republicans. At the mass level, they find self-reported ideologues and self-reported strong ideologues are indistinguishable on like-based ideology. Thus, Bond and Messing's results also suggest Facebook's algorithm is imperfect.

5 On launching zip code targeting in 2011, a Facebook spokesperson told Adweek that “Facebook uses a variety of information to help us determine a person's zip code.” The reporter speculated that ‘this might include a user's IP address and internet service provider as well as the current city and address information users list in their profiles’ (Constine Reference Constine2011). Given my own results, it is unlikely that Facebook determines zip code based on GPS tracking.


Allen, B (2015) New facebook multicultural affinity targeting: more granular segmenting options for brands. Performics (December 2). Accessed 17 January 2018 via\\granular-segmenting-options-for-brands/.Google Scholar
Baker, R, Blumberg, SJ, Brick, JM, Couper, MP, Courtright, M, Dennis, JM, Dillman, D, Frankel, MR, Garland, P, Groves, RM, Kennedy, C, Krosnick, J, Lavrakas, PJ, Lee, S, Link, M, Piekarski, L, Rao, K, Thomas, RK andZahs, D (2010) Research synthesis: AAPOR report on online panels. Public Opinion Quarterly 74, 711781.Google Scholar
Berinsky, AJ, Huber, GA and Lenz, GS (2012) Evaluating online labor markets for experimental research:'s Mechanical Turk. Political Analysis 20, 351368.CrossRefGoogle Scholar
Berinsky, AJ, Margolis, MF and Sances, MW (2014) Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science 58, 739753.CrossRefGoogle Scholar
Boas, TC, Christenson, DP and Glick, DM (2018) Recruiting large online samples in the United States and India: facebook, Mechanical Turk and Qualtrics. Political Science Research and Methods forthcoming.CrossRefGoogle Scholar
Bond, R and Messing, S (2015) Quantifying social media's political space: estimating ideology from publicly revealed preferences on facebook. American Political Science Review 109, 6278.CrossRefGoogle Scholar
Constine, J (2011) Facebook's new zip code Ad targeting nearly doubles click through rates says blinq media. Adweek August 11. Accessed 7 August 2018 via Scholar
Craig, BM, Hays, RD, Pickard, AS, Cella, D, Revicki, DA and Reeve, BB (2013) Comparison of US panel vendors for online surveys. Journal of Medical Internet Research 15, e260.CrossRefGoogle ScholarPubMed
Emba, C (2017) When it comes to Facebook, Russia's 100,000 is worth more than you think. Washington Post (September 11). Accessed 15 October 2018 via Scholar
Guynn, J (2017) Facebook halts ads that exclude racial and ethnic groups. USA Today (November 29). Accessed 17 January 2018 via Scholar
Hirano, S, Lenz, GS, Pinkovskiy, M and Snyder, JM (2015) Voter learning in state primary elections. American Journal of Political Science 59, 91108.CrossRefGoogle Scholar
Jäger, K (2017) The potential of online sampling for studying political activists around the world and Across time. Political Analysis 25, 329343.CrossRefGoogle Scholar
Maheshwari, S and Isaac, M (2016) Facebook will stop some Ads from targeting users by race. New York Times (November 11). Accessed 17 January 2018 via Scholar
Phillips, K (2016) Five verification strategies to minimize online market research fraud. Survey sampling international blog (June 16). Accessed 4 March 2018 via Scholar
Raju, M, Byers, D and Bash, D (2017) Exclusive: Russian-linked facebook ads targeted Michigan and Wisconsin. CNN (October 4). Accessed 17 January 2018 via Scholar
Rosenberg, M, Confessore, N and Cadwalladr, C (2018) How trump consultants exploited the facebook data of millions. New York Times (March 17). Accessed 7 August 2018 via Scholar
Samuels, DJ and Zucco, C (2013) Using facebook as a subject recruitment tool for survey-experimental research. Working paper, Department of Political Science, University of Minnesota. Accessed 17 January 2018 via Scholar
Samuels, DJ and Zucco, C (2014) The power of Partisanship in Brazil: evidence from survey experiments. American Journal of Political Science 58, 212225.CrossRefGoogle Scholar
Sances, MW (2018) Ideology and vote choice in U.S. Mayoral elections: evidence from facebook surveys. Political Behavior forthcoming.CrossRefGoogle Scholar
Zhang, B, Mildenberger, M, Howe, PD, Marlon, J, Rosenthal, S and Leiserowitz, A (2018) Quota sampling using facebook advertisements. Political Science Research and Methods forthcoming.CrossRefGoogle Scholar

Sances Dataset


Sances supplementary material


[Opens in a new window]
PDF 666 KB

Altmetric attention score

Full text views

Full text views reflects PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views.

Total number of HTML views: 42
Total number of PDF views: 360 *
View data table for this chart

* Views captured on Cambridge Core between 29th March 2019 - 4th December 2020. This data will be updated every 24 hours.

Hostname: page-component-b4dcdd7-z76xg Total loading time: 0.753 Render date: 2020-12-04T12:22:21.042Z Query parameters: { "hasAccess": "1", "openAccess": "0", "isLogged": "0", "lang": "en" } Feature Flags last update: Fri Dec 04 2020 12:00:04 GMT+0000 (Coordinated Universal Time) Feature Flags: { "metrics": true, "metricsAbstractViews": false, "peerReview": true, "crossMark": true, "comments": true, "relatedCommentaries": true, "subject": true, "clr": false, "languageSwitch": true }

Send article to Kindle

To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Missing the Target? Using Surveys to Validate Social Media Ad Targeting
Available formats

Send article to Dropbox

To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

Missing the Target? Using Surveys to Validate Social Media Ad Targeting
Available formats

Send article to Google Drive

To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

Missing the Target? Using Surveys to Validate Social Media Ad Targeting
Available formats

Reply to: Submit a response

Your details

Conflicting interests

Do you have any conflicting interests? *