Hostname: page-component-8448b6f56d-cfpbc Total loading time: 0 Render date: 2024-04-25T06:51:09.435Z Has data issue: false hasContentIssue false
  • English
  • Français

Avoiding Post-Treatment Bias in Audit Experiments

Published online by Cambridge University Press:  25 April 2018

Alexander Coppock
Alexander Coppock*
Affiliation:
Yale University e-mail: alex.coppock@yale.edu
Get access Rights & Permissions [Opens in a new window]

Extract

Audit experiments are used to measure discrimination in a large number of domains (Employment: Bertrand et al. (2004); Legislator responsiveness: Butler et al. (2011); Housing: Fang et al. (2018)). Audit studies all have in common that they estimate the average difference in response rates depending on randomly varied characteristics (such as the race or gender) of a requester. Scholars conducting audit experiments often seek to extend their analyses beyond the effect on response to the effects on the quality of the response. Response is a consequence of treatment; answering these important questions well is complicated by post-treatment bias (Montgomery et al., 2018). In this note, I consider a common form of post-treatment bias that occurs in audit experiments.

Type
Short Report
Information
Journal of Experimental Political Science , Volume 6 , Issue 1 , Spring 2019 , pp. 1 - 4
Copyright
Copyright © The Experimental Research Section of the American Political Science Association 2018 

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The data, code, and any additional materials required to replicate all analyses in this article are available at the Journal of Experimental Political Science Dataverse within the Harvard Dataverse Network, at doi:10.7910/DVN/6NVI9C. I would like to thank Ariel White, Noah Nathan, Julie Faller, Saad Gulzar, and Peter Aronow for helpful comments.

References

REFERENCES

Aronow, Peter M., Baron, Jonathon and Pinson, Lauren 2018. “A Note on Dropping Experimental Subjects who Fail a Manipulation Check.” Political Analysis. In press.Google Scholar
Bendick, Marc, Jackson, Charles W. and Reinoso, Victor A. 1994. “Measuring Employment Discrimination through Controlled Experiments.” The Review of Black Political Economy 23 (1): 25–48.Google Scholar
Bertrand, Marianne and Mullainathan, Sendhil 2004. “Are Emily and Greg more employable than Lakisha and Jamal? A Field Experiment on Labor Market Discrimination.” The American Economic Review 94 (4): 991–1013.Google Scholar
Björkman, Martina and Svensson, Jakob 2009. “Power to the People: Evidence from a Randomized Field Experiment of a Community-Based Monitoring Project in Uganda.” Quarterly Journal of Economics 124 (2): 735–69.Google Scholar
Butler, Daniel M. and Broockman, David E. 2011. “Do Politicians Racially Discriminate Against Constituents? A Field Experiment on State Legislators.” American Journal of Political Science 55 (3): 463–77.Google Scholar
Coppock, Alexander 2018. Replication Data for: Avoiding Post-Treatment Bias in Audit Experiments. Harvard Dataverse, v. 4.8.4. doi:10.7910/DVN/6NVI9C.Google Scholar
Costa, Mia 2017. “How Responsive are Political Elites? A Meta-Analysis of Experiments on Public Officials.” Journal of Experimental Political Science 4 (3): 241–54.Google Scholar
Fang, Albert H., Guess, Andrew M. and Humphreys, Macartan 2018. “Can the Government Deter Discrimination? Evidence from a Randomized Intervention in New York City.” Journal of Politics. In press.Google Scholar
Kalla, Joshua, Rosenbluth, Frances and Teele, Dawn Langan 2018. “Are You My Mentor? A Field Experiment on Gender, Ethnicity, and Political Self-Starters.” The Journal of Politics 80 (1): 337–41.Google Scholar
Montgomery, Jacob M., Nyhan, Brendan and Torres, Michelle 2018. “How Conditioning on Post-treatment Variables Can Ruin Your Experiment and What to Do About It.” American Journal of Political Science. In press.Google Scholar
Quillian, Lincoln, Pager, Devah, Hexel, Ole and Midtbøen, Arnfinn H. 2017. “Meta-analysis of Field Experiments Shows No Change in Racial Discrimination in Hiring Over Time.” Proceedings of the National Academy of Sciences 114 (41): 10870–5.Google Scholar
Rondeau, Daniel and List, John A. 2008. “Matching and Challenge Gifts to Charity: Evidence from Laboratory and Natural Field Experiments.” Experimental Economics 11 (3): 253–67.Google Scholar
White, Ariel R., Nathan, Noah L. and Faller, Julie K. 2015. “What Do I Need to Vote? Bureaucratic Discretion and Discrimination by Local Election Officials.” American Political Science Review 109 (1): 129–42.Google Scholar
Zhang, Junni L. and Rubin, Donald B. 2003. “Estimation of Causal Effects via Principal Stratification When Some Outcomes are Truncated by “Death”.” Journal of Educational and Behavioral Statistics 28 (4): 353–68.Google Scholar
Supplementary material: Link

Coppock Dataset

Dataset

https://doi.org/10.7910/DVN/6NVI9C
Link