Skip to main content Accessibility help
Hostname: page-component-7ccbd9845f-2c279 Total loading time: 0.559 Render date: 2023-01-28T11:33:00.475Z Has data issue: true Feature Flags: { "useRatesEcommerce": false } hasContentIssue true

Part IV - Systemic Issues

Published online by Cambridge University Press:  19 November 2018

Robert J. Sternberg
Cornell University, New York
Get access
Publisher: Cambridge University Press
Print publication year: 2018

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)



Alsheikh-Ali, A. A., Qureshi, W., Al-Mallah, M. H., & Ioannidis, J. P. A. (2011). Public availability of published research data in high-impact journals. PLoS ONE, 6(9), e24357. ScholarPubMed
Anderson, M., & Magruder, J. (2017). Split-sample strategies for avoiding false discoveries. w23544. Cambridge, MA: National Bureau of Economic Research. doi:10.3386/w23544CrossRefGoogle Scholar
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017). Sample-size planning for more accurate statistical power: A method adjusting sample effect sizes for publication bias and uncertainty. Psychological Science, 28(11), 15471562. doi:10.1177/0956797617723724CrossRefGoogle ScholarPubMed
Begley, C. G., & Ellis, L. M. (2012). Drug development: Raise standards for preclinical cancer research. Nature, 483(7391), 531533. doi:10.1038/483531aCrossRefGoogle ScholarPubMed
Bem, D. J. (2004). Writing the empirical journal article. In Zanna, M. P. & Darley, J. M. (Eds.), The compleat academic: A practical guide for the beginning social scientist (2nd edn., pp. 185219). Washington, DC: American Psychological Association.Google Scholar
Board of Governors of the Federal Reserve System, Chang, A. C., & Li, P. (2015). Is economics research replicable? Sixty published papers from thirteen journals say “usually not.” Finance and Economics Discussion Series, 2015(83), 126. doi:10.17016/FEDS.2015.083Google Scholar
Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, , Emma, S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14(5), 365376. doi:10.1038/nrn3475CrossRefGoogle ScholarPubMed
Claerbout, J. (1994). Seventeen years of super computing and other problems in seismology. Paper presented at the National Research Council meeting on High Performance Computing in Seismology. Scholar
Crampton, E. W. (1947). The growth of the odontoblast of the incisor teeth as a criterion of vitamin C intake of the guinea pig. Journal of Nutrition, 33(5), 491504. ScholarPubMed
de Groot, A. (1969). Methodology: Foundations of inference and research in the behavioral sciences. The Hague: Mouton.Google Scholar
Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203) 15021505. doi:10.1126/science.1255484CrossRefGoogle ScholarPubMed
Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power. American Statistician, 55(February), 1924. doi:10.1198/000313001300339897CrossRefGoogle Scholar
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196217. doi:10.1207/s15327957pspr0203_4CrossRefGoogle ScholarPubMed
Lakens, D. (2014a). Performing high-powered studies efficiently with sequential analyses: Sequential analyses. European Journal of Social Psychology, 44(7), 701710. doi:10.1002/ejsp.2023CrossRefGoogle Scholar
Lakens, D. (2014b, December 19). The 20% statistician: Observed power, and what to do if your editor asks for post-hoc power analyses. Retrieved November 11, 2017, from
Lakens, (2017). Equivalence tests: A practical primer for tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355362. doi:10.1177/1948550617697177CrossRefGoogle ScholarPubMed
Lin, W., & Green, D. P. (2016). Standard operating procedures: A safety net for pre-analysis plans. PS: Political Science & Politics, 49(3), 495500. doi:10.1017/S1049096516000810Google Scholar
Lindsay, S., Simons, D., & Lilienfeld, S. (2016). Research preregistration 101. Observer, 29(10), 1416.Google Scholar
Lithgow, G. J., Driscoll, M., & Phillips, P. (2017). A long journey to reproducible results. Nature, 548(7668) 387388. doi:10.1038/548387aCrossRefGoogle ScholarPubMed
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., … Yarkoni, T. (2015). Promoting an open research culture. Science, 348(6242), 14221425. doi:10.1126/science.aab2374CrossRefGoogle ScholarPubMed
Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. doi:10.1126/science.aac4716
Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528530. doi:10.1177/1745691612465253CrossRefGoogle ScholarPubMed
Pereira, T. V., & Ioannidis, J. P. A. (2011). Statistically significant meta-analyses of clinical trials have modest credibility and inflated effects. Journal of Clinical Epidemiology, 64(10), 10601069. doi:10.1016/j.jclinepi.2010.12.012CrossRefGoogle ScholarPubMed
Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638641. doi:10.1037/0033-2909.86.3.638CrossRefGoogle Scholar
Savage, C. J., & Vickers, A. J. (2009). Empirical study of data sharing by authors publishing in PLoS journals. Ed. Chris Mavergames. PLoS ONE, 4(9), e7078. doi:10.1371/journal.pone.0007078CrossRefGoogle Scholar
Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 13591366. doi:10.1177/0956797611417632CrossRefGoogle ScholarPubMed
Simons, D. J., Shoda, Y., & Lindsay, D. S. (2017). Constraints on generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, August, 174569161770863. doi:10.1177/1745691617708630CrossRefGoogle ScholarPubMed
Vazire, S. (2017). Quality uncertainty erodes trust in Science. Collabra: Psychology, 3(1), 1. doi:10.1525/collabra.74CrossRefGoogle Scholar
Vazire, S. (2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13(4), 411–17.CrossRefGoogle ScholarPubMed
Veer, A. E. van 't, & Giner-Sorolla, R. (2016). Pre-registration in social psychology – A discussion and suggested template. Journal of Experimental Social Psychology, 67(November), 212. doi:10.1016/j.jesp.2016.03.004CrossRefGoogle Scholar
Vines, T. H., Albert, A. Y. K., Andrew, R. L., Débarre, F., Bock, D. G., Franklin, M. T., … Rennison, D. J. (2014). The availability of research data declines rapidly with article age. Current Biology, 24(1), 9497. ScholarPubMed
Wagenmakers, E.-J., Dutilh, G., & Sarafoglou, A. (2018). The creativity-verification cycle in psychological science: New methods to combat old idols. PsyArXiv. doi:10.17605/OSF.IO/37NTPGoogle ScholarPubMed
Wagenmakers, E.-J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., et al. (2017). Bayesian inference for psychology. Part II: Example Applications with JASP. Psychonomic Bulletin & Review, July. doi:10.3758/s13423-017-1323-7Google Scholar
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61(7), 726728. doi:10.1037/0003-066X.61.7.72CrossRefGoogle ScholarPubMed


Beall, J. (2012). Predatory publishers are corrupting open access. Nature, 489, 179.CrossRefGoogle ScholarPubMed
Beall, J. (2016). Predatory journals: Ban predators from the scientific record. Nature, 534, 326.CrossRefGoogle ScholarPubMed
Berkeley, G. (n.d.) If a tree falls in the forest, Retrieved January 7, 2018 from
Bohannon, J. (2013). Who's afraid of peer review? Science, 342, 6065.CrossRefGoogle ScholarPubMed
Björk, B.-C. (2017). Gold, green, and black open access. Learned Publishing. doi:10.1002/leap.1096CrossRefGoogle Scholar
Enserink, M. (2016). In dramatic statement, European leaders call for “immediate” open access to all scientific papers by 2020. Science. doi:10.1126/science.aag0577Google Scholar
Eysenbach, G. (2006). Citation advantages of open access articles. PLoS Biology, 4(5), e157.CrossRefGoogle Scholar
Fanelli, D. (2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891904.CrossRefGoogle Scholar
Franco, A., Malhorta, N., & Simonovits, G. (2017). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 15021505.CrossRefGoogle Scholar
Harnad, S., & Brody, T. (2004). Comparing the impact of open access (OA) vs.non-OA articles in the same journals. D-Lib Magazine, 10(6). Scholar
Himmelstein, D. S., Romero, A. R., Levernier, J. G., Munro, T. A., McLaughlin, S. R., Tzovaras, B. G., & Greene, C. S. (2017). Sci-hub provides access to nearly all scholarly literature. eLife 2018;7:e32822 DOI: 10.7554/eLife.32822CrossRef
Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124.CrossRefGoogle ScholarPubMed
Laakso, M., and Björk, B.-C. (2012). Anatomy of open access publishing: A study of the longitudinal development and internal structure. BMC Medicine, 10, 124.CrossRefGoogle ScholarPubMed
Lawrence, S. (2001). Online or invisible. Nature, 411, 521.CrossRefGoogle ScholarPubMed
Nicholson, J., & Pepe, A. (2016) 65 out of the 100 most cited papers are paywalled. Retrieved January 18, 2018 from
Pashler, H., & Harris, C. R. (2012). Is the replicability crisis overblown? Three arguments examined. Perspectives in Psychological Science, 7, 531536.CrossRefGoogle ScholarPubMed
Pashler, H., & Wagenmakers, E.-J. (2012). Editors’ introduction to the Special Section on Replicability in Psychological Science. Perspectives in Psychological Science, 7(6), 528530.CrossRefGoogle ScholarPubMed
Pinfield, S., Salter, J., & Bath, P. A. (2016). The “total cost of publication” in a hybrid open-access environment: Institutional approaches to funding journal article-processing charges in combination with subscriptions. Journal of the Association for Information Science and Technology, 67(7), 17511766.CrossRefGoogle Scholar
Shen, C., & Björk, B.-C. (2015). “Predatory” open access: A longitudinal study of article volumes and market characteristics. BMC Medicine, 13, 230.CrossRefGoogle ScholarPubMed
Sorokowski, P., Kulczycki, E., Sorokowska, A., & Pisanski, K. (2017), Predatory journals recruit fake editor. Nature, 543, 481483.CrossRef
Suber, P. (2012). Open access. Cambridge, MA: MIT Press.Google ScholarPubMed
Wicherts, J. M., Borsboom, D., Kats, J., & Molenaar, D. (2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61(7), 726728.CrossRefGoogle ScholarPubMed


Sternberg, R. J., & Fiske, S. E. (Eds.) (2015). Ethical challenges in the behavioral and brain sciences: Case studies and commentaries. New York: Cambridge University Press.CrossRefGoogle Scholar

Save book to Kindle

To save this book to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the or variations. ‘’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats

Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

Available formats

Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

Available formats