No CrossRef data available.
Article contents
Measurement validity and the integrative approach
Published online by Cambridge University Press: 05 February 2024
Abstract
Almaatouq et al. propose a novel integrative approach to experiments. We provide three examples of how unaddressed measurement issues threaten the feasibility of the approach and its promise of promoting commensurability and knowledge integration.
- Type
- Open Peer Commentary
- Information
- Copyright
- Copyright © The Author(s), 2024. Published by Cambridge University Press
References
Alexandrova, A., & Haybron, D. M. (2016). Is construct validation valid? Philosophy of Science, 83(5), 1098–1109. https://doi.org/10.1086/687941CrossRefGoogle Scholar
Almaatouq, A., Alsobay, M., Yin, M., & Watts, D. J. (2021). Task complexity moderates group synergy. Proceedings of the National Academy of Sciences of the United States of America, 118(36), Article e2101062118. https://doi.org/10.1073/pnas.2101062118CrossRefGoogle ScholarPubMed
Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., … Rahwan, I. (2018). The moral machine experiment. Nature, 563(7729), 59–64. https://doi.org/10.1038/s41586-018-0637-6CrossRefGoogle ScholarPubMed
Awad, E., Dsouza, S., Kim, R., Schulz, J., Henrich, J., Shariff, A., … Rahwan, I. (2020). Reply to: Life and death decisions of autonomous vehicles. Nature, 579(7797), E3–E5. https://doi.org/10.1038/s41586-020-1988-3CrossRefGoogle ScholarPubMed
Bauman, C. W., McGraw, A. P., Bartels, D. M., & Warren, C. (2014). Revisiting external validity: Concerns about trolley problems and other sacrificial dilemmas in moral psychology. Social and Personality Psychology Compass, 8(9), 536–554. https://doi.org/10.1111/spc3.12131CrossRefGoogle Scholar
Bigman, Y. E., & Gray, K. (2020). Life and death decisions of autonomous vehicles. Nature, 579(7797), E1–E2. https://doi.org/10.1038/s41586-020-1987-4CrossRefGoogle ScholarPubMed
Bostyn, D. H., Sevenhant, S., & Roets, A. (2018). Of mice, men, and trolleys: Hypothetical judgment versus real-life behavior in trolley-style moral dilemmas. Psychological Science, 29(7), 1084–1093. https://doi.org/10.1177/0956797617752640CrossRefGoogle ScholarPubMed
Engel, D., Woolley, A. W., Jing, L. X., Chabris, C. F., & Malone, T. W. (2014). Reading the mind in the eyes or reading between the lines? Theory of mind predicts collective intelligence equally well online and face-to-face. PLoS ONE, 9(12), e115212. https://doi.org/10.1371/journal.pone.0115212CrossRefGoogle ScholarPubMed
Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456–465. https://doi.org/10.1177/2515245920952393CrossRefGoogle Scholar
Fried, E. I., Flake, J. K., & Robinaugh, D. J. (2022). Revisiting the theoretical and methodological foundations of depression measurement. Nature Reviews Psychology, 1(6), 358–368. https://doi.org/10.1038/s44159-022-00050-2CrossRefGoogle ScholarPubMed
Herrman, H., Kieling, C., McGorry, P., Horton, R., Sargent, J., & Patel, V. (2019). Reducing the global burden of depression: A Lancet–World Psychiatric Association Commission. The Lancet, 393(10189), e42–e43. https://doi.org/10.1016/S0140-6736(18)32408-5CrossRefGoogle ScholarPubMed
Hester, N., Axt, J. R., Siemers, N., & Hehman, E. (2023). Evaluating validity properties of 25 race-related scales. Behavior Research Methods, 55(4), 1758–1777. https://doi.org/10.3758/s13428-022-01873-wCrossRefGoogle ScholarPubMed
Higgins, W. C., Ross, R. M., Langdon, R., & Polito, V. (2023). The “reading the mind in the eyes” test shows poor psychometric properties in a large, demographically representative U.S. sample. Assessment, 30(6), 1777–1789. https://doi.org/10.1177/10731911221124342CrossRefGoogle Scholar
Higgins, W. C., Ross, R. M., Polito, V., & Kaplan, D. M. (2023). Three threats to the validity of the reading the mind in the eyes test: A commentary on Pavolova and Sokolov (2022). Neuroscience and Biobehavioral Reviews, 147, 105088. https://doi.org/10.1016/j.neubiorev.2023.105088CrossRefGoogle Scholar
Kim, Y. J., Engel, D., Woolley, A. W., Lin, J. Y.-T., McArthur, N., & Malone, T. W. (2017). What makes a strong team? Using collective intelligence to predict team performance in League of Legends. In Proceedings of the 2017 ACM conference on computer supported cooperative work and social computing (pp. 2316–2329). Association for Computing Machinery, Portland, Oregon, USA. https://doi.org/10.1145/2998181.2998185CrossRefGoogle Scholar
Kittel, A. F. D., Olderbak, S., & Wilhelm, O. (2022). Sty in the mind's eye: A meta-analytic investigation of the nomological network and internal consistency of the “reading the mind in the eyes” test. Assessment, 29(5), 872–895. https://doi.org/10.1177/1073191121996469CrossRefGoogle Scholar
Lilienfeld, S. O., & Strother, A. N. (2020). Psychological measurement and the replication crisis: Four sacred cows. Canadian Psychology, 61(4), 281–288. https://doi.org/10.1037/cap0000236CrossRefGoogle Scholar
Oakley, B. F., Brewer, R., Bird, G., & Catmur, C. (2016). Theory of mind is not theory of emotion: A cautionary note on the reading the mind in the eyes test. Journal of Abnormal Psychology, 125(6), 818–823. https://doi.org/10.1037/abn0000182CrossRefGoogle Scholar
Olderbak, S., Wilhelm, O., Olaru, G., Geiger, M., Brenneman, M. W., & Roberts, R. D. (2015). A psychometric analysis of the reading the mind in the eyes test: Toward a brief form for research and applied settings. Frontiers in Psychology, 6, 1503. https://doi.org/10.3389/fpsyg.2015.01503CrossRefGoogle Scholar
Silverman, C. (2022). How to read “reading the mind in the eyes”. Notes and Records of the Royal Society of London, 76(4), 683–697. https://doi.org/10.1098/rsnr.2021.0058CrossRefGoogle Scholar
Stevanovic, D., Jafari, P., Knez, R., Franic, T., Atilola, O., Davidovic, N., … Lakic, A. (2017). Can we really use available scales for child and adolescent psychopathology across cultures? A systematic review of cross-cultural measurement invariance data. Transcultural Psychiatry, 54(1), 125–152. https://doi.org/10.1177/1363461516689215CrossRefGoogle ScholarPubMed
Vazire, S., Schiavone, S. R., & Bottesini, J. G. (2022). Credibility beyond replicability: Improving the four validities in psychological science. Current Directions in Psychological Science, 31(2), 162–168. https://doi.org/10.1177/09637214211067779CrossRefGoogle Scholar
Woolley, A. W., Chabris, C. F., Pentland, A., Hashmi, N., & Malone, T. W. (2010). Evidence for a collective intelligence factor in the performance of human groups. Science (New York, N.Y.), 330(6004), 686–688. https://doi.org/10.1126/science.1193147CrossRefGoogle ScholarPubMed
Target article
Beyond playing 20 questions with nature: Integrative experiment design in the social and behavioral sciences
Related commentaries (31)
Against naïve induction from experimental data
Are language–cognition interactions bigger than a breadbox? Integrative modeling and design space thinking temper simplistic questions about causally dense phenomena
Assume a can opener
Beyond integrative experiment design: Systematic experimentation guided by causal discovery AI
Commensurability engineering is first and foremost a theoretical exercise
Confidence in research findings depends on theory
Consensus meetings will outperform integrative experiments
Dimensional versus conceptual incommensurability in the social and behavioral sciences
Discovering the unknown unknowns of research cartography with high-throughput natural description
Diversity of contributions is not efficient but is essential for science
Don't let perfect be the enemy of better: In defense of unparameterized megastudies
Eliminativist induction cannot be a solution to psychology's crisis
Experiment commensurability does not necessitate research consolidation
Explore your experimental designs and theories before you exploit them!
Getting lost in an infinite design space is no solution
Individual differences do matter
Integrative design for thought-experiments
Integrative experiments require a shared theoretical and methodological basis
Is generalization decay a fundamental law of psychology?
Measurement validity and the integrative approach
Neuroadaptive Bayesian optimisation can allow integrative design spaces at the individual level in the social and behavioural sciences… and beyond
Phenomena complexity, disciplinary consensus, and experimental versus correlational research in psychological science
Representative design: A realistic alternative to (systematic) integrative design
Sampling complex social and behavioral phenomena
Some problems with zooming out as scientific reform
Test many theories in many ways
The elephant's other legs: What some sciences actually do
The future of experimental design: Integrative, but is the sample diverse enough?
The miss of the framework
The social sciences needs more than integrative experimental designs: We need better theories
There are no shortcuts to theory
Author response
Replies to commentaries on beyond playing 20 questions with nature