Experimental Instruments for Measuring Cognitive Biases in Financial Decision-Making: A Systematic Review and Methodological Assessment

Héctor Madariaga Tapia; Marta Tostes Vieira

doi:10.5281/zenodo.23103092

Authors

Héctor Madariaga Tapia Pontificia Universidad Católica del Perú https://orcid.org/0009-0002-9215-3778
Marta Tostes Vieira Pontificia Universidad Católica del Perú https://orcid.org/0000-0003-0855-7463

DOI:

https://doi.org/10.5281/zenodo.23103092

Keywords:

Cognitive biases, Behavioral finance, Experimental methods, Measurement instruments, Systematic review, Methodological transparency

Abstract

This study aims to systematically review and evaluate the experimental instruments used to measure cognitive biases in financial decision-making, examining their methodological characteristics and implications for the comparability and transparency of behavioral evidence. Following the PRISMA 2020 guidelines, a systematic review covered the period from 1990 to 2025 and yielded a final corpus of 70 studies. The analysis employed a three-component framework: descriptive extraction (C1), methodological robustness assessment (D1), and procedural reporting audit (E1). Inter-rater agreement was assessed using Cohen’s Kappa. Association and temporal analyses incorporated the Holm adjustment for multiple comparisons.

Experimental tasks and simulations accounted for 94.3% of the studies. The methodological dimensions exhibited heterogeneous profiles: several criteria predominantly received High classifications, although five showed severe ceiling effects, while explicitly documented reliability evidence remained limited and reporting of procedural practices varied across studies. Sensitivity analysis indicated that the methodological profile remained heterogeneous after excluding the five criteria affected by severe ceiling effects from the integrative interpretation.

None of the eight contrasts provided sufficient statistical evidence after the Holm adjustment, although some revealed relevant descriptive patterns and association magnitudes. The review relies on the information reported in the primary studies and on the search and eligibility criteria applied. The C1–D1–E1 framework was developed specifically for this review and has not undergone external validation.

More systematic documentation of measurement properties and procedural practices may strengthen the transparency, comparability, and reproducibility of experimental research in behavioral finance. The study shifts attention from which cognitive biases researchers examine to how they measure such biases experimentally, integrating descriptive extraction, methodological assessment, and procedural auditing. It also provides an empirical foundation for a conceptual proposal encompassing technological, measurement, statistical, and ethical transparency.

Downloads

Download data is not yet available.

References

Aromataris, E., Lockwood, C., Porritt, K., Pilla, B., & Jordan, Z. (Eds.). (2024). JBI manual for evidence synthesis. JBI. https://doi.org/10.46658/JBIMES-24-01

Badola, S., Sahu, A. K., & Adlakha, A. (2024). A systematic review on behavioral biases affecting individual investment decisions. Qualitative Research in Financial Markets, 16(3), 448-476. https://doi.org/10.1108/QRFM-05-2022-0095

Barberis, N., & Thaler, R. (2003). A survey of behavioral finance. En G. M. Constantinides, M. Harris, & R. M. Stulz (Eds.), Handbook of the Economics of Finance (Vol. 1, pp. 1053-1128). Elsevier. https://doi.org/10.1016/S1574-0102(03)01027-6

Biais, B., Hilton, D., Mazurier, K., & Pouget, S. (2005). Judgemental overconfidence, self-monitoring and trading performance in an experimental financial market. The Review of Economic Studies, 72(2), 287-312. https://doi.org/10.1111/j.1467-937X.2005.00333.x

Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., Heikensten, E., Holzmeister, F., Imai, T., Isaksson, S., Nave, G., Pfeiffer, T., Razen, M., & Wu, H. (2016). Evaluating replicability of laboratory experiments in economics. Science, 351(6280), 1433-1436. https://doi.org/10.1126/science.aaf0918

Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637-644. https://doi.org/10.1038/s41562-018-0399-z

Camerer, C. F., & Hogarth, R. M. (1999). The effects of financial incentives in experiments: A review and capital-labor-production framework. Journal of Risk and Uncertainty, 19, 7-42. https://doi.org/10.1023/A:1007850605129

Campbell, M., McKenzie, J. E., Sowden, A., Katikireddi, S. V., Brennan, S. E., Ellis, S., Hartmann-Boyce, J., Ryan, R., Shepperd, S., Thomas, J., Welch, V., & Thomson, H. (2020). Synthesis without meta-analysis (SWiM) in systematic reviews: Reporting guideline. BMJ, 368, 16890. https://doi.org/10.1136/bmj.l6890

Christensen, G., & Miguel, E. (2018). Transparency, reproducibility, and the credibility of economics research. Journal of Economic Literature, 56(3), 920-980. https://doi.org/10.1257/jel.20171350

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104

Critical Appraisal Skills Programme. (2018). CASP checklists. https://casp-uk.net/casp-tools-checklists/

Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281-302. https://doi.org/10.1037/h0040957

Crosetto, P., & Filippin, A. (2016). A theoretical and experimental appraisal of four risk elicitation methods. Experimental Economics, 19, 613-641. https://doi.org/10.1007/s10683-015-9457-9

De Oliveira Cardoso, N., Zorgi Salvador, E., Broch, G., Mette, F. M. B., Yoshinaga, C. E., & De Lara Machado, W. (2024). Measuring behavioral biases in individual investors decision-making and sociodemographic correlations: A systematic review. Qualitative Research in Financial Markets, 16(4), 636-659. https://doi.org/10.1108/QRFM-05-2022-0090

Falk, A., & Heckman, J. J. (2009). Lab experiments are a major source of knowledge in the social sciences. Science, 326(5952), 535-538. https://doi.org/10.1126/science.1168244

Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3(4), 456-465. https://doi.org/10.1177/2515245920952393

Frey, R., Pedroni, A., Mata, R., Rieskamp, J., & Hertwig, R. (2017). Risk preference shares the psychometric structure of major psychological traits. Science Advances, 3(10), e1701381. https://doi.org/10.1126/sciadv.1701381

Glaser, M., & Weber, M. (2007). Overconfidence and trading volume. The Geneva Risk and Insurance Review, 32(1), 1-36. https://doi.org/10.1007/s10713-007-0003-3

Harrison, G. W., & List, J. A. (2004). Field experiments. Journal of Economic Literature, 42(4), 1009-1055. https://doi.org/10.1257/0022051043004577

Hertwig, R., & Ortmann, A. (2001). Experimental practices in economics: A methodological challenge for psychologists? Behavioral and Brain Sciences, 24(3), 383-403. https://doi.org/10.1017/S0140525X01004149

Holt, C. A., & Laury, S. K. (2002). Risk aversion and incentive effects. American Economic Review, 92(5), 1644-1655. https://doi.org/10.1257/000282802762024700

Kahneman, D., & Tversky, A. (1979). Prospect theory: An analysis of decision under risk. Econometrica, 47(2), 263-292. https://doi.org/10.2307/1914185

Kumar, S., & Goyal, N. (2015). Behavioural biases in investment decision making – a systematic literature review. Qualitative Research in Financial Markets, 7(1), 88-108. https://doi.org/10.1108/QRFM-07-2014-0022

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. https://doi.org/10.2307/2529310

Levitt, S. D., & List, J. A. (2007). What do laboratory experiments measuring social preferences reveal about the real world? Journal of Economic Perspectives, 21(2), 153-174. https://doi.org/10.1257/jep.21.2.153

Lönnqvist, J.-E., Verkasalo, M., Walkowitz, G., & Wichardt, P. C. (2015). Measuring individual risk attitudes in the lab: Task or ask? An empirical comparison. Journal of Economic Behavior & Organization, 119, 254-266. https://doi.org/10.1016/j.jebo.2015.08.003

Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741-749. https://doi.org/10.1037/0003-066X.50.9.741

Moore, D. A., & Healy, P. J. (2008). The trouble with overconfidence. Psychological Review, 115(2), 502-517. https://doi.org/10.1037/0033-295X.115.2.502

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., Chou, R., Glanville, J., Grimshaw, J. M., Hróbjartsson, A., Lalu, M. M., Li, T., Loder, E. W., Mayo-Wilson, E., McDonald, S., … Moher, D. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. BMJ, 372, n71. https://doi.org/10.1136/bmj.n71

Pedroni, A., Frey, R., Bruhin, A., Dutilh, G., Hertwig, R., & Rieskamp, J. (2017). The risk elicitation puzzle. Nature Human Behaviour, 1(11), 803-809. https://doi.org/10.1038/s41562-017-0219-x

Smith, V. L. (1976). Experimental economics: Induced value theory. The American Economic Review, 66(2), 274-279. http://www.jstor.org/stable/1817233

Thaler, R. H. (2016). Behavioral economics: Past, present, and future. American Economic Review, 106(7), 1577-1600. https://doi.org/10.1257/aer.106.7.1577

Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases. Science, 185, 1124-1131.

Zahera, S. A., & Bansal, R. (2018). Do investors exhibit behavioral biases in investment decision making? A systematic review. Qualitative Research in Financial Markets, 10(2), 210-251. https://doi.org/10.1108/QRFM-04-2017-0028