Słowa kluczowe
kryzys replikacyjności
niewłaściwe praktyki badawcze
Jak cytować
Liczba wyświetleń: 23
Liczba pobrań: 12
Abstrakt
Znaczna część badań naukowych jest trudna lub nawet niemożliwa do replikowania lub odtworzenia, co określane jest mianem kryzysu replikacji. Jednym z czynników przyczyniających się do tego kryzysu jest niska jakość danych wykorzystywanych w badaniach. Często można to przypisać nieuważnym lub nietypowym respondentom. Eliminacja danych z tych grup może poprawić jakość danych badawczych i potencjalnie zwiększyć prawdopodobieństwo udanej replikacji. Eliminacja takich danych może czasami mieć skutek odwrotny. Metody wykrywania i usuwania nieuważnych i nietypowych respondentów różnią się znacznie, dlatego też dają różne wyniki i mogą być stosowane na wiele sposobów, dodając kolejny poziom złożoności w kontekście replikacji. Głównym celem artykułu jest wskazanie na zagrożenie tkwiące w posługiwaniu się różnymi metodami wykrywania nieuważnych i nietypowych odpowiedzi dla możliwości odtworzenia wyników badania. Artykuł podzielony jest na dwie części. W pierwszej omówiono zagadnienia związane ze źródłami kryzysu replikacji w naukach społecznych i potencjalnego wpływu metod wykrywania nieuważnych odpowiedzi respondentów na możliwości replikowania badań. W drugiej części, na podstawie analizy przypadku jednego z badań zamieszczonych w systemie Open Science Framework (OSF), pokazano, jak subtelny, a zarazem znaczący może być wpływ zastosowanych metod wykrywania i usuwania nieuważnych i nietypowych respondentów na powodzenie replikacji badań. W końcowej części artykułu wskazano na kroki mające na celu ograniczenie problemu z replikacją związaną z wykorzystaniem metod wykrywania nieuważnych i nietypowych respondentów.
Bibliografia
Agnoli, F., Wicherts, J. M., Veldkamp, C. L., Albiero, P., i Cubelli, R. (2017). Questionable research practices among Italian research psychologists. PloS ONE, 12(3), e0172792. DOI: https://doi.org/10.1371/journal.pone.0172792
Aguinis, H., Ramani, R. S., i Alabduljader, N. (2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12(1), 83–110. DOI: https://doi.org/10.5465/annals.2016.0011
Amrhein, V., Trafimow, D., i Greenland, S. (2019). Inferential statistics as descriptive statistics: There is no replication crisis if we don’t expect replication. The American Statistician, 73(sup1), 262–270. DOI: https://doi.org/10.1080/00031305.2018.1543137
Andrade, C. (2021). HARKing, cherry-picking, p-hacking, fishing expeditions, and data dredging and mining as questionable research practices. The Journal of Clinical Psychiatry, 82(1), e1–e3. DOI: https://doi.org/10.4088/JCP.20f13804
Baillie, M., Le Cessie, S., Schmidt, C. O., Lusa, L., Huebner, M., for the Topic Group “Initial Data Analysis” of the STRATOS Initiative. (2022). Ten simple rules for initial data analysis. PLoS Computational Biology, 18(2), e1009819. DOI: https://doi.org/10.1371/journal.pcbi.1009819
Bakker, B. N., Jaidka, K., Dörr, T., Fasching, N., i Lelkes, Y. (2021). Questionable and open research practices: Attitudes and perceptions among quantitative communication researchers. Journal of Communication, 71(5), 715–738. DOI: https://doi.org/10.1093/joc/jqab031
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., Abston, K. A., Bennett, A. A., i Adkins, C. L. (2016). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42(1), 5–20. DOI: https://doi.org/10.1177/0149206315619011
Benjamini, Y. (2020). Selective inference: The silent killer of replicability. Harvard Data Science Review, 2(4). DOI: https://doi.org/10.1162/99608f92.fc62b261
Białek, A., i Wolski, P. (2023). Dwa głosy o kryzysie wiarygodności w psychologii. Przegląd Psychologiczny, 66(1), 9–26. DOI: https://doi.org/10.31648/przegldpsychologiczny.9455
Bridges, A. J. (2022). Hypothesizing after results are known: HARKing. W W. O’Donohue, A. Masuda i S. Lilienfeld (red.), Avoiding questionable research practices in applied psychology (s. 175–190). Springer International Publishing. DOI: https://doi.org/10.1007/978-3-031-04968-2_8
Brzeziński, J. M. (2023). Czy kryzys wiarygodności w psychologii? Przegląd Psychologiczny, 66(1), 27–47. DOI: https://doi.org/10.31648/przegldpsychologiczny.9456
Bybee, S., Cloyes, K., Baucom, B., Supiano, K., Mooney, K., i Ellington, L. (2022). Bots and nots: Safeguarding online survey research with underrepresented and diverse populations. Psychology Sexuality, 13(4), 901–911. DOI: https://doi.org/10.1080/19419899.2021.1936617
Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Nave, G., Nosek, B. A., Pfeiffer, T., Altmejd, A., Buttrick, N., Chan, T., Chen, Y., Forsell, E., Gampa, A., Heikensten, E., Hummer, L., Imai, T., … Wu, H. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2(9), 637–644. DOI: https://doi.org/10.1038/s41562-018-0399-z
Cockburn, A., Dragicevic, P., Besançon, L., i Gutwin, C. (2020). Threats of a replication crisis in empirical computer science. Communications of the ACM, 63(8), 70–79. DOI: https://doi.org/10.1145/3360311
Coiera, E., Ammenwerth, E., Georgiou, A., i Magrabi, F. (2018). Does health informatics have a replication crisis? Journal of the American Medical Informatics Association, 25(8), 963–968. DOI: https://doi.org/10.1093/jamia/ocy028
Cornell, D., Klein, J., Konold, T., i Huang, F. (2012). Effects of validity screening items on adolescent survey data. Psychological Assessment, 24(1), 21–35. DOI: https://doi.org/10.1037/a0024824
Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70(4), 596–612. DOI: https://doi.org/10.1177/0013164410366686
Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. DOI: https://doi.org/10.1016/j.jesp.2015.07.006
DeSimone, J. A., i Harms, P. D. (2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33(5), 559–577. DOI: https://doi.org/10.1007/s10869-017-9514-9
Dunn, A. M., Heggestad, E. D., Shanock, L. R., i Theilgard, N. (2018). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 33(1), 105–121. DOI: https://doi.org/10.1007/s10869-016-9479-0
Duvendack, M., Palmer-Jones, R., i Reed, W. R. (2017). What is meant by “replication” and why does it encounter resistance in economics? American Economic Review, 107(5), 46–51. DOI: https://doi.org/10.1257/aer.p20171031
Ellis, R. J. (2022). Questionable research practices, low statistical power, and other obstacles to replicability: Why preclinical neuroscience research would benefit from registered reports. Eneuro, 9(4), ENEURO.0017-22.2022. DOI: https://doi.org/10.1523/ENEURO.0017-22.2022
Fox, C. R., i Tversky, A. (1995). Ambiguity aversion and comparative ignorance. The Quarterly Journal of Economics, 110(3), 585–603. DOI: https://doi.org/10.2307/2946693
Franco, A., Malhotra, N., i Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505. DOI: https://doi.org/10.1126/science.1255484
Franco, A., Malhotra, N., i Simonovits, G. (2016). Underreporting in psychology experiments: Evidence from a study registry. Social Psychological and Personality Science, 7(1), 8–12. DOI: https://doi.org/10.1177/1948550615598377
Freese, J., i Peterson, D. (2017). Replication in social science. Annual Review of Sociology, 43(1), 147–165. DOI: https://doi.org/10.1146/annurev-soc-060116-053450
Hayes, A. F. (2018). Introduction to mediation: A regression-based approach. Guilford Press.
He, J., Van de Vijver, F. J., Espinosa, A. D., i Mui, P. H. (2014). Toward a unification of acquiescent, extreme, and midpoint response styles: A multilevel study. International Journal of Cross Cultural Management, 14(3), 306–322. DOI: https://doi.org/10.1177/1470595814541424
Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., i DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114. DOI: https://doi.org/10.1007/s10869-011-9231-8
Hudson, R. (2023). Explicating exact versus conceptual replication. Erkenntnis, 88(6), 2493–2514. DOI: https://doi.org/10.1007/s10670-021-00464-z
Ioannidis, J. P. (2008). Why most discovered true associations are inflated. Epidemiology, 19, 640–648. DOI: https://doi.org/10.1097/EDE.0b013e31818131e7
Kam, C. C. S., Meyer, J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512–541. DOI: https://doi.org/10.1177/1094428115571894
Kerr, N. L. (1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. DOI: https://doi.org/10.1207/s15327957pspr0203_4
Krosnick, J. A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. DOI: https://doi.org/10.1002/acp.2350050305
Lakens, D. (2015). On the challenges of drawing conclusions from p-values just below 0.05. PeerJ, 3, e1142. https://peerj.com/articles/1142/ DOI: https://doi.org/10.7717/peerj.1142
Lakens, D., i Evers, E. R. K. (2014). Sailing from the seas of chaos into the corridor of stability: Practical recommendations to increase the informational value of studies. Perspectives on Psychological Science, 9(3), 278–292. DOI: https://doi.org/10.1177/1745691614528520
Leichtmann, B., Nitsch, V., i Mara, M. (2022). Crisis ahead? Why human-robot interaction user studies may have replicability problems and directions for improvement. Frontiers in Robotics and AI, 9, 838116. DOI: https://doi.org/10.3389/frobt.2022.838116
Leys, C., Delacre, M., Mora, Y. L., Lakens, D., i Ley, C. (2019). How to classify, detect, and manage univariate and multivariate outliers, with emphasis on pre-registration. International Review of Social Psychology, 32(1, Article 5. DOI: https://doi.org/10.5334/irsp.289
Lynch Jr, J. G., Bradlow, E. T., Huber, J. C., i Lehmann, D. R. (2015). Reflections on the replication corner: In praise of conceptual replications. International Journal of Research in Marketing, 32(4), 333–342. DOI: https://doi.org/10.1016/j.ijresmar.2015.09.006
Maniaci, M. R., i Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83. Maxwell, S. E., Lau, M. Y., i Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean? American Psychologist, 70(6), 487–498. DOI: https://doi.org/10.1037/a0039400
Meade, A. W., i Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. DOI: https://doi.org/10.1037/a0028085
Muradchanian, J., Hoekstra, R., Kiers, H., i van Ravenzwaaij, D. (2023). The role of results in deciding to publish: A direct comparison across authors, reviewers, and editors based on an online survey. PloS ONE, 18(10), e0292279. DOI: https://doi.org/10.1371/journal.pone.0292279
Muthukrishna, M., Bell, A. V., Henrich, J., Curtin, C. M., Gedranovich, A., McInerney, J., i Thue, B. (2020). Beyond Western, educated, industrial, rich, and democratic (WEIRD) psychology: Measuring and mapping scales of cultural and psychological distance. Psychological Science, 31(6), 678–701. DOI: https://doi.org/10.1177/0956797620916782
Niessen, A. S. M., Meijer, R. R., i Tendeiro, J. N. (2016). Detecting careless respondents in webbased questionnaires: Which method to use? Journal of Research in Personality, 63, 1–11. DOI: https://doi.org/10.1016/j.jrp.2016.04.010
Nosek, B. A., i Errington, T. M. (2020). What is replication? PLoS Biology, 18(3), e3000691. DOI: https://doi.org/10.1371/journal.pbio.3000691
Nosek, B. A., Hardwicke, T. E., Moshontz, H., Allard, A., Corker, K. S., Dreber, A., Fidler, F., Hilgard, J., Kline Struhl, M., Nuijten, M. B., Rohrer, J. M., Romero, F., Scheel, A. M., Scherer, L. D., Schönbrodt, F. D., i Vazire, S. (2022). Replicability, robustness, and reproducibility in psychological science. Annual Review of Psychology, 73(1), 719–748. DOI: https://doi.org/10.1146/annurev-psych-020821-114157
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. DOI: https://doi.org/10.1126/science.aac4716
Oppenheimer, D. M., Meyvis, T., i Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. DOI: https://doi.org/10.1016/j.jesp.2009.03.009
Pagell, M. (2021). Replication without repeating ourselves: Addressing the replication crisis in operations and supply chain management research. Journal of Operations Management, 67(1), 105–115. DOI: https://doi.org/10.1002/joom.1120
Peels, R., i Bouter, L. (2018). The possibility and desirability of replication in the humanities. Palgrave Communications, 4(1), 95. DOI: https://doi.org/10.1057/s41599-018-0149-x
Pfundmair, M, i Lermer, E. (2023) Examining the link between social exclusion and social-risk taking: A correlational and experimental investigation [version 1; peer review: 1 approved with reservations], Routledge Open Research, 2023, 2:4 Last updated: 28 Mar 2023. https://osf.io/umrht DOI: https://doi.org/10.12688/routledgeopenres.17729.1
Ravn, T., i Sørensen, M. P. (2021). Exploring the gray area: Similarities and differences in questionable research practices (QRPs) across main areas of research. Science and Engineering Ethics, 27(4), 40. DOI: https://doi.org/10.1007/s11948-021-00310-z
Richard, F. D., Bond, C. F., i Stokes-Zoota, J. J. (2003). One hundred years of social psychology quantitatively described. Review of General Psychology, 7(4), 331–363. Saad, D. (2021). Nowe narzędzia i techniki zwiększające trafność badań internetowych. Com.Press, 4(1), 106–121. DOI: https://doi.org/10.1037/1089-2680.7.4.331
Scheel, A. M., Schijen, M. R. M. J., i Lakens, D. (2021). An excess of positive results: Comparing the standard psychology literature with registered reports. Advances in Methods and Practices in Psychological Science, 4(2), 1–12. DOI: https://doi.org/10.1177/25152459211007467
Stodden, V. (2015). Reproducing Statistical Results. Annual Review of Statistics and Its Application, 2(1), 1–19. DOI: https://doi.org/10.1146/annurev-statistics-010814-020127
Świątkowski, W., i Dompnier, B. (2017). Replicability crisis in social psychology: Looking at the past to find new pathways for the future. International Review of Social Psychology, 30(1), 111–124. DOI: https://doi.org/10.5334/irsp.66
Tincani, M., i Travers, J. (2019). Replication research, publication bias, and applied behavior analysis. Perspectives on Behavior Science, 42(1), 59–75. DOI: https://doi.org/10.1007/s40614-019-00191-5
Vecchio, R., Caso, G., Cembalo, L., i Borrello, M. (2020). Is respondents’ inattention in online surveys a major issue for research? Economia Agro-Alimentare, 1, 1–18. DOI: https://doi.org/10.3280/ecag1-2020oa10069
Vow els, M. J. (2021). Misspecification and unreliable interpretations in psychology and social science. Psychological Methods, 28, 507–526. DOI: https://doi.org/10.1037/met0000429
Wanat, T. (2010). Niedoskonalości w formulowaniu hipotez badawczych w pracach doktorskich. Zagadnienia Naukoznawstwa, 46(1), 27–41.
Ward, M., i Meade, A. W. (2023). Dealing with careless responding in survey data: Prevention, identification, and recommended best practices. Annual Review of Psychology, 74, 577–596. DOI: https://doi.org/10.1146/annurev-psych-040422-045007
Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., Van Aert, R. C. M., i Van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1832. DOI: https://doi.org/10.3389/fpsyg.2016.01832
Yang, S., Wang, L., Ding, P. (2019). Causal inference with confounders missing not at random. Biometrika, 106(4), 875–888. DOI: https://doi.org/10.1093/biomet/asz048
Licencja
Prawa autorskie (c) 2024 WPiA UAM
Utwór dostępny jest na licencji Creative Commons Uznanie autorstwa 4.0 Międzynarodowe.