Dr Patricio Saavedra


Curriculum vitae



Instituto de Ciencias Sociales

Universidad de O'Higgins



Many Labs 2: Investigating Variation in Replicability Across Samples and Settings


Journal article


R. A. Klein, M. Vianello, F. Hasselman, B. Adams, Reginald B. Adams, Sinan Alper, M. Aveyard, Jordan R. Axt, Mayowa T. Babalola, Š. Bahník, R. Batra, M. Berkics, M. Bernstein, D. Berry, Olga Bialobrzeska, Evans Dami Binan, K. Bocian, M. Brandt, R. Busching, A. Rédei, Huajian Cai, Fanny Cambier, K. Cantarero, Cheryl L. Carmichael, F. Céric, Jesse J. Chandler, Jen-Ho Chang, A. Chatard, E. Chen, Winnee Cheong, D. Cicero, S. Coen, Jennifer A. Coleman, Brian Collisson, M. Conway, Katherine S. Corker, P. Curran, F. Cushman, Z. Dagona, Ilker Dalgar, A. D. Rosa, W. E. Davis, M. D. Bruijn, Leander De Schutter, T. Devos, M. D. Vries, Canay Doğulu, Nerisa Dozo, K. Dukes, Yarrow Dunham, K. Durrheim, C. Ebersole, J. Edlund, Anja Eller, A. English, C. Finck, Natalia Frankowska, M. Freyre, Mike Friedman, E. Galliani, Joshua Chiroma Gandi, Tanuka Ghoshal, S. Giessner, Tripat Gill, Timo Gnambs, Ángel Gómez, Roberto González, J. Graham, Jon E. Grahe, Ivan Grahek, Eva Green, Kakul Hai, M. Haigh, Elizabeth L. Haines, Michael P. Hall, Marie E. Heffernan, Joshua A. Hicks, P. Houdek, Jeffrey R. Huntsinger, H. Huynh, H. Ijzerman, Y. Inbar, Åse Innes-Ker, William Jiménez-Leal, Melissa-Sue John, Jennifer A. Joy-Gaba, Roza G. Kamiloğlu, H. Kappes, Serdar Karabati, H. Karick, Victor N. Keller, Anna Kende, Nicolas O. Kervyn, G. Knežević, Carrie Kovacs, Lacy E. Krueger, German Kurapov, J. Kurtz, D. Lakens, L. Lazarević, C. Levitan, N. Lewis, S. Lins, Nikolette P. Lipsey, J. Losee, E. Maassen, Angela T. Maitner, W. Malingumu, Robyn K. Mallett, S. Marotta, Janko Međedović, Fernando Mena-Pacheco, T. Milfont, Wendy L. Morris, Sean C. Murphy, A. Myachykov, N. Neave, K. Neijenhuijs, A. J. Nelson, Félix Neto, Austin Lee Nichols, A. Ocampo, S. O'Donnell, Haruka Oikawa, M. Oikawa, E. Ong, Gábor Orosz, Malgorzata Osowiecka, Grant Packard, Rolando Pérez-Sánchez, B. Petrović, Ronaldo Pilati, B. Pinter, Lysandra Podesta, Gabrielle Pogge, M. Pollmann, Abraham M. Rutchick, Patricio Saavedra, Alexander K Saeri, E. Salomon, Kathleen Schmidt, Felix D. Schönbrodt, M. Sekerdej, David Sirlopú, Jeanine L. M. Skorinko, M. A. Smith, V. Smith-Castro, K. Smolders, A. Sobkow, W. Sowden, Philipp Spachtholz, M. Srivastava, Troy G Steiner, J. Stouten, Chris N. H. Street, Oskar K. Sundfelt, S. Szeto, E. Szumowska, Andrew C. W. Tang, Norbert K Tanzer, Morgan J. Tear, J. Theriault, M. Thomae, David Torres, Jakub Traczyk, J. Tybur, A. Ujhelyi, R. V. Aert, M. V. Assen, M. Hulst, P. Lange, A. V. Veer, Alejandro Echeverría, L. Vaughn, A. Vázquez, L. D. Vega, Catherine Verniers, M. Verschoor, Ingrid Voermans, M. Vranka, Cheryl A. Welch, A. Wichman, L. Williams, M. Wood, Julie A. Woodzicka, Marta Wrońska, L. Young, J. Zelenski, Zeng Zhi-jia, Brian A. Nosek
Advances in Methods and Practices in Psychological Science, 2018

Semantic Scholar DOI
Cite

Cite

APA   Click to copy
Klein, R. A., Vianello, M., Hasselman, F., Adams, B., Adams, R. B., Alper, S., … Nosek, B. A. (2018). Many Labs 2: Investigating Variation in Replicability Across Samples and Settings. Advances in Methods and Practices in Psychological Science.


Chicago/Turabian   Click to copy
Klein, R. A., M. Vianello, F. Hasselman, B. Adams, Reginald B. Adams, Sinan Alper, M. Aveyard, et al. “Many Labs 2: Investigating Variation in Replicability Across Samples and Settings.” Advances in Methods and Practices in Psychological Science (2018).


MLA   Click to copy
Klein, R. A., et al. “Many Labs 2: Investigating Variation in Replicability Across Samples and Settings.” Advances in Methods and Practices in Psychological Science, 2018.


BibTeX   Click to copy

@article{r2018a,
  title = {Many Labs 2: Investigating Variation in Replicability Across Samples and Settings},
  year = {2018},
  journal = {Advances in Methods and Practices in Psychological Science},
  author = {Klein, R. A. and Vianello, M. and Hasselman, F. and Adams, B. and Adams, Reginald B. and Alper, Sinan and Aveyard, M. and Axt, Jordan R. and Babalola, Mayowa T. and Bahník, Š. and Batra, R. and Berkics, M. and Bernstein, M. and Berry, D. and Bialobrzeska, Olga and Binan, Evans Dami and Bocian, K. and Brandt, M. and Busching, R. and Rédei, A. and Cai, Huajian and Cambier, Fanny and Cantarero, K. and Carmichael, Cheryl L. and Céric, F. and Chandler, Jesse J. and Chang, Jen-Ho and Chatard, A. and Chen, E. and Cheong, Winnee and Cicero, D. and Coen, S. and Coleman, Jennifer A. and Collisson, Brian and Conway, M. and Corker, Katherine S. and Curran, P. and Cushman, F. and Dagona, Z. and Dalgar, Ilker and Rosa, A. D. and Davis, W. E. and Bruijn, M. D. and Schutter, Leander De and Devos, T. and Vries, M. D. and Doğulu, Canay and Dozo, Nerisa and Dukes, K. and Dunham, Yarrow and Durrheim, K. and Ebersole, C. and Edlund, J. and Eller, Anja and English, A. and Finck, C. and Frankowska, Natalia and Freyre, M. and Friedman, Mike and Galliani, E. and Gandi, Joshua Chiroma and Ghoshal, Tanuka and Giessner, S. and Gill, Tripat and Gnambs, Timo and Gómez, Ángel and González, Roberto and Graham, J. and Grahe, Jon E. and Grahek, Ivan and Green, Eva and Hai, Kakul and Haigh, M. and Haines, Elizabeth L. and Hall, Michael P. and Heffernan, Marie E. and Hicks, Joshua A. and Houdek, P. and Huntsinger, Jeffrey R. and Huynh, H. and Ijzerman, H. and Inbar, Y. and Innes-Ker, Åse and Jiménez-Leal, William and John, Melissa-Sue and Joy-Gaba, Jennifer A. and Kamiloğlu, Roza G. and Kappes, H. and Karabati, Serdar and Karick, H. and Keller, Victor N. and Kende, Anna and Kervyn, Nicolas O. and Knežević, G. and Kovacs, Carrie and Krueger, Lacy E. and Kurapov, German and Kurtz, J. and Lakens, D. and Lazarević, L. and Levitan, C. and Lewis, N. and Lins, S. and Lipsey, Nikolette P. and Losee, J. and Maassen, E. and Maitner, Angela T. and Malingumu, W. and Mallett, Robyn K. and Marotta, S. and Međedović, Janko and Mena-Pacheco, Fernando and Milfont, T. and Morris, Wendy L. and Murphy, Sean C. and Myachykov, A. and Neave, N. and Neijenhuijs, K. and Nelson, A. J. and Neto, Félix and Nichols, Austin Lee and Ocampo, A. and O'Donnell, S. and Oikawa, Haruka and Oikawa, M. and Ong, E. and Orosz, Gábor and Osowiecka, Malgorzata and Packard, Grant and Pérez-Sánchez, Rolando and Petrović, B. and Pilati, Ronaldo and Pinter, B. and Podesta, Lysandra and Pogge, Gabrielle and Pollmann, M. and Rutchick, Abraham M. and Saavedra, Patricio and Saeri, Alexander K and Salomon, E. and Schmidt, Kathleen and Schönbrodt, Felix D. and Sekerdej, M. and Sirlopú, David and Skorinko, Jeanine L. M. and Smith, M. A. and Smith-Castro, V. and Smolders, K. and Sobkow, A. and Sowden, W. and Spachtholz, Philipp and Srivastava, M. and Steiner, Troy G and Stouten, J. and Street, Chris N. H. and Sundfelt, Oskar K. and Szeto, S. and Szumowska, E. and Tang, Andrew C. W. and Tanzer, Norbert K and Tear, Morgan J. and Theriault, J. and Thomae, M. and Torres, David and Traczyk, Jakub and Tybur, J. and Ujhelyi, A. and Aert, R. V. and Assen, M. V. and Hulst, M. and Lange, P. and Veer, A. V. and Echeverría, Alejandro and Vaughn, L. and Vázquez, A. and Vega, L. D. and Verniers, Catherine and Verschoor, M. and Voermans, Ingrid and Vranka, M. and Welch, Cheryl A. and Wichman, A. and Williams, L. and Wood, M. and Woodzicka, Julie A. and Wrońska, Marta and Young, L. and Zelenski, J. and Zhi-jia, Zeng and Nosek, Brian A.}
}

Abstract

We conducted preregistered replications of 28 classic and contemporary published findings, with protocols that were peer reviewed in advance, to examine variation in effect magnitudes across samples and settings. Each protocol was administered to approximately half of 125 samples that comprised 15,305 participants from 36 countries and territories. Using the conventional criterion of statistical significance (p < .05), we found that 15 (54%) of the replications provided evidence of a statistically significant effect in the same direction as the original finding. With a strict significance criterion (p < .0001), 14 (50%) of the replications still provided such evidence, a reflection of the extremely high-powered design. Seven (25%) of the replications yielded effect sizes larger than the original ones, and 21 (75%) yielded effect sizes smaller than the original ones. The median comparable Cohen’s ds were 0.60 for the original findings and 0.15 for the replications. The effect sizes were small (< 0.20) in 16 of the replications (57%), and 9 effects (32%) were in the direction opposite the direction of the original effect. Across settings, the Q statistic indicated significant heterogeneity in 11 (39%) of the replication effects, and most of those were among the findings with the largest overall effect sizes; only 1 effect that was near zero in the aggregate showed significant heterogeneity according to this measure. Only 1 effect had a tau value greater than .20, an indication of moderate heterogeneity. Eight others had tau values near or slightly above .10, an indication of slight heterogeneity. Moderation tests indicated that very little heterogeneity was attributable to the order in which the tasks were performed or whether the tasks were administered in lab versus online. Exploratory comparisons revealed little heterogeneity between Western, educated, industrialized, rich, and democratic (WEIRD) cultures and less WEIRD cultures (i.e., cultures with relatively high and low WEIRDness scores, respectively). Cumulatively, variability in the observed effect sizes was attributable more to the effect being studied than to the sample or setting in which it was studied.


Share

Tools
Translate to