Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results

Justin F. Landy; Miaolei Liam; Isabel L. Ding; Domenico Viganola; Warren Tierney; Anna Dreber; Magnus Johannesson; Thomas Pfeiffer; Charles R. Ebersole; Quentin F. Gronau; Alexander Ly; Don Van Den Bergh; Maarten Marsman; Koen Derks; Eric Jan Wagenmaker; Andrew Proctor; Daniel M. Bartels; Christopher W. Bauman; William J. Brady; Felix Cheung; Andrei Cimpian; Simone Dohle; M. Brent Donnellan; Adam Hahn; Michael P. Hall; William Jiménez-Leal; David J. Johnson; Richard E. Lucas; BenoÎt Monin; Andres Montealegre; Elizabeth Mullen; Jun Pang; Jennifer Ray; Diego A. Reinero; Jesse Reynolds; Walter Sowden; Daniel Storage; Runkun Su; Christina M. Tworek; Jay J. Van Bavel; Daniel Walco; Julian Wills; Xiaobing Xu; Kai Chi Yam; Xiaoyu Yang; William A. Cunningham; Martin Schweinsberg; Molly Urwitz; Eric L. Uhlmann; Matú Adamkovic; Ravin Alaei; Casper J. Albers; Aurélien Allard; Ian A. Anderson; Michael R. Andreychik; Peter Babinčák; Bradley J. Baker; Gabriel Baník; Ernest Baskin; Jozef Bavolar; Ruud M.W.J. Berkers; Michal Białek; Joel Blanke; Johannes Breuer; Ambra Brizi; Stephanie E.V. Brown; Florian Brühlmann; Hendrik Bruns; Leigh Caldwell; Jean François Campourcy; Eugene Y. Chan; Yen Ping Chang; Benjamin Y. Cheung; Alycia Chin; Kit W. Cho; Simon Columbus; Paul Conway; Conrad A. Corretti; Adam W. Craig; Paul G. Curran; Alexander F. Danvers; Ian G.J. Dawson; Martin V. Day; Erik Dietl; Johannes T. Doerflinger; Alice Dominici; Vilius Dranseika; Peter A. Edelsbrunner; John E. Edlund; Matthew Fisher; Anna Fung; Oliver Genschow; Timo Gnambs; Matthew H. Goldberg; Lorenz Graf-Vlachy; Andrew C. Hafenbrack; Sebastian Hafenbrädl; Andree Hartanto; Joseph P. Heffner; Joseph Hilgard; Felix Holzmeister; Oleksandr V. Horchak; Tina S.T. Huang; Joachim Hüffmeier; Sean Hughes; Ian Hussey; Roland Imhoff; Bastian Jaeger; Konrad Jamro; Samuel G.B. Johnson; Andrew Jones; Lucas Keller; Olga Kombeiz; Lacy E. Krueger; Anthony Lantian; Justin P. Laplante; Ljiljana B. Lazarevic; Jonathan Leclerc; Nicole Legate; James M. Leonhardt; Desmond W. Leung; Carmel A. Levitan; Hause Lin; Qinglan Liu; Marco Tullio Liuzza; Kenneth D. Locke; Albert L. Ly; Melanie MacEacheron; Christopher R. Madan; Harry Manley; Silvia Mari; Marcel Martončik; Scott L. McLean; Jonathon McPhetres; Brett G. Mercier; Corinna Michels; Michael C. Mullarkey; Erica D. Musser; Ladislas Nalborczyk; Gustav Nilsonne; Nicholas G. Otis; Sarah M.G. Otner; Philipp E. Otto; Oscar Oviedo-Trespalacios; Mariola Paruzel-Czachura; Francesco Pellegrini; Vitor M.D. Pereira; Hannah Perfecto; Gerit Pfuhl; Mark H. Phillips; Ori Plonsky; Maura Pozzi; Danka B. Puric; Brett Raymond-Barker; David E. Redman; Caleb J. Reynolds; Ivan Ropovik; Lukas Röseler; Janna K. Ruessmann; William H. Ryan; Nika Sablaturova; Kurt J. Schuepfer; Astrid Schütz; Miroslav Sirota; Matthias Stefan; Eric L. Stocks; Garrett L. Strosser; Jordan W. Suchow; Anna Szabelska; Kian Siong Tey; Leonid Tiokhin; Jais Troian; Till Utesch; Alejandro Vásquez-Echeverriá; Leigh Ann Vaughn; Mark Verschoor; Bettina Von Helversen; Pascal Wallisch; Sophia C. Weissgerber; Aaron L. Wichman; Jan K. Woike; Iris Žeželj; Janis H. Zickfeld; Yeonsin Ahn; Philippe F. Blaettchen; Xi Kang; Yoo Jin Lee; Philip M. Parker; Paul A. Parker; Jamie S. Song; May Anne Very; Lynn Wong

doi:10.1037/bul0000220

Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results

Justin F. Landy, Miaolei Liam, Isabel L. Ding, Domenico Viganola, Warren Tierney, Anna Dreber, Magnus Johannesson, Thomas Pfeiffer, Charles R. Ebersole, Quentin F. Gronau, Alexander Ly, Don Van Den Bergh, Maarten Marsman, Koen Derks, Eric Jan Wagenmaker, Andrew Proctor, Daniel M. Bartels, Christopher W. Bauman, William J. Brady, Felix CheungAndrei Cimpian, Simone Dohle, M. Brent Donnellan, Adam Hahn, Michael P. Hall, William Jiménez-Leal, David J. Johnson, Richard E. Lucas, BenoÎt Monin, Andres Montealegre, Elizabeth Mullen, Jun Pang, Jennifer Ray, Diego A. Reinero, Jesse Reynolds, Walter Sowden, Daniel Storage, Runkun Su, Christina M. Tworek, Jay J. Van Bavel, Daniel Walco, Julian Wills, Xiaobing Xu, Kai Chi Yam, Xiaoyu Yang, William A. Cunningham, Martin Schweinsberg, Molly Urwitz, Eric L. Uhlmann, Matú Adamkovic, Ravin Alaei, Casper J. Albers, Aurélien Allard, Ian A. Anderson, Michael R. Andreychik, Peter Babinčák, Bradley J. Baker, Gabriel Baník, Ernest Baskin, Jozef Bavolar, Ruud M.W.J. Berkers, Michal Białek, Joel Blanke, Johannes Breuer, Ambra Brizi, Stephanie E.V. Brown, Florian Brühlmann, Hendrik Bruns, Leigh Caldwell, Jean François Campourcy, Eugene Y. Chan, Yen Ping Chang, Benjamin Y. Cheung, Alycia Chin, Kit W. Cho, Simon Columbus, Paul Conway, Conrad A. Corretti, Adam W. Craig, Paul G. Curran, Alexander F. Danvers, Ian G.J. Dawson, Martin V. Day, Erik Dietl, Johannes T. Doerflinger, Alice Dominici, Vilius Dranseika, Peter A. Edelsbrunner, John E. Edlund, Matthew Fisher, Anna Fung, Oliver Genschow, Timo Gnambs, Matthew H. Goldberg, Lorenz Graf-Vlachy, Andrew C. Hafenbrack, Sebastian Hafenbrädl, Andree Hartanto, Joseph P. Heffner, Joseph Hilgard, Felix Holzmeister, Oleksandr V. Horchak, Tina S.T. Huang, Joachim Hüffmeier, Sean Hughes, Ian Hussey, Roland Imhoff, Bastian Jaeger, Konrad Jamro, Samuel G.B. Johnson, Andrew Jones, Lucas Keller, Olga Kombeiz, Lacy E. Krueger, Anthony Lantian, Justin P. Laplante, Ljiljana B. Lazarevic, Jonathan Leclerc, Nicole Legate, James M. Leonhardt, Desmond W. Leung, Carmel A. Levitan, Hause Lin, Qinglan Liu, Marco Tullio Liuzza, Kenneth D. Locke, Albert L. Ly, Melanie MacEacheron, Christopher R. Madan, Harry Manley, Silvia Mari, Marcel Martončik, Scott L. McLean, Jonathon McPhetres, Brett G. Mercier, Corinna Michels, Michael C. Mullarkey, Erica D. Musser, Ladislas Nalborczyk, Gustav Nilsonne, Nicholas G. Otis, Sarah M.G. Otner, Philipp E. Otto, Oscar Oviedo-Trespalacios, Mariola Paruzel-Czachura, Francesco Pellegrini, Vitor M.D. Pereira, Hannah Perfecto, Gerit Pfuhl, Mark H. Phillips, Ori Plonsky, Maura Pozzi, Danka B. Puric, Brett Raymond-Barker, David E. Redman, Caleb J. Reynolds, Ivan Ropovik, Lukas Röseler, Janna K. Ruessmann, William H. Ryan, Nika Sablaturova, Kurt J. Schuepfer, Astrid Schütz, Miroslav Sirota, Matthias Stefan, Eric L. Stocks, Garrett L. Strosser, Jordan W. Suchow, Anna Szabelska, Kian Siong Tey, Leonid Tiokhin, Jais Troian, Till Utesch, Alejandro Vásquez-Echeverriá, Leigh Ann Vaughn, Mark Verschoor, Bettina Von Helversen, Pascal Wallisch, Sophia C. Weissgerber, Aaron L. Wichman, Jan K. Woike, Iris Žeželj, Janis H. Zickfeld, Yeonsin Ahn, Philippe F. Blaettchen, Xi Kang, Yoo Jin Lee, Philip M. Parker, Paul A. Parker, Jamie S. Song, May Anne Very, Lynn Wong

Psychology

Research output: Contribution to journal › Article › peer-review

Abstract

To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Original language	English (US)
Pages (from-to)	451-479
Number of pages	29
Journal	Psychological bulletin
Volume	146
Issue number	5
DOIs	https://doi.org/10.1037/bul0000220
State	Published - 2020

Keywords

conceptual replications
crowdsourcing
forecasting
research robustness
scientific transparency
Crowdsourcing
Humans
Adult
Psychology/methods
Random Allocation
Research Design

ASJC Scopus subject areas

General Psychology

Access to Document

10.1037/bul0000220

Cite this

Landy, J. F., Liam, M., Ding, I. L., Viganola, D., Tierney, W., Dreber, A., Johannesson, M., Pfeiffer, T., Ebersole, C. R., Gronau, Q. F., Ly, A., Bergh, D. V. D., Marsman, M., Derks, K., Wagenmaker, E. J., Proctor, A., Bartels, D. M., Bauman, C. W., Brady, W. J., ... Wong, L. (2020). Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results. Psychological bulletin, 146(5), 451-479. https://doi.org/10.1037/bul0000220

Landy, JF, Liam, M, Ding, IL, Viganola, D, Tierney, W, Dreber, A, Johannesson, M, Pfeiffer, T, Ebersole, CR, Gronau, QF, Ly, A, Bergh, DVD, Marsman, M, Derks, K, Wagenmaker, EJ, Proctor, A, Bartels, DM, Bauman, CW, Brady, WJ, Cheung, F, Cimpian, A, Dohle, S, Donnellan, MB, Hahn, A, Hall, MP, Jiménez-Leal, W, Johnson, DJ, Lucas, RE, Monin, B, Montealegre, A, Mullen, E, Pang, J, Ray, J, Reinero, DA, Reynolds, J, Sowden, W, Storage, D, Su, R, Tworek, CM, Van Bavel, JJ, Walco, D, Wills, J, Xu, X, Yam, KC, Yang, X, Cunningham, WA, Schweinsberg, M, Urwitz, M, Uhlmann, EL, Adamkovic, M, Alaei, R, Albers, CJ, Allard, A, Anderson, IA, Andreychik, MR, Babinčák, P, Baker, BJ, Baník, G, Baskin, E, Bavolar, J, Berkers, RMWJ, Białek, M, Blanke, J, Breuer, J, Brizi, A, Brown, SEV, Brühlmann, F, Bruns, H, Caldwell, L, Campourcy, JF, Chan, EY, Chang, YP, Cheung, BY, Chin, A, Cho, KW, Columbus, S, Conway, P, Corretti, CA, Craig, AW, Curran, PG, Danvers, AF, Dawson, IGJ, Day, MV, Dietl, E, Doerflinger, JT, Dominici, A, Dranseika, V, Edelsbrunner, PA, Edlund, JE, Fisher, M, Fung, A, Genschow, O, Gnambs, T, Goldberg, MH, Graf-Vlachy, L, Hafenbrack, AC, Hafenbrädl, S, Hartanto, A, Heffner, JP, Hilgard, J, Holzmeister, F, Horchak, OV, Huang, TST, Hüffmeier, J, Hughes, S, Hussey, I, Imhoff, R, Jaeger, B, Jamro, K, Johnson, SGB, Jones, A, Keller, L, Kombeiz, O, Krueger, LE, Lantian, A, Laplante, JP, Lazarevic, LB, Leclerc, J, Legate, N, Leonhardt, JM, Leung, DW, Levitan, CA, Lin, H, Liu, Q, Liuzza, MT, Locke, KD, Ly, AL, MacEacheron, M, Madan, CR, Manley, H, Mari, S, Martončik, M, McLean, SL, McPhetres, J, Mercier, BG, Michels, C, Mullarkey, MC, Musser, ED, Nalborczyk, L, Nilsonne, G, Otis, NG, Otner, SMG, Otto, PE, Oviedo-Trespalacios, O, Paruzel-Czachura, M, Pellegrini, F, Pereira, VMD, Perfecto, H, Pfuhl, G, Phillips, MH, Plonsky, O, Pozzi, M, Puric, DB, Raymond-Barker, B, Redman, DE, Reynolds, CJ, Ropovik, I, Röseler, L, Ruessmann, JK, Ryan, WH, Sablaturova, N, Schuepfer, KJ, Schütz, A, Sirota, M, Stefan, M, Stocks, EL, Strosser, GL, Suchow, JW, Szabelska, A, Tey, KS, Tiokhin, L, Troian, J, Utesch, T, Vásquez-Echeverriá, A, Vaughn, LA, Verschoor, M, Helversen, BV, Wallisch, P, Weissgerber, SC, Wichman, AL, Woike, JK, Žeželj, I, Zickfeld, JH, Ahn, Y, Blaettchen, PF, Kang, X, Lee, YJ, Parker, PM, Parker, PA, Song, JS, Very, MA & Wong, L 2020, 'Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results', Psychological bulletin, vol. 146, no. 5, pp. 451-479. https://doi.org/10.1037/bul0000220

@article{729d261111aa499188deda79cd2c135d,

title = "Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results",

abstract = "To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).",

keywords = "conceptual replications, crowdsourcing, forecasting, research robustness, scientific transparency, Crowdsourcing, Humans, Adult, Psychology/methods, Random Allocation, Research Design",

author = "Landy, {Justin F.} and Miaolei Liam and Ding, {Isabel L.} and Domenico Viganola and Warren Tierney and Anna Dreber and Magnus Johannesson and Thomas Pfeiffer and Ebersole, {Charles R.} and Gronau, {Quentin F.} and Alexander Ly and Bergh, {Don Van Den} and Maarten Marsman and Koen Derks and Wagenmaker, {Eric Jan} and Andrew Proctor and Bartels, {Daniel M.} and Bauman, {Christopher W.} and Brady, {William J.} and Felix Cheung and Andrei Cimpian and Simone Dohle and Donnellan, {M. Brent} and Adam Hahn and Hall, {Michael P.} and William Jim{\'e}nez-Leal and Johnson, {David J.} and Lucas, {Richard E.} and Beno{\^I}t Monin and Andres Montealegre and Elizabeth Mullen and Jun Pang and Jennifer Ray and Reinero, {Diego A.} and Jesse Reynolds and Walter Sowden and Daniel Storage and Runkun Su and Tworek, {Christina M.} and {Van Bavel}, {Jay J.} and Daniel Walco and Julian Wills and Xiaobing Xu and Yam, {Kai Chi} and Xiaoyu Yang and Cunningham, {William A.} and Martin Schweinsberg and Molly Urwitz and Uhlmann, {Eric L.} and Mat{\'u} Adamkovic and Ravin Alaei and Albers, {Casper J.} and Aur{\'e}lien Allard and Anderson, {Ian A.} and Andreychik, {Michael R.} and Peter Babin{\v c}{\'a}k and Baker, {Bradley J.} and Gabriel Ban{\'i}k and Ernest Baskin and Jozef Bavolar and Berkers, {Ruud M.W.J.} and Michal Bia{\l}ek and Joel Blanke and Johannes Breuer and Ambra Brizi and Brown, {Stephanie E.V.} and Florian Br{\"u}hlmann and Hendrik Bruns and Leigh Caldwell and Campourcy, {Jean Fran{\c c}ois} and Chan, {Eugene Y.} and Chang, {Yen Ping} and Cheung, {Benjamin Y.} and Alycia Chin and Cho, {Kit W.} and Simon Columbus and Paul Conway and Corretti, {Conrad A.} and Craig, {Adam W.} and Curran, {Paul G.} and Danvers, {Alexander F.} and Dawson, {Ian G.J.} and Day, {Martin V.} and Erik Dietl and Doerflinger, {Johannes T.} and Alice Dominici and Vilius Dranseika and Edelsbrunner, {Peter A.} and Edlund, {John E.} and Matthew Fisher and Anna Fung and Oliver Genschow and Timo Gnambs and Goldberg, {Matthew H.} and Lorenz Graf-Vlachy and Hafenbrack, {Andrew C.} and Sebastian Hafenbr{\"a}dl and Andree Hartanto and Heffner, {Joseph P.} and Joseph Hilgard and Felix Holzmeister and Horchak, {Oleksandr V.} and Huang, {Tina S.T.} and Joachim H{\"u}ffmeier and Sean Hughes and Ian Hussey and Roland Imhoff and Bastian Jaeger and Konrad Jamro and Johnson, {Samuel G.B.} and Andrew Jones and Lucas Keller and Olga Kombeiz and Krueger, {Lacy E.} and Anthony Lantian and Laplante, {Justin P.} and Lazarevic, {Ljiljana B.} and Jonathan Leclerc and Nicole Legate and Leonhardt, {James M.} and Leung, {Desmond W.} and Levitan, {Carmel A.} and Hause Lin and Qinglan Liu and Liuzza, {Marco Tullio} and Locke, {Kenneth D.} and Ly, {Albert L.} and Melanie MacEacheron and Madan, {Christopher R.} and Harry Manley and Silvia Mari and Marcel Marton{\v c}ik and McLean, {Scott L.} and Jonathon McPhetres and Mercier, {Brett G.} and Corinna Michels and Mullarkey, {Michael C.} and Musser, {Erica D.} and Ladislas Nalborczyk and Gustav Nilsonne and Otis, {Nicholas G.} and Otner, {Sarah M.G.} and Otto, {Philipp E.} and Oscar Oviedo-Trespalacios and Mariola Paruzel-Czachura and Francesco Pellegrini and Pereira, {Vitor M.D.} and Hannah Perfecto and Gerit Pfuhl and Phillips, {Mark H.} and Ori Plonsky and Maura Pozzi and Puric, {Danka B.} and Brett Raymond-Barker and Redman, {David E.} and Reynolds, {Caleb J.} and Ivan Ropovik and Lukas R{\"o}seler and Ruessmann, {Janna K.} and Ryan, {William H.} and Nika Sablaturova and Schuepfer, {Kurt J.} and Astrid Sch{\"u}tz and Miroslav Sirota and Matthias Stefan and Stocks, {Eric L.} and Strosser, {Garrett L.} and Suchow, {Jordan W.} and Anna Szabelska and Tey, {Kian Siong} and Leonid Tiokhin and Jais Troian and Till Utesch and Alejandro V{\'a}squez-Echeverri{\'a} and Vaughn, {Leigh Ann} and Mark Verschoor and Helversen, {Bettina Von} and Pascal Wallisch and Weissgerber, {Sophia C.} and Wichman, {Aaron L.} and Woike, {Jan K.} and Iris {\v Z}e{\v z}elj and Zickfeld, {Janis H.} and Yeonsin Ahn and Blaettchen, {Philippe F.} and Xi Kang and Lee, {Yoo Jin} and Parker, {Philip M.} and Parker, {Paul A.} and Song, {Jamie S.} and Very, {May Anne} and Lynn Wong",

note = "Publisher Copyright: {\textcopyright} 2020, American Psychological Association.",

year = "2020",

doi = "10.1037/bul0000220",

language = "English (US)",

volume = "146",

pages = "451--479",

journal = "Psychological bulletin",

issn = "0033-2909",

publisher = "American Psychological Association",

number = "5",

}

TY - JOUR

T1 - Crowdsourcing Hypothesis Tests

T2 - Making transparent how design choices shape research results

AU - Landy, Justin F.

AU - Liam, Miaolei

AU - Ding, Isabel L.

AU - Viganola, Domenico

AU - Tierney, Warren

AU - Dreber, Anna

AU - Johannesson, Magnus

AU - Pfeiffer, Thomas

AU - Ebersole, Charles R.

AU - Gronau, Quentin F.

AU - Ly, Alexander

AU - Bergh, Don Van Den

AU - Marsman, Maarten

AU - Derks, Koen

AU - Wagenmaker, Eric Jan

AU - Proctor, Andrew

AU - Bartels, Daniel M.

AU - Bauman, Christopher W.

AU - Brady, William J.

AU - Cheung, Felix

AU - Cimpian, Andrei

AU - Dohle, Simone

AU - Donnellan, M. Brent

AU - Hahn, Adam

AU - Hall, Michael P.

AU - Jiménez-Leal, William

AU - Johnson, David J.

AU - Lucas, Richard E.

AU - Monin, BenoÎt

AU - Montealegre, Andres

AU - Mullen, Elizabeth

AU - Pang, Jun

AU - Ray, Jennifer

AU - Reinero, Diego A.

AU - Reynolds, Jesse

AU - Sowden, Walter

AU - Storage, Daniel

AU - Su, Runkun

AU - Tworek, Christina M.

AU - Van Bavel, Jay J.

AU - Walco, Daniel

AU - Wills, Julian

AU - Xu, Xiaobing

AU - Yam, Kai Chi

AU - Yang, Xiaoyu

AU - Cunningham, William A.

AU - Schweinsberg, Martin

AU - Urwitz, Molly

AU - Uhlmann, Eric L.

AU - Adamkovic, Matú

AU - Alaei, Ravin

AU - Albers, Casper J.

AU - Allard, Aurélien

AU - Anderson, Ian A.

AU - Andreychik, Michael R.

AU - Babinčák, Peter

AU - Baker, Bradley J.

AU - Baník, Gabriel

AU - Baskin, Ernest

AU - Bavolar, Jozef

AU - Berkers, Ruud M.W.J.

AU - Białek, Michal

AU - Blanke, Joel

AU - Breuer, Johannes

AU - Brizi, Ambra

AU - Brown, Stephanie E.V.

AU - Brühlmann, Florian

AU - Bruns, Hendrik

AU - Caldwell, Leigh

AU - Campourcy, Jean François

AU - Chan, Eugene Y.

AU - Chang, Yen Ping

AU - Cheung, Benjamin Y.

AU - Chin, Alycia

AU - Cho, Kit W.

AU - Columbus, Simon

AU - Conway, Paul

AU - Corretti, Conrad A.

AU - Craig, Adam W.

AU - Curran, Paul G.

AU - Danvers, Alexander F.

AU - Dawson, Ian G.J.

AU - Day, Martin V.

AU - Dietl, Erik

AU - Doerflinger, Johannes T.

AU - Dominici, Alice

AU - Dranseika, Vilius

AU - Edelsbrunner, Peter A.

AU - Edlund, John E.

AU - Fisher, Matthew

AU - Fung, Anna

AU - Genschow, Oliver

AU - Gnambs, Timo

AU - Goldberg, Matthew H.

AU - Graf-Vlachy, Lorenz

AU - Hafenbrack, Andrew C.

AU - Hafenbrädl, Sebastian

AU - Hartanto, Andree

AU - Heffner, Joseph P.

AU - Hilgard, Joseph

AU - Holzmeister, Felix

AU - Horchak, Oleksandr V.

AU - Huang, Tina S.T.

AU - Hüffmeier, Joachim

AU - Hughes, Sean

AU - Hussey, Ian

AU - Imhoff, Roland

AU - Jaeger, Bastian

AU - Jamro, Konrad

AU - Johnson, Samuel G.B.

AU - Jones, Andrew

AU - Keller, Lucas

AU - Kombeiz, Olga

AU - Krueger, Lacy E.

AU - Lantian, Anthony

AU - Laplante, Justin P.

AU - Lazarevic, Ljiljana B.

AU - Leclerc, Jonathan

AU - Legate, Nicole

AU - Leonhardt, James M.

AU - Leung, Desmond W.

AU - Levitan, Carmel A.

AU - Lin, Hause

AU - Liu, Qinglan

AU - Liuzza, Marco Tullio

AU - Locke, Kenneth D.

AU - Ly, Albert L.

AU - MacEacheron, Melanie

AU - Madan, Christopher R.

AU - Manley, Harry

AU - Mari, Silvia

AU - Martončik, Marcel

AU - McLean, Scott L.

AU - McPhetres, Jonathon

AU - Mercier, Brett G.

AU - Michels, Corinna

AU - Mullarkey, Michael C.

AU - Musser, Erica D.

AU - Nalborczyk, Ladislas

AU - Nilsonne, Gustav

AU - Otis, Nicholas G.

AU - Otner, Sarah M.G.

AU - Otto, Philipp E.

AU - Oviedo-Trespalacios, Oscar

AU - Paruzel-Czachura, Mariola

AU - Pellegrini, Francesco

AU - Pereira, Vitor M.D.

AU - Perfecto, Hannah

AU - Pfuhl, Gerit

AU - Phillips, Mark H.

AU - Plonsky, Ori

AU - Pozzi, Maura

AU - Puric, Danka B.

AU - Raymond-Barker, Brett

AU - Redman, David E.

AU - Reynolds, Caleb J.

AU - Ropovik, Ivan

AU - Röseler, Lukas

AU - Ruessmann, Janna K.

AU - Ryan, William H.

AU - Sablaturova, Nika

AU - Schuepfer, Kurt J.

AU - Schütz, Astrid

AU - Sirota, Miroslav

AU - Stefan, Matthias

AU - Stocks, Eric L.

AU - Strosser, Garrett L.

AU - Suchow, Jordan W.

AU - Szabelska, Anna

AU - Tey, Kian Siong

AU - Tiokhin, Leonid

AU - Troian, Jais

AU - Utesch, Till

AU - Vásquez-Echeverriá, Alejandro

AU - Vaughn, Leigh Ann

AU - Verschoor, Mark

AU - Helversen, Bettina Von

AU - Wallisch, Pascal

AU - Weissgerber, Sophia C.

AU - Wichman, Aaron L.

AU - Woike, Jan K.

AU - Žeželj, Iris

AU - Zickfeld, Janis H.

AU - Ahn, Yeonsin

AU - Blaettchen, Philippe F.

AU - Kang, Xi

AU - Lee, Yoo Jin

AU - Parker, Philip M.

AU - Parker, Paul A.

AU - Song, Jamie S.

AU - Very, May Anne

AU - Wong, Lynn

PY - 2020

Y1 - 2020

N2 - To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

AB - To what extent are research results influenced by subjective decisions that scientists make as they design studies? Fifteen research teams independently designed studies to answer five original research questions related to moral judgments, negotiations, and implicit cognition. Participants from 2 separate large samples (total N > 15,000) were then randomly assigned to complete 1 version of each study. Effect sizes varied dramatically across different sets of materials designed to test the same hypothesis: Materials from different teams rendered statistically significant effects in opposite directions for 4 of 5 hypotheses, with the narrowest range in estimates being d = -0.37 to + 0.26. Meta-analysis and a Bayesian perspective on the results revealed overall support for 2 hypotheses and a lack of support for 3 hypotheses. Overall, practically none of the variability in effect sizes was attributable to the skill of the research team in designing materials, whereas considerable variability was attributable to the hypothesis being tested. In a forecasting survey, predictions of other scientists were significantly correlated with study results, both across and within hypotheses. Crowdsourced testing of research hypotheses helps reveal the true consistency of empirical support for a scientific claim. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

KW - conceptual replications

KW - crowdsourcing

KW - forecasting

KW - research robustness

KW - scientific transparency

KW - Crowdsourcing

KW - Humans

KW - Adult

KW - Psychology/methods

KW - Random Allocation

KW - Research Design

UR - http://www.scopus.com/inward/record.url?scp=85081412411&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85081412411&partnerID=8YFLogxK

U2 - 10.1037/bul0000220

DO - 10.1037/bul0000220

M3 - Article

C2 - 31944796

AN - SCOPUS:85081412411

SN - 0033-2909

VL - 146

SP - 451

EP - 479

JO - Psychological bulletin

JF - Psychological bulletin

IS - 5

ER -

Crowdsourcing Hypothesis Tests: Making transparent how design choices shape research results

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this