Measuring the predictability of life outcomes with a scientific mass collaboration

Matthew J. Salganik; Ian Lundberg; Alexander T. Kindel; Caitlin E. Ahearn; Khaled Al-Ghoneim; Abdullah Almaatouq; Drew M. Altschul; Jennie E. Brand; Nicole Bohme Carnegie; Ryan James Compton; Debanjan Datta; Thomas Davidson; Anna Filippova; Connor Gilroy; Brian J. Goode; Eaman Jahani; Ridhi Kashyap; Antje Kirchner; Stephen McKay; Allison C. Morgan; Alex Pentland; Kivan Polimis; Louis Raes; Daniel E. Rigobon; Claudia V. Roberts; Diana M. Stanescu; Yoshihiko Suhara; Adaner Usmani; Erik H. Wang; Muna Adem; Abdulla Alhajri; Bedoor AlShebli; Redwane Amin; Ryan B. Amos; Lisa P. Argyle; Livia Baer-Bositis; Moritz Büchi; Bo Ryehn Chung; William Eggert; Gregory Faletto; Zhilin Fan; Jeremy Freese; Tejomay Gadgil; Josh Gagné; Yue Gao; Andrew Halpern-Manners; Sonia P. Hashim; Sonia Hausen; Guanhua He; Kimberly Higuera; Bernie Hogan; Ilana M. Horwitz; Lisa M. Hummel; Naman Jain; Kun Jin; David Jurgens; Patrick Kaminski; Areg Karapetyan; E. H. Kim; Ben Leizman; Naijia Liu; Malte Möser; Andrew E. Mack; Mayank Mahajan; Noah Mandell; Helge Marahrens; Diana Mercado-Garcia; Viola Mocz; Katariina Mueller-Gastell; Ahmed Musse; Qiankun Niu; William Nowak; Hamidreza Omidvar; Andrew Or; Karen Ouyang; Katy M. Pinto; Ethan Porter; Kristin E. Porter; Crystal Qian; Tamkinat Rauf; Anahit Sargsyan; Thomas Schaffner; Landon Schnabel; Bryan Schonfeld; Ben Sender; Jonathan D. Tang; Emma Tsurkov; Austin van Loon; Onur Varol; Julia Wang; Flora Wang; Samantha Weissman; Kirstie Whitaker; Maria K. Wolters; Wei Lee Woon; Xiafei Wang; Zhi Wang; James Wu; Catherine Wu; Kengran Yang; Jingwen Yin; Bingyu Zhao; Chenyun Zhu; Jeanne Brooks-Gunn; Barbara E. Engelhardt; Moritz Hardt; Dean Knox; Karen Levy; Arvind Narayanan; Brandon M. Stewart; Duncan J. Watts; Sara McLanahan

doi:10.1073/pnas.1915006117

Measuring the predictability of life outcomes with a scientific mass collaboration

Matthew J. Salganik, Ian Lundberg, Alexander T. Kindel, Caitlin E. Ahearn, Khaled Al-Ghoneim, Abdullah Almaatouq, Drew M. Altschul, Jennie E. Brand, Nicole Bohme Carnegie, Ryan James Compton, Debanjan Datta, Thomas Davidson, Anna Filippova, Connor Gilroy, Brian J. Goode, Eaman Jahani, Ridhi Kashyap, Antje Kirchner, Stephen McKay, Allison C. MorganAlex Pentland, Kivan Polimis, Louis Raes, Daniel E. Rigobon, Claudia V. Roberts, Diana M. Stanescu, Yoshihiko Suhara, Adaner Usmani, Erik H. Wang, Muna Adem, Abdulla Alhajri, Bedoor AlShebli, Redwane Amin, Ryan B. Amos, Lisa P. Argyle, Livia Baer-Bositis, Moritz Büchi, Bo Ryehn Chung, William Eggert, Gregory Faletto, Zhilin Fan, Jeremy Freese, Tejomay Gadgil, Josh Gagné, Yue Gao, Andrew Halpern-Manners, Sonia P. Hashim, Sonia Hausen, Guanhua He, Kimberly Higuera, Bernie Hogan, Ilana M. Horwitz, Lisa M. Hummel, Naman Jain, Kun Jin, David Jurgens, Patrick Kaminski, Areg Karapetyan, E. H. Kim, Ben Leizman, Naijia Liu, Malte Möser, Andrew E. Mack, Mayank Mahajan, Noah Mandell, Helge Marahrens, Diana Mercado-Garcia, Viola Mocz, Katariina Mueller-Gastell, Ahmed Musse, Qiankun Niu, William Nowak, Hamidreza Omidvar, Andrew Or, Karen Ouyang, Katy M. Pinto, Ethan Porter, Kristin E. Porter, Crystal Qian, Tamkinat Rauf, Anahit Sargsyan, Thomas Schaffner, Landon Schnabel, Bryan Schonfeld, Ben Sender, Jonathan D. Tang, Emma Tsurkov, Austin van Loon, Onur Varol, Julia Wang, Flora Wang, Samantha Weissman, Kirstie Whitaker, Maria K. Wolters, Wei Lee Woon, Xiafei Wang, Zhi Wang, James Wu, Catherine Wu, Kengran Yang, Jingwen Yin, Bingyu Zhao, Chenyun Zhu, Jeanne Brooks-Gunn, Barbara E. Engelhardt, Moritz Hardt, Dean Knox, Karen Levy, Arvind Narayanan, Brandon M. Stewart, Duncan J. Watts, Sara McLanahan

Social Research and Public Policy

Research output: Contribution to journal › Article › peer-review

Abstract

How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the value of mass collaborations in the social sciences.

Original language	English (US)
Pages (from-to)	8398-8403
Number of pages	6
Journal	Proceedings of the National Academy of Sciences of the United States of America
Volume	117
Issue number	15
DOIs	https://doi.org/10.1073/pnas.1915006117
State	Published - Apr 14 2020

Keywords

Life course
Machine learning
Mass collaboration
Prediction

ASJC Scopus subject areas

General

Access to Document

10.1073/pnas.1915006117

Cite this

Salganik, M. J., Lundberg, I., Kindel, A. T., Ahearn, C. E., Al-Ghoneim, K., Almaatouq, A., Altschul, D. M., Brand, J. E., Carnegie, N. B., Compton, R. J., Datta, D., Davidson, T., Filippova, A., Gilroy, C., Goode, B. J., Jahani, E., Kashyap, R., Kirchner, A., McKay, S., ... McLanahan, S. (2020). Measuring the predictability of life outcomes with a scientific mass collaboration. Proceedings of the National Academy of Sciences of the United States of America, 117(15), 8398-8403. https://doi.org/10.1073/pnas.1915006117

Salganik, MJ, Lundberg, I, Kindel, AT, Ahearn, CE, Al-Ghoneim, K, Almaatouq, A, Altschul, DM, Brand, JE, Carnegie, NB, Compton, RJ, Datta, D, Davidson, T, Filippova, A, Gilroy, C, Goode, BJ, Jahani, E, Kashyap, R, Kirchner, A, McKay, S, Morgan, AC, Pentland, A, Polimis, K, Raes, L, Rigobon, DE, Roberts, CV, Stanescu, DM, Suhara, Y, Usmani, A, Wang, EH, Adem, M, Alhajri, A, AlShebli, B, Amin, R, Amos, RB, Argyle, LP, Baer-Bositis, L, Büchi, M, Chung, BR, Eggert, W, Faletto, G, Fan, Z, Freese, J, Gadgil, T, Gagné, J, Gao, Y, Halpern-Manners, A, Hashim, SP, Hausen, S, He, G, Higuera, K, Hogan, B, Horwitz, IM, Hummel, LM, Jain, N, Jin, K, Jurgens, D, Kaminski, P, Karapetyan, A, Kim, EH, Leizman, B, Liu, N, Möser, M, Mack, AE, Mahajan, M, Mandell, N, Marahrens, H, Mercado-Garcia, D, Mocz, V, Mueller-Gastell, K, Musse, A, Niu, Q, Nowak, W, Omidvar, H, Or, A, Ouyang, K, Pinto, KM, Porter, E, Porter, KE, Qian, C, Rauf, T, Sargsyan, A, Schaffner, T, Schnabel, L, Schonfeld, B, Sender, B, Tang, JD, Tsurkov, E, van Loon, A, Varol, O, Wang, J, Wang, F, Weissman, S, Whitaker, K, Wolters, MK, Woon, WL, Wang, X, Wang, Z, Wu, J, Wu, C, Yang, K, Yin, J, Zhao, B, Zhu, C, Brooks-Gunn, J, Engelhardt, BE, Hardt, M, Knox, D, Levy, K, Narayanan, A, Stewart, BM, Watts, DJ & McLanahan, S 2020, 'Measuring the predictability of life outcomes with a scientific mass collaboration', Proceedings of the National Academy of Sciences of the United States of America, vol. 117, no. 15, pp. 8398-8403. https://doi.org/10.1073/pnas.1915006117

@article{488072a76ddd431db3edbfae26ab8682,

title = "Measuring the predictability of life outcomes with a scientific mass collaboration",

abstract = "How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the value of mass collaborations in the social sciences.",

keywords = "Life course, Machine learning, Mass collaboration, Prediction",

author = "Salganik, {Matthew J.} and Ian Lundberg and Kindel, {Alexander T.} and Ahearn, {Caitlin E.} and Khaled Al-Ghoneim and Abdullah Almaatouq and Altschul, {Drew M.} and Brand, {Jennie E.} and Carnegie, {Nicole Bohme} and Compton, {Ryan James} and Debanjan Datta and Thomas Davidson and Anna Filippova and Connor Gilroy and Goode, {Brian J.} and Eaman Jahani and Ridhi Kashyap and Antje Kirchner and Stephen McKay and Morgan, {Allison C.} and Alex Pentland and Kivan Polimis and Louis Raes and Rigobon, {Daniel E.} and Roberts, {Claudia V.} and Stanescu, {Diana M.} and Yoshihiko Suhara and Adaner Usmani and Wang, {Erik H.} and Muna Adem and Abdulla Alhajri and Bedoor AlShebli and Redwane Amin and Amos, {Ryan B.} and Argyle, {Lisa P.} and Livia Baer-Bositis and Moritz B{\"u}chi and Chung, {Bo Ryehn} and William Eggert and Gregory Faletto and Zhilin Fan and Jeremy Freese and Tejomay Gadgil and Josh Gagn{\'e} and Yue Gao and Andrew Halpern-Manners and Hashim, {Sonia P.} and Sonia Hausen and Guanhua He and Kimberly Higuera and Bernie Hogan and Horwitz, {Ilana M.} and Hummel, {Lisa M.} and Naman Jain and Kun Jin and David Jurgens and Patrick Kaminski and Areg Karapetyan and Kim, {E. H.} and Ben Leizman and Naijia Liu and Malte M{\"o}ser and Mack, {Andrew E.} and Mayank Mahajan and Noah Mandell and Helge Marahrens and Diana Mercado-Garcia and Viola Mocz and Katariina Mueller-Gastell and Ahmed Musse and Qiankun Niu and William Nowak and Hamidreza Omidvar and Andrew Or and Karen Ouyang and Pinto, {Katy M.} and Ethan Porter and Porter, {Kristin E.} and Crystal Qian and Tamkinat Rauf and Anahit Sargsyan and Thomas Schaffner and Landon Schnabel and Bryan Schonfeld and Ben Sender and Tang, {Jonathan D.} and Emma Tsurkov and {van Loon}, Austin and Onur Varol and Julia Wang and Flora Wang and Samantha Weissman and Kirstie Whitaker and Wolters, {Maria K.} and Woon, {Wei Lee} and Xiafei Wang and Zhi Wang and James Wu and Catherine Wu and Kengran Yang and Jingwen Yin and Bingyu Zhao and Chenyun Zhu and Jeanne Brooks-Gunn and Engelhardt, {Barbara E.} and Moritz Hardt and Dean Knox and Karen Levy and Arvind Narayanan and Stewart, {Brandon M.} and Watts, {Duncan J.} and Sara McLanahan",

note = "Publisher Copyright: {\textcopyright} This open access article is distributed under Creative Commons Attribution-NonCommercialNoDerivatives License 4.0 (CC BY-NC-ND).",

year = "2020",

month = apr,

day = "14",

doi = "10.1073/pnas.1915006117",

language = "English (US)",

volume = "117",

pages = "8398--8403",

journal = "Proceedings of the National Academy of Sciences of the United States of America",

issn = "0027-8424",

publisher = "National Academy of Sciences",

number = "15",

}

TY - JOUR

T1 - Measuring the predictability of life outcomes with a scientific mass collaboration

AU - Salganik, Matthew J.

AU - Lundberg, Ian

AU - Kindel, Alexander T.

AU - Ahearn, Caitlin E.

AU - Al-Ghoneim, Khaled

AU - Almaatouq, Abdullah

AU - Altschul, Drew M.

AU - Brand, Jennie E.

AU - Carnegie, Nicole Bohme

AU - Compton, Ryan James

AU - Datta, Debanjan

AU - Davidson, Thomas

AU - Filippova, Anna

AU - Gilroy, Connor

AU - Goode, Brian J.

AU - Jahani, Eaman

AU - Kashyap, Ridhi

AU - Kirchner, Antje

AU - McKay, Stephen

AU - Morgan, Allison C.

AU - Pentland, Alex

AU - Polimis, Kivan

AU - Raes, Louis

AU - Rigobon, Daniel E.

AU - Roberts, Claudia V.

AU - Stanescu, Diana M.

AU - Suhara, Yoshihiko

AU - Usmani, Adaner

AU - Wang, Erik H.

AU - Adem, Muna

AU - Alhajri, Abdulla

AU - AlShebli, Bedoor

AU - Amin, Redwane

AU - Amos, Ryan B.

AU - Argyle, Lisa P.

AU - Baer-Bositis, Livia

AU - Büchi, Moritz

AU - Chung, Bo Ryehn

AU - Eggert, William

AU - Faletto, Gregory

AU - Fan, Zhilin

AU - Freese, Jeremy

AU - Gadgil, Tejomay

AU - Gagné, Josh

AU - Gao, Yue

AU - Halpern-Manners, Andrew

AU - Hashim, Sonia P.

AU - Hausen, Sonia

AU - He, Guanhua

AU - Higuera, Kimberly

AU - Hogan, Bernie

AU - Horwitz, Ilana M.

AU - Hummel, Lisa M.

AU - Jain, Naman

AU - Jin, Kun

AU - Jurgens, David

AU - Kaminski, Patrick

AU - Karapetyan, Areg

AU - Kim, E. H.

AU - Leizman, Ben

AU - Liu, Naijia

AU - Möser, Malte

AU - Mack, Andrew E.

AU - Mahajan, Mayank

AU - Mandell, Noah

AU - Marahrens, Helge

AU - Mercado-Garcia, Diana

AU - Mocz, Viola

AU - Mueller-Gastell, Katariina

AU - Musse, Ahmed

AU - Niu, Qiankun

AU - Nowak, William

AU - Omidvar, Hamidreza

AU - Or, Andrew

AU - Ouyang, Karen

AU - Pinto, Katy M.

AU - Porter, Ethan

AU - Porter, Kristin E.

AU - Qian, Crystal

AU - Rauf, Tamkinat

AU - Sargsyan, Anahit

AU - Schaffner, Thomas

AU - Schnabel, Landon

AU - Schonfeld, Bryan

AU - Sender, Ben

AU - Tang, Jonathan D.

AU - Tsurkov, Emma

AU - van Loon, Austin

AU - Varol, Onur

AU - Wang, Julia

AU - Wang, Flora

AU - Weissman, Samantha

AU - Whitaker, Kirstie

AU - Wolters, Maria K.

AU - Woon, Wei Lee

AU - Wang, Xiafei

AU - Wang, Zhi

AU - Wu, James

AU - Wu, Catherine

AU - Yang, Kengran

AU - Yin, Jingwen

AU - Zhao, Bingyu

AU - Zhu, Chenyun

AU - Brooks-Gunn, Jeanne

AU - Engelhardt, Barbara E.

AU - Hardt, Moritz

AU - Knox, Dean

AU - Levy, Karen

AU - Narayanan, Arvind

AU - Stewart, Brandon M.

AU - Watts, Duncan J.

AU - McLanahan, Sara

N1 - Publisher Copyright: © This open access article is distributed under Creative Commons Attribution-NonCommercialNoDerivatives License 4.0 (CC BY-NC-ND).

PY - 2020/4/14

Y1 - 2020/4/14

N2 - How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the value of mass collaborations in the social sciences.

AB - How predictable are life trajectories? We investigated this question with a scientific mass collaboration using the common task method; 160 teams built predictive models for six life outcomes using data from the Fragile Families and Child Wellbeing Study, a high-quality birth cohort study. Despite using a rich dataset and applying machine-learning methods optimized for prediction, the best predictions were not very accurate and were only slightly better than those from a simple benchmark model. Within each outcome, prediction error was strongly associated with the family being predicted and weakly associated with the technique used to generate the prediction. Overall, these results suggest practical limits to the predictability of life outcomes in some settings and illustrate the value of mass collaborations in the social sciences.

KW - Life course

KW - Machine learning

KW - Mass collaboration

KW - Prediction

UR - http://www.scopus.com/inward/record.url?scp=85083153846&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85083153846&partnerID=8YFLogxK

U2 - 10.1073/pnas.1915006117

DO - 10.1073/pnas.1915006117

M3 - Article

C2 - 32229555

AN - SCOPUS:85083153846

SN - 0027-8424

VL - 117

SP - 8398

EP - 8403

JO - Proceedings of the National Academy of Sciences of the United States of America

JF - Proceedings of the National Academy of Sciences of the United States of America

IS - 15

ER -

Measuring the predictability of life outcomes with a scientific mass collaboration

Abstract

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this