An external stability audit framework to test the validity of personality prediction in AI hiring

Alene K. Rhea, Kelsey Markey, Lauren D’Arinzo, Hilke Schellmann, Mona Sloane, Paul Squires, Falaah Arif Khan, Julia Stoyanovich

    Research output: Contribution to journalArticlepeer-review


    Automated hiring systems are among the fastest-developing of all high-stakes AI systems. Among these are algorithmic personality tests that use insights from psychometric testing, and promise to surface personality traits indicative of future success based on job seekers’ resumes or social media profiles. We interrogate the validity of such systems using stability of the outputs they produce, noting that reliability is a necessary, but not a sufficient, condition for validity. Crucially, rather than challenging or affirming the assumptions made in psychometric testing — that personality is a meaningful and measurable construct, and that personality traits are indicative of future success on the job — we frame our audit methodology around testing the underlying assumptions made by the vendors of the algorithmic personality tests themselves. Our main contribution is the development of a socio-technical framework for auditing the stability of algorithmic systems. This contribution is supplemented with an open-source software library that implements the technical components of the audit, and can be used to conduct similar stability audits of algorithmic systems. We instantiate our framework with the audit of two real-world personality prediction systems, namely, Humantic AI and Crystal. The application of our audit framework demonstrates that both these systems show substantial instability with respect to key facets of measurement, and hence cannot be considered valid testing instruments.

    Original languageEnglish (US)
    Pages (from-to)2153-2193
    Number of pages41
    JournalData Mining and Knowledge Discovery
    Issue number6
    StatePublished - Nov 2022


    • Algorithm Audit
    • Hiring
    • Personality
    • Reliability
    • Stability
    • Validity

    ASJC Scopus subject areas

    • Information Systems
    • Computer Science Applications
    • Computer Networks and Communications


    Dive into the research topics of 'An external stability audit framework to test the validity of personality prediction in AI hiring'. Together they form a unique fingerprint.

    Cite this