Measuring the effectiveness of embedded phishing exercises

Hossein Siadati, Sean Palka, Avi Siegel, Damon McCoy

    Research output: Contribution to conferencePaper

    Abstract

    Embedded phishing exercises, which send test phishing emails, are utilized by organizations to reduce the susceptibility of its employees to this type of attack. Research studies seeking to evaluate the effectiveness of these exercises have generally been limited by small sample sizes. These studies have not been able to measure possible factors that might bias results. As a result, companies have had to create their own design and evaluation methods, with no framework to guide their efforts. Lacking such guidelines, it can often be difficult to determine whether these types of exercises are truly effective, and if reported results are statistically reliable. In this paper, we conduct a systematic analysis of data from a large real world embedded phishing exercise that involved 19,180 participants from a single organization, and utilized 115,080 test phishing emails. The first part of our study focuses on developing methodologies to correct some sources of bias, enabling sounder evaluations of the efficacy of embedded phishing exercises and training. We then use these methods to perform an analysis of the effectiveness of this embedded phishing exercise, and through our analysis, identify how the design of these exercises might be improved.

    Original languageEnglish (US)
    StatePublished - 2017
    Event10th USENIX Workshop on Cyber Security Experimentation and Test, CSET 2017, co-located with USENIX Security 2017 - Vancouver, Canada
    Duration: Aug 14 2017 → …

    Conference

    Conference10th USENIX Workshop on Cyber Security Experimentation and Test, CSET 2017, co-located with USENIX Security 2017
    CountryCanada
    CityVancouver
    Period8/14/17 → …

    ASJC Scopus subject areas

    • Computer Networks and Communications
    • Safety, Risk, Reliability and Quality

    Fingerprint Dive into the research topics of 'Measuring the effectiveness of embedded phishing exercises'. Together they form a unique fingerprint.

  • Cite this

    Siadati, H., Palka, S., Siegel, A., & McCoy, D. (2017). Measuring the effectiveness of embedded phishing exercises. Paper presented at 10th USENIX Workshop on Cyber Security Experimentation and Test, CSET 2017, co-located with USENIX Security 2017, Vancouver, Canada.