Dually Noted: Layout-Aware Annotations with Smartphone Augmented Reality

Jing Qian, Qi Sun, Curtis Wigington, Han L. Han, Tong Sun, Jennifer Healey, James Tompkin, Jeff Huang

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    Sharing annotations encourages feedback, discussion, and knowledge passing among readers and can be beneficial for personal and public use. Prior augmented reality (AR) systems have expanded these benefits to both digital and printed documents. However, despite smartphone AR now being widely available, there is a lack of research about how to use AR effectively for interactive document annotation. We propose Dually Noted, a smartphone-based AR annotation system that recognizes the layout of structural elements in a printed document for real-time authoring and viewing of annotations. We conducted experience prototyping with eight users to elicit potential benefits and challenges within smartphone AR, and this informed the resulting Dually Noted system and annotation interactions with the document elements. AR annotation is often unwieldy, but during a 12-user empirical study our novel structural understanding component allows Dually Noted to improve precise highlighting and annotation interaction accuracy by 13%, increase interaction speed by 42%, and significantly lower cognitive load over a baseline method without document layout understanding. Qualitatively, participants commented that Dually Noted was a swift and portable annotation experience. Overall, our research provides new methods and insights for how to improve AR annotations for physical documents.

    Original languageEnglish (US)
    Title of host publicationCHI 2022 - Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems
    PublisherAssociation for Computing Machinery
    ISBN (Electronic)9781450391573
    StatePublished - Apr 29 2022
    Event2022 CHI Conference on Human Factors in Computing Systems, CHI 2022 - Virtual, Online, United States
    Duration: Apr 30 2022May 5 2022

    Publication series

    NameConference on Human Factors in Computing Systems - Proceedings


    Conference2022 CHI Conference on Human Factors in Computing Systems, CHI 2022
    Country/TerritoryUnited States
    CityVirtual, Online


    • Annotation
    • augmented reality
    • document interaction
    • layout structure
    • paper
    • smartphone
    • text

    ASJC Scopus subject areas

    • Human-Computer Interaction
    • Computer Graphics and Computer-Aided Design
    • Software


    Dive into the research topics of 'Dually Noted: Layout-Aware Annotations with Smartphone Augmented Reality'. Together they form a unique fingerprint.

    Cite this