Visual Interaction with Deep Learning Models through Collaborative Semantic Inference

Sebastian Gehrmann, Hendrik Strobelt, Robert Kruger, Hanspeter Pfister, Alexander M. Rush

    Research output: Contribution to journalArticlepeer-review

    Abstract

    Automation of tasks can have critical consequences when humans lose agency over decision processes. Deep learning models are particularly susceptible since current black-box approaches lack explainable reasoning. We argue that both the visual interface and model structure of deep learning systems need to take into account interaction design. We propose a framework of collaborative semantic inference (CSI) for the co-design of interactions and models to enable visual collaboration between humans and algorithms. The approach exposes the intermediate reasoning process of models which allows semantic interactions with the visual metaphors of a problem, which means that a user can both understand and control parts of the model reasoning process. We demonstrate the feasibility of CSI with a co-designed case study of a document summarization system.

    Original languageEnglish (US)
    Article number8805457
    Pages (from-to)884-894
    Number of pages11
    JournalIEEE Transactions on Visualization and Computer Graphics
    Volume26
    Issue number1
    DOIs
    StatePublished - Jan 2020

    Keywords

    • Deep Learning
    • Human-Centered Design
    • Human-Computer Collaboration
    • Interaction Design
    • Neural Networks

    ASJC Scopus subject areas

    • Software
    • Signal Processing
    • Computer Vision and Pattern Recognition
    • Computer Graphics and Computer-Aided Design

    Fingerprint

    Dive into the research topics of 'Visual Interaction with Deep Learning Models through Collaborative Semantic Inference'. Together they form a unique fingerprint.

    Cite this