DrawTalking: Building Interactive Worlds by Sketching and Speaking

Karl Toby Rosenberg, Rubaiat Habib Kazi, Li Yi Wei, Haijun Xia, Ken Perlin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We introduce DrawTalking, an approach to building and controlling interactive worlds by sketching and speaking while telling stories. It emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study with our prototype shows that the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.

Original languageEnglish (US)
Title of host publicationUIST 2024 - Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
PublisherAssociation for Computing Machinery, Inc
ISBN (Electronic)9798400706288
DOIs
StatePublished - Oct 13 2024
Event37th Annual ACM Symposium on User Interface Software and Technology, UIST 2024 - Pittsburgh, United States
Duration: Oct 13 2024Oct 16 2024

Publication series

NameUIST 2024 - Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology

Conference

Conference37th Annual ACM Symposium on User Interface Software and Technology, UIST 2024
Country/TerritoryUnited States
CityPittsburgh
Period10/13/2410/16/24

Keywords

  • creativity
  • human-AI collaboration
  • multimodal
  • play
  • programmability
  • prototyping
  • sketching

ASJC Scopus subject areas

  • Human-Computer Interaction
  • Software

Fingerprint

Dive into the research topics of 'DrawTalking: Building Interactive Worlds by Sketching and Speaking'. Together they form a unique fingerprint.

Cite this