Generative bookscapes: Towards immersive and interactive book reading

Tae Hong Park, Toshihisa Tsuruoka

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper we introduce the Bookscapes project. Which focuses on augmenting the traditional book-reading experience through generative sound, music, and soundscape creation utilizing sound synthesis, music information retrieval (MIR) techniques, and computational emotion analysis. The current prototype software is implemented in MATLAB and MAX. The MATLAB environment employs real-time convolutional neural networks (CNN)to detect speech when a user is reading a book (e.g. bedtime storytelling for children). MATLAB is also used to adaptively generate background soundscapes by reacting to the reader's voice (when present) and text contained in pages of a given book. The MAX environment is used to generate synthesized sounds and “music” by exploiting the software's well-documented utility as it is the most widely used computer music tool today.

Original languageEnglish (US)
Title of host publicationProceedings of the 2019 International Computer Music Conference, ICMC-NYCEMF 2019 - International Computer Music Conference New York City Electroacoustic Music Festival
PublisherInternational Computer Music Association
Pages207-212
Number of pages6
ISBN (Electronic)0984527486, 9780984527489
StatePublished - Sep 1 2019
Event45th International Computer Music Conference, ICMC 2019 and International Computer Music Conference New York City Electroacoustic Music Festival, NYCEMF 2019 - New York, United States
Duration: Jun 16 2019Jun 23 2019

Publication series

NameProceedings of the 2019 International Computer Music Conference, ICMC-NYCEMF 2019 - International Computer Music Conference New York City Electroacoustic Music Festival

Conference

Conference45th International Computer Music Conference, ICMC 2019 and International Computer Music Conference New York City Electroacoustic Music Festival, NYCEMF 2019
Country/TerritoryUnited States
CityNew York
Period6/16/196/23/19

Keywords

  • Audiobooks
  • Computer music system
  • Generative audio
  • Machine learning
  • MATLAB
  • Soundscapes

ASJC Scopus subject areas

  • Media Technology
  • Music
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Generative bookscapes: Towards immersive and interactive book reading'. Together they form a unique fingerprint.

Cite this