Multimodal transcript of face-to-face group-work activity around interactive tabletops

This paper describes a multimodal system around a multi-touch tabletop to collect different data sources for group-work activities. The system collects data from various cameras, microphones and the logs of the activities performed in the multi-touch tabletop. We conducted a pilot study with 27 students in an authentic classroom to explore the feasibility of capture individual and group interactions of each participant in a collaborative database design activity. From the raw data, we extracted low-level features (e.g. tabletop action, gaze interaction, verbal intervention, emotions) and generated some visualizations as annotated transcripts of what happened in a work session. We evaluated teacher’s perception about how the automated multimodal transcript could potentially support the understanding of group-work activities. Results from teachers’ perceptions pointed out that the multimodal transcript could become a valuable tool to understand group-work rapport and performance.