EEG-Based Machine Learning Models to Evaluate Haptic Delay: Should We Label Data Based on Self-Reporting or Physical Stimulation?

Reliable haptic interfaces augment human-computer interaction via simulated tactile and kinesthetic feedback. As haptic technologies advance, user experience evaluation becomes more crucial. Conventionally, self-reporting is used to evaluate haptic experiences; however, it could be inconsistent or imprecise due to human error. A promising alternative is using neurocognitive methods with machine or deep learning models to evaluate the human haptic experience. Machine and deep learning models can be trained on Electroencephalography (EEG) data labeled based on self-report or actual physical stimulation. As the literature lacks a systematic study on which approach is more robust, we develop a visuo-haptic task to answer this question by examining an important haptic experience, namely, haptic delay. EEG is recorded during the experiment, and participants report whether they detected a delay in the haptic modality through self-report. Four machine/deep learning models were trained twice on the EEG data using the two labeling methods. Models trained with labels from the physical stimuli significantly outperformed those trained with self-reporting labels. Although this finding holds true for one particular haptic experience (haptic delay), it cannot be extrapolated to others; rather, it suggests that EEG data labeling plays a prominent role in evaluating the haptic experience through neurocognitive methods.