Non-contact detection of mentalfatiguefromfacial expressions andheartsignals: A self-supervised-based multimodal fusion method

Detecting mental fatigue is crucial for preventing accidents in critical areas. This paper presents a non-contact multimodal mental fatigue detection method to improve convenience and accuracy. Our approach combines facial expression sequences with physiological signals from frequency-modulated continuous wave (FMCW) radar. We also developed a self-supervised learning-based multimodal framework that enhances detection accuracy. Additionally, we created a specialized dataset using the psychological AX-CPT paradigm and conducted comparative studies with similar public datasets. Experimental results show that our multimodal self-supervised learning strategy significantly improves non-contact mental fatigue detection accuracy, achieving an optimal accuracy of 0.918 on our self-build dataset and performing well on comparable public datasets.

Shengjian Hu, Weining Fang, Haifeng Bao , Tianlong Zhang.Non-contact detection of mentalfatiguefromfacial expressions andheart signals: A self-supervised-based multimodal fusion method. BiomedicalSignalProcessingandControl,https://doi.org/10.1016/j.bspc.2025.107658