Classification of pilots’ mental states using a multimodal deep learning network

In this study, we aimed to investigate the feasibility of a robust detection system of the pilot's mental states (i.e., distraction, workload, fatigue, and normal) based on multimodal biosignals (i.e., electroencephalogram, electrocardiogram, respiration, and electrodermal activity) and a multimodal deep learning (MDL) network. To do this, first, we constructed an experimental environment using a flight simulator in order to induce the different mental states and to collect the biosignals. Second, we designed the MDL architecture – which consists of a convolutional neural network and long short-term memory models – to efficiently combine the information of the different biosignals. Our experimental results successfully show that utilizing multimodal biosignals with the proposed MDL could significantly enhance the detection accuracy of the pilot's mental states.
Source: Biocybernetics and Biomedical Engineering - Category: Biomedical Engineering Source Type: research