DSAIL.(지도교수: 한진영)의 이다은(인공지능융합학과), 손세정(인공지능융합학과), 전효림(인공지능융합학과) 학생들이 연구한 논문 “Learning Co-Speech Gesture for Multimodal Aphasia Type Detection” 이 세계 최고 권위 자연어처리 학회인 EMNLP 2023 (The 2023 Conference on Empirical Methods in Natural Language Processing), Main conference paper로 채택되었습니다. 논문은 23년 12월 싱가포르에서 발표될 예정입니다.

본 논문은 인공지능융합학과 박사과정 및 석사과정 학생들의 협업을 통한 결과물로서, 실어증 유형을 예측하기 위해 음성과 제스처 간의 상관관계를 반영하는 multimodal graph neural network를 제안하였습니다. 논문의 자세한 내용은 다음과 같습니다.

[논문]

Daeun Lee, Sejung Son, Hyolim Jeon, Seungbae Kim and Jinyoung Han, ““Learning Co-Speech Gesture for Multimodal Aphasia Type Detection,” The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), Dec. 2023.

[Abstract].

Aphasia, a language disorder resulting from brain damage, requires accurate identification of specific aphasia types, such as Broca’s and Wernicke’s aphasia, for effective treatment. However, little attention has been paid to developing methods to detect different types of aphasia. Recognizing the importance of analyzing co-speech gestures for distinguish aphasia types, we propose a multimodal graph neural network for aphasia type detection using speech and corresponding gesture patterns. By learning the correlation between the speech and gesture modalities for each aphasia type, our model can generate textual representations sensitive to gesture information, leading to accurate aphasia type detection. Extensive experiments demonstrate the superiority of our approach over existing methods, achieving state-of-the-art results (F1 84.2%). We also show that gesture features outperform acoustic features, highlighting the significance of gesture expression in detecting aphasia types.