TY - GEN

T1 - Knowledge Transfer for on-Device Speech Emotion Recognition With Neural Structured Learning

AU - Chang, Yi

AU - Ren, Zhao

AU - Nguyen, Thanh Tam

AU - Qian, Kun

AU - Schuller, Bjorn W.

N1 - Funding Information: This research was partially funded by the Federal Ministry of Education and Research (BMBF), Germany under the project LeibnizKILabor with grant No. 01DD20003, and the research projects “IIP-Ecosphere”, granted by the German Federal Ministry for Economics and Climate Action (BMWK) via funding code No. 01MK20006A, the Ministry of Science and Technology of the People’s Republic of China (No. 2021ZD0201900), the National Natural Science Foundation of China (No. 62272044), the National High-Level Young Talent Project, and the BIT Teli Young Fellow Program from the Beijing Institute of Technology, China.

PY - 2023

Y1 - 2023

N2 - Speech emotion recognition (SER) has been a popular research topic in human-computer interaction (HCI). As edge devices are rapidly springing up, applying SER to edge devices is promising for a huge number of HCI applications. Although deep learning has been investigated to improve the performance of SER by training complex models, the memory space and computational capability of edge devices represents a constraint for embedding deep learning models. We propose a neural structured learning (NSL) framework through building synthesized graphs. An SER model is trained on a source dataset and used to build graphs on a target dataset. A relatively lightweight model is then trained with the speech samples and graphs together as the input. Our experiments demonstrate that training a lightweight SER model on the target dataset with speech samples and graphs can not only produce small SER models, but also enhance the model performance compared to models with speech samples only and those using classic transfer learning strategies.

AB - Speech emotion recognition (SER) has been a popular research topic in human-computer interaction (HCI). As edge devices are rapidly springing up, applying SER to edge devices is promising for a huge number of HCI applications. Although deep learning has been investigated to improve the performance of SER by training complex models, the memory space and computational capability of edge devices represents a constraint for embedding deep learning models. We propose a neural structured learning (NSL) framework through building synthesized graphs. An SER model is trained on a source dataset and used to build graphs on a target dataset. A relatively lightweight model is then trained with the speech samples and graphs together as the input. Our experiments demonstrate that training a lightweight SER model on the target dataset with speech samples and graphs can not only produce small SER models, but also enhance the model performance compared to models with speech samples only and those using classic transfer learning strategies.

KW - edge device

KW - lightweight deep learning

KW - neural structured learning

KW - Speech emotion recognition

UR - http://www.scopus.com/inward/record.url?scp=85171249470&partnerID=8YFLogxK

U2 - 10.48550/arXiv.2210.14977

DO - 10.48550/arXiv.2210.14977

M3 - Conference contribution

AN - SCOPUS:85171249470

SN - 978-1-7281-6328-4

BT - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 48th IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2023

Y2 - 4 June 2023 through 10 June 2023

ER -