I made a Speech Emotion Recognition application called “I Am You Are” to help people with autism, who experience difficulties in understanding people’s emotions.
Web version demo
Mobile version demo
Motivation
Many people with autism struggle to classify people’s emotions, so they learn how to distinguish emotions according to facial and vocal expressions. Considering that some autistic people find it hard to make eye contact, I wanted to make an emotion recognition program using speech.
Datasets
We used Korean audio speech data from AI-HUB, and used English audio speech data such as RAVDESS, TESS, EMO-DB. All of the data were labed to one of the seven emotions : Sadness, Angry, Fear, Disgust, Neutral, Surprise, Happiness. Due to the poor quality of some of the provided Korean speech data, I had to exclude many Sadness, Happiness, and Surprise data files and record some by myself.
Data pre-processing and creating Models
I used handcrafted features such as MFCC (Mel-Frequency Cepstral Coefficient), MFCC chunks, Mel Spectrogram Frequency, Chroma, Contrast, and Tonnetz as input features. I made several models using open source libraries such as sklearn and torch. I chose the model I made with classifiers, which showed an accuracy of 78%.
Making it into an Application
This was the tricky part because making a deep learning application is not easy. My team and I used flask to create an application that can be used on both web and mobile. I made a speech record function for the users to record their or another person’s speech whenever they feel the need to understand speech emotion.
See how the application works in this video. I used my voice and a speech file of a man speaking Korean dialect. (Although it is mostly in Korean, you can still check how it shows the predicted emotion!)
Presentation
I won the Ministry of Science and ICT Minister Award and People’s choice Award in a competition hosted by the Ministry of Science and ICT and supervised by the National Research Foundation of Korea, South Korea.
Presentation Slide (Korean) Link
Presentation Video (Korean) Link
If you want to see me receiving the award via zoom and giving an acceptance speech, then click!