Multimodal emotion analysis
Reading emotion from counselling audio and text together
Context
I owned the emotion analysis module of a product that analyses counselling sessions.
Text alone loses the emotional signal carried by tone and intonation, so the analysis was hard to trust.
How I solved it
- Rather than scaling the text model, I went after the signal it never had
- tone and intonation are absent from a transcript, so no amount of model size recovers them
- Set a KoBERT text-only baseline first, then added audio on the same data so the difference was the only thing being measured
- Ported TEASEL (a speech prefix on RoBERTa) and validated it on the CMU-MOSI benchmark before moving it to Korean
- Reused the structure from the punctuation restoration research directly: a Conv1d subsampler aligning audio frames to text length, Transformer cross-attention for fusion, and a learned virtual embedding wherever audio was missing
- Built the multimodal preprocessing pipeline and a real-time inference server on BentoML
Result
- F1 score went from 0.71 to 0.81 against the text-only model
- Classified across seven labels
- Ekman’s six basic emotions plus neutral: happiness, anger, disgust, fear, sadness, surprise, neutral
- Trained on roughly 10GB of paired counselling audio and transcripts