Multimodal emotion analysis
Reading emotion from counselling audio and text together
Context
I owned the emotion analysis module of a product that analyses counselling sessions.
Problem
Text alone loses the emotional signal carried by tone and intonation, so the analysis was hard to trust.
How I solved it
- Rather than scaling the text model, I went after the signal it never had — tone and intonation are absent from a transcript, so no amount of model size recovers them
- Designed an end-to-end inference structure that folds audio in as a second modality
- Adopted a fine-grained emotion taxonomy to capture subtler states
- Built the multimodal preprocessing pipeline and a real-time inference server on BentoML
Result
- F1 score improved 10% over the text-only baseline
Impact
Emotional signals that text alone missed now reach the analysis.