Client session summary and speaker diarization
From raw audio to a per-speaker counselling summary
Context
Writing up session notes eats directly into counsellors’ working hours — a 50-minute session carries 20 to 30 minutes of documentation behind it.
Counsellors wrote up their notes by hand after long sessions, spending real time sifting out what each speaker actually said.
How I solved it
- Designed an end-to-end pipeline: speaker diarization, then speech-to-text, then LLM summarisation
- Combined pyannote diarization with Whisper (medium)
- the two return different timelines, so segments are merged at speaker changes to assign each sentence to a speaker
- Delivered per-speaker summaries and a whole-session summary as separate outputs
Result
- Audio in, per-speaker and full summaries out
- the pipeline shipped to production
- Speaker diarization at a 10% error rate (DER)
- Writing up a session now finishes within five minutes of it ending