Korean STT punctuation restoration research
Improving STT output by restoring punctuation from multimodal signals
Context
A university–industry project on STT output quality, the front end of the counselling analysis pipeline.
ASR output arrives without punctuation, which hurts readability and drags down every downstream step — summarisation, emotion analysis, and the rest. But audio is not always available in the pipeline, so the model had to use it when present and still work when absent.
How I solved it
- Surveyed prior work, confirmed where text-only restoration breaks down, and decided to port UniPunc (ICASSP 2022) to Korean
- Encoded text with KoBERT and audio with wav2vec2-xlsr-korean, then aligned the audio frames to text length through a Conv1d subsampler
- Fused the two with Transformer cross-attention
- text as the query, audio as the memory
- Substituted a learned virtual embedding wherever audio was missing, so the model trains multimodally but still runs when no audio arrives
- Put CLOVA ASR in front of it for an end-to-end Streamlit demo that takes a single audio file
Result
- Trained and evaluated on AI Hub welfare call-centre counselling data, classifying four marks
- comma, period, question mark, exclamation mark
- F1 up 15 points over a text-only KoBERT baseline
- The same model survives audio going missing in the pipeline
- it is text-only, so it runs without audio
- The research was handed over to the company