Vietnamese Non-Verbal Text-to-Speech for Conversational Synthesis

Important dates

September 18: Training Data release

September 25: Public Test

October 9: Private Test

October 15: Result announcement

October 25: Paper submission

November 5: Acceptance notification

November 12: Camera-ready (=> Proceedings)

November 15: Workshop date

General Description

 ViNV-TTS is a shared task on synthesizing natural Vietnamese conversational speech that contains non-verbal vocalizations (NVs) - communicative sounds that are not words, such as laughter, breathing, sniffing, and throat-clearing. Given Vietnamese text with inline NV tags at explicit positions, systems must produce speech in which each tagged NV is present, of the correct type, placed at the intended position, and acoustically natural and well-integrated with the surrounding speech, while preserving intelligibility and the identity of the target speaker. 

The task has two tag-conditioned tracks that differ only in speaker setting: 
• Track A (Core): synthesizes speech for a target speaker seen during training. Zero-shot voice cloning is not required.
• Track B (Advanced): synthesizes speech for an unseen target speaker given a short reference audio clip, requiring zero-shot voice cloning.

Systems are evaluated through both human and automatic evaluation. Human evaluation includes speech naturalness, audio quality, NV naturalness and NV placement accuracy. Automatic evaluation includes WER, predicted MOS and speaker similarity.

Dataset

The task releases the first public Vietnamese conversational speech corpus annotated with non-verbal tags, providing resources for training and system development. It is built from Vietnamese conversational sources such as podcasts and talk shows via an automated construction pipeline. 

The corpus is multi-speaker, segmented into short conversational utterances, and annotated with four non-verbal tags - [laughter], [breathing], [sniff], and [throatclearing] - placed inline at their point of occurrence. Each item provides the audio (single-channel, 24 kHz), a transcript containing inline non-verbal tags, per-event non-verbal timing metadata (type, onset, offset), and a speaker ID. Exact corpus statistics (total duration, number of utterances, and per-tag counts) will be finalized and reported at release.

The data is split into public Train/Dev and Public test portions, plus a held-out Private test with two versions: 
• Track A: new text with gold non-verbal tags on speakers seen during training
• Track B: new text with gold non-verbal tags and short reference audio clips from held-out, unseen speakers, requiring zero-shot voice cloning.

Evaluation Metrics

Three main aspects will be used to evaluate the submitted models: Non-Verbal Vocalization (NV) Realization, Speech/Audio Quality, and Speaker Similarity.

• NV Placement Accuracy (NVPA): Measures whether the required non-verbal vocalizations are correctly realized at their intended positions in the synthesized speech. An NV event is considered correctly realized only when the required NV type is present at the intended position. Range: 0 to 1, where higher is better.
• NV Naturalness (NVN): Measures how natural and human-like the synthesized non-verbal vocalizations sound and how naturally they are integrated with the surrounding speech. Scale: 1 to 5, where higher is better.
• Speech Naturalness (SN): Measures the overall naturalness, fluency, and human-likeness of the synthesized speech, including the speech surrounding the non-verbal vocalizations. Scale: 1 to 5, where higher is better.
• Quality (Q): Measures the overall perceptual quality of the synthesized audio, including audible artifacts, distortions, noise, and other audio degradations. Scale: 1 to 5, where higher is better.
• Word Error Rate (WER): Evaluates the intelligibility and content accuracy of the synthesized speech by comparing the recognized speech with the input transcript using a pretrained ASR model (Zipformer). Non-verbal tags are excluded from WER computation. Range: 0 to 1, where lower is better.
• Predicted MOS (pMOS): Measures the predicted perceptual quality and naturalness of the synthesized speech using an automatic MOS prediction model (DNSMOS). Scale: 1 to 5, where higher is better.
• Speaker Similarity (SS): Measures the similarity between the target/reference speaker and the synthesized speech using a pretrained speaker recognition model (ECAPA-TDNN). Range: 0 to 1, where higher is better.

Final Scoring Formula:

• Track A: A = 0.3*NVPA + 0.2*NVN + 0.1*SN + 0.1*Q + 0.1*(1 - WER) + 0.1*pMOS + 0.1*SS;
• Track B: B = 0.25*NVPA + 0.15*NVN + 0.1*SN + 0.1*Q + 0.1*(1 - WER) + 0.1*pMOS + 0.2*SS
*Note: All components of the scoring formula are normalized to [0, 1].

Submission Format

After training their systems, participating teams are required to deploy their models as an API endpoint. During the evaluation phase, the organizers will send inference requests to the submitted API and collect the resulting synthesized speech. Each team will be provided with an allocated evaluation period, which will be announced by the organizers in advance. The API must remain accessible and operational throughout the assigned evaluation period and return the synthesized audio for each valid input request.

Contact

Zalo Group: []

Registration

https://docs.google.com/forms/d/e/1FAIpQLSdT2KQCWZAsb3BYgqfD2uBDXg_SFPt5LN3cUmeHrZ46fZqb6Q/viewform

Organizers

• Nguyễn Hoàng Kỳ - Vbee AITalk JSC - kynh@vbeeai.com
• Trần Sơn Tùng - Vbee AITalk JSC - tungts@vbeeai.com
• Nguyễn Thị Thu Trang - Vbee AITalk JSC - trangntt@vbeeai.com

References

1. Yao, Z., Guo, L., Yang, X., Kang, W., Kuang, F., Yang, Y., ... & Povey, D. (2024, May). Zipformer: A faster and better encoder for automatic speech recognition. In International Conference on Learning Representations (Vol. 2024, pp. 44440-44455).
2. Ye, R., Zhou, Y., Yu, R., Lin, Z., Li, K., Li, X., ... & Wu, Z. (2025). A scalable pipeline for enabling non-verbal speech generation and understanding. arXiv preprint arXiv:2508.05385.