Speech Quality Assessment
Important dates
September 18: Training Data release
September 25: Public Test
October 9: Private Test
October 15: Result announcement
October 25: Paper submission
November 5: Acceptance notification
November 12: Camera-ready (= > Proceedings)
November 15: Workshop date
General Description
A conventional Speech Quality Assessment (SQA) system returns a single number for a recording. That number says how bad the audio is, but not why, and not where. Three recordings can all receive MOS = 2.8 while one suffers from background noise, another from signal distortion, and the third from unnatural pauses — three different faults requiring three different engineering fixes, indistinguishable from the score alone.
This shared task asks for both. Given a degraded speech recording and no reference signal, a system must produce:
- Quality prediction. A single overall quality score in [1, 5] for the whole utterance.
- Fault detection and temporal localization. For each of the three fault types — noise, distortion, and unnatural pause — whether it is present, and the time intervals in which it occurs. A recording with none of the three is labelled "no fault."
A submission must contain both parts for every recording. Where a system predicts one or more faults, each predicted fault must come with its time intervals: a score without localization, or localization without a score, is not a valid entry. Where a system predicts no fault, the interval list is empty and only the quality score is required.
The practical target is automatic diagnosis: telling a developer that a telephony channel, a TTS voice, or an enhancement front-end is failing, what kind of failure it is, and at which second of the audio to listen — without a human auditing every file.
Dataset
Training, validation, and public test data: We use QualiSpeech (Wang et al., ACL 2025) for the training, validation, and public test sets. QualiSpeech provides interpretable annotations rather than a single score: 14,577 English speech samples, roughly half real human speech (from NISQA and GigaSpeech) and half synthetic speech from recent TTS systems (ChatTTS, XTTS v2, CosyVoice, F5-TTS, E2-TTS, OpenVoice, Parler-TTS, VoiceCraft), under diverse degradation conditions.
Dataset page: https://huggingface.co/datasets/tsinghua-ee/QualiSpeech
Licensing. CC BY-NC-SA 4.0. The BVCC-derived portion may not be redistributed and must be obtained through the official download scripts on the dataset page. Registration includes acceptance of these terms.
Private test set: The final ranking is computed on a private test set built and released later by the organizers.
Evaluation Metrics
In this task, three metrics are used: Pearson Correlation Coefficient (PCC), macro-averaged F1-score (F1_macro), and Frame Intersection over Union (IoU).
PCC (Pearson): measures the linear correlation between the predicted and ground-truth quality scores.
F1_macro: F1 is computed per fault type at the utterance level and macro-averaged over the three fault types (noise, distortion, and unnatural pause). The "no fault" class is excluded.
Frame IoU: measures the overlap between the predicted and ground-truth faulty-frame masks on a 20 ms grid:
IoU = |predicted ∩ ground truth| / (|predicted ∪ ground truth| + ε)
where ε = 1e-8 is used for numerical safety. IoU is computed per recording, then averaged within each fault type, and finally macro-averaged over the three fault types. IoU is conditional on ground-truth presence: for a given fault type, only recordings in which that fault occurs in the ground truth are included in the average.
Overall score: The final evaluation score is calculated as follows (higher is better):
Final = 0.40 × PCC + 0.30 × F1_macro + 0.30 × IoU_macro
Submission Format
[]
Contact
Zalo Group: []
Registration
https://forms.gle/4pncFehjS6ufshBg9
Organizers
- Ta Bao Thang, Hanoi University of Science and Technology, tabaothang97@gmail.com
- Do Van Hai, Thuyloi University, haidv@tlu.edu.vn
References
- Example
Sponsors and Partners
