Speech Quality Assessment

Important dates

September 18: Training Data release

September 25: Public Test

October 9: Private Test

October 15: Result announcement

October 25: Paper submission

November 5: Acceptance notification

November 12: Camera-ready (= > Proceedings)

November 15: Workshop date

General Description

A conventional Speech Quality Assessment (SQA) system returns a single number for a recording. That number says how bad the audio is, but not why, and not where. Three recordings can all receive MOS = 2.8 while one suffers from background noise, another from signal distortion, and the third from unnatural pauses — three different faults requiring three different engineering fixes, indistinguishable from the score alone.
This shared task asks for both. Given a degraded speech recording and no reference signal, a system must produce:

1. Quality prediction. A single overall quality score in [1, 5] for the whole utterance.
2. Fault detection and temporal localization. For each of the three fault types — noise, distortion, and unnatural pause —  whether it is present, and the time intervals in which it occurs. A recording with none of the three is labelled "no fault."

A submission must contain both parts for every recording. Where a system predicts one or more faults, each predicted fault must come with its time intervals: a score without localization, or localization without a score, is not a valid entry. Where a system predicts no fault, the interval list is empty and only the quality score is required.
The practical target is automatic diagnosis: telling a developer that a telephony channel, a TTS voice, or an enhancement front-end is failing, what kind of failure it is, and at which second of the audio to listen — without a human auditing every file.

Dataset

Training, validation, and public test data: We use QualiSpeech (Wang et al., ACL 2025) for the training, validation, and public test sets. QualiSpeech provides interpretable annotations rather than a single score: 14,577 English speech samples, roughly half real human speech (from NISQA and GigaSpeech) and half synthetic speech from recent TTS systems (ChatTTS, XTTS v2, CosyVoice, F5-TTS, E2-TTS, OpenVoice, Parler-TTS, VoiceCraft), under diverse degradation conditions.

Dataset page: https://huggingface.co/datasets/tsinghua-ee/QualiSpeech

Licensing. CC BY-NC-SA 4.0. The BVCC-derived portion may not be redistributed and must be obtained through the official download scripts on the dataset page. Registration includes acceptance of these terms.

Private test set: The final ranking is computed on a private test set built and released later by the organizers.
 

Evaluation Metrics

In this task, three metrics are used: Pearson Correlation Coefficient (PCC), macro-averaged F1-score (F1_macro), and Frame Intersection over Union (IoU). 

• PCC (Pearson): measures the linear correlation between the predicted and ground-truth quality scores. 

• F1_macro: F1 is computed per fault type at the utterance level and macro-averaged over the three fault types (noise, distortion, and unnatural pause). The "no fault" class is excluded. 

• Frame IoU: measures the overlap between the predicted and ground-truth faulty-frame masks on a 20 ms grid: 

  IoU = |predicted ∩ ground truth| / (|predicted ∪ ground truth| + ε) 

  where ε = 1e-8 is used for numerical safety. IoU is computed per recording, then averaged within each fault type, and finally macro-averaged over the three fault types. IoU is conditional on ground-truth presence: for a given fault type, only recordings in which that fault occurs in the ground truth are included in the average. 

Overall score: The final evaluation score is calculated as follows (higher is better): 

Final = 0.40 × PCC + 0.30 × F1_macro + 0.30 × IoU_macro 

Submission Format

[]

Contact

Zalo Group: []

Registration

https://forms.gle/4pncFehjS6ufshBg9

Organizers

• Ta Bao Thang, Hanoi University of Science and Technology, tabaothang97@gmail.com
• Do Van Hai, Thuyloi University, haidv@tlu.edu.vn

References

1. Example