Fundamental performance analysis for spoken dialogue systems based on a quantitative simulation approach
Journal
ICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
Journal Volume
2
Pages
1221-1224
Date Issued
2000
Author(s)
Lin B.-S
Abstract
The performance of dialogue systems is mostly measured based on the analysis of a large dialogue corpus. In this way, the dialogue performance can not be obtained before the system is on line, and the dialogue corpus should be recollected if the system is modified. Also, the effect of different factors, including system dialogue strategy, recognition and understanding accuracy or user response pattern, etc., on the dialogue performance can not be quantitatively identified because they can not be precisely controlled in different corpora. In this paper, a fundamental performance analysis scheme for dialogue systems based on a quantitative simulation approach is proposed. With this scheme the fundamental performance of a dialogue system can be predicted and analyzed efficiently without having any real spoken dialogue system implemented or having any dialogue corpus actually collected. How the dialogue performance varies with respect to each factor, from recognition accuracy to dialogue strategy, can be individually identified, because all such factors can be precisely controlled in the simulation. The quality of service for the spoken dialogue system can also be flexibly defined and the design parameters easily determined. This approach is therefore very useful for the design, development and improvement of spoken dialogue although the on-line system and real corpus will eventually be needed in the final evaluation and analysis of the system performance in any case. © 2000 IEEE.
Other Subjects
Linguistics; Pattern recognition systems; Quality of service; Signal processing; Speech processing; Computational linguistics; Computer simulation; Finite automata; Performance; Random processes; Statistical methods; Design parameters; Dialogue strategy; Evaluation and analysis; Performance analysis; Quantitative simulation; Recognition accuracy; Response patterns; Spoken dialogue system; Speech recognition; Speech transmission; Quality of service; Quantitative simulation approach; Spoken dialogue systems
Type
conference paper