Evaluating Modular Dialogue System for Form Filling Using Large Language Models

Hakimov, Sherzod and Weiser, Yan and Schlangen, David

This paper introduces a novel approach to form-filling and dialogue system evaluation by leveraging Large Language Models (LLMs). The proposed method establishes a setup wherein multiple modules collaborate on addressing the form-filling task. The dialogue system is constructed on top of LLMs, focusing on defining specific roles for individual modules. We show that using multiple independent sub-modules working cooperatively on this task can improve performance and handle the typical constraints of using LLMs, such as context limitations. The study involves testing the modular setup on four selected forms of varying topics and lengths, employing commercial and open-access LLMs. The experimental results demonstrate that the modular setup consistently outperforms the baseline, showcasing the effectiveness of this approach. Furthermore, our findings reveal that open-access models perform comparably to commercial models for the specified task.

In Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024) , 2024
[PDF]

@inproceedings{Hakimov-2024,
  title = {Evaluating Modular Dialogue System for Form Filling Using Large Language Models},
  author = {Hakimov, Sherzod and Weiser, Yan and Schlangen, David},
  editor = {Graham, Yvette and Liu, Qun and Lampouras, Gerasimos and Iacobacci, Ignacio and Madden, Sinead and Khalid, Haider and Qureshi, Rameez},
  booktitle = {Proceedings of the 1st Workshop on Simulating Conversational Intelligence in Chat (SCI-CHAT 2024)},
  month = mar,
  year = {2024},
  address = {St. Julians, Malta},
  publisher = {Association for Computational Linguistics},
  url = {https://aclanthology.org/2024.scichat-1.4},
  pages = {36--52}
}