The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling

Madureira, Brielen and Kahardipraja, Patrick and Schlangen, David

Incremental dialogue model components produce a sequence of output prefixes based on incoming input. Mistakes can occur due to local ambiguities or to wrong hypotheses, making the ability to revise past outputs a desirable property that can be governed by a policy. In this work, we formalise and characterise edits and revisions in incremental sequence labelling and propose metrics to evaluate revision policies. We then apply our methodology to profile the incremental behaviour of three Transformer-based encoders in various tasks, paving the road for better revision policies.

In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue , 2023
[PDF]

@inproceedings{Madureira-2023-3,
  title = {The Road to Quality is Paved with Good Revisions: A Detailed Evaluation Methodology for Revision Policies in Incremental Sequence Labelling},
  author = {Madureira, Brielen and Kahardipraja, Patrick and Schlangen, David},
  editor = {Stoyanchev, Svetlana and Joty, Shafiq and Schlangen, David and Dusek, Ondrej and Kennington, Casey and Alikhani, Malihe},
  booktitle = {Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue},
  month = sep,
  year = {2023},
  address = {Prague, Czechia},
  publisher = {Association for Computational Linguistics},
  url = {https://aclanthology.org/2023.sigdial-1.14},
  doi = {10.18653/v1/2023.sigdial-1.14},
  pages = {156--167}
}