The Speech Assistant as In-Car Passenger

A general theme of our work is exploring what it would mean for a human and a dialogue system / conversational agent to be in a truly joint situation. In the project described here, we have looked at a very specific, and very common, type of situation: driving in a car.

Executive summary: We’ve built a dialogue system / in-car speech assistant that reacts to the driving situation and can interrupt itself. We have tested it in a driving simulation, and found that it causes much less distraction for the driver than an assistant that behaves like current commercial systems, which cannot interrupt themselves.

Publications: (Lopez Gambino, Kennington, and Schlangen 2017), (Kennington et al. 2014), (Kousidis et al. 2014), (Kousidis et al. 2014)

1. Maria Soledad Lopez Gambino, Casey Kennington, and David Schlangen Silence, Please! Interrupting In-Car Phone Conversations Proceedings of the First Workshop on Conversational Interruptions in Human-Agent Interactions (CIHAI 2017) 2017 [PDF]

   BibTeX

   @inproceedings{Lopez_Gambino-2017-1,
     author = {Lopez Gambino, Maria Soledad and Kennington, Casey and Schlangen, David},
     editor = {Cafaro, Angelo and Coutinho, Eduardo and Gebhard, Patrick and Potard, Blaise},
     booktitle = {Proceedings of the First Workshop on Conversational Interruptions in Human-Agent Interactions (CIHAI 2017)},
     issn = {1613-0073},
     location = {Stockholm},
     pages = {9 -- 18},
     title = {{Silence, Please! Interrupting In-Car Phone Conversations}},
     volume = {1943},
     year = {2017},
     topics = {},
     domains = {},
     approach = {},
     project = {}
   }
   

   Details
2. Casey Kennington, Spyridon Kousidis, Timo Baumann, Hendrik Buschmeier, Stefan Kopp, and David Schlangen Better Driving and Recall When In-car Information Presentation Uses Situationally-Aware Incremental Speech Output Generation AutomotiveUI 2014: Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications 2014 [Abs] [PDF]

   It is established that driver distraction is the result of sharing cognitive resources between the primary task (driving) and any other secondary task. In the case of holding conversations, a human passenger who is aware of the driving conditions can choose to interrupt his speech in situations potentially requiring more attention from the driver, but in-car information systems typically do not exhibit such sensitivity. We have designed and tested such a system in a driving simulation environment. Unlike other systems, our system delivers infor- mation via speech (calendar entries with scheduled meetings) but is able to react to signals from the environment to interrupt when the driver needs to be fully attentive to the driving task and subsequently resume its delivery. Distraction is measured by a secondary short-term memory task. In both tasks, drivers perform significantly worse when the system does not adapt its speech, while they perform equally well to control conditions (no concurrent task) when the system intelligently interrupts and resumes.

   BibTeX

   @inproceedings{Kennington-2014,
     author = {Kennington, Casey and Kousidis, Spyridon and Baumann, Timo and Buschmeier, Hendrik and Kopp, Stefan and Schlangen, David},
     booktitle = {AutomotiveUI 2014: Proceedings of the 6th International Conference on Automotive User Interfaces and Interactive Vehicular Applications},
     location = {Seattle, Washington, USA},
     pages = {7:1--7:7},
     title = {{Better Driving and Recall When In-car Information Presentation Uses Situationally-Aware Incremental Speech Output Generation}},
     doi = {10.1145/2667317.2667332},
     year = {2014},
     topics = {},
     domains = {},
     approach = {},
     project = {}
   }
   

   Details
3. Spyridon Kousidis, Casey Kennington, Timo Baumann, Hendrik Buschmeier, Stefan Kopp, and David Schlangen Situationally Aware In-Car Information Presentation Using Incremental Speech Generation: Safer, and More Effective Proceedings of the EACL 2014 Workshop on Dialogue in Motion 2014 [Abs] [PDF]

   Holding non-co-located conversations while driving is dangerous (Horrey and Wickens, 2006; Strayer et al., 2006), much more so than conversations with physically present, “situated” interlocutors (Drews et al., 2004). In-car dialogue systems typically resemble non-co-located conversations more, and share their negative impact (Strayer et al., 2013). We implemented and tested a simple strategy for making in-car dialogue systems aware of the driving situation, by giving them the capability to interrupt themselves when a dangerous situation is detected, and resume when over. We show that this improves both driving performance and recall of system-presented information, compared to a non-adaptive strategy.

   BibTeX

   @inproceedings{Kousidis-2014,
     author = {Kousidis, Spyridon and Kennington, Casey and Baumann, Timo and Buschmeier, Hendrik and Kopp, Stefan and Schlangen, David},
     booktitle = {Proceedings of the EACL 2014 Workshop on Dialogue in Motion},
     location = {Gothenburg, Sweden},
     pages = {68--72},
     title = {{Situationally Aware In-Car Information Presentation Using Incremental Speech Generation: Safer, and More Effective}},
     doi = {10.3115/v1/W14-0212},
     year = {2014},
     topics = {},
     domains = {},
     approach = {},
     project = {}
   }
   

   Details
4. Spyridon Kousidis, Casey Kennington, Timo Baumann, Hendrik Buschmeier, Stefan Kopp, and David Schlangen A Multimodal In-Car Dialogue System That Tracks The Driver’s Attention Proceedings of the 16th International Conference on Multimodal Interfaces 2014 [Abs] [PDF]

   When a passenger speaks to a driver, he or she is co-located with the driver, is generally aware of the situation, and can stop speaking to allow the driver to focus on the driving task. In-car dialogue systems ignore these important aspects, making them more distracting than even cell-phone conversations. We developed and tested a “situationally-aware” dialogue system that can interrupt its speech when a situation which requires more attention from the driver is detected, and can resume when driving conditions return to normal. Furthermore, our system allows driver-controlled resumption of interrupted speech via verbal or visual cues (head nods). Over two experiments, we found that the situationally-aware spoken dialogue system improves driving performance and attention to the speech content, while driver-controlled speech resumption does not hinder performance in either of these two tasks.

   BibTeX

   @inproceedings{Kousidis-2014-1,
     author = {Kousidis, Spyridon and Kennington, Casey and Baumann, Timo and Buschmeier, Hendrik and Kopp, Stefan and Schlangen, David},
     booktitle = {Proceedings of the 16th International Conference on Multimodal Interfaces},
     location = {Istanbul, Turkey},
     pages = {26--33},
     title = {{A Multimodal In-Car Dialogue System That Tracks The Driver's Attention}},
     doi = {10.1145/2663204.2663244},
     year = {2014},
     topics = {},
     domains = {},
     approach = {},
     project = {}
   }
   

   Details

A collaboration with the Social Cognitive Systems Group, Prof. Stefan Kopp, and Dr. Timo Baumann, Universität Hamburg.

(This work has been featured in the “Neue Westfälische” and “Westfalenblatt” newspapers on Nov 21 2014.)