Human-Computer Dialog Toolkit


Basically everything humans do with technology includes some form of “dialog” with a computer. Filling out a web form or clicking with a mouse are examples of human-computer communication. This research focusses on a higher-level computer understanding of human communication, i.e., natural language understanding.

Consider a dialog with someone behind the counter at Subway: They might first ask you to select a bread, then ask you what cheese and what meat. Next if you want your sandwich toasted. You might then select toppings. What if approached the counter and said everything you wanted all at once, before you were prompted for anything? A human may remember your order in its entirety and complete the order without having to prompt you for any more choices. Or, perhaps you forgot to specify whether to toast your sandwich or not and the person behind the counter asked you that first instead of what bread you want. This is human to human mixed-initiative interaction. Humans are imperfect and may not remember your order in completion or may even forget to ask you a question such as whether you want to toast your bread. This is where I believe computers can outperform humans using human to computer mixed-initiative interaction.

This toolkit was designed to prototype many human-computer interactions that may occur from a generalized, easy-to-use, programming-not-required model.

Here is a quick poster about the project.

See Research

See Human Computer Interaction on wikipedia.


Perugini, S. & Buck, J.W. (2016). A language-based model for specifying and staging mixed-initiative dialogs. In Proceedings of the Eighth ACM SIGCHI Symposium on Engineering Interactive Computing Systems, New York, NY: ACM Press. (Accepted; to appear.)

Buck, J.W. & Perugini, S. (2016). A tool for staging mixed-initiative dialogs. In Proceedings of the Twenty-seventh Annual Modern Artificial Intelligence and Cognitive Science Conference.

More Coming. . .

  • Any situation that involves ordering, including
    • Kiosks
    • ATMs
    • Gas Stations
    • Digital form completion
  • Smart Phone Technologies
    • Personal Assistant technology (Siri, Google Now, Cortana, etc. )
  • AI (simulating human-human dialog)
  • Cross-platform support! Windows, Mac, Linux
  • Rapidly prototype (or script) a human-computer dialog
  • Change the flexibility of the dialog without any rewriting
    • Allow multiple responses at a time
    • Enforce an order of responses
    • Require sets of responses at a time
    • Allow out-of-turn (out of order) responses
    • A dialog with only 3 states can have 13 different orderings/combinations and there are 8196 combinations (of varying size) of these 13.
  • Automatically generate a client to host the dialog
  • Automatically support natural language processing, including:
    • Category (or question -answer) recognition
    • Misspelled word estimation
    • Implicit information extraction
    • Human Queries
  • No programming required
Downloads (repositories):

Engine & Server