My vision is to build computer systems that can have fluent conversations with humans using English, or another human language, as well as gestures and other non-verbal behaviors.
To have a fluent conversation, the system needs to be aware of the context in which the dialog is taking place. This context includes the human dialog partner, the previous dialog, as well as the surrounding environment. The system should use appropriate linguistic and non-linguistic means to relate to the context. For example, it needs to be able to refer to objects in the environment, and it needs to react to what the user says and does as well as to other changes in the context.
In recent projects, I have, for example, worked on a system that automatically generates English instructions to help a human user solve a task in a 3D virtual environment, or I have helped build an animated figure that can automatically generate the words and gestures to give walking directions across a college campus. My research involves computational modeling and implementation as well as studies of human communication.
Current Projects
The efficacy of a computing-concepts video library for students and peer tutors in multidisciplinary contexts
This multi-institution, NSF-funded IUSE project extends previous work. The earlier project led to the initial development of OCCTIVE, the Online Computing-concepts Toolkit of Interdisciplinary Videos for Education. This video library is intended primarily for use in non-CS courses that use computing, providing students in those courses with strong understanding of foundational computing concepts that will be transferable as they continue to encounter computing within their fields of study. In this new project we are improving the existing videos, adding new videos, and carrying out robust evaluation in order to determine the impact on students’ understanding of and ability to use core computing concepts.
Social robotics
Past Projects
CUE Ethics: Evaluating Frameworks for Incorporating Computing Across the Curriculum
This multi-institution, NSF-funded IUSE project to address computing in undergraduate education resulted in OCCTIVE, the Online Computing-concepts Toolkit of Interdisciplinary Videos for Education. The focus of this video repository is non-CS courses that use computing, providing students in those courses with strong understanding of foundational computing concepts that will be transferable as they continue to encounter computing within their fields of study.
ANTE: A Four-Tier Framework to Boost Visual Literacy for High Dimensional Data
The goal of this project is to develop a system that can present complex, multi-dimensional data in such a way that novice users can make sense of them. The system will automatically generate narratives from data which present relationships in the data through a sequence of visualizations accompanied by natural language text.