SurveyMan: Programming and Automatically Debugging Surveys
University of Massachusetts
October 1, 2015
Olin 107
Lunch will be served at noon in the Crochet Lab (Wold 013).


Surveys can be viewed as programs, complete with logic, control flow, and bugs. Word choice or the order in which questions are asked can unintentionally bias responses. Vague, confusing, or intrusive questions can cause respondents to abandon a survey. Surveys can also have runtime errors: inattentive respondents can taint results. This effect is especially problematic when deploying surveys in uncontrolled settings, such as on the web or via crowdsourcing platforms. Because the results of surveys drive business decisions and inform scientific conclusions, it is crucial to make sure they are correct.

We present SurveyMan, a system for designing, deploying, and automatically debugging surveys. Survey authors write their surveys in a lightweight domain-specific language aimed at end users. SurveyMan statically analyzes the survey to provide feedback to survey authors before deployment. It then compiles the survey into JavaScript and deploys it either to the web or a crowdsourcing platform. SurveyMan's dynamic analyses automatically find survey bugs, and control for the quality of responses. We demonstrate its effectiveness with case studies of social science surveys conducted via Amazon's Mechanical Turk.

Bio: Emma Tosch is a PhD candidate in the College of Computer and Information Sciences at University of Massachusetts Amherst. Her thesis work is on programming languages and tools for data scientists and social scientists. She is the lead author and programmer on the SurveyMan project. Her work on SurveyMan has won first place in the 2014 student research competition, a best paper award at OOPSLA 2014, and a 2015 Outstanding Synthesis Award in the College of Computer and Information Sciences. Emma earned her B.A in English Literature from Wellesley College in 2008 before working at a healthcare IT start up. She obtained a post-baccalaureate certificate and M.A. in Computer Science from Brandeis University in 2011.

Please email Kristina Striegnitz ( if you have any questions concerning the seminar series or if you would like to receive the seminar announcements by email.