This project note describes challenges and procedures undertaken in annotating an audio-visual dataset capturing a multimodal situated collaborative construction task. In the task, all participants begin with different partial information, and must collaborate using speech, gesture, and action to arrive a solution that satisfies all individual pieces of private information. This rich data poses a number of annotation challenges, from small objects in a close space, to the implicit and multimodal fashion in which participants express agreement, disagreement, and beliefs. We discuss the data collection procedure, annotation schemas and tools, and future use cases.
- Multimodal Common Ground Annotation for Partial Information Collaborative Problem Solving
- Yifan Zhu - Brandeis Univ, Waltham, MA 02453 USAChangsoo Jung - Colorado State UniversityKenneth Lai - Brandeis UniversityVideep Venkatesha - Colorado State UniversityMariah Bradford - Colorado State UniversityJack Fitzgerald - Colorado State UniversityHuma Jamil - Colorado State UniversityCarine Graff - Colorado State UniversitySai Kiran Ganesh Kumar - Colorado State UniversityBruce Draper - Colorado State UniversityNathaniel Blanchard - Colorado State UniversityJames Pustejovsky - Brandeis Univ, Waltham, MA 02453 USANikhil Krishnaswamy - Colorado State University
- B Harry (Editor)
- PROCEEDINGS OF THE 21ST JOINT ACL - ISO WORKSHOP ON INTEROPERABLE SEMANTIC ANNOTATION, ISA-21, pp.85-91
- Assoc Computational Linguistics-Acl
- 7
- W911NF-25-1-0096 / U.S. Army Research Office (ARO) HR00112490377 / U.S. Defense Advanced Research Projects Agency (DARPA) Friction for Accountability in Conversational Transactions (FACT) program
- 9924586151001921
- Michtom School of Computer Science; Benjamin and Mae Volen National Center for Complex Systems; Interdepartmental Program in Linguistics and Computational Linguistics
- English
- Conference proceeding