Richard Brutti

Richard Brutti

Computational Linguist

Posts by Collection

portfolio

publications

Attention-based Sequence Classification for Affect Detection

Published in Interspeech, 2018

An attention-based LSTM model for recognizing emotion from speech, using class balancing strategies to improve detection of underrepresented emotional states, as part of the Interspeech ComParE challenge.

Cristina Gorrostieta, Richard Brutti, Kye Taylor, Avi Shapiro, Joseph Moran, Ali Azarbayejani, and John Kane (2018). Attention-based Sequence Classification for Affect Detection. In Proceedings of Interspeech 2018, pages 506-510.
Download Paper

Gender De-Biasing in Speech Emotion Recognition

Published in Interspeech, 2019

We demonstrate that speech emotion recognition models exhibit gender bias with lower accuracy for female speakers, and achieve more equitable performance across genders with a de-biasing training technique.

Cristina Gorrostieta, Reza Lotfian, Kye Taylor, Richard Brutti, and John Kane (2019). Gender De-Biasing in Speech Emotion Recognition. In Proceedings of Interspeech 2019, pages 2823-2827.
Download Paper

Designing multimodal datasets for NLP challenges

Published in arXiv preprint, 2021

Design principles for multimodal NLP datasets that test competence-based comprehension, introducing the Recipe-to-Video Questions (R2VQ) dataset for evaluating commonsense reasoning about actions and events.

James Pustejovsky, Eben Holderness, Jingxuan Tu, Parker Glenn, Kyeongmin Rim, Kelley Lynch, and Richard Brutti (2021). Designing multimodal datasets for NLP challenges. In arXiv preprint arXiv:2105.05999.
Download Paper

Situated UMR for multimodal interactions

Published in SemDial Workshop on the Semantics and Pragmatics of Dialogue, 2021

This paper proposes an extension to Uniform Meaning Representation (UMR) to capture both language and gesture in multimodal dialogue.

Kenneth Lai, Richard Brutti, Lucia Donatelli, and James Pustejovsky (2021). Situated UMR for multimodal interactions. In Proceedings of the 25th Workshop on the Semantics and Pragmatics of Dialogue (SemDial 2021).
Download Paper

Challenges and opportunities in annotating a multimodal collaborative problem-solving task

Published in Interdisciplinary Approaches to Getting AI Experts and Education Stakeholders Talking Workshop, AIEd, 2022

We present an annotation scheme coupling automatically extracted features with collaborative problem solving frameworks for developing AI partners in collaborative learning contexts.

Mariah Bradford, Paige Hansen, Kenneth Lai, Richard Brutti, Rachel Dickler, Leanne Hirshfield, James Pustejovsky, Nathaniel Blanchard, and Nikhil Krishnaswamy (2022). Challenges and opportunities in annotating a multimodal collaborative problem-solving task. In Interdisciplinary Approaches to Getting AI Experts and Education Stakeholders Talking Workshop, AIEd.
Download Paper

Abstract meaning representation for gesture

Published in Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC), 2022

We present Gesture AMR (GAMR), an extension to Abstract Meaning Representation that captures gesture meaning in relation to speech in multimodal dialogue.

Richard Brutti, Lucia Donatelli, Kenneth Lai, and James Pustejovsky (2022). Abstract meaning representation for gesture. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC).
Download Paper

Towards Situated AMR: Creating a corpus of Gesture AMR

Published in International Conference on Human-Computer Interaction, 2022

We extend Abstract Meaning Representation to represent situated multimodal dialogue, focusing on gesture and its integration with speech for grounded meaning (GAMR).

Lucia Donatelli, Kenneth Lai, Richard Brutti, and James Pustejovsky (2022). Towards situated AMR: Creating a corpus of gesture AMR. In International Conference on Human-Computer Interaction, pages 293-312.
Download Paper

SemEval-2022 task 9: R2VQ–competence-based multimodal question answering

Published in Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), 2022

A competence-based question answering challenge using cooking recipes and videos, where questions reflect specific semantic reasoning competencies.

Jingxuan Tu, Eben Holderness, Marco Maru, Simone Conia, Kyeongmin Rim, Kelley Lynch, Richard Brutti, Roberto Navigli, and James Pustejovsky (2022). SemEval-2022 task 9: R2VQ--competence-based multimodal question answering. In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-2022), pages 1244-1255.
Download Paper

Annotating situated actions in dialogue

Published in Proceedings of the Fourth International Workshop on Designing Meaning Representations, 2023

We demonstrate how Abstract Meaning Representation can annotate actions in multimodal interactions, enabling simulation and recreation of annotated events.

Christopher Tam, Richard Brutti, Kenneth Lai, and James Pustejovsky (2023). Annotating situated actions in dialogue. In Proceedings of the Fourth International Workshop on Designing Meaning Representations, pages 45-51.
Download Paper

Evaluating Multimodal Behavior Schemas with VoxWorld

Published in International Conference on Human-Computer Interaction, 2023

Christopher Tam, Richard Brutti, Kenneth Lai, and James Pustejovsky (2023). Evaluating Multimodal Behavior Schemas with VoxWorld. In International Conference on Human-Computer Interaction, pages 578-591.
Download Paper

When text and speech are not enough: A multimodal dataset of collaboration in a situated task

Published in Journal of Open Humanities Data, 2024

A multimodal dataset of collaborative interactions with annotations for gesture, speech, gaze, joint attention, and engagement in a shared task.

Ibrahim Khebour, Richard Brutti, Indrani Dey, Rachel Dickler, Kelsey Sikes, Kenneth Lai, Mariah Bradford, Brittany Cates, Paige Hansen, Changsoo Jung, and others (2024). When text and speech are not enough: A multimodal dataset of collaboration in a situated task. In Journal of Open Humanities Data, 10.
Download Paper

Common ground tracking in multimodal dialogue

Published in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024

A method for automatically identifying shared beliefs and questions under discussion in task-oriented dialogue using multimodal features and formal closure rules.

Ibrahim Khalil Khebour, Kenneth Lai, Mariah Bradford, Yifan Zhu, Richard Brutti, Christopher Tam, Jingxuan Tu, Benjamin A. Ibarra, Nathaniel Blanchard, Nikhil Krishnaswamy, and others (2024). Common ground tracking in multimodal dialogue. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 3587-3602.
Download Paper

Encoding Gesture in Multimodal Dialogue: Creating a Corpus of Multimodal AMR

Published in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 2024

An annotated corpus of multimodal AMR in a task-based setting with temporal alignments to speech and gesture, capturing cross-modal coreference relationships.

Kenneth Lai, Richard Brutti, Lucia Donatelli, and James Pustejovsky (2024). Encoding Gesture in Multimodal Dialogue: Creating a Corpus of Multimodal AMR. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 5806-5818.
Download Paper

Modeling theory of mind in multimodal HCI

Published in International Conference on Human-Computer Interaction, 2024

We apply Simulation Theory of Mind within embodied HCI to model belief attribution in multimodal interactions, demonstrating its application to common ground tracking.

Yifan Zhu, Hannah VanderHoeven, Kenneth Lai, Mariah Bradford, Christopher Tam, Ibrahim Khebour, Richard Brutti, Nikhil Krishnaswamy, and James Pustejovsky (2024). Modeling theory of mind in multimodal HCI. In International Conference on Human-Computer Interaction, pages 205-225.
Download Paper

A Model of Information State in Situated Multimodal Dialogue

Published in Proceedings of the 16th International Conference on Computational Semantics, 2025

We explore how speech, gesture, and action contribute to common ground construction using a simple information state model and multimodal AMR annotations.

Kenneth Lai, Lucia Donatelli, Richard Brutti, and James Pustejovsky (2025). A Model of Information State in Situated Multimodal Dialogue. In Proceedings of the 16th International Conference on Computational Semantics, pages 282-288.
Download Paper

Exploring the Potential of Generative Artificial Intelligence in Enhancing Computational Thinking in Digital Literacy and Computer Science Education

Published in Journal of Research Initiatives, 2026

This conceptual article examines resources for facilitating computational thinking (CT) in Massachusetts schools within the context of the Digital Literacy and Computer Science (DLCS) Curriculum Framework.

Simone Elias, Richard Brutti, and Linda Wilson-Jones (2026). Exploring the Potential of Generative Artificial Intelligence in Enhancing Computational Thinking in Digital Literacy and Computer Science Education. Journal of Research Initiatives, 9(2):11.
Download Paper

More Than “Oh”: Grounding Observable Events with Grunts in Multimodal Dialogue

Published in Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC), 2026

Conversational grunts (minimal vocalizations like oh, mm-hm, and uh-huh) ground information and coordinate understanding in human dialogue, yet computational systems typically treat them as noise rather than meaningful communicative acts.

Richard Brutti, and James Pustejovsky (2026). More Than "Oh": Grounding Observable Events with Grunts in Multimodal Dialogue. In Proceedings of the Fifteenth Language Resources and Evaluation Conference (LREC).
Download Paper

The Grunt Work of Conversational Grounding: Minimal Responses in Multimodal Dialogue

Published in Proceedings of the Annual Meeting of the Cognitive Science Society 48, 2026

Conversational grounding, the establishment of mutual understanding between participants, is fundamental to studies of dialogue. However, methods of grounding multimodal information remain understudied. We investigate how minimal vocalizations (conversational grunts like hm and ooh) encode grounding of verbal information versus observed events in collaborative tasks.

Richard Brutti, and James Pustejovsky (2026). The Grunt Work of Conversational Grounding: Minimal Responses in Multimodal Dialogue. In Proceedings of the Annual Meeting of the Cognitive Science Society (Vol. 48).
Download Paper

Digital Learning and Digital Literacy: Conceptual Distinctions, Theoretical Frameworks, and Educational Implications Across the Educational Continuum

Published in Journal of Research Initiatives, 2026

The terms digital learning and digital literacy recur frequently in educational policy and institutional discourse, yet peer-reviewed scholarship shows they are conceptually distinct constructs operating at fundamentally different levels of analysis.

Simone Elias, Linda Wilson-Jones, and Richard Brutti (2026). Digital Learning and Digital Literacy: Conceptual Distinctions, Theoretical Frameworks, and Educational Implications Across the Educational Continuum. Journal of Research Initiatives, 9(3):14.
Download Paper

Human-Centered AI Governance in K–12 Education: A Qualitative Content Analysis of Policy Gaps in Massachusetts School Districts

Published in Journal of Research Initiatives, 2026

Artificial intelligence (AI) and generative AI tools have rapidly permeated K–12 education, outpacing governance structures. Although a growing body of scholarship proposes multi-level AI governance models for education, empirical evidence on implementing these frameworks at the district policy level remains limited.

Simone Elias, Noran Moffett, Jeremy Womack, and Richard Brutti (2026). Human-Centered AI Governance in K–12 Education: A Qualitative Content Analysis of Policy Gaps in Massachusetts School Districts. Journal of Research Initiatives, 9(3):12.
Download Paper

talks

Processing Events with Grunts

Published:

Events significantly contribute to the multimodal context for speech, and can aid in automatic understanding and interpreting of dialogues, from the perspective of the Common Ground.

teaching