Identifying the Function of Hand Gestures from their Form in Political Speech

Research output: Chapter in Book/Report/Conference proceedingBook chapterResearchpeer-review

Abstract

This paper addresses the automatic classification of the semiotic type of hand gestures produced by six English speaking politicians in different contexts from the annotations of the gesture form and shape. It builds upon and extends to more data the work proposed in [24]. Gestures contribute to the successful delivery of a message in face-to-face communication by reinforcing what is conveyed by speech or adding new information to it. Gestures are multi-functional and can have different meanings depending on the context. The identification of the semiotic type of gestures is the first step towards their automatic interpretation. Moreover, exploiting the relation between gesture form and function, described in the literature, on many types of data, contributes to the automatic generation of gestures e.g. in infocommunication systems. In the present work we annotated the semiotic types of hand gestures produced by Boris Johnson in two question/answer sessions at the British House of Common and added it to the annotations of the gestures of five other politicians in different contexts. We trained various classifiers to identify the semiotic type of the hand gestures. The F1 score obtained by the best performing algorithm on the classification of four semiotic types is 0.68. This result confirms on a larger data set the results obtained in the preceding pilot study indicating that it is possible to identify the semiotic types of hand gestures from their coarse-grained form features in nearly one third of the cases.
Original languageEnglish
Title of host publicationAccentuated Innovations in Cognitive Info-Communication
EditorsRyszard Klempous, Jan Nikodem, Péter Zoltán Baranyi
PublisherSpringer
Publication date2022
Pages203-216
ISBN (Print)978-3-031-10955-3
ISBN (Electronic)978-3-031-10956-0
DOIs
Publication statusPublished - 2022
SeriesTopics in Intelligent Engineering and Informatics
Volume16
ISSN2193-9411

Cite this