The Automatic Annotation of the Semiotic Type of Hand Gestures in Obama’s Humorous Speeches

Research output: Chapter in Book/Report/Conference proceedingArticle in proceedingsResearchpeer-review

Abstract

This paper describes a pilot study act to investigate the semiotic types of hand gestures in video-recorded speeches and their automatic classification. Gestures, which also comprise e.g. head movements and body posture, contribute to the successful delivery of the message by reinforcing what is expressed by speech or by adding new information to what is uttered. The automatic classification of the semiotic type of gestures from their shape description can contribute to their interpretation in human-human communication and in advanced multimodal interactive systems. We annotated and analysed hand gestures produced by Barack Obama during two speeches at the Annual White House Correspondent Dinners and found differences in the contexts in which various hand gesture types were used. Then, we trained machine learning algorithms to classify the semiotic type of the hand gestures. The F-score obtained by the best performing algorithm on the classification of four semiotic types is 0.59. Surprisingly, the shape feature that contributes mostly to classification is the trajectory of the left hand. The results of this study are promising, but they should be tested on more data of different type, produced by different speakers and in more languages.
Original languageEnglish
Title of host publicationProceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)
Number of pages6
Place of PublicationParis, France
PublisherEuropean Language Resources Association
Publication date2018
Pages1067-72
ISBN (Electronic)979-10-95546-00-9
Publication statusPublished - 2018

Cite this