Patrik Jonell

I’m currently a PhD student in the area of adaptive conversational agents at KTH, The Royal Institute of Technology in Stockholm, Sweden. My research interests can be broadly defined as multimodal interaction. At the moment, my main research focus is machine learning for these interactions, specifically for non-verbal behavior generation in conversational agents. My goal is to give conversational agents the ability to adapt their non-verbal behavior in response to the behaviour of a conversation partner, similar to what humans do, and I use data collected from interactions between humans, along with deep learning, to progress toward that goal.

Recent publications

Frontiers in
Computer
Science

Multimodal capture of patient behaviour for improved detection of early dementia: clinical feasibility and preliminary results

Patrik Jonell^†, Birger Moëll^†, Krister Håkansson^†, Gustav Eje Henter, Taras Kucherenko, Olga Mikheeva, Göran Hagman, Jasper Holleman, Miia Kivipelto, Hedvig Kjellström, Joakim Gustafson, and Jonas Beskow

[In review] Frontiers in Computer Science - Human-Media Interaction 2020

Abs Website

Non-invasive automatic screening for Alzheimer’s disease has the potential to improve diagnostic accuracy while lowering healthcare costs. Previous research has shown that patterns in speech, language, gaze, and drawing can help detect early signs of cognitive decline. In this paper, we describe a highly multimodal system for unobtrusively capturing data during real clinical interviews conducted as part of cognitive assessments for Alzheimer’s disease. The system uses nine different sensor devices (smartphones, a tablet, an eye tracker, a microphone array, and a wristband) to record interaction data during a specialist’s first clinical interview with a patient, and is currently in use at Karolinska University Hospital in Stockholm, Sweden. Furthermore, complementary information in the form of brain imaging, psychological tests, speech therapist assessment, and clinical meta-data is also available for each patient. We detail our data-collection and analysis procedure and present preliminary findings that relate measures extracted from the multimodal recordings to clinical assessments and established biomarkers, based on data from 25 patients gathered thus far. Our findings demonstrate feasibility for our proposed methodology and indicate that the collected data can be used to improve clinical assessments of early dementia.

^†Indicates equal contribution
IVA Best Paper

Let’s Face It: Probabilistic Multi-Modal Interlocutor-Aware Generation of Facial Gestures in Dyadic Settings

Patrik Jonell, Taras Kucherenko, Gustav Eje Henter, and Jonas Beskow

In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (IVA’20) 2020

Abs PDF Code Website

To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. One key aspect of this is generation of appropriate non-verbal behavior for the agent, for example facial gestures, here defined as facial expressions and head movements. Most existing gesture-generating systems do not utilize multi-modal cues from the interlocutor when synthesizing non-verbal behavior. Those that do, typically use deterministic methods that risk producing repetitive and non-vivid motions. In this paper, we introduce a probabilistic method to synthesize interlocutor-aware facial gestures - represented by highly expressive FLAME parameters - in dyadic conversations. Our contributions are: a) a method for feature extraction from multi-party video and speech recordings, resulting in a representation that allows for independent control and manipulation of expression and speech articulation in a 3D avatar; b) an extension to MoGlow, a recent motion-synthesis method based on normalizing flows, to also take multi-modal signals from the interlocutor as input and subsequently output interlocutor-aware facial gestures; and c) subjective and objective experiments assessing the use and relative importance of the different modalities in the synthesized output. The results show that the model successfully leverages the input from the interlocutor to generate more appropriate behavior.
ICMI Best Paper

Gesticulator: A framework for semantically-aware speech-driven gesture generation

Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexanderson, Iolanda Leite, and Hedvig Kjellström

In Proceedings of the 22th ACM International Conference on Multimodal Interaction (ICMI’20) 2020

Abs arXiv Code Website

During speech, people spontaneously gesticulate, which plays a key role in conveying information. Similarly, realistic co-speech gestures are crucial to enable natural and smooth interactions with social agents. Current data-driven co-speech gesture generation systems use a single modality for representing speech: either audio or text. These systems are therefore confined to producing either acoustically-linked beat gestures or semantically-linked gesticulation (e.g., raising a hand when saying “high”): they cannot appropriately learn to generate both gesture types. We present a model designed to produce arbitrary beat and semantic gestures together. Our deep-learning based model takes both acoustic and semantic representations of speech as input, and generates gestures as a sequence of joint angle rotations as output. The resulting gestures can be applied to both virtual agents and humanoid robots. Subjective and objective evaluations confirm the success of our approach. The code is publicly available at https://github.com/Svito-zar/gesticulator.
IVA

Can we trust online crowdworkers? Comparing online and offline participants in a preference test of virtual agents

Patrik Jonell^†, Taras Kucherenko^†, Ilaria Torre, and Jonas Beskow

In Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents (IVA’20) 2020

Abs PDF

Conducting user studies is a crucial component in many scientific fields. While some studies require participants to be physically present, other studies can be conducted both physically (e.g. in-lab) and online (e.g. via crowdsourcing). Inviting participants to the lab can be a time-consuming and logistically difficult endeavor, not to mention that sometimes research groups might not be able to run in-lab experiments, because of, for example, a pandemic. Crowdsourcing platforms such as Amazon Mechanical Turk (AMT) or Prolific can therefore be a suitable alternative to run certain experiments, such as evaluating virtual agents. Although previous studies investigated the use of crowdsourcing platforms for running experiments, there is still uncertainty as to whether the results are reliable for perceptual studies. Here we replicate a previous experiment where participants evaluated a gesture generation model for virtual agents. The experiment is conducted across three participant pools - in-lab, Prolific, and AMT - having similar demographics across the in-lab participants and the Prolific platform. Our results show no difference between the three participant pools in regards to their evaluations of the gesture generation models and their reliability scores. The results indicate that online platforms can successfully be used for perceptual evaluations of this kind.}, booktitle = {Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents

^†Indicates equal contribution
GENEA

The GENEA Challenge 2020:Benchmarking gesture-generation systems on common data

Taras Kucherenko^†, Patrik Jonell^†, Youngwoo Yoon^†, Pieter Wolfert, and Gustav Eje Henter

In Proceedings of the GENEA Workshop 2020, part of IVA’20. 2020

Abs PDF

Automatic gesture generation is a field of growing interest, and a key technology for enabling embodied conversational agents. Research into gesture generation is rapidly gravitating towards data-driven methods. Unfortunately, individual research efforts in the field are difficult to compare: there are no established benchmarks, and each study tends to use its own dataset, motion visualisation, and evaluation methodology. To address this situation, we launched the GENEA gesture-generation challenge, wherein participating teams built automatic gesture-generation systems on a common dataset, and the resulting systems were evaluated in parallel in a large, crowdsourced user study. Since differences in evaluation outcomes between systems now are solely attributable to differences between the motion-generation methods, this enables benchmarking recent approaches against one another and investigating the state of the art in the field. This paper provides a first report on the purpose, design, and results of our challenge, with each individual team’s entry described in a separate paper also presented at the GENEA Workshop. Additional information about the workshop can be found at https://genea-workshop.github.io/2020.

^†Indicates equal contribution
CogSci

Embodiment and gender interact in alignment to TTS voices

Michelle Cohn, Patrik Jonell, Taylor Kim, Jonas Beskow, and Georgia Zellou

Proceedings of the 42th Annual Meeting of the Cognitive Science Society, CogSci 2020 2020

Abs PDF

The current study tests subjects’ vocal alignment toward female and male text-to-speech (TTS) voices presented via three systems: Amazon Echo, Nao, and Furhat. These systems vary in their physical form, ranging from a cylindrical speaker (Echo), to a small robot (Nao), to a human-like robot bust (Furhat). We test whether this cline of personification (cylinder\textless mini robot\textless human-like robot bust) predicts patterns of gender-mediated vocal alignment. In addition to comparing multiple systems, this study addresses a confound in many prior vocal alignment studies by using identical voices across the systems. Results show evidence for a cline of personification toward female TTS voices by female shadowers (Echo\textless Nao\textless Furhat) and a more categorical effect of device personification for male TTS voices by male shadowers (Echo\textless Nao, Furhat). These findings are discussed in terms of their implications for models of devicehuman interaction and theories of computer personification.
Frontiers in
Robotics and AI

Towards an Engagement-Aware Attentive Artificial Listener for Multi-Party Interactions

Catharine Oertel, Patrik Jonell, Dimosthenis Kontogiorgos, Kenneth Funes Mora, Jean-Marc Odobez, and Joakim Gustafson

[In review] Frontiers in Robotics and AI - Human-Robot Interaction 2020