Do you like him? - Designing a Multimodal Emotional Expression for a Mobile Robot.
As part of the Master's course "Natural User Interfaces" at Technische Hochschule Ingolstadt, ENAMOUR ("Empathy with aN AutonoMOUs Robot") was developed. The goal was to teach a quadruped robot (Unitree A1) to express convincing emotions: recognisable through body language, facial expressions on an LCD display, and fitting sound signals.
The project was interdisciplinary and brought together four study programmes: Mechanical Engineering (head mechanics), Computer Science (system architecture), Artificial Intelligence (speech recognition), and UX Design (emotion design and user studies). Our 13-person UX team was responsible for the emotional expressiveness of the robot from the first study through to the final implementation.
The result was a functional prototype named "Spike" that responds to voice commands, expresses emotions such as joy, sadness, anger, curiosity, and fear and was presented to a broad audience at the THI On Campus Festival.
As one of the two Project Managers, I took on organisational responsibility for the UX team, alongside my active involvement in design and implementation.
We worked within a Scrum-inspired, self-organised framework adapted to our needs. Each sprint lasted one week (two weeks in exceptional cases). Every sprint meeting consisted of a review, team discussion, and planning for the next cycle.
* Notion with a Kanban board for sprint planning, meeting minutes, and milestone tracking
* FigJam for research, study planning, and creative work steps
* GitHub for version control and team-wide code access
* Discord as the primary communication platform for cross-team exchange
In addition to project management, two dedicated communication partners coordinated the weekly exchange with the Computer Science, AI, and Mechanical Engineering teams. This ensured that technical constraints were fed into our design decisions early on — for example, when the CS team had to switch to high-level movement commands and we adjusted our gesture concepts accordingly.
Quasi-experiment · n = 27 · Online · approx. 15 min
The first study examined how effectively Spike's body movements communicate emotions, and which design direction was preferred for the display face. Participants watched videos of the robot and rated gesture clarity, emotion recognisability, and design variables such as eyes, muzzle, and display style on a Likert scale.
Key findings: Joy and curiosity were well recognised; anger less so. The display was required to show eyes. Participants also requested additional emotional states — sleepiness and affection — which were incorporated in the second iteration of the gesture design.
Quantitative · n = 43 · Online · max. 30 min
Building on Study 1, we tested four display design concepts: Alpha (cartoonish, colourful), Beta (realistic), Gamma (robotic, blue tones), and Delta (cartoonish, monochrome). Each group evaluated their variant using emotion videos, Likert scales, and open comments. The outcome was a combination of Alpha and Gamma: a robotic blue design with emotion-specific colours and cartoon-like symbols for added clarity. Colour gradients and iris details that brought the eyes to life were particularly well received.
Qualitative · n = 8 · In-person · Focus Group + UEQ
The third study evaluated the finished system. Eight participants assessed all eight emotions including sounds and transitions. In addition to a custom questionnaire, the standardised User Experience Questionnaire (UEQ) was used. Emotions were correctly recognised at an average of 4.14 out of 5. The visual display was more convincing than the sound signals. The display design achieved solid UEQ scores; participants were able to reliably identify emotions and follow the interaction concept.
Beyond my role as Project Manager, I was actively involved in the technical implementation of the UX component. Specifically in system architecture and ROS development.
The overall system followed a modular pipeline: voice input → AI intent recognition → JSON transfer via ROS → UX emotion logic → output to display, sound, and movement. Our UX team was responsible for the emotion control layer.
We worked with ROS Noetic in an Ubuntu VM, as ROS was barely functional on Windows or macOS. Communication with other teams was handled via ROS Publisher/Subscriber. Since Python 2.7 does not support common data science libraries, we built our lookup tables as nested dictionaries, which was a easy to implement solution.
- Sprint 1: Technical MVP receive intent, match emotion, output JSON to CS team
- Sprint 2: Dynamic output, intent list finalised, dictionary structure built
- Sprint 3: Synchronisation of motion, sound, and display; randomisation of sounds and movements
for more natural behaviour; emotion state machine for realistic transitions
We also developed a JSON-based intent list format that served as the central interface to the AI team and was refined iteratively throughout the project.
ENAMOUR was successfully demonstrated at the THI On Campus Festival (June 2022) in front of a broad audience including external visitors, fellow students, faculty. Spike moved, responded to commands, and displayed recognisable emotions on its screen.
The visual display and movement werde the strongest channel for conveying emotion; sound was perceived as complementary.
For me personally, this project was an wonderful experience in interdisciplinary collaboration: as Project Manager I maintained structure and communication, as UX Designer I shaped the interaction concept, and as a ROS developer I translated design decisions into running code. Multi-facetted roles like these can be demanding, but all sub-teams worked and collaborated perfectly, which made it possible. But I also learned, that for bigger projects, this kind of context-switching roles should be avoided as much as possible.
Read more:
