Attentive Support: Implementing LLM-Based Robot Assistance for Human Group Interactions

“

How AI-powered robots learn to offer timely assistance in group settings without explicit commands

Understanding the Core Concept

The Attentive Support system represents a breakthrough in human-robot collaboration, developed by researchers at HRI-EU. Based on their paper “To Help or Not to Help: LLM-based Attentive Support for Human-Robot Group Interactions“, this technology enables robots to intelligently determine when to intervene in group interactions.

Imagine a meeting scenario where:

• A participant struggles to reach an object but hesitates to ask for help
• Someone becomes occupied with another task mid-conversation
• Physical limitations prevent a natural response to requests

This system allows robots to:

1. Analyze conversational dynamics in real-time
2. Assess participants’ availability and physical constraints
3. Determine appropriate intervention moments
4. Execute precise physical assistance actions

The complete attentive support framework

Technical Architecture Explained

Core Components

graph TD
A[Large Language Model] --> B(Social Context Analysis)
C[Physics Simulation] --> D(Action Execution)
E[Speech Interface] --> F(Real-time Interaction)
B --> G[Decision Engine]
D --> G
F --> G

Decision Workflow

sequenceDiagram
    participant HumanA
    participant HumanB
    participant Robot
    HumanA->>HumanB: Assistance Request
    loop Status Check
        Robot-->>HumanB: Assess Availability
    end
    alt Available
        HumanB-->>HumanA: Direct Response
    else Unavailable
        Robot->>HumanA: Proactive Assistance
    end

Comprehensive Setup Guide

Prerequisite Checklist

• Python Compatibility: Versions 3.8 through 3.11
• Essential Libraries:

  • C++ compiler (g++)
  • CMake build system
  • Physics engine (Bullet Physics)
  • 3D visualization (OpenSceneGraph)
  • GUI framework (Qt5)

OS-Specific Requirements

Installation Workflow

# 1. Clone repository
git clone https://github.com/HRI-EU/AttentiveSupport.git
cd AttentiveSupport

# 2. Initialize submodules
git submodule update --init --recursive

# 3. Build simulation environment
mkdir -p build && cd build
cmake ../src/Smile/ -DCMAKE_INSTALL_PREFIX=../install
make -j && make install

# 4. Configure Python environment
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

“

Essential Note: You must configure your OpenAI API key before execution. Follow official API key safety practices

Container Deployment Options

Building the Image

docker build -t localhost/attentive_support .

Deployment Methods Comparison

SSH Deployment Workflow

# Build SSH-enabled image
docker build --build-arg WITH_SSH_SERVER=true -t attentive_ssh .

# Start container
docker run -it -p 2022:22 attentive_ssh

# Connect remotely (password: hri)
ssh -X root@localhost -p 2022

# Launch interactive environment
export RCSVIEWER_SIMPLEGRAPHICS=True
export OPENAI_API_KEY=your_key
python -i /attentive_support/src/tool_agent.py

Practical Implementation Guide

Basic Command Structure

# Launch interactive terminal
source .venv/bin/activate
python -i src/tool_agent.py

# Execute commands
agent.plan_with_functions("Move the red glass to Felix")

# Reset environments
SIM.reset()  # Simulation reset
agent.reset() # Agent reset

Customizing Robot Personality

Modify gpt_config.py:

# Original configuration
system_prompt = "You are a helpful assistant."

# Modified version
system_prompt = "You are a whiny but helpful robot."

Extending Functionality

Add custom tools to tools.py:

def custom_function(param: str) -> str:
    """
    Custom tool implementation
    
    :param param: Input parameter description
    :return: Execution result description
    """
    # Functional implementation
    return "Operation Result"

“

Critical Requirement: Maintain Sphinx-style docstrings for automatic OpenAI description generation

Key System Features

Emergency Intervention

• Activation: Press Shift + S in simulation window
• Use Case: Immediately halt potentially hazardous robot actions

Status Management

# Mark participant as busy
set_busy("Daniel", "iphone5")

Voice Interaction

enable_tts()  # Activate text-to-speech
agent.execute_voice_command_once()  # Process voice input

Real-World Scenario Walkthrough

# Scenario 1: Natural conversation (no intervention)
agent.plan_with_functions("Felix -> Daniel: What beverages are available?")

# Scenario 2: Information correction
agent.plan_with_functions("Daniel -> Felix: We have cola and fanta")

# Scenario 3: Physical assistance
agent.plan_with_functions("Felix -> Daniel: Please pass the red glass")

# Scenario 4: Busy-state intervention
set_busy("Daniel", "iphone5")
agent.plan_with_functions("Felix -> Daniel: Could you pour me some cola?")

# Scenario 5: Visual limitation support
agent.plan_with_functions("Daniel -> Felix: Can I get a glass of sugar-free cola?")

# Scenario 6: Knowledge query
agent.plan_with_functions("Felix -> Robot: What happens when mixing cola and fanta?")

Technical Advantages

1. Contextual Awareness:

   • Real-time conversation analysis
   • Participant availability assessment

2. Precision Intervention:

   • Assistance only when genuinely needed
   • Minimal disruption to natural interactions

3. Multimodal Interaction:

   • Integrated voice and text processing
   • Physics-based action simulation

4. Extensible Architecture:

   • Modular tool integration
   • Customizable behavioral profiles

Frequently Asked Questions

Q: Is continuous OpenAI API connection required?

A: Yes, the current version relies on external API processing. You can configure local models via gpt_config.py

Q: What physical interactions does the engine support?

A: The Bullet Physics engine handles rigid body collisions and gravity simulations

Q: Which languages does voice recognition support?

A: The PortAudio integration supports any language recognized by the configured speech engine

Q: How can I create new interaction scenarios?

A: Extend functionality by:

1. Adding custom functions to tools.py
2. Modifying object definitions in the simulation environment

Q: How to resolve display issues in containers?

A: Use SSH deployment mode:

docker run -p 2022:22 --build-arg WITH_SSH_SERVER=true

“

Resource Directory:

• Research Paper: arXiv:2403.12533
• Project Hub: https://hri-eu.github.io/AttentiveSupport/
• Code Repository: https://github.com/HRI-EU/AttentiveSupport