Summary
AntEval evaluates social interaction competency in LLM-driven agents through a multi-agent interaction framework and two quantitative metrics: Information Exchanging Precision (IEP) and Interaction Expressiveness Gap (IEG).
Research paths
How this paper contributes to the site's broader research map.
- Semantic Motion and Embodied InteractionBridge
AntEval evaluates task completion separately from information exchange and intention expression, showing that successful outcomes do not establish that agents communicated effectively.
Research question
Agent social behavior is difficult to improve when evaluation stops at surface fluency or small talk and lacks quantitative measures of whether agents exchange relevant information and express their intentions.
What the paper contributes
- Provides an interaction framework designed to elicit information exchange and intention expression in multi-character settings.
- Introduces Information Exchanging Precision (IEP) for assessing information exchange.
- Introduces Interaction Expressiveness Gap (IEG) for assessing how effectively interaction expresses intended information.
Evidence and evaluation scope
The paper presents evaluations intended to demonstrate the utility of IEP and IEG for diagnosing LLM-agent interaction competency. Exact agent settings, interaction tasks, and metric behavior should be read from arXiv v3.
Scope and limitations
The metrics operationalize selected aspects of social interaction and do not constitute a complete measure of social intelligence, safety, relationship quality, or behavior across all cultures and open-ended settings.
Positioning for related work
AntEval is an evaluation contribution for LLM-driven multi-agent social interaction. The current title and metric names should be used; the earlier title about 'informativeness and expressiveness' is obsolete.
Related-work context
Liang et al. introduce AntEval, a framework for evaluating LLM-driven social interactions using Information Exchanging Precision and Interaction Expressiveness Gap to quantify information exchange and intention expression.
A concise, neutral description of how this paper can be situated in related work.
Official abstract
Large Language Models (LLMs) have demonstrated their ability to replicate human behaviors across a wide range of scenarios. However, their capability in handling complex, multi-character social interactions has yet to be fully explored, primarily due to the absence of robust, quantitative evaluation methods. This gap has slowed the development of agents proficient in more nuanced interactions beyond simple exchanges, for example, small talk. To address this challenge, we introduce the Multi-Agent Interaction Evaluation Framework (AntEval), encompassing a novel interaction framework and evaluation methods. The interaction framework aims to foster an complex interaction environment that bolsters information exchange and intention expression within social interactions. Furthermore, we introduce evaluation methods, including two metrics: Information Exchanging Precision (IEP) and Interaction Expressiveness Gap (IEG), designed for the quantitative and objective assessment of agents' interaction competencies. Our findings highlight the utility of these evaluative methods and show significant potential for improving LLMs' ability to construct agents that interact in a more natural manner with human-like intricacy.
The abstract is reproduced for scholarly identification and remains under the paper publisher/authors’ original copyright; it is not covered by this page’s CC BY license.
Evidence references
| What to verify | Location in the paper |
|---|---|
| Problem statement | Abstract |
| Method and contributions | Abstract; interaction framework and IEP/IEG evaluation sections |
| Evaluation statement | Abstract; agent-interaction evaluation experiments |
Primary source: arXiv record (arXiv:2401.06509v3, revised 2024-03-05).
How to cite
Cite the paper—not this explainer—for scientific claims. Cite this page only when reusing its original commentary.
Yuanzhi Liang, Linchao Zhu, and Yi Yang. “AntEval: Evaluation of Social Interaction Competencies in LLM-Driven Agents.” arXiv (2024). arXiv:2401.06509.
Reuse policy
Original explanatory text on this page is licensed under CC BY 4.0 with attribution and a link to this page. Paper title, abstract, figures, and bibliographic metadata are excluded and retain their original rights. Creative Commons Attribution 4.0 International.