AntEval: Measuring Social Interaction in AI Agents

Why a tabletop game is a useful laboratory

A tabletop role-playing game (TRPG) gives each character a role, private information, and reasons to act without revealing everything directly. Those ingredients create controlled asymmetry: an evaluator can know what one participant should learn from another and what intention sits behind an action. In ordinary open-domain chat, the ground truth for “good social interaction” is much harder to specify. The game is therefore an instrument for measurement, not the paper’s claim about where agents will ultimately be deployed.

The setup also exposes shortcuts. An agent can maximize task success by stating private objectives mechanically or by relying on information that should not be shared. Human players often imply, negotiate, and respond to context instead. By preserving role constraints, AntEval tests whether an interaction carries useful information while maintaining an intelligible social process. The benchmark remains bounded by scripted scenarios and language-based play, but it gives evaluators a way to inspect intermediate behavior that a terminal score cannot reveal.

IEP asks whether knowledge changed

Information Exchange Precision is built around checkable knowledge points. The evaluation considers what participants know before an interaction and what the other party can recover afterward. It is not a word-overlap metric. Mentioning a term does not prove that the listener received the intended fact, and long dialogue receives no automatic bonus. A concise exchange that transmits the relevant information can outperform a verbose conversation that circles around it without changing either participant’s understanding.

Open-text judgments require an evaluator, so the paper uses large language models with explicit criteria and repeated sampling to assess whether information was conveyed. This makes IEP scalable but introduces a second model into the measurement loop. Prompt wording, judge version, aggregation, and human agreement become part of the experimental protocol. From today’s perspective, that dependency looks even more important: stronger judges may improve semantic matching, while shared style preferences between judge and agent can still reward polished but shallow interaction.

IEG measures a gap, not maximal explicitness

Intention Expression Gap uses a virtual Dungeon Master to infer action intentions from the interaction, then compares agent-generated dialogue with human references. The use of a gap is deliberate. If intention cannot be inferred at all, the character’s behavior may be vague. If the character announces a hidden skill check or internal label verbatim, prediction becomes easy but the exchange ceases to resemble natural role play. Good expression lies between opacity and mechanical disclosure.

IEG is therefore not a score where making every intention more explicit is automatically better. It evaluates relative distance from the human reference process under the benchmark’s action structure. That makes it complementary to IEP. A dialogue can transmit facts effectively while expressing motives awkwardly, or maintain natural role behavior while failing to deliver crucial information. Separating the axes prevents one attractive total score from hiding two different product or research problems.

What the experiments reveal about agent communication

The paper evaluates several LLM-driven agent settings and reports that stronger models generally exchange information more effectively, while a gap from human interaction remains. In an analysis of GPT-4 samples with IEP below 50%, human descriptions frequently included “boring,” “mechanical,” “rigid,” and “insincere.” These are pragmatic failures rather than basic syntax failures. The agents can produce correct sentences while failing to respond, adapt, and reveal information in a socially convincing way.

That evidence should not be generalized into a complete ranking of social intelligence. The sample analysis concerns defined TRPG interactions, chosen models, prompts, judges, and knowledge points. Low IEP can arise from withholding, misunderstanding, distraction, or evaluation error; the metric alone does not diagnose every cause. Figure-level examples and the word analysis make the failure legible, while the quantitative tables establish broader patterns. Both are needed before concluding that one agent architecture communicates better than another.

What still holds—and what needs stronger evidence

The 2024 paper predates many current memory systems, tool-using agents, and longer-context orchestration patterns. Those systems can change the meaning of information exchange. If agents share a memory store or can inspect one another’s state, IEP no longer measures communication in the same way. Models and LLM judges have also changed. A modern replication should report the exact agent scaffold, available tools, turn budget, judge prompts, sampling, and whether knowledge could travel through channels outside dialogue.

The conceptual separation still holds. Task success, conversational fluency, information transfer, and intention expression are distinct. AntEval belongs in work on process-level evaluation of multi-agent social interaction, not as a universal chat benchmark or a safety certification. Future extensions could weight knowledge by importance and privacy, measure how many turns it takes to become actionable, and test whether agents correct misunderstandings. Hindsight makes the paper easier to place: its durable contribution is a vocabulary for asking what happened inside a successful interaction.

There is also a precision–privacy tension that the original axes make visible. Maximizing information exchange without regard to authorization can reward oversharing, while minimizing disclosure can look like communication failure. A modern benchmark should label which facts are relevant, permitted, sensitive, or intentionally withheld, then score whether the right recipient learns the right fact at the right time. This would extend IEP without turning all information flow into a virtue and would better match tool-using agents operating with user data.

Longer interactions need temporal accounting as well. Mark when a fact is first offered, when it is acknowledged, when it is used in a decision, and whether later dialogue corrupts it. An agent may briefly acquire the correct information yet act on an outdated version ten turns later. This sequence separates delivery, comprehension, retention, and use—four stages that a single post-conversation judgment can merge. It also exposes whether extra turns repair misunderstandings or merely add fluent noise.

Evidence map

The locations below point to the primary paper so readers can verify the method and reported evidence directly.

Reuse

Original commentary in this note is licensed under CC BY 4.0. The paper title and bibliographic material retain their original rights.