Eliciting LLM Propositional Beliefs in Agentic Tasks: Consistency, Accuracy, and Response Tendencies
How accurately and consistently do LLM agents report their beliefs about an interactive environment, and what can these reports reveal about task success?
Figure 1. Data pipeline. Sampled tasks produce agent trajectories; LLM-authored templates and logical rules are grounded at each turn. Queries at the same recorded prefix elicit self-reported beliefs, which are paired with labels and probabilities from the reference evaluation module.
View full-size figure ↗Abstract
Large language models (LLMs) are increasingly deployed as autonomous agents in interactive environments that require multi-step planning, reasoning, and state tracking. Evaluating an agent’s beliefs about environment states and task progress can therefore help diagnose behavioral failures. However, reference belief labels are often unavailable or difficult to obtain. In this paper, we introduce a general, LLM-assisted evaluation pipeline that combines proposition generation with reference labels. The pipeline grounds LLM-authored proposition templates in agent trajectories and obtains labels from an LLM-assisted reference observer using formal logical rules and Bayesian inference. Using this pipeline, we systematically evaluate models’ self-reported belief consistency and accuracy across World, Epistemic, Affordance, and Decision propositions in ALFWorld, WebShop, and Noisy Cabinets and examine their relationship with task success. T/F-only Affordance and Decision accuracy are positively associated with task success after centering within environments. Furthermore, we find that response tendencies help explain both observed consistency and accuracy variation: the tendency to answer Unknown contributes to cross-model differences in consistency, and a True/False tendency model explains degradation in accuracy and negation consistency as context grows.
Methods and contributions
- Developed an LLM-assisted pipeline that grounds proposition templates in agent trajectories and assigns True / False / Unknown reference labels using formal logical rules and Bayesian inference.
- Evaluated World, Epistemic, Affordance, and Decision propositions in ALFWorld, WebShop, and Noisy Cabinets, measuring accuracy alongside permutation, negation, sequential, and rephrasing consistency.
- Found that Affordance and Decision accuracy is positively associated with task success after centering within each environment, when both reference labels and self-reports are True or False.
- Showed that Unknown-response tendencies contribute to cross-model consistency differences, and developed a response-tendency model linking accuracy and negation consistency to explain their decline along longer interaction histories.