Roadmap on Incentive Compatibility for AI Alignment and Governance in Sociotechnical Systems

The burgeoning integration of artificial intelligence (AI) into human society brings forth significant implications for societal governance and safety. While considerable strides have been made in addressing AI alignment challenges, existing methodologies primarily focus on technical facets, often neglecting the intricate sociotechnical nature of AI systems, which can lead to a misalignment between the development and deployment contexts. To this end, we posit a new problem worth exploring: Incentive Compatibility Sociotechnical Alignment Problem (ICSAP). We hope this can call for more researchers to explore how to leverage the principles of Incentive Compatibility (IC) from game theory to bridge the gap between technical and societal components to maintain AI consensus with human societies in different contexts. We further discuss three classical game problems for achieving IC: mechanism design, contract theory, and Bayesian persuasion, in addressing the perspectives, potentials, and challenges of solving ICSAP, and provide preliminary implementation conceptions.

• incentive compatibility
• collaborative intelligence
• AI alignment
• sociotechnical systems

• Institute for Artificial Intelligence, Peking University — Zhaowei Zhang, Mingzhi Wang, Haoyang Ye, Chengdong Ma, Yaodong Yang
• State Key Laboratory of General Artificial Intelligence, BIGAI — Zhaowei Zhang
• Shanghai Jiao Tong University — Fengshuo Bai
• Zhongguancun Academy — Fengshuo Bai

• National Natural Science Foundation of China, grant 62376013

Zhang, Z., Bai, F., Wang, M., Ye, H., Ma, C., & Yang, Y. (2026). Roadmap on Incentive Compatibility for AI Alignment and Governance in Sociotechnical Systems. In M. Iklé, A. Kolonin, & M. Bennett (Eds.), Artificial General Intelligence: AGI 2025. Lecture Notes in Computer Science, vol. 16058, pp. 370–380. Springer, Cham. https://doi.org/10.1007/978-3-032-00800-8_33