From UAT to AI Evaluation: A Requirements-Based Framework for Human Validation of Enterprise AI Systems
DOI:
https://doi.org/10.64137/31079458/IJCSEI-V2I4P103Keywords:
Acceptance Criteria, AI Defect Taxonomy, AI Evaluation, AI Governance, Business Rule Compliance, Continuous Evaluation, Design Science Research, Enterprise AI, Hallucination Detection, Human-AI Collaboration, Human-in-the-Loop, Requirements Engineering, Risk-Tiered Validation, User Acceptance TestingAbstract
Enterprise organizations are increasingly embedding generative and agentic AI into business applications historically validated through User Acceptance Testing (UAT). Conventional UAT is typically structured around predefined requirements and sufficiently predictable expected outcomes evaluated using pass/fail criteria; generative AI complicates this model because outputs are probabilistic and context-sensitive, and a response may be fluent and confident while incorrect, non-compliant with a business rule, or inconsistent with prior behavior. This conceptual, design-science paper synthesizes requirements engineering, software testing, human-AI interaction, AI evaluation, and AI governance literature into a requirements-based framework for human validation of enterprise AI systems, comprising requirements elicitation, AI-specific acceptance criteria, risk-tiered human validation, and feedback/remediation loops under a governance layer supporting continuous revalidation. The paper contributes an explicit mapping from conventional UAT concepts to AI-specific extensions and an AI defect taxonomy that routes findings to remediation owners. Human validation is positioned as complementary to, not a replacement for, automated evaluation. A generic, composite scenario illustrates the framework's application. Explicit limitations, including the absence of empirical validation, are discussed.
References
[1] “IEEE Standard for Software and System Test Documentation,” IEEE Std 829-2008, pp. 1–150, July 2008, doi: 10.1109/IEEESTD.2008.4578383.
[2] “ISO/IEC/IEEE 29119-1:2022 Software and systems engineering Software testing,” 2022. [Online]. Available: https://www.iso.org/standard/81291.html
[3] S. Amershi et al., “Guidelines for Human-AI Interaction,” Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI ’19, 2019, doi: 10.1145/3290605.3300233.
[4] G. Bansal et al., “Does the Whole Exceed Its Parts? The Effect of AI Explanations on Complementary Team Performance,” Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–16, May 2021, doi: 10.1145/3411764.3445717.
[5] “Ouyang, L., Wu, J., Jiang, X., et al., Training Language Models to Follow Instructions With Human Feedback. arXiv 2203.02155. - References - Scientific Research Publishing,” 2022. https://www.scirp.org/reference/referencespapers?referenceid=4079808
[6] L. Zheng et al., “Judging LLM-as-a-Judge With MT-Bench and Chatbot Arena,” arXiv.org, 2023. https://doi.org/10.48550/arXiv.2306.05685
[7] G. H. Chen, S. Chen, Z. Liu, F. Jiang, and B. Wang, “Humans or LLMs as the Judge? A Study on Judgement Bias,” Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8301–8327, Jan. 2024, doi: 10.18653/v1/2024.emnlp-main.474.
[8] Z. Ji et al., “Survey of Hallucination in Natural Language Generation,” ACM Computing Surveys, vol. 55, no. 12, Nov. 2022, doi: 10.1145/3571730.
[9] S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal, “Detecting Hallucinations in Large Language Models Using Semantic Entropy,” Nature, vol. 630, no. 8017, pp. 625–630, June 2024, doi: 10.1038/s41586-024-07421-0
[10] M. Mohammadi, Y. Li, J. Lo, and W. Yip, “Evaluation and Benchmarking of LLM Agents: A Survey,” Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.2, pp. 6129–6139, Aug. 2025, doi: 10.1145/3711896.3736570
[11] NIST, “AI Risk Management Framework,” Artificial Intelligence Risk Management Framework (AI RMF 1.0), vol. 1, no. 1, Jan. 2023, doi: 10.6028/nist.ai.100-1
[12] NIST, “Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile,” Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, July 2024, doi: 10.6028/nist.ai.600-1
[13] International Organization for Standardization, “ISO/IEC 42001:2023,” ISO, 2023. https://www.iso.org/standard/42001
[14] European Union, “Regulation - EU - 2024/1689 - EN - EUR-Lex,” eur-lex.europa.eu, June 13, 2024. https://eur-lex.europa.eu/eli/reg/2024/1689/oj
[15] “ISO ISO/IEC 250102011, ‘Systems and Software Engineering Systems and Software Quality Requirements and Evaluation (SQuaRE) System and Software Quality Models,’ 2011. - References - Scientific Research Publishing,” Scirp.org, 2026. https://www.scirp.org/reference/referencespapers?referenceid=891857
[16] A. Hevner, S. March, J. Park, and S. Ram, “Design Science in Information Systems Research,” MIS Quarterly, vol. 28, no. 1, pp. 75–105, 2004, doi: 10.2307/25148625.
