A FRAMEWORK FOR METRIC EVALUATION OF ARTIFICIAL INTELLIGENCE SYSTEMS BASED ON QUALITY MODEL
DOI:
https://doi.org/10.26906/SUNZ.2022.2.041Keywords:
artificial intelligence system, quality assessment, evaluation metrics, visualization, frameworkAbstract
Motivation. Nowadays it is crucially important to understand whether systems based on artificial intelligence (AI) can be trusted. Many modern AI systems are built according to the "black box" principle, i.e. it is not clear how they work, but we see only the results of their work. Besides, it is needed tools to compare different AI solutions. When several AIs are competing for use in some system, it is required to determine the best one. The goal of the research is to develop a modelbased framework to evaluate the quality of an AI system (AIS) using metrics and a method for visualizing the evaluation results. Research stages. The article analyzes the models of AIS quality, metrics and types of convolution for its evaluation, proposes a method for evaluating and visualizing the results and describes an example of applying the method. Conclusions. The basic models of quality, combined into a four-level hierarchy are used to assess AIS. The rules of metrics formation and the method of quality calculation using convolutions and visualization of intermediate and final results using radial metric diagrams have been defined for these characteristics. Corresponding quality models, metrics, and evaluation and visualization methods provide implementing automation framework by use of the developed tool. This tool allows the user to create a quality model, set metrics, and enter metrics values. Then, based on these metrics, a generalized quality metric for the system is calculated an d visualized using the radar diagrams. The tool is a desktop application created on .Net Framework platform. The direction of further research. Forthcoming steps can be devoted to development of the model and tools for quality assessment for different domains, considering the aspects of quality evolution.Downloads
References
Trustworthy AI [Text] / R. Chatila, V. Dignum, M. Fisher, F. Giannotti, K. Morik, S. Russell, K. Yeung // Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): collective monograph, edited by B. Braunschweig, M. Ghallab. – Cham: Springer International Publishing, 2021. – Vol. 12600. – Р. 13-39. DOI: 10.1007/978-3-030-69128-8.
A Systematic Review of Explainable Artificial Intelligence in Terms of Different Application Domains and Tasks [Text] / M. R. Islam, M. U. Ahmed, S. Barua, S. Begum // Applied Sciences. – 2022. – Vol. 12. – Article Id: 1353. DOI: 10.3390/app12031353
Харченко В. С., Фесенко Г. В., Ілляшенко О. О. (2022), Базова модель нефункційних характеристик для оцінки якості штучного інтелекту // Радіоелектронні і комп'ютерні системи 2(102). с. 1-14.
ISO/IEC 25010 (2011). ISO/IEC 25010:2011, Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models.
Харченко В.С., Жихарев В.Я., Іллюшко В.М. та ін. (2004), Основи надійності цифрових систем, Харків: Нац. Аерокосм. Ун-т. «ХАІ».
NIST Four Principles of Explainable Artificial Intelligence: Draft NISTIR 8312 / P. J. Phillips, C. A. Hahn, P. C. Fontana, D. A. Broniatowski, M. A. Przybocki, C. A. Hahn, P. C. Fontana. – Gaithersburg: National Institute of Standards and Technology, 2020. – 24 p. DOI: 10.6028/NIST.IR.8312.
European Commission, Directorate-General for Communications Networks, Content and Technology, Ethics guidelines for trustworthy AI, Publications Office, (2019), https://data.europa.eu/doi/10.2759/346720
UNESCO (2021), Recommendation on the Ethics of Artificial Intelligence, https://unesdoc.unesco.org/ark:/48223/pf0000380455, Дата звернення 21.05.2022
ISO/IEC TR 24028:2020. Information technology. Artificial intelligence. Overview of trustworthiness in artificial intelligence [Electronic resource]. – Available at: https://www.iso.org/standard/77608.html. – 10.03.2022.
OECD. Tools for Trustworthy AI: A Framework to Compare Implementation Tools [Electronic resource]. – Available at: https://www.oecd.org/science/tools-for-trustworthy-ai-008232ec-en.htm. – 10.03.2022.
Москаленко В. В. Багатоетапний метод глибинного навчання з попереднім самонавчанням для класифікаційного аналізу дефектів стічних труб [Текст] / В. В. Москаленко, М. О. Зарецький, А. С. Москаленко, А. Г. Коробов, Я. Ю. Ковальський // Радіоелектронні і комп’ютерні системи. – 2021. – № 4. – C. 71-81. DOI: 10.32620/reks.2021.4.06.
Kuchuk, H. System of license plate recognition considering large camera shooting angles [Text] / H. Kuchuk, A. Podorozhniak, N. Liubchenko, D. Onischenko // Radioelectronic and Computer Systems. – 2021. – No. 4. – P. 82-91. DOI: 10.32620/reks.2021.4.07
Some, Evariste & Gondwe, Greg & Rowe, Evan. (2019). Cybersecurity and Driverless Cars: In Search for a Normative Way of Safety. 352-357. 10.1109/IOTSMS48152.2019.8939168.
Gordieiev, O. IT-oriented software quality models and evolution of the prevailing characteristics [Text] / O. Gordieiev, V. Kharchenko // Dependable Systems, Services and Technologies (DESSERT): Proceeding of 9th Int. Conf., 2018. – P. 375-380. DOI: 10.1109/DESSERT.2018.8409162.
