ASSESSING THE QUALITY OF AN AUGMENTED REALITY SYSTEM: AN ANALYSIS OF THE STATE OF RESEARCH
DOI:
https://doi.org/10.26906/SUNZ.2022.4.079Keywords:
augmented reality, quality models, metrics, user experience, usabilityAbstract
Due to the rapid spread of augmented reality technology in various spheres of human activity, the question of quality assessment of these systems has arisen. Although studies in this direction have already been conducted, they covered individual quality indicators and did not offer a single comprehensive model. The ISO-25010 standard was used to assess the quality of augmented reality systems. But this standard does not take into account the special differences of systems and applications of augmented reality, which makes research for this direction extremely relevant. The purpose of this article is to review and systematize existing publications on the topic of augmented reality quality research to determine current results and current research directions for further use in the development of models and metric methods for assessing the quality of augmented reality systems. During the review, it was established that the largest number of studies is devoted to the quality of using augmented reality and the user experience of using such systems. In most cases, it is suggested to conduct an assessment using the heuristic method. Ther e are also attempts to develop metrics based on existing quality standards for software systems or to modernize existing evaluation models. Proposals for the development of separate metric methods for evaluating augmented reality systems are absent or are i n closed access. Based on the results of the review, it can be concluded that when developing models and metrics for assessing the quality of augmented reality systems, it will be necessary to take into account such characteristics as the quality of use, the quality of the user experience, and visual appeal. This list is planned to be supplemented after conducting a more in-depth review of existing research.Downloads
References
Azuma R. T. A Survey of Augmented Reality [Electronic resource] / Ronald T. Azuma // Presence: Teleoperators and Virtual Environments. – 1997. – Vol. 6, no. 4. – P. 355-385. – DOI: https://doi.org/10.1162/pres.1997.6.4.355
Milgram P. A Taxonomy Of Mixed Reality Visual Displays [Electronic resource] / Paul Milgram, Fumio Kishino // IEICE Transactions on Information Systems. – 1994. – E77D, no. 12. – P. 1321–1329. – Mode of access: https://cs.gmu.edu/~zduric/cs499/Readings/r76JBo-Milgram_IEICE_1994.pdf
Kaur K. Designing virtual environments for usability [Electronic resource] : Electronic Thesis or Dissertation / Kaur Kulwinder. – [S. l.], 1998. – Mode of access: http://openaccess.city.ac.uk/7567/
Sutcliffe A. G. Evaluating the usability of virtual reality user interfaces [Electronic resource] / A. G. Sutcliffe, K. Deol Kaur // Behaviour & Information Technology. – 2000. – Vol. 19, no. 6. – P. 415-426. – DOI: http://dx.doi.org/10.1080/014492900750052679
Bowman D. A. A Survey of Usability Evaluation in Virtual Environments: Classification and Comparison of Methods [Electronic resource] / Doug A. Bowman, Joseph L. Gabbard, Deborah Hix // Presence: Teleoperators and Virtual Environments. – 2002. – Vol. 11, no. 4. – P. 404-424. – DOI: https://doi.org/10.1162/105474602760204309
Gabbart J. L. Usability Engineering of Virtual Environments[Electronic resource] /Joseph L. Gabbard, Deborah Hix // Handbook of Virtual Environments (1st Edition). – Boca Raton, CRC Press, 2002. – P. 681-699. DOI: https://doi.org/10.1201/9780585399102
Tromp J. G. Systematic Usability Evaluation and Design Issues for Collaborative Virtual Environments [Electronic resource] / Jolanda G. Tromp, Anthony Steed, John R. Wilson // Presence: Teleoperators and Virtual Environments. – 2003. – Vol. 12, no. 3. – P. 241–267. – DOI: https://doi.org/10.1162/105474603765879512
Gabbard J. L. Researching Usability Design and Evaluation Guidelines for Augmented Reality (AR) Systems [Electronic resource] / Joseph L. Gabbard. – Mode of access: https://www.rkriz.net/sv/classes/ESM4714/Student_Proj/class00/gabbard/index.html
Usability engineering: domain analysis activities for augmented-reality systems [Electronic resource] / Joseph Gabbard [et al.] // Electronic Imaging 2002, San Jose, CA / ed. by A. J. Woods [et al.]. – [S. l.], 2002. – DOI: http://dx.doi.org/10.1117/12.468073
Bach C. Obstacles and Perspectives for Evaluating Mixed Reality Usability [Electronic resource] / Cédric Bach, Dominique L. Scapin // ResearchGate. – Mode of access: https://www.researchgate.net/publication/221104007_Obstacles_and_Perspectives_for_Evaluating_Mixed_Reality_Usability
Träskbäck M. User requirements and usability of mixed reality applications [Electronic resource] : Licentiate thesis / Träskbäck Marjaana. – Helsinki, 2004. – 109 p. – Mode of access: https://aaltodoc.aalto.fi/bitstream/handle/123456789/48/urn012717.pdf?sequence=1&isAllowed=y
Alzua-Sorzabal A. An Experimental Usability Study for Augmented Reality Technologies in the Tourist Sector [Electronic resource] / Aurkene Alzua-Sorzabal, María Teresa Linaza, Marina Abad // Information and Communication Technologies in Tourism 2007. – Vienna. – P. 231-242. – DOI: http://dx.doi.org/10.1007/978-3-211-69566-1_22
Gabbard J. L. Usability Engineering for Augmented Reality: Employing User-Based Studies to Inform Design [Electronic resource] / J. L. Gabbard, J. E. Swan // IEEE Transactions on Visualization and Computer Graphics. – 2008. – Vol. 14, no. 3. – P. 513-525. – DOI: https://doi.org/10.1109/TVCG.2008.24
Kostaras N. N. Assessing the Usability of Augmented Reality Systems / Nektarios N. Kostaras, Michalis N. Xenos // 13th Panhellenic Conference on Informatics, Corfu, 10–12 September 2009. – Los Alamitos, 2009. – P. 197–201
Finstad K. The Usability Metric for User Experience [Electronic resource] / Kraig Finstad // Interacting with Computers. – 2010. – Vol. 22, no. 5. – P. 323-327. – DOI: http://dx.doi.org/10.1016/j.intcom.2010.04.004
Lin H.-C. K. Establishment And Usability Evaluation Of An Interactive Ar Learning System On Conservation Of Fish [Electronic resource] / Hao-Chiang Koong Lin [et al.] // The Turkish Online Journal of Educational Technology. – 2011. – Vol. 10, no. 4. – P. 181–187. – Mode of access: https://files.eric.ed.gov/fulltext/EJ946626.pdf
Rusu C. A Methodology to Establish Usability Heuristics [Electronic resource] / Cristian Rusu [et al.] // The Fourth International Conference on Advances in Computer-Human Interactions, Gosier, 23–28 February 2011. – [S. l.], 2011. – P. 59–62. – Mode of access: https://www.researchgate.net/publication/229040164_A_Methodology_to_establish_usability_heuristics
Ko S. M. Usability Principles for Augmented Reality Applications in a Smartphone Environment [Electronic resource] / Sang Min Ko, Won Suk Chang, Yong Gu Ji // International Journal of Human-Computer Interaction. – 2013. – Vol. 29, no. 8. – P. 501-515. – DOI: https://doi.org/10.1080/10447318.2012.722466
Kalalahti J. Developing Usability Evaluation Heuristics For Augmented Reality Applications [Electronic resource] : Masters thesis / Kalalahti Joanna. – Lappeenranta, 2015. – 90 p. – Mode of access: https://lutpub.lut.fi/bitstream/handle/10024/103081/masters_thesis_joanna_kalalahti_31122014.pdf?sequence=2
Tuli N. Usability Principles for Augmented Reality based Kindergarten Applications [Electronic resource] / Neha Tuli, Archana Mantri // Procedia Computer Science. – 2020. – Vol. 172. – P. 679-687. – DOI: https://doi.org/10.1016/j.procs.2020.05.089
Al-Obaidi A. Usability Principles for Augmented Reality Applications in Education [Electronic resource] / Arwa Al-Obaidi, Master Prince // International Journal of Computer Science and Network Security. – 2022. – Vol. 22, no. 1. – P. 49–54. – DOI: http://dx.doi.org/10.22937/IJCSNS.2022.22.1.8
Expected user experience of mobile augmented reality services: a user study in the context of shopping centres [Electronic resource] / Thomas Olsson [et al.] // Personal and Ubiquitous Computing. – 2011. – Vol. 17, no. 2. – P. 287-304. – DOI: http://dx.doi.org/10.1007/s00779-011-0494-x
Olsson T. User Expectations and Experiences of Mobile Augmented Reality Services [Electronic resource] / Thomas Olsson // Tampere University of Technology. – 2012. – P. 1–102. – Mode of access: https://urn.fi/URN:ISBN:978-952-15-2953-5
Towards a Model of User Experience in Immersive Virtual Environments [Electronic resource] / Katy Tcha-Tokey [et al.] // Advances in Human-Computer Interaction. – 2018. – Vol. 2018. – P. 1–10. – DOI: https://doi.org/10.1155/2018/7827286
Scheibmeir J. Quality Model for Testing Augmented Reality Applications [Electronic resource] / Jim Scheibmeir, Yashwant K. Malaiya // 2019 IEEE 10th Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), New York City, NY, USA, 10–12 October 2019. – [S. l.], 2019. – DOI: https://doi.org/10.1109/UEMCON47517.2019.8992974
David A. The value of visual quality and service quality to augmented reality enabled mobile shopping experience [Electronic resource] / Alsius David [et al.] // Quality Management Journal. – 2021. – Vol. 28, no. 3. – P. 116–127. – DOI: https://doi.org/10.1080/10686967.2021.1920868
Huiyu D. Confusing Image Quality Assessment: Towards Better Augmented Reality Experience [Electronic resource] / Duan Huiyu [et al.] // arXiv Computer Vision and Pattern Recognition. – 2022. – P. 1–13. – DOI: https://doi.org/10.48550/arXiv.2204.04900
Leue M. C. A Theoretical Model of Augmented Reality Acceptance / M. Claudia Leue, Dario tom Dieck, Timothy Jung // Department of Food and Tourism Management Manchester Metropolitan University. – 2014. – P. 1–5. – Mode of access: https://agrilifecdn.tamu.edu/ertr/files/2014/02/enter2014_RN_93.pdf
Understanding users’ continuance intention toward smartphone augmented reality applications [Electronic resource] / Keesung Kim [et al.] // Information Development. – 2014. – Vol. 32, no. 2. – P. 161–174. – DOI: https://doi.org/10.1177/0266666914535119
Towards the development of guidelines for educational evaluation of augmented reality tools [Electronic resource] / Manoela M. O. da Silva [et al.] // 2016 IEEE Virtual Reality Workshop on K-12 Embodied Learning through Virtual & Augmented Reality (KELVAR), Greenville, SC, USA, 19 March 2016. – [S. l.], 2016. – DOI: https://doi.org/10.1109/KELVAR. 2016.7563677
User evaluation of mobile augmented reality scenarios [Electronic resource] / Thomas Olsson [et al.] // Journal of Ambient Intelligence and Smart Environments. – 2012. – Vol. 4, no. 1. – P. 29-47. – DOI: https://doi.org/10.3233/AIS-2011-0127
Dünser A. Evaluating Augmented Reality Systems [Electronic resource] / Andreas Dünser, Mark Billinghurst // Handbook of Augmented Reality. – New York, NY, 2011. – P. 289-307. – DOI: https://doi.org/10.1007/978-1-4614-0064-6_13
Augmented Reality Design Heuristics: Designing for Dynamic Interactions [Electronic resource] / Tristan C. Endsley [et al.] // Proceedings of the Human Factors and Ergonomics Society Annual Meeting. – 2017. – Vol. 61, no. 1. – P. 2100–2104. – DOI: https://doi.org/10.1177/1541931213602007
Graphic User Interface Design Principles for Designing Augmented Reality Applications [Electronic resource] / Afshan Ejaz [et al.] // International Journal of Advanced Computer Science and Applications. – 2019. – Vol. 10, no. 2. – DOI: https://doi.org/10.14569/IJACSA.2019.0100228
Орехова А.А. Качество в использовании интерактивных параллельных программных средств / А. А. Орехова, К. А. Бохан, В. С. Харченко // Системи управління, навігації та зв'язку . - Київ, 2012 . - Т. 1. - Вип. 1 (21). - С. 127-131.
Orechova A. Human-machine interface quality assessment techniques: Green and safety issues [Electronic resource] / Аnastasiia Orechova, Oleksandr Orehov, Vyacheslav Kharchenko // DT 2014 - 10th International Conference on Digital Technologies. – IEEE, 2014. – P. 248-254. – DOI: https://doi.org/10.1109/DT.2014.6868723
