MODEL AND METHOD OF INCLUSIVE INTERACTION WITH VISUAL AND TEXTUAL CONTENT BASED ON CONTEXTUAL GESTURE INTERPRETATION
DOI:
https://doi.org/10.26906/SUNZ.2026.3.057Keywords:
contactless interaction, tracking, gesture recognition, human-computer interaction, Media Pipe Hands, gesture control, interface, model, methodAbstract
Background. Contactless gesture control, as an auxiliary channel of interaction with an interface, has now become widely used in various areas of life, as it provides a convenient, safe, and, if necessary, inclusive way of interacting with visual, graphical, and textual content. The subject of the study is the process of contactless control of basic navigation scenarios for working with visual, textual, and web content using hand tracking methods and contextual interpretation of gesture events. The objective of the study is to develop and experimentally validate a method of contactless interaction with visual and textual content based on RGB hand tracking and contextual interpretation of gestures for implementing practical user scenarios. The main research task of the study is to develop a conceptual model of context-adaptive inclusive user interaction with an information system and the external environment. Methods. Functional modeling, computer vision methods, analysis of hand keypoints, geometric rules for gesture recognition, and experimental testing were used. Results. A method is proposed in which a gesture event is converted into a command only after determining the type and analyzing the context, namely the technical and scenario context. The experimental validation was carried out using five basic gestures and three scenarios. Conclusions. The obtained results confirmed the operability of the method as an auxiliary channel of contactless interaction with visual and textual content.Downloads
References
1. World Health Organization (2023), “Disability”, available at: https://www.who.int/news-room/fact-sheets/detail/disabilityand-health
2. World Health Organization (n.d.), “Hand hygiene”, available at: https://www.who.int/teams/integrated-healthservices/infection-prevention-control/hand-hygiene
3. World Health Organization (2021), “Key facts and figures. World Hand Hygiene Day 2021”, available at: https://www.who.int/campaigns/world-hand-hygiene-day/2021/key-facts-and-figures
4. Vysocký, A., Grushko, S., Oščádal, P., Kot, T., Babjak, J., Jánoš, R., Sukop, M., and Bobovský, Z. (2020), “Analysis of Precision and Stability of Hand Tracking with Leap Motion Sensor”, Sensors, vol. 20, no. 15, article 4088, doi: https://doi.org/10.3390/s20154088 DOI: https://doi.org/10.3390/s20154088
5. Jeon, H., Choi, H., Noh, D., Kim, T., and Lee, D. (2022), “Wearable Inertial Sensor-Based Hand-Guiding Gestures Recognition Method Robust to Significant Changes in the Body-Alignment of Subject”, Mathematics, vol. 10, no. 24, article 4753, doi: https://doi.org/10.3390/math10244753 DOI: https://doi.org/10.3390/math10244753
6. Yarovyi, V., Barkovska, O., Maksymov, D., Ni, Y., and Raptanov, D. (2026), “Methods for recognising hand gestures using computer vision to assist people with movement or speech impairments”, Control, Navigation and Communication Systems, no. 1(83), pp. 135–141, doi: https://doi.org/10.26906/SUNZ.2026.1.135 DOI: https://doi.org/10.26906/SUNZ.2026.1.135
7. Barkovska, O., Ruban, I., Tymoshenko, D., Holovchenko, O., and Yankovskyi, O. (2025), “Research on mobile machine learning platforms for human gesture recognition in human-machine interaction systems”, Technology Audit and Production Reserves, vol. 2, no. 2(82), pp. 6–14, doi: https://doi.org/10.15587/2706-5448.2025.325423 DOI: https://doi.org/10.15587/2706-5448.2025.325423
8. Alsharif, B., Alalwany, E., Ibrahim, A., Mahgoub, I., and Ilyas, M. (2025), “Real-Time American Sign Language Interpretation Using Deep Learning and Keypoint Tracking”, Sensors, vol. 25, no. 7, article 2138, doi: https://doi.org/10.3390/s25072138 DOI: https://doi.org/10.3390/s25072138
9. Yaseen, Kwon, O.-J., Kim, J., Jamil, S., Lee, J., and Ullah, F. (2024), “Next-Gen Dynamic Hand Gesture Recognition: MediaPipe, Inception-v3 and LSTM-Based Enhanced Deep Learning Model”, Electronics, vol. 13, no. 16, article 3233, doi: https://doi.org/10.3390/electronics13163233 DOI: https://doi.org/10.3390/electronics13163233
10. Chen, I.D.S., Yang, C.-M., Wu, S.-S., Yang, C.-K., Chen, M.-J., Yeh, C.-H., and Lin, Y.-H. (2024), “Continuous Recognition of Teachers’ Hand Signals for Students with Attention Deficits”, Algorithms, vol. 17, no. 7, article 300, doi: https://doi.org/10.3390/a17070300 DOI: https://doi.org/10.3390/a17070300
11. Gena, C., Vernero, F., Mattutino, C., Persichella, G., and Lomanto, L. (2019), “Experimenting Touchless Gestural Interaction for a University Public Web-based Display”, Adjunct Proceedings of the 13th Biannual Conference of the Italian SIGCHI Chapter (CHItaly 2019), Padova, Italy, 5 p., available at: https://arxiv.org/abs/2011.09749
12. Drăgoi, M.V., et al. (2025), “An Inclusive Offline Learning Platform Integrating Gesture Recognition and Local AI Models”, Biomimetics, vol. 10, no. 10, article 693, doi: https://doi.org/10.3390/biomimetics10100693 DOI: https://doi.org/10.3390/biomimetics10100693
13. Şen, Z., Çil, E.T., and Serif, T. (2024), “Web Camera-Based Gamified Rehabilitation System for Improving Fine Motor Skills: Pilot Study”, International Journal of Digital Waste Engineering, vol. 1, no. 1, pp. 15–22, doi: https://doi.org/10.5281/zenodo.14580508
14. Husna, R., Brata, K.C., Anggraini, I.T., Funabiki, N., Rahmadani, A.A., and Fan, C.-P. (2025), “An Investigation of Hand Gestures for Controlling Video Games in a Rehabilitation Exergame System”, Computers, vol. 14, no. 1, article 25, doi: https://doi.org/10.3390/computers14010025 DOI: https://doi.org/10.3390/computers14010025
15. Panagiotou, C., Faliagka, E., Antonopoulos, C.P., and Voros, N. (2025), “Multidisciplinary ML Techniques on Gesture Recognition for People with Disabilities in a Smart Home Environment”, AI, vol. 6, no. 1, article 17, doi: https://doi.org/10.3390/ai6010017 DOI: https://doi.org/10.3390/ai6010017
16. Hobbs, T., and Ali, A. (2025), “Smart Home Control Using Real-Time Hand Gesture Recognition and Artificial Intelligence on Raspberry Pi 5”, Electronics, vol. 14, no. 20, article 3976, doi: https://doi.org/10.3390/electronics14203976 DOI: https://doi.org/10.3390/electronics14203976
17. Lopes, M.C., Silva, P.A., Marenco, L., Vilas Boas, E.C., de Carvalho, J.G.A., Ferreira, C.A., Carvalho, A.L.O., Guimarães, C.V.R., Aquino, G.P., and de Figueiredo, F.A.P. (2026), “GECO: A Real-Time Computer Vision-Assisted Gesture Controller for Advanced IoT Home System”, Sensors, vol. 26, no. 1, article 61, doi: https://doi.org/10.3390/s26010061 DOI: https://doi.org/10.3390/s26010061
18. Wang, X., Shen, H., Yu, H., Guo, J., and Wei, X. (2022), “Hand and Arm Gesture-based Human-Robot Interaction: A Review”, Proceedings of the 6th International Conference on Algorithms, Computing and Systems, Association for Computing Machinery, New York, article 12, pp. 1–7, doi: https://doi.org/10.1145/3564982.3564996 DOI: https://doi.org/10.1145/3564982.3564996
19. Yun, G., Kwak, H., and Kim, D.H. (2024), “Single-Handed Gesture Recognition with RGB Camera for Drone Motion Control”, Applied Sciences, vol. 14, no. 22, article 10230, doi: https://doi.org/10.3390/app142210230 DOI: https://doi.org/10.3390/app142210230
20. Marques, P., Váz, P., Silva, J., Martins, P., and Abbasi, M. (2025), “Real-Time Gesture-Based Hand Landmark Detection for Optimized Mobile Photo Capture and Synchronization”, Electronics, vol. 14, no. 4, article 704, doi: https://doi.org/10.3390/electronics14040704 DOI: https://doi.org/10.3390/electronics14040704
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Olesia Barkovska
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
