METHODS FOR RECOGNISING HAND GESTURES USING COMPUTER VISION TO ASSIST PEOPLE WITH MOVEMENT OR SPEECH IMPAIRMENTS

Abstract

Relevance. Real-time hand gesture recognition systems play an important role in the development of modern technologies for human interaction with the “smart environment.” They create conditions for contactless device control and open up new communication opportunities for people with speech or motor impairments. The object of research. The object of research is a real-time hand gesture recognition system using computer vision and machine learning methods. The research problem is to create an accessible and effective means of communication for people with speech or motor impairments, as well as to expand the possibilities of controlling devices and elements of a “smart home” without physical contact or voice commands. Purpose of the article. The purpose of this article is to investigate and compare different methods for real-time hand gesture recognition using computer vision and machine learning. Specifically, the study evaluates the performance of MediaPipe Hands versus YOLOv8 for detecting and classifying hand gestures. The aim is to identify which approach provides higher accuracy and better adaptability for applications such as assistive communication for people with speech or motor impairments. Research results. In the course of the work, a prototype system was developed that combines video processing technologies (GStreamer or FFmpeg), the MediaPipe Hands library for determining key points of the hand, and CNN or LSTM deep learning models for classifying gestures and movement sequences. The results demonstrated the possibility of accurate gesture recognition in real time, as well as flexible adaptation to individual user characteristics. Conclusions. Interpretation of the results showed that the effectiveness of the system is due to the use of a comprehensive approach: the combination of 3D coordinate detection of key hand points with a neural network allowed high stability to be achieved even under changing lighting or background conditions. A distinctive feature of the proposed solution is the ability to recognise sequences of gestures that form phrases or commands (for example, “I want to drink”), as well as a function for teaching user gestures to expand the system’s individual vocabulary. The developed technology can be used in assistive communication systems for people with disabilities, in medical and rehabilitation facilities, as well as for controlling smart home elements. Its implementation contributes to increasing the accessibility of digital technologies and improving the quality of life of users.