INVESTIGATION OF METHODS FOR DETECTING ANOMALIES AT THE STAGE OF DATA PRE-PROCESSING
DOI:
https://doi.org/10.26906/SUNZ.2022.1.052Keywords:
data preprocessing, machine learning, preprocessing, Standard Deviation Method, Local Outlier Factor, Random Forest, KNN, BaggingAbstract
The subject of the research is the methods and means of detecting anomalies in data. The purpose of the article is to improve the quality of data classification by detecting anomalies at the pre-processing stage. Task: to investigate methods for detecting anomalies at the stage of data preprocessing, to determine the decision threshold for each of the methods and to evaluate the quality of classification before and after preprocessing. Methods used are: artificial intelligence methods, machine learning, ensemble methods. The following results were obtained: anomaly detection methods were studied: Standard Deviation Method, Local Outlier Factor method, Isolation Forest method. The dependence of the number of anomalies on the decision threshold for each of the methods is obtained. The evaluation of the quality of data preprocessing was performed using classifiers based on the KNN and Bagging methods. The studied methods are implemented programmatically using the GOOGLE COLAB cloud service based on Jupyter Notebook. Conclusions. The scientific novelty of the results obtained lies in the study of anomaly detection methods at the stage of data preprocessing, the choice of a preprocessing meta-algorithm and the determination of its optimal settings.Downloads
References
Cui Z. G., Cao Y. Wu, Liu H.N, Qiu, Z. F., Chen, C. W. Research on preprocessing technology of building energy consumption monitoring data based on machine learning algorithm. Build. Sci. 2018, Vol. 34 (2), C. 94–99.
Bernhard Schölkopf, Robert C Williamson, Alex J Smola et al. Support vector method for novelty detection Advances in Neural Information Processing Systems, Denver, United States, 2000, P. 582–588.
Zhou Chong, Paffenroth Randy C. Anomaly detection with robust deep autoencoders, ACM SIGKDD International Conference on Knowledge Discovery and Data Mining,Halifax, Canada, 2017, P. 665–674.
Adrian Alan Pol, Gianluca Cerminara, Cécile Germain et al. Detector monitoring with artificial neural networks at the CMS experiment at the CERN Large Hadron Collider. Computing and Software for Big Science. 2019,Vol. 3 (1), P. 3-8.
Stankevicius Mantas, Marcinkevicius Virginijus, Rapsevicius Valdas. Comparison of Supervised Machine Learning Techniques for CERN CMS Of49 fline Data Certification. Doctoral Consortium/Forum@ DB&IS, 2018, P. 170–176.,
Adrian Alan Pol, Virginia Azzolini, Gianluca Cerminara et al. Anomaly detection using Deep Autoencoders for the assessment of the quality of the data acquired by the CMS experiment, EPJ Web of Conferences, EDP Sciences, 2019, Vol. 214, P. 5.
Maxim Borisyak, Artem Ryzhikov, Andrey Ustyuzhanin et al. (1 + epsilon)-class Classification: an Anomaly Detection Method for Highly Imbalanced or Incomplete Data Sets, Journal of Machine Learning Research, 2020, Vol. 21(72), P. 1–22.
Гавриленко С.Ю., Шевердін І. В. Розробка методу ідентифікації стану комп’ютерної системи на основі алгоритму «Isolation Forest», Радіоелектроніка, інформатика, управління, 2021, №.1(56), Р. 105-116.
Support vector method for novelty detection / Bernhard Schölkopf,Robert C Williamson, Alex J Smola et al., Advances in Neural InformationProcessing System, Denver, United States, 2000, P. 582–588.
Большаков А.С., Губанкова Е.В. Обнаружение аномалий в компьютерных сетях с использованием методов машинного обучения, Телекоммуникационные устройства и системы, 2020. Т. 10. № 1. С. 37-42.
Breunig M. M. et al. LOF: identifying density-based local outliers, ACM sigmodrecord – ACM, 2000, Vol. 29 (2), P. 93-104.
Breunig, M. M ., Kriegel, H.P., Ng, R.T ., Сандер, Дж.. LOF: определение локальных выбросов на основе плотности, Материалы Международной конференции ACM SIGMOD 2000 г. по управлению данными. SIGMOD, 2000, С. 93–104.
Zhou, H., Wang, P., & Li, H. Research on adaptive parameter determination in DBSCAN algorithm, Journal of Xi'an University of Technology, 2014, 28(3), P.289-292.
Karami A., Johansson R. Choosing DBSCAN parameters automatically using differential evolution, International Journal of Computer Applications, 2014, Vol. 91(7), P.1-11.
Arthur Zimek, Erich Schubert. Outlier Detection. Encyclopedia of Database Systems, Springe,r New York, 2017. Р.96-106.
Dai, Zhifeng and Xiaomin Chang. Predicting Stock Return with Economic Constraint: Can Interquartile Range Truncate the Outliers, Mathematical Problems in Engineering, 2021, Р. 1-12.
Liu, Fei Tony, Ting, Kai Ming and Zhou, Zhi-Hua. Isolation forest, Proceedings of the 2008 Eighth IEEE International Conference on Data Mining, December 2008, P. 413–422.
