EFFECTIVENESS OF EVOLUTIONARY ALGORITHMS IN NEURAL NETWORK OPTIMIZATION TASKS: UNCONDITIONAL MINIMIZATION, PRUNING, AND HYPERPARAMETER OPTIMIZATION

Abstract

Relevance. Modern neural networks contain a significant number of redundant parameters, which increases training time, energy consumption, and hardware requirements. Standard gradient methods tend to get trapped in local minima on multimodal tasks; magnitude pruning ignores the structural importance of layers. Zero-cost proxies, particularly SynFlow, estimate architecture viability via signal propagation capacity without full training, yet their applicability boundaries remain insufficiently explored. Object of study: neural network optimization methods. Objective: a systematic experimental comparison of evolutionary, gradient-based, and Bayesian methods within a unified framework, with an emphasis on the role of SynFlow as a proxy for evaluating genome (architecture) fitness. Results. At extreme sparsity (98%), the combination of CMA-ES + SynFlow (signal propagation capacity analysis) with the addition of mathematical energy compensation overcomes network degradation and achieves State-of-the-Art performance (F1 = 0.81). This approach outperforms both classic magnitude pruning and gradient-based soft mask methods (SoftMask), requiring orders of magnitude less computational cost compared to approximate training-based approaches. At moderate sparsity (≤ 90%), magnitude pruning retains its superiority. In hyperparameter optimization (HPO), CMA-ES achieves practical equivalence with Optuna TPE in terms of solution quality, while in high-dimensional settings, it requires orders of magnitude less execution time. Using SynFlow as an objective function in HPO allows for accelerating the search by over 250 times, but yields inferior final accuracy compared to full training. Conclusions: Evolutionary algorithms are effective for multimodal tasks and structural optimization problems; for differentiable weight training tasks, Adam prevails (F1 ≈ 0.97 on Digits). Zero-cost proxies are context-specific.