Survey of Optimization Algorithms in Modern Neural Networks

The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many proble...

Deskribapen osoa

Gorde:

Xehetasun bibliografikoak
Egile Nagusiak:	Abdulkadirov, R. I., Абдулкадиров, Р. И., Lyakhov, P. A., Ляхов, П. А., Nagornov, N. N., Нагорнов, Н. Н.
Formatua:	Статья
Hizkuntza:	English
Argitaratua:	2023
Gaiak:	Approximation Spiking neural networks Bilevel optimization Quasi-Newton methods Gradient-free optimization Graph neural networks Quantum neural networks Quantum computations Physics-informed neural networks Optimization methods Fractional order optimization Information geometry
Sarrera elektronikoa:	https://dspace.ncfu.ru/handle/20.500.12258/24231
Etiketak:	Etiketa erantsi Etiketarik gabe, Izan zaitez lehena erregistro honi etiketa jartzen!

id	ir-20.500.12258-24231
record_format	dspace
spelling	ir-20.500.12258-242312023-08-03T08:23:28Z Survey of Optimization Algorithms in Modern Neural Networks Abdulkadirov, R. I. Абдулкадиров, Р. И. Lyakhov, P. A. Ляхов, П. А. Nagornov, N. N. Нагорнов, Н. Н. Approximation Spiking neural networks Bilevel optimization Quasi-Newton methods Gradient-free optimization Graph neural networks Quantum neural networks Quantum computations Physics-informed neural networks Optimization methods Fractional order optimization Information geometry The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many problems, remains the most well-utilized branch of machine learning. Thus, one must select appropriate neural network architectures, data processing, and advanced applied mathematics tools. A common challenge for these networks is achieving the highest accuracy in a short time. This problem is solved by modifying networks and improving data pre-processing, where accuracy increases along with training time. Bt using optimization methods, one can improve the accuracy without increasing the time. In this review, we consider all existing optimization algorithms that meet in neural networks. We present modifications of optimization algorithms of the first, second, and information-geometric order, which are related to information geometry for Fisher–Rao and Bregman metrics. These optimizers have significantly influenced the development of neural networks through geometric and probabilistic tools. We present applications of all the given optimization algorithms, considering the types of neural networks. After that, we show ways to develop optimization algorithms in further research using modern neural networks. Fractional order, bilevel, and gradient-free optimizers can replace classical gradient-based optimizers. Such approaches are induced in graph, spiking, complex-valued, quantum, and wavelet neural networks. Besides pattern recognition, time series prediction, and object detection, there are many other applications in machine learning: quantum computations, partial differential, and integrodifferential equations, and stochastic processes. 2023-08-03T08:22:31Z 2023-08-03T08:22:31Z 2023 Статья Abdulkadirov, R., Lyakhov, P., Nagornov, N. Survey of Optimization Algorithms in Modern Neural Networks // Mathematics. - 2023. - 11 (11), статья № 2466. - DOI: 10.3390/math11112466 http://hdl.handle.net/20.500.12258/24231 en Mathematics application/pdf application/pdf
institution	СКФУ
collection	Репозиторий
language	English
topic	Approximation Spiking neural networks Bilevel optimization Quasi-Newton methods Gradient-free optimization Graph neural networks Quantum neural networks Quantum computations Physics-informed neural networks Optimization methods Fractional order optimization Information geometry
spellingShingle	Approximation Spiking neural networks Bilevel optimization Quasi-Newton methods Gradient-free optimization Graph neural networks Quantum neural networks Quantum computations Physics-informed neural networks Optimization methods Fractional order optimization Information geometry Abdulkadirov, R. I. Абдулкадиров, Р. И. Lyakhov, P. A. Ляхов, П. А. Nagornov, N. N. Нагорнов, Н. Н. Survey of Optimization Algorithms in Modern Neural Networks
description	The main goal of machine learning is the creation of self-learning algorithms in many areas of human activity. It allows a replacement of a person with artificial intelligence in seeking to expand production. The theory of artificial neural networks, which have already replaced humans in many problems, remains the most well-utilized branch of machine learning. Thus, one must select appropriate neural network architectures, data processing, and advanced applied mathematics tools. A common challenge for these networks is achieving the highest accuracy in a short time. This problem is solved by modifying networks and improving data pre-processing, where accuracy increases along with training time. Bt using optimization methods, one can improve the accuracy without increasing the time. In this review, we consider all existing optimization algorithms that meet in neural networks. We present modifications of optimization algorithms of the first, second, and information-geometric order, which are related to information geometry for Fisher–Rao and Bregman metrics. These optimizers have significantly influenced the development of neural networks through geometric and probabilistic tools. We present applications of all the given optimization algorithms, considering the types of neural networks. After that, we show ways to develop optimization algorithms in further research using modern neural networks. Fractional order, bilevel, and gradient-free optimizers can replace classical gradient-based optimizers. Such approaches are induced in graph, spiking, complex-valued, quantum, and wavelet neural networks. Besides pattern recognition, time series prediction, and object detection, there are many other applications in machine learning: quantum computations, partial differential, and integrodifferential equations, and stochastic processes.
format	Статья
author	Abdulkadirov, R. I. Абдулкадиров, Р. И. Lyakhov, P. A. Ляхов, П. А. Nagornov, N. N. Нагорнов, Н. Н.
author_facet	Abdulkadirov, R. I. Абдулкадиров, Р. И. Lyakhov, P. A. Ляхов, П. А. Nagornov, N. N. Нагорнов, Н. Н.
author_sort	Abdulkadirov, R. I.
title	Survey of Optimization Algorithms in Modern Neural Networks
title_short	Survey of Optimization Algorithms in Modern Neural Networks
title_full	Survey of Optimization Algorithms in Modern Neural Networks
title_fullStr	Survey of Optimization Algorithms in Modern Neural Networks
title_full_unstemmed	Survey of Optimization Algorithms in Modern Neural Networks
title_sort	survey of optimization algorithms in modern neural networks
publishDate	2023
url	https://dspace.ncfu.ru/handle/20.500.12258/24231
work_keys_str_mv	AT abdulkadirovri surveyofoptimizationalgorithmsinmodernneuralnetworks AT abdulkadirovri surveyofoptimizationalgorithmsinmodernneuralnetworks AT lyakhovpa surveyofoptimizationalgorithmsinmodernneuralnetworks AT lâhovpa surveyofoptimizationalgorithmsinmodernneuralnetworks AT nagornovnn surveyofoptimizationalgorithmsinmodernneuralnetworks AT nagornovnn surveyofoptimizationalgorithmsinmodernneuralnetworks
_version_	1809808535312138240

Survey of Optimization Algorithms in Modern Neural Networks

Antzeko izenburuak