A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions

In this paper, we propose a natural gradient descent algorithm with momentum based on Dirichlet distributions to speed up the training of neural networks. This approach takes into account not only the direction of the gradients, but also the convexity of the minimized function, which significantly a...

Полное описание

Сохранить в:

Библиографические подробности
Главные авторы:	Abdulkadirov, R. I., Абдулкадиров, Р. И., Lyakhov, P. A., Ляхов, П. А.
Формат:	Статья
Язык:	English
Опубликовано:	2023
Темы:	Dirichlet distributions Natural gradient descent Pattern recognition Machine learning
Online-ссылка:	https://dspace.ncfu.ru/handle/20.500.12258/23476
Метки:	Добавить метку Нет меток, Требуется 1-ая метка записи!

id	ir-20.500.12258-23476
record_format	dspace
spelling	ir-20.500.12258-234762025-02-12T08:32:32Z A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions Abdulkadirov, R. I. Абдулкадиров, Р. И. Lyakhov, P. A. Ляхов, П. А. Dirichlet distributions Natural gradient descent Pattern recognition Machine learning In this paper, we propose a natural gradient descent algorithm with momentum based on Dirichlet distributions to speed up the training of neural networks. This approach takes into account not only the direction of the gradients, but also the convexity of the minimized function, which significantly accelerates the process of searching for the extremes. Calculations of natural gradients based on Dirichlet distributions are presented, with the proposed approach introduced into an error backpropagation scheme. The results of image recognition and time series forecasting during the experiments show that the proposed approach gives higher accuracy and does not require a large number of iterations to minimize loss functions compared to the methods of stochastic gradient descent, adaptive moment estimation and adaptive parameter-wise diagonal quasi-Newton method for nonconvex stochastic optimization. 2023-05-12T12:40:37Z 2023-05-12T12:40:37Z 2023 Статья Abdulkadirov, R.I., Lyakhov, P.A. A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions // Computer Optics. - 2023. - 47 (1), pp. 160-169. - DOI: 10.18287/2412-6179-CO-1147 http://hdl.handle.net/20.500.12258/23476 en Computer Optics application/pdf application/pdf
institution	СКФУ
collection	Репозиторий
language	English
topic	Dirichlet distributions Natural gradient descent Pattern recognition Machine learning
spellingShingle	Dirichlet distributions Natural gradient descent Pattern recognition Machine learning Abdulkadirov, R. I. Абдулкадиров, Р. И. Lyakhov, P. A. Ляхов, П. А. A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions
description	In this paper, we propose a natural gradient descent algorithm with momentum based on Dirichlet distributions to speed up the training of neural networks. This approach takes into account not only the direction of the gradients, but also the convexity of the minimized function, which significantly accelerates the process of searching for the extremes. Calculations of natural gradients based on Dirichlet distributions are presented, with the proposed approach introduced into an error backpropagation scheme. The results of image recognition and time series forecasting during the experiments show that the proposed approach gives higher accuracy and does not require a large number of iterations to minimize loss functions compared to the methods of stochastic gradient descent, adaptive moment estimation and adaptive parameter-wise diagonal quasi-Newton method for nonconvex stochastic optimization.
format	Статья
author	Abdulkadirov, R. I. Абдулкадиров, Р. И. Lyakhov, P. A. Ляхов, П. А.
author_facet	Abdulkadirov, R. I. Абдулкадиров, Р. И. Lyakhov, P. A. Ляхов, П. А.
author_sort	Abdulkadirov, R. I.
title	A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions
title_short	A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions
title_full	A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions
title_fullStr	A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions
title_full_unstemmed	A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions
title_sort	new approach to training neural networks using natural gradient descent with momentum based on dirichlet distributions
publishDate	2023
url	https://dspace.ncfu.ru/handle/20.500.12258/23476
work_keys_str_mv	AT abdulkadirovri anewapproachtotrainingneuralnetworksusingnaturalgradientdescentwithmomentumbasedondirichletdistributions AT abdulkadirovri anewapproachtotrainingneuralnetworksusingnaturalgradientdescentwithmomentumbasedondirichletdistributions AT lyakhovpa anewapproachtotrainingneuralnetworksusingnaturalgradientdescentwithmomentumbasedondirichletdistributions AT lâhovpa anewapproachtotrainingneuralnetworksusingnaturalgradientdescentwithmomentumbasedondirichletdistributions AT abdulkadirovri newapproachtotrainingneuralnetworksusingnaturalgradientdescentwithmomentumbasedondirichletdistributions AT abdulkadirovri newapproachtotrainingneuralnetworksusingnaturalgradientdescentwithmomentumbasedondirichletdistributions AT lyakhovpa newapproachtotrainingneuralnetworksusingnaturalgradientdescentwithmomentumbasedondirichletdistributions AT lâhovpa newapproachtotrainingneuralnetworksusingnaturalgradientdescentwithmomentumbasedondirichletdistributions
_version_	1842245568332562432

A new approach to training neural networks using natural gradient descent with momentum based on Dirichlet distributions

Схожие документы