Sie befinden Sich nicht im Netzwerk der Universität Paderborn. Der Zugriff auf elektronische Ressourcen ist gegebenenfalls nur via VPN oder Shibboleth (DFN-AAI) möglich. mehr Informationen...
Subaging in underparametrized deep neural networks
Ist Teil von
Machine learning: science and technology, 2022-09, Vol.3 (3), p.35013
Ort / Verlag
Bristol: IOP Publishing
Erscheinungsjahr
2022
Quelle
EZB Electronic Journals Library
Beschreibungen/Notizen
Abstract
We consider a simple classification problem to show that the dynamics of finite–width Deep Neural Networks in the underparametrized regime gives rise to effects similar to those associated with glassy systems, namely a slow evolution of the loss function and aging. Remarkably, the aging is sublinear in the waiting time (subaging) and the power–law exponent characterizing it is robust to different architectures under the constraint of a constant total number of parameters. Our results are maintained in the more complex scenario of the MNIST database. We find that for this database there is a unique exponent ruling the subaging behavior in the whole phase.