Mean-field message-passing equations in the Hopfield model and its generalizations

Abstract : Motivated by recent progress in using restricted Boltzmann machines as preprocess-ing algorithms for deep neural network, we revisit the mean-field equations (belief-propagation and TAP equations) in the best understood such machine, namely the Hopfield model of neural networks, and we explicit how they can be used as iterative message-passing algorithms, providing a fast method to compute the local polariza-tions of neurons. In the "retrieval phase" where neurons polarize in the direction of one memorized pattern, we point out a major difference between the belief propagation and TAP equations : the set of belief propagation equations depends on the pattern which is retrieved, while one can use a unique set of TAP equations. This makes the latter method much better suited for applications in the learning process of restricted Boltzmann machines. In the case where the patterns memorized in the Hopfield model are not independent, but are correlated through a combinatorial structure, we show that the TAP equations have to be modified. This modification can be seen either as an alteration of the reaction term in TAP equations, or, more interestingly, as the consequence of message passing on a graphical model with several hidden layers, where the number of hidden layers depends on the depth of the correlations in the memorized patterns. This layered structure is actually necessary when one deals with more general restricted Boltzmann machines.
Type de document :
Article dans une revue
Liste complète des métadonnées

Littérature citée [43 références]  Voir  Masquer  Télécharger

https://hal-ens.archives-ouvertes.fr/hal-02083694
Contributeur : Marine Laffont <>
Soumis le : vendredi 29 mars 2019 - 10:53:28
Dernière modification le : mardi 2 avril 2019 - 02:23:33
Document(s) archivé(s) le : dimanche 30 juin 2019 - 13:36:16

Fichier

1608.01558.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

Collections

Citation

Marc Mézard. Mean-field message-passing equations in the Hopfield model and its generalizations. Physical Review E , American Physical Society (APS), 2016, 95 (2), pp.022117. ⟨10.1103/PhysRevE.95.022117⟩. ⟨hal-02083694⟩

Partager

Métriques

Consultations de la notice

24

Téléchargements de fichiers

16