Phase transition analysis for shallow neural networks with arbitrary activation functions

In this work we extend the statistical physics based analysis of so-called soft committee machines (SCM), two-layered feedforward neural networks with differentiable hidden unit activation, in student–teacher model scenarios. In particular, we study phase transitions with the training set size as observed in off-line learning for SCM with arbitrary activation functions. The analysis is facilitated by expanding the relevant quantities in a basis of Hermite polynomials and truncating the expansion to a relatively small number of terms. We derive a sufficient condition for the presence of a second order phase transition and find an analytical expression for the point in training where an unspecialized phase becomes unstable. In addition we derive an expression for the asymptotic learning behavior for general activation functions. Moreover, we demonstrate that the approach facilitates the treatment of mismatched scenarios, where student and teacher network have different activation functions. This enables the investigation of more realistic learning scenarios within the statistical physics framework.