Neural Networks

In biology, a neural network is a circuit composed of a group of chemically connected or functionally associated neurons. The connections of neurons are modeled by means of weights. A positive weight reflects an excitatory connection, while negative values mean inhibitory connections. All inputs are modified by weights and summed up. This aggregation corresponds to taking linear combinations. Finally, an activation function controls the amplitude of the output. Although each of them being relatively simple, the neurons can build networks with surprisingly high processing power. This gave rise since 1970s to the development of neural networks for solving artificial intelligence problems. Remarkable progress in this direction have been achieved particularly in the last decade. As example, we just mention the neural network Leela Chess Zero that managed to win in May 2019 the Top Chess Engine Championship, defeating the conventional chess engine Stockfish in the final. In this chapter, we get to know how the neuron’s functioning can be mathematically modeled. This is done on example of classification neurons in terms of the generalized linear regression. The generalization refers here to the use of activation functions. First, we focus on the sigmoid activation function and the corresponding logistic regression. The training of weights will be based on the minimization of the average cross-entropy arising from the maximum likelihood estimation. We minimize the average cross-entropy by means of the stochastic gradient descent. Second, the threshold activation function is considered. The corresponding neuron is referred to as perceptron. For the latter, the Rosenblatt learning is shown to provide a correct linear classifier in finitely many iteration steps. After mentioning the XOR problem, which cannot be handled by perceptrons with one layer, we introduce multilayer perceptrons. We point out their importance by stating the universal approximation theorem.