Generalized Optimization Algorithms for Complex Models

Linking the statistic and the machine learning literature, we provide new general results on the convergence of stochastic approximation schemes and inexact Newton methods. Building on these results, we put forward a new optimization scheme that we call generalized inexact Newton method (GINM). We extensively discuss the theoretical and the computational aspects of the GINM. The results apply to both deterministic and stochastic approximation schemes, and are particular effective in the case in which the objective function to be optimized is highly irregular and/or the stochastic equicontinuity hypothesis is violated. Examples are common in dynamic discrete choice models and complex simulation models characterized by nonlinearities and high levels of heterogeneity. The theory is supported by extensive Monte Carlo experiments.