Optimize to generalize in Gaussian processes: An alternative objective based on the Rényi divergence

We introduce an alternative closed-form objective function α-ELBO for improved parameter estimation in the Gaussian process (GP) based on the Rényi α-divergence. We use a decreasing temperature parameter α to iteratively deform the objective function during optimization. Ultimately, our objective function converges to the exact log-marginal likelihood function of GP. At early optimization stages, α-ELBO can be viewed as a regularizer that smoothes some unwanted critical points. At late stages, α-ELBO recovers the exact log-marginal likelihood function that guides the optimizer to solutions that best explain the observed data. Theoretically, we derive an upper bound of the Rényi divergence under the proposed objective and derive convergence rates for a class of smooth and non-smooth kernels. Case studies on a wide range of real-life engineering applications demonstrate that our proposed objective is a practical alternative that offers improved prediction performance over several state-of-the-art inference techniques.