Exponential Families, Rényi Divergence and the Almost Sure Cauchy Functional Equation

If $$P_1,\ldots , P_n$$ P 1 , … , P n and $$Q_1,\ldots ,Q_n$$ Q 1 , … , Q n are probability measures on $$\mathbb {R}^d$$ R d and $$P_1*\cdots *P_n$$ P 1 ∗ ⋯ ∗ P n and $$Q_1*\cdots *Q_n$$ Q 1 ∗ ⋯ ∗ Q n are their respective convolutions, the Rényi divergence $$D_{\lambda }$$ D λ of order $$\lambda \in (0,1]$$ λ ∈ ( 0 , 1 ] satisfies $$D_{\lambda }(P_1*\cdots *P_n||Q_1*\cdots *Q_n)\le \sum _{i=1}^nD_{\lambda }(P_i||Q_i).$$ D λ ( P 1 ∗ ⋯ ∗ P n | | Q 1 ∗ ⋯ ∗ Q n ) ≤ ∑ i = 1 n D λ ( P i | | Q i ) . When $$P_i$$ P i belongs to the natural exponential family generated by $$Q_i$$ Q i , with the same natural parameter $$\theta $$ θ for any $$i=1,\ldots ,n$$ i = 1 , … , n , the equality sign holds. The present note tackles the inverse problem, namely “does the equality $$D_{\lambda }(P_1*\cdots *P_n||Q_1*\cdots *Q_n)=\sum _{i=1}^nD_{\lambda }(P_i||Q_i)$$ D λ ( P 1 ∗ ⋯ ∗ P n | | Q 1 ∗ ⋯ ∗ Q n ) = ∑ i = 1 n D λ ( P i | | Q i ) imply that $$P_i$$ P i belongs to the natural exponential family generated by $$Q_i$$ Q i for every $$i=1,\ldots ,n$$ i = 1 , … , n ?” The answer is not always positive and depends on the set of solutions of a generalization of the celebrated Cauchy functional equation. We discuss in particular the case $$P_1=\cdots =P_n=P$$ P 1 = ⋯ = P n = P and $$Q_1=\cdots =Q_n=Q$$ Q 1 = ⋯ = Q n = Q , with $$n=2$$ n = 2 and $$n=\infty $$ n = ∞ , the latter meaning that the equality holds for all n. Our analysis is mainly devoted to P and Q concentrated on non-negative integers, and P and Q with densities with respect to the Lebesgue measure. The results cover the Kullback–Leibler divergence (KL), this being the Rényi divergence for $$\lambda = 1$$ λ = 1 . We also show that the only f-divergences such that $$D_{f}(P^{*2}||Q^{*2})=2D_{f}(P||Q)$$ D f ( P ∗ 2 | | Q ∗ 2 ) = 2 D f ( P | | Q ) , for P and Q in the same exponential family, are mixtures of KL divergence and its dual.