New Penalized Stochastic Gradient Methods for Linearly Constrained Strongly Convex Optimization

For minimizing a strongly convex objective function subject to linear inequality constraints, we consider a penalty approach that allows one to utilize stochastic methods for problems with a moderate to large number of constraints and/or objective function terms. We provide upper bounds on the distance between the solutions to the original constrained problem and the penalty reformulations, guaranteeing the convergence of the proposed approach. We consider a static method that uses a fixed smoothness parameter for the penalty function as well as a dynamic nested method with a novel way for updating the smoothness parameter of the penalty function and the step-size. In both cases, we apply accelerated stochastic gradient methods and study the expected incremental/stochastic gradient iteration complexity to produce a solution within an expected distance of $$\epsilon $$ ϵ to the optimal solution of the original problem. We show that this complexity is proportional to $$m\sqrt{\frac{m}{\mu \epsilon }}$$ m m μ ϵ , where m is the number of constraints and $$\mu $$ μ is the strong convexity parameter of the objective function, which improves upon existing results when m is not too large. We also show how to query an approximate dual solution after stochastically solving the penalty reformulations, leading to results on the convergence of the duality gap. Moreover, the nested structure of the algorithm and upper bounds on the distance to the optimal solutions allows one to safely eliminate constraints that are inactive at an optimal solution throughout the algorithm, which leads to improved complexity results. Finally, we present computational results that demonstrate the effectiveness and robustness of our algorithm.