Stochastic Multi-level Nested Composition Optimization
Over the past few years, nested composition optimization problems whose objective functions is a composition of exceptions, have received much attention due to their emerging applications including risk-averse optimization and training large-scale graph neural networks. The main difficulty in solving this class of problems is the absence of an unbiased estimator for the gradient of the objective function with a bounded second moment (independent of the problem dimension). In this talk, I will present a few settings under which we can provide efficient algorithms to find an approximate stationary point of the problem. We will further show that the sample complexities of these methods match that of the single level stochastic optimization problems and hence are optimal in terms of dependence on the target accuracy and only polynomially depend on the number of levels in the objective function. Moreover, these methods do not require any mini-batch of samples to be convergent and thus can be easily applied to the online stochastic setting.