It is becoming increasingly clear that implicit regularization afforded by the optimization algorithms play a central role in machine learning, and especially so when using large, deep, neural networks. In this talk I will argue that, for the most part, the optimization geometry induces the inductive bias that enables learning with large deep networks, and will discuss work on understanding and precisely characterizing this inductive bias.