I’ll start by outlining various types of pitfalls in neural net optimization, as well as the most widely used optimization algorithms (SGD, RMSprop, Adam, etc.). Then I’ll consider the use of second-order information to speed up training using the natural gradient algorithm. I’ll look at how two key hyperparameters — learning rates and batch sizes — impact performance. Then I’ll turn to ways of improving generalization; this will include standard techniques like dropout and batch normalization, as well as more recent studies of the interaction between optimization and generalization.