Optimizers

An optimizer is the algorithm that adjusts a model’s parameters during training to make its error smaller. Picture the error as a hilly landscape where every position is one setting of the parameters: the gradient tells you which direction is downhill from where you stand, and plain gradient descent takes a small step that way, over and over. Stochastic gradient descent estimates that direction from a small random batch of data, which is noisy but fast. Momentum keeps some of the previous step’s velocity, so it rolls through small bumps and doesn’t zigzag. Adam also scales each parameter’s step by how large its recent gradients have been, which is why it often works well with little tuning.