Optimization

Optimization is the math of finding the best choice: the lowest cost, the smallest error, the highest score, often subject to limits. Most methods work like walking downhill in fog: look at the slope where you stand (the gradient) and take a step down, repeating until the ground is flat. The catch is that you might settle into a small dip that isn’t the lowest point overall. Convex problems are the friendly case, shaped like a single bowl, so any bottom you reach is the true bottom. Training a neural network is a huge non-convex optimization, while things like linear regression are convex and reliably solvable.