“Gradient descent is a core optimization algorithm in artificial intelligence (AI) and machine learning used to find the optimal parameters for a model by minimizing a cost (or loss) function.” – Gradient descent
Gradient descent is a first-order iterative optimisation algorithm used to minimise a differentiable cost or loss function by adjusting model parameters in the direction of the steepest descent.4,1 It is fundamental in artificial intelligence (AI) and machine learning for training models such as linear regression, neural networks, and logistic regression by finding optimal parameters that reduce prediction errors.2,3
How Gradient Descent Works
The algorithm starts from an initial set of parameters and iteratively updates them using the formula:
?_{new} = ?_{old} - ? ?J(?)where ? represents the parameters, ? is the learning rate (step size), and ?J(?) is the gradient of the cost function J.4,6 The negative gradient points towards the direction of fastest decrease, analogous to descending a valley by following the steepest downhill path.1,2
Key Components
- Learning Rate (?): Controls step size. Too small leads to slow convergence; too large may overshoot the minimum.1,2
- Cost Function: Measures model error, e.g., mean squared error (MSE) for regression.3
- Gradient: Partial derivatives indicating how to adjust each parameter.4
Types of Gradient Descent
| Type | Description | Advantages |
|---|---|---|
| Batch Gradient Descent | Uses entire dataset per update. | Stable convergence.5 |
| Stochastic Gradient Descent (SGD) | Updates per single example. | Faster for large data, escapes local minima.3 |
| Mini-Batch Gradient Descent | Uses small batches. | Balances speed and stability; most common in practice.5 |
Challenges and Solutions
- Local Minima: May trap in suboptimal points; SGD helps escape.2
- Slow Convergence: Addressed by momentum or adaptive rates like Adam.2
- Learning Rate Sensitivity: Techniques include scheduling or RMSprop.2
Key Theorist: Augustin-Louis Cauchy
Augustin-Louis Cauchy (1789-1857) is the pioneering mathematician behind the gradient descent method, formalising it in 1847 as a technique for minimising functions via iterative steps proportional to the anti-gradient.4 His work laid the foundation for modern optimisation in AI.
Biography
Born in Paris during the French Revolution, Cauchy showed prodigious talent, entering École Centrale du Panthéon in 1802 and École Polytechnique in 1805. He contributed profoundly to analysis, introducing rigorous definitions of limits, convergence, and complex functions. Despite political exiles under Napoleon and later regimes, he produced over 800 papers, influencing fields from elasticity to optics. Cauchy served as a professor at the École Polytechnique and Sorbonne, though his ultramontane Catholic views led to professional conflicts.4
Relationship to Gradient Descent
In his 1847 memoir “Méthode générale pour la résolution des systèmes d’équations simultanées,” Cauchy described an iterative process equivalent to gradient descent: updating variables by subtracting a positive multiple of partial derivatives. This predates widespread use in machine learning by over a century, where it powers backpropagation in neural networks. Unlike later variants, Cauchy’s original focused on continuous optimisation without batching, but its core principle remains unchanged.4
Legacy
Cauchy’s method enabled scalable training of deep learning models, transforming AI from theoretical to practical. Modern enhancements like Adam build directly on his foundational algorithm.2,4
References
1. https://www.geeksforgeeks.org/data-science/what-is-gradient-descent/
2. https://www.datacamp.com/tutorial/tutorial-gradient-descent
3. https://www.geeksforgeeks.org/machine-learning/gradient-descent-algorithm-and-its-variants/
4. https://en.wikipedia.org/wiki/Gradient_descent
5. https://builtin.com/data-science/gradient-descent
7. https://www.ibm.com/think/topics/gradient-descent
8. https://www.youtube.com/watch?v=i62czvwDlsw
