The question that changed how I study
A neural network does not understand words, pictures or logs the way we do. The only thing it can actually act on is a derivative. Once I saw that, calculus stopped being a school subject for me and became the engine room of AI.
I am learning the math behind deep learning from scratch, one concept at a time, and writing about it in public. A few weeks into calculus, I was drilling the power rule and the product rule like everyone does. Then one question stopped me: why does a machine that recognises faces care about the slope of a curve?
This post is my answer, written for anyone who knows a little algebra and wants to see why derivatives sit at the heart of every neural network.
What a derivative really tells you
A derivative answers one simple question: if I nudge the input a tiny bit, how much does the output move, and in which direction?
Think of walking on a hill in thick fog. You cannot see the top or the bottom. But you can feel the ground under your feet. If it tilts down to your left, you know which way to step to go lower. That feeling of the ground is the slope, and the slope is the derivative.
Here is a small example. Take this function:
f(x) = x^2 \qquad f'(x) = 2x
At x = 3, the derivative says the slope is 2 × 3 = 6. Let us check that with real numbers instead of trusting the rule. Nudge x from 3 to 3.01. The output goes from 9 to 9.0601. The output moved 0.0601 while the input moved 0.01, which is about 6 times as much. The rule and the reality agree.
So a derivative gives you two pieces of information at once. The sign tells you the direction, and the size tells you how sensitive the output is to that input. Keep those two ideas in your pocket, because a neural network uses nothing else to learn.
How a neural network learns
Strip away the hype and a neural network is just a very large function with a lot of knobs. Those knobs are called weights. Data goes in, the weights shape it, and a prediction comes out.
At the start, the weights are almost random, so the predictions are bad. To measure how bad, we use a loss function. It takes the prediction and the correct answer and returns one number. A big loss means the model is very wrong. A small loss means it is getting close.
So learning means one thing: turn the knobs until the loss gets small. The problem is that a modern network can have millions or even billions of knobs. You cannot turn them one by one and check what happens. You need something that tells you, for every knob at once, which way to turn it and how hard.
That something is the derivative of the loss with respect to each weight. Collect all of those derivatives together and you get what is called the gradient.
Gradient descent with one tiny knob
Let us build the smallest possible model and watch it learn. It has a single weight w, and it predicts by multiplying the input by w. We give it one example: input 2, correct answer 10. The perfect weight is obviously 5, but the model does not know that.
The loss is the squared error between the prediction 2w and the answer 10:
L(w) = (2w - 10)^2 = 4w^2 - 40w + 100
Using nothing more than the power rule, the derivative is:
\frac{dL}{dw} = 8w - 40Now the learning rule. We move the weight a small step against the slope. The step size is called the learning rate, written as the Greek letter eta, and here we set it to 0.05:
w_{\text{new}} = w - \eta \cdot \frac{dL}{dw}Why against the slope? If the derivative is negative, increasing w lowers the loss, so we should increase w. Subtracting a negative number does exactly that. Here is what happens when we start at w = 1:
A real network does the same thing, just with millions of weights updated at the same time.
Why it has to be derivatives
Imagine training without derivatives. For every single weight, you would nudge it, run the whole network again, and see if the loss went up or down. With a million weights, that is a million extra runs for one tiny step of learning. Training a modern model that way would never finish.
Derivatives solve this because of a rule called the chain rule. A neural network is a long chain of small functions, one layer feeding into the next. The chain rule lets us multiply the slopes of those small pieces together to get the slope of the whole thing.
The method that applies the chain rule backwards through the layers, from the loss to the very first weight, is called backpropagation. In one backward pass it calculates the derivative of the loss for every weight in the network. One forward pass to predict, one backward pass to learn. That is the heartbeat of deep learning.
To be fair, there are training methods that skip derivatives, such as evolutionary search. They have their uses, but almost every model you have heard of today was trained with gradients. That is why I say derivatives are the language these networks speak.
This also explains a rule that confused me at first: every piece of a network needs a slope we can calculate. If a layer has no usable derivative, the learning signal cannot pass through it. That idea leads straight to ReLU, the most popular activation function, which I will break down in the next post.
What this changed for me
Before this, I treated derivative rules as tricks to memorise for a test. Now every rule I learn feels like a new tool for one job: telling a model which way to turn its knobs. The power rule, the product rule and the chain rule are not separate chapters anymore. They are parts of the same machine.
If you are learning math for AI too, try this. Every time you meet a new rule, ask yourself what a neural network would use it for. The math gets easier to remember when it has a reason to exist.
I am sharing this journey one concept at a time as I learn it. If something here helped, or if you spotted a mistake, I would genuinely like to hear about it. Next up: ReLU, and the mistake that finally taught me the difference between the height of a function and its slope.