3D Surface Plot

The idea

A function of two variables is a height map: each point (x, y) on the floor has a height z = f(x, y) above it, and the graph is the surface those heights make. Everything multivariable calculus says about f is a statement about that terrain. The gradient ∇f at a point is the direction of steepest ascent and how steep it is. A critical point is where the gradient is zero — the surface is locally flat — and it can be a hilltop, a valley bottom, or a saddle, where the ground rises in one direction and falls in the other.

The ball turns the gradient into motion. At every step it rolls in the direction of −∇f, straight downhill, which is gradient descent in the literal sense. On a bowl it finds the one minimum from anywhere. On a saddle it pauses near the flat point and then rolls off, because a saddle satisfies ∇f = 0 without being a minimum of anything. On a surface with several basins, the basin it ends in is decided entirely by where it was dropped.

That last fact is the whole difficulty of optimisation in more than one variable. Descent is local. It goes down, and it stops when there is no down, whether or not somewhere lower exists over the next ridge.

What to look for

• On 'bowl', drop the ball anywhere and it converges to the same point. The traced path curves toward the centre because the gradient always points at the minimum of a paraboloid.
• On 'saddle', the ball approaches the flat point along the rising axis, slows almost to a stop, then accelerates away along the falling axis. The critical point is real; the optimum is not.
• On 'wells', the two basins have a ridge between them. Balls dropped on either side of the ridge end in different minima, and the ridge itself is a line of unstable equilibria.
• On 'ripple', the many shallow minima trap the ball wherever it happens to be. This is the landscape that makes local search hard, and it is why global methods add noise or restarts.
• Use 'wireframe' to see the grid of the surface itself and read the contour spacing: close lines mean a steep gradient and a fast ball.

Getting it right

• ∇f = 0 is necessary for a minimum but not sufficient. The saddle is the standard counterexample and it is here for that reason.
• The gradient is a direction on the floor, not on the surface. The ball moves in (x, y); its height follows from f.
• "The ball rolls downhill" is exact for the direction and approximate for the speed. Gradient descent takes steps proportional to the gradient, which is not Newtonian rolling with momentum; the trace shows the descent path, not a physics simulation.
• The 'monkey' saddle is degenerate: three valleys meet at a single flat point where the second derivatives all vanish, so the usual second-derivative test cannot classify it. The terrain can.

Turn the knobs

• scenario chooses the surface: 'bowl', 'saddle', 'monkey', 'wells', 'ripple'. Run them in that order; each removes an assumption the previous one let you make.
• scale resizes the landscape without changing its shape or the descent path.
• The camera presets ('3/4', 'top') are the best experiment on this page. From the top, the traced path is a plain 2D descent plot; from three-quarters it is obviously a ball on a hill. Both are the same function.