Resources

Artificial neural networks are graphs that are rought similar to the network of neuron cells in the human brain. They are called "artificial" to distinguish them from biological neural networks.

The vertices in the neural network graph are called nodes, units, or neurons.

The connections in the neural network graph are called edges or weights.

The essential idea for a neural net was shared by Walter Pitts and Warren McCulloch in 1943. ANNs became much more practical with the development of backpropagation in the 1970s.

Gradient descent is a common technique for finding minimal values of a function. If you have a complicated math equation that models average housing price over the days of a year, you can use gradient descent to find the times of the year when prices dip. You begin by choosing a random day of the year and calculating the slope (the gradient) of the function on that day. If the slope is positive, the prices are increasing at that time of year. If the slope is negative, they are increasing. You take a small step in the direction of decreasing slope, either forward or backward, and then repeat the process for this new date. You continue until you get stuck at a point where the slope changes from negative to positive, which can only happen at the bottom of a dip in the curve. At this point, the price is at a minimum value relative to the surrounding dates (a local minimum) but it may not be the absolute lowest price dip throughout the year. Finding the global minimum (absolute lowest value) of a function is difficult; this is why learning/optimization functions often get stuck in local minima.

Taking steps in the direction of the positive gradient (instead of the negative gradient) will lead you uphill to a local maximum of the function. This is called gradient ascent.

In machine learning, gradient descent is used to find a minimum value of the cost function, that is, the point where the error between the predicted values and trained values is minimized.

An excellent overview of different gradient descent techniques for ML is provided by Sebastian Ruder.