Series:
Basic Intuitions of Machine Learning & Deep Learning for beginners

Chapter 4: Deep Learning’s Learning Mechanism
Optimization Paradigm driven by 3 Step Iteration Cycle

Originally published 16 February, 2021
By Michio Suginoo
In this chapter, we will have a quick look at the learning mechanism of Deep Learning.

How does it learn?

You can click on the video below.
Cut a long story short, Deep Learning is an optimization paradigm that learns through repetitions of “3 step Iteration Cycle”: Try and Error and Refine.

The next figure illustrates how this 3 step iteration cycle operates in the structure of “Feedforward Neural Networks”, a prototype of Deep Learning Algorithm, which you saw earlier.
The stack of layers in Deep Learning architecture is comprised of 3 functional sub-divisions: the input layer, the hidden layers, and the output layer.
 
Right after taking the input in the input layer, the model enters into the iteration cycle.
First, “prediction process” takes place in the hidden layers along “the blue arrow on the top”. The system tries the current values of Hyperparameters to generate a “tentative prediction”.

Then, second, “the measurement of the error” takes place in the yellow box. The output layer generates a tentative prediction based on the current values of hyperparameters. Thereafter, the network engages in the process of ‘Error’, using a Cost Function to measure the error between the current tentative prediction and the given actual labels.
  
Third, the refinement of hyperparameters takes place back in the hidden layers along the green arrow at the bottom. This process goes backward the way it came during the process of ‘Try’. So it is called Backward propagation.

And the system repeats this 3 step iteration cycle over and over, until it reduces the error within an acceptable range.

Overall, the hidden layers engage in two learning stages: to try the current values of hyperparameters in the process of ‘Try’; and to update the values of hyperparameters in the process of ‘Refine’.

Updating Hyperparameters

Next, let’ take a look at how the system refines the hyperparameter values.
The 3D chart here is a simplified illustration of the Cost Function.
Remember the ultimate objective of the optimization is to minimize the error, thus, the value of the cost function. In order to minimize the Error, we want to arrive at the bottom of the Cost Function Landscape in the illustration above.

Click the  animation video below. This illustrates the process of Deep Learning Optimization.
It’s like skiing along the downhill. This process is called Gradient Descent. Easier said than done. This chart is a little bit of over-simplification. The reality might be more complex. In Chapter 7, we will face the reality.

In the next chapter, we will have a quick look at the background behind Deep Learning Revolution in recent years.
Donation:
Please feel free to click the bottom below to donate and support
the activities of www.reversalpoint.com
​Copyright © by Michio Suginoo. All rights reserved.