Series:
Basic Intuitions of Machine Learning & Deep Learning for beginners

Chapter 5: Deep Learning Revolution
Engineering Achievement on Steroid

Originally published 16 February, 2021
By Michio Suginoo
In Chapter 3, we saw what constitutes the complexity of Deep Learning. As a recap, it is captured by the size of 4 operating components of Deep Learning:
  • # of neurons or nodes
  • # of hyperparameters
  • # of layers
  • # of connections.
And, today, the complexity of Deep Learning Models is exploding.

Now, the numbers you see in the next picture symbolize the complexity and the scale of Deep Learning algorithm. Professor Fei Fei Li of Stanford presents these figures as a typical specification of deep learning models for Image Recognition
Application in 2015. (Li, 2015
The massive size of these operating components in Millions and Billions implies two things about Deep Learning : 1) its massive computation demand; and 2) “its complex algorithm”.

In addition, as mentioned earlier, Deep Learning is Data hungry.

Overall, Deep Learning demands 3 infrastructures:
  • huge hardware computation power,
  • massive Digital Data Feed, and
  • flexible software ecosystem.

In the 20th century, constraints in these 3 infrastructures hindered the advancement of Deep Learning.
 
Nevertheless, an unexpected revolution took place in an unexpected context.

Thanks to the rising demand for Computer Game, the Game Industry developed effective GPU, Graphic Processing Unit, to perform “real time digital image processing” which demands intensive “parallel computing”.

Today, beyond GPU, and even beyond Cloud, engineers can use high performance computing. This chart illustrates how many connections each hardware can process in Deep Learning development.
Overall, the hardware revolution accelerated the advancement of Deep Learning in recent years.
 
Here is a chart illustrating the evolution of Compute Usage by Deep Learning models. In the recent decade, the compute usage of Deep Learning has exponentially increased.
Now, I would like to draw your attention to the unit of Y axis: Peta-flop per second for one day.

Next, please shift your attention to the box on the right. Beyond Mega, Beyond Giga, and Beyond Tera, now we are in the age of Peta, which is 10 to the power of 15: an astronomical scale.
 
Deep Learning in the age of Peta is an engineering achievement on the Steroid of Computational Power.
 
Inevitably, the intensive compute usage means high “energy consumption”, thus, high Carbon Footprint. As my personal wish, energy efficiency is going to be the key for the next Deep Learning Revolution.
 
Next, let’s check Carbon Footprint of Deep Learning.
Donation:
Please feel free to click the bottom below to donate and support
the activities of www.reversalpoint.com
​Copyright © by Michio Suginoo. All rights reserved.