REVERSAL POINT
  • Home
    • Your Support
    • Privacy Policy
    • Terms of Use
    • Contact
  • Monetary Paradox
    • Monetary Wonderland
    • Shirakawa's Monetary Policy Paradox 1
    • Shirakawa's Monetary Policy Paradox 2
    • Minsky's Non-Neutral Money
    • Monetary Policy Paradox
  • Secular Cycle
    • Blog >
      • Bond Wave >
        • Monetary Regime Cycle and BitCoin
        • Paradigm Shifts in Monetary Regime along Bond Wave
    • Supra-secular rhythm
    • Secular Rhythm of Bond Wave >
      • Bond Wave Mapping 1: Paradigm Transformation in Interntional Monetary Regime
      • Bond Wave Mapping 2: Price & Inflation Cycles
      • Bond Wave Mapping 3: Private Debt Cycle
      • Bond Wave Mapping 4: Fiscal Cycle & Negative Real Yield Cycle
      • Bond Wave Mapping X: Political Cycle
      • Limited Gold Supply was a perennial problem for the Gold Standard: in search for Elastic Money and Scalability
  • Political Philosophy
    • Zeitgeist Zero Hour: Intro
    • Socrates Constitutional Cycle >
      • Socrates' Constitutional Cycle
      • Intrinsic value of Socrates Cycle
      • Contrast between Socrates vs Aristotle
    • Can we preserve democracy? >
      • Terminal Symptom of Democracy in Ancient World, Theoretical Views
      • Paradox of Equality & Aristotelean Paradox Management
      • Aristotelean Preservation of Constitutions
      • Contemporary Liberal Representative Democracy?
    • Old Contents >
      • Socrates' 5 Political Regimes
      • Socrates-Homer Hypothesis
      • Crassus & Trump: Socrates-Homer Hypothesis in Modern Context
  • Others
    • Price Evolution >
      • Oligopoly Price Cycle
      • Deflation >
        • Zero Boundary
        • Anecdote 1874-97
        • Deflationary Innovation
    • Innovation >
      • Introduction AI & ML & DL >
        • Chap1 ML Paradigm
        • Chap2 Generalization of ML
        • Chap3 DL Connectionism
        • Chap4 DL Learning Mechanism: Optimization Paradigm
        • Chap5 DL Revolution
        • Chap6 DL Carbon Footprint
        • Chap7 DL Underspecification
        • Chap8 CNN & Sequence Models
      • Map Risk Clusters of Neighbourhoods in the time of Pandemic
      • Confusing Blockchain >
        • Chapter 1: Linguistic Ambiguity
        • Chapter 2: Limitations in Consensus Protocols
        • Chapter 3-1: Disintermedition Myth-conceptions
        • Chapter 3-2: Autonomous Self-regulating Governance Myth-Conceptions
    • Environmental Distress >
      • Model Risk and Tail Risk of Climate-related Risks
  • Socrates' Constitutional Cycle

Series:
Basic Intuitions of Machine Learning & Deep Learning for beginners


Chapter 5: Deep Learning Revolution
Engineering Achievement on Steroid

Originally published 16 February, 2021
By Michio Suginoo

In Chapter 3, we saw what constitutes the complexity of Deep Learning. As a recap, it is captured by the size of 4 operating components of Deep Learning:
  • # of neurons or nodes
  • # of hyperparameters
  • # of layers
  • # of connections.
And, today, the complexity of Deep Learning Models is exploding.

Now, the numbers you see in the next picture symbolize the complexity and the scale of Deep Learning algorithm. Professor Fei Fei Li of Stanford presents these figures as a typical specification of deep learning models for Image Recognition
Application in 2015. (Li, 2015
Picture
The massive size of these operating components in Millions and Billions implies two things about Deep Learning : 1) its massive computation demand; and 2) “its complex algorithm”.

In addition, as mentioned earlier, Deep Learning is Data hungry.

Overall, Deep Learning demands 3 infrastructures:
  • huge hardware computation power,
  • massive Digital Data Feed, and
  • flexible software ecosystem.

In the 20th century, constraints in these 3 infrastructures hindered the advancement of Deep Learning.
 
Nevertheless, an unexpected revolution took place in an unexpected context.

Thanks to the rising demand for Computer Game, the Game Industry developed effective GPU, Graphic Processing Unit, to perform “real time digital image processing” which demands intensive “parallel computing”.

Today, beyond GPU, and even beyond Cloud, engineers can use high performance computing. This chart illustrates how many connections each hardware can process in Deep Learning development.
Picture
Overall, the hardware revolution accelerated the advancement of Deep Learning in recent years.
 
Here is a chart illustrating the evolution of Compute Usage by Deep Learning models. In the recent decade, the compute usage of Deep Learning has exponentially increased.
Picture
Now, I would like to draw your attention to the unit of Y axis: Peta-flop per second for one day.

Next, please shift your attention to the box on the right. Beyond Mega, Beyond Giga, and Beyond Tera, now we are in the age of Peta, which is 10 to the power of 15: an astronomical scale.
 
Deep Learning in the age of Peta is an engineering achievement on the Steroid of Computational Power.
 
Inevitably, the intensive compute usage means high “energy consumption”, thus, high Carbon Footprint. As my personal wish, energy efficiency is going to be the key for the next Deep Learning Revolution.
 
Next, let’s check Carbon Footprint of Deep Learning.

Donation:
Please feel free to click the bottom below to donate and support
the activities of www.reversalpoint.com

​Copyright © by Michio Suginoo. All rights reserved.

Proudly powered by Weebly