Thermodynamics of Machine Learning
https://arxiv.org/pdf/1807.04162v1.pdf
1. First Law of Learning
2. Maxwell Relation
3. Zeroth Law of Learning
We have formalized representation learning as the process
of minimizing the distortion introduced when we project
the real world (World P) onto the world we desire (World
Q). The projection is naturally described by a set of four
functionals which variationally bound relevant mutual informations
in the real world. Relations between the functionals
describe an optimal three-dimensional surface in a four
dimensional space of optimal states. A single learning objective
targeting points on this optimal surface can express a
wide array of existing learning objectives spanning from unsupervised
learning to supervised learning and everywhere
in between. The geometry of the optimal frontier suggests a
wide array of identities involving the functionals and their
partial derivatives. This offers a direct analogy to thermodynamics
independent of any physical content. By analogy
to thermodynamics, we can begin to develop new quantitative
measures and relationships amongst properties of our
models that we believe will offer a new class of theoretical
understanding of learning behavior
https://arxiv.org/pdf/1807.04162v1.pdf
1. First Law of Learning
2. Maxwell Relation
3. Zeroth Law of Learning
We have formalized representation learning as the process
of minimizing the distortion introduced when we project
the real world (World P) onto the world we desire (World
Q). The projection is naturally described by a set of four
functionals which variationally bound relevant mutual informations
in the real world. Relations between the functionals
describe an optimal three-dimensional surface in a four
dimensional space of optimal states. A single learning objective
targeting points on this optimal surface can express a
wide array of existing learning objectives spanning from unsupervised
learning to supervised learning and everywhere
in between. The geometry of the optimal frontier suggests a
wide array of identities involving the functionals and their
partial derivatives. This offers a direct analogy to thermodynamics
independent of any physical content. By analogy
to thermodynamics, we can begin to develop new quantitative
measures and relationships amongst properties of our
models that we believe will offer a new class of theoretical
understanding of learning behavior