Combinatory Adjoints and Differentiation at MSFP 2022
Our paper Combinatory Adjoints and Differentiation, with Fritz Henglein, Robin Kaarsgaard, Mikkel Kragh Mathiesen, and Robert Schenck, was accepted for presentation at MSFP 2022 in Munich on April 2.
Automatic differentiation is central to tasks such as training neural networks. We describe derivatives as compositions of linear functions, avoiding the enormous matrices that a direct representation could require. A symbolic calculation of adjoints then gives the behaviour of reverse-mode differentiation. Because the representation retains the original program’s parallel operations, it also creates opportunities for optimisation and efficient parallel execution.
Conference programme and abstract.
The Caddie project provides a Standard ML implementation exploring these ideas, with examples of forward-mode and reverse-mode differentiation.