In-depth examination of the mathematical foundations of modern deep learning, surveying current research in the area. Topics include stochastic and distributed optimization, regularization, initialization, network architecture design, and loss function design. Concepts are developed in the context of various application areas including supervised learning, generative modeling, and reinforcement learning.