Theories of Deep Learning

​​https://stats385.github.io/readings​​

Lecture 1 – Deep Learning Challenge. Is There Theory?

Readings

1. 1. 
      
2. ​​Deep Deep Trouble​​
3. ​​Why 2016 is The Global Tipping Point...​​
4. ​​Are AI and ML Killing Analyticals...​​
5. ​​The Dark Secret at The Heart of AI​​
6. ​​AI Robots Learning Racism...​​
7. ​​FaceApp Forced to Pull ‘Racist' Filters...​​
8. ​​Losing a Whole Generation of Young Men to Video Games​​
   1. 
      

Lecture 2 – Overview of Deep Learning From a Practical Point of View

Readings

1. 1. 
      
2. ​​Emergence of simple cell​​
3. ​​ImageNet Classification with Deep Convolutional Neural Networks (Alexnet)​​
4. ​​Very Deep Convolutional Networks for Large-Scale Image Recognition (VGG)​​
5. ​​Going Deeper with Convolutions (GoogLeNet)​​
6. ​​Deep Residual Learning for Image Recognition (ResNet)​​
7. ​​Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift​​
8. ​​Visualizing and Understanding Convolutional Neural Networks​​
   1. 
      

Blogs

1. 1. 
      
2. ​​An Intuitive Guide to Deep Network Architectures​​
3. ​​Neural Network Architectures​​

Videos

1. 1. 
      
2. ​​Deep Visualization Toolbox​​
   1. 
      

Lecture 3

Readings

1. 1. 
      
2. ​​A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction​​
3. ​​Energy Propagation in Deep Convolutional Neural Networks​​
4. ​​Discrete Deep Feature Extraction: A Theory and New Architectures​​
5. ​​Topology Reduction in Deep Convolutional Feature Extraction Networks​​
   1. 
      

Lecture 4

Readings

1. 1. 
      
2. ​​A Probabilistic Framework for Deep Learning​​
3. ​​Semi-Supervised Learning with the Deep Rendering Mixture Model​​
4. ​​A Probabilistic Theory of Deep Learning​​
   1. 
      

Lecture 5

Readings

1. 1. 
      
2. ​​Why and When Can Deep-but Not Shallow-networks Avoid the Curse of Dimensionality: A Review​​
3. ​​Learning Functions: When is Deep Better Than Shallow​​
   1. 
      

Lecture 6

Readings

1. 1. 
      
2. ​​Convolutional Patch Representations for Image Retrieval: an Unsupervised Approach​​
3. ​​Convolutional Kernel Networks​​
4. ​​Kernel Descriptors for Visual Recognition​​
5. ​​End-to-End Kernel Learning with Supervised Convolutional Kernel Networks​​
6. ​​Learning with Kernels​​
7. ​​Kernel Based Methods for Hypothesis Testing​​
   1. 
      

Lecture 7

Readings

1. 1. 
      
2. ​​Geometry of Neural Network Loss Surfaces via Random Matrix Theory​​
3. ​​Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice​​
4. ​​Nonlinear random matrix theory for deep learning​​
   1. 
      

Lecture 8

Readings

1. 1. 
      
2. ​​Deep Learning without Poor Local Minima​​
3. ​​Topology and Geometry of Half-Rectified Network Optimization​​
4. ​​Convexified Convolutional Neural Networks​​
5. ​​Implicit Regularization in Matrix Factorization​​
   1. 
      

Lecture 9

Readings

1. 1. 
      
2. ​​Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position​​
3. ​​Perception as an inference problem​​
4. ​​A Neurobiological Model of Visual Attention and Invariant Pattern Recognition Based on Dynamic Routing of Information​​
   1. 
      

Lecture 10

Readings

1. 1. 
      
2. ​​Working Locally Thinking Globally: Theoretical Guarantees for Convolutional Sparse Coding​​
3. ​​Convolutional Neural Networks Analyzed via Convolutional Sparse Coding​​
4. ​​Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning​​
5. ​​Convolutional Dictionary Learning via Local Processing​​
   1. 
      

To be discussed and extra

• • 
    
• ​​Emergence of simple cell​​ by Olshausen and Field
• ​​Auto-Encoding Variational Bayes​​ by Kingma and Welling
• ​​Generative Adversarial Networks​​ by Goodfellow et al.
• ​​Understanding Deep Learning Requires Rethinking Generalization​​ by Zhang et al.
• ​​Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy?​​ by Giryes et al.
• ​​Robust Large Margin Deep Neural Networks​​ by Sokolic et al.
• ​​Tradeoffs between Convergence Speed and Reconstruction Accuracy in Inverse Problems​​ by Giryes et al.
• ​​Understanding Trainable Sparse Coding via Matrix Factorization​​ by Moreau and Bruna
• ​​Why are Deep Nets Reversible: A Simple Theory, With Implications for Training​​ by Arora et al.
• ​​Stable Recovery of the Factors From a Deep Matrix Product and Application to Convolutional Network​​ by Malgouyres and Landsberg
• ​​Optimal Approximation with Sparse Deep Neural Networks​​ by Bolcskei et al.
• ​​Convolutional Rectifier Networks as Generalized Tensor Decompositions​​ by Cohen and Shashua
• ​​Emergence of Invariance and Disentanglement in Deep Representations​​ by Achille and Soatto
• ​​Deep Learning and the Information Bottleneck Principle​​ by Tishby and Zaslavsky