Some Insights from Doubly Infinite Neural Networks
When and Where
Speakers
Description
The theoretical analysis of infinite-width neural networks has led to many interesting practical results (choice of initialization schemes, choice of Bayesian priors etc.). However, the traditional infinite-width framework focuses on fixed depth networks and omits the large depth behavior of these models. In this talk, I will cover different topics on the infinite-width-then-infinite-depth networks and show how we can leverage some theoretical results to obtain principled algorithms. The following papers will be covered in this talk:
- On the Impact of the activation function on deep neural networks training. ICML 2019. (Joint work with A. Doucet and J. Rousseau)
- Stable ResNet. AISTATS 2021. (Joint work with E. Clerico, B. He, G. Deligiannidis, A. Doucet, and J. Rousseau)
About Soufiane Hayou
Soufiane Hayou obtained his PhD in statistics in 2021 from Oxford where he was advised by Arnaud Doucet and Judith Rousseau. He graduated from Ecole Polytechnique in Paris before joining Oxford.
During his PhD, he worked mainly on the theory of randomly initialized infinite-width neural networks on topics including the impact of the hyperparameters (variance of the weights, activation function) and the architecture (fully-connected, convolutional, skip connections) on how the 'geometric' information propagates inside the network. He is currently a visiting assistant professor of mathematics at the National University of Singapore.