当前位置:首页>>学术讲座
Spectrum concentration in deep residual learning: a free probability approach
 讲座名称 Spectrum concentration in deep residual learning: a free probability approach 讲座时间 2018-12-03 10:30:00 讲座地点 西电北校区主楼III区-401 讲座人 邱才明(Prof. Qiu Caiming) 讲座人介绍 Prof. Qiu Caiming, IEEE Fellow, Specially-appointed Professor of National “Thousand Talents Program”, Distinguished Professor of “Thousand Talents Program” of Shanghai, Chair Professor of Shanghai Jiaotong University, Director of Big Data Engineering Research Center of Shanghai Jiaotong University, Tenured professor of the Tennessee Institute of Technology, USA. Professor Qiu Caiming received a bachelor's degree in science from Xidian University in 1987, a master's degree from the University of Electronic Science and Technology in 1990, and a doctor's degree from the Institute of Technology, New York U 讲座内容 We revisit the weight initialization of deep residual networks  (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with the limiting spectral distribution of \textit{non-Hermitian} random matrices, rather than their conventional \textit{Hermitian} counterparts in the literature.  As a consequence, this new tool enables us to evaluate the singular value spectrum of the input-output Jacobian of a fully-connected deep ResNet in both linear and nonlinear cases. With the powerful tool of free probability, we conduct an asymptotic analysis of the (limiting) spectrum on the single-layer case, and then extend this analysis to the multi-layer case of an arbitrary number of layers. The asymptotic analysis illustrates the necessity and university of rescaling the classical random initialization by the number of residual units $L$, so that the mean squared singular value of the associated Jacobian remains of order $O(1)$, when compared with the large width and depth of the network. We empirically demonstrate that the proposed initialization scheme learns at a speed of orders of magnitudes faster than the classical ones, and thus attests a strong practical relevance of this investigation.

Copyright © 2011-2019 西安电子科技大学