加入收藏夹
联系我们
关于本站
个人主页
西电导航
西安电子科技大学
    当前位置:首页>>学术讲座
Spectrum concentration in deep residual learning: a free probability approach
时间:2018-12-02 12:20    点击:   所属单位:人工智能学院
讲座名称 Spectrum concentration in deep residual learning: a free probability approach
讲座时间 2018-12-03 10:30:00
讲座地点 西电北校区主楼III区-401
讲座人 邱才明(Prof. Qiu Caiming)
讲座人介绍 Prof. Qiu Caiming, IEEE Fellow, Specially-appointed Professor of National “Thousand Talents Program”, Distinguished Professor of “Thousand Talents Program” of Shanghai, Chair Professor of Shanghai Jiaotong University, Director of Big Data Engineering Research Center of Shanghai Jiaotong University, Tenured professor of the Tennessee Institute of Technology, USA.
Professor Qiu Caiming received a bachelor's degree in science from Xidian University in 1987, a master's degree from the University of Electronic Science and Technology in 1990, and a doctor's degree from the Institute of Technology, New York U
讲座内容 We revisit the weight initialization of deep residual networks  (ResNets) by introducing a novel analytical tool in free probability to the community of deep learning. This tool deals with the limiting spectral distribution of \textit{non-Hermitian} random matrices, rather than their conventional \textit{Hermitian} counterparts in the literature.  As a consequence, this new tool enables us to evaluate the singular value spectrum of the input-output Jacobian of a fully-connected deep ResNet in both linear and nonlinear cases. With the powerful tool of free probability, we conduct an asymptotic analysis of the (limiting) spectrum on the single-layer case, and then extend this analysis to the multi-layer case of an arbitrary number of layers. The asymptotic analysis illustrates the necessity and university of rescaling the classical random initialization by the number of residual units $L$, so that the mean squared singular value of the associated Jacobian remains of order $O(1)$, when compared with the large width and depth of the network. We empirically demonstrate that the proposed initialization scheme learns at a speed of orders of magnitudes faster than the classical ones, and thus attests a strong practical relevance of this investigation.
转载请注明出处:西安电子科技大学学术信息网
如果您有学术信息或学术动态,欢迎投稿。我们将在第一时间确认并收录,投稿邮箱: meeting@xidian.edu.cn
Copyright © 2011-2019 西安电子科技大学 
开发维护:电子工程学院网络信息中心  管理员:meeting@xidian.edu.cn 站长统计: