Yang Feng (Chair)

Abstract

Truncation of the singular value decomposition is a true scientific workhorse. But where to Truncate? For 55 years the answer, for many scientists, has been to eyeball the scree plot, an approach which still generates hundreds of papers per year. I will describe ScreeNOT, a mathematically solid alternative deriving from the many advances in Random Matrix Theory over those 55 years. Assuming a model of low-rank signal plus possibly correlated noise, and adopting an asymptotic viewpoint with number of rows proportional to the number of columns, we show that ScreeNOT has a surprising oracle property.

It typically achieves exactly, in large finite samples, the lowest possible MSE for matrix recovery, on each given problem instance – i.e. the specific threshold it selects gives exactly the smallest achievable MSE loss among all possible threshold choices for that noisy dataset and that unknown underlying true low rank model. The method is computationally efficient and robust against perturbations of the underlying covariance structure. (Joint work with Matan Gavish and Elad Romanov, Hebrew University)