INEXACT INTERIOR-POINT METHODS FOR LARGE SCALE LINEAR AND CONVEX QUADRATIC SEMIDEFINITE PROGRAMMING LI LU (B.Sc., SJTU, China) A THESIS SUBMITTED FOR THE DEGREE OF DOCTOR OF PHILOSOPHY DEPARTMENT OF MATHEMATICS NATIONAL UNIVERSITY OF SINGAPORE 2010 To my parents Acknowledgements I would like to express my heartfelt gratitude to my advisor Professor Toh KimChuan, for his invaluable guidance and expertise in optimization, his utmost support and encouragement throughout the past five years Without him, this thesis would never have been possible The way of conducting scientific research, the opening posture towards new ideas and the attitude on teaching that I learned from him would be a lifelong treasure I would like to express my sincere thanks to Professor Zhao Gongyun for his instruction on game theory and numerical optimization, which are the first and the last modules I took during my study in NUS I sincerely thank him for sharing with me his wisdom and experience in the field of numerical computation and optimization theory I am also indebted to Professor Sun Defeng for his continuous effort on conducting weekly optimization seminars in Department of Mathematics, NUS His broad knowledge and enthusiasm on optimization have helped me tremendously in exploring various topics I am also thankful to Dr Liu Yongjin, Dr Yun Sangwoon and Dr Zhao Xinyuan v vi Acknowledgements for their helpful and constructive discussions in many topics related to my thesis This acknowledgement will remain incomplete without expressing my gratitude to my fellow colleagues and friends at NUS, where many happy memories I will carry from My thanks also goes out to National University of Singapore and Department of Mathematics for providing me excellent working conditions and scholarships to complete my study Contents Acknowledgements Summary v xi List of Tables xiii Notation xvi Introduction 1.1 The bottleneck of interior-point methods 1.2 Organization of the thesis 1.3 Convex quadratic SDP 1.4 Sparse covariance selection 10 1.5 Dual-scaling interior-point methods 16 Symmetric cones and Euclidean Jordan Algebras 19 vii viii Contents Polynomial-time inexact interior-point methods for convex quadratic programming over symmetric cones 29 3.1 Convex quadratic symmetric cone programming 30 3.2 An infeasible central path and its neighborhood 33 3.3 An inexact infeasible interior-point algorithm 38 3.4 Proof of Lemma 3.7 44 Inexact primal-dual path-following methods for l1 -regularized logdeterminant semidefinite programming problem 57 4.1 A customized inexact primal-dual interior-point method 58 4.2 Preconditioners 64 4.3 Computation of search direction for the special case (1.8) 66 4.3.1 Computing (∆x, ∆y) first 67 4.3.2 Computing (∆y, ∆u) first 69 Numerical Experiment 71 4.4.1 Synthetic Examples 72 4.4.2 Real world examples 78 4.4 An inexact dual-scaling interior-point method for linear programming problems over symmetric cones 83 5.1 An inexact dual-scaling interior point algorithm 84 5.1.1 Inexact search directions 87 5.2 Verification of the admissible condition (5.13b) 95 5.3 A practical inexact-direction dual-scaling algorithm 98 5.4 Numerical experiments 103 Conclusions and future work 111 Contents Bibliography ix 113 114 Bibliography [6] J A Bilmes, Natural statistical models for automatic speech recognition, PhD thesis, University of California, Berkeley, 1999 [7] P Biswas, K.-C Toh, and Y Ye, A distributed SDP approach for largescale noisy anchor-free graph reailzation with applications to molecular conformation, SIAM J Sci Comput., 30 (2008), pp 1251–1277 [8] B Borchers, SDPLIB 1.2, library of semidefinite programming test problems, Optim Methods Softw., 11/12 (1999), pp 683–690 Interior point methods [9] R Borsdorf and N J Higham, A preconditioned Newton algorithm for the nearest correlation matrix, IMA J Numer Anal., 30 (2010), pp 94–107 [10] S Boyd and L Xiao, Least-squares covariance matrix adjustment, SIAM J Matrix Anal Appl., 27 (2005), pp 532–546 (electronic) [11] S Burer, R D C Monteiro, and Y Zhang, A computational study of a gradient-based log-barrier algorithm for a class of large-scale SDPs, Math Program., 95 (2003), pp 359–379 [12] Z X Chan and D Sun, Constraint nondegeneracy, strong regularity, and nonsingularity in semidefinite programming, SIAM J Optim., 19 (2008), pp 370–396 [13] B Chazelle, C Kingsford, and M Singh, A semidefinite programming approach to side chain positioning with new rounding strategies, INFORMS J Comput., 16 (2004), pp 380–392 [14] S S Chen and R A Gopinath, Model selection in acoustic modeling, in Proc EUROSPEECH’99, Budapest, Hungary, 1999, pp 1087–1090 Bibliography [15] C Choi and Y Ye, Solving sparse semidefinite programs using the dual scaling algorithm with an iterative solver working paper, 2000 [16] G M Crippen and T F Havel, Distance geometry and molecular conformation, vol 15 of Chemometrics Series, Research Studies Press Ltd., Chichester, 1988 [17] J Dahl, L Vandenberghe, and V Roychowdhury, Covariance selection for nonchordal graphs via chordal embedding, Optim Methods Softw., 23 (2008), pp 501–520 [18] G B Dantzig, Linear programming and extensions, Princeton University Press, Princeton, N.J., 1963 [19] A d’Aspremont, Identifying small mean reverting portfolios preprint, 2008 [20] A d’Aspremont, O Banerjee, and L El Ghaoui, First-order methods for sparse covariance selection, SIAM J Matrix Anal Appl., 30 (2008), pp 56–66 [21] J W Demmel, Applied numerical linear algebra, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1997 [22] A P Dempster, Covariance selection, Biometrics, 28 (1972), pp 157–175 [23] A Dobra, Dependency networks for genome-wide data, Tech Report 574, Department of Statistics, University of Washington, 2009 [24] R L Dykstra, An algorithm for restricted least squares regression, J Amer Statist Assoc., 78 (1983), pp 837–842 [25] D Edwards, Introduction to graphical modelling, Springer Texts in Statistics, Springer-Verlag, New York, second ed., 2000 115 116 Bibliography [26] J Fan, Y Feng, and Y Wu, Network exploration via the adaptive LASSO and SCAD penalties, Ann Appl Stat., (2009), pp 521–541 ´ [27] J Faraut and A Koranyi, Analysis on symmetric cones, Oxford Mathematical Monographs, The Clarendon Press Oxford University Press, New York, 1994 Oxford Science Publications [28] R M Freund, Polynomial-time algorithms for linear programming based only on primal scaling and projected gradients of a potential function, Math Programming, 51 (1991), pp 203–222 [29] R W Freund, F Jarre, and S Mizuno, Convergence of a class of inexact interior-point algorithms for linear programs, Math Oper Res., 24 (1999), pp 50–71 [30] R W Freund and N M Nachtigal, A new krylov-subspace method for symmetric indefinite linear systems, in Proc 14th IMACS World Congress on Comput Applied Math., Atlanta, USA, 1994, pp 1253–1256 [31] J Friedman, T Hastie, and R Tibshirani, Sparse inverse covariance estimation with the graphical lasso, Biostat., (2008), pp 432–441 [32] M Fukuda, M Kojima, K Murota, and K Nakata, Exploiting sparsity in semidefinite programming via matrix completion I General framework, SIAM J Optim., 11 (2000/01), pp 647–674 (electronic) [33] T Fushiki, Estimation of positive semidefinite correlation matrices by using convex quadratic semidefinite programming, Neural Comput., 21 (2009), pp 2028–2048 Bibliography 117 [34] Y Gao and D Sun, Calibrating least squares semidefinite programming with equality and inequality constraints, SIAM J Matrix Anal Appl., 31 (2009), pp 1432–1457 [35] T R Golub, D K Slonim, P Tamayo, C Huard, M Gaasenbeek, J P Mesirov, H Coller, M L Loh, J R Downing, M A Caligiuri, and C D Bloomfield, Molecular classification of cancer: class discovery and class prediction by gene expression monitoring, Science, 286 (1999), pp 531–537 [36] I Hedenfalk, D Duggan, Y Chen, M Radmacher, M Bittner, R Simon, P Meltzer, B Gusterson, M Esteller, M Raffeld, Z Yakhini, A Ben-Dor, E Dougherty, J Kononen, L Bubendorf, W Fehrle, S Pittaluga, S Gruvberger, N Loman, O Johannsson, H Olsson, B Wilfond, G Sauter, O.-P Kallioniemi, A Borg, and J Trent, Gene-expression profiles in hereditary breast cancer, N Engl J Med., 344 (2001), pp 539–548 [37] C Helmberg, Semidefinite programming http://www-user tu-chemnitz.de/~helmberg/semidef.html [38] C Helmberg, F Rendl, R J Vanderbei, and H Wolkowicz, An interior-point method for semidefinite programming, SIAM J Optim., (1996), pp 342–361 [39] N J Higham, Computing the nearest correlation matrix—a problem from finance, IMA J Numer Anal., 22 (2002), pp 329–343 [40] G Iyengar, D J Phillips, and C Stein, Approximation algorithms for semidefinite packing problems with applications to MAXCUT and graph 118 Bibliography coloring, in Integer programming and combinatorial optimization, vol 3509 of Lecture Notes in Comput Sci., Springer, Berlin, 2005, pp 152–166 [41] F Jarre and F Rendl, An augmented primal-dual method for linear conic programs, SIAM J Optim., 19 (2008), pp 808–823 [42] D Johnson, G Pataki, and F Alizadeh, The seventh DIMACS implementation challenge: Semidefinite and related optimization problems http://dimacs.rutgers.edu/Challenges/Seventh/, 2000 [43] P Jordan, J von Neumann, and E Wigner, On an algebraic generalization of the quantum mechanical formalism, Ann of Math (2), 35 (1934), pp 29–64 [44] N Karmarkar, A new polynomial-time algorithm for linear programming, Combinatorica, (1984), pp 373–395 √ [45] M Kojima, S Mizuno, and A Yoshise, An O( n L) iteration potential reduction algorithm for linear complementarity problems, Math Programming, 50 (1991), pp 331–342 [46] M Kojima, M Shida, and S Shindoh, Search directions in the SDP and the monotone SDLCP: generalization and inexact computation, Math Program., 85 (1999), pp 51–80 [47] M Kojima, S Shindoh, and S Hara, Interior-point methods for the monotone semidefinite linear complementarity problem in symmetric matrices, SIAM J Optim., (1997), pp 86–125 [48] J Korzak, Convergence analysis of inexact infeasible-interior-point algorithms for solving linear programming problems, SIAM J Optim., 11 (2000), pp 133–148 (electronic) Bibliography [49] E Kranich, Interior point methods for mathematical programming : A bibliography http://netlib.sandia.gov/bib/index.html, 1996 [50] V Krishnamurthy and A d’Aspremont, A pathwise algorithm for covariance selection preprint, 2009 [51] B Kulis, M A Sustik, and I S Dhillon, Low-rank kernel learning with Bregman matrix divergences, J Mach Learn Res., 10 (2009), pp 341– 376 [52] S L Lauritzen, Graphical models, vol 17 of Oxford Statistical Science Series, The Clarendon Press Oxford University Press, New York, 1996 Oxford Science Publications [53] S R Lele, Euclidean Distance Matrix Analysis (EDMA): Estimation of mean form and mean form difference, Math Geology, 25 (1993), pp 573– 602 [54] L Li and K.-C Toh, A polynomial-time inexact primal-dual infeasible path-following algorithm for convex quadratic SDP accepted, Oct 2009 [55] N Linial, E London, and Y Rabinovich, The geometry of graphs and some of its algorithmic applications, Combinatorica, 15 (1995), pp 215–245 ´ [56] L Lovasz, On the Shannon capacity of a graph, IEEE Trans Inform Theory, 25 (1979), pp 1–7 [57] Z Lu, Smooth optimization approach for sparse covariance selection, SIAM J Optim., 19 (2008), pp 1807–1827 [58] , Adaptive first-order methods for general sparse inverse covariance selection preprint, 2009 119 120 Bibliography [59] Z Lu, R D C Monteiro, and J W O’Neal, An iterative solver-based infeasible primal-dual path-following algorithm for convex quadratic programming, SIAM J Optim., 17 (2006), pp 287–310 (electronic) [60] Z.-Q Luo, J F Sturm, and S Zhang, Conic convex programming and self-dual embedding, Optim Methods Softw., 14 (2000), pp 169–218 [61] J Malick, A dual approach to semidefinite least-squares problems, SIAM J Matrix Anal Appl., 26 (2004), pp 272284 (electronic) ă [62] N Meinshausen and P Buhlmann, High-dimensional graphs and variable selection with the lasso, Ann Statist., 34 (2006), pp 1436–1462 [63] S Mizuno and F Jarre, Global and polynomial-time convergence of an infeasible-interior-point algorithm using inexact computation, Math Program., 84 (1999), pp 105–122 [64] R D C Monteiro, Primal-dual path-following algorithms for semidefinite programming, SIAM J Optim., (1997), pp 663–678 [65] R D C Monteiro and Y Zhang, A unified analysis for a class of long-step primal-dual path-following interior-point algorithms for semidefinite programming, Math Programming, 81 (1998), pp 281–299 [66] G E Moore, Cramming more components onto integrated circuits, Electronics, 38 (1965), p 114–117 ´ [67] J J More and Z Wu, Global continuation for distance geometry problems, SIAM J Optim., (1997), pp 814–836 [68] Y Nesterov, Smooth minimization of non-smooth functions, Math Program., 103 (2005), pp 127–152 Bibliography [69] Y Nesterov and A Nemirovskii, Interior-point polynomial algorithms in convex programming, vol 13 of SIAM Studies in Applied Mathematics, Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 1994 ´ ´ [70] A Orden, LP from the 40s to the 90s, Interfaces, 23 (1993), pp 2–12 [71] P M Pardalos, D Shalloway, and G Xue, eds., Global minimization of nonconvex energy functions: molecular conformation and protein folding, DIMACS Series in Discrete Mathematics and Theoretical Computer Science, 23, American Mathematical Society, Providence, RI, 1996 Papers from the DIMACS Workshop held as part of the DIMACS Special Year on Mathematical Support for Molecular Biology at Rutgers University, New Brunswick, New Jersey, March 20–21, 1995 [72] J Pittman, E Huang, H Dressman, C.-F Horng, S H Cheng, M.-H Tsou, C.-M Chen, A Bild, E S Iversen, A T Huang, J R Nevins, and M West, Integrated modeling of clinical and gene expression information for personalized prediction of disease outcomes, in Proc Natl Acad Sci USA, 101(22), 2004, pp 8431–8436 [73] H Qi and D Sun, A quadratically convergent Newton method for computing the nearest correlation matrix, SIAM J Matrix Anal Appl., 28 (2006), pp 360–385 (electronic) [74] B K Rangarajan, Polynomial convergence of infeasible-interior-point methods over symmetric cones, SIAM J Optim., 16 (2006), pp 1211–1229 (electronic) 121 122 Bibliography ă [75] R Rebonato and P Jacke, The most general methodology for creating a valid correlation matrix for risk management and option pricing purposes, J Risk, (1999) [76] R T Rockafellar, Augmented Lagrangians and applications of the proximal point algorithm in convex programming, Math Oper Res., (1976), pp 97–116 [77] R T Rockafellar, Monotone operators and the proximal point algorithm, SIAM J Control Optimization, 14 (1976), pp 877–898 [78] Y Saad, Iterative methods for sparse linear systems, Society for Industrial and Applied Mathematics, Philadelphia, PA, second ed., 2003 [79] K Sachs, O Perez, D Pe’er, D A Lauffenburger, and G P Nolan, Causal protein-signaling networks derived from multiparameter single-cell data, Science, 308 (2005), pp 523–529 [80] K Scheinberg and I Rish, SINCO - a greedy coordinate ascent method for sparse inverse covariance selection problem preprint, 2009 [81] S H Schmieta and F Alizadeh, Associative and Jordan algebras, and polynomial time interior-point algorithms for symmetric cones, Math Oper Res., 26 (2001), pp 543–564 [82] , Extension of primal-dual interior point algorithms to symmetric cones, Math Program., 96 (2003), pp 409–438 [83] N J A Sloane, Challenge problems: Independent sets in graphs http: //www.research.att.com/~njas/, 2005 AT&T Shannon Lab [84] J D Storey and R Tibshirani, Statistical significance for genome-wide studies, in Proc Natl Acad Sci USA, 100(16), 2003, pp 9440–9445 Bibliography [85] J F Sturm, Using SeDuMi 1.02, a MATLAB toolbox for optimization over symmetric cones, Optim Methods Softw., 11/12 (1999), pp 625–653 [86] D Sun and J Sun, Semismooth matrix-valued functions, Math Oper Res., 27 (2002), pp 150–169 [87] D Sun, J Sun, and L Zhang, The rate of convergence of the augmented Lagrangian method for nonlinear semidefinite programming, Math Program., 114 (2008), pp 349–391 [88] M J Todd, Semidefinite optimization, Acta Numer., 10 (2001), pp 515 560 ă ă ă [89] M J Todd, K.-C Toh, and R H Tutuncu, On the Nesterov-Todd direction in semidefinite programming, SIAM J Optim., (1998), pp 769– 796 (electronic) [90] M J Todd and Y Ye, A centered projective algorithm for linear programming, Math Oper Res., 15 (1990), pp 508–529 [91] K.-C Toh, A note on the calculation of step-lengths in interior-point methods for semidefinite programming, Comput Optim Appl., 21 (2002), pp 301–310 [92] , Solving large scale semidefinite programs via an iterative solver on the augmented systems, SIAM J Optim., 14 (2003), pp 670–698 (electronic) [93] , An inexact primal-dual path following algorithm for convex quadratic SDP, Math Program., 112 (2008), pp 221–254 [94] K.-C Toh and M Kojima, Solving some large scale semidefinite programs via the conjugate residual method, SIAM J Optim., 12 (2002), pp 669691 (electronic) 123 124 Bibliography ă ă ă [95] K.-C Toh, M J Todd, and R H Tutuncu, SDPT3—a MATLAB software package for semidefinite programming, version 1.3, Optim Methods Softw., 11/12 (1999), pp 545581 Interior point methods ă ă ă [96] K.-C Toh, R H Tutuncu, and M J Todd, Inexact primal-dual pathfollowing algorithms for a special class of convex quadratic SDP and related problems, Pac J Optim., (2007), pp 135–164 [97] S Turkay, E Epperlein, and N Christofides, Correlation stress testing for value-at-risk, J Risk, (2003), pp 7589 ă ă ă [98] R H Tutuncu, K.-C Toh, and M J Todd, Solving semidefinitequadratic-linear programs using SDPT3, Math Program., 95 (2003), pp 189– 217 Computational semidefinite and second order cone programming: the state of the art [99] U Ueno and T Tsuchiya, Covariance regularization in inverse space preprint, 2009 [100] L Vandenberghe, S Boyd, and S.-P Wu, Determinant maximization with linear matrix inequality constraints, SIAM J Matrix Anal Appl., 19 (1998), pp 499–533 (electronic) [101] M J Wainwright and M I Jordan, Log-determinant relaxation for approximate inference in discrete markov random fields, IEEE Trans Signal Process., 54 (2006), pp 2099–2109 [102] C Wang, D Sun, , and K.-C Toh, Solving log-determinant optimization problems by a newton-cg proximal point algorithm preprint, 2009 Bibliography [103] K Q Weinberger, F Sha, and L K Saul, Learning a kernel matrix for nonlinear dimensionality reduction, in ACM International Conference Proceeding Series, 69, 2004, pp 106–115 [104] J Whittaker, Graphical models in applied multivariate statistics, Wiley Series in Probability and Mathematical Statistics: Probability and Mathematical Statistics, John Wiley & Sons Ltd., Chichester, 1990 ă [105] A Wille, P Zimmermann, E Vranova, A Furholz, O Laule, ´ S Bleuler, L Hennig, A Prelic, P von Rohr, L Thiele, E Zită zler, W Gruissem, and P Buhlmann, Sparse graphical gaussian modeling of the isoprenoid gene network in arabidopsis thaliana, Genome Biol., (2004) [106] F Wong, C K Carter, and R Kohn, Efficient estimation of covariance selection models, Biometrika, 90 (2003), pp 809–830 [107] W B Wu and M Pourahmadi, Nonparameteric estimation of large covariance matrices of longitudinal data, Biometrika, 90 (2003), pp 831–844 [108] M Yamashita, K Fujisawa, and M Kojima, Implementation and evaluation of SDPA 6.0 (semidefinite programming algorithm 6.0), Optim Methods Softw., 18 (2003), pp 491–505 The Second Japanese-Sino Optimization Meeting, Part II (Kyoto, 2002) [109] Y Ye and J Zhang, An improved algorithm for approximating the radii of point sets, in Approximation, randomization, and combinatorial optimization, vol 2764 of Lecture Notes in Comput Sci., Springer, Berlin, 2003, pp 178–187 125 126 Bibliography [110] K Y Yeung, R E Bumgarner, and A E Raftery, Bayesian model averaging: development of an improved multi-class, gene selection and classification tool for microarray data, Bioinformatics, 21 (2005), pp 2394–2402 [111] M Yuan and Y Lin, Model selection and estimation in the gaussian graphical model, Biometrika, 94 (2007), pp 19–35 [112] X Yuan, Alternating direction methods for sparse covariance selection preprint, 2009 [113] S Zhang, A new self-embedding method for convex programming, J Global Optim., (2004), pp 479–496 [114] S Zhang, On alternating direction methods for monotropic semidefinite programming, PhD thesis, National University of Singapore, 2009 [115] Y Zhang, On extending some primal-dual interior-point algorithms from linear programming to semidefinite programming, SIAM J Optim., (1998), pp 365–386 (electronic) [116] X Y Zhao, A semismooth Newton-CG augmented Lagrangian method for large scale linear and convex quadratic SDPs, PhD thesis, National University of Singapore, 2009 [117] X Y Zhao, D Sun, and K.-C Toh, A Newton-CG augmented Lagrangian method for semidefinite programming, SIAM J Optim., 20 (2010), pp 1737–1765 [118] G Zhou and K.-C Toh, Polynomiality of an inexact infeasible interior point algorithm for semidefinite programming, Math Program., 99 (2004), pp 261–282 INEXACT INTERIOR-POINT METHODS FOR LARGE SCALE LINEAR AND CONVEX QUADRATIC SEMIDEFINITE PROGRAMMING LI LU NATIONAL UNIVERSITY OF SINGAPORE 2010 Inexact Interior-Point Methods for Large Scale Linear and Convex Quadratic Semidefinite programming Li Lu 2010 ... developed an inexact primal-dual path-following interior- point method for convex quadratic symmetric xi xii Summary cone programming problems and an inexact dual-scaling interior- point method for linear. .. beyond interior- point methods to consider algorithms based on classical methods for convex programming, such as proximal -point and augmented Lagrangian methods (For details on non -interior- point. .. Therefore the interior- point methods using iterative solvers are commonly called inexact interior- point methods In order to guarantee the global polynomial convergence of inexact interior- point methods,