Book contents
- Frontmatter
- Contents
- List of contributors
- Preface
- Part I Mathematical foundations
- 1 Tensor models: solution methods and applications
- 2 Sparsity-aware distributed learning
- 3 Optimization algorithms for big data with application in wireless networks
- 4 A unified distributed algorithm for non-cooperative games
- Part II Big data over cyber networks
- Part III Big data over social networks
- Part IV Big data over biological networks
- Index
- References
1 - Tensor models: solution methods and applications
from Part I - Mathematical foundations
Published online by Cambridge University Press: 18 December 2015
Book contents
- Frontmatter
- Contents
- List of contributors
- Preface
- Part I Mathematical foundations
- 1 Tensor models: solution methods and applications
- 2 Sparsity-aware distributed learning
- 3 Optimization algorithms for big data with application in wireless networks
- 4 A unified distributed algorithm for non-cooperative games
- Part II Big data over cyber networks
- Part III Big data over social networks
- Part IV Big data over biological networks
- Index
- References
Summary
This chapter introduces several models and associated computational tools for tensor data analysis. In particular, we discuss: tensor principal component analysis, tensor low-rank and sparse decomposition models, and tensor co-clustering problems. Such models have a great variety of applications; examples can be found in computer vision, machine learning, image processing, statistics, and bio-informatics. For computational purposes, we present several useful tools in the context of tensor data analysis, including the alternating direction method of multipliers (ADMM), and the block variables optimization techniques. We draw on applications from the gene expression data analysis in bio-informatics to demonstrate the performance of some of the aforementioned tools.
Introduction
One rich source of big data roots is the high dimensionality of the data formats known as tensors. Specifically, a complex-valued m-dimensional or mth-order tensor (a.k.a. m-way multiarray) can be denoted by ℱ ∈ ℂn1×n2×…×nm, whose dimension in the ith direction is ni, i = 1, …,m. Vector and matrix are special cases of tensor when m = 1 and m = 2, respectively. In the era of big data analytics, huge-scale dense data in the form of tensors can be found in different domains such as computer vision [1], diffusion magnetic resonance imaging (MRI) [2–4], the quantum entanglement problem [5], spectral hypergraph theory [6], and higher-order Markov chains [7]. For instance, a color image can be considered as 3D data with row, column, color in each direction, while a color video sequence can be considered as 4D data, where time is the fourth dimension. Therefore, how to extract useful information from these tensor data becomes a very meaningful task.
On the other hand, the past few years have witnessed an emergence of sparse and low-rank matrix optimization models and their applications in data sciences, signal processing, machine learning, bioinformatics, and so on. There have been extensive investigations on low-rank matrix completion and recovery problems since the seminal works of [8–11]. Some important variants of sparse and low-rank matrix optimization problems such as robust principal component analysis (PCA) [12, 13] and sparse PCA [14] have also been studied. A natural extension of the matrix to higher-dimensional space is the tensor. Traditional matrix-based data analysis is inherently two-dimensional, which limits its ability in extracting information from a multi-dimensional perspective.
- Type
- Chapter
- Information
- Big Data over Networks , pp. 3 - 36Publisher: Cambridge University PressPrint publication year: 2016
References
[1] and , “Compact representation of multidimensional data using tensor rank-one decomposition,” in Proceedings of the Pattern Recognition, 17th International Conference on ICPR, 2004.
[2] , , , , , and , “A polynomial based approach to extract the maxima of an antipodally symmetric spherical function and its application to extract fiber directions from the orientation distribution function in diffusion mri,” in Computational Diffusion MRI Workshop (CDMRI'08), New York, 2008.Google Scholar
[3] and , “On computing the underlying fiber directions from the diffusion orientation distribution function,” in Medical Image Computing and Computer-Assisted Intervention, MICCAI 2008, , , and , eds., 2008.Google Scholar
[4] , , and , “Higher order positive semi-definite diffusion tensor imaging,” SIAM Journal on Imaging Sciences, pp. 416–433, 2010.Google Scholar
[5] and , “The geometric measure of multipartite entanglement and the singular values of a hypermatrix,” J. Math. Phys., vol. 51, p. 072102, 2010.Google Scholar
[6] and , “Algebraic connectivity of an even uniform hypergraph,” Journal of Combinatorial Optimization, vol. 24, pp. 564–579, 2012.Google Scholar
[7] and , “Existence and uniqueness of stationary probability vector of a transition probability tensor,” Department of Mathematics, The Hong Kong Baptist University, Tech. Rep., 2011.Google Scholar
[8] , , and , “Rank minimization and applications in system theory.” in American Control Conference, 2004, pp. 3273–3278.
[9] , , and , “Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization,” SIAM Review, vol. 52, no. 3, pp. 471–501, 2010.Google Scholar
[10] and , “Exact matrix completion via convex optimization,” Foundations of Computational Mathematics, vol. 9, pp. 717–772, 2009.Google Scholar
[11] and , “The power of convex relaxation: near-optimal matrix completion,” IEEE Trans. Inform. Theory, vol. 56, no. 5, pp. 2053–2080, 2009.Google Scholar
[12] , , , and , “Robust principal component analysis?” Journal of ACM, vol. 58, no. 3, pp. 1–37, 2011.Google Scholar
[13] , , , and , “Rank-sparsity incoherence for matrix decomposition,” SIAM Journal on Optimization, vol. 21, no. 2, pp. 572–596, 2011.Google Scholar
[14] , , , and , “A direct formulation for sparse PCAusing semidefinite programming,” SIAM Review, vol. 49, no. 3, pp. 434–448, 2007.Google Scholar
[15] , , , et al., “SPARCoC: a new framework for molecular pattern discovery and cancer gene identification,” PLoS ONE, vol. 10, no. 3, e0117135, 2015.Google Scholar
[16] , , and , “Nuclear-norm-free variational models for background extraction from surveillance video,” Preprint, 2013.
[17] , The Expression of a Tensor or a Polyadic as a Sum of Products. Institute of Technology, 1927.
[18] , “Multiple invariants and generalized rank of a p-way matrix or tensor,” Journal of Mathematical Physics, vol. 7, no. 1, pp. 39–79, 1927.Google Scholar
[19] and , “Analysis of individual differences in multidimensional scaling via an n-way generalization of ‘Eckart–Young’ decomposition,” Psychometrika, vol. 35, no. 3, pp. 283–319, 1970.Google Scholar
[20] , Foundations of the PARAFAC Procedure: Models and Conditions for an “Explanatory” Multimodal Factor Analysis. Los Angeles: University of California at Los Angeles, 1970.Google Scholar
[21] , “Deflation methods for sparse PCA,” in Advances in Neural Information Processing Systems (NIPS), 2008.
[22] , , and , “On the best rank-1 and rank- (r1, r2, …, rn) approximation of higher-order tensors,” SIAM Journal on Matrix Analysis and Applications, vol. 21, no. 4, pp. 1324–1342, 2000.Google Scholar
[23] , , and , “Tensor principal component analysis via convex optimization,” Mathematical Programming, vol. 150, pp. 423–457, 2015.Google Scholar
[24] , “Eigenvalues of a real supersymmetric tensor,” Journal of Symbolic Computation, vol. 40, pp. 1302–1324, 2005.Google Scholar
[25] and , “Tensor approximation and signal processing applications,” in Structured Matrices in Mathematics, Computer Science, and Engineering I, , Ed., Contemporary Mathematics Series, American Mathematical Society, 2001.Google Scholar
[26] , “Singular values and eigenvalues of tensors: a variational approach,” in Proceedings of the IEEE International Workshop on Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP), 2005.
[27] , , , and , “Maximum block improvement and polynomial optimization,” SIAM Journal on Optimization, vol. 22, pp. 87–107, 2012.Google Scholar
[28] , , and , “Z-eigenvalue methods for a global polynomial optimization problem,” Mathematical Programming, Series A, vol. 118, pp. 301–316, 2009.Google Scholar
[29] and , “On the best rank-1 approximation of higher-order supersymmetric tensors,” SIAM Journal on Matrix Analysis and Applications, vol. 23, pp. 863–884, 2002.Google Scholar
[30] and , “Tensor decompositions and applications,” SIAM Review, vol. 51, pp. 455–500, 2009.Google Scholar
[31] and , “Shifted power method for computing tensor eigenpairs,” SIAM J. Matrix Analysis, vol. 32, pp. 1095–1124, 2011.Google Scholar
[32] , , and , “Toxicogenomics in predictive toxicology in drug development,” Chem. Biol., vol. 11, pp. 161–171, 2004.Google Scholar
[33] , , and , “cdna microarrays: Identification of gene signatures and their application in clinical practice,” J. BUON, vol. 12, Suppl. 1, pp. S39–44, 2007.Google Scholar
[34] , “Molecular targets in gynaecological cancers,” Pathology, vol. 39, pp. 26–45, 2007.
[35] , , , and , “A new framework for co-clustering of gene expression data,” in PRIB2011, ser. Lecture Notes in Bio-Informatics, et al., Eds., Springer-Verlag, 2011, vol. 7036, pp. 1–12.Google Scholar
[36] , , , , and , “A unified adaptive co-identification framework for high-d expression data,” in PRIB2012, ser. LectureNotes in Bio-Informatics, Springer-Verlag, 2012, vol. 7632, pp. 59–70.Google Scholar
[37] , “Howdoes gene expression clusteringwork?” Nature Biotechnology, vol. 23, no. 12, pp. 1499–1502, 2005.Google Scholar
[38] , , , and , “Cluster analysis and display of genomewide expression patterns,” Proceedings of theNational Academy of Sciences, vol. 95, no. 25, pp. 14 863–14 868, 1998.Google Scholar
[39] , , , , and , “Systematic determination of genetic network architecture,” Nature Genetics, vol. 22, no. 3, pp. 281–285, 1999.Google Scholar
[40] , , , et al., “Interpreting patterns of gene expression with selforganizing maps: methods and application to hematopoietic differentiation,” Proceedings of the National Academy of Sciences, vol. 96, no. 6, pp. 2907–2912, 1999.Google Scholar
[42] , , , et al., “A systematic comparison and evaluation of biclustering methods for gene expression data,” Bioinformatics, vol. 22, no. 9, pp. 1122–1129, 2006.Google Scholar
[43] , , , et al., “A two-step clustering for 3-d gene expression data reveals the main features of the arabidopsis stress response,” Journal of Integrative Bioinformatics, vol. 4, no. 1, p. 54, 2007.Google Scholar
[44] and , “An effective tri-clustering algorithm combining expression data with gene regulation information.” Gene Regulation and Systems Biology, vol. 3, pp. 49–64, 2008.Google Scholar
[45] and , “Tricluster: an effective algorithm for mining coherent clusters in 3d microarray data,” in Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data, ACM, 2005, pp. 694–705.Google Scholar
[46] , , , , and , “Mining coherent gene clusters from gene-sample-time microarray data,” in Proceedings of the Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, 2004, pp. 430–439.Google Scholar
[47] , , , , and , “Hunting for coherent co-clusters in high dimensional and noisy datasets,” in Data Mining Workshops, 2008. ICDMW'08. IEEE International Conference on, IEEE, 2008, pp. 654–663.Google Scholar
[48] , , , , and , “A generalized maximum entropy approach to bregman co-clustering and matrix approximation,” Journal of Machine Learning Research, vol. 8, pp. 1919–1986, 2007.Google Scholar
[49] , , and , “Comparative analysis of clustering methods for gene expression time course data,” Genetics and Molecular Biology, vol. 27, no. 4, pp. 623–631, 2004.Google Scholar
[50] and , “Judging the quality of gene expression-based clustering methods using gene annotation,” Genome research, vol. 12, no. 10, pp. 1574–1581, 2002.Google Scholar
[52] , Multiway Data Analysis, North-Holland, Amsterdam, 1989, ch. Rank, Decomposition, and Uniqueness for 3-way and N-way Arrays, pp. 7–18.Google Scholar
[53] , , , and , “Tensor completion for estimating missing values in visual data,” in The Twelfth IEEE International Conference on Computer Vision, 2009.
[54] , , and , “Tensor completion and low-n-rank tensor recovery via convex optimization,” Inverse Problems, vol. 27, no. 2, p. 025010, 2011.Google Scholar
[55] , , and , “Estimation of low-rank tensors via convex optimization,” preprint, 2011.
[56] and , “Robust low-rank tensor recovery: models and algorithms,” preprint, 2013.
[57] , , and , “Statistical performance of convex tensor decomposition,” in NIPS, 2011.
[58] , , , and , “Tensor versus matrix completion: a comparison with application to spectral data,” IEEE Signal Processing Letters, vol. 18, no. 7, pp. 403–406, 2011.Google Scholar
[59] , , , and , “Square deal: lower bounds and improved relaxations for tensor recovery,” preprint, 2013.
[60] , , and , “Low-rank tensor completion by Riemannian optimization,” preprint, 2013.
[61] and , “Low-rank matrix and tensor completion via adaptive sampling,” preprint, 2013.
[62] , , and , “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Transactions on Information Theory, vol. 52, pp. 489–509, 2006.Google Scholar
[63] , “Compressed sensing,” IEEE Transactions on Information Theory, vol. 52, pp. 1289–1306, 2006.Google Scholar
[64] , “Interior point methods in semidefinite programming with applications to combinatorial optimization,” SIAM Journal on Optimization, vol. 5, pp. 13–51, 1993.Google Scholar
[65] and , “Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming,” J. Assoc. Comput. Mach., vol. 42, no. 6, pp. 1115–1145, 1995.Google Scholar
[67] and , “CVX: Matlab software for disciplined convex programming, version 1.21,” http://cvxr.com/cvx, 2010.
[68] and , “On the numerical solution of the heat conduction problem in 2 and 3 space variables,” Transactions of the American Mathematical Society, vol. 82, pp. 421–439, 1956.Google Scholar
[69] and , “The numerical solution of parabolic elliptic differential equations,” SIAM Journal on Applied Mathematics, vol. 3, pp. 28–41, 1955.Google Scholar
[70] and , “Splitting algorithms for the sum of two nonlinear operators,” SIAM Journal on Numerical Analysis, vol. 16, pp. 964–979, 1979.Google Scholar
[71] and , Augmented Lagrangian Methods: Applications to the Numerical Solution of Boundary-Value Problems, North-Holland Pub. Co., 1983.Google Scholar
[72] and , Augmented Lagrangian and Operator-Splitting Methods in Nonlinear Mechanics, Philadelphia, Pennsylvania: SIAM, 1989.Google Scholar
[73] , “Splitting methods for monotone operators with applications to parallel optimization,” Ph.D. dissertation, Massachusetts Institute of Technology, 1989.
[74] and , “On the Douglas–Rachford splitting method and the proximal point algorithm for maximal monotone operators,” Mathematical Programming, vol. 55, pp. 293–318, 1992.Google Scholar
[75] , “Applications of the method of multipliers to variational inequalities,” in Augmented Lagrangian Methods: Applications to the Solution of Boundary Value Problems, M.|Fortin and R.|Glowinski, Eds., Amsterdam: North-Holland, 1983.Google Scholar
[76] and , “Alternating direction algorithms for l1 problems in compressive sensing,” SIAM Journal on Scientific Computing, vol. 33, no. 1, pp. 250–278, 2011.Google Scholar
[77] , , , and , “A new alternating minimization algorithm for total variation image reconstruction,” SIAM Journal on Imaging Sciences, vol. 1, no. 3, pp. 248–272, 2008.Google Scholar
[78] and , “The split Bregmanmethod for L1-regularized problems,” SIAM J. Imaging Sci., vol. 2, pp. 323–343, 2009.Google Scholar
[79] and , “Recovering low-rank and sparse components of matrices from incomplete and noisy observations,” SIAM J. Optim., vol. 21, pp. 57–81, 2011.Google Scholar
[80] , “Alternating direction methods for sparse covariance selection,” Journal of Scientific Computing, vol. 51, pp. 261–273, 2012.Google Scholar
[81] , , and , “Sparse inverse covariance selection via alternating linearization methods,” in NIPS, 2010.
[82] , “Alternating direction method of multipliers for sparse principal component analysis,” Journal of the Operations Research Society of China, vol. 1, no. 2, pp. 253–274, 2013.Google Scholar
[83] , , and , “Alternating direction augmented Lagrangian methods for semidefinite programming,” Mathematical Programming Computation, vol. 2, pp. 203–230, 2010.Google Scholar
[84] , , , , and , “Distributed optimization and statistical learning via the alternating direction method of multipliers,” Foundations and Trends in Machine Learning, vol. 3, no. 1, pp. 1–122, 2011.Google Scholar
[85] , , and , “Fixed point and Bregman iterative methods for matrix rank minimization,” Mathematical Programming Series A, vol. 128, pp. 321–353, 2011.Google Scholar
[86] , Nonlinear Programming, 2nd Edn, Belmont, Massachusetts: Athena Scientific, 1999.Google Scholar
[87] , “Convergence of a block coordinate descent method for nondifferentiable minimization,” J. Optim. Theory Appl., vol. 109, no. 3, pp. 475–494, 2001.Google Scholar
[88] and , Parallel and Distributed Computation: Numerical Methods, Prentice-Hall, Inc., Upper Saddle River, NJ, USA, 1989.Google Scholar
[89] and , “On the convergence of the block nonlinear Gauss–Seidel method under convex constraints,” Oper. Res. Lett., vol. 26, no. 3, pp. 127–136, 2000.Google Scholar
[90] and , “On the convergence of the coordinate descent method for convex differentiable minimization,” J. Optim. Theory Appl., vol. 72, no. 1, pp. 7–35, 1992.Google Scholar
[91] and , “On the linear convergence of descent methods for convex essentially smooth minimization,” SIAM Journal on Control and Optimization, vol. 30, no. 2, pp. 408–425, 1992.Google Scholar
[92] and , “Error bounds and convergence analysis of feasible descent methods: a general approach,” Annals of Operations Research, vol. 46, no. 1, pp. 157–178, 1993.Google Scholar
[93] and , “On the convergence of block coordinate descent type methods,” SIAM Journal on Optimization, vol. 23, no. 4, pp. 2037–2060, 2013.Google Scholar
[94] , , , and , “Iteration complexity analysis of block coordinate descent methods,” arXiv preprint arXiv:1310.6957, 2013.
[95] , , and , “On convergence of the maximum block improvement method,” to appear in SIAM Journal on Optimization, 2013.
[96] et al., “Distinct types of diffuse large b-cell lymphoma identified by gene expression profiling,” Nature, vol. 403, no. 6769, pp. 503–511, 2000.Google Scholar
[97] , , , et al., “The AtGenExpress global stress expression data set: protocols, evaluation and model data analysis of UV-b light, drought and cold stress responses,” The Plant Journal, vol. 50, no. 2, pp. 347–363, 2007.Google Scholar
[98] , , , , and , “Edisa: extracting biclusters from multiple time-series of gene expression profiles,” BMC Bioinformatics, vol. 8, no. 1, pp. 334–347, 2007.Google Scholar
[99] , , , and , “Minimum sum-squared residue co-clustering of gene expression data.” in Proceedings of The Fourth SIAM International Conference on Data Mining, vol. 3, SIAM, 2004, pp. 114–125.Google Scholar
[100] , , , and , “Genotyping and genomic profiling of non–small-cell lung cancer: Implications for current and future therapies,” Journal of Clinical Oncology, vol. 31, no. 8, pp. 1039–1049, 2013.Google Scholar
[101] , , , et al., “A novel classification of lung cancer into molecular subtypes,” PloS one, vol. 7, no. 2, p. e31906, 2012.Google Scholar
[102] et al., “Gene expression–based survival prediction in lung adenocarcinoma: a multi-site, blinded validation study,” Nature Medicine, vol. 14, no. 8, pp. 822–827, 2008.Google Scholar