Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Notation
- Contributors
- 1 Introduction to Information Theory and Data Science.
- 2 An Information-Theoretic Approach to Analog-to-Digital Compression
- 3 Compressed Sensing via Compression Codes
- 4 Information-Theoretic Bounds on Sketching
- 5 Sample Complexity Bounds for Dictionary Learning from Vector- and Tensor-Valued Data
- 6 Uncertainty Relations and Sparse Signal Recovery
- 7 Understanding Phase Transitions via Mutual Information and MMSE
- 8 Computing Choice: Learning Distributions over Permutations
- 9 Universal Clustering
- 10 Information-Theoretic Stability and Generalization
- 11 Information Bottleneck and Representation Learning
- 12 Fundamental Limits in Model Selection for Modern Data Analysis
- 13 Statistical Problems with Planted Structures: Information-Theoretical and Computational Limits
- 14 Distributed Statistical Inference with Compressed Data
- 15 Network Functional Compression
- 16 An Introductory Guide to Fano’s Inequality with Applications in Statistical Estimation
- Index
- References
8 - Computing Choice: Learning Distributions over Permutations
Published online by Cambridge University Press: 22 March 2021
Book contents
- Frontmatter
- Dedication
- Contents
- Preface
- Notation
- Contributors
- 1 Introduction to Information Theory and Data Science.
- 2 An Information-Theoretic Approach to Analog-to-Digital Compression
- 3 Compressed Sensing via Compression Codes
- 4 Information-Theoretic Bounds on Sketching
- 5 Sample Complexity Bounds for Dictionary Learning from Vector- and Tensor-Valued Data
- 6 Uncertainty Relations and Sparse Signal Recovery
- 7 Understanding Phase Transitions via Mutual Information and MMSE
- 8 Computing Choice: Learning Distributions over Permutations
- 9 Universal Clustering
- 10 Information-Theoretic Stability and Generalization
- 11 Information Bottleneck and Representation Learning
- 12 Fundamental Limits in Model Selection for Modern Data Analysis
- 13 Statistical Problems with Planted Structures: Information-Theoretical and Computational Limits
- 14 Distributed Statistical Inference with Compressed Data
- 15 Network Functional Compression
- 16 An Introductory Guide to Fano’s Inequality with Applications in Statistical Estimation
- Index
- References
Summary
We discuss the question of learning distributions over permutations of a given set of choices, options or items based on partial observations. This is central to capturing the so-called “choice’’ in a variety of contexts. The question of learning distributions over permutations arises beyond capturing “choice’’ too, e.g., tracking a collection of objects using noisy cameras, or aggregating ranking of web-pages using outcomes of multiple search engines. Here we focus on learning distributions over permutations from marginal distributions of two types: first-order marginals and pair-wise comparisons. We emphasize the ability to identify the entire distribution over permutations as well as the “best ranking’’.
Keywords
- Type
- Chapter
- Information
- Information-Theoretic Methods in Data Science , pp. 229 - 262Publisher: Cambridge University PressPrint publication year: 2021
References
Huang, J., Guestrin, C., and Guibas, L., “Fourier theoretic probabilistic inference over permutations,” J. Machine Learning Res., vol. 10, no. 5, pp. 997–1070, 2009.Google Scholar
Diaconis, P., Group representations in probability and statistics. Institute of Mathematical Statistics, 1988.CrossRefGoogle Scholar
Thurstone, L., “A law of comparative judgement,” Psychol. Rev., vol. 34, pp. 237–286, 1927.Google Scholar
Marschak, J., “Binary choice constraints on random utility indicators,” Cowles Foundation Discussion Paper, 1959.Google Scholar
Yellott, J. I., “The relationship between Luce’s choice axiom, Thurstone’s theory of comparative judgment, and the double exponential distribution,” J. Math. Psychol., vol. 15, no. 2, pp. 109–144, 1977.Google Scholar
Plackett, R., “The analysis of permutations,” Appl. Statist., vol. 24, no. 2, pp. 193–202, 1975.Google Scholar
McFadden, D., “Conditional logit analysis of qualitative choice behavior,” in Frontiers in Econometrics, Zarembka, P., ed. Academic Press, 1973, pp. 105–142.Google Scholar
Debreu, G., “Review of R. D. Luce, ‘individual choice behavior: A theoretical analysis,”’Amer. Economic Rev., vol. 50, pp. 186–188, 1960.Google Scholar
Bradley, R. A., “Some statistical methods in taste testing and quality evaluation,” Biometrics, vol. 9, pp. 22–38, 1953.Google Scholar
McFadden, D., “Econometric models of probabilistic choice,” in Structural analysis of discrete data with econometric applications, Manski, C.F. and McFadden, D., eds. MIT Press, 1981.Google Scholar
Ben-Akiva, M. E. and Lerman, S. R., Discrete choice analysis: Theory and application to travel demand. CMIT Press, 1985.Google Scholar
McFadden, D., “Disaggregate behavioral travel demand’s RUM side: A 30-year retrospective,” Travel Behaviour Res., pp. 17–63, 2001.Google Scholar
Guadagni, P. M. and Little, J. D. C., “A logit model of brand choice calibrated on scanner data,” Marketing Sci., vol. 2, no. 3, pp. 203–238, 1983.Google Scholar
Mahajan, S. and van Ryzin, G. J., “On the relationship between inventory costs and variety benefits in retail assortments,” Management Sci., vol. 45, no. 11, pp. 1496–1509, 1999.Google Scholar
Ben-Akiva, M. E., “Smicture of passenger travel demand models,” Ph.D. dissertation, Department of Civil Engineering, MIT, 1973.Google Scholar
Boyd, J. H. and Mellman, R. E., “The effect of fuel economy standards on the U.S. automotive market: An hedonic demand analysis,” Transportation Res. Part A: General, vol. 14, nos. 5–6, pp. 367–378, 1980.Google Scholar
Cardell, N. S. and Dunbar, F. C., “Measuring the societal impacts of automobile downsizing,” Transportation Res. Part A: General, vol. 14, nos. 5–6, pp. 423–434, 1980.Google Scholar
McFadden, D. and Train, K., “Mixed MNL models for discrete response,” J. Appl. Econometrics, vol. 15, no. 5, pp. 447–470, 2000.Google Scholar
Bartels, K., Boztug, Y., and Muller, M. M., “Testing the multinomial logit model,” 1999, unpublished working paper.Google Scholar
Horowitz, J. L., “Semiparametric estimation of a work-trip mode choice model,” J. Econometrics, vol. 58, pp. 49–70, 1993.CrossRefGoogle Scholar
Koopman, B., “On distributions admitting a sufficient statistic,” Trans. Amer. Math. Soc., vol. 39, no. 3, pp. 399–409, 1936.Google Scholar
Crain, B., “Exponential models, maximum likelihood estimation, and the Haar condition,” J. Amer. Statist. Assoc., vol. 71, pp. 737–745, 1976.Google Scholar
Beran, R., “Exponential models for directional data,” Annals Statist., vol. 7, no. 6, pp. 1162–1178, 1979.Google Scholar
Wainwright, M. and Jordan, M., “Graphical models, exponential families, and variational inference,” Foundations and Trends Machine Learning, vol. 1, nos. 1–2, pp. 1–305, 2008.Google Scholar
Jagabathula, S. and Shah, D., “Infering rankings under constrained sensing,” in Advances in Neural Information Processing Systems, 2009, pp. 753–760.Google Scholar
Jagabathula, S. and Shah, D., “Inferring rankings under constrained sensing,” IEEE Trans. Information Theory, vol. 57, no. 11, pp. 7288–7306, 2011.Google Scholar
Farias, V., Jagabathula, S., and Shah, D., “A data-driven approach to modeling choice,” in Advances in Neural Information Processing Systems, 2009.Google Scholar
Farias, V., Jagabathula, S., and Shah, D., “A nonparametric approach to modeling choice with limited data,” Management Sci., vol. 59, no. 2, pp. 305–322, 2013.Google Scholar
Arrow, K. J., “A difficulty in the concept of social welfare,” J. Political Economy, vol. 58, no. 4, pp. 328–346, 1950.CrossRefGoogle Scholar
Candès, E. J. and Tao, T., “Decoding by linear programimng,” IEEE Trans. Information Theory, vol. 51, no. 12, pp. 4203–4215, 2005.CrossRefGoogle Scholar
Candès, E. J., Romberg, J. K., and Tao, T., “Stable signal recovery from incomplete and inaccurate measurements,” Communications Pure Appl. Math., vol. 59, no. 8, pp. 1207–223, 2006.Google Scholar
Candès, E. J. and Romberg, J., “Quantitative robust uncertainty principles and optimally sparse decompositions,” Foundations Computational Math., vol. 6, no. 2, pp. 227–254, 2006.Google Scholar
Candès, E. J., Romberg, J., and Tao, T., “Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information,” IEEE Trans. Information Theory, vol. 52, no. 2, pp. 489–509, 2006.CrossRefGoogle Scholar
Donoho, D. L., “Compressed sensing,” IEEE Trans. Information Theory, vol. 52, no. 4, pp. 1289–1306, 2006.Google Scholar
Berinde, R., Gilbert, A. C., Indyk, P., Karloff, H., and Strauss, M. J., “Combining geometry and combinatorics: A unified approach to sparse signal recovery,” in Proc. 46th Annual Allerton Conference on Communications, Control, and Computation, 2008, pp. 798–805.Google Scholar
Chatterjee, S., “Matrix estimation by universal singular value thresholding,” Annals Statist., vol. 43, no. 1, pp. 177–214, 2015.Google Scholar
Song, D., Lee, C. E., Li, Y., and Shah, D., “Blind regression: Nonparametric regression for latent variable models via collaborative filtering,” in Advances in Neural Information Processing Systems, 2016, pp. 2155–2163.Google Scholar
Borgs, C., Chayes, J., Lee, C. E., and Shah, D., “Thy friend is my friend: Iterative collaborative filtering for sparse matrix estimation,” in Advances in Neural Information Processing Systems, 2017, pp. 4715–4726.Google Scholar
Shah, N., Balakrishnan, S., Guntuboyina, A., and Wainwright, M., “Stochastically transitive models for pairwise comparisons: Statistical and computational issues,” in International Conference on Machine Learning, 2016, pp. 11–20.Google Scholar
Farias, V. F., Jagabathula, S., and Shah, D., “Sparse choice models,” in 46th Annual Conference on Information Sciences and Systems (CISS), 2012, pp. 1–28.Google Scholar
Negahban, S., Oh, S., and Shah, D., “Iterative ranking from pair-wise comparisons,” in Advances in Neural Information Processing Systems, 2012, pp. 2474–2482.Google Scholar
Negahban, S., Oh, S., and Shah, D., “Rank centrality: Ranking from pairwise comparisons,” Operations Res., vol. 65, no. 1, pp. 266–287, 2016.Google Scholar
Ammar, A. and Shah, D., “Ranking: Compare, don’t score,” in Proc. 49th Annual Allerton Conference on Communication, Control, and Computing, 2011, pp. 776–783.Google Scholar
Ammar, A. and Shah, D., “Efficient rank aggregation using partial data,” in ACM SIGMETRICS Performance Evaluation Rev., vol. 40, no. 1, 2012, pp. 355–366.Google Scholar
Emerson, P., “The original Borda count and partial voting,” Social Choice and Welfare, vol. 40, no. 2, pp. 353–358, 2013.Google Scholar
Hajek, B., Oh, S., and Xu, J., “Minimax -optimal inference from partial rankings,” in Advances in Neural lnformation Processing Systems, 2014, pp. 1475–1483.Google Scholar
Chen, Y. and Suh, C., “Spectral mle: Top-k rank aggregation from pairwise comparisons,” in International Conference on Machine Learning, 2015, pp. 371–380.Google Scholar
Chen, Y., Fan, J., Ma, C., and Wang, K., “Spectral method and regularized mle are both optimal for top-k ranking,” arXiv:1707.09971, 2017.Google Scholar
Jang, M., Kim, S., Suh, C., and Oh, S., “Top-k ranking from pairwise comparisons: When spectral ranking is optimal,” arXiv:1603.04153, 2016.Google Scholar
L. Maystre and Grossglauser, M., “Fast and accurate inference of Plackett–Luce models,” in Advances in Neural Information Processing Systems, 2015, pp. 172–180.Google Scholar
Ammar, A., Oh, S., Shah, D., and Voloch, L. F., “What’s your choice?: Learning the mixed multi-nomial,”in ACM SIGMETRICS Performance Evaluation Rev., vol. 42, no. 1, 2014, pp. 565–566.Google Scholar
Oh, S., Thekumparampil, K. K., and Xu, J., “Collaboratively learning preferences from ordinal data,” in Advances in Neural Information Processing Systems, 2015, pp. 1909–1917.Google Scholar
Lu, Y. and Negahban, S. N., “Individualized rank aggregation using nuclear norm regularization,” in Proc. 53rd Annual Allerton Conference on Communication, Control, and Computing, 2015, pp. 1473–1479.CrossRefGoogle Scholar
Oh, S. and Shah, D., “Learning mixed multinomial logit model from ordinal data,” in Advances in Neural Information Processing Systems, 2014, pp. 595–603.Google Scholar
Shah, N. B., Balakrishnan, S., and Wainwright, M. J., “Feeling the bern: Adaptive estimators for Bernoulli probabilities of pairwise comparisons,” in IEEE International Symposium on Information Theory (ISIT), 2016, pp. 1153–1157.CrossRefGoogle Scholar
Adler, M., Gemmell, P., Harchol-Balter, M., Karp, R. M., and Kenyon, C., “Selection in the presence of noise: The design of playoff systems,” in SODA, 1994, pp. 564–572.Google Scholar
Jamieson, K. G. and Nowak, R., “Active ranking using pairwise comparisons,” in Advances in Neural Information Processing Systems, 2011, pp. 2240–2248.Google Scholar
Karbasi, A., Ioannidis, S., and Massoulié, L., “From small-world networks to comparison-based search,” IEEE Trans. Information Theory, vol. 61, no. 6, pp. 3056–3074, 2015.Google Scholar
Braverman, M. and Mossel, E., “Noisy sorting without resampling,” in Proc. 19th Annual ACM–SlAM Symposium on Discrete Algorithms, 2008, pp. 268–276.Google Scholar
Yue, Y. and Joachims, T., “Interactively optimizing information retrieval systems as a dueling bandits problem,” in Proc. 26th Annual International Conference on Machine Learning, 2009, pp. 1201–1208.Google Scholar
Jamieson, K. G., Katariya, S., Deshpande, A., and Nowak, R. D., “Sparse dueling bandits,” in AISTATS, 2015.Google Scholar
Dudík, M., Hofmann, K., Schapire, R. E., Slivkins, A., and Zoghi, M., “Contextual dueling bandits,” in Proc. 28th Conference on Learning Theory, 2015, pp. 563–587.Google Scholar
Kondor, R., Howard, A., and Jebara, T., “Multi -object tracking with representations of the symmetric group,” in Artificial Intelligence and Statistics, 2007, pp. 211–218.Google Scholar
Jerrum, M., Sinclair, A., and Vigoda, E., “A polynomial-time approximation algorithm for the permanent of a matrix with nonnegative entries,” J. ACM, vol. 51, no. 4, pp. 671–697, 2004.Google Scholar
Saaty, T. L. and Hu, G., “Ranking by eigenvector versus other methods in the analytic hierarchy process,” Appl. Mathe. Lett., vol. 11, no. 4, pp. 121–125, 1998.Google Scholar
Dwork, C., Kumar, R., Naor, M., and Sivakumar, D., “Rank aggregation methods for the web,” in Proc. 10th International Conference on World Wide Web, 2001, pp. 613–622.Google Scholar
Rajkumar, A. and Agarwal, S., “A statistical convergence perspective of algorithms for rank aggregation from pairwise data,” in International Conference on Machine Learning, 2014, pp. 118–126.Google Scholar
Azari, H., Parks, D., and Xia, L., “Random utility theory for social choice,” in Advances in Neural Information Processing Systems, 2012, pp. 126–134.Google Scholar