13 - Recent Trends
Published online by Cambridge University Press: 05 September 2016
Summary
Throughout the book we have covered a number of compact data structures that are well established and tested, and their basic aspects can be considered stable. This chapter is conceived as an epilogue, where we take a look to the future. We describe some recent trends that, while not mature or general enough to deserve a thorough treatment in the book, are certainly promising and likely to be the focus of intense research in the upcoming years. Therefore, the chapter may also serve to guide the readers looking for hot research topics. Its writing style is different from the other chapters, as we discuss the bibliography together with the main material.
First, we consider encoding data structures. These ensure only that a certain set of queries of interest can be answered; the actual data cannot be recovered. Encodings can offer significant space reductions with respect to the data entropy and are interesting because they have the potential to reach the minimum space needed just to answer the desired queries; nothing superfluous is stored. They also define a new concept of entropy, in terms not only of a universe of objects, but also of a set of queries posed on them. Encodings for a few problems exist already, but they are very recent and mostly in a theoretical stage.
Second, we consider repetitive document collections. Many of the largest text collections arising in applications these days are highly repetitive, and thus very compressible if one applies the right compression methods. The compact data structures we have seen, however, focus on statistical compression, which is insensitive to this kind of compressibility. New text indexes building on other compression principles are being designed, although they are generally slower than the classical ones and still do not reach the exact entropy bounds as current statistical methods do.
Finally, we consider secondary memory. Compact data structures use less space than classical ones and thus may fit in smaller and faster memories. However, they have generally less locality of reference, thus they are slower than classical structures when competing in the same level of the memory hierarchy. This is particularly relevant when the datasets are huge and the structures must operate on disk, where the space usage is not so important and locality of reference is crucial.
- Type
- Chapter
- Information
- Compact Data StructuresA Practical Approach, pp. 501 - 548Publisher: Cambridge University PressPrint publication year: 2016
References
and (1999). Geometric range searching and its relatives. In Advances in Discrete and Computational Geometry, volume 223 of Contemporary Mathematics, pages 1–56. AMS Press.
, , , , , , and (2012). A comparison of index-based Lempel-Ziv LZ77 factorization algorithms. ACMComputing Surveys, 45(1), article 5.Google Scholar
, , , , , , and (2002). Efficient tree layout in a multilevel memory hierarchy. CoRR, cs/0211010v2. http:// arxiv.org/abs/cs.DS/0211010.
(2002). External memory data structures. In Handbook of Massive Data Sets, chapter 9, pages 313–357. Kluwer Academic Publishers.
, , , and (2005). Cache-oblivious planar orthogonal range searching and counting. In Proc. 21st ACMSymposium on Computational Geometry (SoCG), pages 160–169.Google Scholar
and , (2007). A Lempel-Ziv text index on secondary storage. In Proc. 18th Annual Symposium on Combinatorial Pattern Matching (CPM), LNCS 4580, pages 83–94.Google Scholar
, , and (2012). Stronger Lempel-Ziv based compressed text indexing. Algorithmica, 62(1), 54–101.Google Scholar
, , and (1996). Hierarchies of indices for text searching. Information Systems, 21(6), 497–514.Google Scholar
, , , , and (2014). Relative FM-indexes. In Proc. 21st International Symposium on String Processing and Information Retrieval (SPIRE), LNCS 8799, pages 52–64.Google Scholar
, , and (2015a). Access, rank, select in grammar-compressed strings. In Proc. 23rd Annual European Symposium on Algorithms (ESA), LNCS 9294, pages 142– 154.Google Scholar
, , , , and (2016). Practical combinations of repetition-aware data structures. CoRR, abs/1604.06002. http://arxiv.org/abs/1604.06002.
, , , , and (2015b). Composite repetitionaware data structures. In Proc. 26th Annual Symposium on Combinatorial Pattern Matching (CPM), LNCS 9133, pages 26–39.Google Scholar
, , , , , , and (2015c). Queries on LZ-bounded encodings. In Proc. 25th Data Compression Conference (DCC), pages 83–92.Google Scholar
, , and (1990). Text Compression. Prentice Hall.
, , , , , and (2015). Random access to grammar-compressed strings and trees. SIAM Journal on Computing, 44(3), 513–539.Google Scholar
, , , , and (1987). Complete inverted files for efficient text retrieval and analysis. Journal of the ACM, 34(3), 578–595.Google Scholar
and (2006). Cache-oblivious string dictionaries. In Proc. 17th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 581–590.Google Scholar
, , , , , , and (2005). The smallest grammar problem. IEEE Transactions on Information Theory, 51(7), 2554–2576.Google Scholar
, , and (2008). Lempel-Ziv factorization using less time & space. Mathematics in Computer Science, 1, 605–623.Google Scholar
, , , , , and (1995). External-memory graph algorithms. In Proc. 6th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 139–149.Google Scholar
, , , , and (2015). Geometric BWT: compressed text indexing via sparse suffixes and range searching. Algorithmica, 71(2), 258–278.Google Scholar
, , , and (2010). I/O-efficient compressed text indexes: From theory to practice. In Proc. 20th Data Compression Conference (DCC), pages 426–434.Google Scholar
and (1996). Efficient suffix trees on secondary storage. In Proc. 7th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 383–391.Google Scholar
and (2012). Improved grammar-based compressed indexes. In Proc. 19th International Symposium on String Processing and Information Retrieval (SPIRE), LNCS 7608, pages 180–192.Google Scholar
, , , and , (2010). Compresse. q-gram indexing for highly repetitive biological sequences. In Proc. 10th IEEE Conference on Bioinformatics and Bioengineering (BIBE), pages 86–91.Google Scholar
, , , and , (2016). Universal indexes for highly repetitive document collections. Information Systems, 61, 1–23.Google Scholar
and (1996). A time and space efficient data structure for string searching on large texts. Information Processing Letters, 58(5), 217–222.Google Scholar
, , , and (2014). Encoding range minima and range top-2 queries. Philosophical Transactions of the Royal Society A, 372(20130131).Google Scholar
, , and (2015). Worst-case optimal tree layout in external memory. Algorithmica, 72(2), 369–378.Google Scholar
and (2011). Robust relative compression of genomes with random access. Bioinformatics, 27, 2979–2986.Google Scholar
, , and (2012). Succinct and I/O efficient data structures for traversal in trees. Algorithmica, 63(1–2), 201–223.Google Scholar
, , , and (2016). I/O-efficient path traversal in succinct planar graphs. Algorithmica. Early view, DOI 10.1007/s00453-015-0086-7.
, , , and (2014). Fast relative Lempel-Ziv self-index for similar sequences. Theoretical Computer Science, 532, 14–30.Google Scholar
and (1995). String matching in Lempel-Ziv compressed strings. In Proc. 27th ACM Symposium on Theory of Computing (STOC), pages 703–712.Google Scholar
and (2014). A uniform paradigm to succinctly encode various families of trees. Algorithmica, 68(1), 16–40.Google Scholar
, , , and (2014). Relative Lempel-Ziv with constant-time random access. In Proc. 21st International Symposium on String Processing and Information Retrieval (SPIRE), LNCS 8799, pages 13–17.Google Scholar
, and (1999). The string B-tree:Anewdata structure for string search in external memory and its applications. Journal of the ACM, 46(2), 236–280.Google Scholar
, , , and , (2007). Compressed representations of sequences and full-text indexes. ACM Transactions on Algorithms, 3(2), article 20.Google Scholar
(2011). Combined data structure for previous- and next-smaller-values. Theoretical Computer Science, 412(22), 2451–2456.Google Scholar
and (2011). Space-efficient preprocessing schemes for range minimum queries on static arrays. SIAM Journal on Computing, 40(2), 465–492.Google Scholar
, , and , (2009). Faster entropy-bounded compressed suffix trees. Theoretical Computer Science, 410(51), 5354–5364.Google Scholar
, , and (2015). Lempel Ziv computation in small space (LZ-CISS). In Proc. 26th Annual Symposium on Combinatorial Pattern Matching (CPM), LNCS 9133, pages 172– 184.Google Scholar
and (2015). Searching and indexing genomic databases via kernelization. Frontiers in Bioengineering and Biotechnology, 3(12).Google Scholar
, , , , and (2012). A faster grammarbased self-index. In Proc. 6th International Conference on Language and Automata Theory and Applications (LATA), LNCS 7183, pages 240–251.Google Scholar
, , and (2014a). Block graphs in practice. In Proc. 2nd International Conference on Algorithms for Big Data (ICABD), pages 30–36.Google Scholar
, Gawrychowski, P., , , and (2014b). LZ77-based self-indexing with faster pattern matching. In Proc. 11th Latin American Theoretical Informatics Symposium (LATIN), LNCS 8392, pages 731–742.Google Scholar
, , and (2015a). Approximate pattern matching in LZ77- compressed texts. Journal of Discrete Algorithms, 32, 64–68.Google Scholar
, , , and (2015b). Relative compressed suffix trees. CoRR, abs/1508.02550. http://arxiv.org/abs/1508.02550.
and (2015a). Encodings of range maximum-sum segment queries and applications. In Proc. 26th Annual Symposium on Combinatorial Pattern Matching (CPM), LNCS 9133, pages 196–206.Google Scholar
and (2015b). Optimal encodings for range top-k, selection, and min-max. In Proc. 42nd International Colloquium on Automata, Languages, and Programming (ICALP), Part I, LNCS 9134, pages 593–604.Google Scholar
, , and (2006). Succinct ordinal trees with level-ancestor queries. ACM Transactions on Algorithms, 2(4), 510–534.Google Scholar
, , , , and (2014). Large-scale pattern search using reduced-space on-disk suffix arrays. IEEE Transactions on Knowledge and Data Engineering, 26(8), 1918–1931.Google Scholar
, , , , , and (2016). Encoding 2-D range maximum queries. Theoretical Computer Science, 609, 316–327.Google Scholar
and , (2009). A compressed text index on secondary memory. Journal of Combinatorial Mathematics and Combinatorial Computing, 71, 127–154.Google Scholar
, , and (2014). Locally compressed suffix arrays. ACM Journal of Experimental Algorithmics, 19(1), article 1.Google Scholar
and (2013). Simpler and faster Lempel Ziv factorization. In Proc. 23rd Data Compression Conference (DCC), pages 133–142.Google Scholar
and (2014). Space efficient linear time Lempel-Ziv factorization for small alphabets. In Proc. 24th Data Compression Conference (DCC), pages 163–172.Google Scholar
and (1999). Efficient cross-trees for external memory. In External Memory Algorithms and Visualization, DIMACS Series in Discrete Mathematics and Theoretical Computer Science. AMS Press.
, , , , and (2013). Encodings for range selection and top-k queries. In Proc. 21st Annual European Symposium on Algorithms (ESA), LNCS 8125, pages 553–564.Google Scholar
, , and (2012). Succinct ordinal trees based on tree covering. ACM Transactions on Algorithms, 8(4), article 42.Google Scholar
, , and (2006). Ordered pattern matching: Towards full-text retrieval. Technical Report TR-06-008, Purdue University.
, , , and (2009). On entropy-compressed text indexing in external memory. In Proc. 16th International Symposium on String Processing and Information Retrieval (SPIRE), LNCS 5721, pages 75–89.Google Scholar
, , , , and (2010). Indexing similar DNA sequences. In Proc. 6th International Conference on Algorithmic Aspects in Information and Management (AAIM), LNCS 6124, pages 180–190.Google Scholar
, , and (2003). An external memory data structure for shortest path queries. Discrete Applied Mathematics, 126, 55–82.Google Scholar
, , and (2015). Linked dynamic tries with applications to LZcompression in sublinear time and space. Algorithmica, 71(4), 969–988.Google Scholar
(2015). Approximation of grammar-based compression via recompression. Theoretical Computer Science, 592, 115–134.Google Scholar
(2016). A really simple approximation of smallest grammar. Theoretical Computer Science, 616, 141–150.Google Scholar
and (2015). Simultaneous encodings for range and next/previous larger/smaller value queries. In Proc. 21st International Conference on Computing and Combinatorics (COCOON), LNCS 9198, pages 648–660.Google Scholar
and (2011). Range selection and median: Tight cell probe lower bounds and adaptive data structures. In Proc. 22nd Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 805–813.Google Scholar
and (1996). Lempel-Ziv parsing and sublinear-size index structures for string matching. In Proc. 3rd South American Workshop on String Processing (WSP), pages 141–155.Google Scholar
, , and (2013a). Lightweight Lempel-Ziv parsing. In Proc. 12th International Symposium on Experimental Algorithms (SEA), pages 139–150.Google Scholar
, , and (2013b). Linear time Lempel-Ziv factorization: Simple, fast, small. In Proc. 24th Annual Symposium on Combinatorial Pattern Matching (CPM), LNCS 7922, pages 189–200.Google Scholar
, , and (2014). Lempel-Ziv parsing in external memory. In Proc. 24th Data Compression Conference (DCC), pages 153–162.Google Scholar
and (2013). Lempel-Ziv factorization: Simple, fast, practical. In Proc. 15th Workshop on Algorithm Engineering and Experiments (ALENEX), pages 103–112.Google Scholar
and (2000). Grammar-based codes: A new class of universal lossless source codes. IEEE Transactions on Information Theory, 46(3), 737–754.Google Scholar
and (2016). Lempel-Ziv computation in compressed space (LZ-CICS). In Proc. 26th Data Compression Conference (DCC), pages 3–12.Google Scholar
and , (1999). Compression of low entropy strings with Lempel-Ziv algorithms. SIAM Journal on Computing, 29(3), 893–911.Google Scholar
and , (2013). On compressing and indexing repetitive sequences. Theoretical Computer Science, 483, 115–133.Google Scholar
, , and (2010). Relative Lempel-Ziv compression of genomes for large-scale storage and retrieval. In Proc. 17th International Symposium on String Processing and Information Retrieval (SPIRE), LNCS 6393, pages 201–206.Google Scholar
, , and (2011). Reference sequence construction for relative compression of genomes. In Proc. 18th International Symposium on String Processing and Information Retrieval (SPIRE), LNCS 7024, pages 420–425.Google Scholar
, , , and (2012). Iterative dictionary construction for compression of large DNA data sets. IEEE/ACM Transactions on Computational Biology and Bioinformatics, 9, 137–149.Google Scholar
and (2000). Off-line dictionary-based compression. Proceedings of the IEEE, 88(11), 1722–1732.Google Scholar
and (1976). On the complexity of finite sequences. IEEE Transactions on Information Theory, 22(1), 75–81.Google Scholar
(2003). Compact suffix array – A space-efficient full-text index. Fundamenta Informaticae, 56(1-2), 191–210.Google Scholar
(2008). Personal communication.
and , (2004). Compressed compact suffix arrays. In Proc. 15th Annual Symposium on Combinatorial Pattern Matching (CPM), LNCS 3109, pages 420–433.Google Scholar
and , (2005). Succinct suffix arrays based on run-length encoding. Nordic Journal of Computing, 12(1), 40–66.Google Scholar
,, and (2004).Advantages of backward searching – efficient secondary memory and distributed implementation of compressed suffix arrays. In Proc. 15th Annual International Symposium on Algorithms and Computation (ISAAC), LNCS 3341, pages 681–692.Google Scholar
, , , and (2010). Storage and retrieval of highly repetitive sequence collections. Journal of Computational Biology, 17(3), 281–308.Google Scholar
, , and (2012). An online algorithm for lightweight grammarbased compression. Algorithms, 5(2), 214–235.Google Scholar
, , , and (2013a). ESP-index: A compressed index based on edit-sensitive parsing. Journal of Discrete Algorithms, 18, 100–112.Google Scholar
, , , and (2013b). Fully-online grammar compression. In Proc. 20th International Symposium on String Processing and Information Retrieval (SPIRE), pages 218–229.Google Scholar
, , and Sinha, R. (2009). Reducing space requirements for disk resident suffix arrays. In Proc. 14th International Conference on Database Systems for Advanced Applications (DASFAA), pages 730–744.Google Scholar
, , and (2001). Representing dynamic binary trees succinctly. In Proc. 12th Annual ACM-SIAM Symposium on Discrete Algorithm (SODA), pages 529–536.Google Scholar
and (2004). Simple implementation of String B-trees. In Proc. 11th International Symposium on String Processing and Information Retrieval (SPIRE), LNCS 3246, pages 214–215.Google Scholar
, , , , , , and (2013a). Suffix array of alignment: A practical index for similar data. In Proc. 20th International Symposium on String Processing and Information Retrieval (SPIRE), LNCS 8214, pages 243–254.Google Scholar
, , , , , , and (2013b). Suffix tree of alignment: An efficient index for similar data. In Proc. 24th International Workshop on Combinatorial Algorithms (IWOCA), LNCS 8288, pages 337–348.Google Scholar
, (2004). Indexing text using the Ziv-Lempel trie. Journal of Discrete Algorithms, 2(1), 87–114.Google Scholar
, and (2016). Faster compressed suffix trees for repetitive text collections. Journal of Experimental Algorithmics, 21(1), article 1.8.Google Scholar
, and (2016). Optimal encodings for range majority queries. Algorithmica, 74(3), 1082–1098.Google Scholar
, , and (2014). Asymptotically optimal encodings for range selection. In Proc. 34th Annual Conference on Foundations of Software Technology and Theoretical Computer Science (FSTTCS), pages 291–302.Google Scholar
, , and (1994). Compression by induction of hierarchical grammars. In Proc. 4th Data Compression Conference (DCC), pages 244–253.Google Scholar
and (2015). Encoding nearest largest values. In Proc. 26th Annual Symposium on Combinatorial Pattern Matching (CPM), LNCS 9133, pages 385–395.Google Scholar
, , , , and (2015). Dynamic index, LZ factorization, and LCE queries in compressed space. CoRR, abs/1504.06954. http://arxiv.org/abs/1504.06954.
and (2011). Lempel-Ziv factorization revisited. In Proc. 22nd Annual Symposium on Combinatorial Pattern Matching (CPM), LNCS 6661, pages 15–26.Google Scholar
and (2016). Space-efficient substring occurrence estimation. Algorithmica, 74(1), 65–90.Google Scholar
and (2006). Time-space trade-offs for predecessor search. In Proc. 38th Annual ACM Symposium on Theory of Computing (STOC), pages 232–240.Google Scholar
and (2016). Computing LZ77 in run-compressed space. In Proc. 26th Data Compression Conference (DCC), pages 23–32.Google Scholar
(2015). Encoding data structures. In Proc. 9th International Workshop on Algorithms and Computation (WALCOM), LNCS 8973, pages 1–7.Google Scholar
, , and (1981). Linear algorithm for data compression via string matching. Journal of the ACM, 28(1), 16–24.Google Scholar
and L. (2008). A compressed self-index using a Ziv-Lempel dictionary. Information Retrieval, 11(4), 359–388.Google Scholar
(2003). Application of Lempel-Ziv factorization to the approximation of grammar-based compression. Theoretical Computer Science, 302(1–3), 211–222.Google Scholar
(2005). A fully linear-time approximation algorithm for grammar-based compression. Journal of Discrete Algorithms, 3(2-4), 416–430.Google Scholar
(2006). Foundations of Multidimensional and Metric Data Structures. Morgan Kaufmann.
(1994). On the Ziv-Lempel proof and related topics. Proceedings of the IEEE, 82, 866–871.Google Scholar
, , , and (2008). Improving suffix array locality for fast pattern matching on disk. In Proc. ACM International Conference on Management of Data (SIGMOD), pages 661–672.Google Scholar
(2013). Array range queries. In Space-Efficient Data Structures, Streams, and Algorithms – Papers in Honor of J. Ian Munro on the Occasion of His 66th Birthday, LNCS 8066, pages 333–350. Springer.Google Scholar
(1977). NP-completeness results concerning data compression. Technical Report 234, Department of Electrical Engineering and Computer Science, Princeton University.
and (1982). Data compression via textual substitution. Journal of the ACM, 29(4), 928–951.Google Scholar
and (1995). The P-range tree: A new data structure for range searching in secondary memory. In Proc. 6th Annual ACM-SIAM Symposium on Discrete Algorithms (SODA), pages 378–387.Google Scholar
(1993). A generalized suffix tree and its (un)expected asymptotic behaviors. SIAM Journal on Computing, 22(6), 1176–1198.Google Scholar
(2008). Algorithms and Data Structures for External Memory. Now Publishers.
and (1994). The sliding-window Lempel-Ziv algorithm is asymptotically optimal. Proceedings of the IEEE, 82, 872–877.Google Scholar
, , , , and (2014). Faster compact on-line Lempel- Ziv factorization. In Proc. 31st International Symposium on Theoretical Aspects of Computer Science (STACS), LIPIcs 25, pages 675–686.Google Scholar
, , , , and (2013). Efficient direct search on compressed genomic data. In Proc. 29th IEEE International Conference on Data Engineering (ICDE), pages 961–972.Google Scholar
and (1977). A universal algorithm for sequential data compression. IEEE Transactions on Information Theory, 23(3), 337–343.Google Scholar
and (1978). Compression of individual sequences via variable length coding. IEEE Transactions on Information Theory, 24(5), 530–536.Google Scholar