Skip to main content Accessibility help
×
Home
Hostname: page-component-6c8bd87754-h9sqt Total loading time: 0.332 Render date: 2022-01-16T10:27:13.042Z Has data issue: true Feature Flags: { "shouldUseShareProductTool": true, "shouldUseHypothesis": true, "isUnsiloEnabled": true, "metricsAbstractViews": false, "figures": true, "newCiteModal": false, "newCitedByModal": true, "newEcommerce": true, "newUsageEvents": true }

3 - Finding Similar Items

Published online by Cambridge University Press:  05 December 2014

Jure Leskovec
Affiliation:
Stanford University, California
Anand Rajaraman
Affiliation:
Milliways Laboratories, California
Jeffrey David Ullman
Affiliation:
Stanford University, California
Get access

Summary

A fundamental data-mining problem is to examine data for “similar” items. We shall take up applications in Section 3.1, but an example would be looking at a collection of Web pages and finding near-duplicate pages. These pages could be plagiarisms, for example, or they could be mirrors that have almost the same content but differ in information about the host and about other mirrors.

We begin by phrasing the problem of similarity as one of finding sets with a relatively large intersection. We show how the problem of finding textually similar documents can be turned into such a set problem by the technique known as “shingling.” Then, we introduce a technique called “minhashing,” which compresses large sets in such a way that we can still deduce the similarity of the underlying sets from their compressed versions. Other techniques that work when the required degree of similarity is very high are covered in Section 3.9.

Another important problem that arises when we search for similar items of any kind is that there may be far too many pairs of items to test each pair for their degree of similarity, even if computing the similarity of any one pair can be made very easy. That concern motivates a technique called “locality-sensitive hashing,” for focusing our search on pairs that are most likely to be similar.

Finally, we explore notions of “similarity” that are not expressible as intersection of sets. This study leads us to consider the theory of distance measures in arbitrary spaces. It also motivates a general framework for locality-sensitive hashing that applies for other definitions of “similarity.”

Applications of Near-Neighbor Search

We shall focus initially on a particular notion of “similarity”: the similarity of sets by looking at the relative size of their intersection. This notion of similarity is called “Jaccard similarity,” and will be introduced in Section 3.1.1. We then examine some of the uses of finding similar sets.

Type
Chapter
Information
Publisher: Cambridge University Press
Print publication year: 2014

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

[1] A., Andoni and P., Indyk, “Near-optimal hashing algorithms for approximate nearest neighbor in high dimensions,” Comm. ACM 51:1, pp. 117–122, 2008.Google Scholar
[2] A.Z., Broder, “On the resemblance and containment of documents,” Proc. Compression and Complexity of Sequences, pp. 21–29, Positano Italy, 1997.Google Scholar
[3] A.Z., Broder, M., Charikar, A.M., Frieze, and M., Mitzenmacher, “Min-wise independent permutations,” ACM Symposium on Theory of Computing, pp. 327–336, 1998.Google Scholar
[4] M.S., Charikar, “Similarity estimation techniques from rounding algorithms,” ACM Symposium on Theory of Computing, pp. 380–388, 2002.Google Scholar
[5] S., Chaudhuri, V., Ganti, and R., Kaushik, “A primitive operator for similarity joins in data cleaning,” Proc. Intl. Conf. on Data Engineering, 2006.Google Scholar
[6] M., Datar, N., Immorlica, P., Indyk, and V.S., Mirrokni, “Locality-sensitive hashing scheme based on p-stable distributions,” Symposium on Computational Geometry pp. 253–262, 2004.Google Scholar
[7] A., Gionis, P., Indyk, and R., Motwani, “Similarity search in high dimensions via hashing,” Proc. Intl. Conf. on Very Large Databases, pp. 518–529, 1999.Google Scholar
[8] M., Henzinger, “Finding near-duplicate web pages: a large-scale evaluation of algorithms,” Proc. 29th SIGIR Conf., pp. 284–291, 2006.Google Scholar
[9] P., Indyk and R., Motwani. “Approximate nearest neighbor: towards removing the curse of dimensionality,” ACM Symposium on Theory of Computing, pp. 604–613, 1998.Google Scholar
[10] U., Manber, “Finding similar files in a large file system,” Proc. USENIX Conference, pp. 1–10, 1994.Google Scholar
[11] M., Theobald, J., Siddharth, and A., Paepcke, “SpotSigs: robust and efficient near duplicate detection in large web collections,” 31st Annual ACM SIGIR Conference, July, 2008, Singapore.Google Scholar
[12] C., Xiao, W., Wang, X., Lin, and J.X., Yu, “Efficient similarity joins for near duplicate detection,” Proc. WWW Conference, pp. 131–140, 2008.Google Scholar
4
Cited by

Send book to Kindle

To send this book to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle.

Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

Available formats
×

Send book to Dropbox

To send content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about sending content to Dropbox.

Available formats
×

Send book to Google Drive

To send content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about sending content to Google Drive.

Available formats
×