5 - Information gain
Published online by Cambridge University Press: 05 July 2012
Summary
Describing objects by features is a very common thing to do. For example, many decision support systems use a tree-like representation of cases, where every branch in the tree corresponds to a feature and its observed value. But which features can be used to model a certain concept? What is the shortest and most meaningful rule with which we can describe a distinct set of objects using our knowledge?
In the previous chapter we saw how similarity measures can be used to group objects into (hopefully) meaningful clusters. Given an information system ℑ, we now want to describe a feature's utility with respect to a given object's classification. Relationally speaking, we need to recursively apply those features fi ϵ F that generate a partition on U that is similar to U/Rt to learn a compressing classifier this way. It appears to be a good idea to start with a feature that appears to be the most “similar” to t. A feature being quite similar to the target function can be assumed to carry relevant information with respect to t. And this leads us to the information-theoretic notion of entropy.
Information gain driven classifier learning
While clustering tries to find hierarchies of groups of objects, so-called decision trees represent a hierarchy of feature-induced partitions. Unlike (unsupervised) similarity measures in clustering, one uses a target-specific information measure called entropy.
People often try to explain Shannon and Weaver's (1949) information-theoretic measure of entropy by the laws of entropy in thermodynamics.
- Type
- Chapter
- Information
- Relational Knowledge Discovery , pp. 92 - 120Publisher: Cambridge University PressPrint publication year: 2012