In the preceding chapter, we saw how central reflection and martingale principles are to a generalized theory of learning. The purpose of this chapter is to examine those principles in more detail. Reflection requires one's future beliefs to cohere with one's prior beliefs. There are many examples of cases where this would not seem advisable.What these examples have in common is that factors other than learning new information have an influence on how beliefs change. If we restrict belief revisions to learning events, the reflection principle can be derived as a requirement of rational belief change in three different ways. The first one, based on dynamic Dutch book arguments, is well known. The other two approaches are new. According to one, reflection principles can be derived within an accuracy framework, in which degrees of belief are regarded as best estimates of truth values. The third approach starts with the value of knowledge theorem of decision theory. The value of knowledge theorem says that you never expect to make worse decisions after having conditioned your prior probabilities on some new information: typically, new information is valuable. The same should be true for learning in general, not just for Bayesian conditioning. Reflection principles are an immediate consequence of requiring that the value of knowledge property holds for generalized learning.

Probabilities of Future Probabilities

Recall the black box learning situation introduced in the previous chapter. You are about to change your degree of belief for some proposition A in response to an unspecified learning experience.

This chapter applies the ideas developed in the preceding chapter to a class of bounded resource learning procedures known as payoff-based models. Payoff-based models are alternatives to classical Bayesian models that reduce the complexity of a learning situation by disregarding information about states of the world. I am going to focus on one particular payoff-based model, the “basic model of reinforcement learning,” which captures in a precise and mathematically elegant way the idea that acts which are deemed more successful (according to some specific criterion) are more likely to be chosen.

What we are going to see is that the basicmodel can be derived from certain symmetry principles, analogous to the derivation of Carnap's family of inductive methods. Studying the symmetries involved in this derivation leads into a corner of decision theory that is relatively unknown in philosophy. Duncan Luce, in the late 1950s, introduced a thoroughly probabilistic theory of individual choice behavior in which preferences are replaced by choice probabilities. A basic constraint on choice probabilities, known as “Luce's choice axiom,” together with the theory of commutative learning operators, provides us with the fundamental principles governing the basic model of reinforcement learning.

Our exploration of the basic model does not, of course, exhaust the study of payoff-based and other learning models. I indicate some other possible models throughout the chapter and in the appendices. The main conclusion is that learning procedures that stay within a broadly probabilistic framework often arise from symmetry principles in a way that is analogous to Bayesian models.

One of the main criticisms of Carnap's inductive logic that Hilary Putnam has raised – alluded to in the epigraph – is that it fails in situations where inductive inference ought to go beyond relative frequencies. It is a little ironic that Carnap and his collaborators could have immediately countered this criticism if they had been more familiar with the work of Bruno de Finetti, who had introduced a formal framework that could be used for solving Putnam's problem already in the late 1930s. De Finetti's central innovation was to use symmetries that generalize exchangeability to various notions of partial exchangeability for inductive inference of patterns.

The goal of this chapter is to show how generalized symmetries can be used to overcome the inherent limitations of order invariant learning models, such as the Johnson–Carnap continuum of inductive methods or the basic model of reinforcement learning. What we shall see is that learning procedures can be modified so as to be able to recognize in principle any finite pattern.

Taking Turns

Order invariant learning rules collapse when confronted with the problem of learning how to take turns. Taking turns is important whenever a learning environment is periodic.

This appendix shows how to apply the work of A. A. J. Marley on commutative learning operators to reinforcement learning. Marley works with a slightly more general set of axioms than the one I use here, which is tailored toward the basic model of reinforcement learning.

Abstract Families

Let's consider the sequences of propensities one for each act. They arise from sequences of choice probabilities that at every stage satisfy Luce's choice axiom. Let be the range of values the random variables, can take on, and let be In many applications, will just be the set of nonnegative real numbers. Let be the set of outcomes, which can often be identified with a subset of the reals.

A learning operator L maps pairs of propensities and outcomes in to. If x is an alternative's present propensity, then L(x, a) is its new propensity if choosing that alternative has led to outcome a. This gives rise to a family of learning operators; for each, can be viewed as an operator from to. We assume that there is a unit element with

The triple is called an abstract family. An abstract family is quasi-additive if there exists a function and a function such that for each x, y in

We say that the process given by the sequences of propensities and the sequence of choices of acts is a reinforcement learning process if there exists an abstract family such that for all n whenever

where e is the unit element of. If such a family is quasi-abstract, then

Marley's Theorem

Some of Marley's principles tell us when an abstract family fits the learning process we are interested in. Let's start by introducing the relevant concepts.

Definition 1An abstract family is strictly monotonic if for all x, y in and each a in

Strict monotonicity says that learning is stable; an outcome has the same effect on how propensities are ordered across all propensity levels.

Learning models usually feature an individual agent who responds in some way to new information. They are monological, in the sense that they ignore that learning often takes place in a social context. The learning models discussed in previous chapters are no exception. In the last two chapters of this book I wish to pursue two issues that are relevant for extending my approach to social epistemology.

Both topics have to do with epistemic disagreement. Disagreement is ubiquitous in many areas of our lives. It is common in cutting-edge science, economics, business, politics, religion, and philosophy, not to mention the many ordinary disagreements we all have to deal with day to day. Many of our disagreements have epistemic aspects. For instance, divergent economic policies may be based on different assessments of data and models. What is the relationship between epistemic disagreements and rational learning? To what extent are divergent opinions compatible with all agents being epistemically rational? Finding answers to these questions is of crucial importance.

This chapter focuses on learning from others. The opinions of other agents who might disagree with you are taken as evidence that can cause you to change your beliefs. We will see that this learning situation requires no substantially new solution. Radical probabilism provides us with all the resources to model updating on the opinions of other agents. The most important part of this chapter is a “rational reconstruction” of a rule for merging the opinions of agents which has been much discussed in the epistemological literature on peer disagreement. This rule, which is known as straight averaging, combines the opinions of a group of agents additively by assigning each of them equal weights. I will show that something close to straight averaging emerges from combing a principle of Carnapian inductive logic with the theory of higher-order probabilities.

So far, we have considered learning models that operate within fixed conceptual frameworks. Fictitious play assumes that states, acts, and outcomes are known; reinforcement learning assumes the same for acts and outcomes. In many situations this is implausible, as Bayes’ friend and curator Richard Price has observed in his reflection on “a person just brought forth into this world.” One may not know the basic constituents of a new environment; accordingly, there may be no fixed conceptual framework for learning.

While this observation is not a knockout criticism of the learning models studied in previous chapters, it does put the spotlight on one of their inherent limitations. In this chapter, we will investigate some ways to overcome that limitation. To set the stage, I will give some context to the question of how much knowledge a learning model presupposes by discussing Savage's distinction between small worlds and large worlds in the context of learning. In a small world, the structure of an epistemic situation is fully known; in a large world, one does not know or anticipate every aspect of the epistemic situation that might be relevant. The distinction between large and small worlds will add yet another layer to our running discussion of bounded rationality.

The second section of this chapter looks at models based on flexible conceptual frameworks that take the epistemic incompleteness of a learning situation into account. They do so by keeping the learning procedure open to conceptual changes.

In Chapter 1, we introduced the two main aspects of Bayesian rational learning: dynamic consistency and symmetry. The preceding chapters focused on symmetries. In particular, we have seen that learning models other than Bayesian conditioning agree that updating on new information should be consistent with one's overall inductive assumptions about the learning situation. In this chapter we return to the issue of dynamic consistency. Recall that Bayesian conditioning is the only dynamically consistent rule for updating probabilities in a special, and particularly important, class of learning situations in which an agent learns the truth of a factual proposition. The basic rationale for dynamic consistency is that an agent's probabilities are best estimates prior and posterior to the learning experience only if they cohere with one another.

This chapter explores dynamic consistency in the context of learning models of bounded rationality. Since these models depart rather sharply from Bayesian conditioning, it is not immediately clear how, or even whether, they can be dynamically consistent. The relevant insights come from Richard Jeffrey's epistemological program of radical probabilism, which holds that Bayesian conditioning is just one among many legitimate forms of learning. After introducing Jeffrey's main ideas, we will see that his epistemology provides a large enough umbrella to include the probabilistic models of learning we have encountered in the preceding chapters, and many more. Two principles of radical probabilism, in particular, will assume a decisive role: Bas van Fraassen's reflection principle and its generalization, the martingale principle, which is due to Brian Skyrms. The two principles extend dynamic consistency to generalized learning processes and thereby allow us to say when such a process updates consistently on new information, even if the content of the information cannot be expressed as an observational proposition.

This chapter is a short introduction to the philosophy of inductive inference. After motivating the issues at stake, I'm going to focus on the two ideas that will be developed in this book: consistency and symmetry.

Consistency is a minimal requirement for rational beliefs. It comes in two forms: static consistency guarantees that one's degrees of beliefs are not self-contradictory, and dynamic consistency requires that new information is incorporated consistently into one's system of beliefs. I am not going to present consistency arguments in full detail; my goal is, rather, to give a concise account of the ideas that underlie the standard theory of probabilistic learning, known as Bayesian conditioning or conditionalization, in order to set the stage for generalizing these ideas in subsequent chapters. Bayesian conditioning provides the basic framework for rational learning from factual propositions, but it does not always give rise to tractable models of inductive inference. In practice, nontrivial inductive inference requires degrees of beliefs to exhibit some kind of symmetry. Symmetries are useful because they simplify a domain of inquiry by distinguishing some of its features as invariant. In this chapter, we examine the most famous probabilistic symmetry, which is known as exchangeability and was studied extensively by Bruno de Finetti in his work on inductive inference.

The kind of probabilism I advocate in this book is very undogmatic in that it typically allows for a range of admissible beliefs in response to the same information. The charge of excessive subjectivity is often brought against such a view. This charge is particularly pressing when the rationality of science is at stake: after all, the emergence of a consensus seems to be one consequence of what it means to respond rationally to scientific evidence. Does radical probabilism have anything to offer that would account for this feature of scientific rationality?

The previous chapter explored learning from other agents. As we have seen, this leads to a consensus only under special conditions, which express an initial agreement among the agents about the structure of the epistemic situation. That agreement is required at some levels in order to get agreement at others is a theme we are going to encounter again in this chapter, in which we set aside learning from others and focus instead on situations where agents revise their opinions based on the same information.

In the epigraph to this chapter, Peirce expresses an optimistic view about this process: learning from the same evidence shall overcome any initial disagreement. There is more than a grain of truth in this statement, as is shown by a number of theorems that I will discuss in this chapter. These theorems go a long way toward showing that probabilistic learning can lead to a rational consensus. However, there are also limits to this. The question of when Jeffrey conditioning leads to a consensus is especially interesting in this regard; we will see that considering Jeffrey conditioning helps reveal a dependence between long-run agreement and whether evidence is solid or soft.