Hostname: page-component-7d684dbfc8-zgpz2 Total loading time: 0 Render date: 2023-10-02T01:25:11.473Z Has data issue: false Feature Flags: { "corePageComponentGetUserInfoFromSharedSession": true, "coreDisableEcommerce": false, "coreDisableSocialShare": false, "coreDisableEcommerceForArticlePurchase": false, "coreDisableEcommerceForBookPurchase": false, "coreDisableEcommerceForElementPurchase": false, "coreUseNewShare": true, "useRatesEcommerce": true } hasContentIssue false

# Appendix C - Marley's Axioms

Published online by Cambridge University Press:  25 October 2017

## Summary

This appendix shows how to apply the work of A. A. J. Marley on commutative learning operators to reinforcement learning. Marley works with a slightly more general set of axioms than the one I use here, which is tailored toward the basic model of reinforcement learning.

Abstract Families

Let's consider the sequences of propensities one for each act. They arise from sequences of choice probabilities that at every stage satisfy Luce's choice axiom. Let be the range of values the random variables, can take on, and let be In many applications, will just be the set of nonnegative real numbers. Let be the set of outcomes, which can often be identified with a subset of the reals.

A learning operator L maps pairs of propensities and outcomes in to. If x is an alternative's present propensity, then L(x, a) is its new propensity if choosing that alternative has led to outcome a. This gives rise to a family of learning operators; for each, can be viewed as an operator from to. We assume that there is a unit element with

The triple is called an abstract family. An abstract family is quasi-additive if there exists a function and a function such that for each x, y in

We say that the process given by the sequences of propensities and the sequence of choices of acts is a reinforcement learning process if there exists an abstract family such that for all n whenever

where e is the unit element of. If such a family is quasi-abstract, then

Marley's Theorem

Some of Marley's principles tell us when an abstract family fits the learning process we are interested in. Let's start by introducing the relevant concepts.

Definition 1An abstract family is strictly monotonic if for all x, y in and each a in

Strict monotonicity says that learning is stable; an outcome has the same effect on how propensities are ordered across all propensity levels.

Type
Chapter
Information
Publisher: Cambridge University Press
Print publication year: 2017

## Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

# Save book to Kindle

To save this book to your Kindle, first ensure coreplatform@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about saving to your Kindle.

Note you can select to save to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be saved to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

Find out more about the Kindle Personal Document Service.

• Marley's Axioms
• Book: The Probabilistic Foundations of Rational Learning
• Online publication: 25 October 2017
• Chapter DOI: https://doi.org/10.1017/9781316335789.013
Available formats
×

# Save book to Dropbox

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Dropbox.

• Marley's Axioms
• Book: The Probabilistic Foundations of Rational Learning
• Online publication: 25 October 2017
• Chapter DOI: https://doi.org/10.1017/9781316335789.013
Available formats
×

# Save book to Google Drive

To save content items to your account, please confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. Find out more about saving content to Google Drive.

• Marley's Axioms
• Book: The Probabilistic Foundations of Rational Learning
• Online publication: 25 October 2017
• Chapter DOI: https://doi.org/10.1017/9781316335789.013
Available formats
×