Skip to main content Accessibility help


  • Access
  • Open access


      • Send article to Kindle

        To send this article to your Kindle, first ensure is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the or variations. ‘’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Using Meta-Analysis for Large-Scale Ecosystem Service Valuation: Progress, Prospects, and Challenges
        Available formats

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Using Meta-Analysis for Large-Scale Ecosystem Service Valuation: Progress, Prospects, and Challenges
        Available formats

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Using Meta-Analysis for Large-Scale Ecosystem Service Valuation: Progress, Prospects, and Challenges
        Available formats
Export citation


This article discusses prospects and challenges related to the use of meta-regression models (MRMs) for ecosystem service benefit transfer, with an emphasis on validity criteria and post-estimation procedures given sparse attention in the ecosystem services literature. We illustrate these topics using a meta-analysis of willingness to pay for water quality changes that support aquatic ecosystem services and the application of this model to estimate water quality benefits under alternative riparian buffer restoration scenarios in New Hampshire's Great Bay Watershed. These illustrations highlight the advantages of MRM benefit transfers, together with the challenges and data needs encountered when quantifying ecosystem service values.


Submitted for the Special Issue associated with the USDA Workshop, Applications and Potential of Ecosystem Services Valuation within USDA – Advancing the Science.


The demand for information on ecosystem service values (ESVs), combined with a lack of time and resources required to conduct primary valuation studies, has led to common use of benefit transfer to quantify these values (Plummer 2009; Bateman et al. 2011a, 2011b; Ferrini, Schaafsma, and Bateman 2015; Johnston and Wainger 2015; Richardson et al. 2015; Johnston, Rolfe, and Zawojska 2018). Benefit transfer is defined as the use of research results from preexisting primary studies at one or more sites or policy contexts (called study sites) to predict welfare estimates such as willingness to pay (WTP) or related information for other, typically unstudied sites or policy contexts (called policy sites) (Johnston et al. 2015a). Among the different approaches to benefit transfer in the literature,1 there is emerging consensus over the advantages of methods that synthesize data from multiple sources, such as meta-analysis (e.g., Rosenberger and Phipps 2007; Boyle et al. 2009; Johnston and Rosenberger 2010; Kaul et al. 2013; Rolfe, Brouwer, and Johnston 2015; Johnston, Rolfe, and Zawojska 2018).

The most common applications of meta-analysis for ESV benefit transfer involve meta-regression models (MRMs). These models generate parametric functions that characterize the systematic influence of economic, ecological (or resource), beneficiary (population), and primary study attributes on comparable measures of value.2 The resulting functions are used to predict values at policy sites where original valuation studies have not been conducted. Explanatory variables in these MRMs can link directly to biophysical models and enable adjustments in ESV estimates for the attributes of ecosystems, regions, populations, and policy contexts. There are many examples of such MRMs applied to different types of environmental and ecosystem service improvements including water quality (Johnston, Besedin, and Stapler 2017), recreational fishing (Johnston et al. 2006), forest recreation (Zandersen and Tol 2009), and wetlands-provided ecosystem services (Brander et al. 2012), among many others.3

Despite this work, further advances in methods and understanding are required if meta-analyses are to be relied on for widespread ESV prediction, particularly within large-scale applications. The capacity of MRMs to support valid and reliable4 benefit transfers depends on multiple criteria, including the use of appropriate econometrics (Nelson and Kennedy 2009; Boyle and Wooldridge 2018) and specifications able to accommodate expected patterns such as the sensitivity of values to scope, scale, and spatial dimensions (Johnston, Besedin, and Stapler 2017; Johnston, Besedin, and Holland 2018; Johnston, Rolfe, and Zawojska 2018; Kling and Phaneuf 2018; Newbold et al. 2018a, 2018b; Smith 2018).5 ESV applications are often deficient in these areas (Bateman et al. 2011b; Richardson et al. 2015; Johnston, Rolfe, and Zawojska 2018). Invariance of transferred welfare estimates to dimensions such as these can signal a lack of construct validity (Johnston, Besedin, and Stapler 2017).6 The estimation of MRMs for ecosystem service valuation also requires robust primary study data for the type of values under study, with sufficient study reporting for metadata development (Loomis and Rosenberger 2006).

In addition, multiple procedures are required to predict and aggregate values for policy sites using MRM or other transfer functions. These procedures must address challenges such as (1) linking biophysical information on the scope and scale of environmental changes to MRM functions, (2) accounting for the extent of the market and other spatial dimensions of households and environmental changes, (3) addressing practical dissimilarities or gaps between MRM variables and measurable conditions at the policy site, (4) accounting for the systematic influence of methodological factors on value estimates, and (5) ensuring that the resulting predictions have theoretical and empirical properties required to ensure validity for the intended uses.

Concerns such as these are sometimes overlooked in the ecosystem services valuation literature, leading to many MRMs that—despite having superficially acceptable statistical performance—are poorly suited to credible benefit transfer applications. As noted by Johnston, Rolfe, and Zawojska (2018, p. 202), “Many applications [of meta-analysis in the ecosystem services] literature lack minimal properties necessary to promote valid and reliable [benefit transfer] estimates.” Even if the underlying MRM is suitable for this purpose, valid transfers also depend on the post-estimation procedures used to predict ESVs from the estimated benefit function. These issues are seldom discussed. The academic literature focuses primarily on estimation and interpretation of the underlying MRMs rather than the evaluation and use of these models for benefit transfer. The practical steps and challenges involved in benefit prediction and aggregation using these MRMs—or properties of the resulting value estimates—are not often considered in detail.

This article discusses prospects and challenges related to the use of MRMs for benefit transfer. We begin with a summary of key issues and challenges associated with the use of MRMs to forecast values for large-scale ESVs, including post-estimation issues given sparse attention in the literature. We illustrate these issues using a meta-analysis of WTP for water quality changes that support aquatic ecosystem services (Johnston, Besedin, and Stapler 2017) and the application of this model to estimate aggregate water quality benefits under alternative riparian buffer restoration scenarios in New Hampshire's Great Bay Watershed. The MRM is reviewed with respect to key validity criteria to highlight both advantages and limitations of the model. Benefit transfers from this MRM are compared for multiple scenarios, scopes, and scales of environmental improvement in the case study area, aggregated over different market extents.

The goal of this illustration is practical. Unlike publications that focus on MRM estimation and interpretation, here we emphasize often-unreported steps and choices associated with the use of MRM benefit functions for ESV prediction, as well as patterns in the resulting welfare forecasts. Attention is given to topics such as the consistency between variables used in published MRMs (typically reflecting data reported by primary studies in the metadata) and biophysical data of the type available to quantify ecosystem service changes. We also consider the properties and construct validity of the resulting value predictions. These illustrations highlight the advantages of MRM benefit transfers together with challenges and data needs.

Meta-Analysis Properties and Benefit Transfer Validity

The potential advantages of meta-analysis for benefit transfer are established in the literature (Rosenberger and Phipps 2007; Nelson and Kennedy 2009; Rosenberger and Johnston 2009; Johnston and Rosenberger 2010; Kaul et al. 2013; Rolfe, Brouwer, and Johnston 2015; Boyle and Wooldridge 2018; Johnston, Rolfe, and Zawojska 2018). Meta-regression analyses synthesize the results of multiple prior studies into a single set of parametric predictors that can be used within benefit transfer, providing a means to ground transfers in a broad base of prior information.7 The resulting function allows predicted values to be tailored to the needs of particular policy evaluations.8

There is emerging consensus over the advantages of methods that synthesize data from the literature in this way, including potential improvements in benefit transfer reliability—or reductions in generalization error (e.g., Rosenberger and Phipps 2007; Boyle et al. 2009; Johnston and Rosenberger 2010; Johnston and Thomassin 2010; Kaul et al. 2013; Rolfe, Brouwer, and Johnston 2015; Johnston, Rolfe, and Zawojska 2018). The capacity of MRMs to synthesize information across studies that vary across commodity, site, policy, and population factors can obviate the need to find one study that is a close match to the policy site across all dimensions (Stapler and Johnston 2009). Meta-analytic transfers can also reduce the risk of error caused by the use of a study site that differs from the policy site in influential ways or a primary study that suffers from measurement error, and can help diagnose and ameliorate the effect of selection biases in the literature (Stanley 2005; Hoehn 2006; Rosenberger and Johnston 2009; Boyle and Wooldridge 2018).

Despite these advantages, the ability of an MRM to provide valid and reliable predictions depends on multiple factors, including the procedures used for data synthesis, statistical analysis, value prediction, and benefit aggregation (Smith and Pattanayak 2002; Nelson and Kennedy 2009; Boyle et al. 2013; Boyle, Kaul, and Parmeter 2015; Nelson, 2015; Boyle and Wooldridge, 2018; Johnston, Rolfe, and Zawojska 2018). Considerations such as these are frequently overlooked in the ecosystem services literature, leading to transfers that violate conditions for construct validity in welfare estimates (Plummer 2009; Bateman et al. 2011b; Johnston and Wainger 2015; Richardson et al. 2015).

Prior reviews of MRMs in environmental economics find that many violate best practice guidelines for econometric analysis (Nelson and Kennedy 2009). Boyle and Wooldridge (2018) discuss additional issues related to metadata construction and MRM specification, as related to the validity and reliability of predictions. They distinguish between MRMs used to understand a body of empirical literature and those used to predict values. Among their key findings is that “no single estimated meta-equation is suitable for prediction in all benefit-transfer applications” (Boyle and Wooldridge 2018, p. 633). Care must be taken to construct metadata and specify equations to predict values grounded in a consistent theoretical definition and commodity type. Hence, “a one-size-fits-all use of an estimated meta-equation to support benefit transfers is likely not appropriate” (Boyle and Wooldridge 2018, p. 633). Kling and Phaneuf (2018), Newbold et al. (2018b), and Moeltner (2019) consider whether and when MRM specifications should be further restricted to impose theoretical properties such as adding-up on benefit transfers, in some cases reaching contrary conclusions.9

Among commonly acknowledged requirements for valid valuation via MRMs is at least a minimal degree of commodity and welfare consistency across metadata observations (Smith and Pattanayak 2002; Bergstrom and Taylor 2006; Loomis and Rosenberger 2006, Nelson and Kennedy 2009; Johnston and Rosenberger 2010; Boyle and Wooldridge 2018; Johnston, Rolfe, and Zawojska 2018). Moeltner and Rosenberger (2014, p. 470) characterize commodity consistency as a situation in which “basic commodities under consideration [within the metadata] must be essentially equivalent.” Welfare consistency implies parallel consistency in the theoretical welfare constructs measured across observations. Many ESV MRMs are estimated using primary study data sets that pool value metrics that have no such relationship, such as estimates of consumer value, producer value, and measures that do not reflect welfare-theoretic values of any type. Prior MRMs have also pooled value estimates linked to dissimilar goods and services such as recreation, flood control, fisheries production, carbon sequestration, raw material provision, nutrient cycling, water supply, existence, aesthetics, and others (Johnston, Rolfe, and Zawojska 2018). The lack of metadata consistency within such models raises validity concerns (Bergstrom and Taylor 2006)—neither theory nor economic intuition justifies the inclusion of such dissimilar welfare measures within the estimation of a single meta-analytic value function.10

Any remaining differences in commodity, site, or population characteristics that might influence value must be captured using right-hand side variables in the MRM. However, even similar environmental changes may be defined in different ways across the literature, and reconciling these measurements across studies is not always straightforward (Smith and Pattanayak 2002; Johnston et al. 2005; Van Houtven, Powers, and Pattanayak 2007; Rolfe, Brouwer, and Johnston 2015). This can lead to trade-offs between the number of studies (N) and independent variables (K) within an MRM (Moeltner, Boyle, and Paterson 2007, p. 252)—“should the researcher discard explanatory variables that are not common to all studies (thus preserve N at the cost of K) or discard observations that do not include all key regressors (thus preserve K at the cost of N)?” Accuracy can be diminished if the resulting variables are quantified in low-resolution or categorical terms (e.g., high, medium, and low) (Johnston et al. 2012; Johnston, Besedin, and Stapler 2017). Moreover, the usefulness of any valuation exercise is reduced if variables are defined in ways that cannot be directly linked to the quantitative biophysical, health, or engineering outputs used to quantify effects for policy analysis (Loomis and Rosenberger 2006; Johnston et al. 2012; Schultz et al. 2012; Boyd et al. 2016).

The validity of MRM benefit transfers further depends on the extent to which these transfers reflect theoretical patterns expected in welfare estimates (Kling and Phaneuf 2018). For example, microeconomic theory suggests that predictions of per household WTP for most ecosystem service changes should respond to the scope and scale of the change, as well as substitutes and spatial dimensions. For nonmarginal changes, diminishing marginal values are typically expected. Hence, linear “scaling-up” of value predictions is unlikely to produce valid estimates, particularly when scaled to a large degree (Bockstael et al. 2000; Bateman et al. 2011b; Brander et al. 2012; Johnston and Wainger 2015). Theory also suggests that most ESVs should be sensitive to household income, among other population attributes, although naïve income adjustments may not always improve transfers (Johnston and Duke 2010). Expectations such as these influence the type of MRM specifications required for validity.

A final set of questions and concerns—and those perhaps given least attention in the literature—relates to the post-estimation steps required to predict and aggregate values from a metafunction. These procedures can have a profound impact on benefit predictions. For example, as shown by Johnston, Besedin, and Ranson (2006) for the treatment of primary study methodological variables when predicting benefits, the use of ad hoc procedures can cause WTP predictions to vary more than 15-fold. This sensitivity can be reduced with best practices for post-estimation value prediction (Stapler and Johnston 2009; Boyle and Wooldridge 2018).

One often overlooked post-estimation concern relates to variable measurement conventions applied in the MRM (discussed previously), including those used to quantify the biophysical scope and scale of ecosystem service changes. First, one must assess whether the ecosystem service of interest at the policy site is a good match for the service(s) covered by the MRM. Assuming a good match exists, one must then link the relevant biophysical changes predicted at a policy site to ecosystem service measures used within the existing MRM and obtain any required data or biophysical model outputs.

For example, Rolfe, Brouwer, and Johnston (2015) meta-analyze per household WTP per kilometer of river in “good” health, where studies in the metadata define good health using different types of biophysical criteria. The usefulness of such results for benefit transfer requires a defensible and replicable way to predict this outcome (kilometers in good health) using biophysical information available to policy analysts. Even for a given benefit function, different analyst assumptions regarding biophysical measures or indices can lead to different benefit predictions. Walsh and Wheeler (2013, p. 81) demonstrate that different assumptions regarding the mathematical structure of water quality indices “can have a profound effect on benefits.” In addition, MRM variables are often defined in ways that are biophysically ambiguous, potentially exacerbating the challenge of linking measured policy effects to MRM benefit estimates.

A second post-estimation concern relates to the role of spatial dimensions in benefit estimation, including the assigned extent of the market for benefit prediction and aggregation. As noted by Bateman et al. (2006, 2011a), Schaafsma (2015), Johnston et al. (2017), and Johnston, Besedin, and Holland (2018), choices such as these can swamp the effects of other procedural choices in benefit transfer. A related challenge is that—depending on the MRM specification—average and aggregate benefits may vary depending on whether average benefits are predicted for one large market or multiple component submarkets. In such cases, is it sufficient to predict values assuming one constant “average” ecosystem service change and baseline over one large market area? Or, should values be predicted for smaller areas, reflecting the specific changes and baselines in those areas? If one chooses the latter approach, what is the appropriate size of the subareas to be considered? When values depend on spatial dimensions (Schaafsma 2015; Glenk et al. 2019), predicted aggregate values for average environmental changes everywhere will not, in general, be equal to the same values averaged over the population for spatially heterogeneous changes. Similar questions apply to other contextual considerations that vary over space such as income and substitutes.

A third consideration is whether the desired policy site ESV scenarios are suitable for valuation using the MRM in question. For example, would a particular scenario require values to be predicted outside the data range of the primary study metadata? With a few exceptions, the literature provides little guidance for these post-estimation procedures, leading to wide variation in applied practice. Compounding this challenge is a tendency of these procedures and assumptions to remain unreported, leading to lack of methodological transparency.

Illustrative Application: Predicting Water Quality Benefits from Buffer Restoration

This section illustrates the use of an applied MRM for ESV benefit transfer. Among the goals of this illustration are to (a) demonstrate the characteristics of the MRM required for valid benefit transfers, (b) clarify the post-estimation steps and choices required to generate and evaluate predictions using the model, and (c) discuss the advantages and disadvantages of the resulting benefit predictions. The intent is to clarify often-undocumented properties and practices that affect transfer value validity. The empirical case study analysis implements a benefit transfer to quantify per household and aggregate market WTP associated with water quality improvements of the type that could result from the restoration of vegetated buffers within New Hampshire's Great Bay watershed. We use an existing MRM for this purpose to enable focus on model evaluation and post-estimation benefit transfer, rather than on estimation of an original MRM.

Benefits are predicted using an application of the MRM estimated by Johnston, Besedin, and Stapler (2017). This model was designed to support benefit transfer for water quality improvements in U.S. water bodies including rivers, lakes, and estuaries and reflects the type of MRM used by the U.S. Environmental Protection Agency (U.S. EPA) for regulatory analysis (U.S. EPA 2009a, 2010, 2012, 2015; Griffths et al. 2012). The metadata were drawn from primary stated preference studies that estimated total (use and nonuse) per household WTP for water quality changes in water bodies that support ecosystem services including aquatic life, recreational uses (e.g., fishing, boating, and swimming), and nonuse values.11 This MRM and metadata have been used as the basis for multiple prior validity tests and analyses (e.g., Newbold et al. 2018b; Moeltner 2019), so that the underlying properties of the meta-analysis are well known. As such, this MRM provides an established foundation from which to demonstrate an applied benefit transfer.

Grounded in this MRM, we present the process used to apply the resulting transfer function, along with assumptions and implications for valid benefit transfer. To illustrate how transferred WTP estimates vary across different types of water bodies, we predict values for improvements to the Great Bay and two of its tributaries, the Squamscott River and Exeter River. Values for improvements to each of these areas are predicted independently, holding conditions at other areas constant. We consider various scenarios of water quality change for these water bodies, based on current baseline quality. We also consider values predicted and aggregated over different market areas. The goal is to consider the responsiveness of WTP estimates to factors such as scope, scale, extent of the market, and baseline condition. We also describe the water quality index (WQI) used to quantify water quality baselines and changes within the MRM, as well as how the index is defined as a function of water quality parameters.

The Meta-Regression Model

As described by Johnston, Besedin, and Stapler (2017), all observations in the original metadata were identified and coded following the guidelines of Stanley et al. (2013). To ensure welfare consistency, observations were restricted to U.S. studies that estimated total (use and nonuse) value, used established stated preference methods, reported comparable Hicksian WTP measures, and provided sufficient detail to verify that valuation methods met minimum quality standards. To ensure commodity consistency, studies were limited to those for which per household WTP estimates could be mapped to water quality changes measured on a standardized 100-point WQI that relates pollutant concentrations to water body suitability for human uses (Walsh and Wheeler 2013). The final metadata included 140 unique observations from 51 stated preference studies published between 1985 and 2013 (Table 1).

Table 1. Primary Studies in the Metadata (willingness to pay [WTP] is per household per year in 2007 USD)

Note: Complete citations to these referenced works are provided by Johnston, Besedin, and Stapler (2017).

The dependent variable in the MRM is the natural log of per household WTP for water quality improvements specified in each original study, with all values adjusted to 2007 USD (these values are subsequently updated to 2016 USD as discussed in Implementing the Benefit Transfer below). Independent variables expected to explain variation in this value measure (and included in the model) characterize (1) the geographic region and affected aquatic resources, (2) affected ecosystem services, (3) populations whose values were measured, (4) baseline resource condition and water quality change, (5) potential substitute resources and complementary land uses, and (6) the primary study methodology and year (Table 2). Emphasis was given to core economic and resource variables directly relevant to benefit transfer. To construct the data set, variables reported directly by primary studies were supplemented with data on study sites, beneficiary populations, and water bodies extracted from readily available national data sets.12

Table 2. Meta-Analysis Variable Descriptions and Mean Metadata Values

Among key variables in the MRM is that characterizing the scope (or size) of the water quality change, measured as a natural log (Ln_QualityChg). The model also includes a spatial index variable (Ln_RelativeSize) that characterizes the size of the affected water body measured using shoreline length in kilometers (geospatial scale), relative to the size of the sampled market area measured in square kilometers (market extent). This specification enables the marginal effect of water body size on WTP to decline as size of the sampled market area increases, and vice versa. The inclusion of market area in this composite index also provides a means to proxy for the effect of distance decay on value, as larger sampled market areas generally imply greater distances to affected water bodies, ceteris paribus (Johnston, Besedin, and Stapler 2017).

To characterize the scope of proportional effects on regional (potentially substitute) water bodies, the variable ProportionChg measures the proportion of water bodies (of the same hydrologic type) affected by the water quality change, within each state. Potential land use complements to water quality (specifically, the lack of complementary land uses) are characterized using Ln_PropAgLand, representing the (natural log of the) proportion the affected area with agricultural land use.13 The model also includes variables characterizing affected uses (or ecosystem services) mentioned in each primary study such as fishing and swimming (or nonuse only), as well as indicators identifying the region of the United States where changes occurred.

The MRM is estimated using unweighted random effects ordinary least squares with robust standard errors, allowing for cross-sectional correlation among observations from the same study (Nelson and Kennedy 2009).14 The model is specified as

(1)$$\bar{y}_{\,js} = \bar{x}_{\,js}{\rm \beta} + {\rm \varepsilon} _{\,js}, $$

where $\bar{y}_{js}$ is the welfare measure for observation s in study j (the natural log of mean or median WTP per household for the specified water quality change), and $\bar{x}_{js}$ is the vector of independent variables. This vector includes natural logs of household income (Ln_Income), water quality change and baseline (Ln_QualityChg; Ln_BaseQuality), relative geospatial scale (Ln_RelativeSize), and agricultural land proportion (Ln_PropAgLand). Other variables enter linearly.15 The vector β represents a conforming parameter vector. Following standard specifications, εjs = e js + u s, where u s represents a systematic, normally distributed, study-level random effect with E(u s) = 0 and Var(u s) = σu2; and e js is a standard iid (independent and identically distributed) estimation level error, distributed with a zero mean and constant variance σe2.

Results are shown in Table 3. Parameter estimates are jointly significant at p<0.01 (χ2 = 729.61, df = 24), with an R2 of 0.633. Signs of significant parameter estimates match those suggested by theory. For example, WTP is positively related to the scope of water quality change (Ln_QualityChg), household income (Ln_Income), improvements to larger water bodies and samples over smaller market extents (Ln_RelativeSize), and improvements that affect a larger proportion of surrounding waters (ProportionChg). WTP is negatively related to improvements in more agricultural areas (Ln_PropAgLand) and samples limited to nonusers (Non_Users). The functional form implies diminishing marginal utility with regard to scope and scale. Other properties of the underlying MRM are discussed by Johnston, Besedin, and Stapler (2017).

Table 3. Meta-Regression Results—Random Effects Model (Johnston, Besedin, and Stapler et al. 2017)

Note: *** p < 0.01, ** p < 0.05, * p < 0.10.

Implementing the Benefit Transfer

Estimating the MRM is only the first step in benefit transfer. Few articles discuss the procedures required to move from the estimated MRM to benefit transfer predictions, along with the choices and challenges that are involved. The objective of this section is to make these procedures transparent and replicable. The basic steps of an ESV transfer are described by Johnston and Wainger (2015).16 Two initial steps include establishing the need for benefit transfer and developing the conceptual foundation for the exercise. The latter involves development of an implicit or explicit “conceptual model of relationships between ecosystem processes and human benefits, including the biophysical pathways through which benefits are realized and their connections to different beneficiary groups” (Johnston and Wainger 2015, p. 247). Such a model should clarify the linkages between actions, changes in ecosystem services, and the effect of these changes on utility (Bateman et al. 2011b; Olander et al. 2018). This conceptual model is developed within a valuation context in which management scenarios, ecosystem services, and affected populations can be defined. ESVs at policy sites are predicted conditional on a set of biophysical changes in ecosystem services. These changes can be quantified in various ways, including biophysical models, field observations, or the use of hypothetical scenarios.

Here, we investigate three distinct water-quality change scenarios for three different water bodies within the Great Bay watershed of New Hampshire, USA (Figure 1): the Great Bay Estuary itself, not including tributaries (Figure 2), and the freshwater and tidal portions of the Exeter-Squamscott tributary (Figure 3). These define the geospatial scale of changes to be considered. Market extents must also be defined—the location of households for whom values are estimated. For the Great Bay Estuary, we evaluate WTP over three different market areas: (1) residents in NH towns immediately adjacent to the bay (Figure 2), (2) residents of NH towns within the entire Great Bay watershed (Figure 1), and (3) all residents of New Hampshire (Figure 1, inset). For the Exeter-Squamscott River, we evaluate WTP for residents in towns adjacent to the upper or lower portion of the river, respectively (Figure 3). The water quality improvements to be considered include those that could potentially arise because of intensive restoration of vegetated riparian buffers around the focal water bodies, of the general type discussed by Johnston, Holland, and Yao (2016) for the nearby Wells National Estuarine Research Reserve. Restoration of this type can improve water quality through various processes, such as nutrient uptake, reduction of storm-water runoff, and shoreline stabilization by riparian buffer vegetation.

Figure 1. Great Bay Watershed in New Hampshire, USA

Note: Water quality index values are calculated using equation (2).

Figure 2. Great Bay Estuary Major Assessment Units and Baseline Water Quality (Table 6)

Notes: The Exeter River is the freshwater portion of the river from the headwaters to the Exeter town center (indicated by hash mark across river), and the Squamscott River is the tidal portion of the river from the Exeter town center to the Great Bay. Baseline water quality is shown for select river segments (Table 6). Water quality index values are calculated using equation (2).

Figure 3. Exeter-Squamscott River Watershed, a Subwatershed in the Southern Portion of the Great Bay Watershed (Hydrologic Unit Code 0106000308)

Quantifying Water Quality Change

A core component of any ESV benefit transfer is quantification of the change in ecosystem services for which values are to be estimated, in units that are meaningful for value prediction. This requires the definition of biophysical units that link ecosystem service changes at the policy site to variables in the MRM that characterize baselines and changes (i.e., scope). Here, this is done using a standard 100-point WQI to quantify water quality changes of the type that could occur because of riparian buffer restoration (Walsh and Wheeler 2013). As described by Johnston, Besedin, and Holland (2018), WQIs combine information on physical and chemical water quality parameters into a single index linked to the ecosystem services or uses provided by a water body (Abbasi 2012; Walsh and Wheeler 2013; Van Houtven et al. 2014). They are among the most common means to measure water quality for valuation and benefit transfer (Griffiths et al. 2012; Walsh and Wheeler 2013). Here we apply the WQI methodology and classification of U.S. EPA (2009b), adapted from Cude's (2001) Oregon Water Quality Index.

Implementing the WQI to quantify baselines and changes entails three steps: (1) obtaining pollutant data and projections for the water body, (2) transforming these data into subindex values, and (3) combining the subindex values into an aggregate WQI score. The pollutants used by the WQI, along with their required units of measure and associated WQI subindex weights, are shown in Table 4. To establish baselines, pollutant data were obtained from the New Hampshire Department of Environmental Services (NHDES). These data were averaged across all sampling periods and monitoring stations for several NHDES Water Quality Assessment Units in each of our three focal water bodies to produce WQI pollutant parameter values for each pollutant subindex.17 These pollutant parameter values were transformed into corresponding subindex values using functions and thresholds in Table 5.18

Table 4. Water Quality Index (WQI) Pollutants, Concentration Units, and Index Weights

Table 5. Water Quality Index Parameter-Subindex Equations

Note: See Table 4 for definition of abbreviations.

Finally, the subindex values weights were used to calculate the WQI for each major water body using the weighted geometric mean, as described by Walsh and Wheeler (2013),

(2)$$WQI = \mathop \prod \limits_{i = 1}^6 Q_i^{W_i},$$

where Q i is the calculated water quality subindex for parameter i and W i is the weight of the ith parameter from Table 4. The resulting baseline WQI values for each assessment unit are shown in Table 6 and vary across study areas (Figures 2 and 3). U.S. EPA's (2009b) water quality classification identifies the minimum WQI value on a 100-point scale required for human uses (e.g., drinking, swimming, fishing, and boating), and links to the MRM variables Ln_QualityChg and Ln_BaseQuality, both of which are measured in WQI units.

Table 6. Baseline Water Quality Estimates

a Water quality data were limited for much of the Exeter River. The “Brentwood” assessment unit was the farthest upstream unit that contained a relatively complete set of pollutant data.

b Beginning in 2016, impoundment area NHIMP600030805-04 behind the Exeter River dam became part of river area NHRIV600030805-32.

Note: NHDES, New Hampshire Department of Environmental Services; WQI, water quality index.

Although these calculations estimate water quality baselines for our case study sites, they are not sufficient to predict water quality changes from particular management interventions at these sites. In some cases, these changes can be quantified using modeling exercises that predict changes in relevant ecosystem service or environmental quality measures that would occur because of specified actions. Spatially explicit forecasts are typically required (Bateman et al. 2011a, 2011b; Ferrini, Schaafsma, and Bateman 2015; Johnston, Besedin, and Holland 2018; Glenk et al. 2019). In many situations, however, models sufficient to predict these changes are unavailable (Olander et al. 2017). In such cases, changes are sometimes quantified using illustrative “what if” scenarios reflecting future conditions that might occur (Johnston and Wainger 2015).

Here, we illustrate benefit transfers using scenarios of potential water quality improvements in affected areas, including 3, 5, 7, and 9-point increases on a 100-point WQI, from minimum baseline conditions in each water body (Table 6). These levels were chosen to fall in the general range of similar changes considered by Johnston, Holland, and Yao (2016) for estuarine systems in Maine, also potentially caused by buffer restoration.19 Using equation (2) and information in Tables 5 and 6, one can further calculate changes in water quality parameters that would be sufficient to generate these WQI changes. For each level of improvement, we forecast WTP for each geospatial scale (water body) and market extent introduced previously. We predict per household and aggregate population-level WTP for each market area.

Setting MRM Variable Levels

Benefit transfer also requires values (or levels) to be chosen for other independent variables in the MRM (Table 2). These levels are inserted into equation (1) for $\bar{x}_{js}$ to predict WTP. Variable levels are chosen to reflect current conditions at the policy site. Selection of these levels often requires intermediate calculations using external data such as spatial landscape (GIS) metrics and U.S. Census data, whereas other values are chosen based on anticipated policy changes. Variables for study methodology are typically assigned their mean values over the metadata, unless other levels reflect “best practices” associated with reduced measurement errors in primary studies (Johnston, Besedin, and Ranson 2006; Stapler and Johnston 2009; Boyle and Wooldridge 2018).

Within the present application, levels for Ln_PropAgLand, Ln_RelativeSize, ProportionChg, and their underlying geospatial components (e.g., shoreline length, watershed area, town area, county area, and agricultural land area) are calculated using GIS data layers available for the study area, based on our predefined policy scenarios and market areas. These calculations follow variable definitions in Table 2. For example, Ln_RelativeSize is calculated by dividing the shoreline length of the focal resource (river or bay) by the size of the market area and then calculating the natural log of this ratio.

Median household income (Ln_Income) and number of households for towns, counties, and states were obtained from U.S. Census data for 2015 (2011–2015 American Community Survey 5-year estimates; Income for the Great Bay watershed was approximated using a household-weighted average for Rockingham and Strafford counties. Household incomes for groups of communities (e.g., communities adjacent to the Exeter River) were calculated as a household-weighted average across the communities.20 The resulting variable levels are presented in Table 7.

Table 7. Geospatial and Socioeconomic Data for Benefit Transfer Scenarios

Values for the remaining (non-methodological) variables were selected based on the scenario. Because none of the scenarios involved multiple geographically distinct water body types, Multi_Body = 0. The Squamscott and Exeter scenarios include a river, so River = 1. We are interested in forecasting WTP for users and nonusers (Non_Users = 0) in New Hampshire (Northeast_US = 1) for water bodies supporting three recreational uses (Swim_Use = 1, Boat_Use = 1, and Game_Fish = 1). For methodological variables, we assume an annual, mandatory, mean payment (LumpSum = 0, Voluntary = 0, and MedianWTP = 0), with StudyYear = 2017. We also set OutlierBids = 1 and NonReviewed = 0, under the assumption that these settings reflect best practices.21 Mean values over the metadata (Table 2) are used for all remaining variables.

Calculating and Aggregating Values

The use of these variable levels for $\bar{x}_{js}$ within equation (1) provides an estimate of $\bar{y}_{js}$, or the natural log of per household WTP, for the particular water quality change scenario, affected area, and assumed market extent. Exponentiating this predicted value according to $WTP = {\rm exp}\lpar \hat{y}_{js} + {\rm \sigma} _{\rm \varepsilon} ^2$/2), provides an estimate of per household WTP, where ${\rm \sigma} _{\rm \varepsilon} ^2$ is the model error variance from Table 3 (Johnston and Besedin 2009). The resulting estimate is denominated in 2007 U.S. dollars and can be converted to current dollars using standard CPI adjustment (here we adjust to 2016 dollars). As this value reflects a mean per household WTP estimate, it can be aggregated across households within the specified market area to estimate aggregate population-level WTP.

This procedure is illustrated in Table 8 for the Squamscott River, 9-point WQI increase scenario, using the minimum baseline water quality from Table 6. Communities in this region include Exeter, Newfields, and Stratham, with a median household income of $86,305.22 Given these conditions, the benefit transfer predicts annual per household WTP = $53.73 (2016 USD) for a WQI change from 71 to 80. When aggregated across all households in the three adjacent communities, the result is a total WTP of $517,754 per year.

Table 8. Illustrating the Benefit Transfer Process for a 9-Point Increase on the 100-Point Water Quality Index (WQI) in the Squamscott River (baseline WQI = 71)

Note: CPI, consumer price index; NHDES, New Hampshire Department of Environmental Services; WTP, willingness to pay.

Predicted Welfare Patterns and Benefit Transfer Validity

Initial insight into the validity of MRM benefit transfers can be gleaned through consideration of value surface patterns implied by the estimated meta-equation, including parameter estimates and functional form. For example, do parameter estimates indicate intuitive responsiveness of WTP to influences such as scope, scale, and extent of the market? However, other types of insight can only be provided through evaluation of the benefit predictions across different management scenarios. For example, are these predictions consistent with theory, intuition, and results of prior primary studies? Does the MRM generate credible predictions for the particular range of environmental changes under consideration?23 Results such as these can help determine whether benefit transfer results are defensible for applied use.

Benefit transfers produce a range of WTP forecasts for water quality improvements in the Great Bay watershed, with results varying as expected over different scenarios. Table 9 and Figure 4 illustrate predicted WTP (per household per year) for 3-, 5-, 7-, and 9-point WQI increases for the three focal water bodies, assuming minimum baseline quality from Table 5. Transfers for Great Bay improvements also consider three different market extents when calculating per household WTP (adjacent towns, Rockingham and Strafford counties, and the entire state of New Hampshire). These are shown both numerically (Table 9) and graphically (Figure 4). Table 10 aggregates these per household estimates over all households within the relevant market areas to generate total, population-level predictions. Reported p-values (Tables 9 and 10) are calculated using Wald χ2 tests (Greene 2012, p. 528), based on the underlying precision of MRM parameter estimates (Table 3).

Note: Three market regions (adjacent towns, two counties, and all of New Hampshire) were assessed for the Great Bay.

Figure 4. Willingness to Pay (per household per year) for 3-, 5-, 7-, and 9-Point Increases in Water Quality on the 100-Point Water Quality Index (WQI) for Three Water Bodies Using the Minimum Baseline WQI Value for Each Water Body from Table 6

Table 9. Predicted Annual per Household Willingness to Pay to Improve Water Quality Index from Minimum Baselines

Notes: *** p < 0.01, ** p < 0.05, * p < 0.10, with p-values calculated using Wald χ2 tests (Greene 2012, p. 528). All values in 2016 USD.

Table 10. Annual Aggregated Willingness to Pay (WTP; millions of dollars) to Improve Water Quality Index from Minimum Baselines

Notes: *** p < 0.01, ** p < 0.05, * p < 0.10, with p-values calculated using Wald χ2 tests (Greene 2012, p. 528). All values in 2016 USD. WTP values for Exeter and Squamscott Rivers are aggregated over adjacent towns, as described in the main text.

The results show multiple patterns in WTP that are relevant to validity assessments and prospective policy applications. For example, annual per household WTP increases as the size of the water quality improvement increases (e.g., from a 3- to 9-point increase) for all focal water bodies (Figure 4), but at a decreasing rate. These are intuitive findings consistent with positive scope sensitivity and diminishing marginal utility. For the Exeter and Squamscott Rivers, mean WTP transfers range from $39 (3-point) to $54 (9-point) per household per year, for households in adjacent communities. These reflect predictions that are consistent with the range of WTP values for similar improvements found in the prior literature (Table 1) and results of other water quality meta-analyses (e.g., Johnston et al. 2005; Van Houtven, Powers, and Pattanayak 2007).

Per household WTP predictions are also similar across these two rivers, reflecting the offsetting effects of different parameters. Although the baseline water quality is better and the size of the improved water body (i.e., the length of the river) is larger in the Exeter River, median household income is higher in the Squamscott River (Tables 6 and 8). Annual per household WTP is greater ($62–$85) for improvements to Great Bay (despite baseline water quality being higher), primarily because of the size of the water body and the relative lack of substitutes within New Hampshire (Tables 6 and 8). Results such as these demonstrate the combined effects of scope, scale, substitutes, and income on WTP predictions.

Results also demonstrate the effects of market extent. As the market area for the Great Bay benefit transfer increases (from towns to counties to state), annual per household WTP decreases. This reflects intuitive effects of distance decay (Sutherland and Walsh 1985; Bateman et al. 2006, 2011a; Schaafsma 2015; Glenk et al. 2019). That is, larger market areas are associated with larger average distances between individuals and improved resources, ceteris paribus, leading to an expectation of lower mean per household WTP (Johnston, Besedin, and Stapler 2017).

Benefit transfers aggregated over an entire market area (or population) can vary because of differences in per household WTP and in the number of households in the market area. Despite comparable per household WTP measures, regional WTP values aggregated across all households in the adjacent communities for the three-town Squamscott River are lower than values for the larger seven-town Exeter River region, because of the larger number of households in the latter region (Figure 3, Table 10). For example, aggregated WTP for a 3-point WQI increase in the Squamscott River is approximately $0.38 million over the adjacent community population, compared with $0.75 million for the Exeter River (Table 10).

Aggregated values for the seven communities immediately adjacent to the Great Bay exceed those for the Exeter and Squamscott Rivers, because of larger per household WTP values and the larger number of households in the region. Moreover, as one expands the assumed extent of the market for the Great Bay benefit transfer (from towns to counties to the state), aggregated population-level WTP increases by roughly a factor of 30. Even though mean per household WTP is lower for larger market areas (because of distance decay), the summation of values over (much) larger populations overwhelms this effect when estimating aggregate population-level WTP. Patterns such as these demonstrate the often-dominant effect of assumed market area (the area over which benefits are aggregated) when conducting benefit transfers (Bateman et al. 2006).

Predicted WTP estimates for all scenarios are statistically significant, with significance positively related to the scope of WQI change and the extent of the market (Tables 9 and 10). Hence, variations in the valuation scenario influence not only mean WTP predictions that emerge from an MRM, but also (unsurprisingly) the relative precision of those predictions. Given findings such as these, practitioners may wish to consider not only the predicted WTP point estimate for a transfer, but also whether it is possible to reject the null hypothesis that this estimate is different from zero. We reject this null hypothesis in all cases, in most cases at p < 0.05 or better.

Challenges for Large-Scale Ecosystem Service Valuation

Results such as these illustrate many of the patterns desired for large-scale benefit transfers, including responsiveness of welfare estimates to factors that should—according to theory—influence per household and aggregate WTP. Moreover, because the MRM includes an internal means to adjust WTP estimates for variations in scope and scale, there is no need to “scale up” estimates (Brander et al. 2012). However, meta-analytic methods are not a panacea. Although the valuation literature tends to emphasize the advantages of these methods, there are also limitations that should be considered. As discussed previously, the validity of any MRM for benefit transfer depends on multiple factors. However, even with an MRM judged to be sufficient across all of these dimensions, challenges for benefit transfer can still arise. This section discusses some of these challenges, using the previous MRM and case study benefit transfer as an illustration.

Among the primary limitations is that all benefit transfer methods—including MRMs—are constrained in terms of their capacity to account for unique attributes of particular policy sites (Bateman et al. 2011a). All valuation contexts are characterized by unique conditions, including ecosystem services, substitutes and complements, populations, geospatial dimensions, and other factors that influence economic values. Although MRM functions can provide a means to adjust WTP predictions for some of these factors, their ability to do so is limited by the set of variables in the model. This is not an inherent shortcoming in meta-analytic methods—the ability to adjust benefit estimates is usually greater within MRM benefit transfer than within other benefit transfer approaches (such as unit value transfer, single-study benefit function transfer, and structural preference calibration24). However, given limits in the information available from primary valuation studies in the literature, these adjustments are still constrained. For example, the illustrated MRM includes only two variables characterizing affected populations (Ln_Income and Non_Users) and three variables characterizing affected uses (Swim_Use, Boat_Use, and Game_Fish). Context-specific effects beyond these variables—for example, possible effects on drinking water (which is not valued by this MRM) or on regionally important cultural resources—cannot be accommodated explicitly without adding corresponding control variables and obtaining metadata that provide sufficient variation in these variables.

Among other context-specific features that can only be accommodated in a limited manner is the effect of complements and substitutes. As discussed by Bateman et al. (2011a), Glenk et al. (2019), and Schaafsma (2015), substitutes and complements for specific ecosystem service improvements are often diverse and vary over space. This leads to two linked challenges for benefit transfers: (1) identifying measures of substitutes and complements that are similarly relevant across sites and (2) obtaining data on these measures from primary studies or supplementary sources. Given these challenges, most MRMs omit variables that capture the effect of substitutes and complements on WTP. The illustrated MRM is comparatively superior in this regard but still includes only one variable that captures spatially variable substitute effects (ProportionChg) and one that captures potential effects of complements (Ln_PropAgLand). As such, the capacity of this (or any) MRM to predict the spatially heterogeneous effects of multiple substitutes and complements on ESVs is limited. If more comprehensive and precise estimates of these effects are required, primary valuation studies should be conducted.25

There are also trade-offs implied by the functional forms for MRMs that influence suitability for particular types of benefit transfer. For example, WTP patterns implied in Figure 4 may lead to questions regarding how the model can and should be used to estimate benefits for successive, small water quality improvements over time. Even cursory examination of this figure suggests that adding-up likely does not apply—predicted WTP for one 9-unit improvement is much less than WTP for a 3-unit improvement multiplied by 3.26 Works such as Newbold et al. (2018b) and Moeltner (2019) have responded with alternative functional forms for this MRM that impose the adding-up property. These functions, however, reduce empirical performance—they have poorer fit to the metadata (Johnston, Rolfe, and Zawojska 2018).27 As noted by Newbold et al. (2018b, p. 544), there is “a tradeoff between improved statistical fit that can be achieved by allowing additional model flexibility […], and consistency with theoretical restrictions that might require reduced flexibility on the other.” Kling and Phaneuf (2018, p. 498) further argue that practical failures of adding-up tests in applied market and nonmarket situations suggest that these tests “are not likely to be fruitful in informing benefit transfer.”28

Trade-offs such as these related to the choice of strong structural versus reduced-form specifications for MRMs are discussed by Bergstrom and Taylor (2006) and Johnston, Rolfe, and Zawojska (2018) and highlight the lack of consensus in this area. As noted by Johnston, Rolfe, and Zawojska (2018, p. 192), “All [MRM model specifications] require assumptions, and the capacity of any assumed specification to approximate all aspects of empirical reality cannot be assured.” Disagreements over the appropriateness of strong structural specifications in MRMs are further related to a tension between two competing motivations for economic meta-analysis. Historically, one of the primary motivations for economic meta-analysis was to allow metadata (from many prior studies) to reveal patterns caused by “misspecification biases and specification searching in empirical economic research” (Stanley 2005, p. 205). Viewed from this perspective, the use of strong structural specifications to compel—ex ante—MRMs to produce certain types of empirical results is antithetical to the core purpose of meta-analysis. In contrast, others have argued that structural specifications are required to ensure welfare-theoretic consistency within meta-analytic transfers (Smith and Pattanayak 2002; Newbold et al. 2018b).

Potential ramifications of MRM specification are further illustrated using Figure 5, which shows the predicted marginal WTP (or demand) for WQI improvements in the Great Bay, by residents of surrounding towns.29 By illustrating marginal rather than total WTP, Figure 5 clarifies welfare patterns implied by Figure 4, and particularly WTP patterns for small changes in water quality. As shown by Figure 5, marginal WTP increases sharply for ΔWQI < 3, suggesting high marginal WTP for the first few units of quality change. Because of this pattern, repeated use of the MRM to predict values for successive, small water quality changes will lead to large aggregate WTP estimates for the combined change, ceteris paribus.30 This is among the motivations used by Newbold et al. (2018b) and Moeltner (2019) to suggest structural adding-up MRM specifications that, among other features, attenuate WTP predictions for small changes.

Notes: Valuation scenario is identical to that in Figure 4 and described in the main text. WQI, water quality index; WTP, willingness to pay.

Figure 5. Predicted Demand for Water Quality Improvements (ΔWQI), Great Bay by Residents of Adjacent Towns

However, it is important to notice that predictions such as these are outside of the range of data support for the MRM—there are no observations of ΔWQI < 2.5 in the metadata. Moreover, none of the metadata observations report WTP for successive water quality changes—all report WTP for one-time improvements. Structural and nonstructural MRMs estimated from these metadata tend to predict similar WTP estimates for one-time changes in water quality within the range of the data. In contrast, debates over structural MRM functional forms often center on WTP predictions for successive, small water quality changes. Because there are no supporting metadata for predictions of this type, there is little capacity to validate these predictions. More informed debates over WTP predictions for small, successive water quality changes will require primary studies that provide credible estimates of these values.

A related issue concerns the capacity of MRMs to predict ESVs over large scales, such as multistate regions or nationwide. As shown by Table 3, the partial elasticity associated with Ln_RelativeSize is small. Recall, this variable is defined as the natural log of the ratio of total shoreline length (kilometers) of the improved resource and the size of the benefit aggregation market area (square kilometers). The inelastic magnitude of this effect implies that per household WTP declines gradually with increases in market area size.31 Given that only one primary study in the metadata considers more than a two-state market area,32 it is unclear to what extent this small effect size applies to larger market areas—as these are also (largely) outside the range of the metadata. The implications for large-scale ESV transfers are implied by Table 10—aggregate values increase rapidly with the size of the assumed market area. Hence, the model will potentially predict large aggregate WTP estimates when applied to regions larger than a single state. Given the lack of support in the metadata for such applications, practitioners should proceed with care.

A final note concerns the accuracy of MRM benefit transfer compared with simpler forms of unit value or benefit function transfer, or naïve MRM benefit transfers that overlook the validity concerns discussed previously. Direct comparisons of this type are beyond the scope of this article but have been conducted elsewhere. For example, Johnston, Besedin, and Stapler (2017) and Johnston, Besedin, and Holland (2018) demonstrate the benefit transfer accuracy gains made possible via the inclusion of spatial variables in MRM benefit transfer for different variants of the same water quality metadata used here. Johnston and Thomassin (2010) use an earlier, U.S.-Canada version of the same metadata to demonstrate that MRM benefit transfer reduces transfer errors compared with unit value benefit transfer. Using other sets of metadata, Moelter and Rosenberger (2014) and Johnston and Moeltner (2014) illustrate Bayesian approaches that can be used to evaluate the extent to which commodity and welfare consistency increase benefit transfer efficiency. This and other recent work is summarized by Johnston, Rolfe, and Zawojska's (2018) review of the recent benefit transfer literature.33


Methodological advances in meta-regression modeling have improved the capacity of benefit transfers to provide valid and reliable welfare predictions. Variable and functional form specifications for these models can be adapted to meet particular benefit transfer needs and theoretical expectations, and there is increasing recognition of the trade-offs implied by different types of MRM specifications. The future seems bright in terms of continued development of meta-analytic techniques to support ecosystem service valuation.

At the same time, many meta-analyses in the ecosystem services literature omit fundamental elements required to ensure valid value predictions, and the steps required to generate benefit predictions from underlying meta-analytic benefit functions are often obscured. Clarity and transparency on issues such as these are required to ensure credible uses of meta-analytic tools. Few meta-analyses in the literature discuss the extent to which the resulting benefit predictions are credible for applied use or evaluate the impact of post-estimation procedures on benefit transfer results. In some cases, simple unit value transfers may be more accurate than those implemented using meta-analysis, particularly if the latter does not follow best-practice standards. The illustrated case study is designed to demonstrate how evaluations of issues such as these can help clarify the suitability of MRM predictions for benefit transfer.

Recognition of these issues is needed. As noted by Johnston, Rolfe, and Zawojska (2018), there has been a proliferation of benefit transfer tools developed outside of the environmental valuation and benefit transfer literature (Bagstad et al. 2013), and government agencies are applying benefit transfer practices that are considered arbitrary and unjustified by valuation researchers (Boyle, Kotchen, and Smith 2017; Smith 2018). Without “cost-effective, straightforward, transferable, scalable, meaningful, and defensible methods” (Olander et al. 2017, p. 170) for benefit transfer, analysts may apply approaches that are unlikely to provide valid and defensible estimates.

1 Introductions to these approaches are provided by Johnston et al. (2015a) and Rosenberger and Loomis (2017).

2 For additional details on MRMs, see Smith and Pattanayak (2002), Bergstrom and Taylor (2006), Moeltner, Boyle, and Paterson (2007), Nelson and Kennedy (2009), Johnston and Rosenberger (2010), Boyle et al. (2013), Boyle, Kaul, and Parmeter (2015), Rolfe, Brouwer, and Johnston (2015), Boyle and Wooldridge (2018), Johnston, Besedin, and Holland (2018), and Johnston, Rolfe, and Zawojska (2018).

3 For additional examples, see reviews in Nelson and Kennedy (2009), Johnston and Rosenberger (2010), Rolfe, Brouwer, and Johnston (2015), and Johnston, Rolfe, and Zawojska (2018).

4 For a formal discussion of valuation validity and reliability, see Bishop and Boyle (2019).

5 Here we define scope as the quantity or quality of an ecosystem service change for which a value is predicted; scale is defined as the geographic area over which the change takes place.

6 As described by Bishop and Boyle (2019, p. 564), construct validity reflects the extent to which a value measure is consistent with “prior expectations about how the true value ought to be related to other variables. Such prior expectations are motivated by theory, intuition, and past empirical evidence.”

7 Results can also help researchers characterize the literature around a topic and allow the analyst to test hypotheses regarding factors that influence study results (Smith and Pattanayak 2002; Boyle and Woodridge 2018).

8 Other (non-regression) types of meta-analysis may also be used for benefit transfer, such as the synthesis of prior valuation estimates to generate mean unit values, for example weighted by measures of study precision. For an introduction to different types of meta-analysis applied in economics, see Stanley and Doucouliagos (2012). Here we emphasize MRMs, as these are the most common form of meta-analysis found in the benefit transfer literature.

9 The adding-up property in MRMs relates to sequential use of the model to predict WTP for additive improvements. Assume that WTPA is predicted WTP for a quality improvement from A to B, WTPB is predicted WTP for a quality improvement from B to C, and WTPC is predicted WTP for a combined improvement from A to C. The adding-up property implies that WTPA + WTPB = WTPC, when each is predicted separately by the MRM, and utility is held constant at the original level for all calculations. For a more formal definition, see Kling and Phaneuf (2018).

10 Recent work demonstrates that gains in transfer reliability may sometimes be achieved by relaxing narrow, ex ante commodity and welfare restrictions in favor of those supported by empirical relationships (e.g., Moeltner and Rosenberger 2014; Johnston and Moeltner 2014; Moeltner 2015). For example, in some cases reliability may be improved by pooling metadata on otherwise similar Hicksian and Marshallian welfare measures (Johnston and Moeltner 2014). Pooling of this type within MRMs has been justified based on structural relationships between these two welfare measures, grounded in microeconomic theory (Londoño and Johnston 2012).

11 The metadata exclude studies focusing on drinking-water quality.

12 External data were extracted from the National Hydrography Dataset; Hydrologic Unit Code Watershed Boundary Dataset; National Land Cover Database; NOAA Global Self-Consistent, Hierarchical, High-Resolution Geography Database; and U.S. Census.

13 Affected areas are defined as counties that intersect or touch affected water bodies. The rationale for this variable is that nonagricultural land uses (e.g., forests and residential) are often associated with recreational and other human uses that can magnify the per household value of water quality improvements (Johnston, Besedin, and Stapler 2017).

14 An alternative approach would be to weight metadata observations such that each study (rather than each observation) is given equal weight—both weighted and unweighted approaches are common in the literature (Boyle and Wooldridge 2018). As noted by Johnston et al. (2005, p. 231), “Although weighting methods prevent studies providing multiple observations from unduly influencing model estimation, they also imply that such studies are no more informative, overall, than others (Bateman and Jones 2003).” Hence, the decision to weight observations in an MRM is typically made on a case-by-case basis. Using earlier versions of these metadata, Johnston, Besedin, and Wardwell (2003) and Johnston et al. (2005) find that results are robust to the use of weighted versus unweighted regression.

15 This specification allows the MRM to reflect curvature in the valuation function, a multiplicative rather than additive effect of independent variables, and the implied constraints that WTP approaches zero when water quality change, income, and the geospatial index variable (Ln_RelativeSize) approach zero.

16 Similar steps are described by Bateman et al. (2011b) for ecosystem services valuation in general.

17 We investigate water quality in each assessment unit rather than averaging pollutant data across the entire water body in order to produce a range of water quality values that could then be used in sensitivity analyses. Average pollutant concentration values for each assessment unit are listed in Bauer and Johnston (2017).

18 These functions are drawn from U.S. EPA (2009b, tables 10-1 and 10-3 and appendix F). There are six water quality subindices in each WQI; however, note that the WQI for freshwater rivers and streams includes biochemical oxygen demand, whereas the WQI for estuaries includes chlorophyll-a.

19 These water quality changes are larger than one would typically experience due to large-scale regulatory changes but can occur in small areas (e.g., because of extensive restoration). As described by Newbold et al. (2018a, p. 475, citing U.S. EPA 2015), “Most contemporary EPA regulations promulgated under the Clean Water Act are estimated to improve water quality by <1% in the vast majority of water bodies.” In contrast, choice experiment attribute levels in Johnston, Holland, and Yao (2016) for intensive riparian restoration at the Wells National Estuarine Research Reserve (restoring up to 500 local riparian acres) allow for up to a maximum of a 20 percentage point increase on a 100-point index of ecological condition, based on “ecological models, field studies, and expert consultations” (p. 743).

20 Because the meta-regression analysis used 2007 U.S. dollars in its estimation process, we convert U.S. Census income data from 2015 USD to 2007 USD using the consumer price index (CPI) from the U.S. Bureau of Labor Statistics (

21 This follows Boyle and Wooldridge (2018) in setting levels for selected variables based on assumed “best practices,” with others set at mean values. For additional discussion of the treatment of methodological variable levels in MRM predictions, see Johnston, Besedin, and Ranson (2006) and Moeltner, Boyle, and Paterson (2007).

22 The entry in Table 8, 11.232, reflects the natural log of this value after adjustment from 2015 to 2007 USD.

23 For example, “most stated preference water quality valuation studies have examined improvements or decrements in the order of 10-20% of the full range of possible water quality levels represented on the [100-point water quality index] scale” (Newbold et al. (2018a, pp. 474–475). MRMs grounded in this literature typically have few or no data points (in the metadata) for very small water quality changes. WTP predictions for very small quality changes may therefore occur outside the range of available data. This can lead to benefit estimates that lack face validity (e.g., seem unreasonably high) or are sensitive to functional forms imposed by the MRM analyst (Johnston, Rolfe, and Zawojska 2018).

24 Johnston et al. (2015a) describe these approaches. Preference calibration requires the analyst to specify a structural utility function that describes individuals’ choices over market and/or nonmarket goods. One then derives analytical expressions that determine a theoretically consistent relationship between benefit measures from existing primary studies. Given the mathematical complexity of these methods and the need to derive closed-form solutions, approaches of this type almost always incorporate a small number of variables (e.g., Smith, Van Houtven, and Pattanayak 2002; Smith, Pattanayak, and Van Houtven 2006).

25 The need for high quality, well-documented primary valuation studies to support policy analysis and benefit transfer has been emphasized repeatedly in the literature (Loomis and Rosenberger 2006; McComb et al. 2006; Rosenberger and Johnston 2009; Johnston and Rosenberger 2010; Johnston et al. 2015b). Related issues are summarized by Johnston et al. (2015b, pp. 565–567), who argue that “[those] who support valuation research have thus far been unable to promote development of a large set of empirical valuation studies specifically designed and documented for benefit transfer applications.”

26 This is only a crude approximation of adding-up. A formal evaluation would adjust baseline WQI to successively higher levels when calculating WTP for each successive improvement. Adjustments would also be required for income effects (Kling and Phaneuf 2018). However, a more formal evaluation by Moeltner (2019) shows that adding-up is violated when using this MRM for benefit transfer.

27 This poor fit is not surprising. As noted by Kling and Phaneuf (2018), past evaluations have shown that the adding-up property is routinely violated in both market and nonmarket situations. Hence, these specifications are imposing a property on the MRM that is likely violated by many of the primary studies in the metadata.

28 Specifications such as that of Johnston, Besedin, and Holland (2018) that define the dependent variable in terms of WTP per unit of change can approximate the adding-up property without sacrificing empirical fit, but this approach does not guarantee that adding-up is satisfied perfectly.

29 All other aspects of the valuation scenario mirror those described previously.

30 Small (negative) WTP adjustments associated with the resulting increases in baseline water quality (Ln_BaseQuality) are insufficient to offset this effect.

31 This pattern implies that distance decay is significant but small in magnitude.

32 Not including the District of Columbia.

33 Prior work does not find that more complex benefit function transfers, such as MRMs, always outperform simpler methods (Johnston, Rolfe, and Zawojska 2018). Although benefit function transfers and data synthesis methods have been shown to outperform unit value transfers, on average (Kaul et al. 2013; Rosenberger 2015), unit value and simpler function transfers have been shown to be more accurate in some cases (e.g., Barton 2002; Lindhjem and Navrud 2008; Johnston and Duke 2010; Bateman et al. 2011a; Klemick et al. 2018; Guignet et al. 2019). The literature has yet to reach consensus on conditions under which this is likely to occur (Johnston, Rolfe, and Zawojska 2018).

Financial support

Supported by National Oceanic and Atmospheric Administration (NOAA) Grant NA14NOS4190145, under subaward 3003668021 from The Nature Conservancy. Opinions do not imply endorsement of the funding agency.


The authors thank David Patrick and Peter Steckler of the New Hampshire Chapter of The Nature Conservancy, Matt Wood of the New Hampshire Department of Environmental Services, and Abigail Kaminski at Clark University for assistance. This article was prepared for the USDA Workshop, Applications and Potential of Ecosystem Services Valuation within USDA – Advancing the Science, Washington, DC, April 23–24, 2019.

Conflict of interest


Transparency and openness promotion statement

Data necessary to replicate results are available in this article, through existing data repositories, or by request. Results in Tables 1–3 are available from Johnston, Besedin, and Stapler (2017); data to replicate these results are available by request from the lead author. Geospatial and population data summarized in Tables 6 and 7 are available from the National Hydrography Dataset (; the Hydrologic Unit Code Watershed Boundary Dataset (; the National Land Cover Database (; the NOAA Global Self-Consistent, Hierarchical, High-Resolution Geography Database (; New Hampshire's Statewide GIS Clearinghouse (; and the U.S. Census ( Water quality monitoring data are available from the New Hampshire Department of Environmental Services (


Abbasi, T. 2012. Water Quality Indices. Amsterdam: Elsevier.
Bagstad, K.J., Semmens, D.J., Waage, S., and Winthrop, R.. 2013. “A Comparative Assessment of Decision-Support Tools for Ecosystem Services Quantification and Valuation.” Ecosystem Services 5: 2739.
Barton, D.N. 2002. “The Transferability of Benefit Transfer: Contingent Valuation of Water Quality Improvements in Costa Rica.” Ecological Economics 42(1–2): 147164.
Bateman, I.J., Brouwer, R., Ferrini, S., Schaafsma, M., Barton, D.N., Dubgaard, A., Hasler, B., et al. 2011a. “Making Benefit Transfers Work: Deriving and Testing Principles for Value Transfers for Similar and Dissimilar Sites Using a Case Study of the Non-market Benefits of Water Quality Improvements across Europe.” Environmental and Resource Economics 50(3): 365387.
Bateman, I.J., Day, B.H., Georgiou, S., and Lake, I.. 2006. “The Aggregation of Environmental Benefit Values: Welfare Measures, Distance Decay and Total WTP.” Ecological Economics 60(2): 450460.
Bateman, I.J., and Jones, A.P.. 2003. “Contrasting Conventional with Multi-level Modeling Approaches to Meta-analysis: Expectation Consistency in UK Woodland Recreation Values.” Land Economics 79(2): 235258.
Bateman, I.J., Mace, G.M., Fezzi, C., Atkinson, G., and Turner, K.. 2011b. “Economic Analysis for Ecosystem Service Assessments.” Environmental and Resource Economics 48(2): 177218.
Bauer, D.M., and Johnston, R.J.. 2017. Buffer Options for the Bay: Economic Valuation of Water Quality Ecosystem Services in New Hampshire's Great Bay Watershed. Great Bay National Estuarine Research Reserve, Greenland, NH.
Bergstrom, J.C., and Taylor, L.O.. 2006. “Using Meta-analysis for Benefits Transfer: Theory and Practice.” Ecological Economics 60(2): 351360.
Bishop, R.C., and Boyle, K.J.. 2019. “Reliability and Validity in Nonmarket Valuation.” Environmental and Resource Economics 72(2): 559582.
Bockstael, N.E., Freeman, A.M., Kopp, R.J., Portney, P.R., and Smith, V.K.. 2000. “On Measuring Economic Values for Nature.” Environmental Science and Technology 34(8): 13841389.
Boyd, J., Ringold, P., Krupnick, A., Johnston, R.J., Weber, M.A., and Hall, K.. 2016. “Ecosystem Services Indicators: Improving the Linkage between Biophysical and Economic Analyses.” International Review of Environmental and Resource Economics 8(3–4): 359443.
Boyle, K.J., Kaul, S., and Parmeter, C.F.. 2015. “Meta-analysis: Econometric Advances and New Perspectives towards Data Synthesis and Robustness.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 383418.
Boyle, K.J., Kotchen, M.J., and Smith, V.K.. 2017. “Deciphering Dueling Analyses of Clean Water Regulations.” Science 358(6359): 4950.
Boyle, K.J., Kuminoff, N.V., Parmeter, C.F., and Pope, J.C.. 2009. “Necessary Conditions for Valid Benefit Transfers.” American Journal of Agricultural Economics 91(5): 13281334.
Boyle, K.J., Parmeter, C.F., Boehlert, B.B., and Paterson, R.W.. 2013. “Due Diligence in Meta-analyses to Support Benefit Transfers.” Environmental and Resource Economics 55(3): 357386.
Boyle, K.J., and Wooldridge, J.M.. 2018. “Understanding Error Structures and Exploiting Panel Data in Meta-analytic Benefit Transfers.” Environmental and Resource Economics 69(3): 609635.
Brander, L.M., Bräuer, I., Gerdes, H., Ghermandi, A., Kuik, O., Markandya, A., Navrud, S., et al. 2012. “Using Meta-analysis and GIS for Value Transfer and Scaling Up: Valuing Climate Change Induced Losses of European Wetlands.” Environmental and Resource Economics 52(3): 395413.
Cude, C.G. 2001. “Oregon Water Quality Index: A Tool for Evaluating Water Quality Management Effectiveness.” Journal of the American Water Resources Association 37(1):125137.
Ferrini, S., Schaafsma, M., and Bateman, I.J.. 2015. “Ecosystem Services Assessment and Benefit Transfer.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 275305.
Glenk, K., Johnston, R.J., Meyerhoff, J., and Sagebiel, J.. 2019. “Spatial Dimensions of Stated Preference Valuation in Environmental and Resource Economics: Methods, Trends and Challenges.” Environmental and Resource Economics.
Greene, W.H. 2012. Econometric Analysis (7th edition). Upper Saddle River, NJ: Prentice Hall.
Griffiths, C., Klemick, H., Massey, M., Moore, C., Newbold, S., Simpson, D., Walsh, P., and Wheeler, W.. 2012. “US Environmental Protection Agency Valuation of Surface Water Quality Improvements.” Review of Environmental Economics and Policy 6(1): 130146.
Guignet, D., Heberling, M.T., Papenfus, M., Griot, O., and Holland, B.. 2019. “Property Values and Water Quality: A Nationwide Meta-analysis and the Implications for Benefit Transfer.” National Center for Environmental Economics Working Paper, 19-05, U.S. Environmental Protection Agency, Washington, DC.
Hoehn, J.P. 2006. “Methods to Address Selection Effects in the Meta Regression and Transfer of Ecosystem Values.” Ecological Economics 60(2): 389398.
Johnston, R.J., and Besedin, E.Y.. 2009. “Estimating Willingness to Pay for Aquatic Resource Improvements Using Benefits Transfer.” In Thurston, H.W., Heberling, M.T., and Schrecongost, A., eds. Environmental Economics for Watershed Restoration. Boca Raton, FL: CRC Press, pp. 95122.
Johnston, R.J., Besedin, E.Y., and Holland, B.M.. 2018. “Modeling Distance Decay within Valuation Meta-analysis.” Environmental and Resource Economics 72(3): 657690.
Johnston, R.J., Besedin, E.Y., Iovanna, R., Miller, C.J., Wardwell, R.F., and Ranson, M.H.. 2005. “Systematic Variation in Willingness to Pay for Aquatic Resource Improvements and Implications for Benefit Transfer: A Meta-analysis.” Canadian Journal of Agricultural Economics 53(2–3): 221248.
Johnston, R.J., Besedin, E.Y., and Ranson, M.H.. 2006. “Characterizing the Effects of Valuation Methodology in Function-Based Benefits Transfer.” Ecological Economics 60(2): 407419.
Johnston, R.J., Besedin, E.Y., and Stapler, R.. 2017. “Enhanced Geospatial Validity for Meta-analysis and Environmental Benefit Transfer: An Application to Water Quality Improvements.” Environmental and Resource Economics 68(2): 343375.
Johnston, R.J., Besedin, E.Y., and Wardwell, R.F.. 2003. “Modeling Relationships between Use and Nonuse Values for Surface Water Quality: A Meta-analysis.” Water Resources Research 39(12): 13631371.
Johnston, R.J., and Duke, J.M.. 2010. “Socioeconomic Adjustments and Choice Experiment Benefit Function Transfer: Evaluating the Common Wisdom.” Resource and Energy Economics 32(3): 421438.
Johnston, R.J., Holland, B.M., and Yao, L.. 2016. “Individualized Geocoding in Stated Preference Questionnaires: Implications for Survey Design and Welfare Estimation.” Land Economics 92(4): 737759.
Johnston, R.J., and Moeltner, K.. 2014. “Meta-modeling and Benefit Transfer: The Empirical Relevance of Source-Consistency in Welfare Measures.” Environmental and Resource Economics 59(3): 337361.
Johnston, R.J., Ranson, M.H., Besedin, E.Y., and Helm, E.C.. 2006. “What Determines Willingness to Pay per Fish? A Meta-analysis of Recreational Fishing Values.” Marine Resource Economics 21(1): 132.
Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R.. 2015a. “Introduction to Benefit Transfer Methods.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 1959.
Johnston, R.J., Rolfe, J., and Zawojska, E.. 2018. “Benefit Transfer of Environmental and Resource Values: Progress, Prospects and Challenges.” International Review of Environmental and Resource Economics 12(2–3): 177266.
Johnston, R.J., and Rosenberger, R.S.. 2010. “Methods, Trends and Controversies in Contemporary Benefit Transfer.” Journal of Economic Surveys 24(3): 479510.
Johnston, R.J., Rosenberger, R.S., Rolfe, J., and Brouwer, R.. 2015b. “Benefit Transfer: The Present State and Future Prospects.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 553574.
Johnston, R.J., Schultz, E.T., Segerson, K., Besedin, E.Y., and Ramachandran, M.. 2012. “Enhancing the Content Validity of Stated Preference Valuation: The Structure and Function of Ecological Indicators.” Land Economics 88(1): 102120.
Johnston, R.J., and Thomassin, P.J.. 2010. “Willingness to Pay for Water Quality Improvements in the United States and Canada: Considering Possibilities for International Meta-analysis and Benefit Transfer.” Agricultural and Resource Economics Review 39(1): 114131.
Johnston, R.J., and Wainger, L.A.. 2015. “Benefit Transfer for Ecosystem Service Valuation: An Introduction to Theory and Methods.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 237273.
Kaul, S., Boyle, K.J., Kuminoff, N.V., Parmeter, C.F., and Pope, J.C.. 2013. “What Can We Learn from Benefit Transfer Errors? Evidence from 20 Years of Research on Convergent Validity.” Journal of Environmental Economics and Management 66(1): 90104.
Klemick, H., Griffiths, C., Guignet, D., and Walsh, P.. 2018. “Improving Water Quality in an Iconic Estuary: An Internal Meta-analysis of Property Value Impacts around the Chesapeake Bay.” Environmental and Resource Economics 69(2): 265292.
Kling, C.L., and Phaneuf, D.J.. 2018. “How Are Scope and Adding Up Relevant for Benefits Transfer?Environmental and Resource Economics 69(3): 483502.
Lindhjem, H., and Navrud, S.. 2008. “How Reliable Are Meta-analyses for International Benefit Transfers?Ecological Economics 66(2–3): 425435.
Londoño, L.M., and Johnston, R.J.. 2012. “Enhancing the Reliability of Benefit Transfer over Heterogeneous Sites: A Meta-analysis of International Coral Reef Values.” Ecological Economics 78: 8089.
Loomis, J.B., and Rosenberger, R.S.. 2006. “Reducing Barriers in Future Benefit Transfers: Needed Improvements in Primary Study Design and Reporting.” Ecological Economics 60(2): 343350.
McComb, G., Lantz, V., Nash, K., and Rittmaster, R.. 2006. “International Valuation Databases: Overview, Methods and Operational Issues.” Ecological Economics 60(2): 461472.
Moeltner, K. 2015. “Optimal Scope and Bayesian Model Search in Benefit Transfer.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 507523.
Moeltner, K. 2019. “Bayesian Nonlinear Meta Regression for Benefit Transfer.” Journal of Environmental Economics and Management 93: 4462.
Moeltner, K., Boyle, K.J., and Paterson, R.W.. 2007. “Meta-analysis and Benefit Transfer for Resource Valuation—Addressing Classical Challenges with Bayesian Modeling.” Journal of Environmental Economics and Management 53(2): 250269.
Moeltner, K., and Rosenberger, R.S.. 2014. “Cross-Context Benefit Transfer: A Bayesian Search for Information Pools.” American Journal of Agricultural Economics 96(2): 469488.
Nelson, J.P. 2015. “Meta-analysis: Statistical Methods.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 329356.
Nelson, J.P., and Kennedy, P.E.. 2009. “The Use (and Abuse) of Meta-analysis in Environmental and Resource Economics: An Assessment.” Environmental and Resource Economics 42(3): 345377.
Newbold, S., Simpson, R.D., Massey, D.M., Heberling, M.T., Wheeler, W., Corona, J., and Hewitt, J.. 2018a. “Benefit Transfer Challenges: Perspectives from U.S. Practitioners.” Environmental and Resource Economics 69(3): 467482.
Newbold, S., Walsh, P.J., Massey, D.M., and Hewitt, J.. 2018b. “Using Structural Restrictions to Achieve Theoretical Consistency in Benefit Transfers.” Environmental and Resource Economics 69(3): 529554.
Olander, L.P., Johnston, R.J., Tallis, H., Kagan, J., Maguire, L.A., Polasky, S., Urban, D., Boyd, J., Wainger, L., and Palmer, M.. 2018. “Benefit Relevant Indicators: Ecosystem Services Measures That Link Ecological and Social Outcomes.” Ecological Indicators 85: 12621272.
Olander, L., Polasky, S., Kagan, J.S., Johnston, R.J., Wainger, L., Saah, D., Maguire, L., Boyd, J., and Yoskowitz, D.. 2017. “So You Want Your Research to Be Relevant? Building the Bridge between Ecosystem Services Research and Practice.” Ecosystem Services 26A: 170182.
Plummer, M.L. 2009. “Assessing Benefit Transfer for the Valuation of Ecosystem Services.” Frontiers in Ecology and the Environment 7(1): 3845.
Richardson, L., Loomis, J., Kroeger, T., and Casey, F.. 2015. “The Role of Benefit Transfer in Ecosystem Service Valuation.” Ecological Economics 115: 5158.
Rolfe, J., Brouwer, R., and Johnston, R.J.. 2015. “Meta-analysis: Rationale, Issues and Applications.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 357381.
Rosenberger, R.S. 2015. “Benefit Transfer Validity and Reliability.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 307326.
Rosenberger, R.S., and Johnston, R.J.. 2009. “Selection Effects in Meta-analysis and Benefit Transfer: Avoiding Unintended Consequences.” Land Economics 85(3): 410428.
Rosenberger, R.S., and Loomis, J.B.. 2017. “Benefit Transfer.” In Champ, P.A., Boyle, K.J., and Brown, T.C., eds. A Primer on Nonmarket Valuation (2nd edition). Dordrecht, the Netherlands: Springer, pp. 431462.
Rosenberger, R.S., and Phipps, T.T.. 2007. “Correspondence and Convergence in Benefit Transfer Accuracy: Meta-analytic Review of the Literature.” In Navrud, S. and Ready, R., eds. Environmental Value Transfer: Issues and Methods. Dordrecht, the Netherlands: Springer, pp. 2343.
Schaafsma, M. 2015. “Spatial and Geographical Aspects of Benefit Transfer.” In Johnston, R.J., Rolfe, J., Rosenberger, R.S., and Brouwer, R., eds. Benefit Transfer of Environmental and Resource Values: A Guide for Researchers and Practitioners. Dordrecht, the Netherlands: Springer, pp. 421439.
Schultz, E.T., Johnston, R.J., Segerson, K., and Besedin, E.Y.. 2012. “Integrating Ecology and Economics for Restoration: Using Ecological Indicators in Valuation of Ecosystem Services.” Restoration Ecology 20(3): 304310.
Smith, V.K. 2018. “Benefits Transfer: Current Practice and Prospects.” Environmental and Resource Economics 69(3): 449466.
Smith, V.K., Houtven, G. Van, and Pattanayak, S.K.. 2002. “Benefit Transfer via Preference Calibration: ‘Prudential Algebra’ for Policy.” Land Economics 78(1): 132152.
Smith, V.K., and Pattanayak, S.K.. 2002. “Is Meta-analysis a Noah's Ark for Non-market Valuation?Environmental and Resource Economics 22(1–2): 271296.
Smith, V.K., Pattanayak, S.K., and Houtven, G. Van. 2006. “Structural Benefit Transfer: An Example Using VSL Estimates.” Ecological Economics 60(2): 361371.
Stanley, T.D. 2005. “Beyond Publication Bias.” Journal of Economic Surveys 19(3): 309345.
Stanley, T.D., and Doucouliagos, H.. 2012. Meta-regression Analysis in Economics and Business. New York: Routledge.
Stanley, T.D., Doucouliagos, H., Giles, M., Heckemeyer, J.H., Johnston, R.J., Laroche, P., Nelson, J.P., et al. 2013. “Meta-analysis of Economics Research Reporting Guidelines.” Journal of Economic Surveys 27(2): 390394.
Stapler, R.W., and Johnston, R.J.. 2009. “Meta-analysis, Benefit Transfer, and Methodological Covariates: Implications for Transfer Error.” Environmental and Resource Economics 42(2): 227246.
Sutherland, R.J., and Walsh, R.G.. 1985. “Effect of Distance on the Preservation Value of Water Quality.” Land Economics 61(3): 282290.
U.S. Environmental Protection Agency (U.S. EPA). 2009a. Environmental Impact and Benefits Assessment for Final Effluent Guidelines and Standards for the Construction and Development Category. EPA-821-R-09-012, Office of Science and Technology, Office of Water, U.S. EPA, Washington, DC.
U.S. Environmental Protection Agency (U.S. EPA). 2009b. “Nonmarket Benefits from Water Quality Improvements.” In Environmental Impact and Benefits Assessment for Final Effluent Guidelines and Standards for the Construction and Development Category. EPA-821-R-09-012, U.S. EPA, Washington, DC, pp. 10-110-27.
U.S. Environmental Protection Agency (U.S. EPA). 2010. Economic Analysis of Final Water Quality Standards for Nutrients for Lakes and Flowing Waters in Florida. Office of Science and Technology, Office of Water, U.S. EPA, Washington, DC.
U.S. Environmental Protection Agency (U.S. EPA). 2012. Economic Analysis of Proposed Water Quality Standards for the State of Florida's Estuaries, Coastal Waters, and South Florida Inland Flowing Waters. Office of Science and Technology, Office of Water, U.S. EPA, Washington, DC.
U.S. Environmental Protection Agency (U.S. EPA). 2015. Benefit and Cost Analysis for the Proposed Effluent Limitations Guidelines and Standards for the Steam Electric Power Generating Point Source Category. EPA-821-R-15-005, Office of Science and Technology, Office of Water, U.S. EPA, Washington, DC.
Van Houtven, G., Mansfield, C., Phaneuf, D.J., von Haefen, R., Milstead, B., Kenney, M.A., and Reckhow, K.H.. 2014. “Combining Expert Elicitation and Stated Preference Methods to Value Ecosystem Services from Improved Lake Water Quality.” Ecological Economics 99: 4052
Van Houtven, G., Powers, J., and Pattanayak, S.K.. 2007. “Valuing Water Quality Improvements in the United States Using Meta-analysis: Is the Glass Half-Full or Half-Empty for National Policy Analysis?Resource and Energy Economics 29(3): 206228.
Walsh, J.P., and Wheeler, W.J.. 2013. “Water Quality Indices and Benefit-Cost Analysis.” Journal of Benefit-Cost Analysis 4(1): 81105.
Zandersen, M., and Tol, R.S.J.. 2009. “A Meta-analysis of Forest Recreation Values in Europe.” Journal of Forest Economics 15(1–2): 109130.