Skip to main content Accessibility help
×
Home

Information:

  • Access
  • Cited by 46

Figures:

Actions:

      • Send article to Kindle

        To send this article to your Kindle, first ensure no-reply@cambridge.org is added to your Approved Personal Document E-mail List under your Personal Document Settings on the Manage Your Content and Devices page of your Amazon account. Then enter the ‘name’ part of your Kindle email address below. Find out more about sending to your Kindle. Find out more about sending to your Kindle.

        Note you can select to send to either the @free.kindle.com or @kindle.com variations. ‘@free.kindle.com’ emails are free but can only be sent to your device when it is connected to wi-fi. ‘@kindle.com’ emails can be delivered even when you are not connected to wi-fi, but note that service fees apply.

        Find out more about the Kindle Personal Document Service.

        Outcomes, not implementation, predict conservation success
        Available formats
        ×

        Send article to Dropbox

        To send this article to your Dropbox account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Dropbox.

        Outcomes, not implementation, predict conservation success
        Available formats
        ×

        Send article to Google Drive

        To send this article to your Google Drive account, please select one or more formats and confirm that you agree to abide by our usage policies. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your <service> account. Find out more about sending content to Google Drive.

        Outcomes, not implementation, predict conservation success
        Available formats
        ×
Export citation

Abstract

To use more effectively the limited resources available for conservation there is an urgent need to identify which conservation approaches are most likely to succeed. However, measuring conservation success is often difficult, as it is achieved outside the project time frame. Measures of implementation are often reported to donors to demonstrate achievement but it is unclear whether they really predict conservation success. We applied a conceptual framework and score-card developed by the Cambridge Conservation Forum (CCF) to a sample of 60 conservation activities to determine the predictive power of implementation measures versus measures of key outcomes (later steps in the models defined in the CCF tools). We show that assessing key outcomes is often more difficult than quantifying the degree of implementation of a project but that, while implementation is a poor predictor of success, key outcomes provide a feasible and much more reliable proxy for whether a project will deliver real conservation benefits. The CCF framework and evaluation tool provide a powerful basis for synthesizing past experience and, with wider application, will help to identify factors that affect the success of conservation activities.

Introduction

Given the limited resources available for conservation it is imperative that funding is channelled towards those approaches that can be proven to be successful (Sutherland et al., 2004; Ferraro & Pattanayak, 2006). Considerable effort has recently been devoted to devising frameworks that help to monitor the progress and success of conservation projects (Sutherland et al., 2004; Hockings et al., 2006; Pullin & Stewart, 2006; Mace et al., 2007; Kapos et al., 2008; Salafsky et al., 2008) but an issue that still hampers these attempts is the fact that the effects of many conservation projects on target populations or habitats only become measurable well beyond the time frame of the usual project cycle (Kapos et al., 2008). Long-term monitoring of targeted species or habitats would be desirable but is often not practical. An alternative approach is to identify predictors of success that are both easier to measure and serve as reliable surrogates for long-term data on the changing status of conservation targets.

The Cambridge Conservation Forum (CCF), a network of conservation organizations and researchers based in and around Cambridge, UK (CCF, 2009), has developed a project evaluation tool (Kapos et al., 2008). This draws on a conceptual framework (Kapos et al., 2008) that is itself an elaboration of the idea of results chains for conservation interventions (Salafsky et al., 2001, 2008). Together the framework and evaluation tool enable us to investigate the link between intermediate steps (such as activities, outputs and outcomes) and the ultimate success (conservation effects) of a project. Here we use a preliminary dataset to test whether measures of project implementation (as typically reported to donors) are useful predictors of conservation effects, or whether measuring subsequent outcomes is a better alternative in terms of how easily the measures can be derived and how well they predict longer-term effects on conservation targets.

Methods

CCF framework and evaluation tool

The CCF framework and evaluation tool begin with the recognition that conservation projects characteristically involve several different types of activity that may each have different outcomes and appropriate measures of success. Based on existing categorizations of conservation action (Salafsky et al., 2002, 2008) and as a result of consultation within CCF, we defined seven broad categories of conservation activity (Table 1) that together encompass most of the work that CCF members and other conservation organizations undertake. Typically conservation projects include several of these activity types (Wilder & Walpole, 2008; Kapos et al., in press).

Table 1 The seven broad categories of conservation activity adopted by CCF as the basis for its evaluation framework. Together these encompass most of the work that CCF members and other conservation organizations undertake.

For each activity type the CCF framework provides a conceptual model of the likely relationships between its successful implementation in a project and conservation impact, making explicit the linkages that are often assumed (Kapos et al., 2008). The models are simplest for species and site management (Fig. 1) and more complex for the other five activity types (e.g. Figs 2 & 3). All models include implementation stages, leading on through a series of outcomes (how an intervention affects the conservation problem of interest), to conservation effect (project-scale effects on the likelihood of persistence of populations or habitats of conservation concern). In all but the two simplest models we identify key outcomes that provide the platform for reducing threats to and/or improving the responses of conservation targets to those threats. For livelihoods-related projects the key outcome is the abandonment of the relevant damaging practices; for policy work, it is the implementation of the policies or legislation promoted; for education and awareness raising, a change in behaviour by the intended audience; for capacity-building, increases in the quantity and/or quality of conservation action; and for research, the application of research results to conservation practice.

Fig. 1 The conceptual models for species and site management that form the framework for evaluating the success of conservation actions are directly analogous and have the simplest structure, with implementation of management actions leading directly to reduction of threats to the conservation target and/or an improvement in its responses to those threats (outcomes), and a consequent conservation effect. The components corresponding to implementation, outcomes and effects are separated by horizontal lines.

Fig. 2 The conceptual model for enhancing and/or providing alternative livelihoods, in which the links between implementation and conservation effect are more complex, involving more intermediate outcomes: livelihoods-related activities aim to develop sustainable management regimes for important natural resources, encourage the development of alternative sources of income, and/or provide direct incentives. If adequate support for the targeted practices exists or is developed, and the necessary conditions and resources for their implementation are in place, they can bring about a change in practice. Finally, if the new practices have been appropriately designed this may lead to reduction in threats and/or improved responses of the focal ecosystems, habitats, species and/or populations, which may ultimately lead to a conservation effect (a change in status). This may or may not be accompanied by real improvements in livelihoods and/or attitudes but these will not affect conservation status, except by affecting the targeted behaviours or practices. Each of these more complex models includes a key outcome (in this case, change in practice, marked with a star) on which threat reduction and/or improvement of responses depends. The components corresponding to implementation, outcomes and effects are separated by horizontal lines. The hexagons show indicative (but not exhaustive) links to other activity types.

Fig. 3 The CCF framework’s model for capacity building efforts, which aim to improve either individuals’ skills or aspects of organizational capacity, or both. If those improvements are in fact applied in conservation, they should lead to more and/or better conservation action to address the problems of interest (the key outcome), leading through reduced threat and/or improved responses to improved status. The components corresponding to implementation, outcomes and effects are separated by horizontal lines. The remaining models in the CCF framework (Table 1) show analogous sequences (Kapos et al., 2008; CCF Outputs, 2009).

The score-card–style evaluation tool developed by CCF has a single questionnaire for each conservation activity type. Each questionnaire is based on the appropriate model and comprised of carefully worded questions that work progressively through implementation and outputs to outcomes and conservation effects (CCF Outputs, 2009). Each question offers four ordered answers reflecting increasing levels of achievement, which can be scored on a 1–4 scale. There are also further unranked options for when insufficient information is available; these are worded to distinguish between cases where relevant information is anticipated and those where information is not being collected at all.

As an example, the question about the key outcome in the questionnaire on livelihoods-related activities is: ‘Has the target audience reduced their use of the damaging practices addressed by the project?’ and the answers offered are: (a) No; (b) Some individuals have reduced their use; (c) Most individuals have reduced their use and/or some have abandoned them entirely; (d) Most individuals have entirely abandoned damaging practices; (e) Too early to tell but information will be available; (f) Not assessed, necessary information neither available nor anticipated. Answers a–d reflect increasing reduction of damaging practices and e and f deal with missing information.

Trial application

To test which type of measures (implementation vs outcome) offers the most practical approach for assessing conservation performance (in terms of how easily they can be derived and how effectively they predict long-term effects), the CCF questionnaires were used by conservation professionals to self-evaluate a set of 60 activities from 26 projects. A few of these individuals were involved in the development of the framework and questionnaire but the majority were not. Their answers to the questionnaire were based on the best available evidence, which ranged from the experience and opinion of project managers, often supported by anecdotal evidence, to quantitative data (e.g. from surveys of agricultural practice and statistics on trade).

We used the completed questionnaires firstly to compare how easy it was to answer questions about implementation, outcome and conservation effect. Secondly, for each evaluated activity, we calculated a mean score for the answers to questions at each of these three levels, restricting the analysis of the outcome level to the section on key outcomes (as defined above). We then used these average scores to test the extent to which measures at the two lower levels (implementation and outcome) were good predictors of the activity's conservation effect. This analysis reduced the sample to the 22 activities for which full information was available for all three levels. However, this reduction in sample size did not introduce bias to the sample: projects with full information did not differ from incomplete projects in their scores for implementation or outcome (medians (ranges) for implementation 3.3 (2.8–3.5) vs 3.5 (3.0–4.0), Wilcoxon test W = 452.5, n = 22 vs 37, P = 0.473; for outcome 3.0 (2.0–3.4) vs 2.5 (2.0–3.0), W = 91.0, n = 22 vs 11, P = 0.245).

Results

As expected, questions were less likely to be answered the further they were along any given model (and thus closer to conservation effect; see Fig. 4 for a detailed example from capacity-building). This was the case irrespective of the type of conservation activity being undertaken (Fig. 5). Questions on key outcomes, despite being less readily answered than those about implementation, were still much easier to answer than those on actual conservation effects.

Fig. 4 The frequency among 12 completed capacity building questionnaires of substantive responses (i.e. a–d, which reflect the level of achievement rather than lack of information) to key questions. The key outcome, in this case increased quantity and/or quality of conservation action, is marked with a star (as in Fig. 3). This pattern is consistent to a greater or lesser extent across activity types.

Fig. 5 Variation in frequency of substantive responses (i.e. a–d, which reflect the level of achievement rather than lack of information) at different points within the CCF framework. Across different activity types the occurrence of substantive responses to key questions regarding implementation (stippled bars) is always higher than for questions about key outcomes (light grey bars), which usually have more substantive responses than questions about conservation effect (dark bars).

With regard to what best predicts conservation effects, we found that the degree of implementation of a given activity was a poor predictor of its conservation effect (Spearman rank correlation: ρ = 0.077, n = 22, P = 0.734; Fig. 6a). In contrast, the extent of achievement of key outcomes was a much more powerful predictor of the effect of an intervention, even in this small sample of activities (ρ = 0.668, n = 22, P < 0.001; Fig. 6b). This result is not linked to any tendency in the methodology to promote linkage between adjacent levels; we tested for a close link between implementation and outcome scores and found none (ρ =0.060, P = 0.792).

Fig. 6 The relationship between the overall conservation effects of activities in 22 conservation projects and (a) the level of their implementation and (b) the degree to which key outcomes are achieved. Answers from the relevant questions have been given simple numerical values for plotting, with 1 being the lowest level of achievement and 4 being the highest. Symbols are sized according to the number of overlapping points, up to a maximum of four.

Discussion

Our results show that measures of key outcomes can potentially serve as powerful predictors of real conservation effects, and thus project success. While assessing outcomes is more difficult than simply reporting on implementation, measures of implementation were unable to predict success and are thus of little value in tracking conservation impacts. This is important, given that most donors have until now typically only required reporting at the level of project implementation. In contrast, because outcomes are influenced by some of the many external factors that affect the linkages between activity and impact, they provide a more reliable basis for assessing project success. Although in future it would be desirable to validate fully this conclusion using a larger sample size, targeting project monitoring and reporting efforts and resources more strongly towards outcome-level measures is likely to improve the assessment of conservation achievements.

The CCF framework and evaluation tool provide a standardized approach to improving on current reporting practice through the identification and assessment of outcomes for different activity types. The CCF tools build on and complement existing approaches for using conservation experience to inform future conservation actions, including The Nature Conservancy's 5-S Framework (TNC, 2003), logical frameworks and results chains (Salafsky et al., 2001) and tools for implementing them, such as Miradi (Miradi, 2007), conservation audits based on the CMP's Open Standards (CMP, 2004; O'Neil, 2007), threat reduction assessment (Salafsky & Margoluis, 1999), and the scoring and evaluation frameworks developed for assessing agri-environment schemes (Carey et al., 2003, 2005), protected area management effectiveness (Hockings et al., 2006; Stolton et al., 2008), and interventions by zoos (Mace et al., 2007). The CCF tools also complement the growing drive for evidence-based conservation (Sutherland et al., 2004; Sutherland, 2005) by providing both a means to assess the effects of whole projects as distinct from single management interventions and a way of helping to ensure that the data needed for evidence-based conservation are collected.

So far, practitioners who have used the CCF tools report taking no more than a day to assess a project, making it difficult to argue that it is too time-consuming to conduct such assessments (given that projects normally represent several person-years of effort). Although further validation of our conclusion about the usefulness of assessing key outcomes is desirable, we believe that our results provide clear evidence that such measures have the potential to allow practitioners to predict the downstream effects of their projects, even when these will only be realized beyond the project timeframe.

The dataset presented in this paper is relatively small and it was not possible to test the robustness of our general conclusion for each activity type independently. The dataset is also relatively lacking in unsuccessful projects, mirroring the more general patterns in the availability of information about conservation actions that fail (Redford & Taber, 2000). A larger database of assessed projects would not only allow us to address this issue but also to investigate several other interesting questions, such as whether the success of different activity types depends on particular conditions, which combinations of activities provide the best synergies, and which resource allocation strategies are most successful. We therefore strongly encourage practitioners to take advantage of the CCF tools, which are freely available online (CCF Outputs, 2009). Results presented here and feedback from practitioners who have trialled the tool already suggest this will help organizations plan interventions, monitor their progress, and obtain meaningful, early measures of their long-term effectiveness. Expanding the database of evaluated projects would also help us build a much-needed resource with which to start asking precise and quantitative questions about what determines conservation success.

Acknowledgements

This work was supported by a grant from the John D. and Catherine T. MacArthur Foundation to the Cambridge Conservation Forum via the University of Cambridge. The member organizations of the Cambridge Conservation Forum (see http://www.cambridgeconservationforum.org.uk) generously supported the participation of many of their staff members in the working groups, workshops and discussion events of this project. Members of the Conservation Measures Partnership and the IUCN Programme Evaluation Group provided helpful input. Earthwatch and the BAT Biodiversity Partnership also helped with testing and refining the evaluation tool. The University of Cambridge Department of Zoology and UNEP-WCMC were especially helpful in accommodating project staff and many project meetings. We are grateful to the following individuals, who participated in the project's working groups and discussions and in testing the evaluation tool: W. Adams, M. Aminu-Kano, M. Ausden, E. Ball, S. Barnard, A. Bowley, E. Bowen-Jones, R. Brett, M. Brooke, P. Brotherton, P. Buckley, N. Bystriakova, D. Coomes, B. Cooper, B. Dickson, J. Doberski, E. van Ek, J. Ekstrom, L. Evans, D. Gibbons, M. Green, R. Green, A. Grigg, D. Hawkins, M. Harris, P. Herkenrath, A. Hipkiss, G. Hirons, R. Hossain, F. Hughes, J. Hughes, J. Hutton, C. Ituarte, D. Kingma, P. Laird, A. Lanjouw, P. Lee, C. Magin, T. Milliken, R. Mitchell, D. Noble, S. O'Connor, T. Oldfield, K. O'Regan, E. Papworth, A. Rodrigues, R. Sinegar, P. Stromberg, W. Sutherland, D. Thomas, R. Trevelyan, G. Tucker, S. Wells, J. Williams and M. Wright. We thank two anonymous referees for helpful comments on an earlier draft of this article.

References

Carey, P.D., Manchester, S.J. & Firbank, L.G. (2005) Performance of two agri-environment schemes in England: a comparison of ecological and multi-disciplinary evaluations. Agriculture Ecosystems and Environment, 108, 178188.
Carey, P.D., Short, S., Morris, C., Hunt, J., Priscott, A., Davis, M. et al. (2003) The multi-disciplinary evaluation of a national agri-environment scheme. Journal of Environmental Management, 69, 7191.
CCF (Cambridge Conservation Forum) (2009) Http://www.cambridgeconservationforum.org.uk [accessed 30 April 2009].
CCF Outputs (2009) Http://www.cambridgeconservationforum.org.uk/measures_outputs.htm [accessed 30 April 2009].
CMP (Conservation Measures Partnership) (2004) Open Standards for the Practice of Conservation. Conservation Measures Partnership, Washington, DC, USA.
Ferraro, P.J. & Pattanayak, S.K. (2006) Money for nothing? A call for empirical evaluation of biodiversity conservation investments. PLoS Biology, 4, e105.
Hockings, M., Stolton, S., Leverington, F., Dudley, N. & Courrau, J. (2006) Evaluating Effectiveness: A Framework for Assessing Management Effectiveness of Protected Areas, 2nd edition. IUCN World Commission on Protected Areas and James Cook University, Brisbane, Australia.
Kapos, V., Balmford, A., Aveling, R., Bubb, P., Carey, P., Entwistle, A. et al. (2008) Calibrating conservation: new tools for measuring success. Conservation Letters, 1, 155164.
Kapos, V., Manica, A., Aveling, R., Bubb, P., Carey, P., Entwistle, A. et al. (in press) Defining and measuring success in conservation. In Trade Offs in Conservation (eds Leader-Williams, N. & Smith, R.). Blackwell, Oxford, UK.
Mace, G., Balmford, A., Leader-Williams, N., Manica, A., Walter, O., West, C.D. & Zimmermann, A. (2007) Measuring conservation success: assessing zoos' contribution. In Zoos in the 21st Century: Catalysts for Conservation? (eds Zimmermann, A., Hatchwell, M., Dickie, L.A. & West, C.), pp. 322342. Cambridge University Press, Cambridge, UK.
Miradi, (2007) Miradi: Adaptive Management Software for Conservation Projects. Conservation Measures Partnership, Bethesda, USA & Benetech, Palo Alto, USA.
O'Neil, E. (2007) Conservation Audits: Auditing the Conservation Process—Lessons Learned, 2003–2007. Conservation Measures Partnership, Bethesda, USA.
Pullin, A.S. & Stewart, G.B. (2006) Guidelines for systematic review in conservation and environmental management. Conservation Biology, 20, 16471656.
Redford, K.H. & Taber, A. (2000) Writing the wrongs: developing a safe-fail culture in conservation. Conservation Biology, 14, 15671568.
Salafsky, N. & Margoluis, R. (1999) Threat reduction assessment: a practical and cost-effective approach to evaluating conservation and development projects. Conservation Biology, 13, 830841.
Salafsky, N., Margoluis, R. & Redford, K. (2001) Adaptive Management: A Tool for Conservation Practitioners. Biodiversity Support Program, Washington, DC, USA.
Salafsky, N., Margoluis, R., Redford, K.H. & Robinson, J.G. (2002) Improving the practice of conservation: a conceptual framework and research agenda for conservation science. Conservation Biology, 16, 14691479.
Salafsky, N., Salzer, D., Stattersfield, A.J., Hilton-Taylor, C., Neugarten, R., Butchart, S.H.M. et al. (2008) A standard lexicon for biodiversity conservation: unified classifications of threats and actions. Conservation Biology, 22, 897911.
Stolton, S., Hockings, M., Dudley, N., MacKinnon, K., Whitten, T. & Leverington, F. (2008) Management Effectiveness Tracking Tool—Reporting Progress at Protected Area Sites, 2nd edition. WWF, Gland, Switzerland.
Sutherland, W.J. (2005) How can we make conservation more effective? Oryx, 39, 12.
Sutherland, W.J., Pullin, A.S., Dolman, P.M. & Knight, T.M. (2004) The need for evidence-based conservation. Trends in Ecology & Evolution, 19, 305308.
TNC (The Nature Conservancy) (2003) The Five-S Framework for Site Conservation: A Practitioner's Handbook for Site Conservation Planning and Measuring Conservation Success. The Nature Conservancy, Arlington, USA.
Wilder, L. & Walpole, M. (2008) Disentangling the links between conservation and poverty reduction in practice. Oryx, 42, 539547.

Biographical sketches

The authors, who made up the steering committee for the Cambridge Conservation Forum (CCF) collaborative project on Harmonizing Measures of Conservation Success, all work in conservation research and/or practice with organizations that are members of CCF. CCF exists to strengthen links and develop new synergies across the diverse community of conservation practitioners and researchers based in and around Cambridge, UK, working at local, national and international levels, and is currently comprised of more than 40 member organizations.